Possible bad memory management in Fortran-C interfacing

[mEm]

Hi everyone,

Basically, I want to call a Fortran procedure from a C function.

Since the actual code would be quite “messy”, I reproduced the situation in a much readable format.

Fortran

! ------------------------------------------------------
!    PURE FORTRAN CORE
! ------------------------------------------------------
module mod_internal_
   implicit none
contains
   function actualProc(f, n) result(res)
      integer(kind = 4), intent(in) :: n
      real(kind = 8), intent(in)    :: f(n)
      real(kind = 8) :: res(n)
      integer :: i
      do i = 1, n
         res(i) = f(i) * n
      enddo
   end function
end module

module mod2
   use mod_internal_
   implicit none
   private
   public :: actualFctWrapper 
contains
   function actualFctWrapper(f, n) result(res)
      integer(kind = 4), intent(in) :: n
      real(kind = 8), intent(in)    :: f(n)
      real(kind = 8), allocatable, target :: res(:)

      res = actualProc(f, n)
   end function
end module

! ------------------------------------------------------
!    FORTRAN-C interface
! ------------------------------------------------------
module mod_cintf
   use iso_c_binding
   implicit none
   private :: evalFct_c_
   procedure(fctintf), private, pointer :: fptr_int_ => null()
   abstract interface
      function fctintf(f, n) result(res)
         integer(kind = 4), intent(in) :: n
         real(kind = 8), intent(in)    :: f(n)
         real(kind = 8), allocatable, target :: res(:)
      end function
   end interface

   interface
      subroutine acquireEvalFctPtr(fct) bind(c, name="acquireEvalFctPtr")
         import c_funptr
         type(c_funptr), value :: fct
      end subroutine
   end interface

contains
   
   subroutine evalFct_c_(f, n, res)
      real(c_double), intent(in), target :: f
      integer(c_int), intent(in) :: n
      real(kind = 8), allocatable, target :: res_(:)
      type(c_ptr), intent(inout) :: res

      integer(kind = 4) :: n_, i
      real(kind = 8), pointer :: f_(:)

      n_  = int(n, kind=4)
      call c_f_pointer(c_loc(f), f_, [n_])

      res_ = fptr_int_(f_, n_)
      print *, " Evaluation from within Fortran:"
      do i = 1, n_
         write(*, fmt='(5x, f12.5)', advance='no') res_(i)
      enddo
      print *
      res  = c_loc(res_)
   end subroutine

   subroutine acquireEvalFct(fct)
      procedure(fctintf), pointer, intent(in) :: fct

      fptr_int_ => fct
      call acquireEvalFctPtr(c_funloc(evalFct_c_))
   end subroutine
end module


! ------------------------------------------------------
!    main
! ------------------------------------------------------
program test
   use mod2
   use mod_cintf
   implicit none

   call acquireEvalFct(actualFctWrapper)
end program test

C

#include <stdio.h>

void (*__fptr)(double**, int*, double**);

void acquireEvalFctPtr(void (*fct)(double**, int*, double**)) {
   double _rv[5] = {1.f, 2.f, 5.f, 4.f, 5.f};
   int    _ival  = 2;
   double *_res;
   
   __fptr = fct;

   // now call function pointer (implementation comes from Fortran)
   __fptr((double **) &_rv, &_ival, &_res);
   printf("\n  Evaluation result is: \n");
   for (unsigned i = 0; i < _ival; ++i)
      printf("     %12.5f", _res[i]);
}

Issue

Now, the actual memory allocation happens in the Fortran procedure mod_internal_ actualProc().
This procedure is externally exposed via a wrapper actualFctWrapper(), which is there for setting up before actually calling it (in reality interfaces differ, that’s why there exist the wrapper, to ease/automate its call). In this latter procedure, the function result is an allocatable because it is deferred shape.
When calling the set function pointer in C __fptr((double **) &_rv, &_ival, &_res), this get to the Fortran_C interface procedure evalFct_c_(), which internally calls the internal function pointer which has been previously set to the actual (wrapper) evaluating function.

Now, the issue: how to free this allocated memory?
I put a free() at the end of the C code, but in the real code base I get an exception Invalid address specified to RtlValidateHeap( 000001C0F5B70000, 000001C0F76A7E50 ), while this does not happen in the example provided here. Cannot judge which one is the real faulty one…

Also, another concern I had is that res_ in evalFct_c_() should be implicitly deallocated at exit. So, I wonder if this might cause, at some later point some access violation or similar. Or that memory might be reused, overriding its older values, invalidating the use of the result _res from an ancient call from the C side __fptr((double **) &_rv, &_ival, &_res);.

Hope I was clear enough.

Thanks to everyone answering.

[ivanpribec]

@mEm

Also, another concern I had is that res_ in evalFct_c_() should be implicitly deallocated at exit. So, I wonder if this might cause, at some later point some access violation or similar. Or that memory might be reused, overriding its older values, invalidating the use of the result _res from an ancient call from the C side __fptr((double **) &_rv, &_ival, &_res);.

Yes. The allocatable res_ will be deallocated when evalFct_c_ exits, rendering the pointer res invalid. You will need to use a pointer array to make the memory persist after the Fortran function is called, and deallocate it manually afterwards, with a routine such as:

subroutine f_dealloc(ptr) bind(c)
  type(c_ptr), value :: ptr
  real(8), pointer :: res(:)
  if (c_associated(ptr)) then
     call c_f_pointer(ptr,res)
     deallocate(res)
  end if
end subroutine

Btw, can’t the size of the array returned from the Fortran routine be determined in advance? That would make things much simpler. I also don’t understand the need for the global function pointer __fptr; can’t you just call fct(...) directly? As a side-note, some categories of names with leading underscores are reserved by the C language.

[mEm]

Thanks @ivanpribec .

@ivanpribec

You will need to use a pointer array to make the memory persist after the Fortran function is called, and deallocate it manually afterwards, with a routine such as:

So you basically say to change from real(kind = 8), allocatable, target :: res_(:) to real(kind = 8), pointer :: res_(:) in evalFct_c_, and later call f_dealloc() on the C side ? Did I understand correctly ?

@ivanpribec

Btw, can’t the size of the array returned from the Fortran routine be determined in advance?

It could, it would need some few changes, but definitely possible. I think I will finally consider implementing it.

@ivanpribec

I also don’t understand the need for the global function pointer __fptr; can’t you just call fct(...) directly?

Do you mean on the C side?

@ivanpribec

As a side-note, some categories of names with leading underscores are reserved by the C language.

My bad, thanks for the suggestion. I will move them after.

[FortranFan]

@mEm

… I want to call a Fortran procedure from a C function …

@mEm,

You show a lot of complicated code but it’s unclear what exactly you are trying to do: what exactly is the Fortran procedure you want to invoke in a C function?

It will also help if you can explain why you want to “call a Fortran procedure from a C function,” meaning is the calling program (and the memory management) in C, or is C simply some callback library or some such? It will be useful if you can confirm there is some good reason the code cannot be all Fortran because otherwise that can be a preferable design.

[mEm]

@FortranFan ,

there are basically two cores, the main in Fortran (where memory allocation basically happen), the “secondary” in C.
The C one is needed since it is the one integrating some OpenCL GPU parallelisation. Because of this, the C core, even though implementing some unrelated stuff, depends on the main Fortran one because it needs some base algorithmic infomation regarding data and their dimensions. Everything is passed via pointers (in the sense, no memory copies are neeed since this would be read-only data on the C side).
Now, at some point I realised I needed to “move” a tiny part of the Fortran core to the C one. Namely, this evaluation function, which by itself is an internal function pointer (what there I call actualProc()) to an external procedure provided by the user (via a dedicated API call). So, instead of “rewriting” this function pointer call in C (which would require definition of some structs etc., I wanted to be able to call it from C via a wrapper, using the internal Fortran data to set up the right function call.
The only problem is that at the time I developed all this, I did not have in mind this possible extension, otherwise I would have written this part already in C…

[FortranFan]

@mEm

… there are basically two cores, the main in Fortran (where memory allocation basically happen), the “secondary” in C. …

Thanks @mEm for the explanation. So a quick question first: given your scenario, why not work with C interoperable types and kinds everywhere in your Fortran code e.g., use c_double kind defined as named constant in iso_c_binding and work with real(c_double) everywhere. Why deal with a “magic number” of real(kind = 8) in your code?

[mEm]

@FortranFan

So a quick question first: given your scenario, why not work with C interoperable types and kinds everywhere in your Fortran code e.g., use c_double kind defined as named constant in iso_c_binding and work with real(c_double) everywhere

This remark hits me differently, since from the beginning I felt I had (and somewhat wanted to) develop (some of) the Fortran core in C (and than call it from Fortran). However, this had originated to be directly linked to the code base of the office I was working on, which is in a mix of old and modern Fortran. And they wanted me to do it in Fortran since it was the only language almost all of them was capable of understanding.
In the meantime, things evolved quite fast, and I am now realising that those constraints are now affecting further developments.
I am trying to do it as best as possible in order to avoid having to re-write big chunks of code.

@FortranFan

Why deal with a “magic number” of real(kind = 8) in your code?

I put that for showing purposes only, but I normally use the iso_fortran_env module from which I get the desired kinds. Will pay attention to that in any case, thanks!

[ivanpribec]

Design issues aside, you probably want to add bind(c) on the evaluation function, which is passed to C.

[ivanpribec]

Here’s a slightly simplified organization of more or less the same functionality, but without any pointers on the Fortran side:

// eval.c
#include <stdio.h>
#include <stdlib.h>

// Callback function with dynamic return size
// 
// 1) Call with lres = -1 to perform a result size query
// 2) Call the function with res allocated to the size returned in step 1
//
typedef void (*Fcn)(
    const double * /* f */, 
    int /* n */, 
    double * /* res */,
    int * /* lres */);

#define N 5
void eval(Fcn f)
{
    double rv[N] = {1., 2., 3., 4., 5.}; // No `f` suffix for double
    double *res = NULL;
    int lres = -1;

    // 1) Result size query and allocation
    f(rv,N,res,&lres);
    res = malloc(lres * sizeof(double));
    
    // 2) Evaluate result
    f(rv,N,res,&lres);

    // Do more C stuff
    puts("\n  Evaluation result is:");
    for (int i = 0; i < lres; i++)
        printf("     %12.5f", res[i]);
    puts("");

    // Free memory
    free(res);
}
#undef N

! eval_fortran.f90

module mod_internal_
   use, intrinsic :: iso_c_binding, only: c_int, c_double
   implicit none
   private

   public :: ip, wp, actualProc

   integer, parameter :: ip = c_int
   integer, parameter :: wp = c_double

contains
   function actualProc(f, n) result(res)
      integer(ip), intent(in) :: n
      real(wp), intent(in)    :: f(n)
      real(wp) :: res(n)
      integer :: i
      do i = 1, n
         res(i) = f(i) * n
      enddo
   end function
end module

module mod2
   use mod_internal_
   implicit none
   private
   public :: actualFct
contains
   ! A convenience wrapper which returns an allocatable array
   function actualFct(f, n) result(res)
      integer(ip), intent(in) :: n
      real(wp), intent(in)    :: f(n)
      real(wp), allocatable, target :: res(:)
      res = actualProc(f, n)
   end function
end module

module c_mod
   use, intrinsic :: iso_c_binding, only: c_int, c_double
   implicit none
   public

   abstract interface
      subroutine c_fcn(f,n,res,lres) bind(c)
         import
         integer(c_int), value :: n
         real(c_double), intent(in) :: f(n)
         real(c_double), intent(inout) :: res(*)
         integer(c_int), intent(inout) :: lres
      end subroutine
   end interface

   interface
      subroutine eval(fcn) bind(c,name="eval")
         import
         procedure(c_fcn) :: fcn
      end subroutine
   end interface

end module

program main
   use c_mod
   implicit none
   call eval(myfcn)
contains
   ! User callback
   subroutine myfcn(f,n,res,lres) bind(c)
      use mod2
      real(c_double), intent(in) :: f(n)
      integer(c_int), value :: n
      real(c_double), intent(inout) :: res(*)
      integer(c_int), intent(inout) :: lres
      if (lres == -1) then
         ! Respond to result size query
         lres = n
         return
      end if
      ! Call Fortran procedure
      res(1:lres) = actualFct(f,n)
   end subroutine
end program

~/fortran$ gcc -Wall -c eval.c
~/fortran$ gfortran -Wall eval_fortran.f90 eval.o
~/fortran$ ./a.out

  Evaluation result is:
          5.00000         10.00000         15.00000         20.00000         25.00000

[FortranFan]

@mEm

… how to free this allocated memory? …

See the work by @ivanpribec where you can use raw C memory management functions, that is your design option #1.

Alternatively, you can work with managed functions provided via enhanced interoperability with C introduced starting Fortran 2018 and manage your memory that way whilst using the commonly recommended assumed-shape received arguments in your Fortran code. like so

• Fortran

! ------------------------------------------------------
!    PURE FORTRAN CORE
! ------------------------------------------------------
module kinds_m
   use, intrinsic :: iso_c_binding, only : CI => c_int, CD => c_double
end module 
module mod_internal_
   use kinds_m, only : CI, CD
contains
   pure function actualProc( f, n ) result(res)
      import :: CI, CD
      ! Argument list
      real(kind=CD), intent(in)    :: f(:)
      integer(kind=CI), intent(in) :: n
      ! Function result
      real(kind=CD) :: res(n)
      res = n * f
   end function
end module
module mod2
   use kinds_m, only : CI, CD
   use mod_internal_, only : actualProc
contains
   subroutine actualFctWrapper(f, n, res) bind(C, name="actualFctWrapper")
      real(kind=CD), intent(in)  :: f(:)
      integer(kind=CI), intent(in), value :: n
      real(kind=CD), allocatable, intent(inout) :: res(:)
      res = actualProc( f, n )
      print *, "In actualFctWrapper: res = ", res
   end subroutine
end module
! ------------------------------------------------------
!    FORTRAN-C interface
! ------------------------------------------------------
module mod_cintf
   use kinds_m, only : CI, CD
   use mod2, only : actualFctWrapper
   interface
      subroutine C_EvalFunc(pFunc) bind(C, name="C_EvalFunc")
         import :: actualFctWrapper
         procedure(actualFctWrapper) :: pFunc
      end subroutine
      subroutine C_FreeResult() bind(C, name="C_FreeResult")
      end subroutine
   end interface
end module
! ------------------------------------------------------
!    main
! ------------------------------------------------------
program test
   use mod2, only : actualFctWrapper 
   use mod_cintf, only : C_EvalFunc, C_FreeResult
   call C_EvalFunc(actualFctWrapper)
   ! do whatever work is needed followed by "clean up"
   call C_FreeResult()
end program test     

• C

#include <stdio.h>
#include "ISO_Fortran_binding.h"

typedef void(*EvalFunc)(const CFI_cdesc_t *, int, CFI_cdesc_t *);
void C_EvalFunc( EvalFunc );
void C_FreeResult();
CFI_CDESC_T(1) f;
CFI_CDESC_T(1) res;

void C_EvalFunc( EvalFunc evalfunc ) {
   enum N { N = 5 };
   int irc;
   CFI_rank_t rank;
   CFI_index_t extent[1];
   double _rv[N] = {1.f, 2.f, 5.f, 4.f, 5.f};

   irc = 0;
   rank = 1;
   extent[0] = (CFI_index_t)N;
   irc = CFI_establish((CFI_cdesc_t *)&f, (void *)_rv,
      CFI_attribute_other,
      CFI_type_double, sizeof(double), rank, extent);
   if (irc != CFI_SUCCESS) {
      // error handling
      printf("In C_EvalFunc: CFI_establish for f failed.");
      return;
   }
   irc = CFI_establish((CFI_cdesc_t *)&res, NULL,
      CFI_attribute_allocatable,
      CFI_type_double, sizeof(double *), rank, NULL);
   if (irc != CFI_SUCCESS) {
      // error handling
      printf("In C_EvalFunc: CFI_establish for res failed.");
      return;
   }
   evalfunc((CFI_cdesc_t *)&f, (int)N, (CFI_cdesc_t *)&res);
}
void C_FreeResult() {
   // Free the CFI decriptor object for Fortran-C interoperability
   int irc = CFI_deallocate((CFI_cdesc_t *)&res);
   return;
}

C:\temp>cl /c /W3 /EHsc c.c
Microsoft (R) C/C++ Optimizing Compiler Version 19.31.31105 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

c.c

C:\temp>ifort /c /standard-semantics /free p.f
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.8.0 Build 20221119_000000
Copyright (C) 1985-2022 Intel Corporation.  All rights reserved.


C:\temp>link p.obj c.obj /subsystem:console /out:p.exe
Microsoft (R) Incremental Linker Version 14.31.31105.0
Copyright (C) Microsoft Corporation.  All rights reserved.


C:\temp>p.exe
 In actualFctWrapper: res =  5.00000000000000 10.0000000000000
 25.0000000000000 20.0000000000000 25.0000000000000

Thus you have a couple of choices here, you can be glad to read.

[mEm]

@ivanpribec @FortranFan thanks for your answers.

Between your approaches I am this time more keen to the one proposed by @ivanpribec . I find it much less verbose on the C side.

I made it finally to work using raw C allocation:

double freq__[2] = {1., 2.};
int    nfreqs__ = 2;
int    jtmp__, itc__, nnl__;
double *temp__;
...
temp__ = (double *) malloc(sizeof(double) * jtmp__);
...
extdata__.evalFunc_ptr__(&itc__, &nfreqs__, freq__, &nnl__, temp__);
...
free(temp__);

I do find strange the fact that I cannot get it right to pass those ints by value, i.e. having this C function declaration typedef void (*evalFct)(int, int, const double*, int, double*); and calling it with

extdata__.evalFunc_ptr__(itc__, nfreqs__, freq__, nnl__, temp__);

as you @ivanpribec also suggest. I get undefined addresses on the Fortran side using

subroutine evalFunc_C_to_F_wrapper_(itc, nf, f, idim, res)
      integer(c_int), value         :: itc
      integer(c_int), value         :: nf, idim
      real(c_double), intent(in), target :: f
      real(c_double), intent(inout)      :: res(*)
      real(kind = RDP), allocatable      :: res_(:, :)

which is quite strange…

@ivanpribec

Design issues aside, you probably want to add bind(c) on the evaluation function, which is passed to C.

Also, doing this the compiler complains about error #6117: A module procedure in a submodule that is not a separate module procedure cannot have a binding label. since it is a local procedure defined in a submodule.

[FortranFan]

@mEm

… I am this time more keen to the one proposed by @ivanpribec . I find it much less verbose on the C side …

@mEm

… calling it with

extdata__.evalFunc_ptr__(itc__, nfreqs__, freq__, nnl__, temp__);

as you @ivanpribec also suggest. I get undefined addresses on the Fortran side using

subroutine evalFunc_C_to_F_wrapper_(itc, nf, f, idim, res)
..
      real(kind = RDP), allocatable      :: res_(:, :)

@mEm ,

It’s unclear what exactly you’re doing. Ostensibly it would appear you are dealing on the Fortran side with a subprogram interface along the following lines:

module m
   use, intrinsic :: iso_c_binding, only : c_int
contains
   subroutine sub( n ) bind(C, name="sub")
      integer(c_int), allocatable, intent(inout) :: n
      n = 42
   end subroutine 
end module 

Toward which you are trying do the following on the C side with the hope of eschewing verbosity:

#include <stdlib.h>
void sub( int * );
int main() {
    int *x;
    x = (int *)malloc( sizeof(int *) );
    *x = 0;
    sub(x);
    free(x);
}

But try it, you will see that it won’t work, the Fortran subprogram in such a case is not interoperable with such an approach due to the ALLOCATABLE attribute of argument n.

It is with this in mind as you’d shown in your original post that I pointed to you the enhanced interoperability facilities from Fortran 2018 that allow the Fortran subprogram - as shown - to be consumed via a C companion processor. Sure, it leads to a bit of additional verbosity on the C side but the advantage is all in Fortran - the code there remains compact and efficient and one can interoperate arguments which are of assumed-shape and/or have the ALLOCATABLE attribute and so forth.

// option that works
#include <stdio.h>
#include "ISO_Fortran_binding.h"
void sub( CFI_cdesc_t * );
int main() {
    CFI_CDESC_T(0) cx;
    int irc;
    irc = CFI_establish((CFI_cdesc_t *)&cx, NULL,
      CFI_attribute_allocatable,
      CFI_type_int, sizeof(int), 0, NULL);
    sub((CFI_cdesc_t *)&cx);
    printf("C main - x: %d", *(int *)cx.base_addr);
    irc = CFI_deallocate((CFI_cdesc_t *)&cx);
}

C:\temp>ifort /c /free /standard-semantics m.f
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.9.0 Build 20230302_000000
Copyright (C) 1985-2023 Intel Corporation.  All rights reserved.


C:\temp>cl /c /W3 /EHsc c.c
Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31937 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

c.c

C:\temp>link c.obj m.obj /subsystem:console /out:c.exe
Microsoft (R) Incremental Linker Version 14.34.31937.0
Copyright (C) Microsoft Corporation.  All rights reserved.


C:\temp>c.exe
C main - x: 42