Memory mapped files in Fortran?

Language enhancement
7

I played with this briefly in the past, using the C route. My motivation were memory-mapped arrays following this paper:

Rojc, B., & Depolli, M. (2021). A Resizable C++ Container using Virtual Memory. In ICSOFT (pp. 481-488).
https://www.scitepress.org/Papers/2021/105571/105571.pdf

My experiment can be found here: Virtual memory array in Fortran based on POSIX system headers · GitHub. I admit I didn’t fully read the documentation for mmap and just blindly followed the article. I copied the parameter values out of mman.h header file on my Mac.

The purpose of the original work above was to use the mmap-ed container for fast-generation of point clouds, as described in:

Depolli, M., Slak, J., & Kosec, G. (2022). Parallel domain discretization algorithm for RBF-FD and other meshless numerical methods for solving PDEs. Computers & Structures, 264, 106773. https://doi.org/10.1016/j.compstruc.2022.106773

Another interesting use of memory-mapped arrays is for cyclic buffers. An example can be found in the following work (not easy reading):

Kummerländer, A., Dorn, M., Frank, M., & Krause, M. J. (2023). Implicit propagation of directly addressed grids in lattice Boltzmann methods. Concurrency and Computation: Practice and Experience, 35(8), e7509. https://doi.org/10.1002/cpe.7509

Concerning the syntax, vendors already implement extensions to the open statement. For example if you look at Intel’s documentation, the items in green are Intel-specific: OPEN Statement Specifiers. But I don’t think any vendor has extensions in the allocate statement.

Is there a particular reason you want the memory mapping to work with allocatable arrays? In your snippet,

open(unit=lu,file="foo.bin",form="unformatted")
allocate( a(n), mmap=lu )

does n take on the value from the file, or does it provide access to up to n elements?

With pointer arrays, in principle you could do something like:

real, pointer :: mm_a(:)   ! prefix with mm_, so we remember to deallocate

call mmap_from_file("foo.bin",mm_a)
! ...
call mmap_destroy(mm_a)

The caller need not know that this is a OS-specific feature implemented using C-binding.

A slightly-more complicated option, would be to make the mapped file it’s own type a la Boost:

type(mapped_file) :: mfile
real, pointer :: mm_a(:)

mfile = mapped_file(filename,action="readwrite"[,length=maxlength,offset=0])
call mmap_from_file(mfile,mm_a)
if (associated(mm_a)) then
   ! ...
end if
nullify(mm_a)

! finalizer takes care of closing mfile

Pointer arrays, and subroutines are both native elements of Fortran. Something like this could surely find a home in the Fortran stdlib.

The reason that solution worked well in C is because in C arrays are typically just pointers to the initial element, not because mmap would be a native part of C. I’d also expect you’d have to change at least a second line:

if (enough_ram) free(arr) else munmap(arr,...);

So I think with pointer arrays you can replicate what you had in C:

real, pointer :: a(:)
if (enough_ram) then
   allocate(a(n))
else
   call mmap(...)
end if

! ... use a ...
! not a single line needs to change

if (enough_ram) then
   deallocate(a)
else
   call munmap(...)
end if
1 Like