Memory mapped files in Fortran?

C has memory mapped files. Among different possible usages they allow:

  • accessing the content of a file as if it was an array in memory
  • allocating very large arrays beyond the RAM+swap size: the content is basically on disk and cached in RAM; of course there’s no magic and it can’t be as fast as having the content only in RAM, but in some cases (and assuming fast disks such as SSDs) the performance penalty can be relatively limited

Would it be possible to get these features in Fortran? Or is it too OS-dependent, and therefore non portable?

real, allocatable :: a(:)
open(unit=lu,file="foo.bin",form="unformatted")
allocate( a(n), mmap=lu )
1 Like

You can open a file with access="direct" and recl=something. Getting the file to have the correct size I’m not as sure about. I haven’t had much reason to use them. Anyway, we already have this with direct access files.

A memory mapped file can also be accessed by several processes. It’s a way to share data between processes, very different from message queues or pipes.

Having this would be incredible. The memory mapped file system APIs are generally not trivial to use, so having this bundled in the language would be great!

The implementation is OS-dependent, but the concept exists in all platforms I know. I don’t feel that’s an argument against: ˋdo concurrent` concrete implementation is highly OS-dependent too.

Many languages have OS-dependent features: in Python, the function to open a shared memory file has a different signature between Windows and Linux (mmap — Memory-mapped file support — Python 3.12.1 documentation). On systems without memory mapped files, the module is simply absent (micropython for example).

Yes, I have used them only in the cases described above, but there are more more potential applications.

Indeed in Fortran it could be an optional module ISO_MMAP

OK, but it’s not as flexible. For instance I had a recent case in C where a bunch of large arrays where needed, but the code had also to be able to run on machines with limited RAM. So for these arrays we just did something like:

if (enough_ram) arr = malloc(...) else arr = mmap(...);

And we didn’t have to change a single line in the rest of the code. In our case the performance penalty was acceptable (30% slower), because the array were accessed sequentially with many computations on chunks that were already in cache.

Granted, memory mapping can probably be used in Fortran through C interoperability, but generally speaking I prefer having the useful features directly native in the language.

By the way, to quickly get a file of the desired size (say n default reals) I would write:

open(newunit=lu, file="foo.bin", acces="stream")
write(lu, pos=n*storage_size(0.0)/file_storage_size) 0.0
1 Like

In that case you could probably do something like

real, pointer :: arr(:)

call c_f_pointer(c_ptr=c_get_mem(...), fptr=arr, shape=[...])
void* c_get_mem(...) {
  return enough_ram ? malloc(...) : mmap(...);
}

:wink:

I played with this briefly in the past, using the C route. My motivation were memory-mapped arrays following this paper:

Rojc, B., & Depolli, M. (2021). A Resizable C++ Container using Virtual Memory. In ICSOFT (pp. 481-488).
https://www.scitepress.org/Papers/2021/105571/105571.pdf

My experiment can be found here: Virtual memory array in Fortran based on POSIX system headers · GitHub. I admit I didn’t fully read the documentation for mmap and just blindly followed the article. I copied the parameter values out of mman.h header file on my Mac.

The purpose of the original work above was to use the mmap-ed container for fast-generation of point clouds, as described in:

Depolli, M., Slak, J., & Kosec, G. (2022). Parallel domain discretization algorithm for RBF-FD and other meshless numerical methods for solving PDEs. Computers & Structures, 264, 106773. https://doi.org/10.1016/j.compstruc.2022.106773


Another interesting use of memory-mapped arrays is for cyclic buffers. An example can be found in the following work (not easy reading):

Kummerländer, A., Dorn, M., Frank, M., & Krause, M. J. (2023). Implicit propagation of directly addressed grids in lattice Boltzmann methods. Concurrency and Computation: Practice and Experience, 35(8), e7509. https://doi.org/10.1002/cpe.7509


Concerning the syntax, vendors already implement extensions to the open statement. For example if you look at Intel’s documentation, the items in green are Intel-specific: OPEN Statement Specifiers. But I don’t think any vendor has extensions in the allocate statement.

Is there a particular reason you want the memory mapping to work with allocatable arrays? In your snippet,

open(unit=lu,file="foo.bin",form="unformatted")
allocate( a(n), mmap=lu )

does n take on the value from the file, or does it provide access to up to n elements?

With pointer arrays, in principle you could do something like:

real, pointer :: mm_a(:)   ! prefix with mm_, so we remember to deallocate

call mmap_from_file("foo.bin",mm_a)
! ...
call mmap_destroy(mm_a)

The caller need not know that this is a OS-specific feature implemented using C-binding.

A slightly-more complicated option, would be to make the mapped file it’s own type a la Boost:

type(mapped_file) :: mfile
real, pointer :: mm_a(:)

mfile = mapped_file(filename,action="readwrite"[,length=maxlength,offset=0])
call mmap_from_file(mfile,mm_a)
if (associated(mm_a)) then
   ! ...
end if
nullify(mm_a)

! finalizer takes care of closing mfile

Pointer arrays, and subroutines are both native elements of Fortran. Something like this could surely find a home in the Fortran stdlib.

The reason that solution worked well in C is because in C arrays are typically just pointers to the initial element, not because mmap would be a native part of C. I’d also expect you’d have to change at least a second line:

if (enough_ram) free(arr) else munmap(arr,...);

So I think with pointer arrays you can replicate what you had in C:

real, pointer :: a(:)
if (enough_ram) then
   allocate(a(n))
else
   call mmap(...)
end if

! ... use a ...
! not a single line needs to change

if (enough_ram) then
   deallocate(a)
else
   call munmap(...)
end if

Could be both. I could be also a single statement that creates a temporary files are maps it to memory. I haven’t really thought about the right syntax to use.

In C mmap() is a kind of extension of malloc(), as it can even be used to allocate regular memory without any backing file, as malloc() does. So it looked natural to me to include this feature in allocate() (but again I’m not talking about the syntax, just about the principle).

You’re right, implementing it in stdlib first is probably the right thing to do (although it will address only pointers, not allocatables, but that’s OK).

Generally speaking, sure. In our particular case it was useless as the array was needed until the very end of the execution, and the temporary file was automatically removed at the end of the execution.

Memory mapped files can be created using access=‘stream’, which gives byte addressing.

But why have the data in a file ?
Create a derived type structure with allocatable components and then most file data structures can be replicated in memory.
Just buy enough memory for the project. Now it is a very big database that can’t fit into memory.

Now with OpenMP, the strategy is managing cache, so memory is used for data structures as disk files once were. Data structures can also be more flexible as the analysis proceeds and data components can be re-allocated.

Opening a file in stream access doesn’t mean it’s memory mapped at all (?)

Thanks for the tip :slight_smile: ! But it’s not always that simple…

It’s not only about amount of memory, it’s also about sharing memory between processes (not necessarily a large amount).

It is sort-of memory-mapped, in a sense that you can read/write at any point of the file, using pos= specifier, bit it is not in the full sense of mmap in C, which gives you ability to use the file as an array.

I see that you are using MAP_PRIVATE | MAP_ANONYMOUS. According to this answer on SO (which is not plain accurate on all points, though), this is a combination that allocates a virtual memory volume, without any backing file, and the interesting point is that on Linux malloc() precisely uses mmap() with this combination for allocations above 128kBi.

Some MAP_xxx flags are also unclear to me.

  • It seems that with MAP_PRIVATE (without MAP_ANONYMOUS) it’s possible to specify a backing file, but nothing is actually written (backed) in it, so I don’t get the point of specifying a file…
  • “true” memory mapping of physical files requires MAP_SHARED
    • I’ve then experienced that MAP_NORESERVE was required if the size of the mapping was exceeding the RAM+swap size.
    • MAP_NORESERVE is a Linux extension, and my understanding is that when used, mmap() does not check that the requested volume can fit in the RAM+swap… But I but don’t get why some swap space is needed at all when a physical file is present for backing the memory pages.
  • There’s also MAP_POPULATE, and my understanding is that it is equivalent to touching all the pages
  • And finally, I don’t know how to use mmap() for inter-proces communication :sweat_smile:

I have a related question that is more general: are these parameter values (such as MAP_SHARED, etc…) specified by the Posix standard, or do they possibly depend on the compiler?

Here’s what I have on my systems:

On Ubuntu in /usr/include/x86_64-linux-gnu/bits/mman-linux.h:

#define MAP_SHARED	0x01		/* Share changes.  */
#define MAP_PRIVATE	0x02		/* Changes are private.  */
#ifdef __USE_MISC
# define MAP_SHARED_VALIDATE	0x03	/* Share changes and validate
					   extension flags.  */
# define MAP_TYPE	0x0f		/* Mask for type of mapping.  */
#endif

/* Other flags.  */
#define MAP_FIXED	0x10		/* Interpret addr exactly.  */
#ifdef __USE_MISC
# define MAP_FILE	0
# ifdef __MAP_ANONYMOUS
#  define MAP_ANONYMOUS	__MAP_ANONYMOUS	/* Don't use a file.  */
# else
#  define MAP_ANONYMOUS	0x20		/* Don't use a file.  */
# endif
# define MAP_ANON	MAP_ANONYMOUS
/* When MAP_HUGETLB is set bits [26:31] encode the log2 of the huge page size.  */
# define MAP_HUGE_SHIFT	26
# define MAP_HUGE_MASK	0x3f
#endif

On Darwin in /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/sys/mman.h:

/*
 * Flags contain sharing type and options.
 * Sharing types; choose one.
 */
#define MAP_SHARED      0x0001          /* [MF|SHM] share changes */
#define MAP_PRIVATE     0x0002          /* [MF|SHM] changes are private */
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
#define MAP_COPY        MAP_PRIVATE     /* Obsolete */
#endif  /* (!_POSIX_C_SOURCE || _DARWIN_C_SOURCE) */

/*
 * Other flags
 */
#define MAP_FIXED        0x0010 /* [MF|SHM] interpret addr exactly */
#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
#define MAP_RENAME       0x0020 /* Sun: rename private pages to file */
#define MAP_NORESERVE    0x0040 /* Sun: don't reserve needed swap area */
#define MAP_RESERVED0080 0x0080 /* previously unimplemented MAP_INHERIT */
#define MAP_NOEXTEND     0x0100 /* for MAP_FILE, don't change file size */
#define MAP_HASSEMAPHORE 0x0200 /* region may contain semaphores */
#define MAP_NOCACHE      0x0400 /* don't cache pages for this mapping */
#define MAP_JIT          0x0800 /* Allocate a region that will be used for JIT purposes */

To find the system mmap header, I used the file:

// find_mman.c
#include <sys/mman.h>

which I compiled with gcc -H -c find_mman.c to find the paths of the included headers.

Note I truncated the file output.

I haven’t checked the Posix standard, but I assume the values of the constants are system-dependent.

An easy way to get them is as follows:

// export_mman.c
#include <stdio.h>
#include <sys/mman.h>

int main(void)
{
    printf("integer(c_int), parameter :: MAP_SHARED = %d\n", MAP_SHARED);
    printf("integer(c_int), parameter :: MAP_PRIVATE = %d\n", MAP_PRIVATE);
    return 0;
}
$ gcc-13 export_mman.c && ./a.out > "mman.fi"
$ cat mman.fi 
integer(c_int), parameter :: MAP_SHARED = 1
integer(c_int), parameter :: MAP_PRIVATE = 2

I couldn’t find any mandated values for the constants either. This means that in such cases we need some C code one way or another.

A demo code with a simplified Fortran interface: GitHub - PierUgit/fmmap: Memory Mapped files in Fortran

1 Like