We had exactly this problem with our electronic structure code The Elk Code. Data had to be written so that it could be used for the next run and some arrays were too large to keep in memory.
We solved it in two ways: the first was to switch some of the variables to single precision (also suggested by MarDie above). This did not significantly affect the accuracy of the results and was faster as a bonus. For example, large arrays are calculated in single precision but are then contracted and accumulated in smaller double precision arrays.
The second way, was to implement a simple RAM disk in Fortran for direct-access files (see the module modramdisk.f90 in the elk/src directory).
The ‘disk’ is an allocatable array of allocatable arrays (I think this is a Fortran 2003 feature):
! record data stored as 4-byte words
type, private :: rec_t
integer(4), allocatable :: dat(:)
end type
! RAM disk file consisting of the filename and an array of records
type, private :: file_t
character(len=:), allocatable :: fname
type(rec_t), allocatable :: rec(:)
end type
! arrays of files constituting the RAM disk
type(file_t), allocatable, private :: file(:)
which are allocated on-the-fly. All variables (real, complex, integer) are stored in 4-byte integer arrays using the Fortran transfer function.
Here is an example of writing to the RAM disk:
! write to RAM disk if required
if (ramdisk) then
call putrd(fname,ik,v1=vkl(:,ik),n1=nstsv,nzv=nstsv*nstsv,zva=evecsv)
end if
! write to disk if required
if (wrtdsk) then
! find the record length
inquire(iolength=recl) vkl(:,ik),nstsv,evecsv
open(206,file=fname,form='UNFORMATTED',access='DIRECT',recl=recl)
write(206,rec=ik) vkl(:,ik),nstsv,evecsv
close(206)
end if
By default, the record is written to both the RAM disk and the regular disk, although the latter can be disabled.
Here is code which reads from the RAM disk:
! read from RAM disk if required
if (ramdisk) then
call getrd(fname,ik,tgs,v1=vkl_,n1=nstsv_,nzv=nstsv*nstsv,zva=evecsv)
if (tgs) goto 10
end if
open(206,file=fname,form='UNFORMATTED',access='DIRECT',recl=recl)
read(206,rec=ik) vkl_,nstsv_,evecsv
close(206)
10 continue
If a record is not in the RAM disk, then the code seamlessly falls back to reading it from the regular disk. This can happen if the maximum number of records was exceeded or the data is in the RAM disk on a different node. Fortunately for our code, each node mostly reads the data it has written and not the data written by another node.
This simple RAM disk sped up the code enormously because the big direct access files were not written to or read from the networked filesystem. (We’re currently working on a way to read the RAM disk of a different node using one-sided MPI communication, but it’s not that easy.)
Lastly, there may be another method you could use. We thought about trying this but it we needed root access on the cluster so we didn’t pursue it. It’s to use the temporary BeeGFS
file system BeeOND. The idea was to stitch together the Linux RAM disks (tmpfs) with BeeOND into a coherent networked file system independent from the cluster filesystem. For each run, the files needed by the code would be copied to the BeeOND disk, the code would be run and the files copied back to the regular file system. In principle we could devote the RAM of entire nodes to the file system, and make the disk as large as needed.
I’m not sure if there are any gotchas with this approach: you’d certainly need to get the system administrator to set it up and build it into the batch submission system, not something that you could do at user level.