Memory usage and allocation question

1

Hi all,

I have a question about how Fortran allocates memory. I have a large Fortran program not written by myself, It allocates a large amount of memory at the beginning (I believe). However, if I watch ram usage while the program is running, the ram usage increases slowly over time instead of all at once in the beginning. This is annoying, as it makes it more difficult to predict exactly how much ram will actually all be used.

Any idea what is going on here? It is possible that while the ALLOCATE calls are in the beginning of the program that the memory is not actually allocated until it is needed? Or is it more likely there is an ALLOCATE being called during the programs running that I simply am unaware of?

Thanks!

1 Like
2

Is this a legacy program? It might be allocating a huge memory pool and accessing it every now and then and also have random allocatable arrays that are allocated dynamically. Another options is that there’s memory leaks and that’s why the usage increases

3

If it increases steadily and indefinitely, you may want to check that you don’t have a memory leak.

4

How do you monitor the memory that is used by the program?

There is a difference between the “virtual memory” used by the program, and the “resident memory” used by the program: if the program allocates say 10GB, the reported virtual memory used will immediately increase by 10GB, but the reported resident memory used will only gradually increase as the memory is effectively accessed.

5

Thank you for the replies.

The program isn’t legacy, but the programmer (and his style) is. ( I hope that makes sense)

I have access to the code base, and have some reasonable understanding of how it works. I have looked for and cannot find any allocate calls in loops; they are all at the initiation of the program (as long as i haven’t made any mistakes searching, FORTRAN is not my first language).

I monitor the memory using TOP. I do not know if this is virtual or resident, nor did I know there was a difference. Which one does TOP show? What is the best way to measure virtual vs resident memory?

Thanks

6

Hey, The output from the top command should list both virtual and resident memory (VIRT and RES I think).

7

If you have the NAG Fortran Compiler, nagfor(1), compile and link with -mtrace=all. This will print a line telling you where in the source an allocation or deallocation happens.

If you don’t, and are on Linux, you should use “strace -e %memory ./a.out” to probe the userspace/kernel transactions about memory.

If your compiler uses malloc(3), you can turn on malloc-tracing. Start the program under the debugger, maybe like so

MALLOC_TRACE=/tmp/malloc_trace.txt gdb -ex "start" -ex "call (void)mtrace()" -ex "cont"  -ex "quit" ./a.out

Finally, Massif, is another useful tool for getting information about memory consumption of any program.

2 Likes
8

Default top reports virtual memory usage under the column heading “VIRT” and actual memory resident under the column heading “RES”. I believe most Fortran compilers make at least some use of the stack during execution, although large local arrays and allocated variables are dynamically allocated during execution. It is possible your program is calling malloc directly, so you might grep for that.

Of possible interest may be:
.
.Track memory usage in Fortran 90
Stack-Overflow
.valgrind

9

Is the memory being allocated as allocatable arrays or pointer arrays. If the latter I would look for a memory leak. Also, I would personally avoid using valgrind on Fortran codes. My experience is it will give false positives. If you are on Linux and have access to Intel’s ifx compiler, you might try using the code sanitizer options.(-fsanitize=address). I’ve found it to be more reliable than valgrind.

1 Like
10

With a memory leak, the virtual memory is continuously increasing during the execution. So that’s a the first thing to look at.

11

Thanks all for the suggestions.

When I get the time, I’ll try some of the things suggested

12

In addition to several excellent suggestions on how to track down a possible memory leak listed above, perhaps you are worried about the amount of memory used as you need a value to give a job scheduler or are hitting a limit on how many program instances you can run on a particular node. In that case you might want the program to always generate a usage report. There are several possible ways to do that internally or externally ( e.g. the time(1) command, an accounting command often available via a job scheduler ( such as LSF, OpenPBS, Torque, Slurm, …).

It is much easier now to call C routines such as getrusage but an often overlooked
option on Linux is to open and print data from the /proc/self directory, where you can usually (varies a bit by platform) get memory high-water marks, system and user CPU time and all kinds of info for making a nice resource report without doing
anything other than OPEN and READ/WRITE calls. For example

program main
  implicit none
  real :: r(1000000000/4), rnd
  write(*,*)'just the array should take=',nint(real(storage_size(r))/8.0*size(r)/1024),'KB'
  write(*,*)'but resident size likely smaller'
  call pstatus()
  write(*,*)'until used or intialized'
  r=0
  call pstatus()
  call random_number(rnd)
  write(*,*)'so compiler does not optimize away',r(size(r)*rnd)
contains
subroutine pstatus()
!@(#) print Linux status for current PID
character(len=256) :: iomsg, line
integer :: iostat, lun
  open(newunit=lun,file='/proc/self/status',iostat=iostat,iomsg=iomsg)
  if(iostat.eq.0)then
     do
        read(lun,'(a)',iostat=iostat)line
        if(iostat.ne.0)exit
        if(index(line,'Vm').eq.1)then
	   if(index(line,'RSS').ne.0)then
              write(*,'(*(a))')trim(line),' <==='
	   else
              write(*,'(a)')trim(line)
	   endif
        endif
     enddo
  endif
  close(unit=lun,iostat=iostat)
end subroutine pstatus
end program main

Shows that just declaring an array usually does not cause it to be allocated on the spot, but initializing it usually does (which can be a good or bad thing depending on
where in the scope you initialize it, how often, …). So you might initialize the arrays in a debug version to eliminate a lot of small allocations if looking for the leak, for example.

 just the array should take=      976563 KB
 but resident size likely smaller
VmSize:   983936 kB
VmLck:         0 kB
VmRSS:      5440 kB <===
VmData:     1088 kB
VmStk:         0 kB
VmExe:        64 kB
VmLib:      2816 kB`
 until used or intialized
VmSize:   986048 kB
VmLck:         0 kB
VmRSS:    984128 kB <===
VmData:   979712 kB
VmStk:         0 kB
VmExe:        64 kB
VmLib:      2880 kB

although particularly when using high optimizations you may see a surprising variation in the numbers reported.

If the coding style is to allocate everything with declarations or in a common or global area in global scope (basically you had little choice but to do that without using custom loader commands with most f77 compilers) you could simple use

$ size build/*/app/mem|column -t
text  data  bss         dec         hex       filename
5581  1436  1000000448  1000007465  3b9ae729  build/gfortran_688A5DB7BAD2F9DA/app/mem
5989  1484  1000000448  1000007921  3b9ae8f1  build/gfortran_E167FD2A985B468F/app/mem

(The number of interest is usually under “dec” and in bytes.)

and see the size the program is declaring at startup what it is going to use, but that often just gives you a
minimum now (which can still be useful).

13

ALLOCATE does not take physical memory pages. It is the use of this memory that takes the memory pages. So if the program progressively defines the values in the arrays, this may appear as progressive allocation of memory pages.
ALLOCATE will commit virtual memory.
You should be able to identify virtual memory committed vs physical memory allocated to identify if this is happening.