Heap, stack, and thread-safety

1

I am totally ignorant about this topic, so what I am going to say/ask may be wrong or make little sense.

I just read that heap-allocated arrays might not be thread-safe, because “data stored in Heap-memory are visible to all threads”. This sounds like a rather obscure claim, and I do not know whether it is valid.

It seems to me that gfortran saves arrays on the heap by default, although I have not found the official documentation about this aspect. ifort has an option -heap-arrays to do so. The gfortran counterpart of -heap-arrays seems to be -fno-stack-arrays. Thanks for correcting me if I am wrong.

If the abovementioned claim is valid, does it mean that Fortran code compiled with gfortran without any option or ifort with -heap-arrays is generally not thread-safe? That would be very concerning, isn’t it? Thus I doubt whether the claim is true and I hope not.

Something that might be related: When I compile a program involving large automatic arrays with gfortran -fmax-stack-var-size=64 in order to avoid stack overflows, the following message is emitted:

Error: Array ‘XXX’ at (1) is larger than limit set by ‘-fmax-stack-var-size=’, moved from stack to static
storage. This makes the procedure unsafe when called recursively, or concurrently from multiple threads. 
Consider increasing the ‘-fmax-stack-var-size=’ limit (or use ‘-frecursive’, which implies unlimited 
‘-fmax-stack-var-size’) - or change the code to use an ALLOCATABLE array. If the variable is never
accessed concurrently, this warning can be ignored, and the variable could also be declared with the
SAVE attribute. [-Werror=surprising]
2

All threads share the same virtual address space, so data created and stored in one thread is addressable from other threads. I don’t think it matters whether that memory is heap or stack. When some entity is local to a thread, that just means that each thread has its own copy that can be modified separately from the other threads. This is mostly just a software feature, and the effort to support that programming model is shared between the compiler and the operating system.

Going beyond that thread model can involve many different approaches. There can be multiple cores per cpu, multiple cpus per node, and multiple nodes. Memory can be shared among threads, among cores on a cpu, and between cpus on a node. Usually memory that is “shared” by different nodes is simulated shared memory (using message passing and synchronization flags, or shared i/o devices, or shared network resources).

1 Like
3

TLDR; static variables are not thread safe (or recursion safe), and under some conditions a compiler may make a variable static when you might not expect it.

The message from gfortran is actually a reasonable one, but some background information can make things a bit clearer.

Let’s start with saved variables. A saved variable is one that lives for the entire execution of the program. It may only be accessible from a certain scope, but it never “goes away.” These variables are not thread-safe, because different threads will be accessing the same memory. I.e., their memory is reserved at program startup, which does not know how many threads there will be since threads can be spawned dynamically.

Next, let’s talk about local (stack) variables. When a procedure begins execution, it reserves a “stack frame,” big enough to hold all the local variables. Once the procedure is finished, the entire stack frame is “released,” cleaning up all the local variables. When multiple threads are executing, they each get their own stack frame. Thus local variables are thread safe.

But, stack frames can only be so big. So what to do with variables that are too big to fit in a stack frame (usually large arrays)? Before there was such a thing as multi-core cpus and threads, recursive procedures, or automatic arrays, they just got treated as saved variables. I.e. we know how big they are, so we just reserve space for them in the same “static storage” location created at program startup. This was sometimes referred to as “the heap,” hence the name of that ifort option, but static storage is really more appropriate, which is what that gfortran message says. The heap is now the name of the place that dynamically allocated variables go so saying that statically allocated variables are placed there may be true, but can be a bit misleading.

We said earlier that saved (statically allocated) variables aren’t thread safe (they’re not safe in recursion either), and so you get that message. The 2018 standard says that all procedures are recursive by default, so as compilers start to be more compliant they will stop doing this. gfortran has an option to turn this on and tells you about it, -frecursive. This basically stops it from using static storage for unsaved variables. I.e. All local variables will either go on the stack or be dynamically allocated. You can do this manually by making the variable allocatable.

2 Likes
4

Thank you @everythingfunctional for the explanation. So it is true that Fortran code compiled with gfortran without any option or ifort with -heap-arrays is generally not thread-safe, right?

5

It’s slightly more nuanced. More like “Fortran code compiled with gfortran without any option or ifort with -heap-arrays is potentially not thread-safe.” I think the only general case where it will be not thread-safe is in procedures that have local variables that are arrays declared with a large, constant size. I think in all other cases the local variables will either fit on the stack, or must be allocated dynamically (since the size isn’t known until procedure execution begins), and those cases are thread-safe.

2 Likes
6

Using -heap-arrays in ifort (or the equivalent in gfortran) has no effect on whether or not code is thread-safe. Automatic (local arrays whose size depends on run-time values) and temporary arrays, which these options affect, are local to the thread instance, with the location typically being stored on the stack or in registers.

I will ignore the use of global variables, which @everythingfunctional explained in detail. What you may have to worry about is whether the compiler uses any static (fixed) storage for local variables or data structures it uses to access variables. In Fortran 2008, procedures could be declared RECURSIVE, which in effect meant that static storage was not used except for variables you explicitly made static (COMMON/SAVE/module variables). This changed in Fortran 2018 where RECURSIVE was now the default. but ifort, at least, does not do this unless you have also said -standard-semantics. (I don’t know gfortran’s behavior here.)

If thread safety is important, at a minimum all of your procedures should have the RECURSIVE prefix (harmless if F2018 semantics are being used), and you don’t use SAVE, COMMON or module variables without providing some mechanism to synchronize access.

2 Likes
7

Thanks for the clarification on ifort’s behavior. Is there a situation in which ifort would place a local variable in static storage? I.e. in which this code would not be thread/recursion safe?

subroutine foo(...)
  integer, parameter :: work_size = ...
  integer :: work_array(work_size)

  ...
end subroutine

It is worth making note for those who weren’t aware, (re: my explanation for saved variables), any variable declared in the main program, any module variable, any variable that appears in a common block, any variable declared explicitly with the save attribute, and any variable with an initialization in its declaration, is a saved variable.

1 Like
8

Sure - it does this all the time if default-recursive is not enabled (asking for OpenMP support gets you this.) But more important, data structures passed to I/O routines, descriptors used to pass certain arguments, and even local scalars can be put in static storage. ifort defaults tend to be those that help performance. These do change over time (-realloc-lhs is now the default, for example), but it’s best to not make assumptions - use RECURSIVE in all your procedures in a threaded application.

4 Likes
9

Thank you @RonShepard @everythingfunctional @sblionel for very informative explanations.

According to

and

I would suppose that the following module is thread-safe if compiled with gfortran or ifort -heap-arrays .

module s_mod
implicit none
private
public :: sfunc

contains

function sfunc(n) result(s)
! The purpose of this stupid function is to test a large automatic array.
integer, intent(in) :: n
real :: s
real :: x(n, n)
x = 1.0
s = sum(x)
end function sfunc

end module s_mod
10

It is true that for variables or arrays that are on the local thread stack, these will be thread safe, as they are not seen easily by other threads, although this totally misses the point of being thread-safe.

It is true that most private copies or local arrays in a multi-thread program are located on the “thread-stack”, while allocated arrays are located in the single heap.

There can be shared arrays on the primary thread stack (automatic or local arrays outside the OMP region) or the heap (most allocated or shared arrays). Consideration of “thread-safe” only applies to these variables or arrays that can be seen my multiple/all threads, essentially “shared arrays”. So being heap-allocated or located on the primary thread stack is not really what defines thread-safe.

“Thread-safe” is more an issue for variables that are not obviously shared. A good example of arrays or variables that are not thread-safe are those in a “utility” routine (that can be called by all threads) that have the save attribute, eg counters or status variables.

To make a multi-thread program thread safe is due to you, the programmer. For arrays (or importantly variables) to be “thread-safe” is more a program design approach.

For designing an OpenMP program to target performance, it can be preferable to have small private arrays on the local thread stack, while large shared arrays on the heap. It is better that shared arrays can span many memory pages to target efficiency related to memory consistency, but this is a very different matter to thread-safety.

1 Like
11

Again, -heap-arrays makes no difference as to thread safety here.

Your procedure sfunc is not thread-safe unless marked RECURSIVE or compiled with full Fortran 2018 semantics. Variable s could be statically allocated, and different threads might overwrite each other.

1 Like
12

@zaikunzhang ,

In thread you quoted, @sblionel had advised, " it’s best to not make assumptions - use RECURSIVE in all your procedures in a threaded application" and @sblionel had stated -heap-arrays in and of itself was not connected with thread-safety in any way and yet you provided to show an example involving your sfunc procedure where both the pieces of advice are not heeded by you. This is a similar pattern to your other recent threads with floating-point calculations involving compiler optimization using Intel Fortran. What do you think might be behind the pattern, is it a language vernacular barrier you think?

14

thanks

15

Any code meant to be multithreaded should be compiled with the -fopenmp option. You don’t need any other compiler option.

Of course this doesn’t ensure the thread-safety if you have written the code the wrong way, but at least this tells the compiler to not do anything on its own that is potentially not thread-safe.

1 Like