Hello,
When using OpenMP I have had in the past (say 15-20 years ago) bad experiences with I/O operations (including simple write(,)) in parallel regions. So I took the habit to always put the I/Os inside critical regions. However I never sorted out this topic: does OpenMP specifies something about thread-safety of the I/Os, or does it entirely depend on the implementations?
Thanks
You are not alone. Every time there is an I/O, I go for !$omp critical all the way. However I think things are somewhat better nowadays.
Thread-safety requires many things. Non-static data is the most obvious one (no save etc.) The annoying part here is procedures called must be recursive, even though what we mean by that in this case is re-entrant, not recursive. I find that more annoying than I/O.
Now, I/O is another bag of worms here. Thread-safety in the ārun-time environmentā is what you are asking for, and I donāt think there is a standard to speak of, but itās rather implementation-dependent. I am pretty sure some compilers do provide ways for run-time safety (and even for a more elegant re-entrancy), but we definitely need more information from experts in the topic.
One thing is sure, we need a standard here. Desperately - and not tomorrow, yesterday.
Since 2018 Fortran procedures are recursive by default. The attribute non_recursive exists too.
Hereās an example:
program iadd_example
print *, iadd(13,29)
contains
!> Integer addition using bitwise operators (carrying)
pure function iadd(a,b) result(c)
integer, intent(in) :: a, b
integer :: c
if (b == 0) then
c = a
else
c = iadd(ieor(a,b),shiftl(iand(a,b),1))
end if
end function
end program
You may need to use compiler flags to enforce the standardized behavior. For example with the new Intel Fortran compiler:
$ ifort -standard-semantics iadd_example.f90
$ ./a.out
42
At least in gfortran 11.2, this doesnāt seem to be implemented. I have to explicitly declare as recursive all the procedures called in a parallelized region, even though said procedures are designed to be re-entrant right from the beginning. If I donāt tell the compiler those are recursive, it wonāt work (well, unless I dodge them with critical regions.)
I am also very much interested if there is a flag for run-time environment thread safety (this includes I/O), especially in gfortran.
Edit: a.out is still a thing? I forgot it existed, and there was a GNU/Linux conference back in 2014 with people complaining a.out is no more.
While the a.out format is no longer used the default name of an executable on Linux conpiled by gcc/gfortran is still a.out. It is, however just a name.
Maybe I am misunderstanding what you mean here, because I have been heavily using OpenMP for years, and never declared subroutines called in parallel region as ārecursiveā (I am mainly using ifort, but occasionnaly gfortran too). When compiling with -fopenmp the compiler takes care of what is needed for the routines (no static variablesā¦)
I agree that !$OMP CRITICAL might be a conservative approach, but it also has a useful function in that it groups output for each thread, which makes the resulting output easier to read.
I do not typically place write (*,fmt=ā¦) in a critical region, but again, if I want the output grouped, I will resort to critical. I have not found corruption problems when mixing writes.
Having reports from all threads mixed in a file or to the screen can be very confusing and requires extra labeling of each line for clarity.
I am sure the option ā-fopenmpā implies ā-frecursiveā.
Indeed. But beyond formating issues I remember crashes with unprotected writes. Even with unformatted and direct access writes, where each thread was writing to different places in the file (so no collision possible).
I use Windows 10, so given the buffered I/O, I would definately not use direct access writes outside a critical region.
My approach to shared I/O is even more conservative. I have a long multi-thread compute phase so use a diferent unformatted sequential output file for each thread. Then at end of compute reporting phase, I do that in a critical region and report results to multiple shared text files. I do not write to direct access binary files in this phase. My final reporting is a single thread reduction program of shared summary text files and also in Excel.
So I have not encountered shared direct access files in the OMP region, although I do access them in other single thread parts of the program. (Structural Finite Element analysis package). My present OpenMP skyline solver is an in-memory solver (limited to 64 GBytes installed memory), so again no file access in this OpenMP phase.
The old F standard (which I donāt think it was created with thread safety in mind,) actually followed a ācold turkeyā policy with that. I/O inside a pure or recursive function was treated as an error. It was also able to catch less obvious pitfalls, such as declarations like real :: foo=1.0 which implies persistence. Back then I liked that Gordian knot approach of F. But today, I would say just avoid I/O in parallelization, but when itās really needed, critical regions is the way to go, at least until we have a concrete standard.
I can get the reason for pure procedures, which are supposed to have no side effect, but not for the recursive routines. Whatās wrong with I/Os in a recursive routine?
I actually expect any compiler to report an error for such a declaration in a recursive routine (or for a routine compiled with -fopenmp)
As things stand, the issues and challenges and the fixes and workarounds you experience in this area will vary with compiler implementations and also their versions.
If you are āmainly using ifort,ā you may also want to inquire at the Intel Fortran forum:
Also, please note there is an outstanding support request with Intel re: thread-safety of list-directed I/O . So until that gets resolved, you may want to be careful with list-directed IO in any concurrent execution scenarios, OpenMP or otherwise.
And with Intel Fortran, you may want to try to their latest oneAPI version and start with small reproducer-type test cases attempting other IO (besides list-directed) with OpenMP and see if you get them to work. If such cases fail, either post them at the above Intel forum and request assistance with them; or, if you are able to buy their support subscription, then do so to get more direct service via their Software Center:
https://supporttickets.intel.com/?lang=en-US
The small cases can also help you with gfortran and in case of any problems, you may want to follow-up at GFortran - GCC Wiki.
Actually I am OK with my current practice of putting all I/Os in critical sections. I was just wondering of the thread-safety of I/Os was covered by the standards (Fortran and/or OpenMP) or not. It seems that the answer is ānoā (in which case one should always use critical sections if one wants the code to be portable and not depend on a specific compiler).
Do you have a link?
I/O is not going to be thread-safe without a critical section one way or another. Either users add that when necessary or Fortran implementations add it in every single I/O statement, since they canāt know the calling context in all cases. Nobody is going to accept the latter overhead, so just do the right thing and add critical sections. Note also that OpenMP is not relevant to Fortran standardization and the committee is not going to consider doing anything about a problem caused by OpenMP, for which a reasonable solution already exists.
Also, when I/O operates on files that are visible to multiple processes, e.g. stdout when MPI is used, the locking needs to happen across that scope, which is not something a Fortran implementation should do. Again, MPI has methods for solving this already.
Well, itās actually not just OpenMP. The do concurrent feature opens the possibility of simultaneous writes by different threads.
I am not convinced that writes are permitted inside do concurrent, even if some compilers allow this.
C1143 A reference to an impure procedure shall not appear within a DO CONCURRENT construct.
C15107 A pure subprogram shall not contain a print-stmt, open-stmt, close-stmt, backspace-stmt, endfile-stmt, rewind-stmt, flush-stmt, wait-stmt, or inquire-stmt.
C15108 A pure subprogram shall not contain a read-stmt or write-stmt whose io-unit is a file-unit-number or *.
NOTE 4
The above constraints are designed to guarantee that a pure procedure is free from side effects (modifications of data visible outside the procedure), which means that it is safe to reference it in constructs such as DO CONCURRENT and FORALL, where there is no explicit order of evaluation.
@JeffH I am not convinced either, but couldnāt find a clear and formal restriction. For, you found that within a do concurrent constructs must call only pure procedures and that pure procedures shall not contain read/write (for the most part), but not that the do concurrent construct itself shall not contain read/writeā¦
The standard permits IO within a DO CONCURRENT block construct, just that there are some requirements that are listed in the standard on read / write operations and note several aspects of the behavior are marked as processor-dependent. This aspect is among the reasons why I mentioned compiler implementations specifically in the comment upthread and suggested to look into them e.g., with Intel, etcā¦
The following conforms:
integer :: x(3)
do concurrent ( integer :: i = 1:size(x) )
print *, x(i)
end do
end
Even if it is not really I/O, a āwrite(string,fmt) variableā into a character variable might pose problems in the runtime environment (same for read). At least gfortran uses file unit numbers and other shared data structures internally. Not long ago, this part was not thread-safe due to data races in the implementation. It should be thread-safe since version 12 I think.