# Problems with coarrays
**URL:** https://fortran-lang.discourse.group/t/problems-with-coarrays/2565
**Category:** Uncategorized
**Created:** [January 12, 2022, 4:22am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565 "2022-01-12T04:22:47Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 12, 2022, 4:22am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/1 "2022-01-12T04:22:47Z")
Hello all!
I seem to be having trouble and I’m wonding if anyone could help me…
I’m at a stand still until someone or some tutorial out there can help me.
I’m running on windows10 in the WSL. I’ve got caf compiling and cafrun works, but the coarrays just do not work.
Any ideas?
Example;
```auto
! Program Paratest
! Written by Frank Meyer
! Created Jan 9, 2022
! Version 0.1a
! Description: Testing some sync issues.
program Paratest
implicit none
! Create co-arrays for testing
integer :: i_test1[*]
! Other variables for testing purposes.
integer :: i_loop1, i_loop2, i_me, i_all
! Set variable for work
i_loop1 = 1
i_loop2 = 2
i_me = this_image()
i_all = num_images()
print *, i_me, " says; ", i_loop1 ! Does printing work? Good...
sync all
i_loop2 = i_me + 1 ! Can we access the coarray?
if (i_loop2 .gt. i_all) then
i_loop2 = 1
end if
i_test1[i_loop2] = i_me ! Put our number over one.
sync all
print *, i_me, " now says;", i_test1 ! Did it get here?
do i_loop1 = 1, i_all ! Add a dynamic value to it.
i_test1[i_loop1] = i_test1[i_loop1] + 1
end do ! Equiv to i_test1 += i_all
sync all
print *, i_me, " finally says;", i_test1 ! See the failure...
end program Paratest
```
Output is;
```
2 says; 1
3 says; 1
4 says; 1
7 says; 1
10 says; 1
11 says; 1
12 says; 1
14 says; 1
15 says; 1
8 says; 1
16 says; 1
1 says; 1
6 says; 1
9 says; 1
13 says; 1
5 says; 1
1 now says; 16
2 now says; 1
5 now says; 4
6 now says; 5
7 now says; 6
8 now says; 7
9 now says; 8
10 now says; 9
13 now says; 12
3 now says; 6
11 now says; 10
12 now says; 11
14 now says; 13
15 now says; 14
16 now says; 15
4 now says; 5
1 finally says; 22
2 finally says; 5
3 finally says; 12
5 finally says; 16
7 finally says; 19
8 finally says; 19
9 finally says; 21
10 finally says; 23
11 finally says; 18
13 finally says; 26
4 finally says; 10
6 finally says; 13
14 finally says; 27
12 finally says; 27
15 finally says; 24
16 finally says; 30
```
Note that 3 says 6 early in the run proving that sync all did not stop thread 3 from reaching past the sync all before the rest of the threads.
Knarfnarf.
---
### Author:  [@FortranFan](https://fortran-lang.discourse.group/u/FortranFan)
#### Post date: [January 12, 2022, 4:56am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/2 "2022-01-12T04:56:34Z")
@Knarfnarf , welcome! Perhaps you want to try your program with another processor also to better understand the issue(s) with cafrun?
[**Intel oneAPI**](https://fortran-lang.discourse.group/t/intel-releases-oneapi-toolkit-free-fortran-2018/471) is available currently without requiring a commercial license.
---
### Author:  [@Euler-37](https://fortran-lang.discourse.group/u/Euler-37)
#### Post date: [January 12, 2022, 11:58am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/3 "2022-01-12T11:58:38Z")
> [@Knarfnarf](#):
>
> `print *, i_me, " now says;", i_test1 ! Did it get here?`
```auto
sync all
print *, i_me, " now says;", i_test1 ! Did it get here?
sync all
```
And two `sync all`,maybe it is a bug for open-coarrays.The result of `ifort` is right.
---
### Author:  [@themos](https://fortran-lang.discourse.group/u/themos)
#### Post date: [January 12, 2022, 12:19pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/4 "2022-01-12T12:19:36Z")
I think there are two errors: One is the placement of the first sync all and the other is non-use of atomic intrinsic atomic\_add for the +1 operation.
Your first sync does nothing useful. Your second is before the “now says”. As soon as images get past that sync, some other image can change some image’s i\_test1 (by rushing ahead to i\_loop1) so the print shows the changed value.
Try only having “sync all” around the last two prints (the final one can is not needed, of course) and call atomic\_add.
---
### Author:  [@everythingfunctional](https://fortran-lang.discourse.group/u/everythingfunctional)
#### Post date: [January 12, 2022, 9:51pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/5 "2022-01-12T21:51:22Z")
> [@Knarfnarf](#):
>
> ```auto
> do i_loop1 = 1, i_all ! Add a dynamic value to it.
> i_test1[i_loop1] = i_test1[i_loop1] + 1
> end do
>
> ```
This is your problem. You’re having each image update every value, with no synchronization and not atomically. Thus, some values may not be correctly updated.
Also, since there is no barrier between this loop and the previous print statement, some images may be updating the value before other images have a chance to print it.
The suggestions from others should give some clues as to how to fix these problems.
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 13, 2022, 3:25am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/6 "2022-01-13T03:25:56Z")
WOW! That oneAPI was impossible to setup! Is there a smaller group of installs available? Something less than 50? Or maybe an install.sh to do it all automatically?
Knarfnarf
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 13, 2022, 3:43am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/7 "2022-01-13T03:43:09Z")
Well… The sync all’s do have a reason, the first one keeps the print messages from the front aligned, the second after rotating everything helps those changes stay aligned, and the last should be all we need to get the last prints to be correct… And adding a sync all in the middle of the loop actually did nothing.
But I’m curious; what’s the code for atomic\_add look like?
Knarfnarf
---
### Author:  [@FortranFan](https://fortran-lang.discourse.group/u/FortranFan)
#### Post date: [January 13, 2022, 3:49am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/8 "2022-01-13T03:49:59Z")
> [@Knarfnarf](#):
>
> … That oneAPI was impossible to setup! Is there a smaller group of installs available?..
If your interest is only in Fortran, you can try proceeding with just Intel’s HPC toolkit, that shouldn’t be too difficult to setup and give you the IFORT compiler (and IFX, a **beta** version of Intel’s LLVM-based compiler): [Download the Intel® HPC Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/hpc-toolkit-download.html)
---
### Author:  [@themos](https://fortran-lang.discourse.group/u/themos)
#### Post date: [January 13, 2022, 10:11am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/9 "2022-01-13T10:11:21Z")
> **code**
>
> ```auto
> program Paratest
> use iso_fortran_env , only: atomic_int_kind, lock_type
> implicit none
> integer(atomic_int_kind) :: i_test1[*]
> Type(lock_type) :: test_lock[*]
> integer :: i_loop1, i_loop2, i_me, i_all
>
> ! Set variable for work
> i_loop1 = 1
> i_loop2 = 2
> i_me = this_image()
> i_all = num_images()
>
> print *, i_me, "a says; ", i_loop1 ! Does printing work? Good...
> ! sync all ! original, useless because of next sync
> i_loop2 = i_me + 1 ! Can we access the coarray?
> if (i_loop2 .gt. i_all) then
> i_loop2 = 1
> end if
> i_test1[i_loop2] = i_me ! Put our number over one.
> sync all ! original, must wait for i_test1 to be updated by other image
> print *, i_me, "b says;", i_test1 ! Did it get here?
> sync all ! needed, stops other images rushing ahead and changing
> ! i_test1 before we print it above
> do i_loop1 = 1, i_all ! Add a dynamic value to it.
>
> !!$ !Alt critical
> !!$ critical
> !!$ i_test1[i_loop1] = i_test1[i_loop1]+1
> !!$ end critical
>
> !!$ !Alt atomic
> !!$ call atomic_add( i_test1[i_loop1] ,1)
>
> !!$ !Alt locks
> !!$ Lock(test_lock[i_loop1])
> !!$ i_test1[i_loop1] = i_test1[i_loop1]+1
> !!$ Unlock(test_lock[i_loop1])
>
> !!$ !Alt race condition
> !!$ i_test1[i_loop1] = i_test1[i_loop1]+1
>
> end do ! Equiv to i_test1 += i_all
> sync all ! original, needed, otherwise i_test1 will be printed
> ! before all images have finished updating it
> print *, i_me, "c says;", i_test1 ! See the failure...
> end program Paratest
>
> ```
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 13, 2022, 6:57pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/10 "2022-01-13T18:57:59Z")
The website for oneAPI does look great, but the instructions are not download/install/done… If I can’t get this to work any other way I’ll look into it, but it did seem to require a rather large amount of requirements and dependencies. As it is, I think I’ve got the WSL and cygwin compiling/running now and they both have the same behavior. Obviously it is my code that is flawed, just how to fix it will be the issue.
Knarfnarf
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 13, 2022, 7:01pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/11 "2022-01-13T19:01:21Z")
Interesting!
So am I correct in seeing 3 solutions there along with the original code?
Knarfnarf
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 13, 2022, 7:09pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/12 "2022-01-13T19:09:42Z")
So, not sure why this comment was erased, but; redoing it.
My answer seems to be to add a sync all after the second round of printing to stop the early error in the second round of testing, and to use atomic\_add statements to perform the changes for the last round of testing. This has worked perfectly.
Due to issues cutting and pasting to this forum, I’ll add the code as another reply.
Knarfnarf
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 13, 2022, 7:16pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/13 "2022-01-13T19:16:06Z")
```auto
! Program Paratest
! Written by Frank Meyer
! Created Jan 9, 2022
! Version 0.2a
! Description: Testing some sync issues.
program Paratest
implicit none
! Create co-arrays for testing
integer :: i_test1[*]
! Other variables for testing purposes.
integer :: i_loop1, i_loop2, i_me, i_all, i_locked
integer :: i_images(2)
! Set variable for work
i_loop1 = 1
i_loop2 = 2
i_me = this_image()
i_all = num_images()
i_images(1) = i_me
print *, i_me, " says; ", i_loop1 ! Does printing work?
sync all
i_loop2 = i_me + 1 ! Can we access the coarray?
if (i_loop2 .gt. i_all) then
i_loop2 = 1
end if
i_test1[i_loop2] = i_me ! Put our number over one.
sync all
print *, i_me, " now says;", i_test1 ! Did it get here?
sync all
do i_loop1 = 1, i_all ! Add a dynamic value to it.
call atomic_add(i_test1[i_loop1], 1)
end do ! Equiv to i_test1 += i_me
sync all
print *, i_me, " finally says;", i_test1 ! See the Success (values = i_me-1+16)...
end program Paratest
```
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 16, 2022, 6:52am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/14 "2022-01-16T06:52:29Z")
For another task, it looks like the third method might work, if it does then that is the last key to the puzzle! Interesting that you test the lock on a non-critical variable to prove the ability to reach the critical one… I will have to study this!
Knarfnarf
---
### Author:  [@themos](https://fortran-lang.discourse.group/u/themos)
#### Post date: [January 16, 2022, 9:56am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/15 "2022-01-16T09:56:09Z")
“Critical protects code, locks protect variables” is what to remember.
There is no “critical variable” or “non-critical variable”.
Locks are more fine-grained and offer more control (you can do other things while waiting for a lock).
But EVENTs is where the performance is and should be the first to try.
---
### Author:  [@msz59](https://fortran-lang.discourse.group/u/msz59)
#### Post date: [January 16, 2022, 12:37pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/16 "2022-01-16T12:37:13Z")
> [@Knarfnarf](#):
>
> WOW! That oneAPI was impossible to setup! Is there a smaller group of installs available?
I have no experience with Windows/WSL, so I do not know how linux software is installed there. In _regular_ Linux environment, both RHEL-like and Ubuntu/Debian-like, you can add an Intel oneAPI repository to teh system and then try to install just the fortran compiler. All needed extra packages will be automatically installed as dependencies.
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [January 17, 2022, 12:58am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/17 "2022-01-17T00:58:46Z")
Yeah… I’m running this in windows right now and WOW! to get CAF installed was VERY confusing. It looked like I needed the core components (with dependencies) then at least two other products (with different dependencies)… The website LOOKED nice, but was very unclear as to what did or did not have CAF or was compatible to what I was looking for. Your mileage my vary.
Knarfnarf
---
### Author:  [@msz59](https://fortran-lang.discourse.group/u/msz59)
#### Post date: [January 17, 2022, 6:27pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/18 "2022-01-17T18:27:25Z")
CAF comes (used to come) with opencoarrays which are used for `gfortran`. With Intel’s oneAPI `ifort` you don’t need CAF.
You are right, however, that CAF has become hard to install on Linuxes. Fedora and other RHEL-like distros abandoned opencoarrays quite a while ago. Ubuntu had `open-coarrays-bin` package until 18.04 but in 20.04 it is not available anymore. Only `libcoarrays-*` packages left but you have to know how to compile to use them.
---
### Author:  [@Knarfnarf](https://fortran-lang.discourse.group/u/Knarfnarf)
#### Post date: [February 2, 2023, 4:08am UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/19 "2023-02-02T04:08:44Z")
As it turns out; I’ve have very good success with caf under WSL (a few issues with the version of Ubuntu that gets installed missing… everything! … and needing to be pointed to a better repository, updated, upgraded, updated, and upgraded AGAIN!). The install of CAF was instantaneous once I started using HomeBrew!
But the issue here is that CAF does NOT push to other images reliably. You CAN pull from another image with good reliability so long as you do so with a type that has a long integer as an iteration counter to compare with the iteration of this image. That way you can be sure that the data being pulled is good.
I have a parrallel sort program that does this and it’s proven to work reliably.
Knarfnarf
---
### Author:  [@Federchen](https://fortran-lang.discourse.group/u/Federchen)
#### Post date: [February 3, 2023, 3:39pm UTC](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565/20 "2023-02-03T15:39:32Z")
> But the issue here is that CAF does NOT push to other images reliably.
If I understand you correctly, then see that document on page 26 for coarray get and put operations:
[https://doku.lrz.de/download/attachments/43320950/Advanced\_Fortran\_Coarrays.pdf](https://doku.lrz.de/download/attachments/43320950/Advanced_Fortran_Coarrays.pdf) .
From my own experiences, ‘unreliable’ data transfer through coarray put operations are common if the SPMD coarray runtime (i.e. the runtime of the current coarray team) got ‘destroyed’ by violating the execution segment ordering in that coarray team, i.e. some programming mistake with a synchronization or atomic operation. Any following data transfer or synchronization may or may not succeed, the ‘destroyed’ runtime becomes unreliable.
The only way, that I am aware of, to recover from a ‘destroyed’ coarray runtime is to leave the current coarray team (END TEAM) and then to re-enter (CHANGE TEAM) the coarray team, if this is still possible. Of course this makes no sense if the execution segment ordering is violated again and again, it should not be violated at all.
[Next page](https://fortran-lang.discourse.group/t/problems-with-coarrays/2565.md?page=2)