# Fortran coarray shared memory

**URL:** <https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178>\
**Category:** Help\
**Created:** [May 8, 2021, 3:15am UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178 "2021-05-08T03:15:54Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![haomiao](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/haomiao/32/546_2.png) [@haomiao](https://fortran-lang.discourse.group/u/haomiao)\
**Post date:** [May 8, 2021, 3:15am UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/1 "2021-05-08T03:15:54Z")

</div>

How to implement shared memory in fortran coarray? It’s to me that every image have a copy of local variable. Although fortran is a global address space language (PGAS) which means that one image is easy to access the data of other images. As to me, this is valid only for fortran coarrays. How about the local variable?  
In the source code of neural-fortran [https://github.com/modern-fortran/neural-fortran](https://github.com/modern-fortran/neural-fortran) and the subrotuine of `train_batch` , there is a code snippet

```fortran
    real(rk), intent(in) :: x(:,:), y(:,:), eta

    indices = tile_indices(im)
    is = indices(1)
    ie = indices(2)

    call db_init(db_batch, self % dims)
    call dw_init(dw_batch, self % dims)

    do concurrent(i = is:ie)
      call self % fwdprop(x(:,i))
      call self % backprop(y(:,i), dw, db)
      do concurrent(n = 1:nm)
        dw_batch(n) % array = dw_batch(n) % array + dw(n) % array
        db_batch(n) % array = db_batch(n) % array + db(n) % array
      end do
    end do

```

It’s clearly that the `is` and `ie` are the index of data for present image. But we simply use the `x(:,i)`, `y(x:i)` to access the data which are not coarray by the definition. Does it means that every image has a copy of all the x and y. If so, when the number of image is incrased, the memory will blow up ?

In the paper of “A parallel Fortran framework for neural networksand deep learning”, the author says that

> Elapsed times on up to 12 parallel images on a shared-memory system are shown in Figure 4.

What’s the meaning of shared-memory system? I do not get the meaning of shared-memory as every image has a copy of the dataset.

To be clear, does fortran coarray supports shared memory ? If yes, how to implement the shared memory code?

---

<div class="post-metadata">

**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\
**Post date:** [May 8, 2021, 4:09am UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/2 "2021-05-08T04:09:15Z")

</div>

Welcome @haomiao!

In the paper, by “shared-memory system” I mean a single-node, multi-core computer. It means that the processes (images) physically share RAM, but it doesn’t mean that each image has a copy of every other image’s data.

The importance of mentioning the “shared-memory system” in the paper was to make note that there is no communication through the network.

From the Fortran code point of view, it makes no difference if you’re running on a single-node or multi-node machine. For example, running a program on 8 images on one 8-core machine, vs. 2 4-core machines, should yield the same results (assuming no race conditions) and memory use by each image.

Note that neural-fortran doesn’t use coarrays but only collective subroutines–`co_sum` and `co_broadcast`. Each image allocates a different section of the array (`is:ie`). Decomposition is done by the library (in `tile_indices`), but the communication is done by the compiler or a coarrays-implementing library.

---

<div class="post-metadata">

**Author:** ![haomiao](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/haomiao/32/546_2.png) [@haomiao](https://fortran-lang.discourse.group/u/haomiao)\
**Post date:** [May 8, 2021, 7:27am UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/3 "2021-05-08T07:27:52Z")

</div>

Thanks @milancurcic  
For example, when I run the following code

```fortran
program main
    integer :: i
    integer :: a(4) = [1,2,3,4]
    do i = 1, 4
        if(this_image() == i) a(1) = i*2
    end do
    print*, "process index: ", this_image(), a
end program main

```

and run `cafrun -n 4 ./a.out`  
The results are

> process index: 4 8 2 3 4  
> process index: 1 2 2 3 4  
> process index: 2 4 2 3 4  
> process index: 3 6 2 3 4

clearly there are four copies of `a`.

Just like the `neural-fortran` to declare `x(:,:)` , there is no difference to me as I declare the `a` is the local array. In my opinion, there is a copy of `x(:,:)` in other images.

---

<div class="post-metadata">

**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\
**Post date:** [May 8, 2021, 2:36pm UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/4 "2021-05-08T14:36:37Z")

</div>

You’re right, all images have a local copy of all variables, no matter if they’re declared as coarray or not. The allocatable arrays (non-coarrays) don’t have to be allocated on all images, and they don’t need to have the same extent (lower and upper bounds). Coarrays do need to be allocated with the same lower and upper bounds on all images, and on all images at the same time (allocating a coarray is a blocking operation).

Back to neural-fortran, I looked at how it’s implemented and it seems like all images have a copy of the full input arrays `x` and `y`. Each image only trains on its own portion of the workload, after which the weights and biases are globally updated using `co_sum`. Of course, this is not optimal for memory use, but allows a very simple high-level API which uses exactly the same code for serial and parallel execution. In other words, the parallel work distribution is managed inside of `network_type % train_batch()`, and not outside of it.

A more optimal approach memory-wise would be for each image to read only its portion of the data, and call `network_type % fwdprop()`, `network_type % backprop()`, and `network_type % update()` directly. You’d need to take care of data exchange yourself, but it’s simple enough, you can just follow how that’s implemented in `network_type % train_batch()`.

---

<div class="post-metadata">

**Author:** ![billlong](https://avatars.discourse-cdn.com/v4/letter/b/71e660/32.png) [@billlong](https://fortran-lang.discourse.group/u/billlong)\
**Post date:** [May 8, 2021, 9:25pm UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/5 "2021-05-08T21:25:28Z")

</div>

Each image has its own variables, which is true for both coarrays and noncarrays. The difference with coarrays is that there is a corresponding coarray on every image that can be accessed from a different image. The method used for addressing coarrays on other images depends on the hardware and OS support. For Linux-based systems (common in HPC) there is an OS plug-in called XPMEM that allows a process to access memory in a different process in the same shared-memory space. For the case of a single node with many cores, and one image for each core, each image is a separate process, and memory addressing to the remote process can be done through XPMEM. (The addressing possibility is available in the hardware - XPMEM mainly enforces memory space security between processes.) Within a node memory can be accessed without employing the systems internal network. There have been hardware designs that had true global addressing (the upper bits of the memory address were the image number-1 based on the initial image, and routing tables were built into the unified memory and network hardware. However, these machines are (were) pretty rare because of the cost. But the performance on distributed memory coarray codes was pretty spectacular. Typically there was only 1 process/node so all codes with more than one image were distributed memory. Remote accesses did not involve a library call - the compiler just generated inline code to stuff the correct bits into the upper part of the address and a simple load or store instruction was executed.

---

<div class="post-metadata">

**Author:** ![FortranFan](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@FortranFan](https://fortran-lang.discourse.group/u/FortranFan)\
**Post date:** [May 8, 2021, 10:59pm UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/6 "2021-05-08T22:59:56Z")

</div>

> [@haomiao](#):
>
> Does it means that every image has a copy of all the x and y. If so, when the number of image is incrased, the memory will blow up ? …

@haomiao ,

Welcome to this forum.

Re: “when the number of image is incrased, the memory will blow up ?” is your concern that each image having its own copy of some 'large" memory dataset will cause an issue?

By the way, note the `ALLOCATABLE` attribute and how the memory usage can be controlled depending on the needs across images:

```Fortran
   integer, allocatable :: a(:)
   if ( this_image() == 1 ) then
      a = [1, 2, 3, 4]
   end if
   print *, "On image ", this_image(), "; allocated(a)? ", allocated(a)
   if ( allocated(a) ) print *, "a = ", a 
end

```

> On image 3 ; allocated(a)? F  
> On image 2 ; allocated(a)? F  
> On image 4 ; allocated(a)? F  
> On image 1 ; allocated(a)? T  
> a = 1 2 3 4

> [@haomiao](#):
>
> … does fortran coarray supports shared memory ? If yes, how to implement the shared memory code?

Re: “does fortran coarray supports shared memory?” do you mean the use of some “global” data that can be accessed (shared) the same on all the images?

---

<div class="post-metadata">

**Author:** ![haomiao](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/haomiao/32/546_2.png) [@haomiao](https://fortran-lang.discourse.group/u/haomiao)\
**Post date:** [May 9, 2021, 12:44am UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/7 "2021-05-09T00:44:32Z")

</div>

@FortranFan  
Yes, it’s exactly what you have said. How could we use “global” data that can be shared on all images if we don’t allocate the variable on each image. As I think, the `OPENMP` could do this easily, and `MPI` could allocate the shared memory. But how about the `coarray`? We could allocate `a` in image 1, but how other images get the `a` if we do not allocate `a` on them?

---

<div class="post-metadata">

**Author:** ![themos](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/themos/32/521_2.png) [@themos](https://fortran-lang.discourse.group/u/themos)\
**Post date:** [May 9, 2021, 7:22am UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/8 "2021-05-09T07:22:26Z")

</div>

They always reference a[1]?

---

<div class="post-metadata">

**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\
**Post date:** [May 9, 2021, 4:48pm UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/9 "2021-05-09T16:48:25Z")

</div>

> [@haomiao](#):
>
> But how about the `coarray` ? We could allocate `a` in image 1, but how other images get the `a` if we do not allocate `a` on them?

You can’t allocate a coarray on some of the images and not on others. It has to happen on all images. Or at least on all images on the same team, IIRC.

---

<div class="post-metadata">

**Author:** ![FortranFan](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@FortranFan](https://fortran-lang.discourse.group/u/FortranFan)\
**Post date:** [May 9, 2021, 6:46pm UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/10 "2021-05-09T18:46:25Z")

</div>

> [@haomiao](#):
>
> Yes, it’s exactly what you have said. How could we use “global” data that can be shared on all images if we don’t allocate the variable on each image. …

That’s what I suspected and that’s why I asked to confirm.

Re: “How could we use “global” data that can be shared on all images if we don’t allocate the variable on each image,” to the best of my knowledge, the Fortran `coarray` model does not allow what you seek,

What you are asking appears more like a threading approach with multiple threads and single data model. Fortran `coarray` is not that.

---

<div class="post-metadata">

**Author:** ![everythingfunctional](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/everythingfunctional/32/176_2.png) [@everythingfunctional](https://fortran-lang.discourse.group/u/everythingfunctional)\
**Post date:** [May 10, 2021, 11:46pm UTC](https://fortran-lang.discourse.group/t/fortran-coarray-shared-memory/1178/11 "2021-05-10T23:46:36Z")

</div>

That’s more or less what coarrays are. A coarray must have (or be allocated to) the same shape and size on every image (at least within a team), but any image can access any image’s data directly.

However, with collective subroutines, one can share data between images without using a coarray. I.e. `co_broadcast` can send the data from one image to all the others. `co_sum` can gives you the total from all images.

There are a variety of techniques possible with just the above features, and we haven’t even gotten into teams and `event_type` yet. But which technique/strategy you use depends heavily on the problem you’re trying to solve. “How do I have global data?” isn’t a specific enough question to give an appropriate answer, and it’s quite possible the answer to your question may be “having global data might not be a good design in this case”.