# Coarrays: control over the number of images

**URL:** <https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632>\
**Category:** Uncategorized\
**Created:** [September 20, 2024, 9:44am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632 "2024-09-20T09:44:46Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 20, 2024, 9:44am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/1 "2024-09-20T09:44:46Z")

</div>

I am experimenting with coarrays for an actual application, rather than getting a feeling for what you can do with them, and I am facing a particular question regarding the number of images:

For some algorithms the number of images merely means that the work is divided over whatever number of images is available. In this case I want each image to take care of a single subregion in my complete model region. But these subregions are determined outside my particular program.

If I start my program using all defaults, the number of images may be 24, whereas I only need 3.  
I can control the number via environment variables or via compile time options, but what do I do with extra images, like when I forget to specify that my program needs only 3 images for the case at hand and I still get 24? Should I stop the remaining 21 or make sure they occupy very little memory and do not do anything except wait?

What is a “good” way to handle this situation?

---

<div class="post-metadata">

**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\
**Post date:** [September 20, 2024, 11:14am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/2 "2024-09-20T11:14:56Z")

</div>

A naive suggestion could be to abort the program:

```fortran
if (num_images() /= 3) error stop "abort: more images than necessary."

```

This would force the calling process (or person that invoked it) to adjust the number of images beforehand.

Another idea would be to group your images into teams, and have the other team do something else at the time. I don’t know if stopping one team would “return” them into the pool of available system resources, or if the members of that team would wait in front of an (implicit) barrier at the `end program`/`stop` statement until the rest of images are complete.

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 20, 2024, 11:25am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/3 "2024-09-20T11:25:27Z")

</div>

I thought of that naïve suggestion, but that assumes that the user starts the program themselves (interactively) and knows how to set the number of images. It is definitely a possibility and simple.

I had not thought about teams though, it is worthwhile to explore that possibility!

---

<div class="post-metadata">

**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\
**Post date:** [September 20, 2024, 11:30am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/4 "2024-09-20T11:30:43Z")

</div>

Using environment variables you can in principle fine-tune the process pinning: [https://www.intel.com/content/www/us/en/developer/articles/technical/distributed-memory-coarray-fortran-with-the-intel-fortran-compiler-for-linux-essential.html](https://www.intel.com/content/www/us/en/developer/articles/technical/distributed-memory-coarray-fortran-with-the-intel-fortran-compiler-for-linux-essential.html)

I guess you’ll need to expose these “levers” in a wrapper shell script, that doesn’t involve the user knowing anything about CAF or the specific implementation settings.

> **Example of a wrapper script**
>
> ⚠ This code was generated by ChatGPT and has not been tested. ⚠
> 
> Save the block below as `wrapper.sh`. Assumes the CAF executable was created using the Intel Fortran compiler
> 
> ```auto
> #!/bin/bash
> 
> # Default value for --num_images
> NUM_IMAGES=3
> 
> # Parse arguments
> for arg in "$@"; do
> if [[$arg == --num_images=*]]; then
> NUM_IMAGES="${arg#*=}"
> else
> OTHER_ARGS+="$arg "
> fi
> done
> 
> # Check if NUM_IMAGES is a valid number and greater than or equal to 3
> if ((NUM_IMAGES < 3)); then
> echo "Error: --num_images cannot be less than 3." >&2
> exit 1
> fi
> 
> # Set the environment variable and run ./caf_fortran with the other arguments
> FOR_COARRAY_NUM_IMAGES="$NUM_IMAGES" ./caf_fortran $OTHER_ARGS
> 
> ```

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 20, 2024, 1:23pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/5 "2024-09-20T13:23:39Z")

</div>

We use all manner of scripts, so that is not a big deal. But I do want to hide switches that do not really matter to the user.

---

<div class="post-metadata">

**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\
**Post date:** [September 20, 2024, 1:57pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/6 "2024-09-20T13:57:37Z")

</div>

From what I understood, your application has a kind of domain decomposition approach to dealing with parallelism which is divided in a pre-processing step and then your application which would be the “processing step” right? If so, I guess that looking at this in the same manner as you would an MPI application would be the safest. So, better to stick with ensuring that the pre- and processing steps are setup with the same amount of images (processes in the MPI jargon). If you do not want the user to mingle with this setting for your processing step, could you insert a scripting approach that detects how many domains were used for the decomposition before running your program? and then pass that to `cafrun` … this is at least what I would try to keep some sanity in my mind 🙂

---

<div class="post-metadata">

**Author:** ![everythingfunctional](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/everythingfunctional/32/176_2.png) [@everythingfunctional](https://fortran-lang.discourse.group/u/everythingfunctional)\
**Post date:** [September 20, 2024, 1:58pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/7 "2024-09-20T13:58:52Z")

</div>

Teams actually seem like a pretty convenient way to address this problem, as the rest of the code doesn’t need to worry about getting the unused images to participate in any collective operations. I.e. some psuedo-code

```auto
I_participate = participates(this_image())
team_number = merge(1, 2, I_participate)
form team (team_number, my_team)
if (I_participate) then
  change team (my_team)
    call do_the_work
  end team
end if
end

```

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 20, 2024, 2:13pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/8 "2024-09-20T14:13:08Z")

</div>

Yes, it is a domain-decomposition problem. And as Brad illustrated, using teams is probably the most convenient way. Thanks, everyone!

---

<div class="post-metadata">

**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\
**Post date:** [September 20, 2024, 2:24pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/9 "2024-09-20T14:24:53Z")

</div>

Very curious to know about your experience down the road with coarray and domain decomposition 🙂 … So far I’ve just played around with coarray but haven’t used it seriously.

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 20, 2024, 2:37pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/10 "2024-09-20T14:37:02Z")

</div>

Well, my first attempts worked fine, but they were very simple. I am now experimenting with a program that actually reads the partitioning (and the connections between the partitions) from input files. That will be close to the actual application and I can certainly post about this.

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 26, 2024, 9:11am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/11 "2024-09-26T09:11:29Z")

</div>

I have made progress:

- The program works, in the sense that the results look plausible
- I have for now avoided the use of teams. Instead, it demands the right number of images.
- I want to write down my experiences, but that will take a wee bit of time 🙂

Of course my test case/demo is far too small and simple to see whether the performance has improved, but that is of secondary importance.

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 27, 2024, 6:25am UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/12 "2024-09-27T06:25:53Z")

</div>

Actually, adding teams to avoid setting the number of images to use was quite easy.

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [September 30, 2024, 6:39pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/13 "2024-09-30T18:39:24Z")

</div>

FYI: I have described my experiment here: [memos-on-programming/doc at main · arjenmarkus/memos-on-programming · GitHub](https://github.com/arjenmarkus/memos-on-programming/tree/main/doc). The repository also contains the program itself and the input files.

---

<div class="post-metadata">

**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\
**Post date:** [October 16, 2024, 3:04pm UTC](https://fortran-lang.discourse.group/t/coarrays-control-over-the-number-of-images/8632/14 "2024-10-16T15:04:33Z")

</div>

Meanwhile I used some procrastination to create a version of the program that uses MPI instead of coarrays. That took me a lot more thinking, partly to understand the MPI functions/subroutines, partly because there is a confusing aspect in my example, but I blame it partly on MPI itself. Anyway, to my surprise it gives exactly the same results as the coarray version. Which I chose to take to be a good omen.
