Coarrays: Not ready for prime time

66

@nncarlson

FYI

I was able to build a version of MPICH 4.0.2 using both gcc 11.2 and Intel oneAPI 2021.5 with the gforker process manager only plus the ch3:nemesis device which is suppose to default to shared memory on single node systems. I ran your index-map finite volume test case using the gcc 11 version of MPI along with an attempt (that failed) to run coarrays. The test system
was an AMD Ryzen 5 5600X (6 core) CPU, 64Gbytes DDR4 3200 memory (running at 3200) on a Linux Mint 20.3 (Ubuntu 20.04 LTS based - Kernel 5.4.0). The processor frequency on all cores is locked at 3600 Ghz (no frequency scaling).

The following are my GCC11 MPI times (as reported by your code)

1 2 4 6
56.09 37.08 30.05 24.25

Not sure why these are much lower than your values but these are
the numbers printed by your code.

I couldn’t get CAF to run for more than 1 process. Probably something related to how
I built openCoarrays. I’ll try Intel next.

Edit.

MPI times with ifort 2021.5 using MPICH 4.0.2 built with gforker and ch3:nemesis device

1 2 4 6
55.23 36.95 26.64 24.62

67

That’s most likely because you didn’t modify the parameter setting to NZ=513 (at the top of disk-fv-parallel.F90) and were using the default value of NZ=257. Sorry that I wasn’t clear about what I used in my post. Also your Ryzen 5600X is likely faster, core-to-core, than my 2920X which is getting a little old now.

68

O.K. MPICH 4.0 times with Intel and GCC 11 for NZ=513.

GCC
1       2        4         6
256.95  122.79   74.63     60.95

Intel
1       2        4         6
511.65  251.53   141.71    108,79

type or paste code here

Previous Intel times are actually GCC so disregard.

Interesting that Intel with the same compile options and MPICH version is slower that GCC by almost a factor of 2.

I tried running Intel CAF using MPICH 4.0 instead of Intel MPI and the run appeared to hang

69

@FortranFan I just noticed your message from 3 days ago. I’m stretched really thin these days so I’m not likely to do the evaluation on my own, but I’ve long had a virtual open-door policy in that I try to make myself available for interactive discussion and pair programming. It both helps me focus and feels more efficient because questions can be answered in real time. Next week, Wednesday and Thursday are best. The follow week, Thursday is best. My availability is publicly viewable at Google Calendar - Sign in to Access & Edit Your Schedule and I generally accept all calendar invitations.

70

@nncarlson. I haven’t tried rerunning GCC-CAF. Life got in the way. I’ll try again this weekend if I can find the time. I need to revisit my openCoarray’s build to make sure it’s actually using standard MPICH and not the Intel version. I think I needed to set MPI_HOME explicitly instead of relying on cmake to find the MPI version I wanted to use. Thats the problem when you have multiple MPI versions on one system. Always a chance of your application grabbing the wrong one.