Recommended way to use LFortran with mpi

Compilers LFortran
1

Hi,

I would like to try compiling an existing fairly large MPI-based scientific Fortran code with LFortran. The code currently uses use mpi with standard point-to-point and collective operations.

What is the recommended way to use MPI with LFortran today?

I found lfortran/fortran_mpi and the earlier POT3D example. Is fortran_mpi still the recommended approach, and is it sufficiently complete to serve as a mostly drop-in replacement for use mpi in an existing code?

Alternatively, is it now possible/recommended to use LFortran directly with an existing OpenMPI installation, or to build OpenMPI itself with LFortran?

Thanks!

1 Like
2

You need to build a version of MPI such as MPICH or Open MPI and tell it to use LFortran to build the Fortran interfaces and modules . I think a gfortran module might also work so if you have access to a prebuilt version that uses gfortran to build the mpi and mpi_08 modules. I’ll let @certik correct me if I’m wrong.

2 Likes
3

So lfortran/fortran_mpi should work and we can easily fix it for any bugs that you might discover. I think that’s the cleanest way to call MPI from Fortran, as it creates very thin wrappers and calls the C MPI directly, so you install whatever C MPI you want, and then just compile your Fortran code with lfortran/fortran_mpi and things should just work.

If you do not want this approach, then we can also work on getting LFortran working with mpich and openmpi directly, which I think we have all the features in LFortran to support that, so we just need to try it and fix any bugs that we discover (I think we haven’t tried it yet).

@rsci if your code is open source, I can have a look and fix all bugs. If it is not, then simply report all bugs into our issue tracker: Issues · lfortran/lfortran · GitHub and ping me and we’ll take care of it. If your project is large, almost for sure you’ll discover some bugs, so just report them and we’ll fix them.

2 Likes
4

Thanks a lot! I’ll givelfortran/fortran_mpi a try first then and see how far I get.

1 Like
5

A small progress update.

I’ve been testing LFortran 0.64.0 with lfortran/fortran_mpi, using Open MPI through its C interface. So far, a simple MPI program works, and several small tests within the codebase are compiling and passing, including MPI tests with multiple ranks.

The main limitation on the MPI side at the moment is interface coverage in fortran_mpi: a number of standard interfaces are still missing, including routines such as MPI_Abort and MPI_Send, some standard MPI datatypes, and overloads for commonly used collectives. I’ve been adding the missing wrappers as needed.

I’ve also encountered a few compiler issues. So far I have reduced two of them to small standalone reproducers.

1. dot_product on a derived-type component section with --separate-compilation

module repro
  implicit none
  type :: vector_t
    integer :: x(1)
  end type
contains
  subroutine transform_index(a, result)
    type(vector_t), intent(in) :: a
    integer, intent(out) :: result
    result = dot_product(a%x(:), a%x)
  end subroutine
end module

With

lfortran --separate-compilation -c repro.f90

LFortran segfaults during compilation, while GFortran accepts it. dot_product(a%x, a%x) and sum(a%x(:)) both compile successfully. Without --separate-compilation, the module itself compiles, although I encountered the crash later when compiling a program using the routine.

2. Imported constant used to initialize a same-named component

module constants
  implicit none
  real, parameter :: value = 42.0
end module

module profile
  use constants, only: value
  implicit none

  type :: profile_t
    real :: value = value
  end type
end module

LFortran reports

semantic error: Initialization of `value` must reduce to a compile time constant.

while GFortran accepts it. Aliasing the imported constant, for example default_value => value, and using that in the initializer works around the problem.

6

@rsci beautiful, I can reproduce all 3 bugs!

Note: use the latest release (currently 0.66.0), as a lot of bugs are fixed. I will fix the above quickly. Regarding the fortran_mpi issue, you already added those missing interfaces? If so, can you send us a PR? Otherwise I’ll do it.

7

Thanks! I’ve submitted a PR with some of the missing interfaces (with quite a bit of help from Codex), and I’ll probably need to add more as I progress with the compilation.

8

I see it: Add MPI interfaces for scalar communication and communicator splitting by ggoyman · Pull Request #139 · lfortran/fortran_mpi · GitHub, thanks!

9

Both of these are fixed now.

10

A quick progress update.

I updated LFortran and made another attempt at compiling the code. I managed to get quite a bit further this time, encountering what seem to be a few compiler bugs along the way (sorry for the issue spam!):

At this point I think the next compilation pass may actually be the final one, or at least quite close.

One question is about default real precision. The production builds rely on compiler-wide promotion of default REAL to 64 bit (-r8 with Intel and -s real64 with Cray; GFortran has -fdefault-real-8). I don’t see an equivalent working option in LFortran at the moment. Is there already another way to achieve the same semantics? If not, how difficult would it be to implement support for this?

On the MPI side, it also looks like quite a bit more of the Fortran interface still needs to be implemented in fortran_mpi. I’m thinking of trying the other route you mentioned earlier: building the Open MPI Fortran bindings with LFortran.

11

Those are beautiful bug reports, thanks for that. I am already fixing all of them. I’ll ping you when it’s done.

Looks like we have an issue for that already: [feature request] add -fdefault-real-8 and -fdefault-double-8 · Issue #11500 · lfortran/lfortran · GitHub

We already support promoting integers, so we just need to do it for real also. I don’t think it’s inherently difficult, probably just a lot of corner cases.

I recently added support for LFortran, GFortran, Flang and Ifx: CI: run gfortran, flang, ifx and lfortran with Open MPI and MPICH via pixi by certik · Pull Request #140 · lfortran/fortran_mpi · GitHub. They all use the same openmpi or mpich, and then compile the wrappers and run tests as well as the POT3D code in parallel. I really like this approach, I think it fixes all MPI issues in Fortran and it works for all compilers, no need to be building openmpi with every Fortran compiler that you want to use. Yes, we need to add more features.

But the other approach of getting openmpi/mpich itself working with LFortran we also have to eventually support, so if you want, report us all bugs there and we’ll fix it.

1 Like
12

I tried building Open MPI 5.0.11 directly with the current LFortran main, and it actually got quite far. There were two main obstacles:

  1. Open MPI’s configure test for the size of Fortran CHARACTER returned 16 instead of 1. I reduced this to an LFortran issue and reported it here:

The underlying CHARACTER array layout is actually correct (C_LOC gives a one-byte stride), but CHARACTER array elements are passed incorrectly through an implicitly interfaced external call. For now I worked around the configure test with ompi_cv_fortran_sizeof_CHARACTER=1.

  1. The shared build gets through compilation but fails when libtool invokes LFortran as the linker driver with options such as -shared, -rpath, -soname, etc. The LFortran-generated objects themselves can be linked into a shared library with GCC. As a workaround, I configured Open MPI with --disable-shared --enable-static.

With these two workarounds, Open MPI 5.0.11 builds and installs successfully.

I then compiled a small MPI smoke test with LFortran and linked the final executable with GCC. I tested integer and real broadcasts, blocking and nonblocking send/receive, MPI_Waitall, MPI_Allreduce, MPI_Reduce, MPI_Barrier, MPI_Comm_split, communicator freeing, and status handling.

The one expected failure is MPI_CHARACTER: a character broadcast returns from MPI_Bcast, but the receiving rank crashes when the received string is subsequently used. This looks consistent with #14153.

So at this point the native Open MPI route looks quite promising. For non-character communication, the use mpi bindings already seem usable.

2 Likes
13

@rsci I have all your issues above fixed in main except four, which have all PRs submitted, just waiting for the CI. Hopefully by tomorrow everything will be in.

Regarding your OpenMPI bug, it’s actually a bug in OpenMPI itself which uses non-conforming Fortran code and relies on internal ABI of Fortran compilers. I recently submitted a fix to NetCDF fixing exactly this issue: Fix non-conforming type mismatches at character(kind=c_char) void * arguments by certik · Pull Request #492 · Unidata/netcdf-fortran · GitHub and we should submit a similar fix to OpenMPI. They should not rely on internal ABI, but rather strictly use bind(c) which is guaranteed to have the same C ABI across all compilers. But in the meantime, it looks like the ABI that they want is actually better than the current one we have in LFortran, so I am updating it to use the better one, but I do not want to commit to it, I want to keep the freedom to change internal ABI as we see fit, for performance reasons. But at least for now this should unblock you.

Overall great work, hopefully we’ll get everything fixed and we’ll see how far you can take it. I think there will also be a bunch of runtime bugs also, but just report them all and we’ll fix them quickly.

14

Andrej, Can you explain what you mean by an internal ABI. I thought OpenMPI implemented BIND(C) interfaces. I believe that is what the mpi_08 module brings in. In fact that was the main driver for the C-interop extensions in Fortran 2008 wasn’t it. As to NetCDF. I wrote the original C-Interop interfaces for NetCDF but gave the code to Unidata/UCAR with the caveat that I wouldn’t be involved in direct support (was too busy with other things at the time). I haven’t looked at the code in about 10 years so your fix will probably prompt me to go back and look at the code again :grinning_face_with_smiling_eyes:. When I wrote the interfaces C-interop was in its very early years (2005 - 2010) and some things where not implemented correctly in several compilers so I had to work around some things that weren’t supported until later version of Fortran. Also, I was trying to duplicate the interfaces defined by the C wrappers generated by the old cfortrah.h macros and support Fortran 77 along with Fortran 90/95.

1 Like
15

@rwmsu yes, look at this issue CHARACTER array elements are passed incorrectly to an implicitly interfaced external procedure · Issue #14153 · lfortran/lfortran · GitHub, the reproducer there uses external check and then it calls it directly from C using a certain ABI. That is not conforming, as the check function is not bind(c) and doesn’t have an interface. So you can’t assume a particular ABI.

1 Like
16

@rsci all your issues are now fixed except CHARACTER array elements are passed incorrectly to an implicitly interfaced external procedure · Issue #14153 · lfortran/lfortran · GitHub (I am still working on it).

17

The convention of passing the character string addresses and appending the ‘hidden’ length arguments at the end has been around since the first f77 compiler from Bell Labs in 1978. Early on, a few compilers used different conventions. But pretty much everyone does it the Bell Labs way now. Though you have to watch out for the size of the hidden length. On some compilers it has been int or long - as that was all that was needed with 32-bit ABIs. But nowadays should really be size_t.

Another detail is when optional value arguments are present. The convention is to have a hidden boolean argument for each of these that needs to follow all the hidden character string lengths.

1 Like
18

Thanks @wspector. Yes, I am going to use the same convention, to be compatible. But I think no code should depend on it, my understanding is that bind(c) is sufficient for everything.

19

I think bind(c) requires an explicit interface, right? There is no portable way for a fortran program to access the bind(c) name without an explicit interface to do that translation.

20

Yes. Is there any reason why OpenMPI cannot supply these explicit bind(c) interfaces?