Interfacing Fortran code from Python

Help
1

I’m a Python programmer looking to Fortran as a way to speed up code that couldn’t be handled by array operations with Numpy and SciPy. (There are of course many other ways you can do this but I was also motivated to learn Fortran to understand legacy code in my field, computational chemistry). My goal would be to include Fortran code under the hood to handle slow operations for my Python packages. This code could be packaged for example by having separate Fortran packages included with git submodules and then compiling platform specific packages and supplying them with conda/pip. I’ve explored a number of different alternatives of calling Fortran code from Python, all with their pros and cons.

Explicit C inteface
Coding explicit C interface in Fortran and calling that with Python libraries like ctypes, cffi etc. This is perfectly general but also quite annoying on the Fortran side in terms of having to write and maintain a separate API. The coding on the Python side is also annoying in terms of having to keep track of dimensions of the arrays sent back and forth etc. The implementations I have seen often uses a two-step procedure on the Python side, with a low-level interface and on top of that a more Pythonic one for the end user of the package.

Automatic C interface generation
Here some code generator takes care of writing the interface automatically, with alternatives like f2py, fmodpy etc. This is very convenient as you don’t have to write a separate API and the generated Python wrappers also take care of many things under the hood so that the programmer can directly send NumPy arrays. f2py is a mature project included in numpy, but doesn’t support Fortran derived types, and development seems to have halted completely. fmodpy seems very promising as although it still does not support derived types, it’s under active development and derived types are on the roadmap.

Probably one of the ways which Fortran will survive in the future is in this type of use case. Even if the interface generation question is solved I still see some issues in the short terms such as support for Windows which is expected by Python users.

Curious if anyone has any experience or views on this.

8 Likes
2

Welcome to the forum.

The Fortran Wiki lists Fortran-Python interoperability tools. F90wrap is an F90 to Python interface generator with derived type support that is actively developed. There is an associated paper from Journal of Physics: Condensed Matter.

4 Likes
3

I personally like the explicit C interface route quite a lot. But my perspective is more that of a Fortran programmer looking for a way to make libraries accessible in Python. Using the explicit C route I get a C API “for free” along the way and I have the possibility create a multilayer API with several entry points and different abstraction levels for Python frameworks to hook into.

The other way around, from a Python programmers view looking for compiled extension modules, a multilayer API with different entry points doesn’t really make sense, because the compiled extension module has to fit only one particular purpose. I would go a step further than automatic interface generation and rather go directly to automatic code generation for this purpose, maybe pyccel is the right tool for the job here:

4 Likes
4

I also prefer the Explicit C Interface route. It’s 100% standard conforming, and once you write it, it’s done and you can use it with any language that can interoperate with C. That is better, in my opinion, than having to run your code through a magical black box (f2py or swig or similar) that probably forces you to write code in a very rigid way or requires special directives, and likely is going to produce some python-specific binary file that is only going to work with a very specific version of python.

4 Likes
5

I agree with others here that the standard CFI is very nice and robust and is the gateway to any language including python. I have written a few examples and exercises on this topic in the past. Here is one. You can find a more sophisticated case (e.g. involving callbacks) in this library. Would be happy to help further if needed.

Regarding compilers, Intel classic and oneAPI Fortran compilers are available free of charge to Windows users. NAG is also available for Windows (but for a fee, I believe). You could also easily install Gfortran binaries for Windows, or install Microsoft WSL and forget Windows altogether.
p.s. There is also the NVIDIA Fortran compiler for Windows which is also available free of charge.

4 Likes
6

Thanks for all the helpful answers! After giving it some thought and trying out writing a cffi interface I think that this route is quite acceptable. In my view the advantage of a separate Fortran implementation with an API is that the code is persistent and re-usable. There are so many re-implementations of basically the same algorithm in different languages and apps. An implementation in something like pyccel, pythran or Numba risk not being usable after a few years as the package ecosystem changes. Fortran code on the other hand… So even if my Python package is no longer used, the underlying implementations of the algorithms might.

The next problem that I am trying to solve is to build this conveniently so that it can be packaged with conda/pip. Most users of my tools would expect to be able to install with conda/pip on any operating system without having to worry about installing compilers on their own. conda/conda-forge does provide packages for Fortran compilers (i.e. gfortran). Is there any way to use fpm to build shared libraries?

6 Likes
7

FPM and Conda are certainly good viable options (I have tried neither though, only know others who have done so). If you want to lift the burden of compilation and package management from the user, especially if your code gets complex and large, you could write platform-agnostic CMake build scripts that compile your code for the three major platforms (Windows, Linux, macOS, or more, if there is user base), then put the shared library files all together in a Python package structure and upload it to PyPI. There is again this example Python packaging I have done here that includes example setup files. Once your package is ready, building and uploading it to PyPI is as simple as running the following few commands (inside the package root folder, assuming Windows OS Anaconda Command Prompt),

python -m pip install --upgrade pip
python -m pip install --user --upgrade setuptools wheel
pip install --user --upgrade twine

python setup.py sdist bdist_wheel
REM to upload the package to PyPI test portal:
twine upload --repository testpypi dist/*
REM to upload the package to the main portal, only after you test the package on testpypi:
twine upload --repository pypi dist/*

(REM is Windows Batch comment marker and is ignored on the command line). The final structure of the Python package (before running the above commands), that would be built and uploaded to PyPI, is very much like the ones on this GitHub release page: libparamonte_python_darwin_x64.tar.gz, libparamonte_python_linux_x64.tar.gz, libparamonte_python_windows_x64.zip, except that your PyPI package can combine all platform-specific shared files into the same package (if you like so). The important required build files for the PyPI package are setup.py and MANFEST.in.
I have been long trying to write a post on how to do this with an example Python-Fortran package from start to end, but have not had the time for it yet. Until then, I would be happy to provide further help here, if needed.

4 Likes
8

@KjellJorner Thanks for posting here. We are developing many of the tools such as fpm that should help with this task in the future. I also recommend the C interface route as others have mentioned. See this recommendation that I wrote up about 10 years ago:

Back then I would recommend Cython, but you don’t have to use it, you can use any other way to interface C from Python.

Note that I would like to generate the C interface and Python wrappers automatically down the road:

I would probably have it checked in a repository, so that you don’t have to rerun it every time.

4 Likes
9

Thanks for the suggestions, @shahmoradi and @certik! I did some further investigations with the cffi, ctypes and Cython. I would say that from my perspective they are all fairly similar, but maybe the differences become more apparent when working with much larger APIs. The most pragmatic approach seems to be to just pre-build the platform specific shared library file and include that with the Python package in a directory. Cython and cffi have the possibility to build module files with setup tools but I haven’t found any way to link the Fortran shared library statically. So you end up having to build wheels and patch them with tools like auditwheel and delocate (with no real option on Windows). Here it would be neat if fpm would have an option to build shared libraries. At the moment I have been doing that with CMake. If you guys would be interested in writing something up on Fortran-C interfacing for the Fortran-lang documentation I would be happy to contribute.

I think that LFortran is a really good idea and would love to see that extended to automatic interface generation. Personally I prototype all my Python code in Jupyter Lab and have found that it dramatically increases my productivity. When LFortran starts supporting arrays I see myself working mostly there and then transferring the code to an IDE for refactoring.

8 Likes
10

LFortran is such a great idea. OpenCoarrays also has a jupyter binder for parallel Coarray Fortran. Other than these two, the only prototyping options that I know of and use occasionally, are the online Fortran compilers like Tutorialspoint Fortran compiler.

3 Likes
11

@KjellJorner yes that is the idea with LFortran. It was easy to get a prototype up, it is much harder to deliver a working production ready compiler, but we are getting there. You can follow our progress here:

MVP: Roadmap to compile the SNAP project (#313) · Issues · lfortran / lfortran · GitLab

I expect to be very usable by the end of the summer.

3 Likes
12

Here is my preliminary working example that actually does what I want. First the C wrapper module in Fortran. Comments welcome.

module mod_api
  use, intrinsic :: iso_c_binding, only: c_int, c_double
  use mod_main, only: cone_angle
  implicit none
  
contains

  subroutine cone_angle_c(n_atoms, coordinates, radii, index_metal, alpha, normal, cone_indices) &
       bind(c, name="cone_angle")
    integer(c_int), value, intent(in) :: index_metal, n_atoms
    real(c_double), intent(in) :: radii(n_atoms), coordinates(3,n_atoms)
    real(c_double), intent(out) :: alpha, normal(3)
    integer(c_int), intent(out) :: cone_indices(3)
    call cone_angle(coordinates, radii, index_metal, alpha, normal, cone_indices)
  end subroutine cone_angle_c
  
end module mod_api

Then the Python wrapper which exposes the cone angle function which is what the end user would be interacting with.

from ctypes import byref, c_int, c_double
from pathlib import Path

import numpy as np
from numpy.ctypeslib import load_library, as_array, as_ctypes

# Load library
lib_path = Path(__file__).parent.parent / "lib"
lib = load_library("libconeangle", str(lib_path))

# Expose function
def cone_angle(coordinates, radii, index_metal):
    # Convert input to numpy arrays
    radii = np.asarray(radii, dtype=np.float64)
    coordinates = np.asarray(coordinates, dtype=np.float64)

    # Set up c types
    c_n_atoms = c_int(len(radii))
    c_coordinates = as_ctypes(coordinates)
    c_radii = as_ctypes(radii)
    c_index_metal = c_int(index_metal)
    c_alpha = c_double()
    c_normal = as_ctypes(np.empty(3, dtype=np.float64))
    c_cone_indices = as_ctypes(np.empty(3, dtype=np.int32))

    # Run calculation
    lib.cone_angle(c_n_atoms, byref(c_coordinates), byref(c_radii), c_index_metal, byref(c_alpha), byref(c_normal), byref(c_cone_indices))
    cone_angle = np.rad2deg(c_alpha) * 2
    normal = as_array(c_normal)
    cone_indices = as_array(c_cone_indices)
    cone_indices = list(cone_indices[cone_indices > 0])
    return cone_angle, normal, cone_indices
6 Likes
13

Hi Kjell @KjellJorner, welcome to the Discourse! We would very much appreciate a contribution like this for the Fortran-lang website. A dedicated ‘mini-book’ on Fortran-C interoperability is definitely needed. See below for some links on how to get started writing a ‘mini-book’ and how to contribute to the website. We’re also happy to provide any help you need getting started.

https://github.com/fortran-lang/fortran-lang.org/blob/master/MINIBOOKS.md

https://github.com/fortran-lang/fortran-lang.org/blob/master/CONTRIBUTING.md

5 Likes
14

Thanks for the links @lkedward! Do you do any prototyping of minibooks on something like GitBook or HackMD where you could do real-time collaborative editing? Think that might lower the barriers for people to contribute to this and then it can be moved to git fork when more mature.

1 Like
15

Just an update on this topic. I joined in on an open issue on this topic on the fortran-lang GitHub. Me and @awvwgk have started a prototype mini book on HackMD and anyone interested can join in. Create an account on HackMD and send me a message and I can add you to the team.

7 Likes
16

I have been working on a project that seems to parallel this well, I would like to interface between python and existing fortran code directly, instead of the existing I/O which uses text files. I am using the approach of writing a wrapper function with c_iso_binding, and then wrapping this with cython. This seems to work pretty well, but I have two questions I hope the group could help with.

  1. Is there any work on automatic wrapper code generation? This part is time consuming and fragile, and not a lot of fun for functions with 20+ arguments. I also ran into some errors in this process, and the debugging was a nightmare.

  2. Is it possible to work with allocatable fortran arrays with this workflow? I think the holdup is cython, but I can’t find a working example.

Any help is appreciated! I am just starting to learn fortran and I find it very interesting. I think for me, it will remain in the near future much more practical with a convenient interface to python.

2 Likes
17

@nedlrichards, welcome to the Forum! Yes, there several automatic wrapper generation tools, the most mature of which is f2py. See the links above.

Thanks to @hsnyder we now also have a prototype automatic wrapper in LFortran: src/lfortran/codegen/asr_to_py.cpp · 00c9351cad5f0606fb41cd6d8056e9dc4a52c9d5 · lfortran / lfortran · GitLab, currently it’s using iso_c_binding and Cython. If anyone is interested to contribute, definitely let us know. I am happy to get you up to speed.

Regarding allocatable arrays, you either have to allocate / deallocate from Fortran (just write some subroutines to do that and expose them to Python), or you have to allocate from Python or the Cython wrappers and only pass an already allocated array to Fortran. It’s much easier if you just use NumPy to allocate the arrays (numpy.empty()) and never store any pointers to it from Fortran, then there will be no memory leaks as NumPy will use Python’s GC to deallocate.

5 Likes