Packaging a `fpm` project with Python bindings, a little guide and insights from our experience

At our institute we have been migrating a lot of our codebase to a cleaner and more structured one, taking advantage of modern Fortran constructs and fpm. The resulting product is: GitHub - ipqa-research/yaeos: Thermodynamic Equations of State, Fortran library with both automatic and anallytical derivation capabilities. It exploits Fortran OOP maybe too much but it helped us to obtain an easy to extend library.

While for a medium-level Fortran user using this library is (in our opinion) pretty easy, two problems came to our minds:

  • Students in our University don’t have much programming base knowledge and the entry to a compiled language sometimes that experience makes them quit (which we don’t want)
  • Interpreted languages are well-established in scientific computing in the world (besides if they wrap compiled languages or not)

Considering those things, we thought that including a Python interface in our codes would solve those two problems, we all know that f2pycan be a great tool for this, but this only happens with basic types and since the deprecation of distutils the packaging of a f2py based project became a bit hard. After a LOT of trial and error, we share here what we think are good guidelines for a project with a large codebase based on fpm (which could also work for meson and CMake projects I think).

Most of the comments are implemented in the python-api (still not merged) branch of our repositorio (GitHub - ipqa-research/yaeos at 5-python-api)

  1. Write a C-API that uses basic f2py-compatible types on the project root directory (yaeos/c_interface/yaeos_c.f90 at 5-python-api · ipqa-research/yaeos · GitHub): This will make your code viable to be called from Python and any other language since it is a C API, so it is a two birds with one stone scenario. In our case, our codes depend heavily in a polymorphic object ArModel, from which we define a list of containers (yaeos/c_interface/yaeos_c.f90 at 5-python-api · ipqa-research/yaeos · GitHub) that are hidden to the user.
  2. Compilation: Install the fpm based library with fpm install --profile release --prefix PYTHON_API and, of course, your desired flags. Then the C-API callable from Python can be compiled with f2py -m my_api -c C_API.f90 -LPYTHON_API/lib -lmylib. This generates an importable Python extension.
  3. Automatic generation of distributable wheel: The correct way of distributing Python packages is with wheel distributions, these can be generated with the software cibuildwheel. To make the package installable with cibuildwheel the package should be structured as a meson-python project. Which requires a meson.build file yaeos/python/meson.build at main · ipqa-research/yaeos · GitHub that should automate the two steps named before. All this can be automated on multiple OS with a GitHub Action like in: yaeos/.github/workflows/CI.yml at 5-python-api · ipqa-research/yaeos · GitHub
  4. Publish the package: Now the generated wheels can be uploaded to PyPI with twine as a normal Python wheel.

A minimal example can also be seen here: GitHub - SalvadorBrandolin/fortran_meson_py: Example of building python API of a Fortran project (that runs with fpm) with meson

12 Likes

Thanks for sharing. I have a few questions.

I noticed the methods aren’t bind(c). Is this because f2py writes it own wrappers which also cover the name mangling?

Could steps 2-4 be automated in GitHub Actions (or other workflow for that matters)? It seems like it could be a relatively straightforward way to get Fortran code directly into Python registries and with it into the hands of Python users.

I noticed the methods aren’t bind(c). Is this because f2py writes it own wrappers which also cover the name mangling?

Yes, f2py deals with everything. This still is a prototype only focused on Python, so I’ve missed including bind(C) and name mangling in the source code. But it should be there on the final version to be a true C interface

Could steps 2-4 be automated in GitHub Actions (or other workflow for that matters)? It seems like it could be a relatively straightforward way to get Fortran code directly into Python registries and with it into the hands of Python users.

Yes, that can be automated with CI, right now we only reach the wheels generation but after that uploading to PyPI is simple. We haven’t included that yet but is straightforward. An old and abandoned package of mine did that Update README.md · fedebenelli/PyForFluids@9ca6a66 · GitHub here is the workflow, it no longers works since it’s deprecated behaviour but the logic is the same that a new build should do (make wheels, share that wheels to another job that handles the publication)

1 Like

This is great! Thanks, I will study it!

Definitely we need to have something like this as a boilerplate template that people can use to start a new project easily.

Another useful addition would be option to also build it as a conda package.

1 Like

Yes, that is a good idea! I’ve never seen how is the workflow to publish as a conda package, since I always went straight to PyPI, but it can be a good option!

On a side note, maybe extra options to generate templates with fpm would be nice? so one could do fpm new my_project --python and besides the general structure that is already built the basics for a Python API would be generated

1 Like

We have now added automatic upload to PyPI on our CI. It is pretty straightforward.

It is needed to allow uploads from GitHub from the PyPI site, there is an option to “add a new publisher”

And the extra job should be like this (this version does not have it, but there should be some restriction to not upload at any push, but just on pushes on main or on each release/tag):

1 Like

All this took us a long time to elaborate. Maybe some page dedicated to this can be made on some section of the fortran-lang website? Something as “Alternative ways of distributing a fpm library”

3 Likes

Thanks for sharing, I think it would be really beneficial to have it on the website. feel free to open a PR for this on GitHub - fortran-lang/webpage: New Fortran webpage , you can refer to Add : Python Fortran Rosetta Stone (Transfer content from fortran90.org #112) by henilp105 · Pull Request #185 · fortran-lang/webpage · GitHub for the template/markdown .

I’m giving this a go - I’ll post a link to it when there’s actually something to it - but I’m highly encouraged so far.

Key things to keep in mind thus far:

  • f2py lower cases everything
  • f2py ‘promotes’ subroutines to functions if they are simple

Thanks!

@fedebenelli Really cool. I’ve messed around with f2py before but never really fully understood how you could use it with something complicated. Also, I didn’t really want to learn meson, yet another build system. I think the missing piece of the puzzle that you show here is you just have meson call fpm to build the library, and then f2py is really only building the C interface and linking to the library that has already been built. The beauty of this is that you keep the library as an fpm package that is usable from Fortran, and the Python interface is something extra.

I have one suggestion for you though. It looks like you have global variables in your C interface, so it’s not threadsafe. Take a look at what I did here: radbelt/src/radbelt_c_module.f90 at master · jacobwilliams/radbelt · GitHub I’m using an allocatable class and some pointer/integer transfers to make a thread-safe C interface callable from python. The Python API just keeps track of the ints which get converted on the Fortran side.

Thanks for the input! Yes, we were not sure if we should keep everything inside the Fortran package or provide the extra things as well… extra things. We think that leaving outside the C interface and the Python API was the cleanest solution, in our opinion.

I’ll look at the links you shared. I’m pretty sure that there are better ways to how handle the accessibility of allocated models to the external languages, thread safety was not in our minds because we do not work much with multi-threaded code, but it can become important for an external user at some point!

Hi everyone! Thank you for all the comments.

Just here to inform you that the explanation in the minimum example’s README is complete

https://github.com/SalvadorBrandolin/fortran_meson_py

Also, I have included a branch with a cookiecutter template. It’s like a fpm new command but includes all the directories and files for the Python API.

The template have the classic say_hello initial subroutine, the C API for the say_hello subroutine and the Python API for the say_hello function generated with f2py. To create a project from the template you can do:

cookiecutter https://github.com/SalvadorBrandolin/fortran_meson_py --checkout cookies

Then you can install the Python API with:

cd python
pip install .

If you want an editable installation:

cd python
pip install -r requirements-build.txt
pip install -e . --no-build-isolation

Then from a Python interpreter:

from my_project import say_hello

say_hello()

Just waiting to see those amazing and fast Fortran libraries with Python API! :smiley:

4 Likes