While for a medium-level Fortran user using this library is (in our opinion) pretty easy, two problems came to our minds:
Students in our University don’t have much programming base knowledge and the entry to a compiled language sometimes that experience makes them quit (which we don’t want)
Interpreted languages are well-established in scientific computing in the world (besides if they wrap compiled languages or not)
Considering those things, we thought that including a Python interface in our codes would solve those two problems, we all know that f2pycan be a great tool for this, but this only happens with basic types and since the deprecation of distutils the packaging of a f2py based project became a bit hard. After a LOT of trial and error, we share here what we think are good guidelines for a project with a large codebase based on fpm (which could also work for meson and CMake projects I think).
Compilation: Install the fpm based library with fpm install --profile release --prefix PYTHON_API and, of course, your desired flags. Then the C-API callable from Python can be compiled with f2py -m my_api -c C_API.f90 -LPYTHON_API/lib -lmylib. This generates an importable Python extension.
Automatic generation of distributable wheel: The correct way of distributing Python packages is with wheel distributions, these can be generated with the software cibuildwheel. To make the package installable with cibuildwheel the package should be structured as a meson-python project. Which requires a meson.build file yaeos/python/meson.build at main · ipqa-research/yaeos · GitHub that should automate the two steps named before. All this can be automated on multiple OS with a GitHub Action like in: yaeos/.github/workflows/CI.yml at 5-python-api · ipqa-research/yaeos · GitHub
Publish the package: Now the generated wheels can be uploaded to PyPI with twine as a normal Python wheel.
I noticed the methods aren’t bind(c). Is this because f2py writes it own wrappers which also cover the name mangling?
Could steps 2-4 be automated in GitHub Actions (or other workflow for that matters)? It seems like it could be a relatively straightforward way to get Fortran code directly into Python registries and with it into the hands of Python users.
I noticed the methods aren’t bind(c). Is this because f2py writes it own wrappers which also cover the name mangling?
Yes, f2py deals with everything. This still is a prototype only focused on Python, so I’ve missed including bind(C) and name mangling in the source code. But it should be there on the final version to be a true C interface
Could steps 2-4 be automated in GitHub Actions (or other workflow for that matters)? It seems like it could be a relatively straightforward way to get Fortran code directly into Python registries and with it into the hands of Python users.
Yes, that can be automated with CI, right now we only reach the wheels generation but after that uploading to PyPI is simple. We haven’t included that yet but is straightforward. An old and abandoned package of mine did that Update README.md · fedebenelli/PyForFluids@9ca6a66 · GitHub here is the workflow, it no longers works since it’s deprecated behaviour but the logic is the same that a new build should do (make wheels, share that wheels to another job that handles the publication)
Yes, that is a good idea! I’ve never seen how is the workflow to publish as a conda package, since I always went straight to PyPI, but it can be a good option!
On a side note, maybe extra options to generate templates with fpm would be nice? so one could do fpm new my_project --python and besides the general structure that is already built the basics for a Python API would be generated
And the extra job should be like this (this version does not have it, but there should be some restriction to not upload at any push, but just on pushes on main or on each release/tag):
All this took us a long time to elaborate. Maybe some page dedicated to this can be made on some section of the fortran-lang website? Something as “Alternative ways of distributing a fpm library”
@fedebenelli Really cool. I’ve messed around with f2py before but never really fully understood how you could use it with something complicated. Also, I didn’t really want to learn meson, yet another build system. I think the missing piece of the puzzle that you show here is you just have meson call fpm to build the library, and then f2py is really only building the C interface and linking to the library that has already been built. The beauty of this is that you keep the library as an fpm package that is usable from Fortran, and the Python interface is something extra.
I have one suggestion for you though. It looks like you have global variables in your C interface, so it’s not threadsafe. Take a look at what I did here: radbelt/src/radbelt_c_module.f90 at master · jacobwilliams/radbelt · GitHub I’m using an allocatable class and some pointer/integer transfers to make a thread-safe C interface callable from python. The Python API just keeps track of the ints which get converted on the Fortran side.
Thanks for the input! Yes, we were not sure if we should keep everything inside the Fortran package or provide the extra things as well… extra things. We think that leaving outside the C interface and the Python API was the cleanest solution, in our opinion.
I’ll look at the links you shared. I’m pretty sure that there are better ways to how handle the accessibility of allocated models to the external languages, thread safety was not in our minds because we do not work much with multi-threaded code, but it can become important for an external user at some point!
Also, I have included a branch with a cookiecutter template. It’s like a fpm new command but includes all the directories and files for the Python API.
The template have the classic say_hello initial subroutine, the C API for the say_hello subroutine and the Python API for the say_hello function generated with f2py. To create a project from the template you can do:
cookiecutter https://github.com/SalvadorBrandolin/fortran_meson_py --checkout cookies
