Very often, I need to write Python wrappers for my Fortran libraries where I create a shared library of my Fortran code and plug it into Python through its C API. The shared library will have, at least, dependencies on libgfortran and libquadmath.
I was wondering what do you think is the best strategy: embed all the gfortran dependencies or ask the user to install gfortran through homebrew for MacOS and msys2 for windows. On linux, the situation is much easier where most of the time libgfortran is installed by default.
Good question. IMO, asking the user to install gfortran, while doable, makes the distribution process unnecessarily complex. Another drawback, is that each package will be quite heavy because of the dependencies.
There is the same issue with intel fortran, since you need to distribute libifcoremd, libifportmd, libmmd… And if you are on Windows you need to make sure the end user has compatible C++ redistribuables (vcruntime).
I am not too familiar with Python package management as I usually rather work with nuget packages. What I would do in this case is create one package that contains the redistribuable libraries (and no other code), and a second package with my library that simply reference the first one. By doing so, you can also control the version that is used to run your lib.
I have always included the GNU Fortran dependencies when distributing Windows libraries or executables that need them. Speaking from experience, Windows users, including developers, will not tolerate having to install msys2.
You’ll probably have much better luck suggesting installing Homebrew on macOS. Distributing a shared library for macOS is tricky, though. Won’t it need to be signed and notarized?
I also thought that asking the Windows user to install msys2 might be cumbersome. I am not familiar with the macOS ecosystem and I don’t know if it needs to be signed or notarized
After looking at this discussion, it seems that for macOS homebrew is the way to go.
I distribute an OSX binary created from gfortran and have had no user complaints. It seems that with recent versions of OSX there is a popup that warns the user that the binary might be dangerous, but apparently users are used to that and click through it. I add the options -static-libgfortran -static-libgcc
to the compile as few users will have those libraries and even fewer will know how to install them. If you have read that OSX doesn’t support static executables, I would characterize that statement as “just close enough to the truth to be misleading”. You can’t include the OSX libraries that are present in every installation of OSX, but you can include any other libraries your program needs.
I’d love to create the executable with a github action, but so far haven’t gotten that to work. If anyone has a suggestion…
Currently, this is what I am doing. I use the static linking for macOS and Windows to avoid dependencies on libgfortran and libquadmath. On Linux (centos and Debian) the static linking is not possible. I was just wondering what would be the best way in the case I cannot use the static linking.
On Centos 7 and Debian 12, I have tried the static linking for libgfortran and libquadmath and I got an error from the linker saying that the static versions of the libraries (libgfortran*.a and libquadmath*.a) are not available.
After some googling, I found that in fact many distributions do not provide the static versions of libgfortran and libquadmath.
I might have missed something but I did not succeed in statically linking on Linux.
This might be a solution for Windows users Quick Installer for Windows for quickly installing and setting the gfortran compiler as well as all necessary libraries.
Making your library into a conda package is another solution to this problem. It does mean your users have to install conda and perhaps be required to understand conda environments (although there are ways to build a stand-alone installer for a conda package that somewhat hides that). The gfortran and intel runtime dependencies are available there, as well as many other dependencies in the scientific ecosystem. Also, it’s cross-platform.
Static linking with gfortran libraries requires static versions of the libraries, which GCC does not distribute by default. There is no way but to rebuild gfortran from the source for static libraries. I raised this issue to GCC developers several years ago and there was little interest in providing the static versions of the libraries by default.
There is the long standing discussion in the Linux community about static vs dynamic libraries. When I started with Linux 20 years ago, I learned to only use dynamic libraries for everything, as most distributions still recommend today. But I changed my mind, I think static libraries are better in almost every case (the final executable is smaller, faster, no dependencies, etc.). Only with some foundational system libraries like libc it makes sense to use shared libraries in my opinion. Another is a Python extension module.
Compilers should always allow static linking if requested.
Static linking with gfortran libraries requires static versions of the
libraries, which GCC does not distribute by default. There is no way but
to rebuild gfortran from the source for static libraries. I raised this
issue to GCC developers several years ago and there was little interest
in providing the static versions of the libraries by default.
Indeed there is widespread and unjustified hostility to static linking
that I have never understood, however your comment may be unnecessarily
discouraging to potential users. Over the years I have installed gfortran
on Windows, FreeBSD, NetBSD and OSX from package sources and the -static
option has worked in all of them without further action. So gfortran
packagers seem to understand the utility of static linking and make
provision for it.
Personally, I have always favored static libs over dynamic ones. This goes back 20 or so years now when I was running primarily on Cray XT systems that only supported static linking for applications running on the backend. The micro-kernel based OS used on the backend didn’t support dynamic libs. A Cray support engineer told me that the reason Cray didn’t support dynamic libs was that they needed to be read at the start of each process in a run from hard disks that were usually mounted on slow NFS systems. For an application that might use thousands of processes, this lead to the possibility that the disk access requests to roll in the dynamic libs might swamp the system. This has been fixed for the most part over the years placing the libs on a parallel file system instead of NFS. The big issue with static libs is some linkers have a hard time finding all the entry points with just one reference to the lib in a link statement. ie. you have to sometimes do stupid tricks like
Cray now allows you to link in dynamic libs because as mentioned some system libs and other components (MPI etc) don’t ship with a static lib by default anymore.
The big advantage of dynamic/shares libraries is the possibility to centrally deploy (security) fixes. With static libraries, each dependent software needs to be rebuilt with the fixed libraries. One example where something similar to static linking went terribly wrong was the log4j security issue in Java.
The downside of shared linking are incompatibilities caused by breaking changes in the libraries. Theoretically, semantic versioning can solve that but practically it still causes problems.
Linux distributions “solve” the issues related to shared libraries by sticking to exactly one veraion of the library. Hence, each package has to be compiled especially for the target version of the OS and security fixes are often backported because real updates will break things.
Conda shows that dynamic linking and support for various library versions is possible
I have some codes where it is necessary to add a library multiple times with linux, but it is not necessary with MacOS. I assume this is a difference in the ld utility. Is there an ld option that can be used to eliminate this under linux, and to make the linux ld work the way it does on MacOS?
The only other fix seems to be to combine some of the libraries, which introduces some additional build problems. I would like to end up with similar (or identical) compile and load statements on the two OSs.
