I’m currently actively contributing to the development of Spack package manager, which is a package manager designed for supercomputers. The core dev team is in Lawrence Livermore National Laboratory (LLNL) and the project is part of the Exascale Computing Project (ECP). Since its inception, Spack has been used to conveniently manage software installation on various HPC centers, including current pre-exascale systems Summit (ORNL) and Perlmutter (NERSC). I believe Spack will also be in use for the upcoming exascale systems Frontier (ORNL), El Capitan (LLNL), and Aurora (ANL).
I’ve been attending the Spack weekly telcons at Wednesdays 9a Pacific and recently worked with one of the core devs to fix the installation of the fpm package in Spack.
Now, FPM is more just a package. FPM is also a build system. AFAIK Spack supports multiple build systems, including Automake, CMake, and Ninja. It would be nice if we can “elevate” FPM from a normal package in Spack into a supported build system, and hopefully enlarge the user base of FPM!
This thread is meant for brainstorming this idea (FPM as a build system in Spack) as well as trying to make it happen (hopefully we can get the Spack devs to chime along here).
Also, summoning @awvwgk since he’s listed as the FPM maintainer in Spack.
I thought about adding FPM as build system for spack, had a quick look through the code base but decided that it is probably too early. The situation might be different if there were actual packages only relying on fpm as build system.
There are a few issues, basically fpm is currently not playing nice with other package managers. For fpm packages especially libraries will not interact well with other spack packages, similarly, executables with fpm will currently circumvent spack when pulling in their dependencies.
Maybe a harsh opinion here, as a package maintainer for conda-forge and contributor to MSYS2, Spack and homebrew I currently wouldn’t consider a project relying only on fpm as build system for packaging.
Of course, as an fpm developer I would like to see fpm integrate well with those packaging ecosystem at some point, but we are not there yet.
But there are actual packages only relying on fpm as the sole build system, for instance, @rouson 's reference-counter, which he just recently fully migrated from CMake to FPM. (Full disclosure: I’m getting paid to work on this code)
Indeed, and there will be more Fortran packages in the future relying only on fpm. It is a good thing that fpm finds adoption as build system in our community. The problem however is that this creates a separate ecosystem at the moment, which cannot interact with other build systems, like meson, CMake or setuptools to name a few.
I think the timing can’t be more perfect. The notion of having multiple build systems is still being actively discussed within the Spack devs. It actually came up during last week’s telcon: Telcon: 2022 05 11 · spack/spack Wiki · GitHub
I assume that this is not a popular question here, but can someone tell me why not using spack instead of fpm?
From the marketing perspective, having a dedicated Fortran package manager is certainly nice, but as a software developer there are already too many competing distribution channels.
I also think that a package manager and a build system have different functions: with a build system, one can build software if certain prerequisites are fulfilled while a package manager ensures that prerequisites are fulfilled.
From what I understand, Spack is strictly a package manager, while FPM is both a package manager and a build system. Here I’m more on the side of leveraging FPM as a build system within Spack, such that Fortran software that rely on FPM to build can be easily added into Spack too.
Well, that’s the right question to ask. The answer it that this is currently not possible, but would be desirable going forward.
As pointed out, fpm is both a build system and a package manager, within a package manager like spack, fpm however should only be a build system and spack should handle the package management. The question is how to make this switch over work seamlessly, preferably without disturbing the user experience on either the fpm or the spack side.
I recently had a scheduled 30-minute Zoom call with a collaborator who was interested in using the reference-counter repository that contains an updated version of code from my book. I asked if the collaborator had tried fpm (I had been recommending it to him for sometime). He said no. I asked if I could demonstrate fpm as long as I didn’t distract us for more than 3 minutes. He said yes. I started a timer. When the alarm sounded 3 minutes later, I had replaced the repository’s CMake build system with fpm.toml file that contained a single-digit number of lines, almost all of which were boilerplate and yet fpm could build the reference-counter library, a small app that my collaborator provided to demonstrate an issue, and fpm could build and run the test suite.
I used “make” for ~15 years. I used cmake for ~12 years. Within one month of experiencing fpm, I switched almost every project I work on to using fpm. Words cannot express how much better my life is. There’s no way to recover all the time I spent debugging problems with Makefiles and CMakeLists.txt files in my past life.
The above story really sums it up for me. Other build systems have more capabilities than fpm in some respects. However, I maintain that 95% of Fortran programmers only need what fpm offers on at least 95% of their projects. And for those projects, the difference in quality of life is gargantuan. Wild horses could not drag me back to other build systems since I discovered fpm. I hope this explains it.
Also, fpm isn’t just a Fortran build system despite the name. It does quite well with C too. I’m adding a fpm “build system” to OpenCoarrays, wherein the core library is in C and almost the only Fortran code is the test suite. I expect to be able to replace O(1000) lines of CMakeLists.txt with O(10) lines in a fpm.toml file. We’ll keep the CMake build system for the foreseeable future because OpenCoarrays is used in settings where some CMake capabilities are critical, e.g., generating shared libraries. But for the vast majority of everyday use cases, fpm suffices. I put “build system” in quotes above because I feel a little silly for me to describe my single-digit boilerplate fpm.toml file suffices.
I’ve never been happier with the way I compile code in the nearly 43 years that I’ve been writing Fortran.
I think comparing CMake from 12 years ago is not fair. I’m quite happy with the current CMake capabilities for Fortran, but the statement with a lot of boilerplate code is certainly true: GitHub - MarDiehl/fortran-skeleton: Cmake-based fortran skeletons. I believe one advantage of FPM is the fact that all the boilerplate (definition of compiler options etc) is centrally managed.
Sounds like I really should try FPM, but
makes me wonder how well FPM is suited for distributing software via Conda, launchpad, spack, Open Build Service, AUR, etc. It is already a pain to support so many channels, having FPM in addition means more work, not less.
My discomfort is pretty much summarized in the following statement:
because I like the Unix philosophy of one tool per task.
Coming back to the original post: For me it would be nice if FPM superior capabilities as a build system can be combined with package managers like spack. I would use FPM’s capabilities as a package manager for prototyping, but I prefer the traditional approach of having a language-independent package manager as part of the operating system.
Disclaimer: I’m fully aware that my opinion is against the mainstream and people prefer using pip, npm, and cargo. But as an Arch Linux user, this is just a hack for operating systems with broken or non-existing package manager.
It’s true that the swarm of packaging and build systems is a pain… But I like fpm for its simplicity. We could have that slogan: the power of simplicity!
For example, gtk-fortran can now be used as a fpm dependency. I fear it would be difficult to adapt the project to fpm, but finally it was quite simple. The only problems I fixed were:
in the src/ directory, there were two included files and they had the .f90 extension. The fix was to rename them with an .inc extension (hum… bad idea, GitHub now thinks those files are C++ ).
Old code concerning OS dependent functions was another problem: two Fortran files with the same Fortran module name. It was fixed by removing them, as I realized, discussing with GTK people, that those functions should not be used by users (the API should not have been made public).
In the fpm.toml I have just put a [build] section with link = [... with the list of GTK libraries. And the user has just to add in its fpm.toml a [dependencies] section with one line for gtk-fortran. The power of simplicity…
Note that for building the whole library and examples and install it in the system, CMake remains the build system of the project. fpm is just for using the library as a fpm dependency.
The intended usage of fpm is that a project does not rely on the installed system packages, but instead sandboxes the required Fortran dependencies for each project.
I have been very happy that this is the case, as I work on many Fortran projects with various inter-dependencies, all frequently requiring different versions of each other, none of which have really reached a very stable API. If fpm worked like a traditional package manager, my life would be a nightmare of dependency version hell.
I’ve seen languages get this horribly wrong (Python and Ruby for example), and languages do the sandbox thing like fpm from the get go and have much better success (Haskell and Rust for example), so I decided that was absolutely the way to do it when I first was working on the prototype.
Fortran also has the added complexity of projects compiled with different compilers, possibly even different versions of the compiler, not being compatible. What do you do if two projects you want to use are distributed having been compiled with different compilers?
I understand that Spack has attempted to, and possibly has, solved this problem to some extent, but I haven’t really tried to use it much, so I don’t know what it would be like to try have fpm interact with it. Would the idea be to have fpm use Spack to fetch a compiled version of a dependency instead of fpm fetching and compiling the source code for it?
I’m not opposed to having fpm play nicer with other package mangers and build systems, I just haven’t seen a clear explanation of what the expected workflow would look like.
Others have already answered (fpm is a build system and a package manager in one, while Spack is not a build system for Fortran). The same question is why not to use Spack instead of Cargo (for Rust)? It’s just so much easier to use Cargo instead of Spack for Rust. I use Spack quite often, and it’s great if you are a user of packages (especially from various languages), but if you are a developer, fpm and Cargo is better in my personal opinion. It’s also better than CMake (and I like CMake!), as @rouson said.
However, we absolutely have to be very usable from Spack as well as making Spack packages usable from fpm:
I’ve also began using Spack at work (super-computing centre). Currently we use it to provide a default set of environment modules, customized for the compilers which are available and catering to users needs.
I suppose it would be possible to teach Spack how to read fpm manifest files. Based upon the dependency section Spack could implicitly create a set of resources required for each fpm package, cross-checking them with any other dependencies in the spec. It could also inspect the link section of the manifest to find any external libraries which are needed (e.g. BLAS or LAPACK).
Next, fpm would need to be extended, to use Spack as the dependency fetcher (fpm/dependency.f90 at main · fortran-lang/fpm · GitHub) instead of git. This would be controlled by an environment variable, leaving the current dependency fetching mechanisms unchanged for regular users.
).