Optional dependencies with fpm

Help
1

In neural-fortran I use a few different dependencies. Fpm makes this very easy to do via fpm.toml, e.g. see neural-fortran/fpm.toml at main · modern-fortran/neural-fortran · GitHub.

Now, I want to make the HDF5 dependency optional, because a small fraction of users needs it, and because it adds considerable set-up complexity to build the project.

I figured out how to do this with CMake, where I block out sections of code with CPP macros, and tell CMake to not pull the dependencies if that macro is not defined.

How would I best do this with fpm?

I don’t think it’s possible to mark dependencies as optional in fpm, and then use an fpm flag for example.

A possible workaround could be to have two separate fpm.toml files, e.g. fpm.toml and fpm-with-hdf5.toml. Both would need to enable preprocessing, and the latter would need to define the macro. In that case in fpm.toml (no HDF5) I would have:

# no [build] rules
# no [dependencies] rules

[preprocess]
[preprocess.cpp]
suffixes = ["f90"]

and in fpm-with-hdf5.toml I would have:

[build]
external-modules = "hdf5"
link = ["hdf5", "hdf5_fortran"]

[dependencies]
functional = { git = "https://github.com/wavebitscientific/functional-fortran" }
h5fortran = { git = "https://github.com/geospace-code/h5fortran" }
json-fortran = { git = "https://github.com/jacobwilliams/json-fortran" }

[preprocess]
[preprocess.cpp]
macros = ["USE_KERAS_HDF5"]
suffixes = ["f90"]

However, in the first case, I still get

 $ fpm build
 + mkdir -p build/dependencies
<ERROR> *cmd_build* Target error: Unable to find source for module dependency: "functional" used by "././src/nf/nf_keras_submodule.f90"
STOP 1

I guess that this is because fpm doesn’t preprocess the source files before establishing the dependencies.

Any advice on how to work around this with the current fpm? This is with v0.10.1.

Thanks!

1 Like
2

I had to do the same with PETSc a while back for some training courses at Imperial with fpm. I can’t remember off the top of my head what I did, but I will look if I still have the packages stored somewhere and get back to you.

1 Like
3

I think this is just this old issue: Optional dependencies · Issue #84 · fortran-lang/fpm · GitHub, nobody has implemented it yet. PRs welcome. :slight_smile:

4 Likes
4

As a workaround I ended up pulling out the code that depends on HDF5 and other dependencies and putting in its own standalone package: GitHub - neural-fortran/nf-keras-hdf5: Keras HDF5 adapter for neural-fortran, which now uses neural-fortran as a required dependency.

On the Fortran end, this was only possible thanks to the use of submodules, which allowed separating the implementation (in nf-keras-hdf5) from the interface (in neural-fortran). Thanks to @rouson who convinced me to start using submodules more than 2 years ago, much to my resistance.

1 Like
5

I think it would be great to have optional dependencies in fpm and if we find consensus on the syntax, it’d be relatively easy to implement. In my view, any “optional” feature should be bound to the definition of a boolean macro, for the reason discussed here above (calls to modules/functions will have to be conditional, etc.). I like the way it was proposed on the wake of Cargo. Here, one would have an optional package with a boolean

[dependencies]
mpi = { version="*", optional=true }

Being optional would automatically define a features table with the same name as the dependency, where the optional features can be set:

[features]
mpi.preprocess.cpp = ["WITH_MPI"]
mpi.fortran.implicit_typing = true

Because currently we already support dependency based cpp macros without a feature table:

[dependencies]
a = { git = "https://github.com/b/c",  preprocess.cpp.macros=["MY_MACRO"]}

I would also support a “simplified” syntax that, instead of grouped “features”, allows the definition of these flags directly in the dependency field:

[dependencies]
a = { git = "https://github.com/a/b",  optional=true, fortran.implicit_typing=true, preprocess.cpp.macros=["WITH_A"]}

These options would of course not be enforced when the dependency is not present.

3 Likes
6

PS: on trunk, the HDF5 dependency is now a metapackage and you might want to give it a try:

[dependencies]
hdf5 = "*"
7

Regarding syntax for optional dependencies, poetry might be a good source of inspiration, especially the concept of (optional) groups.

Managing dependencies | Documentation | Poetry - Python dependency management and packaging made easy (python-poetry.org)

You end up with something like so:

...
[tool.poetry.dependencies]
python = ">=3.9,<4.0"
numpy = "^1.23.5"
...

[tool.poetry.group.docs]
optional = true
[tool.poetry.group.docs.dependencies]
mkdocs = "^1.4.2"
mkdocs-material = "^9.5.3"
...

[tool.poetry.group.dev]
optional = true
[tool.poetry.group.dev.dependencies]
pytest = "^8.1.1"
pytest-cov = "^5.0.0"
...

You activate optional groups with --with:

poetry install --with dev,docs
4 Likes
8

Great! I’ll try it once it’s in a tagged release.

1 Like
9

I have the same kind of problem and idea to convert a project to fpm. I have a computing project that can be built with GTK or be used on CLI. For the moment, I use two scripts with_GUI.sh and without_GUI.sh to compile and run it.

I have just made a quick test on a toy project where I have created two manifests fpm1.toml and fpm2.toml with different src directories (src1 and src2). The manifest fpm.toml used by fpm is a symbolic link:

$ ln -sf fpm1.toml fpm.toml
$ fpm build
 Hello, essai!

It works, but when you switch you are obliged to first clean the build directory:

$ ln -sf fpm2.toml fpm.toml && fpm clean --skip 
$ fpm build
 Hello, essai2!

else fpm will not detect that the manifest has changed and will not rebuild the project.

Well, it seems a decent workaround, the commands being recorded in the terminal history. I will try to adapt my project with this solution.

10

I’ve been hit by this question today too.

My problem is a touch different however, I want to disable any external dependencies. (Some systems do not have git or external internet on compute nodes which could be tricky to handle).

I can easily turn on/off the dependency using the cpp preprocessor, i.e.

#ifdef EXTDEPS                               
  use stdlib_hash_64bit, only: fnv_1a_hash   
#endif

but fpm still looks for that dependency. None of this is truly end of world stuff (it is easy enough for me to get a git exectuable manually, and point fpm at a local copy of stdlib), but it would be nice for others to have a way to avoid it.

11

Isn’t what you want to achieve easier to handle by coping the dependencies and using a path identifier to the location of your dependencies?

[dependencies]
stdlib = { path = "../stdlib" }

(Path relative to the root of current project, no need for git)

12

You may want to have a look at the open issue 773 on GitHub. It presents different ways to deal with the problem you are facing.

If you want to keep the preprocessing mechanism you can also do as follows:

#ifdef EXTDEPS                               
#include <stdlib.inc>
#endif

and in the file stdlib.inc

use stdlib_hash_64bit, only: fnv_1a_hash

It’s a bit hacky but it does the job.

13

Thanks both!

I want to keep it simple for others to use, hence the disabling entirely. When we are running this code in full production mode, I don’t envision any issues with having dependencies (we would just compile once on the head node), but we are doing some profiling at the moment which is compiling on the compute nodes which are restricted somewhat.

I will update my notes to reflect the path version of dependencies, and also do the #include method.

2 Likes