Optional dependencies with fpm

Help
1

In neural-fortran I use a few different dependencies. Fpm makes this very easy to do via fpm.toml, e.g. see neural-fortran/fpm.toml at main · modern-fortran/neural-fortran · GitHub.

Now, I want to make the HDF5 dependency optional, because a small fraction of users needs it, and because it adds considerable set-up complexity to build the project.

I figured out how to do this with CMake, where I block out sections of code with CPP macros, and tell CMake to not pull the dependencies if that macro is not defined.

How would I best do this with fpm?

I don’t think it’s possible to mark dependencies as optional in fpm, and then use an fpm flag for example.

A possible workaround could be to have two separate fpm.toml files, e.g. fpm.toml and fpm-with-hdf5.toml. Both would need to enable preprocessing, and the latter would need to define the macro. In that case in fpm.toml (no HDF5) I would have:

# no [build] rules
# no [dependencies] rules

[preprocess]
[preprocess.cpp]
suffixes = ["f90"]

and in fpm-with-hdf5.toml I would have:

[build]
external-modules = "hdf5"
link = ["hdf5", "hdf5_fortran"]

[dependencies]
functional = { git = "https://github.com/wavebitscientific/functional-fortran" }
h5fortran = { git = "https://github.com/geospace-code/h5fortran" }
json-fortran = { git = "https://github.com/jacobwilliams/json-fortran" }

[preprocess]
[preprocess.cpp]
macros = ["USE_KERAS_HDF5"]
suffixes = ["f90"]

However, in the first case, I still get

 $ fpm build
 + mkdir -p build/dependencies
<ERROR> *cmd_build* Target error: Unable to find source for module dependency: "functional" used by "././src/nf/nf_keras_submodule.f90"
STOP 1

I guess that this is because fpm doesn’t preprocess the source files before establishing the dependencies.

Any advice on how to work around this with the current fpm? This is with v0.10.1.

Thanks!

1 Like
2

I had to do the same with PETSc a while back for some training courses at Imperial with fpm. I can’t remember off the top of my head what I did, but I will look if I still have the packages stored somewhere and get back to you.

1 Like
3

I think this is just this old issue: Optional dependencies · Issue #84 · fortran-lang/fpm · GitHub, nobody has implemented it yet. PRs welcome. :slight_smile:

2 Likes
4

As a workaround I ended up pulling out the code that depends on HDF5 and other dependencies and putting in its own standalone package: GitHub - neural-fortran/nf-keras-hdf5: Keras HDF5 adapter for neural-fortran, which now uses neural-fortran as a required dependency.

On the Fortran end, this was only possible thanks to the use of submodules, which allowed separating the implementation (in nf-keras-hdf5) from the interface (in neural-fortran). Thanks to @rouson who convinced me to start using submodules more than 2 years ago, much to my resistance.

1 Like
5

I think it would be great to have optional dependencies in fpm and if we find consensus on the syntax, it’d be relatively easy to implement. In my view, any “optional” feature should be bound to the definition of a boolean macro, for the reason discussed here above (calls to modules/functions will have to be conditional, etc.). I like the way it was proposed on the wake of Cargo. Here, one would have an optional package with a boolean

[dependencies]
mpi = { version="*", optional=true }

Being optional would automatically define a features table with the same name as the dependency, where the optional features can be set:

[features]
mpi.preprocess.cpp = ["WITH_MPI"]
mpi.fortran.implicit_typing = true

Because currently we already support dependency based cpp macros without a feature table:

[dependencies]
a = { git = "https://github.com/b/c",  preprocess.cpp.macros=["MY_MACRO"]}

I would also support a “simplified” syntax that, instead of grouped “features”, allows the definition of these flags directly in the dependency field:

[dependencies]
a = { git = "https://github.com/a/b",  optional=true, fortran.implicit_typing=true, preprocess.cpp.macros=["WITH_A"]}

These options would of course not be enforced when the dependency is not present.

3 Likes
6

PS: on trunk, the HDF5 dependency is now a metapackage and you might want to give it a try:

[dependencies]
hdf5 = "*"
7

Regarding syntax for optional dependencies, poetry might be a good source of inspiration, especially the concept of (optional) groups.

Managing dependencies | Documentation | Poetry - Python dependency management and packaging made easy (python-poetry.org)

You end up with something like so:

...
[tool.poetry.dependencies]
python = ">=3.9,<4.0"
numpy = "^1.23.5"
...

[tool.poetry.group.docs]
optional = true
[tool.poetry.group.docs.dependencies]
mkdocs = "^1.4.2"
mkdocs-material = "^9.5.3"
...

[tool.poetry.group.dev]
optional = true
[tool.poetry.group.dev.dependencies]
pytest = "^8.1.1"
pytest-cov = "^5.0.0"
...

You activate optional groups with --with:

poetry install --with dev,docs
4 Likes
8

Great! I’ll try it once it’s in a tagged release.

1 Like