How to use fpm with external pre-compiled modules and libraries

I am a newcomer to fpm and I am trying to build a library project with different options. I am the main developer of the library CrysFML (https://code.ill.fr/scientific-software/crysfml). We use Cmake for building the library with ifort or gfortran for the three platforms Windows, Linux and MacOS with two options: (1) console and (2) using an external library for making GUIs.

I wanted to use something simpler than Cmake to build the library and I came aware of fpm.
I have found that for building console programs fpm is very simple and efficient. However, I had to use a script, invoking fpm with response files and other scripts for renaming files, for handling different compilers and different source files containing the same module name but different content (by automatically renaming before invoking fpm). This is not a problem, but something that I have not found, clearly explained in the documentation, is how to use an external library that is installed locally (*.mod files and library) outside the CrysFML directory. After trying several options using [dependencies] in the fpm.toml file, I never succeeded in building the GUI version of the library CrysFML that uses the commercial Winteracter library.
In the Windows (Linux or MacOS is the same) platform, the (simplified) structure of the two directories is the following

CrysFML
├── Program_Examples
│   └── Subdirectories with different programs
├── rsp
│   └── Directory with different response files
├── fpm.toml
├── make_CrysFML_fpm.bat
├── src
│   └── All module source files

Winter
├── lib.i64	(ifort 64 bits)
│   └── Directory containing *.mod, winter.lib and some *.obj files 
├── src
│   └── All interface modules source files

My questions are the following:
What have I to write in the fpm.toml file to make fpm finding the pre-compiled *.mod files of the Winteracter library? The winteracter.mod (within lib.i64) is used by one module of CrysFML when the script for building it is invoked with the appropriate response file.
I have tried to put in the compiler flags the include item “/I…\winter\lib.64” but fpm is always trying to find the source file of the module winteracter which is in “…/…/winter/src”. Is fpm unable to work with pre-compiled objects or mod files?
Thank you in advance for providing some tips.

========================== Content of some files ====================
File fpm.toml

[build]
auto-executables = false
auto-tests = false
auto-examples = false

[install]
library = true

[library]
source-dir = "Src"
include-dir = "Src"

Response file ifort_debug_win.rsp

options install --prefix ifort_debug_win
options --profile debug
options --compiler ifort
options --flags "/I..\..\wint\lib.i64"

Script file: make_CrysFML_fpm.bat

@echo off
rem .
rem  Attempt to create a unified build method for CrysFML using fmp
rem .
echo ---- Construction of the CrysFML library for 64 bits using gfortran, ifort or ifx (oneAPI) ----
echo ---- The building procedure installs also some executable programs of the Program_Examples subdirectory
echo      Default: ifort compiler in release mode. Equivalent to the first example below
echo      Examples of using the script:
echo             make_CrysFML_fpm  ifort
echo             make_CrysFML_fpm  ifort debug
echo             make_CrysFML_fpm  gfortran
echo             make_CrysFML_fpm  gfortran debug
echo             make_CrysFML_fpm  ifx
echo             make_CrysFML_fpm  ifx debug
echo     For using the Winteracter library add the word "win" as the last argument (without quotes)
echo ----
   (set _DEBUG=N)
   (set _COMP=ifort)
   (set _WINT=N)
rem > Arguments ----
:LOOP
    if [%1]==[debug]    (set _DEBUG=Y)
    if [%1]==[ifort]    (set _COMP=ifort)
    if [%1]==[ifx]      (set _COMP=ifx)
    if [%1]==[gfortran] (set _COMP=gfortran)
    if [%1]==[win]      (set _WINT=win)
    shift
    if not [%1]==[] goto LOOP
rem .
rem  First change the extensions of files that are optionally used in fpm to "xxx" by
rem  invoking the tochange.bat script in the Src directory.
cd .\Src
   if [%_WINT%]==[win] (
          call tochange xxx win
   ) else (
          call tochange xxx
          )
   )
cd ..
    if [%_COMP%]==[ifort] (
      cd .\Src
      ren  CFML_GlobalDeps_Windows_Intel.xxx  CFML_GlobalDeps.f90
      cd ..
      if [%_WINT%]==[win] (
          if [%_DEBUG%]==[Y] (
             fpm @./rsp/ifort_debug_win
          ) else (
             fpm @./rsp/ifort_release_win
          )
      ) else (
          if [%_DEBUG%]==[Y] (
             fpm @./rsp/ifort_debug
          ) else (
             fpm @./rsp/ifort_release
          )
      )
      cd .\Src
      ren CFML_GlobalDeps.f90 CFML_GlobalDeps_Windows_Intel.xxx
      cd ..
    )
    if [%_COMP%]==[ifx] (
      cd .\Src
      ren  CFML_GlobalDeps_Windows_Intel.xxx  CFML_GlobalDeps.f90
      cd ..
      if [%_DEBUG%]==[Y] (
         fpm @./rsp/ifx_debug
      ) else (
         fpm @./rsp/ifx_release
      )
      cd .\Src
      ren CFML_GlobalDeps.f90 CFML_GlobalDeps_Windows_Intel.xxx
      cd ..
    )
    if [%_COMP%]==[gfortran] (
      cd .\Src
      ren  CFML_GlobalDeps_Windows.xxx  CFML_GlobalDeps.f90
      cd ..
      if [%_DEBUG%]==[Y] (
         fpm @./rsp/gf_debug
      ) else (
         fpm @./rsp/gf_release
      )
      cd .\Src
      ren  CFML_GlobalDeps.f90 CFML_GlobalDeps_Windows.xxx
      cd ..
    )
rem .
rem  Undo the changes of extensions to be compatilbe with Cmake
rem .
cd .\Src
   if [%_WINT%]==[win] (
          call tochange f90 win
   ) else (
          call tochange f90
          )
   )
cd ..

Example or running the script with the option win

c:\ILL_Git\CrysFML>make_CrysFML_fpm ifort debug win
---- Construction of the CrysFML library for 64 bits using gfortran, ifort or ifx (oneAPI) ----
---- The building procedure installs also some executable programs of the Program_Examples subdirectory
     Default: ifort compiler in release mode. Equivalent to the first example below
     Examples of using the script:
            make_CrysFML_fpm  ifort
            make_CrysFML_fpm  ifort debug
            make_CrysFML_fpm  gfortran
            make_CrysFML_fpm  gfortran debug
            make_CrysFML_fpm  ifx
            make_CrysFML_fpm  ifx debug
    For using the Winteracter library add the word "win" as the last argument (without quotes)
----
---- Changing the extension of some *.f90 files to *.xxx to maintain compatibility with FPM
<ERROR>Unable to find source for module dependency: "winteracter" used by ".\Src\CFML_IO_Messages.f90"
STOP 1
---- Changing the extension of *.xxx files to *.f90 to maintain compatibility with CMake
1 Like

The external-modules option tells fpm that the modules are not included in the project or it’s dependencies.

The link option tells fpm to include external libraries in the link command.

Those should hopefully get you far enough to get your project working, but don’t hesitate to come back and ask more questions if you encounter more difficulties.

1 Like

Thank you very much for your tips! I can build the project completely adding to the fpm.toml file the lines in bold face:
…
[library]
source-dir = “Src”
include-dir = "…/…/wint/lib.i64"

[build]
auto-executables = false
auto-tests = false
auto-examples = false
external-modules = "winteracter"
link=[“winter”,“user32”, “gdi32”, “comdlg32”, “winspool”, “winmm”, “shell32”, “advapi32”, “htmlhelp”]

The only problem is that I cannot find the way to make the fpm.toml totally independent of the operating system. In Windows, Linux or MacOS the include directories may be in different places. I have to use an external script for renaming fpm_linux.toml to fpm.toml and so on. It would be very interesting if some kind of “if-then-else” or “select case” were introduced directly in fpm. Otherwise, can we use environment variables in fpm.toml? In such a case, what is the syntax for referring these variables?

I have still another question about the syntax of multiple executables. Using [[executable]] many times work perfectly and all are built and installed correctly. However if I try to use the syntax (as prescribed in the Manifest reference)
executable = [
{ name = “a-prog” },
{ name = “app-tool”, source-dir = “tool” },
]

I have different errors depending of what is the preceding section.
If I just write executable = [{} …], I get Key executable is not allowed in install table

If I use [[executable]] like in:
[[executable]]
executable = [
{ name = “cryscalcon”, source-dir = “Program_Examples/Cryst_calculator_console”, main = “cryscalc_con.f90”},
{ name = “hkl_gen”, source-dir = “Program_Examples/hkl_gen”, main = “hkl_gen.f90”},
{ name = “Formal_Charges”, source-dir = “Program_Examples/BondStr/FormalCharges”, main = “Formal_Charges.f90”},
{ name = “Bond_Str”, source-dir = “Program_Examples/BondStr/Src”, main = “Bond_Str.f90”}
]

Key executable is not allowed as executable entry
STOP 1

If I use [executable] instead of [[executable]] there is no error but the executables are not compiled nor installed. What is the proper syntax for using tables with the list of executables?

1 Like

There is an issue open for this at GitHub: Command line argument to specify manifest location · Issue #611 · fortran-lang/fpm · GitHub

I believe the debate on how to extend the manifest syntax to support conditional options is still open. Perhaps you can comment here on what you’d like as a user: Lua scripting · Issue #623 · fortran-lang/fpm · GitHub

1 Like

Hi, I wanted to try something similar to the problem stated at the beginning of this thread and for testing I took the h5fortran library. I opened an issue but would like to bring it here because it is most likely not a problem with the library itself, I feel like I’m doing something wrong but can’t figure it out:

Working environment: WSL2 + gfortran 9.4.0 + fpm 0.7.0
Following the instruction I compiled successfully HDF5 at a directory resembling:
//mnt/d/tmp/third_party_hdf5/release

I have h5fortran in a directory like
//mnt/d/tmp/fortran-lang/h5fortran

When I run “fpm build” I get the following error:

libh5fortran.a done.
char_repeat_read failed.
[ 88%] Compiling...
/usr/bin/ld: build/gfortran_XXXXXXX/h5fortran/example_ex_fcn.c.o: in function main':  ex_fcn.cpp:(.text+0x0): multiple definition of main'; build/gfortran_XXXXXXX/h5fortran/example_char_repeat_read.f90.o:char_repeat_read.f90:(.text.startup+0x0): first defined here
/usr/bin/ld: cannot find -lhdf5_hl_fortran
/usr/bin/ld: cannot find -lhdf5_fortran
/usr/bin/ld: cannot find -lhdf5_hl
/usr/bin/ld: cannot find -lhdf5collect2: error: ld returned 1 exit status
Compilation failed for object " char_repeat_read "stopping due to failed compilation
STOP 1

I modified the fpm.toml file with these lines to make hdf5 visible:

...
[library]
include-dir = ["../../../tmp/third_party_hdf5/release/include","../../../tmp/third_party_hdf5/release/include/static","../../../tmp/third_party_hdf5/release/lib"]

[build]
auto-tests = false
external-modules = ["hdf5", "h5lt"]

link = ["hdf5_hl_fortran", "hdf5_fortran", "hdf5_hl", "hdf5"] 
...

also tried the following commands:

export HDF5LIB="//mnt/d/tmp/third_party_hdf5/release/lib/"
export HDF5INC="//mnt/d/tmp/third_party_hdf5/release/include/"
fpm build --profile release --flag "-I$HDF5INC -L$HDF5LIB"

Any advise? Thanks

Any output when you run this - not sure if this is available via wsl2:

pkg-config --cflags --libs hdf5

if so you can include these as exports:

$(pkg-config --cflags --libs hdf5)

Just did a quick “sudo apt install pkg-config” and got it :+1:

So, doing:

export HDF5LIB="/mnt/d/tmp/third_party_hdf5/release/lib"
export HDF5INC="/mnt/d/tmp/third_party_hdf5/release/include"
fpm build --profile release --flag "-I$HDF5INC -L$HDF5LIB" $(pkg-config --cflags --libs hdf5)

Gives me:

Package hdf5 was not found in the pkg-config search path.
Perhaps you should add the directory containing `hdf5.pc'
to the PKG_CONFIG_PATH environment variable
No package 'hdf5' found

Plus a bunch of errors of the kind:

...
H5_gen.F90:(.text+0x2cf2): undefined reference to `h5dread_f_c'
...
/usr/bin/ld: /mnt/d/tmp/third_party_hdf5/release/lib/libhdf5.a(H5Zdeflate.c.o): in function `H5Z__filter_deflate':
H5Zdeflate.c:(.text+0x122): undefined reference to `compress2'
/usr/bin/ld: H5Zdeflate.c:(.text+0x22b): undefined reference to `inflateInit_'
/usr/bin/ld: H5Zdeflate.c:(.text+0x25d): undefined reference to `inflate'
/usr/bin/ld: H5Zdeflate.c:(.text+0x27e): undefined reference to `inflateEnd'
/usr/bin/ld: H5Zdeflate.c:(.text+0x2d4): undefined reference to `inflateEnd'
/usr/bin/ld: H5Zdeflate.c:(.text+0x366): undefined reference to `inflateEnd'
/usr/bin/ld: /mnt/d/tmp/third_party_hdf5/release/lib/libhdf5.a(H5PLint.c.o): in function `H5PL__close':
H5PLint.c:(.text+0x4d1): undefined reference to `dlclose'
/usr/bin/ld: /mnt/d/tmp/third_party_hdf5/release/lib/libhdf5.a(H5PLint.c.o): in function `H5PL__open':
H5PLint.c:(.text+0x533): undefined reference to `dlopen'
/usr/bin/ld: H5PLint.c:(.text+0x54e): undefined reference to `dlsym'
/usr/bin/ld: H5PLint.c:(.text+0x5c1): undefined reference to `dlerror'
/usr/bin/ld: /mnt/d/tmp/third_party_hdf5/release/lib/libhdf5.a(H5PLplugin_cache.c.o): in function `H5PL__find_plugin_in_cache':     
H5PLplugin_cache.c:(.text+0x359): undefined reference to `dlsym'
collect2: error: ld returned 1 exit status
<ERROR> Compilation failed for object " char_repeat_read "
<ERROR>stopping due to failed compilation
STOP 1

Following the error message at the begining I tried adding the folder containing hdf5.pc by:

export PKG_CONFIG_PATH=//mnt/d/tmp/third_party_hdf5/release/lib/pkgconfig
export HDF5LIB=//mnt/d/tmp/third_party_hdf5/release/lib
export HDF5INC=//mnt/d/tmp/third_party_hdf5/release/include

fpm build --profile release --flag "-I$HDF5INC -L$HDF5LIB" $(pkg-config --cflags --libs hdf5)

Now I just get:

UNKNOWN COMPOUND SHORT KEYWORD:I in -I//mnt/d/tmp/third_party_hdf5/release/include

pkg-config is adding a layer of confusion, because you are trying to link with self built packages, not system installed packages. I’m not sure how to solve your original question, but it is not likely pkg-config will help unless you prefer to install h5fortran using apt in the WSL Ubuntu, and link to that version instead.

Exactly, I want precisely to avoid installing globally to understand how to link with locally compiled libraries. Still working on that one, thanks for the tip regarding pkg-config!!