The recompilation cascade issue and its solutions

Help
1

Hello,

The submodules have brought a solution to the well-know recompilation cascade issue when using modules, by allowing to separate the implementation from the interface. But I was wondering if the recompilation cascade could be avoided in some cases without the use of submodules.

If in a library module one doesn’t change at all the PUBLIC part (that is the interfaces of the public routines do not change, no public routine is added/suppressed, and no public variable or type is added/suppressed/modified), then it is safe to NOT recompile a source file that uses this module when rebuilding the executable?

Note: how to detect/decide that the public part has not changed is another topic that is not the point of my question

2 Likes
2

I think theoretically, it should be possible to avoid recompilation in that case. There is a short discussion on this topic in Build systems that know when to recompile -- revised

2 Likes
3

From my own experience submodules solve the recompilation problem only partially, as you still require some modules for interface declaration. This is particularly true if type-bound procedures are used, where importing the interface also imports the implementation. This has been discussed previously, including solutions involving lots of boiler-plate code.

Moreover just looking at public declaration and interfaces is not sufficient. For example, changing the procedure body might lead to different register usage, which then might change how the routine must be called internally (e.g. stack layout and which registers to use to pass arguments). I have seen this happen but did not try to understand the details by looking at the compiler produced assembler code.

However, gfortran does a really good job at avoiding to update (and change time stamp) the mod files. And cmake (or ninja?) sets up dependency such that recompilation only occurs if mod has been changed. That’s also why I occasionally observe that implementation seeps into internal details of the interfaces.
(For example, if a non-public routine is added, then in general there is no recompilation.)

4

This is the kind of problem that I feared…

Would declaring the routine with BIND(C) be a solution? The C interface is supposed to be independant from the implementation in any case… Would be a trick (I don’t like such tricks, but…)

Or any compiler option that would ensure that the interface doesn’t depend on the implementation…

What about ifort? A solution could also be to compared the mod files as suggested in the other thread, and overwrite the old one only if the new one differs.

5

There was an interesting discussion a while back:

This should answer some of your questions, in particular regarding ifort. Maybe newer ifort versions (llvm based) are different, but I doubt it.

6

It looks like ifort is particularly bad at this :frowning: (I knew that in the past, but was hoping it could have improved).

This is actually the only reason that I can see to have processor-dependent binary mod files instead of more or less standardized text files.

This can be powerful in some cases, but in practice this can also be highly frustrating to not have any control on the mod files.

7

Let me describe a bit the context. We have a huge data processing software, which is made of application units (I would say modules, but this would introduce some confusion with the Fortran module), and librairies.

Unit_A
Unit_B
Unit_C
…

Library_a
Library_b
Library_c
…

(the _a and _A have no relationship: Unit_A can use any library for instance)

Both the units and libraries are versioned independently: Unit_A can be at version 45, Unit_B at version 10, Library_a at version 150, etc…

Because there are tons of units, the software is never complied and linked globally. A unit or library is compiled only when a new version is released, and the link is “on demand” with only the required processing units and libraries.

To add a bit of complexity, different versions can cohabit, and the user can pick any available version (the latest by default) to link with:
Unit_A v40
Unit_A v41
…
Unit_A v45

There is a constraint: once a Unit_X version “n” has been compiled against a Library_y version “m”, then all newer versions of Library_y (version p>=m) are required to be compatible with Unit_X version “n”: the objective is of course to not have to recompile this version of Unit_X.

With include files provided by the libraries it’s manageable: any modification in the implementation of the library won’t change the interface described in the include file. It’s a fully different story if the library provides mod files, and if a modification in the implementation can potentially change the description of the interface in the mod file…

This is close to a no-show…

8

There is an excellent approach to implementing cascade-free and parallel build documented at
https://aoterodelaroza.github.io/devnotes/modern-fortran-makefiles/ and https://www.osti.gov/biblio/1393322. I have implemented this in a substantial existing modern Fortran codebase, and it really does work very effectively. On a 128-core node I can get around 100 compiles running simultaneously with “make -j” and > 10x improvement in build speed.

1 Like