I come to bury .mod files not to praise them

Advocacy
1

TLDR:We have lived with .mod files for 35 years at least. That’s enough. It is time to give them up. Efficient recompilation requires fine-grained analysis. What can replace them? And what other wins can an alternative enable?

To kick off the discussion, I want to present a half-baked idea: the lazy compiler leveraging a database.

The lazy compiler, given a filename that is recognised as Source (a “load”) proceeds to complete the stages up to semantic analysis. If valid, it stores information about the canonicalised lexed source, the program units, the scopes and symbol tables, the module USE dependencies, etc in the database. Once it reads the source, what happens to the file containing the source is immaterial (e.g. it can be piped in). The context is a user-selected Fortran identifier (e.g. MAIN_WEATHER or LIB_MPI). A duplicated global identifier is identified as an error by consulting the current database (with current compilers you might get a linker error, or nonsensical/unintended results).

A special option on the compiler enables the next stage, code generation from the digested source in the database. First, the database context is consulted for existing compilations of the global identifiers. The first time, the database has none. The compiler walks the dependency chain already in the context and determines an order of compilation (including parallel jobs) and executes them. A subsequent time, after a “load” of new versions of some global identifier, a program unit (or possibly a more fine-grained entity) can be determined to not be affected by the change, and the last recorded object code is marked as valid without new code generation needed.

The essence is that the compiler is in charge of what needs to be recompiled and when, not the build system.

Shall we build it?

The evil that men do lives after them

4 Likes
2

After handling *.h files in C/C++ and paths using #include "lib/core/my_lib" and #include <core/my_lib> and the same way of doing things in Python I feel that .mod files are the lesser of evils. With CMake you can very nicely say where they go and installing them is simple, you can always just put everything into your install_prefix/modules or something like that. CMake also gives you a good way to control that and with the FPM it is super easy.

What kind of workflows/scenarios are you having where the mod files annoy you? I am working with fairly large codebases and using reusable libraries across them and .mod files have not been a concern at all…I am curious.

Compiling is also simple-ish right, the compiler knows what it needs to recompile if you do a change. In C++ you have the same problem when you modify the one header file that is everywhere and god it is awful. In this day an age, unless you’re building on a microwave you can do at least make -j4 and make things go really fast.

1 Like
3

I think the title of this thread is misleading.

The issue is not with .mod files, but with the fact that the compiler is still just a compiler, and @themos proposes it could be more.

Modules and submodules are fine, even though the incompatible formats are annoying. But after 35 years, the compiler still expects the source files to be given in a certain order (because we seem to live in a C world, no matter the language), and it has no concept of “cached build”.

If you take Go, for example:

They went with import instead of include precisely to avoid the header-guard-hell. And their packages (directory-bound, akin to our module+submodule) have an associated cache.

So, if you have a set of packages prone to slow compilation (e.g., gotk3, since cgo is slow as hell), they only have to be compiled once for the current Go version —and used by multiple projects afterwards.

4

Yes, that’s the plan for LFortran. It’s obvious the compiler has to be in charge of building things, not the build system. fpm should communicate the file dependencies and other compilation info, but the compiler should actually do the compilation. It could possibly even just have things (i.e. the database) stored in memory, so doesn’t even need to read anything on recompilation, just the changed file.

There are multiple ways to design it. Zig does something similar also. One has to experiment to see which way is the fastest.

The motivation here is twofold:

  • speed of compilation and recompilation (that is probably the number one motivation for me)
  • easier handling for the user, no need to worry about .mod, paths and other issues

Depending on which one of these two you prioritize, it will also change the design possibly.

5

Under prefix_install/modules, could you use sub modules, like prefix_install/modules/aMod, prefix_install/modules/bMod, …

In c, you can use include/GL, include/myinc to organise.

Thanks.

6

But doesn’t the compiler have a more accurate view of the module and include file dependencies?

Currently fpm uses an approximate parser that sets a number of assumptions on the source formatting when determining dependencies. The last time I checked there were limits like no broken module import statements, free-form syntax, no include file parsing, rudimentary sub-module treatment, no special treatment for keyword clashes, e.g. subroutine interface() could trip the parser.

Some Fortran compilers (nagfor, ifx) already support emitting this information via the Make target syntax, so it can be stored in a dependency file and later parsed by the build tool. Ninja uses this strategy for C and C++ headers, but the authors of Ninja already noted that,

It turns out that for large projects (and particularly on Windows, where the file system is slow) loading these dependency files on startup is slow.

The lazy compiler design where the compiler is in charge of traversing the build graph would not have this issue of inter-tool communication.

It does seem like this could be an advantage in the agentic programming era, an agent would be able to rebuild and test the code quicker and with better build correctness guarantees such that files that have changed are indeed recompiled, or if files are deleted, that parts necessary are rebuilt.

It has bothered me that CMake also uses their own custom yacc-based Fortran parser and there are documented issues of this breaking builds. I remember a case where CMake failed with the example,

submodule (foo) foo_sm
contains
    module logical function bar()
    end function
end submodule foo_sm

CMake’s dependency scanner emitted a rule expecting a module named logical. The “fix” was to switch to,

module function bar()
   logical :: bar
end function

If it were the lazy compiler driving the build, I would expect these sort of bugs to vanish.

2 Likes
7

Yes. Probably the dependencies should be handled by the compiler itself. But I think fpm knows: which files to compile (including all package dependencies) as well as any specific compiler flags per file and globally. So all this info should be given to the compiler.

Alternatively, and I also thought about it, the compiler can just follow and understand the fpm conventions and it will just build things directly without even fpm doing anything. So the compiler itself would watch if you add a new file. That would also work.

8

Imho, efficient recompilation requires foremost code decoupling on the user’s side. This is the elephant in the room, and this is (among other reasons) why Fortran cannot do without language support for modern abstraction capabilities.

Improvements to the mechanics of recompilation on the compiler’s side would, of course, be welcome. But they would merely amount to treatments of a symptom of the problem (recompilation cascades), and not the disease (tight coupling).

To cure the latter, traits-based generics and OOP facilities must make it into the language, that will allow for the use of all modern software engineering practices (i.e. dependency inversion, dependency injection (DI), and construction and use of DI containers).

1 Like
9

I would not support this statement. I think it makes sense to have separate tools for different tasks. Figuring out dependency and producing fast machine code are different things, even though both need to parse at lease some of the code

I certainly see the advantage monolithic “Programming Systems” like Rust or Julia (probably also Zig, I don’t know it) where everything is bundled together. Still, I think that modularity is good. It would for example allow to integrate GCC-based Rust compiler into the existing ecosystem. Python, despite having only a single interpreter of relevance, has now support for multiple build systems.

The difficulty is of course to agree on a standard for the communication between compilers and build systems:

  • for Python it took years until the current solution emerged.
  • C++ has a proposal for such a standard, which is stalled.
2 Likes
10

There is a recent talk on YouTube, here, starting around 19 minutes in that discusses this. Since C++ has modules in C20, this is getting quite a bit of attention. Sounds like there is a push to develop a standard way for the compiler to return the dependencies for a given file in C++ to drop the need for CMake to bring its own parser.

11

Producing fast machine code and compilation speed are indeed two different things (two different modes in a compiler).

But figuring out dependencies is related to compilation speed. So if you care about compilation speed, then it’s highly relevant.

12

I don’t know any of the details of C++ modules, but from this comment it looks like the difference is that the file names are referenced in C++ whereas in fortran it is the module name. Those are not one-to-one, a file can contain multiple modules, and multiple files can contain multiple copies of those modules which might differ by algorithm, OS environment, compiler, and even compiler version or compiler options. The fortran standard committee has always just ignored this issue, and that is why external facilities like make, Cmake, and now fpm have been developed to fill in that missing connection. This has been compounded over the decades because fortran traditionally has not supported a standard preprocessor, which can also be used with the combination of macros and conditional compilation logic to also allow a single file to work with various combinations of compilers, OS, OpenMP, and algorithm options. So now we are in a position where it is impossible for a fortran compiler to look at just the source code in various files and determine which combinations are compatible or desired. The compiler cannot read the programmer’s mind, and there is no standard way for the programmer to convey that information to the compiler.

The discussion in this thread assumes there is a solution to this problem, and it is just a matter of choosing how to implement it. I doubt that there is even a solution, at least not one that is backward compatible (e.g. with fortran modules) or one that is compatible with existing practice (e.g. de facto C preprocessor, macros defined on the compiler command line, makefiles, etc.).

13

But figuring out dependencies is related to compilation speed. So if you care about compilation speed, then it’s highly relevant.

That makes sense to me. @certik is my following understanding correct or at least close to what you have in mind when developing LFortran? It’s the naive understanding of a compiler and build system user with no experience in developing such tools.

There are two contributions to compilation speed:

  1. The raw performance: How fast is a single file/compilation unit compiled given that all dependencies are fulfilled. The potential for parallelizing that is rather small and this is entirely the responsibility of the compiler
  2. The coordination/orchestration: In which order does the compilation happen. Depending on the structure of the project, there is a huge potential for parallelization. That is currently the task of the build system.

For point 2, i.e. what is discussed here, two design choices can be made:

  1. A static plan for compilation order that is fixed once the dependencies are known.
  2. Dynamic scheduling, e.g. in the form of a thread pool.

For an optimal plan, it is important to know/estimate the duration of each compilation step. This information can only come from the compiler.

Other aspects that came to my mind are:

  1. Interprocedural optimization (IPO) complicates the picture and additional information is needed to get the optimal performance.
  2. For dynamic scheduling, knowing the compile time is less relevant.
14

The file is dead. Program units are the top level construct of Fortran. I will propose a new outer level, CONTEXT. They are bags of program units and other CONTEXTs. The compiler will detect the presence of more than one global identifier (or main program unit) in a CONTEXT and report it as an error.

Fixed form is dead. It is automatically converted to free form which is the only form that syntax/semantic analysis acts on and reports on.

USE without ONLY is dead. It’s as bad as implicit typing. I would mark it obsolete. The compiler will find out which names are accessed, why shouldn’t YOU know when a name you did not expect was public in module A suddenly starts being resolved in your code?

Dependencies and fast code generation (you can put a hyphen in both places) go together. When the compiler knows that only one loop was affected by a change, it can deploy the optimization resources efficiently, to either finish the compilation earlier or to do it better.

1 Like
15

USE without ONLY is dead. It’s as bad as implicit typing.

It’s bad because Fortran has no namespaces. only helps to avoid conflicts that exists because of that, but an IMHO better solution would be the introduction of namespaces.

2 Likes
16

How about types with overloaded operators? I do not like typing,

   use dual_numbers, only: dual_t, operator(+), operator(-), &
       operator(*), operator(=), ...
4 Likes
17

The absence of “namespaces” (qualified imports) doesn’t have that much to do with it. It’s bad because Fortran programmers typically do not care to do proper encapsulation. They prefer to use dozens upon dozens of global variables and loose collections of module procedures. Hence the use of

use mymod

or
use mymod, only: mod_var1, mod_var2, mod_procedure1, mod_procedure2.

The second form is equally bad, given that the language allows one to simply write:

use mymod, only: MyType

I do not recall a single modern Fortran code, even a predominantly object-oriented one among those that I’ve seen published on Github, that refrained from using the first two forms. All of them violated the principle of encapsulation/data abstraction, to various degrees.

18

That’s literally the issue number one at Namespace for modules · Issue #1 · j3-fortran/fortran_proposals · GitHub. In two weeks it will be 7 years! By now it could have been in every compiler. I even brought it to the committee and made it discuss it, see the issue for details. Then it was put for F28, but I think it’s now dropped. There were some objections (see at the very bottom of the issue) which I think got resolved in the comments there, but I think it’s not on the list anymore.

I think LFortran is now advanced enough that we can prototype it, which would resolve all objections that something wouldn’t work, because we would make it work. So far I don’t see any blockers.

5 Likes
19

Lack of namespaces was astonishing to me when I was learning Fortran. It almost made me abandon the language when first learning (sometimes when I see things that Fortran is missing or does strangely it almost makes me wish I did at that point). I hope it gets reconsidered for a future version (but at that point I don’t know if I’ll be fully converted to C++ since Fortran has a lot of misses).

1 Like
20

The absence of “namespaces” (qualified imports) doesn’t have that much to do with it. It’s bad because Fortran programmers typically do not care to do proper encapsulation.

That is a question of programming paradigm. For object-oriented style (as in C++), a single import would indeed be sufficient. I believe that your example suggests that each module implements a single type (or a few types) and all procedures are bound to that type:

use mymod, only myType
type(myType) :: myVariable

call myVariable%procedure1

In that case, only is not even needed because everything besides myType can be declared as private.

When using functional style, a module will contain many functions that operate on it’s arguments only. In that case, namespaces would indicate where these functions come from.

Note: When writing this, I realized that the object-oriented capabilities of Fortran can be used to emulate namespaces. If myType has no members, i.e. a state, myVariable would be the namespace. I still have to decide whether I like this hack.

2 Likes