A streamlined preprocessor written in Fortran

Announcements
1

Since the on-going work on such features as standardizing new templating
and generic programming features in the Fortran standard is expected
to take a while there is related functionality that can be supplied
immediately by preprocessors.

There are several already listed in addition to the often-cited
fpp(1), cpp(1), and m4(1):

but I was looking for something written in Fortran that supports
some form of templating and direct support of blocks of plain text
and did not quite see what I wanted, so there is now

Which except for bug fixes I will leave as-is for stability; but I am
branching a version called “pref” instead of “prep” for prototyping
new ideas formed from lessons learned with “prep”.

Looking for feedback on “prep” and related topics, such as whether
your use of “fpm” is limited by needing preprocessing, if you use
other preprocessors not listed at the Wiki (is so, adding them to the Wiki would be useful), …

4 Likes
2

There are tons of pre-processors for Fortran. What are the pros and cons of your pre-processor over others?

I can see an advantage that it is written in Fortran (thus can be easily incorporated in fpm).

3

It allows for templating via variable expansion using a POSIX-shell syntax which many are familiar with, and allows for plain text blocks to be written to a file, allowing markdown, HTML, and Latex to be easily incorporated straight in the code file. Uniquely to my knowledge the plain text blocks can be filtered into a CHARACTER array definition or comments, including a basic flag (just preliminary) that allows the same text block to be formatted for use with Ford or Doxygen as well as as-is comments. I use preprocessing for documentation more than for conditional compilation, which is the focus of preprocessors like cpp and fpp. It intentionally does not use a pound prefix by default so that files (especially C) that already contain cpp(1) directives can be mixed in without collisions.

It also allows environment variables to be imported, and by default $IFNDEF|$IFDEF will test for environment variables as well as variables declared with $DEFINE. This allows for easier communication with the programming environment than fpp(1) or cpp(1) which require that kind of information to be passed in on the command line.

I would like to see fpm(1) hide the preprocessing stage more like C/cpp(1) does, where you would maintain the input files in the user space but the post-processed files would be placed in a directory in build/ much like the *.o files are now, avoiding the collision issues and file clutter most Fortran preprocessors require where you have the input files and output files both visible in user-space; and preprocessing is particularly problematic with fpm if the preprocessor is not easily bundled with fpm and useable in an environment not easily modified with external dependencies by the user. prep(1) is designed with that in mind, but I think the issues of preprocessing and fpm(1) are not near being resolved in the manner I am hoping for.

Some of the cons are it is not reversible, as is Dan Nagle’s superset of coco(1); it does not directly allow the use of popen (or any non-Fortran-standard feature) like ufpp(1); which basically allows for directly using any other shell language to generate code (a very powerful feature, but fraught with dependency issues if not used judiciously) and does not have direct macro parameters, which I am not particularly fond of. f90ppr and ufpp allow for more general logical expressions, supporting expressions and reals. I could switch in M_calculator in less than a few hours but I think there are drawbacks to making a preprocessor too language-like. I allow reuse of blocks but not true looping or a “for” statement. The preprocessor is intentionally kept to what I think is the basic core functionality , but that is limiting in some respects. Fortran pre-processing has been an issue for decades and even though there are a number of pre-processors none has seemed to win out. As Fortran has expanded in capability and standarization has become more common some of the conditional compilation issues have reduced in recent years but are not gone; and the lack of templating and documenation features is not likely to be standardized soon so I am trying to consolidate a solution for those issues into preprocessing in a sustainable but hopefully useful manner.

On the con side cpp(1) and fpp(1) are often tightly coupled with the compiler, allowing for far more pre-defined variables. Allowing for importing of environment variables addresses this to a small extent.

I am not inserting "# NNN “filename” lines for the debugger, but line counts and input files are tracked so that would be trivial to add.

I am not warning about line length nor doing automatic continuation lines. Again, easy to add but there should at least be a warning, as the changes to Fortran to eliminate the 132 column default line length limit will not be available on all compilers for a while although a lot of compilers already allow that.

2 Likes
4

I know this topic is hotly debated and disliked by the Fortran committee. But it would have been great if the committee took the issue of a Fortran preprocessor more seriously. If there are so many users actively using it, to the extent that they have to create new preprocessors for their needs, and if all major compilers support some form of preprocessor that is heavily used by the Fortran programmers (including the committee members), that is really pointing to some gap in the language.

One major application of preprocessor that will not be resolved even when templates become part of Fortran, is the issue of code sections that belong to a particular platform or parallelism paradigm. Let’s agree with the committee’s consensus that the use of preprocessor encourages bad programming habits. What is the alternative in such use cases that I mentioned? Code duplication? My comment here is more of a request for help than attempting to argue for the usefulness of preprocessor. The question is whether the benefits of standard preprocessor recognized by all compilers outweigh the drawbacks.

8 Likes
5

Here is the discussion about pre-processor regarding the standards committee: The preprocessor should be standardized · Issue #65 · j3-fortran/fortran_proposals · GitHub

1 Like
6

Perhaps ironically, I fall into the camp of disliking preprocessing, at
least to the extent of feeling it is a “necessary evil”, not a desirable
language feature.

Preprocessing can run the gamit from something that redefines the syntax
of the language (see RATFOR) to what Fortran provides (the INCLUDE
directive) with a lot in-between (m4, cpp, fpp, …).

The COCO preprocessor did momentarily make the cut as a standard
pre-processor, but it sounds like you might be familiar with that history
already. Looking to COCO as having been an “acceptable” preprocessor
somewhat influenzed what I added to prep(1).

To see how preprocessing can go wrong, see C (no amount of flames will
change my opinion :>).

Along those lines long ago I wrote a paper on the pros of
minimizing preprocessing and how to isolate it in wrapper routines
whenever possible. I might see if I can add that to the prep(1) or
pref(1) repository.

As a quick summary, one Fortran-ish way to minimize preprocessing is to
create programming-environment specific directores for routines, like

gfortran_mswindows/ gfortran_posix/ generic/

and then use the INCLUDE path option (usually “-I”) found on most
compilers where you have duplicate filenames in the directories and choose
one (the biggest issue here being excessive duplication of code). Some
compilers will find the first instance of the file and use it, so you
can always add -Igeneric last, but others will complain if duplicate
filenames are found.

You should always isolate code requiring extensive preprocessing in it’s
own routines in its own files as much as possible, so the body of your
code is preprocessor-free.

If there is a lot of routines involved, especially if they are re-usuable
making a separate library of the programming-environment-specific routines
is almost always a good idea.

A lot of conditional compilation can now be eliminated by using the
ISO_C_BINDING interface, as a lot of it in the past involved C interfaces
directly or indirectly (using a Fortran extension that provided the
same function as a C function).

When the conditional code still compiles on all your platforms you can
compile all the versions and use function pointers or pass procedure
names but even in those rare cases you have to be careful not to impact
performance.

I hope the stdlib project will hide preprocessing for a large amount of
users in the near future by providing things like a “POSIX” interface,
but stdlib will require some form of versioning or preprocessing for
system-related and compiler-specific issues. Even issues like support
of REAL128 kinds, and certainly an extension such as REAL256 and CUDA
features make me pretty certain the need will not be going away anytime
soon.

In the case of prep(1), better support in the Fortran standard of generic programming and
perhaps some support of plain text blocks will/could go a long way towards
reducing any need for it but in the meantime I hope it will be useful.
Fortran has multiple compiler providers, unlike a lot of recent languages.
This tends to promote the need for conditional compilation.

4 Likes
7

Thanks for the extensive response and advice. I began some serious projects in Fortran several years ago with the initial goal of full-standard compliance. But soon it became clear to me that it is not possible in any efficient manageable way (not that it is a limitation of Fortran. Any language without preprocessing capabilities would present the same challenges). For example, one project involves three implementations (serial, Coarray, and MPI parallelism). Although MPI and some existing Coarray implementations are consistent with each other and can be mixed, it is completely illogical to combine the two in this particular case. But duplicating the entire code base for each paradigm is also nonsensical. The only resolution was to use conditional compilation. Fortunately, the preprocessing features of some compilers are partially consistent with each other, like those of Intel and GNU. But it would be great if these features were standardized so that users know what can be relied upon and what is portable across standard-conforming compilers. I am generally hesitant of using external community-driven preprocessors, especially if it written in some other non-Fortran language. Even though they may be super-beneficial from the developer’s perspective, they generally add a layer of dependency which can complicate and lengthen the build process for the end-users and be a barrier toward full automation of the library build.

4 Likes
8

Indeed. One way to use conditional compilation is to include different modules (as a whole) using a build system. This works as long as the rest of the code uses exactly the same API. If it does not, then you need some kind of pre-processing.

3 Likes
9

I particularly like submodules for this.

For each instance where I need conditional compilation, I like to have a single module file which provides consistent interfaces, and then each conditional option has a submodule file which implements the interfaces in the module file in an option-specific way. IMO this minimises the room for error, removes the preprocessor dependence, and as a bonus minimises re-compilation when options are changed.

9 Likes
10

@veryreverie great idea! This might be the first use case where I would actually like submodules. :wink:

1 Like
11

Have you still got a copy of the paper somewhere and if yes, could you make it publicly available?

12

will this preprocessor be considered a replacement of fypp?

13

@aradi and I would have to brainstorm what the best path forward is. Given that fypp can in principle execute any Python statement, it’s impossible to create a pure C++ implementation. However, in practice we don’t seem to use this feature in stdlib, so in practice a usable subset of fypp should be possible to implement in C++.

14

Some thoughts and clarifications …

I asked about the paper and I cannot distribute it even though it is still available; but a new version today would be considerably different, emphasizing templating and conditional compilation of various parallel-related methods and how to minimize preprocessing in regards to those aspects of coding – and when abstracting the parallel methods is preferred instead.

CRAY was the only one supporting F-/Coarrays at the time and GPUs were almost exclusively accessed via boutique languages then so it needed an update anyway.

prep(1) was not intended as a replacement for fypp which did not exist when the ancestors of prep(1) such as ufpp(1) were created, but as a Fortran-based and Fortran-targeted tool, which has been lacking in general for decades (accept that COCO was implemented in Fortran). I need a preprocessor with the longevity and portability of Fortran that is maintainable by Fortran-centric programmers, and to me that means it is a major plus if it is written in Fortran.

I hope that fpm(1) in particular can support user-specified pre-processing agnostically as is possible with make(1); and that prep(1) would be a good preprocessor solution for a good part of the community, as it is for me. It has proven to be very portable and easily built in any Fortran environment, including a nearly exclusively-Fortran environment.

I would like to see a method evolve in fpm(1) like I described where pre-processing is treated somewhat like compiling in that the post-processed files are created in the build/ directory and then compiled from there, continuing on the theme in fpm that output files are removed from the user directory and can exist for each custom build simultaneously.

In my own build environment ( which now coexists with fpm in everything I have managed to extract into a github repository) only the prep(1) files exist in the source directories and all source files requiring preprocessing create output in a scratch directory which is then compiled and placed into an ar(1)-build library so no .f90 or .o files exist afterward. This is much cleaner and allows for concurrent builds compared to a typical make(1) build; so I would really like to see fpm(1) continue on a somewhat similar path. make(1) (even before gmake) actually supports libraries in such a way that a conditional rebuild can be triggered from the entry in the libXXX.a file instead of the “.o” file, and by including compiler-specific make files can support multiple compilers relatively elegantly but it sure was not obvious to me how to do that at first years ago; but fpm(1) does much the same with option-and-compiler-specific build/ subdirectories; so I really like that.

Neither fpp/cpp nor fypp quite fits my needs and I have several million lines of code that depends on a prep-like preprocessor primarily to allow for documentation to be included in the source files and for parallel programming and so do others so support is on-going.

Templating has only recently become a significant need, as I mostly deal with 64-bit floats and default intrinsic types or at least user-defined types defined with a small variation of intrinsic types myself, but I see the need; so one near-term planned upgrade is to improve the templating features.

So prep(1) is something I will continue to need, but from the (lack of) response I saw in any adoption of it here as a starting point for a ubiquitious tool my only current plans are to improve the template capabilities, to possibly add an option for generating a single output file for project-specific reasons, to make an fpp-compatible mode and to make it support more pre-defined macro values. That is particularly vexing as it is an important feature but making calls to the compiler to dump the pre-defined macros (which initially seemed promising) and then importing them had more issues than I hoped.

So I do not see prep(1) replacing all the various pre-processors (including fypp(1) or any other new or existing processor) but I see it as a tool I need that is a platform-independent Fortran-based tool that lets me write better code faster that I can easily extend when I need too. It sounded like it would fit a niche for others too, and if so they are welcome to use it in any fashion but I did not see any major indication here with this effort that it or any other preprocessor will be a single standard preprocessor for Fortran any time soon without J3 and the compiler vendors getting behind such an effort, although I took a stab at making prep(1) a seed for starting a community-based effort in that direction (in particular with fpm(1) usage). Didn’t see much of a spark or feedback though, so I wanted to get back to a few other things for now, aside from a newer expression parser I have as the current one in prep(1) has been quite adequate for what I need, but leverages some old F77 code that could stand a refresh.

I still think there would be great value in a preprocessor being distributed with fpm(1). When I distribute something like a library myself I distribute the output files from prep(1) as .f90 files and rarely as .F90 files (assuming fpp/cpp is available). Over 90% of the files in my GPF github repository are actually maintained as .ff (my preferred prep(1) file suffix) but all distributed as .f90 files. It would be useful to be able to distribute files that needed more extensive preprocessing as-is but I don’t see that being a good path forward yet except for fpp-compatible directives (for now).

Actual answers …

So I think guidelines about preprocessing would still be useful and that perhaps we can start a Fortran Wiki or fortran-lang collaborative document about it; but I cannot distribute my old paper.

prep(1) apparently did not light any fires but I use it myself and the source is a simple Fortran code (and Fortran is notoriously good at upward compatibility) so it is a good tool to depend on if it is a good fit for you; but I think fypp(1) also looks like it will be around a while too.

I am more a fan of the idea of fpm(1) supporting preprocessing in general than a particular one, but think even given that that having one bundled with fpm(1) would be useful, and personally would prefer it to be Fortran-based like Nagle’s COCO(1) or prep(1).

Along those lines I think I want a starting base of a Fortran-based fpp-ish preprocessor available first, and then some in-source documentation support and templating added to that; but others have other needs. The biggest fight there is that fpp(1) itself is not standardized and varies widely in how or if it supports macros with parameters.

3 Likes
15

Has it been decided that fypp is preferred over prep? If so, what are the reasons?

16

There were enough interesting suggestions I created a new release of prep(1) that incorporates them all (except for a modern modularized rewrite of the expression parser, I just have not had time (I started it off with some OLD code)) in

Release v8.1.1 of prep(1) - a Fortran preprocessor · urbanjost/prep · GitHub

Thanks for the feedback. Suggestions still welcome! The processing of extended markdown and tex files; and the --debug mode along with the $HELP cribsheet for interactive tests and “$BLOCK SET” are my favorites so far.

2 Likes