New release of fpx

Announcements
1

I just published the first release of fpx, an extended Fortran preprocessor.

This project aims at providing a simple, embeddable, open-source preprocessor written in modern Fortran. fpx is mostly compliant with a C preprocessor, fine-tuned for the specificity of the Fortran language. fpx is an embeddable preprocessor. It can be used as a command-line tool or directly embedded into any solution.

fpx supports:

  • conditional compilation with #if, #ifdef, #ifndef, #elif, #else,#endif

  • simple macros and functions like macros with #define, #undef, defined, and !defined

  • simple arithmetic and bitwise operations with +, -, *, **, /, >, <, >=, =<, ||, &&, |, ^, &, ! and ~.

  • include files with #include

  • variadic macros with __VA_ARGS__, and __VA_OPT__,

  • build-in macros as __LINE__, __FILE__, __FILENAME__, __TIME__, __DATE__, __TIMESTAMP__

  • stringification # and concatenation ##

  • and more…

Command line

The preprocessor fpx can be used from the command line using your favorite shell. The following options are available:

Option Definition
-D Define a with no value.
-D= Define a with as its value.
-U Undefine ’
-I Add to the end of the global include paths.
-h, -? Display this help.
-o Output file path with name and extension.
-v Display the version of the program.

Using the file preprocessor could not be easier. The function simply takes as arguments the input and output file paths.

Embedded

program test
    use fpx_parser
    
    call preprocess('tests/input.f90', 'tests/output.f90')
end program

*For more examples, please refer to the Documentation.
At the moment, the fpx passes all the tests from LFortran and tests of flang targeting modern Fortran. Some of the flang tests might differ since they are quite opiniated, especialy concerning the rules for line continuation in and after macro substitution.

16 Likes
2

Before proceeding with the development of fpx, I would like to propose this survey to determine which functionalities are the most useful. I collected some ideas of mine, but you have more, do not hesitate to share.

  • Support #foreach(a,b,c,…) keyword, to build multiple versions of the same function with various kinds
  • Look for #include files in PATH
  • Read response files (.rsp) to substitute intel fpp with fpx
  • Read global macro definitions from a configuration file
  • Evaluate string intrinsic functions at preprocessing time (e.g. __FILE__(index(‘', back=.true.):index(’.', back=.true.)) to return the filename.)))
  • Other ideas?
0 voters
3

Another feature that would be nice would be to automatically invoke the preprocessor by the fortran compiler, i.e. without a separate preprocessor step and without a separate intermediate output file. Requiring a separate output file makes debugging more difficult because the line numbers will refer to the intermediate file rather than to the original source file, and it makes the build process more difficult because of the extra step required in the *.f90 → *.tmp → *.o process.

4

Do you see this as complementing the proposed standard preprocessor for F202Y or as a direct replacement with additional features?

5

I think the preprocessed files offer hints for the correct next line number in the form # <number>, don’t they? (at least ifort/ifx do).

As @rwmsu posted (while I started writing this), there’s already an effort to standardize the preprocessor, and also an effort to obviate the need of #foreach (...) through the auto-generic subprograms.

So, how would an external preprocessor fit in, when it’s supposed to always be run by the compiler —regardless of flag or file extension?

6

That’s the idea. I would like to offer the possibility to substitute fpp or cop with fpx without any additional step. I started to collect some info about it. The results of my research are available in the Documentation. Intel for instance proposes the fpp-name flag to pass the name of another preprocessor. All I am missing is to be able to read response file used to pass the command line arguments.

Besides this fpx already supports reading from stdin.

Finally, as mentioned by @jwmwalrus, the #line is used to keep track of the original file line number. Both fpp and fpp work this way. It’s not supported in fpx yet though. Maybe this should be the next thing to do.

7

Features I find useful are an option to recognize environment variables as defining a macro, a block text feature than can optionally export the text to a file for post-processing, an option to name the block of text and reuse it for templating, allowing #include to include files via curl/wget options, allowing #include to filter the plain text input into a character array definition, into base64 data (allowing including PNG, PNM, GIF, … files, for example) are features I find I use more than the base if/else/elseif/endif selection options.

1 Like
8

Thanks for bringing this to my attention. I was not aware of this. I hope it can finally make it to the standard.

That said, history showed that it is not that easy to standardize the preprocessor (see the coco case). The big difference I see with the coco attempt though is that, this time, the proposal is much closer to a C compliant preprocessor, with tweaks, which I am perfectly aligned with.

(@jwmwalrus, I try to answer to your comment in the same post.)

I do not see fpx as competing with that proposal since, 202Y is not near soon and from what I understand there is no guaranty that the proposal makes it to the final standard. In the mean time there is room for fpx.

Moreover, as @certik advocates very often, there is a need for prototyping new features. So if fpx can be used as a prototype and finally gets retired in a decade or so, it will have served its purpose.

From what I can tell fpx is quite compatible with the j3 paper, so it can really be used as a prototype and codes compatible with it will remains compatible with a standard preprocessor.

Finally, I have seen recently that @FedericoPerini added some preprocessing support to fpm. This shows that there is a need for an embeddable preprocessor, which is not possible if it is part or the compiler.

I would be more than happy to work with the authors of the proposal to see how fpx can get closer to their proposal.

2 Likes
9

Something I have never provided in a preprocessor but I found intriguing in coco was reversibility, where all data in the original file was retained in comments and could be recovered.

Originally, a small cpp processor (based on stripping down prep) that did not include macros was proposed as being included with fpm and fpm.toml being automatiically run through it since TOML does not directly provide conditionals. At the time a standard preprocessor was considered to be relatively imminent and fpm itself was in its infancy so it was decided to put off such a feature; and fpm.toml looked like it was going to have profile definitions; and the main need at the time looked to be to allow specifying different compiler options. Since then a need for other selections like dependencies being looked for locally versus via curl have created other reasons a preprocessor could be useful for running fpm.toml through.

1 Like
10

Thanks to those who answered the poll. From the results, it seems that the #foreach and #include directives are on top of the list.

In addition, I have been discussing with @bonachea and @pjfasano from the j3 subgroup about pre-processing. Looks like we have a lot in common, and they suggested that fpx could become a ‘reference‘ implementation of the current j3 papers:

After reviewing the papers, it does not look too hard to make fpx `standard` compliant. It already support most directives (only missing #line_number, #pragma, #error, and #warning), function-like macros, variadic macros and most arithmetic operations (missing x:y?z). There is a notable difference about line continuation, but that should be easy to standardize.
I wanted to start working on the #include directive, until I noticed that the papers do not clearly mention the order in which the paths are resolved for and “filename“.
My approach would be to stick to what intel’s fpp is doing:

For #include “filename”, file names are searched for in the following order:

  • In the directory in which the source file resides

  • In the directories specified by the I or Y preprocessor option

  • In the default directory

For #include <filename>, filenames are searched for in the following order:

  • In the directories specified by the I or Y preprocessor option

  • In the default directory

There is still an open question regarding the ‘default’ directory. Should it be the directories specified in the INCLUDE environmental variable? Or something else?

1 Like
11

Having looked at that issue for prep a behavior that varies between fpp and cpp varients is if you #include “filename” in a nested set of files and they are not in the same directory is the name read relative to the directory the file is found in or from one of the previous levels? It seems intuitive to read relative to where the file is found but last time I looked some preprocessors used the initial directory where everything started and did not consider that a bug at the time.

In recent years the question about whether to do anything with the input files regarding encoding comes up. So far I do not thing any cpp-like processor does but if not also doing so be careful about requiring a fixed line width in bytes, as UTF-8 strings can easily exceed the old 132 or 72 character limits. Fortran has gone to allowing very long lines so an option to split lines at a certain column number is less desirable anyway.

A “super-include” might convert non-ASCII-7 characters in strings to C-style escape sequences, which more and more Fortran compilers support as an extension, or converting binary files into base64 strings as options. There are very few restrictions on what the Fortran INCLUDE statement can take as an argument, but #include usually is limited to just filenames; but I think the Godbolt site and a few others will resolve a URL or curl statement. Handy on web applications but probably not going to make it into a standard, but INCLUDE is surprisingly very free to allow almost any syntax

12

I noticed that there are variants in the search order and even in the relative paths reference. If you look at the Windows doc, it states that

Syntax Form Action
Quoted form The preprocessor searches for include files in this order:

  1. In the same directory as the file that contains the #include statement.

  2. In the directories of the currently opened include files, in the reverse order in which they were opened. The search begins in the directory of the parent include file and continues upward through the directories of any grandparent include files.

  3. Along the path that’s specified by each /I compiler option.

  4. Along the paths that are specified by the INCLUDE environment variable. |
    | Angle-bracket form | The preprocessor searches for include files in this order:

  5. Along the path that’s specified by each /I compiler option.

  6. When compiling occurs on the command line, along the paths that are specified by the INCLUDE environment variable. |

Point 2 of the “quoted form” is a variant of what you were discussing, from what I understand.

I’m not going to support url because I would like to keep a pure Fortran solution, but that’s an interesting idea :+1: .
I must admit that I dropped the problem about line length simply by excluding support to fixed form. If you check the papers, this is also what the j3 group is suggesting. Supporting legacy fixed form is a whole new challenge, that I am not ready to take.

13

New release of fpx v1.2.0

Standard compliance

This version is compliant with the J3 standard draft. Or at least, to the best of my understanding. The standard compliance is activated with the command line argument --std.

#include file search order improvement

For #include “file”:

  1. Directory of the parent source file
  2. Directories specified by the -I option (global%includedir)
  3. Directories in INCLUDE environment variable
  4. Current working directory

For #include :

  1. Directories specified by the -I option (global%includedir)
  2. Directories in INCLUDE environment variable
  3. Current working directory

Support #for directive

fpx now supports the non-standard directive #for.

  • Basic iteration:
#for T in [integer, real, complex]

type(T) :: value

#endfor

! Expands to:

type(integer) :: value

type(real)    :: value

type(complex) :: value
  • Using a macro list:
#define NUMERICS [integer, real, complex]

#for T in NUMERICS

type(T) :: value

#endfor
  • Nested loops:
#define CONCAT(a,b) a##b

#for T in [integer, real]

#for R in [32,64]

type(CONCAT(T,R)) :: value

#endfor

#endfor
  • Generic procedure generation:
#define NUMERICS [integer, real, complex]
#define CONCAT(a,b) a##b

#for T in NUMERICS

module procedure CONCAT(add_,T)

#endfor
5 Likes
14

Hi @davidpfister I have one question regarding the generic procedure generation. I’m imagining the use case of replacing the use of fypp by fpx, currently with fypp one can create generic procedures by:

#:for T in NUMERICS

module procedure add_${T}$

#:endfor

Which has the advantage of making clear the procedure name and where one is “injecting” the preprocessing.

With the “CONCAT” example you show, I get the feeling that it “obfuscates” reading the declaration to know what’s the procedure name and what is the preprocessed variable injected in (think about a large project like stdlib where the macro declaration is not in the few lines preceding its use).

Would there be a way if mimicking some where closely the fypp style with fpx ?

Another question/sugestion: Have you thought about adding fpx to the Compiler Explorer Fortran libraries? that would be very useful to experiment with it online :slight_smile:

15

Originally, I designed fpx to be as standard as possible. The idea was to use it as a drop-in replacement for cpp or fpp.
Then I wanted to add a loop functionality, for which I took inspiration from Fypp. That’s why I chose the syntax #for rather than #do, #forall, or #foreach. For the moment, the #for loop creates a regular macro that is expanded like any regular macro, hence the use of CONCAT.

Now, it’s true that it obfuscates the code, and that the syntax in Fypp is much more readable. Now, it’s my turn to have a question for you: do you think the ${T} function should be general or only valid within the #for loop?

If there is interest in introducing ${T} in fpx, that should be fairly easy.

Concerning Compiler Explorer, I did not think about it. But if there is interest, I can look it up.

16

With fypp there’s also the use-case of parameter value assignment with the syntax ${ }$. This doesn’t necessarily live within a loop. So I guess making it a little more general would be useful. Globaly speaking, with your latest version it sparked my curiosity to see if fpx could be a more Fortranic replacement for fypp, at least with some of the most frequently used feutures.

I think that this would be a greate way of allowing people to explorer and play with the package before commiting to using it.
If of any help, you can take a look at these GH issues/PRs that allowed having fpm and stdlib as libraries in CE:

FPM in CE : [LIB REQUEST]: Add Fortran Package Manager (library) · Issue #6884 · compiler-explorer/compiler-explorer · GitHub
stdlib in CE : [LIB REQUEST] Fortran stdlib · Issue #6025 · compiler-explorer/compiler-explorer · GitHub

1 Like
17

The value assignment ${T} is a no-brainer. But it would take much more than this to replace fypp. For instance, syntax like #:for ik,it,ii in LINALG_INT_KINDS_TYPES, with multiple variables is not possible yet, and test conditions like #:if not rk in ["sp","dp"]are also not supported.
On the other hand, fpx support conditional expressions like #if x?y:z and variadic macros.
So if you were to adapt an existing code it would still require quite some rewriting.

Thanks for the links. I will check it out.

1 Like
18

Fpx now supports the ${x} syntax for macro insertion. I also revisited the design of the for directive to match the functionalities from fypp required in stdlib.

For instance, example like

#:for rk,rt,ri in REAL_KINDS_TYPES
#:if not rk in ["sp","dp"]
module subroutine stdlib${ii}_${ri}sysvx( fact, uplo, n, nrhs, a, lda, af, ldaf, ipiv, b,ldb, x, ldx, rcond, &
…
#:endif
#:endfor

Are manageable in fpx as follows:

#for rk in REAL_KINDS
#define rt real(rk)
#if rk == sp
#define ri s
#else
#define ri r
#endif
#if rk != sp && rk != dp
     module subroutine stdlib${ii}_${ri}sysvx( fact, uplo, n, nrhs, a, lda, af, ldaf, ipiv, b,ldb, x, ldx, rcond, &
…
#endif
#endfor

The code is available on the master branch and I will make a new release soon

1 Like