# Implementing fypp like pre-processing functionality in fpm

**URL:** <https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550>\
**Category:** Uncategorized\
**Created:** [July 12, 2021, 10:38pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550 "2021-07-12T22:38:02Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\
**Post date:** [July 12, 2021, 10:38pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/1 "2021-07-12T22:38:02Z")

</div>

In the short run we should simply make `fpm` call `fypp` in PATH to pre-process `.fypp` files automatically. That might end up being the only thing we will do for `.fypp` files.

A separate question is what it would take to rewrite `fypp` or something similar in functionality into Fortran, so that we can link it with `fpm`, so that `fpm` is standalone (does not depend on Python). The main target is `stdlib`. Let’s look at the usage in `stdlib`. Below I collected the main uses cases. Did I miss any?

We can use these to design a pre-processor that is possible to implement in Fortran. If these are all or most of the use cases, they can be separated into two independent kinds:

- generic over any type (either one or a combination of integer, real, complex, logical) and kind
- generic over any array rank

The first one is I think quite simple to implement (i.e., design a syntax, preferably as close to fypp as possible) and implement in Fortran. The second one is more complicated, as one has to do things like select subarrays and handle declarations properly, but still it seems very much doable.

Both of these main use cases seem to 100% fit as a subset into the generics effort in the Fortran Standards Committee. So in fact we _should_ design this well as a community and use stdlib and other codes as examples of real world usage. Then the generics subgroup at the committee can take this and design some syntax for the language. Our pre-processor would work as a prototype for this. (The generics effort in Fortran is wider, we are also designing how to do “templates/traits”, but that is a separate problem that we do not seem to need here.)

# Use Cases

## 1: Loop over all integer, real, complex and logical kinds

Loop over all integer and real types and kinds:

- [stdlib/src/stdlib\_math.fypp at 492543d0086ff3821ecf2ee8f0a9cf46a945a7ac · fortran-lang/stdlib · GitHub](https://github.com/fortran-lang/stdlib/blob/492543d0086ff3821ecf2ee8f0a9cf46a945a7ac/src/stdlib_math.fypp)

```fortran
#:set IR_KINDS_TYPES = INT_KINDS_TYPES + REAL_KINDS_TYPES
...
    #:for k1, t1 in IR_KINDS_TYPES
    elemental function clip_${k1}$(x, xmin, xmax) result(res)
        ${t1}$, intent(in) :: x
        ${t1}$, intent(in) :: xmin
        ${t1}$, intent(in) :: xmax
        ${t1}$ :: res

        res = max(min(x, xmax), xmin)
    end function clip_${k1}$
    #:endfor

```

To loop over complex and logical kinds also: [stdlib/src/stdlib\_optval.fypp at 492543d0086ff3821ecf2ee8f0a9cf46a945a7ac · fortran-lang/stdlib · GitHub](https://github.com/fortran-lang/stdlib/blob/492543d0086ff3821ecf2ee8f0a9cf46a945a7ac/src/stdlib_optval.fypp)

```fortran
#:set KINDS_TYPES = REAL_KINDS_TYPES + INT_KINDS_TYPES + CMPLX_KINDS_TYPES + &
  & [('l1','logical')]

```

This is by far the most common use case. This one would be easy (I think) to implement in Fortran.

## 2: Loop over all array ranks

Example: [stdlib/src/stdlib\_stats\_moment\_scalar.fypp at 492543d0086ff3821ecf2ee8f0a9cf46a945a7ac · fortran-lang/stdlib · GitHub](https://github.com/fortran-lang/stdlib/blob/492543d0086ff3821ecf2ee8f0a9cf46a945a7ac/src/stdlib_stats_moment_scalar.fypp)

```fortran
  #:for k1, t1 in INT_KINDS_TYPES
    #:for rank in REDRANKS
      #:set RName = rname("moment_mask_scalar",rank, t1, k1, 'dp')
      module function ${RName}$(x, order, dim, center, mask) result(res)
        ${t1}$, intent(in) :: x${ranksuffix(rank)}$
        integer, intent(in) :: order
        integer, intent(in) :: dim
        real(dp), intent(in) :: center
        logical, intent(in) :: mask${ranksuffix(rank)}$
        real(dp) :: res${reduced_shape('x', rank, 'dim')}$

        if (dim >= 1 .and. dim <= ${rank}$) then
          res = sum(( real(x, dp) - center)**order, dim, mask) / count(mask, dim)
        else
          call error_stop("ERROR (moment): wrong dimension")
        end if

      end function ${RName}$
    #:endfor
  #:endfor

```

This use case also seems possible to do in Fortran.

# 3. Different code path for complex type

Example: [stdlib/src/stdlib\_stats\_cov.fypp at 492543d0086ff3821ecf2ee8f0a9cf46a945a7ac · fortran-lang/stdlib · GitHub](https://github.com/fortran-lang/stdlib/blob/492543d0086ff3821ecf2ee8f0a9cf46a945a7ac/src/stdlib_stats_cov.fypp)

```auto
  #:for k1, t1 in RC_KINDS_TYPES
    #:set RName = rname("cov",2, t1, k1)
    module function ${RName}$(x, dim, mask, corrected) result(res)
...
       ${t1}$ :: res(merge(size(x, 1), size(x, 2), mask = 1<dim)&
                          , merge(size(x, 1), size(x, 2), mask = 1<dim))
...
          #:if t1[0] == 'r'
            res = matmul( transpose(center), center)
          #:else
            res = matmul( transpose(conjg(center)), center)
          #:endif
...
    end function

```

This could be done by introducing a function in the pre-processor such as:

```auto
          #:if is_real(t1)
            res = matmul( transpose(center), center)
          #:else
            res = matmul( transpose(conjg(center)), center)
          #:endif

```

# 4. rname

Example: [stdlib/src/stdlib\_stats\_cov.fypp at 492543d0086ff3821ecf2ee8f0a9cf46a945a7ac · fortran-lang/stdlib · GitHub](https://github.com/fortran-lang/stdlib/blob/492543d0086ff3821ecf2ee8f0a9cf46a945a7ac/src/stdlib_stats_cov.fypp)

```auto
    #:set RName = rname("cov",2, t1, k1)
    module function ${RName}$(x, dim, mask, corrected) result(res)

```

I actually don’t know what this does. I assume it creates the name of the function somehow. This seems minor and possible to do in Fortran also.

# 5. Loop over each dimension up to a rank, and `select_subarray`

Example: [stdlib/src/stdlib\_stats\_moment.fypp at 492543d0086ff3821ecf2ee8f0a9cf46a945a7ac · fortran-lang/stdlib · GitHub](https://github.com/fortran-lang/stdlib/blob/492543d0086ff3821ecf2ee8f0a9cf46a945a7ac/src/stdlib_stats_moment.fypp)

```auto
          #:for fi in range(1, rank+1)
          case(${fi}$)
            if (present(center)) then
              do i = 1, size(x, ${fi}$)
                res = res + (x${select_subarray(rank, [(fi, 'i')])}$ - center)**order
              end do
            else
              allocate(mean_, source = mean(x, ${fi}$))
              do i = 1, size(x, ${fi}$)
                res = res + (x${select_subarray(rank, [(fi, 'i')])}$ - mean_)**order
              end do
              deallocate(mean_)
            end if
          #:endfor

```

---

<div class="post-metadata">

**Author:** ![nshaffer](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/nshaffer/32/161_2.png) [@nshaffer](https://fortran-lang.discourse.group/u/nshaffer)\
**Post date:** [July 13, 2021, 12:46am UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/2 "2021-07-13T00:46:29Z")

</div>

Leading up to the 2008 standard, there was a draft standard with a full blown lexical macro specification a la Lisp (or more recently Julia). It was dropped in later revisions. I don’t have the link handy now, but I will track it down this evening if someone doesn’t beat me to it.

Something like that is probably too involved for a simple preprocessor. But my memory is that the specification included several examples relevant to generic programming, both type and rank. It would be useful to study and compare with the `fypp`'s approach.

Edit:

> **[07-007r2.pdf](https://j3-fortran.org/doc/year/07/07-007r2.pdf)**
>
> 6.91 MB

Pages 45-53

---

<div class="post-metadata">

**Author:** ![aradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/aradi/32/3659_2.png) [@aradi](https://fortran-lang.discourse.group/u/aradi)\
**Post date:** [July 13, 2021, 7:48am UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/3 "2021-07-13T07:48:11Z")

</div>

But please, let’s do not burry it within fpm. It should be a library, used by fpm with an optional command-line interface, so that also non-fpm based projects can use it…

---

<div class="post-metadata">

**Author:** ![aradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/aradi/32/3659_2.png) [@aradi](https://fortran-lang.discourse.group/u/aradi)\
**Post date:** [July 13, 2021, 8:02am UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/4 "2021-07-13T08:02:27Z")

</div>

Oh, that’s very interesting and a much more integrated approach than fypp (although kinda too verbose for my taste). However, we have to decide, whether we want to have “smart” (context sensitive) or “dumb” (context insensitive) macros in general. Fypp does the latter, this is where the clumsy syntax comes from. It parses only its own directives, but never the Fortran code itself. Therefore, it needs a robust syntax to make sure, it only manipulates what it is supposed to.

The example in the draft

```auto
DEFINE MACRO my_generic_interface(typename,array_of_kinds)
  MACRO INTEGER :: i, kind
  INTERFACE my_generic_procedure
  MACRO DO i=1, SIZE(array_of_kinds)
    ! Necessary in order to evaluate kind to an integer:
    MACRO kind = array_of_kinds(i)
    MACRO IF (kind>0) THEN
      SUBROUTINE MySpecificProcedure_%%kind(X)
        typename(kind), INTENT(IN) :: X <-- That is tricky IMO
      END SUBROUTINE
    MACRO END IF
  MACRO END DO
  END INTERFACE
END MACRO my_generic_interface

```

needs more intelligence to decide, whether `typename(kind)` must be substituted with the macro arguments or not (as it is not in a line starting with the `MACRO` keyword). It could also occur in following lines:

```auto
! dummy comment typename(kind) should be not replaced (or should it?)

write(*,*) "typename(kind) should not be substituted here either"

write(*,*) "What about this?&
typename(kind)&
&"  

```

If one starts to parse the Fortran code as well, the pre-processor should be rather part (or use parts) of a compiler in order for the parsing capabilities.

---

<div class="post-metadata">

**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\
**Post date:** [July 13, 2021, 2:53pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/5 "2021-07-13T14:53:53Z")

</div>

> [@aradi](#):
>
> But please, let’s do not burry it within fpm. It should be a library, used by fpm with an optional command-line interface, so that also non-fpm based projects can use it…

Of course. This would be a standalone fpm package. The fpm itself would just depend on it and use it.

> [@aradi](#):
>
> whether we want to have “smart” (context sensitive) or “dumb” (context insensitive) macros in general.

I think the “dumb” approach is the way to go. However, since this is a pre-processor for Fortran, the pre-processor should know and have built-in support for knowing the kinds, types, features for any rank array declarations etc. But it would not parse any Fortran statements, from that perspective it would be “dumb”. Pretty much `fypp` is almost there, it’s just the Python evaluation that is an issue, but if `fypp` has more built-in support for most of the frequent tasks that currently are done in Python, then the Fortran version can just be a subset of `fypp`.

Regarding the macro proposal, I don’t like it at all, it is verbose and much less readable than fypp to me. I talked with several Fortran users who also don’t like it.

---

<div class="post-metadata">

**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\
**Post date:** [July 13, 2021, 9:48pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/6 "2021-07-13T21:48:27Z")

</div>

Did anybody ever use [COCO](http://www.daniellnagle.com/coco.html)? I’ve not studied the syntax but I think there was an implementation somewhere written in Fortran right? Maybe that could be useful?

---

<div class="post-metadata">

**Author:** ![nshaffer](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/nshaffer/32/161_2.png) [@nshaffer](https://fortran-lang.discourse.group/u/nshaffer)\
**Post date:** [July 14, 2021, 12:26pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/7 "2021-07-14T12:26:48Z")

</div>

I don’t think CoCo has a looping construct. It does let you define a parameterized text block and then expand it with actual arguments (search that page for “fermion” to find an example). You could use that to manually loop over types, say, but it will get out of hand for procedures that are fully type-rank-kind generic.

CoCo is GPL, so there’s nothing stopping us from using its source as a starting point, maybe extending it with a loop directive.

---

<div class="post-metadata">

**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\
**Post date:** [July 14, 2021, 1:45pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/8 "2021-07-14T13:45:27Z")

</div>

> [@nshaffer](#):
>
> CoCo is GPL, so there’s nothing stopping us from using its source as a starting point, maybe extending it with a loop directive.

Unfortunately the license does stop us to link it with fpm and distribute fpm with CoCo built-in. However, Dan Nagle (the author) might be willing to relicense it.

---

<div class="post-metadata">

**Author:** ![FortranFan](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@FortranFan](https://fortran-lang.discourse.group/u/FortranFan)\
**Post date:** [August 19, 2021, 1:02pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/9 "2021-08-19T13:02:17Z")

</div>

> [@certik](#):
>
> … A separate question is what it would take to rewrite `fypp` or something similar in functionality into Fortran, so that we can link it with `fpm` , so that `fpm` is standalone (does not depend on Python). The main target is `stdlib` . Let’s look at the usage in `stdlib` . Below I collected the main uses cases. Did I miss any?
> 
> We can use these to design a pre-processor that is possible to implement in Fortran. If these are all or most of the use cases, they can be separated into two independent kinds:
> 
> - generic over any type (either one or a combination of integer, real, complex, logical) and kind
> - generic over any array rank
> 
> The first one is I think quite simple to implement (i.e., design a syntax, preferably as close to fypp as possible) and implement in Fortran. The second one is more complicated, as one has to do things like select subarrays and handle declarations properly, but still it seems very much doable. …

@certik’s idea of a preprocessor in Fortran itself is extremely relevant and important given other discussions c.f. [Proposal: moving fftpack under fortran-lang - #29 by aradi](https://fortran-lang.discourse.group/t/proposal-moving-fftpack-under-fortran-lang/1711/29)

Fortran has a few crucial gaps when it comes to Generics but yet is high-level enough that the need for preprocessing to achieve generic algorithms (and containers) can be limited to particular aspects only. And it is conceivable to do with a Fortran processor-based facility that is both aware of the Fortran language semantics and syntax and can employ it effectively.

---

<div class="post-metadata">

**Author:** ![aradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/aradi/32/3659_2.png) [@aradi](https://fortran-lang.discourse.group/u/aradi)\
**Post date:** [August 19, 2021, 1:27pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/10 "2021-08-19T13:27:22Z")

</div>

Please note, that in cases like [Proposal: moving fftpack under fortran-lang - #29 by aradi](https://fortran-lang.discourse.group/t/proposal-moving-fftpack-under-fortran-lang/1711/29), having the preprocessor in Fortran would only give any advantage over the current situation, if the **preprocessor was always compiled with exactly the same compiler as the project**. Otherwise, it won’t have access to the compiler-dependent Fortran internals, like the `real_kinds` array. Probably every compilation of a preprocessor dependent project would then have to start with the compilation of the preprocessor first. Doable, but extra effort when building projects.

---

<div class="post-metadata">

**Author:** ![Lufla](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/lufla/32/2836_2.png) [@Lufla](https://fortran-lang.discourse.group/u/Lufla)\
**Post date:** [July 4, 2024, 2:06pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/11 "2024-07-04T14:06:53Z")

</div>

> [@certik](#):
>
> In the short run we should simply make `fpm` call `fypp` in PATH to pre-process `.fypp` files automatically. That might end up being the only thing we will do for `.fypp` files.
> 
> A separate question is what it would take to rewrite `fypp` or something similar in functionality into Fortran, so that we can link it with `fpm`, so that `fpm` is standalone (does not depend on Python). The main target is `stdlib`. Let’s look at the usage in `stdlib`. Below I collected the main uses cases. Did I miss any?
> 
> We can use these to design a pre-processor that is possible to implement in Fortran. If these are all or most of the use cases, they can be separated into two independent kinds:
> 
> - generic over any type (either one or a combination of integer, real, complex, logical) and kind
> - generic over any array rank
> 
> The first one is I think quite simple to implement (i.e., design a syntax, preferably as close to fypp as possible) and implement in Fortran. The second one is more complicated, as one has to do things like select subarrays and handle declarations properly, but still it seems very much doable.
> 
> Both of these main use cases seem to 100% fit as a subset into the generics effort in the Fortran Standards Committee. So in fact we _should_ design this well as a community and use stdlib and other codes as examples of real world usage. Then the generics subgroup at the committee can take this and design some syntax for the language. Our pre-processor would work as a prototype for this. (The generics effort in Fortran is wider, we are also designing how to do “templates/traits”, but that is a separate problem that we do not seem to need here.)

Hi @aradi !

In fpm manifest they say that fpm works with fypp for pre-processor.

```auto
To use multiple preprocessors, for example cpp and fypp, specify them like this:

Example

[preprocess]
[preprocess.cpp]
[preprocess.fypp]

```

But I can’t did it works. I think is better to delete the information in manifest or, if works, how can a do to use it?

---

<div class="post-metadata">

**Author:** ![aradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/aradi/32/3659_2.png) [@aradi](https://fortran-lang.discourse.group/u/aradi)\
**Post date:** [July 4, 2024, 3:13pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/12 "2024-07-04T15:13:58Z")

</div>

@Lufla Not sure, I have never tried to use fpm with a preprocessor so far (I mainly use CMake for most of my fypp-preprocessed projects). Maybe you could open a separate topic and tag it with the “fpm” label here on fortran-lang.discourse? I am sure, the fpm-developers can answer it very quickly and provide an example.

---

<div class="post-metadata">

**Author:** ![FedericoPerini](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/federicoperini/32/1750_2.png) [@FedericoPerini](https://fortran-lang.discourse.group/u/FedericoPerini)\
**Post date:** [July 4, 2024, 4:21pm UTC](https://fortran-lang.discourse.group/t/implementing-fypp-like-pre-processing-functionality-in-fpm/1550/13 "2024-07-04T16:21:26Z")

</div>

@Lufla fpm currently only supports `cpp` as a preprocessor, I have no idea how that could have ended up into the documentation.