Automatic Differentiation Built Into LFortran

Language enhancement
#1

@certik, @yizhang and everyone_else_interested:
I think it would be of immense benefit and forward-thinking to have automatic differentiation built into the LFortran compiler as a first class citizen.

Based on:

    1. its usefulness/utility in the present/future and even past
    1. trend in (array) programming languages:
    • Julia
    • Dex
    • Futhark
    • Jax - Python library
    • a C++ proposal - not yet ratified
    • etc.
    1. not playing catch-up again in the future as we currently are in Fortran to other languages.
7 Likes
#2

Here is an issue for this feature:

Feel free to add your comments there.

If anyone is interested in working on this, please let me know.

4 Likes
#3

For LLVM-based compilers (such as LFortran), it should be possible in principle to use Enzyme AD which performs the differentiation on the level of the LLVM IR.

There are issues open on AD at both the stdlib and J3 Fortran proposals repositories:

The C++ proposal can be found here.

Leaving all implementation details aside, the part interesting for the Fortran community is how to embed AD within our source code. Here’s a non-exhaustive list of approaches:

  1. a directive based formalism that co-exists independent of the base language
  2. by introducing new commands (differentiate block construct + jacobian/hessian commands to retrieve the derivatives)
  3. a (parameterized) derived type which supports AD
  4. as a new (dummy) variable attribute
6 Likes
#4

Thanks @certik, I will go through that.

2 Likes
#5

I am so glad it is also under J3 consideration. Thanks for the link.

2 Likes
#6

I think it’s probably wildly optimistic to expect that the Fortran Committee would ever add anything like this to the language. Getting it into a compiler (i.e. LFortran) is likely the only hope we have. I agree, Fortran needs this. Differentiation is such a huge component in so many areas of scientific and technical computing, and Fortran has fallen behind in this as in so much else.

3 Likes
#7

Building AD adjoints for blas/lapack routines is the most needed but also most daunting task on this roadmap.The problem with compiler level AD is the lack of knowledge of non-scalar structure that is ubiquitous in numerical computing, as it is not able to see matrix/jacobian structures in dgtsv, dgbsv, dsysv etc. So even we support Enzyme out of box we’ll need augment it significantly to make it usable/scalable for everyday Fortran applications.

1 Like
#8

You mean, something like this?

#9

Yes. That’s not something that can be done efficiently by simply using compiler-level AD constructs.

#10

For BLAS the adjoint routines have already appeared: Algorithm 1005: Fortran Subroutines for Reverse Mode Algorithmic Differentiation of BLAS Matrix Operations: ACM Transactions on Mathematical Software: Vol 46, No 1

For other matrix problems, this document may be of help. (Thanks again to @lkedward for posting it in a previous discussion.)