# Automatic differentiation of Fortran code, opinions? **URL:** \ **Category:** Help\ **Created:** [October 25, 2020, 5:23pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369 "2020-10-25T17:23:30Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![lanast](https://avatars.discourse-cdn.com/v4/letter/l/cdc98d/32.png) [@lanast](https://fortran-lang.discourse.group/u/lanast)\ **Post date:** [October 25, 2020, 5:23pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/1 "2020-10-25T17:23:30Z")
Hello community, I am planning to implement the automatic differentiation functionality on code which I have developed in Fortran. What are your opinions regarding the numerous packages listed on [http://www.autodiff.org](http://www.autodiff.org)? Does anybody have good experience with one or anothe and would recommend specific package? Thank you Kind regards ---
**Author:** ![shahmoradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/shahmoradi/32/3151_2.png) [@shahmoradi](https://fortran-lang.discourse.group/u/shahmoradi)\ **Post date:** [October 25, 2020, 6:47pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/2 "2020-10-25T18:47:06Z")
I have had a similar question and need for years. Alas, I have not succeeded in my search. There is only one Fortran AD tool listed on the website that you mentioned, and to my knowledge, it is commercial. I have been thinking of starting an open-source project on implementing AD for a while but that has not materialized yet. If you are successful in your search for an AD tool in Fortran, please let us know here. Thanks! ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 25, 2020, 6:55pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/3 "2020-10-25T18:55:08Z")
I have used [DNAD](https://github.com/joddlehod/dnad/blob/master/dnad.F90) succesfuly in the past to compute Jacobian matrices. Here is the [link](https://www.sciencedirect.com/science/article/pii/S0010465513000027) to a paper with the description. I believe a short introduction can also be found in the PhD thesis of Joshua Hodson: > [Numerical Analysis and Spanwise Shape Optimization for Finite Wings of Arbitrary Aspect Ratio, Joshua D. Hodson, Utah State University, 2019](https://digitalcommons.usu.edu/etd/7574/) A simple usage example is given in the thesis: ```Fortran program Circle #ifdef dnad use dnadmod #define real type(dual) #endif implicit none REAL, parameter :: pi=3.1416 real :: r,a write(*,*) "Enter a radius: " read(*,*) r a = pi*r**2 write(*,*) "Area = ",a end program Circle ``` The program is then compiled using `gfortran –Ddnad –Dndv=1 dnad.F90 Circle.F90`, which replaces the lower-case `real` variables with a variables of `type(dual)`. Running the program produces the output: ```plaintext $ ./a.out Enter a radius 5 1 Area = 78.5400009 31.4159985 ``` We see that on printing the area, the program also prints the derivative vector, which in this example is equal to circumreference, \partial A/\partial r = 2 \pi r. In my application - evaluating small Jacobian matrices of size 9 x 9, the DNAD code proved just as fast as hand-coded Jacobians. One downside is that the way the code is structured currently requires the user to specify the number of design variables at compile time, meaning it cannot be integrated at multiple places where a different number of design variables would be needed. I’ve been thinking about doing some experiments to combine DNAD with the fypp preprocessor to make it more flexible. A second code I’ve experimented with is [Tapenade](http://tapenade.inria.fr:8080/tapenade/index.jsp), which is an on-line Automatic Differentiation Engine. It can also be installed locally as a set of Java classes. This then allows you to integrate it in a Makefile (I haven’t tried it). Compared to DNAD, which only implements _forward mode_ automatic differentiation (using operator overloading), Tapenade also supports _reverse mode_ differentation. There are however some [limitations](http://www-sop.inria.fr/ecuador/tapenade/userdoc/build/html/tapenade/faq.html#OL1) with respect to which language features are supported for differentation. From scrolling through the documentation I believe that if you stick to “procedural-style” with subroutines and functions operating on the default Fortran types (reals and integers) it should work quite well. One of the hurdles I’ve met is how to differentiate code which relies upon BLAS or LAPACK. In principle I believe it should be possible to download the reference libraries from netlib, and replace all `real` variables with `type(dual)` and use DNAD, or run BLAS and LAPACK through Tapenade, but it feels kind of daunting. Recently, a set of reverse mode differentiated BLAS routines was published in [TOMS](https://dl.acm.org/doi/abs/10.1145/3382191), but I haven’t found time to dig deeper. ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 25, 2020, 7:46pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/4 "2020-10-25T19:46:04Z")
While the list of Fortran AD packages on [autodiff.org](http://autodiff.org) is indeed large, I believe that most of those are “dead” tools. The biggest users of AD currently seem to be the machine learning frameworks including TensorFlow, MXNet, PyTorch, and Zygote. I found the overview given in the paper [Automatic differentiation in ML: Where we are and where we should be going](https://papers.nips.cc/paper/8092-automatic-differentiation-in-ml-where-we-are-and-where-we-should-be-going) to be quite informative. The NAG AD Suite, that I believe @shahmoradi has in mind as a commercial solution, seems to have a strong focus on C++. In fact the [NAG website](https://www.nag.com/content/algorithmic-differentiation-software) says: > For simulation programs written in Fortran a version of the NAG Fortran Compiler has been extended to serve as a pre-processor to dco. where [dco/c++](https://www.stce.rwth-aachen.de/research/software/dco/cpp) is the tool for automatic differentiation. ---
**Author:** ![septc](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/septc/32/77_2.png) [@septc](https://fortran-lang.discourse.group/u/septc)\ **Post date:** [October 25, 2020, 9:45pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/5 "2020-10-25T21:45:19Z")
Also, this paper was very readable & informative to me (to get the basic understanding of what is forward/reverse AD modes etc…) > **[Automatic differentiation in machine learning: a survey](https://arxiv.org/abs/1502.05767)** > > Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in > machine learning. Automatic differentiation (AD), also called algorithmic > differentiation or simply "autodiff", is a family of techniques similar to but > more general than... Now I’ve been learning how to use AD from time to time, and it seems very useful indeed (I was amazed that the differentiation of log det(A) etc can be done very straightforwardly (e.g. by PyTorch), though it may be not surprising for people familiar with AD.) ---
**Author:** ![gardhor](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gardhor/32/88_2.png) [@gardhor](https://fortran-lang.discourse.group/u/gardhor)\ **Post date:** [October 25, 2020, 9:55pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/6 "2020-10-25T21:55:12Z")
For some of my codes, some AD tools were needed. Therefore, years ago, I’ve developed part of those tools for a specific application. But more recently, I’ve started to set up some AD fortran library: “[github.com/lauvergn/QuantumModelLib/tree/OOP\_branch/SRC/dnSLib](http://github.com/lauvergn/QuantumModelLib/tree/OOP_branch/SRC/dnSLib)” Right now, this library is working for scalar multivariate functions and up to the third derivatives. It is based on overloading the operators (±/= …) and all intrinsic fortran functions (sin, cos, exp …). The library is still part of a code, but it should be relatively easy to extract. Another library is dealing with matrix, but it is less complet. ---
**Author:** ![gardhor](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gardhor/32/88_2.png) [@gardhor](https://fortran-lang.discourse.group/u/gardhor)\ **Post date:** [October 25, 2020, 10:07pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/7 "2020-10-25T22:07:28Z")
> [@ivanpribec](#): > > One of the hurdles I’ve met is how to differentiate code which relies upon BLAS or LAPACK. In principle I believe it should be possible to download the reference libraries from netlib, and replace all `real` variables with `type(dual)` and use DNAD, or run BLAS and LAPACK through Tapenade, but it feels kind of daunting. For some matrix/vector operations (addition, multiplication …), it will work, although, there are some more efficient procedures. However, a diagonalization is more tricky due to the iterative procedure. Anyway, they are procedures to get derivatives of the eigenvalues and eigenvectors. ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [October 26, 2020, 4:59am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/8 "2020-10-26T04:59:38Z")
@shahmoradi, first of all, welcome to the community! If you are interested, I am happy to write a prototype in LFortran, we have an issue open for it: > **[Add automatic differentiation (#97) · Issues · lfortran / lfortran](https://gitlab.com/lfortran/lfortran/-/issues/97)** > > This has been requested several times. The first step would be to add code that takes an ASR of some Fortran function... Let me know if you would be interested in working on this together. If so, let’s brainstorm how it should work. ---
**Author:** ![gardhor](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gardhor/32/88_2.png) [@gardhor](https://fortran-lang.discourse.group/u/gardhor)\ **Post date:** [October 26, 2020, 6:51am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/9 "2020-10-26T06:51:28Z")
I’ll be happy to be part of the discussion about AD in particular on adding a new type and overloading operator and intrinsic functions. However, I’m not sure about the meaning of ASR! ---
**Author:** ![lkedward](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/lkedward/32/72_2.png) [@lkedward](https://fortran-lang.discourse.group/u/lkedward)\ **Post date:** [October 26, 2020, 8:31am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/10 "2020-10-26T08:31:55Z")
Hi @lanast, I have been through many of the Fortran options at [http://www.autodiff.org](http://www.autodiff.org), and as others have mentioned, the vast majority of them are no longer available or are limited to old subsets of Fortran. May I ask which form of AD you require, Forward or Backward? For Forward mode, there are several OO libraries available, such as in [flibs](http://flibs.sourceforge.net/automdiff.html). The only option for Backward mode that I am aware of is [Tapenade](http://flibs.sourceforge.net/automdiff.html) which I can recommend highly. It is robust and works well with a wide-subset of modern Fortran. I can confirm that it can be integrated with a makefile quite nicely when installed locally. Moreover, the source-code transformation approach to AD will always provide the best performance since you can exploit compiler optimization of derivative code. Tapenade used to be closed-source and licensed for commercial use, but it now looks like it’s open source under MIT which is brilliant! * * * > [@ivanpribec](#): > > One of the hurdles I’ve met is how to differentiate code which relies upon BLAS or LAPACK May I ask which BLAS/LAPACK operations you need derivatives for @ivanpribec? My understanding of this is that it isn’t usually a good idea to differentiate such routines, rather it is better to use the normal BLAS/LAPACK operations to implement the analytic derivative. I would recommend the following report on this topic: [An extended collection of matrix derivative results for forward and reverse mode algorithmic differentiation](https://people.maths.ox.ac.uk/gilesm/files/NA-08-01.pdf). I’ve applied this successfully in a simple MATLAB potential flow solver to get reverse mode derivatives from the matrix inverse op. ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 26, 2020, 8:53am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/11 "2020-10-26T08:53:36Z")
I’m working on a dryer optimization problem. To calculate the objective function I need to solve a 1D diffusion equation discretized with finite differences in space and the Crank-Nicholson method in time. This leads to a tridiagonal matrix problem which I solve using `dgtsv`. Right now I’m using a derivative-free optimization code, but I’m not happy with the convergence rate. Thank you for the recommendation. ---
**Author:** ![lkedward](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/lkedward/32/72_2.png) [@lkedward](https://fortran-lang.discourse.group/u/lkedward)\ **Post date:** [October 26, 2020, 9:10am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/12 "2020-10-26T09:10:36Z")
> [@ivanpribec](#): > > This leads to a tridiagonal matrix problem which I solve using `dgtsv` . I see, so this is an adjoint problem then. I think you just need to solve the transpose system to back-propagate your sensitivities (see sec 2.3.1 in above referece); this will also be tridiagonal so you can still use `dgtsv` for the backward problem. ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [October 26, 2020, 3:42pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/13 "2020-10-26T15:42:00Z")
Perfect. @gardhor, will you be able to join our monthly Fortran call? [Fortran Monthly Call: October 2020](https://fortran-lang.discourse.group/t/fortran-monthly-call-october-2020/347) If so, let’s discuss it there. ---
**Author:** ![gardhor](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gardhor/32/88_2.png) [@gardhor](https://fortran-lang.discourse.group/u/gardhor)\ **Post date:** [October 26, 2020, 4:56pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/14 "2020-10-26T16:56:24Z")
> [@certik](#): > > will you be able to join our monthly Fortran call? When the October fortran call is due? ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [October 27, 2020, 5:13pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/15 "2020-10-27T17:13:27Z")
It will be in about 45 min if you can make it, see the link above for connection information. ---
**Author:** ![simong](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@simong](https://fortran-lang.discourse.group/u/simong)\ **Post date:** [October 27, 2020, 7:33pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/16 "2020-10-27T19:33:22Z")
I first wrote an _ad_ implementation about 15 years ago. In the recent lockdown I decided to re-implement it to be more extensible to higher derivatives. Currently it supports up to and including 4th: [my autodiff](https://github.com/sgeard/autodiff) ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [October 27, 2020, 7:43pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/17 "2020-10-27T19:43:44Z")
@simong excellent, thanks for posting it! We discussed this at our latest call, summary is here: [Fortran Monthly Call: October 2020](https://fortran-lang.discourse.group/t/fortran-monthly-call-october-2020/347/11). Can you please also post your code to this issue: > > > It was discussed at the October 2020 Fortran call to add a section to our website about autodiff, similar to https://www.juliadiff.org/,... That way we can include it at the new “autodiff” section at our website that we just agreed at the call to create. ---
**Author:** ![simong](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@simong](https://fortran-lang.discourse.group/u/simong)\ **Post date:** [October 27, 2020, 10:17pm UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/18 "2020-10-27T22:17:55Z")
Have just posted as requested. ---
**Author:** ![shahmoradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/shahmoradi/32/3151_2.png) [@shahmoradi](https://fortran-lang.discourse.group/u/shahmoradi)\ **Post date:** [October 29, 2020, 7:06am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/19 "2020-10-29T07:06:54Z")
Thanks, @certik for the link. This would be great! Aside from isolate usages, AD is absolutely required to popularize Fortran in the Machine Learning community. ---
**Author:** ![gardhor](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gardhor/32/88_2.png) [@gardhor](https://fortran-lang.discourse.group/u/gardhor)\ **Post date:** [May 11, 2022, 9:28am UTC](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369/20 "2022-05-11T09:28:38Z")
I’ve just added an automatic differentiation library on githhub. [AD\_dnSVM](https://github.com/lauvergn/AD_dnSVM) It uses forward mode up to the third derivative and as many independent variable. [Next page](https://fortran-lang.discourse.group/t/automatic-differentiation-of-fortran-code-opinions/369.md?page=2)