The difficulties of using Fortran in SciPy

The lack of Windows compiler is just one problem. There are other issues using Fortran in SciPy that were raised in previous discussions (Using reserved words as variables - #23 by ilayn), such as:

  • The SciPy developers and maintainers generally aren’t familiar with Fortran and prefer C++ for new development (the number of students exposed to Python and C++ is bigger than Fortran, so who can blame them)
  • The Fortran routines in SciPy are fixed-form, use implicit typing, and use an “old” programming style, that makes them difficult to read and extend
  • NumPy (and hence SciPy) using C array layout by default; this means that temporary copies are sometimes needed when Fortran routines are to be called.
  • Sparse algebra codes in Fortran use 1-based indexing, whereas the SciPy sparse matrices use 0-based indexing (in principle this is addressable nowadays in Fortran through custom array bounds, but it would require work that not many are willing to invest)

Meanwhile, processors have become faster, the Python interpreter has improved, and JIT compilation has flourished, meaning that algorithms are often “fast enough” even when implemented directly in Python.

Anyways, I found the Quansight article interesting. Thanks for sharing it. Even cooler now that I know the author @h-vetinari is also part of our community.

Speaking for myself as a former academic user, I had access to both MATLAB and Mathematica at university, and hence no incentive to contribute to SciPy in that period. Now I mainly work with compiled languages, and only use Python for plotting and scripting.

It’s hard to say whats the reason for the insufficient investment in open-source Fortran compilers. Here are just a few of my speculations:

  • Lack of Fortran users able and willing to contribute to open-source compiler development; Fortran users are generally domain experts first, and programmers second. We tend to take tools such as compilers for granted.
  • Fortran users tend to be part of enterprises and their needs are met by the proprietary compilers from numerous vendors including Intel, NAG, NVIDIA, Silverfrost, previously also Lahey and Absoft.
  • Fortran stakeholders (users, educators, and even the committee) are poor communicators. In other programming language communities, much more effort (and money) goes toward dissemination, education, conferences, and even advertising, frankly. Essentially, they make themselves heard, whereas Fortran users often don’t even mention what language they work in for fear of being someone’s joke.

Just as an example regarding the last point, both Bjarne Stroustroup (C++) and Guido van Rossum (Python) appeared on the Lex Fridman channel. Bjarne regularly appears at C++ conferences, talking about how C++ is important, and permeates all domains of computing, as do speakers from a bunch of other companies (Adobe, Bloomberg, NVIDIA, Apple, Google, etc.). (I found a recent talk at Strangeloop, “The Economics of Programming Languages”, enlightening.)

In contrast, look at the pages of WRF, ICON, or ECMWF, some of the main operational weather forecast models in daily use. It takes a dozen of clicks before you find out these things are implemented in Fortran. It’s similar with Ansys CFX, CalculiX (in use at MTU Aero Engines). I suspect (but don’t know for sure) that STAR-CCM+ (from Siemens) and LS-DYNA (Ansys) are also in Fortran. AVBP is written in Fortran and in use by the industry including Safran Group, Air Liquide, Total, and Airbus. OpenFAST is a wind turbine simulation tool in Fortran from the DoE. OpenRadioss for shock and impact problems is also Fortran.

Essentially, the weather, transport, and energy sectors are all heavy users of Fortran (which isn’t surprising given the maturity of these industries), but few realize this. However little of the money trickles back into Fortran compiler development.

Apart from the DoE (that has funded flang development), Siemens (that has funded GCC development through their child organization Mentor), or the (German) Sovereign Tech Fund (that has funded LFortran development), I’m not aware of any direct support from enterprises toward gfortran, flang, or LFortran. (Just for clarity, I mean direct support from parties that are not also chip vendors.) Hopefully, we can make this change in the future.

Edit: I believe that Intel and AMD contribute to GCC occasionally to add support for their new architectures. Not sure if they contribute to the Fortran frontend in terms of language features and bug fixes.