The counter-intuitive rise of Python in scientific computing

We really have to be realistic here. @ivanpribec I don’t agree that fortran is as easy to program in as Python. There are pitfalls and in particular, users have to compile the code.

Consider person A, he develops some code in fortran and finds some weird behaviour. He wants to plot some of the data structures while running to get an idea of the flow.

If you haven’t already setup your own program to be linkable with X plotting library of fortran, then this is a nightmare to begin with (yeah, fpm will help), but problem is that users have to deal with things such as 1) adding use statements, 2) adding temporary variables, 3) fixing build scripts to add libraries for plotting, 4) be sure about types etc. passing down correct stuff.

Yeah, “block” is your best friend in this case! :wink:

But really, in Python, any body can just do 3 lines of code to get a functional plot.
Agreed that documentation, infrastructure around is a big plus in the community, but to say that Python and fortran are equally agile is simply not true. :slight_smile:

What we surely should aim for is a thriving community, increasing package repositories and improving documentation whole-wide to make it fresh and shiny again. But even then. I still think that users will be more “productive” in scripting languages.

Not to bash, just to add to the thread :wink:

Also, being a supporter at an HPC site, I can safely say that the majority of scientists are not interested in learning a language deep enough to be fluent without having documentation in a tab nearby. They care about their end-goal; the problem at hand. They don’t even care about performance (as long as it takes some days).

3 Likes

Finally some responses which return to the source of the thread :slight_smile:

I don’t think I made any statements about Fortran being easier than Python. Only that in the case of the story in the blog post, it was not the agility of Python that lead to triumph over Fortran for the specific problem, but the well developed community infrastructure. Of course you could argue that Python being easy and more accessible (and as @billlong mentioned, being taught at universities) is partially the reason why more libraries are developed in it.

I think there are pitfalls to using Python too. Being a Python user myself, I know I have struggled a lot with pip, conda, virtual environments, incompatibility between Python 2 and 3, wrong python paths, Python classes and decorators, etc. While Python can be used interactively in the interpreter or in a Jupyter notebook, I usually find myself putting things in a script, and running from a shell. This is probably just a consequence of not being bothered to configure an IDE, and being so used to the shell <-> editor cycle from Fortran.

Concerning use statements, Python has these too -> import. While it works great for pre-installed pacakages, I always need a documentation tab open to figure out how the folder structure relates to the import names in my own libraries. I’ve also had problems with absolute and relative import paths, when interacting with Python users doing developmnet in PyCharms on Windows.

While the dynamic typing in Python makes the language feel simple, I have had cases before where it introduced subtle bugs. A simple example that comes to my mind is

a = np.array([1,2,3])
b = a

where the array b is now only a pointer to a. Modifying elements of b will also modify elements of a. In my experience, explaining this to a beginner can be challenging. A second example is passing the wrong dummy variables in a function call. In recent codes I also see Python users specifying type hints which brings us half-way back to static typing like in Fortran.

But I agree that for the average scientific user, which is just interested in solving a problem with some existing algorithm implementation, it will be easier to accomplish in Python.

4 Likes

@zerothi, I think part of your comment mirrors what I wrote above: Fortran is currently not as easy to use as Python.

The part where we might differ is in what “might be” in the future: I think if we develop all the tools and compilers, Fortran might be as easy to use as Python. I understand if you are skeptical.

3 Likes

Having used a Fortran-based routine called INSPECT() that when passed an array lets you interact with it using matlab-like commands both from text and stdin that allows for plotting; as well as writing the output to a process, as a NAMELIST file, and as a Matlab input file I know that it is not the language per-se but the easily available tools that can make a language feel simple. You can make system calls at the prompt so you can call other simple utilties that compare the NAMELIST file to previous dumps and give a statistical summary of the data as well. All Fortran except a binding to low-level graphics. Easier to use than any other utility I have personally tried with a built-in user manual for when I forget. It can also be called as a stand-alone program
that can replay the commands entered when called from the program if journaling was turned on. I mention it because I know python-only programmers that dump their data and call inspect(1). They do not care what language it was written in, just that it is easy to use. Python is in several ways a proof of how it is possible to make C/C++/Fortran easier to use as it is a scripting language that depends on those other languages for much of it’s functionality.

1 Like

Personally, I don’t use Python for scientific computing. Perhaps because I learned Python 15 years after Fortran… But ten years after learning Python, I have finally never tried things like scipy or numpy, which are probably wonderful tools. I love Fortran because it’s made for crunching numbers and it’s perennial (no problem to run a 25 years old code). When the CPU temperature is rising, I know it is for good reasons.

But I understand that Python is attractive and I love it too : you can do so much by writing so few lines… The syntax is sobre (like a Google search engine homepage…) And, the standard library is huge: https://docs.python.org/3/library/index.html
Nothing to install, just type “import this_library”… I am using Python for scripts, sometimes big scripts like in gtk-fortran (regular expressions…). And of course, an interpreter is cool when you want to quickly try some commands…

Of course, as stated by Ivan, you can also have a lot of troubles with some Python features…

Finally, I think the Fortran standard library could be a killer feature, then fpm, then a Fortran interpreter.

3 Likes

I personally do think Fortran language itself also requires an urgent boost to serve well the scientific and technical computing arenas. And this is in addition to everything needed in its “ecosystem”: standard library, interpreter and other enhancements in tooling, etc.

The notion of Turing-complete may be too abstract and even too basic any longer for any practical utility with modern languages. Nonetheless, one can think of an even more abstract notion of feature-completeness that can be more practical and which can resonate intuitively with those coding-in-anger regularly in scientific and technical domains. This notion is decidedly dependent on the time period, the state-of-art in computing at a given instance in time. On this count, Fortran has perennially fallen short of the needs of practitioners since ANSI FORTRAN publication back in 1978. And the deficit continues to this day.

I had previously noted 6 items in this thread which are crucial for immediate survival of the language as a legitimate contender in the scientific and technical computing space.

However I do believe there is a list of miscellaneous items in the syntactic sugar category that can make coding in Fortran far more safe, straightforward, and appealing and enjoyable, especially for scientists and engineers who should be and want to be domain experts in their respective fields, and not language masters:

  • Why are we still required to include implicit none in every scope in Fortran? Particularly in INTERFACE .. END INTERFACE constructs that has its own scope and which made it into the language much after the legacy with IMPLICIT statements. This is especially pernicious, particularly with any respect to diversity and inclusivity which is all the rage these days with social justice. Because the common refrain is “I don’t mind adding that statement” in all my codes, so nor should you - this is inherently disrespectful. Well, there is a vast group of scientists and engineers who are either unable or unwilling to follow the discipline in terms of this verbosity. And the cost of any indiscipline can be prohibitively expensive, particularly in commercial environments. And continuously excluding the inclinations of those wanting to advance to a modern state where the desire to be explicit with types is the default “value proposition” from language design considerations is forcefully ignored not only belittles any concept of type safety but disrespects the most precious commodity, the time of practitioners.

  • There are so many domains now where 100% of the computing is performed using IEEE floating-point arithmetic. Then, why are we still required to go through a long preamble with one’s own KIND module that sets the desired floating-point KINDs for use with IEEE formats. Fortran can do with built-in aliases, say ieee_fpb32, ieee_fpb64, etc. (or something less verbose) that coders can immediately employ toward floating-point computations with an intuitive grasp and confidence in the types.

  • Same applies to other intrinsic types and their kinds, especially INTEGER and CHARACTER with `c_char’ kind.

  • And of course, a built-in string type. Sure, initial experimentation in a standard-library years after ISO_VARYING_STRING is a decent reset, but it is the base language itself that needs this facility to be included intrinsically.

The rate of change with all of these features in standard Fortran is either too slow or zero. Fortran is headed to science museum archives unless things change for the better at a much faster pace.

3 Likes

FortranFan’s attempt to make IMPLICIT NONE a political issue is strained.

Do people agree with me that the standards committee should make implicit typing obsolescent? Every compiler will retain it, but people who insisted on implicit typing would have to use a std=legacy option.

2 Likes

I’m not suggesting that gfortran or any other compiler remove the ability to compile code with implicit typing, just that users who still want this feature should use a compiler option that acknowledges their code is not standard-conforming. A large fraction would already be using the std=legacy option for reasons Steve Kargl described.

Sometimes I forget to use IMPLICIT NONE and get syntax errors that puzzle me. I doubt I am the only one. Making IMPLICIT NONE the default would save people writing new code a little time.

1 Like

At least for new codes such errors can be prevented on the level of tooling.

As an example using the new Fortran package manager:

~/fortran$ fpm new test_fpm
 + mkdir -p test_fpm
 + cd test_fpm
 + mkdir -p test_fpm/src
 + mkdir -p test_fpm/test
 + mkdir -p test_fpm/app
 + git init test_fpm
Initialized empty Git repository in /home/fortran/test_fpm/.git/
~/fortran$ cd test_fpm/
~/fortran/test_fpm$ cat src/test_fpm.f90 
module test_fpm
  implicit none
  private

  public :: say_hello
contains
  subroutine say_hello
    print *, "Hello, test_fpm!"
  end subroutine say_hello
end module test_fpm

Or using code completion upon pressing tab (in my case using Sublime Text):

implicit_none1

implicit_none2

The code completion templates can be customized up to ones preferences.

1 Like

It’s NOT political, but social - a fundamental difference, please make an effort to understand - see this also: https://mailman.j3-fortran.org/pipermail/j3/2020-June/012160.html. The social considerations underpin everything, increasingly the functioning of ISO and ANSI and their committees also.

Progress on any effort is almost always smacked down with “we’ve always done it this way, that’s how it should remain, we don’t mind continuing with it, we don’t think it’s trouble for us” and then tacitly conveying to take it or leave it. And people will leave.

The resistance to making implicit none the default in Fortran is but a small (and some will think trivial) example yet it’s a classic illustration of which one can see the whole picture of also what holds the language back.

Converse to the cliched expression “you’d me at hello,” countless scientists and engineers are lost to Fortran at implicit none itself, often the first section in an introduction to the language. And like with this story, many a battle is lost.

Please note these workarounds are very difficult and often impossible to implement and deploy and adopt, across constructs in the language itself such as interface .. end interface and across workflows spanning simple editors such as Notepad to Visual Studio Code to Visual Studio IDE and many other.

The standards committee has made some formerly valid code obsolescent. I like to use FORALL one-liners, but gfortran -std=f2018 xforall.f90 on

integer, parameter :: n = 3
integer            :: i,ivec(n)
j = 2
forall (i=1:n) ivec(i) = i*j
print*,ivec
end

says

xforall.f90:4:28:

4 | forall (i=1:n) ivec(i) = i*j
  |                            1
Warning: Fortran 2018 obsolescent feature: FORALL construct at (1)

and adding the option -Werror turns that warning into an error. If Fortran 2025 (or whatever) made implicit typing obsolescent, it could be arranged that only people who compiled with -std=f2025 or the equivalent in other compilers saw the warning, and only those who compiled with std=f2025 -Werror got an error.

2 Likes

There is a thread open at the j3-fortran proposals page on the topic of removing implicit typing: https://github.com/j3-fortran/fortran_proposals/issues/90

I can merely suggest to continue the discussion on implicit typing there.

2 Likes

I, of course, agree that implicit none has to go, and it is another one of those things holding back Fortran. It is literally the first thing I have to tell people when I try to introduce them to Fortran: “OK, you have to put this magic line in every file to disable 50 year old nonsense behavior”. It’s not a good look. We need to hide the nonsense, not keep it front and center for all to see (the same for implicit save). I have never seen anyone use these “features” except by accident. Every beginner has been bitten by these ridiculous features. I see it over and over and over again, year after year. If you want to use 50 year old nonsense, you need to use a compiler flag. That is not too much to ask.

5 Likes

My point remains about implicit none becoming the default in Fortran. Note this is entirely separate from removing implicit typing, there is a subtle but crucial difference here.

But my bigger point is there are critical gaps and deficiencies in Fortran language because of which there have been rise of many other programming languages and paradigms ever since the beginning in scientific computing, but which started to accelerate in the 1970s. Python among the latest in the top spot for scientific computing even. One may be inclined to brush the top spots off as occupied by fads, nonetheless a larger concern remains which is the slide of Fortran.

To stop the bleeding, one has to address all the wounds. Many of the issues have to do with features, or thereof, in the language standard itself.

I think the treatment of Fortran in this blog post vs. Python is rather unfair. If Bob is not a good researcher, that is not a weakness of Fortran. I highly doubt if many people (perhaps including the authors of this blog post) know much about modern Fortran syntax and capabilities. Even the savviest self-proclaimed Fortran programmers that I have seen in academia do not really know modern Fortran features beyond F90 or F2003. They are simply not aware of it.
and then there is the issue of bad Fortran education and courses in academia. I have not seen any Fortran course based on Fortran 2008 or 2018, or even 2003. The most progressive educators that I have seen teach Fortran 90. and that explains why many people do not know about modern Fortran language.

4 Likes

I think Fortran may have stabilized. It is ranked 32nd with 0.45% share in the December 2020 TIOBE index, compared to 30th and 0.456% share in January 2011.

Damian Rouson and myself teach Fortran 2018 in our courses, and are always looking for people who would like to learn it. If you know of anybody interested in a training session, please send them our way. everythingfunctional.com or sourceryinstitute.org

2 Likes

As far as argument matching Fortran did not allow a 1x1 array, which led to a very heavy use of passing scalars as arrays; which I never saw a compiler have trouble with until very recently, so that would break a lot of code. If Fortran was done from scratch everything could be an array where the undeclared size would default to 1x1 and a lot of that code would work.

The first three compilers I checked all have a switch to imply “implicit none”. Several compilers like the Intel compiler allow for installation of a default centralized configuration file where you can change all the compiler defaults. So this is something that compiler developers have suppied an option for supporting, indicating there is indeed a need or demand for it. Perhaps vendors could address this issue by supplying links to their compilers that implied strict use of modern programming, like implying implicit none and forcing allowing no deprecated features. Something like fpm would have to still have a way to call compilers like gfortran using default behavior, but might encourage “best practices”
by having an option or default to call a compiler with as many of these switches on as the compiler supports. For example, for gfortran

 gfortran -free-form -ffree-line-length-none -coarray=single -fimplicit-none -std=f2018 -fmodule-private -Wargument-mismatch ...

F was an attempt to force “good” practice and (though I applaud it) did not take the world by storm, but perhaps the vendors are another avenue to help deprecate old practices.

Remember one of the advantages of Fortran is there is all that old code out there that lies behind a lot of “modern” utilities. I remember someone looking totally perplexed that when they wanted to build some Python packages from scratch that they needed a Fortran compiler.

So, for gfortran there could be a “gf” link for good fortran, and a “goof” link for “good old obsolescent fortran” that just use different switches. Make a couple of little bash(1) files and get everyone you can to use “gf” and see how it goes. Make sure the scripts record usage so you can see who "goof"s and help them port their code :slight_smile:

2 Likes

I think Fortran gets it right. A scalar is not a 1x1 array, or a 1-element vector. A scalar has rank 0, but it can be broadcast so that arithmetic operations of scalars with arrays work.

2 Likes