Improving Fortran standardization process (lessons from C++23 getting multidimensional arrays)

1

The new C++ 23 standard will get first class multidimensional arrays, which was the last main missing feature of C++ compared to Fortran. The main issue of using C++ for numerical computing was that each project used a different multidimensional array library (or reinvented their own) with different syntax and semantics, which prevented easy reuse of array code between projects. That will now be fixed with a unified syntax, so I expect the C++ array libraries will converge and one will be able to write numerical array code in C++ that will be reusable across projects. And with that I also expect compilers to be able to optimize multidimensional arrays more.

We can use this particular example to learn quite a few nice lessons for Fortran, and I am hoping we can improve our standardization process accordingly:

  • GitHub is used to track the proposals for this feature: P2128 Multidimensional subscript operator · Issue #845 · cplusplus/papers · GitHub, you can see all the revisions and the current status of the feature

  • Final revision paper: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2128r6.pdf

  • The paper is nicely formatted with links, colors, syntax highlighting, etc.

  • They have a prior compiler implementation (in an open source compiler) and an online demo (both linked from the paper)

  • They have deprecated the rarely used comma operator syntax in [ ] in C++20, and then reused the syntax for an expected widely used feature (multidimensional arrays) in C++23. Read the “What about comma expressions?” in the FAQ part of the paper for more details.

  • I assume compilers will support the old syntax with a compiler option, so that old code that would use this rare feature does not break.

  • We have a lot of old stuff in Fortran like that, which is rarely used, but which prevents new more widely used features to be introduced. There are multiple ways we can introduce new features and keep old code compiling. Not requiring implicit none comes to mind. There are many other such examples.

  • They release on a 3 year cycle

Here are issues at our J3 GitHub repository that suggest to fix some of these:

The most frequent argument that I hear why the above cannot be done is that it is too difficult, it would not work, it is not allowed by ISO rules and other objections. The C++ process clearly shows that this is not too difficult, it works, and it is allowed.

So the main reason why the above is not done is that we do not want to do it. That is indeed a very good argument (I am serious): just the fact that something is good for C++ does not necessarily mean it is good for Fortran. There are good arguments why it is better to release on a 5 (or more) years cycle, why not to use github, why not to use better formatting for papers, and why to forever keep typing implicit none (as an example). So let’s have that discussion.

Why not do the above? That is my vision how the Fortran Standards Committee could work in the future.

18 Likes
2

Skimming the links you provided, the syntax for referring to an array element will be x[i,j,k], as in Python/NumPy or R. Will you be able to refer to array sections in C++, for example x[i,j,0:n]? When you pass such an array to a C++ function, will you also need to pass the array dimensions, as is done with the 1D array of C and C++? If working with multidimensional arrays can be fast, convenient, and safe in C++, a “rude” question that will be asked is “Who needs Fortran?”. A personal answer is “Those who don’t want to or can’t learn a huge and complicated language such as C++, or who already have a large Fortran code base”, but more general and persuasive answers will also be needed.

3 Likes
3

That’s the key aspect here: vision.

Toward that, the standard bearers and the practitioners need to become brutally frank with themselves and each other to ask and answer

For whom Fortran, for what?

If the primary goal of an extremely, extremely slow, limited, and ad hoc process to update the language is to maintain existing codebases in certain large institutions who have big $$ contracts with compiler vendors and the purpose is mostly to manage the path of such codebases toward their eventual destiny i.e., extinction and if that is what fuels the heavy resistance even toward long-term enhancements (e.g., deletion of implicit mapping, earliest it can be realized is circa 2040), then it should be somehow conveyed honestly.

Fortran will always be around for old codebases, toy programs, and individual/small team efforts.

The question really is about fundamental and fulfilling endeavors in scientific and technical computing: few managers will support and sign a plan toward a new, ambitious project that has Fortran as the main programming paradigm. Time to start addressing the reasons behind this. The language standard development with a vision, as stated by @certik in the original post, is the right place to start.

4 Likes
4

C++ is evolving aggressively, while Fortran still cannot drop implicit none… If Fortran won’t have generic programming in the next five years, I cannot think of any reasons for people to start a new project with Fortran.

5 Likes
5

The earliest generic programming can even be part of the Fortran standard is the 202Y revision which itself is very likely >5 years away.

Robust and reliable processor implementations of generic programming might only see the light of day more than 5 to 10 years after the official publication of 202Y, many implementations will ignore or refuse to commence work on generic programming until after such a publication. The users of some compilers, like a widely used open-source implementation, even face the risk of never seeing said feature getting implemented in that processor.

In the realm of big, ambitious new efforts in scientific and technical computing today, there is no raison d’être for Fortran.

1 Like
6

If that’s the case, I think Fortran users should consider migrating to C++ and Chapel.

7

Indeed, the “default future” (assuming our current process) for generic programming is likely 5 to 10 years (or more) from now, and that still assumes we do a good job in the generics subgroup, which is by no means guaranteed. If you are interested, please join our efforts at GitHub - j3-fortran/generics.

However, it does not need to be this way!

One (out of many) examples how things could go is that we develop a solid proposal for generics, implement in Flang, LFortran, GFortran (and perhaps one or two commercial compilers join forces!) and use it as a community in many projects. This in turn will make it natural for the standards committee to approve relatively quickly. This is not the only way how it can get done. I am just posting it as an idea. All that is needed to make this happen is enough community determination and excitement to make this happen.

9 Likes
8

Some generic (!) observations):

  • The C++ committee is far larger than the Fortran committee (by about 10X, I think).
  • The speed of getting things done in Fortran is not directly a result of the process, but rather (in my opinion), that the actual work of designing new features is done by only a handful of members, none of whom are among those complaining the most about lack of progress.

I had high hopes for the GitHub repository to be a place where proposals for new features were actively developed. Instead it seems to be largely a lot of furniture rearranging and arguments about removing features from the language. It does, at least, provide a list of items to consider, but the committee will still have to do the grunt-work of specification and integration.

There is a Generics subcommittee working on a proposal for the next revision - I don’t know how far they have gotten with it. We are, at the moment, focused on finishing F202X, but I would like to start active discussion of potential 202Y features very soon.

8 Likes
9

As far as I understand Fortran history, since Fortran 90 the language seems driven by the committee.
But what you propose looks more like the situation in the 60’s & 70’s, when the standard committee was rather running after the compilers developers, trying to find a common ground after the explosive success of the language.
An alliance between major free licensed compilers to work in an orderly manner on a common extension to the language could be an interesting strategy. And as the work would be in open access, proprietary compilers could follow what is happening.

5 Likes
10

Even the F77 standardization process was largely driven by the committee. As I understand it

  1. PARAMETERs were invented by the committee.
  2. IF…THEN…ELSE was common in Fortran preprocessors, but not in compilers.
  3. CHARACTER was not common in compilers.
  4. LOGICAL was common in preprocessors, but not in compilers.

There were a number of common extensions to F66 that were not adopted by the committee, see MIL-STD 1753, published just before the F77 standard.

4 Likes
11

You don’t think the fact that only a handful of members work on new features is a direct result of the process? If the process was better, don’t you think more people would participate? Radical changes are needed in how Fortran is developed. Many suggestions have been made on the GitHub site, but so far, none of them have been implemented. For example, why isn’t the LaTeX code on GitHub so people can propose edits to it and show diffs of proposed changes. Wouldn’t that be better than whatever you are doing now? I think people will come out of the woodwork to help if the process was more open.

2 Likes
12

I still have high hopes for it.

To take it to the next level, the committee leadership itself (i.e., you) has to embrace it and use it to open up the process, as @jacobwilliams said. People will come to help. All you would have to do is what the C++ committee does.

Steve, why don’t you want to use GitHub?

2 Likes
14

Careful… there is lots of wisdom in experience. Being dismissive of those who have gone before is a path that leads to unnecessary mistakes. I say this as a 30 year old (a “young dude”).

6 Likes
15

Everybody: please be respectful. Let’s discuss how to improve our process, but let’s not do this old vs young debate. Anybody can participate, no matter what age.

5 Likes
16

IMHO, there is a lots of beating around the bush going on here. Nice formatting of papers, faster development cycles and frequent release of standards are all results of vast pool of eagerly waiting volunteers. C++ and Python are general purpose languages with a large support base amongst computer science and programming enthusiasts. Their coffers are brimming with programming interns, and experienced developers, and language theory experts. Software boom in everyday life ensures thrust from easily available funds.

Fortran on the other hand is a niche language. Even though the overall number of scientific programmers is increasing every year, a large number of these refuse to go beyond Fortran 90. They are serious about science and will invest only as much in learning programming as absolutely necessary. Add to that the, not unfounded, belief that modern and modular practices put unnecessary burden on processors. I know so many good scientists who see their foray into anything above Fortran 90 as unnecessary distraction from science. They are happy not being computer scientists. Their codes are not beasts, but they do run efficiently on massive machines. Any argument about modern practices is bound to result into disdain. And, I see nothing wrong with them, science does not discriminate between languages! There is no need to force feed somebody if he is already doing fine. I feel majority of Fortran users care for results, rather than our fetishes with language features.

Then, there are those at the interface of CS and science. Some of them see Fortran as a liability, remaining fantasize its resurrection. To the latter I would say, with whatever reason you have for the love, you are heavily outnumbered by apathetic Fortranners, who would not even fund, let alone join, your mission and cut-throat C++ers. Everything wrong with committee, standards etc will fall into place if there are enough users of modern Fortran. Accept that there are far few of them. LFortran will, hopefully, bite into userbases of MATLAB, Python and Julia. That should provide some impetus. You can even try the semi-commercial model of Julia, along with aggressive marketing. However, those migrating to C++ are hard to win back.

Personally, I do not love Fortran. But, I see it as the most appropriate tool available to me at the moment. Using C++ would be drinking from a fire hose, while Python is like digging mountain with a scoop. I am ready to switch if there is a better alternative. There is none at the moment, I believe. Chapel seems to be going after the fortune of X10 and Fortress. Julia is yet to prove its mettle on HPC.

fortran-pi
fortran-pi1063Ă—536 90.8 KB

Edit: Picture

8 Likes
17

In all processes, I am most looking forward to the realization of generics. For the Fortran community to form a code ecology, generics seem to be indispensable. This can be seen from the current situation of stdlib.

Generics are arranged in 202Y (maybe in 2027?). It will also take some years for each compiler to implement generics. For developers, the waiting time (maybe 8~10 years, maybe we are “old” at that time) seems a bit far away … Modern Internet-related technology innovation is relatively fast.

However, it does not need to be this way!

If we can try to implement generics or something in LFortran beforehand, it may be a thing to celebrate. It can be a syntax feature extended by the compiler. This way there is a lot of pressure for LFortran.

From the perspective of the gradual development of things, I can’t expect drastic changes in the Fortran committee, but we already have the community and organization, and I look forward to the friendly and bright ecology of Fortran.

I have only been in programming for 2 years (only Fortran). If there is any wrong point of view, please forgive me :heart:

10 Likes
18

I second this idea. From a perspective of an end-user, we could have two versions of Fortran:

  • A community driven Fortran compiler (or more aggressively a Fortran programming environment) as a test bed of new features. Because a lot of the “new” features we try to implement on Fortran, for example, the standard library, package manager or generic programming are very mature concepts, end users should be welcome/encouraged to use this version. In a word, this is the “pioneer version” of Fortran.
  • Enterprise compilers that strictly follow the Fortran standard. Features that has been extensively tested by the community and cautiously discussed by the committee will be added to these compilers. Correspondingly this is the “LTS” of Fortran.

This model has been successfully applied by other projects, Fedora and RedHat is one of them. Fedora is sponsored by RedHat alone. In the case of Fortran, instead of one sponsor we have many many potential sponsors like IBM, Cray, Intel, NVIDIA, NAG, or even Apple. If the community could join forces to build a “Fortran programming ecosystem”, the resurrection is not impossible.

As an end-user I’m already using stdlib and fpm, and I am telling you guys, these two tools have brought great conveniences to my research and I’m really thankful to that! If LFortran becomes mature and if it supports generic programming, I would be more than happy to use it simply because I don’t have to write func_sp, func_dp, func_int32, func_int64 again and again.

Come on Fortran, we need a come back!

12 Likes
19

Ondrej, you certainly know that I DO use GitHub, have started proposals there and have actively participated in issue discussions. I also promote the proposals GitHub often. Why would you suggest otherwise?

It is said “a poor workman blames his tools”. The converse is also true - good tools do not ensure quality work. I do not for a second believe that broader use of GitHub (or any other tool of your choice) would magically increase the speed of development of the Fortran standard. There’s nothing wrong with the tools we are using today - what we lack is people willing to sit down and write down detailed specifications, syntax rules and edits to integrate new features into the standard. Instead there’s a lot of handwaving and nebulous suggestions that someone else should do the hard work (see the parable of “The Little Red Hen”.)

Specifically regarding editing the text of the standard, if there were assistant editors a shared source management system would make sense. Malcolm DOES use one (svn at present), and I have raised the idea with him of finding another committee member to assist with the grunt work of integrating edits from papers. I will note that opening the source of the standard to all and sundry would bring down the wrath of the ISO gods.

7 Likes
20

Steve, you mainly answered a question “why don’t you want to use GitHub in general?” and your answer is great, thank you for that. I know that you use GitHub in general, and I especially want to thank you for your participation in the J3 incubator repository. It is greatly appreciated. I know you are not against GitHub in general.

I thought it would be clear from the context, but I realized after I asked that I should have been specific, and that is my fault. What I am really asking is:

  • Why don’t you want to use GitHub in the J3 and WG5 committees, just like C++ uses GitHub for their committee?

You answered it somewhat:

I do not for a second believe that broader use of GitHub (or any other tool of your choice) would magically increase the speed of development of the Fortran standard.

Of course it is not the tools that magically fix things. Good guitarist can play great on any guitar. The best guitar cannot make me a good guitarist on it own. Good tools do not ensure quality work.

That is why the J3 incubator repository did not yet raise to its full potential. Because the best tools cannot magically fix things.

Here is the real reason:

It is the inability of the committee to accommodate and work with those people in the wider community who want to help! We have submitted several proposals via the incubator repository, and they were only considered because I have essentially forced their considerations at the plenary while in Vegas. And at the last meeting they were just ignored as out of order (it was for 202Y, not 2X).

People see this and decide “why should I bother proposing anything and reworking the proposals if the committee will just ignore my work?”.

If instead people can see that their proposals are regularly considered (no matter if 2X or 2Y), you will see tons of good proposals.

Instead what people see is the WG5 leader publicly dismissing the J3 incubator repository as:

I had high hopes for the GitHub repository to be a place where proposals for new features were actively developed. Instead it seems to be largely a lot of furniture rearranging and arguments about removing features from the language. It does, at least, provide a list of items to consider, but the committee will still have to do the grunt-work of specification and integration.

And just like that, with this paragraph, you have discouraged tons of people to participate, because you have dismissed the repository as a vehicle for the community to contribute, and delegated it to “furniture rearranging and arguments about removing features from the language.”. Because who wants to participate in furniture rearrangement? I certainly don’t. What would be the point?

So let’s try to fix that: the J3 incubator repository is not for some minor tweaks. The goal of the repository is to allow the community to collaborate on proposals and submit them to the committee. And the goal is for the committee to consider every (high quality) proposal that anybody submits via its current interface at j3-fortran.org. Last time I talked with you about this goal, you were supportive of this goal.

Steve, do you not want to see your “high hopes” (per your comment) realized? I think you do. Then let’s make it happen. You are the WG5 leader. Why don’t you make a plan how you want to see this happen?

I am here to help. Tell us the plan, and we can help.

5 Likes
21

I agree with @certik and others that the standards development workflow and processes must be significantly improved to allow broader participation.

And I agree with @sblionel that it’s down to a few people to do the grunt work to incorporate changes to the standard.

From my own experience and observations it seems it’s down to 10% of any group that will do the hard work. We see that with the Fortran Standard, stdlib, fpm, etc. This may be a law of (human) nature.

Yes, developing the standard openly on GitHub would be a significant improvement. And of course, people would not just come out of the woodwork to make edits. It would still be down to the few.

If we want more people to do the work (including improving the processes), we need more people to join the committee. It’s free and anybody can become an alternate (non-voting) member to a paying member organization. If we want anything (e.g. exceptions, GitHub, whatever…) to happen or happen faster, find 50 people who want that thing and have them join the committee. And maybe 5 of them will do the grunt work to make it happen.

And with 100 or more members, the need to scale up (i.e. develop on GitHub) would become more obvious to the current members.

I think @certik’s work from the start of his joining J3 serves as evidence that change is slow but possible. It may not seem like things are changing for the Fortran Standard, but they are, and this will only become more apparent going forward.

So, you want any aspect of Fortran Standard to change? Find many people who want that too, have them join, some may even vote, and eventually the change will happen.

8 Likes