Improving Fortran standardization process (lessons from C++23 getting multidimensional arrays)

22

I see the current process as working, but not well. No one aspect is a deal breaker, but taken together there is a lot of opportunity for increased convenience and productivity being left on the table. Some examples:

At present, if I want to see the changes between revisions of a proposal I must open both versions and manually do a visual comparison. If we were using GitHub, it has tools to automatically highlight the differences for me.

For discussions (at least the “approved place”) is separate from the proposal itself. If I want to participate I must go one place to see the proposal, and another to see the discussion (if there is a recorded discussion at all). And non-committee members can’t see that discussion, so it’s no wonder they don’t understand what the committee is doing.

If I want to understand what edits a proposal makes to the standard I must

  • Find the edits paper
  • Find the version of the standard it’s based on
  • Find the corresponding lines in the standard for each individual edit

all manually. If we were using GitHub, all of that is done automatically.

If I want to see all of the papers related to a proposal, it is significant effort to find all of the paper numbers associated with it and access each individual paper. Starting from the final edits paper is at least doable, as they general refer back to previous papers. But going the other direction is nearly impossible. There is no correlation between a paper and it’s follow-on paper. If we were using GitHub, many (most?) of the hyperlinks would be provided automatically.

Like I said, no one of these is a deal breaker alone. But the committee is practically avoiding ALL the modern conveniences and tools that most software developers have become accustomed to. Imagine a member of the C++ committee got curious about contributing to Fortran. They would immediately see that they would have to abandon nearly ALL the tools and conveniences they’ve gotten used to working with on the C++ standard. That’s pretty much a non-starter.

Imagine asking a bunch of contractors to help you renovate a 1960’s house, but telling them they can’t use any tools made after 1990. You probably wouldn’t get many takers. If we want to attract more contributors, we need to meet people where they’re at, not tell them our tools would work just fine if they put in the effort.

7 Likes
23

@everythingfunctional thanks for the comment. That nicer GitHub style workflow is just a convenience. Yes, it would lower the bar and reach a wider audience, but even if the committee will not use it, other members (such as myself and Zach did many times in the past) can create the proper GitHub issues and keep them updated about what the committee is doing. And we can even add the diffs and other things.
That indeed is not a deal breaker.

However, there is a deal breaker.

The deal breaker is that the committee is not setup to consider and review any proposals beyond 2X and provide feedback to them. The committee has done it exactly twice, on two proposals that I asked at plenary to get reviewed some time ago. After that, it did not do it again. In my opinion it is this exact problem that prevents new contributors to create proposals at our J3 repository.

However, I think Steve is on board with this change, based on his comment here: Proposal For J3 Committee Rules · Issue #98 · j3-fortran/fortran_proposals · GitHub that:

All papers submitted are at least looked at at the next meeting (unless we overlook something, which happens but is rare.)

Steve, would you be willing to guarantee that every proposal will get looked at and discussed?

Honestly, I think nothing else is needed. I am offering to lead the discussion on the 2Y proposals, as I have done at our (last?) meeting in Vegas, and take care of the organization of it. If you want me to.

24

I can’t guarantee anything. I can make suggestions, but it’s ultimately up to the committee as a whole to decide what they want to do. The role of Convenor has no real authority. That most of the committee is willing to go along with my suggestions is nice, but it helps that I have a feel for what they’ll put up with. Note also that I’m NOT the J3 Chair, and at J3-only meetings I am really just another member.

I DO think it’s time to start discussing 2Y features. Note that it is WG5 that selects features, so just discussing them with J3 leaves out many interested parties. How about this - Ondrej, if you’re willing, put together a list of 2Y proposals in a single document, with links to the GitHub, and send it to me. I’ll add some text and then send it to the WG5 list to get people started on thinking about what we might do. It would be great if more members engaged in the discussions at GitHub, but not all members will. I will want to have some discussion at the Manchester (I hope!) WG5 meeting next year, but just putting a list in front of people will get some thinking.

As for tools - there are some benefits from using a tool such as GitHub but I’m doubtful you’d get most of the committee to make a wholesale change to using it. I don’t think agonizing over that right now is helpful.

4 Likes
25

Proposal consideration

If you would be willing to make a suggestion, that would be awesome. Thank you for this. I understand that your powers are limited. And a suggestion would go a long way.

202Y Proposals

Perfect, I will do that. Would it be ok with you if I involve anybody who is interested in setting up this list?

This draft would be preliminary and WG5 can change it or do whatever it wants with it, it’s not binding. However, my goal would be to go out of my way to get as many people involved in this as possible. What this would do is that it would get people’s skin in the game. There will be two possible outcomes for every feature on this list:

  1. WG5 selects the feature. I will then ask people who advocated for this feature, meaning they want it, to please help us write the proposals to get it done.

  2. WG5 does not select the feature. Now in this case I will advocate for us in the WG5 committee to write a feedback why we did not select the feature. What this will do is provide a good solid high quality feedback to people who proposed this feature.

Both are good outcomes. There is also a third possibility:

  1. Finally there might also be features not on this list that WG5 will select. In this case our job will be to get the wider community excited that these are good features.

And what I am really hoping is that we will manage to have a frank discussion while producing this list, and we will quickly realize that we have to select, or down select the features, or prioritize them. And obviously having proposals ready for them would really help.

Btw, we are quite ready with LFortran to actually prototype some simpler features, which is something I really wanted to do for every proposal to have a prior compiler implementation.

When is the Manchaster meeting? I can only see a meeting in July 18-22, 2022 in Boulder, CO here: WG5 Fortran - Meetings.

Tools

I am not worried about the tools, the above two items are much higher priority for me.

Thank you Steve, I really appreciate this workflow offer. I will keep you updated.

1 Like
26

@everythingfunctional is exactly right here. It’s hard to imagine a system that would be more hostile to getting new contributors to participate. This cannot simply be ignored.

1 Like
27

If you like, but I didn’t think of it as needing anything other than collecting and collating. No attempt at ranking. I will introduce it as “I asked Ondrej to collect…”.

I realize I was a bit off in describing how features are selected. Each National Body (J3 for the US) submits a list of requests. WG5 then votes on all the requests across the countries that submit. Historically, J3 has submitted the most requests, Germany and UK usually have a couple. Sometimes Japan offers one. (Canada is the only other country currently participating actively, and I don’t recall they have ever had their own requests. IBM Canada’s rep is officially part of J3 so he votes there.)

Given the abbreviated meeting schedule for the next two J3 meetings, it’s not clear to me that there will be time available to discuss 2Y ideas, but who knows? (Unfortunately, I will miss the 3rd and 4th day of the October meeting.)

We have only a few days before the October meeting, so I don’t know how much time members will have to think about the proposals.

1 Like
28

Is their more information available on how to join the national bodies other than J3? The J3 website states:

J3 meetings, documents, and membership are open to anyone worldwide. Meeting information and documents are available from this website; membership information may be obtained from any committee member. [emphasis added]

Their seem to be many members at the Discourse from UK and Germany who might be interested in getting more involved with the committee work, but don’t know where to start.

1 Like
29

For Germany, DIN is the national body, and for UK, BSI. I’ll comment, though, that direct membership in a National Body is NOT free (I pay INCITS $1300/year for example), and sometimes there are various restrictions on intellectual property. I don’t know if other NBs have the “alternate member” policy INCITS does for the US, where a primary member can name anyone as an alternate at no charge. Quite a few participants in J3 work are alternates to primary members and there’s no requirement that they be from the US. The downside is that you don’t get a formal vote if the primary member also votes, but you can do everything else.

The approach I would suggest is to find an existing J3 member to name you as an alternate, if your desire is to contribute to the standards process. I have two alternates already (Malcolm Cohen is one, because his employer no longer pays for membership!) But if you want to get your own vote, then you’ll have to contact your National Body organization and see what their requirements are for joining.

1 Like
30

Thank you @sblionel and @certik for the plan. I volunteer to help Ondrej with it.

@ivanpribec and others who are interested in joining J3, please speak with @gak, @rouson, and/or @certik who are active here and also J3 members (among others). If I understand it correctly, a paying Org can have any number of alternates. Gary had me on board within a day once he realized that I was interested in joining.

2 Likes
31

I noticed that there are a number of proposals being tracked for Fortran 202y in the Issues · j3-fortran/fortran_proposals · GitHub repo.

I would like to make a change to the “Labels”. The standard doesn’t fare to its sections as “Chapters”, but as “Clauses”. I’d like to make that change to the labels, unless I hear some kind of violent opposition.

We should use terms that the Standard uses, in general.

2 Likes
32

Gary, great, please go ahead. I didn’t know they were called Clauses and not Chapters. @sblionel asked us (@certik) to classify all proposals by Clause number, thus the classification work I just began. It’s a work in progress and so far I only covered the last page (8) of the issues, and working from the back. I will continue classifying in the following days.

1 Like
33

Yes, we are also working on extracting the ideas and organize it by title, link to the issue, standard Clause (@milancurcic already started this work), I’ll write a script for that soon. In the meantime, @gak and others: please help us classify each such proposal by the clause it belongs to. That way we can get ourselves organized.

34

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1684r3.html
mdarray is very likely to be voted into C++ 26. Also, modules are being actively improved in C++. There is not so much time left for Fortran.

3 Likes
35

I glanced at a couple of the documents that are linked in this thread, and I don’t want to take the time to understand the language lawyer stuff. Can someone quickly summarize what are the features of this new C++ array functionality? Does this bring C++ up to modern fortran standards, or is it more like f77-level functionality?

36

Sort of, potentially also more powerful/customizable once you combine it with the whole C++ template machinery (depending how you see it). The general idea is there are two new containers mdarray (similar to allocatable arrays) and mdspan (closer to Fortran pointer arrays or assumed-shape array dummy arguments; can also be strided/non-contiguous). The new containers build upon the new multi-dimensional operator[] syntax (P2128R6 - Multidimensional subscript operator).

Writing a 3D seven-point stencil code may look as follows (input and output here are pointers to some kind of floating point buffer;N, M, and O are the “array” extents; v is a C++ range - a light-weight iterator-like abstraction):

image

This examples also makes use of the C++ standard parallelism features to state the iteration space can be updated in parallel and unsequentially. An analogous example of arrays and loop nests in Fortran could be:

real, pointer, contiguous :: b(:,:,:), a(:,:,:)
integer :: i, j, k

do concurrent(i = 2,size(a,1)-1, &
              j = 2,size(a,2)-1, &
              k = 2,size(a,3)-1)

  b(i,j,k) = ( a(i,j,k-1) + &
               a(i-1,j,k) + &
               ... )
end do

The goal for mdspan/mdarray is also they become the standard C++ multi-dimensional array solution. The lack of proper multi-dimensional arrays in C++ has led to the appearance of several competing solutions in practice, which can be hindering at times:

image
image732×187 41.3 KB

My guess is that libraries which already use Eigen, Blaze or other array libraries will continue to do so (the damage has already been done…). Upcoming versions however may add an operator which converts their own array container to a mdspan (essentially something like the Fortran array descriptor). C++ allows implicit conversions when passing dummy arguments, so going forward, if your C++ procedures/API accept a mdspan dummy argument, it might be usable with all existing (and future) array libraries.

As I’ve shown in Resistance to modernization - #13 by ivanpribec, you could also cast a Fortran array descriptor to a mdspan (or vice-versa), to call a C++ library from Fortran (or vice-versa).

*The snippets above are taken from the talk C++ Standard Parallelism - Bryce Adelstein Lelbach - CppNow 2022.

7 Likes
37

One could also argue the time of C++ is over, with so many languages appearing that aim to replace C++ (Rust, Carbon, etc.). If you think C++ is “winning”, how is this for an opinion from a long-time C++ programmer who also participated in the standardization process:

It’s been over a year since I touched C++ professionally and I am surprised to say I don’t miss it in the slightest. A committee that doesn’t listen, tooling that doesn’t care about user experience, claims of anything being inactionable when in reality people don’t want to do the work, the constant fear, uncertainty, and doubt masqueraded around regarding the ABI.

I would highly recommend the following article by Jeff Atwood - The Magpie Developer, and the article quoted therein by Andy Hunt and Dave Thomas - Imaginate. (Andy Hunt and Dave Thomas are the authors of the widely known book - The Pragmatic Programmer.)

As Jeff Atwood says:

Don’t feel inadequate if you aren’t lining your nest with the shiniest, newest things possible. Who cares what technology you use, as long as it works, and both you and your users are happy with it?

That’s the beauty of new things: there’s always a new one coming along. Don’t let the pursuit of new, shiny things accidentally become your goal. Avoid becoming a magpie developer. Be selective in your pursuit of the shiny and new, and you may find yourself a better developer for it.

The recent thread - Altair Radioss - FEM solver goes open source - is an excellent example. To me, the mixture of F77 and F90 source code looks like a mess, the coding style is one of the oddest I’ve ever seen. But if you look at their Testimonials page, the crash simulator is used by excellent professors and engineering departments, and the solver has received pledges of support from AMD, Arm, and Intel, three of the top CPU manufacturers.

12 Likes
38

In my opinion, Fortran only has a chance if a progressive, try-it-out quickly and make progress this week (rather than some year of this decade) is forked and developed independently of the standard. The standard is what is (and has been) killing Fortran for decades.

1 Like
39

What exactly has been stopping any compiler developer, commercial or open source, from doing that for the last 30 years?

40

Demand, obviously. But if this discourse can muster enough people to be the driving force of demand, things could change.

2 Likes
41

Nothing.

In fact the situation with FORTRAN encouraged quite a few from “doing” exactly “that” and out came MATLAB, Octave, R, Python, Julia, …

3 Likes