SciPy: "there is interest in adopting this (PRIMA) implementation, but little appetite for taking on more Fortran code"

1

In the latest discussion on the inclusion of PRIMA into SciPy, a major SciPy maintainer said the following.

Update from the community call: there is interest in adopting this implementation, but little appetite for taking on more Fortran code (because we’re already working to get rid of it in several other components). A feasible route forward could be to translate the Fortran implementation to C, using e.g. f2c, and to use that. Or perhaps there is interest in doing an official Cython / Pythran / C / C++ port?

See ENH: The Fortran 77 implementation of COBYLA is buggy and challenging to maintain. Switch to the PRIMA implementation? · Issue #18118 · scipy/scipy · GitHub for the original post. You may want to input your opinions / arguments under that thread.

This is not a good sign for the Fortran community according to the broken windows theory. Such viewpoints are contagious and will spread quickly in a community as vigorous and populous as Python.

(For more information about PRIMA, see the discussions under another thread.)

2 Likes
2

I would stick to Fortran. Then all the adopters would have to use a port, if they want.

For me, this is like confronting a bully at school. He wants you to surrender. If you surrender, he wins.

4 Likes
3

We all agree, but at some point they may chose to convert the code to C on their own. We have to convince them this is not the right way.

2 Likes
4

Yes, that is the point. They will have to convert… They will use non-original code, but a port.

1 Like
5

One idea here would be to (in parallel to interfacing in python the Fortran code as discussed in the original thread) convert the current repo into an fpm package. It could be tested and used “out of the shelf” with fpm. What do you think @zaikunzhang ?

1 Like
6

@hkvzjal Yes. That would call attention into the fpm initiative.

1 Like
7

F2c is not an option for PRIMA, because f2c can only translate Fortran 77. Would the SciPy maintainers be willing to incorporate PRIMA if there were C wrappers for it? How difficult would it be to create such wrappers? Most of the R packages I see that have Fortran code have thin C wrappers that R calls.

2 Likes
8

With the iso_c_binding+ctypes approach there would be no need for a C wrapper but directly a python wrapper calling the signature functions written in the Fortran module wrapper. This could reduce the 3-languages issue.

2 Likes
9

Thank you @conradoat for your comment and suggestion.

My goal is to make Professor Powell’s solvers as accessible as possible to scientists, engineers, and algorithm researchers. I am not particularly in favor of or against any language. I hope that everyone can easily use Powell’s solvers in her/his favorite languages.

The first implementation of PRIMA is in modern Fortran simply because Powell’s implementation was in Fortran 77. Using Fortran, I can systematically verify the bit-to-bit faithfulness of the modernized implementation (not only “faithful up to an epsilon”). In addition, the intrinsic support for matrix-vector calculations is a strong advantage when developing reference implementations (or templates) of numerical solvers — most numerical algorithms are combinations of such calculations anyway.

The major motivation for developing the modern Fortran version is to provide a reference for the implementation in other languages, namely Python, MATLAB, C++, Julia, and R. A reference implementation must be structured, modularized, readable, understandable, and extendable. The original Fortran 77 code is a true masterpiece, but it is not proper at all for being used as a reference implementation. You do not want to use a spaghetti-style codebase with 244 GOTOs as a reference, or your implementation will be of the same style.

Putting it more straightforwardly, I implemented the modern-Fortran version of PRIMA in order to develop versions that are entirely Fortran-free.

Coming back to the point, I fully understand why the SciPy community has “little appetite for taking on more Fortran code”. The reputation of Fortran has been damaged over the years. I do not agree with the damaged reputation, and I feel sorry for those who do not have a chance to know (or refuse to know) modern Fortran due to this false reputation, but I do not blame them. It is the responsibility of the Fortran community to re-establish the reputation. I do not regard it as a bully to request non-Fortran implementations. It is not a question of surrendering or not.

I do consider myself a member of the Fortran community. Taking my share of the aforementioned responsibilty, I will try to promote the usage of modern Fortran via the PRIMA project. There is nothing more convincing than a successful real-life project.

For the inclusion of PRIMA in SciPy, I will keep communicating with both the SciPy and the Fortran community (e.g., those on this discourse), trying to find the best route. As pointed out by others, f2c is not an option due to its incapability of handling modern Fortran. Official C++ and Python implementations are being planned, but they will not be delivered in the near future. The most probable and practical solution, as suggested here and under the other thread, is to wrap the modern Fortran implementation of PRIMA using iso_c_bindings + ctypes or similar facilities. I hope the SciPy maintainers will accept this solution.

9 Likes
10

@hkvzjal @conradoat Thank you for mentioning this. Absolutely. Making PRIMA an FPM package is the next thing to do for the modern-Fortran version of PRIMA. Another one is auto-documentation with FORD or Sphinx.

I think these are low-hanging apples. The only obstacle is that I am too busy right now. PRIMA has been a black hole that absorbs all my time and energy in the past three years. I have to catch up with my other projects or my career is in real danger.

2 Likes
11

@zaikunzhang I understand your point. I think the way to re-establish the reputation is to present quality software in modern Fortran. This is the spirit of the fpm initiative.

2 Likes
12

Here is my response posted under ENH: The Fortran 77 implementation of COBYLA is buggy and challenging to maintain. Switch to the PRIMA implementation? · Issue #18118 · scipy/scipy · GitHub . You are particularly welcome to join the discussion over there. Given my ignorance of Python, your input will be quite valuable. Thank you very much.

4 Likes
13

You surely know the fortran code in detail, but I wanted to point out that Powell’s publications date back to the 1960s and 1970s, so the first versions of his methods were written well before the f77 standard.

On the Python topic, it is ridiculous on the face of it for someone to suggest using f2c to convert a modern fortran code to c. That simply shows how out of touch some programmer are, by decades.

3 Likes
14

Very true. But that also shows all the work that needs to be done to change that perception.

1 Like
15

Hi @RonShepard , thank you for pointing this out. It is correct that Professor Powell became an active and soon famous numerical analyst in the 1960s due to his works on the Davidon-Fletcher-Powell method and conjugate direction method. I guess he started his career at Harwell while coding in Fortran 6X.

However, the first of the solvers under discussion, namely COBYLA, was released on May 7th, 1992, with the paper published in 1994. Therefore, the code was in quite standard Fortran 77.

Unfortunately, even though all the solvers under discussion were developed between the 1990s and 2010s, Professor Powell did not adopt Fortran 90. If he had, my life in the past three years (and probably in some years to come) would have been (would be) quite different.

16

Is PRIMA thread safe (mostly no global or saved variables, and all routines recursive)?
I think that would be valuable.
Generally, as a user, I feel more comfortable with a thread safe library that can run concurrently as I have less thing to worry about.

2 Likes
17

Hi @egio , thank you for raising this important point!

Yes, as far as I can see. There is no global variable / module variable. (BTW, how to test / verify thread safety systematically?)

I have been keeping thread safety in my mind when developing PRIMA. In fact, I asked a question here about thread safety some time ago — almost all my posts on this forum arise from PRIMA.

18

I’m thinking about a Python user perspective, they may want to run it in a thread safe (with multiple Python threads) and concurrently, solving a problem with a function that it is solving inside another problem (although it may be stupid to do that).

Without global and saved variables I think it should be ok. And of course if there is an error the routines shouldn’t stop but report back an error code.
And, of course, they don’t have to access the operative system.
The previous thread on thread safety seems to be quite exhaustive: Thread safety of modern Fortran libraries: what and how?.

19

Thank you @egio for your insights from a Python user’s perspective. This is important for PRIMA.

Without global and saved variables I think it should be ok.

I confirm that there is no such variables.

And of course if there is an error the routines shouldn’t stop but report back an error code.

PRIMA invokes error stop when there is an error. This occurs only once in the code, at the following place:

For the moment, I do not supply a code number to error stop. I will revise the code to assign an informative code to error stop. I remember that I did not do that because some compilers (e.g., Absoft af95, Oracle sunf95) do not support it. Now that I have dropped the support for discontinued compilers, it becomes doable.

And, of course, they don’t have to access the operative system.

Does write belong to “access the operative system”? That is the only possibility. All other code in PRIMA does nothing but numerical calculations. The usage of write is limited to a few lines in prima/fortran/common/{fprint.f90, debug.F90}. They can be easily removed / turned off if needed.

20

There is a discussion about this topic on the Front Page of Hacker News:

https://news.ycombinator.com/news

(Of course, its position is falling with time going. It is about 20 at the time of writing. )

This is the second time that PRIMA appears as a front-page story within a week (indeed, three days), the first one being on 16 May. Interestingly and encouragingly, many comments there show that there is still quite strong support for Fortran in the technology community.

2 Likes