SciPy: "there is interest in adopting this (PRIMA) implementation, but little appetite for taking on more Fortran code"

[egio]

There is another crucial point if one want to integrate it in scipy.
It should never fail due to stack exhaustion, as you well know.

[ChrisRackauckas]

@zaikunzhang

There is a discussion about this topic on the Front Page of Hacker News:

(Of course, its position is falling with time going. It is about 20 at the time of writing. )

This is the second time that PRIMA appears as a front-page story within a week (indeed, three days), [. Interestingly and encouragingly, many comments there show that there is still quite strong support for Fortran in the technology community.

Yeah I saw this pop up and was confused. I develop the Julia SciML organization https://sciml.ai/ and we’d be more than happy to work with you to get wrappers for PRIMA into Optimization.jl’s general interface.

Optimization.jl: A Unified Optimization Package · Optimization.jl

Please get in touch and we can figure out how to set this all up. I’d be curious to try this out and do some benchmarks against nlopt methods.

[Pap]

I’ll have to censor myself, because a proper answer to SciPy’s maintainer statement, “we’re already working to get rid of it”, can only be offensive and far from being polite. Therefore I will just express my humble opinion that “Sci” and “Py” don’t go together, so “SciPy” can only be interpreted as a joke. And yes, I am aware it is popular, despite the fact it is… what it is (self-censoring here).

[hkvzjal]

Hi @Pap it would be shameful to shame SciPy’s work, honestly their library has been an excellent work to enable new scientist-in-the-making to discover complex algorithms in a Top-down approach which has indeed empowered a lot of people!!

I have used SciPy for years to learn or to benchmark many codes before pushing the performance barriers in Fortran…

There are many extremally well known python APIs out there using SciPy behind the curtains.

We should work much more closely with them, it would be extremally beneficial for both communities!!

And they seem to be indeed very open to the guidance we could provided to move forward: please follow the latest comments here : ENH: The Fortran 77 implementation of COBYLA is buggy and challenging to maintain. Switch to the PRIMA implementation? · Issue #18118 · scipy/scipy · GitHub

[Pap]

@hkvzjal

We should work much more closely with them, it would be extremally beneficial for both communities!!

I would probably agree, if they didn’t say that they are just trying to get rid of Fortran, so everything new they might add should not be in Fortran. They even consider using… f2c as a solution, to convert the code to “holy” C (and I don’t mean HolyC by T.A. Davis). I honestly don’t see any room for cooperating.

[hkvzjal]

The answer from one of the maintainers after a proposal on a Fortran wrapper to avoid f2py…

That wrapper code looks pretty reasonable and is a lot easier to understand than f2py imho. I think we could give that a try.
(…)
Yes you are right. My question was what not using f2py would mean - but bind(c) takes care of all that, and is way better than the unreadable generated code that f2py emits to deal with name mangling.

Please take the time to fully read! you could be surprised what a small amount of kind sharing and help can do.

[lmiq]

It would be very nice if, later, you write a post here or somewhere else explaining what was the best interfacing solution you arrived at with them. Having that clearly documented will be very helpful for others developing Fortran routines to interface with python or C.

[Beliavsky]

@zaikunzhang tweeted that

COBYQA by Tom M. Ragonneau and Zaikun Zhang is becoming the next derivative-free (a.k.a. zeroth-order / gradient-free) optimization solver in SciPy.

Congratulations, and thanks for showing that it is possible to get modern Fortran into non-Fortran-specific libraries, with some persistence.

ENH: add cobyqa to scipy.optimize.

[zaikunzhang]

Thank you @Beliavsky !

However, COBYQA is not PRIMA, although both are derivative-free (a.k.a. zeroth-order/gradient-free) optimization (DFO) packages.

COBYQA is a new algorithm, developed by Tom M. Ragonneau (my former Ph.D. student) and Zaikun Zhang (me) as part of Tom’s Ph.D. thesis during 2019–2022. It is coded in Python (with Cython). Tom did all the coding.

PRIMA is a package that I have been working on for modernizing Powell’s DFO solvers. Powell coded them in F77, and I have reimplemented them in modern Fortran during 2020-present.

PRIMA is also being integrated into SciPy:

• ENH: The Fortran 77 implementation of COBYLA is buggy and challenging to maintain. Switch to the PRIMA implementation? · Issue #18118 · scipy/scipy · GitHub
• RFC: Tracking progress to integrate with SciPy · Issue #112 · libprima/prima · GitHub

However, it needs much more work due to the multiple languages involved. The community (including the SciPy maintainers) is providing a lot of help, but more is needed. I am completely ignorant about Python, so I have to rely on community help.

I hope the success of COBYQA will help us to promote PRIMA in multiple ways.

Thank you.