Let's get LFortran to compile Minpack

We recently announced that LFortran can parse all of SciPy into AST: LFortran can now parse all of SciPy to AST.

We have now identified the exact work needed to fully compile Minpack (the exact code from SciPy) and run an example test:

It’s about 10 isolated features (none of them major), and we have checked that by commenting those features out in the Minpack source code that everything compiles, links and runs (obviously it is giving incorrect answers and might not run through the correct code path due to the workarounds). Here is an example of the full compilation and run of the commented out Minpack source code: Compile minpack · Issue #512 · lfortran/lfortran · GitHub.

After the 10 features are implemented, the SciPy’s Minpack original unmodified source code should fully compile with LFortran. After that it might be giving incorrect answers, in which case we will debug those bugs and fix them. That’s it!

This will be a big milestone for LFortran, as it becomes usable with some production codes.

We are looking for more contributors to help us get there faster. If you are interested, please let us know. I am happy to meet over video to get you started. It’s not complicated at all. I am offering free mentoring. I am hoping that it will become common place for all of us here to be able to simply fix things in the compiler itself when needed. For example the LFortran’s runtime library is written in Fortran, so for that it’s basically like developing any other Fortran library.

10 Likes

Ok, so thanks to Smit, Konrad, Pranav, Gagandeep, Ubaid, Naman and many others, with the latest LFortran as of couple minutes ago (94ab497) we can now compile Minpack and it even runs (almost) correctly!

Use the scipy21 branch of my Minpack fork, commit 0cd235c. At this point, I think there is just one minor change in the source code: XX implied do loop · certik/minpack@123becc · GitHub (disabling implied do loops, #981), the rest are just cosmetic changes to build systems, printing, etc.

Here is how it looks like:

minpack/lf(scipy21) $ lfortran --version
LFortran version: 0.18.0-96-g94ab497ac
Platform: macOS ARM
Default target: arm64-apple-darwin21.3.0
minpack/lf(scipy21) $ FC=lfortran cmake ..
-- The C compiler identification is AppleClang 13.0.0.13000029
-- The CXX compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The Fortran compiler identification is unknown
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - failed
-- Check for working Fortran compiler: /Users/ondrej/bin/lfortran
-- Check for working Fortran compiler: /Users/ondrej/bin/lfortran - works
-- Checking whether /Users/ondrej/bin/lfortran supports Fortran 90
-- Checking whether /Users/ondrej/bin/lfortran supports Fortran 90 - yes


Configuration results
---------------------
Fortran compiler: /Users/ondrej/bin/lfortran
Build type: Debug
Fortran compiler flags: 
Installation prefix: /usr/local
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/ondrej/repos/lfortran/minpack/lf
minpack/lf(scipy21) $ make
Scanning dependencies of target minpack
[  2%] Building Fortran object src/CMakeFiles/minpack.dir/minpack.f90.o
[  4%] Building Fortran object src/CMakeFiles/minpack.dir/covar.f.o
[  7%] Building Fortran object src/CMakeFiles/minpack.dir/errjac.f.o
[  9%] Building Fortran object src/CMakeFiles/minpack.dir/hybipt.f.o
[ 12%] Building Fortran object src/CMakeFiles/minpack.dir/lhesfcn.f.o
[ 14%] Building Fortran object src/CMakeFiles/minpack.dir/lmdipt.f.o
[ 17%] Building Fortran object src/CMakeFiles/minpack.dir/ocpipt.f.o
[ 19%] Building Fortran object src/CMakeFiles/minpack.dir/r1updt.f.o
[ 21%] Building Fortran object src/CMakeFiles/minpack.dir/vecjac.f.o
[ 24%] Building Fortran object src/CMakeFiles/minpack.dir/chkder.f.o
[ 26%] Building Fortran object src/CMakeFiles/minpack.dir/dmchar.f.o
[ 29%] Building Fortran object src/CMakeFiles/minpack.dir/fdjac1.f.o
[ 31%] Building Fortran object src/CMakeFiles/minpack.dir/hybrd1.f.o
[ 34%] Building Fortran object src/CMakeFiles/minpack.dir/lmder1.f.o
[ 36%] Building Fortran object src/CMakeFiles/minpack.dir/lmpar.f.o
[ 39%] Building Fortran object src/CMakeFiles/minpack.dir/qform.f.o
[ 41%] Building Fortran object src/CMakeFiles/minpack.dir/rwupdt.f.o
[ 43%] Building Fortran object src/CMakeFiles/minpack.dir/dogleg.f.o
[ 46%] Building Fortran object src/CMakeFiles/minpack.dir/fdjac2.f.o
[ 48%] Building Fortran object src/CMakeFiles/minpack.dir/hybrd.f.o
[ 51%] Building Fortran object src/CMakeFiles/minpack.dir/lmder.f.o
[ 53%] Building Fortran object src/CMakeFiles/minpack.dir/lmstr1.f.o
[ 56%] Building Fortran object src/CMakeFiles/minpack.dir/qrfac.f.o
[ 58%] Building Fortran object src/CMakeFiles/minpack.dir/ssqfcn.f.o
[ 60%] Building Fortran object src/CMakeFiles/minpack.dir/dpmpar.f.o
[ 63%] Building Fortran object src/CMakeFiles/minpack.dir/grdfcn.f.o
[ 65%] Building Fortran object src/CMakeFiles/minpack.dir/hybrj1.f.o
[ 68%] Building Fortran object src/CMakeFiles/minpack.dir/lmdif1.f.o
[ 70%] Building Fortran object src/CMakeFiles/minpack.dir/lmstr.f.o
[ 73%] Building Fortran object src/CMakeFiles/minpack.dir/qrsolv.f.o
[ 75%] Building Fortran object src/CMakeFiles/minpack.dir/ssqjac.f.o
[ 78%] Building Fortran object src/CMakeFiles/minpack.dir/enorm.f.o
[ 80%] Building Fortran object src/CMakeFiles/minpack.dir/hesfcn.f.o
[ 82%] Building Fortran object src/CMakeFiles/minpack.dir/hybrj.f.o
[ 85%] Building Fortran object src/CMakeFiles/minpack.dir/lmdif.f.o
[ 87%] Building Fortran object src/CMakeFiles/minpack.dir/objfcn.f.o
[ 90%] Building Fortran object src/CMakeFiles/minpack.dir/r1mpyq.f.o
[ 92%] Building Fortran object src/CMakeFiles/minpack.dir/vecfcn.f.o
[ 95%] Linking Fortran static library libminpack.a
[ 95%] Built target minpack
Scanning dependencies of target example_hybrd
[ 97%] Building Fortran object examples/CMakeFiles/example_hybrd.dir/example_hybrd.f90.o
warning: Format statement is not implemented yet, for now we will ignore it
  --> /Users/ondrej/repos/lfortran/minpack/examples/example_hybrd.f90:61:6 - 64:58
   |
61 |    1000 format(5x, "FINAL L2 NORM OF THE RESIDUALS", d15.7// &
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
...
   |
64 |               5x, "FINAL APPROXIMATE SOLUTION"//(5x, 3d15.7))
   | ...^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ignored for now


Note: if any of the above error or warning messages are not clear or are lacking
context please report it to us (we consider that a bug that needs to be fixed).
[100%] Linking Fortran executable example_hybrd
[100%] Built target example_hybrd
minpack/lf(scipy21) $ ctest 
Test project /Users/ondrej/repos/lfortran/minpack/lf
    Start 1: example_hybrd
1/1 Test #1: example_hybrd ....................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.01 sec

When you run the example by hand:

minpack/lf(scipy21) $ ./examples/example_hybrd 
3.00000000000000000e+00
OK
Done
1.23108081174292243e-09 18 1
Computed:
-5.70654512542618897e-01
-6.81628341216289013e-01
-7.01732451306508187e-01
-7.04212939735556764e-01
-7.01369048406216167e-01
-6.91865644625619836e-01
-6.65792012683941503e-01
-5.96034200634259048e-01
-4.16412062945815342e-01

Reference:
-5.70654499999999953e-01 -6.81628299999999965e-01 -7.01732499999999981e-01 -7.04212900000000031e-01 -7.01369000000000020e-01 -6.91865599999999970e-01 -6.65792000000000050e-01 -5.96034199999999958e-01 -4.16412100000000007e-01

For comparison here is GFortran’s output on exactly the same Minpack source code:

minpack/gf(scipy21) $ ./examples/example_hybrd 
   3.0000000000000000     
 OK
 Done
   1.1926358347598092E-008          14           1
 Computed:
 -0.57065451160065905      -0.68162834229123059      -0.70173245256347117      -0.70421294008375235      -0.70136904762728880      -0.69186564337991419      -0.66579201215468919      -0.59603420128081663      -0.41641206299847200     
 Reference:
 -0.57065449999999995      -0.68162829999999996      -0.70173249999999998      -0.70421290000000003      -0.70136900000000002      -0.69186559999999997      -0.66579200000000005      -0.59603419999999996      -0.41641210000000001     

This is not bad at all.

Now, there are still many bugs. One fun one is this one: Passing array section `a(1:m,n)` as `a(1,n)` · Issue #976 · lfortran/lfortran · GitHub, where LFortran thought that enorm(m,a(1,j)) means an implicitly defined enorm function that accepts an integer and a real argument, so it constructs an implicit interface with that signature. But that’s a mistake, in reality enorm accepts an integer and a real array argument… So the syntax is completely ambiguous, and the only solution is to use the same ABI for both alternatives (since we do not know which alternative the actual implementation of enorm will use). As a consequence, the enorm function currently returns incorrect results, and you can see it in the output above, some of the numbers are inaccurate due to this.

There might be more bugs.

If you want to help us get this all fixed up and “hardened”, please get in touch. The more hands on this, the better. As you can see, it’s pretty much all there for Minpack, these are all isolated bugs that we will just fix one by one.

This is the final sprint towards the finish line.

16 Likes

Ok, another month later, thanks to Smit, Pranav, Gagandeep and many others, we fixed several of the implicit interface issues, the values now look a lot closer.

GFortran:

$ gf/examples/example_hybrd 
   3.0000000000000000     
 OK
 Done
   1.1926358347598092E-008          14           1
 Computed:
 -0.57065451160065905      -0.68162834229123059      -0.70173245256347117      -0.70421294008375235      -0.70136904762728880      -0.69186564337991419      -0.66579201215468919      -0.59603420128081663      -0.41641206299847200     
 Reference:
 -0.57065449999999995      -0.68162829999999996      -0.70173249999999998      -0.70421290000000003      -0.70136900000000002      -0.69186559999999997      -0.66579200000000005      -0.59603419999999996      -0.41641210000000001     

LFortran:

$ lf/examples/example_hybrd
3.00000000000000000e+00
OK
Done
3.20285655606697024e-09 15 1
Computed:
-5.70654512008728076e-01
-6.81628341355134837e-01
-7.01732451588553130e-01
-7.04212939801392657e-01
-7.01369048199748324e-01
-6.91865644303652494e-01
-6.65792012508090947e-01
-5.96034200751178744e-01
-4.16412063091758433e-01

Reference:
-5.70654499999999953e-01 -6.81628299999999965e-01 -7.01732499999999981e-01 -7.04212900000000031e-01 -7.01369000000000020e-01 -6.91865599999999970e-01 -6.65792000000000050e-01 -5.96034199999999958e-01 -4.16412100000000007e-01

Still not the same. It could be some numerical issues stemming from different order of operations, or possibly the usual single/double precision corruption, but it could also be another bug. We’ll have to investigate. We agree to about 1e-10:

>>> -5.70654512008728076e-01 - (-0.57065451160065905)
-4.080690230168216e-10
>>> -4.16412063091758433e-01 - (-0.41641206299847200)
-9.328643413297755e-11

If you want to help, just reproduce the above and put printing statements and see when it starts to differ, then create a minimal reproducible example and report it as a bug. Use the scipy21 branch here: GitHub - certik/minpack at scipy21 and follow the instructions in the README.

Overall, this is excellent progress, I am very happy about this.

10 Likes

Another BIG push, with Ensure current_body is not nullptr by certik · Pull Request #1094 · lfortran/lfortran · GitHub we now get bit to bit exactly the same floating point numbers as GFortran.

GFortran

   3.0000000000000000     
   1.1926358347598092E-008          14           1
 Computed:
 -0.57065451160065905      -0.68162834229123059      -0.70173245256347117      -0.70421294008375235      -0.70136904762728880      -0.69186564337991419      -0.66579201215468919      -0.59603420128081663      -0.41641206299847200     
 Reference:
 -0.57065449999999995      -0.68162829999999996      -0.70173249999999998      -0.70421290000000003      -0.70136900000000002      -0.69186559999999997      -0.66579200000000005      -0.59603419999999996      -0.41641210000000001     

LFortran:

3.00000000000000000e+00
1.19263583475980921e-08 14 1
Computed:
-5.70654511600659053e-01
-6.81628342291230593e-01
-7.01732452563471165e-01
-7.04212940083752348e-01
-7.01369047627288800e-01
-6.91865643379914186e-01
-6.65792012154689195e-01
-5.96034201280816633e-01
-4.16412062998471999e-01

Reference:
-5.70654499999999953e-01 -6.81628299999999965e-01 -7.01732499999999981e-01 -7.04212900000000031e-01 -7.01369000000000020e-01 -6.91865599999999970e-01 -6.65792000000000050e-01 -5.96034199999999958e-01 -4.16412100000000007e-01

There are some printing differences, and I am sure there are still tons of bugs, and this is just one example of Minpack, etc.

But I think we have it!

9 Likes

Another month, and I think we have it.

CI: Update Minpack (all tests now fully work) by certik · Pull Request #1228 · lfortran/lfortran · GitHub

LFortran compiles Minpack as is from SciPy (no modifications) and all 5 tests that I had in my legacy github repository. All tests return bit to bit exact answers as GFortran. We test this at our CI and check the answers, so Minpack should now always work with LFortran. We now have to harden everything, if you discover any bugs with Minpack, please report them and we’ll fix it and add tests.

Going forward, here is what we should do and I think if you are interested to help, you can:

  • Try to compile SciPy with GFortran, except Minpack with LFortran and try to get all integration issues resolved
  • Document features that we need for the rest of SciPy
  • Compile Minpack to WebAssembly via LLVM and get it working with pyodide.

We have a parallel effort to compile fpm, progressing very well.

12 Likes

How about the modernized Minpack?

@jacobwilliams unfortunately you use some modern features that we don’t support yet, so you’ll have to wait, or help us. :slight_smile: We are now focusing on fpm, so once fpm fully compiles, the modernized Minpack will probably not be that hard to get working also.

1 Like

My plan is to keep producing modern Fortran libraries. Then people might want to use them, and maybe that will somehow lead to more pressure and resources becoming available for work on the various compilers. That won’t happen with projects clinging to old FORTRAN 77 code. But I have to leave it to people smarter than me to actually work on the compilers. :slight_smile:

4 Likes

We might be able to do the modern Minpack in parallel with fpm, I just looked at it, and it’s just a few features left to fix/implement in LFortran. @jacobwilliams, if we get modern Minpack compiling, will you become our beta tester?

6 Likes