Is that last statement true? I thought the whole point of WSL was to be ‘native’ and lighter weight than say vbox. A quick google search turns up benchmarks for wsl2 that show comparable performance to bare metal linux and certainly not 10x slower. And that is consistent with my experience. And if you are finding native python and cython performance acceptable replacements it would seem top performance is not a driving metric.
1 Like
I own a windows laptop and WSL setup is fraught by many, many traps. I have one WSL instance that runs with near native speed, one that runs 10x slower. No idea why, and I’ve already wasted a wildly disproportionate amount of time on the setup in general, and on this question in particular. That’s despite being not exactly a beginner[1], so I imagine that many casual users will give up if they encounter these issues. Overall, I stand by the statement that the WSL is far from a panacea on windows.
e.g. knowing how to deal with broken system clock in WSL or the boundaries between the various FSes, resp. changes in case-sensitivity ↩︎
1 Like
Interesting. Granted my experience is only one person. And people doing benchmarks might be using simple setups on plain or fresh installations. One suggestion though it sounds like you’ve moved on, if you are seeing factors of 10, I’d suspect a file system issue.
1 Like
And regardleas of difficulties using WSL, many institutes (mine included) won’t permit its usage at all due to cybersecurity concerns (as in, it bypasses all of their cybersecurity measures). So it’s a non-starter for a lot of us!
2 Likes
Yes, WSL seems like a nice bonus to support, but a solid Windows support must support the native Windows API, MSVC, including the ability to create a GUI (that’s more relevant to other applications than SciPy), etc. If Windows by default switches to WSL, then that can change things. Until then I would treat it as a bonus, but not a replacement for native support.
4 Likes
100%!! As of the current state of affairs it should by no means be seen as a replacement.
1 Like
As someone who started off in university computing services the requirement to work with several os’s was important. Today I have to work with people who use a variety of os’s. Having enough ‘desktop’ systems to run a range of native linux operating systems would be very expensive, so the functionality provided by windows hyper-v and wsl is extremely useful. This machine has native installs of ubuntu and Windows and you can chose which os to run at boot time. The native unbuntu install provides access to the nvidia gpu using the nvidia hpc offering. I have 2 linux os’s available via wsl and 6 linux os’s available via hyper-v. Access to the gpu for Fortran programming is only available (understandably) via the native ubuntu install. Horses for courses. I can live with the limitations of the WSL and hyper-v access.
2 Likes
I think that the point @certik is trying to make and that I also think is important is that: while WSL enables a lot of things, it should not be taken as an excuse for dropping native support for windows.
A developer can play around with several linux distros through WSL, that helps a lot, but if you develop software that has to run natively for non-developers then WSL is not a solution.
Unless at a given point one can develop and run linux software on windows without having to know that a WSL, or whatever other system is working as middle-man, then, the current state of affairs is just a poor-half-backed solution for production software deployment.
3 Likes
and let’s not forget! This will soon be a larger problem.
2 Likes
@h-vetinari Terrific article! Great work by everybody involved!
A few random comments in no particular order:
- Dumping all the old F77 code in a monorepo where it is the foundation of much of the world’s scientific programming stack, while being barely maintained by people who don’t know Fortran is just a catastrophe. I guess everybody knows that now.
- It’s not just a catastrophe for Python, it’s a catastrophe for Fortran as well. This old code “poisons the mind” against Fortran of anybody that has to deal with it. I too hate F77, it was replaced decades ago and the fact that people are still using it is just awful. It’s long past time to move on.
- Not blaming anybody…it is what it is. There was no active open source Fortran community when SciPy began. But now we have one, so there are possibilities that didn’t exist before. We just have to figure them out. The Fortran ecosystem, tooling and compiler situation is also improving slowly but surely. If it’s too late and the SciPy plan is to just convert it all to C++ then that’s too bad since C++ is a terrible programming language for math. Don’t come after me C++ folks.
- The difficulty of modernizing old F77 code is usually exaggerated. I don’t think it’s that hard at all. There are even some automated tools to do it, but the last mile has to be done by somebody that knows Fortran. However, the place to do that modernization is not within the SciPy monorepo.
- Many of my modern Fortran libraries are modernizations of the ancient code that still lives in the SciPy monorepo, and have been modernized and expanded beyond what you have there. These libraries are robust, fast, and can be used for decades, that is the strength of Fortran. They can be used in Fortran projects via the Fortran Package Manager, they can have exposed C-interfaces so could be used by any other programming language, and we could create Python extension modules for them as well for use in Python or maybe even incorporation into SciPy.
- But to do that we need help from the Python experts. I only just discovered that
ctypeswas not a good way to call Fortran code from Python! I don’t personally know how something likef2pyfits into Modern Fortran code. - Fully understand that most everybody is a volunteer and work on what they want (same for me!).
21 Likes
Well said @jacobwilliams. I think it’s an important point that SciPy is essentially monorepo of separate packages dependent in places on shared cores. There may only be a handful of people in the world with a broad understanding of the entire library. Though I’m I SciPy developer, I wasn’t even aware of the details of the build issues until I read @h-vetinari’s article. Most SciPy developers work only on the pieces of the library where they have expertise.
So far the choice of C++ was made only for scipy.special, where I’m part of a small group working on a header only library of Cuda compatible special function implementations that could also be used in CuPy, PyTorch, Jax, etc. The hope is to help fill the gap in special function availability within these other libraries, and provide a single source of truth for implementations of these functions. scipy/19404 contains a discussion about this, and you can see it in action in scipy/19601. I think there are good technical reasons to use C++ here. For other Fortran libraries in SciPy, I think the options are still open. For rewrites, I think the preferred option might be the interpreted shell/compiled core approach, with performance critical parts written in Cython or Pythran. I’m personally supportive of SciPy switching to actively maintained modern Fortran replacements where quality ones exist, but I’m only a small voice in this matter, and I think I lot depends on the build issues.
I mostly agree, but don’t feel quite as negatively about it. The earliest SciPy developers did have Fortran knowledge, but it was only a small group, and their greatest success was in building a community. I think SciPy and eventually the Scientific Python stack did a great job democratizing access to scientific computing tools, in doing this, a lot of technical debt was taken on. Whether it ends up being catastrophe or eucatastrophe is unsettled but I think there’s hope.
Also, I’ve been following some of the progress with lfortran and am consistently impressed and am very sympathetic with the collective efforts to try to build a strong modern Fortran community. @jacobwilliams, I’m aware of your modern Fortran libraries and have just now put together that you’re the one behind them. Great work there as well.
3 Likes
The SciPy 1.13.0 Release Notes
Under scipy.special improvements says
- All Fortran code, namely,
AMOS,specfun, andcdfliblibraries that the majority of special functions depend on, is ported to Cython/C.
Have you used recent versions of Msys2? You can use GNU tools to build native windows applications in the UCRT64 environment. If you need POSIX, you still have that via the Msys2 environment or via Cygwin.
@Beliavsky yes, SciPy’s goal is to remove all of Fortran. However, if they accept PRIMA, that would be awesome.
Thank you @certik for mentioning PRIMA. See below for the progress:
The Python binding has been finished. I will merge the PR after some verification is done.
7 Likes
Have you used recent versions of Msys2? You can use GNU tools to build native windows applications in the UCRT64 environment. If you need POSIX, you still have that via the Msys2 environment or via Cygwin.
I never regretted the switch to Linux, so I did not really looked for alternatives. But I check Msys2 regularly because I want to provide a package for the software I maintain via Msys2 on Windows. Unfortunately, the unavailability of MPI-enabled libraries (in particular, FFTW and HDF5) means that dependencies cannot be fulfilled.
As a side note: Msys2 is heavily inspired by pacman, the package manager of Arch Linux. As an Arch user, I would say that the copy is not as good as the original. Issues with missing MPI libraries are unknown to me and the package manager is not an alien to the OS but part of it.
2 Likes
@zaikunzhang excellent! It’s on our plan with LFortran also: TODO for beta · Issue #3806 · lfortran/lfortran · GitHub, we still have some issues to wrap up first, but at least it is on the radar now.
2 Likes
@certik Great! Thank you for keeping PRIMA on your radar! I will include LFortran in the CI of PRIMA once you succeed on your side.
1 Like
LPython 0.21 was announced on Phoronix yesterday:
4 Likes