Python wheels on GitHub Actions using Fortran, F2PY (NumPy), Meson, and cibuildwheel

Help
1

Hi.

I am writing a Python library that uses Fortran code. To distribute the package, I use

  1. F2PY (NumPy) to generate the Python interfaces,
  2. Meson (Ninja backend) to compile the Fortran extensions, and
  3. cibuildwheel to generate the wheels automatically using GitHub Actions.

The problem I face concerns the compilation on Windows. Based on NumPy and SciPy’s GitHub repos, I installed rtools to have a proper installation of mingw-w64.

  1. The compilations work for Python 3.7 to 3.10 on the runner windows-2019 but not windows-2022. Is there any known reason?
  2. The compilation for Python 3.11 does not work (independently of the Windows version).

The errors I get are as follows.

    [4/132] Compiling C object python/pdfo/fuobyqa.cp311-win_amd64.pyd.p/meson-generated_.._fuobyqamodule.c.obj
    FAILED: python/pdfo/fuobyqa.cp311-win_amd64.pyd.p/meson-generated_.._fuobyqamodule.c.obj
    "cc" "-Ipython/pdfo\fuobyqa.cp311-win_amd64.pyd.p" "-Ipython/pdfo" "-I..\..\python\pdfo" "-IC:\Users\runneradmin\AppData\Local\Temp\pip-build-env-b17p49bp\overlay\Lib\site-packages\numpy\core\include" "-IC:\Users\runneradmin\AppData\Local\Temp\pip-build-env-b17p49bp\overlay\Lib\site-packages\numpy\f2py\src" "-Ipython\pdfo" "-IC:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\python.3.11.2\tools\Include" "-fvisibility=hidden" "-fdiagnostics-color=always" "-DNDEBUG" "-D_FILE_OFFSET_BITS=64" "-Wall" "-Winvalid-pch" "-O3" "-DMS_WIN64=" "-DNPY_NO_DEPRECATED_API=NPY_1_9_API_VERSION" -MD -MQ python/pdfo/fuobyqa.cp311-win_amd64.pyd.p/meson-generated_.._fuobyqamodule.c.obj -MF "python/pdfo\fuobyqa.cp311-win_amd64.pyd.p\meson-generated_.._fuobyqamodule.c.obj.d" -o python/pdfo/fuobyqa.cp311-win_amd64.pyd.p/meson-generated_.._fuobyqamodule.c.obj "-c" python/pdfo/fuobyqamodule.c
    python/pdfo/fuobyqamodule.c:88:10: fatal error: threads.h: No such file or directory
       88 | #include <threads.h>
          |          ^~~~~~~~~~~
    compilation terminated.

Did any face (and hopefully patch) similar issues before?
Thank you :grin:

2 Likes
2

Hello @ragonneau Tom, welcome to the discourse!

It would be helpful to have a minimal example — a repo containing a small piece of Fotran code and your building scripts that can reproduce the problem.

3

Could you post a link to the Actions Workflow logs and to the cibuildwheel configuration options?
You are attempting to use glibc on Windows, which if done through a msys2 shell should be okay. Will probably be able to say more once I’ve looked at how the wheels are made.

1 Like
4

A quick search points to a lack of support for the C11 standard threading library. You might try installing MVS2022 in your system which should provide the required library.

1 Like
5

I built a minimal example at GitHub - ragonneau/test-fortran-python. However, it is so simple that F2PY does not attempt to load threads.h, I would need to complexify it. Meanwhile, I uploaded the log of the compilation of the whole project for Python 3.11 on Windows 2019 at 16_Build wheel for cp311-win_amd64.txt · GitHub. I cannot upload it here, as I am a new user of the Fortran Discourse.

6

The Python 3.11 build probably fails because of pyproject-metadata

2023-04-19T10:49:53.5970336Z   Collecting pyproject-metadata>=0.7.1
2023-04-19T10:49:53.6098211Z     Downloading pyproject_metadata-0.7.1-py3-none-any.whl (7.4 kB)
2023-04-19T10:49:53.8786430Z     Link requires a different Python (3.11.2 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2023-04-19T10:49:53.8797275Z     Link requires a different Python (3.11.2 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2023-04-19T10:49:53.8806649Z     Link requires a different Python (3.11.2 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2023-04-19T10:49:53.8816475Z     Link requires a different Python (3.11.2 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
2023-04-19T10:49:53.8827776Z     Link requires a different Python (3.11.2 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)

As for the general Windows failure, can you try and use msys compilers instead of the R-project GCC you are using?

Here is a simplified workflow that we use to create the wheels for fpm.

name: Build

on: [push, pull_request]

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.sys.os }}
    runs-on: ${{ matrix.sys.os }}
    strategy:
      fail-fast: false
      matrix:
        sys:
          - { os: windows-2019, shell: "msys2 {0}" }
          - { os: ubuntu-20.04, shell: bash }
          - { os: macos-11, shell: bash }
    defaults:
      run:
        shell: ${{ matrix.sys.shell }}
    steps:
      - uses: actions/checkout@v3

      - if: runner.os == 'Windows'
        uses: msys2/setup-msys2@v2
        with:
          msystem: MINGW64
          path-type: inherit
          install: >-
            mingw-w64-x86_64-gcc-fortran

      - name: Build wheels
        run: |
          pipx run cibuildwheel
        env:
          CIBW_ARCHS_WINDOWS: auto64
          CIBW_ARCHS_MACOS: x86_64
          CIBW_ENVIRONMENT_WINDOWS: CC=gcc CXX=g++ FC=gfortran
          CIBW_ENVIRONMENT_MACOS: CC=clang CXX=clang++ FC=gfortran-11

      - uses: actions/upload-artifact@v3
        with:
          path: ./wheelhouse/*.whl

Notes

  • The C-runtime library for Windows is not the same used by GNU compilers, so that can be a potential source of problems for your package
  • The Ninja build system on Windows (at least for CMake) overrides the compiler defaults internally to the MSVC compilers, I don’t think that’s the case for Meson but remember that for future reference
  • If you attempt to produce a wheel for Mac arm64 you will probably struggle since you will have to install a GNU cross-compiler for arm. We have a working solution for this, but it is not a trivial task.
2 Likes
7

Hi @gnikit,

I tried your solution on my project (See the repo pdfo/pdfo on GitHub at python/dev), and it still does not work. The compilation for Python 3.11 still renders the exact same error. Do you have any suggestions?

Regards,
Tom.

8

Hello @gnikit .

In order to try your suggestion, I started a new branch on the repo that Tom @ragonneau is working on.

[Disclaimer: I am totally ignorant about Python, so what I do below may turn out completely wrong.]

Testing wheel...
  
  + pip install virtualenv -c 'C:\Program Files (x86)\pipx\.cache\2b6868c5f98c489\Lib\site-packages\cibuildwheel\resources\constraints-python38.txt'
  Collecting virtualenv
    Using cached virtualenv-20.21.0-py3-none-any.whl (8.7 MB)
  Collecting distlib<1,>=0.3.6 (from virtualenv)
    Using cached distlib-0.3.6-py2.py3-none-any.whl (468 kB)
  Collecting filelock<4,>=3.4.1 (from virtualenv)
    Using cached filelock-3.11.0-py3-none-any.whl (10.0 kB)
  Collecting platformdirs<4,>=2.4 (from virtualenv)
    Using cached platformdirs-3.2.0-py3-none-any.whl (14 kB)
  Installing collected packages: distlib, platformdirs, filelock, virtualenv
  Successfully installed distlib-0.3.6 filelock-3.11.0 platformdirs-3.2.0 virtualenv-20.21.0
  + python -m virtualenv --no-download 'D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test'
  created virtual environment CPython3.8.10.final.0-64 in 1645ms
    creator CPython3Windows(dest=D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test, clear=False, no_vcs_ignore=False, global=False)
    seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\runneradmin\AppData\Local\pypa\virtualenv)
      added seed packages: pip==23.0.1, setuptools==67.4.0, wheel==0.38.4
    activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
  + where python
  D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test\Scripts\python.exe
  D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\build\venv\Scripts\python.exe
  C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe
  C:\ProgramData\Chocolatey\bin\python.exe
  + pip install 'D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\repaired_wheel\pdfo-1.2.1-cp38-cp38-win_amd64.whl'
  Processing d:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\repaired_wheel\pdfo-1.2.1-cp38-cp38-win_amd64.whl
  Collecting scipy>=1.1.0
    Downloading scipy-1.10.1-cp38-cp38-win_amd64.whl (42.2 MB)
       --------------------------------------- 42.2/42.2 MB 18.2 MB/s eta 0:00:00
  Collecting numpy>=1.17.0
    Downloading numpy-1.24.2-cp38-cp38-win_amd64.whl (14.9 MB)
       --------------------------------------- 14.9/14.9 MB 59.8 MB/s eta 0:00:00
  Installing collected packages: numpy, scipy, pdfo
  Successfully installed numpy-1.24.2 pdfo-1.2.1 scipy-1.10.1
  + bash D:\a\pdfo\pdfo/tools/github/cibw_test_command.sh
  + python -m unittest pdfo.testpdfo
  E
  ======================================================================
  ERROR: runTest (pdfo.tests.test_pdfo.TestPDFO)
  PDFO tests on unconstrained and constrained problems.
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test\lib\site-packages\pdfo\_uobyqa.py", line 112, in uobyqa
      from .gethuge import gethuge
  ImportError: DLL load failed while importing gethuge: The specified module could not be found.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test\lib\site-packages\pdfo\tests\test_pdfo.py", line 128, in runTest
      global_res = func_solver(**args)
    File "D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test\lib\site-packages\pdfo\_uobyqa.py", line 117, in uobyqa
      import_error_so('gethuge')
    File "D:\a\_temp\msys64\tmp\cibw-run-fohq48qx\cp38-win_amd64\venv-test\lib\site-packages\pdfo\_dependencies.py", line 3628, in import_error_so
      raise ImportError('{} is missing. Please reinstall {} and ensure the '
  ImportError: gethuge is missing. Please reinstall pdfo and ensure the fulfilment of the requirements (numpy>=1.20.0).
  
  ----------------------------------------------------------------------
  Ran 1 test in 0.054s
  
  FAILED (errors=1)
  
  Testing unconstrained problems ...
  Error: Command bash D:\a\pdfo\pdfo/tools/github/cibw_test_command.sh failed with code 1. None
  
                                                             ✕ 18.66s
creating virtual environment...
installing cibuildwheel...
Error: Process completed with exit code 1.
0s

The full workflow log is here:
pdfo.log.txt (478.3 KB)

It would be nice if you could check what is wrong. Thank you very much.

9

After digging into the files, it seems that if either __MINGW32__ or __MINGW64__ are defined, then NumPy (numpy/numpy/core/include/numpy/npy_os.h on GitHub) defines NPY_OS_MINGW. When this variable is defined, then F2PY should not attempt to load threads.h (following the issue #18910 on NumPy’s repo).

Hence, it seems that neither __MINGW32__ nor __MINGW64__ are defined in my case, which is weird.

Thanks for your help,
Cheers,
Tom.

10

I will try and have a look early next week, but in general testing for cross-compiled wheels might be an issue (not sure if that’s what’s going on here)

2 Likes
11

I found the solution to my problem today, I post it here in case it is helpful to someone else in the future.

I simply added the argument default_options: ['c_std=c99', 'fortran_std=legacy'] to the call to project in the main meson.build (as did SciPy).

2 Likes
12

Do you still want me to have a look or did you fix your problems?

13

Hello @gnikit . Thank you for keeping this in mind. I think @ragonneau has solved the problem himself. Here is the new workflow file:

Many thanks again.