Packaging for MSYS2

@zoziha asked me about a MSYS2 version of stdlib when we were preparing the release for version 0.1.0 and while I’m not much of a Windows user, the MSYS2 tool chain is something I found very useful. In particular because its package manager, pacman, is based on Arch Linux’ tool chain, the one I’m using for my own machine.

Arch Linux is known for its incredible comprehensive documentation, if you don’t know about it yet, checkout the Arch Linux Wiki. The other nice thing is its very accessible packaging tool chain and this is something MSYS2 inherited.

This post will show you how to package the Fortran standard library (or other projects) for MSYS2.

Step 0: Setup

This will only work on Windows, if you don’t have a Windows machine, you can still learn how you could package for MSYS2, but you won’t be able to test out any of the examples below.

To install MSYS2, checkout their webpage

Download and run the installer to get the MSYS2 tool chain. We will use one of the MinGW terminals for the following steps (mingw64, mingw32 or ucrt64 will work here, all examples are with mingw64).

Step 1: Dependencies

Before we can start building stdlib, we first need the fypp preprocessor, which is unfortunately not yet packaged for MSYS2. Therefore, we will start with packaging fypp.

First create a directory for your build files and one for fypp in particular

❯ mkdir ~/mingw-w64
❯ cd ~/mingw-w64
❯ mkdir fypp
❯ cd fypp

In the fypp directory we create a file named PKGBUILD with the content:

_pyname=fypp
_realname=${_pyname}
pkgbase=mingw-w64-python-${_realname}
pkgname=("${MINGW_PACKAGE_PREFIX}-python-${_realname}")
pkgver=3.1
pkgrel=1
pkgdesc="Python powered Fortran preprocessor (mingw-w64)"
url="https://fypp.readthedocs.io/"
license=("BSD-2-Clause")
arch=('any')
mingw_arch=('mingw32' 'mingw64' 'ucrt64' 'clang64' 'clang32')
depends=("${MINGW_PACKAGE_PREFIX}-python")
makedepends=("${MINGW_PACKAGE_PREFIX}-python-setuptools")
source=("${_realname}-${pkgver}.tar.gz"::"https://github.com/aradi/fypp/archive/${pkgver}.tar.gz")
sha256sums=('0f66e849869632978a8a0623ee510bb860a74004fdabfbfb542656c6c1a7eb5a')

prepare() {
  cd ${srcdir}
  rm -rf python-build-${CARCH} | true
  cp -r "${_pyname}-${pkgver}" "python-build-${CARCH}"
}

build() {
  msg "Python build for ${CARCH}"
  cd "${srcdir}/python-build-${CARCH}"
  MSYS2_ARG_CONV_EXCL="--prefix=;--install-scripts=;--install-platlib=" \
      ${MINGW_PREFIX}/bin/python setup.py build
}

package() {
  cd "${srcdir}/python-build-${CARCH}"
  MSYS2_ARG_CONV_EXCL="--prefix=;--install-scripts=;--install-platlib=" \
    ${MINGW_PREFIX}/bin/python setup.py install --prefix=${MINGW_PREFIX} --root="${pkgdir}" --optimize=1 --skip-build
}

What is happening here? A PKGBUILD is written in bash, we can define variables and functions here, certain functions are picked up by our packaging script and are used as hooks to produce the actual package. Also, a number of environment variables is predefined for us, like MINGW_PACKAGE_PREFIX, CARCH, MINGW_PREFIX, srcdir and pkgdir.

We define a _pyname and _realname variable, here they are both named fypp. Since we are creating a MinGW package, we have to introduce the prefix MINGW_PACKAGE_PREFIX to ensure we can only provide and use packages for one terminal. Also, we prefix our package name with python since we are packaging a Python project. In Arch as well in MSYS2 Python is always Python 3, no need to worry about this.

The next variables (pkgver, pkgrel, pkgdesc, url, license) provide the meta data of the project, we can infer this from the documentation of the project we are packaging. For the MSYS2 toolchain we always use arch=('any') since the actual architecture selection happens with the mingw_arch variables, we can use almost all supported architectures here, since we are dealing with a Python package.

Finally, we have to select our build and run dependencies, for building we need setuptools and for running Python, of course. To actually retrieve the sources we are building from, the source variable is specified, every entry in the source array requires a checksum in sha256sums. You can download the tarball and run sha256sum in a terminal to get the correct value. Having checksums is an important integrity check to ensure the downloaded content is valid.

The functions, prepare, build and package are the respective stages of building our project. Since they just invoke setup.py to deal with all the things. We will only mention one detail here: the MSYS2_ARG_CONV_EXCL argument, which makes sure that generated paths are compatible with Windows.

Step 2: Building a package

To actually build a package we use the makepkg-mingw script, with

❯ MINGW_ARCH=mingw64 makepkg-mingw --syncdeps --cleanbuild --log --force

The MINGW_ARCH environment variable should match the architecture of your terminal, because you can only use MinGW packages with the same architecture in your current terminal.

After the successful build we can use pacman to install fypp for this terminal:

❯ pacman -U mingw-w64-x86_64-python-fypp-3.1-1-any.pkg.tar.zst

Checkout that fypp is working with

❯ fypp --version
fypp 3.1

Step 3: Packaging stdlib

The Fortran standard library is CMake based, therefore we will create a build file for a CMake project.

The name stdlib is a bit vague, therefore we will go with the CMake project name which is fortran_stdlib. Always keep in mind to play nice with all the other packages in your package manager when choosing a name.

❯ cd ~/mingw-w64
❯ mkdir fortran_stdlib
❯ cd fortran_stdlib

Again, we create a file called PKGBUILD

_fname=stdlib
_realname=fortran_${_fname}
pkgbase=mingw-w64-${_realname}
pkgname="${MINGW_PACKAGE_PREFIX}-${_realname}"
pkgver=0.1.0
pkgrel=1
arch=('any')
mingw_arch=('mingw32' 'mingw64' 'ucrt64')
pkgdesc="Fortran standard library (mingw-w64)"
url="https://github.com/fortran-lang/stdlib"
depends=("${MINGW_PACKAGE_PREFIX}-gcc-libs"
         "${MINGW_PACKAGE_PREFIX}-gcc-libgfortran")
makedepends=("${MINGW_PACKAGE_PREFIX}-gcc-fortran"
             "${MINGW_PACKAGE_PREFIX}-python-fypp"
             "${MINGW_PACKAGE_PREFIX}-cmake"
             "${MINGW_PACKAGE_PREFIX}-ninja")
options=('strip')
license=('MIT')
source=(${_realname}-${pkgver}.tar.gz::"https://github.com/fortran-lang/stdlib/archive/refs/tags/v${pkgver}.tar.gz")
sha256sums=('0c715b9fc15102817c591d78c4a21ebba392512e8feb18fa5d0bdbcdebdcd52c')

build() {
  cd "${srcdir}/${_fname}-${pkgver}"
  local _build="build_${CARCH}"
  local _fc="${MINGW_PREFIX}/bin/gfortran"

  FC="${_fc}" \
  MSYS2_ARG_CONV_EXCL="-DCMAKE_INSTALL_PREFIX=" \
  ${MINGW_PREFIX}/bin/cmake \
    -GNinja \
    -DCMAKE_INSTALL_PREFIX=${MINGW_PREFIX} \
    -DCMAKE_BUILD_TYPE=Release \
    -B"${_build}"
  cmake --build "${_build}"
}

package() {
  cd "${srcdir}/${_fname}-${pkgver}/build_${CARCH}"

  DESTDIR="${pkgdir}" \
  cmake --install .

  install -Dm0644 ${srcdir}/${_fname}-${pkgver}/LICENSE ${pkgdir}${MINGW_PREFIX}/share/licenses/${_realname}/LICENSE
}

Note that we now actually depend on our python-fypp package for building stdlib. We also have to limit the mingw_arch entries a bit because the clang based MinGW terminals don’t have a Fortran compiler yet.

With this we start the makepkg-mingw script again to build our package (remember to select the MINGW_ARCH matching your terminal):

❯ MINGW_ARCH=mingw64 makepkg-mingw --syncdeps --cleanbuild --log --force

After the successful build we can use pacman to install stdlib for this terminal:

❯ pacman -U mingw-w64-x86_64-fortran_stdlib-0.1.0-1-any.pkg.tar.zst

If you have pkg-config installed you can check whether stdlib is working now

❯ pkg-config fortran_stdlib --modversion
0.1.0

Step 4: Using stdlib

Finally, we want to make use of our installed stdlib version. For a quickstart we will use the CMake template for stdlib:

❯ mkdir ~/sources
❯ cd sources
❯ git clone https://github.com/fortran-lang/stdlib-cmake-example
❯ cd stdlib-cmake-example

As a ready to use CMake integration of stdlib, this project will automatically pick up a system wide installation of stdlib instead of building it on demand:

❯ cmake -B _build -G Ninja
-- The Fortran compiler identification is GNU 10.3.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: C:/msys64/mingw64/bin/gfortran.exe - skipped
-- Checking whether C:/msys64/mingw64/bin/gfortran.exe supports Fortran 90
-- Checking whether C:/msys64/mingw64/bin/gfortran.exe supports Fortran 90 - yes
-- Setting build type to 'RelWithDebInfo' as none was specified.
-- fortran_stdlib: Find installed package
-- test-drive: Find installed package
-- Could NOT find test-drive (missing: test-drive_DIR)
-- Retrieving test-drive from https://github.com/fortran-lang/test-drive
-- Configuring done
-- Generating done
-- Build files have been written to: C:/msys64/home/awvwgk/sources/stdlib-cmake-example/_build
❯ ninja -C _build
ninja: Entering directory `_build/'
[28/28] Linking Fortran executable _deps\test-drive-build\test\test-drive-tester.exe

The project also contains a simple example binary for using stdlib:

❯ ./_build/app/stdlib-example.exe
2021-10-04 19:34:04.238: INFO: This project uses the Fortran standard library!

And that’s it, now you know how to create MinGW package files for the MSYS2 tool chain. In principle you can use the above PKGBUILD files as template to package a lot of Python and CMake based projects (I haven’t tried building LFortran here yet).

4 Likes

Is it possible to use fpm to compile stdlib in msys2? In this way, there is no need to rely on fypp.

I don’t have a good computer knowledge. I saw the PR of the fpm and fypp packages you @awvwgk submitted in the msys2/MINGW-packages, where I am learning how to create msys2 packages. Because the environment I use is mainly the msys2 environment and the wsl environment, if possible, I may be able to participate in the maintenance of the msys2-stdlib package in the future. :slight_smile:
Thank you for your excellent work!

You are right, in principle it is possible to use fpm for building
stdlib. However, there is a caveat when packaging software with
multiple build systems since you want to produce a package that works
for many users. In case of stdlib CMake is still the primary build
system and produces the most feature complete package, including the
library, the module files, a CMake config and pkgconf export.
Especially the latter two ensure that stdlib can get reused in many
other build systems like CMake, meson, make, …

Frankly, I don’t see fpm here yet, because in fpm we would build our
dependencies on demand rather than finding them. I don’t see this as an
issue as long as fpm produces executables. Once we want integrate fpm
with packaging systems and provide libraries, this will be of course an
issue he have to solve. It is in fact a quite fundamental issue of
build systems like cargo or stack when providing libraries that should
be in included in system package managers.

1 Like

A MinGW package of fypp is now available in the MSYS2 distribution: Base Package: mingw-w64-python-fypp - MSYS2 Packages. Step 1 and 2 are now not necessary anymore, but I leave them in the tutorial since they could be helpful as templates for packaging other Python projects.