Using Julia libraries in Fortran with StaticCompiler.jl

This actually works today with StaticCompiler.jl[1]

julia> using Bessels, StaticCompiler

julia> compile_shlib(besselj1, Tuple{Float64}, name="besselj1_") # Fortran conventions require the name to end with an underscore.
"/home/mason/bessel_test/Bessels_besselj1.so"

Here is me calling it from Fortran:

program call_jl_besselj1
  implicit none
  integer, parameter :: p = selected_real_kind(15, 307)
  real(p) :: x, r
  real(p), external :: julia_besselj1

  x = 1.234_p
  r = julia_besselj1(x)
  print *, "res: ", r
end program call_jl_besselj1
bessel_test$  gfortran call_jl_bessels.f90 Bessels_besselj1.o -o call_jl_bessels
bessel_test$ ./call_jl_bessels 
 res:   0.50677700893555877    

The object file for the compiled Bessels.besselj1 is only 16Kb, so firmly in the ‘small binary’ category:

bessel_test$ du -h Bessels_besselj1.o
16K     Bessels_besselj1.o

[1] StaticCompiler.jl is pretty experimental, but should be somewhat stable for the use case here of a very simple function like Bessels.besseljl1, since almost all the serious work is being done by GPUCompiler.jl which is a much more serious project.

Nonetheless, binaries produced in this way need to be seriously tested before they are trusted.

7 Likes

I’m a little sad, I couldn’t manage to build the static library from Julia myself.

What I did:

[gbellomi@login2 lib]$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.3 (2022-05-06)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using Bessels

julia> using StaticCompiler

julia> compile_shlib(besselj1, Tuple{Float64}, name="besselj1_")
"/scratch/gbellomi/call_julia_bessels/lib/Bessels_besselj1.so"

julia> 

# No error message here, I just closed julia with ctrl+d

[gbellomi@login2 lib]$ ls
Bessels_besselj1.o  Bessels_besselj1.so

# I see what I expect in the lib directory

[gbellomi@login2 lib]$ cd ..

# I go "upstairs" to call gfortran on both the fortran source (located in test/) and the julia object

[gbellomi@login2 call_julia_bessels]$ gfortran test/bench.f90 lib/Bessels_besselj1.o -o test.o
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: lib/Bessels_besselj1.o: in function `julia_sincos_domain_error_601':
text:(.text+0x2069): undefined reference to `jl_invoke'
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: text:(.text+0x2071): undefined reference to `jl_throw'
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: lib/Bessels_besselj1.o: in function `gpu_gc_pool_alloc':
text:(.text+0x20c0): undefined reference to `jl_throw'
collect2: error: ld returned 1 exit status

# for some reason I cannot link

Originally I wanted to setup an fpm project, so that’s the reason for the “complex” directory tree, but I thought the errors were related to me not knowing how to link properly with fpm so I tried doing it directly with a single gfortran call, as in @Mason’s example. I’ve also tried to do everything in the same directory, encountering the very same problems:

[gbellomi@login2 call_julia_bessels]$ cp test/bench.f90 lib/bench.f90
[gbellomi@login2 call_julia_bessels]$ cd lib/
[gbellomi@login2 lib]$ ls
bench.f90  Bessels_besselj1.o  Bessels_besselj1.so
[gbellomi@login2 lib]$ gfortran bench.f90 Bessels_besselj1.o -o bench.o
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: Bessels_besselj1.o: in function `julia_sincos_domain_error_601':
text:(.text+0x2069): undefined reference to `jl_invoke'
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: text:(.text+0x2071): undefined reference to `jl_throw'
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: Bessels_besselj1.o: in function `gpu_gc_pool_alloc':
text:(.text+0x20c0): undefined reference to `jl_throw'
collect2: error: ld returned 1 exit status

So there might be problems in the Julia generated library itself, right?

Try to link against the shared object, also you are linking an executable rather than creating an object file. Here is my suggestion on how to change the link line:

-gfortran test/bench.f90 lib/Bessels_besselj1.o -o test.o
+gfortran test/bench.f90 lib/Bessels_besselj1.so -o tester "-Wl,-rpath,$PWD/lib"

The rpath is to be on the safe side when running the executable later.

1 Like

Sure, that’s what I initially tried, but given it wasn’t working (same two undefined references) I just went to a verbatim copy of what the original example was doing.

I’ll give a try to your below suggestion, later or tomorrow, thanks!

EDIT: @awvwgk yes, as suspected, still the same problem. In fact exactly the same error message I got before in my first attempt (the fpm way, putting a suitable link entry in the toml file, etc)

$ gfortran test/bench.f90 lib/Bessels_besselj1.so -o tester "-Wl,-rpath,$PWD/lib"
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: lib/Bessels_besselj1.so: undefined reference to `jl_invoke'
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: lib/Bessels_besselj1.so: undefined reference to `jl_throw'
collect2: error: ld returned 1 exit status
$ mv lib/Bessels_besselj1.so lib/libBessels_besselj1.so
$ fpm @benchmark
bench.f90                              done.
bench                                  failed.
[100%] Compiling...
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: lib//libBessels_besselj1.so: undefined reference to `jl_invoke'
/home/gbellomi/spack/opt/spack/linux-centos7-sandybridge/gcc-8.3.0/binutils-2.38-u44qdep7hvpfnojgzv54twl7jmclt4mv/bin/ld: lib//libBessels_besselj1.so: undefined reference to `jl_throw'
collect2: error: ld returned 1 exit status
<ERROR> Compilation failed for object " bench "
<ERROR>stopping due to failed compilation
STOP 1
my fpm.toml
name = "call_julia_bessels"
version = "0.0.0"
license = "unlicense"
author = "gb@Ulysses"
maintainer = "gbellomi@sissa.it"
[build]
link = "Bessels_besselj1"
auto-tests = true
[install]
library = false
my fpm.rsp
# Custom FPM profiles to run profilable but fast benchmarks
@benchmark
options test
options --compiler gfortran
options --flag "-O3 -funroll-loops -march=native -g -pg -L lib/"
@profile
options	test
options	--compiler gfortran
options --runner gprof
options	--flag "-O3 -funroll-loops -march=native -g -pg -L lib/"
my tree
$ tree
.
├── build
│   ├── cache.toml
│   ├── dependencies
│   └── gfortran_C8CECF3FB2C6F510
│       ├── call_julia_bessels
│       │   ├── test_bench.f90.o
│       │   ├── test_bench.f90.o.digest
│       │   └── test_bench.f90.o.log
│       └── test
│           └── bench.log
├── fpm.rsp
├── fpm.toml
├── lib
│   ├── Bessels_besselj1.o
│   └── libBessels_besselj1.so
├── README.md
├── src
└── test
    └── bench.f90

gfortran \neq Fortran

Also, presumably StaticCompiler.jl employs LLVM with Clang or some such, meaning there is in effect a C companion processor. Under the circumstances, it would appear the Fortran caller should use the Fortran standard interoperability facility with a C companion processor.

compile_shlib(besselj1, Tuple{Float64}, name="besselj1")
program call_jl_besselj1
  use, intrinsic :: iso_c_binding, only : Float64 => c_double
  interface
     function julia_besselj1( x ) result(r) bind(C, name="besselj1")
        ! Argument list
        real(Float64), intent(in) :: x
        ! Function result
        real(Float64) :: r
     end function 
  end interface
  real(Float64) :: x, r

  x = 1.234_Float64
  r = julia_besselj1(x)
  print *, "res: ", r
end program call_jl_besselj1

Edit: fixed the name of the processor to its “correct” one.

1 Like

Oh perfect, this indeed could be the right, compiler independent, safe approach. Nice!

Anyway, the fact the jl_invoke and jl_throw are indeed missing is easily demonstrated:

[gbellomi@login2 lib]$ nm *Bessels* | grep jl
                 U jl_invoke
                 U jl_throw
                 U jl_invoke
                 U jl_throw
[gbellomi@login2 lib]$ ls
Bessels_besselj1.o  libBessels_besselj1.so

Full look into libBessels_besselj1.so
[gbellomi@login2 lib]$ nm libBessels_besselj1.so 
0000000000203030 B __bss_start
0000000000203030 b completed.6943
                 w __cxa_finalize@@GLIBC_2.2.5
0000000000000580 t deregister_tm_clones
0000000000000610 t __do_global_dtors_aux
0000000000202e10 t __do_global_dtors_aux_fini_array_entry
0000000000202e18 d __dso_handle
0000000000202e20 d _DYNAMIC
0000000000203030 D _edata
0000000000203038 B _end
0000000000002724 T _fini
0000000000000650 t frame_dummy
0000000000202e08 t __frame_dummy_init_array_entry
0000000000002cc0 r __FRAME_END__
0000000000203000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
0000000000002a80 r __GNU_EH_FRAME_HDR
0000000000002700 t gpu_gc_pool_alloc
00000000000026e0 t gpu_malloc
00000000000026f0 t gpu_report_oom
0000000000000508 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 U jl_invoke
                 U jl_throw
0000000000002670 T julia_besselj1_
0000000000001360 t julia__besselj1_588
0000000000001110 t julia_cos_sum_612
0000000000000660 t julia_paynehanek_599
0000000000000950 t julia_sincos_595
0000000000002680 t julia_sincos_domain_error_601
                 U malloc@@GLIBC_2.2.5
00000000000005c0 t register_tm_clones
0000000000203030 d __TMC_END__
Full look into Bessels_besselj1.o
[gbellomi@login2 lib]$ nm Bessels_besselj1.o 
00000000000020a0 t gpu_gc_pool_alloc
0000000000002080 t gpu_malloc
0000000000002090 t gpu_report_oom
                 U jl_invoke
                 U jl_throw
0000000000002010 T julia_besselj1_
0000000000000d00 t julia__besselj1_588
0000000000000ab0 t julia_cos_sum_612
0000000000000000 t julia_paynehanek_599
00000000000002f0 t julia_sincos_595
0000000000002020 t julia_sincos_domain_error_601
0000000000000000 r .LCPI0_0
0000000000000008 r .LCPI0_1
0000000000000010 r .LCPI0_2
0000000000000000 r .LCPI1_0
0000000000000018 r .LCPI1_1
0000000000000060 r .LCPI1_10
0000000000000068 r .LCPI1_11
0000000000000070 r .LCPI1_12
0000000000000078 r .LCPI1_13
0000000000000080 r .LCPI1_14
0000000000000088 r .LCPI1_15
0000000000000090 r .LCPI1_16
0000000000000098 r .LCPI1_17
00000000000000a0 r .LCPI1_18
00000000000000a8 r .LCPI1_19
0000000000000020 r .LCPI1_2
00000000000000b0 r .LCPI1_20
00000000000000b8 r .LCPI1_21
00000000000000c0 r .LCPI1_22
00000000000000c8 r .LCPI1_23
00000000000000d0 r .LCPI1_24
00000000000000d8 r .LCPI1_25
00000000000000e0 r .LCPI1_26
00000000000000e8 r .LCPI1_27
00000000000000f0 r .LCPI1_28
00000000000000f8 r .LCPI1_29
0000000000000028 r .LCPI1_3
0000000000000100 r .LCPI1_30
0000000000000108 r .LCPI1_31
0000000000000110 r .LCPI1_32
0000000000000118 r .LCPI1_33
0000000000000120 r .LCPI1_34
0000000000000128 r .LCPI1_35
0000000000000130 r .LCPI1_36
0000000000000138 r .LCPI1_37
0000000000000010 r .LCPI1_38
0000000000000030 r .LCPI1_4
0000000000000038 r .LCPI1_5
0000000000000040 r .LCPI1_6
0000000000000048 r .LCPI1_7
0000000000000050 r .LCPI1_8
0000000000000058 r .LCPI1_9
0000000000000140 r .LCPI2_0
0000000000000148 r .LCPI2_1
0000000000000020 r .LCPI2_10
0000000000000190 r .LCPI2_11
0000000000000198 r .LCPI2_12
00000000000001a0 r .LCPI2_13
00000000000001a8 r .LCPI2_14
00000000000001b0 r .LCPI2_15
00000000000001b8 r .LCPI2_16
00000000000001c0 r .LCPI2_17
0000000000000150 r .LCPI2_2
0000000000000158 r .LCPI2_3
0000000000000160 r .LCPI2_4
0000000000000168 r .LCPI2_5
0000000000000170 r .LCPI2_6
0000000000000178 r .LCPI2_7
0000000000000180 r .LCPI2_8
0000000000000188 r .LCPI2_9
00000000000001c8 r .LCPI3_0
00000000000001d0 r .LCPI3_1
0000000000000210 r .LCPI3_10
0000000000000218 r .LCPI3_11
0000000000000220 r .LCPI3_12
0000000000000228 r .LCPI3_13
0000000000000230 r .LCPI3_14
0000000000000238 r .LCPI3_15
0000000000000240 r .LCPI3_16
0000000000000248 r .LCPI3_17
0000000000000250 r .LCPI3_18
0000000000000258 r .LCPI3_19
0000000000000030 r .LCPI3_2
0000000000000260 r .LCPI3_20
0000000000000268 r .LCPI3_21
0000000000000270 r .LCPI3_22
0000000000000278 r .LCPI3_23
0000000000000280 r .LCPI3_24
0000000000000288 r .LCPI3_25
0000000000000290 r .LCPI3_26
0000000000000298 r .LCPI3_27
00000000000002a0 r .LCPI3_28
00000000000002a8 r .LCPI3_29
00000000000001d8 r .LCPI3_3
00000000000002b0 r .LCPI3_30
00000000000002b8 r .LCPI3_31
00000000000001e0 r .LCPI3_4
00000000000001e8 r .LCPI3_5
00000000000001f0 r .LCPI3_6
00000000000001f8 r .LCPI3_7
0000000000000200 r .LCPI3_8
0000000000000208 r .LCPI3_9
0000000000000040 r .L_j_const2
                 U malloc

Probably the most interesting thing, let’s look for all what is unlinked:

[gbellomi@login2 lib]$ nm *Bessels* | grep U
                 U jl_invoke
                 U jl_throw
                 U malloc
0000000000002a80 r __GNU_EH_FRAME_HDR
                 U jl_invoke
                 U jl_throw
                 U malloc@@GLIBC_2.2.5

Hi @rgba, so the problem here is that StaticCompiler.jl in it’s standalone executable mode can’t use julia’s runtime at all, which also means no error handling. Digging into this, it seems that on julia version 1.7.3, an optimization is not occurring that allows it to elide the error branch, and so the binary is not working.

We are currently working on redirecting errors from the runtime to regular exit codes in this PR: Rebase of errors-are-fine by gbaraldi · Pull Request #90 · tshort/StaticCompiler.jl · GitHub but it’s not fully functional yet.

However, in the meantime if you are willing to try this on julia 1.8.2, it should work! If it still has this problem on 1.8.2, then maybe try doing

using Pkg; pkg"add StaticCompiler#master"
1 Like

I was able to reproduce the same problem you experienced on 1.7.3 and it was resolved by updating to 1.8.2.

1 Like

Thanks for the clarification!

Yes, I suspected that Julia version could have been too old for this (since you clearly stated this is a fairly recent achievement). The problem is that I do not have Julia 1.8.2 on spack, 1.7.3 is the preferred version on there:

[gbellomi@login2 lib]$ spack info julia
MakefilePackage:   julia

Description:
    The Julia Language: A fresh approach to technical computing

Homepage: https://julialang.org

Preferred version:  
    1.7.3        https://github.com/JuliaLang/julia/releases/download/v1.7.3/julia-1.7.3.tar.gz

Safe versions:  
    master       [git] https://github.com/JuliaLang/julia.git on branch master
    1.8.0-rc1    https://github.com/JuliaLang/julia/releases/download/v1.8.0-rc1/julia-1.8.0-rc1.tar.gz
    1.7.3        https://github.com/JuliaLang/julia/releases/download/v1.7.3/julia-1.7.3.tar.gz
    1.7.2        https://github.com/JuliaLang/julia/releases/download/v1.7.2/julia-1.7.2.tar.gz
    1.7.1        https://github.com/JuliaLang/julia/releases/download/v1.7.1/julia-1.7.1.tar.gz
    1.7.0        https://github.com/JuliaLang/julia/releases/download/v1.7.0/julia-1.7.0.tar.gz
    1.6.6        https://github.com/JuliaLang/julia/releases/download/v1.6.6/julia-1.6.6.tar.gz
    1.6.5        https://github.com/JuliaLang/julia/releases/download/v1.6.5/julia-1.6.5.tar.gz
    1.6.4        https://github.com/JuliaLang/julia/releases/download/v1.6.4/julia-1.6.4.tar.gz

Deprecated versions:  
    1.6.3        https://github.com/JuliaLang/julia/releases/download/v1.6.3/julia-1.6.3.tar.gz
    1.6.2        https://github.com/JuliaLang/julia/releases/download/v1.6.2/julia-1.6.2.tar.gz
    1.6.1        https://github.com/JuliaLang/julia/releases/download/v1.6.1/julia-1.6.1.tar.gz
    1.6.0        https://github.com/JuliaLang/julia/releases/download/v1.6.0/julia-1.6.0.tar.gz
    1.5.4        https://github.com/JuliaLang/julia/releases/download/v1.5.4/julia-1.5.4.tar.gz
    1.5.3        https://github.com/JuliaLang/julia/releases/download/v1.5.3/julia-1.5.3.tar.gz
    1.5.2        https://github.com/JuliaLang/julia/releases/download/v1.5.2/julia-1.5.2.tar.gz
    1.5.1        https://github.com/JuliaLang/julia/releases/download/v1.5.1/julia-1.5.1.tar.gz
    1.5.0        https://github.com/JuliaLang/julia/releases/download/v1.5.0/julia-1.5.0.tar.gz
    1.4.2        https://github.com/JuliaLang/julia/releases/download/v1.4.2/julia-1.4.2.tar.gz
    1.4.1        https://github.com/JuliaLang/julia/releases/download/v1.4.1/julia-1.4.1.tar.gz
    1.4.0        https://github.com/JuliaLang/julia/releases/download/v1.4.0/julia-1.4.0.tar.gz
    1.3.1        https://github.com/JuliaLang/julia/releases/download/v1.3.1/julia-1.3.1.tar.gz
    1.2.0        https://github.com/JuliaLang/julia/releases/download/v1.2.0/julia-1.2.0.tar.gz
    1.1.1        https://github.com/JuliaLang/julia/releases/download/v1.1.1/julia-1.1.1.tar.gz
    1.0.0        https://github.com/JuliaLang/julia/releases/download/v1.0.0/julia-1.0.0.tar.gz
    0.6.2        https://github.com/JuliaLang/julia/releases/download/v0.6.2/julia-0.6.2.tar.gz
    0.5.2        https://github.com/JuliaLang/julia/releases/download/v0.5.2/julia-0.5.2.tar.gz
    0.5.1        https://github.com/JuliaLang/julia/releases/download/v0.5.1/julia-0.5.1.tar.gz
    0.5.0        https://github.com/JuliaLang/julia/releases/download/v0.5.0/julia-0.5.0.tar.gz
    0.4.7        https://github.com/JuliaLang/julia/releases/download/v0.4.7/julia-0.4.7.tar.gz
    0.4.6        https://github.com/JuliaLang/julia/releases/download/v0.4.6/julia-0.4.6.tar.gz
    0.4.5        https://github.com/JuliaLang/julia/releases/download/v0.4.5/julia-0.4.5.tar.gz
    0.4.3        https://github.com/JuliaLang/julia/releases/download/v0.4.3/julia-0.4.3.tar.gz

Variants:
    Name [Default]     When    Allowed values    Description
    ===============    ====    ==============    ============================

    openlibm [on]      --      on, off           Use openlibm instead of libm
    precompile [on]    --      on, off           Improve julia startup time

Build Dependencies:
    blas  dsfmt  lapack             libgit2  libuv        libwhich  mbedtls  nghttp2   openlibm  patchelf  perl          unwind    zlib
    curl  gmp    libblastrampoline  libssh2  libuv-julia  llvm      mpfr     openblas  p7zip     pcre2     suite-sparse  utf8proc

Link Dependencies:
    blas  dsfmt  lapack             libgit2  libuv        llvm     mpfr     openblas  p7zip  suite-sparse  utf8proc
    curl  gmp    libblastrampoline  libssh2  libuv-julia  mbedtls  nghttp2  openlibm  pcre2  unwind        zlib

Run Dependencies:
    None

Before leaving the task to spack (which means compiling from source, a HUGE pain, mainly due to LLVM size) I tried getting a binary for julia 1.8.2 from the official website, but it was not working on my HPC cluster. Namely, when launched, it displayed the logo and version info, then this error message

SYSTEM: caught exception of type

after which it was completely frozen (not even exiting with ctrl+d, I needed to kill the whole ssh terminal).

Unfortunately deployment on servers can be quite a hassle (for whatever software, this is not specific to Julia).

Ah, yes, the joys of clusters. You could try with 1.8.0-rc1, or master. Both should work (depending on which master that is I guess)

This I have no idea about, since I’ve never actually written or modified a build system in my life. As long as you can call julia from the cmake file though, then I don’t see why you couldn’t do it.

i.e. one could write a bash script that does it easily, so presumably make could also manage.