Strange behaviour of Intel ifort/ifx compilers with the MKL library on Linux

Compilers
1

Hi all,

I have my own numerical modern Fortran CFD code (open source) running simulations (since 2005). Part of it is solving a sparse matrix A x = b by an iterative method.

Two options are available: 1) Based on Sparsekit’s Bi Conjugate Gradient stabilized with an ILUT preconditioner (by Yousef Saad) and an alternative version that uses OpenMP and b) using Intel’s MKL with the Restarted Generalized Minimal Residual (FGMRES) method (also by Yousef Saad) with an ILUT preconditioner (with multiple threads). One can choose the amount of cores to be used.

The syntax of the MKL library calls have changed. Initially everything looked fine. However, testing the new syntax with large matrices has become problematic.

Characteristics of the test cases (all restarting from a converged solution):

  1. N = 55,823 2D, small
  2. N = 3,950,370 3D medium, about 4M
  3. N = 5,225,299 3D bit larger, about 5M
  4. N = 8,238,210 3D large, about 8M
  5. N = 9,221,750 3D large, about 9M

An overview of a few (older) Ubuntu computers with ifort/ifx versions

  1. tennekes, Ubuntu 18.04, ifort 17.0.4 20170411, with the old syntax of solver_mkl.f90
  2. huygens, Ubuntu 22.04, ifx 2024.1.0 20240308, ifx/ifort using the old MKL interface
  3. stevin, Ubuntu 24.04, ifx 2025.3.2 20260112, ifx with the old MKL interface
  4. leeghwater, Ubuntu 26.04 ifx 2026.1.0 20260617 ifx with the new MKL interface

ALL test cases, run successfully on all puters, with SparsKit2 BiCGstab (OMP).

However… with the MKL library that is not the case:

  1. tennekes, Ubuntu 18.04, ifort, ALL test cases
  2. huygens, Ubuntu 22.04, ifx 2024.1, only cases 1, 2, and 3 (4 and 5 have segmentation faults)
  3. stevin, Ubuntu 24.04, ifx 2025.3.2, only cases 1, 2, and 3 (4 and 5 have SIGSEGV)
  4. leeghwater, Ubuntu 26.04 ifx 2026.1, only cases 1, 2, and 3 (4 and 5 have SIGSEGV)

With gdb on the Ubuntu 24.04, ifx 2025.3.2 machine:

Program received signal SIGSEGV, Segmentation fault.
0x0000155231770825 in mkl_blas_avx2_xdcopy () from /opt/intel/oneapi/mkl/2025.3/lib/libmkl_avx2.so.2
(gdb) bt
#0 0x0000155231770825 in mkl_blas_avx2_xdcopy () from /opt/intel/oneapi/mkl/2025.3/lib/libmkl_avx2.so.2
#1 0x0000155550673a89 in mkl_iss_dfgmres () from /opt/intel/oneapi/mkl/2025.3/lib/libmkl_core.so.2
#2 0x00001555542812b0 in dfgmres_ () from /opt/intel/oneapi/mkl/2025.3/lib/libmkl_intel_lp64.so.2
#3 0x0000155554def6c4 in dfgmres_ () from /opt/intel/oneapi/mkl/2025.3/lib/libmkl_rt.so.2
#4 0x000000000042991f in me$solve_mkl_p_ (debug=, iodbg=, iodef=, ncel=, nnz=,
acsr=…, aclc=…, arwc=…, ithreads=, rtolp=, atolp=, rhs=…, sol=…, istatus=)
at solver_mkl.f90:341

Without gdb:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libc.so.6 0000150167C45330 Unknown Unknown Unknown
libmkl_avx2.so.2 000014FE4D370825 mkl_blas_avx2_xdc Unknown Unknown
libmkl_core.so.2 000015016C473A89 mkl_iss_dfgmres Unknown Unknown
libmkl_intel_lp64 00001501700812B0 DFGMRES Unknown Unknown
libmkl_rt.so.2 0000150170BEF6C4 DFGMRES Unknown Unknown
dolfyn_MKL 000000000042991F me$solve_mkl_p_ 341 solver_mkl.f90

Same case on the Ubuntu 26.04 ifx 2026.1 machine:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libc.so.6 00000867DB445CB0 Unknown Unknown Unknown
libmkl_avx2.so.3 0000086290F3EF35 mkl_blas_avx2_xdc Unknown Unknown
libmkl_intel_thre 00000865592BB977 mkl_blas_dcopy Unknown Unknown
libmkl_core.so.3 000008655F8EDC92 mkl_blas_lp64_dco Unknown Unknown
libmkl_intel_thre 00000865595AC35A mkl_spblas_lp64_m Unknown Unknown
libmkl_intel_thre 0000086559AF03A0 mkl_sparse_d_defa Unknown Unknown
dolfyn_MKL 00000000004276AF me$solve_mkl_p_ 396 solver_mkl.f90

Another interesting issue… FGMRES uses a temporary array defined as:

 ! default max iter
 mkl_maxiter = min(150,Ncel)
    
 if(.not. allocated(mkl_tmp)) allocate(mkl_tmp(Ncel*(2*mkl_maxiter+1) + & 
                                      (mkl_maxiter*(mkl_maxiter+9))/2 + 1 ))

One can allocate some more by changing the 2 in Ncel x ( 2 x mkl_maxiter + 1) into 8… then on the Ubuntu 24.04, ifx 2025.3.2 machine, with case 4, then the segmentation fault happens after 13 gmres iterations (instead of bailing out in the first iteration). Another strange thing happened a few times. Rerunning the same case after a segmentation fault occurred, a few time it seg faulted after 3 iterations. The same case, the same data set… this suggests something very deep.

I ran completely out of ideas. Tried (of course) “ulimit -s unlimited”, “-heap-arrays”, using “-lmkl_rt”, static and dynamic linking, using 1 or more cores…

It does not matter if one adjusts “mkl_maxiter = min(150,Ncel)” to a lower or higher value, dfgmres_init will reset it to the default min(150,Ncel). In the past, because A changes continually, I used a lower value of 20. Also dfgmres_check does not complain. It cannot be the coding (I have been using cases with N = 6M as well).

Has anyone any idea what to try? The Intel Fortran / MKL Forum does not seems to be available at the moment.

Many thanks in advance for looking at this post :- )

Henk

2

mkl_blas_avx2_xdcopy

Because the crash is with AVX2 routine, one possibility could be alignment issue. I don’t remember, whether ifx allocate statement aligns allocated memory, but perhaps the default memory alignment is less than MKL routine expects.

3

allocate(mkl_tmp(Ncel*(2*mkl_maxiter+1) + &
(mkl_maxiter*(mkl_maxiter+9))/2 + 1 ))

Also, could you check the actual size you’re trying to allocate? If you’re using 32 bit default integer for size computation, there’s a possibility of an integer overflow.