Basic OpenMP Test Program Not Exiting

[hirschbergerm]

I’m running the following basic parallelized Hello World program in VS Code:

program Hello

  Use omp_lib

  Implicit None

  INTEGER :: nthreads
  nthreads = 4

  CALL OMP_SET_NUM_THREADS(nthreads)

  write(*,*) omp_get_num_procs()
  write(*,*) omp_get_max_threads()
  write(*,*) omp_get_num_threads()

  !$OMP PARALLEL
    PRINT *, "Hello from process: ", OMP_GET_THREAD_NUM()
  !$OMP END PARALLEL

end program Hello

This correctly prints out “Hello from process: X” 4 times, but then the program doesn’t exit. Is this expected behavior? I haven’t found any posts online that address this.

[Arjen]

No, this is not what I would expect. It seems that the call to OMP_SET_NUM_THREADS does not work properly. If I leave it out, the program does finish (on my laptop it uses 24 cores by default). This is the behaviour with gfortran. If I use ifx, thne the original program stops nicely.

One curious thing: with both compilers (and with or without the call to OMP_SET_NUM_THREADS) I get the value 1 for OMP_GET_NUM_THREADS. Odd.

[PierU]

Hello,

Definitely not expected. Which platform/compiler/version are you using?

@Arjen

One curious thing: with both compilers (and with or without the call to OMP_SET_NUM_THREADS) I get the value 1 for OMP_GET_NUM_THREADS. Odd.

This is the normal output of this function if called from outside of a parallel region. It returns the number of threads that are currently active.

[Arjen]

Oops, yes, you’re absolutely right!

[hirschbergerm]

Hey thanks for the reminder, I’ll remember to include platform/compiler/version going forward.

I’m using gfortran 12.3.0. I’m just using the terminal in VS Code Windows 10 64-bit.

EDIT:

Actually I just tested Arjen’s idea of removing OMP_SET_NUM_THREADS() and the program finished executing.

Why would this be the case. It’s not urgent right now, but in the future I think that’s a variable I’d like to be able to control.

[PierU]

@hirschbergerm

I’m using gfortran 12.3.0. I’m just using the terminal in VS Code Windows 10 64-bit.

With gfortran 13.2 under the MSYS2 environment on Windows, this program correctly exits. I suspect some bad interaction (?) between gfortran and VS code.

[quote=“hirschbergerm, post:5, topic:9413”]
Actually I just tested Arjen’s idea of removing OMP_SET_NUM_THREADS() and the program finished executing.

Why would this be the case. It’s not urgent right now, but in the future I think that’s a variable I’d like to be able to control.[/quote]

For performance reasons (I think), it’s generally recommended to set the number of threads using the OMP_NUM_THREADS environment variable once for all rather than using the omp_set_num_threads() routine. That said, it shoud be work… Did you try using the NUM_THREADS() clause instead of the routine?
!$OMP PARALLEL NUM_THREADS(nthreads)

[hirschbergerm]

Using the NUM_THREADS clause fixed the issue for me! Thank you so much!

I wonder why calling omp_set_num_threads() lead to the issue?

[ivanpribec]

Maybe it is this issue

OpenMP and thread-safety of I/Os (write, read...)

Actually I am OK with my current practice of putting all I/Os in critical sections. I was just wondering of the thread-safety of I/Os was covered by the standards (Fortran and/or OpenMP) or not. It seems that the answer is “no” (in which case one should always use critical sections if one wants the code to be portable and not depend on a specific compiler). Do you have a link?

Can you try to use the modified program:

  !$OMP PARALLEL
    !$OMP CRITICAL
    PRINT *, "Hello from process: ", OMP_GET_THREAD_NUM()
    !$OMP END CRITICAL
  !$OMP END PARALLEL

Or maybe this other modification:

!$OMP PARALLEL
   id = omp_get_thread_num()
   print *, "Hello from process: ", id
!$OMP END PARALLEL

Just curious if it changes anything.

[ivanpribec]

Maybe this is relevant too:

The GNU Fortran runtime library uses various C library functions that depend on the locale, such as strtod and snprintf . In order to work correctly in locale-aware programs that set the locale using setlocale , the locale is reset to the default “C” locale while executing a formatted READ or WRITE statement. On targets supporting the POSIX 2008 per-thread locale functions (e.g. newlocale , uselocale , freelocale ), these are used and thus the global locale set using setlocale or the per-thread locales in other threads are not affected. However, on targets lacking this functionality, the global LC_NUMERIC locale is set to “C” during the formatted I/O. Thus, on such targets it’s not safe to call setlocale concurrently from another thread while a Fortran formatted I/O operation is in progress. Also, other threads doing something dependent on the LC_NUMERIC locale might not work correctly if a formatted I/O operation is in progress in another thread.

Source: Thread-safety of the runtime library (The GNU Fortran Compiler)

@PierU

That said, it shoud be work… Did you try using the NUM_THREADS() clause instead of the routine?
!$OMP PARALLEL NUM_THREADS(nthreads)

I don’t see major differences at the application level: Compiler Explorer

[JohnCampbell]

This problem with using CALL OMP_SET_NUM_THREADS(nthreads) was introduced after Gfortran ver 11.1.0 on the Windows version supplied by equation.com
implementation. I have not been able to get a response to reporting this issue.
I have not tested other sources of Windows versions of Gfortran, since 11.1.0.

As noted, using the NUM_THREADS clause fixed the issue, but this requires a rework of all previous codes.

I have previously noted this issue, including at the end of:

OpenMP and FORTRAN

Hi, I put my 30-year old FORTRAN code on 2D Euler-flow turbulence simulations here: It runs fine in my 12-core-CPU-15GB RAM laptop with all cores getting triggered, but wondering if I still missed any trick in optimization like gfortran -O3 -Ofast -openmp hooks. Added openmp stuff like: USE OMP_LIB !$OMP PARALLEL Am I missing any other trick in the book … Thanks, Suki

Non Windows versions do not have this problem.

[JohnCampbell]

I have an update to this problem of windows version of Gfortran, provided by equation.com
It appears that this problem of the program not correctly stopping was introduced after Ver 11.1.0 when equation.com’s relaease was changed to using pthread-w32.

Since Ver 11.1, one of my large computation problem has no longer worked with pthread-w32.
To give some background: this problem involves a finite element solution of a 20 GByte matrix for repeated (50,000) loading vectors across 10 threads. Each itteration requires reading the 20 GByte matrix from memory, through L3 cache to the 10 cores. I repeatidly use !$OMP BARRIER to align the threads so they can share the same information in L3 cache. This cache alignment reduced the itteration time from 11 seconds to 2.4 seconds, which is a significant improvement.
This problem was exaserbated by Using Windows 10 on an AMD Ryzen 5900x, when microsoft was messing with changes to suit Intel processors / room heaters.

I have now identified that a cause of my problem is that equation.com’s change to use of pthread-w32 for Gfortran can not handle the large numbers of !$OMP BARRIER access.

I have developed a “minimal workable example” which on my Windows 10 shows in Task Manager that the number Handles explodes to over 16 million; then the program crashes.

I am wondering if others can test this attached Gfortran OpenMP program, which runs on Windows using pthread-w32. Perhaps others could test if runs to completion on other OS/thread management to see if a this problem is more extensive that equation.com’s implementation using pthreads.

To rum, the following may need to be changed:

1. integer :: threads_to_use = 10 ! ( I have a processor configured to 12 threads )
2. integer*8 function HighResTimer_tick () ! I use winapi QUERYPERFORMANCECOUNTER (rate=10^7)
   any SYSTEM_CLOCK with a rate > 10^6 could suffice
3. The program multiplies 2 matrices, using less than70 MBytes and runs for about 4 minutes for a sucessful test, or less if it fails ! (not really testing L3 cache issue)

I am listing the test program and batch file to run the test.
I use another batch file to select which version of Gfortran to use( I have 16 versions from 7.1 to 17 )
I monitor the CPU usage handles on Task Manager, while minimal other information is available.
Success is the program reports “##### Test completed ####” while failure is a different exit !
The program is:

!  program to test effect of OMP BARRIER on OMP_MATMUL
!  This program works with equation.com Gfortran 11.1.0 and earlier but fails with subsequent versions that use pthread-w32

!  If versions earlier than 11.1.0 did not have the bug, that should be a bug of "pthread-w32," not a gfortran bug. 
!  Jenn-Ching Luo <@equation.com>

 program omp_speed_test
    implicit none

    integer, parameter :: si=4, sj=4, sk=4
    real*8, allocatable :: A(:,:), B(:,:), C(:,:), D(:,:)
    real*8    :: cl(si), loop_fac

    integer   :: pass, N,N1,N2,NS, loop, M = 100

    integer   :: j, k, k1,k2,j1,j2,i1,i2

    real      :: start, finish, delay, ticks_per_sec, delay_t, delay_b
    integer*8, external :: HighResTimer_rate

    integer   :: threads_to_use = 10
    logical   :: use_barrier  = .true.
    integer   :: barrier_step = 4, pass_barrier
    integer*8 :: barrier_count, barrier_tick

    ticks_per_sec = HighResTimer_rate ()

    open ( unit=11, file='omp_test.log', position='append' )

    call report_compiler (11)
    call report_compiler (6)

    NS = sj*threads_to_use
    N1 = ( 1000 / NS ) * NS
    N2 = ( 1300 / NS ) * NS
    
!    do pass = 1,2
     pass = 1
     do N = N1,N2,NS
      write ( *,10) 'Pass = ',pass,' N = ',n,' Threads_to_use = ',threads_to_use
      write (11,10) 'Pass = ',pass,' N = ',n,' Threads_to_use = ',threads_to_use
   10 format ( / a,i0,a,i0,a,i0 )

   ! Initialize matrices
       start = elapse_sec ()
       allocate ( A(N,N), B(N,N), C(N,N), D(N,N) )
 
       call random_number (A)
       call random_number (B)
       C = 0
       D = 0
       finish = elapse_sec () - start
       write ( *,11) "initialise time: ", finish, " seconds"
       write (11,11) "initialise time: ", finish, " seconds"
 
   ! matmul
       start = elapse_sec ()
 !      D = matmul (A, B)         ! eliminate single thread calculation
       D = A + B
       finish = elapse_sec () - start
       write ( *,11) "int matmul time: ", finish, " seconds"
       write (11,11) "int matmul time: ", finish, " seconds"
 
   ! for sub-matrix size s, this could be achieved with
       C = 0.0
       loop_fac = 1.0 / dble(M)
       barrier_count = 0
       barrier_tick  = 0
       start = elapse_sec ()
       write (*,*) '... start'
 
     !$OMP  PARALLEL DO    &
     !$OMP&  SHARED ( A,B,C,N,M, loop_fac, barrier_step, use_barrier )    &
     !$OMP&  PRIVATE ( j1,j2,j,k1,k2,k,i1,i2, cl, loop, pass_barrier )    &
     !$OMP&  num_threads (threads_to_use)                 &
     !$OMP&  REDUCTION(+ : barrier_count, barrier_tick)   &
     !$OMP&  SCHEDULE (DYNAMIC)
 
       do j1 = 1,N,sj
         j2 = min (j1+sj-1,N)
         pass_barrier = 0
         do loop = 1,M     ! repeat calculation of j1:j2 column block with all threads with barrier start
 
!           if ( mod(loop, barrier_step) == 1 ) then
 
           do k1 = 1,N,sk
             k2 = min (k1+sk-1,N)

             pass_barrier = pass_barrier +1
             if ( mod(pass_barrier, barrier_step) == 1 ) then
              call test_barrier ( use_barrier, barrier_count, barrier_tick )
             end if

             do i1 = 1,N,si
               i2 = min(i1+si-1,N)
               do j = j1,j2
                 cl = C(i1:i2,j) 
                 do k = k1, k2
                   cl = cl + A(i1:i1+si-1,k) * B(k,j)
                 end do
                 C(i1:i2,j) = cl*loop_fac
               end do
             end do
           end do
         end do
 
       end do
     !$OMP END PARALLEL DO
 
       finish = elapse_sec () - start
       delay  = real(barrier_tick) / ticks_per_sec 
       delay_t = delay / real (threads_to_use)
       delay_b = delay / real (barrier_count)
       write ( *,11) " sub3 loops time: ",finish, " seconds : err =", max_err ( c, d ), si, sj, sk
       write (11,11) " sub3 loops time: ",finish, " seconds : err =", max_err ( c, d ), si, sj, sk
       write ( *,12) barrier_count, barrier_tick, delay_t, delay_b
       write (11,12) barrier_count, barrier_tick, delay_t, delay_b
    11 FORMAT ( a,F10.5, a,ES12.4, 4I4)
    12 FORMAT ( "  barriers used   ",i0," : ticks ", i0,f10.5," sec / thread ",es11.3," sec / barrier")
 
       deallocate ( A, B, C, D )

       close ( unit=11 )
       open ( unit=11, file='omp_test.log', position='append' )

     end do
     write ( *,*) "Pass",pass," completed"
     write (11,*) "Pass",pass," completed"
!    end do

    write ( *,19)
    write (11,19)
 19 format ( / "##### Test completed ####" )

  contains

    real function elapse_sec ()
    integer*8 :: tick, rate, last = -1
    real      :: sec
    call system_clock ( tick, rate)
     if ( last < 0 ) last = tick
     sec = dble (tick-last) / dble (rate)
     last = tick
     elapse_sec = sec
  end function elapse_sec

  real*8 function max_err ( c, d )
  real*8 :: c(:,:), d(:,:)
  integer :: i, j
  real*8 x
    x = 0
    do i = 1,size(c,2)
      do j = 1,size(c,1)
        x = max ( x, abs(c(j,i)-d(j,i)) )
      end do
    end do
    max_err = x
  end function max_err
  
  end program omp_speed_test

   subroutine test_barrier ( use_barrier, barrier_count, barrier_tick )
      LOGICAL   use_barrier
      INTEGER*8 barrier_count, barrier_tick, ticks
      integer*8, external :: HighResTimer_tick
!
       if ( use_barrier ) then
         ticks = HighResTimer_tick ()
!$OMP BARRIER
         ticks = HighResTimer_tick () - ticks
         barrier_tick  = barrier_tick  + ticks
         barrier_count = barrier_count + 1
       else
         ticks = HighResTimer_tick ()
         ticks = HighResTimer_tick () - ticks
         barrier_tick  = barrier_tick  + ticks
         barrier_count = barrier_count + 1
       end if

   end subroutine test_barrier

    integer*8 function HighResTimer_tick ()
      use ISO_C_BINDING
      
      interface
        function QUERYPERFORMANCECOUNTER(tick) bind(C, name="QueryPerformanceCounter")
          use ISO_C_BINDING
          !GCC$ ATTRIBUTES STDCALL :: QUERYPERFORMANCECOUNTER
          logical(C_BOOL) QUERYPERFORMANCECOUNTER
          integer(C_LONG_LONG) tick
        end function QUERYPERFORMANCECOUNTER
      end interface
!
      integer(C_LONG_LONG) :: tick
      logical(C_BOOL)      :: ll
!
      ll    = QUERYPERFORMANCECOUNTER (tick)
      HighResTimer_tick = tick
    end function HighResTimer_tick

    integer*8 function HighResTimer_rate ()
      use ISO_C_BINDING

      interface
        function QUERYPERFORMANCEFREQUENCY(tick_rate) bind(C, name="QueryPerformanceFrequency") 
          use ISO_C_BINDING
          !GCC$ ATTRIBUTES STDCALL :: QUERYPERFORMANCEFREQUENCY
          logical(C_BOOL) QUERYPERFORMANCEFREQUENCY
          integer(C_LONG_LONG) tick_rate
        end function QUERYPERFORMANCEFREQUENCY
      end interface
!
      logical(C_BOOL)      :: ll
      integer(C_LONG_LONG) :: tick_rate = -1
!
      if ( tick_rate < 0 ) then
        ll    = QUERYPERFORMANCEFREQUENCY (tick_rate)
        write (*,*) 'HighResTimer has', tick_rate,' ticks per second'
      end if
      HighResTimer_rate = tick_rate
    end function HighResTimer_rate

   subroutine report_compiler (lu)
    use iso_fortran_env
     implicit none
     integer :: lu
     integer :: ia(8)
     character*3 :: label(12) = [ 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun','Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec' ]

      call Date_and_Time (VALUES=ia)

!      hour = ia(5)
!      am   = ' am'
!      IF (hour >= 12) am = ' pm'
!      IF (hour >  12) hour = hour-12
!      IF (hour ==  0) hour = 12
!
      write (lu,11) 'Date : ', ia(5), ia(6), ia(7),  ia(3), label(ia(2)), ia(1)
      write (lu,10) 'Vern : ', compiler_version ()
      write (lu,10) 'Opts : ', compiler_options ()  
   11 format (a8,I2,':',I2.2,':',I2.2, ' on ', I2,'_',A3,'_',I4 /1x)
   10 format (a8,a)

   end subroutine report_compiler

My windows batch test is:

set program=%1

del %program%.exe

set basic=-g -fimplicit-none -O2 -march=native -fopenmp
set vec=-g -fimplicit-none -O3 -march=native -ffast-math -fopenmp -fstack-arrays

gfortran -v %program%.f90 %basic% -o %program%.exe

%program%

notepad omp_test.log

Typical output is:

 Date : 15:59:30 on 13_Aug_2026

 Vern : GCC version 14.2.0
 Opts : -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512vbmi -mno-avx512ifma -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mno-pku -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni -mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex -mno-avxvnniint16 -mno-sm3 -mno-sha512 -mno-sm4 -mno-apxf -mno-usermsr --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=512 -mtune=znver3 -mthreads -g -O2 -fimplicit-none -fopenmp

Pass = 1 N = 1000 Threads_to_use = 10
initialise time:    0.01166 seconds
int matmul time:    0.00082 seconds
 sub3 loops time:    5.25245 seconds : err =  1.9908E+00   4   4   4
  barriers used   1562500 : ticks 362272084   3.62272 sec / thread   2.319E-05 sec / barrier

Pass = 1 N = 1040 Threads_to_use = 10
initialise time:   -0.00011 seconds
int matmul time:    0.00096 seconds
 sub3 loops time:    5.67820 seconds : err =  1.9963E+00   4   4   4
  barriers used   1690000 : ticks 383856581   3.83857 sec / thread   2.271E-05 sec / barrier
 Date : 16:01:19 on 13_Aug_2026

 Vern : GCC version 15.1.0
 Opts : -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512vbmi -mno-avx512ifma -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mno-pku -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni -mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex -mno-avxvnniint16 -mno-sm3 -mno-sha512 -mno-sm4 -mno-apxf -mno-usermsr -mno-avx10.2 -mno-amx-avx512 -mno-amx-tf32 -mno-amx-transpose -mno-amx-fp8 -mno-movrs -mno-amx-movrs --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=512 -mtune=znver3 -mthreads -g -O2 -fimplicit-none -fopenmp

Pass = 1 N = 1000 Threads_to_use = 10
initialise time:    0.01105 seconds
int matmul time:    0.00071 seconds
 sub3 loops time:    5.17701 seconds : err =  1.9924E+00   4   4   4
  barriers used   1562500 : ticks 351932019   3.51932 sec / thread   2.252E-05 sec / barrier

Pass = 1 N = 1040 Threads_to_use = 10
initialise time:    0.00114 seconds
int matmul time:    0.00095 seconds
 sub3 loops time:    5.60472 seconds : err =  1.9937E+00   4   4   4
  barriers used   1690000 : ticks 373581438   3.73581 sec / thread   2.211E-05 sec / barrier
 Date : 16:02:12 on 13_Aug_2026

 Vern : GCC version 16.1.0
 Opts : -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512vbmi -mno-avx512ifma -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mno-pku -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni -mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex -mno-avxvnniint16 -mno-sm3 -mno-sha512 -mno-sm4 -mno-apxf -mno-usermsr -mno-avx10.1 -mno-avx10.2 -mno-amx-avx512 -mno-amx-tf32 -mno-amx-fp8 -mno-movrs -mno-amx-movrs -mno-avx512bmm --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=512 -mtune=znver3 -mthreads -g -O2 -fimplicit-none -fopenmp

Pass = 1 N = 1000 Threads_to_use = 10
initialise time:    0.01130 seconds
int matmul time:    0.00080 seconds
 sub3 loops time:    5.22905 seconds : err =  1.9936E+00   4   4   4
  barriers used   1562500 : ticks 360070759   3.60071 sec / thread   2.304E-05 sec / barrier

Pass = 1 N = 1040 Threads_to_use = 10
initialise time:   -0.00067 seconds
int matmul time:    0.00105 seconds
 sub3 loops time:    5.66370 seconds : err =  1.9947E+00   4   4   4
  barriers used   1690000 : ticks 382216620   3.82217 sec / thread   2.262E-05 sec / barrier
 Date : 16:02:48 on 13_Aug_2026

 Vern : GCC version 17.0.0 20260719 (experimental)
 Opts : -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512vbmi -mno-avx512ifma -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mno-pku -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni -mno-avx512fp16 -mno-avxifma -mno-avxvnniint8 -mno-avxneconvert -mno-cmpccxadd -mno-amx-fp16 -mno-prefetchi -mno-raoint -mno-amx-complex -mno-avxvnniint16 -mno-sm3 -mno-sha512 -mno-sm4 -mno-apxf -mno-usermsr -mno-avx10.1 -mno-avx10.2 -mno-amx-avx512 -mno-amx-fp8 -mno-movrs -mno-amx-movrs -mno-avx512bmm --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=512 -mtune=znver3 -mthreads -g -O2 -fimplicit-none -fopenmp

Pass = 1 N = 1000 Threads_to_use = 10
initialise time:    0.01109 seconds
int matmul time:    0.00071 seconds
 sub3 loops time:    5.34333 seconds : err =  1.9918E+00   4   4   4
  barriers used   1562500 : ticks 375211537   3.75212 sec / thread   2.401E-05 sec / barrier

Pass = 1 N = 1040 Threads_to_use = 10
initialise time:   -0.00053 seconds
int matmul time:    0.00094 seconds
 sub3 loops time:    5.78664 seconds : err =  1.9940E+00   4   4   4
  barriers used   1690000 : ticks 399258873   3.99259 sec / thread   2.362E-05 sec / barrier
 Date : 16:03:19 on 13_Aug_2026

 Vern : GCC version 11.1.0
 Opts : -march=znver3 -mmmx -mpopcnt -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mavx -mavx2 -msse4a -mno-fma4 -mno-xop -mfma -mno-avx512f -mbmi -mbmi2 -maes -mpclmul -mno-avx512vl -mno-avx512bw -mno-avx512dq -mno-avx512cd -mno-avx512er -mno-avx512pf -mno-avx512vbmi -mno-avx512ifma -mno-avx5124vnniw -mno-avx5124fmaps -mno-avx512vpopcntdq -mno-avx512vbmi2 -mno-gfni -mvpclmulqdq -mno-avx512vnni -mno-avx512bitalg -mno-avx512bf16 -mno-avx512vp2intersect -mno-3dnow -madx -mabm -mno-cldemote -mclflushopt -mclwb -mclzero -mcx16 -mno-enqcmd -mf16c -mfsgsbase -mfxsr -mno-hle -msahf -mno-lwp -mlzcnt -mmovbe -mno-movdir64b -mno-movdiri -mmwaitx -mno-pconfig -mno-pku -mno-prefetchwt1 -mprfchw -mno-ptwrite -mrdpid -mrdrnd -mrdseed -mno-rtm -mno-serialize -mno-sgx -msha -mshstk -mno-tbm -mno-tsxldtrk -mvaes -mno-waitpkg -mwbnoinvd -mxsave -mxsavec -mxsaveopt -mxsaves -mno-amx-tile -mno-amx-int8 -mno-amx-bf16 -mno-uintr -mno-hreset -mno-kl -mno-widekl -mno-avxvnni --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=512 -mtune=znver3 -mthreads -g -O2 -fimplicit-none -fopenmp

Pass = 1 N = 1000 Threads_to_use = 10
initialise time:    0.01100 seconds
int matmul time:    0.00075 seconds
 sub3 loops time:    9.19393 seconds : err =  1.9920E+00   4   4   4
  barriers used   1562500 : ticks 319735470   3.19735 sec / thread   2.046E-05 sec / barrier

Pass = 1 N = 1040 Threads_to_use = 10
initialise time:    0.00033 seconds
int matmul time:    0.00090 seconds
 sub3 loops time:   10.19006 seconds : err =  1.9948E+00   4   4   4
  barriers used   1690000 : ticks 344237948   3.44238 sec / thread   2.037E-05 sec / barrier

Pass = 1 N = 1080 Threads_to_use = 10
initialise time:    0.00301 seconds
int matmul time:    0.00101 seconds
 sub3 loops time:   11.21647 seconds : err =  1.9935E+00   4   4   4
  barriers used   1822500 : ticks 373678068   3.73678 sec / thread   2.050E-05 sec / barrier

Pass = 1 N = 1120 Threads_to_use = 10
initialise time:    0.00229 seconds
int matmul time:    0.00107 seconds
 sub3 loops time:   12.41673 seconds : err =  1.9937E+00   4   4   4
  barriers used   1960000 : ticks 406578270   4.06578 sec / thread   2.074E-05 sec / barrier

Pass = 1 N = 1160 Threads_to_use = 10
initialise time:    0.00416 seconds
int matmul time:    0.00113 seconds
 sub3 loops time:   13.75136 seconds : err =  1.9941E+00   4   4   4
  barriers used   2102500 : ticks 451013781   4.51014 sec / thread   2.145E-05 sec / barrier

Pass = 1 N = 1200 Threads_to_use = 10
initialise time:    0.00498 seconds
int matmul time:    0.00120 seconds
 sub3 loops time:   14.82806 seconds : err =  1.9959E+00   4   4   4
  barriers used   2250000 : ticks 463302533   4.63303 sec / thread   2.059E-05 sec / barrier

Pass = 1 N = 1240 Threads_to_use = 10
initialise time:    0.00286 seconds
int matmul time:    0.00120 seconds
 sub3 loops time:   16.33693 seconds : err =  1.9970E+00   4   4   4
  barriers used   2402500 : ticks 509941473   5.09941 sec / thread   2.123E-05 sec / barrier

Pass = 1 N = 1280 Threads_to_use = 10
initialise time:    0.00637 seconds
int matmul time:    0.00138 seconds
 sub3 loops time:   18.11096 seconds : err =  1.9949E+00   4   4   4
  barriers used   2560000 : ticks 554212626   5.54213 sec / thread   2.165E-05 sec / barrier
 Pass           1  completed

##### Test completed ####

If anyone is able to confirm my findings, or find a problem with my approach it would be appreciated.

The test is basically a multi-threaded matrix multiplier with (an extreme number of) embedded !$OMP BARRIER placed to align the L3 cache (which is not required for this small memory test!)