Do concurrent + NVIDIA GPU offload: does it support external procedures?

Help
1

I’ve been experimenting with do concurrent and NVIDIA’s automatic GPU offload (nvfortran -stdpar=gpu), and I’ve run into a pattern that makes me think external module procedures called from a do concurrent construct aren’t currently supported for offload. I’d like to check with the community whether this matches your understanding, or whether I’m missing a flag/attribute somewhere.

Here’s what I tested, going from “definitely works” to “breaks”:

Case 1 — Internal (contained) pure subroutine, single file

Everything — the main program and a pure subroutine — lives in one file, with the subroutine in a contains block of the program.

program random_2d_example
    implicit none
    
    ! Declare variables
    integer, parameter :: n_rows = 4
    integer, parameter :: n_cols = 4
    integer :: i, j
    real, dimension(n_rows, n_cols) :: r
    real, dimension(n_rows, n_cols) :: c, p
    
    ! Initialize random number generator array
    call random_number(r)
    
    ! Generate values and process them through the pure routine
	do concurrent (i=1:n_cols,j=1:n_rows) 
            call initial_micro(r(i, j), c(i, j))
            p(i, j) = c(i, j)
    end do

contains

    pure subroutine initial_micro(r_val, c_val)
        real, intent(in)  :: r_val
        real, intent(out) :: c_val

        c_val = r_val
    end subroutine initial_micro

end program random_2d_example

Result: compiles and offloads fine.

nvfortran -stdpar=gpu -gpu=cc61 -gpu=mem:managed -Minfo=accel test.f90 -o test
random_2d_example:
     15, Generating NVIDIA GPU code
         15,   ! blockidx%x threadidx%x auto-collapsed
             Loop parallelized across CUDA thread blocks, CUDA threads(32) collapse(2) ! blockidx%x threadidx%x collapsed-innermost
     15, Generating implicit copy(c(:,:)) [if not already present]
         Generating implicit copyin(r(:,:)) [if not already present]
         Generating implicit copyout(p(:,:)) [if not already present]
initial_micro:
     22, Generating implicit acc routine seq
         Generating acc routine seq
         Generating NVIDIA GPU code

Case 2 — pure subroutine inside an internal module, still single file

Same file, but now routine lives in a module (in the same source file).

! --- Microphysics Module ---
module micro_mod
    implicit none
    
    ! Keep public interfaces clean and structured
    private
    public :: initial_micro

contains

    pure subroutine initial_micro(r_val, c_val)
        real, intent(in)  :: r_val
        real, intent(out) :: c_val

        c_val = r_val
    end subroutine initial_micro

end module micro_mod


! --- Main Program ---
program random_2d_example
    use micro_mod, only: initial_micro
    implicit none
    
    integer, parameter :: n_rows = 5
    integer, parameter :: n_cols = 4
    integer :: i, j
    real, dimension(n_rows, n_cols) :: r
    real, dimension(n_rows, n_cols) :: c, p
    
    ! Initialize random numbers
    call random_number(r)
    
    ! Offload execution to GPU via do concurrent
    do concurrent (i = 1:n_rows, j = 1:n_cols) 
        call initial_micro(r(i, j), c(i, j))
        p(i, j) = c(i, j)
    end do

    print *, "Program executed successfully. First element p(1,1):", p(1, 1)

end program random_2d_example

Result: still compiles and offloads fine.

nvfortran -stdpar=gpu -gpu=cc61 -gpu=mem:managed -Minfo=accel test_module.f90 -o test_module
initial_micro:
     11, Generating implicit acc routine seq
         Generating acc routine seq
         Generating NVIDIA GPU code
random_2d_example:
     36, Generating NVIDIA GPU code
         36,   ! blockidx%x threadidx%x auto-collapsed
             Loop parallelized across CUDA thread blocks, CUDA threads(32) collapse(2) ! blockidx%x threadidx%x collapsed-innermost
     36, Generating implicit copy(c(:,:)) [if not already present]
         Generating implicit copyin(r(:,:)) [if not already present]
         Generating implicit copyout(p(:,:)) [if not already present]

Case 3 — Same module, but split into a separate source file

module micro_mod
    implicit none
    private
    public :: initial_micro

contains

    elemental subroutine initial_micro(r_val, c_val)
        real, intent(in)  :: r_val
        real, intent(out) :: c_val

        c_val = r_val
    end subroutine initial_micro

end module micro_mod

program random_2d_example
    use micro_mod, only: initial_micro
    implicit none
    
    integer, parameter :: n_rows = 5
    integer, parameter :: n_cols = 4
    integer :: i, j
    real, dimension(n_rows, n_cols) :: r
    real, dimension(n_rows, n_cols) :: c, p
    
    call random_number(r)
    
    do concurrent (i = 1:n_rows, j = 1:n_cols) 
        call initial_micro(r(i, j), c(i, j))
        p(i, j) = c(i, j)
    end do

    print *, "Program executed successfully. First element p(1,1):", p(1, 1)

end program random_2d_example

Result:

nvfortran -stdpar=gpu -gpu=cc61 -gpu=mem:managed -c micro_mod.f90
nvfortran -stdpar=gpu -gpu=cc61 -gpu=mem:managed -c main.f90
NVFORTRAN-S-1074-Procedure call in Do Concurrent is not supported yet (main.f90: 14)
 0 inform,   0 warnings,   1 severes, 0 fatal for random_2d_example

My question

Based on this, it looks like offloading a do concurrent loop that calls an external module procedure (i.e. a procedure compiled in a separate translation unit and pulled in via use) is not currently supported, even though the exact same procedure works fine when it’s defined internally to the program or in a module within the same file.

Is this a known/expected limitation of nvfortran’s -stdpar GPU offload today (e.g. related to whole-program analysis or LTO requirements for offload regions), or is there a way to make external procedures visible to the offload compiler (some combination of IPA/LTO flags, or explicit interface attributes) that I’m missing?

2

this is a limitation yes, you have to have them in the same translation unit or via an include statement.

3

"This is interesting — it’s also the first time I’ve come across the use of an include statement in this context.

nvfortran -stdpar=gpu -gpu=cc61 -gpu=mem:managed main.f90 -o main.out
./main.out
 Program executed successfully. First element p(1,1):   0.9079230

Single command: The other interesting part is that a single command is enough to build and run the whole program — no separate compilation step needed."

4

They are ugly but useful for exactly this application, in this you can see I make a lot of use of them.

5

The compilation is simple. Just one line. So it is very useful. Adding a statement at the top is not a big trouble. The good thing is that we do not need to rewrite the whole program.