# Fortran projects running on GPUs in production **URL:** \ **Category:** Help\ **Created:** [April 4, 2022, 12:02am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118 "2022-04-04T00:02:05Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![fortran4r](https://avatars.discourse-cdn.com/v4/letter/f/9de0a6/32.png) [@fortran4r](https://fortran-lang.discourse.group/u/fortran4r)\ **Post date:** [April 4, 2022, 12:02am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/1 "2022-04-04T00:02:05Z")
Are there any projects using Fortran with GPU in production? ---
**Author:** ![CRquantum](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/crquantum/32/730_2.png) [@CRquantum](https://fortran-lang.discourse.group/u/CRquantum)\ **Post date:** [April 4, 2022, 3:03am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/2 "2022-04-04T03:03:38Z")
There must be a way. At least for Intel, as their OneAPI name suggested and their firm willingness into GPU business. By the way, I am pretty sure they never give up GPU, and as GPU computation catches more and more attentions considering its potential in doing big matrix operations such as solving some partial differential equations, fluid dynamics, heat transfer, imaging processing (convolution matrix), cellular automata, etc. Those stuff previously are suitable for Xeon Phi (slower but have many threads and high bandwidth memory), now Xeon Phi is basically replaced by GPUs. In short, I see no reason why Intel Fortran cannot offload openMP or something from Intel CPU to Intel GPU, they have some links as below, [https://www.intel.com/content/www/us/en/develop/documentation/get-started-with-cpp-fortran-compiler-openmp/top.html](https://www.intel.com/content/www/us/en/develop/documentation/get-started-with-cpp-fortran-compiler-openmp/top.html) [https://www.intel.com/content/www/us/en/developer/videos/three-quick-practical-examples-openmp-offload-gpus.html#gs.vnh4qj](https://www.intel.com/content/www/us/en/developer/videos/three-quick-practical-examples-openmp-offload-gpus.html#gs.vnh4qj) I have a similar thread, but I am not very sure how to really offload to Intel GPU, I think I am almost there but still need one more step to make it work. In fact the examples are all in OneAPI’s example folder, if their instruction on how to offload openMP to GPU a little clear (especially on Windows) it will be great, > [@How to use IFX and offload openMP to GPU?](https://fortran-lang.discourse.group/t/how-to-use-ifx-and-offload-openmp-to-gpu/3104): > > Dear all, I noticed in some cases perhaps Julia may be faster as @certik showed below, [https://fortran-lang.discourse.group/t/simple-summation-8x-slower-than-in-julia/1171?u=crquantum](https://fortran-lang.discourse.group/t/simple-summation-8x-slower-than-in-julia/1171) I also noticed @mohoree said Intel MKL’s routines can perform like 5x faster than not using MKL, [https://fortran-lang.discourse.group/t/simple-summation-8x-slower-than-in-julia/1171/98?u=crquantum](https://fortran-lang.discourse.group/t/simple-summation-8x-slower-than-in-julia/1171/98) I also notice @gnikit mentioned MKL too, [https://fortran-lang.discourse.group/t/equivalent-gfort-and-ifort-compi…](https://fortran-lang.discourse.group/t/equivalent-gfort-and-ifort-compilation/3092/5) I think gfortran or Lfortran can also offload openMP to GPU, if not now, in the near future it should be possible. But I guess, if the application needs to frequently exchange data from CPU through memory to GPU, then the speedup by GPU will be ruined by the slow memory speed (bandwidth). For example, a typical DDR4 2666 memory’s bandwidth is just about 30 GB/s. GPU’s GDDR5 or something could be 10x faster. In fact, I suspect the reason that apple’s M1 chip can achieve very high performance, one importance reason is that the memory on M1 chip Mac has quite high bandwidth, like 200 - 400 GB/s if I remember correctly. That is almost the same speed as L2 or L3 cache (but cache should have at least 10x shorter latency than memory). That also explains why M1 Mac is very good image/video processing, as those applications typically will be benefit from high speed memory (images are just some big arrays in the memory). ---
**Author:** ![shahmoradi](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/shahmoradi/32/3151_2.png) [@shahmoradi](https://fortran-lang.discourse.group/u/shahmoradi)\ **Post date:** [April 4, 2022, 5:03pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/3 "2022-04-04T17:03:21Z")
[Amber Molecular Dynamics](https://ambermd.org/index.php) is one of the most popular Molecular Dynamics packages whose kernel is modern Fortran with GPU support. ---
**Author:** ![snano](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@snano](https://fortran-lang.discourse.group/u/snano)\ **Post date:** [April 5, 2022, 6:25am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/4 "2022-04-05T06:25:38Z")
> Quantum ESPRESSO toward the exascale > Abstract: Quantum ESPRESSO is an open-source distribution of computer codes for quantum-mechanical materials modeling, based on density-functional theory, pseudopotentials, and plane waves, and renowned for its performance on a wide range of hardware architectures, from laptops to massively parallel computers, as well as for the breadth of its applications. In this paper, we present a motivation and brief review of the ongoing effort to port Quantum ESPRESSO onto heterogeneous architectures based on hardware accelerators, which will overcome the energy constraints that are currently hindering the way toward exascale computing. > Section [III C](https://aip.scitation.org/doi/10.1063/5.0005082#s3C) describes the current status, capabilities, and some benchmarks for Quantum ESPRESSO on NVIDIA graphics processing units (GPUs), one of the leading candidate architectures for the future “exascale computer.” The benchmarks are designed for relatively small machines and do not aim at showing performances on large-sized systems. They aim instead at pointing out bottlenecks, inefficiencies, and the minimum size of calculations that saturate the computational power of a GPU. Taken from the (reference: J. Chem. Phys. **152** , 154105 (2020). Mostly, I would say 99% of the codebase is Fortran90 or Fortran200x. For more details, see the main website: [https://www.quantum-espresso.org/](https://www.quantum-espresso.org/) ---
**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\ **Post date:** [April 5, 2022, 3:25pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/5 "2022-04-05T15:25:13Z")
IBM’s [GRAF](https://www.ibm.com/weather/industries/cross-industry/graf) is a proprietary fork of [MPAS](https://mpas-dev.github.io/atmosphere/atmosphere.html) that runs on GPUs. I heard that it uses OpenACC directives but I don’t know this for a fact. ---
**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\ **Post date:** [April 5, 2022, 3:30pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/6 "2022-04-05T15:30:05Z")
Peter Ukkonen runs a [fork of neural-fortran](https://github.com/peterukk/rte-rrtmgp-nn) on GPUs using OpenACC directives and delegating `matmul` calls to cuBLAS ([Paper](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021MS002875)). This is academic research, so probably not what you’d call true production, but these projects often have the end-goal of being used in production a few years down the road. ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [April 5, 2022, 11:00pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/7 "2022-04-05T23:00:50Z")
Here is a list of HPC codes that all run on GPU: [https://www.nvidia.com/content/intersect-360-HPC-application-support.pdf](https://www.nvidia.com/content/intersect-360-HPC-application-support.pdf), most of the top 10 codes are in Fortran. ---
**Author:** ![gnikit](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gnikit/32/1213_2.png) [@gnikit](https://fortran-lang.discourse.group/u/gnikit)\ **Post date:** [April 5, 2022, 11:15pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/8 "2022-04-05T23:15:29Z")
I don’t think this qualifies as a Fortran project, but PETSc which you can call from Fortran is adding/has added [GPU support](https://petsc.org/release/overview/gpu_roadmap) ---
**Author:** ![gak](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gak/32/146_2.png) [@gak](https://fortran-lang.discourse.group/u/gak)\ **Post date:** [April 8, 2022, 2:11am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/9 "2022-04-08T02:11:10Z")
Not a direct pointer, but many of the applications that use OpenACC for GPU acceleration are in Fortran as well. When I was last an NVIDIA, there were over 100 GPU-accelerated production apps with OpenACC. Most of those were primarily in Fortran, but I can’t lay my hands on a definitive list at the moment. ---
**Author:** ![septc](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/septc/32/77_2.png) [@septc](https://fortran-lang.discourse.group/u/septc)\ **Post date:** [April 8, 2022, 5:59am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/10 "2022-04-08T05:59:40Z")
I am also interested in OpenACC, which I got to know from a recent tutorial seminar and seems relatively simple to use. VASP (an electronic structure program) also seems to support GPU via OpenACC. > **[NVIDIA GPU Accelerated VASP 6 uses OpenACC to Deliver 15X More Performance |...](https://developer.nvidia.com/blog/nvidia-gpu-accelerated-vasp-6-uses-openacc-to-deliver-15x-more-performance/)** > > Developers of the world’s leading HPC application for atomic scale modelling, Vienna Ab initio Simulation Package (VASP), rolled out VASP 6.1.0 which ports new and expanded acceleration in NVIDIA GPUs… - [VASP Electronic Structure Prediction Application and GPU Acceleration | NVIDIA](https://www.nvidia.com/en-sg/data-center/gpu-accelerated-applications/vasp/) - [OpenACC GPU port of VASP - Vaspwiki](https://www.vasp.at/wiki/index.php/OpenACC_GPU_port_of_VASP) ---
**Author:** ![rwmsu](https://avatars.discourse-cdn.com/v4/letter/r/48db29/32.png) [@rwmsu](https://fortran-lang.discourse.group/u/rwmsu)\ **Post date:** [April 8, 2022, 1:32pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/11 "2022-04-08T13:32:24Z")
I think both the Abaqus and LS-DYNA FEM codes have some level of GPU support and both are (mostly) Fortran. ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [April 10, 2022, 6:10pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/12 "2022-04-10T18:10:40Z")
> [@gak](#): > > but I can’t lay my hands on a definitive list at the moment. I think the link I [posted](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/7) above might be exactly the list that you have in mind. ---
**Author:** ![gak](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gak/32/146_2.png) [@gak](https://fortran-lang.discourse.group/u/gak)\ **Post date:** [April 10, 2022, 6:27pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/13 "2022-04-10T18:27:05Z")
Yours was from November 2017. 🙂 I was thinking of an OpenACC list that was more recent. I thought they had a definitive OpenACC list on the [openacc.org](http://openacc.org) web site, but didn’t see it. ---
**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\ **Post date:** [April 10, 2022, 8:38pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/14 "2022-04-10T20:38:16Z")
If you find a more recent list, please share it! It’s hard to find these documents, but they are very helpful to see the status of Fortran. I did a big search around 2017, that document was the best I was able to find. ---
**Author:** ![pcosta](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/pcosta/32/5504_2.png) [@pcosta](https://fortran-lang.discourse.group/u/pcosta)\ **Post date:** [June 6, 2023, 3:16pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/15 "2023-06-06T15:16:21Z")
[CaNS](https://github.com/CaNS-World/CaNS) runs on many GPUs powered by OpenACC directives and an NVIDIA pencil domain decomposition library. ---
**Author:** ![Beliavsky](https://avatars.discourse-cdn.com/v4/letter/b/ba8739/32.png) [@Beliavsky](https://fortran-lang.discourse.group/u/Beliavsky)\ **Post date:** [June 6, 2023, 4:10pm UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/16 "2023-06-06T16:10:28Z")
A case-insensitive grep (actually Windows findstr) of my [list of Fortran codes on GitHub](https://github.com/Beliavsky/Fortran-code-on-GitHub) for “gpu” gives the following. If my description of a project does not include “gpu”, as it did not for CaNS, the project will be missed. [Nbody6++GPU - Beijing version](https://github.com/nbody6ppgpu/Nbody6PPGPU-beijing): N-body star cluster simulation code, by Rainer Spurzem and team. It is an offspring of Sverre Aarseth’s direct N-body [codes](http://www.sverre.com/). [POT3D: High Performance Potential Field Solver](https://github.com/predsci/POT3D): computes potential field solutions to approximate the solar coronal magnetic field using observed photospheric magnetic fields as a boundary condition. A version of POT3D that includes GPU-acceleration with both MPI+OpenACC and MPI+OpenMP was released as part of the Standard Performance Evaluation Corporation’s (SPEC) beta version of the SPEChpc™ 2021 benchmark suites. [Fluid Transport Accelerated Solver (FluTAS)](https://github.com/Multiphysics-Flow-Solvers/FluTAS): modular, multiphysics code for multiphase fluid dynamics simulations. The code is written following a “functional programming” approach and aims to accommodate various independent modules. One of the main purposes of the project is to provide an efficient framework able to run both on many-CPUs (MPI) and many-GPUs (MPI+OpenACC+CUDA-Fortran). [IMEXLB-1.0](https://github.com/lucaso19891019/IMEXLB-1.0): Lattice Boltzmann Method (LBM) proxy application code-suite for heterogeneous platforms (such as ThetaGPU). A ProxyApp, by definition, is a proxy for a full-fledged application code that simulates a wider array of problems. [MGLC](https://github.com/cheryli/MGLC): multi-GPU parallel implementation of LBM(Lattice Boltzmann Method), using OpenACC to accelerate codes on single GPU and MPI for inter-GPU communication [fft-overlap](https://github.com/dappelha/fft-overlap): efficient implementations of ffts on multiple GPUs and across multiple nodes, by dappelha. Overlapping data transfer on multiple levels. [arrayfire-fortran](https://github.com/arrayfire/arrayfire-fortran): Fortran wrapper for ArrayFire, a general purpose GPU library. [Eigensolver\_gpu](https://github.com/NVIDIA/Eigensolver_gpu): generalized eigensolver for symmetric/hermetian-definite eigenproblems with functionality similar to the DSYGVD/X or ZHEGVD/X functions available within LAPACK/MAGMA, by Josh Romero et al. This solver has less dependencies on CPU computation than comparable implementations within MAGMA, which may be of benefit to systems with limited CPU resources or to users without access to high-performing CPU LAPACK libraries. [GraSPH](https://github.com/edoyango/GraSPH): Smoothed-particle Hydrodynamics (SPH) program originally intended for simulations of bulk granular material as well as fluids, by Edward Yang. Src\_CAF contains code intended to run multi-core configuration enabled with the Coarray Fortran 2008 features, and src\_GPU contains code intended to run on a CUDA-enabled GPU. [CUDA Fortran: Fortran programming on GPU](https://github.com/Koushikphy/Intro-to-CUDA-Fortran): a complete introduction for beginners by Koushik Naskar [ExaTENSOR](https://github.com/ORNL-QCI/ExaTENSOR): basic numerical tensor algebra library for distributed HPC systems equipped with multicore CPU and NVIDIA (or AMD) GPU, by Dmitry I. Lyakh. The hierarchical task-based parallel runtime of ExaTENSOR is based on the virtual tensor algebra processor architecture, i.e. a software processor specialized to numerical tensor algebra workloads on heterogeneous HPC systems (multicore/KNL, NVIDIA or AMD GPU). [FGPU](https://github.com/LLNL/FGPU): code examples focusing on porting FORTRAN codes to run DOE heterogenous architecture CPU+GPU machines, from LLNL. The purpose of these is to provide both learning aids for developers and OpenMP and CUDA code examples for testing vendor compilers capabilities. [GPU programming with OpenMP offloading](https://github.com/csc-training/openmp-offload): exercises and other material for [course](https://events.prace-ri.eu/event/1273/), by Jussi Enkovaara et al. [gpu-tips](https://github.com/dappelha/gpu-tips): Fortran examples of CUDA and directives tips and tricks for IBM Power + Nvidia Systems, by dappelha [nbody-ifx-do-concurrent](https://github.com/sarojadhikari/nbody-ifx-do-concurrent): N-body Fortran code port to test ifx (Intel Fortran) GPU offload of `do concurrent`, by Saroj Adhikari [Tensor Algebra Library Routines for Shared Memory Systems: Nodes equipped with multicore CPU, NVIDIA GPU, AMD GPU, and Intel Xeon Phi (TAL\_SH)](https://github.com/DmitryLyakh/TAL_SH): implements basic tensor algebra operations with interfaces to C, C++11, and Fortran 90+, by Dmitry I. Lyakh [BDpack](https://github.com/amir-saadat/BDpack): GPU-enabled Brownian dynamics package for simulation of polymeric solutions, by Amir Saadat. An associated paper is [Computationally efficient algorithms for incorporation of hydrodynamic and excluded volume interactions in Brownian dynamics simulations: A comparative study of the Krylov subspace and Chebyshev based techniques](https://pubmed.ncbi.nlm.nih.gov/24832302/), A. Saadat and B. Khomami, J. Chem. Phys., 140, 184903 (2014). [A Fortran Electronic Structure Programme (AFESP)](https://github.com/brianz98/A-Fortran-Electronic-Structure-Programme): project based on the [Crawford Group’s C++ Programming Tutorial in Chemistry](https://github.com/CrawfordGroup/ProgrammingProjects), but written in Fortran, by Kirk Pearce et al. The end goal of this project will be performing HF, MP2, CCSD, and CCSD(T), as per the original tutorial, but with additional support for multicore processors (modern CPUs, GPUs). [qe-gpu](https://github.com/fspiga/qe-gpu): GPU-accelerated Quantum ESPRESSO using CUDA Fortran, by Filippo Spiga ---
**Author:** ![RobertPincus](https://avatars.discourse-cdn.com/v4/letter/r/a183cd/32.png) [@RobertPincus](https://fortran-lang.discourse.group/u/RobertPincus)\ **Post date:** [June 7, 2023, 2:41am UTC](https://fortran-lang.discourse.group/t/fortran-projects-running-on-gpus-in-production/3118/17 "2023-06-07T02:41:03Z")
[ICON](https://code.mpimet.mpg.de/projects/iconpublic), the global model developed in Germany to simulate weather and climate, is [used on GPUs](https://doi.org/10.5194/gmd-15-6985-2022).