# Performance portable LLM inference in Fortran (beating llama.cpp/vllm on H200)

**URL:** <https://fortran-lang.discourse.group/t/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200/11309>\
**Category:** LFortran\
**Created:** [November 16, 2026, 5:06am UTC](https://fortran-lang.discourse.group/t/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200/11309 "2026-11-16T05:06:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\
**Post date:** [November 16, 2026, 5:06am UTC](https://fortran-lang.discourse.group/t/performance-portable-llm-inference-in-fortran-beating-llama-cpp-vllm-on-h200/11309/1 "2026-11-16T05:06:51Z")

</div>

I think Fortran might be a very solid language for AI, but it has not been used much in it yet. I think it has a lot of potential, because it has arrays, fast numerics, it’s simple, parallelism in the language (both on-node and GPU via do concurrent as well as across-nodes via coarrays), has high-performance compilers and Fortran has always been used to run fast on any platform.

So let’s see how it can work. Let’s take one of the best latest local models, I picked Qwen3.8-27B-8bit (released 3 months ago), but any other recent model can be used also.

Let’s write the inference for this model in Fortran: [GitHub - certik/qwen3.8-27B: Qwen3.8-27B (MLX 8-bit) inference: C reference, Fortran (LFortran GPU) and CUDA · GitHub](https://github.com/certik/qwen3.8-27B).

Let’s use do concurrent for GPU offloading, but no other directive/pragmas (so no OpenACC, no OpenMP, nothing), just pure Fortran, as it’s meant to be used.

Let’s start with H200 from NVIDIA. On the machine that I have access to, the inference is limited by the available bandwidth to about 170 tok/s based on theoretical bandwidth, but based on measured memory speed more like 150 tok/s. Let’s write our own C + CUDA version (available at the link above), optimize it. Then let’s benchmark llama.cpp, vllm and finally LFortran-compiled Fortran inference above with CUDA GPU offloading.

| Code | tok/s |
| --- | --- |
| C + CUDA | 120 |
| LFortran | 102 |
| vllm | 92 |
| llama.cpp | 72 |

There are various options for vllm and llama.cpp, I let AI agent iterate with both for quite a while to extract as much performance as possible, for llama it used flash attention, vllm uses cutlass, etc. Other combinations were slower. Maybe it’s possible to make them run even faster, but the above was the best I was able to get.

I also tried nvfortran that I had access to from NVHPC 26.5 (May 2026 release), but it can’t compile the code, so I had an agent rewrite it so that nvfortran can compile it, but it only ran at 5 tok/s on the GPU, so I didn’t include it in the table.

Regarding performance portability, the above code, unmodified, runs on any CPU via any Fortran compiler. And it also runs on Apple GPU via LFortran, right now at 3.4 tok/s, but I had it running at almost 6 tok/s, so the performance regressed a little bit, but I’ll fix it. The performance peak is about 17 tok/s. So not a bad start.

Some thoughts:

I expected that we will have to extend Fortran with some additional real kinds (bf16, and various other reduced precision), but we didn’t, just standard Fortran is enough.

I expected that we need some directives to handle memory movement, but it looks like the compiler is able to figure everything out by itself. @sumseq is that what you would also expect? If this translates to other codes, then we’ll be in great shape with Fortran.

The Fortran code is not too bad, about 2,000 lines of kernels, but they can be simplified, I had to do some of the optimizations by hand in Fortran, and we should build them into the compiler instead. The C+CUDA version has fewer kernels and is about 3x shorter as a result, and 20% faster. So we should do the same in Fortran.

So there is opportunity for improvement for sure, this is just the start. But hopefully it shows that it is possible to use Fortran for LLM inference and run on GPUs and run faster than some of the established inference frameworks.
