While working on an OpenACC GPU port of a CFD solver with OpenACC, I wanted to maintain bit-reproducibility between existing CPU calculations and GPU-accelerated kernels — particularly for mathematical transcendental functions (sin, cos, exp, etc.).
I tested the bitwise-reproducible algorithms from Arteaga et al. (2014). By wrapping the original bitrep C++ source with a Fortran interface and adding OpenACC/OpenMP directives (plus disabling compilers fused multiply-add optimizations) I achieved reproducibility between CPU and GPU in pure transcendental kernels.
Here a few note:
Bit-reproducible results have been tested on several architectures of NVIDIA and AMD GPUs.
Performance penalty ~3-4× slowdown compared to standard math libraries
Equivalent performance on OpenACC/OpenMP with NVHPC SDK compiler (as expected)
Haven’t yet tested directly in CUDA/HIP kernels to determine if the performance loss could be reduced by removing the OpenACC/OpenMP overhead
I’m also porting a weather model to a GPU platform project, using the bitrep library to achieve bitwise reproducibility verification across CPU and GPU platforms. Tests passed with the NVIDIA HPC SDK (CPU vs GPU) and with Flang (Sugon CPU vs GPU). My bitrep github repo: qingu/bitrep: reproducible CPU/GPU transcendental functions with Fortran interface
Have you tested bitwise reproducibility across different compilers on the CPU platform?
CPU IEEE float and GPU IEEE float are different, so a cross-backend mismatch is actually expected. A good way to (kinda) fight it is through same-backend regression and analysis instead.
Thanks for sharing your implementation. Once I’ve finished investigating CUDA support and its performance—which currently shows similar performance slowdowns—I’ll test reproducibility between different compilers using binary output testing as you did.
Thanks for sharing your trauma notes! I am facing a similar issue with MPI_Reduce. Arteaga’s paper presents a solution for reproducibility in reductions, but I haven’t investigated it yet.
MOM6 uses an order-invariant extended fixed-precision sum, as described Hallberg and Adcroft (2014). Values are converted to fixed precision and distributed into bins which are reproducibly summed as integers, with accounting for carry bits and overflow. There is some redundancy in the range of the bins, so that each can tolerate a certain number of carrys before pushing upward (which is a serialized, and potentially inefficient, step).
A newer paper with a very similar approach (to me at least) is Ahrens et al. (2020).
On the transcendental side, I now have properly ported the functions to CUDA and I measured the same performance overhead as with the OpenACC/OpenMP implementation.
This week I tested reproducibility by dumping the bits in binary files (as you did) and comparing them on different compilers and architectures. I achieve bit-wise reproducibility across: