# Supporting BFLOAT16 in Fortran: “NOT recommended"?
**URL:** https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637
**Category:** Language enhancement
**Created:** [March 15, 2024, 8:25pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637 "2024-03-15T20:25:21Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 15, 2024, 8:25pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/1 "2024-03-15T20:25:21Z")
Low-precision floating-point arithmetic is [one of the magics that enable us to have the powerful AI today](https://pytorch.org/blog/what-every-user-should-know-about-mixed-precision-training-in-pytorch/) (see also [discussions on Hacker News](https://news.ycombinator.com/item?id=39713255)). It is the workhorse on modern hardware like GPU and [TPU](https://en.wikipedia.org/wiki/Tensor_Processing_Unit), although [not as much on CPU](https://scicomp.stackexchange.com/questions/35187/is-half-precision-supported-by-modern-architecture) (yet).
Being a language aimed at supporting HPC on modern hardware, Fortran cannot afford beig left behind in terms of low-precision floating-point arithmetic. Fortunately, REAL16 (half-precision real) has been introduced in F2023, which is fabulous! As of March 2024, I only know [`nagfor`](https://fortran-lang.discourse.group/t/nagfor-7-1-supports-half-precision-floating-point-numbers-but-with-many-bugs/7488) and [`nvfortran`](https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-ref-guide/index.html#data-types-fort-real2) support it, the latter without providing intrinsics like `abs`, `exp`, …
However, the Low-precision floating-point arithmetic being used in AI training is mostly [BFLOAT16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format). This non-IEEE arithmetic has a precision lower than but a range equal to single precision. No Fortran compiler on the market supports it now, as far as I know.
Indeed, I read today that there was a discussion about introducing BFLOAT16 to `iso_fortran_env`, but the conclusion was that
> JOR recommends that J3 does not proceed with adding any explicit support for BFLOAT16.
See [https://j3-fortran.org/doc/year/20/20-118.txt](https://j3-fortran.org/doc/year/20/20-118.txt).
Will LFortran support BFLOAT16 @certik ? I am told that [LLVM Flang supports it with `real(kind = 3)`](https://news.ycombinator.com/item?id=39714226), but LLVM Flang is not ready to use as far as I understand. I look forward to trying in in [PRIMA](http://www.libprima.net).
* * *
* * *
**Referenes**
> [@Discussion on Fortran's type-kind distinction](https://fortran-lang.discourse.group/t/discussion-on-fortrans-type-kind-distinction/2144/6):
>
> I wonder if selected\_real\_kind was addressing the problem from the 60s, 70s and early 80s, that has disappeared by the time it became available in the standard and compilers. Now, can it happen in the future that we will see more variety again? I think it can. We might have both half precision (real16?) as well as bfloat16. What is nice about the selected\_real\_kind approach is that it is hardware independent. You select the properties of floating point that you need, and your code should run …
> [@\`nagfor\` 7.1 supports half-precision floating-point numbers, but with many bugs](https://fortran-lang.discourse.group/t/nagfor-7-1-supports-half-precision-floating-point-numbers-but-with-many-bugs/7488):
>
> Half-precision real has been widely available on GPUs with important applications in machine learning. On CPUs, it seems less available. I discovered just a few days ago that nagfor now supports REAL16, half-precision real. This is very interesting. However, I found a couple of bugs after a quick try on [PRIMA](http://www.libprima.net). Here is an example, which has been submitted to NAG confirmed as a bug by NAG. Code (also available at [test\_compiler/test\_isorth.f90 at master · zequipe/test\_compiler · GitHub](https://github.com/zequipe/test_compiler/blob/master/test_isorth.f90)): ! test…
> [@Use case of single-precision real number](https://fortran-lang.discourse.group/t/use-case-of-single-precision-real-number/4965):
>
> Could anyone please explain to me in what cases single-precision real numbers are more suitable than double-precision ones?
---
### Author:  [@rbitr](https://fortran-lang.discourse.group/u/rbitr)
#### Post date: [March 15, 2024, 10:27pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/2 "2024-03-15T22:27:07Z")
It’s probably just random chance but I feel like I encounter FP16 much more often than BF16. For ML I’d like to see a breakdown of which major models use which and in which formats.
I’d rather see FP8 or other quantization support, if i had to prioritize.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 16, 2024, 8:23am UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/3 "2024-03-16T08:23:47Z")
> [@rbitr](#):
>
> It’s probably just random chance but I feel like I encounter FP16 much more often than BF16.
> For ML I’d like to see a breakdown of which major models use which and in which formats.
For your reference, see
[To Bfloat or not to Bfloat? That is the Question!](https://www.cerebras.net/machine-learning/to-bfloat-or-not-to-bfloat-that-is-the-question/#:~:text=Our%20experiments%20demonstrated%20that%20choosing,maintaining%20the%20same%20accuracy%20level)
[How to select half precision (BFLOAT16 vs FLOAT16) for your trained model?](https://stackoverflow.com/questions/69399917/how-to-select-half-precision-bfloat16-vs-float16-for-your-trained-model)
[Mixed Precision Training: Difference between BF16 and FP16](https://www.reddit.com/r/MachineLearning/comments/vndtn8/d_mixed_precision_training_difference_between/)
[BFloat16: The secret to high performance on Cloud TPUs](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus)
The general view seems to be “use BFLOAT16 if it is supported on your platform”, to benefit from its high speed (hardware support needed) and low storage while maintaining the same range as single precision. But this really depends on your hardware and language. As mentioned, the availability of BFLOAT16 is still limited on CUP, and does not exist with Fortran (“supporting it in Fortran is not recommended” …).
You may also search for [“machine learning float16 or bfloat16”](https://www.google.com/search?q=machine+learning+float16+or+bfloat). For traditional HPC, the situation may not necessarily be the same.
---
### Author:  [@PierU](https://fortran-lang.discourse.group/u/PierU)
#### Post date: [March 16, 2024, 11:56am UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/4 "2024-03-16T11:56:41Z")
> [@zaikunzhang](#):
>
> JOR recommends that J3 does not proceed with adding any explicit support for BFLOAT16.
Note that this has strictly no consequence on the actual support of bloats16’s by the compilers. It’s just that there’s no dedicated constant in `iso_fortran_env`. Conversely, if such constant was adopted, it would just be `-1` in most compilers.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 16, 2024, 12:50pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/5 "2024-03-16T12:50:20Z")
> [@PierU](#):
>
> Note that this has strictly no consequence on the actual support of bloats16’s by the compilers. It’s just that there’s no dedicated constant in `iso_fortran_env`.
Being neither a standard committee member nor a compiler developer, I guess the inclusion of `BFLOAT16` in `iso_fortran_env` would encourage rather than discourage vendors to support it. In contrast, the recommendation for not including/supporting it in `iso_fortran_env` can hardly be an encouragement.
Excluding `BFLOAT16` is almost excluding Fortran from training large machine learning and AI models, which is not necessarily the most important business but cannot be ignored either.
---
### Author:  [@rwmsu](https://fortran-lang.discourse.group/u/rwmsu)
#### Post date: [March 16, 2024, 3:46pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/6 "2024-03-16T15:46:55Z")
Kind of interesting that if you do a web search on half or mixed precision applications you see several articles pop up that describe research in using half precision for linear algebra (definitely something the HPC community cares about). In particular there appears to be a lot of interest in using iterative refinement techniques to recover standard (32 bit) precision from mixed half-precsion linear solvers etc. I guess this is driven mostly by ML training requirements but given potential performance difference of half-precision over standard precision on even modestly priced GPUs you would think this is something the standard committee would want to support. Since NVidia, AMD and I presume Intel hobble the double precision performance on their consumer graphics cards, some kind of mixed precision solution to HPC type problems on inexpensive hardware seems like something Fortran should be supporting.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 16, 2024, 4:13pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/7 "2024-03-16T16:13:25Z")
> [@rwmsu](#):
>
> Kind of interesting that if you do a web search on half or mixed precision applications you see several articles pop up that describe research in using half precision for linear algebra (definitely something the HPC community cares about). In particular there appears to be a lot of interest in using iterative refinement techniques to recover standard (32 bit) precision from mixed half-precsion linear solvers etc.
Low/mixed-precision computing has been an intriguing research topic in the numerical analysis community for **many (~20)** years. It is **much before** low-precision training shows its power in machine learning. See [the review paper](https://www.sciencedirect.com/science/article/pii/S0010465508003846?casa_token=AtovCLvDDpQAAAAA:pDIgKrZvi2YzjLwhfbuv9QXPRTYN7gM2LzORxmWpQYQ1EPinXuuKtqQ42fR52scOH_m5_5EYrPo) from 2009 for more information. [Jack Dongarra](https://en.wikipedia.org/wiki/Jack_Dongarra) (Turing Award winner) is one of the co-authors.
Another important figure in mixed-precision computing was [Nick Higham](https://nhigham.com/) (FRS, former SIAM president), who sadly [passed away](https://www.lms.ac.uk/news/nick-higham) recently.
As a computational/applied mathematician, I have kept a keen interest in low/mixed-precision computing. Even though I have not made (published) any essential contribution yet, I am confident to say that it is one of the most important topics in modern scientific computing, even if it did not turn out that useful to machine learning and AI.
Unfortunately, as a language for scientific computing, Fortran is essentially out of this game (up to now) — it is indeed excluding itself from the game.
---
### Author:  [@PierU](https://fortran-lang.discourse.group/u/PierU)
#### Post date: [March 16, 2024, 6:01pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/8 "2024-03-16T18:01:55Z")
> [@zaikunzhang](#):
>
> Excluding `BFLOAT16` is almost excluding Fortran from training large machine learning and AI models, which is not necessarily the most important business but cannot be ignored either.
Again, nobody wants to -and more than that nobody can- prevent any compiler from supporting a `bfloat16` real. I can’t imagine that the decision from a compiler editor to bring `bfloat16` support depend in any way on the presence of a constant in `iso_fortran_env` (before `iso_fortran_env` did exist, many compilers were supporting 128bits reals).
_nvfortran_ supports `bfloat16` because it makes perfect sense for Nvidia: they need it to operate their GPUs.
_nagfor_ supports `bfloat16` because they are multi-platform
_Intel Fortran_ doesn’t support `bfloat16` because Intel does’t have any hardware with them (in the same way, Intel Fortran does not support OpenMP offloading to GPUs other than Intel… at least it was the case until recently)
_gfortran_, well, I guess they lack resources for implementing everything they would like.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 17, 2024, 7:43am UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/9 "2024-03-17T07:43:31Z")
> [@PierU](#):
>
> Again, nobody wants to -and more than that nobody can- prevent any compiler from supporting a `bfloat16` real. I can’t imagine that the decision from a compiler editor to bring `bfloat16` support depend in any way on the presence of a constant in `iso_fortran_env` (before `iso_fortran_env` did exist, many compilers were supporting 128bits reals).
Fair enough. However, if we only rely on vendor extensions for important features of the languages, what is the point of having a standard in the first place?
---
### Author:  [@PierU](https://fortran-lang.discourse.group/u/PierU)
#### Post date: [March 17, 2024, 8:44am UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/10 "2024-03-17T08:44:50Z")
There’s a confusion here. Again, introducing a constant in `iso_fortran_env` wouldn’t mean that supporting the corresponding type would be _required_. Compilers are not _required_ to support a `real128` type: just, if they do, they can set the kind value in `real128` (and if they don’t the constant is -1). In that sense, the support of a `real128` type, if different from the default `real` and `double precision` ones, is an extension to the standard.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 17, 2024, 1:14pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/11 "2024-03-17T13:14:02Z")
> [@PierU](#):
>
> There’s a confusion here. Again, introducing a constant in `iso_fortran_env` wouldn’t mean that supporting the corresponding type would be _required_. Compilers are not _required_ to support a `real128` type: just, if they do, they can set the kind value in `real128` (and if they don’t the constant is -1−1-1). In that sense, the support of a `real128` type, if different from the default `real` and `double precision` ones, is an extension to the standard.
I know very well that the inclusion of `bfloat16` in `iso_fortran_env` never means requesting/ensuring support for it. It is clear to me that Fortran standards impose only the existence of the default and the double-precision real, and will unlikely impose anything else in the foreseeable future.
However, IMHO, the inclusion or exclusion of a kind value in `iso_fortran_env` still means something. The interpretation of this “something” may vary from person to person, but it is definitely not nothing. Otherwise, the standard committee would not have had a vote to decide it.
---
### Author:  [@PierU](https://fortran-lang.discourse.group/u/PierU)
#### Post date: [March 17, 2024, 9:03pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/12 "2024-03-17T21:03:14Z")
> [@zaikunzhang](#):
>
> I know very well that the inclusion of `bfloat16` in `iso_fortran_env` never means requesting/ensuring support for it
The problem is that the title of this topic, and a part of your text as well, can mean something different to the readers…
> [@zaikunzhang](#):
>
> Otherwise, the standard committee would not have had a vote to decide it.
Knowing the arguments for/against the proposal inside the subgroup and committee would be interesting, in particular why `real16` had been accepted and not `bfloat16`. A part of the answer is maybe that the kind constants in `iso_fortran_env` do not attempt to describe a specific implementation. For instance `real32` can be any floating point stored in 32 bits, not necessarily the IEEE flavor. The same with `real128`, which can be in practice either the IEEE flavor, or the “double double” one.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 17, 2024, 11:39pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/13 "2024-03-17T23:39:19Z")
> [@PierU](#):
>
> A part of the answer is maybe that the kind constants in `iso_fortran_env` do not attempt to describe a specific implementation. For instance `real32` can be any floating point stored in 32 bits, not necessarily the IEEE flavor. The same with `real128`, which can be in practice either the IEEE flavor, or the “double double” one.
Maybe this was one of the reasons.
---
### Author:  [@certik](https://fortran-lang.discourse.group/u/certik)
#### Post date: [March 18, 2024, 8:57pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/14 "2024-03-18T20:57:42Z")
I think we need both IEEE f16, and bf16 in Fortran.
What are the kind numbers that other compilers use for f16 and bf16?
---
### Author:  [@rwmsu](https://fortran-lang.discourse.group/u/rwmsu)
#### Post date: [March 18, 2024, 9:53pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/15 "2024-03-18T21:53:18Z")
Nvidia web sites say that fp16 is real(2) but is only fully supported on gpu/tpu. However, the source file for iso\_fortran\_env in latest nvidia HPC-SDK compilers doesn’t list a KIND parameter with a value of 2 for a real value (ie no REAL16). I don’t know about BFLOAT16. I don’t see any mention of anything other than the IEEE FP16.
---
### Author:  [@certik](https://fortran-lang.discourse.group/u/certik)
#### Post date: [March 18, 2024, 10:07pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/16 "2024-03-18T22:07:02Z")
I see: f16 (IEEE) could be kind=2, and bf16 could be kind=3?
---
### Author:  [@rwmsu](https://fortran-lang.discourse.group/u/rwmsu)
#### Post date: [March 18, 2024, 10:13pm UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/17 "2024-03-18T22:13:35Z")
The following code works with nvfortran 24.3.
```auto
Program test16
USE ISO_FORTRAN_ENV
real :: a32
real(2) :: a16
a32 = 4.0
a16 = 2.0_2
print *,' 32 / 16 = ', a32/a16
stop
end program test16
```
On Linux mint system
nvfortran -o test16.x testfp16.f90
./test16.x
32 / 16 = 2.000000
I presume the systems that cling to using the number of bytes as the KIND value will probably default to 3 but the standard says it can be anything. Probably best to check with NAG to see what they do since they are the only ones I know of that don’t use REAL\_KINDS=[4,8] etc.
Edit.
As to the KIND variable names for ISO\_FORTRAN\_ENV, I think just using REAL16 for IEEE f16
and REAL16E (extended real16) or REAL16B for BFLOAT16 might be the path of least resistance but that makes to much sense (at least to me) so the standards folks would never adopt it.
---
### Author:  [@zaikunzhang](https://fortran-lang.discourse.group/u/zaikunzhang)
#### Post date: [March 19, 2024, 4:24am UTC](https://fortran-lang.discourse.group/t/supporting-bfloat16-in-fortran-not-recommended/7637/18 "2024-03-19T04:24:24Z")
> [@rwmsu](#):
>
> I presume the systems that cling to using the number of bytes as the KIND value will probably default to 3 but the standard says it can be anything. Probably best to check with NAG to see what they do since they are the only ones I know of that don’t use REAL\_KINDS=[4,8] etc
```fortran
! kinds.f90
program kinds
use iso_fortran_env, only : REAL16, REAL32, REAL64, REAL128
implicit none
print *, REAL16, REAL32, REAL64, REAL128
end program kinds
```
```plaintext
$ nagfor kinds.f90 && ./a.out
NAG Fortran Compiler Release 7.2(Shin-Urayasu) Build 7201
[NAG Fortran Compiler normal termination]
16 1 2 3
```