# 1 billion row challenge in Fortran : mmap + OpenMP + branchless scan

**URL:** <https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254>\
**Category:** Uncategorized\
**Created:** [January 23, 2024, 8:30am UTC](https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254 "2024-01-23T08:30:42Z")\
**Posts on this page:** 4\
**Page:** 3

<div class="post-metadata">

**Author:** ![sshestov](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@sshestov](https://fortran-lang.discourse.group/u/sshestov)\
**Post date:** [March 18, 2024, 12:28pm UTC](https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254/41 "2024-03-18T12:28:31Z")

</div>

Disagree: I’m also using mmap and my times are way more consistent. Just tried several times right now and constantly get ~2.98s (3.54s for your code).

In the measurements.txt I have 413 different cities (the command is still running, but that’s the way how it was created) and this is exactly that number I obtain in my case.

---

<div class="post-metadata">

**Author:** ![emiruz](https://avatars.discourse-cdn.com/v4/letter/e/e9a140/32.png) [@emiruz](https://fortran-lang.discourse.group/u/emiruz)\
**Post date:** [March 19, 2024, 2:07pm UTC](https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254/42 "2024-03-19T14:07:00Z")

</div>

Final version is [here](https://github.com/emiruz/1brc/blob/main/1brc_hash_mmap_openmp.f90). \<6s on 4 cores, on my XPS13 laptop 🙂 I’m happy with it.  
If you run it with fortran-13 it doesn’t seem to have the thread stall problem.

---

<div class="post-metadata">

**Author:** ![sshestov](https://avatars.discourse-cdn.com/v4/letter/s/bcef8e/32.png) [@sshestov](https://fortran-lang.discourse.group/u/sshestov)\
**Post date:** [March 19, 2024, 2:44pm UTC](https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254/43 "2024-03-19T14:44:34Z")

</div>

Cool, congrats! On my PC it takes ~3.2 s (`gfortran 11.4`). I tried it with `ifx` and it crashes (which actually can mean not a standard behavior of your program; I saw `gfortran` was more forgiving).

Do I understand correctly that for the hash table you do not use a regular array, but create … who is it … linked list? Also in your main program you define `type(row_ptr) :: hash_tbl(parts, hash_tbl_size)`. It might be faster if you re-arrange the dimensions, in fortran we have many-dimensional arrays stored in the memory not as in C.

---

<div class="post-metadata">

**Author:** ![emiruz](https://avatars.discourse-cdn.com/v4/letter/e/e9a140/32.png) [@emiruz](https://fortran-lang.discourse.group/u/emiruz)\
**Post date:** [March 19, 2024, 3:47pm UTC](https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254/44 "2024-03-19T15:47:43Z")

</div>

> [@sshestov](#):
>
> Cool, congrats! On my PC it takes ~3.2 s (`gfortran 11.4`).

Thank you for trying it! Its fastest with Fortran 13, so that may speed it up a little on your machine. Also setting `integer :: parts = 16` to either the number of cores you have (if no hyperthreading) or to twice the number of cores (if hyper threading). to be twice the number of cores: that makes quite a big difference.

> [@sshestov](#):
>
> I tried it with `ifx` and it crashes (which actually can mean not a standard behavior of your program; I saw `gfortran` was more forgiving).

I’ve coded a long time (~20 years), but this is my first Fortran program 🙂 I was delighted by how easy the language is to learn compared to my recent experiments (Prolog, Haskell). I haven’t tried it with `ifx`: I’ll give it a go and try to standardise it. I’m not surprised it doesn’t work under `ifx` because of the big number stuff in the hashing function, integer kinds, and the use of the transfer function: different assumptions about endians, wrapping behaviour, etc would produce different results.

> [@sshestov](#):
>
> Do I understand correctly that for the hash table you do not use a regular array, but create … who is it … linked list?

No, it uses a regular array. I have an array of type(row\_ptr):

type :: row\_ptr  
type(row), pointer :: p =\> null()  
end type row\_ptr

That allows me to address the content of individual cells.

> [@sshestov](#):
>
> It might be faster if you re-arrange the dimensions, in fortran we have many-dimensional arrays stored in the memory not as in C.

Ok, I’ll try it!

[Previous page](https://fortran-lang.discourse.group/t/1-billion-row-challenge-in-fortran-mmap-openmp-branchless-scan/7254.md?page=2)