# Big Data in Fortran

**URL:** <https://fortran-lang.discourse.group/t/big-data-in-fortran/4208>\
**Category:** Uncategorized\
**Created:** [August 24, 2022, 1:39am UTC](https://fortran-lang.discourse.group/t/big-data-in-fortran/4208 "2022-08-24T01:39:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ali](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ali/32/751_2.png) [@Ali](https://fortran-lang.discourse.group/u/Ali)\
**Post date:** [August 24, 2022, 1:39am UTC](https://fortran-lang.discourse.group/t/big-data-in-fortran/4208/1 "2022-08-24T01:39:44Z")

</div>

What is the best approach for handling Big Data in Fortran? How is it possible to create arrays like [tall arrays](https://de.mathworks.com/help/matlab/import_export/tall-arrays.html) for out-of-memory data in MATLAB?

---

<div class="post-metadata">

**Author:** ![urbanjost](https://avatars.discourse-cdn.com/v4/letter/u/0ea827/32.png) [@urbanjost](https://fortran-lang.discourse.group/u/urbanjost)\
**Post date:** [August 24, 2022, 2:11am UTC](https://fortran-lang.discourse.group/t/big-data-in-fortran/4208/2 "2022-08-24T02:11:05Z")

</div>

There are other methods, but the simplest one is typically to store the data in direct access files and read sections into arrays in memory as needed. This is often done with small functions that allow you to load the data using something that appears like array syntax. There are multiple libraries available for different scenarios available also. If starting to work with large complex datasets you might start with reading about HDF5 as well.

---

<div class="post-metadata">

**Author:** ![jeremie.vandenplas](https://avatars.discourse-cdn.com/v4/letter/j/76d3ee/32.png) [@jeremie.vandenplas](https://fortran-lang.discourse.group/u/jeremie.vandenplas)\
**Post date:** [August 24, 2022, 7:10am UTC](https://fortran-lang.discourse.group/t/big-data-in-fortran/4208/3 "2022-08-24T07:10:05Z")

</div>

What kind/amount of Big Data do you mean?  
Recently I used [h5fortran](https://github.com/geospace-code/h5fortran). As @urbanjost mentioned it, if you work with large complex datasets, I would suggest to give it a try.

---

<div class="post-metadata">

**Author:** ![gnikit](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/gnikit/32/1213_2.png) [@gnikit](https://fortran-lang.discourse.group/u/gnikit)\
**Post date:** [August 24, 2022, 8:13am UTC](https://fortran-lang.discourse.group/t/big-data-in-fortran/4208/4 "2022-08-24T08:13:04Z")

</div>

As others have mentioned, HDF5 is probably the easiest way of doing so. An added benefit, which I found extremely useful, is the ability to visualise the data interactively in a Jupyter notebook using Python.

---

<div class="post-metadata">

**Author:** ![Ali](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ali/32/751_2.png) [@Ali](https://fortran-lang.discourse.group/u/Ali)\
**Post date:** [August 24, 2022, 10:54am UTC](https://fortran-lang.discourse.group/t/big-data-in-fortran/4208/5 "2022-08-24T10:54:09Z")

</div>

Thanks @urbanjost, @jeremie.vandenplas!

Regarding @jeremie.vandenplas 's question: The data is millions or billions of real kind=8.