What is the best approach for handling Big Data in Fortran? How is it possible to create arrays like tall arrays for out-of-memory data in MATLAB?
1 Like
There are other methods, but the simplest one is typically to store the data in direct access files and read sections into arrays in memory as needed. This is often done with small functions that allow you to load the data using something that appears like array syntax. There are multiple libraries available for different scenarios available also. If starting to work with large complex datasets you might start with reading about HDF5 as well.
4 Likes
What kind/amount of Big Data do you mean?
Recently I used h5fortran. As @urbanjost mentioned it, if you work with large complex datasets, I would suggest to give it a try.
2 Likes
As others have mentioned, HDF5 is probably the easiest way of doing so. An added benefit, which I found extremely useful, is the ability to visualise the data interactively in a Jupyter notebook using Python.
2 Likes
Thanks @urbanjost, @jeremie.vandenplas!
Regarding @jeremie.vandenplas 's question: The data is millions or billions of real kind=8.
1 Like