# fortranDF: A simple data frame library for Fortran **URL:** \ **Category:** Announcements\ **Created:** [February 22, 2024, 2:19am UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446 "2024-02-22T02:19:54Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![jaiken](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jaiken/32/4356_2.png) [@jaiken](https://fortran-lang.discourse.group/u/jaiken)\ **Post date:** [February 22, 2024, 2:19am UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/1 "2024-02-22T02:19:54Z")
Through the needs of a different project that I am working on, I have created a library that implements a data frame data structure called fortranDF (fortran Data Frames): > **[GitHub - jaiken17/fortranDF: FortranDF is a library that implements a...](https://github.com/jaiken17/fortranDF)** > > FortranDF is a library that implements a data\_frame data structure similar to those used in other languages. - jaiken17/fortranDF The idea is to be somewhat similar to the data frames found in Pandas and R, where the data is column-based. Each column can be a different intrinsic type, but all data within a column must be of the same type. Each column also can have a header, which can be used to retrieve the column associated to said header. Here is some example code that builds a data frame with an integer column and a real column and prints the whole data frame to the screen: ```fortran program example use iso_fortran_env,only: OUTPUT_UNIT use df_precision,only: rk use df_fortranDF,only: data_frame implicit none type(data_frame) :: df call df%new() call df%append([1.0_rk,2.0_rk,3.0_rk],"real_data") call df%append([1,2,3],"integer_data") call df%write(OUTPUT_UNIT) end program example ``` Right now the library is just the data container/structure, with some very limited IO procedures (write to screen and read from a file with a very specific format). I hope to change this in the future (ideally add at least support for CSV format) but for now IO is delegated to the user with the use of extended types. As an example for extending the `data_frame` type, I have also made [fortranMR](https://github.com/jaiken17/fortranMR) (fortran MESA reader) public, which is a very bare bones and unpolished library for reading in a type of file that the stellar evolution program [MESA](https://github.com/mesaHub/MESA) outputs. fortranMR is really for my own use, but it happens to also serve as decent example of making user-defined extensions to the `data_frame` type. I am interested to hear any feedback for anything about this project, including feature ideas, code criticism, documentation, or even whether or not you think this would be useful to use. ---
**Author:** ![Beliavsky](https://avatars.discourse-cdn.com/v4/letter/b/ba8739/32.png) [@Beliavsky](https://fortran-lang.discourse.group/u/Beliavsky)\ **Post date:** [February 22, 2024, 2:10pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/2 "2024-02-22T14:10:43Z")
Thanks for your project. As a heavy user of Python pandas, I have wondered how closely Fortran could replicate a pandas or R data frame. For my own use I wrote a “data frame” type in Fortran restricted to double precision values, character column headers, and type(date) (defined by me) index values, and it was useful. If a data frame has n1 rows and n2 columns, you could add an allocatable component `valid(:,:)` with shape `[n1, n2]` that is `.true.` where data exists. You could set non-existent real data to NaN, but there is no equivalent for data of other types. If `valid` is not allocated it means all data exists. Instead of having columns (1-D arrays) of arbitrary type, another implementation of a data frame would have it be composed of 2-D arrays of arbitrary type, since it is more efficient to deal with a 2-D array than an array of 1-D arrays, and since data frames will often be composed of groups of columns of the same type. ---
**Author:** ![everythingfunctional](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/everythingfunctional/32/176_2.png) [@everythingfunctional](https://fortran-lang.discourse.group/u/everythingfunctional)\ **Post date:** [February 22, 2024, 3:06pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/3 "2024-02-22T15:06:13Z")
I haven’t had a chance to look at your library closely yet, but one of the things I used to really like about Pandas was the ability to read in a file like ```plaintext City, State, Mayor, Population Nowhere, CA, Person, N/A Somewhere, AL, , 2000 ``` and be able to do something like (I may have the indexing backwards) ```python print(data["Nowhere"]["Mayor"]) # prints "Person" print(data["Somewhere"]["Population"]) # prints 2000 ``` and have things not explode. Part of what makes that possible is Python’s duck-typing. Not sure how close you can get to that in Fortran, but it’s what made data exploration in Python so easy. ---
**Author:** ![jaiken](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jaiken/32/4356_2.png) [@jaiken](https://fortran-lang.discourse.group/u/jaiken)\ **Post date:** [February 22, 2024, 4:13pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/4 "2024-02-22T16:13:03Z")
> If a data frame has n1 rows and n2 columns, you could add an allocatable component `valid(:,:)` with shape `[n1, n2]` that is `.true.` where data exists. Originally, I had thought about letting each column have its own length, but at some point I made the decision to enforce all columns to be the same length for the ease of implementation. With my current implementation, it is possible for the user to append columns all of the same length but where the ‘good’ data ends early (and then keep track of the size of the ‘good’ data on their own). Currently I have a class member called `nrows` that is just scalar, but it might be worth adding a `max_nrows` scalar and turn `nrows` into a rank 1 array to keep track of the ‘good’ data. But I do like your idea of a mask array as well. > Instead of having columns (1-D arrays) of arbitrary type, another implementation of a data frame would have it be composed of 2-D arrays of arbitrary type, since it is more efficient to deal with a 2-D array than an array of 1-D arrays, and since data frames will often be composed of groups of columns of the same type. This is also a very good idea. I think I had thought about something similar, but again thought it would be too difficult initially (mainly due to wanting to associate headers with columns). I have some new ideas on how to deal with the headers though, so I think I will revisit this. Thanks for your input @Beliavsky. ---
**Author:** ![jaiken](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jaiken/32/4356_2.png) [@jaiken](https://fortran-lang.discourse.group/u/jaiken)\ **Post date:** [February 22, 2024, 5:35pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/5 "2024-02-22T17:35:31Z")
I didn’t know that this was possible in Python. I will admit that I haven’t used data frames in Python _that much_ but I did really like the idea of that type of container. I think there would be a way to get similar functionality to your code snippet (albeit in more lines). I will assume that ```plaintext City, State, Mayor, Population Nowhere, CA, Person, N/A Somewhere, AL, , 2000 ``` is already stored in a `data_frame` object called `df` and where `City, State, Mayor, Population` are the headers. We could then have something along the lines of ```fortran index = findloc(df%getch("City"),"Nowhere",dim=1) print*, df%getch("Mayor",index) ! prints Person index = findloc(df%getch("City"),"Somewhere",dim=1) print*, df%getch("Population",index) ! prints 2000 ``` where `index` is an integer, and the value 2000 must be a character string. This functionality could be hidden away into a function and overloaded with the `getch` functions. It seems to me that the `"City"` column is being treated as a second header but I would want to look more into the specific behaviour that Python allows before implementing this. This is a very interesting suggestion @everythingfunctional, thanks for the input. ---
**Author:** ![scottza](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@scottza](https://fortran-lang.discourse.group/u/scottza)\ **Post date:** [February 23, 2024, 8:38am UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/6 "2024-02-23T08:38:43Z")
In case this project grows more complex at some point, it can be worth keeping the [Apache Arrow](https://arrow.apache.org) project in mind when thinking about the in-memory storage format. ---
**Author:** ![samharrison7](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/samharrison7/32/3868_2.png) [@samharrison7](https://fortran-lang.discourse.group/u/samharrison7)\ **Post date:** [February 23, 2024, 8:49am UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/7 "2024-02-23T08:49:48Z")
Great work @jaiken! > [@jaiken](#): > > or even whether or not you think this would be useful to use It definitely will be useful! I’m convinced that a lack of data analysis libraries like Pandas is holding people making from moving over to Fortran. If you couple this with interactive notebooks via LFortran, then we’re onto a winner. Coincidently, creating a Fortran dataframe library has been on my to-do list for far too long, so I’m very pleased to see this! > [@scottza](#): > > In case this project grows more complex at some point, it can be worth keeping the [Apache Arrow](https://arrow.apache.org) project in mind when thinking about the in-memory storage format. It might be worth keeping [Polars](https://pola.rs/) in mind too (which follows the Arrow model). I think that is written in Rust. It would be nice if we have a Fortran Arrow-like library that outperforms Polars (because Polars boasts about how fast it is!). ---
**Author:** ![jaiken](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jaiken/32/4356_2.png) [@jaiken](https://fortran-lang.discourse.group/u/jaiken)\ **Post date:** [February 26, 2024, 5:40pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/8 "2024-02-26T17:40:16Z")
Thanks @samharrison7 and @scottza for the input. So far, I haven’t worried much about performance as I really just wanted something that worked and wouldn’t take _too_ long to build. But as I progress with the project, I will keep performance in mind and will definitely look more closely into Polars and Apache Arrow. ---
**Author:** ![omdaniel](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/omdaniel/32/5820_2.png) [@omdaniel](https://fortran-lang.discourse.group/u/omdaniel)\ **Post date:** [March 20, 2025, 2:40pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/9 "2025-03-20T14:40:15Z")
Another plug for a native Fortran interface to Apache Arrow is that if Fortran is storing data in-memory using Arrow one would be able to use other data analysis tool in Python or R to analyze the in-memory data without having to save to the file system storage first. ---
**Author:** ![Beliavsky](https://avatars.discourse-cdn.com/v4/letter/b/ba8739/32.png) [@Beliavsky](https://fortran-lang.discourse.group/u/Beliavsky)\ **Post date:** [March 21, 2025, 1:26pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/10 "2025-03-21T13:26:34Z")
My dataframe type, specialized to integer indices, string column names, and double precision data is > **[GitHub - Beliavsky/DataFrame: DataFrame for Fortran for floating point data](https://github.com/Beliavsky/DataFrame)** > > DataFrame for Fortran for floating point data I use it for time series data, where dates can be represented as YYYYMMDD integers. The README gives an example of using it to analyze stock price data. The code below shows how one can subset data ```fortran program xdataframe_loc use dataframe_mod, only: DataFrame, display, random type(DataFrame) :: df call random(df, 4, 3) df%index = df%index * 10 call display(df, title="df") ! subset by row and column numbers call display(df%icol([2,3]), title="df%icol([1,3])") call display(df%irow([2,4]), title="df%irow([2,4])") ! subset by index values and column names call display(df%loc(rows=[20, 40]), title="df%loc(rows=[20, 40]") call display(df%loc(columns=["C1", "C3"]), title="df%loc(columns=['C1', 'C3'])") call display(df%loc(rows=[20, 40], columns=["C1", "C3"]), & title="df%loc(rows=[20, 40], columns=['C1', 'C3']") call display(df%loc()) end program xdataframe_loc ``` with output ```auto df index C1 C2 C3 10 0.9584 0.1462 0.3937 20 0.5586 0.2374 0.3336 30 0.1630 0.3181 0.6878 40 0.5074 0.0701 0.7673 df%icol([1,3]) index C2 C3 10 0.1462 0.3937 20 0.2374 0.3336 30 0.3181 0.6878 40 0.0701 0.7673 df%irow([2,4]) index C1 C2 C3 20 0.5586 0.2374 0.3336 40 0.5074 0.0701 0.7673 df%loc(rows=[20, 40] index C1 C2 C3 20 0.5586 0.2374 0.3336 40 0.5074 0.0701 0.7673 df%loc(columns=['C1', 'C3']) index C1 C3 10 0.9584 0.3937 20 0.5586 0.3336 30 0.1630 0.6878 40 0.5074 0.7673 df%loc(rows=[20, 40], columns=['C1', 'C3'] index C1 C3 20 0.5586 0.3336 40 0.5074 0.7673 index C1 C2 C3 10 0.9584 0.1462 0.3937 20 0.5586 0.2374 0.3336 30 0.1630 0.3181 0.6878 40 0.5074 0.0701 0.7673 ``` ---
**Author:** ![rngil](https://avatars.discourse-cdn.com/v4/letter/r/b19c9b/32.png) [@rngil](https://fortran-lang.discourse.group/u/rngil)\ **Post date:** [September 30, 2025, 3:36am UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/11 "2025-09-30T03:36:07Z")
For those interested, here’s my take ([DataFort](https://github.com/rngil/DataFort)) on dataframes that I’ve working on for some time, loosely based on `fortranDF`. I also ran a very minor benchmark against polars (100,000 x 5 dummy data) | Benchmark | DataFort (s) | Polars (s) | Faster | | --- | --- | --- | --- | | 1. Creating DataFrame (5 cols) | 0.001421 | 0.003715 | **DataFort (2.6x)** | | 2. Statistics (mean, std, min, max) | 0.001112 | 0.001962 | **DataFort (1.8x)** | | 3. Filtering (Temperature \> 25.0) | 0.004858 | 0.014884 | **DataFort (3.1x)** | | 4. Sorting by column | 1.199529 | 0.007562 | **Polars (159x)** | | 5. Selecting 3 columns | 0.000871 | 0.000513 | **Polars (1.7x)** | | 6. Slicing rows [1000:5000] | 0.000206 | 0.000364 | **DataFort (1.8x)** | | 7. Math transforms (log, sqrt) | 0.000865 | 0.002430 | **DataFort (2.8x)** | | 7b. Apply custom function (x² + 2x + 1) | 0.000433 | 0.003364 | **DataFort (7.8x)** | | 8. Inner join on column | 0.417289 | 0.005924 | **Polars (70x)** | | 9. Writing to CSV | 0.198987 | 0.028333 | **Polars (7.0x)** | | 10. Reading from CSV | 0.456994 | 0.012179 | **Polars (38x)** | I should probably use something other than quicksort for my sorting. ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [September 30, 2025, 1:20pm UTC](https://fortran-lang.discourse.group/t/fortrandf-a-simple-data-frame-library-for-fortran/7446/12 "2025-09-30T13:20:04Z")
There are few things that you could try to improve some of those slow parts: - Sorting: `stdlib` has some nice and performant sorting algorithms that you could test [sorting – Fortran-lang/stdlib](https://stdlib.fortran-lang.org/page/specs/stdlib_sorting.html), and if you need to with respect to multiple indexes, one strategy that I’ve used for sorting COO matrices is (just two levels in this case) would be to use count sort for the exterior sorting, and then a quick sort for the interior index. - For reading CSV files, you could also profit from the accelerated numerical conversion procedures [str2num – Fortran-lang/stdlib](https://stdlib.fortran-lang.org/page/specs/stdlib_str2num.html)