I am using the NetCDF library in Fortran to store the output of my numerical simulations.
I was wondering what people think about the following?
I have to store complex matrices and wondering whether it is better to write the REAL and IMAGINARY parts separately (there is no native complex external data type in NetCDF) or define a new compound type with the nf90_def_compound function.
I have a similar problem, at the moment I have stored the imaginary part and the real separately.
I’m wondering if would be better to add a new dimension to my matrices and use transfer or something similar to transform a complex one in a real one.
My produced netcdf should be compatible with xarray so I may check which kind of netcdf (or hdf5) file xarray is producing when one of its data array is complex.
I could read back in python the netcdf files with the imaginary and real parts separated but then I had to combine them back in python to work with them.
It works but it feels clumsy.
This is my main motivation for favouring either a compound type approach or like you say store an additional dimension. Having the real and imaginary parts separate might lead one to forget about it when reading the data
I asked myself a similar question a while ago when outputting quaternions (an extension of complex numbers) using HDF5.
Initially, I’ve created a compound type consisting of four named doubles. This means that N quaternions can be stored in an array of shape N. The advantage was that information about the data was hardcoded into the type. However, the disadvantage was more complicated processing which I do in Python. To avoid a special case for quaternions, I changed output to a 4D array, i.e. the output is now stored as an double array of shape (N,4). This is consistent with the way I store stress and strain tensors, namely as (N,3,3) arrays.
For quaternions, I would therefore recommend to store data always as floating point arrays of leading/trainling dimension 4. However, this is mainly due to the lack of build-in support for quaternions in any language that I’m aware of. In contrast, most languages support complex numbers. Whether it is convenient to use the build-in complex types depends on the capabilities of the NetCDF wrappers to the languages you are using. In any case, I would store the data together, either as a complex number or as two floating point numbers. In the latter case, adding metadata to inform users about the nature of the data would be important.
Thank you @MarDie . Could you please provide how you defined the compound type for quaternions. I am doing the processing of the data in python which as you mention supports complex numbers
The code is available on DAMASK/HDF5_utilities.f90 at bd78bf9d1d709797e47d3d4221c4db2a48556c44 · eisenforschung/DAMASK · GitHub but it is using HDF5, not NetCDF.