Using a trained neural network model in a fortran code

gradureau · March 23, 2023, 4:33pm

Hey everyone,

So I am working on a software written in fortran and my aim is to include machine learning models to this code.

so far I am thinking to train my models with python’s keras or Julia’s Flux library and I have seen that neural-fortran seems to be able to import a pretrained model saved in HDF5 and use it in a fortran code, which is exactly what I need. I have three questions concerning neural-fortran :

Yet, the github documentation (GitHub - modern-fortran/neural-fortran: A parallel framework for deep learning) doesn’t say how to compile neural-fortran with any compilers (I compile my project with mpif90, how do include neural-fortran in your project?). Could anyone help on this point?

Moreover this solution is great for keras models, but so far, I have seen that Julia’s Flux models are saved in the .bson format. Does anyone know if there would be a way to save a Flux pretrained model in HDF5, so that it could be compatible with neural-fortran?

Finally, neural-fortran doesn’t seem to take into account batchnormalisation layers. Is it going to be included in a future version of neural-fortran?

Thank you very much for your answers!

milancurcic · March 23, 2023, 8:24pm

Thanks @gradureau for your interest.

Can you open an issue in the repo describing your set-up and the issues that you encounter? Even an “I don’t know where to start is valid”, but it would be helpful to know more about the environment (compilers, OS, build system, etc.)

I’m not familiar with Flux, but I suppose bson → json (using some non-Fortran tool at least) should be straightforward, and from there writing an adapter for a model stored in json to neural-fortran model constructor should also be a data plumbing kind of work.

We have an issue open for it (Add batch normalization and Adam optimizer · Issue #114 · modern-fortran/neural-fortran · GitHub), so there’s at least some desire for it. Can you help us implement it?

rwmsu · March 23, 2023, 11:27pm

The Fortran-Keras Bridge project (GitHub - scientific-computing/FKB: A two-way deep learning bridge between Keras and Fortran) has/had minimal support for both a batch normalization and a dropout layer. Batch normalization was only supported for prediction using pre-trained models. No support for training in Fortran. In my most recent attempt to develop a NN model with Keras, batch normalization was the only technique that allowed me to get anything approaching a converged solution most of the time.

gradureau · March 24, 2023, 8:58am

Thank you @milancurcic

So I will use neural-fortran in a supercalculator with OS linux-gnu, CentOS Linux
release 7.8.2003 and mpif90 uses the compiler ifort 19.0.5.281. As for the build
system, I don’t know what it is. Could you give me some examples or explain me
how to search for the information?
Hum, does neural-fortran also read json files then?
Hum, yes, I could try to implement it. Yet I am mostly a machine learning user so I’ll
have to do this in my free time. I’ll look into this.

Sincerely
Gonzague Radureau

gradureau · March 24, 2023, 9:02am

Hum, very interesting, thank you very much @rwmsu !
Yet I then have the same questions as for neural-fortran :

How can you include this code into your fortran project?
Is it possible to use this code with Julia’s Flux pretrained models?

For the last question I am investigating on the possibility of saving Flux pretrained models in a keras compatible format. If you have any other options, I would be more then happy to hear them.

milancurcic · March 24, 2023, 2:54pm

How do you compile your supercalculator? Is it with CMake, Make, or perhaps a custom build script? If you can get your application to build with fpm, that would be the easiest way to add neural-fortran as a dependency.

Neural-fortran uses json-fortran to parse a JSON from a string attribute of a Keras HDF5 file. So JSON reading is accessible, but of course the code to map the JSON fields to an internal representation of the model would need to be written separately. Here’s an example for Keras:

github.com

modern-fortran/neural-fortran/blob/f328c8df55f7945242f0d340c8c04dccc6676042/src/nf/nf_network_submodule.f90#L85


      
            ! Loop over each layer in order and call the init methods.
            ! This will allocate the data internal to each layer (e.g. weights, biases)
            ! according to the size of the previous layer.
            do n = 2, size(res % layers)
              call res % layers(n) % init(res % layers(n - 1))
            end do
          
          end function network_from_layers
          
          
          module function network_from_keras(filename) result(res)
            character(*), intent(in) :: filename
            type(network) :: res
            type(keras_layer), allocatable :: keras_layers(:)
            type(layer), allocatable :: layers(:)
            character(:), allocatable :: layer_name
            character(:), allocatable :: object_name
            integer :: n
          
            keras_layers = get_keras_h5_layers(filename)

gradureau · March 28, 2023, 1:41pm

Hey @rwmsu ,

so I am trying to use FKB in my project, since it is lighter and has more options that are useful for me. Yet I am struggling with the use of convert_weights.py, since it is searching for a ‘model_config’ key in the HDF5 saved model and my version of tensorflow doesn’t produce this key. Which version of tensorflow is required for FKB? I am using the 2.21.0.

Thank you for your answers!
Gonzague Radureau

rwmsu · March 28, 2023, 2:03pm

Sorry. I haven’t actually used FKB, just took a look at what it could do and decided to use the Intel oneAPI AI toolkit implementation which I think is threaded (at least on Intel processors). It ran fast enough to handle all but the biggest cases I was trying to run in a reasonable length of time. Fortunately, I was able to get a small allocation on a cluster with several V100s so I ran the largest cases there.

milancurcic · March 28, 2023, 2:18pm

FKB was based on TensorFlow v1, before Keras became part of TF. There’s an open PR from 2021 that updates it to TF v2, however, it has not been reviewed or merged. You can try using the branch from that PR directly. Unfortunately, I don’t think FKB is maintained anymore. It was part of Jordan Ott’s PhD thesis work and I believe Jordan has since moved on to other work in the industry. Nevertheless, it would at this point probably be more trouble than worth it to try to use FKB. Most features that FKB brought to the table via glue scripts has been implemented in neural-fortran thanks to a NASA contract that asked for those features specifically. Plus, FKB uses an early version of neural-fortran that has since been significantly improved (correctness and compute performance-wise) in the upstram repo.

I think the shortest path to batchnorm layers in Fortran is for us to implement them together. It shouldn’t be too much trouble.

gradureau · March 28, 2023, 2:32pm

Hum, interesting, thank you very much ! Then as for neuran-fortran, I think I am understanding how to compile now. Using gfortran, I run the command

gfortran mycode.f90 -o mycode -Lpath/to/lib -Ipath/to/include -llibrary.

It seems to work on my computer. Yet I the calculation space I am working on, hdf5 isn’t installed, so I have to figure out a way to do this. Thank you very much for your help !

gradureau · April 4, 2023, 4:16pm

Hello everyone,

So some update on this. I am still blocked on the compilation of a code with neural-fortran. I am using gfortran 11.3.0 and I still use the command :

gfortran mycode.f90 -o mycode -Lpath/to/lib -Ipath/to/include -lneural

to compile my code. Lets say I compile the code simple.f90 with this command I get the following message :

/bin/ld: …/neural-fortran/build/lib//libneural.a(nf_keras_submodule.f90.o): warning: relocation against __json_value_module_MOD___vtab_json_value_module_Json_core' in read-only section .text’
/bin/ld : …/neural-fortran/build/lib//libneural.a(nf_io_hdf5_submodule.f90.o) : dans la fonction « __nf_io_hdf5_MOD_get_hdf5_dataset_real32_4d » :
nf_io_hdf5_submodule.f90:(.text+0x63) : référence indéfinie vers « __h5fortran_MOD___vtab_h5fortran_Hdf5_file »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x106) : référence indéfinie vers « __h5fortran_MOD_hdf_initialize »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x12a) : référence indéfinie vers « __h5fortran_MOD_hdf_get_shape »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x3f2) : référence indéfinie vers « __h5fortran_MOD_h5read_4d »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x40a) : référence indéfinie vers « __h5fortran_MOD_hdf_finalize »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x444) : référence indéfinie vers « __h5fortran_MOD___final_h5fortran_Hdf5_file »
/bin/ld : …/neural-fortran/build/lib//libneural.a(nf_io_hdf5_submodule.f90.o) : dans la fonction « __nf_io_hdf5_MOD_get_hdf5_dataset_real32_2d » :
nf_io_hdf5_submodule.f90:(.text+0x512) : référence indéfinie vers « __h5fortran_MOD___vtab_h5fortran_Hdf5_file »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x5b5) : référence indéfinie vers « __h5fortran_MOD_hdf_initialize »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x5d9) : référence indéfinie vers « __h5fortran_MOD_hdf_get_shape »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x791) : référence indéfinie vers « __h5fortran_MOD_h5read_2d »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0x7aa) : référence indéfinie vers « __h5fortran_MOD_hdf_finalize »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xa26) : référence indéfinie vers « __h5fortran_MOD___final_h5fortran_Hdf5_file »
/bin/ld : …/neural-fortran/build/lib//libneural.a(nf_io_hdf5_submodule.f90.o) : dans la fonction « __nf_io_hdf5_MOD_get_hdf5_dataset_real32_1d » :
nf_io_hdf5_submodule.f90:(.text+0xb75) : référence indéfinie vers « __h5fortran_MOD___vtab_h5fortran_Hdf5_file »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xc4f) : référence indéfinie vers « __h5fortran_MOD_hdf_initialize »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xc70) : référence indéfinie vers « __h5fortran_MOD_hdf_get_shape »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xd89) : référence indéfinie vers « __h5fortran_MOD_h5read_1d »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xda1) : référence indéfinie vers « __h5fortran_MOD_hdf_finalize »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xdd6) : référence indéfinie vers « __h5fortran_MOD___final_h5fortran_Hdf5_file »
/bin/ld : …/neural-fortran/build/lib//libneural.a(nf_io_hdf5_submodule.f90.o) : dans la fonction « __nf_io_hdf5_MOD_hdf5_attribute_string » :
nf_io_hdf5_submodule.f90:(.text+0xe64) : référence indéfinie vers « __h5global_MOD_h5f_acc_rdonly_f »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xe96) : référence indéfinie vers « __h5f_MOD_h5fopen_f »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xec3) : référence indéfinie vers « __h5a_MOD_h5aopen_by_name_f »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xeda) : référence indéfinie vers « __h5a_MOD_h5aget_type_f »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xef7) : référence indéfinie vers « __h5a_MOD_h5aread_ptr »
/bin/ld : nf_io_hdf5_submodule.f90:(.text+0xf0e) : référence indéfinie vers « __h5f_MOD_h5fclose_f »
/bin/ld : …/neural-fortran/build/lib//libneural.a(nf_keras_submodule.f90.o) : dans la fonction « __nf_keras_MOD_get_keras_h5_layers » :
nf_keras_submodule.f90:(.text+0x4) : référence indéfinie vers « __json_value_module_MOD___vtab_json_value_module_Json_core »
/bin/ld : nf_keras_submodule.f90:(.text+0x2c8) : référence indéfinie vers « __json_value_module_MOD_json_parse_string »
/bin/ld : nf_keras_submodule.f90:(.text+0x2fc) : référence indéfinie vers « __json_value_module_MOD_json_get_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x319) : référence indéfinie vers « __json_value_module_MOD_json_count »
/bin/ld : nf_keras_submodule.f90:(.text+0x46e) : référence indéfinie vers « __json_value_module_MOD_json_value_get_child_by_index »
/bin/ld : nf_keras_submodule.f90:(.text+0x4f6) : référence indéfinie vers « __json_value_module_MOD_json_get_string_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x527) : référence indéfinie vers « __json_value_module_MOD_json_get_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x5ab) : référence indéfinie vers « __json_value_module_MOD_json_get_string_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x668) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x8b6) : référence indéfinie vers « __json_value_module_MOD_json_value_destroy »
/bin/ld : nf_keras_submodule.f90:(.text+0x97c) : référence indéfinie vers « __json_value_module_MOD_json_get_integer_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0xb6c) : référence indéfinie vers « __json_value_module_MOD_json_get_string_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0xbc6) : référence indéfinie vers « __json_value_module_MOD_json_get_integer_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0xc59) : référence indéfinie vers « __json_value_module_MOD_json_get_integer_vec_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0xce0) : référence indéfinie vers « __json_value_module_MOD_json_get_string_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0xdc8) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x1006) : référence indéfinie vers « __json_value_module_MOD_json_get_integer_vec_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x10f5) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x11ad) : référence indéfinie vers « __json_value_module_MOD_json_get_integer_vec_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x1248) : référence indéfinie vers « __json_value_module_MOD_json_get_integer_vec_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x1339) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x160d) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x184c) : référence indéfinie vers « __json_value_module_MOD_json_get_real32_vec_by_path »
/bin/ld : nf_keras_submodule.f90:(.text+0x197c) : référence indéfinie vers « __functional_MOD_reverse_r4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x1c08) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x1cb0) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x1d50) : référence indéfinie vers « __functional_MOD_reverse_i4 »
/bin/ld : nf_keras_submodule.f90:(.text+0x215b) : référence indéfinie vers « __json_value_module_MOD_json_value_destroy »
/bin/ld: warning: creating DT_TEXTREL in a PIE
collect2: error: ld returned 1 exit status

It seems that it has trouble using the submodules. Could anyone help me with this?

milancurcic · April 4, 2023, 4:23pm

Hi Gonzague,

I think the issue is that you only ask the linker to link against neural-fortran (-lneural), but you also need to link against all dependencies (HDF5, h5fortran, json-fortran, functional-fortran).

I suggest trying fpm which takes care of all the tedious compilation and linking details for you.

gradureau · April 5, 2023, 8:49am

Hey,

so I have tried to run the following command compiling with cmake :

gfortran simple.f90 -o simple -Ipath/to/include -Lpath/to/lib -lneural -lfunctional -lh5fortran -ljsonfortran

then I ran this command compiling with fpm :

gfortran simple.f90 -o simple -Ipath/to/include -Lpath/to/lib -lneural-fortran

I don’t have any errors anymore, comming from jsonfortran and functional, but I still get errors from h5fortran.

When I compile with cmake, no .mod is generated in the include for h5fortran.

To sum up all of my remaining issues seem to come from the h5fortran library. What could I do to solve this issue? Could it be due to the gfortran version I am using? (gfortran 11.3.0 > gfortran 9.4.0 tested according to the github)

rwmsu · April 5, 2023, 11:19am

Are your h5fortran issues related to not including the proper HDF5 libs in your link statement or are they errors with h5fortran.

Did you include the links to the actual hdf5 libraries, ie

-L$(HDF5LIBPATH) -lhdf5 -lhdf5_fortran -lhdf5_hl -lhdf5hl_fortran

In the process of trying to get neural-fortran to compile and run on a Cray with Cray’s compilers along with gfortran and ifort, I had issues with h5fortran’s use of unlimited polymorphic values and select type with one compiler (don’t remember which) so I just created my own set of hdf5 utitlities (based on h5fortran) but replaced the UP logic with type specific functions and used a generic interface. If you look at the hdf5 routines actually used in neural-fortran there are only a relative few so bringing in all of h5fortran is a bit of overkill. The same with functional because only one function (reverse) is used.

I’ve only played around a little with fpm but I second Milan’s suggestion you try that instead of cmake.

gradureau · April 5, 2023, 11:55am

Hey @rwmsu ,

Thank you so much, it works now with the fpm compilation of neural-fortran.
Yet it doesn’t with the cmake compilation. So far I’ve been
tying both, since I’m planing to use this code in the computation space of my institute and they have cmake, but not fpm. I’ll have to try using the binary of fpm in this computation space to make this compilation work.

Anyways thank you very much for the help. For those who may be interested by the solution, here is the command line I use to compile a code with neural-fortran

gfortran mycode.f90 -o mycode -Ipath/to/nf/include -Ipath/to/hdf5/include -Lpath/to/nf/lib -Lpath/to/hdf5/lib -lneural-fortran -lhdf5 -lhdf5_fortran -lhdf5_hl -lhdf5hl_fortran

I compile neural-fortran with the fpm command.

gradureau · April 25, 2023, 9:40am

Hello everyone,

So on my personal computer everything works great, but not
on the computing space of the institute I’m working in.

On my computer I compiled everything with fpm and gfortran 11.3.0
and hdf5 1.10.7.

Now on the computing space, I compile everything with fpm using
ifort 2021.3.0 20210609 and hdf5 1.12.1. Here is the command line :

fpm build --profile release --flag “-fno-frontend-optimize -I$HDF5INC -L$HDF5LIB” --compiler ifort

Everything compiles until the point where the examples get compiled. I then get the
errors :

ifort: command line warning #10006: ignoring unknown option ‘-fno-frontend-optimize’
build/ifort_1D32607880784D78/neural-fortran/libneural-fortran.a(src_nf_nf_network_submodule.f90.o) : In the function « nf_network_mp_train_ » :
nf_network_submodule.f90:(.text+0x900b) : undefined reference to « for_rtl_ICAF_BROADCAST_DESCR »
nf_network_submodule.f90:(.text+0x904e) : undefined reference to « for_rtl_ICAF_BROADCAST_DESCR »
build/ifort_1D32607880784D78/neural-fortran/libneural-fortran.a(src_nf_nf_dense_layer_submodule.f90.o) : In the function « nf_dense_layer_mp_update_ » :
nf_dense_layer_submodule.f90:(.text+0x24c8) : undefined reference to « for_rtl_ICAF_BROADCAST_OP_DESCR »
nf_dense_layer_submodule.f90:(.text+0x251c) : undefined reference to « for_rtl_ICAF_BROADCAST_OP_DESCR »
build/ifort_1D32607880784D78/neural-fortran/libneural-fortran.a(src_nf_nf_dense_layer_submodule.f90.o) : In the function « nf_dense_layer_mp_init_ » :
nf_dense_layer_submodule.f90:(.text+0x3077) : undefined reference to « for_rtl_ICAF_BROADCAST_DESCR »
build/ifort_1D32607880784D78/neural-fortran/libneural-fortran.a(src_nf_nf_conv2d_layer_submodule.f90.o) : In the function « nf_conv2d_layer_mp_update_ » :
nf_conv2d_layer_submodule.f90:(.text+0x43af) : undefined reference to « for_rtl_ICAF_BROADCAST_OP_DESCR »
nf_conv2d_layer_submodule.f90:(.text+0x440d) : undefined reference to « for_rtl_ICAF_BROADCAST_OP_DESCR »
Compilation failed for object " simple "
stopping due to failed compilation
STOP 1

Is there a way to overcome this issue?

milancurcic · April 25, 2023, 11:10am

What happens if you disable coarrays via compiler flag? I don’t recall the exact syntax from the top of my head, but I think the neural-fortran README has examples for compiling with ifort in serial mode.

-fno-frontend-optimize is a gfortran option. Don’t use it when compiling with ifort.

gradureau · April 25, 2023, 1:25pm

Yes, indeed it works if I disable the coarrays. Here is the command line I enter
to compile neural-fortran :

fpm build --profile release --flag “-I$HDF5INC -L$HDF5LIB -coarray=single” --compiler ifort

Now for instance if I want to compile individually simple.f90 with ifort, I have to enter the command :

ifort simple.f90 -o simple -I$NFINC -I$HDF5INC -L$NFLIB -L$HDF5LIB -lneural-fortran -lhdf5 -lhdf5_fortran -lhdf5_hl -lhdf5hl_fortran -coarray=single

Thank you very much for the help. I should now be able to use neural fortran and contribute if needed.

gradureau · April 25, 2023, 2:47pm

Ok, I spoke too fast. The compilation works, but the executables don’t
do anything. From what I saw on the internet, this should be
due to memory issues, which shouldn’t be the case for me.

So I didn’t see any examples in the README about compiling
neural-fortran with intel in fpm. How would I need to do that?

milancurcic · April 25, 2023, 6:53pm

Can you post the program and the output?

A few of the included examples segfault with recent Intel OneAPI versions (2021 and later I think) but I haven’t looked yet why that is.

Topic		Replies	Views
Fortran For Building ML Models Tutorials	5	1082	July 12, 2023
Python interoperability with Fortran	14	2233	April 15, 2021
Fortran and Neural Networks	17	6641	November 14, 2021
Help, Convert from python to Fortran	7	882	September 25, 2023
Fortran frameworks/Libraries for Robotics/Artificial Intelligence Help	4	1453	October 7, 2020

Using a trained neural network model in a fortran code

Related topics