Hi, is anybody aware if there is any work on creating an API for ONNX in Fortran?
I saw that PyTorch can default to ONNX for saving any kind of models, and Keras seems to have already some solutions as well, this enables moving models from torch to keras quite easily see this blog
If I saw correctly the core libraries are in C, and wrappers for other languages are already in place.
As I understand it, an ONNX “record” (for my ignorance of the correct terminology here) is a protobuf message, so a Fortran reader or writer would boil down to bindings to the protobuf C library.
This is something completely new to me, have to sit and read on that one 
So when you say “binding the protobuf C library” do you mean enabling to reconstruct the graph and data used to build the model stored in the file? In that case it would be necessary to reconstruct the layers in order to do an actual inference, no? for instance, mapping it to neural-fortran’s layers to do the actual evaluation. From what I understood in the onnx page they already put at disposal the runtimes to evaluate the models. I guess several approaches can be taken here 
Yes, I think there are two parts to this:
Reading protobuf in Fortran would only lead toward item 1. Layers would still need to be constructed on the Fortran ML framework end.
It’s only the ONNX format part that’s common to frameworks that implement it. Constructing the internal data structures that do the inference are still specific to each framework.
A protobuf Fortran library would be useful for other applications as well…
I’ll give it a try to create a Fortran wrapper, to see how far can I get. Could you advise me on a good tool for kick-off a semi-automatic wrapping creation?
I got the onnx library which contains the include and lib folders (there is a static lib and the dynamic lib), in the include folder there are several .h ( onnxruntime_c_api.h, onnxruntime_cxx_api.h, onnxruntime_cxx_inline.h, etc ). Since the cxx headers are “thin wrappers” to the C api as they say in the web site, I guess the best would be to wrap directly based on this one “onnxruntime_c_api.h”.
I was reading about shroud and it seems quite interesting to facilitate creating the necessary interfaces. Would you advise for something different? (My end goal would be to link to either an executable or dynamic library created mainly with Fortran)
I don’t have experience auto-generating wrappers. I’ve only used such libraries as a consumer (generated with SWIG).
My suggestion would be to start small (pick a minimal ONNX model file), and only implement functions needed to read and parse the features in it. This should help to better understand how the process works, and you could then research automating the rest if there are two many to do by hand.
The h2m tool developed by an intern working with Dan Nagle, the former chair of the US portion of the Fortran standard committee, auto-generates Fortran modules from C header files: the repository is here and a paper about h2m is here.
Another tool that auto-generates Fortran interface bodies from C or C++ is the SWIG-Fortran extension to SWIG. There’s a paper about it here and a corresponding fork of SWIG here. This SWIG Fortran repository has more recent commits than h2m, but I suspect h2m might be more lightweight and simple because it focuses strictly on C without the considerable complexities of C++.
@rouson, perhaps you can influence the Fortran 202Y worklist to extend to further enhanced interoperability with a C companion processor such that certain sections of C code (type definitions, function prototypes, even certain C functions) can be directly “imported” into Fortran source - why not?!! This can possibly be a use case for the current initiative to consider standard preprocessing as part of Fortean 202Y. This can obviate the need for such tools!
@FortranFan I like that idea a lot in principle, but I wonder if it is workable in practice. Are there examples of other languages that have done something similar? I haven’t actually used either of the tools that I mentioned. I’ve always wondered how the tools deal with certain inherent ambiguities in any language translation. For example, does the tool translate a C float** into a Fortran type(c_ptr) or a Fortran 2D real array? I assume the tools have some set of assumptions they make or some way for users to influence the option choice. Were this to become standardized, I imagine that unambiguous decisions would have to be hardwired into the standard and the choice might not match what many users want.
Thank you @rouson for the tip, I did not know h2m, will definitely take a look!!
This
and this
Got me thinking, wouldn’t it a direct interoperability with C++ facilitate some boilerplate? I’m thinking that for instance if one could map (in a standardized way), lets say an allocatable array into a std::vector, that could bring some clarity to what today it seems to me, an obfuscating step of using type(c_ptr). It is very practical indeed, but I feel that inter-operating with type(c_ptr) just goes against the adage “explicit is better than inplicit”. (this might be a very naive idea)
If I have a class with an allocatable array as a member, in order to map it as a struct, both the class and the array, should have a wrapper declaration with type(c_ptr)… not very fond of that but it does get the job done…
@FortranFan some discussion of C++ interoperability came up in a side conversation during last week’s J3 meeting. I don’t think any such capability is likely to happen for the foreseeable future. The complexities are truly daunting. Starting with the simple issue we’re discussing, how would one construct the array descriptor? A float** is just a memory address. Sure the associated pointer can be used to index through an array, but in the case that the associated memory was allocated by C++, how does the compiler discover the shape? Would it be allocatable or pointer? I imagine it would have to be a pointer because otherwise, Fortran would be in charge of deallocating memory that was allocated by C++. If it’s a pointer, would Fortran be allowed to reallocate the pointer? Hopefully not, but how will that be prevented? We can easily run into all the usual pitfalls of dangling pointers and memory leaks and such.
In addition to these ambiguities, interfacing Fortran with templated C++, which by some definitions is all of modern C++, is a nightmare. I worked on one project where one of the team members wrote upwards of 40K lines of Perl scripts just for generating Fortran interfaces corresponding to templated C++ header files. That project eventually collapsed under its own weight and was later reincarnated by instead contributing better modern Fortran support to the SWIG language-interoperability tool (see the link below). It seemed like good work, but I don’t think it ever got merged into the main branch of SWIG. I suspect the issues involved are just to challenging to be practical.
Worse yet might be when we introduce templates in Fortran. Then we have a combinatorial explosion of cases to handle in both directions of the interaction between the two languages. With shock waves traveling in both directions, maybe it could be called a combinatorial implosion. 
@hkvzjal did you make any progress with the onnx binding?
I also need to do inference on a pretrained model. I tried the neural-fortran, which is quite elegant, but does not support all the layers that I need. I’m now trying to use the onnx c-api for inference.
@SiggyF what kind of model are you working with? I’m working on a pure fortran transformer inference “framework”, although it’s a bit of a mess right now. If you’re doing something with language models I’d be interested to hear about it.
We’ve submitted a proposal for an ML for Fortran workshop at SC24. If it’s accepted I’d love to see your transformer framework there.
Not really, I expended some time trying to understand the C API but didn’t have enough to push all the way, at the end I found these Fortran bindings to libtorch GitHub - Cambridge-ICCS/FTorch: A library for directly calling PyTorch ML models from Fortran. that are rather easy to link and use, so I went with that as I simultaneously test native Fortran implementations.
Yes, definitely interested.