# ONNX - Library/vendor agnostic API for inferencing DL models **URL:** \ **Category:** Help\ **Created:** [February 7, 2023, 6:38pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147 "2023-02-07T18:38:15Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [February 7, 2023, 6:38pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/1 "2023-02-07T18:38:16Z")
Hi, is anybody aware if there is any work on creating an API for [ONNX](https://onnx.ai/) in Fortran? I saw that PyTorch can default to ONNX for saving any kind of models, and Keras seems to have already some solutions as well, this enables moving models from torch to keras quite easily [see this blog](https://medium.com/analytics-vidhya/pytorch-to-keras-using-onnx-71d98258ad76) If I saw correctly the core libraries are in C, and wrappers for other languages are [already in place](https://onnxruntime.ai/docs/api/). ---
**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\ **Post date:** [February 7, 2023, 7:15pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/2 "2023-02-07T19:15:58Z")
As I understand it, an ONNX “record” (for my ignorance of the correct terminology here) is a protobuf message, so a Fortran reader or writer would boil down to bindings to the [protobuf C library](https://github.com/protobuf-c/protobuf-c). ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [February 7, 2023, 7:36pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/3 "2023-02-07T19:36:15Z")
> [@milancurcic](#): > > to bindings to the [protobuf C library](https://github.com/protobuf-c/protobuf-c) This is something completely new to me, have to sit and read on that one 😃 So when you say “binding the protobuf C library” do you mean enabling to reconstruct the graph and data used to build the model stored in the file? In that case it would be necessary to reconstruct the layers in order to do an actual inference, no? for instance, mapping it to neural-fortran’s layers to do the actual evaluation. From what I understood in the onnx page they already put at disposal the runtimes to evaluate the models. I guess several approaches can be taken here 🤔 ---
**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\ **Post date:** [February 7, 2023, 8:00pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/4 "2023-02-07T20:00:06Z")
Yes, I think there are two parts to this: 1. Parsing the ONNX format (protobuf under the hood); 2. Constructing the layers on the Fortran ML engine end. Reading protobuf in Fortran would only lead toward item 1. Layers would still need to be constructed on the Fortran ML framework end. It’s only the ONNX format part that’s common to frameworks that implement it. Constructing the internal data structures that do the inference are still specific to each framework. ---
**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\ **Post date:** [February 7, 2023, 8:34pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/5 "2023-02-07T20:34:23Z")
A protobuf Fortran library would be useful for other applications as well… ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [February 8, 2023, 8:18am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/6 "2023-02-08T08:18:37Z")
I’ll give it a try to create a Fortran wrapper, to see how far can I get. Could you advise me on a good tool for kick-off a semi-automatic wrapping creation? I got the onnx library which contains the include and lib folders (there is a static lib and the dynamic lib), in the include folder there are several .h ( onnxruntime\_c\_api.h, onnxruntime\_cxx\_api.h, onnxruntime\_cxx\_inline.h, etc ). Since the cxx headers are “thin wrappers” to the C api as they say in the web site, I guess the best would be to wrap directly based on this one “onnxruntime\_c\_api.h”. I was reading about [shroud](https://shroud.readthedocs.io/en/develop/introduction.html) and it seems quite interesting to facilitate creating the necessary interfaces. Would you advise for something different? (My end goal would be to link to either an executable or dynamic library created mainly with Fortran) ---
**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\ **Post date:** [February 8, 2023, 5:45pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/7 "2023-02-08T17:45:03Z")
I don’t have experience auto-generating wrappers. I’ve only used such libraries as a consumer (generated with SWIG). My suggestion would be to start small (pick a minimal ONNX model file), and only implement functions needed to read and parse the features in it. This should help to better understand how the process works, and you could then research automating the rest if there are two many to do by hand. ---
**Author:** ![rouson](https://avatars.discourse-cdn.com/v4/letter/r/ce7236/32.png) [@rouson](https://fortran-lang.discourse.group/u/rouson)\ **Post date:** [February 21, 2023, 11:42pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/8 "2023-02-21T23:42:32Z")
The h2m tool developed by an intern working with Dan Nagle, the former chair of the US portion of the Fortran standard committee, auto-generates Fortran modules from C header files: the repository is [here](https://github.com/dan-nagle/h2m) and a paper about h2m is [here](https://dl.acm.org/doi/pdf/10.1145/3266145.3266147). ---
**Author:** ![rouson](https://avatars.discourse-cdn.com/v4/letter/r/ce7236/32.png) [@rouson](https://fortran-lang.discourse.group/u/rouson)\ **Post date:** [February 21, 2023, 11:49pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/9 "2023-02-21T23:49:52Z")
Another tool that auto-generates Fortran interface bodies from C or C++ is the SWIG-Fortran extension to SWIG. There’s a paper about it [here](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8745480) and a corresponding fork of SWIG [here](https://github.com/swig-fortran/swig). This SWIG Fortran repository has more recent commits than h2m, but I suspect h2m might be more lightweight and simple because it focuses strictly on C without the considerable complexities of C++. ---
**Author:** ![FortranFan](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@FortranFan](https://fortran-lang.discourse.group/u/FortranFan)\ **Post date:** [February 22, 2023, 3:16am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/10 "2023-02-22T03:16:48Z")
> [@rouson](#): > > The h2m tool … auto-generates Fortran modules from C header files … @rouson, perhaps you can influence the Fortran 202Y worklist to extend to further enhanced interoperability with a C companion processor such that certain sections of C code (type definitions, function prototypes, even certain C functions) can be directly “imported” into Fortran source - why not?!! This can possibly be a use case for the current initiative to consider standard preprocessing as part of Fortean 202Y. This can obviate the need for such tools! ---
**Author:** ![rouson](https://avatars.discourse-cdn.com/v4/letter/r/ce7236/32.png) [@rouson](https://fortran-lang.discourse.group/u/rouson)\ **Post date:** [February 22, 2023, 4:35am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/11 "2023-02-22T04:35:15Z")
@FortranFan I like that idea a lot in principle, but I wonder if it is workable in practice. Are there examples of other languages that have done something similar? I haven’t actually used either of the tools that I mentioned. I’ve always wondered how the tools deal with certain inherent ambiguities in any language translation. For example, does the tool translate a C `float**` into a Fortran `type(c_ptr)` or a Fortran 2D `real` array? I assume the tools have some set of assumptions they make or some way for users to influence the option choice. Were this to become standardized, I imagine that unambiguous decisions would have to be hardwired into the standard and the choice might not match what many users want. ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [February 23, 2023, 7:32am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/12 "2023-02-23T07:32:19Z")
Thank you @rouson for the tip, I did not know h2m, will definitely take a look!! This > [@FortranFan](#): > > can be directly “imported” into Fortran source - why not?!! and this > [@rouson](#): > > does the tool translate a C `float**` into a Fortran `type(c_ptr)` or a Fortran 2D `real` array? Got me thinking, wouldn’t it a direct interoperability with C++ facilitate some boilerplate? I’m thinking that for instance if one could map (in a standardized way), lets say an allocatable array into a std::vector, that could bring some clarity to what today it seems to me, an obfuscating step of using type(c\_ptr). It is very practical indeed, but I feel that inter-operating with type(c\_ptr) just goes against the adage “explicit is better than inplicit”. (this might be a very naive idea) If I have a class with an allocatable array as a member, in order to map it as a struct, both the class and the array, should have a wrapper declaration with type(c\_ptr)… not very fond of that but it does get the job done… ---
**Author:** ![rouson](https://avatars.discourse-cdn.com/v4/letter/r/ce7236/32.png) [@rouson](https://fortran-lang.discourse.group/u/rouson)\ **Post date:** [February 27, 2023, 10:10am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/13 "2023-02-27T10:10:05Z")
@FortranFan some discussion of C++ interoperability came up in a side conversation during last week’s J3 meeting. I don’t think any such capability is likely to happen for the foreseeable future. The complexities are truly daunting. Starting with the simple issue we’re discussing, how would one construct the array descriptor? A `float**` is just a memory address. Sure the associated pointer can be used to index through an array, but in the case that the associated memory was allocated by C++, how does the compiler discover the shape? Would it be `allocatable` or `pointer`? I imagine it would have to be a `pointer` because otherwise, Fortran would be in charge of deallocating memory that was allocated by C++. If it’s a `pointer`, would Fortran be allowed to reallocate the pointer? Hopefully not, but how will that be prevented? We can easily run into all the usual pitfalls of dangling pointers and memory leaks and such. In addition to these ambiguities, interfacing Fortran with templated C++, which by some definitions is all of modern C++, is a nightmare. I worked on one project where one of the team members wrote upwards of 40K lines of Perl scripts just for generating Fortran interfaces corresponding to templated C++ header files. That project eventually collapsed under its own weight and was later reincarnated by instead contributing better modern Fortran support to the SWIG language-interoperability tool (see the link below). It seemed like good work, but I don’t think it ever got merged into the main branch of SWIG. I suspect the issues involved are just to challenging to be practical. > **[Automated Fortran–C++ Bindings for Large-Scale Scientific Applications](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8745480&casa_token=G-Vx-B2PgN0AAAAA:pv5u8bZ9UdcIIuLM69JuWDKQvMzh9ulhy3xaIsw1ygbypx_NdvsBWS_ESTHz_CzUiwcuEM2paj_G&tag=1)** > > Although many active scientific codes use modern Fortran, most contemporary scientific software libraries are implemented in C and C++. Providing their numerical, algorithmic, or data management features to Fortran codes requires writing and... ---
**Author:** ![rouson](https://avatars.discourse-cdn.com/v4/letter/r/ce7236/32.png) [@rouson](https://fortran-lang.discourse.group/u/rouson)\ **Post date:** [February 27, 2023, 10:13am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/14 "2023-02-27T10:13:02Z")
Worse yet might be when we introduce templates in Fortran. Then we have a combinatorial explosion of cases to handle in both directions of the interaction between the two languages. With shock waves traveling in both directions, maybe it could be called a combinatorial implosion. 🙂 ---
**Author:** ![SiggyF](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/siggyf/32/4375_2.png) [@SiggyF](https://fortran-lang.discourse.group/u/SiggyF)\ **Post date:** [February 27, 2024, 9:48am UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/15 "2024-02-27T09:48:22Z")
@hkvzjal did you make any progress with the onnx binding? I also need to do inference on a pretrained model. I tried the neural-fortran, which is quite elegant, but does not support all the layers that I need. I’m now trying to use the onnx c-api for inference. ---
**Author:** ![rbitr](https://avatars.discourse-cdn.com/v4/letter/r/f1d935/32.png) [@rbitr](https://fortran-lang.discourse.group/u/rbitr)\ **Post date:** [February 27, 2024, 12:10pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/16 "2024-02-27T12:10:00Z")
@SiggyF what kind of model are you working with? I’m working on a pure fortran transformer inference “framework”, although it’s a bit of a mess right now. If you’re doing something with language models I’d be interested to hear about it. ---
**Author:** ![milancurcic](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/milancurcic/32/2_2.png) [@milancurcic](https://fortran-lang.discourse.group/u/milancurcic)\ **Post date:** [February 27, 2024, 5:30pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/17 "2024-02-27T17:30:10Z")
We’ve submitted a [proposal](https://github.com/ml4fortran/sc24) for an ML for Fortran workshop at SC24. If it’s accepted I’d love to see your transformer framework there. ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [February 27, 2024, 6:32pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/18 "2024-02-27T18:32:34Z")
Not really, I expended some time trying to understand the C API but didn’t have enough to push all the way, at the end I found these Fortran bindings to libtorch [GitHub - Cambridge-ICCS/FTorch: A library for directly calling PyTorch ML models from Fortran.](https://github.com/Cambridge-ICCS/FTorch) that are rather easy to link and use, so I went with that as I simultaneously test native Fortran implementations. ---
**Author:** ![rbitr](https://avatars.discourse-cdn.com/v4/letter/r/f1d935/32.png) [@rbitr](https://fortran-lang.discourse.group/u/rbitr)\ **Post date:** [February 27, 2024, 8:02pm UTC](https://fortran-lang.discourse.group/t/onnx-library-vendor-agnostic-api-for-inferencing-dl-models/5147/19 "2024-02-27T20:02:48Z")
Yes, definitely interested.