# Types from Fortran to Python via Opaque Pointers **URL:** \ **Category:** Uncategorized\ **Created:** [November 13, 2023, 3:15pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782 "2023-11-13T15:15:41Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 13, 2023, 3:15pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/1 "2023-11-13T15:15:41Z")
Thought this might be of interest to some people here. > **[Types from Fortran to Python via Opaque Pointers](https://rgoswami.me/posts/types-fortran-python-opaque/)** > > An exploration of the opaque object and pointer interface approach to unleash more modern features than those offered by the ISO Fortran standard. > Background Since this has been covered a few times before, just some quick pointers1. The ultimate... There’s nothing on the Fortran / C side which hasn’t been discussed here already, but the `PyCapsule` approach is new (AFAIK). ---
**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\ **Post date:** [November 13, 2023, 11:42pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/2 "2023-11-13T23:42:06Z")
All the C code is gibberish to me, but the basic idea of using pointers to access complex Fortran types from Python is the same idea as I showed in my 2018 paper: [Application of Modern Fortran to Spacecraft Trajectory Design and Optimization](https://ntrs.nasa.gov/api/citations/20180000413/downloads/20180000413.pdf). See Fig. 20. It’s a good technique, and you don’t need `f2py` (or `swig` or anything extra) if you just write the wrappers yourself. ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 13, 2023, 11:56pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/3 "2023-11-13T23:56:36Z")
Thanks for the reference, will keep it in mind for my next post 🙂 . Generally the idea of using pointers for interfaces is very very old, back to `swig` and `f2py` as well, since (e.g.) `f2py` generates wrappers to interact with modules and functions (i.e. it “flattens” them to subroutines). Personally though, I never have time to write or maintain hand-crafted wrappers (across operating systems), so`f2py` support is crucial for my libraries (like this [Gauss-Jacobi Quadrature code](https://github.com/HaoZeke/GaussJacobiQuad)) and this is the general attitude of most people at the intersection of Python, C++ and Fortran I’d guess, so the challenges are to: - Generate bindings with as little user interaction as possible. - Interface to native Python classes instead of the (pretty awkward) `ctypes` library (which generally requires a layer on-top to be palpable). The importance of these points cannot be overstated, the whole `scipy` debacle is in part because no one wants to write / maintain an interface to Python-C. Also I’ve not benchmarked it yet, but I suspect the overhead of `ctypes` is more than that of direct `Python-C-Fortran` bindings. Some other historical examples include: - M. G. Gray, R. M. Roberts, and T. M. Evans, “Shadow-object interface between Fortran 95 and C++,” _Computing in Science Engineering_, vol. 1, no. 2, pp. 63–70, Mar. 1999, doi: [10.1109/5992.753048](https://doi.org/10.1109/5992.753048). - R. Bader, “A Fortran binding for the GNU scientific library,” _SIGPLAN Fortran Forum_, vol. 26, no. 2, pp. 4–11, Aug. 2007, doi: [10.1145/1279941.1279942](https://doi.org/10.1145/1279941.1279942). - A. Pletzer, D. McCune, S. Muszala, S. Vadlamani, and S. Kruger, “Exposing Fortran Derived Types to C and Other Languages,” _Computing in Science Engineering_, vol. 10, no. 4, pp. 86–92, Jul. 2008, doi: [10.1109/MCSE.2008.94](https://doi.org/10.1109/MCSE.2008.94). - V. K. Decyk, “A method for passing data between C and Opaque Fortran 90 pointers,” _SIGPLAN Fortran Forum_, vol. 27, no. 2, pp. 2–7, Aug. 2008, doi: [10.1145/1408643.1408644](https://doi.org/10.1145/1408643.1408644). And a bunch of others, though none discuss the binding to Python, which is less trivial than one might expect (and only handled in an automated, maintained manner by `f2py`). ---
**Author:** ![rfarmer](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@rfarmer](https://fortran-lang.discourse.group/u/rfarmer)\ **Post date:** [November 14, 2023, 10:58am UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/4 "2023-11-14T10:58:35Z")
I will point out that f2py is not the only automated set of bindings out ther. My library [GitHub - rjfarmer/gfort2py: Library to allow calling fortran code from python](https://github.com/rjfarmer/gfort2py) automates the creation of the bindings between fortran and python. As long as your code is in a module then you need to make no changes to your code. ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 6:14pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/5 "2023-11-14T18:14:49Z")
Thanks, that is a nice intiative. Some thoughts (just from a quick look) - I [discussed](https://rgoswami.me/posts/cython-derivedtype-f2py/) and [discarded](https://rgoswami.me/posts/fortran-oop-python/) dictionaries as an interface (wasn’t considered pythonic enough) at one point but the more the merrier. - Have you tried compiling SciPy’s fortran code / the F2PY test suite? - GFortran only as a dependency is a pretty severe restriction (and I personally dislike `ctypes`, not portable at all, makes for ugly boilerplate calls) Feel free to join the NumPy slack’s `f2py` channel if you’d like to discuss these more as well 🙂 ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [November 14, 2023, 6:25pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/6 "2023-11-14T18:25:36Z")
> [@rgoswami](#): > > I personally dislike `ctypes`, not portable at all Question, when you say “not portable” what do you mean? Honestly, ctypes is the only approach I have found to enable using which ever compiler I want to build the shared library, and have this shared library being **independent** of the python version. If what you don’t like is the maintenance of the intermediate wrappers that’s one thing and yes that’s true, but for the rest I’m not getting the **non-portability** ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 6:33pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/7 "2023-11-14T18:33:51Z")
Perhaps I am mistaken but it seemed [from the documentation](https://docs.python.org/3/library/ctypes.html#accessing-functions-from-loaded-dlls) that one would have to have different calls for different operating systems. To me this says “not portable” though I suppose it could be seen also as maintenance burden. ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 6:56pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/8 "2023-11-14T18:56:13Z")
In general, I don’t really see how `ctypes` is a good idea, **yes** if you know exactly what compiler you used and what arguments and ensure compatibility you can make a call, but this is pretty much the same as glorified object renaming (see @sblionel’s very valid argument against [that here](https://fortran-lang.discourse.group/t/exploring-iso-c-binding-and-type-bound-procedures/2447/7)) Sure the Python libraries `ctypes` and `np.ctypes` help a bit, make the calls slightly less uglier than mangling symbols and making calls by hand, but it is still essentially the same approach with the same drawbacks. Works for personal projects where users are expected to “read the documentation” and compile accordingly for functionality… Not so useful as a basis for a generic tool. ---
**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\ **Post date:** [November 14, 2023, 8:13pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/9 "2023-11-14T20:13:57Z")
I also don’t understand what you are saying here. Once you define the C-compatible interface using the C-interoperability features on the Fortran side, you can call it just like any other C routine. It doesn’t matter the compiler or operating system. It’s way less burden than injecting another tool into build or run time. Maybe there is some misscommunication here… ---
**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\ **Post date:** [November 14, 2023, 8:16pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/10 "2023-11-14T20:16:24Z")
Yes, exactly. I have found the same. Build the shared library (with C-compatible interfaces) and that’s all you need. To me that is way easier and more portable. (vs trying to fight SWIG to get it to do what you want and then deal with extra platform and/or python-specific libs floating around. i’ve never used f2py). ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 8:27pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/11 "2023-11-14T20:27:51Z")
> [@jacobwilliams](#): > > It’s way less burden than injecting another tool into build or run time. I think we’re talking at cross purposes though, as can be seen in SciPy today (and the past 20+ years) the dominant way to interface standard Fortran libraries with Python has been via additional tools because no one has written / maintained wrappers to the underlying codes. > i’ve never used f2py As a quick primer, what it does is: - Generate fortran code (for compatibility) - Writes out C code interfacing to the fortran code - Importantly writes out Python-C code Finally the resulting fortran files (including the wrappers) and the C code (which links to Python at compile time) is linked into a nice executable importable module. This is fully automated as in the example above. For example, without having to know much about Fortran or Python you can [pass arrays or strings or callback functions](https://numpy.org/doc/stable/f2py/f2py-examples.html) to and from both languages via `f2py`. If you have the luxury of `bind(c)` types you can [generate direct interfaces](https://github.com/HaoZeke/GaussJacobiQuad), tested on all three operating systems without handling `.dll` files specially. Certainly one could argue that everyone should write their own wrappers with well designed APIs exposing exactly the functionality they need. I don’t think the general attitude that `ctypes` does the same thing is helpful in the least for **general** binding tools which can and are in use for production Python code. > Build the shared library (with C-compatible interfaces) and that’s all you need. Generally, python users would be interested in something which you can grab from PyPI. I’m not sure this would work for distributing wheels, but the `f2py` compiled modules certainly do (e.g. SciPy). As noted above, if `ctypes` works for you, it works. However, from the perspective of “will this help people actually keep Fortran in Python” I don’t see it helping at all. Plus there’s a nice segfault when there’s a failure in `ctypes` which is never fun (noted by @certik). Perhaps the closest argument is that with `bind(c)` ensuring uniformity of calls, there is no real benefit (beyond needing to write things by hand) compared to `ctypes` (assuming that it is trivial to ensure that the `dll` or `so` is in the right place at the right time when called from `python`). However, `f2py` supports way more things than `bind(c)`, and also works for older pre-`bind(c)` code as well. It also has nice helpers for `numpy` arrays in particular. ---
**Author:** ![hkvzjal](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/hkvzjal/32/3055_2.png) [@hkvzjal](https://fortran-lang.discourse.group/u/hkvzjal)\ **Post date:** [November 14, 2023, 8:54pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/12 "2023-11-14T20:54:26Z")
So, just to be clear, I have no dogma here, just practical experience. When I first found out about f2py I was very happy because I thought it would help me create a python API out of kernels and libraries I maintain for other stuff. But no, f2py (as I understood) was basically behaving as a “build-system” in its own, an imposing 2 key restrictions that I found were a **no-go** : 1) An **upstream dependency** of my kernels (which are a mix of Fortran/C/C++) to the given python version and 2) An **upstream dependency** to a subset of Fortran standards… I guess in summary the problem for my use case is that I’m not writing Fortran code as backend for python, but actually for full-fledge software, the python API is just a subset. The 2nd point is kind of also true for iso\_c\_binding, one can not directly bind a derived type which contains allocatables or pointers or other derived types as members… but if I already have to circumvent that for the iso\_c\_binding interoperability or for f2py, I might just as well do it for the one that imposes the less amount of restrictions. And being able to compile a shared object (well 2, one for windows and one for linux) that I just compile once, create a .whl that I can then install in any version of python from 3.6 all the way to 3.11 was honestly a marvel I never thought possible. I do think f2py is very practical, I’m not trying to diminish it in any way, but it has constraints that are too heavy if one already has a large machinery running and wants to extract a few bits out of it without duplication of kernels and downsizing to a subset of the language. At the end the problem of this discussion is what does “the general case” actually mean? we all have our use cases, experiences and biases… could this discussion help maybe bring some of the drawbacks and strong points of each approach together to see if something could evolve? ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 9:02pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/13 "2023-11-14T21:02:22Z")
> [@hkvzjal](#): > > 1. An **upstream dependency** of my kernels (which are a mix of Fortran/C/C++) to the given python version This is not true, though it could be said to be historically ambiguous. `f2py` will generate `.pyf` files from which it will generate `.f90` and `.c` files. At **compile** time, the user is required to link to a `python` version to get a python extension module (import blah) which can also be packaged into a `wheel` suitable for PyPI distribution. In fact, if the shared library is built with `meson`, or can be linked in at compile time, there’s no need to recompile the shared library at all. It is just linked to the wrapper. The confusion comes from when `f2py` was used with `distutils` which limited how it could be integrated into other projects. Today, ([NumPy 1.26](https://github.com/numpy/numpy/releases/tag/v1.26.0) onwards) `f2py` will no longer force the use `distutils` or inflict `setup.py` files on the user, but will instead generate a `meson.build` file which can be integrated into an existing project. > 1. An **upstream dependency** to a subset of Fortran standards This is true, but as noted it is true for `iso_c_binding` as well. In fact, `f2py` supports a wider range of features than the `iso_c_binding` interface, e.g. F77 support, so arguably the more permissive case here is `f2py`. Personally it took me a long time to write all the (mechanically similar) `pointer` and `c_ptr` interfaces to the trivial derived type shown in the post, so I would prefer to have it automagically generated. The last part is the interaction with `python` itself, and this is the part which `f2py` does and is non-trivial to implement by hand (e.g. all the Python-C code in the post) but does make it more “pythonic”. > And being available to compile a shared object (well 2, one for windows and one for linux) that I just compile once, create a .whl that I can the install in any version of python from 3.6 all the way to 3.11 was honestly a marvel I never thought possible. This is nice but it is actually not true. Yes you can install it in any version of python, but you must ensure the system dependencies are the same for your shared library. This is one of the reasons it can’t be used to upload to PyPI. Also it is non-trivial to ensure `LD_LIBRARY_PATH` and other variables are setup to have access to the shared library. I know it is common in scientific computing to have environment variables which set the locations of shared libraries but it isn’t a very good practice… > I do think f2py is very practical, I’m not trying to diminish it in any way, but it has constraints that are too heavy if one already has a large machinery running and wants to extract a few bits out of it without duplication of kernels and downsizing to a subset of the language. I understand, but the point of these discussions is to find good directions for `f2py` to grow and to try to remove (what I feel are) some slight misunderstandings about its features and use. > At the end the problem of this discussion is what does “the general case” actually mean? we all have our use cases, experiences and biases… could this discussion help maybe bring some of the drawbacks and strong points of each approach together to see if something could evolve? +1, at the moment, what I meant by the general case would be viewed from the “python” side (e.g. SciPy) instead of the `fortran` / scientific computing side. ---
**Author:** ![rfarmer](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@rfarmer](https://fortran-lang.discourse.group/u/rfarmer)\ **Post date:** [November 14, 2023, 11:20pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/15 "2023-11-14T23:20:37Z")
No I haven’t tried compiling f2py’s or scipy’s test suite as most of the code isn’t in a module. So I’m not personally interested in supporting it. Yes a hard dependency on gfortran is both gfort2py’s greatest strength and greatest weakness. Sure I can’t support other compilers on the other hand my life is much simpler as I don’t have to worry about what other compilers are doing. I’m not sure why you think ctypes isn’t portable, I have 8 lines of code in gfort2py that care about the OS (and that’s just for selecting the extension for shared libraries). It even works on [Windows](https://fortran-lang.discourse.group/t/gfort2py-v2-0-0-release/6038/2) without me having even tried at that point to get it to work on Windows. > Works for personal projects where users are expected to “read the documentation” and compile accordingly for functionality… Not so useful as a basis for a generic tool. Yet gfort2py works without any modification to the source (so no annotations and no limitations on intent). A user of gfort2py doesn’t even need to know ctypes is, it’s entirely wrapped and transparent to the user. Much like f2py wraps the messiness of building a python-c interface to present something useful to the user. > I don’t think the general attitude that `ctypes` does the same thing is helpful in the least for **general** binding tools which can and are in use for production Python code. Except it can. Sure ctypes on it’s own doesn’t help. But you wrap ctypes and then you can do anything you want. Gfort2py has support for scalars, explicit arrays, strings, assumed shape arrays, alloctable arrays, and derived types. Things not currently supported are more due to my time commitment instead of any fundamental limitations of using ctypes. ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 11:28pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/16 "2023-11-14T23:28:26Z")
> [@rfarmer](#): > > Things not currently supported are more due to my time commitment instead of any fundamental limitations of using ctypes. No. This is simply not true. Let me put this in a simpler context. Since shared libraries cannot be built and uploaded to PyPI (without going through actually writing Python-C code / the `auditwheel` process etc. see `cibuildwheel` for more information), there **will never** be a `pip install` version of a library which interfaces through `ctypes` which means as far as **fortran in python** as far as the Python ecosystem is concerned it is a non-starter. > Yet gfort2py works without any modification to the source (so no annotations and no limitations on intent). A user of gfort2py doesn’t even need to know ctypes is, it’s entirely wrapped and transparent to the user. For **fortran** programmers who want to provide Python bindings, `ctypes` will be fine. Along with the note from the readme about `LD_LIBRARY_PATH`. For more of an understanding of these issues there is the post on SciPy’s build system issues, and the simple matter that as it stands, no user ever needs to compile SciPy’s fortran code themselves, they get it in a binary wheel, which `gfort2py` cannot do because of a `ctypes` limitation. Note that you **can** build a local wheel and distribute (with a prayer that all the system dependencies are the same) it but it needs to be “compatible” ([via cibuildwheel](https://github.com/pypa/cibuildwheel) typically) for PyPI distribution. The limitation has been listed earlier as well, users **need** to have the library (either the `.dll` or the `.so`) at runtime, and this path needs to be correctly set for every new library. This is in sharp contrast to the `wheel` approach where `python` handles this automagically. When was the last time a `pip install` required setting an environment variable? There are simply very different expectations for different communities. Fortran programmers are expected to handle their libraries themselves, and the wrappers, Python users are largely **not**. > I have 8 lines of code in gfort2py that care about the OS (and that’s just for selecting the extension for shared libraries). Compared to the 0 lines needed from having actual C-Python code. I’m not saying it is too hard to support and `gfort2py` is excellent work. However, we have differing veiws on what counts as portable. Portability is not the same as popularity or ease. That it is simple to have `gfortran` only doesn’t make it acceptable for cases like `scipy` or the rest of the Python community, no matter how useful and elegant it is for the fortran programmers who use only `gfortran`. I do want to be clear about the fact that `gfort2py` is a good project with clear goals and does the Fortran community a lot of good. However, ignoring the real reasons why it is not acceptable for more general use is also not a great attitude. Another point is closer to the compiler issue. As noted here: [https://fortran-lang.discourse.group/t/the-difficulties-of-using-fortran-in-scipy/](https://fortran-lang.discourse.group/t/the-difficulties-of-using-fortran-in-scipy/) One of the main issues is the lack of compatible Fortran compilers. This means that `gfortran` on Windows depending on where it comes from will be unable to link correctly to the `C` libraries… but this is an issue with compiling the library… until you try to share the `.dll` file with someone else and they can’t find symbols because their `gfortran` is different… It is however impossible to describe the problems of portability like this. A practical example as mentioned, is to see if it is possible to get a `pip` install friendly binding + library out of this method, and then to see if it can be reliably built across the ~53 different configurations NumPy supports. ---
**Author:** ![rfarmer](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@rfarmer](https://fortran-lang.discourse.group/u/rfarmer)\ **Post date:** [November 14, 2023, 11:46pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/17 "2023-11-14T23:46:26Z")
> [@rgoswami](#): > > However, ignoring the real reasons why it is not acceptable for more general use is also not a great attitude. Which I never did, my original post was simply pointing out that f2py is not the only game in town. I have never claimed it would be a replacement for use in scipy. > No. This is simply not true. Here I was simply discussing the implementation of Fortran features not the annoyance of getting libraries working on different machines. ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 14, 2023, 11:51pm UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/18 "2023-11-14T23:51:42Z")
> [@rfarmer](#): > > f2py is not the only game in town For use **by** the Python ecosystem it is, because: > the annoyance of getting libraries working on different machines. Is a large part of working in Python. Also it isn’t just that the libraries are hard to install, its also that `ctypes` is fundamentally unable to work without the shared library at hand, so it cannot be used generally. ---
**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\ **Post date:** [November 15, 2023, 12:29am UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/19 "2023-11-15T00:29:26Z")
> [@rgoswami](#): > > Since shared libraries cannot be built and uploaded to PyPI (without going through actually writing Python-C code / the `auditwheel` process etc. see `cibuildwheel` for more information), there **will never** be a `pip install` version of a library which interfaces through `ctypes` which means as far as **fortran in python** as far as the Python ecosystem is concerned it is a non-starter. Just FYI: it may be true for `pip` (I don’t know what auditwheel, cibuildwheel is, frankly), but not for `conda`. All these problems are solved by `conda` (i.e., you build the libs yourself and people can install them just fine for each platform and you don’t have to mess around with `LD_LIBRARY_PATH`). That’s what I do. I can write Fortran code of any complexity, and compile it as a shared lib with whatever build system I want. That includes manually writing interfaces using the C-binding feature of Fortran (which isn’t really that much more work than writing a C header file, for example). Then I can call it from any version of Python I want. It is deployed to the users as a conda package which takes care of the dependencies and the dll/so/dylib path issues. ---
**Author:** ![rgoswami](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/rgoswami/32/23_2.png) [@rgoswami](https://fortran-lang.discourse.group/u/rgoswami)\ **Post date:** [November 15, 2023, 12:42am UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/20 "2023-11-15T00:42:50Z")
Yup, `conda` / `spack` `nix` etc. will all take libraries built this way. Even `lmod` can be used to hack paths semi-automatically (as is done in most HPC centers). Or `direnv` will also automatically change path variables based on your directory so you can have your paths the way you want them… However, I see that as a Scientific community thing and is fundamentally off-topic here IMO. Python as used commonly is the focus of my post. This means `pip installable` libraries or things which help the existing ecosystem (e.g. NumPy / SciPy). Other approaches are either stop gap solutions or may miss the point completely. I don’t want to get into the many problems of `conda` or its ecosystem (because it would take us too far afield, but again, it has been discussed quite often in the SciPy / python packaging ecosystem). My favorite example of this logic is always the symbol renaming example, which can be used to interact with basically anything anytime but is never a good idea. Everything else forms a spectrum between fully supported and hacking compiled objects. As another example, it is why even the excellent libraries you maintain haven’t made their way into SciPy. It is of course unfair to expect the community here to work directly on F2PY (as an example) but it would atleast help to recognize what it brings beyond what is currently practiced. The only reason I am belaboring the point is that if the efforts were made to integrate from both ends, it would be more productive. The point of these posts is to try to get more of the Fortran community interested in opening issues w.r.t F2PY with the goal of getting modern Fortran into SciPy and just more generally acceptable. Honestly the community here has the most positive opinion of `ctypes` I’ve ever seen. Even the [NumPy documentation](https://numpy.org/doc/stable/reference/routines.ctypeslib.html) is quick to point out flaws: ![image](https://global.discourse-cdn.com/free1/uploads/fortran_lang/original/2X/9/953c0ce6942285d69dd306f4332bfb0826c1c0ea.png) The [source code itself](https://github.com/python/cpython/blob/3.12/Lib/ctypes) better explains why it isn’t considered to be too great. Now these may not have been encountered by people here yet, but these warnings are there for a reason, NumPy has a lot more users, and so sees a lot more sharp edges of the `ctypes` interface in general. ---
**Author:** ![jacobwilliams](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/jacobwilliams/32/10_2.png) [@jacobwilliams](https://fortran-lang.discourse.group/u/jacobwilliams)\ **Post date:** [November 15, 2023, 12:57am UTC](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782/21 "2023-11-15T00:57:25Z")
> [@rgoswami](#): > > However, I see that as a Scientific community thing and is fundamentally off-topic here IMO. Python as used commonly is the focus of my post. This means `pip installable` libraries or things which help the existing ecosystem (e.g. NumPy / SciPy). Other approaches are either stop gap solutions or may miss the point completely. I don’t want to get into the many problems of `conda` or its ecosystem (because it would take us too far afield, but again, it has been discussed quite often in the SciPy / python packaging ecosystem). Well I think you are underestimating the use cases of conda here. It solves significant problems that the rest of the Python packaging dumpster fire does not. The scientific community are the people using these libraries so it isn’t an insignificant thing. [Next page](https://fortran-lang.discourse.group/t/types-from-fortran-to-python-via-opaque-pointers/6782.md?page=2)