Fortran called by Python/Vice-Versa

Hi all, I am using a column physics model (actually a standalone part of a climate model that simulates certain processes, and can be ‘inserted’ into the larger models to add in these processes) that is about 50,000 lines of Fortran90.

It has about maybe 40 .F90 files, they may have 2-3 subroutines per file. I don’t know exactly what calls what first (more or less) and what iterates over what, so it’s a sort of spaghetti right now. I do understand one subroutine very well. This subroutine I have basically been tasked with putting a machine learning algorithm in its place, and keeping the rest the same.

So I have say 100 subroutines, only 1 needs replacing. To do machine learning my boss basically wants me to do it in python and this seems a fair choice. What this means is that somehow, fortran will have to once call a python machine learning script and get answers out of it and carry on, or that I will have to face the rather more daunting task of writing python that calls all the rest of the fortran code and puts it into somehow the same order/create the same model with Fortran doing the legwork and Python pulling the strings, so as to make ML bit easy to do.

There are pros and cons to both quite obviously. One thing I’m stuck at is I’ve basically never done anything like this before. Do you guys know how to do anything like this? What exists to get fortran to be called by python (and vice versa)? And if I had to have python calling all the fortran subroutines etc. would I almost have to write a mini-overarching python code of this whole model and how would I be able to understand it?

e.g. if I have to take all the code to python and write “python call this fortran module … then call this module” but there are subroutines in here all calling each other, how does one go about ‘understanding’ or ‘organising’ what is being done so that the fortran is not discarded, but called correctly (e.g. I assume I have to call in python, but the fortran subroutines call each other also and it seems this process becomes incredibly messy/subject to so many failures).

Any and all advice to someone out of my depth would be most welcome! (I think this may be called wrapping?)

2 Likes

A similar question was posted on Reddit a few hours ago, with 14 replies so far. I mention this so that people do not spend time writing replies that were already posted elsewhere.

Ha! :slight_smile: I was hoping to find others here who may know (assume not everyone here is on reddit, in fact I didn’t think anyone here would be). :slight_smile: I’ll (or can) remove the question then if it helps? :slight_smile: Don’t mean to take up people’s time twice.

I don’t think you need to remove the question, but you could follow the discussion on Reddit for a few days, and if you still have questions, you can summarize that discussion here and ask further questions.

Ok sure. Thanks for the advice.

One of them is really interesting. Instead of embedding, wrapping, calling C/C++ or F2PY, communicate Fortran executable with Python script via MPI messages.

Fortran Keras Bridge aims to solve this problem.

The limitations are that:

  • You’re limited to Keras (and Tensorflow v1) on the Python side
  • You’re limited to fully-connected feed-forward networks on the Fortran side (but for a column physics model I think that should be sufficient)

What is the specific Machine learning algorithm that you need? Depending on the algorithm, it may be much easier to implement the ML algorithm in Fortran, or even better, there might be existing Fortran implementations of it. Many ML algorithms were initially written in Fortran before being translated to other languages and popularized.

1 Like

This is the post that interested me the most. As he says “yup, python and fortran can easily exchange data.” as “the mpiexec command starts multiple programs separated by “:”, they will have the same “communicator object”, and use the same underlying mpi library”.

One belief I have of this approach you may be knowledgeable about is that currently this works for only one python script/one fortran script. I.e. I think I would have to find a way for a fortran subroutine to call the MPI and run separately a fortran/python ‘talking’ pair and that this pair gives information back to the model itself. Do you know if this has to be true or how the approach they advise could be implemented in a model? (i.e. when there are many .F90s and you only want one of them to talk to a python script and exchange info at the right time)

I’m not sure yet. I think the original idea was, with some communication between python/fortran established, to test a few algorithms and identify ones based on their performance. That said I think there was some gut instinct that we may find from this approach that it shows recurrent neural networks maybe the way to go as a guess, but the acid test being testing various algorithms when python/fortran are talking to each other.

Off topic: Love your user name. I spent a year at NM Tech in Socorro. Ages ago.

1 Like

How many files and subroutines you have is kind of irrelevant. From your description it seems the subroutines are compiled into a single Fortran executable and now you just need to adapt or create a subroutine which performs part of the computations in Python?

It is possible to embed a Python interpreter in a C/C++ application (and hence Fortran too due to Fortran-C interoperability). Whether this will be easier/better than communicating via MPI processes is hard to say.

Is the Fortran column physics model parallelized? Would the ML part need to be called independently on each thread/process?