Workflow needed to incorporate code from program A into program B

[t1mc0up]

If possible, I would like some advice regarding an efficient workflow to add code from one fortran program (A) to another program (B).

In particular, to add the “runon” function in SWAP (https://www.swap.alterra.nl/) to the MAESPA program (https://bitbucket.org/remkoduursma/maespa/src/master/).

(1) How should I approach this task and is there a formal description, e.g. refactoring? re-engineering? that helps to refine searches on Google etc. I have bought Milan’s book, which is very helpful regarding refactoring; but it doesn’t quite answer this workflow.

(2) What toolsets are available to analyse fortran code (free form, f90) ?
As a novice, I am able to compile both programs and generate the documentation and graphs using doxygen in Code::Blocks. This shows some internal relationships. But I seem to have only commercial options (e.g. Understand, SciTools) to analyse the fortran code. Unlike options for C++. e.g. Sourcetrails. Eclipse plugin “Photran” offers some options for refactoring.

Thanks for any suggestions and help.

[everythingfunctional]

In general, the process could be fairly simple if both programs are written in a fairly modular and decoupled way. Just add the call(s) to the functions, the appropriate use statements, and make sure all the source files are copied into the project.

If the communication between the programs needs to be more complicated, or if there is coupling making things more complicated than simply call the procedure you’re interested, things get exponentially more difficult. At that point your idea of using some tools to analyze the code to give you more insight into the inner workings and various views is the way to go.

Also, shameless plug, Damian Rouson and I are providing training and consulting services together. Check out our websites and get in touch with us if you’re at all interested in getting a quote.

• https://everythingfunctional.com
• http://www.sourceryinstitute.org/

We’re currently working on a project that has a similar aspect to it.

[t1mc0up]

Thanks for replying!

“Adding the Call to the function” is something I can look at.

The question I posed had an additional purpose, which is answered by your offer of training and consulting services. If I cannot make progress, then this is a valid option.

In my toolset I have two software stacks: one is in C/C++ and the other is in Fortran. Both approaches require porting code within the respective languages.

For the C/C++ toolset this has been relatively easy to achieve and the community is wide and diverse enough to search/find a solution. So there I have made progress. And I will continue with this learning process.

This is not the case with Fortran and the time spent searching online for a viable workflow, not even a solution, has been expensive. It is not so apparent how to port blocks of code between Fortran programs.

Having bought Milan’s book “Modern Fortran” to gain insights into this process, I have found it useful and covers some very important territory using a concise, step-wise method. However it is not agile enough with respect to this need to extend a Fortran program using external code snippets or blocks. By agility I include things such as tools to analyse the source, e.g. Sourcetrails for C++ etc.

I have wondered whether this is due to academics being primarily involved in wrangling Fortran as part of their research, as well as the struggle to find research funding. That means Fortran code that is no longer developed alongside research effectively becomes abandonware, unsupported and poorly documented…

This “agility” to wrangle Fortran source code is important.

I will look at your website and take it from there. Thanks!

[FortranFan]

@t1mc0up

However it is not agile enough with respect to this need to extend a Fortran program using external code snippets or blocks. By agility I include things such as tools to analyse the source, e.g. Sourcetrails for C++ etc.

I have wondered whether this is due to academics being primarily involved in wrangling Fortran as part of their research, as well as the struggle to find research funding. That means Fortran code that is no longer developed alongside research effectively becomes abandonware, unsupported and poorly documented…

This “agility” to wrangle Fortran source code is important.

I agree with your assessment.

The challenges of working with Fortran are so great, few colleagues and no managers I know of appear willing to commit to working with Fortran in the teams I have worked. The end result being codebases after codebases becoming “abandonware” in my experience.

This recent site is tremendous progress to stem the tide, though much progress will be needed to advance the Fortran ecosystem, especially in terms of tooling.

The efforts toward a Fortran “standard library” also will prove very valuable for Fortran and its users: GitHub - fortran-lang/stdlib: Fortran Standard Library

@t1mc0up, if you can share your learnings from your “C/C++ toolset”, that will be very helpful particularly about aspects that you feel would be applicable to Fortran i.e., after you have finished @milancurcic’s book and get to appreciate and differentiate Fortran relative to C-like languages.

[nshaffer]

Your problem could be simple or complicated depending on what you mean by this:

@t1mc0up

In particular, to add the “runon” function in SWAP (https://www.swap.alterra.nl/ ) to the MAESPA program (https://bitbucket.org/remkoduursma/maespa/src/master/ ).

Are you adding a new model provided by SWAP into the MAESPA program? Are you swapping out a model in MAESPA for one in SWAP? Do results from one model need to feed into the other one? Before any particular approach or tool can be recommended, we need more information about what the objective is.

[FortranFan]

@t1mc0up

If possible, I would like some advice regarding an efficient workflow to add code from one fortran program (A) to another program (B).

In the meantime, if you can provide a brief but specific description of what you wish to achieve with these programs, some reader or other can provide a workaround until you decide to get professional input as suggested upthread.

You may note with GCC/gfortran and Intel Fortran toolsets on Windows and Linux platforms, you can setup archive (static) and/or shared/dynamic (DLL on Windows) of common data and methods including object-oriented (OO) design along with modular, functional, and to some extent generic programming paradigms toward your programs in Fortran (or C, C++ as well given enhanced interoperability with C facilities in Fortran).

Unfortunately, the information toward AGILE development options in Fortran are still in a highly nascent stage. Readers at site such as this can start providing feedback that can eventually feed into an ecosystem for such development in Fortran.

[t1mc0up]

@FortranFan: Thanks for the suggestion, I will come back to sharing my experience with the C/C++ toolset. Probably in a separate post. For now I will outline how I have approached this workflow regarding the Fortran runon subroutine.

@nsnaffer: Which brings me to your pragmatic questions and addressed below.

Central to posting my question was a workflow that helped me understand and identify the most efficient and effective path.

I used the steps below as an initial workflow (tools are doxygen and grep).

1. Understand the physical process you are modelling (e.g. diagram of runon routing and runon infiltration). This helps to reduce the complexity and keep it as simple as possible. In this example “runon” refers to runon routing (input a timeseries file, *.csv) and runon infiltration (infiltration subroutine and water balance output).

The results (i.e. runoff timeseries) from another model (including SWAP, or other…) or a previous MAESPA run become input for MAESPA as a "runon timeseries.

2. Code::Blocks: build and create documentation (doxygen, including graphs). No comparison with Sourcetrails, but there is a request on Github to include fortran. Is this a sign of the times?

3. Look for keyword ‘runon’ in SWAP source directory: “grep -nrHIiF – Runon .”
   Note that the full stop at the end means “search inside current directory”.

4. Ask yourself if this is complicated or simple. In particular whether the SWAP subroutines are complex.

@nsnaffer: “Are you adding a new model provided by SWAP into the MAESPA program?
Are you swapping out a model in MAESPA for one in SWAP?”

This is a crucial question and one I cannot answer at this stage.
I don’t have the experience to see where the dragons are; i.e. is this feasible?

Alternatively, are you adding the MAESPA canopy model provided by into the SWAP program?
Because MAESPA can run without input from the soil water module (“reforged” from the SPA fortran model). Maybe “reforging” fortran is the label we could use to describe this process?

I had hoped there would be a workflow that helps to avoid wasting time.

[t1mc0up]

[ivanpribec]

I think it would help to clarify whether:

1. you are trying to couple two models of different phenomena (available in two different codes)?
2. you want to replace part of the model in code A, with a different model (constitutive equation) of the same physical process available in code B.

In case of 1), I would follow the advice of chapter 8 in the book of Damian Rouson et al. - Scientific Software Design: The Object-Oriented Way, and build a “puppeteer” which handles the transfer of data between codes. This way you don’t need to modify the existing codes, but just build a sort of adaptor between them.

In case of 2) my work-flow would be something like:

• Choose the main code we plan to extend (based upon documentation, programming style, available features, user support)
• Identify parts of the code A (mathematical model) that we would like to replace, or the regions where we would like to incorporate features of the other model
• Follow the high-level description of the other model in published works (scientific papers) to build your own implementation. The code B can help guide the implementation process in case of missing information.

[t1mc0up]

Thanks for your guidance @ivanpribec:

Code A is a 1D water balance model which was coupled with a crop model, WOFOST
Code B is a 3D fortran model for trees (energy balance) which was coupled with a soil model (SPA). MAESPA couples the water balance and energy balance. For micro-climate modelling, MAESPA’s water balance is missing some important soil water functions. I would like to start with adding runon infiltration to gauge the workflow.

At this stage I am using this exercise to reflect on a workflow that expedites this process.

My preference is for your case (2)

I would like code B, MAESPA, to be able to process runon data.
One thought is to use subroutine(s) from SWAP to achieve this.

" * Choose the main code we plan to extend (based upon documentation, programming style, available features, user support)."
I would like to extend code B, MAESPA

" * Identify the regions (blocks?) of code B (mathematical model) where we would like to incorporate features of the other model."
Modify watbal.f90 - which is based on SPA water balance routines, with various modifications - and contains all the subroutines for calculating the water balance.
Two subroutines inside watbal.f90 that seem to be initial candidates :
INFILTRATE and WATBALLAY

“* Follow the high-level description of the other model in published works (scientific papers) to build your own implementation. Code A can help guide the implementation process in case of missing information.” And adding to this the suggestion from @everythingfunctional:
" Just add the call(s) to the functions, the appropriate use statements, and make sure all the source files are copied into the project"

Does this mean I add from SWAP the call(s) and appropriate USE statements to the MAESPA watbal.f90 file, where relevant (e.g. runon input, runon infiltration, water balance output functions)?

Then make sure the relevant SWAP *.f90 files are also copied to the MAESPA project folder.

[ivanpribec]

Yes, that seems right to me.

I didn’t look deeper into the call graphs, but if the SWAP code depends upon some global state variables, you might need to perform an initialization call.

Concerning the files, you could either copy them into MAESPA or compile them into either a static or shared library (DLL), and then just link them with a flag like -lswap during the MAESPA compilation process.

[t1mc0up]

@FortranFan and @ivanpribec: thanks for the guidance regarding static and dynamic libraries.
and the possible requirement for initialisation.

@FortranFan: “providing feedback into an ecosystem for such development in Fortran”. That would be an important and helpful outcome.