Flags for optimization fortran code

1

Dear all,

I am brand new in this slist. So welcome everybody - I am glad to join this forum. I have also limited English so please forgive me my mistakes.

I work on ocean modelling. Recently we took into account large ROMS (Regional Ocean Modelling System) code and adapted it for our domain. It works, we have some problems with keeping it working, but our main problem is the time of integration. Because of problems with programing environment (PE), the calculation of one model day takes near two hours. So it is really too long. The model is not small, the size of domain is 2700x3200 and it has 40 vertical levels. One snapshot saved to he netcdf file has about 15 GB. It is fortran code compiled using gfortran (gnu fortran compiler). We use 13 nodes, and each node has 24 cores (intel cluster at Informatic Centre of Technical University of Gdańsk).

Additional information is here:

  • we use mpif90:

mpif90 --version
GNU Fortran (GCC) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc

I am not aware about all of the needed libraries - we use netcdf (but it is not parallel, the version is over 4.5).

The loader is also mpif90

to me important variables defined in makefile are as a follow:

               FC := mpif90
#           FFLAGS := -frepack-arrays -fallow-argument-mismatch
           FFLAGS := -frepack-arrays -Wno-argument-mismatch

       FIXEDFLAGS := -ffixed-form
        FREEFLAGS := -ffree-form -ffree-line-length-none
              CPP := /usr/bin/cpp
         CPPFLAGS := -P -traditional -w              # -w turns of warnings
           INCDIR := /usr/include /usr/local/bin
            SLIBS := -L/usr/local/lib -L/usr/lib -L/usr/lib64
            ULIBS :=

ifdef USE_DEBUG
           FFLAGS += -g -O0
           FFLAGS += -fbounds-check
           FFLAGS += -fbacktrace
           FFLAGS += -fcheck=all
           FFLAGS += -fsanitize=address -fsanitize=undefined
           FFLAGS += -finit-real=nan -ffpe-trap=invalid,zero,overflow
 else
           FFLAGS += -O3
           FFLAGS += -ffast-math  -malign-double -fforce-addr -funroll-loops
 endif

of course USE_DEBUG is off.

I am not going to make changes in the code, because it is still under developing so i am thinking about compiler flags. So, could you be so kind and give me advice what flags, or what changes should i do to have better performance of the model.

Thank you in advance for your help and any advice,

all the best

Jaromir

3 Likes
2

Dear Jaromir, thanks for the question and welcome to the forum!

The only missing flag that I can see is -march=native. Make sure you do not define USE_DEBUG. Otherwise it seems you are already using all flags. For production, you can also try the Intel compiler that can usually deliver better performance than GFortran.

3 Likes
3

Dear Certic,
thank you for your suggestion. We will check this flag, also we are going to switch to intel compiler as soon as it will be possible. But because of ‘size’ of the code and also size of domain we would like to make sure it works properly and we have no problems with PE
with best regards.
Jaromir

1 Like