I am brand new in this slist. So welcome everybody - I am glad to join this forum. I have also limited English so please forgive me my mistakes.
I work on ocean modelling. Recently we took into account large ROMS (Regional Ocean Modelling System) code and adapted it for our domain. It works, we have some problems with keeping it working, but our main problem is the time of integration. Because of problems with programing environment (PE), the calculation of one model day takes near two hours. So it is really too long. The model is not small, the size of domain is 2700x3200 and it has 40 vertical levels. One snapshot saved to he netcdf file has about 15 GB. It is fortran code compiled using gfortran (gnu fortran compiler). We use 13 nodes, and each node has 24 cores (intel cluster at Informatic Centre of Technical University of Gdańsk).
Additional information is here:
- we use mpif90:
GNU Fortran (GCC) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc
I am not aware about all of the needed libraries - we use netcdf (but it is not parallel, the version is over 4.5).
The loader is also mpif90
to me important variables defined in makefile are as a follow:
FC := mpif90 # FFLAGS := -frepack-arrays -fallow-argument-mismatch FFLAGS := -frepack-arrays -Wno-argument-mismatch FIXEDFLAGS := -ffixed-form FREEFLAGS := -ffree-form -ffree-line-length-none CPP := /usr/bin/cpp CPPFLAGS := -P -traditional -w # -w turns of warnings INCDIR := /usr/include /usr/local/bin SLIBS := -L/usr/local/lib -L/usr/lib -L/usr/lib64 ULIBS := ifdef USE_DEBUG FFLAGS += -g -O0 FFLAGS += -fbounds-check FFLAGS += -fbacktrace FFLAGS += -fcheck=all FFLAGS += -fsanitize=address -fsanitize=undefined FFLAGS += -finit-real=nan -ffpe-trap=invalid,zero,overflow else FFLAGS += -O3 FFLAGS += -ffast-math -malign-double -fforce-addr -funroll-loops endif
of course USE_DEBUG is off.
I am not going to make changes in the code, because it is still under developing so i am thinking about compiler flags. So, could you be so kind and give me advice what flags, or what changes should i do to have better performance of the model.
Thank you in advance for your help and any advice,
all the best