# Domain decomposition with the Hivemind **URL:** \ **Category:** Projects\ **Created:** [November 10, 2023, 12:24pm UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759 "2023-11-10T12:24:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kimala](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/kimala/32/2707_2.png) [@kimala](https://fortran-lang.discourse.group/u/kimala)\ **Post date:** [November 10, 2023, 12:24pm UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759/1 "2023-11-10T12:24:08Z")
Hello everyone. First of all, let me thanks everyone for this incredible community that makes me learn something new every day about my computational mother tongue. In the last 6 months I have worked a lot with MPI, to handle data exchange in a domain decomposition environment implemented with a Z-pencil strategy. Specifically, I use the code [NEMO](https://www.nemo-ocean.eu) to do some idealized ocean simulations. This code has a finite difference core and it employs a one-point [halo/overlap/ghostcells](https://www.researchgate.net/profile/Keith-Lindsay/publication/268055930/figure/fig9/AS:669448730058765@1536620423489/An-illustration-of-a-ghost-cell-or-halo-region-showing-the-halo-for-a-particular-block.png) to facilitate computations of the finite differences. My task in the past months was to collect extra points in this halo/overlap/ghostcells in order to compute operations with larger support (in my case, a gaussian filtering and a discrete fast wavelet transform). To do so, I have implemented 2 kinds of routines: the first kind, in several variants, to enlarge my halo region of a given h number of points (in each direction, meaning that from an n \times m rectangle I can go to a (n+2h) \times (m+2h) rectangle). The second kind actually communicates only one side of the operation, so only does right to left, or left to right, top to bottom or bottom to top. I know that I can construct the first kind of routine with the second, but I developed the second kind because it was incorrect to apply the first kind for the wavelet transform (details upon request). I was in a hurry, so my routines are definitely not optimal, and I have learned MPI with a sink or swim approach: if it works and it is not slower than sequential, it is a win. Now, I would like to gather here the collective intelligence of this hivemind regarding domain decomposition handling, given the following problem: ![North-Sea-proxy](https://global.discourse-cdn.com/free1/uploads/fortran_lang/original/2X/4/4cbcb6affba130714b3c26982c21f5c1f7ad787a.png) Consider this geometry, as a non-trivial example of a domain with non regular boundaries. Each square is a sub-domain, handled with MPI. Each of this subdomains has the same dimension, let’s say N\_i \times N\_j \times N\_k, where i means longitude, j latitude and k is the vertical. The MPI parallelization is only done in i and j, and you can assume that these two corresponds to the first and second dimension of a fortran array. Inside each subdomain there will be room for openMP, openACC, fortran do concurrent or CUDA kernels for basic parallel operations on the array. An overlap of h points is given, where this h can be tuned by the operator followings her/his/their needs. The end goal would be solving an elliptic PDE on this domain, so there is also a linear system at some point, to solve, but how to solve it is not the main discussion. I would like to discuss about how you would target the management of multiple sub-domains, their connections, their numbering, what would be a good data structure for them and so on. I know my base software quite well, NEMO, and I am starting looking upon [CROCO](https://www.croco-ocean.org), I have read Milan’s book almost from cover to cover, thus I have an idea of the general picture as he implemented a similar thing with a regular domain and Fortran coarrays. However, I would like to have a discussion with people more experienced than me regarding this task, their experience and their workaround. No lines of codes are expected, but rather a discussion of the strategies for an efficient handling of the problem. Feel free to share papers on the topic. Ocean and Climate modeling is one of the fields where Fortran is still the main language, even when it comes to porting to GPU most of the efforts are done to adapt existing fortran codes, so I hope this topic will interest you and pose you a design challenge. On my side, I will be very happy to read every contribution, summarize them from time to time, I will as well come back with more specific topics. In the best case scenario, I would like to summarize at one point what the Fortran community regards as the best practice for such a topic. (Needless to say that whatever comes out of this discussion is not going to be monetized) I look forward to read your thoughts. ---
**Author:** ![Arjen](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@Arjen](https://fortran-lang.discourse.group/u/Arjen)\ **Post date:** [November 10, 2023, 1:59pm UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759/2 "2023-11-10T13:59:23Z")
I have not read your post yet, but this article might be of some use (as a first guess 🙂): Fortran High-Level Synthesis: Reducing the barriers to accelerating HPC codes on FPGAs Gabriel Rodriguez-Canal, Nick Brown, Tim Dykes, Jessica R. Jones, and Utz-Uwe Haus 1EPCC, The University of Edinburgh 2HPC/AI EMEA Research Lab, Hewlett Packard Enterprise arXiv:2308.13274v1 [cs.DC] 25 Aug 2023 ---
**Author:** ![rwmsu](https://avatars.discourse-cdn.com/v4/letter/r/48db29/32.png) [@rwmsu](https://fortran-lang.discourse.group/u/rwmsu)\ **Post date:** [November 10, 2023, 3:33pm UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759/3 "2023-11-10T15:33:06Z")
As far as data structures, I would look all the CFD literature related to cut-cell cartesian and immersed boundary methods. I think some of them discuss data structures. One issue is how the underlying grid was generated. Is it structured (as in a cartesian or tensor-product grid) or is it a collection of unstructured hex or quad cells. Also, do you plan on using Adaptive Mesh Refinement (AMR). If so you want to look at Marsha Berger’s work on block structrured AMR. If the grid is structured, the biggest issue is usually defining the connectivity between blocks and/or individual cells. A set of data structures (derived types) that define the following: global domain, subblocks in domain, cells in blocks, and connectivity (both physically and in terms of parallel communciation) would be my decomposition of the problem. Although, you stated that solvers were not your current interest, I think you should look at the following book > **[Amazon.com: Domain Decomposition: Parallel Multilevel Methods for Elliptic...](https://www.amazon.com/Domain-Decomposition-Multilevel-Differential-Equations/dp/052149589X)** > > Amazon.com: Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations: 9780521495899: Smith, Barry, Bjorstad, Petter, Gropp, William Note: although some people use the phrase “domain decomposition” to refer to mesh partitioning. They are NOT the same thing (but one leads to the other) ---
**Author:** ![rsci](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@rsci](https://fortran-lang.discourse.group/u/rsci)\ **Post date:** [November 10, 2023, 8:09pm UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759/4 "2023-11-10T20:09:17Z")
Perhaps this article will be useful to you. > **[Advanced parallel implementation of the coupled ocean–ice model FEMAO...](https://gmd.copernicus.org/articles/14/843/2021/)** > > Abstract. In this paper, we present a parallel version of the finite-element model of the Arctic Ocean (FEMAO) configured for the White Sea and based on MPI technology. This model consists of two main parts: an ocean dynamics model and a surface ice... ---
**Author:** ![Rob777](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@Rob777](https://fortran-lang.discourse.group/u/Rob777)\ **Post date:** [November 13, 2023, 10:43am UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759/5 "2023-11-13T10:43:26Z")
If the mesh topology is going to be multi-block structured, some data structures to handle it are presented in [https://www.amazon.it/Handbook-Grid-Generation-Joe-Thompson/dp/0849326877/ref=tmm\_hrd\_swatch\_0?\_encoding=UTF8&qid=&sr=](https://www.amazon.it/Handbook-Grid-Generation-Joe-Thompson/dp/0849326877/ref=tmm_hrd_swatch_0?_encoding=UTF8&qid=&sr=) The 2nd edition linked above is a well-known reference for multi-block structured meshing. ---
**Author:** ![kimala](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/kimala/32/2707_2.png) [@kimala](https://fortran-lang.discourse.group/u/kimala)\ **Post date:** [July 11, 2024, 9:21am UTC](https://fortran-lang.discourse.group/t/domain-decomposition-with-the-hivemind/6759/6 "2024-07-11T09:21:20Z")
I will revive this post as I will soon start modernizing a code with the aim of making it MPI aware. The code uses a cartesian three dimensional grid and performs stencil operations on this grid. I will follow @milancurcic [worked example for the tsunami simulator](https://github.com/modern-fortran/tsunami/blob/master/src/ch08/mod_field.f90) to design a derived data type (DDT). So I want to collect suggestions from you on what this derived data type should be performing. To me, the main two things that are needed are - halo exchange in non-blocking fashion through MPI (this is the requirement for my work) - I/O from and to NetCDF files, with automated rebuild (i.e. from `np` different output files, one for each subdomain, to one big file for the global domain) but it can be extended - possibly support, in the same DDT, other paradigms for the halo exchange (such as coarrays) - other type of files would be of interest (eg paraview, ascii or whatsoever) The specific code will simulate a particular ocean dynamics, but I think this could be of interest in other fields as well and could provide a nice tool to have in the shed for the finite difference people. Let me know what would be your wishlist for such a DDT.