# Omp\_proc\_bind / omp\_places **URL:** \ **Category:** Help\ **Created:** [October 19, 2024, 7:47am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733 "2024-10-19T07:47:34Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![PierU](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/pieru/32/1848_2.png) [@PierU](https://fortran-lang.discourse.group/u/PierU)\ **Post date:** [October 19, 2024, 7:47am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/1 "2024-10-19T07:47:34Z")
Hi, When using OpenMP, if I want to ensure that a given thread is attached to the same physical core for the whole program life, it seems that setting the environment variable [`OMP_PROC_BIND`](https://www.openmp.org/spec-html/5.0/openmpse52.html) to `true` is enough. Is that correct? Now, I have a machine with 2 CPUs, each one having 16 cores. When running a program with 32 threads, do I have to specify something special (possibly with [`OMP_PLACES`](https://www.openmp.org/spec-html/5.0/openmpse53.html) ?) to ensure that the the threads 0-15 are attached to the first CPU and the threads 16-31 to the second CPU, or is it granted by default? Another case: on this 32 cores machines, I want to use only 16 threads and be sure that they are all on the same CPU. I understand that I should set `OMP_PLACES` to `sockets`: is that correct? And finally, about the allocations: in a NUMA scheme, I assume that the memory allocated within a given thread is as far as possible placed on the physical RAM that is “attached” to the CPU where the thread is running. And that it happens at the “first touch” and not necessarily when “allocate” is executed (at least on the main current OS’s, which do “lazy allocations”). Is that correct ? ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 19, 2024, 11:30am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/2 "2024-10-19T11:30:30Z")
Does the CPU support hyper-threading? I assume it only has 2 NUMA domains (1 for each socket)? > [@PierU](#): > > or is it granted by default? I think the general answer is no, if you don’t specify, it is implementation-defined. You can check what occurs in practice with OMP\_DISPLAY\_AFFINITY=1. For instance, in the libgomp [docs](https://gcc.gnu.org/onlinedocs/libgomp/OMP_005fPLACES.html) they say threads can be moved between CPUs: > If `OMP_PLACES` and `GOMP_CPU_AFFINITY` are unset and `OMP_PROC_BIND` is either unset or `false` , threads may be moved between CPUs following no placement policy. Moreover, the libgomp the docs [state](https://gcc.gnu.org/onlinedocs/libgomp/OMP_005fPROC_005fBIND.html): > When undefined, `OMP_PROC_BIND` defaults to `TRUE` when `OMP_PLACES` or `GOMP_CPU_AFFINITY` is set and `FALSE` otherwise. So setting `OMP_PLACES=sockets` will automatically bind/pin the OpenMP threads when using libgomp (but not necessarily in libiomp). * * * For libiomp, the thread affinity interface is documented [here](https://www.intel.com/content/www/us/en/docs/cpp-compiler/developer-guide-reference/2021-10/thread-affinity-interface.html). The OpenMP variables are documented [here.](https://www.intel.com/content/www/us/en/docs/fortran-compiler/developer-guide-reference/2024-2/supported-environment-variables.html) By default, the thread binding is set to FALSE, which is equivalent to KMP\_AFFINITY=none which means: > Does not bind OpenMP threads to particular thread contexts; however, if the operating system supports affinity, the compiler still uses the OpenMP thread affinity interface to determine machine topology. I don’t understand the implications of the second part of the sentence. * * * Especially on HPC systems you need to be careful, because sometimes the variables are already set by the admins/modules. I’ve encountered an environment where `KMP_` variables where set, and took precedence over `OMP_` variables. To use the OpenMP interface, I had to explicitly unset the `KMP_AFFINITY` variable. If not set correctly, it’s easy to slow down everything by mixing up MPI ranks and OpenMP threads in ways that don’t respect the machine topology or clash at the software level. * * * Probably not relevant to your question, but on MacOS I’m not sure OpenMP thread affinity is supported at all. The OS kernel passes work to threads as it deems best and the affinity variables are ignored. At least that’s what I’ve inferred from the following threads: - [macos - How to set processor affinity on OS X? - Super User](https://superuser.com/questions/149312/how-to-set-processor-affinity-on-os-x) - [parallel processing - How can I get an OpenMP program to use multiple cores on macOS - Stack Overflow](https://stackoverflow.com/questions/58404423/how-can-i-get-an-openmp-program-to-use-multiple-cores-on-macos) - [Thread affinitization: pinning Julia threads to cores - General Usage - Julia Programming Language](https://discourse.julialang.org/t/thread-affinitization-pinning-julia-threads-to-cores/58069) - [[Apple M1] Force an execution on a specif… - Apple Community](https://discussions.apple.com/thread/252525908?sortBy=rank) A while ago I did some experiments with XCode [Instruments](https://help.apple.com/instruments/mac/current/#/dev7b09c84f5) (the native Mac profiler), and one can see the thread switching which takes place. ---
**Author:** ![PierU](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/pieru/32/1848_2.png) [@PierU](https://fortran-lang.discourse.group/u/PierU)\ **Post date:** [October 19, 2024, 1:31pm UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/3 "2024-10-19T13:31:45Z")
The CPUs are 2x [Intel Xeon Gold 6346](https://www.intel.fr/content/www/fr/fr/products/sku/212457/intel-xeon-gold-6346-processor-36m-cache-3-10-ghz/specifications.html). AFAIK s single Intel Xeon processor does not have several NUMA domains, so on this machine there are probably 2 NUMA domains (one for each CPU). And yes it has hyperthreading, although I don’t want to use it. > [@ivanpribec](#): > > I think the general answer is no, if you don’t specify, it is implementation-defined. You can check what occurs in practice with OMP\_DISPLAY\_AFFINITY=1. > > For instance, in the libgomp [docs](https://gcc.gnu.org/onlinedocs/libgomp/OMP_005fPLACES.html) they say threads can be moved between CPUs: > > > If `OMP_PLACES` and `GOMP_CPU_AFFINITY` are unset and `OMP_PROC_BIND` is either unset or `false` , threads may be moved between CPUs following no placement policy. But I was implicitly talking about the case where `OMP_PROC_BIND` would be set to `true`. I have tested with `OMP_DISPLAY_AFFINITY=1`, and at least with the Intel compiler the threads are bound to the cores in order: ```auto % setenv OMP_NUM_THREADS 32 && setenv OMP_PROC_BIND true && setenv OMP_DISPLAY_AFFINITY 1 && ./a.out OMP: pid 300862 tid 300875 thread 13 bound to OS proc set {13} OMP: pid 300862 tid 300862 thread 0 bound to OS proc set {0} OMP: pid 300862 tid 300868 thread 6 bound to OS proc set {6} OMP: pid 300862 tid 300863 thread 1 bound to OS proc set {1} OMP: pid 300862 tid 300864 thread 2 bound to OS proc set {2} OMP: pid 300862 tid 300877 thread 15 bound to OS proc set {15} OMP: pid 300862 tid 300871 thread 9 bound to OS proc set {9} OMP: pid 300862 tid 300865 thread 3 bound to OS proc set {3} OMP: pid 300862 tid 300869 thread 7 bound to OS proc set {7} OMP: pid 300862 tid 300876 thread 14 bound to OS proc set {14} OMP: pid 300862 tid 300870 thread 8 bound to OS proc set {8} OMP: pid 300862 tid 300867 thread 5 bound to OS proc set {5} OMP: pid 300862 tid 300874 thread 12 bound to OS proc set {12} OMP: pid 300862 tid 300873 thread 11 bound to OS proc set {11} OMP: pid 300862 tid 300872 thread 10 bound to OS proc set {10} OMP: pid 300862 tid 300866 thread 4 bound to OS proc set {4} OMP: pid 300862 tid 300882 thread 20 bound to OS proc set {20} OMP: pid 300862 tid 300886 thread 24 bound to OS proc set {24} OMP: pid 300862 tid 300878 thread 16 bound to OS proc set {16} OMP: pid 300862 tid 300890 thread 28 bound to OS proc set {28} OMP: pid 300862 tid 300888 thread 26 bound to OS proc set {26} OMP: pid 300862 tid 300889 thread 27 bound to OS proc set {27} OMP: pid 300862 tid 300881 thread 19 bound to OS proc set {19} OMP: pid 300862 tid 300884 thread 22 bound to OS proc set {22} OMP: pid 300862 tid 300883 thread 21 bound to OS proc set {21} OMP: pid 300862 tid 300893 thread 31 bound to OS proc set {31} OMP: pid 300862 tid 300891 thread 29 bound to OS proc set {29} OMP: pid 300862 tid 300887 thread 25 bound to OS proc set {25} OMP: pid 300862 tid 300880 thread 18 bound to OS proc set {18} OMP: pid 300862 tid 300892 thread 30 bound to OS proc set {30} OMP: pid 300862 tid 300885 thread 23 bound to OS proc set {23} OMP: pid 300862 tid 300879 thread 17 bound to OS proc set {17} ``` (setting `OMP_PROC_BIND` to `close` or `spread` gives the same pattern) With only 16 threads, the threads are spread over the 2 CPUs: ```auto % setenv OMP_NUM_THREADS 16 && setenv OMP_PROC_BIND true && setenv OMP_DISPLAY_AFFINITY 1 && ./a.out OMP: pid 302731 tid 302731 thread 0 bound to OS proc set {0} OMP: pid 302731 tid 302732 thread 1 bound to OS proc set {2} OMP: pid 302731 tid 302733 thread 2 bound to OS proc set {4} OMP: pid 302731 tid 302734 thread 3 bound to OS proc set {6} OMP: pid 302731 tid 302735 thread 4 bound to OS proc set {8} OMP: pid 302731 tid 302736 thread 5 bound to OS proc set {10} OMP: pid 302731 tid 302737 thread 6 bound to OS proc set {12} OMP: pid 302731 tid 302738 thread 7 bound to OS proc set {14} OMP: pid 302731 tid 302739 thread 8 bound to OS proc set {16} OMP: pid 302731 tid 302745 thread 14 bound to OS proc set {28} OMP: pid 302731 tid 302742 thread 11 bound to OS proc set {22} OMP: pid 302731 tid 302740 thread 9 bound to OS proc set {18} OMP: pid 302731 tid 302741 thread 10 bound to OS proc set {20} OMP: pid 302731 tid 302743 thread 12 bound to OS proc set {24} OMP: pid 302731 tid 302746 thread 15 bound to OS proc set {30} OMP: pid 302731 tid 302744 thread 13 bound to OS proc set {26} ``` `OMP_PROC_BIND` must be set to `close` to get them on a single CPU: ```auto % setenv OMP_NUM_THREADS 16 && setenv OMP_PROC_BIND close && setenv OMP_DISPLAY_AFFINITY 1 && ./a.out OMP: pid 303690 tid 303700 thread 10 bound to OS proc set {10} OMP: pid 303690 tid 303704 thread 14 bound to OS proc set {14} OMP: pid 303690 tid 303699 thread 9 bound to OS proc set {9} OMP: pid 303690 tid 303692 thread 2 bound to OS proc set {2} OMP: pid 303690 tid 303693 thread 3 bound to OS proc set {3} OMP: pid 303690 tid 303690 thread 0 bound to OS proc set {0} OMP: pid 303690 tid 303701 thread 11 bound to OS proc set {11} OMP: pid 303690 tid 303703 thread 13 bound to OS proc set {13} OMP: pid 303690 tid 303705 thread 15 bound to OS proc set {15} OMP: pid 303690 tid 303691 thread 1 bound to OS proc set {1} OMP: pid 303690 tid 303697 thread 7 bound to OS proc set {7} OMP: pid 303690 tid 303694 thread 4 bound to OS proc set {4} OMP: pid 303690 tid 303696 thread 6 bound to OS proc set {6} OMP: pid 303690 tid 303695 thread 5 bound to OS proc set {5} OMP: pid 303690 tid 303698 thread 8 bound to OS proc set {8} OMP: pid 303690 tid 303702 thread 12 bound to OS proc set {12} ``` I tried playing with `OMP_PLACES`, but it seems that it’s not needed here. ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 19, 2024, 4:44pm UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/4 "2024-10-19T16:44:38Z")
> [@PierU](#): > > AFAIK s single Intel Xeon processor does not have several NUMA domains, so on this machine there are probably 2 NUMA domains (one for each CPU). Some older ones do (did). The [CoolMUC2](https://doku.lrz.de/linux-cluster-segments-10745971.html) cluster has these dual-socket 28-core “Haswell” servers with two Intel Xeon E5-2697 v3 CPUs each: ```txt ivan@cm2login2:~> lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 56 On-line CPU(s) list: 0-55 Thread(s) per core: 2 Core(s) per socket: 14 Socket(s): 2 NUMA node(s): 4 ... Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz ... NUMA node0 CPU(s): 0-6,28-34 NUMA node1 CPU(s): 7-13,35-41 NUMA node2 CPU(s): 14-20,42-48 NUMA node3 CPU(s): 21-27,49-55 ``` In this CPU package the 14 physical cores are divided into this dual-ring layout (image source: [Analysis of Intel’s Haswell Microarchitecture Using the ECM Model and Microbenchmarks | SpringerLink](https://doi.org/10.1007/978-3-319-30695-7_16)) with two independent memory controllers: ![image](https://global.discourse-cdn.com/free1/uploads/fortran_lang/original/2X/1/1856b1452aaa081791277a8f655272e30492ec37.png) The 7-cores per NUMA domain are tied to this “Cluster on Die” mode (section 2.5 in the paper above) where the cores are divided equally among memory controllers. This page contains some further die configurations: [The Magic Inside the Uncore - Intel Xeon E5 Version 3: Up to 18 Haswell EP Cores](https://www.anandtech.com/show/8423/intel-xeon-e5-version-3-up-to-18-haswell-ep-cores-/4) ---
**Author:** ![septc](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/septc/32/77_2.png) [@septc](https://fortran-lang.discourse.group/u/septc)\ **Post date:** [October 20, 2024, 6:08am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/5 "2024-10-20T06:08:15Z")
I’ve never tried “thread pinning”, so interested in what kind of situations it is useful. Is it basically for machines with multiple CPUs (like Xeon above with two sockets)? If so, it is not necessary to care about it for a single-CPU machine like Ryzen? (I’ve tried asking it with ChatGPT4o-mini, and the answer is something like it is useful also for Ryzen, so I am confused…) > **Output of ChatGPT4o-mini** > > [Q] Is thread pinning important for Ryzen? > > [A] Yes, thread pinning can be important for AMD Ryzen CPUs, especially in certain contexts: > > 1. **Multi-core Performance** : Ryzen CPUs often have multiple cores and threads. Pinning threads to specific cores can help improve cache locality and reduce cache misses, leading to better performance in multi-threaded applications. > 2. **Game and Application Optimization** : Many games and applications benefit from thread pinning, as it can help maintain consistent performance and reduce latency by keeping critical threads on dedicated cores. > 3. **NUMA Considerations** : Some Ryzen systems, particularly those with higher core counts, may exhibit NUMA-like behavior, where memory access times vary based on core and memory location. Pinning threads can help ensure optimal memory access patterns. > 4. **High-Performance Workloads** : For workloads that demand high performance, such as video rendering or scientific computations, thread pinning can optimize resource utilization and reduce contention. > > While Ryzen CPUs manage threads well, especially with their efficient architecture, thread pinning can still provide benefits in specific scenarios where performance consistency and optimization are crucial. ---
**Author:** ![JohnCampbell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@JohnCampbell](https://fortran-lang.discourse.group/u/JohnCampbell)\ **Post date:** [October 21, 2024, 5:57am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/6 "2024-10-21T05:57:51Z")
I have been trying to understand this issue with Ryzen 5900X (12 cores), with Windows 10 and Gfortran Ver 11. I have a large run that requiresd 10 threads, all of which are using a shared 25 GByte array. Maintaining each thread to the same cpu and also having all threads running at the same rate could optimise the use of L3 cache, but this has not been achievable. ( poor alignment of threads has extended a 3-hour run to 8 hours, so it is a noticeable problem. This change was observed with OS updates) “Task Manager” indicates that the threads are moving about between the 24 logical cpu’s. I have disabled “hyper-threading” as having 2 threads on the same physical CPU appears to provide inferior performance, but the core usage is still varying considerably. I have not been able to improve performance by combining environment variable `OMP_PROC_BIND` with Gfortran on Windows. I have not been able to lock each thread to a single cpu, so I would be interested in what you may be able to achieve. However, by using !$OMP BARRIER, I have improved performance, assumed to be related to L3 cache efficiency. I will be interested to see if Win 11 plus 24H2 helps in my case, as thread management appears to be very much managed by the OS. I have no experience of Linux behaviour to know how the OS compare. ---
**Author:** ![PierU](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/pieru/32/1848_2.png) [@PierU](https://fortran-lang.discourse.group/u/PierU)\ **Post date:** [October 21, 2024, 9:25am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/7 "2024-10-21T09:25:40Z")
> [@ivanpribec](#): > > Some older ones do (did). OK, I checked this one, and there are 2 NUMA domains: ```auto % lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 57 bits virtual CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 106 Model name: Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz Stepping: 6 CPU MHz: 1067.798 CPU max MHz: 3600.0000 CPU min MHz: 800.0000 BogoMIPS: 6200.00 Virtualization: VT-x L1d cache: 1.5 MiB L1i cache: 1 MiB L2 cache: 40 MiB L3 cache: 72 MiB NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art ar ch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulq dq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4 _1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm a bm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjus t bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512 ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xget bv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoi nvd dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip p ku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpop cntdq rdpid fsrm md_clear pconfig flush_l1d arch_capabilities ``` ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 21, 2024, 10:21am UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/8 "2024-10-21T10:21:04Z")
That is interesting. It looks like hyper-threading (HT) is disabled at the level of BIOS; the Intel product specification for [Xeon Gold 6346](https://ark.intel.com/content/www/us/en/ark/products/212457/intel-xeon-gold-6346-processor-36m-cache-3-10-ghz.html) says that [Intel HT](https://www.intel.com/content/www/us/en/gaming/resources/hyper-threading.html) is available. (HT it is rarely useful in BW-limited computational codes anyways.) I was looking at an Intel Xeon Platinum 8380 (also a 3rd Gen. Xeon SP using Icelake cores), also dual socket but with 40 cores per CPU: ```txt Model name: Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 40 Socket(s): 2 Caches (sum of all): L1d: 3.8 MiB (80 instances) L1i: 2.5 MiB (80 instances) L2: 100 MiB (80 instances) L3: 120 MiB (2 instances) NUMA: NUMA node(s): 4 NUMA node0 CPU(s): 0-19,80-99 NUMA node1 CPU(s): 20-39,100-119 NUMA node2 CPU(s): 40-59,120-139 NUMA node3 CPU(s): 60-79,140-159 ``` I failed to find any architecture images for these specific models, but according to the following articles these SMPs use a mesh architecture: - [Deep Dive Into Intel’s “Ice Lake” Xeon SP Architecture](https://www.nextplatform.com/2021/04/19/deep-dive-into-intels-ice-lake-xeon-sp-architecture/) - [Intel® Xeon® Processor Scalable Family Technical Overview](https://www.intel.com/content/www/us/en/developer/articles/technical/xeon-processor-scalable-family-technical-overview.html) - [3rd Gen Intel® Xeon® Scalable Processors](https://www.intel.com/content/www/us/en/products/docs/processors/xeon/3rd-gen-xeon-scalable-processors-brief.html) - [Technical Overview Of The 4th Gen Intel® Xeon® Scalable processor family](https://www.intel.com/content/www/us/en/developer/articles/technical/fourth-generation-xeon-scalable-family-overview.html) I believe in my case that Sub-NUMA Clustering (the CPU package is logically split into 2 NUMA domains) is turned on. ---
**Author:** ![PierU](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/pieru/32/1848_2.png) [@PierU](https://fortran-lang.discourse.group/u/PierU)\ **Post date:** [October 21, 2024, 2:50pm UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/9 "2024-10-21T14:50:11Z")
> [@septc](#): > > I’ve never tried “thread pinning”, so interested in what kind of situations it is useful. Is it basically for machines with multiple CPUs (like Xeon above with two sockets)? If so, it is not necessary to care about it for a single-CPU machine like Ryzen? It can be beneficial in many different cases, but it seems to me that it is the most useful in case of a NUMA architecture: it can be multiples sockets, but it can also be a single socket with a CPU that has a internal NUMA architecture (as mentioned by @ivanpribec ) ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [October 22, 2024, 11:17pm UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/10 "2024-10-22T23:17:37Z")
I just re-read today the thread affinity chapter in [Using OpenMP—The Next Step: Affinity, Accelerators, Tasking, and SIMD](https://direct.mit.edu/books/monograph/4482/Using-OpenMP-The-Next-StepAffinity-Accelerators). There is a lot to process there (the chapter is 67 pages long!), but it provides a much more understandable explanation compared to the OpenMP standard text, including some architectural background on NUMA architectures. To summarize the main concepts of the chapter: - **OpenMP Places** - A place is a hardware resource that can execute an OpenMP thread. - **Affinity Policy** - While the place list defines all the resources available to the OpenMP runtime scheduler, the affinity policy may be adjusted on each (nested) parallel region, and controls _how_ the threads should be scheduled relative to each other. The two concepts together specify the thread affinity. In absence of either one of these, or both, implementation-defined default settings are used. The interaction is summarized by the following table (Fig 4.4, page 160), ![image](https://global.discourse-cdn.com/free1/uploads/fortran_lang/original/2X/c/c42b0b010e88d9893c7c83b95f04b165983b991f.png) Here are the tips and tricks from the concluding remarks: - The `OMP_PLACES` and `OMP_PROC_BIND` environment variables have default settings. We strongly recommend setting these explicitly, or checking the documentation for the defaults. - The use of environment variable `OMP_DISPLAY_ENV` is strongly recommended to verify the affinity settings. [The book was published before `OMP_DISPLAY_AFFINITY` was introduced in OpenMP 5.0.] - The place list is static. It is defined upon program startup and cannot be modified. - Threads are not allowed to migrate between places. - If a place contains multiple resource numbers, all numbers in the list are equal from a scheduling point of view. - The affinity _policy_ [emphasis mine] may be adapted on each region. - The abstract names are preferred when specifying the place list. - An implementation may support additional abstract names to support specific arhictectural features. - When more control over the placement is needed, the interval notation provides for a compact way to define the places. This is not only less error-prone, but also easier to adapt to other platforms. * * * > [@PierU](#): > > Another case: on this 32 cores machines, I want to use only 16 threads and be sure that they are all on the same CPU. I understand that I should set `OMP_PLACES` to `sockets`: is that correct? Is that 16 _hardware_ threads (“places”) or OpenMP threads? You can limit the places to a single socket using `OMP_PLACES=sockets(1)`, but you still can’t control which socket. To control exactly which you’d need to use the interval notation `{ : [: ]}`. So you could chose: - socket 1 (NUMA node0): `OMP_PLACES={0:16:1}`, which expands to `{0,1,...,15}`, or - socket 2 (NUMA node1): `OMP_PLACES={16:16:1}`, which expands to `{16,17,...31}`. > [@PierU](#): > > I assume that the memory allocated within a given thread is as far as possible placed on the physical RAM that is “attached” to the CPU where the thread is running. And that it happens at the “first touch” and not necessarily when “allocate” is executed (at least on the main current OS’s, which do “lazy allocations”). Is that correct ? I’ll quote from the book here, > With the _First Touch_ placement policy, the thread (or process) that “touches” the data for the first time, has ownership of the corresponding page. This defines the _home_ node for that page. More specifically, the first thread that accesses a page, memory capacity issues aside, has the data allocated in the memory closest to this thread. This tends to work okay if each thread only needs to work on blocks of data it initialized and allocated itself. However it potentially creates problems if a master thread initializes the data (or reads it via I/O), and other threads need to access it. (For instance if the master thread was on socket 2, and the worker threads in a parallel section got placed on socket 1.) ---
**Author:** ![ivanpribec](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/ivanpribec/32/3290_2.png) [@ivanpribec](https://fortran-lang.discourse.group/u/ivanpribec)\ **Post date:** [November 30, 2024, 9:36pm UTC](https://fortran-lang.discourse.group/t/omp-proc-bind-omp-places/8733/11 "2024-11-30T21:36:20Z")
> [@ivanpribec](#): > > Probably not relevant to your question, but on MacOS I’m not sure OpenMP thread affinity is supported at all. Just wanted to confirm this on macOS (Intel Inside) using GCC (libgomp) I get the warning: ```fortran $ OMP_PLACES=cores OMP_NUM_THREADS=4 ./build/main libgomp: Affinity not supported on this configuration ``` If I ask to display affinity information it shows: ```txt level 1 thread 0x7ff84cf22dc0 affinity 0-11 level 1 thread 0x700002f21000 affinity 0-11 level 1 thread 0x700003124000 affinity 0-11 level 1 thread 0x700003327000 affinity 0-11 OMP: pid 4891 tid 259 thread 0 bound to OS proc set {undefined} OMP: pid 4891 tid 7171 thread 3 bound to OS proc set {undefined} [... truncated ...] ``` AFAIK, the ARM-based Macs don’t have [SMT](https://en.wikipedia.org/wiki/Simultaneous_multithreading), but they do have performance- and efficiency-cores. Would be interesting to find out if libomp (LLVM) supports affinity on the newer Macs.