# How to correctly determine the \`-march\` flag when working with Intel oneAPI on Intel processors? **URL:** \ **Category:** Help\ **Created:** [February 19, 2022, 3:03am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826 "2022-02-19T03:03:47Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 19, 2022, 3:03am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/1 "2022-02-19T03:03:47Z")
The testing environment is Ubuntu 20.04.3 LTS installed on a machine with dual Intel Xeon E5-2699 v4 and Supermicro X10DAi motherboard. I try to compile and test [VASP.6.3.0](https://www.vasp.at/info/post#NEW%20RELEASE%3a%20VASP.6.3.0) with recent/latest Intel oneAPI base and hpc toolkits. I found that in order to pass all the tests, the following compiler option must be used: ```auto FFLAGS += -march=core-avx2 ``` As a comparison, the following compiler option will trigger some test errors: ```auto FFLAGS += -xHOST ``` So I want to know how to correctly determine the `-march` flag when working with Intel oneAPI on Intel processors, for example, this case, `-march=core-avx2`? The following is my test environment hardware and software specific information, for reference only: ```auto werner@X10DAi-00:~$ inxi -Cxxx CPU: Topology: 2x 22-Core model: Intel Xeon E5-2699 v4 bits: 64 type: MT MCP SMP arch: Broadwell rev: 1 L2 cache: 110.0 MiB flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 387287 Speed: 1200 MHz min/max: 1200/3600 MHz Core speeds (MHz): 1: 1200 2: 1202 3: 1202 4: 1202 5: 1200 6: 1202 7: 1203 8: 1201 9: 1204 10: 1201 11: 1654 12: 2007 13: 2204 14: 2200 15: 1245 16: 1202 17: 1202 18: 1202 19: 1203 20: 1202 21: 1203 22: 1202 23: 1202 24: 1201 25: 1202 26: 1202 27: 1201 28: 1202 29: 1202 30: 1202 31: 2066 32: 1202 33: 1202 34: 1202 35: 1203 36: 1202 37: 1202 38: 1202 39: 1202 40: 1202 41: 1200 42: 1516 43: 1200 44: 1200 45: 1200 46: 1202 47: 1200 48: 1200 49: 1200 50: 1200 51: 1201 52: 1201 53: 1201 54: 1201 55: 1200 56: 1201 57: 1204 58: 1200 59: 1200 60: 1609 61: 1871 62: 2200 63: 1251 64: 1201 65: 1201 66: 1201 67: 1200 68: 1203 69: 1200 70: 1201 71: 1201 72: 1201 73: 1201 74: 1201 75: 1200 76: 1200 77: 1200 78: 1201 79: 1203 80: 1523 81: 1201 82: 1200 83: 1200 84: 1201 85: 1201 86: 1200 87: 1200 88: 1204 werner@X10DAi-00:~$ inxi -Mxxx Machine: Type: Desktop System: Supermicro product: X10DAi v: 123456789 serial: Mobo: Supermicro model: X10DAI v: 1.02 serial: UEFI: American Megatrends v: 3.2 date: 12/16/2019 werner@X10DAi-00:~$ inxi -Sxxx System: Host: X10DAi-00 Kernel: 5.8.0-43-generic x86_64 bits: 64 compiler: N/A Desktop: GNOME 3.36.9 tk: GTK 3.24.20 wm: gnome-shell dm: GDM3 3.36.3 Distro: Ubuntu 20.04.3 LTS (Focal Fossa) ``` See [here](https://www.vasp.at/forum/viewtopic.php?f=2&t=18392#p21492) for related discussions. Regards, HZ ---
**Author:** ![msz59](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@msz59](https://fortran-lang.discourse.group/u/msz59)\ **Post date:** [February 19, 2022, 12:58pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/2 "2022-02-19T12:58:02Z")
You can find the proper _codename_ in the first line of `inxi -Cxxx` output: > arch: Broadwell thus, use `-march=broadwell` If not available you can always leave the detection of the proper _family_ to the compiler, using `-march=native` Interestingly, this `native` value to the `-march` option is not mentioned in `man ifort`, checked in versions 2021.3.0, 2021.4.0 BTW, maybe somebody knows: I have a desktop machine equipped with i5-10505 CPU, codename “Comet Lake”. There is no `-march=cometlake` option available. The closest codename seems to be “Icelake” but assembler sources generated using `native` and `icelake` do differ. So, which codename is appropriate for that CPU? ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 19, 2022, 1:56pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/3 "2022-02-19T13:56:14Z")
The [march document](https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/compiler-reference/compiler-options/compiler-option-details/code-generation-options/march.html) has the following processor name: ```auto broadwell ``` I mean: The initial letter is lowercase, while inxi’s result is capitalized. Which one should I use? As for `-march=native`, I noticed intel has [`xHost`](https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/compiler-reference/compiler-options/compiler-option-details/code-generation-options/xhost-qxhost.html#xhost-qxhost) code generation option, which is similar to the function of `-march=native`. But as we have discussed [here](https://www.vasp.at/forum/viewtopic.php?f=2&t=18392#p21492), `-xHOST` doesn’t work for some vasp tests. BTW, the following gcc command confirms that `-march=native` is equivalent to `-march=broadwell` on my machine: ```auto $ gcc -march=native -Q --help=target|grep -- '^[]*-march=' -march= broadwell ``` Regards, HZ ---
**Author:** ![msz59](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@msz59](https://fortran-lang.discourse.group/u/msz59)\ **Post date:** [February 19, 2022, 2:27pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/4 "2022-02-19T14:27:09Z")
Use lowercase Easy to check: > ifort -march=Broadwell source.f90 > ifort: command line warning #10148: option ‘-march=Broadwell’ not supported ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 19, 2022, 2:31pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/5 "2022-02-19T14:31:16Z")
Great. The following also do the trick: ```auto $ ifort -march=Broadwell >/dev/null ifort: command line warning #10148: option '-march=Broadwell' not supported ifort: command line error: no files specified; for help type "ifort -help" ``` ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 19, 2022, 2:46pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/6 "2022-02-19T14:46:07Z")
> [@msz59](#): > > You can find the proper _codename_ in the first line of `inxi -Cxxx` output: > > > arch: Broadwell > > thus, use `-march=broadwell` > If not available you can always leave the detection of the proper _family_ to the compiler, using `-march=native` Wonderful advice. Based on your above comment, I’ve confirmed that both of the following settings solve the problem discussed here: `FFLAGS += -march=broadwell` or `FFLAGS += -march=native` Regards, HZ ---
**Author:** ![sblionel](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/sblionel/32/853_2.png) [@sblionel](https://fortran-lang.discourse.group/u/sblionel)\ **Post date:** [February 19, 2022, 4:38pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/7 "2022-02-19T16:38:45Z")
-xHost won’t work if you run the executable on a system with a smaller instruction set. The VASP page referenced before includes a mention of MPI. If the program ran on a cluster with some nodes that didn’t support the compiling node’s instruction set, that can cause problems. ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 20, 2022, 1:52am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/8 "2022-02-20T01:52:57Z")
@sblionel 1. What do you mean by saying “a smaller instruction set”? 2. Do you mean `-march=native` is more portable than `-xHost`? Regards, HZ ---
**Author:** ![CRquantum](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/crquantum/32/730_2.png) [@CRquantum](https://fortran-lang.discourse.group/u/CRquantum)\ **Post date:** [February 20, 2022, 2:30am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/9 "2022-02-20T02:30:25Z")
If you compile on machine A, and run on machine B, where A and B uses the same CPU architecture, like both are broadwell, or whatever. Then -xHost should just work fine. However, if B and A does not uses the same CPU architecture, there is a chance that as Dr. Fortran @sblionel said, if you still use -xHost on the compiling machine A, the code will run on A but may not run on machine B. In this case, if you insist on compiling on A, you need to make sure use -march=XXX where XXX is for B, or for both A and B or the common denominator. ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 20, 2022, 3:35am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/10 "2022-02-20T03:35:57Z")
> [@CRquantum](#): > > In this case, if you insist on compiling on A, you need to make sure use -march=XXX where XXX is for B, or for both A and B or the common denominator. 1. Do you mean use `-march` solely or in combination with `-xHost`? 2. IMO, it’s not so convenient, even difficult, to find the acceptable instruction sets or common definitions for A and B. 3. Do you mean by this way, `-march` can facilitate the cross-compiling? Regards, HZ ---
**Author:** ![CRquantum](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/crquantum/32/730_2.png) [@CRquantum](https://fortran-lang.discourse.group/u/CRquantum)\ **Post date:** [February 20, 2022, 4:35am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/11 "2022-02-20T04:35:07Z")
If you use -march=XXX, then do not use -xHost. Yes -march=XXX can create code for intel’s CPU with codename XXX. -xHost basically is for compiling and running on the same ‘Host’ machine. You just need to google the codename of your CPU, in your case, search intel ark xeon 2699v4, > **[Intel® Xeon® Processor E5-2699 v4 (55M Cache, 2.20 GHz) - Product...](https://ark.intel.com/content/www/us/en/ark/products/91317/intel-xeon-processor-e52699-v4-55m-cache-2-20-ghz.html)** > > Intel® Xeon® Processor E5-2699 v4 (55M Cache, 2.20 GHz) quick reference with specifications, features, and technologies. ![image](https://global.discourse-cdn.com/free1/uploads/fortran_lang/original/2X/8/8b1b04b62efb6fffcb58493f74656c7a76614c71.png) so it is broadwell, if want to run on you B which is xeon 2699v4, you need to use -march=broadwell If you A and Haswell which is earlier than Boradwell, and you want to run on both A and B, you use -march=haswell. Yes, it seems there is no one-size-fits-all option, it requires you to know a little bit about the intel CPU codename. Actually Haswell, Broadwell, until coffeelake, and some Skylake there are no big difference, they all use AVX2. so -march=core-avx2 should just work. If you can try to compile and run on the same machine, then -xHost is the best. ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 20, 2022, 7:05am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/12 "2022-02-20T07:05:29Z")
> [@CRquantum](#): > > You just need to google the codename of your CPU, in your case, search intel ark xeon 2699v4, As commented [here](https://unix.stackexchange.com/questions/230634/how-to-find-out-intel-architecture-family-from-command-line), the most convenient methods are still the following ones: ```auto $ gcc -march=native -Q --help=target|grep -- '^[]*-march=' -march= broadwell $ cat /sys/devices/cpu/caps/pmu_name broadwell ``` > [@CRquantum](#): > > If you can try to compile and run on the same machine, then `-xHost` is the best. The problem is: for some specific scenarios, `-xHost` doesn’t work, while `-march=native` does, just as the issue discussed [here](https://www.vasp.at/forum/viewtopic.php?f=2&t=18392#p21492). That’s the main reason I asked this question. Regards, HZ ---
**Author:** ![sblionel](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/sblionel/32/853_2.png) [@sblionel](https://fortran-lang.discourse.group/u/sblionel)\ **Post date:** [February 20, 2022, 3:56pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/13 "2022-02-20T15:56:24Z")
The issue you linked doesn’t suggest that `-xHost` “doesn’t work”, but rather that it is inappropriate for a mixed execution environment. What `-xHost` does (I know, because I’m the one who implemented it) is query the CPU type. If it is an Intel processor, it applies the appropriate `-x` option for that processor, and that includes a check at the start of execution that the program is running on an Intel processor supporting the specified instruction set (or larger). If the compiling CPU is non-Intel, then it applies the appropriate `-march` option and omits the startup check. Even with `-march`, if you execute on a processor that doesn’t support the same instruction set as the compiling processor, the program may fail with an invalid instruction fault. (Note that “Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.” [Performance Index (intel.com)](https://edc.intel.com/content/www/us/en/products/performance/benchmarks/overview/?r=322126669)) ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 21, 2022, 1:48am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/14 "2022-02-21T01:48:27Z")
> [@sblionel](#): > > The issue you linked doesn’t suggest that `-xHost` “doesn’t work”, but rather that it is inappropriate for a mixed execution environment. Thank you for your in-depth explanation. By “a mixed execution environment”, do you mean situations where execution and compilation processors belong to different architectures? According to the description [here](https://www.vasp.at/forum/viewtopic.php?f=2&t=18049#p20327), the tests that trigger vasp error is caused by the following reason: ```auto Hi John, Hi Roger, I can confirm that this is an issue that is triggered by requesting AVX512 instructions (-xCORE-AVX512 or -xHost on an applicable host), and disappears when limiting things to AVX2. We have not seen this before because only recently we acquired an AVX512 capable machine (a Cascade Lake Xeon). I reproduced this with the Intel 19.1.2.254 compilers (which means some 2020 version of Parallel Studio --- confusing). I have not checked whether it is solved in the new oneAPI distros, but will try to do so ASAP. ``` Based on the above comment, this issue is attributed to a problem with the current Intel oneAPI compilers & tools. Regards, HZ ---
**Author:** ![sblionel](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/sblionel/32/853_2.png) [@sblionel](https://fortran-lang.discourse.group/u/sblionel)\ **Post date:** [February 21, 2022, 3:22pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/15 "2022-02-21T15:22:48Z")
> [@Hongyi](#): > > Thank you for your in-depth explanation. By “a mixed execution environment”, do you mean situations where execution and compilation processors belong to different architectures? Yes, exactly. But no. the comment does not suggest “a problem with the current Intel oneAPI compilers & tools”. AVX512 vectorization can cause computations to occur in a different order, and floating-point computations can be sensitive to that. If the VASP test is looking for a specific exact value for some computation, any different instruction sequence, while mathematically still correct, can result in small (or sometimes not so small) changes in the final value. I suggest you take a look at [Improving Numerical Reproducibility in C/C++/Fortran (supercomputing.org)](http://sc13.supercomputing.org/sites/default/files/WorkshopsArchive/pdfs/wp129s1.pdf) ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 22, 2022, 1:29am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/16 "2022-02-22T01:29:40Z")
> [@sblionel](#): > > But no. the comment does not suggest “a problem with the current Intel oneAPI compilers & tools”. AVX512 vectorization can cause computations to occur in a different order, and floating-point computations can be sensitive to that. Thank you for your detailed explanation. > [@sblionel](#): > > If the VASP test is looking for a specific exact value for some computation, any different instruction sequence, while mathematically still correct, can result in small (or sometimes not so small) changes in the final value. Yes. This is exactly what happened in VASP validation test. > [@sblionel](#): > > I suggest you take a look at [Improving Numerical Reproducibility in C/C++/Fortran (supercomputing.org)](http://sc13.supercomputing.org/sites/default/files/WorkshopsArchive/pdfs/wp129s1.pdf) I looked it over, but can’t find a recommended compilation option from your lecture above to deal with the VASP tests discussed here. Regards, HZ ---
**Author:** ![sblionel](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/sblionel/32/853_2.png) [@sblionel](https://fortran-lang.discourse.group/u/sblionel)\ **Post date:** [February 22, 2022, 2:32pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/17 "2022-02-22T14:32:35Z")
The simplest thing, I think, is to not use -xHost when running the VASP validation test. You shouldn’t be using -xHost in an MPI environment anyway, unless all your nodes have the same processor. If the VASP validation test is looking for an exact FP value, that’s its error. If it accepts a small variation, perhaps the range needs to be widened a bit to accommodate more advanced vectorizations. ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 22, 2022, 3:05pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/18 "2022-02-22T15:05:46Z")
> [@sblionel](#): > > The simplest thing, I think, is to not use -xHost when running the VASP validation test. You shouldn’t be using -xHost in an MPI environment anyway, unless all your nodes have the same processor. What about `-march=native` then? > [@sblionel](#): > > If the VASP validation test is looking for an exact FP value, that’s its error. If it accepts a small variation, perhaps the range needs to be widened a bit to accommodate more advanced vectorizations. I’m not so sure either, so I have raised this question on the VASP forum. Regards, HZ ---
**Author:** ![sblionel](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/sblionel/32/853_2.png) [@sblionel](https://fortran-lang.discourse.group/u/sblionel)\ **Post date:** [February 22, 2022, 3:23pm UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/19 "2022-02-22T15:23:23Z")
> [@Hongyi](#): > > What about `-march=native` then? That’s a gfortran option - at least, it’s not documented for ifort. But for gfortran, it behaves much the same as ifort’s -`xHost`, and I would not recommend that in a mixed-node environment. Instead, verify the instruction set level of all of the execution nodes and use the greatest common set. ---
**Author:** ![Hongyi](https://avatars.discourse-cdn.com/v4/letter/h/5fc32e/32.png) [@Hongyi](https://fortran-lang.discourse.group/u/Hongyi)\ **Post date:** [February 23, 2022, 1:19am UTC](https://fortran-lang.discourse.group/t/how-to-correctly-determine-the-march-flag-when-working-with-intel-oneapi-on-intel-processors/2826/20 "2022-02-23T01:19:42Z")
Got it. Thank you very much. HZ