Some possibilities to consider:
I recall that Intel’s compilers can be slow with compile-time computation, as discussed here. (I guess this is just a more general version of the previously discussed implied do-loops.)
Certain compiler options can make compilation take far longer, as you are aware. I recall that -ipo with ifort can slow down compilation appreciably (same for -flto for ifx). Which compiler options are you using?
Another possibility: This is probably not your problem as what I was doing was far outside normal practice, but I have run into slow compilation times when I had a lot of interfaces (~800 functions). Specifically, I had a program which generated Fortran code including a huge number of derived types and type-bound operators. You can read about the problem on Intel’s forum here, though the bug report focuses on ifort/ifx being unable to compile the code at all, not compilation being slow. As I recall, ifort/ifx would get progressively slower as I added more interfaces/derived types, though I think ifort/ifx didn’t take that long in all cases that would compile. nvfortran was exceedingly slow for this case but would eventually compile the code. In contrast, gfortran and nagfor were quite fast and had no issues.