Books about optimization of numerical software?

[meow464]

There are:

1. Language level optimizations;
2. Compiler pass optimizations;
3. Things that basically have to be hand written in assembly (think OpenBLAS).

I would like resources on any of them but specially 2 and 3. I tried search engines but all results are on software for mathematical optimization (operational research).

I was able to find 3 books but they are all from ~2000 and I guess the basics should remain the same but CPUs have changed a bit since then: multi core is much more popular, performance and efficiency cores, vectorization, etc.

1. Performance Optimization of Numerically Intensive Codes;
2. Software Optimization for High-Performance Computers;
3. Software Optimization Cookbook: High-Performance Recipes for the Intel Architecture;

[oscardssmith]

Honestly, one of the best resources here is Julia’s source code, libraries, and tutorials.

• https://www.youtube.com/channel/UCDtsHjkOEMHYPGgpKX8VOPg is a series of 12 videos on all the numeric methods needed for solving differential equations (including preconditioning, sparse methods, implicit methods, autodifferentiation, PDE stencils, gpu programming, profiling and optimization, probabilistic programming, and sensitivity analysis. For a text form, DifferentialEquations.jl: Scientific Machine Learning (SciML) Enabled Simulation and Estimation · DifferentialEquations.jl has a ton of information about benefits of different methods.
• GitHub - JuliaLinearAlgebra/Octavian.jl: Multi-threaded BLAS-like library that provides pure Julia matrix multiplication is a Matmul implimentation written in a high level language that beats MKL in performance. It’s not the absolutely most readable (it aims for optimal performance) but it’s way better than anything in pure assembly.
• julia/base/compiler at master · JuliaLang/julia · GitHub is fairly dense, but it shows the code for a lot of core compiler techniques from effect analysis, to abstract interpretation (for type inference), to IR manipulation (SSA, phi nodes etc)
• Juliacon (starting tomorrow) will have a bunch of talks about language level features from high performance garbage collection, to how to write a LIBM in a high level language., to GPU optimization. (schedule here, https://pretalx.com/juliacon-2022/schedule/ sign up is free)
  Let me know if you have more specific questions.

[ivanpribec]

Introduction to High Performance Computing for Scientists and Engineers | CRC Press

Also see the book pages at Georg Hager's Blog | HPC Book

[Carltoffel]

Georg Hager and Gerhard Wellein are very nice people, I attended their performance optimization course half a year ago. Since the course material is online accessible, maybe you want to check it out:

https://moodle.nhr.fau.de/course/view.php?id=4

[meow464]

to how to write a LIBM in a high level language

I was not able to find this one. Could you link it please? Also will those talks be available on youtube afterwards?

[oscardssmith]

Optimizing Floating Point Math in Julia | JuliaCon 2022 (it’s my talk). All talks are going on to youtube as soon as they go live. Also for the source code, julia/base/special at master · JuliaLang/julia · GitHub has most of the implementations (the code organization isn’t perfect but it’s mostly decent)

[septc]

These books also seem very readable and informative (though I have read only 10 % of each yet…). Both were published in 2021, so I guess they are relatively new.

Parallel and High Performance Computing

The Art of Writing Efficient Programs | Packt

[Jamie]

Goedecker, Goove names come to my mind. Two different authors.