Edgar Dijsktra wrote in his 1972 ACM Turing Award lecture (EWD340) titled the The Humble Programmer,
To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming had become an equally gigantic problem. In this sense the electronic industry has not solved a single problem, it has only created them, it has created the problem of using its products. To put it in another way: as the power of available machines grew by a factor of more than a thousand, society’s ambition to apply these machines grew in proportion, and it was the poor programmer who found his job in this exploded field of tension between ends and means. The increased power of the hardware, together with the perhaps even more dramatic increase in its reliability, made solutions feasible that the programmer had not dared to dream about a few years before. And now, a few years later, he had to dream about them and, even worse, he had to transform such dreams into reality! Is it a wonder that we found ourselves in a software crisis? No, certainly not, and as you may guess, it was even predicted well in advance; but the trouble with minor prophets, of course, is that it is only five years later that you really know that they had been right.
Given that the selling point of these GPUs is supposed to be AI and LLMs, Dijkstra had to say the following:
As an aside I would like to insert a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem. Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity.
I wish they would call AI by its more descriptive name: a multivariate least-squares fit to data.
They used to be called neural networks, then it became machine learning and now it’s AI. However, the basic technique (which now requires this absurd amount of computer power) is the back propagation algorithm for calculating the gradient of a cost function (usually the square of the difference between the data and the neural network output).
The problems with least-squares fitting apply to both small and large data and parameter sets: they are susceptible to over fitting, they don’t do well with discontinuous or unbounded data, and they can be poor at extrapolation.
When the AI hype has settled down, we’ll have another, potentially useful, numerical technique to including in our tool set, like numerical linear algebra or the FFT.
Incidentally, we use neural network models in our work to fit the non-linear optical response of materials. We certainly don’t call it AI though.
I would say (but I don’t claim I’m right) that ML uses large scale non-linear optimization and prediction, and that AI uses ML.
Cerebras Systems has announced its third-generation wafer-scale engine (WSE-3) for AI: the chip size is a whole wafer! (46,225 mm²) 
With 4,000 billions transistors… and 900,000 cores.
I don’t know the price.
Scientific and technological progress is never ending: each time it solves a problem, it creates two new ones. Engineers and scientists can never be bored…
All these terms are usually represented in an onion-like structure where AI encompasses them all: AI > ML > Neural Networks.
But at the end of the day, it’s only statistics (quite advanced I must concede)!
Much machine learning is trying to predict binary or categorical outputs, for which least-squares is not the proper criterion. Of course, statisticians have been employing techniques such as logistic regression for these problems for decades – Alan Miller has Fortran codes for this.
Super Micro Computer is now a $60 billion market cap company because of its sales of servers with NVIDIA GPUs. One of their brochures says
Software Development Kits are critical for getting the maximum performance from a GPU. The following SDKs are supported on the Supermicro Universal GPU servers:
• The NVIDIA HPC SDK C, C++, and Fortran compilers support GPU acceleration of HPC modeling and simulation applications with standard C++ and Fortran, OpenACC® directives, and CUDA®. GPU-accelerated math libraries maximize performance on common HPC algorithms, and optimized communications libraries enable standards-based multi-GPU and scalable systems programming. Performance profiling and debugging tools simplify porting and optimization of HPC applications, and containerization tools enable easy deployment on-premises or in the cloud. The HPC SDK provides the tools needed to build NVIDIA GPU-accelerated HPC applications.
• AMD ROCm™ is an open software platform allowing researchers to tap the power of AMD Instinct™ accelerators to drive scientific discoveries. The ROCm platform is built on the foundation of open portability, supporting environments across multiple accelerator vendors and architectures. With ROCm 5.0, AMD extends its platform powering top HPC and AI applications with AMD Instinct MI200 series accelerators, increasing accessibility of ROCm for developers and delivering outstanding performance across critical workloads. In addition, various precision-based math is available, which enables acceleration for a wide range of workloads.
Yes, most of the ML references I’m familiar with tend to break down ML into two camps, regression (multi-variate LS fitting) and classification (selection between various classes of data, ie image processing, language processing etc). There are now about as many books about ML/AI as there are grains of sand on a beach but one I can recommend for someone with no ML experience who just wants an easy to follow and not math heavy introduction to ML is Perrotta’s “Programming Machine Learning” book. Its only drawback is its based on TensorFlow/Keras where most of academia now uses PyTorch (or so I’m told). Just using this book I was able to go from not knowing anything about ML (and very little Python) to putting together a regression model for some large scale CFD simulations (using the Intel oneAPI ML toolkits) in a couple of weeks.
Also in the news this month:
AVERAGE CPU USE AT 13 PERCENT
Cast AI researchers also took a look at how many of the provisioned CPUs developers actually use. That number came in at 13 percent on average. They looked to see if the numbers were any better in larger clusters, but in those with 1,000 CPUs or more, CPU utilization only reached 17 percent.
Clusters with 30,000 or more CPUs reached 44 percent utilization, but only represented 1 percent of the systems they looked at.
All this pointed to huge CPU overprovisioning, with the bulk of compute power sits idle.
So despite having so much hardware, we really don’t know what to do with it.
Greg Wilson of the Software Carpentry project wrote a nice article for Scientific American titled “Where’s the Real Bottleneck in Scientific Computing” about the software/hardware imbalance. As he concludes in that article (and also echoing Dijkstra’s thoughts):
Faster chips and more sophisticated algorithms aren’t enough—if we really want computational science to come into its own, we have to tackle the bottleneck between our ears.