(I have tried all except Copilot, which works with specific IDEs.) Are there others? It would be interesting to come up with a benchmark of prompts in English asking an LLM to generate Fortran code and compare the performance of the LLMs in terms of their speed and code correctness. Maybe tasks from Project Euler, Rosetta Code, or Advent of Code? Comparisons could be made across programming languages. In the future it may be more important that an LLM can write correct code in a language than that humans can. “One-shot” tests of whether an LLM can generate correct code from a prompt can be supplemented by multi-shot tests where the LLM is given compiler error messages, is informed of run-time errors, and is given unit tests that verify a program and is allowed to iterate. Ideally an LLM incorporates a search engine so that it links to online code for a task if it exists.
The ability to translate code in Python/NumPy or Matlab to other languages can also be compared.
From my personal perspective, Claude 3 Sonnet seems superior to ChatGPT 3.5 and Gemini for Fortran tasks. I haven’t done any systematic benchmarks though. I only prompt in Polish, never in English, and Claude appears to generate the best Fortran code despite its Polish language skills being quite evidently worse among these models.
By the way, I really recommend trying this: https://lmstudio.ai/ There is a vast selection of models which can be downloaded and run locally from LM Studio. Edit: If a downloaded model doesn’t load in LM Studio, you may need to deactivate GPU Acceleration in the Hardware Settings.
Earth system models (ESMs) are vital for understanding past, present, and future climate, but they suffer from legacy technical infrastructure. ESMs are primarily implemented in Fortran, a language that poses a high barrier of entry for early career scientists and lacks a GPU runtime, which has become essential for continued advancement as GPU power increases and CPU scaling slows. Fortran also lacks differentiability - the capacity to differentiate through numerical code - which enables hybrid models that integrate machine learning methods. Converting an ESM from Fortran to Python/JAX could resolve these issues. This work presents a semi-automated method for translating individual model components from Fortran to Python/JAX using a large language model (GPT-4). By translating the photosynthesis model from the Community Earth System Model (CESM), we demonstrate that the Python/JAX version results in up to 100x faster runtimes using GPU parallelization, and enables parameter estimation via automatic differentiation. The Python code is also easy to read and run and could be used by instructors in the classroom. This work illustrates a path towards the ultimate goal of making climate models fast, inclusive, and differentiable.