# Numeta: A JIT Compiler for Python that Generates Fortran Code

**URL:** <https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711>\
**Category:** Announcements\
**Created:** [October 11, 2024, 5:13pm UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711 "2024-10-11T17:13:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrea](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@andrea](https://fortran-lang.discourse.group/u/andrea)\
**Post date:** [October 11, 2024, 5:13pm UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711/1 "2024-10-11T17:13:23Z")

</div>

Hey everyone!

I’ve been having a lot of fun working on a side project called Numeta. It’s a super simple Just-In-Time (JIT) compiler for Python that focuses on metaprogramming. Think of it as a dumb and lightweight version of Numba, but instead of trying to do all the fancy bytecode stuff, I’m keeping it simple by trying not to read the AST and generating good old Fortran code (because real programmers write Fortran in any language, no, actually because its very easy to [translate Numpy code to Fortran](https://fortran-lang.org/learn/rosetta_stone/) ).

The idea is to use Python’s type hints to tell apart what’s compiled and what’s done at runtime. So far, it’s been really fun seeing this thing take shape! It’s in a very alpha stage (currently depends on `gcc` and `gfortran`, but there’s no inherent blocker for using other compilers, provided they support `iso_c_binding`). The code generation and JITting works, and I’m currently trying to figure out the next steps for the project.

Right now, I’m considering some possible directions:

1. General improvements (e.g., adding support for return types is a must).
2. Expand support for more NumPy functionalities. This might require linking against a LAPACK library, which introduces challenges in managing LAPACK as a dependency.
3. Make it possible to run the code without the decorator (as in Numba), mainly to facilitate debugging.
4. Expose more of Fortran’s functionalities directly, though this could complicate the previous point.
5. Change the backend to translate directly to a compiler intermediate representation (preferably at a higher level). While this would involve significant work, it could remove the need to generate source code, making the JIT process more efficient and seamless.

However, here’s a quick taste of what it does:

```python
import numeta as nm

@nm.jit
def mixed_loops(n, array: nm.f8[:, :]) -> None:
    for i in range(n):
        for j in nm.frange(n):
            array[j, i] = i + j

```

This generates this Fortran code (n=3) that after is compiled and executed:

```fortran
subroutine mixed_loops(fc_n1, array) bind(C)
    use iso_c_binding, only: c_int64_t
    use iso_c_binding, only: c_size_t
    use iso_c_binding, only: c_double
    implicit none
    integer(c_size_t), dimension(0:1), intent(in) :: fc_n1
    real(c_double), dimension(0:(fc_n1(1)+(-1_c_int64_t)), 0:(fc_n1(0)+(-1_c_int64_t))), intent(inout) :: array
    integer(c_int64_t) :: fc_i1
    integer(c_int64_t) :: fc_i2
    integer(c_int64_t) :: fc_i3
    do fc_i1 = 0_c_int64_t, 2_c_int64_t
        array(0, fc_i1) = (0_c_int64_t + fc_i1)
    end do
    do fc_i2 = 0_c_int64_t, 2_c_int64_t
        array(1, fc_i2) = (1_c_int64_t + fc_i2)
    end do
    do fc_i3 = 0_c_int64_t, 2_c_int64_t
        array(2, fc_i3) = (2_c_int64_t + fc_i3)
    end do
end subroutine mixed_loops

```

If anyone’s curious or has suggestions (especially if you know a reasonable way to ship or link LAPACK dependencies, or have insights on the directions I’ve mentioned), I’d love to hear your thoughts. I’ve just started adding more features and would appreciate any feedback.

Feel free to check it out here: [GitLab Repo](https://gitlab.com/andrea_bianchi/numeta)

---

<div class="post-metadata">

**Author:** ![Beliavsky](https://avatars.discourse-cdn.com/v4/letter/b/ba8739/32.png) [@Beliavsky](https://fortran-lang.discourse.group/u/Beliavsky)\
**Post date:** [October 11, 2024, 5:58pm UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711/2 "2024-10-11T17:58:44Z")

</div>

Welcome to the forum, and thanks for informing us of your interesting project. Are you aware of pyccel, which “accelerate[s] _Python_ functions by converting them to _Fortran_ or _C_ functions.”?

> **[GitHub - pyccel/pyccel: Python extension language using accelerators](https://github.com/pyccel/pyccel)**
>
> Python extension language using accelerators

The developers wrote about it in [Journal of Open Source Software: Pyccel: a Python-to-X transpiler for scientific high-performance computing](https://joss.theoj.org/papers/10.21105/joss.04991), claiming orders-of-magnitude speedups.

---

<div class="post-metadata">

**Author:** ![andrea](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@andrea](https://fortran-lang.discourse.group/u/andrea)\
**Post date:** [October 12, 2024, 10:58am UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711/3 "2024-10-12T10:58:32Z")

</div>

Thanks! I wasn’t aware of it, but it sounds cool. I’ll check it out and try to replicate their benchmarks when I can. I did a quick comparison of Numeta vs. Numba (just because I had it installed) using matrix multiplication to have an idea:

```python
import numpy as np
import numeta as nm
import numba as nb
import time

n = 500
iterations = 1000 

a = np.random.rand(n, n)
b = np.random.rand(n, n)
c_nm = np.zeros_like(a)
c_np = np.zeros_like(a)

@nm.jit
def matmul(a: nm.f8[:, :], b: nm.f8[:, :], c: nm.f8[:, :]):
    c[:] = 0.0
    for i in nm.frange(a.shape[0]):
        for k in nm.frange(a.shape[1]):
            c[i, :] += a[i, k] * b[k, :]

start = time.time()
matmul(a, b, c_nm)
numeta_compile_time = time.time() - start
print('Numeta compilation and first execution time:', numeta_compile_time)

start = time.time()
for _ in range(iterations):
    matmul(a, b, c_nm)
numeta_exec_time = (time.time() - start) / iterations
print(f'Numeta execution time for {iterations} iterations:', numeta_exec_time)

@nb.jit
def matmul_np(a, b, c):
    c[:] = 0.0
    for i in range(a.shape[0]):
        for k in range(a.shape[1]):
            #c[i, :] += a[i, k] * b[k, :]
            for j in range(b.shape[1]):
                c[i, j] += a[i, k] * b[k, j]

start = time.time()
matmul_np(a, b, c_np)
numba_compile_time = time.time() - start
print('Numba compilation and first execution time:', numba_compile_time)

start = time.time()
for _ in range(iterations):
    matmul_np(a, b, c_np)
numba_exec_time = (time.time() - start) / iterations
print(f'Numba execution time for {iterations} iterations:', numba_exec_time)

np.testing.assert_allclose(c_nm, c_np)
print("Results match between Numeta and Numba implementations.")

```

And i got very competitive results (I had to remove the array operation in the Numba implementation because it was degrading the performance too much):

```auto
Numeta compilation and first execution time: 0.15936017036437988
Numeta execution time for 1000 iterations: 0.015630727052688597
Numba compilation and first execution time: 0.33567142486572266
Numba execution time for 1000 iterations: 0.019384422063827515
Results match between Numeta and Numba implementations.

```

(Even though I think it is more like a gcc vs llvm comparison)

---

<div class="post-metadata">

**Author:** ![FedericoPerini](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/federicoperini/32/1750_2.png) [@FedericoPerini](https://fortran-lang.discourse.group/u/FedericoPerini)\
**Post date:** [October 12, 2024, 12:32pm UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711/4 "2024-10-12T12:32:17Z")

</div>

> [@andrea](#):
>
> a reasonable way to ship or link LAPACK dependencies

Welcome @andrea, you may want to check our modern Fortran implementation of LAPACK in the [Fortran Standard Library](https://github.com/fortran-lang/stdlib), its packed into few source files and you can easily integrate it with either fpm or CMake

---

<div class="post-metadata">

**Author:** ![andrea](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@andrea](https://fortran-lang.discourse.group/u/andrea)\
**Post date:** [October 13, 2024, 3:06pm UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711/5 "2024-10-13T15:06:23Z")

</div>

Sounds very cool. Maybe I could add fpm as a dependency, and if there’s a way to access the interface from each package, I could make them callable from Python (though it wouldn’t be easy, especially since I’d need to add support for derived types). I’m still quite new to this—I’m just a chemist, not a software engineer, and I’m not very familiar with fpm, so I might be off on some details. Do you know if each package includes an interface file that I could parse to create the Python wrapper?

---

<div class="post-metadata">

**Author:** ![certik](https://yyz2.discourse-cdn.com/free1/user_avatar/fortran-lang.discourse.group/certik/32/4_2.png) [@certik](https://fortran-lang.discourse.group/u/certik)\
**Post date:** [October 17, 2024, 4:08am UTC](https://fortran-lang.discourse.group/t/numeta-a-jit-compiler-for-python-that-generates-fortran-code/8711/6 "2024-10-17T04:08:26Z")

</div>

There is a fairly complete list of Python compilers at the bottom of [https://lpython.org/](https://lpython.org/). I need to add yours in. 🙂