Kick-off for fpm leadership team

With the NumFOCUS application of Fortran-lang and the STF project on the package registry, it is time to get more concrete with the long-term goals for the development of fpm.

Therefore, we want to create a working group to design the roadmap for fpm. If anyone is interested in joining a kick-off meeting, please let us know here. Note that this requires a more long-term commitment, of at least the next six months while the STF project is running, preferably beyond.

11 Likes

Sounds great count me in!

4 Likes

Iā€™m in.

3 Likes

Iā€™m also interested.

4 Likes

Perfect, thank you everybody. I am in too.

Letā€™s meet once a week over video initially, to resolve and make decisions about some big issues that we need, so that the STF projects can implement it.

Can you please fill out this doodle form:

Doodle

And letā€™s see if we can agree on a slot.

3 Likes

Iā€™d like to stay involved.

5 Likes

Weā€™ll meet tomorrow on Tuesday at 11am Mountain time (10am Pacific, 1pm Eastern). I sent a calendar invite to everybody who has responded in the Doodle poll so far.

1 Like

Do you have a meeting link or should I create one? Also, could you please post the time in UTC as well (otherwise Iā€™ll be paranoid about messing up my timezone calculation).

I sent the link as part of the calendar invite. It starts in 30 minutes from now. I just also sent a private email to everybody with the link. Let me know if you got it.

Agenda from my side (please post post what you want to discuss as well):

  • Pros and cons of central registry (why did Zig decide not to have it?)
  • How will we host it, and choosing a person or team responsible for keeping the server up
  • Use namespace + project (like github) or just project (like PyPi and Cargo)?
  • Enforcing prefix module naming convention at the fpm level and registry level
  • Mechanics how we will make decisions (in a timely manner), how will will design things, and hearing from past experience where we failed to do so and how to fix it.

I am hoping to keep the meeting 40 minutes long.

My summary (please correct me below if I misrepresented something):

  • Pros of a central registry:
    • robust distribution, compared to a user removing a package, say, from github
    • possibly faster download
    • you can mirror it
    • we control it
  • Cons: somebody has to maintain, and paying for server costs, overhead of resolving disputes (if any), etc.
  • Overall it seems we agreed we want a registry
  • We will work on finding a person/team to maintain it and have to figure out hosting
  • We should use a namespace + project, like github (it prevents the name squatting issue, a user/organization reserves a namespace on a first serve first come basis, but one can always choose a good looking name, just like you can still do that on github; then you can use any name you want for the project, and nobody else can access the namespace, unless you give permissions)
  • prefix: we should enforce prefixes by default; and provide a command line option to opt-out;
  • later we can discuss how to design a language feature such as namespaces for modules and implement in compilers + get it approved by the committee
  • We will meet weekly for now, and we will try to find a person or a team to lead fpm, who will push to make decisions quicker; in case of big disagreements we will meet over video (like we did just now) with all interested parties and try to resolve the issue.
4 Likes

A few notes I made:

  • Local and central registries share the same architecture; this would allow having organizations (companies, institutes, ā€¦) or even an entire country to deploy itā€™s own registry behind a firewall.
  • Existing method with git repositories should continue to work if possible. (Which namespace do such packages belong to? Does it matter?) If not possible, the sooner we break compatibility the better.
  • Introducing a version field specifier for the fpm TOML manifest could allow for breaking changes.
  • For namespaces we should evaluate design choices taken in Juliaā€™s Pkg.jl, PyPI, Spack, and Cargo/crates.io.

A few questions:

  • Since fpm also has support for C and C++ projects, should those also be prefixed? This also extends to Fortran external sub-programs (i.e. standalone functions and subroutines)
  • Will there be other requirements on packages, for example explicit default visibility in modules?
4 Likes

Should we attempt to reach out to some of these teams and see if they have any advice for us/lessons learned?

I think we should. The prefix behaviour should be the same across C/C++/Fortran to minimise user error while writing fpm.toml files.

What things in C and C++ should be prefixed? All functions, classes and structs? We actually have the same problem in Fortran with external procedures (e.g. that arenā€™t in a module). Should we also enforce that all Fortran projects have all procedures in modules? Is there a way in C/C++ to prevent name clashes?

1 Like

Given that you canā€™t enforce it for C procedures, Iā€™d be hesitant to require this for Fortran.

In C++ you have namespaces, e.g.

namespace fpm {
   // ...
}

In C things can go wrong in very subtle ways due to macros, function-like macros and inline functions. Say you offer an API with the following interface:

// process_api.h
void process(int N, double x[N]);

Now someone else might inadvertently use N, a capitalized identifier as a constant, e.g.

// definitions.h

#define N  42

Next, they write a main program:

#include "definitions.h" 
#include "process_api.h"  // boom!

int main(void) {
    double x[4] = {1.,2.,3.,4.};
    process(4,x);
    return 0;
}

This will fail to compile as the substitution of N will invalidate the header. You can fix it by swapping the order the headers are imported in. But some day you might import a third header, or hit the via a transitive header.

You can also suffer clashes with the standard library; if you would pick I as a dummy variable in a header, all it takes to break the code is to #include <complex.h> which provides I as a macro for the imaginary unit. Not to mention the potential clashes with the name process itself.

What can you do to avoid the problem? In general you prefix all public and internal functions. For the dummy arguments in the public header you can either omit the variables names, use comments or prefixed identifiers:

void pp_process(int, double []);                 // omit
void pp_process(int /*n*/, double[] /*x*/);      // inline comments
void pp_process(int pi_n, double pi_x[pi_n]);    // prefixed (pp - public, pi - internal)

You can also face clashes with member names in struct and union types. Hereā€™s a mixed C/C++ example:

// main.cpp
#include "ISO_Fortran_binding.h"

template<int rank>
class FortranArray {
private:
    CFI_CDESC_T(rank) desc;
};

int main(int argc, char const *argv[])
{
    FortranArray<5> array;
    /* code */
    return 0;
}

The code look okays, but in fact there is a name clash between the template parameter, and a member of the unnamed C descriptor struct hidden underneath the CFI_CDESC_T macro.

$ g++ main.cpp 
In file included from main.cpp:1:
main.cpp:6:2: error: declaration of ā€˜CFI_rank_t FortranArray<rank>::<unnamed struct>::rankā€™ shadows template parameter
    6 |  CFI_CDESC_T(rank) desc;
      |  ^~~~~~~~~~~
main.cpp:3:10: note: template parameter ā€˜rankā€™ declared here
    3 | template<int rank>
      |          ^~~

For C++, namespaces should take care of most of the problems. But for a C header, to be on the safe side you need to prefix all identifiers: types (both struct or not), struct and union members, variables, enumerations, macros, functions, and function-like macros. You can take the "ISO_Fortran_binding.h" header as an example; everything is prefixed with CFI_.

For C, I imagine this could be difficult to enforce. Perhaps some large C projects have developed tools to maintain name hygiene. (cc @vmagnin, maybe you are familiar with how GTK handles this?)

I donā€™t know if they have specific tools. I have found the GTK coding style document:
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/docs/CODING-STYLE.md
After a quick read, I have not found advice on naming. It seems implicit that you should use prefixesā€¦
In gtk-fortran, there are interfaces for ~10000 GTK/GLib functions and ~2500 enums, and wa can they are systematically prefixed with the name or an abbreviation (g for GLib) of the sub-library:
https://raw.githubusercontent.com/vmagnin/gtk-fortran/gtk4/src/gtk-fortran-index.csv
https://github.com/vmagnin/gtk-fortran/blob/gtk4/src/gtk-enumerators.lis

Reading the GTK/GLib .h files, itā€™s also the same for #define, structures, etc. (those things are not imported by the gtk-fortran python wrapper).

Thereā€™s some interesting discussions on why Cargo doesnā€™t have namespaces around the web. I read a bit about it a while back, but couldnā€™t find the exact same threads. Thereā€™s a summary here though with a lot of links that might be worth reading: No Namespaces in Rust is a Feature :: Sam Sieber

1 Like

I am sorry if I enter this discussion late, as I am no active fpm userā€¦ please discard if not meaningful.
Could fpm just itself build a unique global entry point (aka module) for each package, that wraps all public names in the package? Think

module my_package
    implicit none

    ! My package has potentially conflicting names
    type, public :: t
        character(len=12) :: x = 'Hello world!'
    end type t

    contains

       subroutine do_sth(this)
          type(t) :: this
          print *, 'this = ',this
       end subroutine do_sth

end module my_package

! Let fpm automatically generate this
module fpm_my_package

    ! Wrap all public stuff
    use my_package, only: fpm_t => t
    use my_package, only: fpm_do_sth => do_sth

    public :: fpm_t
    public :: fpm_do_sth

end module fpm_my_package

program with_fpm
    use fpm_my_package
    type(fpm_t) :: x
    call fpm_do_sth(x)
end program with_fpm
1 Like

It can. In fact this might be another nice thing to enforce possibly with warnings, to make sure every package has such a module, named as the package.

(If you are proposing this as a solution to fix the ā€œglobal colliding module namesā€ problem, then it would not fix it, as some compilers would fail.)

1 Like

You mean some of the major compilers do not support renaming? If you see the global renaming module as an ā€œalmost namespaceā€, I think the upside would have been that for most cases, the user could just not mind; but in case his codes had collisions, they could revert to using the namespaced interface.

Also note that this interface would also work for C-bound routines I think.