I think this is headed in the right direction of a workable solution.
1 Like
Then why to use fpm in the first place? If a user has full control on all the packages he’s combining (they are all in his stack or from collaborators), shouldn’t he be able to take responsibility over his actions and join them at his own risk?
So a non-conforming project could be used (but not registered), but maybe with a loud warning punched directly in the face, so to warn that this is highly discouraged behavior unless you really know what you are doing (basically self-managing a nonconforming ecostystem).
I mean there is nothing worse here than the usual CMake projects, where there is no enforcing whatsoever on module names and clashes could happen anytime. So if CMake allows for brewing dangerous combinations and fpm does not, in any way, people that do not want to migrate to the new established convention would go the full outcast way, and never change the build system. If you allow them, instead, to keep doing their stuff with just the work needed to change build system (which already can be nontrivial, adn that’s unavoidable), they could gradually fall in love with the new improved workflows and at some point evaluate going to full conformance, for their own benefit (that they had a taste to).
On a second thought I believe I’m convinced that this could be a very well crafted common ground. Afterall fpm is a package manager and should ensure a sane package management, it’s in the name. Allowing nonconforming users to still build is an appreciated concession, that already gives much in terms of “low barrier for gradual adoption”.
Last thing: is there a problem in allowing to prefix with both the custom namespace and the full package name, if one wants? It would allow for easier branding of the main (“top level”) module, in the many cases where it may apply. Than the specific modules, that inevitably would be present in a large project, could reduce their name length with the custom prefix. As an example Test-Drive could have a main module called test_drive and all the different modules that define the different parts of the library could be prefixed with the shorter td, undoubtedly better than test_drive_something, that could even mislead users to think that the main library is ‘test’ and driver_something is a part of it. Still use td might look too generic and obscure in high level user code, while use test_drive is quite representative and clear for the reader.
I don’t think there are inherent technical problems with this: package name clash is (will be) already forbidden by fpm, and the proposal is to check against “custom prefix” clash too. I do not see the need to enforce a xor instead of an and here (I understand that anyway choosing the full name of another package would qualify as an illegale custom prefix, right?). Probably it would just be a matter of taste about what you want or do not want to allow, and the perceived freedom for the community, to work out their best liked setup. Let me be clear: naming in software design is a though thing to do, involving both practical, technical and creative aspects. You don’t want to get stuck with a name that you don’t like (for whatever taste reasons), you don’t want to get stuck with an impractical name either. Changing things after 1.0 is very painful and no one wants that. A too strict convention would reduce too much the perceived room for this delicate decisional process. Let’s leave at least some space for designing choices.
After writing: I still have this bad feeling we are just trying to circumvent a fundamental issue in the language and that the proper solution would be to take action at that level. It can become really frustrating to know that the standard would come to the issue in several years, so we are forced to work our way around this with poor compromises in the package manager. 
I think what @milancurcic summarized plus @aradi’s addition is the way to go.
I’ll add one more addition on top: with module-namespace = false the idea is that you will use it for end applications that are huge. Fpm will refuse to use any such package as a dependency by default. However, we’ll provide an option to override, on a command line (not in fpm.toml). That way there is no limitation and if you know what you are doing, you can. But by default fpm will not allow that, thus ensuring that modules cannot collide.
3 Likes
Oh, wow that’s the thing! 
.
I like it very much for that you are right that an official toml entry should imply best practice level settings. So indeed, unsafe mixing of packages could not be allowed by a legal toml configuration.
With an explicit flag every time you invoke fpm you are promising that you know what you are potentially getting yourself into, so you have the feeling that that’s not a best practice. Perfect. (if you write the fpm command with the flag in a response file, yaml script for CI or whatever you’re basically signing a written contract containing that promise). I think this could be the decisive new idea on the table.
1 Like
It appears there is convergence here on what to do OSTD (outside the standard) with fpm and that’s great to see.
@certik et al., is there anything that can be introduced in the language standard, say in Fortran 202Y and/or 203X, that can enable better packaging ecosystem for Fortran? You will be well aware of the issue threads that are open at the J3 Fortran site. Plus the committee is considering preprocessing facilities for 202Y.
It will be good to also keep the language standard in mind and consider viable options to continue evolving it toward better and better consumption of Fortran in a standard manner well into the future.
@FortranFan I think Fortran has to have a package style like Python, so nesting modules: Allow modules to be nested like in Python · Issue #86 · j3-fortran/fortran_proposals · GitHub.
We have it half implemented in LPython already, so we can prototype this in LFortran as well very soon.
We just need to start making progress on these “basic” language features.
2 Likes
One comment on that: The two-letter namespace will be very soon crowded/exhausted, IMO, if all projects register an additional two-letter alternative of their projects name as well. I would opt for one prefix per project, consistently on all levels.
Also, on the question of how to handle - and _ in project names: I’d opt for stripping them completely in the name space prefixes. A name space prefix should not contain any underscores. Because then, whenever you see a module name someprefix_somesuffix_somesuffix2, it would be clear that the part of the module name until the first underscore is the library namespace prefix. As discussed above, the toml-f library should then be identified by the tomlf prefix, and test-drive should use the prefix testdrive, so no ambiguosity could occur, whether test_drive_something is part of the test-drive library or is a test module in an arbitrary project.
5 Likes
I like the way of declaring namespaces in fpm.toml for fpm packages, packages should need a dedicated namespace and be explicit.
Can we extend the fpm-specific comment syntax for source code, like openmp, such as
!$fpm import 'a.types'
program main
use types
...
Specify explicitly which package’s types module to use in this source file. This may require fpm to be able to handle complex header file paths in the backend, so that users are not forced to obey the package_xx_xx naming style.
1 Like
I agree with @aradi. I am worried too that the two letter namespace will be exhausted soon. Python handles this by using longer prefixes and renaming at import, such as:
import numpy as np
import scipy as sp
etc.
Python packages are not typically named just np.
In Fortran unfortunately we do not have this module renaming capability (and accessing symbols via the module name), that is the proposal here: Namespace for modules · Issue #1 · j3-fortran/fortran_proposals · GitHub.
@milancurcic the reason you use nf_layer is because neuralfortran_layer is too long?
How are we going to handle this in the central registry? First come first serve? What if I have a library which also uses the nf prefix? I will be forced to rename my prefix. Do you agree?
So if we are going to force people to rename their prefix, because two letters will be exhausted soon (just 676 combinations!), we might as well force people from the beginning to use the prefix that is equal to the repository name.
(I think we all agree that we will always allow fpm to relax the restrictions with a command line option; the discussion is about the default behavior as well as enforced restrictions at the registry.)
1 Like
While the two letter prefixes being exhausted quickly is unfortunate, I don’t think the plan to is dictate they be exactly two letters. So once the two letter prefixes are exhausted, we’ll move on to three and four letters, where the space grows quickly (26^n letters); 475254 total possibilities with 1-4 letters.
2 Likes
Yes, and that’s exactly the point I was trying to convey before. Python packages have the renaming to overcome the problem with having a recognizable and well thought name (that would inevitably be relatively long), but still avoid polluting the source code with long strings. Forcing a unique prefix would lead to either choosing long prefixes (negating the point) or using a short alias everywhere… which in the end means your package is called that way. Who cares about the name in the toml if all appearance in source code are with a different name. Doesn’t it just makes more confusing for people? Like I see everywhere this nf thing (note that even the “main” module could not be named neuralfortran, but nf_main or something along the lines) but then if I put nf in the toml nothing is found (the same if in the future would exist a fpm add command to interactively modify the toml, which I think would be natural to have, once the registry is hosted somewhere).
I’m not sure I fully understand this. Overall I agree, but I don’t see that much difference in the ‘additional’ vs ‘unique’ adjective. That’s because I think, one way or the other, it would/should be forbidden to set a prefix that correspond to the project name of an already registered package (even if they are using a custom prefix so technically there is not a clash). Allowing this would make very confusing the discovery of packages, example:
-
user 1 registers a package named ExactDiagonalization, but finding the name too long to be used as a prefix, he opts for
ED. First come first served. -
user 2 has another exact diagonalization library and founds that he can’t use
EDas a prefix. The next obvious thing to do is to pick the fullExactDiagonalizationprefix so he goes that way.
Do you see how confusing this becomes for people navigating the ecosystem? Fpm should anyway disallow taking another package’s name as your prefix.
If you really want to avoid the quick pollution of short names then you don’t want to allow for custom prefixes at all and return to the first point: no freedom, you get to choose only the package name and prefix with that all your modules.
Yes but note that the subspace of actually usable stuff is much smaller: no one would want to prefix with abcd or xxx (unless is a spicy package). Let’s not reason as they are hashes, if that’s the prefix of all your modules (so the first thing you see in the API of a library) it has to be meaningful and good-looking, catchy, etc.
1 Like
Just to clarify, I am fine with two letter prefixes (name space identifiers), provided they are used consistently for all modules of a library. And I also support the idea of choosing a prefix (even a two letter one, if you wish), which is an abbreviation of the project name, if the project name is too long.
What I am opposing, is the possibility to register multiple prefixes for a project. Library authors should decide, what the namespace of their library is and stick with that in all modules, independent whether those are high level “public” modules, or “private” modules with implementation details.
Yes. I meant the package name in the manifest to also be “nf” but it’s not yet only by omission. I’ll fix that.
I don’t know how best to handle this from the registry point of view. If you allow two packages in the registry to have the same namespace (my “nf” and your “nf”), they could still co-exist, they just couldn’t be combined as dependencies in a project.
Alternatively, you could treat it as a first-come-first-serve, with some exceptions, for example by reviewing on a case-by-case basis if an existing squatter name is blocking a more mature and established library from entering the registry, something like PyPI. But we already have requirements for a package to enter a registry, so squatter names would be unlikely to happen.
1 Like
Is this shorthand because the programmer must use the np and sp names when they access the contents of the modules?
If there are two, e.g. numpy, modules that you want to use simultaneously in python, then how does the import statement look to allow that?
I’ve never had to do that in practice, because PyPi only allows to have one package with a given name.
But if you had two numpy packages with the same name, I think you could use something like:
import numpy as np1
import sys
sys.path.insert(0, "/path/to/other/numpy")
import numpy as np2
1 Like
Yeap, in addition by default imports are resolved based on the PYTHONPATH variable
In general for distributed modules I agree, but for building applications or creating a package composed of multiple existing packages I would want to be able to opt out in the fpm.toml file. If I gathered a number of existing packages up into a single package I would not want to have to rename all the components; and if I was building a program with fpm and wanted it to be self-contained, especially on an isolated machine I would like to be able to copy all the components into a single repository I think. I need to rebuild a version that enforces the rule to play with it. Perhaps placing the individual projects into subdirectories under the project root (or links) and continuing to use them as dependencies would be fine (maybe even preferable) ; but I think it needs some real-world testing. If the default remains the same except that the package needs an additional line added to the fpm.toml file to be used as an external package but warns you it is not distributable unless you have an explicit line opting out it would be inhibiting, I think. Since gfortran names the mod files with the name of the module I wonder how many current packages currently conform? You can also place modules straight in applications and examples and tests that are intended just for a single application as well that I do not think should be tested (that is, I think the constraint should only apply to files in src/). The original behavior was far more of a bother when building applications than warranted; but a good idea for distributed packages where it should encourage the pattern from the start. Any requirement enforced just be the tool (fpm) and not the language needs an option to opt out; especially when existing alternative methods (make, Cmake, …) have no such restriction.
4 Likes