Fpm version 0.8.2 released: centralized registry playground

Announcements
1

Dear fortran-lang community,

We are excited to release a new version of fpm: v0.8.2.

This release introduces fpm support for uploading packages to the fpm-registry server directly from the command-line interface, via

fpm publish --token <token-here>

fpm will now interact with a web interface that will help to manage the namespaces & packages.

We have made an experimental registry available online at https://registry-frontend.vercel.app/.

We welcome testing from the fortran-lang community: here are a few steps to get started.

First of all, you will need to register a user account.
For uploading your package to the registry, you will have to step by step follow the following points:

  1. Register yourself as a user. You will require a unique username, email & password to set up your account.
  2. For uploading a package from fpm, you will have to first create a namespace. A namespace represents a collection of packages. Each package is published under a namespace in order to avoid collision of same package names. Namespace names will be unique always.
  3. Now, that you will have created a namespace with a unique name and a nice description. You can go to dashboard by from the dropdown options in the Navigation bar on top.
  4. In the dashboard, you can see the namespace that has been created by you. You can now generate a token for this namespace.
  5. Use this token to upload packages from the fpm using the CLI:
fpm publish --token <token-here>

After completing the above steps, you will receive a response in the fpm command line interface whether your upload was successful or not.

If your upload was successful, you can now again go to the registry frontend and check the dashboard. It should display the package uploaded by you.
You can now Add/Remove maintainers to your package. Mantainers have the rights to operate on the same package.

*** Please note: the current registry is a playground: its database will be fully deleted once its functionality is established. Please do not use it for production yet! more information will follow then. ****

Thanks to everyone who contributed to this release! We are eager to receive community feedback and suggestions for user experience here on Fortran Discourse.

Find the full release here.

This release was supported by the Sovereign Technology Fund (STF).

@arteevraina @henilp105 @minhdao @awvwgk @certik

16 Likes
2

Great!

First problem, I have just tried it on Linux Ubuntu 22.10:

$ fpm publish --token e...................................3
<ERROR> *cmd_publish* Tmp directory error: Couldn't determine system temporary directory.
STOP 1

The temporary directory is however the classical /tmp/

And a naive question concerning the token, should we save it? or is it just for one upload?

Remark: after the creation of the namespace, you are automatically redirected to the dashboard.

3 Likes
3

Hello @vmagnin , the token would be valid for 1 week duration only.but you can always generate a new one.

3 Likes
4

Thank you @vmagnin - this is very useful. I will ask @minhdao to open a PR for this issue.

5 Likes
5

The help key on the test centralized registry just echos “help”. If would be useful if a description like the OP shows. It would be useful for “fpm help publish” referenced a URL to go to to setup up registering and/or where documentation could be found. Some clarification is needed on whether the namespace is for all of a user’s projects or a unique one is required per project, if the package files must be buildable with fpm 8.0+ (ie. conform the the default namespace requirements starting with 8.0), etc.

A great step forward for fpm.

5 Likes
6

The way it is being designed is that it will be representative of an organization, so it could be “my own” namespace for my personal projects, but not necessarily. @henilp105 @arteevraina @minhdao please correct me if I’m wrong but:

  • namespace maintainers and admins oversee all packages in the namespace
  • package maintainers only apply to the package.
  • package and module names are unique within the namespace

This allows to handle changes in both package and organization maintainers flexibly.

6 Likes
7

Just to clarify: the above registry at https://registry-frontend.vercel.app/ is just a temporary experimental link, that we will eventually take down and all data will be lost. Don’t use it for production. But please do use it for testing and report all feedback to us. Once things are in good enough shape, we will then setup the official registry on some subdomain of fortran-lang.org.

6 Likes
8

I published my first package! It worked just like described.

image
image2634Ă—1442 184 KB

A big congratulations to all the developers involved! I really hope this leads to a new renaissance for Fortran by making it easier to build large projects and catalyzing the development of high quality and reliable/stable software packages.

12 Likes
9

Just wondering, will already registered packages be transferred to the official registry once it is launched? Is there any strategy under discussion how to control/prevent namespace squatting?

1 Like
10

Not with this revision - please consider the current one just as a playground. Once it is being tested enough that there is common ground on the rules and the functionality, then we will start backing up the database for the foreseeable future.

4 Likes
11

These are all great questions. For the name squatting in particular, I haven’t give it much thought yet, but at least from a user’s perspective the process of reclaiming a PyPi package name was quite painless, so we might want to use something similar to that.

2 Likes
12

So should this be the forum for discussion or a github repo or …? I have a few notes and quite a few questions on what should and should not be available. The oddest one is that the repository displays a list of dependencies for a project; yet I see no way to register a package with dependencies; even registered dependencies.

Running Notes 
==============================================================================================
TEMP SPACE
When looking for temporary file to create archive file in, looks at
TMPDIR,TMP,TEMP but if not set does not default to /tmp on Linux. Also
assume values ends in /. Should use join_path so that is not required,
as it is common to not include trailing slash.
==============================================================================================
ARCHIVES NEED TO BE TRANSPARENT
In verbose mode the commands being executed should be shown. The method
used requires that tar and git be not only available, but that the
package is checked in with git(1). fpm(1) has not required a particular
SVC previously. A dry run option that perhaps does a "tar tvfz" to show
exactly what is being included should be available. Just executing a
command that collects files from your system and exports them requires
too much trust. Maybe two steps are required -- create archive; export
archive.

creates /tmp/fpm/ tar file; so tar required? On MSWindows zip? Or must
package from ULS?  User needs to know "git archive" is being used, that
.gitignore is obeyed. there  is a need to know what will be in archive
file; that git is required to make the archive; and whether the current
branch, the head of the current branch or the master is used.
==============================================================================================
METADATA
Should note in documentation that license field is required, and validated
so custom licenses are blocked.  {"code":400,"message":"Invalid license
identifier Public Domain. Please check the SPDX license identifier list."}
name, license, and version are not only required, but validated. So
version must follow rules too.

It does not appear to use homepage and description, which even if in
fpm.toml and fields in the repository is not used?
==============================================================================================
Documentation and/or command should say 

 + to see https://registry-frontend.vercel.app/manage/projects
 + show example of using a repository as a dependency

==============================================================================================
SEARCH
Need a CLI interface to the repository like fpm-search.

Search on WWW needs a "browse" mode; not everyone will know to use "." to see all.
==============================================================================================
PUBLISH COMMAND
If you enter "fpm publish" it just stops without indicating token is required or prompting
==============================================================================================
REGISTRY PAGE
Not obvious how to edit description, what is used. expressions, graphics, tables, ...
==============================================================================================
NO DEPENDENCIES, EVEN DEVELOPER OR TEST

# <ERROR> Do not publish packages containing git dependencies. 'M_framework' is a git dependency.

A repository cannot be published with a dependency, even test.dependencies or dev-dependencies.
Many repositories that build a library or module without a dependency use dependencies for 
testing, example programs, ...  that will be very limiting.

At least a dependency that is registered needs allowed. If it is assumed a registered package
has already been checked for non-registered dependencies recursive checks on the dependencies
should not be required.

Registery page has a "dependencies" column?  But publish(1) does not allow dependencies?
==============================================================================================
FUNCTION_PAYLOAD_TOO_LARGE

Getting message about being too big with several repositories.
==============================================================================================

# what happens if you publish same package? If version is not changed, not published.

# any checksums generated?

# any hierarchial organization allowed? can I have a subgroup for sorting, graphics, ...?

# applications versus libraries/modules

# nothing avoiding collisions globally? 

# once published, can I delete?

# should be able to do dry run and see what is in tar file

# so how do I use as a dependency?
  Do not see description of syntax for using package as a dependency
  or how to use local repository 

# verbose mode should be echoing command executed

# https://fpm.fortran-lang.org/en/spec/manifest.html

Will there be just one repository or can someone set up their own
2 Likes
13

Congrats, fpm team! This seems like a significant feature update–why only bugfix release instead of 0.9.0? I don’t think it’s a big deal either way, just curious.

3 Likes
14

I asked myself the same question.
Is there a roadmap toward 1.0 ?

15

Since, this is just an experimental version release, we kept it as a bug fix version.

I suppose we are forecasting to release 0.9.0 when things become stable after the testing period.

3 Likes
16

My response was very different. Perhaps some is planned, but (I did not make a rigorous examination) the functionality appears to be independent of fpm. Like fpm-search before it,
I think it should be a seperate plugin command (ie. fpm-package) but unlike fpm-search it
should be distributed directly in the fpm github package. So the single-file build would just
build fpm but with fpm build you could download and install other apps like this from the fpm package. Unless there is deeply shared functionality that is not just the use of shared modules I think fpm(1) should stay at 0.8.1 and this should be a plugin that can be changed and deployed independent of the fpm(1) command myself. So my surprise was not that it was not fpm 0.9.0, but that it was not fpm 0.8.1 !

If we just keep adding programs to the inside of fpm we end up with software bloat.

2 Likes
17

So what is the limit on size? The largest directory I tried expanded (ie. not as a .tgz archive) was 108Mbytes; which does not seem excessively big; but several of the packages I tested got

Uploading package …
Request Entity Too Large

FUNCTION_PAYLOAD_TOO_LARGE

18

Virtually every package I have has a dependency on a unit testing framework package, or command line parsers and I/O routines; and so far I have to remove all dependencies. At a minimum if a dependency is registered it should be allowed. If no registered package is allowed to have the Fortran-stdlib package as a dependency it would greatly discourage its use, for example.

What is the current design goal for package dependencies?

1 Like
19

We are currently using vercel serverless functions to host our API. Which has a limit on request & response bodies of 4.5 mb.

20

Thank you @urbanjost, this is exactly the thoughtful feedback we were hoping to have! There are rules that we haven’t defined yet and we hope we can find the best fit with the community.

We will need time to review all your comments but regarding dependencies, here is what we had discussed internally:

  • the registry database should contain all sources needed to build all packages. So, all registry-fit packages should only include dependencies from the same registry
  • we have discussed about implementing package validation on the fpm side before upload. Fpm should first build the project and ensure the build and all tests pass, and dependencies are ok before uploading. not much progress has been made yet, we haven’t well clarified the rules
  • some important packages (blas, lapack, stdlib, MPI, etc….) will be implemented by fpm as “metapackages” I.e. always available at the users request, just via name (and perhaps version). So the registry won’t have to include them
  • this means the registry will be built from the bottom up, starting from the simplest packages that have no dependencies at all
4 Likes