I wanted to announce that I was finally able to make open-source a framework for a Fortran compiler test suite that I’ve been working on.
There is still plenty to do, but this demonstrates a proof of concept. Namely there is still work to do on deciding how to organize the tests, and how one might want to filter/organize the outputs, and there aren’t many tests in the suite yet. But the framework can interact with a test case in any way needed, i.e. command line arguments, files, standard input, standard output/error, etc.
If you are at all interested in using this or contributing to it, feel free to reach out and ask me questions.
I noticed the remarks by @aradi about their testing framework and how that brought various compiler bugs to light. Perhaps these bugs can inspire more tests?
I’d love to be able to automate the production of the feature support tables that have historically been produced, like the one below for example, and have it published somewhere. That’s a somewhat longer term goal, as we need a suite of tests for all those features first.
@everythingfunctional Looks great! I think it would be very helpful for the community if we could have the equivalent of cppreference’s compiler support table: C++ compiler support - cppreference.com
Not just what compiler supports what features, but also when support was added. Obviously this is no mean feat, but an RSE can dream
Also some more repos with collections of feature examples that might be interesting:
IMO it would be better to start with/prioritize by the features people care about, rather than a particular standard in which they were introduced. You could organize them in the table by when they were introduced, as has been done in the past.
I’d also love to start measuring how well compilers catch invalid code and produce good error messages. I’m not entirely sure how to organize such a test suite yet, but this framework should support it.
@everythingfunctional Very nice idea! Quick question: Do you plan to add all community provided tests to the repository with the framework? Or would it be possible to have one repo with the framework and a separate one with the tests?
I could imagine scenarios, when one wants to use the framework, but without the community added tests. As an example, I could track all the reproducers of compiler bugs, which I had to write workarounds for in Fortuno. Then, whenever a new release of a given compiler appears, I just run the compiler-tester with the project specific tests, and can check, whether the issue had been fixed and the workarounds can be removed. I’d probably even consider to put those tests into the project repository itself. (For sure, they might/should be also added to the community driven repository, but having those relevant for a given project in the project’s own repository ensures an easier workflow for the developer of that project…)
I think it should work either way. I’d love to have the “community maintained” tests (whatever anyone is willing to write) in this repo, but the framework works just fine pointing at any suite of tests. I.e.
I’m also very interested in adding the features to the framework and come up with an organizational structure or tag convention that makes selecting relevant tests and formatting the outputs convenient for use cases like this.
I have had several test frameworks targeted at different uses. None is quite the same as ths one, but for automation the lowest-maintenance solution was to write out a simple default ASCII report but have an option to write out the information in a NAMELIST format. NAMELIST groups are so easy to read and write that they make a very natural format for passing date into filter programs that then can convert it to SQL inserts to load a database or SQLite3 file, an HTML document, or a CSV file (easily imported into most spreadsheets). More recently perhaps TOML or JSON files would be candidates for the metafile but NAMELIST is built into Fortran and there is a python module for reading them as well that has made it ideal.
More recently I have been consolidating some of the modules most used for testing into a collection for further refinement but my goal is focused primarily on supplementing the fpm subcommand test so that anyone that sets their tests up to be run via “fpm test” can then use the exact same CD/CI scripts in github to automate their execution; without giving up the ability to run the tests outside of fpm and to still be able to generate more presentable reports or pass the test results to spreadsheets, databases, web pages, … The examples at unit_test_results show an example of output automatically formatted as HTML, CSV, and tex t. So there might be a few ideas used in M_framework that could be useful albeit probably not directly transferable if looking for low-maintenance hopefully long-lived solutions for automating testing.
Even though M_framework is a WIP for the foreseeable future it is functional. The majority of the more popular dozen or so modules in general-purpose-fortran | General Purpose Fortran Cooperative use it if you are interested in examples. All the tests for M_strings use it, for example.
So there are a few possible automation concepts used there that might be useful food for thought.
