Unit testing report generation

1

Over time a large amount of unit and regression testing has been developed for code I work with.
Of course the important part of unit testing is to verify changes work as intended without adverse effects, often on the spot or as part of batch CD/CI verification. But sometimes reports are desired.

Different tools seem to be used with different languages but there are things like TAP (Test Anything Protocol); but I do not see many consumers that generate anything but basic reports.
Is there anything out there that is language-agnostic being used to conglomerate unit testing data for reporting?

Currently one feature the current local tools have that we use to handle the report issues is that everyone directly or indirectly can generate CSV files. These are then collected into SQLite files or databases which can then be used to generate historical information or to compare to previous run times or just as a proof of record that tests were run and so on. CSV imports into most spread sheets and databases. HTML imports decently into most word processors. The sqlite3 tool can generate both of those.

I made a mini-version at M_framework that demonstrates the basic features I am looking for, and demonstrates how the generators and consumers are compartmentalized.

The CVS output probably looks like what you would expect; an HTMLfile shows some of the fun javascript lets you have as another example.

I use M_framework(3f) for several dozen fpm(1) packages as it is a much lighter build than the GPF (General Purpose Fortran) package. It does not include the data-fitting and numeric file comparison tools and so on but works well as a small dependency with fpm and github CD/CI. I made it primarily just for my github fpm packages but anyone interested is free to use it.

That being said, I was wondering if there is a generic tool being used, perhaps or probably in another language like Rust or Julia or Python I am overlooking?

1 Like
2

CMake gives you something like this with CDash. I haven’t looked very deep into it, however the gist is ctest will generate XML reports, that can then be displayed by a CDash server.

You can look at the public projects using CDash here: CDash. You’ll notice that they eat their own dog food; CMake, CDash, ParaView, VTK, and I guess other software from Kitware is all tested with CDash.

1 Like
3

Used Cdash (and Jenkins) for several years via a browser; in all that time I never used it outside of a GUI and did not know it was useable that way. A lot of overhead but that definitely needs a second look. Some issues with putting up a database in some of the environments I need something in but it looks very appealing. Thanks!

4

In the past I’ve used JUnit XML and uploaded the test reports to Jenkins. There are a few tools out there that can take the JUnit XML and convert it to html or parse it natively and display in their UI. Not sure if that is better than ctest and cdash though.