I wanted to figure out which unit test framework I should consider for a new Fortran project. There are a ton listed in the Fortran wiki. I tried to weed out the ones that looked immature or unmaintained, and I still got a list of SIX options. So… what do you guys use? What do you recommend?
As author of the ftnunit package I can tell you that it is not unmaintained 
It is rather mature, as far as I am concerned, so requires little or no changes, though ideas are welcome. I recently added support for its use in the context of CMake/CTest, but have not found the opportunity yet to publish that (and I should probalby move the stuff to Github, as SF has become invisible).
As for your poll, I am not sure what I would recommend. I know mine best, but it does not mean it IS the best.
5 Likes
I voted for test-drive, but that’s a little unfair on the others as I’ve only really used test-drive in angst. It worked really well and had the same kind of “well that was surprisingly easy” feel as pytest (if you’re familiar with that framework for Python). Especially coupled with fpm: fpm test.
1 Like
There are more frameworks in my list:
f90tw: provides Fortran wrappers for a limited subset of the Boost and Google test frameworks functionality, by loudove. At the same time, it offers a rather simple mechanism for setting up and managing test suites in Fortran projects.
fort_test: very lightweight testing framework that supports basic assertions and running of test sets, by Thomas Marks
fortran-testanything: test library supporting the Test Anything Protocol (TAP) inspired by Perl’s Test::More module, by dennisdjensen
fortran_test_helper: library to provide assistance to testing, by Jason Christopherson
fortran-unit-test: unit test library in Fortran to encourage scientific programmer to write tests, by dongli
Fortran-Unit-Test: shows how to link Fortran code with C/C++ and a testing framework such as Google Test, by Rich Morrison
Fortran_UnitTest: unit test library based on OOP, by zhenkunl. It is strongly inspired by Zofu, and its output format is derived from fortran-unit-test.
Fortran Unit Test Library: pure Fortran library using Object-Oriented Programming (OOP), by zhenkunl. It is strongly inspired by Zofu, and its output format is derived from fortran-unit-test.
Fortran Unit Test Framework (FortUTF): unit test framework written purely in Fortran to be compatible with as many projects as possible, by Kristian Zarębski
Fortran Unit Testing Objects (Fortuno): flexible and extensible Fortran unit testing framework for testing serial, MPI-parallelized and coarray-parallelized applications, by Bálint Aradi
ftest: minimalistic unit testing, by Ladislav Méri
FUnit: lightweight library for writing, administering, and running unit tests in Fortran, by Andrey Pudov. It is an instance of the xUnit architecture and provides Fortran programmers a basic testing functionality with a flexible variety of user interfaces.
FyTest — Instant Fortran unit testing: lightweight unit testing framework for Fortran, by Bálint Aradi. Thanks to its header-only design, it can be easily bundled with any Fortran project without creating extra dependencies.
M_framework: aggregate of Fortran modules useful for creating terminal messages, comparing expected values to results, writing logfiles and playback journals and performing unit tests for Fortran, by urbanjost
par-funnel: unit test parameterizer using namelist, by Tomohiro Degawa. Par-funnel is not a unit test framework but is intended to be used with other unit test frameworks.
pFUnit: unit testing framework enabling JUnit-like testing of serial and MPI-parallel software written in Fortran, from Goddard-Fortran-Ecosystem. Limited support for OpenMP is also provided in the form of managing exceptions in a thread-safe manner.
tap: minimal producer implementation of the “Test Anything Protocol” (TAP) in Fortran 90, from gzahl
test-drive: lightweight, procedural unit testing framework based on nothing but standard Fortran, by Sebastian Ehlert and Jeremie Vandenplas. Integration with meson, cmake and Fortran package manager (fpm) is available.
Testing Or ASsertion Toolkit (TOAST): unit testing library by thomasms and Bob Apthorpe
Veggies: unit testing framework written using functional programming principles, with the ability to test parallel code, by Brad Richardson et al. As many of its procedures as possible are marked with the pure keyword, while still allowing the framework to test impure code.
XFunit: object oriented framework to implement unit testing in Fortran 2018, by Fran Martinez Fadrique. Unit testing with XFunit is structured in suites and units. A test suite typically define all test to be implemented for a Fortran module.
2 Likes
test-drive I recommend because it’s a Fortran-lang maintained project.
I haven’t used it in my personal projects, though.
Consider also the Fortran testing unframework, or:
- A test suite is a Fortran program.
- Each test is an
iftest of alogicalexpression that reports tostderron a failure and is silent otherwise. - At the end of the test suite report to
stdoutif one or more tests failed or all succeeded. - On one or more failures, stop the program with a non-zero exit code.
Example: neural-fortran/test/test_dense_layer.f90 at main · modern-fortran/neural-fortran · GitHub
3 Likes
I haven’t used it extensively, but my impression was that test-drive was simpler and less featureful (so probably easier to learn), but similar in design to veggies. I’m a big fan of the extra features in veggies, plus I wrote it, so that’s what I use in my projects, but it’s perfectly reasonable to start with what is easier to learn.
2 Likes
Yes, all my tests use this approach — just a bunch of programs with error stop in them. Either fpm test or ctest works with this approach. I found it sufficient for my purposes.
But if you want more regular output and more “order”, you can use any of the test frameworks above.
2 Likes
Oh! I got a couple of questions for you: What is the difference between Vegetables, Veggies, and Garden? Why was Vegetables replaced? Why do people write “Veggies / Garden” as if they are equivalents? Are they? If so, why? At least the README files look like clones of each other.
What are some of the extra features in Veggies / Garden that you’d like to highlight? I have never used unit testing in any formal way.
Vegetables was the original name of the project. I then split it into Veggies and Garden. Veggies doesn’t require support for the parallel features of the language (i.e. coarrays), where Garden enables testing of parallel code. In all other respects they are identical.
- Nicely formatted output with color, and an option to report what was checked even in passing tests
- A nesting/hierarchical structure to the test suite
- Having parameterized tests (i.e. the test accepts an input) which facilitates
- providing specific examples on which to execute the test
- randomized testing (which includes finding “simplest” failing examples)
- Ability to select a subset of tests to execute
- A companion tool, cart, that can automatically generate the driver program
Those things do add some complexity to the framework, although not much that users of the framework really have to contend with if they don’t use them. Like I said, if you don’t need or want those features there are simpler options, but I find a lot of value in having them.
7 Likes
You had me sold at…
I do like a colourful terminal! (The other stuff sounds awesome too, I must give Veggies/Garden a go.)
@everythingfunctional Thanks! Veggies & Garden look super interesting.
What is a driver program?
Driver is one of those fuzzy terms, that can be used for many purposes. The definition on Wikipedia says:
A driver in software provides a programming interface to control and manage specific lower-level interfaces that are often linked to a specific type of hardware, or other low-level service.
In the context of the current discussion, I think Brad is referring to a test driver, i.e. a program which invokes the tests (the “low-level service” in this case). The test driver could perhaps expose command-line arguments to select tests by name, or a subset of tests, change the verbosity of the test output, and other things.
For example when you use the built-in testing facilities in CMake, it provides ctest as the test driver program.
@everythingfunctional, I see that you’re using ANSI escape sequences for color, as do I.
I’ve been wondering, is there some more modern way to make colors? For example, a command like git diff will show red and green color in the terminal, and yet git diff > file.txt does not have any escape sequences in the text file. Does git somehow detect whether its output is being redirected or not, and then enable or disable color?
There may be, but ANSI escape sequences seemed to work well enough for my purposes so I went with it.
I believe so. I can think of a handful of programs that seem to do that.
I’d welcome a PR to add any improvements to the colorization.
1 Like
Slightly OT, but I was curious, too, and asked a ChatGPT. (The C-code works!)
Summary
This behavior is usually controlled by checking the output file descriptor with system calls like isatty() in Unix-like systems. If isatty() returns true, it means the output is going to a terminal, and the program can safely use escape sequences to format the output with color. If it returns false, indicating that the output is being redirected, the program usually outputs plain text without any escape sequences.
#include <stdio.h>
#include <unistd.h>
int main() {
if (isatty(STDOUT_FILENO)) {
printf("Standard output is a terminal.\n");
// You can include terminal-specific code here, like color output
} else {
printf("Standard output is not a terminal.\n");
// Output for redirection or piping, no special terminal codes
}
return 0;
}
To create a similar functionality in Fortran, you can use the isatty function, but keep in mind that Fortran doesn’t have a built-in isatty function like C does. Instead, you can use the ISO C Binding feature of Fortran to call the C isatty function. Here is a minimal example:
program isatty_example
use, intrinsic :: iso_c_binding, only : c_int
implicit none
interface
function isatty(fd) bind(C, name="isatty")
import c_int
integer(c_int), value :: fd
integer(c_int) :: isatty
end function isatty
end interface
integer(c_int) :: result
! STDOUT_FILENO is typically 1 in Unix-like systems
result = isatty(1)
if (result /= 0) then
print *, 'Standard output is a terminal.'
else
print *, 'Standard output is not a terminal.'
end if
end program isatty_example
2 Likes
I think you forgot to copy the minimal example. I’m curious to see it.
2 Likes
I found that gfortran has an isatty function that you can use directly without the C binding. So I guess your options are either a gfortran-specific GNU-extension method, or a unix-specific C binding method. Not sure if anything is more portable than that.
Here’s a minimal example using gfortran:
program color_demo
use iso_fortran_env, only: output_unit
implicit none
character, parameter :: esc = char(27)
! ANSI escape color codes: https://stackoverflow.com/a/54062826/4347028
character(len = *), parameter :: &
fg_bright_red = esc//"[91m", &
fg_bright_green = esc//"[92m", &
color_reset = esc//"[0m"
character(len = :), allocatable :: bright_red, bright_green, reset
! Is stdout a terminal? Otherwise, it's a file
if (isatty(output_unit)) then
! Use escape codes for color
bright_red = fg_bright_red
bright_green = fg_bright_green
reset = color_reset
else
! Don't attempt any coloring
bright_red = ""
bright_green = ""
reset = ""
end if
print *, bright_red//"here's red text"//reset &
//" and then normal text"
print *, bright_green//"here's green text"//reset &
//" and then normal text"
end program color_demo
Output:
Be careful to always concatenate reset after your text. If you’ve ever had a program permanently mess up your terminal color, it’s because they forgot to reset.
@everythingfunctional I might send a PR later this month, but I’m swamped with Advent of Code for now 
3 Likes
The isatty() function is an operating system call, not a C library call. It is a POSIX function, so the use should be portable to any posix-compliant OS. Of course, the API is defined in C, as all POSIX functions are defined. The two choices are to try to call the OS function directly from fortran, or to write a wrapper in C and then call that from fortran. In this case, there are some error codes that are set, and those are easiest to reference from C, so maybe the wrapper approach is indicated. Also, the argument to the function is a file descriptor, so there is the issue of associating a fortran unit with a C file descriptor.
edit: I also noticed the previous discussion topic. Should INQUIRE have an ISATTY parameter?
1 Like
That is commonly done on GNU/Unix systems by calling isatty(3c). ISATTY() is a very common Fortran extension available in multiple compilers; or on “POSIX systems” call the C routine via an ISO_C_Binding interface. The strongest reason for using the extension is it is usually implemented on all supported platforms like MSWindows. The POSIX routines are only available on MSWIndows via things like CygWin or the Linux subsystem … .
In some cases you can guess whether the output is a tty or not by calling INQUIRE; but I have found the results to vary wildly. Sometimes the query will return “stdout” as a filename, or a null name or a name like “/dev/ppty” when the file is assigned to stdout but varies so much from compiler to compiler that using INQUIRE is not portable without conditionals for each compiler.
It is such a common extension it would be nice if ISATTY was part of the standard (along with many “POSIX” routines practically every compiler supports with a proprietary interface) but I would not wait for that unless you are exceptionally patient.
1 Like