Simple unit testing in fortran with cmake/ctest

Help
1

Hi everyone,

Can a Fortran program return 0 or 1 after end of execution?

I need it for unit testing with cmake/ctest …

In C/C++ we can do

int main()
{
   cout << "Hello world!" << endl;
   return 0;
}

I saw the different unit testing frameworks recommended by the fortran-wiki, and I think they’re overly complicated, and can be simplified if we’re using cmake/ctest for unit testing our code.

  • cmake/ctest deems a test to be successful if the program returns an error code 0,
  • and it deems the test as failed, if the program returns an error code 1 (also any other non-zero error code).

So, the simplest unit test framework would be to just return 0 or 1 based on whether the code succeeded or not.

Though I don’t know if that is possible in Fortran.

This old article mentions a way, but it is so convoluted, that I don’t want to use it: Unit testing with Fortran and CTest

A better solution that I came up with, is to …

  1. Create a function foo in Fortran that runs all the unit tests, and returns either 0 or 1 based on whether the tests failed or passed.
  2. Call this foo from a C program.
  3. In C, return whatever was returned from foo

As in …

int main()
{
  return foo();
}

This, in theory, should be much more simpler than what’s mentioned in the article, and definitely more simpler than many unit test frameworks recommended by the Fortran wiki.

Is there any other simpler option?

Thanks!

2

I believe this is what you are looking for:

! succeed.f90
stop 0
end

! fail.f90
stop 1
end

# CMakeLists.txt
cmake_minimum_required(VERSION 3.12)
project(test_example LANGUAGES Fortran)

add_executable(succeed succeed.f90)
add_executable(fail fail.f90)

include(CTest)

add_test(NAME succeed COMMAND succeed)

add_test(NAME fail COMMAND fail)
set_property(TEST fail PROPERTY WILL_FAIL TRUE)

~/test_example/build$ ctest
Test project /Users/ivan/test_example/build
    Start 1: succeed
1/2 Test #1: succeed ..........................   Passed    0.21 sec
    Start 2: fail
2/2 Test #2: fail .............................   Passed    0.08 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.29 sec
3 Likes
3

I’m a little curious about the bigger picture here.

If all your tests are being executed from a single Fortran program, what’s the point in using CTest?

I understand they seem complicated, but testing is quite frequently complex. They are complicated because they are trying to accommodate the complexities of testing.

I know it initially seems this simple, but very quickly you run into:

  • What gets reported if something fails?
  • Where do I start looking to try and fix a bug?
  • How do I know what is being tested?
  • Are the tests useful as code examples? (i.e. documentation for typical usage)
  • Can I tell that my tests are passing for the right reasons?

There’s always room for exploration and innovation in this space, but I’m curious if you’ve thought about what the experience of using such a test suite would be like.

4

You might also consider error stop 1. The value can be a character or an integer, literal or variable, and if an integer it should be <256 on unix/posix machines. The difference is that you (maybe) will get a traceback with error stop, depending on the compiler and load options.

5

In fact error stop would already be sufficient. As explained by this StackOverflow answer: fortran - What is the difference between STOP and ERROR STOP? - Stack Overflow,

There is (at least) one aspect where compilers are allowed to vary in detail. If no integer stop code is given, on stop the standard recommends a zero value be passed as the exit code of the process (where such a concept is supported). If no integer stop code is given on error stop, then Fortran 2018 recommends a non-zero value be passed as the exit code.

At least on Mac, both gfortran and ifort satisfy this recommendation.

6

CTest can also use regular expressions to decide if a test passed or failed, so you don’t have to necessarily rely on the 0/non-zero convention:

! succeed.f90
print *, "Success"
end

In the CMake file you need to set the PASS_REGULAR_EXPRESSION property:

add_test(NAME succeed COMMAND succeed)
set_property(TEST succeed 
    PROPERTY PASS_REGULAR_EXPRESSION "Success")

Obviously, with regular expressions you can write all sorts of filters making CTest quite flexible.

CTest is quite powerful if you take a look at Testing With CMake and CTest guide. Unfortunately there is a gap when it comes to having good Fortran examples. It can also output the test results as an XML which can then be transformed into a website.

1 Like
7

Yeah, CTest is plenty powerful, and there are plenty of reasons to want to use it, but my concern was about effectively not using it for any of the reasons it would make sense to.

8

One drawback of CTest, as I understand its usage, is that the tests are all separate programs. That makes it a wee bit cumbersome if you want to have a large number of unit tests. Of course, you can put a bunch of tests in one program, but how do you tell then which of them failed?

It seems more useful for system or integration tests.

9

The solution for this is to create a test driver that links many small tests into a single executable. In fact CMake already has this with the create_test_sourcelist option, but it assumes tests have a standard C interface:

// foo.cxx
int foo(int argc, char* argv[]) {
   /* test goes here */
}

! foo.f90
! return 0 for SUCCESS, or non-zero for FAILURE
function foo(argc, argv) bind(c)
use iso_c_binding
integer(c_int), value :: argc
type(c_ptr), value :: argv
integer(c_int) :: foo
! ... test goes here ...
end function

Afterwards the tests can also be registered individually:

set (TestsToRun foo.f90 <more test files here>)
add_executable(test_driver test_driver.f90 ${TestsToRun})

# Add all the ADD_TEST for each test
foreach (test ${TestsToRun})
  get_filename_component (TName ${test} NAME_WE)
  add_test (NAME ${TName} COMMAND test_driver ${TName})
endforeach ()

To catch Fortran stop codes, the test driver would need to be written in Bash, or a different convention with subprograms needs to be adopted. However packing the tests into the driver could probably be automated using a Python or CMake script.

10

We would put unit tests regarding different features in different programs.

For mathematics module, there would be one program, with many unit test functions.

For physics module, there would be another program, with many unit test functions etc.

i.e tests for a single module will be written in a single file.

CTest would inform if any function in the program failed.

Using C macros, we would print out the file name, function name, and line number where the test failed.

If something failed, the developer has to see the printed out error, and check the location in code that emitted the error.

Nothing more is required in my opinion.

Fancy error reporting is useful, but over-complicates either the unit test framework, or how we use the unit test framework.

I don’t have a Fortran example, here’s an example from C++.

int
test_001_001_001()
{
    Bitset bit;

    bit.SetBit(1);
    bit.SetBit(3);

    CTEST_EQ(bit.IsBitSet(1), true);
    CTEST_EQ(bit.IsBitSet(2), false);
    CTEST_EQ(bit.IsBitSet(3), true);

    return 0;
}

int
main()
{
    CRET(test_001_001_001());
    return 0;
}

The main function will call many unit tests, and the CTEST_EQ will test them. If something fails, the CTEST_EQ macro will throw an error, and print out the code which caused the error, along with the location of the code.

11

My ftnunit package uses batch files and shell scripts to make sure the test program runs through the whole set of tests, even if the program crashes at some point.

12

If you’re getting into unit testing, I highly recommend reading “The Art of Unit Testing”.

My bit of advice. Unit testing provides two major benefits:

  1. In development of the tests it provides a structured framework for specifying clearly and precisely the requirements/expected behavior of the software
  2. In the future the tests serve as documentation and examples of how the code is intended to be used and expected to behave

The automated checking that those requirements/expectations are met, while valuable, is secondary. If your unit testing is concerned only with the automated checking part you won’t get nearly the benefits from point 1 as you could, and you’ll likely lose all of the benefits you could have had from point 2.

Good luck in your unit testing adventure.

13

I haven’t had access to NAG Fortran for some years but when I did I found that

stop 42

would stop the program with the code 52, because it was printing that in octal not decimal. Does NAG or any other compiler still do that?

1 Like
14 
program stopcode
   stop 42
end program

$ gfortran stopcode.f90 && a.out; echo $?
STOP 42
42
$ nagfor stopcode.f90 && a.out; echo $?
NAG Fortran Compiler Release 7.1(Hanzomon) Build 7114
[NAG Fortran Compiler normal termination]
STOP: 42
42

However, changing the value to 256 results in

$ gfortran stopcode.f90 && a.out; echo $?
STOP 256
0

and changing the value to -1 results in

$ nagfor stopcode.f90 && a.out; echo $?
NAG Fortran Compiler Release 7.1(Hanzomon) Build 7114
[NAG Fortran Compiler normal termination]
STOP: -1
255

Only the low-order bits are passed from the fortran program to the OS.

2 Likes