Difficulties with Test Metrics


Nice article. A few opinions I developed in my short experience:

  • 80% of failures will happen in 20% of the codebase. Actual numbers may be very different (90/10? 99/1?), but it’s yet another example of the 80/20 rule.
  • Code that’s more difficult to test is more likely to fail.
  • That’s also why code coverage is almost meaningless unless it’s 100%, and even then it’s not bullet proof for the reasons you explained in the article.
  • TDD is mostly an academic exercise that goes against getting work done by deadline. It’s nice when you have plenty of time, but otherwise not worth it (please don’t shoot me, this is just an opinion, and I appreciate TDD being useful for others!).
  • The only real test is production.
  • No matter what, most code will fail in production, eventually.
1 Like

I’m glad you liked it. I understand the opinion on TDD. It takes a decent amount of practice to get used to and not have it slow you down, but it my experience it makes deadlines easier to meet because:

  • I force myself to come up with a design that’s more testable (i.e. less likely to fail as you admitted)
  • I don’t spend nearly as much time tracking down bugs in untested code introduced who knows how long ago
    • i.e. if there’s a bug, it’s something I’ve added in the last few minutes
  • I can make large scale refactorings fearlessly, because I know the tests cover every piece of functionality that’s important
  • The tests are an easy place to figure out how code I haven’t worked on in a while is supposed to work, meaning I can get back up to speed and make changes faster than if I didn’t have them

But, all your other points are more or less correct. And of course, TDD doesn’t save you from requirements you didn’t even know you had :wink:

1 Like

My personal experience with testing software in a more systematic way probably began with the book Software Testing by Boris Beizer. He clearly describes that code coverage is only a first level in a whole hierarchy of testing strategies. And when it comes to metrics, the book “Safer C” by Les Hatton introduced me to such measures as cyclometric complexity and others. (Another author whom I can recommend on this topic is Norman Fenton - just google his name together with “software metrics”).

Whatever strategy you follow, the ideal is of course to make sure that your software does what it is supposed to do. And that is where you get into difficulties: actual requirements for the software are indeed almost always discovered as an afterthought. I have seen approaches where listing the requirements is the first thing you need to do and while that makes sense, it is also very difficult or impossible to be sure that you have them all. And when you do have them all, you can be sure that they will change as soon as you stop writing them down.

That said, I agree with the article that blindly staring at code coverage (lines of code or paths or decisions or what not) is not the way to achieve the ultimate goal. It is at best a means to get some confidence in your work. But so is test-driven development or requirements analysis.

Something I found quite remarkable regarding thorough testing is an article I read some time about a seemingly very simple algorithm: determining the norm of a vector. ff you are interested: https://www.researchgate.net/publication/298896236. It is Fortran-related and describes four methods to calculate the norm and how they perform in terms of robustness and accuracy.

Just a warning though: it may be disheartening that even a simple algorithm like determining the norm of a vector should require so much care :). Think of the care we should take to test/check our software for, say, solving the Schrödinger equation on a non-trivial problem or the three-body problem of classical mechanics.

We can only try and try and try … but let us continue to try with this worthwile goal in mind: robust software that does what we expect from it. And of course, let us keep our manager happy with some solid numbers.


My favorite book on testing is “The Art of Unit Testing” by Roy Osherove, which stresses the importance of your tests being understandable.