Unit tests, code coverage and the hidden trap…

Quality has always been a “hot” subject in software engineering, and several good development practices used in Agile development, unit testing for example, have been created to improve the quality of the softwares developed.

Programme Managers, Project Managers and Assurance Quality Managers haven’t just been looking for techniques to improve quality, they’ve also been looking for ways to monitor the quality of the projects and the application of these techniques.

This has lead to the development of numerous tools to check the unit testing coverage, the style of the code and its compliance to well accepted rules. Plus, of course, dashboards that gather all that information and present it in a user friendly way.

One of the most commonly used metrics is the code coverage of the unit tests. We often see in Agile teams rules like, “the line coverage should at least 80%”, or “the branch coverage should be at least 70%”. But…herein lies the hidden trap.The code coverage is not enough to verify the quality of the tests. Recently, I saw some unit tests that were exercising the code, but did not have real assertions. The test would only check that the data received was not nil. Something like:

* Example of a useless test.
* As Mashooq Badar explained in a previous entry
* of this blog, tests should have explicit names.
public void
testThatTheCodeRunsWithoutCheckingTheReturnedValue() {
// Prepare Context
// create mock object for instance using Mockito
AnInterfaceToMock mockObject =
// Create instance of class under test
// and inject mock objects
ClassUnderTest anInstance = new ClassUnderTest();
// Run code to test
List result = anInstance.operationUnderTest();
// Assertions

This test is almost useless, as it does not check the list of data it received. We do not know whether or not the ‘operationUnderTest’ performed its job properly. The only thing this test shows is that the ‘operationUnderTest’ did not throw up an exception. A change in the code of the class under test will probably not affect this test. However, the quality dashboard will present a healthy project with a high level of code coverage and instil a false confidence in the managers’ minds. There is no tool that can perform that extra check, for now.

This is one of the reasons why code reviews have to be performed, published and monitored as well.

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>