Reducing the cost of unit tests

When writing unit tests and looking at what tests other have written I have noticed a few patterns.

  • Developers might not write a test because the test file didn't exist. Creating a stub, adding it to the build, etc was a mental barrier.
  • Once a unit test file existed for a class the presumed coverage of class was dramatically higher than the reality. More than likely it only contains one test.
    • Developers will rarely go back and create tests for code that already exists.
    • Most unit tests are very similar and there are opportunities for automating the whole process of creating tests.

I ended up writing a tool that parses the code and outputs the stub of a unit tests for a class.

  • A stubbed out test for each non private function in the class.
  • A subclass for the class to expose any protected functions for testing, with stubs
  • A basic sanity test that just calls each function with default arguments.
  • Creates a test case function for each test, populated with default values and return value.

The developer than only has to go through and filling in the data for each test. This tool brought the cost of creating unit tests from annoying to dirt cheap. Because the stubs for all fo the functions where there there was no question of what had tests and what did not. You simple went from top to bottom filling in the tests. The code coverage of classes that have tests from this tool were always higher than those that had been hand rolled previously. Because the tool would automatically insert basic tests for int (-1, 0, 1) and string (null, "", "abc"), as well as just calling every function it was surprising how often it cought bugs after generating the test without even filling in all of the stubs.

Overall this tool reduced the cost of getting unit tests written has resulted in more tests, more consistent tests, higher code coverage, catching bugs sooner, and ultimately higher quality of the code.

How much cheaper could the tests be made? Can it be integrated into the function documentation or into the class itself so the test exists with the code it is testing rather than needing to make a new thing somewhere else? How about in richer tooling that automatically executes functions with smart input and reporting the results to the users editor without having to even create a test file?

Previous
Previous

Code coverage and more using Valgrind's Callgrind

Next
Next

DistCC