Unit testing has become incredibly popular during the past years. As a consequence I sometimes feel like we’ve lost the focus on why we write unit test and what we expect as a return on investment.
The success of unit testing probably comes from the simplicity of the appraoch. However, we’ve learned since then that unit testing is not so easy neither. A high percentage of code coverage does not necessary mean quality software, doeasn’t mean that border cases have been covered, and can even impede software maintenance if the test suite is poorly organized.
Don’t get me wrong, I do value the benefit of unit testing. But unit testing for the sake of unit testing has no value to me. If you think an other form of automated testing will be a most rewarding strategy, then do it. If you go for unit tests, the important questions to ask are:
- Did the unit tests revealed bug in the production code?
- Did you organise the unit test in a meaningful way?
- Did you spend time to identify and test border cases?
A clear “yes” to these three questions indicates an intelligent and probably rewarding testing effort. A pertinent test suite should reveal the bugs that you introduce in the system — and don’t pretend you don’t introduce any. A well-organized test suite should give you a high-level view of the features in the module. A well-crafted test suite should cover the nasty use cases of the system before they hurt you when the system is in production.
There is an abundant literature about unit testing, but nothing that I read seemed to cover the reality of the unit tests that I write. I therefore analysed the nature of my own unit tests and came with a personal categorization which differs from existing one. Here is the 4 categories I’ve identified as well as a short description of the pro/cons of each kind.
Basic unit tests
A basic unit test is a test suite which covers one unique class and test each method in a more-or-less individual fashion. This is the core idea of unit test where each tiny functionality is tested in full isolation, possible with the help of mock objects to break the dependencies. As a result, the test should be repeatable and also independent (of the environment and of other modules).
Problem with real unit tests are:
- Testing micro-functionality is perceived as boring, exhaustive or laborious, especially if mock objects are over-used
- Tests are tightly coupled with the implementation details and need frequent refactoring
- Fixture overlap and dependency between tests leads to code duplication
- Relevance of such test can be discussed as the coupling between low-level and high-level fault is only partially understood.
Unit test with indirect assertions
For basic unit tests, the subject under test (SUT) is the very one for which we assert the behavior. We perform an action on the SUT and ensures it behaves correctly. This is however sometimes not possible as soon as the system become a bit more complicated. As a consequence, the actions are performed on the SUT, but we rely on a level of indirection for the assertions; we then assert the behavior of another object than the SUT. This is for instance the case when we mock the database and want to ensure that the rows are correctly altered.
- Coupling with the implementation is still high
- Fake objects migth be used instead of Mocks — they contain state and aren’t purely hard-code anymore
- Behaviour of the SUT is harder to understand due to the level of indirection
Inflection point unit test
Inflection points — a term coined by Michael Feather if I’m right — are somehow the entry points to a given software module. For utility libraries or services, the inflection points correspond to the public API. Testing these specific points is the most rewarding strategy to me, and other people think the same.
- Inflection points are less subject to changes, and are closer to a black-box form of testing
- Tests become the first client of the interface and give you a change to check if the API is practical
- After having covered all the common use cases of the inflection point, the test coverage of the underlying classes should be close to 100%. If not, this indicates a potential design weakness or useless code.
Dynamic unit tests
I qualify tests as “dynamic” when their execution change from one run to the other. Primary goal of such test is to simulate the dynamicity and variability of the productive system. Such test are however quite far away from the concept of basic unit tests and could be considered to some extend as integration tests; they are however still repeatable and independent of other modules. Execution in the real system may indeed be aftected by either
- Threading issues
- Contextual data, e.g. cache
- Nature of the input
Most of these tests rely on randomization, for instance to generate input data or disrupt the scheduling of threads. Though it’s more complicated, randomization and fuzzing have been proved as effective techniques to detect issues which would never arise with fixed execution condition. Think for instance about phase of the moon bugs and date problems.
://ewernli 