28 December 2010

What makes code testing worthwile?
On the “automated” in “unit testing”.

In the software engineering community, a lot has already been written and said about the benefits of writing tests along with the “normal” development of software, yet the matter is still widely misunderstood. Most people who hear “testing”, think “unit testing” and what they really mean is “automated unit testing”, where the “automated” part is not any less important then the “unit” part.

As a result of this many programmers, even through they hear about the benefits of testing at every software conference, still look at testing as some kind of chore, where in fact automated tests are so much of a productivity helper that, if they are done correctly, doing work without them will seem much more mundane! It is important to understand that the big benefits of “testing” are in fact the benefits of having as many kinds of tests well automated as it is efficient with relation to the time it takes to automate them.

TDD, BDD and various other specific techniques that are fashionable at the moment, while there is much time devoted to talking about them, are of little use until some fundamental issues aren’t more well understood by every institution, company, manager and programmer willing to improve programming productivity and reduce the number of bugs in their software.

Especially, every management person in the software buisness should understand how the software development process is realised by the programmer in his or her daily work. Traditionally, a developer will introduce changes or new features in a series of small changes in the code, with each change being followed by manually testing if it had the desired effect. If the change affects the software in the middle of some long and complicated interaction with the user, the process of doing the test after each small change might require a series of steps that takes a long time to execute. Often, a simple syntax mistake like missing a semicolon or a comma forces the programmer to repeat this procedure.

So, if an effective process of testing the code will not be explicitly set up, testing will be done anyway, yet in the least effective way possible. Thus, estabilishing a good, automated software testing process will provide large time savings by shortening the change-test cycle. To convey this point, programmers and technical managers should tell their CEOs about “automatic tests” and not about “unit tests”, TDD or whatnot.

It is also interesting how the tools and techniques that are currently widely available handle the automation aspect of testing. Testing frameworks should:

a) allow to catch as many bugs as possible

b) make test specification as fast as possible

c) make test execution as fast as possible

While there has been some superficial improvement in b) with the advent of BDD, it seems right now one has to choose between a) and c) exclusively. For example for database backed applications test suites either have to use mocks and avoid the database which makes them quite a bit less likely to catch certain types of bugs or otherwise the test suite gets dog slow as it grows. There is surely a lot of room for improvement here. I wonder what would an ideal solution targeting a), b) and c) look like?

Comment