Designing for Debuggability

"Designing for testability" is a way of writing code so that testing is easier. "Designing for debuggibility", in contrast, is a way of writing tests so that development is easier.

Designing for testability means that when you write code, you design it in such a way that it can be tested. For example, if you're developing a client/server system, make it possible to create a stub API so that the client can be tested isolation, without having to setup and configure a server. The advantage of this is that it's easier to write tests against single components of the system, instead of trying to test the whole system all at once. It makes the tests easier to maintain (since the client tests aren't affected by changes in how the server is set up). It also makes it easier to debug the tests when they fail, since you know the problem is in the client, not the server.

Designing for debuggability is the dual of designing for testabilty. The idea is that when you write a test, you should be writing it with the expectation that some day the test will fail. That's what a good test does -- it fails; it highlights regressions or misunderstandings. And since the goal of the test is for it to fail someday, it's in your interest to make sure that when that day comes it's easy to figure out what went wrong. That means you should keep the test very simple and direct -- you don't want to have to debug the test before it's even possible to start debugging the code. It also means making sure that the error message you get when the test fails is very obvious. Don't settle for "AssertionError". Instead, try for "AssertionError: too slow, expected at least 3000 widgets per second, but got only 2200". Try to avoid even things as simple as if-statements and for-loops, because they make the test more complicated.

Some tests are inherently complicated. This is especially true of load tests, which run a long time and use a lot of input. For those tests, designing for debuggability might mean keeping a detailed log of everything that happened in the test. It might mean making sure that the inputs are deterministic instead of randomly generated (or at least it means being able to reuse the same random seed that caused the failure initially). It might mean crafting the test so that a small set of inputs get magnified into a whole ton of load.

For example, at work I'm testing a client-server system. I have some end-to-end tests that start up the server and then run various commands against the client. Now suppose one of those commands ends up killing the server. It would be helpful if the test told me "command X killed the server, and here's the stderr log from the server process". It's less helpful if the only error message I get is that the command right after command X failed with "connection refused". And it's even less helpful if the test didn't capture stderr, so that I have to run all the steps in the test by hand in order to see the error message from the server.