Debugging with Code Coverage

Lately I've been spending a lot of time debugging regression tests. (That is, I started with a codebase and some tests that passed, but after I changed the codebase, the tests don't pass anymore). Figuring out why the tests are failing can be pretty tedious, especially if I didn't write the test in the first place and therefore have no idea what it's actually trying to test.

So here's an item I'm putting on my Christmas wish list: a regression test framework that takes advantage of code coverage metrics to help you localize the error. The idea is that you would run the unit test on the working codebase to get a baseline code coverage pattern. Then you'd run it again on the broken codebase, and do a diff on the code coverage pattern. I bet that in a large percentage of failures, this would quickly highlight what area the problem is in.

I'm only thinking of using a single bit per line of code, indicating whether that line was executed or not. A more elaborate system might do something with execution counts (how often the line was executed), and/or execution order. But I think that a simple bit-per-line approach would get you 90% of the utility, while remaining very simple and understandable.

The target system for something like this would be a large codebase (100KLOC or more), where several people are making changes. So you can't fit the entire system in your head, and the bug that you triggered might well be manifesting in code that you've never seen before.

I've been mulling over this idea for a couple weeks now, and I wanted to share it and see what people think. Comments? Has it been done already?

Followups to Debugging with Code Coverage:

• New Job

Posted on December 19, 2003 06:08 PM
More programming articles