Debugging with Code Coverage

Lately I've been spending a lot of time debugging regression tests. (That is, I started with a codebase and some tests that passed, but after I changed the codebase, the tests don't pass anymore). Figuring out why the tests are failing can be pretty tedious, especially if I didn't write the test in the first place and therefore have no idea what it's actually trying to test.

So here's an item I'm putting on my Christmas wish list: a regression test framework that takes advantage of code coverage metrics to help you localize the error. The idea is that you would run the unit test on the working codebase to get a baseline code coverage pattern. Then you'd run it again on the broken codebase, and do a diff on the code coverage pattern. I bet that in a large percentage of failures, this would quickly highlight what area the problem is in.

I'm only thinking of using a single bit per line of code, indicating whether that line was executed or not. A more elaborate system might do something with execution counts (how often the line was executed), and/or execution order. But I think that a simple bit-per-line approach would get you 90% of the utility, while remaining very simple and understandable.

The target system for something like this would be a large codebase (100KLOC or more), where several people are making changes. So you can't fit the entire system in your head, and the bug that you triggered might well be manifesting in code that you've never seen before.

I've been mulling over this idea for a couple weeks now, and I wanted to share it and see what people think. Comments? Has it been done already?

Followups to Debugging with Code Coverage:

New Job

Posted on December 19, 2003 06:08 PM
More programming articles

Comments

Will this link help? It takes a different tack on which tests to prioritize etc.
http://portal.acm.org/citation.cfm?id=566172.566187&dl=portal&dl=ACM&type=series&idx=SERIES392&part=Proceedings&WantType=Proceedings&title=International%20Symposium%20on%20Software%20Testing%20and%20Analysis
Effectively prioritizing tests in development environment
ISSN:0163-5948 `

Posted by: Govind at December 20, 2003 12:45 PM

UGG Bailey Button bootsis a new style in 2009.The classic cardy uggs boots is another hot boots that worth of buying.And the classic tall ugg boots will make your winter amusing.And now uggs on sale,if you are looking for such a boot,the ugg boots is good choice this year.

Posted by: uggs on sale at November 17, 2009 08:30 PM

Uggs on sale now.Classic cardy boots ugg is a special boot that makes you different from the other girls.And the Bailey Button UGGs is hot in 2009.If you want to get a warm shoe in cold winter,I think ugg classic tall boots is a good choice for you.

Posted by: ugg classic cardy boots at November 17, 2009 09:01 PM

rolex replica
A.Lange & Sohne replica
Alain Silberstein replica

Posted by: rolex replica at November 19, 2009 01:03 AM

We are the best online sales for the china wholesale . Here you can have a large of choices of kinds Ugg Boots,Converse Shoes,Timberland Boots,puma shoes,Nike Shox Shoes ,Nike Dunk SB Shoes,Nike Air Max，Links Of London,Tiffany Jewelry,Dior Handbags?,jimmy choo handbags ,Cartier Watches, 8GB Mp4 Players,Bluetooth Car DVDs. All our cheap online cheap goods are high quality and original packages, and best service. We offer our customers the best service, 7 days arrive at your door.Enjoy your easy and happy shopping with us.

Posted by: cheap goods sale. at November 20, 2009 01:52 AM

A.Lange & Sohne watch for sale
Audemars Piguet watch for sale
Bell & Ross watch for sale
Breguet watch for sale Rolex Datejust II watch for sale
Rolex Datejust watch for sale
Rolex Submariner watch for sale
Yachtmaster watch for sale/a>
Rolex Day-Date II watch for sale

Posted by: Panerai watch for sale at November 20, 2009 02:42 AM

discount nike air yeezy

wholesale online"> china wholesale online

discount ugg

boots
nike free shoes
furture
ugg boots
Discount store

Posted by: nike-airyeezy at November 20, 2009 02:56 AM

Supra Shoes
Supra Skytop Shoes
Supra Society
Supra Thunder
Supra Suprano Hi
Supra Strapped
Supra Trinity
Supra Skylow 1.5
Supra Dixon
Diablo 1.5
Supra Cruizer
Supra Vaider Shoes
Supra Cuban

Posted by: Supra Shoes at November 20, 2009 04:48 AM

Burberry watches
Bvlgari watches
Cartier watches
chopard watches
Concord watches
croum watches
DeWitt watches
ebel watches
Ferrari watches

Posted by: Bell & Ross watches at November 20, 2009 08:09 PM

Ferrari watch for sale
Franck Muller watch for sale
Glashutte watch for sale
Graham watch for sale
Hermes watch for sale
Hublot watch for sale
IWC watch for sale
Jaquet droz watch for sale
Lady Watch for sale
Longines watch for sale

Posted by: Ferrari watch for sale at November 20, 2009 08:18 PM

fake Tiffany Earrings
fake Tiffany Necklaces
fake Tiffany sets
fake Tiffany Accessories
Tiffany Necklaces replica
Tiffany sets replica
Tiffany Accessories replica

Posted by: replica Tiffany bangles at November 20, 2009 08:36 PM

replica chopard
replica Concord
replica croum
replica DeWitt
replica ebel
replica Ferrari

Posted by: Lady Watch watches at November 20, 2009 08:37 PM

croum replica
DeWitt replica
ebel replica
Ferrari replica
Franck Muller replica
Glashutte replica

Posted by: A.Lange & Sohne replica at November 21, 2009 03:36 AM

Hermes watches replica
Hublot watches replica
IWC watches replica
Jaquet droz watches replica
Lady watches replica
Longines watches replica
Louis Vuitton watches replica

Posted by: Rolex GMT watches at November 22, 2009 12:05 PM

cheap Christian Louboutin sandal
cheap Jimmy Choo Handbags
cheap Jimmy Choo Shoes
cheap Yves Saint Laurent Shoes

discount Christian Louboutin Boot
discount Christian Louboutin Pump
discount Christian Louboutin sandal
discount Jimmy Choo Handbags

Posted by: discount Jimmy Choo Handbags at November 23, 2009 01:42 PM

discount Alexander McQueen
discount Chloe

discount Christian Louboutin Evening
discount Christian Louboutin Pumps
discount Christian Louboutin Sandals
discount Christian Louboutin Wedges
discount Jimmy Choo
discount Manolo Blahnik

Posted by: discount Manolo Blahnik at November 23, 2009 01:54 PM

cheap Christian Louboutin
cheap Jimmy Choo
cheap Yves Saint Laurent
discount Christian Louboutin
discount Jimmy Choo
discount Yves Saint Laurent

Posted by: Christian Louboutin Boot on sale at November 23, 2009 02:46 PM

People all over the world know the abercrombie and fitch,but not everyone really knows how fashion the abercrombie is,hollister is the Legend maker. Everybody wears the hollister clothing would be the abercrombie mens and the abercrombie womens, if you want know you can search the Ruehl No.925 or abercrombie outlet in the www.google.com .

Posted by: fitch at November 24, 2009 09:26 PM

Laptop Battery Laptop Battery Laptop Batteries
Laptop Batteries discount laptop battery
discount laptop battery
notebook battery notebook battery
computer battery computer battery
replacement laptop battery replacement laptop battery
notebook batteries notebook batteries

Posted by: Laptop Battery at November 24, 2009 10:36 PM

rongTiffany Jewelry Online - Discount Tiffany & Co Jewelry On Sale

Hot Sell Tiffany Jewelry - Tiffany Jewelry Online

Return To Tiffany Double Heart Detachable Pendant Necklace - Tiffany Jewelry Online

Tiffany&Co 1837 Titanium I.D. Bracelet - Tiffany Jewelry Online
Tiffany & Co Bead Bracelet - Tiffany Jewelry Online

Tiffany&Co Atlas Round Pendant Necklace - Tiffany Jewelry Online
Tiffany&Co Bar Pendant Necklace - Tiffany Jewelry Online
Tiffany&Co Bead Necklace(N242) - Tiffany Jewelry Online
Tiffany&Co Bean Pendant Necklace(N157) - Tiffany Jewelry Online

Tiffany&Co Elsa Peretti Apple Pendant Necklace - Tiffany Jewelry Online
Tiffany&Co Butterfly Pendant Necklace - Tiffany Jewelry Online

Posted by: at November 25, 2009 02:10 AM

China Electronics Wholesale China

Phone Cheap Cell Phones

href="http://www.thechinaphone.com/index.php?main_page=advanced_search_result&keyword=China%20Wholesale">China

Wholesale

replica handbags
Louis Vuitton Replica
Louis Vuitton handbags
fake Louis Vuitton bags
fake Louis Vuitton
replica rolex
replica rolex
fake rolex
buy cheap fake rolex
rolex Submariner

Posted by: lovebo at November 29, 2009 10:19 AM

Just wanted to say great job with the blog, today is my first visit here and I’ve enjoyed reading your posts so far
ugg bailey button
Wow, my ugg classic mini will not be coming off now! I’ve had them on for 12hrs strait and I do not want to take them off. Thanks for everything, well worth the wait.

Posted by: ugg bailey button boots at November 30, 2009 06:28 AM

replica watches
Rolex Day Date watch for sale
replica movado
Cartier watches
Rolex Masterpiece watches
Breitling SWISS Replica

Posted by: Rolex Day Date watch for sale at November 30, 2009 04:54 PM

Posted by: Jimmy Choo Handbags at December 1, 2009 04:54 AM

nike air max 2009
discount air max 2009
nike mens low dunk
nike air max 360
cheap adidas shoes
Adidas 35TH Anniversary
Women's Adidas 35th Anniversary
adidas adicolor shoes
dicount UGG Lo Pro Button
ugg new arrivals
cheap ugg nightfall boots
ugg sundance boots
ugg ultra tall boots sale
ugg mayfaire boots on sale

Posted by: sindee at December 1, 2009 11:18 AM

discount Christian Louboutin
cheap Christian Louboutin Shoes
discount Christian Louboutin Wedges
Christian Louboutin Wedges on sale
Christian Louboutin Boots on sale
Tiffany&Co Round Tag Necklace

Posted by: replica Audemars Piguet at December 1, 2009 11:39 PM