Saturday, April 24, 2010

Do Unit Tests and Quality Code Result in Fewer Bugs?

At the end of my team's last release, I set out to prove that unit tests and quality code lead to fewer bugs.

Here are the tools I used.

Jira

The first thing to know is that the team I'm on keeps its features in Jira, and when we commit code to Subversion, we include the Jira ID in the comments.

Below, you'll see how this helped me relate code to features.

The second thing to know is when we find bugs in a feature during its development, in Jira we link those bugs back to the parent feature.

This allowed me to measure how many bugs had been reported for each feature at release.

SvnPlot

There's an open source project called svnplot that allows you to export the information stored in svn to a SQLite database. To do this, follow the instructions in Step 1 of the Quick Start.

You end up with a couple tables that include rows for each revision, including the comment and the files that were added, removed, and changed.

ANTLR

I used ANTLR C# grammar to help map the files changed to classes.

NCover

I used NCover to measure unit test coverage of the added and changed classes.

NDepend

And finally, I used NDepend to measure code quality of those classes, as well as to query the coverage metrics.

Results

This is the part where I'd love to show some fancy graphs proving my point. Unfortunately, the results weren't exactly conclusive.

I ended up dropping the idea of trying to correlate code quality to number of bugs altogether, because I just ended up with a mess.

The results for how unit test coverage related to bugs is sort-of presentable, at least, but they don't show a nice correlation:

Feature LOC Changed / Bug Average Coverage (%)
1 521 65
2 290 38
3 165 0
4 140 54
5 125 21
6 76 25
7 74 77
8 59 30
9 2 80

I'd like to think I'll try again another day, but given those results, I just don't know. I can come up with many reasons why things didn’t come out the way I'd hoped:

  • The amount of change for each feature was very diverse. Some features involved lots of code changes and others involved very few, so that the features may not really be comparable in this way.
  • Some features involved mostly changing UI code, where the coverage can't be high, because we don't currently have a good way to unit test our UI code.
  • Some features, especially Feature 9, had high test coverage, but the unit tests were very low on actual assertions.
  • There's just not enough data.

In the end, if my argument to the business is that unit tests and code quality are important because they reduce the number of bugs, results like these won't help me make my case.

So, if you know of more persuasive results—either your own or stuff out there in the literature—I'd love to hear about it :-)

14 comments:

  1. interesting. it would be also interesting to see what the correlation would be between LOCC/B and, say, cyclomatic complexity. this should be easy given that you use ndepend. i (somewhat naively) mandate correlation between CC and the level of unit testing. if you have CC of say 20 (!) then there are 20 linearly independent paths through code and thus you should have at least 20 unit tests. how CC relates to bugs etc i really don't know...

    ReplyDelete
    Replies
    1. Great Article. Thank you for sharing! Really an awesome post for every one.

      IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      Delete
  2. That exclusively an individual place which holds total consumers lack by free playstation codes were our value.

    ReplyDelete
  3. Woooow.. They're absolutely cute and funny. Thanks for sharing. I think, I can start to color them all, so if you have some more, you have to share it now! While waiting you can find me on https://sexcamhere.com/, watching amazing girls in action. But if you don't have it, you'll be welcome there anyway. So, if you want to enjoy the best sexy view, that's the right place.

    ReplyDelete
  4. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. dpe result teletalk com bd

    ReplyDelete
  5. Great knowledge, do anyone mind merely reference back to it free psn codes

    ReplyDelete
  6. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome! שאלון אישיות

    ReplyDelete

  7. اعالى الخليج تقدم افضل خدمات نقل العفش الدولى المتميزه باسعار متميزة ومنها :

    شركة شحن عفش من الرياض الى الامارات
    نقل عفش من الرياض الى الاردن شركة شحن عفش من الرياض الى الاردن

    ReplyDelete
  8. Speedily this specific web page could without doubt become popular involving most writing a blog along with site-building persons, automobile thoughtful blogposts or possibly testimonials. windows 7 iso

    ReplyDelete
  9. These 'normality cases' must be qualified with the for the most part anticipated outcomes or just not utilized. (Remember, results and execution of an item can be sensibly inferred from a client's tribute just as straightforwardly expressed).how to check jamb result

    ReplyDelete