Saturday, April 24, 2010

Do Unit Tests and Quality Code Result in Fewer Bugs?

At the end of my team's last release, I set out to prove that unit tests and quality code lead to fewer bugs.

Here are the tools I used.

Jira

The first thing to know is that the team I'm on keeps its features in Jira, and when we commit code to Subversion, we include the Jira ID in the comments.

Below, you'll see how this helped me relate code to features.

The second thing to know is when we find bugs in a feature during its development, in Jira we link those bugs back to the parent feature.

This allowed me to measure how many bugs had been reported for each feature at release.

SvnPlot

There's an open source project called svnplot that allows you to export the information stored in svn to a SQLite database. To do this, follow the instructions in Step 1 of the Quick Start.

You end up with a couple tables that include rows for each revision, including the comment and the files that were added, removed, and changed.

ANTLR

I used ANTLR C# grammar to help map the files changed to classes.

NCover

I used NCover to measure unit test coverage of the added and changed classes.

NDepend

And finally, I used NDepend to measure code quality of those classes, as well as to query the coverage metrics.

Results

This is the part where I'd love to show some fancy graphs proving my point. Unfortunately, the results weren't exactly conclusive.

I ended up dropping the idea of trying to correlate code quality to number of bugs altogether, because I just ended up with a mess.

The results for how unit test coverage related to bugs is sort-of presentable, at least, but they don't show a nice correlation:

Feature LOC Changed / Bug Average Coverage (%)
1 521 65
2 290 38
3 165 0
4 140 54
5 125 21
6 76 25
7 74 77
8 59 30
9 2 80

I'd like to think I'll try again another day, but given those results, I just don't know. I can come up with many reasons why things didn’t come out the way I'd hoped:

  • The amount of change for each feature was very diverse. Some features involved lots of code changes and others involved very few, so that the features may not really be comparable in this way.
  • Some features involved mostly changing UI code, where the coverage can't be high, because we don't currently have a good way to unit test our UI code.
  • Some features, especially Feature 9, had high test coverage, but the unit tests were very low on actual assertions.
  • There's just not enough data.

In the end, if my argument to the business is that unit tests and code quality are important because they reduce the number of bugs, results like these won't help me make my case.

So, if you know of more persuasive results—either your own or stuff out there in the literature—I'd love to hear about it :-)

22 comments:

  1. interesting. it would be also interesting to see what the correlation would be between LOCC/B and, say, cyclomatic complexity. this should be easy given that you use ndepend. i (somewhat naively) mandate correlation between CC and the level of unit testing. if you have CC of say 20 (!) then there are 20 linearly independent paths through code and thus you should have at least 20 unit tests. how CC relates to bugs etc i really don't know...

    ReplyDelete
    Replies
    1. Great Article. Thank you for sharing! Really an awesome post for every one.

      IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      Delete
  2. That exclusively an individual place which holds total consumers lack by free playstation codes were our value.

    ReplyDelete
  3. Woooow.. They're absolutely cute and funny. Thanks for sharing. I think, I can start to color them all, so if you have some more, you have to share it now! While waiting you can find me on https://sexcamhere.com/, watching amazing girls in action. But if you don't have it, you'll be welcome there anyway. So, if you want to enjoy the best sexy view, that's the right place.

    ReplyDelete
  4. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. dpe result teletalk com bd

    ReplyDelete
  5. Great knowledge, do anyone mind merely reference back to it free psn codes

    ReplyDelete
  6. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome! שאלון אישיות

    ReplyDelete

  7. اعالى الخليج تقدم افضل خدمات نقل العفش الدولى المتميزه باسعار متميزة ومنها :

    شركة شحن عفش من الرياض الى الامارات
    نقل عفش من الرياض الى الاردن شركة شحن عفش من الرياض الى الاردن

    ReplyDelete
  8. Speedily this specific web page could without doubt become popular involving most writing a blog along with site-building persons, automobile thoughtful blogposts or possibly testimonials. windows 7 iso

    ReplyDelete
  9. These 'normality cases' must be qualified with the for the most part anticipated outcomes or just not utilized. (Remember, results and execution of an item can be sensibly inferred from a client's tribute just as straightforwardly expressed).how to check jamb result

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. You make so many great points here that I read your article a couple of times. Your views are in accordance with my own for the most part. This is great content for your readers. PSC Result 2019

    ReplyDelete
  12. Blog for share this post its helpful to all PSC Exam Result 2019
    Primary School PSC Exam Result 2019 has been Published. All Board in Bangladesh they can Download her exam result SMS and Web. PSC Result are Published at 10 AM our Prime Minister call a Press brief and announced exam result all over Bangladesh.

    ReplyDelete
  13. I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Interested in internet marketing services company? visit marketing 1on1

    ReplyDelete
  14. I just want to let you know that I just check out your site and I find it very interesting and informative.. top solar power companies in phoenix

    ReplyDelete
  15. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. water damage restoration los angeles

    ReplyDelete
  16. I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. Enchanted Florist

    ReplyDelete
  17. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. embroidered patches

    ReplyDelete