Saturday, April 24, 2010

Do Unit Tests and Quality Code Result in Fewer Bugs?

At the end of my team's last release, I set out to prove that unit tests and quality code lead to fewer bugs.

Here are the tools I used.

Jira

The first thing to know is that the team I'm on keeps its features in Jira, and when we commit code to Subversion, we include the Jira ID in the comments.

Below, you'll see how this helped me relate code to features.

The second thing to know is when we find bugs in a feature during its development, in Jira we link those bugs back to the parent feature.

This allowed me to measure how many bugs had been reported for each feature at release.

SvnPlot

There's an open source project called svnplot that allows you to export the information stored in svn to a SQLite database. To do this, follow the instructions in Step 1 of the Quick Start.

You end up with a couple tables that include rows for each revision, including the comment and the files that were added, removed, and changed.

ANTLR

I used ANTLR C# grammar to help map the files changed to classes.

NCover

I used NCover to measure unit test coverage of the added and changed classes.

NDepend

And finally, I used NDepend to measure code quality of those classes, as well as to query the coverage metrics.

Results

This is the part where I'd love to show some fancy graphs proving my point. Unfortunately, the results weren't exactly conclusive.

I ended up dropping the idea of trying to correlate code quality to number of bugs altogether, because I just ended up with a mess.

The results for how unit test coverage related to bugs is sort-of presentable, at least, but they don't show a nice correlation:

Feature LOC Changed / Bug Average Coverage (%)
1 521 65
2 290 38
3 165 0
4 140 54
5 125 21
6 76 25
7 74 77
8 59 30
9 2 80

I'd like to think I'll try again another day, but given those results, I just don't know. I can come up with many reasons why things didn’t come out the way I'd hoped:

  • The amount of change for each feature was very diverse. Some features involved lots of code changes and others involved very few, so that the features may not really be comparable in this way.
  • Some features involved mostly changing UI code, where the coverage can't be high, because we don't currently have a good way to unit test our UI code.
  • Some features, especially Feature 9, had high test coverage, but the unit tests were very low on actual assertions.
  • There's just not enough data.

In the end, if my argument to the business is that unit tests and code quality are important because they reduce the number of bugs, results like these won't help me make my case.

So, if you know of more persuasive results—either your own or stuff out there in the literature—I'd love to hear about it :-)

31 comments:

  1. interesting. it would be also interesting to see what the correlation would be between LOCC/B and, say, cyclomatic complexity. this should be easy given that you use ndepend. i (somewhat naively) mandate correlation between CC and the level of unit testing. if you have CC of say 20 (!) then there are 20 linearly independent paths through code and thus you should have at least 20 unit tests. how CC relates to bugs etc i really don't know...

    ReplyDelete
    Replies
    1. Great Article. Thank you for sharing! Really an awesome post for every one.

      IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      Delete
  2. That exclusively an individual place which holds total consumers lack by free playstation codes were our value.

    ReplyDelete
  3. Woooow.. They're absolutely cute and funny. Thanks for sharing. I think, I can start to color them all, so if you have some more, you have to share it now! While waiting you can find me on https://sexcamhere.com/, watching amazing girls in action. But if you don't have it, you'll be welcome there anyway. So, if you want to enjoy the best sexy view, that's the right place.

    ReplyDelete
  4. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. dpe result teletalk com bd

    ReplyDelete
  5. Great knowledge, do anyone mind merely reference back to it free psn codes

    ReplyDelete
  6. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome! שאלון אישיות

    ReplyDelete

  7. اعالى الخليج تقدم افضل خدمات نقل العفش الدولى المتميزه باسعار متميزة ومنها :

    شركة شحن عفش من الرياض الى الامارات
    نقل عفش من الرياض الى الاردن شركة شحن عفش من الرياض الى الاردن

    ReplyDelete
  8. Speedily this specific web page could without doubt become popular involving most writing a blog along with site-building persons, automobile thoughtful blogposts or possibly testimonials. windows 7 iso

    ReplyDelete
  9. These 'normality cases' must be qualified with the for the most part anticipated outcomes or just not utilized. (Remember, results and execution of an item can be sensibly inferred from a client's tribute just as straightforwardly expressed).how to check jamb result

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. You make so many great points here that I read your article a couple of times. Your views are in accordance with my own for the most part. This is great content for your readers. PSC Result 2019

    ReplyDelete
  12. Blog for share this post its helpful to all PSC Exam Result 2019
    Primary School PSC Exam Result 2019 has been Published. All Board in Bangladesh they can Download her exam result SMS and Web. PSC Result are Published at 10 AM our Prime Minister call a Press brief and announced exam result all over Bangladesh.

    ReplyDelete
  13. I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Interested in internet marketing services company? visit marketing 1on1

    ReplyDelete
  14. I just want to let you know that I just check out your site and I find it very interesting and informative.. top solar power companies in phoenix

    ReplyDelete
  15. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. water damage restoration los angeles

    ReplyDelete
  16. I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. Enchanted Florist

    ReplyDelete
  17. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. embroidered patches

    ReplyDelete
  18. Fabulous post, you have denoted out some fantastic points, I likewise think this s a very wonderful website. I will visit again for more quality contents and also, recommend this site to all. Thanks. medical cleaning services

    ReplyDelete
  19. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! birre inglesi online

    ReplyDelete
  20. DPE Result 2020 (Directorate of Primary Education) will publish in the last week in April 2020! That’s Attractive and Great news for all DPE Examinee and Their Guardians. The Education Board of Bangladesh and the honorable chairman has announced that they’ll publish the DPE Result 2020 as soon as possible and it will be the last week of coming April. A large Number of DPE Examine and other People don’t know how to check SSC Result 2020 Fast. PSC Scholarship Result 2020 will available on Education Board Official Website, Teletalk Result Website and also on PEC/ Ebtedayee Scholarship Result 2020. Check from this Website now.
    If you are reading this post, don’t worry about checking SSC Exam Result 2020. Here are the complete Information and Guideline about SSC Result 2020 of All Education Board Bangladesh.

    National University Admission Authority has been published NU Degree 3rd Year Result 2020 Pdf Download (Session 2015-16) on the official website today. Now you can check it from here.

    ReplyDelete
  21. SSC Routine 2020 (Secondary School Certificate Exam Routine 2020) has been publish on the last month 2019! That’s Attractive and Great news for all SSC Examinee and Their Guardians. The Education Board of Bangladesh and all Education Board chairman has announced the SSC New Routine 2020 on last month. A large Number of SSC Examine and other People don’t know how to check SSC Routine 2020 Fast. The Education Board authority also published HSC Routine 2020 on their official website. Now you can see the HSC New Routine 2020 quickly.
    There are another good news for the National University Students of Bangladesh that the NU Degree 2nd Year Routine 2020 PDF Download has been published on the official website today.

    ReplyDelete
  22. DSS & DNC Job circular has been published by the Bangladesh government. The New Exam date of DSS Job 2020 has been published by the authority on their official website today. Now you can check it from here. DNC Authority also published a new job circular on their official website today. Now you can check DNC Job Circular 2020 from here.

    You can also visit another option if you need it. In Bangladesh there is an exam is going to start from 1 February 2020. That named SSC Exam. We have made a suggestion for the exam you can check this SSC Suggestion 2020 from here.

    BB Combined 7 Bank 771 Posts Job Circular 2020 PDF Download

    BREB Job Circular & Online Application 2020 PDF Download

    ReplyDelete
  23. Please continue this great work and I look forward to more of your awesome blog posts. hardwood floor water damage repair warping

    ReplyDelete
  24. What a fabulous post this has been. Very useful post. I am grateful to you and expect more number of posts like these. Thank you very much. Job Circular

    ReplyDelete
  25. The online ordering of any product, making a stock trade, or buying an airline ticket, for example, best programming keyboard are some work that programming needs to bind internally for the end users.

    ReplyDelete