Saturday, April 24, 2010

Do Unit Tests and Quality Code Result in Fewer Bugs?

At the end of my team's last release, I set out to prove that unit tests and quality code lead to fewer bugs.

Here are the tools I used.

Jira

The first thing to know is that the team I'm on keeps its features in Jira, and when we commit code to Subversion, we include the Jira ID in the comments.

Below, you'll see how this helped me relate code to features.

The second thing to know is when we find bugs in a feature during its development, in Jira we link those bugs back to the parent feature.

This allowed me to measure how many bugs had been reported for each feature at release.

SvnPlot

There's an open source project called svnplot that allows you to export the information stored in svn to a SQLite database. To do this, follow the instructions in Step 1 of the Quick Start.

You end up with a couple tables that include rows for each revision, including the comment and the files that were added, removed, and changed.

ANTLR

I used ANTLR C# grammar to help map the files changed to classes.

NCover

I used NCover to measure unit test coverage of the added and changed classes.

NDepend

And finally, I used NDepend to measure code quality of those classes, as well as to query the coverage metrics.

Results

This is the part where I'd love to show some fancy graphs proving my point. Unfortunately, the results weren't exactly conclusive.

I ended up dropping the idea of trying to correlate code quality to number of bugs altogether, because I just ended up with a mess.

The results for how unit test coverage related to bugs is sort-of presentable, at least, but they don't show a nice correlation:

Feature LOC Changed / Bug Average Coverage (%)
1 521 65
2 290 38
3 165 0
4 140 54
5 125 21
6 76 25
7 74 77
8 59 30
9 2 80

I'd like to think I'll try again another day, but given those results, I just don't know. I can come up with many reasons why things didn’t come out the way I'd hoped:

  • The amount of change for each feature was very diverse. Some features involved lots of code changes and others involved very few, so that the features may not really be comparable in this way.
  • Some features involved mostly changing UI code, where the coverage can't be high, because we don't currently have a good way to unit test our UI code.
  • Some features, especially Feature 9, had high test coverage, but the unit tests were very low on actual assertions.
  • There's just not enough data.

In the end, if my argument to the business is that unit tests and code quality are important because they reduce the number of bugs, results like these won't help me make my case.

So, if you know of more persuasive results—either your own or stuff out there in the literature—I'd love to hear about it :-)

49 comments:

  1. interesting. it would be also interesting to see what the correlation would be between LOCC/B and, say, cyclomatic complexity. this should be easy given that you use ndepend. i (somewhat naively) mandate correlation between CC and the level of unit testing. if you have CC of say 20 (!) then there are 20 linearly independent paths through code and thus you should have at least 20 unit tests. how CC relates to bugs etc i really don't know...

    ReplyDelete
  2. That exclusively an individual place which holds total consumers lack by free playstation codes were our value.

    ReplyDelete
  3. Woooow.. They're absolutely cute and funny. Thanks for sharing. I think, I can start to color them all, so if you have some more, you have to share it now! While waiting you can find me on https://sexcamhere.com/, watching amazing girls in action. But if you don't have it, you'll be welcome there anyway. So, if you want to enjoy the best sexy view, that's the right place.

    ReplyDelete
  4. What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. dpe result teletalk com bd

    ReplyDelete
  5. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better! Cheers, keep doing awesome! שאלון אישיות

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. You make so many great points here that I read your article a couple of times. Your views are in accordance with my own for the most part. This is great content for your readers. PSC Result 2019

    ReplyDelete
  8. Blog for share this post its helpful to all PSC Exam Result 2019
    Primary School PSC Exam Result 2019 has been Published. All Board in Bangladesh they can Download her exam result SMS and Web. PSC Result are Published at 10 AM our Prime Minister call a Press brief and announced exam result all over Bangladesh.

    ReplyDelete
  9. I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Interested in internet marketing services company? visit marketing 1on1

    ReplyDelete
  10. I just want to let you know that I just check out your site and I find it very interesting and informative.. top solar power companies in phoenix

    ReplyDelete
  11. Excellent article. Very interesting to read. I really love to read such a nice article. Thanks! keep rocking. water damage restoration los angeles

    ReplyDelete
  12. I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing. Enchanted Florist

    ReplyDelete
  13. Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include. embroidered patches

    ReplyDelete
  14. Fabulous post, you have denoted out some fantastic points, I likewise think this s a very wonderful website. I will visit again for more quality contents and also, recommend this site to all. Thanks. medical cleaning services

    ReplyDelete
  15. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! birre inglesi online

    ReplyDelete
  16. SSC Routine 2020 (Secondary School Certificate Exam Routine 2020) has been publish on the last month 2019! That’s Attractive and Great news for all SSC Examinee and Their Guardians. The Education Board of Bangladesh and all Education Board chairman has announced the SSC New Routine 2020 on last month. A large Number of SSC Examine and other People don’t know how to check SSC Routine 2020 Fast. The Education Board authority also published HSC Routine 2020 on their official website. Now you can see the HSC New Routine 2020 quickly.
    There are another good news for the National University Students of Bangladesh that the NU Degree 2nd Year Routine 2020 PDF Download has been published on the official website today.

    ReplyDelete
  17. Please continue this great work and I look forward to more of your awesome blog posts. hardwood floor water damage repair warping

    ReplyDelete
  18. What a fabulous post this has been. Very useful post. I am grateful to you and expect more number of posts like these. Thank you very much. Job Circular

    ReplyDelete
  19. The online ordering of any product, making a stock trade, or buying an airline ticket, for example, best programming keyboard are some work that programming needs to bind internally for the end users.

    ReplyDelete
  20. This is a brilliant post I seen because of offer it. It is truly what I needed to see trust in future you will proceed for sharing such an amazing post 먹튀폴리스꽁머니

    ReplyDelete
  21. I wanted to thank you for this great read!! I definitely enjoying every little bit of this post. 안전놀이터

    ReplyDelete
  22. National University is published the nu honours 4th year exam result 2021 on online. Students now can check the result from nu.ac.bd/results as well as examresulthub.com

    ReplyDelete
  23. 20 Public General, Science and Technology University will publish the GST primary selection result on Very Soon. According to the news, Guccho Admission Result will be published gstadmission.ac.bd.
    sorkari chakrir khobor 2021 in our website.
    All Source Find- jobnewsbd24.com/sitemap

    ReplyDelete
  24. i predicted to make you the little remark simply to express profound gratitude a ton indeed approximately the putting tests you have recorded in this text. It is honestly charitable with you to deliver straightforwardly exactly what numerous people may have promoted for a virtual book to get a few benefit for themselves, especially considering the way which you may simply have done it on the off threat that you wanted. Those focuses additionally served like a respectable technique to be sure that the rest have similar longing absolutely like my very own private to peer a variety of extra as some distance as this trouble. I'm sure there are notably extra lovable conferences front and middle for people who inspect your blog. I'm very intrigued along with your composing abilities and furthermore with the layout in your blog. Is that this a paid challenge or did you modify it yourself? Whichever manner hold up the great best composition, it's miles uncommon to peer an terrific weblog like this one nowadays.. I assume other web site proprietors need to receive this site as a model, spotless and terrifi consumer friendly style and plan, no longer to mention the substance. You are a expert on this theme! 먹튀사이트

    ReplyDelete
  25. first-rate article. Think so new form of factors have integrated into your article. Sitting tight to your next article . I am glad to locate this post very beneficial for me, as it carries lot of facts. I constantly opt to examine the best and glad i discovered this issue in you put up. Thank you . This definitely allows me in my paintings. Lol, thanks on your comment! Wink glad you discovered it useful . These are only a couple of factors to think about while seeking out sports activities betting books and structures. Precise day! The way you have got written your publish is so best. Very expert . Hold writing this brilliant post. Admiring the effort and time you put into your weblog and distinctive statistics you provide! Happy to talk your blog, i appear to be forward to extra dependable articles and i assume we all wish to thank such a lot of correct articles, weblog to percentage with us. Terrific put up i should say and thanks for the facts. Training is truely a sticky situation. However, continues to be most of the leading subjects of our time. I respect your publish and stay up for extra . I am for the first time here. I found this board and i in locating it truly helpful & it helped me out lots. I hope to provide something returned and help others which includes you helped me 안전놀이터

    ReplyDelete
  26. your article were given me thinking about some thing. Who will we touch if we need anesthesia quick? I have heard there are some cell offerings on line, and now i understand which of these services is pleasant at their task. I am glad my buddies told me approximately them. Wow! Such an fantastic and useful post that is. I truely genuinely like it. It's so correct and so exquisite. I'm simply surprised. I am hoping which you retain to do your work like this inside the future also. Fantastic dispatch! I'm certainly getting apt to over this info, is truly neighborly my friend. Likewise notable weblog right here amongst most of the expensive data to procure. Reserve up the useful technique you're doing right here. Wow, what's extraordinary article. It's far precisely what i were trying to find. i would love to inform you please keep sharing those type of information. If feasible, thank you. The writer has inspired me with the best high-quality work here. I'll ensure that i percentage this article with as many humans as i can. I suppose it is crucial to elevate cognizance approximately this situation. The most effective manner to achieve this is through sharing. Genuinely, this text is genuinely one of the very first-rate in the history of articles. I am a antique ’article’ collector and i on occasion examine a few new articles if i discover them interesting. And i found this one pretty charming and it ought to move into my collection. Excellent paintings! That is my first time i visit here. I found so many enjoyable stuff in your weblog, mainly its discussion. From the lots of feedback for your articles, i wager i'm not the simplest one having all the leisure right here! Preserve up the good work. I've been meaning to write down some thing like this on my internet site and you've got given me an idea. better than ordinary blog submit and accurate first-rate content, this become a brief substance for me and all of us. You ought to maintain sharing any such substance. I suppose that is a elegant article, and the content material published is super. This content will assist me to complete a paper that i've been operating on for the closing 2 weeks. It was a hard 2 weeks, however i am glad the work is accomplished now. 먹튀신고

    ReplyDelete
  27. first-rate inspiring article. You have an amazing website. It's miles certainly clean, and quality to navigate. I think the internet site is exceptional, appears superb and really clean to follow . This is very pleasant and remarkable preserve writing such form of fantastic blogs it was a splendid risk to go to this type of site and i am happy to recognise. Thanks so much for giving us a hazard to have this possibility.. Extraordinarily fascinating alternate happy that i ran over such beneficial submit. Keep doing exceptional. Satisfied to be a chunk of your internet group 메이저놀이터

    ReplyDelete
  28. Thanks for the exceptional proportion. Your article has proved your hard work and experience you've got were given on this discipline. First rate . I like it studying. I’m certain i'm able to at closing make a move the use of your guidelines on those things i ought to by no means had been capable of touch by myself. You had been so modern to let me be one of these to gain out of your useful statistics. Please understand how a lot i'm thankful. In reality respectable submit. I simply located your blog and wished to mention that i have certainly cherished surfing around your weblog entries. Regardless i'll be subscribing for your nourish and i accept as true with you compose afresh soon! I'm in reality taking part in reading your properly written articles. It seems like you spend a whole lot of time and effort in your weblog. I've bookmarked it and i'm searching forward to studying new articles. That is extraordinarily exciting substance! I've absolutely favored perusing your focuses and feature arrived at the realization which you are ideal approximately a widespread lot of them. You're excellent. I am thrilled and lucky to return on in your web page, i truly preferred the top notch article in your web page. Thanks for this useful data. I additionally determined very thrilling statistics 실시간바카라

    ReplyDelete
  29. i experience so fortunate because i've a blogger such as you that gives fresh ideas depending at the every day scenario. To be very honest, your blogs are easy to examine and apprehend. Desirable success along with your destiny articles as well. thank you for this engaging article. On occasion you operate actual-existence examples on your articles which can be best for me as a reader, but lamentably, from time to time i didn’t get your point inside the put up. It seems fake or irrelevant. Nicely, it’s a very good one. Top notch put up i need to say and thanks for the statistics. Education is truely a sticky concern. But, continues to be many of the leading subjects of our time. I respect your post and sit up for more. I truely cherished reading your blog. It changed into very well authored and smooth to apprehend. Not like other blogs i've examine which can be sincerely no longer that right. Thanks alot! In case you once in a while plan on the use of the web browser that’s now not an problem, however if you’re making plans to browse the internet alot out of your pmp then the ipod’s large screen and better browser can be important. I assume this is a really precise article. You are making this facts thrilling and engaging. You give readers a lot to reflect onconsideration on and i respect that kind of writing. Excellent factors you wrote here.. Fantastic stuff... I think you've made a few actually thrilling factors. Preserve up the good paintings 먹튀검증사이트

    ReplyDelete
  30. i continuously visit your weblog and retrieve everything you publish here but i by no means commented however nowadays once i noticed this submit, i couldn't prevent myself from commenting right here. Satisfactory mate! Extraordinary! I am surprised at how nicely you use words to get your factor across. I'd be interested by reading extra of your work . I used to be just surfing thru the internet seeking out a few statistics and got here across your weblog. I am impressed by way of the information which you have on this blog. It indicates how well you apprehend this difficulty. Bookmarked this web page, will come lower back for more. This text was written through a actual questioning author. I agree some of the with the stable factors made through the writer. I’ll be again quickly . Thank you for posting this info. I simply need to can help you know that i just take a look at out your web site and i locate it very exciting and informative. I cannot wait to study lots of your posts. Very first-rate!!! After i search for this i found this weblog at the pinnacle of all blogs in seek engine. For the very first time, i study your weblog and without a doubt loved it. Your way of delivering content is surely superb and considerable. Desire you may maintain this and offer new, particular and strong content material out of your facet. For me, it’s a five stars article . 먹튀검증업체

    ReplyDelete
  31. excellent blog. I loved reading your articles. This is clearly a fantastic read for me. I have bookmarked it and i am looking ahead to studying new articles. Maintain up the good paintings! Effortlessly, the thing is truely the high-quality subject matter on this registry related issue. I suit in along with your conclusions and will eagerly look forward to your next updates. Absolutely, this newsletter is absolutely one of the very fine within the history of articles. I'm a antique ’article’ collector and i occasionally read a few new articles if i find them interesting. And i found this one pretty charming and it need to move into my collection. Superb work! I trully appretiate your work and recommendations given through you is useful to me. I will proportion this data with my family & pals. That is a extremely good internet site, hold the positive critiques coming. write my dissertation this is a notable inspiring . I am pretty an awful lot thrilled with your properly paintings. You placed truly very beneficial information. I am trying to analyzing your next publish. !!!! This newsletter is clearly contains lot extra facts about this topic. We've got read your all of the statistics a few factors are also good uncooked and a few typically are first-rate. Amazing publish i would love to thank you for the efforts 우리카지노

    ReplyDelete
  32. that is a definitely proper tip specially to those new to the blogosphere. Brief however very accurate records… many thank you for sharing this one. A need to study article! I really like and admire your blog article. A good buy thank you once more. Hold writing. Correct day! I actually would love to offer you a large thumbs up for your fantastic data you've got proper right here on this put up. I will be coming returned in your net website for more speedy. Hello there, youu have achieved an remarkable job. I’ll truely digg it and in my opinion endorse to my buddies. I don’t even know how i completed up right here, but i perception this submit turned into as soon as high-quality. I don’t apprehend who you’re however without a doubt you’re going to a famous blogger in case you show up to aren't already . Thank you masses for sharing! 토토사이트

    ReplyDelete
  33. When you're feeling stressed, your body releases a large amount of acetycholine. Acetycholine is an amino acid that is present in the human brain and spinal cord. 1000mg Delta 8 Gummies Sour Burst

    ReplyDelete
  34. Thanks for the complete test. Now all doubts are solved.

    ReplyDelete
  35. This comment has been removed by the author.

    ReplyDelete
  36. Thanks for the detailed article on this topic. I would like to see more such awesome articles from you.

    ReplyDelete
  37. I visit some sites and sites everyday to read articles, however this website offers feature based posts. If you need a visa to enter Turkey, you may apply for a Turkish visa and fill the visa application form Turkey. It is way easy and quick to get a Turkish e-Visa by following three simple steps.

    ReplyDelete
  38. Thanks for sharing this valuable piece of information. Keep sharing more such awesome articles in the future. Goodbye!

    ReplyDelete
  39. Nice article thank you for this. Check the Kenya visa requirements before you apply for the Kenya visa through online visa application. The e visa application offers the fast visa services. Thank you

    ReplyDelete
  40. Great post I have to say and thanks for the info. Education is definitely a complicated subject. However, it remains one of the main themes of our time. I appreciate your post and I look forward to more.

    ReplyDelete
  41. Thanks for sharing this information.Keep it up...What is Emergency India e-Visa? Indian Emergency e-Visa works faster than normal e-Visa as it cuts down on your time to process visa applications. Emergency visa to India cost depends on your nationality & your visa type.

    ReplyDelete
  42. Hey everyone, Do you know about the Indian visas? Tourists can now apply for a visa for India or eVisa for India. In 5 to 10 minutes you can fill your application form online through Indian Visa Portal. And you can read more information about Visa on our India Visa Blog.


    ReplyDelete
  43. thank you for sharing this amazing and wonderful article, if you want modded apps and games for free do check out apkframe
    scoompa video mod apk

    ReplyDelete