Comparison of Seven Bug Report Types: A Case-Study of ... - IIIT

More documents

Recommendations

Info

TABLE IXPERCENTAGE OF BUGS IN EACH BUG TYPE WHICH ALSO HAVEREGRESSION LABEL (RF: % OF REGRESSION BUGS)Bug Type Num %RFCRASH 207 7%SECURITY 0 0%CLEAN 0 0%POLISH 1 0.034%PERFORMANCE 15 0.51%USABILITY 0 0%TABLE XPERCENTAGE OF BUGS HAVING STATUS ”WON’T FIX” IN EACH BUG TYPE(MM: MEDIAN MTTR)Bug Type Num %WF %CL MMCRASH 1145 60.42% 25.51% 1320.91REGR. 501 26.44% 15.35% 390.97SECUR. 47 2.48% 11.38% 24.25CLEAN 103 5.43% 12.67% 4276.29POLISH 60 3.17% 20.27% 1643.94PERF. 68 3.59% 30.09% 1561.34USAB. 12 0.63% 25.53% 3195.29TABLE XIPERCENTAGE OF BUGS FOUND TO BE DUPLICATE IN EACH BUG TYPE ANDAMOUNT OF TIME REQUIRED IN THEIR RESOLUTIONBug Type Dupl.(%) MMCRASH 1783 (47.2%) 26.15REGRES. 775 (26.4%) 23.52SECUR. 31 (7.8%) 17.52CLEAN 39 (4.9%) 35.21POLISH 40 (15.3%) 24.13PERF. 24 (14.2%) 362.305USAB. 16 (42.1%) 1143.67TABLE XIIPERCENTAGE OF BUGS VERIFIED IN EACH BUG TYPE ((MM: MEDIANMTTR))Bug Type Num(%) MMCRASH 476 (12.26%) 167.8REGR. 855 (29.10%) 151.83SECUR. 51 (12.81%) 369.71CLEAN 64 (8%) 562.17POLISH 91 (34.86%) 474.95PERF. 33 (19.53%) 768.39USAB. 6 (15.79%) 6398.59of regression bugs are usability bugs because it is oftenchallenging to uncover them with test cases as usability bugmay be user specific. Some usability feature that is bug for oneuser may be perfectly fine for the other user or some specificbug may arise after performing series of steps by a specificuser [12].E. Status of Closed BugsA Closed bug can have different status: such as verified,duplicate, WontFix. An analysis of different close bug statuscan have implications in understanding software projectprocess. For example, a large percentage of duplicate bugreports indicate that bug reporters are not able to find the bugalready reported to the issue tracking system or large numberof users are getting impacted by that bug. A large percentageof WontFix status (closed issues) for feature requests indicatesa mismatch between users need and developers priority. TableX displays the absolute number and percentage of bug reportsin each of the categories with the WontFix Label. In TableX “Num” represents actual count of WontFix bugs in eachbug type, “%WF” represents distribution of total wontfix bugsin each category, and “%CL” represents percenatge of closedand WontFix bugs in each category. According to GoogleChromium bug reporting guidelines 11 , the label WontFix isassigned to issues that cannot be reproduced, working asintended or obsolete. Issues tagged as WontFix are closedwithout taking any action. As shown in the Table X, thepercentage of WontFix varies for different types of bug reports.We observe that a large percentage (60.42%) of WonFix bugsbelong to crash-type. We perform a manual inspection of thecrash bug reports that are not fixed to indentify the underlyingreasons. We notice that not able to reproduce a bug is oneof the major reasons for assigning the label WontFix to crash11 http://www.chromium.org/for-testers/bug-reporting-guidelinesbug reports. We also notice cases where a crash bug reportsare closed as they become obsolete and does not occur in thebuild or milestone which is different than the version on whichthe bug was observed. Regression bug reports also contributea significant percentage (26.44%) to WontFix bugs, whereascontribution of remaining five types of bug reports (security,cleanup, performance, polish and usability) is comparativelyvery small (less than 6%).We calculate number of duplicate bugs for each bug categorymotivated by the fact that large number (25-30%) ofall reported bugs are duplicate [13]–[15]. Large number ofduplicate bugs in a particular bug type can be an indicationthat large number of users are getting affected and can haveimplication on bug prioritizing. Duplicate bug report detectionhave received huge attention from the software engineeringresearch community [11], [14]. Table XI shows that crashhave highest number of duplicate bugs 1783, i.e 68.2% ofall labeled duplicate bugs. Result shows that 47.2%, 42.1%and 26.4% of all crash, usability and regression bugs areduplicate respectively. Table XI shows median MTTR valueof usability bugs is fairly high as compared to othercategories. Research community can use these results to tunetheir duplicate detection approach/tool, and further researchcan be done to handle duplicate bugs as per the category theybelong to. We also study number of bugs verified in eachbug category. In Chrome project a bug fix can be verifiedby reporter or the testing team 12 . We calculated number ofbugs verified in each category to identify whether there is anyinterest from reporters or testing team to verify bugs from aparticular bug type. Table XII shows that percentage of bugsthat are verified is higher in polish and regression bugcategory as compared to other bugs.12 http://www.chromium.org/for-testers/bug-reporting-guidelines524
TABLE XIIINUMBER OF STARSCRASH REGR. SECUR. CLEAN POLISH PERF. USAB.Min 1 0 0 0 0 0 1Q1 3 1 1 1 1 2 2Median 7 2 1 1 2 2 3Q3 12 5 2 2 3 5 9Max 179 647 59 28 40 78 391TABLE XIVNUMBER OF COMMENTSCRASH REGR. SECUR. CLEAN POLISH PERF. USAB.Min 0 0 1 1 1 1 2Q1 1 1 10 3 5 6 5Median 2 2 15 5 7 8 8Q3 3 5 22 8.5 12 14 26Max 219 647 81 71 76 59 126TABLE XVMILESTONE CHANGECRASH REGR. SECUR. CLEAN POLISH PERF. USAB.Num. BRs at leastone MoveFrom/BulkMove 2137 (56%) 2034 (69%) 37 (9.29%) 151 (18.9%) 165 (63.2%) 93 (55%) 33 (86.8%)Avg. 1.048 1.25 1 1.093 1 1.064 1.121Std. Dev. 0.350 0.209 0 0.420 0 0.383 0.477F. Number of Stars and Amount of DiscussionIn addition to bug reporter and bug fixer, several developersor users contribute to the threaded discussion and are interestedon activities related to the bug report. We compute number ofstars and number of comments associated to a bug report.Our interest is to measure number of developers or personsshowing interest and collaborating for different bug types. InGoogle Chrome project, any user can star 13 an issue indicatingusers interest on a specific issue. The issue tracking systemautomatically sends an email notification (triggered by changein the status of the issue) to all the users who starred the issue.We extract both number of start and comments for all the bugreports using the programming APIs. Table XIII reveals that ingeneral crash, regression, performance and usability bugsreceive more stars than security, cleanup and polish bugs.The Q2 and Q3 value for number of stars for crash bugs ishighest in contrast to the other six types of bugs.The Google Issue Tracker provides a facility for users to discussand communicate with each-other by posting comments.The issue tracker is not just a database for archiving bugsbut also serves as an application that acts as a central pointof communication for the project team [16]. We calculate thenumber of comments posted for each bug report and reportthe minimum, Q1, median, Q3 and maximum values in TableXIV. We observe that in general the number of commentsper bug report posted for security bug reports is relativelyhigher than other bug reports. The number of commentsper bug report (based on Q2 and Q3 values) for crash andregression bug reports is the lowest in contrast to other bugreports. Large number of personal interested (large star count13 http://code.google.com/p/support/wiki/IssueTrackeror lots of comments) in a bug is an indication of bug popularityand may impact bug fixing time [17]. Panjer et al. [17]foundthat bugs having less than four comments (less discussion) getfixed faster as compared to other bugs. Our empirical resultsshows a similar phenomenon; crash and regression have lowerQ2 value for number of comments and MTTR as comparedto other bug categories, refer to Table XIV and Table IV.G. Milestone Change FrequencyTable XV shows the absolute numbers and percentage ofmilestone changes for seven different types of bug reports.We notice that in general the percentage of milestone changesis high (more than 50%). The percentage of milestone changesfor crash, regression, polish, performance and usability bug reportsis more than 50%. We did a manual inspection of developercomments for bug reports undergoing milestone changeevent to understand the reason behind milestone changes. Wenotice comments mentioning approval of milestone changes ifthe reported bug is not a blocker for the currently assignedmilestone. According to the official blog 14 by Chromiumproject team, Google Chrome is following the release earlyand release often policy (shorter release cycle : once everysix weeks). A blog 15 mentions that because of the rapidrelease cycle, if a feature is not ready then it is moved to thenext release (new release milestones are created and ticketsare moved). A study by Baysal et al. [18], compares thelifespan of major releases of FireFox and Chrome browsersand characterizes Chrome as a fast evolving system with short14 http://blog.chromium.org/2010/07/release-early-release-often.html15 http://blog.assembla.com/assemblablog/tabid/12618/bid/36341/Secrets-of-rapid-release-cycles-from-the-Google-Chrome-team.aspx525
Page 1 and 2: 2012 19th Asia-Pacific Software Eng
Page 3 and 4: TABLE IIICOMPONENT AND REPORTER ENT
Page 6 and 7: Fig. 8. Bug Type: Crash Fig. 9. Bug
Page 10: version cycles. As shown in Table X

Comparison of Seven Bug Report Types: A Case-Study of ... - IIIT

Create successful ePaper yourself

Delete template?

Save as template?