Comparison of Seven Bug Report Types: A Case-Study of ... - IIIT

More documents

Recommendations

Info

TABLE ICLOSELY RELATED WORK LISTED IN A CHRONOLOGICAL ORDERBug Type Study Dataset Study Objectives1 Usability Twidale and Mozilla and GNOMENichols (2005)[5]2 Security Gegick et al.(2010) [3]3 Performance, SecurityZaman et al.(2011) [4]4 Crash Khomh et al.(2011) [6]5 Performance Zaman et al.(2012) [7]Cisco Software SystemFirefox ProjectFirefox ProjectMozilla Firefox and Google ChromeUsability bug report analysis to characterize howdevelopers address and resolve issuesAutomatic identification (statistical modeling) of securitybugs based on natural language informationpresent in bug reportsComparison between Security and Performance bugson various aspectsTriaging of crash-types based on entropy regiongraphA qualitative study on performance bugs across fourdimensions (Impact, Context, Fix and Fix validation)TABLE IIGOOGLE CHROME BROWSER EXPERIMENTAL DATASET DETAILSField Value Field ValueFirst Issue ID 2 CRASH Bug Reports 3778Last Issue ID 111889 REGRESSION Bug Reports 2938Reporting Date of First Issue Id 30-08-2008 SECURITY Bug Reports 398Reporting Date of Last Issue Id 29-01-2012 CLEAN Bug Reports 800Issues Downloaded 93951 POLISH Bug Reports 261Issues [state=closed and status=fixed ,verified, duplicate] 53110 PERFORMANCE Bug Reports 169Issues unable to download 17938 USABILITY Bug Reports 38differences between commercial and open-source projects andthe pattern of discourse emerging across projects [5].Gegick et al. [3] present a technique to automaticallyidentify security bugs by text-mining the natural languageinformation present in a bug report. Their research shows thatsecurity bugs are often mislabeled as non-security due to lackof security domain knowledge. They derive a statistical modelfrom a labeled training dataset and then apply the inducedmodel on test dataset. Their solution approach was able toidentify a high percentage of security bugs which were labeledas non-security [3].Khomh et al. [6] present a method for triaging crash-typesbased on the concept of entropy region graph and introduce atechnique that in addition to the frequency counts also capturesthe distribution of the occurrences of crash-types among theusers of a system.Zaman et al. [4] perform an empirical analysis of securityand performance bugs (non-functional bug types) from Firefoxopen-source project and compare various aspects associatedwith the two types of bugs. The study by Zaman et al [4].reveals insights on aspects like: how often the two types ofbugs are tossed and reopened, triage time, characteristics ofbug fixes, number of developers involved and number of filesimpacted.Zaman et al. [7] investigate performance bug reports ofMozilla Firefox and Google Chrome across four dimensions(Impact, Context, Fix and Fix validation) and study severalcharacteristics of performance bugs and make recommendationson how to improve the process of identifying, trackingand fixing performance bugs.In context to existing literature and closely related work, thispaper makes the following unique and novel contributions.1) A first empirical study comparing seven different typesof bug reports (cleanup, crash, polish, performance,regression, security and usability) on a popular opensourceproject (Google Chromium Browser) across multipledimensions (MTTR, priority, frequently occurringterms, bug fixing process quality, continuity and efficiencyperformance characteristics, component and reporterentropy, number of stars, comments, descriptionlength and milestone changes).2) The results and insights presented in this paper answersseveral research questions such as the degree of differencebetween metrics such as mean time to repair,priority and mile-stone changes across seven differenttypes of bug reports. The study provides evidencesof correlation between certain terms and bug reporttypes. Empirical results demonstrating similarities anddifference in bug fixing process quality, continuity andefficiency across different bug reports are presented.III. EXPERIMENTAL DATASETWe conduct experiments on publicly-available data fromopen-source projects so that our experiments can be replicatedand our results can be used for benchmarking or comparison.Google Chromium is a popular and widely used opensource browser 4 and Google Chromium bug database hasalso been used for mining software repositories experiments 5by researchers. We download bug report data from GoogleChromium issue tracker 6 using the issue tracker Java APIs 7 .4 http://www.chromium.org5 http://2011.msrconf.org/msr-challenge.html6 http://code.google.com/p/chromium/issues/list7 http://code.google.com/p/support/wiki/IssueTrackerAPI518
TABLE IIICOMPONENT AND REPORTER ENTROPY VALUES FOR GOOGLE CHROME BROWSER PROJECTProject Entropy CRASH REGR. SECUR. CLEAN POLISH PERF. USAB.Chrome Component 0.552 (2922) 0.666 (2808) 0.577 (330) 0.6345 (761) 0.637 (246) 0.777 (155) 0.764 (36)[Browser] Reporter 0.890 (2468) 0.871 (2365) 0.781 (365) 0.895 (744) 0.905 (214) 0.932 (148) 0.976 (24)Fig. 1.Component and reporter entropy graphs showing the position of seven types of bug reports [Google Chrome Browser Project]Table II shows the dataset (year 2008 to 2012) used forour experiments. As shown in the Table II, we download bugreports from start date (30-August-2008) till (29-Jan-2012).We were not able to download all the bug reports in thespecified range due to permission denied or not availableerrors. Table II shows count of closed bug reports that areresolved as fixed, verified, or duplicate. Table II also showsnumber of bug reports with respect to each of the sevenbug types. As bug labeling is not a mandatory task in bugfixing process (for Google Chrome project), not every bugreport is labeled as crash, regression, security, cleanup, polish,performance or usability. Hence, we check for the label of eachbug report and include it in our experimental dataset only if istagged (manual labeling by developers) with one of the predefinedcategories. As shown in Table II, the total number ofbug reports which are closed, verified or duplicate and belongsto one of the pre-defined categories (crash, regression, security,cleanup, polish, performance and usability) are 8030 (as onebug report can have multiple labels). Assigning meaningfullabels 8 or manual tagging is used by Google Chrome projectteam members to classify bug report. A bug report can havemultiple labels, for example, label for bug type (such assecurity, performance or usability), milestone changes, statusor priority. We perform a manual inspection of Google Chromeissues to identify the labels (and their variations) for the sevendifferent types of bug reports. For example, we categorize abug report to type crash if it has a “Crash” or “Type-Crash”or “Stability-Crash” keywords in the label field. We extractbug report priority by checking for the presence of the label“pri” in a given bug report. Milestone change informationis extracted using the keyword “movedfrom-”, “movedfrom”,“moved from m” or “bulkmove”. Other standard fields suchas bug report state, status, time taken to fix the bug are directlyaccessible using issue tracker API.8 http://code.google.com/p/support/wiki/IssueTrackerIV. EMPIRICAL ANALYSIS AND RESULTSEach of the following subsections focuses on one particulardimension or a set of coherent metrics across which the sevendifferent types of bug reports are compared and contrasted.A. Entropy across Components and ReportersWe compare various types of bug reports based on a metriccalled “Entropy”. Khomh et al. [6] define a term called ascrash-entropy which is employed in their proposed techniqueto prioritize bug reports for the purpose of triaging. Theyargue that for bug-reports which are linked to crash-types, itis not only important to prioritize based on the raw frequency(number of bug reports for the specific crash-type) but alsobased on the number of users affected by it (distribution acrossusers). Khomh et al [6]. define crash entropy as a metricthat quantifies the distribution of the occurrences of variouscrash-types across users . Two crash types may have the samenumber of bug-reports belonging to their respective categoriesbut the distribution across users can be different.Similarly, we apply the main concept (of measuring Entropy)in the context of this study and present our insights.However, there are differences between the applications of theconcept in the work done by Khomh et al. [6] and our work.Our objective is to measure, compare and contrast the entropyof seven types of bug reports across components and reporters.The formula for entropy (as used in [6] and [8]) is shown inEquation 1:i=nH n (BT )=− p i ∗ log n (p i ) (1)∑i=1In Equation 1, BT denotes bug types (one of the seven categories:crash, regression, security, clean, polish, performanceand usability). n is the total number of unique componentsor reporters (depending on the application for components orreporters) in the dataset. H n (BT ) denotes the entropy for a specificbug-type and p i is the probability of a specific component519
Page 1: 2012 19th Asia-Pacific Software Eng
Page 6 and 7: Fig. 8. Bug Type: Crash Fig. 9. Bug
Page 8 and 9: TABLE IXPERCENTAGE OF BUGS IN EACH
Page 10: version cycles. As shown in Table X

Comparison of Seven Bug Report Types: A Case-Study of ... - IIIT

Create successful ePaper yourself

Delete template?

Save as template?