Implementing and Evaluating a Fault-proneness Prediction Model to Focus Testing in a Telecom Java Legacy System

Abstract

Assuring high quality software is perceived as a key factor to succeed in the software industry. However, deliveries of products of poor quality are still common due to time constraints and limited resources. One way to address this problem is to try to make the software testing process more efficient, and hence find more faults in less time or with fewer resources. For example, fault proneness prediction models can be built on the basis of historic data about changes and faults combined with measures of the structural properties of the software. Assuming that a sufficiently accurate model can be built on the basis of the available data, the model can be applied on a forthcoming release of the software system – giving predictions that identify those software modules that are likely to contain faults. Having identified the fault-prone modules, the testing activities can focus on those modules to improve testing efficiency.

This thesis investigates how one can build, evaluate and use class level fault-proneness prediction models for the purpose of focusing testing in the context of a large, evolving Java legacy system. First, different data mining techniques for building the prediction model were evaluated both in terms of classification accuracy and cost-effectiveness. The results of this evaluation suggested that the best model was very accurate and could, for example, identify as many as 60 percent of the faults in only 10 percent of the system. Second, the practical usefulness of a prediction model based unit testing activity was evaluated: The model was used to rank classes in an upcoming release according to their fault proneness. The most fault prone classes then received additional unit testing, by augmenting the test suite to achieve higher branch and loop coverage on the fault-prone classes, assuming this would reveal additional faults. A relatively large number of faults were identified and corrected within a small subset of the most fault-prone classes, confirming the more theoretical cost-effectiveness estimates: in practice, the model had selected classes with at least ten times higher fault density than what would be expected based on fault data collected from previous releases. The time spent on the focused unit testing activity was recorded. Then interviews were conducted with the developers to elicit expert estimates on the potential cost savings of having performed the focused testing activity and thus preventing some faults from slipping through to later phases. The evaluation indicated that cost saved was 91.7 hours and time consumed during unit testing was 49.5 hours, indicating more than 50 percent cost reduction due to the focused testing activity.