Reporting framework-based software process improvement : A quantitative and qualitative review of 71 experience reports of CMM-based SPI

Abstract

Software development projects have a notoriously high failure rate. Software process improvement (SPI) frameworks have since the early 1990-ies been a suggested remedy for this. The Capability Maturity Model (CMM) is such a framework, but the actual process of implementing the CMM has proven difficult for many software organizations. Another problem is that documentation of the actual benefits of CMM-based SPI (CBS) is vague. In a pursuit of rectifying the situation we present a quantitative and qualitative review of 71 published case stories of CBS. With the data collected we set out to examine several issues: first, the potential for software organizations for learning from and reproducing the almost non-exclusively positive results of CBS reported in case stories, second, to what degree the calculations of Return On Investment (ROI) present believable numbers, and last, if CBS is something that is beneficial for the software industry as a whole. We found that, first, because case stories are largely reported by companies that are unrepresentative for the industry as a whole, the average company will have problems learning from and reproducing the results reported. Secondly, we found that calculations of ROI in general in the literature are of doubtful quality, but with a few prominent and notable exceptions which indicate that viable calculations of ROI for CBS are possible. Finally, we present a reasoning that indicates that CBS probably is beneficial for the software industry as a whole. Drawing on a tradition in the SPI literature of collecting "success factors" for CBS in assisting implementation, we also present a list of all explicitly reported "success" and "non-success"-factors found in the case stories.