How much empirical evidence is there to support claims made by software engineering tool providers about the benefits of their tools, and how valid is this evidence?

Abstract

Software engineering (SE) practitioners should consider the empirical validity of claims set forth by SE tool providers in correspondence with an Evidence-based Software Engineering approach, but to do so, empirical evidence needs to be available. As far as I have managed to determine, I conduct the first study looking into claims made by SE tool providers and subsequently the amount and validity of empirical evidence available directly from the tool providers. The study indicates that 61% of the tools have associated claims with no valid empirical material to support them, while only 22% can be said to have a certain minimum amount of valid empirical evidence to back up some of the linked claims. Furthermore, in cases where independent empirical evidence is available, there are examples of that material giving a false impression unless thoroughly scrutinized. Additionally, for the vast majority of cases, data intended as empirical evidence is considered generally invalid and is largely based on case studies lacking in necessary detail. Half of the 23 organizations contacted responded and of these only two thirds sent material. The results are derived from critical appraisal of that material, as well as material found on SE tool providers respective product websites, and are discussed with relation both to SE tool adoption and marketing. The study contributes by concluding that SE practitioners need to employ their power as consumers to enforce a paradigm change so that more valid independent empirical data will surface, to the definite benefit of the SE practitioners and the potential benefit of the tool providers. Further research is suggested, in particular looking into the selection process of reference clients and the measurements behind the stated benefits of case study companies.