The background information on subjects in program comprehension studies

Abstract

Program comprehension is a very important skill a software engineer need. Many researchers conduct experiments on program comprehension in order to improve tools, documentation, and maintenance guidelines supporting program comprehension. Individual programmers’ productivity might vary significantly even though they have similar background. Thus, the subjects’ background is very important when conducting and analyzing experiments on program comprehension. The survey presented in this short Master thesis identifies subjects background information reported in software experiments on program comprehension. The background information reported in 24 articles was systematically analyzed in order to answer what kind of background information is reported and how the background information was used in the analysis.

The articles reports many different background variables, but the overall impression of the background information reported in program comprehension experiments is that it is rather arbitrary and small. The analysis shows that there is a need for standards and guidelines of how to collect and report subjects’ background information. The survey shows also that almost no background information of the subjects is used in the experiments’ analysis. The articles in this survey provide so little information about the subjects’ background that it is difficult to perform replications and meta-analysis. This thesis aims to make researchers more aware of the subjects’ background in their experiments and reports.

On the basis of the results of the analysis I have suggested background variables that should be collected in comprehension studies and proposed a background questionnaire. The questionnaire was used in an experiment with 24 subjects from the industry. I report here experiences with the questionnaire.