| dc.description.abstract | To conduct controlled experiments is the classical method for identifying cause-effect relationships. This helps researchers evaluate and validate their research results. This master thesis quantitatively systematically surveys the present practices of reporting and using time in 113 controlled software engineering experiments published in 103 scientific articles in the period 1993-2002. The survey can be regarded as an extension of the study of Sjøberg et al. [6], who investigated several other aspects of the same collection of controlled experiments.
The need for an analysis of time in controlled SE-experiments was recognized based on the results of an earlier survey conducted by Jo E. Hannay at SIMULA Research Laboratory. He identified that time often is reported as dependent variable. Due to the dependent variable’s important role in experiments as provider of the effect construct in the causal relationship, it was relevant to find out how time is used.
The investigation of time in controlled SE-experiments focuses on the following aspects; the overall frequency of reporting time, the use of time, the recording of time, the specification of time in terms of time units, and the validity threats that time can constitute. These aspects were regarded as the most important features of time, and hence, would give a thorough and appropriate picture of time in controlled SE-experiments.
The main results of the survey of the aspects of time, is first of all, that time is reported in a substantial majority of controlled SE-experiments. Second, in most of the controlled SE-experiments time is used as dependent variable, or more precise, as measure of subjects’ experimental tasks. However, in an extensive number of controlled SE-experiments time is not explicitly described as dependent variable, although it is used in that sense. Third, time measures are found to be recorded by subjects, experimenters and tools. An interesting finding in relation to this aspect is that the largest proportion of controlled SE-experiments did not explicitly report their time recording. Fourth, a majority of controlled SE-experiments measure the time in minutes. Last, but not least, very few controlled SE-experiments were found to address time in relation to validity threats.
The main conclusions and recommendations of the research, is first of all, that SE-researchers documenting controlled experiments should explicitly describe time as dependent variable in the experimental design, when time is used as dependent variable. This is obviously important due to the vital role of a dependent variable in experimentation. Second, SE-researchers should consistently describe how time is recorded in controlled experiments, due to validations of the experiment results and to possible replications. Third, SE-researchers should specify time measures in minutes, because it provides an appropriate granularity of the time data. And fourth, aspects of time constitute threats to both internal and external validity, and SE-researchers should therefore assess time in relation to possible validity threats of the controlled experiments.
The hope of this master thesis is that the results, discussions, conclusions and recommendations of the research provide useful insight to SE-researchers, so that they can improve the design of time in their controlled experiments. This is vital in order to advance a state-of-the art practice in empirical software engineering research. | nor |