Evaluating Methods for Data Collection during Software Engineering Experiments

Abstract

The research presented in this thesis evaluates three methods for data collection during software engineering experiments: the concurrent and retrospective think-aloud methods and the feedback-collection method. A controlled experiment was conducted with 39 students. The participants conducted three change tasks on a 3600 LOC Java application by means of JBuilder 9. The objective was to evaluate the three methods with regard to type of collected information, interaction between observers and participants, and participants perceptions. For participants assigned to the concurrent think-aloud and feedback-collection conditions, the experiment was followed by a questionnaire to evaluate the method to which they had been assigned. The participants were asked to rate their agreement or disagreement to a number of statements. Two weeks after the experiment, group interviews were conducted to gain further insights into their experiences.

The results showed differences in type of collected information between the three data collection methods. The concurrent think-aloud method provided more information on participants task-performing actions than the other two methods. The retrospective think-aloud method provided more information on experimental material. The feedback-collection method provided more information on participants plans, strategies, comprehension and problems in the task-solving process, and participants perceptions of the experiment. Information on the conduct of the experiment can be useful in revealing problem-solving processes. Collected information on the experimental context and participants perceptions can be useful in validating data obtained by other data sources, and also in revealing ethical and organisational problems when conducting software engineering experiments.

It has been demonstrated that it is not easy to assure that observers follow the given guidelines, and more thorough training for the observers is required. Despite the guidelines stating that interaction with the participants was not allowed, 9% (retrospective think-aloud) and 2% (concurrent think-aloud) of the statements in the protocols were related to interaction initiated by observers.

The results showed differences between participants regarding their perception of the methods. These results may help identifying ethical or other issues with the experiment design, which may influence the experimental conduct. There were some indications that the concurrent think-aloud method and the feedback-collection method affected the performance of the task. Some participants felt that the verbalisation procedure was disturbing. A few participants in both think-aloud methods claimed that the verbalisation helped them to structure their thoughts. Some participants assigned to the methods applied in retrospect (feedback-collection method and retrospective think-aloud method) reported problems with recalling their thoughts. One participant in each group reported being uncomfortable with the experiment situation. This information, even subjective in its nature, can be useful insight for improving experiment design.