Process Data Analysis in ILSAs

Abstract

Computational advancements in the last couple of decades have brought forth a new era of international large-scale assessments (ILSAs) in which the administration of computer-based tests is becoming the norm. Beyond collecting the correct/incorrect answers for each item, computer-based assessments are also able to collect a range of actions performed by the respondents in the computer testing application during the course of the test administration. Both respondents’ actions – starting a unit, clicking a button, spending time until inputting or submitting an answer, and so forth – and their overall behavior – what they do with their keyboards, mice, and even their own eyes – are recorded as a new set of data called “process data,” which are normally time-stamped and can be stored in the so-called log files. There may be plenty of useful insight into the respondent’s cognitive process, and process data can potentially become a relevant element in the scoring process of an assessment, validate test score interpretations, to name a few possibilities for the analysis of such data. This chapter aims to contribute to the body of knowledge in this area by offering (1) an introduction to process data, what kind of data it contains, its relation to the cognitive process, and how it can be organized into a proposed six-layered ecological framework that facilitates its analysis; (2) a literature review of 37 seminal and state-of-the-art publications produced in the last decade on the topic, which are then analyzed both in their chronological perspective as well as in how they fit into the ecological framework; and (3) a discussion of the potential and limitations of using process data in the assessment framework, including a view of what could be the next steps in the analysis of process data from ILSAs.