Process data analysis in problem-solving tasks

Abstract

Problem-solving skills, especially problem-solving skills in technology-rich environments, are critical in today's world as one of the 21st-century skills. International large-scale assessments, such as the OECD's Programme for International Student Assessment (PISA) and the Programme for the International Assessment of Adult Competencies (PIAAC), have highlighted the importance of problem-solving and included it as a core domain in their assessments. These assessments have been widely administered on computers, producing performance data and process data that record a detailed history of the human-computer interaction. Examples of process data in our studies consist of action sequences and timestamps for each action (e.g., mouse clicks and keystrokes). This doctoral dissertation aims to deepen the understanding of problem-solving through the analysis of process data that capture problem-solvers' response processes. The thesis begins with an extended abstract. The extended abstract offers a comprehensive overview of the project and delves into a literature review, covering topics such as problem-solving theories, process data analysis, and latent variable model estimation. Furthermore, this abstract presents the theoretical foundations and methods used, summarizes the main findings, and concludes with a discussion of the contributions and limitations of the project. The second part of this dissertation consists of four articles. To improve the understanding of problem-solving, we analyze process data from PISA 2012 and PIAAC 2012 with the aim of identifying solution patterns (Article I) and validating cognitive processes involved in problem-solving (Article II) within the framework of latent variable modeling. Both Articles I and II define different types of process-based measures, and analyzing them together increases the computational burden. To provide a fast estimation method for high-dimensional latent variable models, we propose to use higher-order Laplace approximations in Articles III and IV. In addition, our approach can simultaneously account for a mixture of ordinal, continuous, and count variables, as well as the dependencies of observed variables from the same item. In summary, this dissertation highlights the potential of process data to improve the understanding of how respondents solve problems and provides tools to increase the estimation efficiency when modeling process data and performance data jointly within the framework of generalized linear latent variable models. Our findings can potentially benefit educational practice by helping students reflect on their response processes, aiding teachers in tailoring their instructions for different students and tasks, and providing materials for test developers to validate task design and for training programs aiming at improving problem-solving skills.