Sascha Wüstenberg

Dynamic Problem Solving: A New Assessment Perspective

This article addresses two unsolved measurement issues in dynamic problem solving (DPS) research: (a) unsystematic construction of DPS tests making a comparison of results obtained in different studies difficult and (b) use of time-intensive single tasks leading to severe reliability problems. To solve these issues, the MicroDYN approach is presented, which combines (a) the formal framework of linear structural equation models as a systematic way to construct tasks with (b) multiple and independent tasks to increase reliability. Results indicated that the assumed measurement model that comprised three dimensions, information retrieval, model building, and forecasting, fitted the data well (n = 114 students) and could be replicated in another sample (n = 140), showing excellent reliability estimates for all dimensions. Predictive validity of school grades was excellent for model building but nonexistent for the other two MicroDYN dimensions and for an additional measure of DPS. Implications are discussed.

Assessing complex problem-solving skills with multiple complex systems

In this paper we propose the multiple complex systems (MCS) approach for assessing domain-general complex problem-solving (CPS) skills and its processes knowledge acquisition and knowledge application. After defining the construct and the formal frameworks for describing complex problems, we emphasise some of the measurement issues inherent in assessing CPS skills with single tasks (i.e., fixed item difficulty, low or unknown reliability, and a large impact of random errors). With examples of the MicroDYN test and the MicroFIN test (two instances of the MCS approach), we show how to adequately score problem-solving skills by using multiple tasks. We discuss implications for problem-solving research and the assessment of CPS skills in general.

Assessment of Complex Problem Solving: What We Know and What We Don’t Know

ABSTRACT Complex Problem Solving (CPS) is seen as a cross-curricular 21st century skill that has attracted interest in large-scale-assessments. In the Programme for International Student Assessment (PISA) 2012, CPS was assessed all over the world to gain information on students’ skills to acquire and apply knowledge while dealing with nontransparent problem situations. This review article summarizes current developments in the assessment of CPS using multiple complex systems and focuses on four topics: (a) What is CPS and why is it relevant, (b) What do we know about CPS, (c) What do we gain from its assessment, and (d) What there is to come. More specifically, we (a) elaborate on how changes in today’s world call for the assessment of CPS, (b) summarize recent empirical results on CPS, pinpoint how log file analyses of students’ behavioral data (c) help to understand differences in CPS, and (d) may deepen the understanding of CPS.

Development of learning to learn skills in primary school

In Finland, schools’ effectiveness in fostering the development of transversal skills is evaluated through large-scale learning to learn (LTL) assessments. This article presents how LTL skills—general cognitive competences and learning-related motivational beliefs—develop during primary school and how they predict pupils’ CPS skills at the end of sixth grade. The six-year follow-up of 608 pupils shows that cognitive competences demonstrated in the beginning of the first grade in a learning preparedness test predict both later cognitive LTL competences and CPS, but their development is not fully determined by earlier individual differences in learning preparedness in the first grade. Motivational beliefs begin to be related to cognitive LTL performance gradually from age 10 on, and they may have a slightly stronger effect on CPS than on cognitive LTL performance. It is concluded that the development of CPS is partly depending on pupils’ initial learning preparedness and the development of their LTL skills.

A longitudinal study of higher-order thinking skills: working memory and fluid reasoning in childhood enhance complex problem solving in adolescence.

Scientists have studied the development of the human mind for decades and have accumulated an impressive number of empirical studies that have provided ample support for the notion that early cognitive performance during infancy and childhood is an important predictor of later cognitive performance during adulthood. As children move from childhood into adolescence, their mental development increasingly involves higher-order cognitive skills that are crucial for successful planning, decision-making, and problem solving skills. However, few studies have employed higher-order thinking skills such as complex problem solving (CPS) as developmental outcomes in adolescents. To fill this gap, we tested a longitudinal developmental model in a sample of 2,021 Finnish sixth grade students (M = 12.41 years, SD = 0.52; 1,041 female, 978 male, 2 missing sex). We assessed working memory (WM) and fluid reasoning (FR) at age 12 as predictors of two CPS dimensions: knowledge acquisition and knowledge application. We further assessed students’ CPS performance 3 years later as a developmental outcome (N = 1696; M = 15.22 years, SD = 0.43; 867 female, 829 male). Missing data partly occurred due to dropout and technical problems during the first days of testing and varied across indicators and time with a mean of 27.2%. Results revealed that FR was a strong predictor of both CPS dimensions, whereas WM exhibited only a small influence on one of the two CPS dimensions. These results provide strong support for the view that CPS involves FR and, to a lesser extent, WM in childhood and from there evolves into an increasingly complex structure of higher-order cognitive skills in adolescence.

Shifts in the Assessment of Problem Solving

In recent decades, the types of problems encountered in daily life have changed considerably. In particular, the technology-based society of today increasingly requires people to deal with dynamically changing, intransparent, and complex problems that cannot be solved only by applying factual knowledge. Moreover, in education and at work, such complex problems often need to be addressed collaboratively. Problem solving skills belong to the skills needed to successfully master such problems and are therefore considered to be crucial in the twenty-first century, but we can differentiate between problem solving on an individual level and collaborative problem solving. In the first part of this chapter, we provide insight into individual complex problem solving (CPS) research, with a special focus on its assessment. Specifically, we elaborate on different computer-based approaches to the measurement of CPS and their benefits and limitations. On the basis of the results of various empirical studies, we describe to which extent the CPS assessment has an added value in explaining the scholastic achievement and offer insights into a particular development of the assessment of this skill in large-scale educational contexts. Specifically, we elaborate on the assessment of CPS in the most influential large-scale project worldwide – the Programme for International Student Assessment (PISA) – and illustrate the process by describing example tasks. Further, we elaborate on how research on CPS has expanded towards the conceptually related skill of collaborative problem solving (ColPS) and discuss specific challenges in its assessment. With respect to the recent inclusion of ColPS in large-scale assessments, we present the assessment approach of another large-scale worldwide project – The Assessment and Teaching of Twenty-first Century Skills (ATC21S) – and describe how ColPS was assessed in the PISA 2015 cycle. Finally, we outline the importance of problem solving concepts in general and CPS specifically for future research, and discuss open research questions in this field.

The Heidelberg Inventory of Geographic System Competency Model

The concept “system” is fundamental to many disciplines. It has an especially prominent place in geography education, in which additionally, the spatial perspective is central. Empirically validated competency models dealing specifically with geographic systems—as well as adequate measurement instruments—are still lacking. Therefore, based on the theoretically-guided development of a Geographic System Competency (GSC) model, the aim was to build and evaluate such a measurement instrument, with the help of probabilistic measurement models. The competency model had three dimensions: (1) “comprehend and analyze systems”, (2) “act towards systems” and (3) “spatial thinking”, whereby dimension (2) was changed to “evaluating possibilities to act towards systems” after a thinking-aloud study. A Cognitive Lab (CogLab) and two quantitative studies (Q1 n = 110, Q2 n = 324) showed divergent results. Dimension (2) could not be identified in both quantitative studies. Whereas Dimensions (1) and (3) constituted separate dimensions in Q1, in Q2 the two-dimensional model did not fit significantly better than the one-dimensional model. Besides showing the close relationship between spatial and systemic thinking in geographic contexts, which are thus both needed in modeling GSC, the project highlights the need for more research in this central area of geography education.

Longitudinal evidence for a continuing cognitive cascade. Working memory and fluid intelligence enhance complex problem solving in adolescence.

Scientists have studied the development of the human mind for decades and have accumulated an impressive number of empirical studies that have provided ample support for the notion that early cognitive performance during infancy and childhood is an important predictor of later cognitive performance during adulthood. As children move from childhood into adolescence, their mental development increasingly involves higher-order cognitive skills that are crucial for successful planning, decision-making, and problem solving skills. However, few studies have employed higher-order thinking skills such as complex problem solving (CPS) as developmental outcomes in adolescents. To fill this gap, we tested a longitudinal developmental model in a sample of 2,021 Finnish sixth grade students (M = 12.41 years, SD = 0.52; 1,041 female, 978 male, 2 missing sex). We assessed working memory (WM) and fluid reasoning (FR) at age 12 as predictors of two CPS dimensions: knowledge acquisition and knowledge application. We further assessed students’ CPS performance 3 years later as a developmental outcome (N = 1696; M = 15.22 years, SD = 0.43; 867 female, 829 male). Missing data partly occurred due to dropout and technical problems during the first days of testing and varied across indicators and time with a mean of 27.2%. Results revealed that FR was a strong predictor of both CPS dimensions, whereas WM exhibited only a small influence on one of the two CPS dimensions. These results provide strong support for the view that CPS involves FR and, to a lesser extent, WM in childhood and from there evolves into an increasingly complex structure of higher-order cognitive skills in adolescence.