Ronny Scherer

Factors of problem-solving competency in a virtual chemistry environment: The role of metacognitive knowledge about strategies

The ability to solve complex scientific problems is regarded as one of the key competencies in science education. Until now, research on problem solving focused on the relationship between analytical and complex problem solving, but rarely took into account the structure of problem-solving processes and metacognitive aspects. This paper, therefore, presents a theoretical framework, which describes the relationship between the components of problem solving and strategy knowledge. In order to assess the constructs, we developed a virtual environment which allows students to solve interactive and static problems. 162 students of grade 10 and the upper secondary level completed the tests within a cross-sectional survey. In order to investigate the structure of problem-solving competency, we established measurement models representing different theoretical assumptions, and evaluated model fit statistics by using confirmatory factor analyses. Results show that problem-solving competency in virtual environments comprises to three correlated abilities: achieving a goal state, systematical handling of variables, and solving analytical tasks. Furthermore, our study provides empirical evidence on the distinction between analytical and complex problem solving. Additionally, we found significant differences between students of grades 10 and 12 within the problem-solving subscales, which could be explained by gaming experience and prior knowledge. These findings are discussed from a measurement perspective. Implications for assessing complex problem solving are given.

Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: an application of multilevel bifactor structural equation modeling

Research on educational effectiveness most often uses student assessments of classroom instruction for measuring aspects of teaching quality. Given that crucial inferences on the success of education are based on these assessments, it is essential to ensure that they provide valid indicators. In this study, we illustrate the application of an innovative application of a multilevel bifactor structural equation model (ML-BFSEM) to examine the validity of student assessments. Analyzing a large-scale data set of 12,077 fourth-grade students in three countries (Finland, Norway, and Sweden), we find that (i) three aspects of teaching quality and subject domain factors can be established; (ii) metric and scalar invariance could be established for the ML-BFSEM approach across countries; and (iii) significant relations between students’ assessments of how easy the teacher is to understand and achievement in all subjects exist. In support of substantive research, we demonstrate a methodological approach for representing the complex nature of student assessments of teaching quality. We finally encourage substantive and methodological researchers to advance the ML-BFSEM.

The quest for comparability: Studying the invariance of the teachers' sense of self-efficacy (TSES) measure across countries

Teachers’ self-efficacy is an important motivational construct that is positively related to a variety of outcomes for both the teachers and their students. This study addresses challenges associated with the commonly used ‘Teachers’ Sense of Self-Efficacy (TSES)’ measure across countries and provides a synergism between substantive research on teachers’ self-efficacy and the novel methodological approach of exploratory structural equation modeling (ESEM). These challenges include adequately representing the conceptual overlap between the facets of self-efficacy in a measurement model (cross-loadings) and comparing means and factor structures across countries (measurement invariance). On the basis of the OECD Teaching and Learning International Survey (TALIS) 2013 data set comprising 32 countries (N = 164,687), we investigate the effects of cross-loadings in the TSES measurement model on the results of measurement invariance testing and the estimation of relations to external constructs (i.e., working experience, job satisfaction). To further test the robustness of our results, we replicate the 32-countries analyses for three selected sub-groups of countries (i.e., Nordic, East and South-East Asian, and Anglo-Saxon country clusters). For each of the TALIS 2013 participating countries, we found that the factor structure of the self-efficacy measure is better represented by ESEM than by confirmatory factor analysis (CFA) models that do not allow for cross-loadings. For both ESEM and CFA, only metric invariance could be achieved. Nevertheless, invariance levels beyond metric invariance are better achieved with ESEM within selected country clusters. Moreover, the existence of cross-loadings did not affect the relations between the dimensions of teachers’ self-efficacy and external constructs. Overall, this study shows that a conceptual overlap between the facets of self-efficacy exists and can be well-represented by ESEM. We further argue for the cross-cultural generalizability of the corresponding measurement model.

Is it time for a new measurement approach? A closer look at the assessment of cognitive adaptability in complex problem solving

In an ever-changing society, in which scientific knowledge increases rapidly, and workplace demands shift toward “twenty-first century skills,” the ability to adapt ones thinking, drive, and emotions to changing and novel problem situations has become essential (OECD, 2013). As a consequence, a number of researchers from different disciplines developed assessment tools that allow for inferences on the level of cognitive adaptability an individual is able to achieve (Martin and Rubin, 1995; Ployhart and Bliese, 2006; Birney et al., 2012; Martin et al., 2013; Cole et al., 2014). However, the limited evidence on construct validity that comes along with existing assessments that do not provide interactive and dynamic performance tasks points to the need of considering alternative assessment methods (Bohle Carbonell et al., 2014). With the advancement of computer-based assessments of complex problem solving (CPS) in educational contexts, new opportunities of measuring adaptability occur, which may overcome these challenges (Wood et al., 2009; Greiff and Martin, 2014). Specifically, the rich data obtained from such assessments, which, for instance, include information on response times, sequences of actions, and the navigation through the assessment), allow researchers to study cognitive adaptability in more depth, as they go beyond mere performance measures (i.e., correct vs. incorrect). The main message of this article consequently reads: It is time to exploit the potential of computer-based assessments of CPS in order to measure cognitive adaptability as a twenty-first century skill.

The Relations Among School Climate, Instructional Quality, and Achievement Motivation in Mathematics

Instructional quality is considered to be an important classroom variable, as it is significantly related to student achievement and motivation in mathematics. Existing studies in educational effectiveness furthermore identified a positive relation between instructional quality and school climate, suggesting that the school environment plays a significant role in teachers’ instructional practices. In order to bring together these two core findings, the relations among different aspects of school climate, instructional quality, and students’ achievement motivation for the TIMSS 2011 grade eight mathematics data sets comprising 50 countries are investigated. In particular, the role of instructional quality as a potential mediator between school climate and student motivation is examined, thereby focusing on three aspects of school climate (emphasis on academic success, safety, and order in schools) and three aspects of achievement motivation (self-concept, intrinsic value, and extrinsic value). In general, there was a significant positive relation between instructional quality and achievement motivation at the classroom level in mathematics; in some countries, a partial mediation of instructional quality between school climate and achievement motivation was apparent. Four main patterns of relations occurred. These findings are discussed with respect to implications for educational effectiveness research.

Psychometric Challenges in Modeling Scientific Problem-Solving Competency: An Item Response Theory Approach

The ability to solve complex problems is one of the key competencies in science. In previous research, modeling scientific problem solving has mainly focused on the dimensionality of the construct, but rarely addressed psychometric test characteristics such as local item dependencies which could occur, especially in computer-based assessments. The present study consequently aims to model scientific problem solving by taking into account four components of the construct and dependencies among items within these components. Based on a data set of 1,487 German high-school students of different grade levels, who worked on computer-based assessments of problem solving, local item dependencies were quantified by using testlet models and Q 3 statistics. The results revealed that a model differentiating testlets of cognitive processes and virtual systems fitted the data best and remained invariant across grades.

The Contribution of International Large-Scale Assessments to Educational Research: Combining Individual and Institutional Data Sources

ABSTRACT The present paper aims to discuss how data from international large-scale assessments (ILSAs) can be utilized and combined, even with other existing data sources, in order to monitor educational outcomes and study the effectiveness of educational systems. We consider different purposes of linking data, namely, extending outcomes measures, analyzing differences over time or across cohorts, and supplementing context information. These linking strategies are illustrated by a non-exhaustive selection of studies that exploited ILSAs to investigate a wide range of educational topics. We conclude that the main contribution of ILSA to educational research lies in the ways they facilitate analyses of educational policy and policy-related issues at the institutional level by means of cross-country analyses. However, the scope of these studies also covers high-quality data on lower levels of the educational system.

Still Comparing Apples With Oranges

Measurement invariance testing has become an integral part of themethodological repertoire in psychological assessment. Whenever researchers attempt to compare scales or test scores across groups or over time, it needs to be ensured that they do not compare “apples with oranges.” Measurement invariance testing is the road that leads to the information about whether the comparisons of means or variances are valid. In other words, invariance testing examines the between-group or between-time points variation in factor means and variances by ruling out differences in measurement (Bauer, 2017). This ensures the fairness of scales or tests and therefore contributes to craftingavalidityargument (Kane, 2013). In this editorial, we briefly review some of the principles and practices of measurement invariance testing in psychological assessment contexts and encourage EJPA authors to critically review the limitations of the commonly used approach to invariance testing – multi-group confirmatory factor analysis. We suggest considering alternatives to this approach, for instance, when submitting to EJPA and we discuss the potential of these alternative approaches.

Some critical reflections on the special issue: Current innovations in computer-based assessments

Technology and computers change and penetrate our lives to an extent that was unthinkable 30 years ago, and it is the mission of Computers in Human Behavior to advance our knowledge on how humans interact with, make use of, and are influenced by computers. The nine papers in this special issue “Current Innovations in Computer-Based Assessment” reflect the innovations and advances computers have brought to the ways we assess psychological attributes of a variety of populations including primary and secondary students, students in higher education, or adults in the work force

The Development of Scientific Strategy Knowledge Across Grades

In this study, we developed a new test on scientific strategy knowledge and investigated the construct validity of the resulting test scores. Moreover, measurement invariance across grade levels has been analyzed to ensure the generalizability of the assessment. Furthermore, convergent and discriminant validity were investigated. A total of N = 1,182 German high school students of Grade Levels 8, 10, and 12 completed tasks on strategy knowledge, fluid intelligence, content knowledge, interest in science, and scientific self-concept within a cross-sectional study. Multigroup confirmatory factor analysis was used to check for measurement invariance. Our results show that scalar invariance holds across grades and that there are significant differences in performance favoring students of higher grade levels. Furthermore, fluid intelligence and content knowledge are relevant predictors of strategy knowledge, whereas gender and motivational constructs do not show significant effects. Implications for developmental studies on strategy knowledge and assessment practice are discussed.