Joost Meijer

Metacognitive activities in text-studying and problem-solving: Development of a taxonomy

This article describes the construction of a hierarchical taxonomy of metacognitive activities for the interpretation of thinking-aloud protocols of students in secondary education, who studied texts on history and physics. After testing an initial elaborate taxonomy on a restricted number of protocols by multiple raters, it appeared that the interrater correspondence was well below par. The categories in the taxonomy were too highly specified. Categories were combined and tested on new protocols in a cyclic fashion. The revised taxonomy was then used for coding 16 history protocols and 16 physics protocols. Frequencies of occurrence of metacognitive activities were obtained, as well as judgements of the quality of the metacognitive activities of the participants. There is a reasonable correlation between the frequency method and the quality method for coding thinking-aloud protocols. Also, there is a substantial covariation of the number of metacognitive activities across both tasks.

Learning potential and anxious tendency: Test anxiety as a bias factor in educational testing

Abstract Contrary to conventional educational testing, in so-called dynamic assessment subjects are allowed to consult help during testing or are offered prior training. The differential results of both testing procedures are sometimes ascribed to the idea that dynamic tests reflect the breadth of the zone of proximal development on top of independent achievement. Alternative explanations claim that conventional tests are more strongly biased towards various characteristics of persons, which have a negative influence on performance, when compared to dynamic tests. In this study, it was hypothesised that static as well as dynamic assessment is biased towards anxious tendencies of subjects, but the former more strongly than the latter. In order to investigate this supposition, the performance of subjects on dynamic and static tests was systematically compared and related to measures of test anxiety in a longitudinal experiment. In the experiment, repeated measures of independent mathematics achievement as well as mathematics learning potential were gathered among students of secondary education in the Netherlands. Prior to every mathematics test, subjects filled out a test anxiety questionnaire. After every mathematics test, subjects filled out a general state anxiety questionnaire. The participating subjects were students from secondary education, either preparing for higher vocational training or university, aged approximately 15 years on average. The results of the experiment showed that lack of self-confidence is an important constituent factor of test anxiety, apart from worry and emotionality. The data supported the assumption that such testing procedures are less biased towards anxiety than conventional tests, but it was not established that dynamic testing procedures render results that are not biased by test anxious tendencies.

STRESS IN THE RELATION BETWEEN TRAIT AND STATE ANXIETY

The relations of stress, trait anxiety, and state anxiety with cognitive performance are the central focus. Stress is conceived of as an external manipulation, trait anxiety as a relatively stable personality characteristic, and state anxiety as transient manifest feelings of insecurity. The influence of these factors on cognitive performance is considered. It was argued that the apparent interaction of stress and trait anxiety with cognitive performance can be traced to the interaction among stress, trait anxiety, and state anxiety. It was hypothesized that state anxiety increases more rapidly as a function of trait anxiety under stressful circumstances than neutral or relatively reassuring circumstances. The negative relation between state anxiety and cognitive performance is assumed to be constant irrespective of stress. Experimental evidence supporting the latter supposition is presented. However, doubts are cast on classical intervention methods for invoking stress. Rather than assuming that these methods invoke stress, it is proposed that feelings of insecurity, associated with ambiguity, are determinants of experienced stress.

The predictive and discriminant validity of the zone of proximal development

BACKGROUND Dynamic measurement procedures are supposed to uncover the zone of proximal development and to increase predictive validity in comparison to conventional, static measurement procedures. AIMS Two alternative explanations for the discrepancies between static and dynamic measurements were investigated. The first focuses on Vygotskys learning potential theory, the second considers the role of anxiety tendency during test taking. If test anxious tendencies are mitigated by dynamic testing procedures, in particular the availability of assistance, the concept of the zone of proximal development may be superfluous in explaining the differences between the outcomes of static and dynamic measurement. SAMPLE Participants were students from secondary education in the Netherlands. They were tested repeatedly in grade three as well as in grade four. Participants were between 14 and 17 years old; their average age was 15.4 years with a standard deviation of .52. METHOD Two types of mathematics tests were used in a longitudinal experiment. The first type of test consisted of open-ended items, which participants had to solve completely on their own. With the second type of test, assistance was available to participants during the test. The latter so-called learning test was conceived of as a dynamic testing procedure. Furthermore, a test anxiety questionnaire was administered repeatedly. Structural equation modelling was used to analyse the data. RESULTS Apart from emotionality and worry, lack of self-confidence appears to be an important constituent of test anxiety. The learning test appears to contribute to the predictive validity of conventional tests and thus a part of Vygotskys claims were substantiated. Moreover, the mere inclusion of a test anxiety factor into an explanatory model for the gathered data is not sufficient. Apart from test anxiety and mathematical ability it is necessary to assume a factor which may be construed as mathematics learning potential. CONCLUSION The results indicate that the observed differences between a conventional, static testing procedure and an experimental, dynamic testing procedure for mathematics cannot be explained sufficiently by a differential bias towards test anxiety. The dynamic testing approach renders scores which add to the predictive validity of conventional testing procedures. Since this gain in predictive validity is not a result of the removal of bias towards test anxiety, this result should be understood as supportive for the validity of the concept of the zone of proximal development.

Influence of Test Anxiety on Measurement of Intelligence

In this study a measurement model for a test anxiety questionnaire was investigated in a sample of 207 Dutch students in the first grade of junior secondary vocational education. The results of a confirmatory factor analysis showed that a model for test anxiety with three factors for worry, emotionality, and lack of self-confidence is associated with a significantly better fit than a model comprised of only the first two factors. The relations of the three test anxiety factors to scores on intelligence tests for measuring verbal ability, reasoning, and spatial ability were examined. The results indicated that test anxiety appears to be transitory: the negative relation between test anxiety and test performance promptly fades away. Finally, we examined whether a distinction can be made between highly test anxious students with low performance due to worrisome thoughts (interference hypothesis) or low ability (deficit hypothesis). Results do not support the deficit hypothesis because the scores of all highly test anxious students increased in a less stressful situation.

An Instrument for the Assessment of Cross-Curricular Skills

An instrument was constructed to assess the level of competence of students (aged 15-16) on eight cross-curricular skills. The instrument, further to be called the cross-curricular skills test (CCST), consisted of 56 multiple-choice items. In the first section cross-curricular skills are defined as general skills which can be taught and practiced in curricula for different disciplines. In the second section we describe how eight cross-curricular skills were selected for testing. We also explain why we decided to compose the test of multiple-choice items, an item-format that is not a customary one for measuring general skills. In the third section we report administration of the test to 465 students in a pilot study and 9,000 students in the main study. Quantitative data are presented concerning p -values, reliability, internal structure, and construct validity of the CCST. Qualitative data were derived from observations, thinking aloud protocols and analysis of the test content. The data warrant the claim that CCST is a valid and reliable test for cross-curricular skills. It is suitable for students of 15-16 years of age and allows administration in large samples. In the final section we discuss some limitations of the test and suggest directions for future research on assessment of cross-curricular skills.

Test anxiety and intelligence testing: A closer examination of the stage-fright hypothesis and the influence of stressful instruction

Abstract The influence of test anxiety and the content of instruction (stressful versus reassuring) on measurements of intelligence were investigated. It was expected that components of test anxiety would show differential effects on test performance. A Latin Square design was used to unravel the effects of test type and test order. Furthermore, effects of type of instruction, stressful versus reassuring, were studied by means of a within-subjects design. Test anxiety was measured with the Revised Worry-Emotionality Questionnaire. Measurements for verbal ability, reasoning, and memory were administered. Performance on memory tests showed less vulnerability to test anxiety compared with the other tests, with a picture recall test being insensitive. The negative effect of test anxiety was mostly confined to the beginning of a test session, independent of the type of test. Partial support for the so-called stage-fright hypothesis was found. The effect of instructional content was equivocal.

ANALYSIS OF SOLVING PROBLEMS

In the context of a larger research project we collected thinking-aloud protocols from pupils of secondary schools when they tried to solve mathematical problems. Pupils solved arithmetic story problems and geometrical problems. The protocols were analysed by using interpretation models. These models were founded on a general problem-solving model. The analysis provided a description of the processes that pupils used to solve the mathematical tasks. Further explication resulted from simulation of solving word problems, using a LISP-based program, SSPS (Simulation of Story Problem Solving).

Multi-domain, multi-method measures of metacognitive activity: what is all the fuss about metacognition … indeed?

Studies about metacognition, intelligence and learning have rendered equivocal results. The mixed model assumes joint as well as independent influences of intelligence and metacognition on learning results. In this study, intelligence was measured by standard tests for reasoning, spatial ability and memory. Participants were 13-year-old school students. Measures of metacognitive activity were gathered by analyses of thinking-aloud protocols within two task domains, i.e., history and physics. Prior knowledge and learning results were measured by tests constructed by the researchers. The results showed that metacognitive activity did not relate to learning results in either task domain. For history, the learning result was only determined by prior knowledge. For physics, intelligence influenced the learning result via prior knowledge, but the effect of execution activity, i.e., exercise, appeared more important. It is possible that ‘learning by doing’, i.e., exercise, is a powerful mean for promoting the application of knowledge in physics.

The factor structure and criterion validity of a test for assessment of cognitive abilities. Comparing students in special and regular education

As a consequence of a recent reform of junior secondary education in the Netherlands, instruments for the assessment of cognitive abilities of students who are eligible for special educational support must be designed. In constructing a new test battery contemporary theory on crystallized and fluid intelligence (Gf-Gc theory) was considered. Complete coverage of the factors in Gf-Gc theory was not striven for, because the status of some of these factors in relation to referral to education with special educational support is not entirely clear. The sample in the study consisted of students from regular and special primary as well as secondary education, aged between 11 and 15 years. Approximately two-thirds of the sample was of Dutch origin, whereas the remaining third was of a different ethnic origin. Results show that the factors verbal-crystallized ability, fluid-reasoning ability, spatial-visualization ability and memory capacity can explain the positive correlations between the tests in the battery for the greater part. Students from regular and special education can be distinguished on the basis of their cognitive profiles reasonably accurate, thus supporting the criterion validity of the test battery. Students requiring special educational support are characterized by a significant discrepancy between short-term recall and long-term retention and retrieval.RésuméUne des conséquences de la réforme récente de l’enseignement secondair aux Pays-Bas est la nécessité de développer des instruments pour l’évaluation des capacités cognitives des enfants qui ont besoin d’un enseignement spécial. Dans la construction d’un nouvel ensemble de tests on a tenu compte de la théorie contemporaine qui fait une distinction entre l’intelligence cristallisée et l’intelligence fluide (la théorie Gf-Gc). On n’a pas cherché à couvrir tous les facteurs de la théorie Gf-Gc, parce que le rôle de ces facteurs dans la décision d’orientation d’un enfant vers l’enseignement spécial n’est pas tout à fait clair. L’âge des enfants sélectionnés pour cette recherche était de 11 à 15 ans. C’étaient des él’`eves de l’enseignement primaire et secondaire, général et spécial. Les deux tiers environ des enfants étaient d’origine hollandaise, les autres étaient d’une autre origine ethnique. Les résultats ont montré que les facteurs de la capacité verbale-cristallisée, la capacité de raisonnement-fluide, la capacité spatiale-visuelle et la capacité de la mémoire peuvent expliquer en bonne partie les corrélations positives entre les tests. Le profil cognitif permet de déterminer de façon raisonnablement précise l’orientation d’un enfant vers l’enseignement général ou spécial. Ce résultat confirme la validité-critère de l’ensemble des tests. Un décalage significatif entre les performances de la mémpire à court terme et de la mémoire à long terme caractérise les enfants qui ont besoin du soutien de l’enseignement spécial.