John Hattie

Methodology Review: Assessing Unidimensionality of Tests and ltenls:

Various methods for determining unidimensionality are reviewed and the rationale of these methods is as sessed. Indices based on answer patterns, reliability, components and factor analysis, and latent traits are reviewed. It is shown that many of the indices lack a rationale, and that many are adjustments of a previous index to take into account some criticisms of it. After reviewing many indices, it is suggested that those based on the size of residuals after fitting a two- or three-parameter latent trait model may be the most useful to detect unidimensionality. An attempt is made to clarify the term unidimensional, and it is shown how it differs from other terms often used inter changeably such as reliability, internal consistency, and homogeneity. Reliability is defined as the ratio of true score variance to observed score variance. Inter nal consistency denotes a group of methods that are intended to estimate reliability, are based on the vari ances and the covariances of test items, and depend on only one administration of a test. Homogeneity seems to refer more specifically to the similarity of the item correlations, but the term is often used as a synonym for unidimensionality. The usefulness of the terms in ternal consistency and homogeneity is questioned. Uni dimensionality is defined as the existence of one latent trait underlying the data.

The Relationship Between Research and Teaching: A Meta-Analysis

A review of various models of the relationship between research and teaching in universities is presented, and the evidence necessary to assess each model is outlined. A meta-analysis of 58 studies demonstrates that the relationship is zero. Suggestions for future directions are provided, and it is argued that a major goal could be to adopt strategies that enhance the relationship between research and teaching.

Effects of Learning Skills Interventions on Student Learning: A Meta-Analysis

The aim of this review is to identify features of study skills interventions that are likely to lead to success. Via a meta-analysis we examine 51 studies in which interventions aimed to enhance student learning by improving student use of either one or a combination of learning or study skills. Such interventions typically focused on task-related skills, self-management of learning, or affective components such as motivation and self-concept. Using the SOLO model (Biggs & Collis, 1982), we categorized the interventions (a) into four hierarchical levels of structural complexity and (b) as either near or far in terms of transfer. The results support the notion of situated cognition, whereby it is recommended that training other than for simple mnemonic performance should be in context, use tasks within the same domain as the target content, and promote a high degree of learner activity and metacognitive awareness.

Adventure Education and Outward Bound: Out-of-Class Experiences That Make a Lasting Difference

The purpose of this meta-analysis is to examine the effects of adventure programs on a diverse array of outcomes such as self concept, locus of control, and leadership. The meta-analysis was based on 1,728 effect sizes drawn from 151 unique samples from 96 studies, and the average effect size at the end of the programs was .34. In a remarkable contrast to most educational research, these short-term or immediate gains were followed by substantial additional gains between the end of the program and follow-up assessments ( ES = .17). The effect sizes varied substantially according the particular program and outcome and improved as the length of the program and the ages of participants increased. Too little is known, however, about why adventure programs work most effectively.

The Relationship Between Self and Achievement/Performance Measures:

This meta-analysis examines the relationship between the various self-measures and measures of performance and achievement. The statistical results of 128 studies are transformed to a common measure, namely, correlation coefficients. These studies represent a total sample of 202,823 persons and produce a data base of 1,136 correlations between self-ratings and performance measures. A range in the relationship of —.77 to .96 was reported with an “average” correlation of .21. It was found that this average relationship was modified by a number of variables. The more significant modifiers of the average relationship were the grade-level of subjects, socioeconomic status, ethnicity, ability of subjects, self-term used in the study, name of self-test used, type and name of performance/achievement measures, and the reliability of both the self-ratings and performance/achievement measures.

Measuring the Effects of Schooling

A simple model to measure the effects of innovation and schooling is proposed. Using a synthesis of 134 meta-analyses, it is demonstrated that educational innovations can be expected to change average achievement outcomes by .4 standard deviations and affective outcomes by .2 standard deviations. The implications of this model are discussed and the major critical determinants of effective education relate to factors under the control of teachers and students rather than to home, curricula or administrative effects.

A Review of the Research on Interventions for Attention Deficit Hyperactivity Disorder: What Works Best?

This meta-analysis examined 74 studies in which there had been an intervention that aimed to improve the behavioral, cognitive, and/or social functioning of people with attention deficit hyperactivity disorder (ADHD), or attention deficit disorder (ADD). Overall, there were larger effects of the various interventions on behavioral than on educational outcomes. These overall effects were larger for medical interventions than for educational, psychosocial, or parent training interventions, but there was little support for flow-over effects, from the reduction in behavior problems to enhanced educational outcomes. The effects on educational outcomes were greater for educational interventions than for other types of intervention.

Classroom Composition and Peer Effects.

Abstract This chapter examines the extent to which the composition of classes affects learning outcomes. The aim is to explore peer effects when students are organized into classes on the basis of ability, ethnicity, or gender, as well as the effects of multigrade and multi-age classes and class size. The argument is defended that these composition factors affect only the probability that differential instruction and learning occur and that, at best, their influences are indirect. Teachers appear not to change their teaching activities when class composition is changed and most often the power of peer effects is rarely realized. Any direct effects of class composition are less related to learning outcomes and more related to equity and expectation effects by teachers and other participants (students, parents, and principals). Whether a school tracks by ability or not, reduces class sizes, implements multigrade/multi-age or single-level classes, or has coeducational or single-sex classes, appears less consequential than whether it attends to the nature and quality of instruction in the classroom, whatever the between-class variability in achievement. The learning environments within the classroom, and the mechanisms and processes of learning that they foster, are by far the more powerful. Good teaching can occur independently of the class configuration or homogeneity of the students within the class.

Cultural Differences in the Use of Strategies for Self-Regulated Learning:

This article reports the results of a study that compared the strategies used by three different groups of upper secondary school students to regulate their own learning processes: Australian students, Japanese students at school in Japan, and Japanese students studying in Australian schools. Although students in the three groups used a similar range of strategies, the pattern of use for each cultural group varied. Variations in the pattern of strategy use were also associated with academic achievement. The structuring of the physical environment for study purposes and the checking of one’s work were two of the most important strategies for each of the groups. The Japanese students used memory strategies significantly more than did the Australian students. Furthermore, although Japanese students studying in Australia resembled their Australian counterparts more than their Japanese counterparts on many of the strategies, they still attached significantly greater importance to the use of memorization than did the Australian students. This finding is discussed in the light of cultural and educational differences between the two groups in terms of their beliefs regarding the relationship between memorization and understanding.

Assessment and Classroom Learning: a deductive approach

(1998). Assessment and Classroom Learning: a deductive approach. Assessment in Education: Principles, Policy & Practice: Vol. 5, No. 1, pp. 111-122.