Daniel H. Caro

An Analytical Approach to Study Educational Inequalities: 10 Hypothesis Tests in PIRLS 2006.

The paper draws on the work of Willms [2006. Learning divides: Ten policy questions about the performance and equity of schools and schooling systems. Montreal: UNESCO Institute for Statistics] to present an analytical approach to the study of academic achievement disparities related to family socioeconomic status. The approach is illustrated by evaluating 10 hypotheses with two-level and three-level hierarchical linear models using data from the Progress in International Reading Literacy Study 2006. For each hypothesis, the underlying theory, statistical model, and critical model test are presented and the results are discussed. The analytical approach can be generalized to other studies and data sets. The results help to understand how inequalities are configured at the within-country and between-country levels.

Cultural, social, and economic capital constructs in international assessments: an evaluation using exploratory structural equation modeling

The article employs exploratory structural equation modeling (ESEM) to evaluate constructs of economic, cultural, and social capital in international large-scale assessment (LSA) data from the Progress in International Reading Literacy Study (PIRLS) 2006 and the Programme for International Student Assessment (PISA) 2009. ESEM integrates the theory-generating approach of exploratory factor analysis (EFA) and theory-testing approach of confirmatory factor analysis (CFA). It relaxes the zero-loading restriction in CFA, allowing items to load on different factors simultaneously, and it provides measurement invariance tests across countries not available in EFA. A main criticism of international LSA studies is the extended use of indicators poorly grounded in theory, like socioeconomic status, that prevent the study of mechanisms underlying associations with student outcomes. This article contributes to addressing this criticism by providing statistical criteria to evaluate the fit of well-defined sociological constructs with the empirical data.

Achievement inequalities in Hamburg schools: how do they change as students get older?

A handful of studies have found evidence of a gap in academic achievement between students of high- and low-socioeconomic status (SES) families. Furthermore, some scholars argue that the gap tends to widen as students get older. Evidence is, however, inconclusive and relies mostly on limited methodological designs. Drawing on the Hamburg School Achievement Census 1996 to 2000 (LAU 5, 7, and 9), the authors examined the trajectory of the math and reading achievement gap associated with SES from the age of 10 to 15 years by means of cross-random-effects, panel data, and hierarchical linear models (HLM). The use of 3 time points and statistical techniques well suited to the longitudinal data contribute to the methodological advance of measurement of the gap trend. The findings indicate a narrowing gap over the course of secondary school years and are interpreted in light of the relatively open and egalitarian school policies in Hamburg.

Socioeconomic background, education, and labor force outcomes: evidence from a regional US sample

This paper examines the long-term association of family socioeconomic status (SES), educational, and labor force outcomes in a regional US longitudinal sample (N = 2264). The results offer insights into the mechanisms underlying the role of family SES in transitions from secondary schooling to early work experiences. It was found that the academic achievement gap associated with SES widens during secondary schooling due in part to course-level tracking. Family SES relates to college enrollment mainly via its association with academic gains in school, but also through family income and father’s occupational status. Family SES is weakly but significantly related to adult offspring’s earnings but more strongly related to occupational status. Educational qualifications and cognitive skills make independent contributions to the explanation of labor force outcomes.

Performance Status and Change--Measuring Education System Effectiveness with Data from PISA 2000-2009.

Reports of international large-scale assessments tend to evaluate and compare education system performance based on absolute scores. And policymakers refer to high-performing and economically prosperous education systems to enhance their own systemic features. But socioeconomic differences between systems compromise the plausibility of those comparisons and references. The paper applies conceptual and methodological approaches from educational effectiveness research to investigate how effectively education systems perform and how effectively they change their performance over time by accounting for socioeconomic differences between systems and cohorts (assessment cycles). Data from 4 cycles of the Programme for International Student Assessment (PISA) are analysed. Results indicate that the quality of systems is evaluated differently if assessed by absolute performance scores or effectiveness measures. The study contributes to methodological developments of effectiveness research in international large-scale assessments and provides relevant information for policymakers to further look into policies, structures, and reform measures that have favoured effectiveness.

Ten Hypotheses about Tolerance toward Minorities among Latin American Adolescents

Discrimination on the grounds of sexual orientation, race, mental and/or physical disability, religious background, HIV/AIDS status, and ethnic origin affects the well-being of minorities and society in general. Recent research in North America underscores the importance of contact with diverse networks, intergroup discussions, a social dominance orientation, religious beliefs, and the school climate, among other factors, to explain tolerance for minorities. Theoretical and empirical work in Latin America is less extensive and has been limited by the lack of quantitative data. This article evaluates 10 different hypotheses about tolerance, using data from eighth-grade students in six Latin American countries that participated in the International Civic and Citizenship Education Study in 2009. Overall, the results provide support for most hypotheses. Notably, countries with relatively more positive views of minorities tend to exhibit very small differences in tolerance attitudes among students of varying socio-demographic characteristics and, apparently, also more effective mechanisms for promoting tolerance through schools.

Causal Mediation in Educational Research: An Illustration Using International Assessment Data

Abstract This paper applies the causal mediation framework proposed by Kosuke Imai and colleagues (Imai, Keele, & Tingley, 2010) to educational research by examining the causal mediating role of early literacy activities in parental education influences on reading performance. The paper argues that the study of causal mediation is particularly relevant in educational settings because learning outcomes result from complex interactions involving multiple actors. The analyses use retrospective longitudinal data from the Progress in International Reading Literacy Study (PIRLS) 2011. The value of international assessment data for producing evidence of causation is discussed. The paper aims to contribute to the study of causation with observational data from educational studies.

Tackling the remaining attainment gap between students with and without immigrant background: an investigation into the equivalence of SES constructs

In England, students with immigrant background exhibit lower educational attainment than those without immigrant background. Family socioeconomic status (SES) helps explain differences in educational attainment, but a gap remains that differs in size for students with different immigrant backgrounds. While the explanatory repertoire for the remaining gap is broad, it has been neglected to comprehensively investigate whether family SES constructs are equivalent across students with different immigrant backgrounds. Using data from the first wave of the Children of Immigrants Longitudinal Survey in Four European Countries (CILS4EU) for England (n = 4,315), the paper applies exploratory structural equation modelling (ESEM) to evaluate measurement invariance of family background constructs across students without and with immigrant background, specifically Pakistani/Bangladeshi immigrant background. Results suggest differences in the structure of family SES indicators across groups and in their association with educational attainment. Complementary variables are suggested to enhance family SES indicators. Findings are relevant to researchers investigating educational inequalities related to immigrant background.

Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

This study evaluated rater accuracy with rater-monitoring data from high stakes examinations in England. Rater accuracy was estimated with cross-classified multilevel modelling. The data included face-to-face training and monitoring of 567 raters in 110 teams, across 22 examinations, giving a total of 5500 data points. Two rater-monitoring systems (Expert consensus scores and Supervisor judgement of correct scores) were utilised for all raters. Results showed significant group training (table leader) effects upon rater accuracy and these were greater in the expert consensus score monitoring system. When supervisor judgement methods of monitoring were used, differences between training teams (table leader effects) were underestimated. Supervisor-based judgements of raters’ accuracies were more widely dispersed than in the Expert consensus monitoring system. Supervisors not only influenced their teams’ scoring accuracies, they overestimated differences between raters’ accuracies, compared with the Expert consensus system. Systems using supervisor judgements of correct scores and face-to-face rater training are, therefore, likely to underestimate table leader effects and overestimate rater effects.

Addressing omitted prior achievement bias in international assessments: an applied example using PIRLS-NPD matched data

Abstract Omitted prior achievement bias is pervasive in international assessment studies and precludes causal inference. For example, reported negative associations between student-oriented teaching strategies and student performance are against expectations and might actually reflect omitted prior achievement bias. Namely, that these teaching strategies are negatively correlated with unobserved prior achievement performance, because teachers offer more support to lower performing students, and not that these strategies cause lower performance. This paper examines omitted prior achievement bias in teaching effects with prior achievement data available for students in England participating in the Progress in International Reading Literacy Study. Further, it proposes an analytical approach to account for omitted prior achievement bias in international estimates. The paper argues that the bias is not simply a technical artefact, but reflects educational mechanisms unobserved in international assessment studies, which can be captured with matched assessment data-sets or with evidence from previous studies. Estimates of these mechanisms can be used to postulate scenarios of the bias across education systems and thereby adjust international estimates of teaching effects as if prior achievement were observed. Potentials and limitations of this approach for studying educational effectiveness with international assessment data are discussed.