Eduardo Backhoff

Theory of Test Translation Error.

In this article, we present a theory of test translation whose intent is to provide the conceptual foundation for effective, systematic work in the process of test translation and test translation review. According to the theory, translation error is multidimensional; it is not simply the consequence of defective translation but an inevitable fact derived, among many other reasons, from the tension between translation error dimensions—broad categories of translation errors such as those related to semantics, register, or the construct being measured—and the fact that languages encode meaning in different ways. While it cannot be eliminated, translation error can be minimized. The extent to which the translation of a test item is acceptable or objectionable can be understood as a probabilistic space defined by the frequency and the severity of translation errors. Accordingly, the translation of an item can be objectionable because it has few severe errors, many mild errors, or many severe errors. To illustrate the theory, we discuss the methods used in an investigation that examined the quality of items translated in México in the TIMSS-1995 international test comparison and present results from that investigation. The theory can contribute to ensuring proper implementation of test adaptation guidelines in international comparisons and to improving activities related to test translation and test translation review.

ETHNIC AND RACIAL DIFFERENCES ON THE STANDARD PROGRESSIVE MATRICES IN MEXICO

Ravens Standard Progressive Matrices test was administered to a representative sample of 920 white, Mestizo and Native Mexican Indian children aged 7-10 years in Mexico. The mean IQs in relation to a British mean of 100 obtained from the 1979 British standardization sample and adjusted for the estimated subsequent increase were: 98.0 for whites, 94.3 for Mestizos and 83.3 for Native Mexican Indians.

Principal Perceptions and Student Achievement in Reading in Korea, Mexico, and the United States: Educational Leadership, School Autonomy, and Use of Test Results.

This study compared PISA 2009 student reading literacy scores with principal perceptions across three countries with varying levels of student performance: Korea, Mexico, and the United States. Seventy-five countries participated in PISA 2009, which measured 15-year-old children’s reading achievement and principal perceptions. The study explored the relationship of principals’ perceived levels of leadership, school autonomy, and use of test results with student attainment of reading literacy. School variables were treated as covariates when each effect of principal leadership was interpreted. All variables were included in a multilevel model and analyzed simultaneously. The means and standard deviations of outcome variables and the explanatory and control variables for the model of the study were calculated by taking into account sampling weights, as well as plausible values for reading literacy scores. SAS PROC MIXED was used to fit hierarchical linear models for the study. There was a positive relationship between student achievement in reading literacy and testing to improve instruction in all three countries—Korea, Mexico, and the United States—and there was a negative relationship between student achievement and lack of resources. Social, economic, and cultural status showed a positive relationship with reading literacy. To be specific, testing to improve instruction can be beneficial in all three countries when it is defined as using tests to group students for instructional purposes and to identify aspects of instruction or the curriculum that could be improved. Results also indicate that students are likely to achieve better if principals perceive that there are no shortages of personnel and equipment.

Language Shift and the Inclusion of Indigenous Populations in Large-Scale Assessment Programs

Indicators of academic achievement for bilingual students can be inaccurate due to linguistic heterogeneity. For indigenous populations, language shift (the gradual replacement of one language by another) is a factor that can increase this heterogeneity and poses an additional challenge for valid testing. We investigated whether and how indigenous populations can be validly included in a large-scale assessment program. We gave Mexican preschool Mayan students aged 5 to 6 years the same set of mathematics items in three versions: (1) original in Spanish, (2) Mayan translation, and (3) content equivalent, developed from scratch in Mayan. Also, we collected information on the students’ and the teachers’ use of Spanish and Mayan, and on the communities’ support of the two languages. Students performed poorly on the three versions. Generalizability theory-based analyses revealed considerable performance inconsistency across items and language versions and low generalizability and dependability coefficients. This performance instability appears to stem from a dwindling support of the Mayan language in Mayan schools and communities. Fair, valid assessment of indigenous populations in either their languages or their countries’ dominant languages appears to be difficult to accomplish with current testing models and policies.

The Measurement of Translation Error in PISA-2006 Items: An Application of the Theory of Test Translation Error

We examined the translation of PISA test items based on the theory of test translation error (TTTE), which has proven to allow detection of translation errors with unprecedented levels of detail. Translation error (TE) is defined as the lack of equivalence between the original and translated versions of items on multiple translation error dimensions (TEDs) that involve design, language, and content. According to the theory, TE results not only from poor translation, but also from factors that are beyond the translators’ skills (e.g., languages encode meaning in different ways). We examined the Mexican, Spanish language translation of science and mathematics PISA 2006 items. A panel comprising teachers, translators, a linguist, a test developer, and a measurement specialist examined the translation of 193 text analytical units (55 pieces of introductory text and 138 items) and identified and coded the TEs identified on ten TEDs. For each item, TE was measured as the number of different TEDs on which the review panel identified TEs. To determine which TEDs are critical to student performance, we examined the correlation between TE and item difficulty (percentage of correct answers and mean proportional score, respectively for dichotomous and non-dichotomous items) considering different sets of TEDs. The highest correlations were observed for the sets that included the dimensions, Grammar, Semantics, Register, Information, Construct, and Culture. We also observed different magnitudes of correlations for science and mathematics items and a stronger, statistically significant correlation for translated items whose translation the review panel identified more objectionable than for the rest of the items. These results confirm that language- and content-related TEs may threat the validity of translated items. They speak to the value using the TTTE as a formative evaluation tool that PISA countries can use to operationalize translation guidelines.