Lorena Llosa

Standards-based classroom assessments of English proficiency: A review of issues, current developments, and future directions for research

With the United States’ adoption of a standards-based approach to education, most attention has focused on the large-scale, high-stakes assessments intended to measure students’ mastery of standards for accountability purposes. Less attention has been paid to the role of standards-based assessments in the classroom. The purpose of this paper is to discuss key issues and challenges related to the use of standards-based classroom assessments to assess English language learners’ English proficiency. First, the paper describes a study of a standards-based classroom assessment of English proficiency in a large urban school district in California. Second, using this study as an example and drawing from the literature in language testing on classroom assessment, this paper highlights the major issues and challenges involved in using English proficiency standards as the basis for classroom assessment. Finally, the article outlines a research agenda for the field given current developments in the areas of English proficiency standards and classroom assessment.

Using program evaluation to inform and improve the education of young English language learners in US schools

The purpose of this paper is to discuss how program evaluation can be conducted and communicated in ways that meaningfully affect the education of English language learners (ELLs) in US schools. First, the paper describes the Waterford Early Reading Program Evaluation, a large-scale evaluation of a reading intervention implemented in schools with substantial populations of ELLs in a large urban school district in California. Second, using the Waterford evaluation as an example, this paper discusses the conditions necessary for conducting an evaluation that yields useful information about a programs implementation and effectiveness. The paper also highlights the importance of communicating those findings in a clear way so as to be meaningful to stakeholders and decision-makers in order to facilitate the goal of improving the education of young ELLs.

Impact of a Large-Scale Science Intervention Focused on English Language Learners

The authors evaluated the effects of P-SELL, a science curricular and professional development intervention for fifth-grade students with a focus on English language learners (ELLs). Using a randomized controlled trial design with 33 treatment and 33 control schools across three school districts in one state, we found significant and meaningfully sized intervention effects on a researcher-developed science assessment and the state science assessment. Subgroup analyses revealed that the P-SELL intervention had a positive and significant effect for each language proficiency group (ELLs, recently reclassified ELLs, former ELLs, and non-ELLs) on the researcher-developed assessment. The intervention also had a positive effect for former ELLs and non-ELLs on the state science assessment, but for ELLs and recently reclassified ELLs, the effect was not statistically significant.

The Challenges of Writing Exposition: Lessons From a Study of ELL and Non-ELL High School Students

The purposes of the study we describe here were (a) to identify the challenges that English Language Learner and non–English Language Learner high school students in an urban public school district experience when composing in the genre of exposition, a genre considered to be central to advanced academic literacy; and (b) to relate these challenges to characteristics of the writing they produce. We present a descriptive inventory of the challenges these adolescents faced when composing in the genre of exposition and also compare the challenges that the 2 groups experienced both in relation to each other and in relation to characteristics of the writing they produced. Finally, we relate our findings to implications for improving writing instruction and assessment for adolescents.

Construct Relevant or Irrelevant? The Role of Linguistic Complexity in the Assessment of English Language Learners' Science Knowledge.

This article addresses the issue of language-related construct-irrelevant variance on content area tests from the perspective of systemic functional linguistics. We propose that the construct relevance of language used in content area assessments, and consequent claims of construct-irrelevant variance and bias, should be determined according to the degree of correspondence between language use in the assessment and language use in the educational contexts in which the content is learned and used. This can be accomplished by matching the linguistic features of an assessment and the linguistic features of the domain in which the assessment is measuring achievement. This represents a departure from previous work on the assessment of English language learners’ content knowledge that has assumed complex linguistic features are a source of construct irrelevant variance by virtue of their complexity.

Toward a More Nuanced Approach to Program Effectiveness Assessment: Hierarchical Linear Models in K–12 Program Evaluation

Most K–12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K–12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K–12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.

Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks

This article reports on an investigation of the role raters’ language background plays in raters’ assessment of test takers’ speaking ability. Specifically, this article examines differences between American and Indian raters in their scores and scoring processes when rating Indian test takers’ responses to the Test of English as a Foreign LanguageTM Internet-Based Test (TOEFL iBT®) Speaking tasks. Three American and three Indian raters were asked to score 60 speech samples from 10 Indian test takers’ responses to TOEFL iBT Speaking tasks and to perform think-aloud protocols while scoring. The data were analyzed with Multifaceted Rasch and verbal protocol analyses. Findings indicate that Indian raters were better than American raters at identifying and understanding features of Indian English in the test takers’ responses. However, Indian and American raters did not differ in their use of scoring criteria, their attitudes toward Indian English, or in the internal consistency and severity of the scores.

Assessing English Learners' Progress: Longitudinal Invariance of a Standards-Based Classroom Assessment of English Proficiency

Assessing and monitoring student progress is becoming increasingly important in classrooms and for accountability purposes. Yet, in order to interpret changes in assessment results from one year to the next as reflecting differences in underlying ability rather than as variations in the measurement, the assessments used should be measuring the same constructs over time. Gathering evidence of an assessments longitudinal invariance is particularly important when the assessments used are based on teacher judgments because teacher judgments are often viewed as inconsistent, and different teachers may be involved each year. This study examined the extent to which a standards-based classroom assessment based on teacher judgments measures English proficiency consistently over time by examining its longitudinal invariance using confirmatory factor analysis. Results indicate that the English Language Development Classroom Assessment measures the same overall construct of English proficiency in Grades 2, 3, and 4: Invariance was established for the overall factor structure of the assessment. However, only partial invariance was established for the first- and second-order factor loadings. These findings suggest that, when the focus is overall language proficiency, teacher judgments could be used to make meaningful determinations of student progress from one year to the next.

School Resources in Teaching Science to Diverse Student Groups: An Intervention’s Effect on Elementary Teachers’ Perceptions

Elementary school teachers’ perceptions of school resources (i.e., material, human, and social) for teaching science to diverse student groups were examined across three school districts from one state. As part of a 3-year curricular and professional development intervention, we examined the effect on teachers’ perceptions after their first year of participation. The study involved 103 fifth-grade teachers from 33 schools participating in the intervention and 116 teachers from 33 control schools. The teachers completed a survey at the beginning and end of the school year. As a result of the intervention, teachers in the treatment group reported more positive perceptions of school resources than teachers in the control group.

Comparability of students’ writing performance on TOEFL iBT and in required university writing courses

Investigating the comparability of students’ performance on TOEFL writing tasks and actual academic writing tasks is essential to provide backing for the extrapolation inference in the TOEFL validity argument (Chapelle, Enright, & Jamieson, 2008). This study compared 103 international non-native-English-speaking undergraduate students’ performance on two TOEFL iBT® writing tasks with their performance in required writing courses in US universities as measured by instructors’ ratings of student proficiency, instructor-assigned grades on two course assignments, and five dimensions of writing quality of the first and final drafts of those course assignments: grammatical, cohesive, rhetorical, sociopragmatic, and content control. Also, the quality of the writing on the TOEFL writing tasks was compared with the first and final drafts of responses to written course assignments using a common analytic rubric along the five dimensions. Correlations of scores from TOEFL tasks (Independent, Integrated, and the total Writing section) with instructor ratings of students’ overall English proficiency and writing proficiency were moderate and significant. However, only scores on the Integrated task and the Writing section were correlated with instructor-assigned grades on course assignments. Correlations between scores on TOEFL tasks and all dimensions of writing quality were positive and significant, though of lower magnitude for final drafts than for first drafts. The TOEFL scores were most highly correlated with cohesive and grammatical control and had the lowest correlations with rhetorical organization. The quality of the writing on the TOEFL tasks was comparable to that of the first drafts of course assignment but not the final drafts. These findings provide backing for the extrapolation inference, suggesting that the construct of academic writing proficiency as assessed by TOEFL “accounts for the quality of linguistic performance in English-medium institutions of higher education” (Chapelle, Enright, & Jamieson, 2008, p. 21).