Lindsay Clare Matsumura

Measuring Instructional Quality in Accountability Systems: Classroom Assignments and Student Achievement

This article reports the technical quality of a measure describing the quality of classroom assignments piloted in the Los Angeles Unified School Districts proposed new accountability system. For this study, 181 teachers were sampled from 35 schools selected at random. Participating teachers submitted three language arts assignments with samples of student work (N = 50). Results indicated a fair level of agreement among the raters who scored the assignments and a high level of internal consistency within each dimension of assignment quality. Results presented a mixed picture with regard to the stability of the ratings at the different levels of schooling and the number of assignments needed to yield a consistent estimate of quality. However, secondary students who received higher quality assignments produced higher quality written work and scored higher as a group on the reading and language portions of the Stanford Achievement Test, 9th edition (Stanford 9) adjusted for student background and prior achievement.

Implementing Literacy Coaching The Role of School Social Resources

This study investigates the influence of a school’s pre-existing social resources on the implementation of a comprehensive literacy-coaching program (Content-Focused Coaching [CFC]). Elementary schools were randomly assigned to receive a CFC-trained coach (n = 15 schools) or to continue with the literacy coaching resources that are standard for the district (n = 14 schools). Ninety-six fourth- and fifth-grade teachers participated in the study (n = 63 CFC and n = 33 comparison). Survey results indicate that teachers in the CFC schools participated more frequently in the coaching activities that emphasized planning and reflecting on instruction, enacting instruction, and building knowledge of the theories underlying effective reading comprehension instruction compared to teachers in the comparison schools. After 1 year, teachers strongly believed that CFC coaching helped improve their instructional practice. Principal leadership was the key resource supporting implementation of the program positively predicting greater teacher participation in coaching activities and perceived usefulness of these activities along with coaches’ training in the CFC program and less experienced teachers. Unexpectedly, a school’s pre-existing culture of teacher collaboration negatively predicted teachers’ coaching experiences. CFC coach interviews contribute to understanding the interactions of social resources within schools that facilitated or hindered program implementation. Implications for the design and implementation of effective instructional coaching policies in districts are discussed.

Leadership for Literacy Coaching: The Principal's Role in Launching a New Coaching Program.

Purpose: This study investigated the relationship between principal leadership and variation in teachers’ participation in a new literacy coaching program: Content-Focused Coaching® (CFC). Research design: Twenty-nine schools were randomly assigned to participate in the CFC program or to serve as a comparison. Interviews were conducted with elementary school principals and coaches, and teachers completed surveys describing their experiences with their new coach. Correlation analyses investigated the relationship between the categories of principal support and the frequency of teachers’ participation in individual coaching activities. Principals’ actions and beliefs were also compared across schools, with teachers’ relatively high and low participation in coaching, to identify patterns in principal leadership. Findings: Principal leadership was significantly associated with the frequency with which teachers conferred with their new CFC coach and were observed by their new coach as teaching reading comprehension lessons. Principal behaviors associated with teachers’ increased engagement with coaches included actively participating in the CFC program and publicly endorsing the coach as a source of literacy expertise to teachers. Principal beliefs regarding a literacy coach’s role and responsibilities were associated with the frequency with which teachers opened their classrooms to the new coaches. Implications: This study provides insight into the features of principal leadership that may support coaches in engaging with teachers and gaining access to their classrooms. Observing teachers’ lessons is a critical dimension of effective coaching and a difficult task for coaches to accomplish. Learning how principals can positively contribute to this process could help schools and districts make more effective use of their literacy coaching resources.

The Effect of Content-Focused Coaching on the Quality of Classroom Text Discussions.

This study examines the effect of a comprehensive literacy-coaching program focused on enacting a discussion-based approach to reading comprehension instruction (content-focused coaching [CFC]) on the quality of classroom text discussions over 2 years. The study used a cluster-randomized trial in which schools were assigned to either CFC or standard practice in the district for literacy coaching. Observers rated classroom text discussions significantly higher in CFC schools. Teachers in the CFC schools participated more frequently in coaching activities that emphasized planning and reflecting on instruction, enacting lessons in their classrooms, building knowledge of the theory underlying effective pedagogy, and differentiating instruction than did the teachers in the comparison condition. Qualitative analyses of coach interviews identified substantive differences in the professional development support available to coaches, scope of coaches’ job responsibilities, and focus of coaching resources in the CFC schools and comparison schools.

The Quality of Writing Tasks and Students’ Use of Academic Language in Spanish

This study investigates the quality of the writing tasks assigned to native Spanish speakers in bilingual (Spanish-English) contexts, and the relationship between task quality and students’ use of an academic register in their native language. Fifty-six language arts tasks were collected from 26 grade 4 and 5 teachers, and four student writing samples were collected in response to each task (N = 224). Multilevel modeling revealed that variation in students’ use of key features of academic language in their writing was associated with the cognitive demand of writing tasks. Findings suggest that students’ opportunities to respond to challenging tasks when writing in their native language are rare and that the rigor of writing tasks may relate to students’ production and development of academic language.

Automatic Scoring of an Analytical Response-To-Text Assessment

In analytical writing in response to text, students read a complex text and adopt an analytic stance in their writing about it. To evaluate this type of writing at scale, an automated approach for Response to Text Assessment RTA is needed. With the long-term goal of producing informative feedback for students and teachers, we design a new set of interpretable features that operationalize the Evidence rubric of RTA. When evaluated on a corpus of essays written by students in grades 4-6, our results show that our features outperform baselines based on well-performing features from other types of essay assessments.

Assessing Students’ Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring

This paper presents an investigation of score prediction based on natural language processing for two targeted constructs within analytic text-based writing: 1) students’ effective use of evidence and, 2) their organization of ideas and evidence in support of their claim. With the long-term goal of producing feedback for students and teachers, we designed a task-dependent model, for each dimension, that aligns with the scoring rubric and makes use of the source material. We believe the model will be meaningful and easy to interpret given the writing task. We used two datasets of essays written by students in grades 5–6 and 6–8. Our experimental results show that our task-dependent model (consistent with the rubric) performs as well as if not outperforms competitive baselines. We also show the potential generalizability of the rubric-based model by performing cross-corpus experiments. Finally, we show that the predictive utility of different feature groups in our rubric-based modeling approach is related to how much each feature group covers a rubric’s criteria.

Student Writing Accepted as High-Quality Responses to Analytic Text-Based Writing Tasks

Literacy standards increasingly emphasize the importance of analytic text-based writing. Little consensus exists, however, around what high-quality student responses should look like in this genre. In this study, we investigated fifth-grade students’ writing in response to analytic text-based writing tasks (15 teachers, 44 writing tasks, 88 pieces of student writing). We used qualitative analytic techniques to investigate teachers’ stated assessment criteria and to compare teachers’ and researchers’ assessments of student writing nominated by teachers as high quality. We then characterized patterns in students’ responses. Our results showed no variation in teachers’ stated assessment criteria but substantial variation in students’ writing with respect to 3 researcher-defined elements: cognitive process, use of text evidence, and explication. Only 22% of student responses nominated by teachers as high quality were close to fully realizing the task potential.