Keith Zvoch

Teacher–Student Relationships Among Behaviorally At-Risk African American Youth From Low-Income Backgrounds: Student Perceptions, Teacher Perceptions, and Socioemotional Adjustment Correlates

This investigation examines teacher—student relationships among African American youth from low-income backgrounds (N = 193). Students and their teachers completed measures of teacher—student relationship quality and measures pertaining to emotional, behavioral, and school-related adjustment. Results indicated that African American youth who fell above the clinical cut point on the externalizing scale of the Child Behavior Checklist (n = 64) reported lower trust in relationships with teachers than did similarly matched students who did not have clinically significant externalizing symptomology. In addition, teachers rated students in the externalizing subgroup as lower in relational closeness and greater in relational conflict. Multiple regression analyses applied to data obtained from the behavioral risk group indicated that both student and teacher perceptions of teacher—student relationship quality were associated with student- and teacher-rated emotional, behavioral, and school-related adjustment. The strength of these associations varied as a function of data source and the specific relationship dimension investigated. Implications for future research and practice efforts aimed at building positive teacher—student relationships are discussed.

Treatment Fidelity in Multisite Evaluation: A Multilevel Longitudinal Examination of Provider Adherence Status and Change.

Program implementation data obtained from the repeated observation of teachers delivering one of two early childhood literacy programs to economically disadvantaged students in a large southwestern school district were analyzed to estimate protocol adherence levels at the onset of the intervention as well as the change in adherence over the intervention period. Application of multilevel growth models to the classroom observation data revealed that fidelity to program protocol varied within and between treatment sites during the initial observation and over time. An exploratory examination of select teacher, classroom, and site characteristics indicated that the background characteristics of teachers and contextual factors in the treatment environment were associated with the fidelity outcomes. These results provide some insight into the range of factors that are associated with protocol adherence and highlight the challenge of achieving and maintaining fidelity to a treatment intervention that is delivered by multiple providers over multiple treatment sites.

How Does Fidelity of Implementation Matter? Using Multilevel Models to Detect Relationships between Participant Outcomes and the Delivery and Receipt of Treatment.

Multilevel modeling techniques facilitated examination of relationships between fidelity indicators and outcomes associated with a summer literacy intervention. Three-level growth models were specified to capture the extent to which students experienced instruction and to demonstrate the ways in which dosage–response relationships manifest in program evaluation contexts. The observation that outcome-related deviations from program protocol occurred both at the provider and at the recipient levels suggests that evaluators will often need to conceptualize, measure, and model “treatment fidelity” as a multilevel, multidimensional construct.

The Inventory of Teacher-Student Relationships: Factor Structure, Reliability, and Validity Among African American Youth in Low-Income Urban Schools

This study investigates the factor structure, reliability, and validity of the Inventory of Teacher-Student Relationships (IT-SR), a measure that was developed by adapting the widely used Inventory of Parent and Peer Attachments (Armsden & Greenberg, 1987) for use in the context of teacher-student relationships. The instrument was field tested with a sample of African American students from low-income backgrounds (N = 171). An exploratory factor analysis was estimated with a randomly selected half of the sample. Three factors pertaining to Communication, Trust, and Alienation in relationships emerged. A confirmatory factor analysis was run on the remainder of the sample. Goodness-of-fit indices indicated that the three factor structure fit the data reasonably well. Scores on each of the three factors correlated with scores on other, existing measures of teacher-student relationship quality as well as with indicators of emotional, behavioral, and school-related adjustment. Implications for research on teacher-student relationships are discussed.

A Multilevel Multisite Outcomes-by-Implementation Evaluation of an Early Childhood Literacy Model.

The evaluation of an intervention delivered across multiple treatment sites presents a unique opportunity for evaluators to gauge the manner and degree to which the “impact” of treatment varies across implementation conditions and different target populations. However, the availability of implementation data for each treatment site, while presenting the opportunity for more sophisticated impact assessment, also presents an analytic challenge. In the following, multilevel growth models were applied to data describing students and classroom sites to demonstrate how the multilevel modeling framework can be used to analyze data obtained from a multisite program implementation. Results of the investigation indicated that increased adherence to the program model was not associated with more positive recipient outcomes. Further examination of the null finding indicated that the highest and lowest rates of literacy growth observed in the study were concentrated in several low-implementing sites. Implications for multisite evaluation design and practice are discussed.

Longitudinal Effects of School Context and Practice on Middle School Mathematics Achievement.

ABSTRACT The authors analyzed mathematics achievement data from a longitudinally matched student cohort from a large southwestern U.S. school district to investigate school context and practice effects on the academic performance and growth of middle school students. Investigation of the degree to which aspects of the school environment related to mathematics achievement outcomes revealed 2 distinct patterns. School context, as measured by student and school demographic characteristics, related closely to mathematics performance levels but had little relationship with mathematics growth rates. The opposite was true for aspects of school practice. Teacher educational attainment and the mathematics curricula delivered to students were not related to student performance levels but were moderately associated with mathematics growth rates. These results suggest that the effect of some policy-relevant school variables may be difficult to identify when student achievement is studied at a single point in time. However, investigation of school impacts on student achievement may be facilitated when an analytic strategy that takes into account the time-dependent and cumulative nature of schooling is adopted.

Measuring and Evaluating School Performance: An Investigation of Status and Growth-Based Achievement Indicators

Several school performance indices were evaluated using achievement data from multiple longitudinally matched middle school student cohorts in a large district in the southwestern United States. Measures designed to index the within-cohort achievement status and growth of students as well as the status and growth-based changes that occur over years between different student cohorts were studied. Within- and between-cohort status and growth-based estimates of school performance were generated by applying simple and relatively complex statistical models to the longitudinal time-series data obtained on students. Results indicated that within-cohort status-based estimates were closely related to student demographics and between-cohort estimates were associated with cohort enrollment size and initial performance level. These results suggest that schools may often be rewarded and penalized on the basis of factors over which school personnel have limited or no control. Implications for the measurement and evaluation of school performance are discussed.

Summer School and Summer Learning: An Examination of the Short- and Longer Term Changes in Student Literacy

Research Findings: Multiple student cohorts were longitudinally tracked and student participation in a summer program bridging the 1st- and 2nd-grade academic years was recorded to examine selection and efficacy issues related to a summer school implementation in the Pacific Northwest. The estimation of regression discontinuity models uncovered evidence of a local average treatment effect. At the cutscore for program admission, participating students had estimated summer oral reading fluency gains approximately 0.40 SD larger than those of nonparticipants. Further examination of the literacy outcomes among the sample of cutscore eligible students revealed that struggling readers who participated in the summer program increased their level of reading fluency relative to struggling readers who declined an invitation to participate. However, the advantage gained by cutscore eligible participants was not sustained over the subsequent academic year. Practice or Policy: These results suggest that supplemental summer instruction delivered to at-risk students may promote literacy gains during the otherwise challenging summer months and thereby serve as a useful intermediary tool for K–12 stakeholders seeking to keep struggling readers on track toward proficiency.

Sample Exclusion and Student Attrition Effects in the Longitudinal Study of Middle School Mathematics Performance

Achievement data from a longitudinally matched student cohort from a large school district in the southwestern United States were analyzed to investigate sample exclusion and student attrition effects on estimates of student, school, and district mathematics performance. Use of 2- and 3-level longitudinal growth models to estimate the growth trajectories of middle school students revealed that mathematics performance differed across 2 sample conditions. Relative to the achievement outcomes associated with a sample that included all students from the longitudinal cohort, district and school achievement were generally higher and student group performance more similar in the smaller, more advantaged student sample used for district accountability reporting. Further investigation of the school performance estimates showed that cross-sample changes in student achievement outcomes were closely related to the proportion of students from special student populations who were excluded from the district accountability sample. The achievement differences and the differential patterns of association demonstrated in this study suggest that conclusions drawn about district and school performance and relationships between student characteristics and student achievement outcomes may depend to some degree on which students are included in an analytic sample. Investigators seeking to take advantage of longitudinal designs in school effectiveness research are cautioned to closely examine their data for nonrandom student attrition and document the impact of sample exclusion and student attrition effects in the research and accountability reports that are produced from longitudinal data sets.

A Longitudinal Examination of the Academic Year and Summer Learning Rates of Full- and Half-Day Kindergartners

Literacy data collected over the course of 2 academic years were used to estimate the rate at which full- and half-day kindergartners acquired literacy skills during kindergarten, 1st grade, and the intervening summer. Application of piecewise growth models to the time series data obtained on a sample of students from a large Southwestern school district revealed that economically disadvantaged full-day kindergartners gained literacy skills at a relatively faster rate than their more economically advantaged half-day peers during the kindergarten year. The accelerated gain experienced by full-day kindergartners was sufficient to enable a reversal of the initial performance deficit observed upon entry to kindergarten. However, over the summer between kindergarten and 1st grade, a different pattern of development was observed. Full-day alumni experienced a literacy fallback, whereas their half-day peers maintained the literacy gains acquired during kindergarten, leading to a second performance reversal by the beginning of the 1st grade. The gap in performance (half-day advantage) then remained constant as the literacy growth of full- and half-day alumni was equivalent over the 1st-grade school year. Implications for evaluating the efficacy of school-based initiatives like full-day kindergarten and more generally the effectiveness of schools and schooling are discussed.