Michael Seltzer

The Metric Matters: The Sensitivity of Conclusions About Growth in Student Achievement to Choice of Metric

A central means of assessing the health of educational systems involves using longitudinal data to examine patterns of academic growth across a series of grades (e.g., the extent to which the range of achievement among students widens or narrows over time and whether students tend to “slump” in particular grades). The purpose of this article is to illustrate, through a series of analyses based on a longitudinal study of reading achievement in the Chicago Public Schools, that the conclusions one draws about patterns of academic growth and, in turn, the decisions one makes regarding the kinds of interventions that may be needed can be extremely sensitive to the metric used (i.e., whether achievement is measured using grade equivalents or based on item response theory metrics).

Bayesian Analysis in Applications of Hierarchical Models: Issues and Methods

In applications of hierarchical models (HMs), a potential weakness of empirical Bayes estimation approaches is that they do not to take into account uncertainty in the estimation of the variance components (see, e.g., Dempster, 1987). One possible solution entails employing a fully Bayesian approach, which involves specifying a prior probability distribution for the variance components and then integrating over the variance components as well as other unknowns in the HM to obtain a marginal posterior distribution of interest (see, e.g., Draper, 1995; Rubin, 1981). Though the required integrations are often exceedingly complex, Markov-chain Monte Carlo techniques (e.g., the Gibbs sampler) provide a viable means of obtaining marginal posteriors of interest in many complex settings. In this article, we fully generalize the Gibbs sampling algorithms presented in Seltzer (1993) to a broad range of settings in which vectors of random regression parameters in the HM (e.g., school means and slopes) are assumed multivariate normally or multivariate t distributed across groups. Through analyses of the data from an innovative mathematics curriculum, we examine when and why it becomes important to employ a fully Bayesian approach and discuss the need to study the sensitivity of results to alternative prior distributional assumptions for the variance components and for the random regression parameters.

Examining Relationships Between Where Students Start and how Rapidly they Progress: Using New Developments in Growth Modeling to Gain Insight into the Distribution of Achievement Within Schools

Studying change in student achievement is of central importance in numerous areas of educational research, including efforts to monitor school performance, investigations of the effects of educational interventions over time, and school effects studies focusing on how differences in school policies and practices relate to differences in student progress. In this article, we argue that in studying patterns of change, it is often important to consider the relationship between where students start (i.e., their initial status) and how rapidly they progress (i.e., their rates of change). Drawing on recent advances in growth modeling methodology, we illustrate the potential value of such an approach in the context of monitoring school performance. In particular, we highlight the ways in which attending to initial status in analyses of student progress can help draw attention to possible concerns regarding the distribution of achievement within schools. To convey the logic of our approach and illustrate various analysis possibilities, we fit a series of growth models to the time series data for students in several schools in the Longitudinal Study of American Youth (LSAY) sample. In a final section, we discuss some of the possibilities that arise in employing a modeling approach of this kind in evaluating educational programs and in conducting school effects research.

Sensitivity analysis for fixed effects in the hierarchical model : A Gibbs sampling approach

Many recent applications of the two-level hierarchical model (HM) have focused on drawing inferences concerning fixed effects—that is, structural parameters in the Level 2 model that capture the way Level 1 parameters (e.g., children’s rates of cognitive growth, within-school regression coefficients) vary as a function of Level 2 characteristics (e.g., children’s home environments and educational experiences; school policies, practices, and compositional characteristics). Under standard assumptions of normality in the HM, point estimates and intervals for fixed effects may be sensitive to outlying Level 2 units (e.g., a child whose rate of cognitive growth is unusually slow or rapid, a school at which students achieve at an unusually high level given their background characteristics, etc.). A Bayesian approach to studying the sensitivity of inferences to possible outliers involves recalculating the marginal posterior distributions of parameters of interest under assumptions of heavy tails, which has the effect of downweighting extreme cases. The goal is to study the extent to which posterior means and intervals change as the degree of heavy-tailedness assumed increases. This strategy is implemented in the HM setting via a new Monte Carlo technique termed Gibbs sampling (Gelfand & Smith, 1990; cf Tanner & Wong, 1987) and is illustrated through reanalyses of the data from a study of vocabulary growth in children (Huttenlocher, Haight, Bryk, Seltzer, & Lyons, 1991). The Gibbs sampling approach presented builds on the work of Dempster, Laird, and Rubin (1980); Little and Rubin (1987); and Lange, Little, and Taylor (1989) concerning the use of the t distribution in robust statistical estimation. Extensions of this approach are discussed in the final section of the article.

Studying Variation in Program Success A Multilevel Modeling Approach

Multilevel modeling techniques, although used extensively in numerous areas of social science research including demography, studies of school organization, and research on cognitive development, have been used surprisingly infrequently in multisite evaluation studies. The goal of this article is to illustrate several ways in which multilevel modeling techniques can help to broaden the kinds of questions that we are able to address in multisite evaluations. In particular, it is shown how these techniques provide a means of moving beyond estimating overall, average program effects to investigations of how differences in various aspects of implementation across sites relate to differences in program success.

Furthering Our Understanding of the Effects of Educational Programs via a Slopes-as-Outcomes Framework

The designs that are commonly employed in evaluation studies give rise to samples of data that have a multilevel structure—for example, students are nested within different classrooms or schools, which, in turn, are assigned to different program types. In this article, I show how a multilevel analysis strategy developed by Burstein termed slopes-as-outcomes provides a means of addressing questions connected with equity in evaluation research (e.g., Do students in a particular program appear to attain high levels of achievement irrespective of their initial levels of achievement?). Furthermore, I discuss how this strategy creates possibilities for studying how differences in program implementation relate to differences in program effectiveness across sites, thereby helping to illuminate those factors connected with program success.

Symptom improvement and its temporal course in short-term dynamic psychotherapy - a growth curve analysis

Using hierarchical linear model procedures, growth curve analyses were performed to examine the course, rate, and correlates of symptom improvement during short-term anxiety-provoking psychotherapy (STAPP) and a 2-year posttermination period. The Symptom Checklist-90 was used to measure general symptomatology. The sample consisted of 15 patients who were found suitable for STAPP. Most had a diagnosis of anxiety. Therapists were in postgraduate manual-guided STAPP training. Results showed that three of four patients made a reliable and clinically significant symptom improvement over the course of treatment. Patients improved at a steady rate during treatment as well as after treatment. Average improvement was large and significant during treatment, while small and marginally significant after treatment. Improvement rates varied significantly over the course of treatment and were faster for patients less rigid in their personality functioning. —J Nerv Ment Dis 183:242-248, 1995

Modeling Heterogeneity in Relationships Between Initial Status and Rates of Change: Treating Latent Variable Regression Coefficients as Random Coefficients in a Three-Level Hierarchical Model

In studies of change in education and numerous other fields, interest often centers on how differences in the status of individuals at the start of a period of substantive interest relate to differences in subsequent change. In this article, the authors present a fully Bayesian approach to estimating three-level Hierarchical Models in which latent variable regression (LVR) coefficients capturing the relationship between initial status and rates of change within each of J schools (Bw j , j = 1, …, J) are treated as varying across schools. Specifically, the authors treat within-group LVR coefficients as random coefficients in three-level models. Through analyses of data from the Longitudinal Study of American Youth, the authors show how modeling differences in Bwj as a function of school characteristics can broaden the kinds of questions they can address in school effects research. They also illustrate the possibility of conducting sensitivity analyses using t distributional assumptions at each level of such models (termed latent variable regression in a three-level hierarchical model [LVR-HM3s]), and present results from a small-scale simulation study that help provide some guidance concerning the specification of priors for variance components in LVR-HM3s. They outline extensions of LVR-HM3s to settings in which growth is nonlinear, and discuss the use of LVR-HM3s in other types of research including multisite evaluation studies in which time-series data are collected during a preintervention period, and cross-sectional studies in which within-cluster LVR slopes are treated as varying across clusters.

Self-Concept Improvement During and After Short-Term Anxiety-Provoking Psychotherapy: A Preliminary Growth Curve Study

Using hierarchical linear model procedures (Bryk & Raudenbush, 1987, 1992) growth curve analyses were performed to examine the course, rate, and correlates of self-concept development during 20 sessions long short-term anxiety-provoking psychotherapy (STAPP; Sifneos, 1992) and a two-year posttermination period. The control coefficient from the SASB Introject Questionnaire (Benjamin, 1984) were used to capture aspects of client self-concept. The sample consisted of 13 clients suitable for STAPP and with mainly anxiety diagnoses. Therapists were in postgraduate manual-guided STAPP training. Results showed that self-concept improvement during as well as after treatment occurred at a steady yet significantly variable rate among clients. Clients showed significant growth toward self-freeing after termination of therapy. This growth was faster for clients with greater initial symptom improvement. This finding is discussed in relation to Howards three-phase model of psychotherapy outcome.

Sensitivity Analysis for Hierarchical Models Employing t Level-1 Assumptions

Much work on sensitivity analysis for hierarchical models (HMs) has focused on level-2 outliers (e.g., in multisite evaluations, a site at which an intervention was unusually successful). However, efforts to draw sound conclusions concerning parameters of interest in HMs also require that we attend to extreme level-1 units (e.g., a person in the treatment group at a particular site whose post-test score [yij ] is unusually small vis-á-vis the other members of that person’s group). One goal of this article is to examine the ways in which level-1 outliers can impact the estimation of fixed effects and random effects in HMs. A second goal is to outline and illustrate the use of Markov Chain Monte Carlo algorithms for conducting sensitivity analyses under t level-1 assumptions, including algorithms for settings in which the degrees of freedom at level 1 (v1 ) is treated as an unknown parameter.