Leigh Burstein

Analyzing Multilevel Data in the Presence of Heterogeneous Within-Class Regressions

The concerns of this investigation are multiple sources of complications in the analysis of multilevel educational data. The focus is on problems that arise when within-group regressions of outcome on input are related to teacher/ school characteristics. Single-level and multilevel analytical approaches are applied to hypothetical data for which the relationship between teacher/class quality and the heterogeneity of within-class slopes is varied systematically. It is shown that single-level analyses and the proposed multilevel approaches can all yield mislèading estimates of teacher/class effects on mean class outcomes. However, selected multilevel methods provide some indication of misspecification and can identify the direction of the bias in estimating teacher/class effects on mean class outcomes.

Assessing Differences between Grouped and Individual-Level Regression Coefficients: Alternative Approaches.

Several techniques for assessing differences between least-squares estimators of regression coefficients from group and individual-level data are summarized. The structural equations approach (Hannan and Burstein, 1974) and the X-rule approach (Firebaugh, 1978) generate estimates of predicted bias when individual-level data are available, and it is important to determine the consequences ofgrouping according to a known procedure. While the X-rule provides slightly more accurate forecasts, both approaches identify the same methods of grouping as resulting in either large or small bias. An approach based on the statistical significance of differences between estimators at two levels (Feige and Watts, 1972) is also described along with an alternative test based on a different interpretation of the standard test for significant differences among regression models. The empirical results indicate that both the Feige- Watts test and the proposed alternative significance test behave well for cases with small differences between group and individual-level estimates. However, the Feige- Watts test was less satisfactory when group and individual-level estimates diverged greatly. The utility and suitability of all four approaches, when (a) data are grouped by a nominal characteristic, (b) there are multiple regressors, or (c) individual-level data cannot be reconstructed, are discussed.

MULTILEVEL INVESTIGATIONS OF SYSTEMATICALLY VARYING SLOPES: ISSUES, ALTERNATIVES, AND CONSEQUENCES

Publisher Summary This chapter discusses the issues, alternative, and consequences of mul;tilevel investigations of systematically varying slopes. Generally, the situations of interest are large-scale non-experimental studies of educational effects. The focus may be on either the program—for example, curricular innovation, school type—the school, the classroom/teacher, or some combination, as the sources of educational effects. Regardless, the intent is to identify the antecedents of student performance and attitude and estimate the magnitude of their effects. The typical study involves two-stage sampling. Either a sample, which can be random, representative, or convenience, of schools is drawn and students are sampled randomly or on a stratified random basis from within schools. Alternatively, a sample of classrooms is chosen, either randomly or exhaustively within a sample of schools, and all the students within the classrooms comprise the total study sample. In either case, this situation yields data with dependencies among observations within the first-stage sampling units; that is, there are correlations among individuals in the same macro units.

Collecting evaluation data : problems and solutions

Perspectives on Data Collection in Evaluations - Leigh Burstein and Howard E Freeman PART ONE: DATA SENSITIVITIES Data Collection Strategies in the Minneapolis Domestic Assault Experiment - Richard A Berk and Lawrence W Sherman Observer Studies - Lee Sechrest Data Collection by Remote Control Data Collection Issues in the Evaluation of the Effect of Television Violence on Elementary School Children - Ronald C Kessler et al PART TWO: MOUNTING LARGE-SCALE FIELD STUDIES A Tale of Two Surveys - Ronald M Anderson, Lu Ann Aday, and Gretchen Voorhis Fleming Lessons from the Best and Worst of Times in Program Evaluation Field Sampling Problems in Data Collection for Evaluating Research - Sandra H Berry Measuring Unfiled Claims in the Health Insurance Experiment - William H Rogers and Joseph P Newhouse Issues of Data Collection in Assessing Programs Involving Crime Reduction - Charles Mallar and Irving Piliavin The Job Corps and Supported Work Evaluations Some Failures in Designing Data Collection That Distort Results - Albert J Reiss Jr PART THREE: USE OF RECORD INFORMATION Program Evaluation and the Use of Extant Data - Eleanor Chelimsky Identification of Treatment Conditions Using Standard Record-Keeping Systems - J Ward Keesling Using Longitudinal Earnings Data from Social Security Records to Evaluate Job-Training Programs - Howard S Bloom An Information System for Planning and Evaluating Geriatric Care - George L Maddox The Duke Older Americans Resources and Services Program From Science to Technology - William D Neigher and Daniel B Fishman Reducing Problems in Mental Health Evaluation by Paradigm Shift REPRISE Data Quality Issues in Evaluation Research - Peter H Rossi Summary Comments

Methodological Considerations in Interpreting Research on Self-Concept

The purposes of this paper are twofold: (1) to provide a brief review of methods used in validating construct interpretat ions of self-concept measurements and show where the papers in this special issue of Journal o f Youth andAdolescence fall within this domain; and (2) to point out what we consider to be the strengths and l imitations o f each paper. As a point o f departure, a couple of general comments about this issue of the journal are in order. In the literature on self-concept (esteem, image, perception view) 4 only scant at tention has been paid to methodological problems (see Crowne and Stephens, 1961; Gordon, 1969; Shavelson, et al. 1976; Wylie, 1961, 1974). An issue of a journal devoted to this topic, then, certainly is a welcome sight. Fur thermore, studies of self-concept have focused primarily on young children, schoolchildren without special attention to adolescents, or children in clinical settings. The recognition of the

The Use of Existing Data Bases in Program Evaluation and School Improvement.

The purpose of this article is to comment on practice in the use of existing data bases in program evaluation and school improvement and to explore directions of increased and improved use. The reports impetus is the role that existing data play in current efforts in local school districts and the possibilities for the future. To a great extent, however, we will have to rely on experiences from other areas of social inquiry outside of education and on national and state, rather than local, practices in education. To date, the record of local district practice in maintaining and using data archives is limited, whereas extended, multipurpose secondary data examination is commonplace elsewhere. Nonetheless, we assume that the current state of affairs in information maintenance and use in local districts is

Data Collection The Achilles Heel of Evaluation Research

Recent advances in design, measurement, and analysis can have only a marginal impact on the integrity of evaluation studies because the evaluation of social programs is fundamentally dependent on the quality of the collected data. The effects of data collection procedures and their consequences on the integrity of evaluation conclusions are explicated. Data collection faults occurring in evaluation studies are enumerated and illustrated. A research agenda is proposed for improving data collection in social program evaluations.

Concomitants of Growth in Mathematics Achievement During the Population A School Year

Publisher Summary This chapter describes concomitants of growth in mathematics achievement during the population a school year. The concepts employed in the analyses which serve as the focus of this chapter are defined at different levels within the school system. Some characterize the individual student, and hence are available for all students within each school. Others, such as those characterizing teachers and classroom instruction are available only for classrooms, and hence are the same for all students within a classroom. Still others, such as school characteristics, are defined at a level which makes them the same for all classrooms within a school. The fact that the concepts are defined at different levels within the schooling system makes the conceptual and analytical work more difficult. The major outcomes in the study are facets of mathematics achievement: arithmetic, algebra, and geometry. The achievement tests administered to all students, and from which the three subtests were derived, employed a core examination, and one of four rotated forms.

Concluding Thoughts: What We Know, What It Means

Publisher Summary This chapter examines the mathematics test results. The amount of information in Second International Mathematics Study (SIMS) about cognitive achievement is the most substantial ever collected in a comparative study. It is found that not only is the item pool larger, but also the tests were administered on two occasions, and numerous sorts of contexts are available to help interpret them. The descriptions can be about levels of achievement and amounts of learning during a year, both placed in a context of the curriculum and what teachers say was taught, how it was taught, and the structural characteristics of the system. The performance of the Japanese on the cognitive test is, of course, exemplary. Two crucial features of that system, facets in which the system is more comprehensive than are the others, may explain the performance. First, given the SIMS test, the Japanese curriculum includes the most content of the eight systems. Secondly, the Japanese curriculum is comprehensive is that it is experienced by virtually all students.

Book Reviews: Ward, Joe H., Jr. & Jennings, Earl.Introduction to linear models. Englewood Cliffs, N.J.: Prentice-Hall, 1973. 333 + xvii pp.,

AIKEN, L. R. Ability and creativity in mathematics. Review of Educational Research, 1973,43, 405-442. KRUTETSKII, V. A. Three papers in J. Kilpatrick & I. Wirszup (Eds.). Soviet studies in the psychology of learning and teaching mathematics. Vol. 2. The structure of mathematical abilities. Stanford, Calif.: School Mathematics Study Group, 1969, pp. 5-111. RIMOLDI, H. J. A., AGHI, M., & BURDER, G. Some effects of logical structure, language, and age in problem solving in children. Journal of Genetic Psychology, 1968, 112, 127-143. STANLEY, J. C, KEATING, D.P., & Fox, L. H. (Eds.). Mathematical talent: Discovery, description, and development. Baltimore: The Johns Hopkins University Press, 1974.