Kenneth A. Sirotnik

An Analysis of Variance Framework for Matrix Sampling

This paper, a discussion of the methodology of matrix sampling, and the empirical and theoretical research on matrix sampling, attempts to demonstrate the f llowing points: 1. Matrix sampling can be viewed as a simple two factor, random model analysis of variance design, the matrix sampling formulas for estimating the mean and variance being simply the point estimate formulas for estimating components of the underlying linear model. 2. These formulas can be based on the weakest possible set of assumptions, viz., random and independent sampling of examinees and items. No assumptions about the statistical nature of the data need be made. 3. The literature is unclear about what effect the above sampling assumptions have upon matrix sampling in the estimation of the mean and, especially, the variance. 4. Of the three alternative procedures suggested for dealing with negative variance estimates in multiple matrix sampling--equating the negative estimate to zero, Winsorizing the distribution of estimates, or treating all estimates alike regardless of sign--the third procedure appears to be most promising. A simulation study is necessary to determine the shape of the distribution of variance component estimates for matrix sampling as well as the relative efficiency of the three methods for handling negative estimates.

The Contextual Correlates of the Relative Expenditures of Classroom Time on Instruction and Behavior: An Exploratory Study of Secondary Schools and Classes:

Multivariate relationships are described between the proportions of classroom time observed in instructional and behavioral modalities and six potentially important contextual variable subsets. Canonical regressions and correlations were computed between each contextual variable subset and the proportions of observed time measures both between schools and within schools (at the classroom level). The magnitudes of resultant relationships were not great. Comparatively speaking, however, important differences emerged between (a) analyses at school and classroom levels, (b) junior and senior high school classes, (c) contextual variable subsets, and (d) the sizes of associations with instructional and behavioral time. It is recommended that future “quantity of schooling” studies include time spent on discipline and control (as well as on task) and be both substantively and analytically sensitive to both school and classroom effects.

Data Collection The Achilles Heel of Evaluation Research

Recent advances in design, measurement, and analysis can have only a marginal impact on the integrity of evaluation studies because the evaluation of social programs is fundamentally dependent on the quality of the collected data. The effects of data collection procedures and their consequences on the integrity of evaluation conclusions are explicated. Data collection faults occurring in evaluation studies are enumerated and illustrated. A research agenda is proposed for improving data collection in social program evaluations.

The Validity of Administrator Performance Assessment Systems: The ADI As a Case-in-Point.

A number of administrator performance assessment centers and systems are in use or are being advocated for use for diagnosis, certification, and selection. However, the construct and predictive validity of such systems are still open to serious questions. The present study investigated both types of validity for one such system, the Administrator Diagnostic Inventory (ADI). Based on the results of this study and the review of research on performance assessment centers more generally, it is concluded that caution and a lot more research accompany the use of performance assessment systems, particularly for purposes of certification and selection.