Lorrie A. Shepard

Psychometricians’ Beliefs About Learning:

The author contends that disputes within the measurement community about what constitutes legitimate test preparation and whether “teaching to the test” is good or bad for student learning can be explained by differences in measurement specialists beliefs about learning. Qualitative analysis of interview data from a nationally representative sample of 50 district testing directors revealed that approximately half of the measurement specialists operate from implicit learning theories that advocate, first, close alignment of tests with curriculum and, second, judicious teaching of tested content. Historical quotations are used to show that these beliefs, associated with criterion-referenced testing, derive from behaviorist learning theory, which requires sequential mastery of constituent skills and explicit testing of each learning step. The sequential, facts-before-thinking model of learning is contradicted, however, by a substantial body of evidence from cognitive psychology. Implicit beliefs should be made explicit because an understanding of learning theory assumptions is fundamental to evaluating evidence of testing effects and therfore to framing validity investigations.

Kindergarten Readiness and Retention: A Qualitative Study of Teachers’ Beliefs and Practices

The issues concerning teachers’ beliefs about and use of retention were explored in a qualitative study. Clinical interviews with teachers, participant observation in kindergarten classes, analysis of documents, and interviews with parents revealed that teachers’ beliefs about the development of school readiness could be described and ordered along a dimension of nativism, that these beliefs relate to their use of retention as a solution to unreadiness or incompetence, and that elements of the organization of the schools in which they teach may also account for beliefs and practices. Teachers’ endorsement of retention diverges both from extant propositional knowledge and from the perceptions of other interested groups.

Escalating Academic Demand in Kindergarten: Counterproductive Policies

Academic demands in kindergarten and first grade are considerably higher today than 20 years ago and continue to escalate. Downward shifts of what were next-grade expectations into the earliest grades are the result of large-scale social trends, for example, the universality of kindergartens, as well as day-to-day pressures felt by teachers, from accountability gates and demands for acceleration from middle-class parents. Narrow emphasis on isolated reading and numeracy skills is detrimental even to the children who succeed and is especially harmful to children labeled as failures. Policies such as raising the entrance age, readiness screening; and kindergarten retention are intended to solve the problem of inappropriate academic demand by removing younger or unready children. Research evidence does not support the efficacy of these policies. Rather, these practices contribute to the continued escalation of curriculum as teachers adjust their teaching to an older and more able group.

Accounting for Statistical Artifacts in Item Bias Research.

Theoretically preferred IRT bias detection procedures were applied to both a mathematics achievement and vocabulary test. The data were from black and white seniors on the High School and Beyond data files. To account for statistical artifacts, each analysis was repeated on randomly equivalent samples of blacks and whites (n’s = 1,500). Furthermore, to establish a baseline for judging bias indices that might be attributable only to sampling fluctuations, bias analyses were conducted comparing randomly selected groups of whites. To assess the effect of mean group differences on the appearance of bias, pseudo-ethnic groups were created, that is, samples of whites were selected to simulate the average black-white difference. The validity and sensitivity of the IRT bias indices was supported by several findings. A relatively large number of items (10 of 29) on the math test were found to be consistently biased; they were replicated in parallel analyses. The bias indices were substantially smaller in white-white analyses. Furthermore, the indices (with the possible exception of χ2) did not find bias in the pseudo-ethnic comparison. The pattern of between-study correlations showed high consistency for parallel ethnic analyses where bias was plausibly present. Also, the indices met the discriminant validity test—the correlations were low between conditions where bias should not be present. For the math test, where a substantial number of items appeared biased, the results were interpretable. Verbal math problems were systematically more difficult for blacks. Overall, the sums-of-squares statistics (weighted by the inverse of the variance errors) were judged to be the best indices for quantifying ICC differences between groups. Not only were these statistics the most consistent in detecting bias in the ethnic comparisons, but they also intercorrelated the least in situations of no bias.

Comparison of Procedures for Detecting test-Item Bias with both Internal and External Ability Criteria

Test bias is conceptualized as differential validity. Statistical techniques for detecting biased items work by identifying items that may be measuring different things for different groups; they identify deviant or anomalous items in the context of other items. The conceptual basis and technical soundness were reviewed for the following item bias methods: transformed item difficulties, item discriminations, one- and three-parameter item characteristic curve methods, and chi-square methods. Sixteen bias indices representing these approaches were computed for black-white and Chicano-white comparisons on both the verbal and nonverbal Lorge-Thorndike Intelligence Tests. In addition, bias indices were recomputed for the Lorge-Thorndike tests using an external criterion. Convergent validity among bias methods was examined in correlation matrices, by factor analysis of the method correlations, and by ratios of agreements in the items found to be “most biased” by each method. Although evidence of convergent validity was found, there will still be important practical differences in the items identified as biased by different methods. The signed full chi-square procedure may be an acceptable substitute for the theoretically preferred but more costly three-parameter signed indices. The external criterion results also reflect on the validity of the methods; arguments were advanced, however, as to why internal bias methods should not be thought of as proxies for a predictive validity model of unbiasedness.

Standard setting issues and methods.

Previous methodological reviews and the contro versy regarding the adequacy of standard-setting technology are summarized. The judgmental nature of all standard-setting methods is examined, and the debate about whether fallible standards are better than none is recast in the context of three different test uses: pupil diagnosis, pupil certification (for high school graduation or professional licensure), and program evaluation. Exemplary standard-set ting methods are reviewed, representing the follow ing major approaches: (1) judgments of test con tent ; (2) judgments about mastery-nonmastery groups; (3) norms and passing rates; (4) empirical methods for discovering standards; and (5) empiri cal methods for adjusting cutoff scores, given a standard on an external criterion measure. Stan dards based on the performance of judged mastery groups (the Contrasting Groups method) and cer tain uses of normative data are likened to Known Groups validation. Recommendations are made for selecting standard-setting techniques depending on test use, including pupil diagnosis, pupil certifica tion, and program evaluation. Future research on standard setting is discussed in the context of im proving practical aspects of judgmental methods.

Characteristics of Pupils Identified as Learning Disabled

This study was aimed at describing the characteristics of school-aged children whom educators had identified as learning disabled (LD). A probability sample of 800 was selected from the population of all children served as learning disabled in the state of Colorado. A coding form was used by trained coders to extract relevant features from the case files of the children. The sample was characterized by (1) distributions of single variables (e.g., below grade level achievement, discrepancy between IQ and achievement, medical indicators), and (2) hierarchical classification creating clusters or subgroups within the LD sample. Fewer than half the sample exhibited characteristics consistent with definitions of LD in federal regulations and professional literature. Included in this group were subgroups of hyperactive, brain-injured children, children with significant discrepancies between IQ and achievement and those with signs of perceptual processing disorders. Slightly more than half the sample did not match conventional definitions of LD but exhibited learning problems such as language interference, emotional disorders, or mild retardation. The inclusion of the latter groups among the learning disabled is a particular problem in the validation of the construct and will confound research on prevalence rates and treatment efficacy.

An Evaluation of the Regression Discrepancy Method for Identifying Children With Learning Disabilities

A learning disability (LD) exists if a childs academic achievement lags significantly behind intellectual ability and there is no other known cause for the discrepancy. The Regression Discrepancy Method using multiple regression for identifying LD children directly parallels the theoretical definition. It involves giving both an ability and an achievement test, which are normed together. An anticipated achievement score is computed for each child based on ability, grade level, and sex. Then, for each ability score, the 10% whose actual achievement is most discrepant from their anticipated achievement are identified as likely LD. In comparing this method with other identification techniques, the author discusses its advantages and limitations, pointing out that while it is conceptually and methodologically superior to other approaches, it is nonetheless seriously deficient as a sole criterion for LD identification. In an area so fraught with definitional and instrumentation problems, provision should be made to collect data independently and to trust only those diagnoses that are consistent and independently verifiable.

An Evaluation of the Identification of Learning Disabled Students in Colorado

The purpose of the study was to evaluate the process whereby children are identified as learning disabled (LD) and the validity of subsequent educational placements: (a) a representative sample of 1,000 cases were analyzed both quantitatively and qualitatively; (b) extensive questionnaires were administered to professionals involved in LD identification; (c) definitions and assessment procedures were evaluated. Only 28% of the LD cases met strict criteria for LD; another 15% showed weak signs of the handicap. The remaining 57% currently in1 LD programs were better described by other indicators such as non-English dominant (7%), other handicaps (10%), slow learners (11%), minor behavior problems (4%), etc. The costs of LD identification were found to be excessive; each year almost half of the resources available for LD pupils was spent on assessment and staffing. Policy implications are discussed.

Effects of Kindergarten Retention at the End of First Grade.

The purpose of the study was to assess the effect of kindergarten retention on firstgrade achievement and adjustment. Forty children who had been retained in kindergarten were identified from schools that practiced kindergarten retention at a high rate. Control children were selected from schools matched on socioeconomic and achievement level, but that did not practice retention in kindergarten. Then, control children were selected individually to match retained children on sex, birthdate, socioeconomic level, second language, and beginning kindergarten readiness scores. The two groups, which were equally young and unready at the start of kindergarten, were compared at the end of first grade on seven outcome measures; the retained children were then completing three years of school and the control children two. There were no differences between the retained and control children on teacher ratings of reading achievement, math achievement, social maturity, learner self-concept, or attention. The groups also did not differ in CTBS math scores; the only difference occurred on the CTBS reading test, where the retained group was one month ahead. Based on parent interview data, children who had spent an extra year before first grade were not much different from those deemed at risk but not retained, except that, on average, retained children had slightly more negative attitudes toward school. The study findings are consistent with other available research on transition programs that show no academic benefit for the extra year and, when examined, a negative impact on social-emotional outcomes.