Lyle V. Jones

Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement.

Three alternative procedures to adjust significance levels for multiplicity are the traditional Bonferroni technique, a sequential Bonferroni technique developed by Hochberg (1988), and a sequential approach for controlling the false discovery rate proposed by Benjamini and Hochberg (1995). These procedures are illustrated and compared using examples from the National Assessment of Educational Progress (NAEP). A prominent advantage of the Benjamini and Hochberg (B-H) procedure, as demonstrated in these examples, is the greater invariance of statistical significance for given comparisons over alternative family sizes. Simulation studies show that all three procedures maintain a false discovery rate bounded above, often grossly, by α (or α/2). For both uncorrelated and pairwise families of comparisons, the B-H technique is shown to have greater power than the Hochberg or Bonferroni procedures, and its power remains relatively stable as the number of comparisons becomes large, giving it an increasing advantage when many comparisons are involved. We recommend that results from NAEP State Assessments be reported using the B-H technique rather than the Bonferroni procedure.

The Rational Origin for Measuring Subjective Values

Abstract A method is proposed and empirically demonstrated for extending Thurstones law of comparative judgment so as to transform psychological qualities into an additive measurement scale. Application of the method yields results supporting the contention that subjective values can be measured on an additive scale, an equal unit scale with a meaningful zero point. * This paper reports research partially supported by the Quartermaster Food and Container Institute for the Armed Forces.

Recent Trends in Achievement Levels of Black and White Youth

For nearly 100 years educational and psychological studies have recorded lower average cognitive test scores for black respondents than for white respondents. In the 1960s a major sample survey, Equality of Educational Opportunity (Coleman et al., 1966), was undertaken to describe the educational differences between white and black students. At about the same time, a large number of federal and state programs were initiated to correct legal, economic, educational, and social imbalance among racial groups. It is not feasible to perform controlled experiments in such programs, and, without experimental control, it is not possible to determine accurately their effects. However, it is possible to monitor change over time in national indicators of achievement.

Monitoring the Mathematics Achievement of Black Students

Findings from the National Assessment of Educational Progress for 1973 and 1978 are reviewed. They show improvement in levels of mathematics achievement for black students at ages 9 and 13, compared with a decline for white students at those ages. In a special National Assessment of mathematics in 1975 -76 for ages 13 and 17, substantial differences are found between average mathematics achievement scores of white and black youth. Based on a multiple regression analysis at age 17, more than half of the total variance in mathematics achievement scores is accounted for by regression, with schoolto-school differences in background variables and individual background differences within school about equally influential. About half of the white-black mean difference is accounted for by regression and, in this accounting, school differences in background variables play a more prominent role than individual differences within school. A particularly influential predictor of mathematics achievement is the number of high school algebra and geometry courses taken. Marked differences are found between predominantly black and predominantly white high schools in the average numbers of such courses taken. The adoption of policies that reduce those differences would be expected to result in relatively higher levels of mathematics achievement for black students.

The predictability of words and their grammatical classes as a function of rate of deletion from a speech transcript

Summary The “cloze” procedure, which requires raters to fill in words deleted from a transcript and provides a measure of the predictability of speech, was used to study the predictability of the various grammatical classes and of specific items drawn from them, and changes in such predictability with systematic variation in rate of textual deletion (from every second item deleted to every sixth item deleted). A transcript of running speech elicited by TAT cards constituted the text and two measures were obtained indicating success in verbatim identification (V) and success in form-class identification (FC); the relation between these two measures was also examined. The main findings are (a) that success both in FC and V completion increases moderately with decreasing frequency of deletion, but that raters do fairly well, particularly for FC scores, even when every second item is deleted; (b) that there are considerable differences in performance between grammatical classes both for FC and V scores, these differences being less for the former scores than the latter; and (c) that within any given grammatical class there are still substantial differences among the items with regard to both verbatim and form-class predictability. It was argued that the determinants of FC and V predictability are rather different, FC predictability being more dependent upon the relatively close grammatical environment whereas V predictability depends more on both close and remote topical content or semantic features of discourse. It was also suggested that these determinants may vary with the particular grammatical class under consideration. While grammatical or form-class contraints are tighter than contraints on specific identification it was suggested that the V or V/FC measure provides an underestimate of the actual predictability of the sense of the discourse, and that the communicative significance of a particular level of V or V/FC success may vary with the grammatical class involved, being greater for semantic than for syntactic items.

Decision making in a sequential search task

Ss were instructed to find the largest of a set of 200 different numbers, observed one at a time (when only the number currently observed could be declared the largest). Numbers sampled from one of three distributions, one with positive skew, one with negative skew, and one rectangular, were presented to three samples of 22 Ss, tested in small groups. The rectangular-distribution condition also was administered individually to a fourth sample of 22 Ss. Results failed to show effects of the distribution of numbers upon S’s performance. However, as predicted, Ss tested in groups tended to observe more cards before stopping than those tested individually. Data are analyzed in an effort to evaluate the adequacy of alternative “stopping rules” as strategies descriptive of Ss performance.

A History of the National Assessment of Educational Progress and Some Questions About Its Future

The initial aim of the National Assessment of Educational Progress (NAEP) was to develop and then to employ procedures for monitoring achievement in the nation. Here, a history of NAEP is presented, with a special focus on procedural changes from 1969 to date. Then some questions are posed about future shifts in the purposes of NAEP. In offering answers to the questions, I argue that NAEP should continue to monitor achievement trends in the nation and in the participating states and that, because those objectives are not compatible with NAEP as a high-stakes accountability assessment program, NAEP should avoid serving this added purpose.

Indicators of Precollege Education in Science and Mathematics, A Preliminary Review.

Many studies point to the inadequacy of precollege education in the United States. How can it be improved? The development of effective policy requires information on the condition of education and the ability to measure change. This book lays out a framework for an efficient monitoring system. Key variables include teacher quality and quantity, course content, instructional time and enrollment, and student achievement.

The Classification of Parts of Speech for the Characterization of Aphasia

Evidence from recent research suggests that aphasia patients may be distinguished, one from another, by assessing the extent to which their speech departs from normal in terms of frequencies of usage of words from various grammatical classes. A system is proposed for classifying words into distinct grammatical classes. The system is designed to provide ease of classification, for speech either from aphasic patients or normal speakers, and to entail consistent application of grammatical criteria. It lends itself to inclusion in a computer programme for processing English language texts. Some potential uses of the classification system are briefly noted.

An Assessment of Research-Doctorate Programs in the United States: Mathematical and Physical Sciences.

1 Front Matter 2 I. Origins of Study and Selection of Programs 3 II. Methodology 4 III. Chemistry Programs 5 IV. Computer Science Programs 6 V. Geoscience Programs 7 VI. Mathematics Programs 8 VII. Physics Programs 9 VIII. Statistics/Biostatistics Programs 10 IX. Summary and Discussion 11 Minority Statement 12 Appendix A: Letter to Institutional Coordinators and Accompanying Survey Form (Measures 01-03) 13 Appendix B: Survey of Earned Doctorates (Measures 04-07) 14 Appendix C: Letter to Evaluators and Specimen of the Instrument Used in the Reputational Survey (Measures 08-11) 15 Appendix D: The ARL Library Index (Measure 12) 16 Appendix E: Data on Faculty Research Support and R&D Expenditures (Measures 13 and 14) 17 Appendix F: Data on Publication Records (Measures 15 and 16) 18 Appendix G: Conference on the Assessment of Quality of Graduate Education Programs--Participants and Summary 19 Appendix H: Planning Committe for the Study of the Quality of Research-Doctorate Programs 20 Appendix I: Region and State Codes for the United States and Possessions