Randy Elliot Bennett

Formative Assessment: A Critical Review.

This paper covers six interrelated issues in formative assessment (aka, ‘assessment for learning’). The issues concern the definition of formative assessment, the claims commonly made for its effectiveness, the limited attention given to domain considerations in its conceptualisation, the under‐representation of measurement principles in that conceptualisation, the teacher‐support demands formative assessment entails, and the impact of the larger educational system. The paper concludes that the term, ‘formative assessment’, does not yet represent a well‐defined set of artefacts or practices. Although research suggests that the general practices associated with formative assessment can facilitate learning, existing definitions admit such a wide variety of implementations that effects should be expected to vary widely from one implementation and student population to the next. In addition, the magnitude of commonly made quantitative claims for effectiveness is suspect, deriving from untraceable, flawed, dated, or unpublished sources. To realise maximum benefit from formative assessment, new development should focus on conceptualising well‐specified approaches built around process and methodology rooted within specific content domains. Those conceptualisations should incorporate fundamental measurement principles that encourage teachers and students to recognise the inferential nature of assessment. The conceptualisations should also allow for the substantial time and professional support needed if the vast majority of teachers are to become proficient users of formative assessment. Finally, for greatest benefit, formative approaches should be conceptualised as part of a comprehensive system in which all components work together to facilitate learning.

Construction versus choice in cognitive measurement : issues in constructed response, performance testing, and portfolio assessment

Contents: Preface. R.E. Bennett, On the Meanings of Constructed Response. R.E. Traub, On the Equivalence of the Traits Assessed by Multiple-Choice and Constructed-Response Tests. R.E. Snow, Construct Validity and Constructed-Response Tests. S. Messick, Trait Equivalence as Construct Validity of Score Interpretation Across Multiple Methods of Measurement. R.J. Mislevy, A Framework for Studying Differences Between Multiple-Choice and Free-Response Test Items. K.K. Tatsuoka, Item Construction and Psychometric Models Appropriate for Constructed Responses. N.J. Dorans, A.P. Schmitt, Constructed Response and Differential Item Functioning: A Pragmatic Approach. J. Braswell, J. Kupin, Item Formats for Assessment in Mathematics. R. Camp, The Place of Portfolios in Our Changing Views of Writing Assessment. D.P. Wolf, Assessment as an Episode of Learning. D.H. Gitomer, Performance Assessment and Educational Measurement. C.A. Dwyer, Innovation and Reform: Examples from Teacher Assessment. T.W. Hartle, P.A. Battaglia, The Federal Role in Standardized Testing. S.P. Robinson, The Politics of Multiple-Choice Versus Free-Response Assessment.

Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized Tests

Three open-ended response types—mathematical expression (ME), generating examples (GE), and graphical modeling (GM)—are described that could broaden the conception of mathematical problem solving used in computerized admissions tests. ME presents single-best-answer problems that call for an algebraic formalism, the correct rendition of which can take an infinite number of surface forms. GE presents loosely structured problems that can have many good answers taking the form of a value, letter pattern, expression, equation, or list. GM asks the examinee to represent a given situation by plotting points on a grid; these items can have a single best answer or multiple correct answers. For the three basic types, sample items are provided, the examinee interfaces and approaches to automated scoring are described, and research results are reported. It is illustrated how ME, GE, and GM can be combined to form extended constructed-response problems, and adescription is offered of how item classes might be used as a basis for creating production-ready scoring keys.

Agreement between expert-system and human raters' scores on complex constructed-response quantitative items.

This study evaluated agreement betwen expert-system and human scores on 12 constructed-response algebra word problems taken by Graduate Record Examination General Test examinees. Ideas are discussed for using expert systems in large-scale assessment programs and in interactive diagnostic assessment.

Using multimedia in large-scale computer-based testing programs

To measure problem solving and related cognitive constructs effectively, future generations of tests will need to include tasks more like those actually encountered in academic and work settings. The advent of computer-based tests makes the inclusion of such performance tasks more feasible. One way in which some tasks might be made more relevant is through multimedia, particularly by incorporating dynamic stimuli, such as audio, video, and animation. We give examples of how such stimuli might be employed to assess skill in history, physical education, and the sciences. In addition, we pose an initial set of issues for consideration by operational testing programs.

The Relationship of Expert-System Scored Constrained Free-Response Items to Multiple-Choice and Open-Ended Items

This study examined the relationship of an expert- system scored constrained free-response item (requir ing the student to debug a faulty computer program) to two other item types: (1) multiple-choice and (2) free- response (requiring production of a program). Confir matory factor analysis was used to test the fit of a three-factor model to these data and to compare the fit of the model to three alternatives. These models were fit using two random-half samples, one given a faulty program containing one bug and the other a program with three bugs. A single-factor model best fit the data for the sample taking the one-bug constrained free re sponse and a two-factor model fit the data somewhat better for the second sample. In addition, the factor intercorrelations showed this item type to be highly re lated to both the free-response and multiple-choice measures.

Assessing Complex Problem Solving Performances

Abstract Computer-based simulations can give a more nuanced understanding of what students know and can do than traditional testing methods. These extended, integrated tasks, however, introduce particular problems, including producing an overwhelming amount of data, multidimensionality, and local dependence. In this paper, we describe an approach to understanding the data from complex performances based on Evidence-Centred Design (ECD), a methodology for devising assessments and for using the evidence observed in complex student performances to make inferences about proficiency. We use as an illustration the National Assessment of Educational Progress (NAEP) Problem Solving in Technology-Rich Environments Study, which is being conducted to exemplify how non-traditional skills might be assessed in a sample-based national survey. The paper focuses on the inferential uses of Evidence-Centred Design, especially how features are extracted from student performance, how these extractions are evaluated, and how the evaluations are accumulated to make evaluative judgements.

Inner-city adults with severe reading difficulties : A closer look

Relatively little is known about the characteristics of inner-city adults who seek assistance from literacy programs. Increased knowledge about this population will enhance the development of more effective programs, as well as policy options. This study describes the characteristics of 280 adults, ages 16 to 63, who came to an adult literacy program that focused on severe reading difficulties. The program, located within a hospital complex in a large, urban area, attracted these individuals through an extensive multimedia outreach effort. Results suggested that the adults who sought help were generally characterized by a vast array of cognitive, academic, and social difficulties. In addition, the extent of these difficulties increased dramatically as literacy level declined. These findings suggest that comprehensive educational, social, and vocational services may be needed to help adults with severe reading difficulties cope with the diverse and severe problems they face.

Internal Construct Validity of a College Admissions Test Across Handicapped and Nonhandicapped Groups

This study investigated the comparability of SAT Verbal and Mathematical scores for nonhandicapped examinees taking standard administrations and handicapped populations taking extended-time, special administrations. A simple two-factor confirmatory model based on Verbal and Mathematical item parcels was posed and tested for invariance across two sets of handicapped and nonhandicapped samples, with each set taking a different form of the SAT. Results showed the two common-factor model to provide a reasonable fit in all populations with the mathematical factor generally fitting somewhat better than the verbal one. Compared with the nonhandicapped population, the factors tended to be less correlated in most of the handicapped groups. In addition, some evidence of verbal method factors defined by the antonym and reading comprehension item types was evident. Finally, there was some suggestion that one of the two Mathematical item types led to different observed score scale units for learning disabled students taking a cassette administration.

A Review of Automatically Scorable Constructed-Response Item Types for Large-Scale Assessment

The use of automated scanning of test sheets, beginning in the 1930s, led to widespread use of the multiple-choice format in standardized testing. New forms of automated scoring now hold out the possibility of making a wide range of constructed-response item formats feasible for use on a large-scale basis. We describe new developments in five domains: mathematical reasoning, algebra problem solving, computer science, architecture, and natural language. For each one, we describe the task as presented to the examinee, the methods used to score the response, and the psychometric properties of the item responses. We then highlight general challenges and issues spanning these technologies. We conclude by offering our views on the ways in which such technologies are likely to shape the future of testing.