John A. Stahl

Judge Consistency and Severity across Grading Periods.

The purpose of this research project was to confirm that differences in the severity of judges and the stringency of grading periods occur, regardless of the nature of the assessment or the examination materials used. Three rather different examinations that require judges were analyzed, using an extended Rasch model to determine whether differences in judge severity and grading-period stringency were observable for all three examinations. Significant variation in judge severity and some variation across grading periods were found on all three examinations. This implies that regardless of the nature of the examination, items, or judges, examinee/measures cannot be considered independent of the particular judges involved unless correction for severity is made systematically. Accounting for judge severity and gradinig-period stringency is extremely important when pass/fail decisions that are meant to generalize to competence are made, as in certification examinations.

Interjudge Reliability and Decision Reproducibility

The purpose of this article is to discuss the importance of decision reproducibility for performance assessments. When decisions from two judges about a students performance using comparable tasks correlate, decisions have been considered reproducible. However, when judges differ in expectations and tasks differ in difficulty, decisions may not be independent of the particular judges or tasks encountered unless appropriate adjustments for the observable differences are made. In this study, data were analyzed with the Facets model and provided evidence that judges grade differently, whether or not the scores given correlate well. This outcome suggests that adjustments for differences among judge severities should be made before student measures are estimated to produce reproducible decisions for certification, achievement, or promotion.

Content Validity Revisited Transforming Job Analysis Data into Test Specifications

This article describes a process for linking job analysis data to test specifications. Rasch rating scale analysis is used to order the ratings of importance assigned to task and content items by practicing laboratory managers. This ordering produces variables representing the range of task and content items from most to least important. The variables also provide an objective frame of reference for review of the data by the experts. The Rasch calibrations for each task and content item are transformed to percentages based on useful limits. From the transformed percentages, test specifications reflecting practice patterns in the field of laboratory management are developed.

Impact of Examiners on Candidate Scores: An Introduction to the Use of Multifacet Rasch Model Analysis for Oral Examinations.

This article presents an introduction to Rasch model analysis and its application to examinations that require examiners. The introduction begins with a discussion of the Rasch model and its assumptions and then presents the multifacet Rasch model to include examiners. The article concludes with an example of an application to an oral examination administered by a medical specialty board.

Applications of conjoint measurement in admission and professional certification programs

Abstract The establishment and management of standards is at the heart of professional education and certification. Usually paper and pencil tests are not sufficient and further evidence must be collected by observing and judging instances of professional performance. The normative methods that have served as defaults for this purpose are devoid of competence specifics. For some years now a few professional associations have been using probabilistic conjoint measurement as the means for developing, applying and maintaining their standards. This chapter describes how this has been done and how well it has worked in practice.