Gregory E. Stone

A standardized rubric to evaluate student presentations

Objective. To design, implement, and assess a rubric to evaluate student presentations in a capstone doctor of pharmacy (PharmD) course. Design. A 20-item rubric was designed and used to evaluate student presentations in a capstone fourth-year course in 2007–2008, and then revised and expanded to 25 items and used to evaluate student presentations for the same course in 2008–2009. Two faculty members evaluated each presentation. Assessment. The Many-Facets Rasch Model (MFRM) was used to determine the rubrics reliability, quantify the contribution of evaluator harshness/leniency in scoring, and assess grading validity by comparing the current grading method with a criterion-referenced grading scheme. In 2007–2008, rubric reliability was 0.98, with a separation of 7.1 and 4 rating scale categories. In 2008–2009, MFRM analysis suggested 2 of 98 grades be adjusted to eliminate evaluator leniency, while a further criterion-referenced MFRM analysis suggested 10 of 98 grades should be adjusted. Conclusion. The evaluation rubric was reliable and evaluator leniency appeared minimal. However, a criterion-referenced re-analysis suggested a need for further revisions to the rubric and evaluation process.

Two short forms of the Agnew Relationship Measure: The ARM-5 and ARM-12

Abstract This article reports the development and psychometric properties of two short forms of the 28-item Agnew Relationship Measure, the ARM-12 and ARM-5. For the ARM-12, results of previous research were used together with conceptual considerations to select three items to represent each of four ARM subscales: Bond, Partnership, Confidence, and Openness. For the ARM-5, item-analytic principles were used to select five items to represent overall alliance. In all three ARMs, client and therapist versions were constructed to contain parallel items. We drew data to assess reliability and validity from three UK trials of brief therapy for depression. Results indicated that the two short ARMs have acceptable psychometric properties and that they converged with each other and with the full ARM.

Evaluating the Quality of Ratings Obtained from Standard-Seting Judges

The purpose of this study is to describe a new approach based on Rasch measurement theory for examining the quality of ratings obtained from standard-setting judges. This new approach is based on the adaptation and modification of several indicators of rating errors. To explore the usefulness of the Rasch measurement theory, the ratings of the nine judges who took part in the standard-setting process on a certification examination for nurses were examined. Ratings were obtained for 213 multiple-choice items on this examination. The data suggest that the judges vary significantly in their views of the essential items required for certification. Furthermore, the data suggest that there is statistically significant variability in the judged essentiality of these items.

High school health-education teachers' perceptions and practices related to teaching HIV prevention.

BACKGROUND HIV/AIDS is one of the leading causes of illness and death in the United States with individuals between the ages of 13 and 19 years being especially vulnerable for infection. The purpose of this study was to examine the attitudes, perceptions, and instructional practices of high school health teachers toward teaching HIV prevention. METHODS A total of 800 surveys were sent to a national random sample of high school health teachers and 50% responded. RESULTS There was almost complete agreement (99%) among respondents that HIV prevention instruction is needed. The factors that emerged as significantly influencing the attitudes and perceptions of high school health teachers about teaching HIV prevention were related to teacher preparation, training, and years of experience teaching health education. A state mandate requiring HIV prevention instruction was significantly associated with higher teacher efficacy expectations and more perceived benefits, but did not have a significant influence in relation to practices in the classroom. Characteristics of high school health teachers that were significantly related to attitudes, perceptions, and instructional practices included the instructors age, sex, and race/ethnicity. CONCLUSIONS High school health teachers who reported the least experience teaching health education had the least supportive attitudes, perceived the most barriers, and had the lowest efficacy and outcome expectations related to teaching about HIV prevention. Whereas these findings support the importance of teacher preparation and training, they also suggest that more recent college graduates may not be fully prepared to provide effective instruction in HIV prevention.

Comparing Construct Definition in the Angoff and Objective Standard Setting Models Playing in a House of Cards Without a Full Deck

Typical validation studies on standard setting models, most notably the Angoff and modified Angoff models, have ignored construct development, a critical aspect associated with all conceptualizations of measurement processes. Stone compared the Angoff and objective standard setting (OSS) models and found that Angoff failed to define a legitimate and stable construct. The present study replicates and expands this work by presenting results from a 5-year investigation of both models, using two different approaches (equating and annual standard setting) within two testing settings (health care and education). The results support the original conclusion that although the OSS model demonstrates effective construct development, the Angoff approach appears random and lacking in clarity. Implications for creating meaningful and valid standards are discussed.

Objective Standard Setting for Judge-Mediated Examinations

Judge-mediated examinations are defined as those for which expert evaluation (using rubrics) is required to determine correctness, completeness, and reasonability of test-taker responses. The use of multifaceted Rasch modeling has led to improvements in the reliability of scoring such examinations. The establishment of criterion-referenced standards for these assessments has, however, remained a labor-intensive practice. This article explores a method for incorporating the establishment of minimal competency within the measurement model itself. Two high-stakes testing boards participated in a controlled experiment comparing a multifaceted Rasch modeled approach to a traditional modified borderline approach often used for standard setting in judge-mediated situations. Criterion points from each were compared to assess differences and support validity. Results of the experiment suggest that modeling standard setting as part of the performance evaluation is an effective and efficient way to define minimal competence.

Objective Standard Setting in Educational Assessment and Decision Making

Assessment and evaluation at all levels of educational systems have become policy priorities for many countries. Two common reasons for this are student learning expectations and accountability. Although much effort has been put into the creation and refinement of content standards, standardized tests, and methods for using testing results, there has been less attention paid to the development of performance standards (proficiency levels) that greatly affect policy decision making. The present study investigates the Objective Standard Setting Model (OSS) as an improved criterion-referenced method for setting multilevel performance standards. To demonstrate how OSS can be employed for performance standard setting, our study used data from preservice teachers completing an assessment literacy test in a pre- and posttest environment. Using OSS, performance standard levels of proficient and excellent were established with clear content-related descriptions of growth in student content mastery.

Benjamin D. Wright: A Higher Standard

In 1992, I approached Ben Wright about the vexing problem facing the testing organization with whom I worked. The establishment of effective criterion-referenced standards that could be used without inevitable adjustment seemed out of reach when employing most traditional models. Soon Ben would become my professor, mentor and friend. With his irrepressible energy, he quickly produced a wealth of published and, more importantly, unpublished thought on the matter. During the next 4 years I was fortunate enough to work with Ben in the development of what is now called the Objective Standard Setting model. His vision helped assessment to define a new pathway to equity and meaningful measurement. As he had done for so many others before me including my father, his wisdom and inspiration would help me to find a new and passionate career, and to share that experience with new generations. This paper presents the development of criterion-referenced standard setting and the vital role Benjamin Wright would play in this important pursuit.