Joel A. Gordon

The psychometric properties of five scoring methods applied to the script concordance test.

Purpose The Script Concordance Test (SCT) is designed to measure cognitive ability related to successful clinical decision making. An SCTs usefulness for medical education depends on establishing its construct validity. The SCTs present construct relates examinees scores to experts’ response patterns, which does not require a single-best-answer format. Because medical education assessments do require a single best answer, the authors compared the psychometric properties of two aggregate scoring methods with three single-best-answer scoring methods for an SCT. Method A nephrology SCT was developed and administered to 85 examinees. Examinees’ scores derived from a key developed using eight experts and a traditional aggregate scoring method on a five-point Likert-based scale were compared with four alternate scoring methods (one method eliminated the multipoint Likert-type scale and three eliminated the Likert-type scale and employed single-best-answer scoring). Results Two of the four alternate scoring methods performed as well as the traditional Likert-type aggregate scoring method. Scores from all five methods were highly intercorrelated. In addition, each method produced scores similarly correlated with level of experience, and none exhibited an intermediate effect. Conclusions Single-best-answer scoring with three answer choices produced results similar to aggregate scoring on a Likert-type scale. Because SCT items appear to assess an examinees understanding of the interrelatedness of medical knowledge, single-best-answer scoring on an SCT may be valid as an educational assessment. More research is needed to assess differential validity compared with multiple-choice question exams and the predictive validity related to clinical performance.

Hydroxyeicosatetraenoic acid oxidation in chinese hamster ovary cells: a peroxisomal metabolic pathway

To evaluate the peroxisomal requirement for beta-oxidation of hydroxyeicosatetraenoic acids (HETES), we tested 5-, 12- and 15-HETE oxidation in wild-type and mutant Chinese hamster ovary (CHO) cells. Mutant CHO cells contain peroxisomal ghosts, have random cytosolic localization of catalase and lack two of the enzymes necessary for peroxisomal beta-oxidation. Reverse-phase HPLC indicated that 33% of 12-HETE radioactivity was converted by wild-type CHO cells during a 2 h incubation to one major and several minor polar metabolites. Wild-type CHO cells also converted 15-HETE to one major and several minor polar metabolites. Neither 12- nor 15-HETE were converted to any metabolites by the mutant CHO cell lines, despite appreciable cellular uptake of these hydroxyeicosanoids. 5-HETE was not converted to any metabolic products by either the wild-type or the mutant CHO cells. Docosahexaenoic acid beta-oxidation was substantially reduced in the mutants as compared to the wild-type cells, palmitic acid beta-oxidation was reduced to an intermediate extent in the mutants, but octanoate beta-oxidation and citrate synthase activity were not impaired. Protein immunoblotting for mitochondrial manganese superoxide dismutase indicated a single band of identity at 20 kDa in both wild-type and mutant CHO cells. Since mutant CHO cells fail to convert 12- and 15-HETE to oxidative metabolites but contain normal mitochondrial enzymatic activities, intact peroxisomes appear to be the organelle responsible for HETE oxidation.

Recommendations for Assigning Weights to Component Tests to Derive an Overall Course Grade

Background: Performance-based assessments are often not reliable enough to be used as the sole method for determining a grade. Purpose: Using more than 1 assessment format has several advantages and can improve the quality of the information used to calculate course grades. To achieve a valid and reliable total score representing a combination of multiple assessment formats requires appropriate weighting procedures. Methods: The classical test theory rationale and methodology for appropriately weighting in-course assessments is presented. Two reliability-related equations are used to make decisions about weighting for a computerized performance assessment and a pattern recognition multiple-choice exam. Results: Using the equations presented, the outcomes of various weighting scenarios are graphically presented. This technique produced weights that allow the instructor to obtain acceptable reliability while retaining a substantial emphasis on performance assessment. Conclusion: The weights assigned to component tests used to derive a total score have important reliability and validity implications. Course instructors need to consider both empirical reliability and logical validity evidence in determining component weights. When used in conjunction with classical methods, objectively scored item formats can augment performance assessments and enhance overall validity and reliability.

A Prelude to Modeling the Expert: A Generalizability Study of Expert Ratings of Performance on Computerized Clinical Simulations.

Context: The University of Iowa College of Medicine has developed a series of computer-based clinical simulations and successfully integrated them into the clinical clerkship curriculum. The computerized patient simulations provide a high degree of realism in simulating a clinical encounter. In an effort to improve the validity of our clinical skills assessment, we have initiated testing research utilizing these simulations. Because of the high costs associated with employing expert raters for performance scoring, automated scoring was deemed essential.Purpose: This study is designed to address the preliminary research questions related to utilizing the simulations for performance assessment and developing a psychometrically sound automated scoring mechanism. Specifically, it addresses issues of reliability in relation to rater and simulation characteristics, and provides essential data required for designing a sound methodology to obtain ratings for modeling.Design: The judgements of 3 expert clinician/raters, grading the responses of 69 third-year medical students, to 2 computerized simulations, are analyzed in a generalizabilty study. A random effects (persons by raters) ANOVA was performed to estimate variance components for modeling. A case facet was added to the anlaysis to provide data regarding performance assessment characteristics. Estimation of the magnitude of each variance component represents the outcome of the generalizability study. Variance estimates are used in the decision study phase of the research.Results: Only moderate levels of inter-rater reliability were obtained. Four or more raters were indicated to obtain adequate reliability. A high level of task/simulation specificity was found. Three or more simulations were indicated for performance assessment. Suggestions for improving ratings were offered.

Raising Medical Students' Awareness of Nutrition and Fitness in Disease Prevention: Nutrition and Fitness Program at the University of Iowa

Abstract: At the University of Iowa we devised a learning experience, called the Nutrition and Fitness Program, for third-year medical students. The program was designed to raise awareness of the role of nutrition and exercise in the prevention and treatment of disease. Students spent one afternoon learning about their personal health risk factors, such as body mass index, percent body fat, other anthropometric measures such as waist, hip and mid-arm circumference, blood lipids, bone-mass density, dietary analysis, and fitness assessment. Students spent another afternoon visiting the cardiac rehabilitation center. At the end of each rotation, students gathered for a heart-healthy meal that served as a focus for a discussion with dietitians about important nutrition issues. The literature and our work with medical students support the need and acceptance of a personalized, practical approach to nutrition education. By offering medical students the opportunity to learn about their own nutrition and fitness risk factors, this Nutrition and Fitness Program appears to have played an important role in the students’ medical education by narrowing the gap between the “science of nutrition” and the “application of nutrition”. Students appreciated learning more about their own health factors and felt that personalizing the information made the learning more valuable and would help in counseling their future patients more effectively.