Matthew Lineberry

Threats to validity in the use and interpretation of script concordance test scores

Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high‐stakes testing.

Consequences Validity Evidence: Evaluating the Impact of Educational Assessments.

Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors’ purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%–20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

Setting mastery learning standards.

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance, not curricular time. Learners practice and retest repeatedly until they reach a designated mastery level; the final level of achievement is the same for all, although time to mastery may vary. Given the unique properties of mastery learning assessments, a thoughtful approach to establishing the performance levels and metrics that determine when a learner has demonstrated mastery is essential. Standard-setting procedures require modification when used for mastery learning settings in health care, particularly regarding the use of evidence-based performance data, the determination of appropriate benchmark or comparison groups, and consideration of patient safety consequences. Information about learner outcomes and past performance data of learners successful at the subsequent level of training can be more helpful than traditional information about test performance of past examinees. The marginally competent “borderline student” or “borderline group” referenced in traditional item-based and examinee-based procedures will generally need to be redefined in mastery settings. Patient safety considerations support conjunctive standards for key knowledge and skill subdomains and for items that have an impact on clinical outcomes. Finally, traditional psychometric indices used to evaluate the quality of standards do not necessarily reflect critical measurement properties of mastery assessments. Mastery learning and testing are essential to the achievement and assessment of entrustable professional activities and residency milestones. With careful attention, sound mastery standard-setting procedures can provide an essential step toward improving the effectiveness of health professions education, patient safety, and patient care.

Making the case for mastery learning assessments: key issues in validation and justification.

Theoretical and empirical support is increasing for mastery learning, in which learners must demonstrate a minimum level of proficiency before completing a given educational unit. Mastery learning approaches aim for uniform achievement of key objectives by allowing learning time to vary and as such are a course-level analogue to broader competency-based curricular strategies. Sound assessment is the cornerstone of mastery learning systems, yet the nature of assessment validity and justification for mastery learning differs in important ways from standard assessment models. Specific validity issues include (1) the need for careful definition of what is meant by “mastery” in terms of learners’ achievement or readiness to proceed, the expected retention of mastery over time, and the completeness of content mastery required in a particular unit; (2) validity threats associated with increased retesting; (3) the need for reliability estimates that account for the specific measurement error at the mastery versus nonmastery cut score; and (4) changes in item- and test-level score variance over retesting, which complicate the analysis of evidence related to reliability, internal structure, and relationships to other variables. The positive and negative consequences for learners, educational systems, and patients resulting from the use of mastery learning assessments must be explored to determine whether a given mastery assessment and pass/fail cut score are valid and justified. In this article, the authors outline key considerations for the validation and justification of mastery learning assessments, with the goal of supporting insightful research and sound practice as the mastery model becomes more widespread.

Educational interventions for international medical graduates: a review and agenda.

International medical graduates (IMGs) play key roles in the health systems of their host countries, but face unique challenges, which makes the provision of effective, tailored support for IMGs essential.

Recommendations for reporting mastery education research in medicine (ReMERM)

Guidelines for reporting several types of medical studies have been described in the literature. However, there are no current guidelines to report studies on mastery learning curriculum development and trainee evaluation in medical education. Such guidelines will be important because medical education is moving toward a competency-based model. The authors sought to define standards for the evaluation of mastery learning curricula using previously published guidelines in related fields and expert review. The authors reviewed previously published guidelines from clinical medicine, medical education, and the social sciences. Six authors with expertise in mastery learning curricula, performance assessment, and medical education compiled and reached agreement about a list of guidelines. The authors later circulated the list to 12 other experts and made revisions. A final list of guidelines was established and received group consensus. The Reporting Mastery Education Research in Medicine (ReMERM) guidelines have 22 categories with 38 items considered to be imperative for reporting a mastery learning research study. Details about each item, with a specific focus on those unique to mastery learning, are discussed. The ReMERM guidelines highlight the importance of developing rigorous curricula that embody reliable measures which yield valid decisions about achievement among medical learners. These guidelines should improve the quality of reporting and help educators, authors, peer reviewers, journal editors, and readers to better understand and evaluate mastery learning research. With this shift to competency-based medical education, the ReMERM guidelines should help meet medical educators’ needs to achieve these new goals.

Validity evidence for a patient note scoring rubric based on the new patient note format of the United States medical licensing examination

Purpose This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup. Method Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass–fail rates between the rubric and the previous global rating. Response process was examined using rater feedback. Results Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass–fail rates. Raters’ feedback indicated that they required more training for scoring the DDX and Workup dimensions. Conclusions There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.

Comparative Research on Training Simulators in Emergency Medicine: A Methodological Review

Summary Statement Recent reviews and research agenda in simulation-based training for health care have repeatedly called for more studies comparing the effectiveness of different simulation modalities and approaches. Although a small body of such comparative research does exist, it is apparent that this line of research comes with particular methodological challenges. In this review, 20 studies comparing simulation modalities in emergency medicine were analyzed in terms of key methodological factors. Results of this literature review show that (1) past studies are largely underpowered to detect effects of the magnitude likely to be found, (2) researchers seeking to demonstrate equivalence of training approaches do not use appropriate statistical tests of equivalence, and (3) studies often use performance criterion test beds that may not support valid conclusions about trainees’ future performance on the job. We discuss strengths and weaknesses of reviewed studies along these issues and recommend options for improving future comparative research in simulation-based training.

Script concordance tests: strong inferences about examinees require stronger evidence.

On our first point, that disagreements of scoring panellists on SCT items are often illogical and have no place in a test’s scoring key, Lubarsky et al. note that ‘response variability among the members of an SCT panel is... a key determinant of the test’s discriminatory power’, where discrimination is generally between residents and panellists. As Cook et al. have noted, medical education research has put too much emphasis and uncritical faith in such knowngroups validity evidence. A host of facets differentiate earlyfrom late-career professionals, including psychosocial development with age, shifts in demographic characteristics, and differences in exposure to trends in medical practice, for instance. If panellists outscore residents, how do we know that this reflects their ‘clinical reasoning skill’ rather than any of a dozen other constructs?

Psychometric properties of the Fundamentals of Endoscopic Surgery (FES) skills examination

BackgroundThe Fundamentals of Endoscopic Surgery (FES) manual skills examination is a simulation-based assessment of five foundational skills in endoscopic surgery. With the FES skills exam becoming part of the board certification process in general surgery, continual investigation is needed to determine the validity with which the exam is supporting inferences and decision-making about examinees, as well as how it might be improved.MethodsThe present study retrospectively analyzed performance and demographic details for the initial 344 examinees completing the FES skills exam.ResultsThe five tasks showed distinct degrees of difficulty, with Loop Reduction being especially difficult for examinees. Tasks related to one another positively but moderately, suggesting that the exam assesses both general and task-specific skills. The number of lower-endoscopic cases completed by an examinee strongly predicted performance, while upper endoscopy experience and career level (e.g., resident vs. fellow vs. practicing) did not. Hand dominance and the type of simulator used were not found to be related to scores. However, three demographic variables that related to one another—gender, glove size, and height—were also related to performance and pass/fail status.ConclusionsThis study’s results generally support the validity argument for the FES skills exam while pointing to additional investigations to be undertaken as the exam is applied more broadly.