Comparing the Ottawa Emergency Department Shift Observation Tool (O-EDShOT) to the traditional daily encounter card: measuring the quality of documented assessments.

Abstract

OBJECTIVES\nThe Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a workplace-based assessment designed to assess a trainee s performance across an entire shift. It was developed in response to validity concerns with traditional end-of-shift workplace-based assessments, such as the daily encounter card. The O-EDShOT previously demonstrated strong psychometric characteristics; however, it remains unknown whether the O-EDShOT facilitates measurable improvements in the quality of documented assessments compared to daily encounter cards.\n\n\nMETHODS\nThree randomly selected daily encounter cards and three O-EDShOTs completed by 24 faculty were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published 9-item quantitative measure of the quality of a completed workplace-based assessment. Automated-CCERR (A-CCERR) scores, which do not require raters, were also calculated. Paired sample t tests were conducted to compare the quality of assessments between O-EDShOTs and DECs as measured by the CCERR and A-CCERR.\n\n\nRESULTS\nCCERR scores were significantly higher for O-EDShOTs (mean(SD)\u2009=\u200925.6(2.6)) compared to daily encounter cards (21.5(3.9); t(23)\u2009=\u20095.2, p\u2009<\u20090.001, d\u2009=\u20091.1). A-CCERR scores were also significantly higher for O-EDShOTs (mean(SD)\u2009=\u200918.5(1.6)) than for daily encounter cards (15.5(1.2); t(24)\u2009=\u20098.4, p\u2009<\u20090.001). CCERR items 1, 4 and 9 were rated significantly higher for O-EDShOTs compared to daily encounter cards.\n\n\nCONCLUSIONS\nThe O-EDShOT yields higher quality documented assessments when compared to the traditional end-of-shift daily encounter card. Our results provide additional validity evidence for the O-EDShOT as an assessment tool for capturing trainee on-shift performance that can be used as a stimulus for actionable feedback and as a source for high-quality workplace-based assessment data to inform decisions about emergency medicine trainee progress and promotion.