Nikki L. Bibler Zaidi

Variance in attributes assessed by the multiple mini-interview

Abstract Introduction: While the extant literature has explored the impact of stations on multiple mini- interview (MMI) scores, the influence of station scenarios has been largely overlooked. Method: A subset of MMI scores was purposively sampled from admissions data at one US medical school. Generalizability (G) theory was used to estimate variance components attributable to applicants and two facets of generalization – scenarios, the content of the station, and items, the attributes assessed. Results: G study suggests that the greatest amount of variance is attributable to the main effect of the scenario (s) facet and the interaction between applicant and scenario (ps), which account for 77% of the total variance. The item facet (i) accounts for only 0.6% of total variance; likewise, the scenario-item interaction (si) accounts for only 1.4% of the total variance. Discussion: While the researchers expected to find a large variance component associated with the scenario-item interaction, this analysis does not support this assumption. The researchers interpret the small scenario-item interaction as a result of variance attributable to the item facet being subsumed by the variance attributable to the content of the scenarios. Conclusions: The results of this study reinforce the need to examine psychometric properties of the MMI.

Electronic health record as an educational intervention in medical error reduction

What problems were addressed? Safe and effective use of electronic health records (EHR) is a critical aspect of medical student education. Medical students are exposed to EHR user behaviours in practice that may include copying and pasting preexisting EHR data, over-reliance on EHR data as the ‘source of truth’, and other time-saving behaviours that may have unintended and unsafe consequences. These inappropriate user behaviours can result in medical documentation errors and increase the risks of patient harm. What was tried? In early 2017, we piloted an academic EHR environment to provide a real-world EHR experience to pre-clerkship medical students. Second-year medical students (n = 169) participated in a half-day session addressing core competencies in safe and ethical EHR use and common errors propagated in EHRs. The session included a handson exercise focused on reconciliation of patientprovided data not currently reflected in the EHR. Students were given a pre-recorded simulated patient encounter that introduced inconsistencies with the data presented to them in the EHR. They were then responsible for updating the EHR with the newly obtained data and removing the erroneous data. Students completed a voluntary survey immediately before (T1) and after (T2) the educational session, and then again halfway through the clerkship year after students had hands-on experience with other EHRs (T3). Students were asked to reflect what percentage of patient-specific data should come directly from the EHR as compared with patientreported information at T1, T2 and T3. What lessons were learned? Data from the survey and the reconciliation exercise were used to examine students’ perceptions on the source of information they feel is necessary to form clinical judgements. On average, only 50% of the discrepancies between the patient interview video and the EHR were appropriately updated in the EHR. A total of 60 students (36%) completed surveys at each time-point and were included in the analysis. Mean scores for the percentage of data that should come from the EHR versus the patient significantly differed across time-points (F (2, 118) = 5.759, p = 0.004). Post hoc analysis using Tukey’s Honest Significance Difference (HSD) revealed a significant reduction in percentage from T1 to T3 (T1 = 62.70 20.00; T3 = 54. 92 18.67) and T2 to T3 (T2 = 61.33 20.44; T3 = 54.92 18.67) at p < 0.05. These results illustrate that with exposure medical students’ perceptions shift to match behavior, with a higher reliance on EHR-provided data versus data derived directly from a patient encounter. Over-reliance on the EHR as the ‘source of truth’ comes from many causes; regardless, it is a behaviour that can propagate medical errors and contribute to a loss of patient centricity. Students and educators have a responsibility to thoroughly understand, and ameliorate, patterns of EHR use which can contribute to time saving but are potentially dangerous. For example, copying EHR data without reconciling veracity is a critical behaviour we consider inappropriate and worth addressing. This study illustrated that the best EHR practices taught to pre-clerkship students diminish over time. Addressing these issues must start early during the pre-clerkship experience but requires ongoing reinforcement over time.