Harley Baker

Online spaced education generates transfer and improves long-term retention of diagnostic skills: a randomized controlled trial.

BACKGROUND Retention of learning from surgical training is often limited, especially if the knowledge and skills are used infrequently. Using histopathology diagnostic skills as an experimental system, we compared knowledge transfer and retention between bolus Web-based teaching (WBT) modules and online spaced education, a novel email-based method of online education founded on the spacing effect. STUDY DESIGN All US urology residents were eligible to participate. Enrollees were randomized to 1 of 2 cohorts. Cohort 1 residents received 3 cycles/repetitions of spaced education on prostate-testis histopathology (weeks 1 to 16) and 3 WBT modules on bladder-kidney (weeks 14 to 16). Cohort 2 residents received 3 cycles of spaced education on bladder-kidney (weeks 1 to 16) and 3 WBT modules on prostate-testis (weeks 14 to 16). Each daily spaced education email presented a clinical scenario with histopathology image and asked for a diagnosis. Participants received immediate feedback after submitting their answers. Each cycle/repetition was 4 weeks long and consisted of 20 questions with unique images. WBT used the identical content and delivery system, with questions aggregated into three 20-question modules. Long-term retention of all 4 topics was assessed during weeks 18 to 45. RESULTS Seven-hundred and twenty-four urology residents enrolled. Spaced education and WBT were completed by 77% and 66% of residents, respectively. Spaced education and WBT generated mean long-term score increases of 15.2% (SD 15.3%) and 3.4% (SD 16.3%), respectively (p < 0.01). Spaced education increased long-term learning efficiency 4-fold. CONCLUSIONS Online spaced education generates transfer of histopathology diagnostic skills and substantially improves their long-term retention. Additional research is needed to determine how spaced education can optimize learning, transfer, and retention of surgical skills.

A Multi-institutional Randomized Controlled Trial of Adjuvant Web-based Teaching to Medical Students

Purpose To investigate the impact of an adjuvant Web-based teaching program on medical students’ learning during clinical rotations. Method From April 2003 to May 2004, 351 students completing clinical rotations in surgery–urology at four U.S. medical schools were invited to volunteer for the study. Web-based teaching cases were developed covering four core urologic topics. Students were block randomized to receive Web-based teaching on two of the four topics. Before and after a designated duration at each institution (ranging one to three weeks), students completed a validated 28-item Web-based test (Cronbach’s alpha = .76) covering all four topics. The test was also administered to a subset of students at one school at the conclusion of their third-year to measure long-term learning. Results Eighty-one percent of all eligible students (286/351) volunteered to participate in the study, 73% of whom (210/286) completed the Web-based program. Compared to controls, Web-based teaching significantly increased test scores in the four topics at each medical school (p < .001, mixed analysis of variance), corresponding to a Cohen’s d effect size of 1.52 (95% confidence interval [CI], 1.23–1.80). Learning efficiency was increased three-fold by Web-based teaching (Cohen’s d effect size 1.16; 95% CI 1.13–1.19). Students who were tested a median of 4.8 months later demonstrated significantly higher scores for Web-based teaching compared to non-Web-based teaching (p = .007, paired t-test). Limited learning was noted in the absence of Web-based teaching. Conclusions This randomized controlled trial provides Class I evidence that Web-based teaching as an adjunct to clinical experiences can significantly and durably improve medical students’ learning.

An Online Spaced-Education Game to Teach and Assess Medical Students: A Multi- Institutional Prospective Trial

Purpose To investigate whether a spaced-education (SE) game can be an effective means of teaching core content to medical students and a reliable and valid method of assessing their knowledge. Method This nine-month trial (2008–2009) enrolled students from three U.S. medical schools. The SE game consisted of 100 validated multiple-choice questions–explanations in preclinical/clinical domains. Students were e-mailed two questions daily. Adaptive game mechanics re-sent questions in three or six weeks if answered, respectively, incorrectly or correctly. Questions expired if not answered on time (appointment dynamic). Students retired questions by answering each correctly twice consecutively (progression dynamic). Posting of relative performance fostered competition. Main outcome measures were baseline and completion scores. Results Seven-hundred thirty-one students enrolled. Median baseline score was 53% (interquartile range [IQR] 16) and varied significantly by year (P < .001, dmax = 2.08), school (P < .001, dmax = 0.75), and gender (P < .001, d = 0.38). Median completion score was 93% (IQR 12) and varied significantly by year (P = .001, dmax = 1.12), school (P < .001, dmax = 0.34), and age (P = .019, dmax = 0.43). Scores did not differ significantly between years 3 and 4. Seventy percent of enrollees (513/731) requested to participate in future SE games. Conclusions An SE game is an effective and well-accepted means of teaching core content and a reliable and valid method to assess student knowledge. SE games may be valuable tools to identify and remediate students who could benefit from additional educational support.

Online "spaced education progress-testing" of students to confront two upcoming challenges to medical schools.

Purpose U.S. medical students will soon complete only one licensure examination sequence, given near the end of medical school. Thus, schools are challenged to identify poorly performing students before this high-stakes test and help them retain knowledge across the duration of medical school. The authors investigated whether online spaced education progress-testing (SEPT) could achieve both aims. Method Participants were 2,648 students from four U.S. medical schools; 120 multiple-choice questions and explanations in preclinical and clinical domains were developed and validated. For 34 weeks, students randomized to longitudinal progress-testing alone (LPTA) received four new questions (with answers/ explanations) each week. Students randomized to SEPT received the identical four questions each week, plus two-week and six-week cycled reviews of the questions/explanations. During weeks 31–34, the initial 40 questions were re-sent to students to assess longer-term retention. Results Of the 1,067 students enrolled, the 120-question progress-test was completed by 446 (84%) and 392 (74%) of the LPTA and SEPT students, respectively. Cronbach alpha reliability was 0.87. Scores were 39.9%, 51.9%, 58.7%, and 58.8% for students in years 1–4, respectively. Performance correlated with Step 1 and Step 2 Clinical Knowledge scores (r = 0.52 and 0.57, respectively; P < .001) and prospectively identified students scoring below the mean on Step 1 with 75% sensitivity, 77% specificity, and 41% positive predictive value. Cycled reviews generated a 170% increase in learning retention relative to baseline (P < .001, effect size 0.95). Conclusions SEPT can identify poorly performing students and improve their longer-term knowledge retention.

Do Chief Resident Scores on the In-Service Examination Predict Their Performance on the American Board of Urology Qualifying Examination?

PURPOSE The American Urological Association In-Service Examination and the American Board of Urology Qualifying Examination are written multiple choice tests that cover all domains in urology. We investigated whether In-Service Examination performance could identify chief residents who scored in the lowest quartile on the Qualifying Examination. MATERIALS AND METHODS All urology chief residents in the United States and Canada in 2008 and 2009 were eligible to participate in this study. In-Service Examination 2008 and Qualifying Examination 2009 performance data were obtained from the American Urological Association and American Board of Urology, respectively. Data were analyzed with the Pearson correlation and descriptive statistics. RESULTS Of the 257 American and Canadian chief residents who completed the Qualifying Examination 2009, 194 (75%) enrolled in this study and were included in analysis. Overall In-Service Examination 2008 scores correlated significantly with Qualifying Examination 2009 scores (r=0.55, p<0.001), accounting for 30% of score variance. Substantial variability in In-Service Examination-Qualifying Examination rankings was notable among individual residents. An In-Service Examination 2008 cutoff percentile rank of 40% identified chief residents in the lowest quartile on the Qualifying Examination 2009 with 71% sensitivity, 77% specificity, and a likelihood ratio of 3.1 and 0.4 (positive and negative likelihood ratios, respectively). CONCLUSIONS The substantial variability of In-Service Examination-Qualifying Examination performance among individual chief residents limits In-Service Examination predictive utility. A single In-Service Examination score should not be used to make a high stakes judgment about an individual resident. In-Service Examination scores should be used as 1 part of an overall evaluation program to prospectively identify residents who could benefit from additional educational support.