Gut | 2019
PWE-112\u2005Ercp dops assessments: evidence of validity and competency development during training
Abstract
Introduction Formative direct observation of procedural skills (DOPS) in ERCP consist of 27 assessable items located within 7 domains and an overall competence rating. Despite their implementation in 2016, validity evidence remain lacking. We aimed to evaluate DOPS scores to appraise validity and competency development during ERCP training. Methods This prospective UK-wide study analysed ERCP DOPS submitted to the JETS e-portfolio between July 2016-October 2018. Reliability was measured using Cronbach’s alpha. DOPS scores were benchmarked using the contrasting groups method to establish consequential validity. The percentage of competent scores were averaged for each item, domain and overall rating and stratified by lifetime procedure count to plot learning curves and provide discriminative validity. Multivariable generalising estimating equations were performed to identify trainee-level predictors of overall procedural competence. Results In total, 818 DOPS submitted from 80 UK centres were analysed. DOPS were completed for 109 trainees (ST–: 26%, ST6: 22%, ST–: 21%, research fellow: 7%, consultant/associate specialist: 24%). 5 items were unassessed in >50% of DOPS and were excluded from Cronbach’s alpha analyses; assessment of the remaining 22 items yielded a statistic of 0.95, indicating high reliability. Attaining competency in 87% of assessed items per DOPS provided the optimal benchmark score (false positive: 10%; false negative: 2%). Competency acquisition occurred in the domain sequence of: ‘pre-procedural’, ‘intubation and positioning’, ‘post-procedural’, ‘endoscopic non-technical skills’, ‘execution of selected therapy’, and ‘cannulation and imaging’ (figure 1). Trainees surpassed the 87% competency threshold after 20–49 procedures (mean: 89%). After 300 procedures, the competency threshold was reached for ‘selective cannulation’ (89%), but not reached for the items of: stenting (plastic: 73%; metal: 70%), sphincterotomy (80%) and sphincteroplasty (56%). On multivariable analysis, lifetime procedure count (P<0.001), easier case difficulty (P<0.001) and lifetime DOPS count >10 (P=0.002) predicted overall procedural competence, but not trainee specialty (P=0.525), grade (P=0.076) or prior gastroscopy certification (P=0.886).Abstract PWE-112 Figure 1 Conclusion This study provides novel validity, reliability and learning curve data in support of ERCP DOPS. Whilst competency in the majority of DOPS items may be attained after 20–49 procedures, this may still be insufficient for selective cannulation and higher-level therapeutic competencies.