Marieke van der Schaaf

Curriculum development for the workplace using Entrustable Professional Activities (EPAs) : AMEE Guide No. 99

Abstract This Guide was written to support educators interested in building a competency-based workplace curriculum. It aims to provide an up-to-date overview of the literature on Entrustable Professional Activities (EPAs), supplemented with suggestions for practical application to curriculum construction, assessment and educational technology. The Guide first introduces concepts and definitions related to EPAs and then guidance for their identification, elaboration and validation, while clarifying common misunderstandings about EPAs. A matrix-mapping approach of combining EPAs with competencies is discussed, and related to existing concepts such as competency milestones. A specific section is devoted to entrustment decision-making as an inextricable part of working with EPAs. In using EPAs, assessment in the workplace is translated to entrustment decision-making for designated levels of permitted autonomy, ranging from acting under full supervision to providing supervision to a junior learner. A final section is devoted to the use of technology, including mobile devices and electronic portfolios to support feedback to trainees about their progress and to support entrustment decision-making by programme directors or clinical teams.

Vertical integration in medical school: effect on the transition to postgraduate training

Medical Education 2010: 44 : 272–279

Graduates from vertically integrated curricula

Background:  Vertical integration (VI) has been recommended as an undergraduate medical curriculum structure that fosters the transition to postgraduate training. Our definition of VI includes: (1) the provision of early clinical experience; (2) the integration of biomedical sciences with clinical cases; (3) long clerkships during the final year; and (4) increasing levels of clinical responsibility for students. The aim of the current study is to support the hypothesis that medical graduates from VI programmes meet the expectations of postgraduate supervisors better than those from non‐VI curricula.

Essential facets of competence that enable trust in medical graduates: a ranking study among physician educators in two countries

Abstract One way to operationalize the assessment of trainees in a competency-based context is to determine whether they can be entrusted with critical activities. To determine which facets of competence (FOCs) are most informative for such decisions, we performed a Delphi study among Dutch educators. In the current study, the resulting list of facets of competence was evaluated among experienced Dutch and German clinical educators to determine which facets appear most relevant and to evaluate the agreement among experts in different countries as a support for their external validity. Eight Dutch and eight German experts scored each FOC on a five-point scale for relevance. A rank-order comparison showed that there was almost full agreement about the top 10 FOCs, among which ‘Scientific and empirical grounded method of working’, ‘Knowing and maintaining own personal bounds and possibilities’, ‘Active professional development’, ‘Teamwork and collegiality’, ‘Active listening to patients’, and ‘Verbal communication with colleagues and supervisors’. We conclude that these facets of competence may be used in a training for educators who need to make entrustment decisions about trainees.

Construct validation of progress testing to measure knowledge and visual skills in radiology

Background: The Dutch Radiology Progress Test (DRPT) monitors the acquisition of knowledge and visual skills of radiology residents in the Netherlands. Aim: We aimed to evaluate the quality of progress testing in postgraduate radiology training by studying the reliability of the DRPT and finding an indication for its construct validity. We expected that knowledge would increase rapidly in the first years of residency, leveling-off in later years, to allow for the development of visual skills. We hypothesized that scores on the DRPT reflect this pattern. Methods: Internal consistencies were estimated with Cronbachs alpha. Performance increase over program years were tested using one-way analysis of variance. Results: Data were available for 498 residents (2281 test results). Reliabilities were around Cronbachs alpha 0.90. There was a significant difference in the mean test results between the first three years of residency. After the fourth year no significant increase in test scores on knowledge could be measured on eight tests. The same pattern occurred for scores on visual skills. However, visual skills scores tend to increase more sharply than knowledge scores. Conclusion: We found support for the reliability and construct validity of the DRPT. However, assessment on visual skill development needs further exploration.

Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: A Delphi study.

BACKGROUND True competency-based medical education should produce graduates meeting fixed standards of competence. Current postgraduate programmes are usually based on a pre-determined length of time in training making them poorly suited for an individual approach. The concept of entrustable professional activities (EPAs) enables a more flexible, personalised and less time-dependent approach to training programmes. An EPA is a unit of professional practice, to be entrusted to a trainee to execute without supervision once they demonstrate sufficient competence. As EPAs relate competencies to clinical practice, they structure training and assessment more logically according to the way clinicians actually work. A first step in building an EPA-based curriculum is to identify the core EPAs of the profession. OBJECTIVES The aim of this study was to identify EPAs for postgraduate training in anaesthesiology and to provide an example of how an existing curriculum can be transformed into an EPA-based curriculum. DESIGN A modified Delphi method was used as a consensus approach applying three Delphi rounds. SETTING Postgraduate specialty training in anaesthesiology in the Netherlands. PARTICIPANTS All programme directors in anaesthesiology in the Netherlands except for a single programme director who was involved as a researcher in this study and could not participate. MAIN OUTCOME MEASURES Agreement among participants on a list of EPAs. Agreement was specified as a consensus rate of more than 80%. RESULTS In this study, 27 programme directors (69% overall response rate) reached consensus on a set of 45 EPAs that describe a curriculum in anaesthesiology for the Netherlands. CONCLUSION This study is a first step toward a more contemporary curriculum in competency-based postgraduate anaesthesiology training.

Volumetric and two-dimensional image interpretation show different cognitive processes in learners

RATIONALE AND OBJECTIVES In current practice, radiologists interpret digital images, including a substantial amount of volumetric images. We hypothesized that interpretation of a stack of a volumetric data set demands different skills than interpretation of two-dimensional (2D) cross-sectional images. This study aimed to investigate and compare knowledge and skills used for interpretation of volumetric versus 2D images. MATERIALS AND METHODS Twenty radiology clerks were asked to think out loud while reading four or five volumetric computed tomography (CT) images in stack mode and four or five 2D CT images. Cases were presented in a digital testing program allowing stack viewing of volumetric data sets and changing views and window settings. Thoughts verbalized by the participants were registered and coded by a framework of knowledge and skills concerning three components: perception, analysis, and synthesis. The components were subdivided into 16 discrete knowledge and skill elements. A within-subject analysis was performed to compare cognitive processes during volumetric image readings versus 2D cross-sectional image readings. RESULTS Most utterances contained knowledge and skills concerning perception (46%). A smaller part involved synthesis (31%) and analysis (23%). More utterances regarded perception in volumetric image interpretation than in 2D image interpretation (Median 48% vs 35%; z = -3.9; P < .001). Synthesis was less prominent in volumetric than in 2D image interpretation (Median 28% vs 42%; z = -3.9; P < .001). No differences were found in analysis utterances. CONCLUSIONS Cognitive processes in volumetric and 2D cross-sectional image interpretation differ substantially. Volumetric image interpretation draws predominantly on perceptual processes, whereas 2D image interpretation is mainly characterized by synthesis. The results encourage the use of volumetric images for teaching and testing perceptual skills.

Exploring the role of assessment criteria during teachers’ collaborative judgement processes of students’ portfolios

Student portfolios are increasingly used for assessing student competences in higher education, but results about the construct validity of portfolio assessment are mixed. A prerequisite for construct validity is that the portfolio assessment is based on relevant portfolio content. Assessment criteria, are often used to enhance this condition. This study aims to identify whether assessment criteria can improve content, argumentation and communication during teacher moderation while judging student portfolios. Six teachers scored 32 student portfolios in dyads with and without assessment criteria. Their judgement processes were qualitatively analysed. Results indicated that the quality of their judgement processes was low, since teachers based their judgements mainly on their own personal opinion and less on evidence found in the portfolio. Teachers barely paid attention to quality checks and easily agreed with each other. When teachers used assessment criteria, the quality of their judgements slightly improved. They based their judgements more on relevant evidence, used less personal experiences and more often checked the quality of their judgement processes. It is concluded that the quality of teacher portfolio judgement is low, and that the use of assessment criteria can enhance its quality.

The influence of a vertically integrated curriculum on the transition to postgraduate training.

Background: Recently, many medical curricula have been changed into vertically integrated programmes. One of the aims of vertical integration is to facilitate the transition from theoretical to clinical education and from medical school to postgraduate training. Aims: The aim of this study was to determine whether a vertically integrated curriculum affects the transition from medical school to postgraduate training. Method: We carried out a survey study among graduates of two cohorts of the Utrecht Medical School, who followed either the traditional or the innovative, vertically integrated, curriculum. Topics of the questionnaire were: (a) activities since medical school, (b) required amount of time and number of applications to get admitted to residency, (c) the process of making career choices. Results: Graduates from the vertically integrated curriculum had made their definite career choice earlier compared to those who followed a traditional programme. Graduates of the new curriculum also needed less time and fewer applications to obtain a residency position. Conclusions: A vertically integrated curriculum at medical school positively affects the transition to postgraduate training. Additional research, among a larger population, is required to determine which components of the curriculum cause this effect and to specify under which conditions these effect occurs.

The impact of various transitions in the medical education continuum on perceived readiness of trainees to be entrusted with professional tasks

Background: Medical trainees go through various transitions during the medical education continuum. Aims: The aim of this study was to understand how transitions in licensure and increased responsibility may affect trainees’ competence development. Method: We carried out a questionnaire study in Leeds (UK). Trainees and supervisors were asked to determine the trainees competence at different stages of training to carry out a diversity of medical activities on a five-point Likert scale. The questionnaires were completed by final-year medical students, trainees of foundation year 1 (FY1) and 2 (FY2), and their supervisors. Results: For all activities listed: (1) the trainees’ presumed competence increased over time and (2) the mean scores given by trainees were significantly higher than the mean scores given by supervisors across all years the questionnaire covered. As estimated by both trainees and supervisors, the impact of a responsibility transition, i.e., from medical school to FY1, is larger than that of a licensure transition, i.e., from FY1 (pre-MD) to FY2 (post-MD). Conclusions: The transition to formal responsibility with a license to practice medicine seems to have less impact on confidence of trainees and their supervisors to execute critical activities than the transition to informal but significantly higher responsibility.