Debbie Jaarsma

Programmatic assessment of competency-based workplace learning: when theory meets practice

BackgroundIn competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment. Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice.MethodsIn a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned.ResultsThe programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges.ConclusionsA programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.

Clarifying students' feedback-seeking behaviour in clinical clerkships

Context  Why and how do students seek feedback on their performance in the clinical workplace and which factors influence this? These questions have remained largely unanswered in research into workplace learning during clinical clerkships. Research on feedback has focused mainly on feedback providers. Whether and how feedback recipients actively seek feedback are under‐examined issues. Research in organisational psychology has proposed a mechanism whereby feedback seeking is influenced by motives and goal orientation mediated by the perceived costs and benefits of feedback. Building on a recently published model of resident doctors’ feedback‐seeking behaviour, we conducted a qualitative study to explore students’ feedback‐seeking behaviours in the clinical workplace.

Development and validation of a competency framework for veterinarians

Changing demands from society and the veterinary profession call for veterinary medical curricula that can deliver veterinarians who are able to integrate specific and generic competencies in their professional practice. This requires educational innovation directed by an integrative veterinary competency framework to guide curriculum development. Given the paucity of relevant information from the veterinary literature, a qualitative multi-method study was conducted to develop and validate such a framework. A competency framework was developed based on the analysis of focus group interviews with 54 recently graduated veterinarians and clients and subsequently validated in a Delphi procedure with a panel of 29 experts, representing the full range and diversity of the veterinary profession. The study resulted in an integrated competency framework for veterinary professionals, which consists of 16 competencies organized in seven domains: veterinary expertise, communication, collaboration, entrepreneurship, health and welfare, scholarship, and personal development. Training veterinarians who are able to use and integrate the seven domains in their professional practice is an important challenge for todays veterinary medical schools. The Veterinary Professional (VetPro) framework provides a sound empirical basis for the ongoing debate about the direction of veterinary education and curriculum development.

Peer group reflection helps clinical teachers to critically reflect on their teaching

Background: Student evaluations can help clinical teachers to reflect on their teaching skills and find ways to improve their teaching. Studies have shown that the mere presentation of student evaluations is not a sufficient incentive for teachers to critically reflect on their teaching. Aim: We evaluated and compared the effectiveness of two feedback facilitation strategies that were identical except for a peer reflection meeting. Method: In this study, 54 clinical teachers were randomly assigned to two feedback strategies. In one strategy, a peer reflection was added as an additional step. All teachers completed a questionnaire evaluating the strategy that they had experienced. We analysed the reflection reports and the evaluation questionnaire. Results: Both strategies stimulated teachers to reflect on feedback and formulate alternative actions for their teaching practice. The teachers who had participated in the peer reflection meeting showed deeper critical reflection and more concrete plans to change their teaching. All feedback strategies were considered effective by the majority of the teachers. Conclusions: Strategies with student feedback and self-assessment stimulated reflection on teaching and helped clinical teachers to formulate plans for improvement. A peer reflection meeting seemed to enhance reflection quality. Further research should establish whether it can have lasting effects on teaching quality.

Feedback-giving behaviour in performance evaluations during clinical clerkships

Abstract Context: Narrative feedback documented in performance evaluations by the teacher, i.e. the clinical supervisor, is generally accepted to be essential for workplace learning. Many studies have examined factors of influence on the usage of mini-clinical evaluation exercise (mini-CEX) instruments and provision of feedback, but little is known about how these factors influence teachers’ feedback-giving behaviour. In this study, we investigated teachers’ use of mini-CEX in performance evaluations to provide narrative feedback in undergraduate clinical training. Methods: We designed an exploratory qualitative study using an interpretive approach. Focusing on the usage of mini-CEX instruments in clinical training, we conducted semi-structured interviews to explore teachers’ perceptions. Between February and June 2013, we conducted interviews with 14 clinicians participated as teachers during undergraduate clinical clerkships. Informed by concepts from the literature, we coded interview transcripts and iteratively reduced and displayed data using template analysis. Results: We identified three main themes of interrelated factors that influenced teachers’ practice with regard to mini-CEX instruments: teacher-related factors; teacher–student interaction-related factors, and teacher–context interaction-related factors. Four issues (direct observation, relationship between teacher and student, verbal versus written feedback, formative versus summative purposes) that are pertinent to workplace-based performance evaluations were presented to clarify how different factors interact with each other and influence teachers’ feedback-giving behaviour. Embedding performance observation in clinical practice and establishing trustworthy teacher–student relationships in more longitudinal clinical clerkships were considered important in creating a learning environment that supports and facilitates the feedback exchange. Conclusion: Teachers’ feedback-giving behaviour within the clinical context results from the interaction between personal, interpersonal and contextual factors. Increasing insight into how teachers use mini-CEX instruments in daily practice may offer strategies for creating a professional learning culture in which feedback giving and seeking would be enhanced.

A framework for the ethics review of education research

Health professions education is a rapidly growing domain of research. Along with this rapid growth comes an increased concern for quality. One element of quality refers to the ethical conduct of researchers towards research participants. Recently, editors of major medical education journals formulated criteria for research ethics, including the condition that experimental research must be approved by an ethics committee if one is available. In some countries, including the Netherlands, health professions education research is not habitually reviewed, usually because it is not recognised policy for ethics review boards (ERBs) to include such research in their considerations. However, in other countries, ERBs for biomedical research do accept education research projects for review. Legislation governing ERBs usually reflects the Nuremberg Code and the Declaration of Helsinki, which were designed to ensure the protection of patients in medical research and were not specifically intended for application in education research. Education research bodies, such as the American Educational Research Association (AERA) Janet and the Australian Association for Research in Education (AARE), have developed codes of research conduct, but in the medical domain much more stringent and legal procedures apply.

Critically reflective work behavior of health care professionals

Introduction: Better understanding of critically reflective work behavior (CRWB), an approach for work‐related informal learning, is important in order to gain more profound insight in the continuing development of health care professionals. Methods: A survey, developed to measure CRWB and its predictors, was distributed to veterinary professionals. The authors specified a model relating CRWB to a Perceived Need for Lifelong Learning, Perceived Workload, and Opportunities for Feedback. Furthermore, research utilization was added to the concept of CRWB. The model was tested against the data, using structural equation modeling (SEM). Results: The model was well represented by the data. Four factors that reflect aspects of CRWB were distinguished: (1) individual CRWB; (2) being critical in interactions with others; (3) cross‐checking of information; and (4) openness to new findings. The latter 2 originated from the factor research utilization in CRWB. The Perceived Need for Lifelong Learning predicts CRWB. Neither Perceived Workload nor Opportunities for Feedback of other practitioners was related to CRWB. Discussion: The results suggest that research utilization, such as cross‐checking information and openness to new findings, is essential for CRWB. Furthermore, perceptions of the need for lifelong learning are more relevant for CRWB of health care professionals than qualities of the workplace.

Teachers’ perceptions of aspects affecting seminar learning: a qualitative study

BackgroundMany medical schools have embraced small group learning methods in their undergraduate curricula. Given increasing financial constraints on universities, active learning groups like seminars (with 25 students a group) are gaining popularity. To enhance the understanding of seminar learning and to determine how seminar learning can be optimised it is important to investigate stakeholders’ views. In this study, we qualitatively explored the views of teachers on aspects affecting seminar learning.MethodsTwenty-four teachers with experience in facilitating seminars in a three-year bachelor curriculum participated in semi-structured focus group interviews. Three focus groups met twice with an interval of two weeks led by one moderator. Sessions were audio taped, transcribed verbatim and independently coded by two researchers using thematic analysis. An iterative process of data reduction resulted in emerging aspects that influence seminar learning.ResultsTeachers identified seven key aspects affecting seminar learning: the seminar teacher, students, preparation, group functioning, seminar goals and content, course coherence and schedule and facilities. Important components of these aspects were: the teachers’ role in developing seminars (‘ownership’), the amount and quality of preparation materials, a non-threatening learning climate, continuity of group composition, suitability of subjects for seminar teaching, the number and quality of seminar questions, and alignment of different course activities.ConclusionsThe results of this study contribute to the unravelling of the ‘the black box’ of seminar learning. Suggestions for ways to optimise active learning in seminars are made regarding curriculum development, seminar content, quality assurance and faculty development.

A retrospective analysis of veterinary medical curriculum development in The Netherlands

Over the past two decades, the Faculty of Veterinary Medicine of Utrecht University (FVMU) has introduced major curriculum changes to keep pace with modern veterinary educational developments worldwide. Changes to program outcomes have been proposed according to professional and societal demands, with more attention paid to generic competencies and electives and species/sector differentiation. Furthermore, changes in educational approaches and the educational organization have been proposed, aiming at a transition from teacher-centered education toward more student-centered education. Curriculum development is a complex and difficult process, with many elements interacting. For a new curriculum to become valid, curriculum elements and their interrelation-such as statements of intent (also called outcomes, goals, or objectives), content, teaching and learning strategies, assessment strategies, and context-need to be addressed in the educational philosophy (i.e., the intended curriculum). This paper describes a document analysis of the major curriculum reforms of the FVMU. Curriculum committee reports were critically analyzed to gain insight into the intentions of the curriculum designers and the match between the curriculum elements, as described by Prideaux. The results show that the reports paid considerable attention to generic competency training, especially to academic training, and to the introduction of more student-centered teaching and learning strategies. However, little attention was paid to assessment strategies and the statements of intent were defined rather broadly. Curriculum evaluation (i.e., what is delivered to the students and how is the curriculum experienced) is needed at all curriculum levels. Possible mismatches between levels need to be identified.

Determinants of Adherence to Treatment in Hypertensive Patients of African Descent and the Role of Culturally Appropriate Education

Background In Western countries, better knowledge about patient-related determinants of treatment adherence (medication and lifestyle) is needed to improve treatment adherence and outcomes among hypertensive ethnic minority patients of African descent. Objective To identify patient-related determinants of adherence to lifestyle and medication recommendations among hypertensive African Surinamese and Ghanaian patients with suboptimal treatment results (SBP≥140) living in the Netherlands and how culturally appropriate hypertension education (CAHE) influenced those determinants. Methods This study analysed data of 139 patients who participated in the CAHE trial. Univariate logistic regression analysis was used to measure the association between patient-related determinants (medication self-efficacy, beliefs about medication and hypertension, social support, and satisfaction with care) and treatment adherence. We also tested whether CAHE influenced the determinants. Results Medication self-efficacy and social support were associated with medication adherence at baseline. At six months, more medication self-efficacy and fewer concerns about medication use were associated with improved medication adherence. Self-efficacy was also associated with adherence to lifestyle recommendations at baseline. CAHE influenced patients’ illness perceptions by creating more understanding of hypertension, its chronic character, and more concerns about the associated risks. Conclusion In this high-risk population, health care providers can support medication adherence by paying attention to patients’ medication self-efficacy, the concerns they may have about medication use and patients’ perceptions on hypertension. The CAHE intervention improved patients’ perception on hypertension.