Peter van Beukelen

Programmatic assessment of competency-based workplace learning: when theory meets practice

BackgroundIn competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment. Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice.MethodsIn a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned.ResultsThe programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges.ConclusionsA programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.

Clarifying students' feedback-seeking behaviour in clinical clerkships

Context  Why and how do students seek feedback on their performance in the clinical workplace and which factors influence this? These questions have remained largely unanswered in research into workplace learning during clinical clerkships. Research on feedback has focused mainly on feedback providers. Whether and how feedback recipients actively seek feedback are under‐examined issues. Research in organisational psychology has proposed a mechanism whereby feedback seeking is influenced by motives and goal orientation mediated by the perceived costs and benefits of feedback. Building on a recently published model of resident doctors’ feedback‐seeking behaviour, we conducted a qualitative study to explore students’ feedback‐seeking behaviours in the clinical workplace.

Development and validation of a competency framework for veterinarians

Changing demands from society and the veterinary profession call for veterinary medical curricula that can deliver veterinarians who are able to integrate specific and generic competencies in their professional practice. This requires educational innovation directed by an integrative veterinary competency framework to guide curriculum development. Given the paucity of relevant information from the veterinary literature, a qualitative multi-method study was conducted to develop and validate such a framework. A competency framework was developed based on the analysis of focus group interviews with 54 recently graduated veterinarians and clients and subsequently validated in a Delphi procedure with a panel of 29 experts, representing the full range and diversity of the veterinary profession. The study resulted in an integrated competency framework for veterinary professionals, which consists of 16 competencies organized in seven domains: veterinary expertise, communication, collaboration, entrepreneurship, health and welfare, scholarship, and personal development. Training veterinarians who are able to use and integrate the seven domains in their professional practice is an important challenge for todays veterinary medical schools. The Veterinary Professional (VetPro) framework provides a sound empirical basis for the ongoing debate about the direction of veterinary education and curriculum development.

Perceptions of learning as a function of seminar group factors

Context  Small‐group learning is advocated for enhancing higher‐order thinking and the development of skills and attitudes. Teacher performance, group interaction and the quality of assignments have been shown to affect small‐group learning in hybrid and problem‐based curricula.

Peer group reflection helps clinical teachers to critically reflect on their teaching

Background: Student evaluations can help clinical teachers to reflect on their teaching skills and find ways to improve their teaching. Studies have shown that the mere presentation of student evaluations is not a sufficient incentive for teachers to critically reflect on their teaching. Aim: We evaluated and compared the effectiveness of two feedback facilitation strategies that were identical except for a peer reflection meeting. Method: In this study, 54 clinical teachers were randomly assigned to two feedback strategies. In one strategy, a peer reflection was added as an additional step. All teachers completed a questionnaire evaluating the strategy that they had experienced. We analysed the reflection reports and the evaluation questionnaire. Results: Both strategies stimulated teachers to reflect on feedback and formulate alternative actions for their teaching practice. The teachers who had participated in the peer reflection meeting showed deeper critical reflection and more concrete plans to change their teaching. All feedback strategies were considered effective by the majority of the teachers. Conclusions: Strategies with student feedback and self-assessment stimulated reflection on teaching and helped clinical teachers to formulate plans for improvement. A peer reflection meeting seemed to enhance reflection quality. Further research should establish whether it can have lasting effects on teaching quality.

Critically reflective dialogues in learning communities of professionals

Communities in which professionals share and create knowledge potentially support their continued learning. To realise this potential more fully, members are required to reflect critically. For learning at work such behaviour has been described as critically reflective work behaviour, consisting of six aspects: challenging groupthink, critical opinion sharing, an openness about mistakes, asking for and giving feedback, experimentation and research utilisation. We studied whether and how these aspects can be distinguished in dialogues of seven different communities of veterinary professionals (critically reflective dialogues). Our exploration of the nature of critically reflective dialogues resulted in an analytical framework. Within each aspect four different modes of communication were identified: interactive, on an individual basis, non-reflective and restricted. We assume that professionals use learning opportunities most in the interactive mode of communication. The framework was employed to study the extent to which dialogues showed these modes of critically reflective dialogues. The results demonstrate that in these communities the modes of communication within aspects were largely non-interactive (i.e., individual, non-reflective). The developed framework discriminates between communities in terms of their critically reflective dialogues. Interventions to improve the effectiveness of learning communities should focus on enhancement of members addressing each others reasons and reflections.

Feedback-giving behaviour in performance evaluations during clinical clerkships

Abstract Context: Narrative feedback documented in performance evaluations by the teacher, i.e. the clinical supervisor, is generally accepted to be essential for workplace learning. Many studies have examined factors of influence on the usage of mini-clinical evaluation exercise (mini-CEX) instruments and provision of feedback, but little is known about how these factors influence teachers’ feedback-giving behaviour. In this study, we investigated teachers’ use of mini-CEX in performance evaluations to provide narrative feedback in undergraduate clinical training. Methods: We designed an exploratory qualitative study using an interpretive approach. Focusing on the usage of mini-CEX instruments in clinical training, we conducted semi-structured interviews to explore teachers’ perceptions. Between February and June 2013, we conducted interviews with 14 clinicians participated as teachers during undergraduate clinical clerkships. Informed by concepts from the literature, we coded interview transcripts and iteratively reduced and displayed data using template analysis. Results: We identified three main themes of interrelated factors that influenced teachers’ practice with regard to mini-CEX instruments: teacher-related factors; teacher–student interaction-related factors, and teacher–context interaction-related factors. Four issues (direct observation, relationship between teacher and student, verbal versus written feedback, formative versus summative purposes) that are pertinent to workplace-based performance evaluations were presented to clarify how different factors interact with each other and influence teachers’ feedback-giving behaviour. Embedding performance observation in clinical practice and establishing trustworthy teacher–student relationships in more longitudinal clinical clerkships were considered important in creating a learning environment that supports and facilitates the feedback exchange. Conclusion: Teachers’ feedback-giving behaviour within the clinical context results from the interaction between personal, interpersonal and contextual factors. Increasing insight into how teachers use mini-CEX instruments in daily practice may offer strategies for creating a professional learning culture in which feedback giving and seeking would be enhanced.

Assessment of competence in clinical reasoning and decision‐making under uncertainty: the script concordance test method

Real‐life, complex problems often require that decisions are made despite limited information or insufficient time to explore all relevant aspects. Incorporating authentic uncertainties into an assessment, however, poses problems in establishing results and analysing their methodological qualities. This study aims at developing a test on clinical decision‐making in veterinary medicine and establishing its reliability and validity. The test is based on the script concordance test method and covers a large sample of authentic cases and uncertainties. The answer key was compiled with reference to the professional judgements and decisions of a panel of experienced practitioners. From a substantive appraisal of the cases and items, the analysis of the test results and the responses from the experienced practitioners, it is concluded that this test validly represents the problems, decisions and uncertainties of clinical practice. In spite of the hindrances caused by the uncertainties included in the test, the reliability and validity of the test and its results could be evaluated and proved to meet measurement criteria.

Critically reflective work behavior of health care professionals

Introduction: Better understanding of critically reflective work behavior (CRWB), an approach for work‐related informal learning, is important in order to gain more profound insight in the continuing development of health care professionals. Methods: A survey, developed to measure CRWB and its predictors, was distributed to veterinary professionals. The authors specified a model relating CRWB to a Perceived Need for Lifelong Learning, Perceived Workload, and Opportunities for Feedback. Furthermore, research utilization was added to the concept of CRWB. The model was tested against the data, using structural equation modeling (SEM). Results: The model was well represented by the data. Four factors that reflect aspects of CRWB were distinguished: (1) individual CRWB; (2) being critical in interactions with others; (3) cross‐checking of information; and (4) openness to new findings. The latter 2 originated from the factor research utilization in CRWB. The Perceived Need for Lifelong Learning predicts CRWB. Neither Perceived Workload nor Opportunities for Feedback of other practitioners was related to CRWB. Discussion: The results suggest that research utilization, such as cross‐checking information and openness to new findings, is essential for CRWB. Furthermore, perceptions of the need for lifelong learning are more relevant for CRWB of health care professionals than qualities of the workplace.

Teachers’ perceptions of aspects affecting seminar learning: a qualitative study

BackgroundMany medical schools have embraced small group learning methods in their undergraduate curricula. Given increasing financial constraints on universities, active learning groups like seminars (with 25 students a group) are gaining popularity. To enhance the understanding of seminar learning and to determine how seminar learning can be optimised it is important to investigate stakeholders’ views. In this study, we qualitatively explored the views of teachers on aspects affecting seminar learning.MethodsTwenty-four teachers with experience in facilitating seminars in a three-year bachelor curriculum participated in semi-structured focus group interviews. Three focus groups met twice with an interval of two weeks led by one moderator. Sessions were audio taped, transcribed verbatim and independently coded by two researchers using thematic analysis. An iterative process of data reduction resulted in emerging aspects that influence seminar learning.ResultsTeachers identified seven key aspects affecting seminar learning: the seminar teacher, students, preparation, group functioning, seminar goals and content, course coherence and schedule and facilities. Important components of these aspects were: the teachers’ role in developing seminars (‘ownership’), the amount and quality of preparation materials, a non-threatening learning climate, continuity of group composition, suitability of subjects for seminar teaching, the number and quality of seminar questions, and alignment of different course activities.ConclusionsThe results of this study contribute to the unravelling of the ‘the black box’ of seminar learning. Suggestions for ways to optimise active learning in seminars are made regarding curriculum development, seminar content, quality assurance and faculty development.