Kiki M. J. M. H. Lombarts

Doctor performance assessment in daily practise : Does it help doctors or not? A systematic review

Context  Continuous assessment of individual performance of doctors is crucial for life‐long learning and quality of care. Policy‐makers and health educators should have good insights into the strengths and weaknesses of the methods available. The aim of this study was to systematically evaluate the feasibility of methods, the psychometric properties of instruments that are especially important for summative assessments, and the effectiveness of methods serving formative assessments used in routine practise to assess the performance of individual doctors.

Doctors' perceptions of why 360-degree feedback does (not) work: a qualitative study.

Objectives  Delivery of 360‐degree feedback is widely used in revalidation programmes. However, little has been done to systematically identify the variables that influence whether or not performance improvement is actually achieved after such assessments. This study aims to explore which factors represent incentives, or disincentives, for consultants to implement suggestions for improvement from 360‐degree feedback.

A Systematic Review of the Impact of Physicians’ Occupational Well-Being on the Quality of Patient Care

BackgroundIt is widely held that the occupational well-being of physicians may affect the quality of their patient care. Yet, there is still no comprehensive synthesis of the evidence on this connection.PurposeThis systematic review studied the effect of physicians’ occupational well-being on the quality of patient care.MethodsWe systematically searched PubMed, Embase, and PsychINFO from inception until August 2014. Two authors independently reviewed the studies. Empirical studies that explored the association between physicians’ occupational well-being and patient care quality were considered eligible. Data were systematically extracted on study design, participants, measurements, and findings. The Medical Education Research Study Quality Instrument (MERSQI) was used to assess study quality.ResultsUltimately, 18 studies were included. Most studies employed an observational design and were of average quality. Most studies reported positive associations of occupational well-being with patient satisfaction, patient adherence to treatment, and interpersonal aspects of patient care. Studies reported conflicting findings for occupational well-being in relation to technical aspects of patient care. One study found no association between occupational well-being and patient health outcomes.ConclusionsThe association between physicians’ occupational well-being and health care’s ultimate goal—improved patient health—remains understudied. Nonetheless, research up till date indicated that physicians’ occupational well-being can contribute to better patient satisfaction and interpersonal aspects of care. These insights may help in shaping the policies on physicians’ well-being and quality of care.

Development of a system for the evaluation of the teaching qualities of anesthesiology faculty

GRADUATE medical education is one of the core missions of academic medical centers, wherein medical specialists are responsible for teaching and supervising their future colleagues. However, being a medical specialist is no longer a sufficient qualification or proxy for competence in medical education aimed at training residents. This is particularly true given the modernization requirements for competency-based teaching and training promoted by accreditation institutions in some countries (such as the Accreditation for Council for Graduate Medical Education in the United States). These modernization efforts accelerate faculty development of clinicianeducators needed to achieve and maintain the highest standard of postgraduate medical education. An effective faculty development track should include measuring medical teaching effectiveness. This requires valid and reliable instruments, as well as providing the findings in a clear and concise format to faculty. Several studies have found that systematic and constructive feedback can result in improved teaching. There are few published and validated evaluation systems or even instruments aimed at supporting the graduate medical education qualities of clinical faculty. In anesthesiology, there are few published instruments and systems, and the existing ones tend to focus on faculty evaluation by residents only without any self-evaluations by faculty. To ensure actual behavioral change, individuals must usually undergo a stepwise change process. Evaluation insights obtained from feedback should be followed by creating positive intentions to change, trying out new behaviors and integrating them into practice. Supporting this change process has been shown to be effective. To support the specialty-specific evaluation of teaching qualities of anesthesiology faculty in an academic medical center, we developed the System for Evaluation of Teaching Qualities (SETQ) comprising (1) a Webbased self-evaluation by faculty, (2) a Web-based residents’ evaluation of faculty, (3) individualized faculty feedback, and (4) individualized faculty follow-up support. This paper has three main objectives: (1) to investigate the psychometric properties of the two instruments underlying the SETQ system, (2) to explore the relationship between residents’ evaluation and faculty self-evaluation, and (3) to gauge the feasibility of reliably using residents’ evaluation of faculty by estimating the number of such evaluations needed per faculty. We also place these objectives in context by describing SETQ. SETQ was initially developed in the anesthesiology department of a large academic medical center that has over 7,000 staff (including about 500 faculty and 400 residents) in the Netherlands. It was later expanded to include specialty-specific modules for internal medicine, surgery, and obstetrics and gynecology. At the time of writing, most of the remaining specialties have signed up for SETQ, resulting in more than 90% faculty coverage in 2009. SETQ is receiving nationwide attention.

Factors influencing residents' evaluations of clinical faculty member teaching qualities and role model status.

Medical Education 2012: 46: 381–389

Good Clinical Teachers Likely to be Specialist Role Models: Results from a Multicenter Cross-Sectional Survey

Context Medical educational reform includes enhancing role modelling of clinical teachers. This requires faculty being aware of their role model status and performance. We developed the System for Evaluation of Teaching Qualities (SETQ) to generate individualized feedback on previously defined teaching qualities and role model status for faculty in (non) academic hospitals. Objectives (i) To examine whether teaching qualities of faculty were associated with their being seen as a specialist role model by residents, and (ii) to investigate whether those associations differed across residency years and specialties. Methods & Materials Cross-sectional questionnaire survey amongst 549 Residents of 36 teaching programs in 15 hospitals in the Netherlands. The main outcome measure was faculty being seen as specialist role models by residents. Statistical analyses included (i) Pearsons correlation coefficients and (ii) multivariable logistic generalized estimating equations to assess the (adjusted) associations between each of five teaching qualities and ‘being seen as a role model’. Results 407 residents completed a total of 4123 evaluations of 662 faculty. All teaching qualities were positively correlated with ‘being seen as a role model’ with correlation coefficients ranging from 0.49 for ‘evaluation of residents’ to 0.64 for ‘learning climate’ (P<0.001). Faculty most likely to be seen as good role models were those rated highly on ‘feedback’ (odds ratio 2.91, 95% CI: 2.41–3.51), ‘a professional attitude towards residents’ (OR 2.70, 95% CI: 2.34–3.10) and ‘creating a positive learning climate’ (OR 2.45, 95% CI: 1.97–3.04). Results did not seem to vary much across residency years. The relative strength of associations between teaching qualities and being seen as a role model were more distinct when comparing specialties. Conclusions Good clinical educators are more likely to be seen as specialist role models for most residents.

Peer mentoring in doctor performance assessment: strategies, obstacles and benefits.

Medical Education 2010: 44 : 140–147

Frequency and determinants of residents' narrative feedback on the teaching performance of faculty: narratives in numbers.

Purpose Physicians involved in residency training often receive feedback from residents on their teaching. Research shows that learners value narrative feedback, but knowledge of the frequency and determinants of narrative feedback in teaching performance evaluation is lacking. This study aims to identifythe frequency with which residentsgave positive comments and suggestions for improvement to faculty, and the factors influencing that frequency. Method From September 2008 through May 2010, the authors collected data, using a validated formative feedback system (System for Evaluation of Teaching Qualities). The authors used univariate and multivariable analysis to investigate the associations between participants’ characteristics, including faculty members’ teaching performance, and the frequency of the two types of narrative comments. Results In total, 659 residents (79% of 839) completed 6,216 evaluations on 917 faculty (95% of 964), resulting in 11,574 positive comments and 4,870 suggestions for improvement. On average, faculty members received 13 positive comments and 5 suggestions for improvement. Multivariable analysis showed that higher teaching performance was associated with higher numbers of positive comments (regression coefficient 0.538; 95% confidence interval: 0.464 to 0.613) and with lower numbers of suggestions for improvement (−0.802; −0.911 to −0.692), both P < .0001. Nonacademic hospitals, participation in teacher training, and female residents’ evaluation were statistically significant determinants of receiving more narrative feedback. Conclusions Residents provided narrative feedback that paralleled and elaborated on quantitative evaluations they provided; therefore, faculty would be wise to attend to narrative feedback. Analysis of the quality of narrative feedback is needed to understand its effectiveness.

Effect of the learning climate of residency programs on faculty's teaching performance as evaluated by residents.

Background To understand teaching performance of individual faculty, the climate in which residents’ learning takes place, the learning climate, may be important. There is emerging evidence that specific climates do predict specific outcomes. Until now, the effect of learning climate on the performance of the individual faculty who actually do the teaching was unknown. Objectives This study: (i) tested the hypothesis that a positive learning climate was associated with better teaching performance of individual faculty as evaluated by residents, and (ii) explored which dimensions of learning climate were associated with faculty’s teaching performance. Methods and Materials We conducted two cross-sectional questionnaire surveys amongst residents from 45 residency training programs and multiple specialties in 17 hospitals in the Netherlands. Residents evaluated the teaching performance of individual faculty using the robust System for Evaluating Teaching Qualities (SETQ) and evaluated the learning climate of residency programs using the Dutch Residency Educational Climate Test (D-RECT). The validated D-RECT questionnaire consisted of 11 subscales of learning climate. Main outcome measure was faculty’s overall teaching (SETQ) score. We used multivariable adjusted linear mixed models to estimate the separate associations of overall learning climate and each of its subscales with faculty’s teaching performance. Results In total 451 residents completed 3569 SETQ evaluations of 502 faculty. Residents also evaluated the learning climate of 45 residency programs in 17 hospitals in the Netherlands. Overall learning climate was positively associated with faculty’s teaching performance (regression coefficient 0.54, 95% confidence interval: 0.37 to 0.71; P<0.001). Three out of 11 learning climate subscales were substantially associated with better teaching performance: ‘coaching and assessment’, ‘work is adapted to residents’ competence’, and ‘formal education’. Conclusions Individual faculty’s teaching performance evaluations are positively affected by better learning climate of residency programs.

Factors predicting doctors' reporting of performance change in response to multisource feedback.

BackgroundMulti-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60–70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change.MethodsThis observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists’ reported change in actual practice.ResultsIn total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p < 0.05) as did negative scores offered by colleagues. (regression coefficient beta = −0.157, p < 0.05). The explained variance of these two variables combined was 34 percent.ConclusionsPerceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors.