Martin Roberts

Factors associated with variability in the assessment of UK doctors’ professionalism: analysis of survey results

Objectives To investigate potential sources of systematic bias arising in the assessment of doctors’ professionalism. Design Linear regression modelling of cross sectional questionnaire survey data. Setting 11 clinical practices in England and Wales. Participants 1065 non-training grade doctors from various clinical specialties and settings, 17 031 of their colleagues, and 30 333 of their patients. Main outcome measures Two measures of a doctor’s professional performance using patient and colleague questionnaires from the United Kingdom’s General Medical Council (GMC). We selected potential predictor variables from the characteristics of the doctors and of their patient and colleague assessors. Results After we adjusted for characteristics of the doctor as well as characteristics of the patient sample, less favourable scores from patient feedback were independently predicted by doctors having obtained their primary medical degree from any non-European country; doctors practising as a psychiatrist; lower proportions of white patients providing feedback; lower proportions of patients rating their consultation as being very important; and lower proportions of patients reporting that they were seeing their usual doctor. Lower scores from colleague feedback were independently predicted by doctors having obtained their primary medical degree from countries outside the UK and South Asia; currently employed in a locum capacity; working as a general practitioner or psychiatrist; being employed in a staff grade, associate specialist, or other equivalent role; and with a lower proportion of colleagues reporting they had daily or weekly professional contact with the doctor. In fully adjusted models, the doctor’s age, sex, and ethnic group were not independent predictors of patient or colleague feedback. Neither the age or sex profiles of the patient or colleague samples were independent predictors of doctors’ feedback scores, and nor was the ethnic group of colleague samples. Conclusions Caution is necessary when considering patient and colleague feedback regarding doctors’ professionalism. Multisource feedback undertaken for revalidation using the GMC patient and colleague questionnaires should, at least initially, be principally formative in nature.

Factors affecting patients' trust and confidence in GPs: evidence from the English national GP patient survey.

Objectives Patients’ trust in general practitioners (GPs) is fundamental to effective clinical encounters. Associations between patients’ trust and their perceptions of communication within the consultation have been identified, but the influence of patients’ demographic characteristics on these associations is unknown. We aimed to investigate the relative contribution of the patients age, gender and ethnicity in any association between patients’ ratings of interpersonal aspects of the consultation and their confidence and trust in the doctor. Design Secondary analysis of English national GP patient survey data (2009). Setting Primary Care, England, UK. Participants Data from year 3 of the GP patient survey: 5 660 217 questionnaires sent to patients aged 18 and over, registered with a GP in England for at least 6 months; overall response rate was 42% after adjustment for sampling design. Outcome measures We used binary logistic regression analysis to investigate patients’ reported confidence and trust in the GP, analysing ratings of 7 interpersonal aspects of the consultation, controlling for patients’ sociodemographic characteristics. Further modelling examined moderating effects of age, gender and ethnicity on the relative importance of these 7 predictors. Results Among 1.5 million respondents (adjusted response rate 42%), the sense of ‘being taken seriously’ had the strongest association with confidence and trust. The relative importance of the 7 interpersonal aspects of care was similar for men and women. Non-white patients accorded higher priority to being given enough time than did white patients. Involvement in decisions regarding their care was more strongly associated with reports of confidence and trust for older patients than for younger patients. Conclusions Associations between patients’ ratings of interpersonal aspects of care and their confidence and trust in their GP are influenced by patients’ demographic characteristics. Taking account of these findings could inform patient-centred service design and delivery and potentially enhance patients’ confidence and trust in their doctor.

Multisource Feedback in Evaluating the Performance of Doctors: The Example of the UK General Medical Council Patient and Colleague Questionnaires

Purpose Internationally, there is increasing interest in monitoring and evaluating doctors’ professional practice. Multisource feedback (MSF) offers one way of collecting information about doctors’ performance. The authors investigated the psychometric properties of two questionnaires developed for this purpose and explored the biases that may exist within data collected via such instruments. Method A cross-sectional study was conducted in 11 UK health care organizations during 2008–2011. Patients (n = 30,333) and colleagues (n = 17,012) rated the professional performance of 1,065 practicing doctors, using the General Medical Council Patient Questionnaire (PQ) and Colleague Questionnaire (CQ). The psychometric properties of the questionnaires were assessed, and regression modeling was used to explore factors that influenced patient and colleague responses on the core questionnaire items. Results Although the questionnaires demonstrated satisfactory internal consistency, test–retest reliability, and convergent validity, patient and colleague ratings were highly skewed toward favorable impressions of doctor performance. At least 34 PQs and 15 CQs were required to achieve acceptable reliability (G > 0.70). Item ratings were influenced by characteristics of the patient and colleague respondents and the context in which their feedback was provided. Conclusions The PQ and CQ are acceptable for the provision of formative feedback on a doctor’s professional practice within an appraisal process. However, biases identified in the questionnaire data suggest that caution is required when interpreting and acting on this type of information. MSF derived from these questionnaires should not be used in isolation to inform decisions about a doctor’s fitness to practice medicine.

Understanding high and low patient experience scores in primary care: analysis of patients' survey data for general practices and individual doctors.

Objectives To determine the extent to which practice level scores mask variation in individual performance between doctors within a practice. Design Analysis of postal survey of patients’ experience of face-to-face consultations with individual general practitioners in a stratified quota sample of primary care practices. Setting Twenty five English general practices, selected to include a range of practice scores on doctor-patient communication items in the English national GP Patient Survey. Participants 7721 of 15 172 patients (response rate 50.9%) who consulted with 105 general practitioners in 25 practices between October 2011 and June 2013. Main outcome measure Score on doctor-patient communication items from post-consultation surveys of patients for each participating general practitioner. The amount of variance in each of six outcomes that was attributable to the practices, to the doctors, and to the patients and other residual sources of variation was calculated using hierarchical linear models. Results After control for differences in patients’ age, sex, ethnicity, and health status, the proportion of variance in communication scores that was due to differences between doctors (6.4%) was considerably more than that due to practices (1.8%). The findings also suggest that higher performing practices usually contain only higher performing doctors. However, lower performing practices may contain doctors with a wide range of communication scores. Conclusions Aggregating patients’ ratings of doctors’ communication skills at practice level can mask considerable variation in the performance of individual doctors, particularly in lower performing practices. Practice level surveys may be better used to “screen” for concerns about performance that require an individual level survey. Higher scoring practices are unlikely to include lower scoring doctors. However, lower scoring practices require further investigation at the level of the individual doctor to distinguish higher and lower scoring general practitioners.

Development of workplace-based assessments of non-technical skills in anaesthesia*

Non‐technical skills are recognised as crucial to good anaesthetic practice. We designed and evaluated a specialty‐specific tool to assess non‐technical aspects of trainee performance in theatre, based on a system previously found reliable in a recruitment setting. We compared inter‐rater agreement (multir‐ater kappa) for live assessments in theatre with that in a selection centre and a video‐based rater training exercise. Twenty‐seven trainees participated in the first in‐theatre assessment round and 40 in the second. Round‐ 1 scores had poor inter‐rater agreement (mean kappa = 0.20) and low reliability (generalisability coefficient G = 0.50). A subsequent assessor training exercise showed good inter‐rater agreement, (mean kappa = 0.79) but did not improve performance of the assessment tool when used in round 2 (mean kappa = 0.14, G = 0.42). Inter‐rater agreement in two selection centres (mean kappa = 0.61 and 0.69) exceeded that found in theatre. Assessment tools that perform reliably in controlled settings may not do so in the workplace.

Safety culture and the 5 steps to safer surgery: an intervention study

BACKGROUND Improvements in safety culture have been postulated as one of the mechanisms underlying the association between the introduction of the World Health Organisation (WHO) Surgical Safety Checklist with perioperative briefings and debriefings, and enhanced patient outcomes. The 5 Steps to Safer Surgery (5SSS) incorporates pre-list briefings, the three steps of the WHO Surgical Safety Checklist (SSC) and post-list debriefings in one framework. We aimed to identify any changes in safety culture associated with the introduction of the 5SSS in orthopaedic operating theatres. METHODS We assessed the safety culture in the elective orthopaedic theatres of a large UK teaching hospital before and after introduction of the 5SSS using a modified version of the Safety Attitude Questionnaire - Operating Room (SAQ-OR). Primary outcome measures were pre-post intervention changes in the six safety culture domains of the SAQ-OR. We also analysed changes in responses to two items regarding perioperative briefings. RESULTS The SAQ-OR survey response rate was 80% (60/75) at baseline and 74% (53/72) one yr later. There were significant improvements in both the reported frequency (P<0.001) and perceived importance (P=0.018) of briefings, and in five of the six safety culture domain scores (Working Conditions, Perceptions of Management, Job Satisfaction, Safety Climate and Teamwork Climate) of the SAQ-OR (P<0.001 in all cases). Scores in the sixth domain (Stress Recognition) decreased significantly (P=0.028). CONCLUSIONS Implementation of the 5SSS was associated with a significant improvement in the safety culture of elective orthopaedic operating theatres.

Why do GPs exclude patients from participating in research? An exploration of adherence to and divergence from trial criteria

BACKGROUND The role of GPs in recruiting or excluding participants critically underpins the feasibility, external validity and generalizability of primary care research. A better understanding of this role is needed. AIM To investigate why GPs excluded potentially eligible participants from a large scale randomized controlled trial (RCT), to determine the proportion of patients excluded on account of trial eligibility compared with other reasons, and to explore the impact of such exclusions on the management and generalizability of RCTs. DESIGN AND SETTING Secondary analysis of data from the CoBalT study, a multi-centre general-practice-based RCT investigating cognitive behavioural therapy as an adjunct to pharmacotherapy for treatment-resistant depression. METHOD GPs were asked to screen patient lists generated from computerized record searches for trial eligibility and to provide narrative reasons for excluding patients. These reasons were coded independently by two researchers, with a third researcher resolving discrepancies. RESULTS Thirty-one percent (4750/15,379) of patients were excluded at the GP screening stage, including 663 on patient lists that remained unscreened. Of the 4087 actively excluded patients, 67% were excluded on account of trial exclusion criteria, 20% for other criteria (half of which were comorbid conditions) and 13% without reason. CONCLUSION Clear, comprehensive criteria, particularly with regards to comorbidities, are required for GPs to confidently screen patients for potential participation in research. Future studies should promote inclusivity and encourage GPs to adopt a liberal approach when screening patient lists. This would enhance the validity and generalizability of primary care research and encourage greater patient autonomy.

Web-Based Textual Analysis of Free-Text Patient Experience Comments From a Survey in Primary Care

Background Open-ended questions eliciting free-text comments have been widely adopted in surveys of patient experience. Analysis of free text comments can provide deeper or new insight, identify areas for action, and initiate further investigation. Also, they may be a promising way to progress from documentation of patient experience to achieving quality improvement. The usual methods of analyzing free-text comments are known to be time and resource intensive. To efficiently deal with a large amount of free-text, new methods of rapidly summarizing and characterizing the text are being explored. Objective The aim of this study was to investigate the feasibility of using freely available Web-based text processing tools (text clouds, distinctive word extraction, key words in context) for extracting useful information from large amounts of free-text commentary about patient experience, as an alternative to more resource intensive analytic methods. Methods We collected free-text responses to a broad, open-ended question on patients’ experience of primary care in a cross-sectional postal survey of patients recently consulting doctors in 25 English general practices. We encoded the responses to text files which were then uploaded to three Web-based textual processing tools. The tools we used were two text cloud creators: TagCrowd for unigrams, and Many Eyes for bigrams; and Voyant Tools, a Web-based reading tool that can extract distinctive words and perform Keyword in Context (KWIC) analysis. The association of patients’ experience scores with the occurrence of certain words was tested with logistic regression analysis. KWIC analysis was also performed to gain insight into the use of a significant word. Results In total, 3426 free-text responses were received from 7721 patients (comment rate: 44.4%). The five most frequent words in the patients’ comments were “doctor”, “appointment”, “surgery”, “practice”, and “time”. The three most frequent two-word combinations were “reception staff”, “excellent service”, and “two weeks”. The regression analysis showed that the occurrence of the word “excellent” in the comments was significantly associated with a better patient experience (OR=1.96, 95%CI=1.63-2.34), while “rude” was significantly associated with a worse experience (OR=0.53, 95%CI=0.46-0.60). The KWIC results revealed that 49 of the 78 (63%) occurrences of the word “rude” in the comments were related to receptionists and 17(22%) were related to doctors. Conclusions Web-based text processing tools can extract useful information from free-text comments and the output may serve as a springboard for further investigation. Text clouds, distinctive words extraction and KWIC analysis show promise in quick evaluation of unstructured patient feedback. The results are easily understandable, but may require further probing such as KWIC analysis to establish the context. Future research should explore whether more sophisticated methods of textual analysis (eg, sentiment analysis, natural language processing) could add additional levels of understanding.

A study to assess the influence of interprofessional point of care simulation training on safety culture in the operating theatre environment of a university teaching hospital

ABSTRACT Interprofessional point of care or in situ simulation is used as a training tool in our operating theatre directorate with the aim of improving crisis behaviours. This study aimed to assess the impact of interprofessional point of care simulation on the safety culture of operating theatres. A validated Safety Attitude Questionnaire was administered to staff members before each simulation scenario and then re-administered to the same staff members after 6–12 months. Pre- and post-training Safety Attitude Questionnaire—Operating Room (SAQ-OR) scores were compared using paired sample t-tests. Analysis revealed a statistically significant perceived improvement in both safety (p < 0.001) and teamwork (p = 0.013) climate scores (components of safety culture) 6–12 months after interprofessional simulation training. A growing body of literature suggests that a positive safety culture is associated with improved patient outcomes. Our study supports the implementation of point of care simulation as a useful intervention to improve safety culture in theatres.

GPAQ-R: development and psychometric properties of a version of the General Practice Assessment Questionnaire for use for revalidation by general practitioners in the UK

BackgroundThe General Practice Assessment Questionnaire (GPAQ) has been widely used to assess patient experience in general practice in the UK since 2004. In 2013, new regulations were introduced by the General Medical Council (GMC) requiring UK doctors to undertake periodic revalidation, which includes assessment of patient experience for individual doctors. We describe the development of a new version of GPAQ – GPAQ-R which addresses the GMC’s requirements for revalidation as well as additional NHS requirements for surveys that GPs may need to carry out in their own practices.MethodsQuestionnaires were given out by doctors or practice staff after routine consultations in line with the guidance given by the General Medical Council for surveys to be used for revalidation. Data analysis and practice reports were provided independently.ResultsData were analysed for questionnaires from 7258 patients relating to 164 GPs in 29 general practices. Levels of missing data were generally low (typically 4.5-6%). The number of returned questionnaires required to achieve reliability of 0.7 were around 35 for individual doctor communication items and 29 for a composite score based on doctor communication items. This suggests that the responses to GPAQ-R had similar reliability to the GMC’s own questionnaire and we recommend 30 completed GPAQ-R questionnaires are sufficient for revalidation purposes. However, where an initial screen raises concern, the survey might be repeated with 50 completed questionnaires in order to increase reliability.ConclusionsGPAQ-R is a development of a well-established patient experience questionnaire used in general practice in the UK since 2004. This new version can be recommended for use in order to meet the UK General Medical Council’s requirements for surveys to be used in revalidation of doctors. It also meets the needs of GPs to ask about patient experience relating to aspects of practice care that are not specific to individual general practitioners (e.g. receptionists, telephone access) which meet other survey requirements of the National Health Service in England. Use of GPAQ-R has the potential to reduce the number of surveys that GPs need to carry out in their practices to meet the various regulatory requirements which they face.