Alec Knight

How effective are selection methods in medical education? A systematic review

Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non‐academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.

What are the essential features of a successful surgical registry? a systematic review

Objective The regulation of surgical implants is vital to patient safety, and there is an international drive to establish registries for all implants. Hearing loss is an area of unmet need, and industry is targeting this field with a growing range of surgically implanted hearing devices. Currently, there is no comprehensive UK registry capturing data on these devices; in its absence, it is difficult to monitor safety, practices and effectiveness. A solution is developing a national registry of all auditory implants. However, developing and maintaining a registry faces considerable challenges. In this systematic review, we aimed to identify the essential features of a successful surgical registry. Methods A systematic literature review was performed adhering to Preferred Reporting Items for Systematic Review and Meta-Analysis recommendations. A comprehensive search of the Medline and Embase databases was conducted in November 2016 using the Ovid Portal. Inclusion criteria were: publications describing the design, development, critical analysis or current status of a national surgical registry. All registry names identified in the screening process were noted and searched in the grey literature. Available national registry reports were reviewed from registry websites. Data were extracted using a data extraction table developed by thematic analysis. Extracted data were synthesised into a structured narrative. Results Sixty-nine publications were included. The fundamentals to successful registry development include: steering committee to lead and oversee the registry; clear registry objectives; planning for initial and long-term funding; strategic national collaborations among key stakeholders; dedicated registry management team; consensus meetings to agree registry dataset; established data processing systems; anticipating challenges; and implementing strategies to increase data completion. Patient involvement and awareness of legal factors should occur throughout the development process. Conclusions This systematic review provides robust knowledge that can be used to inform the successful development of any UK surgical registry. It also provides a methodological framework for international surgical registry development.

What are the requirements for developing a successful national registry of auditory implants? A qualitative study

Objectives Hearing loss is an area of unmet need, and industry is targeting this field with a growing range of surgically implanted hearing devices. Currently, there is no comprehensive UK registry capturing data on these devices; in its absence, it is difficult to monitor clinical and cost-effectiveness and develop national policy. Recognising that developing such a registry faces considerable challenges, it is important to gather opinions from stakeholders and patients. This paper builds on our systematic review on surgical registry development and aims to identify the specific requirements for developing a successful national registry of auditory implants. Design Qualitative study. Participants Data were collected in two ways: (1) semistructured interviews with UK professional stakeholders; and (2) focus groups with patients with hearing loss. The interview and focus group schedules were informed by our systematic review on registry development. Data were analysed using directed content analysis. Judges mapped the themes obtained against a conceptual framework developed from our systematic review on registry development. The conceptual framework consisted of five categories for successful registry development: (1) planning, (2) registry governance, (3) registry dataset, (4) anticipating challenges, (5) implementing solutions. Results Twenty-seven themes emerged from 40 semistructured interviews with professional stakeholders and 18 themes emerged from three patient focus groups. The most important factor for registry success was high rates of data completion. Benefits of developing a successful registry of auditory implants include: strengthening the evidence base and regulation of auditory implants, driving quality and safety improvements, increased transparency, facilitating patient decision-making and informing policy and guidelines development. Conclusions This study identifies the requirements for developing a successful national registry of auditory implants, benefiting from the involvement of numerous professional stakeholder groups and patients with hearing loss. Our approach may be used internationally to inform successful registry development.

Mental Wellbeing Impact Assessment (MWIA) in the workplace

Purpose The Mental Wellbeing Impact Assessment (MWIA) is an evidence-based tool that guides decision makers, such as policy makers and service managers, about the potential impacts of a new programme or policy change. It was initially used in urban regeneration but has subsequently been used in housing, children’s centres and education. The purpose of this paper is to report, for the first time, on the strengths and weaknesses of using the MWIA in the workplace. Design/methodology/approach Feedback was collected from staff who participated in stakeholder workshops as part of the MWIA process at two different public sector organisations. Findings The MWIA can be used as an effective workplace assessment tool and is valuable as both a diagnostic tool and as an intervention in its own right. The MWIA generates tailored action plans focussed on addressing the organisation or team-specific issues. The weaknesses of the MWIA in the workplace are mainly focussed around management cooperation and commitment to the process which should be screened for prior to engaging in the full stakeholder workshop. Originality/value This is the first report of MWIA’s use in the workplace but suggests that it is a useful tool which can be used to support workplace wellbeing, especially in relation to a policy or organisational change. Further studies should be carried out to fully understand the impact of the MWIA in the workplace.

Setting standards and monitoring quality in the NHS 1999-2013: a classic case of goal conflict: English National Health Service

Abstract 2013 saw the National Health Service (NHS) in England severely criticized for providing poor quality despite successive governments in the previous 15 years, establishing a range of new institutions to improve NHS quality. This study seeks to understand the contributions of political and organizational influences in enabling the NHS to deliver high‐quality care through exploring the experiences of two of the major new organizations established to set standards and monitor NHS quality. We used a mixed method approach: first a cross‐sectional, in‐depth qualitative interview study and then the application of principal agent modeling (Waterman and Meier broader framework). Ten themes were identified as influencing the functioning of the NHS regulatory institutions: socio‐political environment; governance and accountability; external relationships; clarity of purpose; organizational reputation; leadership and management; organizational stability; resources; organizational methods; and organizational performance. The organizations could be easily mapped onto the framework, and their transience between the different states could be monitored. We concluded that differing policy objectives for NHS quality monitoring resulted in central involvement and organizational change. This had a disruptive effect on the ability of the NHS to monitor quality. Constant professional leadership, both clinical and managerial, and basing decisions on best evidence, both technical and organizational, helped one institution to deliver on its remit, even within a changing political/policy environment. Application of the Waterman–Meier framework enabled an understanding and description of the dynamic relationship between central government and organizations in the NHS and may predict when tensions will arise in the future.

The NICE alcohol misuse standard – evaluating its impact

Purpose The purpose of this paper is to explore factors affecting implementing the National Institute for Health and Care Excellence (NICE) quality standard on alcohol misuse (QS11) and barriers and facilitators to its implementation. Design/methodology/approach Qualitative interview study analysed using directed and conventional content analyses. Participants were 38 individuals with experience of commissioning, delivering or using alcohol healthcare services in Southwark, Lambeth and Lewisham. Findings QS11 implementation ranged from no implementation to full implementation across the 13 statements. Implementation quality was also reported to vary widely across different settings. The analyses also uncovered numerous barriers and facilitators to implementing each statement. Overarching barriers to implementation included: inherent differences between specialist vs generalist settings; poor communication between healthcare settings; generic barriers to implementation; and poor governance structures and leadership. Research limitations/implications QS11 was created to summarise alcohol-related NICE guidance. The aim was to simplify guidance and enhance local implementation. However, in practice the standard requires complex actions by professionals. There was considerable variation in local alcohol commissioning models, which was associated with variation in implementation. These models warrant further evaluation to identify best practice. Originality/value Little evidence exists on the implementing quality standards, as distinct from clinical practice guidelines. The authors present direct evidence on quality standard implementation, identify implementation shortcomings and make recommendations for future research and practice.

Exploring the potential uses of value-added metrics in the context of postgraduate medical education

Abstract Context: Increasing pressure is being placed on external accountability and cost efficiency in medical education and training internationally. We present an illustrative data analysis of the value-added of postgraduate medical education. Method: We analysed historical selection (entry) and licensure (exit) examination results for trainees sitting the UK Membership of the Royal College of General Practitioners (MRCGP) licensing examination (N = 2291). Selection data comprised: a clinical problem solving test (CPST); a situational judgement test (SJT); and a selection centre (SC). Exit data was an applied knowledge test (AKT) from MRCGP. Ordinary least squares (OLS) regression analyses were used to model differences in attainment in the AKT based on performance at selection (the value-added score). Results were aggregated to the regional level for comparisons. Results: We discovered significant differences in the value-added score between regional training providers. Whilst three training providers confer significant value-added, one training provider was significantly lower than would be predicted based on the attainment of trainees at selection. Conclusions: Value-added analysis in postgraduate medical education potentially offers useful information, although the methodology is complex, controversial, and has significant limitations. Developing models further could offer important insights to support continuous improvement in medical education in future.

Setting standards and monitoring quality in the NHS 1999–2013: a classic case of goal conflict

Abstract 2013 saw the National Health Service (NHS) in England severely criticized for providing poor quality despite successive governments in the previous 15 years, establishing a range of new institutions to improve NHS quality. This study seeks to understand the contributions of political and organizational influences in enabling the NHS to deliver high‐quality care through exploring the experiences of two of the major new organizations established to set standards and monitor NHS quality. We used a mixed method approach: first a cross‐sectional, in‐depth qualitative interview study and then the application of principal agent modeling (Waterman and Meier broader framework). Ten themes were identified as influencing the functioning of the NHS regulatory institutions: socio‐political environment; governance and accountability; external relationships; clarity of purpose; organizational reputation; leadership and management; organizational stability; resources; organizational methods; and organizational performance. The organizations could be easily mapped onto the framework, and their transience between the different states could be monitored. We concluded that differing policy objectives for NHS quality monitoring resulted in central involvement and organizational change. This had a disruptive effect on the ability of the NHS to monitor quality. Constant professional leadership, both clinical and managerial, and basing decisions on best evidence, both technical and organizational, helped one institution to deliver on its remit, even within a changing political/policy environment. Application of the Waterman–Meier framework enabled an understanding and description of the dynamic relationship between central government and organizations in the NHS and may predict when tensions will arise in the future.

Setting standards and monitoring quality in the NHS 1999–2013

Abstract 2013 saw the National Health Service (NHS) in England severely criticized for providing poor quality despite successive governments in the previous 15 years, establishing a range of new institutions to improve NHS quality. This study seeks to understand the contributions of political and organizational influences in enabling the NHS to deliver high‐quality care through exploring the experiences of two of the major new organizations established to set standards and monitor NHS quality. We used a mixed method approach: first a cross‐sectional, in‐depth qualitative interview study and then the application of principal agent modeling (Waterman and Meier broader framework). Ten themes were identified as influencing the functioning of the NHS regulatory institutions: socio‐political environment; governance and accountability; external relationships; clarity of purpose; organizational reputation; leadership and management; organizational stability; resources; organizational methods; and organizational performance. The organizations could be easily mapped onto the framework, and their transience between the different states could be monitored. We concluded that differing policy objectives for NHS quality monitoring resulted in central involvement and organizational change. This had a disruptive effect on the ability of the NHS to monitor quality. Constant professional leadership, both clinical and managerial, and basing decisions on best evidence, both technical and organizational, helped one institution to deliver on its remit, even within a changing political/policy environment. Application of the Waterman–Meier framework enabled an understanding and description of the dynamic relationship between central government and organizations in the NHS and may predict when tensions will arise in the future.

Evaluation of two selection tests for recruitment into radiology specialty training

BackgroundThis study evaluated whether two selection tests previously validated for primary care General Practice (GP) trainee selection could provide a valid shortlisting selection method for entry into specialty training for the secondary care specialty of radiology.MethodsWe conducted a retrospective analysis of data from radiology applicants who also applied to UK GP specialty training or Core Medical Training. The psychometric properties of the two selection tests, a clinical problem solving (CPS) test and situational judgement test (SJT), were analysed to evaluate their reliability. Predictive validity of the tests was analysed by comparing them with the current radiology selection assessments, and the licensure examination results taken after the first stage of training (Fellowship of the Royal College of Radiologists (FRCR) Part 1).ResultsThe internal reliability of the two selection tests in the radiology applicant sample was good (α ≥ 0.80). The average correlation with radiology shortlisting selection scores was r = 0.26 for the CPS (with p < 0.05 in 5 of 11 shortlisting centres), r = 0.15 for the SJT (with p < 0.05 in 2 of 11 shortlisting centres) and r = 0.25 (with p < 0.05 in 5 of 11 shortlisting centres) for the two tests combined. The CPS test scores significantly correlated with performance in both components of the FRCR Part 1 examinations (r = 0.5 anatomy; r = 0.4 physics; p < 0.05 for both). The SJT did not correlate with either component of the examination.ConclusionsThe current CPS test may be an appropriate selection method for shortlisting in radiology but would benefit from further refinement for use in radiology to ensure that the test specification is relevant. The evidence on whether the SJT may be appropriate for shortlisting in radiology is limited. However, these results may be expected to some extent since the SJT is designed to measure non-academic attributes. Further validation work (e.g. with non-academic outcome variables) is required to evaluate whether an SJT will add value in recruitment for radiology specialty training and will further inform construct validity of SJTs as a selection methodology.