Elizabeth Gibbons

Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist

BackgroundThe COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114).Methods75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item.Results88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.ConclusionsResults indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.

How to routinely collect data on patient-reported outcome and experience measures in renal registries in Europe: an expert consensus meeting

Despite the potential for patient-reported outcome measures (PROMs) and experience measures (PREMs) to enhance understanding of patient experiences and outcomes they have not, to date, been widely incorporated into renal registry datasets. This report summarizes the main points learned from an ERA-EDTA QUEST-funded consensus meeting on how to routinely collect PROMs and PREMs in renal registries in Europe. In preparation for the meeting, we surveyed all European renal registries to establish current or planned efforts to collect PROMs/PREMs. A systematic review of the literature was performed. Publications reporting barriers and/or facilitators to PROMs/PREMs collection by registries were identified and a narrative synthesis undertaken. A group of renal registry representatives, PROMs/PREMs experts and patient representatives then met to (i) share any experience renal registries in Europe have in this area; (ii) establish how patient-reported data might be collected by understanding how registries currently collect routine data and how patient-reported data is collected in other settings; (iii) harmonize the future collection of patient-reported data by renal registries in Europe by agreeing upon preferred instruments and (iv) to identify the barriers to routine collection of patient-reported data in renal registries in Europe. In total, 23 of the 45 European renal registries responded to the survey. Two reported experience in collecting PROMs and three stated that they were actively exploring ways to do so. The systematic review identified 157 potentially relevant articles of which 9 met the inclusion criteria and were analysed for barriers and facilitators to routine PROM/PREM collection. Thirteen themes were identified and mapped to a three-stage framework around establishing the need, setting up and maintaining the routine collection of PROMs/PREMs. At the consensus meeting some PROMs instruments were agreed for routine renal registry collection (the generic SF-12, the disease-specific KDQOL™-36 and EQ-5D-5L to be able to derive quality-adjusted life years), but further work was felt to be needed before recommending PREMs. Routinely collecting PROMs and PREMs in renal registries is important if we are to better understand what matters to patients but it is likely to be challenging; close international collaboration will be beneficial.

Functionality and feedback: a protocol for a realist synthesis of the collation, interpretation and utilisation of PROMs data to improve patient care

Introduction The feedback and public reporting of PROMs data aims to improve the quality of care provided to patients. Existing systematic reviews have found it difficult to draw overall conclusions about the effectiveness of PROMs feedback. We aim to execute a realist synthesis of the evidence to understand by what means and in what circumstances the feedback of PROMs data leads to the intended service improvements. Methods and analysis Realist synthesis involves (stage 1) identifying the ideas, assumptions or ‘programme theories’ which explain how PROMs feedback is supposed to work and in what circumstances and then (stage 2) reviewing the evidence to determine the extent to which these expectations are met in practice. For stage 1, six provisional ‘functions’ of PROMs feedback have been identified to structure our review (screening, monitoring, patient involvement, demand management, quality improvement and patient choice). For each function, we will identify the different programme theories that underlie these different goals and develop a logical map of the respective implementation processes. In stage 2, we will identify studies that will provide empirical tests of each component of the programme theories to evaluate the circumstances in which the potential obstacles can be overcome and whether and how the unintended consequences of PROMs feedback arise. We will synthesise this evidence to (1) identify the implementation processes which support or constrain the successful collation, interpretation and utilisation of PROMs data; (2) identify the implementation processes through which the unintended consequences of PROMs data arise and those where they can be avoided. Ethics and dissemination The study will not require NHS ethics approval. We have secured ethical approval for the study from the University of Leeds (LTSSP-019). We will disseminate the findings of the review through a briefing paper and dissemination event for National Health Service stakeholders, conferences and peer reviewed publications.

Patient reports of the outcomes of treatment: a structured review of approaches

Patient reports or ratings are essential for measuring the quality of patient care. Measures designed for this purpose tend to focus on the processes and structures of care rather than the outcomes of it. The latter is arguably the most valid indicator of the quality of care patients receive. Typically this information is gathered by probing patient satisfaction with treatment as part of an investigation of satisfaction with hospital care. More recently patient ratings of the outcome of treatment have been obtained to measure treatment efficacy in clinical trials. However, a more direct approach is to ask patients to assess the benefit of treatment on their current health status. We performed a structured literature review on patient reported satisfaction with outcomes of treatment and direct patient assessments of the same. The purpose of this was to identify suitable candidate questions for a short instrument to tap patient evaluations of in-patient hospital interventions. Articles were included if they dealt with patient satisfaction or patient assessment of the outcomes of treatment. Articles were excluded if they dealt more generally with patient satisfaction with care. We identified 169 papers, 79 were included in the review. The findings of this review suggest that there are a number of benefits of directly asking patients to assess the outcome of hospital treatment. Importantly this approach reflects outcomes relevant to the patient and is also more likely to reflect patient report in routine clinical practice. There is also evidence that such approaches have face validity and construct validity. The problems associated with this approach (i.e. response bias), are those common to patient reported outcome surveys, but employing appropriate strategies can minimize them. Furthermore, employing a simple set of questions that asks patients to assess the outcomes of treatment they receive can be time and resource efficient in comparison to administering lengthy measures. This approach could be tested for potential generic use as an evaluative measure for patients in hospital settings.

A structured review of patient‐reported outcome measures for patients with skin cancer, 2013

The collection of patient‐reported outcome measures (PROMs) within the national PROMs programme for elective procedures is now established mandatory practice in the NHS with high response rates and completion.

Systematic review of measurement properties of patient-reported outcome measures used in patients undergoing hip and knee arthroplasty

Objectives To identify patient-reported outcome measures (PROMs) that have been developed and/or used with patients undergoing hip or knee replacement surgery and to provide a shortlist of the most promising generic and condition-specific instruments. Methods A systematic review of the literature was performed to identify measures used in patients undergoing hip and knee replacement and extract and evaluate information on their methodological quality. Results Thirty-two shortlisted measures were reviewed for the quality of their measurement properties. On the basis of the review criteria, the measures with most complete evidence to date are the Oxford Hip Score (OHS) (for patients undergoing hip replacement surgery) and the Oxford Knee Score (OKS), with OKS-Activity and Participation Questionnaire (for patients undergoing knee replacement surgery). Conclusion A large number of these instruments lack essential evidence of their measurement properties (eg, validity, reliability, and responsiveness) in specific populations of patients. Further research is required on almost all of the identified measures. The best-performing condition-specific PROMs were the OKS, OHS, and Western Ontario and McMaster Universities Osteoarthritis Index. The best-performing generic measure was the Short Form 12. Researchers can use the information presented in this review to inform further psychometric studies of the reviewed measures.

Themes in daily life of adolescents and young adults with congenital bleeding disorders: a qualitative study.

P. LIMPERG,* M. PETERS,† E. GIBBONS,‡ M. COPPENS,§ C. VALK,§ M. GROOTENHUIS* and L. HAVERMAN* *Psychosocial Department, Emma Children’s Hospital, AMC, Amsterdam, the Netherlands; †Department of PediatricHematology, Emma Children’s Hospital and Hemophilia Comprehensive Care Treatment Center, AMC, Amsterdam, the Netherlands; ‡Health Services Research Unit, Nuffield Department of Population Health, University of Oxford, Oxford, England; and §Department of Vascular Medicine, Hemophilia Comprehensive Care Treatment Center, AMC, Amsterdam, the Netherlands

The Outcomes and Experiences Questionnaire: development and validation

Background This report presents evidence regarding the development and validation of a new questionnaire, the Outcomes and Experiences Questionnaire (OEQ). The rationale for the questionnaire is to bring together into one short instrument questions about two distinct domains – patients’ reports of the outcomes of their care and how they experience care. Methods The OEQ was developed from literature reviews, iterative drafting and discussion within the research group and cognitive testing with a sample of patients who had a hospital experience. Two validation studies were carried out with an eleven item OEQ. The goals of the studies were to examine response rates and to test specific hypotheses of how OEQ should relate to other variables normally collected in the two studies. In the first study, the OEQ was added to the follow-up questionnaires for patients (n=490) receiving surgery for hip or knee replacement or varicose vein procedures participating in the national Patient Reported Outcome Measures (PROMs) program permitting the analysis of the OEQ against change scores for the measures obtained before and after surgery. In the second study the OEQ was included in a sample of patients (n=586) who had been selected to receive the National Health Service (NHS) inpatient survey from three contrasting hospital trusts. Results Results from study one provided consistent and substantial evidence of construct validity of OEQ particularly for those receiving hip or knee replacement. The OEQ sub-scales behaved differently and as predicted against other PROMs variables. Again hypotheses of how the two sub-scales regarding outcomes and experiences would relate to the existing domains of patient experience in the inpatient survey were broadly confirmed in study two. Conclusion The report provides encouraging evidence of the OEQ’s capacity to assess distinct reports from patients about outcomes and experiences of care within a single short questionnaire.

Patients’ attitudes and perceptions of two health-related quality-of-life questionnaires used to collect patient-reported outcome measures in the English National Health Service: A qualitative study of patients undergoing cardiac interventions

Objectives: To explore patients’ views on the EuroQol-5D and Coronary Revascularisation Outcome Questionnaire, tools currently used for collecting patient-reported outcome measures in the English National Health Service. The key questions were as follows: (1) whether patients consider them sensitive enough to detect change in their health after cardiovascular disease interventions and (2) whether they consider the health-related quality-of-life questions as meaningful. Methods: Data were collected on patients’ views using focus groups. We held four focus groups selecting participants on the basis of their baseline and follow-up EuroQol-5D scores. Data were analysed using framework analysis and grounded theory. Results: Focus group participants confirmed that they had derived substantial health benefits from their cardiac interventions despite the lack of measurable effects on the EuroQol-5D scores. Participants felt that the EuroQol-5D questionnaire was limited because of the following reasons: Their health fluctuates from day to day. They had difficulty assessing their general health status on the visual analogue scale. They felt that the Coronary Revascularisation Outcome Questionnaire was limited because of the following reasons: They did not understand the clinical terms used. The impact of tiredness on their quality of life was not captured. They were unable to distinguish between the effects of their heart condition and other health issues. Additionally, neither questionnaire considers the adjustments people have made to their domestic arrangements to improve their health-related quality of life. Conclusion: This study provides evidence that the two questionnaires do not capture some aspects of health that patients consider important. Furthermore, the presence of co-morbidities masks the symptoms relating to the heart disease and the effect of their cardiac interventions. Future work on patient-reported outcome measures should consider developing new questionnaires that address these major concerns.

The Long-Term Conditions Questionnaire: conceptual framework and item development

Purpose To identify the main issues of importance when living with long-term conditions to refine a conceptual framework for informing the item development of a patient-reported outcome measure for long-term conditions. Materials and methods Semi-structured qualitative interviews (n=48) were conducted with people living with at least one long-term condition. Participants were recruited through primary care. The interviews were transcribed verbatim and analyzed by thematic analysis. The analysis served to refine the conceptual framework, based on reviews of the literature and stakeholder consultations, for developing candidate items for a new measure for long-term conditions. Results Three main organizing concepts were identified: impact of long-term conditions, experience of services and support, and self-care. The findings helped to refine a conceptual framework, leading to the development of 23 items that represent issues of importance in long-term conditions. The 23 candidate items formed the first draft of the measure, currently named the Long-Term Conditions Questionnaire. Conclusion The aim of this study was to refine the conceptual framework and develop items for a patient-reported outcome measure for long-term conditions, including single and multiple morbidities and physical and mental health conditions. Qualitative interviews identified the key themes for assessing outcomes in long-term conditions, and these underpinned the development of the initial draft of the measure. These initial items will undergo cognitive testing to refine the items prior to further validation in a survey.