Caroline B. Terwee

The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study

BackgroundAim of the COSMIN study (COnsensus-based Standards for the selection of health status Measurement INstruments) was to develop a consensus-based checklist to evaluate the methodological quality of studies on measurement properties. We present the COSMIN checklist and the agreement of the panel on the items of the checklist.MethodsA four-round Delphi study was performed with international experts (psychologists, epidemiologists, statisticians and clinicians). Of the 91 invited experts, 57 agreed to participate (63%). Panel members were asked to rate their (dis)agreement with each proposal on a five-point scale. Consensus was considered to be reached when at least 67% of the panel members indicated ‘agree’ or ‘strongly agree’.ResultsConsensus was reached on the inclusion of the following measurement properties: internal consistency, reliability, measurement error, content validity (including face validity), construct validity (including structural validity, hypotheses testing and cross-cultural validity), criterion validity, responsiveness, and interpretability. The latter was not considered a measurement property. The panel also reached consensus on how these properties should be assessed.ConclusionsThe resulting COSMIN checklist could be useful when selecting a measurement instrument, peer-reviewing a manuscript, designing or reporting a study on measurement properties, or for educational purposes.

Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist

BackgroundThe COSMIN checklist is a standardized tool for assessing the methodological quality of studies on measurement properties. It contains 9 boxes, each dealing with one measurement property, with 5–18 items per box about design aspects and statistical methods. Our aim was to develop a scoring system for the COSMIN checklist to calculate quality scores per measurement property when using the checklist in systematic reviews of measurement properties.MethodsThe scoring system was developed based on discussions among experts and testing of the scoring system on 46 articles from a systematic review. Four response options were defined for each COSMIN item (excellent, good, fair, and poor). A quality score per measurement property is obtained by taking the lowest rating of any item in a box (“worst score counts”).ResultsSpecific criteria for excellent, good, fair, and poor quality for each COSMIN item are described. In defining the criteria, the “worst score counts” algorithm was taken into consideration. This means that only fatal flaws were defined as poor quality. The scores of the 46 articles show how the scoring system can be used to provide an overview of the methodological quality of studies included in a systematic review of measurement properties.ConclusionsBased on experience in testing this scoring system on 46 articles, the COSMIN checklist with the proposed scoring system seems to be a useful tool for assessing the methodological quality of studies included in systematic reviews of measurement properties.

The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content.

BackgroundThe COSMIN checklist (COnsensus-based Standards for the selection of health status Measurement INstruments) was developed in an international Delphi study to evaluate the methodological quality of studies on measurement properties of health-related patient reported outcomes (HR-PROs). In this paper, we explain our choices for the design requirements and preferred statistical methods for which no evidence is available in the literature or on which the Delphi panel members had substantial discussion.MethodsThe issues described in this paper are a reflection of the Delphi process in which 43 panel members participated.ResultsThe topics discussed are internal consistency (relevance for reflective and formative models, and distinction with unidimensionality), content validity (judging relevance and comprehensiveness), hypotheses testing as an aspect of construct validity (specificity of hypotheses), criterion validity (relevance for PROs), and responsiveness (concept and relation to validity, and (in) appropriate measures).ConclusionsWe expect that this paper will contribute to a better understanding of the rationale behind the items, thereby enhancing the acceptance and use of the COSMIN checklist.

Minimal changes in health status questionnaires: distinction between minimally detectable change and minimally important change

Changes in scores on health status questionnaires are difficult to interpret. Several methods to determine minimally important changes (MICs) have been proposed which can broadly be divided in distribution-based and anchor-based methods. Comparisons of these methods have led to insight into essential differences between these approaches. Some authors have tried to come to a uniform measure for the MIC, such as 0.5 standard deviation and the value of one standard error of measurement (SEM). Others have emphasized the diversity of MIC values, depending on the type of anchor, the definition of minimal importance on the anchor, and characteristics of the disease under study. A closer look makes clear that some distribution-based methods have been merely focused on minimally detectable changes. For assessing minimally important changes, anchor-based methods are preferred, as they include a definition of what is minimally important. Acknowledging the distinction between minimally detectable and minimally important changes is useful, not only to avoid confusion among MIC methods, but also to gain information on two important benchmarks on the scale of a health status measurement instrument. Appreciating the distinction, it becomes possible to judge whether the minimally detectable change of a measurement instrument is sufficiently small to detect minimally important changes.

Physical activity questionnaires for adults: a systematic review of measurement properties

Many questionnaires have been developed to measure physical activity (PA), but an overview of the measurement properties of PA questionnaires is lacking. A summary of this information is useful for choosing the best questionnaire available. Therefore, the objective of this study was to evaluate and compare measurement properties of self-administered questionnaires assessing PA in adults. We searched MEDLINE, EMBASE and SportDiscus®, using ‘exercise’, ‘physical activity’, ‘motor activity’ and ‘questionnaire’ as keywords. We included studies that evaluated the measurement properties of self-report questionnaires assessing PA. Article selection, data extraction and quality assessment were performed by two independent reviewers. The quality and results of the studies were evaluated using the Quality Assessment of Physical Activity Questionnaires (QAPAQ) checklist. Construct validity, reliability and responsiveness were rated as positive, negative or indeterminate, depending on the methods and results. We included 85 (versions of) questionnaires. Overall, the quality of the studies assessing measurement properties of PA questionnaires was rather poor. Information on content validity was mostly lacking. Construct validity was assessed in 76 of the questionnaires, mostly by correlations with accelerometer data, maximal oxygen uptake or activity diaries. Fifty-one questionnaires were tested for reliability. Only a few questionnaires had sufficient construct validity and reliability, but these need to be further validated. Responsiveness was studied for only two questionnaires and was poor. There is a clear lack of standardization of PA questionnaires, resulting in many variations of questionnaires. No questionnaire or type of questionnaire for assessing PA was superior and therefore could not be strongly recommended above others. In the future, more attention should be paid to the methodology of studies assessing measurement properties of PA questionnaires and the quality of reporting.

Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments

ObjectivesFor the measurement of patient-reported outcomes, such as (health-related) quality of life, often many measurement instruments exist that intend to measure the same construct. To facilitate instrument selection, our aim was to develop a highly sensitive search filter for finding studies on measurement properties of measurement instruments in PubMed and a more precise search filter that needs less abstracts to be screened, but at a higher risk of missing relevant studies.MethodsA random sample of 10,000 PubMed records (01-01-1990 to 31-12-2006) was used as a gold standard. Studies on measurement properties were identified using an exclusion filter and hand searching. Search terms were selected from the relevant records in the gold standard as well as from 100 systematic reviews of measurement properties and combined based on sensitivity and precision. The performance of the filters was tested in the gold standard as well as in two validation sets, by calculating sensitivity, precision, specificity, and number needed to read.ResultsWe identified 116 studies on measurement properties in the gold standard. The sensitive search filter was able to retrieve 113 of these 116 studies (sensitivity 97.4%, precision 4.4%). The precise search filter had a sensitivity of 93.1% and a precision of 9.4%. Both filters performed very well in the validation sets.ConclusionThe use of these search filters will contribute to evidence-based selection of measurement instruments in all medical fields.

Effectiveness of exercise therapy: A best-evidence summary of systematic reviews

The purpose of this project was to summarise the available evidence on the effectiveness of exercise therapy for patients with disorders of the musculoskeletal, nervous, respiratory, and cardiovascular systems. Systematic reviews were identified by means of a comprehensive search strategy in 11 bibliographic databases (08/2002), in combination with reference tracking. Reviews that included (i) at least one randomised controlled trial investigating the effectiveness of exercise therapy, (ii) clinically relevant outcome measures, and (iii) full text written in English, German or Dutch, were selected by two reviewers. Thirteen independent and blinded reviewers participated in the selection, quality assessment and data-extraction of the systematic reviews. Conclusions about the effectiveness of exercise therapy were based on the results presented in reasonable or good quality systematic reviews (quality score > or = 60 out of 100 points). A total of 104 systematic reviews were selected, 45 of which were of reasonable or good quality. Exercise therapy is effective for patients with knee osteoarthritis, sub-acute (6 to 12 weeks) and chronic (> or = 12 weeks) low back pain, cystic fibrosis, chronic obstructive pulmonary disease, and intermittent claudication. Furthermore, there are indications that exercise therapy is effective for patients with ankylosing spondylitis, hip osteoarthritis, Parkinsons disease, and for patients who have suffered a stroke. There is insufficient evidence to support or refute the effectiveness of exercise therapy for patients with neck pain, shoulder pain, repetitive strain injury, rheumatoid arthritis, asthma, and bronchiectasis. Exercise therapy is not effective for patients with acute low back pain. It is concluded that exercise therapy is effective for a wide range of chronic disorders.

Physical Activity Questionnaires for Youth: A Systematic Review of Measurement Properties

Because of the diversity in available questionnaires, it is not easy for researchers to decide which instrument is most suitable for his or her specific demands. Therefore, we systematically summarized and appraised studies examining measurement properties of self-administered and proxy-reported physical activity (PA) questionnaires in youth.Literature was identified through searching electronic databases (PubMed, EMBASE using ‘EMBASE only’ and Sport Discus®) until May 2009. Studies were included if they reported on the measurement properties of self-administered and proxy-reported PA questionnaires in youth (mean age <18 years) and were published in the English language. Methodological quality and results of included studies was appraised using a standardized checklist (qualitative attributes and measurement properties of PA questionnaires [QAPAQ]).We included 54 manuscripts examining 61 versions of questionnaires. None of the included questionnaires showed both acceptable reliability and validity. Only seven questionnaires received a positive rating for reliability. Reported validity varied, with correlations between PA questionnaires and accelerometers ranging from very low to high (previous day PA recall: correlation coefficient [r] = 0.77). In general, PA questionnaires for adolescents correlated better with accelerometer scores than did those for children.From this systematic review, we conclude that no questionnaires were available with both acceptable reliability and validity. Considerably more high-quality research is required to examine the validity and reliability of promising PA questionnaires for youth.

The Dutch version of the knee injury and osteoarthritis outcome score: A validation study

BackgroundThe Knee Injury and Osteoarthritis Outcome Score (KOOS) was constructed in Sweden. This questionnaire has proved to be valid for several orthopedic interventions of the knee. It has been formally translated and validated in several languages, but not yet in Dutch. The purpose of the present study was to evaluate the clinimetric properties of the Dutch version of the KOOS questionnaire in knee patients with various stages of osteoarthritis (OA).MethodsThe Swedish version of the KOOS questionnaire was first translated into Dutch according to a standardized procedure and second tested for clinimetric quality. The study population consisted of patients with different stages of OA (mild, moderate and severe) and of patients after primary TKA, and after a revision of the TKA. All patients filled in the Dutch KOOS questionnaire, as well as the SF-36 and a Visual Analogue Scale for pain. The following analyses were performed to evaluate the clinimetric quality of the KOOS: Cronbachs alpha (internal consistency), principal component analyses (factor analysis), intraclass correlation coefficients (reliability), spearmans correlation coefficient (construct validity), and floor and ceiling effects.ResultsFor all patients groups Cronbachs alpha was for all subscales above 0.70. The ICCs, assessed for the patient groups with mild and moderate OA and after revision of the TKA patients, were above 0.70 for all subscales. Of the predefined hypotheses 60% or more could be confirmed for the patients with mild and moderate OA and for the TKA patients. For the other patient groups less than 45% could be confirmed. Ceiling effects were present in the mild OA group for the subscales Pain, Symptoms and ADL and for the subscale Sport/Recreation in the severe OA group. Floor effects were found for the subscales Sport/Recreation and Qol in the severe OA and revision TKA groups.ConclusionBased on these different clinimetric properties within the present study we conclude that the KOOS questionnaire seems to be suitable for patients with mild and moderate OA and for patients with a primary TKA. The Dutch version of the KOOS had a lower construct validity for patients with severe OA on a waiting list for TKA and patients after revision of a TKA. Further validation studies on the Dutch version of the KOOS should also include a knee specific questionnaire for assessing the construct validity.

Are factor analytical techniques used appropriately in the validation of health status questionnaires? A systematic review on the quality of factor analysis of the SF-36

Factor analysis is widely used to evaluate whether questionnaire items can be grouped into clusters representing different dimensions of the construct under study. This review focuses on the appropriate use of factor analysis. The Medical Outcomes Study Short Form-36 (SF-36) is used as an example. Articles were systematically searched and assessed according to a number of criteria for appropriate use and reporting. Twenty-eight studies were identified: exploratory factor analysis was performed in 22 studies, confirmatory factor analysis was performed in five studies and in one study both were performed. Substantial shortcomings were found in the reporting and justification of the methods applied. In 15 of the 23 studies in which exploratory factor analysis was performed, confirmatory factor analysis would have been more appropriate. Cross-validation was rarely performed. Presentation of the results and conclusions was often incomplete. Some of our results are specific for the SF-36, but the finding that both the application and the reporting of factor analysis leaves much room for improvement probably applies to other health status questionnaires as well. Optimal reporting and justification of methods is crucial for correct interpretation of the results and verification of the conclusions. Our list of criteria may be useful for journal editors, reviewers and researchers who have to assess publications in which factor analysis is applied.