Rudolf W. Poolman

What makes a measurement instrument valid and reliable

High quality instruments are useful tools for clinical and research purposes. To determine whether an instrument has high quality, measurement properties such as reliability and validity need to be assessed, using standardised criteria. This paper discusses these quality domains and measurement properties using the standardised criteria that were recently published by the COSMIN group. Examples are given of studies evaluating the measurement properties of instruments frequently used in trauma. This paper presents a helpful tool for readers who want to evaluate or assess the quality of a measurement instrument on reliability and validity.

Surgical Compared with Conservative Treatment for Acute Nondisplaced or Minimally Displaced Scaphoid Fractures: A Systematic Review and Meta-Analysis of Randomized Controlled Trials

BACKGROUND There is a current trend in orthopaedic practice to treat nondisplaced or minimally displaced fractures with early open reduction and internal fixation instead of cast immobilization. This trend is not evidence-based. In this systematic review and meta-analysis, we pool data from trials comparing surgical and conservative treatment for acute nondisplaced and minimally displaced scaphoid fractures, thus aiming to summarize the best available evidence. METHODS A systematic literature search of the medical literature from 1966 to 2009 was performed. We selected eight randomized controlled trials comparing surgical with conservative treatment for acute nondisplaced or minimally displaced scaphoid fractures in adults. Data from included studies were pooled with use of fixed-effects and random-effects models with standard mean differences and risk ratios for continuous and dichotomous variables, respectively. Heterogeneity across studies was assessed with calculation of the I(2) statistic. RESULTS Four hundred and nineteen patients from eight trials were included. Two hundred and seven patients were treated surgically, and 212 were treated conservatively. Most trials lacked scientific rigor. Our primary outcome parameter, standardized functional outcome, which was assessed for 247 patients enrolled in four trials, significantly favored surgical treatment (p < 0.01). With regard to our secondary parameters, we found heterogeneous results that favored surgical treatment in terms of satisfaction (assessed in one study), grip strength (six studies), time to union (three studies), and time off work (five studies). In contrast, we found no significant differences between surgical and conservative treatment with regard to pain (two studies), range of motion (six studies), the rates of nonunion (six studies) and malunion (seven studies), and total treatment costs (two studies). The rate of complications was higher in the surgical treatment group (23.7%) than in the conservative group (9.1%), although this difference was not significant (p = 0.13). There was a nearly significantly higher rate of scaphotrapezial osteoarthritis in the surgical treatment group (p = 0.05). CONCLUSIONS Based on primary studies with limited methodological quality, this study suggests that surgical treatment is favorable for acute nondisplaced and minimally displaced scaphoid fractures with regard to functional outcome and time off work; however, surgical treatment engenders more complications. Thus, the long-term risks and short-term benefits of surgery should be carefully weighed in clinical decision-making.

The Harris hip score: Do ceiling effects limit its usefulness in orthopedics?

Background and purpose The Harris hip score (HHS), a disease-specific health status scale that is frequently used to measure the outcome of total hip arthroplasty, has never been validated properly. A questionnaire is suitable only when all 5 psychometric properties are of sufficient quality. We questioned the usefulness of the HHS by investigating its content validity. Methods We performed a systematic review based on a literature search in PubMed, Embase, and the Cochrane Library for descriptive studies published in 2007. 54 studies (59 patient groups) met our criteria and were included in the data analysis. To determine the content validity, we calculated the ceiling effect (percentage) for each separate study and we pooled data to measure the weighted mean. A subanalysis of indications for THA was performed to differentiate the populations for which the HHS would be suitable and for which it would not. A ceiling effect of 15% or less was considered to be acceptable. Results Over half the studies (31/59) revealed unacceptable ceiling effects. Pooled data across the studies included (n = 6,667 patients) suggested ceiling effects of 20% (95%CI: 18–22). Ceiling effects were greater (32%, 95%CI:12–52) in those patients undergoing hip resurfacing arthroplasty. Interpretation Although the Harris hip score is widely used in arthroplasty research on outcomes, ceiling effects are common and these severely limit its validity in this field of research.

Reporting of outcomes in orthopaedic randomized trials : Does blinding of outcome assessors matter?

BACKGROUND Randomization, concealment of treatment allocation, and blinding are all known to limit bias in clinical research. Nonsurgical studies that fail to meet these standards have been reported to inflate the differences between treatment and control groups. While surgical trials can rarely blind surgeons or patients, they can often blind outcome assessors. The aim of this systematic review was threefold: (1) to examine the reporting of outcome measures in orthopaedic trials, (2) to determine the feasibility of blinding in published orthopaedic trials, and (3) to examine the association between the magnitude of treatment differences and the blinding of outcome assessors. METHODS We identified and reviewed thirty-two randomized, controlled trials published in The Journal of Bone and Joint Surgery (American Volume) in 2003 and 2004 for the appropriate use of outcome measures. These trials represented 3.4% of all 938 studies published during that time-period. All thirty-two trials were reviewed by two authors for (1) the outcome measures used and (2) the blinding of outcomes assessors. We calculated the magnitude of the treatment effect of the use of blinded compared with unblinded outcome assessors. RESULTS Ten (31%) of the thirty-two randomized controlled trials used a modified outcome instrument. Of the ten trials, four failed to describe how the outcome instrument was modified. Nine of the ten articles did not describe how the modified instrument was validated and retested. Sixteen of the thirty-two randomized controlled trials did not report blinding of outcome assessors when blinding would have been possible. Among the studies with continuous outcome measure, unblinded outcomes assessment was associated with significantly larger treatment effects than blinded outcomes assessment (standardized mean difference, 0.76 compared with 0.25; p = 0.01). Similarly, in the studies with dichotomous outcomes, unblinded outcomes assessments were associated with significantly greater treatment effects than blinded outcomes assessments (odds ratio, 0.13 compared with 0.42; p < 0.001). The ratio of odds ratios (unblinded to blinded outcomes assessment) was 0.31, suggesting that unblinded outcomes assessment was associated with a potential for exaggeration of the benefit of the effectiveness of a treatment in our cohort of studies. CONCLUSIONS In future orthopaedic randomized controlled trials, emphasis should be placed on detailed reporting of outcome measures to facilitate generalization and the outcome assessors should be blinded, when possible, to limit bias.

Does a "Level I Evidence" rating imply high quality of reporting in orthopaedic randomised controlled trials?

BackgroundThe Levels of Evidence Rating System is widely believed to categorize studies by quality, with Level I studies representing the highest quality evidence. We aimed to determine the reporting quality of Randomised Controlled Trials (RCTs) published in the most frequently cited general orthopaedic journals.MethodsTwo assessors identified orthopaedic journals that reported a level of evidence rating in their abstracts from January 2003 to December 2004 by searching the instructions for authors of the highest impact general orthopaedic journals. Based upon a priori eligibility criteria, two assessors hand searched all issues of the eligible journal from 2003–2004 for RCTs. The assessors extracted the demographic information and the evidence rating from each included RCT and scored the quality of reporting using the reporting quality assessment tool, which was developed by the Cochrane Bone, Joint and Muscle Trauma Group. Scores were conducted in duplicate, and we reached a consensus for any disagreements. We examined the correlation between the level of evidence rating and the Cochrane reporting quality score.ResultsWe found that only the Journal of Bone and Joint Surgery – American Volume (JBJS-A) used a level of evidence rating from 2003 to 2004. We identified 938 publications in the JBJS-A from January 2003 to December 2004. Of these publications, 32 (3.4%) were RCTs that fit the inclusion criteria. The 32 RCTs included a total of 3543 patients, with sample sizes ranging from 17 to 514 patients. Despite being labelled as the highest level of evidence (Level 1 and Level II evidence), these studies had low Cochrane reporting quality scores among individual methodological safeguards. The Cochrane reporting quality scores did not differ significantly between Level I and Level II studies. Correlations varied from 0.0 to 0.2 across the 12 items of the Cochrane reporting quality assessment tool (p > 0.05). Among items closely corresponding to the Levels of Evidence Rating System criteria assessors achieved substantial agreement (ICC = 0.80, 95%CI:0.60 to 0.90).ConclusionOur findings suggest that readers should not assume that 1) studies labelled as Level I have high reporting quality and 2) Level I studies have better reporting quality than Level II studies. One should address methodological safeguards individually.

Infected nonunion of the long bones

Background: Although definitions vary, infected nonunion has been defined as a state of failure of union and persistence of infection at the fracture site for 6 to 8 months.>).1 Infected nonunions of the supracondylar region of the femur are uncommon and are mostly due to a severe open fracture with extensive comminution and segmental bone loss or after internal fixation of a comminuted closed fracture.2 Associated factors include exposed bone devoid of vascularized periosteal coverage for more than 6 weeks, purulent discharge, a positive bacteriological culture from the depth of the wound, and histologic evidence of necrotic bone containing empty lacunae. Soft-tissue loss with multiple sinuses, osteomyelitis, osteopenia, complex deformities with limb-length inequality, stiffness of the adjacent joint, polybacterial multidrug-resistant infection, and smoking all complicate treatment and recovery.3 Although uncommon in incidence, infected nonunions of the long bones present a great challenge to the orthopaedic surgeon in providing optimal treatment of this entity. To give direction to the optimal strategy, this systematic review was performed. Objective: We aimed to review the highest level of available evidence on the operative management of infected nonunions of the long bones.

Outcome Instruments: Rationale for Their Use

The number of outcome instruments available for use in orthopaedic observational studies has increased dramatically in recent years. Properly developed and tested outcome instruments provide a very useful tool for orthopaedic research. Criteria have been proposed to assess the measurement properties and quality of health-status instruments. Unfortunately, not all instruments are developed with use of strict quality criteria. In this article, we discuss these quality criteria and provide the reader with a tool to help select the most appropriate instrument for use in an observational study. We also review the steps for future use of outcome instruments, including the standardization of their use in orthopaedic research.

Hamstring tendon autograft better than bone patellar-tendon bone autograft in ACL reconstruction: a cumulative meta-analysis and clinically relevant sensitivity analysis applied to a previously published analysis.

Background Current debate on treatment options for anterior cruciate ligament (ACL) reconstruction complicate the choice between hamstring and bone patellartendon bone autografts. We hypothesized a priori that cumulative meta-analysis (a form of sensitivity analysis) might show that the evidence for reduction of morbidity by hamstring grafts could have been reached at an earlier time. Furthermore, we hypothesized a priori that modern state-of-the-art hamstring graft fixation technique would give similar results regarding stability as bone-patellar tendon-bone autografts. Methods We performed a cumulative meta-analysis and sensitivity analysis based on femoral graft fixation techniques to compare hamstring autograft and bonepatellar tendon-bone autografts in ACL reconstruction derived from a previously published meta-analysis. Results Cumulatively, that hamstring autograft reduces anterior knee pain had already reached statistical significance in 2001 (relative risk 0.49 (95%CI: 0.32–0.76; p = 0.001, I2 = 0%)). The modern endobutton hamstring graft fixation technique (2 studies) yielded similar stability in the Lachman test as bone-patellar tendon-bone grafts, with a relative risk of 1.1 (95%CI: 0.82–1.5; p = 0.6, I2 = 0%). Exclusion of the endobutton group explains the increased laxity in the hamstring graft group. Interpretation Cumulative meta-analysis strengthens the evidence for reduced morbidity using hamstring tendon autograft for anterior cruciate ligament reconstruction. Sensitivity analysis focusing on state-of-the-art hamstring graft fixation techniques further weakens the evidence that bone-patellar tendon-bone autografts provide better stability.

Systematic review: Do patient expectations influence treatment outcomes in total knee and total hip arthroplasty?

ObjectiveThis systematic review aims to summarise all the available evidence related to the association between pre-operative patient expectations (outcome expectations, process expectations and self efficacy expectations) and 5 different treatment outcomes (overall improvement, pain, function, stiffness and satisfaction) in patients with total knee or total hip arthroplasty at three different follow-op periods (>6 weeks; >6 weeks- ≤6 months; >6 months).MethodsEnglish and Dutch language articles were identified through PubMed, EMBASE.com, PsycINFO, CINAHL and The Cochrane Library from inception to September 2012. Articles assessing the association between pre-operative patient expectations and treatment outcomes for TKA/THA in either adjusted or unadjusted analysis were included. Two reviewers, working independently, determined eligibility, rated methodological quality and extracted data on study design, population, expectation measurements, outcome measurements and strength of the associations. Methodological quality was rated by the same reviewers on a 19 item scale. The scores on the quality assessment were taken into account when drawing final conclusions.ResultsThe search strategy generated 2252 unique references, 18 articles met inclusion criteria. Scores on the methodological quality assessment ranged between 6% and 79%. Great variety was seen in definitions and measurement methods of expectations. No significant associations were found between patient expectations and overall improvement, satisfaction and stiffness. Both significant positive and non-significant associations were found for the association between expectations and pain and function.ConclusionsThere was no consistency in the association between patients’ pre-operative expectations and treatment outcomes for TKA and THA indentified in this systematic review. There exists a need for a sound theoretical framework underlying the construct of ‘patient expectations’ and consistent use of valid measurement instruments to measure that construct in order to facilitate future research synthesis.

Methods to diagnose acute anterior cruciate ligament rupture: a meta-analysis of physical examinations with and without anaesthesia.

PurposeThe aims of this meta-analysis were to determine the sensitivity and specificity of the Lachman, pivot shift and anterior drawer test for acutecomplete ACL rupture in the office setting and under anaesthesia. It was hypothesized that the Lachman test is the most sensitive and the pivot shift test the most specific. Secondly, it was hypothesized that the sensitivity and specificity of all three exams increases when the examination is performed under anaesthesia.MethodsAn electronic database search was performed using MEDLINE and EMBASE. All cross-sectional and cohort studies comparing one or more physical examination tests for diagnosing acute complete ACL rupture to an accepted reference standard such as arthroscopy, arthrotomy and MRI were included.ResultsTwenty studies were identified and included. The overall sensitivity of the Lachman test was 0.81 and the specificity 0.81; with anaesthesia, the sensitivity was 0.91 and the specificity 0.78. For the anterior drawer test, the sensitivity was 0.38 and the specificity 0.81; with anaesthesia, the sensitivity was 0.63 and the specificity 0.91. The sensitivity of the pivot shift test was 0.28 and the specificity 0.81; with anaesthesia, the sensitivity was 0.73 and the specificity 0.98.ConclusionIn the office setting, the Lachman test has the highest sensitivity for diagnosing an acute, complete ACL rupture, while all three tests had comparable specificity. When the examination was performed under anaesthesia, the Lachman test still obtained the highest sensitivity, but the pivot shift test was the most specific.Level of evidenceMeta-analysis of diagnostic test accuracy, Level II.