Jørgen Hilden

Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors

Background: Clinical trials are commonly done without blinded outcome assessors despite the risk of bias. We wanted to evaluate the effect of nonblinded outcome assessment on estimated effects in randomized clinical trials with outcomes that involved subjective measurement scales. Methods: We conducted a systematic review of randomized clinical trials with both blinded and nonblinded assessment of the same measurement scale outcome. We searched PubMed, EMBASE, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press and Google Scholar for relevant studies. Two investigators agreed on the inclusion of trials and the outcome scale. For each trial, we calculated the difference in effect size (i.e., standardized mean difference between nonblinded and blinded assessments). A difference in effect size of less than 0 suggested that nonblinded assessors generated more optimistic estimates of effect. We pooled the differences in effect size using inverse variance random-effects meta-analysis and used metaregression to identify potential reasons for variation. Results: We included 24 trials in our review. The main meta-analysis included 16 trials (involving 2854 patients) with subjective outcomes. The estimated treatment effect was more beneficial when based on nonblinded assessors (pooled difference in effect size −0.23 [95% confidence interval (CI) −0.40 to −0.06]). In relative terms, nonblinded assessors exaggerated the pooled effect size by 68% (95% CI 14% to 230%). Heterogeneity was moderate (I2 = 46%, p = 0.02) and unexplained by metaregression. Interpretation: We provide empirical evidence for observer bias in randomized clinical trials with subjective measurement scale outcomes. A failure to blind assessors of outcomes in such trials results in a high risk of substantial bias.

Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors

Objective To evaluate the impact of non-blinded outcome assessment on estimated treatment effects in randomised clinical trials with binary outcomes. Design Systematic review of trials with both blinded and non-blinded assessment of the same binary outcome. For each trial we calculated the ratio of the odds ratios—the odds ratio from non-blinded assessments relative to the corresponding odds ratio from blinded assessments. A ratio of odds ratios <1 indicated that non-blinded assessors generated more optimistic effect estimates than blinded assessors. We pooled the individual ratios of odds ratios with inverse variance random effects meta-analysis and explored reasons for variation in ratios of odds ratios with meta-regression. We also analysed rates of agreement between blinded and non-blinded assessors and calculated the number of patients needed to be reclassified to neutralise any bias. Data Sources PubMed, Embase, PsycINFO, CINAHL, Cochrane Central Register of Controlled Trials, HighWire Press, and Google Scholar. Eligibility criteria for selecting studies Randomised clinical trials with blinded and non-blinded assessment of the same binary outcome. Results We included 21 trials in the main analysis (with 4391 patients); eight trials provided individual patient data. Outcomes in most trials were subjective—for example, qualitative assessment of the patient’s function. The ratio of the odds ratios ranged from 0.02 to 14.4. The pooled ratio of odds ratios was 0.64 (95% confidence interval 0.43 to 0.96), indicating an average exaggeration of the non-blinded odds ratio by 36%. We found no significant association between low ratios of odds ratios and scores for outcome subjectivity (P=0.27); non-blinded assessor’s overall involvement in the trial (P=0.60); or outcome vulnerability to non-blinded patients (P=0.52). Blinded and non-blinded assessors agreed in a median of 78% of assessments (interquartile range 64-90%) in the 12 trials with available data. The exaggeration of treatment effects associated with non-blinded assessors was induced by the misclassification of a median of 3% of the assessed patients per trial (1-7%). Conclusions On average, non-blinded assessors of subjective binary outcomes generated substantially biased effect estimates in randomised clinical trials, exaggerating odds ratios by 36%. This bias was compatible with a high rate of agreement between blinded and non-blinded outcome assessors and driven by the misclassification of few patients.

The Area under the ROC Curve and Its Competitors

The area under the receiver operating characteristic (ROC) curve is a popular measure of the power of a (two-disease) diagnostic test, but it is shown here to be an inconsistent criterion: tests of indistinguishable clinical impacts may have different areas. A class of diagnosticity measures (DMs) of proven optimality is proposed instead. Once a regret(-like) measure of diagnostic uncertainty is agreed upon, the associated DM is uniquely defined and, indeed, calculable from the ROC curve configuration. Two scaled variants of the ROC are introduced and used to advantage in the analysis. They may also be helpful to students of medical decision making.

Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies

BACKGROUND Blinding patients in clinical trials is a key methodological procedure, but the expected degree of bias due to nonblinded patients on estimated treatment effects is unknown. METHODS Systematic review of randomized clinical trials with one sub-study (i.e. experimental vs control) involving blinded patients and another, otherwise identical, sub-study involving nonblinded patients. Within each trial, we compared the difference in effect sizes (i.e. standardized mean differences) between the sub-studies. A difference <0 indicates that nonblinded patients generated a more optimistic effect estimate. We pooled the differences with random-effects inverse variance meta-analysis, and explored reasons for heterogeneity. RESULTS Our main analysis included 12 trials (3869 patients). The average difference in effect size for patient-reported outcomes was -0.56 (95% confidence interval -0.71 to -0.41), (I(2)=60%, P=0.004), i.e. nonblinded patients exaggerated the effect size by an average of 0.56 standard deviation, but with considerable variation. Two of the 12 trials also used observer-reported outcomes, showing no indication of exaggerated effects due lack of patient blinding. There was a larger effect size difference in 10 acupuncture trials [-0.63 (-0.77 to -0.49)], than in the two non-acupuncture trials [-0.17 (-0.41 to 0.07)]. Lack of patient blinding also increased attrition and use of co-interventions: ratio of control group attrition risk 1.79 (1.18 to 2.70), and ratio of control group co-intervention risk 1.55 (0.99 to 2.43). CONCLUSIONS This study provides empirical evidence of pronounced bias due to lack of patient blinding in complementary/alternative randomized clinical trials with patient-reported outcomes.

Comparison of ultrasound, radiography, and clinical examination in the diagnosis of acute maxillary sinusitis: a systematic review

The objective of this study was to assess the discriminative properties of the methods for diagnosing acute maxillary sinusitis (AMS) in unselected patients. The study design was a systematic review of evaluation studies identified by using Medline, by searching reference lists, by hand searches, and by contacting investigators. Evaluation studies were conducted anywhere in the world. Subjects were adults with suspected AMS. Main outcome measures were: sensitivity, specificity, positive and negative likelihood ratios of the primary studies, weighted means of these parameters in each comparison (clinical examination, radiography, and ultrasound compared to a reference standard in diagnosing AMS), and summary ROC curves and their Q* points where sensitivity equals specificity. For the years from 1962 to present, 49 study reports were found; 11 articles on studies that included a total of 1144 patients were eligible. Compared to sinus puncture, radiography was the most accurate method for diagnosing AMS: the Q* point on the summary ROC curve was 0.82 (95% confidence interval, CI, 0.78-0.85). Ultrasound was slightly less accurate than radiography compared to sinus puncture (Q* 0.80, 95% CI 0.76-0.83). Only two articles reported clinical examination compared to sinus puncture and the Q* for them was 0.75 (95% 0.58-0.86). Clinical examination is a rather unreliable method for diagnosing AMS, even in the hands of experienced specialists. Using radiography or ultrasound improves the accuracy of diagnosis. The diagnosis of AMS is rarely studied in primary care settings. Future comparative trials should preferably combine diagnosis and treatment, evaluating the two aspects of clinical management as unit.

Risk assessment of left ventricular systolic dysfunction in primary care : cross sectional study evaluating a range of diagnostic tests

Abstract Objectives: To assess the probability of left ventricular systolic dysfunction without echocardiography in patients from general practice. Design: Cross sectional study using multivariate regression models to examine the relation between clinical variables and left ventricular systolic dysfunction as determined by echocardiography. Setting: Three general practices in Copenhagen. Subjects: 2158 patients aged >40 years were screened by questionnaires and case record reviews; 357 patients with past or present signs or symptoms of heart disease were identified, of whom 126 were eligible for and consented to examination. Main outcome measures: Clinical variables that were significantly (P<0.05) related to ejection fraction 0.45 and their predictive value for left ventricular systolic dysfunction. Results: 15 patients (12%) had left ventricular systolic dysfunction. The prevalence was significantly related to three questions: does the electrocardiogram have Q waves, left bundle branch block, or ST-T segment changes? (P=0.012); is resting supine heart rate greater than the simultaneous diastolic blood pressure? (P=0.002); and is plasma N-terminal atrial natriuretic peptide>0.8 nmol/l? (P=0.040)? Only one of 60 patients with a normal electrocardiogram had systolic dysfunction (2%, 95% confidence interval 0% to 9%) regardless of response to the other two questions. The risk of dysfunction was appreciable in patients with a yes answer to two or three questions (50%, 27% to 73%). Conclusions: A normal electrocardiogram implies a low risk of left ventricular systolic dysfunction. Patients can be identified for echocardiography on the basis of an abnormal electrocardiogram combined with increased natriuretic peptide concentration or a heart rate greater than diastolic blood pressure, or both. Key messages Early treatment of left ventricular systolic dysfunction reduces morbidity, but diagnosis relies on echocardiography This study examines methods for assessing the risk of left ventricular systolic dysfunction in patients from primary care with past or present signs or symptoms of heart disease Risk can be assessed by three factors: QRS or ST-T changes in the electrocardiogram; increased plasma concentration of N-terminal atrial natriuretic peptide; and tachycardia (supine resting heart rate>diastolic blood pressure) Risk of systolic dysfunction was very low in patients with normal electrocardiographic results Risk was high in patients who had an abnormal electrocardiogram in combination with at least one other abnormal result

Cross sectional study estimating prevalence of heart failure and left ventricular systolic dysfunction in community patients at risk

OBJECTIVE To examine a general practice population to measure the prevalence of signs and symptoms of heart failure (SSHF) and left ventricular systolic dysfunction (LVSD). DESIGN Cross sectional screening study in three general practices followed by echocardiography. SETTING AND PATIENTS All patients ⩾ 50 years in two general practices and ⩾ 40 years in one general practice were screened by case record reviews and questionnaires (n = 2158), to identify subjects with some evidence of heart disease. Among these, subjects were sought who had SSHF (n = 115). Of 357 subjects with evidence of heart disease, 252 were eligible for examination, and 126 underwent further cardiological assessment, including 43 with SSHF. MAIN OUTCOME MEASURES Prevalence of SSHF as defined by a modified Boston index, LVSD defined as an indirectly measured left ventricular ejection fraction ⩽ 0.45, and numbers of subjects needing an echocardiogram to detect one case with LVSD. RESULTS SSHF afflicted 0.5% of quadragenarians and rose to 11.7% of octogenarians. Two thirds were handled in primary care only. At ⩾ 50 years of age 6.4% had SSHF, 2.9% had LVSD, and 1.9% (95% confidence interval 1.3% to 2.5%) had both. To detect one case with LVSD in primary care, 14 patients with evidence of heart disease without SSHF and 5.5 patients with SSHF had to be examined. CONCLUSION SSHF is extremely prevalent in the community, especially in primary care, but more than two thirds do not have LVSD. The number of subjects with some evidence of heart disease needing an echocardiogram to detect one case of LVSD is 14.

Sudden infant death in Copenhagen 1956--1971. I. Infant feeding.

ABSTRACT. 131 cases of the Sudden Infant Death Syndrome (SIDS) in the municipality of Copenhagen 1956–71 (incidence 0.92 SIDS cases per 1000 live births) were investigated on the basis of police reports and infant health visitors records. Fewer SIDS cases were breast‐fed than controls from the second week of life to four months of age. No significant differences were found with respect to the concentrations of fresh cows milk dilutions, age at introduction of solid food, or number of meals per day. From 1956 to 1971 breastfeeding was declining and solids introduced earlier, while the incidence of SIDS remained constant. The lifespan for SIDS cases who had never been breast‐fed was equal to that of cases who had. There is no evidence that SIDS victims had their first solid food during the last days of life. The results are discussed in the light of the hypersensitivity, immuno‐incompetence, and high solute feeding hypotheses. It is concluded that feeding does not seem to be responsible for the occurrence of SIDS. The lower frequency of breast‐feeding among SIDS cases is ascribed to various factors known to be associated with SIDS.

Threshold electrical stimulation (TES) in ambulant children with CP: a randomized double-blind placebo-controlled clinical trial

A randomized double‐blind placebo‐controlled clinical trial was carried out to determine whether a group of stable children with cerebral palsy (36 males, 21 females; mean age 10 years 11 months, range 5 to 18 years) would improve their motor skills after 12 months of threshold electrical stimulation (TES). Two thirds received active and one third received inactive stimulators. For the primary outcome we constructed a set of plausible motor function tests and studied the change in summary indices of the performance measurements. Tests were videotaped and assessed blindly to record qualitative changes that might not be reflected in performance measurements. We also judged range of motion, degree of spasticity, and muscle growth measured by CT. Fifty seven of 82 outpatients who were able to walk at least with a walker, completed all 12 months of treatment (hemiplegia n=25, diplegia n=32). There was no significant difference between active and placebo treatment in any of the tested groups, nor combined. Visual and subjective assessments favoured TES (ns), whereas objective indices showed the opposite trend. We conclude that TES in these patients did not have any significant clinical effect during the test period.

A controlled comparison of myelography, computed tomography, and magnetic resonance imaging in clinically suspected lumbar disc herniation

Study design. A controlled prospective blinded study. Objectives. To compare the diagnostic power of myelography, computed tomography and magnetic resonance imaging in the diagnosis of low lumbar disc herniation. Methods. Eighty patients with monoradicular sciatica were examined by myelography, computed tomography, and magnetic resonance imaging, and all underwent subsequent surgery. The images were evaluated twice in a blinded fashion, and the diagnostic power of the modalities was expressed by a decision-analytic regret function. Results. In 57 patients (71%) a disc herniation at the expected level was disclosed at surgery. The largest amount of diagnostic information was gained from computed tomography, followed by magnetic resonance imaging and myelography. Both computed tomography and magnetic resonance imaging were significantly informative, whereas this was not the case for myelography. Conclusion. The results indicate that computed tomography or magnetic resonance imaging should be the first choice for imaging in patients with suspected lumbar disc herniation.