J M Bland

Measuring agreement in method comparison studies.

Agreement between two methods of clinical measurement can be quantified using the differences between observations made using the two methods on the same subjects. The 95% limits of agreement, estimated by mean difference 1.96 standard deviation of the differences, provide an interval within which 95% of differences between measurements by the two methods are expected to lie. We describe how graphical methods can be used to investigate the assumptions of the method and we also give confidence intervals. We extend the basic approach to data where there is a relationship between difference and magnitude, both with a simple logarithmic transformation approach and a new, more general, regression approach. We discuss the importance of the repeatability of each method separately and compare an estimate of this to the limits of agreement. We extend the limits of agreement approach to data with repeated measurements, proposing new estimates for equal numbers of replicates by each method on each subject, for unequal numbers of replicates, and for replicated data collected in pairs, where the underlying value of the quantity being measured is changing. Finally, we describe a nonparametric approach to comparing methods.

Measurement in Medicine: the Analysis of Method Comparison Studies†

In medicine we often want to compare two different methods of measuring some quantity, such as blood pressure, gestational age, or cardiac stroke volume. Sometimes we compare an approximate or simple method with a very precise one. This is a calibration problem, and we shall not discuss it further here. Frequently, however, we cannot regard either method as giving the true value of the quantity being measured. In this case we want to know whether the methods give answers which are, in some sense, comparable. For example, we may wish to see whether a new, cheap and quick method produces answers that agree with those from an established method sufficiently well for clinical purposes. Many such studies, using a variety of statistical techniques, have been reported. Yet few really answer the question “Do the two methods of measurement agree sufficiently closely?” In this paper we shall describe what is usually done, show why this is inappropriate, suggest a better approach, and ask why such studies are done so badly. We will restrict our consideration to the comparison of two methods of measuring a continuous variable, although similar problems can arise with categorical variables.

COMPARING METHODS OF MEASUREMENT: WHY PLOTTING DIFFERENCE AGAINST STANDARD METHOD IS MISLEADING

When comparing a new method of measurement with a standard method, one of the things we want to know is whether the difference between the measurements by the two methods is related to the magnitude of the measurement. A plot of the difference against the standard measurement is sometimes suggested, but this will always appear to show a relation between difference and magnitude when there is none. A plot of the difference against the average of the standard and new measurements is unlikely to mislead in this way. We show this theoretically and by a practical example.

Interaction revisited: the difference between two estimates

We often want to compare two estimates of the same quantity derived from separate analyses. Thus we might want to compare the treatment effect in subgroups in a randomised trial, such as two age groups. The term for such a comparison is a test of interaction. In earlier Statistics Notes we discussed interaction in terms of heterogeneity of treatment effect.1–3 Here we revisit interaction and consider the concept more generally. The comparison of two estimated quantities, such as means or proportions, each with its standard error, is a general method that can be applied widely. The two estimates should be independent, not obtained from the same individuals—examples are the results from subgroups in a randomised trial or from two independent studies. The samples should be large. If the estimates are E 1 and E 2 with standard errors SE( E 1) and SE( E 2), then the difference d = E 1- E 2 has standard error SE( d )=√[SE( E …

Endovascular versus surgical treatment in patients with carotid stenosis in the Carotid and Vertebral Artery Transluminal Angioplasty Study (CAVATAS): a randomised trial

Background: Percutaneous transluminal angioplasty and tenting (endovascular treatment) can be used to treat carotid stenosis, but risks and benefits are uncertain. We therefore compared endovascular treatment with conventional carotid surgery. Methods: In a multicentre clinical trial, we randomly assigned 504 patients with carotid stenosis to endovascular treatment (n=251) or carotid endarterectomy (n=253). For endovascular patients treated successfully, we used stents in 55 (26%) and balloon angioplasty alone in 158 (74%). An independent neurologist followed up patients. Analysis was by intention to treat. Findings: The rates of major outcome events within 30 days of first treatment did not differ significantly between endovascular treatment and surgery (6·4% vs 5·9%, respectively, for disabling stroke or death; 10·0% vs 9·9% for any stroke lasting more than 7 days, or death). Cranial neuropathy was reported in 22 (8·7%) surgery patients, but not after endovascular treatment (p<0·0001). Major groin or neck haematoma occurred less often after endovascular treatment than after surgery (three [1·2%] vs 17 [6·7%], p<0·0015). At 1 year after treatment, severe (70–99%) ipsilateral carotid stenosis was more usual after endovascular treatment (25 [14%] vs seven [4%], p<0·001). However, no substantial difference in the rate of ipsilateral stroke was noted with survival analysis up to 3 years after randomisation (adjusted hazard ratio=1·04, 95% CI 0·63–1·70, p=0·9). Interpretation: Endovascular treatment had similar major risks and effectiveness at prevention of stroke during 3 years compared with carotid surgery, but with wide CIs. Endovascular treatment had the advantage of avoiding minor complications.BACKGROUND Percutaneous transluminal angioplasty and stenting (endovascular treatment) can be used to treat carotid stenosis, but risks and benefits are uncertain. We therefore compared endovascular treatment with conventional carotid surgery. METHODS In a multicentre clinical trial, we randomly assigned 504 patients with carotid stenosis to endovascular treatment (n=251) or carotid endarterectomy (n=253). For endovascular patients treated successfully, we used stents in 55 (26%) and balloon angioplasty alone in 158 (74%). An independent neurologist followed up patients. Analysis was by intention to treat. FINDINGS The rates of major outcome events within 30 days of first treatment did not differ significantly between endovascular treatment and surgery (6.4% vs 5.9%, respectively, for disabling stroke or death; 10.0% vs 9.9% for any stroke lasting more than 7 days, or death). Cranial neuropathy was reported in 22 (8.7%) surgery patients, but not after endovascular treatment (p<0.0001). Major groin or neck haematoma occurred less often after endovascular treatment than after surgery (three [1.2%] vs 17 [6.7%], p<0.0015). At 1 year after treatment, severe (70-99%) ipsilateral carotid stenosis was more usual after endovascular treatment (25 [14%] vs seven [4%], p<0.001). However, no substantial difference in the rate of ipsilateral stroke was noted with survival analysis up to 3 years after randomisation (adjusted hazard ratio=1.04, 95% CI 0.63-1.70, p=0.9). INTERPRETATION Endovascular treatment had similar major risks and effectiveness at prevention of stroke during 3 years compared with carotid surgery, but with wide CIs. Endovascular treatment had the advantage of avoiding minor complications.

Applying the right statistics: analyses of measurement studies

The study of measurement error, observer variation and agreement between different methods of measurement are frequent topics in the imaging literature. We describe the problems of some applications of correlation and regression methods to these studies, using recent examples from this literature. We use a simulated example to show how these problems and misinterpretations arise. We describe the 95% limits of agreement approach and a similar, appropriate, regression technique. We discuss the difference vs. mean plot, and the pitfalls of plotting difference against one variable only. We stress that these are questions of estimation, not significance tests, and show how confidence intervals can be found for these estimates. Copyright

Agreement Between Methods of Measurement with Multiple Observations Per Individual

Limits of agreement provide a straightforward and intuitive approach to agreement between different methods for measuring the same quantity. When pairs of observations using the two methods are independent, i.e., on different subjects, the calculations are very simple and straightforward. Some authors collect repeated data, either as repeated pairs of measurements on the same subject, whose true value of the measured quantity may be changing, or more than one measurement by one or both methods of an unchanging underlying quantity. In this paper we describe methods for analysing such clustered observations, both when the underlying quantity is assumed to be changing and when it is not.

Measurement error and correlation coefficients.

Measurement error is the variation between measurements of the same quantity on the same individual.1 To quantify measurement error we need repeated measurements on several subjects. We have discussed the within-subject standard deviation as an index of measurement error,1 which we like as it has a simple clinical interpretation. Here we consider the use of correlation coefficients to quantify measurement error. A common design for the investigation of measurement error is to take pairs of measurements on a group of subjects, as in table 1. When we have pairs of observations it is natural to plot one measurement against the other. The resulting scatter diagram (see figure 1 may tempt us to calculate a correlation coefficient between the first and second measurement. There are difficulties in interpreting this correlation coefficient. In general, the correlation between repeated measurements will depend on the variability between subjects. Samples containing subjects who differ greatly will produce larger correlation coefficients than will samples containing similar …

A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement

The intraclass correlation coefficient (rI) has been advocated as a statistic for assessing agreement or consistency between two methods of measurement, in conjunction with a significance test of the difference between means obtained by the two methods. We show that neither technique is appropriate for assessing the interchangeability of measurement methods. We describe an alternative approach based on estimation of the mean and standard deviation of differences between measurements by the two methods.

Preterm delivery: effects of socioeconomic factors, psychological stress, smoking, alcohol, and caffeine

Abstract Objective: To examine the relation between preterm birth and socioeconomic and psychological factors, smoking, and alcohol and caffeine consumption. Design: Prospective study of outcome of pregnancy. Setting: District general hospital in inner London. Participants: 1860 consecutive white women booking for delivery; 1513 women studied after exclusion because of multiple pregnancy and diabetes, refusals, and loss to follow up. Measurements: Gestational age was determined from ultrasound and maternal dates; preterm birth was defined as less than 37 completed weeks. Independent variables included smoking, alcohol and caffeine consumption, and a range of indicators of socioeconomic status and psychological stress. Main results: Unifactorial analyses showed that lower social class, less education, single marital status, low income, trouble with “nerves” and depression, help from professional agencies, and little contact with neighbours were all significantly associated with an increased risk of preterm birth. There were no apparent effects of smoking, alcohol, or caffeine on the length of gestation overall, although there was an association between smoking and delivery before 32 weeks. Cluster analysis indicated three subgroups of women delivering preterm: two predominantly of low social status and a third of older women with higher social status who did not smoke. Mean gestational age was highest in the third group. Conclusions: Adverse social circumstances are associated with preterm birth but smoking is not, apart from an association with very early births. This runs counter to findings for fetal growth (birth weight for gestational age) in this study, where a strong effect of smoking on fetal growth was observed but there was no evidence for any association with psychosocial factors.