Joseph L. Fleiss

Intraclass correlations: Uses in assessing rater reliability.

Reliability coefficients often take the form of intraclass correlation coefficients. In this article, guidelines are given for choosing among six different forms of the intraclass correlation for reliability studies in which n target are rated by k judges. Relevant to the choice of the coefficient are the appropriate statistical model for the reliability and the application to be made of the reliability results. Confidence intervals for each of the forms are reviewed.

The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability

or weighted kappa (Spitzer, Cohen, Fleiss and Endicott, 1967; Cohen, 1968a). Kappa is the proportion of agreement corrected for chance, and scaled to vary from -1 to +1 so that a negative value indicates poorer than chance agreement, zero indicates exactly chance agreement, and a positive value indicates better than chance agreement. A value of unity indicates perfect agreement. The use of kappa implicitly assumes that all disagreements are equally serious. When the investigator can specify the relative seriousness of each kind of disagreement, he may employ weighted kappa, the proportion of weighted agreement corrected for chance. For measuring the reliability of quantitative scales, the product-moment and intraclass correlation coefficients are widely

Frequency domain measures of heart period variability and mortality after myocardial infarction.

BackgroundWe studied 715 patients 2 weeks after myocardial infarction to establish the associations between six frequency domain measures of heart period variability (HPV) and mortality during 4 years of follow-up. Methods and ResultsEach measure of HPV had a significant and at least moderately strong univariate association with all-cause mortality, cardiac death, and arrhythmic death. Power in the lower-frequency bands–ultra low frequency (ULF) and very low frequency (VLF) power–had stronger associations with all three mortality end points than power in the higher-frequency bands-low frequency (LF) and high frequency (HF) power. The 24-hour total power also had a significant and strong association with all three mortality end points. VLF power was the only variable that was more strongly associated with arrhythmic death than with cardiac death or all-cause mortality. In multivariate Cox regression models using a step-up approach to evaluate the independent associations between frequency domain measures of heart period variability and death of all causes, ULF power was selected first (i.e., was the single component with the strongest association). Adding VLF or LF power to the Cox regression model significantly improved the prediction of outcome. With both ULF and VLF power in the Cox regression model, the addition of the other two components, LF and HF power, singly or together, did not significantly improve the prediction of all-cause mortality. We explored the relation between the heart period variability measures and all-cause mortality, cardiac death, and arrhythmic death before and after adjusting for five previously established postinfarction risk predictors: age, New York Heart Association functional class, rales in the coronary care unit, left ventricular ejection fraction, and ventricular arrhythmias detected in a 24-hour Holter ECG recording ConclusionsAfter adjustment for the five risk predictors, the association between mortality and total, ULF, and VLF power remained significant and strong, whereas LF and HF power were only moderately strongly associated with mortality. The tendency for VLF power to be more strongly associated with arrhythmic death than with all-cause or cardiac death was still evident after adjusting for the five covariates. Adding measures of HPV to previously known predictors of risk after myocardial infarction identifies small subgroups with a 2.5-year mortality risk of approximately 50%.

A semi-structured clinical interview for the assessment of diagnosis and mental state in the elderly: the Geriatric Mental State Schedule: I. Development and reliability

A standardized, semi-structured interview for examining and recording the mental state in elderly subjects is described. It allows the classification of patients by symptom profile and can demonstrate changes in that profile over time. It is believed that good reliability is demonstrated between psychiatric raters both for psychiatric diagnosis made on the basis of the schedule findings and for individual items. The Geriatric Mental State Schedule (GMS) consists mainly of items from the eighth edition of the PSE (Wing et al. 1967), together with additional items from the PSS (Spitzer et al. 1964), and extra sections dealing with disorientation and other cognitive abnormalities. Modifications have been introduced to facilitate interviewing elderly subjects.

The ability of several short-term measures of RR variability to predict mortality after myocardial infarction.

BACKGROUND We studied 715 patients 2 weeks after myocardial infarction to test the hypothesis that short-term power spectral measures of RR variability (calculated from 2 to 15 minutes of normal RR interval data) will predict all-cause mortality or arrhythmic death. METHODS AND RESULTS We performed power spectral analyses on the entire 24-hour RR interval time series. To compare with the 24-hour analyses, we selected short segments of ECG recordings from two time periods for analysis: 8 AM to 4 PM and midnight to 5 AM. The former corresponds to the time interval during which short-term measures of RR variability would most likely be obtained. The latter, during sleep, represent a period of increased vagal tone, which may simulate the conditions that exist when patients have a signal-averaged ECG recorded, ie, lying quietly in the laboratory. Four frequency domain measures were calculated from spectral analysis of heart period data over a 24-hour interval. We computed the 24-hour power spectral density and calculated the power within three frequency bands: (1) 0.0033 to < 0.04 Hz, very low frequency power, (2) 0.04 to < 0.15 Hz, low frequency power, and (3) 0.15 to 0.40 Hz, high frequency power. In addition, we calculated the ratio of low to high frequency power. These measures were calculated for 15-, 10-, 5-, and 2-minute segments during the day and at night. Mean power spectral values from short periods during the day and night were similar to 24-hour values, and the correlations between short segment values and 24-hour values were strong (many correlations were > or = 0.75). Using the optimal cutpoints determined previously for the 24-hour power spectral values, we compared the survival experience of patients with low values for RR variability in short segments of ECG recordings to those with high values. We found that power spectral measures of RR variability were excellent predictors of all-cause, cardiac, and arrhythmic mortality and sudden death. Patients with low values were 2 to 4 times as likely to die over an average follow-up of 31 months as were patients with high values. The power spectral measures of RR variability did not predict arrhythmic or sudden deaths substantially better than all-cause mortality. CONCLUSIONS Power spectral measures of RR variability calculated from short (2 to 15 minutes) ECG recordings are remarkably similar to those calculated over 24 hours. The power spectral measures of RR variability are excellent predictors of all-cause mortality and sudden cardiac death.

Measuring Agreement between Two Judges on the Presence or Absence of a Trait

At least a dozen indexes have been proposed for measuring agreement between two judges on a categorical scale. Using the binary (positive-negative) case as a model, this paper presents and critically evaluates some of these proposed measures. The importance of correcting for chance-expected agreement is emphasized, and identities with intraclass correlation coefficients are pointed out.

RR Variability in Healthy, Middle-Aged Persons Compared With Patients With Chronic Coronary Heart Disease or Recent Acute Myocardial Infarction

BACKGROUND The purpose of this investigation was to establish normal values of RR variability for middle-aged persons and compare them with values found in patients early and late after myocardial infarction. We hypothesized that presence or absence of coronary heart disease, age, and sex (in this order of importance) are all correlated with RR variability. METHODS AND RESULTS To determine normal values for RR variability in middle-aged persons, we recruited a sample of 274 healthy persons 40 to 69 years old. To determine the effect of acute myocardial infarction RR variability, we compared measurements of RR variability made 2 weeks after myocardial infarction (n = 684) with measurements made on age- and sex-matched middle-aged subjects with no history of cardiovascular disease (n = 274). To determine the extent of recovery of RR variability after myocardial infarction, we compared measurements of RR variability made in the group of healthy middle-aged persons with measurements made in 278 patients studied 1 year after myocardial infarction. We performed power spectral analyses on continuous 24-hour ECG recordings to quantify total power, ultralow-frequency (ULF) power, very-low-frequency (VLF) power, low-frequency (LF) power, high-frequency (HF) power, and the ratio of LF to HF (LF/HF) power. Time-domain measures also were calculated. All measures of RR variability were significantly and substantially lower in patients with chronic or subacute coronary heart disease than in healthy subjects. The difference from normal values was much greater 2 weeks after myocardial infarction than 1 year after infarction, but the fractional distribution of total power into its four component bands was similar for the three groups. In healthy subjects, ULF power did not change significantly with age; VLF, LF, and HF power decreased significantly as age increased. Patients with chronic coronary heart disease showed little relation between power spectral measures of RR variability and age. Patients with a recent myocardial infarction showed a strong inverse relation between VLF, LF, and HF power and age and a weak inverse relation between ULF power and age. ULF power best separates the healthy group from either of the two coronary heart disease groups. Differences in RR variability between men and women were small and inconsistent among the three groups. CONCLUSIONS All measures of RR variability were significantly and substantially higher in healthy subjects than in patients with chronic or subacute coronary heart disease. The difference between healthy middle-aged persons and those with coronary heart disease was much greater 2 weeks after myocardial infarction than 1 year after infarction, but the fractional distribution of total power into its four component bands was similar for the healthy group and the two coronary heart disease groups. Values of RR variability previously reported to predict death in patients with known chronic coronary heart disease are rarely (approximately 1%) found in healthy middle-aged individuals. Thus, when measures of RR variability are used to screen groups of middle-aged persons to identify individuals who have substantial risk of coronary deaths or arrhythmic events, misclassification of healthy middle-aged persons should be rare.

Correlations among time and frequency domain measures of heart period variability two weeks after acute myocardial infarction

Seven hundred fifteen participants from a multicenter natural history study of acute myocardial infarction were studied (1) to determine the correlations among time and frequency domain measures of heart period variability, (2) to determine the correlations between the measures of heart period variability and previously established post-infarction risk predictors, and (3) to determine the predictive value of time domain measures of heart period variability for death during follow-up after acute myocardial infarction. Twenty-four hour electrocardiographic recordings obtained 11 +/- 3 days after acute myocardial infarction were analyzed and 11 measures of heart period variability were computed. Each of 4 bands in the heart period power spectrum had 1 or 2 corresponding variables in the time domain that correlated with it so strongly (r greater than or equal to 0.90) that the variables were essentially equivalent: ultra low frequency power with SDNN* and SDANN index,* very low frequency power and low-frequency power with SDNN index,* and high-frequency power with r-MSSD* and pNN50.* As expected from theoretical considerations, SDNN and the square root of total power were almost perfectly correlated. Correlations between the time and frequency domain measures of heart period variability and previously identified postinfarction risk predictors, e.g., left ventricular ejection fraction and ventricular arrhythmias, are remarkably weak. Time domain measures of heart period variability, especially those that measure ultra low or low-frequency power, are strongly and independently associated with death during follow-up. * Defined in Table II.

Stability over time of variables measuring heart rate variability in normal subjects

Abstract Both time and frequency domain measures of heart rate (HR) variability have been used to assess autonomic tone in a variety of clinical conditions. Few studies in normal subjects have been performed to determine the stability of HR variability over time, or the correlation between and within time and frequency domain measures of HR variability. Fourteen normal subjects aged 20 to 55 years were studied with baseline and placebo 24-hour ambulatory electrocardiograms performed 3 to 65 days apart to assess the reproducibility of the following time domain measures of cycle length variability: the standard deviation of all normal cycle intervals; mean normal cycle interval; mean day normal cycle interval; night/day difference in mean normal cycle interval; root-mean-square successive cycle interval difference; percentage of differences between adjacent normal cycle length intervals that are >50 ms computed over the entire 24-hour electrocardiographic recording (proportion of adjacent intervals >50 ms); and the frequency domain measures of high (0.15 to 40 Hz), low (0.003 to 0.15) and total (0.003 to 0.40) power. The mean and standard deviations of these measures were virtually identical between placebo and baseline measurements and within the studied time range. Variables strongly dependent on vagal tone (high-frequency, low-frequency and total power, root-mean-square successive difference, and percentage of differences between adjacent normal cycle intervals >50 ms computed over the entire 24-hour electrocardiographic recording) were highly correlated (r > 0.8). It is concluded that measures of HR variability are stable over short periods of time. Certain time and frequency domain variables are highly correlated and may serve as surrogates for each other, and no placebo effect on these variables is evident.

Analysis of data from multiclinic trials

Because the clinics in a multiclinic randomized clinical trial represent neither fixed stratification effects nor random classificatory effects, the appropriate analysis of data from such a trial has been the subject of controversy and debate. The following are some of the elements of controversy that are discussed and for which some bases for resolution are proposed. Is it ever valid to ignore the effects of clinics in the analysis? Is it ever valid to drop clinics from the analysis? Is a multiclinic clinical trial similar in structure or not to a single-clinic clinical trial in which patients have been stratified on a classificatory factor? Assuming that clinics will be taken account of in the analysis, should it be the weighted or the unweighted average of within-clinic treatment differences that is to be taken as the best estimate of the overall difference between the treatments? How should the data be analyzed if there is evidence of treatment-by-clinic interaction?