Jonathan W. Bartlett

Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables

Clinical practice involves measuring quantities for a variety of purposes, such as aiding diagnosis, predicting future patient outcomes, and serving as endpoints in studies or randomized trials. Measurements are almost always prone to various sorts of errors, which cause the measured value to differ from the true value; accordingly, studies investigating measurement error frequently appear in this and other journals. The importance of measurement error depends upon the context in which the measurements in question are to be used. For example, a certain degree of measurement error may be acceptable if measurements are to be used as an outcome in a comparative study such as a clinical trial, but the same measurement errors may be unacceptably large to make measurements usable in individual patient management, such as screening or risk prediction. In the past 20 years many papers have been published advocating how studies of measurement error should be analyzed, with a paper by Bland and Altman1 being one of the most cited and well known examples. There has been much controversy concerning the choice of parameter to be estimated and reported, and consequently confusion surrounding the meaning and interpretation of results from studies investigating measurement error. In this paper we first distinguish between the general concepts of agreement and reliability to aid researchers in considering which are relevant for their particular application. We then review the statistical methods that can be used to investigate and quantify agreement and reliability, dealing separately with the different types of measurement error study, while emphasizing the largely common techniques that should be used for data analysis. We reiterate that the judgment of whether agreement or reliability are acceptable must be related to the clinical application, and cannot be proven by a statistical test. We highlight the fact that reliability depends on the population in which measurements are made, and not just on the measurement errors of the measurement method. We discuss the advantages of method comparison studies making at least two measurements with each measurement method on each subject. A key advantage is that the cause of a correlation between paired differences and means in the so-called Bland–Altman plot can be determined, in contrast to when only a single measurement is made with each method. Throughout the paper, we try to emphasize that calculated values of agreement and reliability from measurement error studies are estimates of parameters, and as such we should report such estimates with CIs to indicate the uncertainty with which they have been estimated. We restrict our attention to measurements of a continuous quantity; alternative methods are required for categorical data2.

Tracking atrophy progression in familial Alzheimer's disease: a serial MRI study

BACKGROUND Serial MRI scanning of autosomal dominant mutation carriers for Alzheimers disease provides an opportunity to track changes that could predate symptoms or clinical diagnosis of the disease. We used hierarchical modelling to assess how hippocampal and whole-brain volumes change as familial Alzheimers disease progresses from the presymptomatic stage through to diagnosis. METHODS Nine mutation carriers had serial clinical assessments and volumetric MRI scans (41 scans: range 3-8 per patient) at different clinical stages (presymptomatic, mild cognitive impairment, or clinical Alzheimers disease). 25 healthy controls had serial scanning (54 scans: range 2-4 per patient) for comparison. We measured whole brain and total hippocampal volumes using semi-automated techniques, and adjusted for total intracranial volume. Hierarchical models were developed to estimate differences in volume and atrophy rate between mutation carriers and controls in relation to when the disease was clinically diagnosed. FINDINGS Mutation carriers had significantly increased hippocampal and whole-brain atrophy rates compared with controls and these differences increased with time. Differences in hippocampal and whole-brain atrophy rates between controls and mutation carriers were evident 5.5 and 3.5 years, respectively, before diagnosis of Alzheimers disease. At a cross-sectional level, differences in mean hippocampal volume between mutation carriers and controls became significant 3 years before clinical diagnosis, whereas differences in mean brain volumes became significant only 1 year before diagnosis. INTERPRETATION Structural changes can be seen on MRI scans that predate the clinical onset of familial Alzheimers disease. Longitudinal measures of atrophy rates can identify differences between mutation carriers and controls 2-3 years earlier than cross-sectional volumetric measures.

Head size, age and gender adjustment in MRI studies: a necessary nuisance?

Imaging studies of cerebral volumes often adjust for factors such as age that may confound between-subject comparisons. However the use of nuisance covariates in imaging studies is inconsistent, which can make interpreting results across studies difficult. Using magnetic resonance images of 78 healthy controls we assessed the effects of age, gender, head size and scanner upgrade on region of interest (ROI) volumetry, cortical thickness and voxel-based morphometric (VBM) measures. We found numerous significant associations between these variables and volumetric measures: cerebral volumes and cortical thicknesses decreased with increasing age, men had larger volumes and smaller thicknesses than women, and increasing head size was associated with larger volumes. The relationships between most ROIs and head size volumes were non-linear. With age, gender, head size and upgrade in one model we found that volumes and thicknesses decreased with increasing age, women had larger volumes than men (VBM, whole-brain and white matter volumes), increasing head size was associated with larger volumes but not cortical thickness, and scanner upgrade had an effect on thickness and some volume measures. The effects of gender on cortical thickness when adjusting for head size, age and upgrade showed some non-significant effect (women>men), whereas the independent effect of head size showed little pattern. We conclude that age and head size should be considered in ROI volume studies, age, gender and upgrade should be considered for cortical thickness studies and all variables require consideration for VBM analyses. Division of all volumes by head size is unlikely to be adequate owing to their non-proportional relationship.

Automated cross-sectional and longitudinal hippocampal volume measurement in mild cognitive impairment and Alzheimer's disease

Volume and change in volume of the hippocampus are both important markers of Alzheimers disease (AD). Delineation of the structure on MRI is time-consuming and therefore reliable automated methods are required. We describe an improvement (multiple-atlas propagation and segmentation (MAPS)) to our template library-based segmentation technique. The improved technique uses non-linear registration of the best-matched templates from our manually segmented library to generate multiple segmentations and combines them using the simultaneous truth and performance level estimation (STAPLE) algorithm. Change in volume over 12months (MAPS-HBSI) was measured by applying the boundary shift integral using MAPS regions. Methods were developed and validated against manual measures using subsets from Alzheimers Disease Neuroimaging Initiative (ADNI). The best method was applied to 682 ADNI subjects, at baseline and 12-month follow-up, enabling assessment of volumes and atrophy rates in control, mild cognitive impairment (MCI) and AD groups, and within MCI subgroups classified by subsequent clinical outcome. We compared our measures with those generated by Surgical Navigation Technologies (SNT) available from ADNI. The accuracy of our volumes was one of the highest reported (mean(SD) Jaccard Index 0.80(0.04) (N=30)). Both MAPS baseline volume and MAPS-HBSI atrophy rate distinguished between control, MCI and AD groups. Comparing MCI subgroups (reverters, stable and converters): volumes were lower and rates higher in converters compared with stable and reverter groups (p< or =0.03). MAPS-HBSI required the lowest sample sizes (78 subjects) for a hypothetical trial. In conclusion, the MAPS and MAPS-HBSI methods give accurate and reliable volumes and atrophy rates across the clinical spectrum from healthy aging to AD.

A meta-analysis of hippocampal atrophy rates in Alzheimer's disease

Hippocampal atrophy rates are useful in both diagnosing and tracking Alzheimers disease (AD). However, cohorts and methods used to determine such rates are heterogeneous, leading to differences in reported annualised rates. We performed a meta-analysis of hippocampal atrophy rates in AD patients and matched controls from studies reported in the peer-reviewed literature. Studies reporting longitudinal volume change in hippocampi in AD subjects together with controls were systematically identified and appraised. All authors were contacted either to confirm the results or to provide missing data. Meta-analysis and meta-regression were then performed on this data. Nine studies were included from seven centres, with data from a total of 595 AD and 212 matched controls. Mean (95% CIs) annualised hippocampal atrophy rates were found to be 4.66% (95% CI 3.92, 5.40) for AD subjects and 1.41% (0.52, 2.30) for controls. The difference between AD and control subject in this rate was 3.33% (1.73, 4.94).

A randomised controlled trial of cognitive behaviour therapy and motivational interviewing for people with Type 1 diabetes mellitus with persistent sub-optimal glycaemic control: a Diabetes and Psychological Therapies (ADaPT) study.

OBJECTIVES To determine whether (i) motivational enhancement therapy (MET) + cognitive behaviour therapy (CBT) compared with usual care, (ii) MET compared with usual care, (iii) or MET + CBT compared with MET was more effective in improving glycaemic control when delivered by general nurses with additional training in these techniques. DESIGN A three-arm parallel randomised controlled trial as the gold standard design to test the effectiveness of psychological treatments. SETTING The recruiting centres were diabetes clinics in seven acute trusts in south-east London and Greater Manchester. PARTICIPANTS Adults (18-65 years) with a confirmed diagnosis of type 1 diabetes for a minimum duration of 2 years and a current glycated (or glycosylated) haemoglobin (HbA1c) value between 8.2% and 15.0%. INTERVENTIONS The control arm consisted of usual diabetes care which varied between the hospitals, but constituted at least three monthly appointments to diabetes clinic. The two treatments arms consisted of usual care with MET and usual care with MET + CBT. MAIN OUTCOME MEASURES The primary outcome was HbA1c at 12 months from randomisation. Secondary outcome measures were 1-year costs measured by the Client Service Receipt Inventory at baseline, 6 months and 12 months; quality of life-years [quality-adjusted life-years (QALYs)] measured by the SF-36 (Short Form-36 Health Survey Questionnaire) and EQ-5D (European Quality of Life-5 Dimensions) at baseline and 12 months. RESULTS One thousand six hundred and fifty-nine people with type 1 diabetes were screened and 344 were randomised to MET + CBT (n = 106), MET (n = 117) and to usual care (n = 121). The 12-month follow-up rate for HbA1c was 88% (n = 305). The adjusted mean 12-month HbA1c was 0.45% lower in those treated with MET + CBT [95% confidence interval (CI) 0.16% to 0.79%, p = 0.008] than for usual care; 0.16% lower in those treated with MET (95% CI 0.20% to 0.51%, p = 0.38) than for usual care; and 0.30% lower with MET + CBT than with MET (95% CI -0.07% to 0.66%, p = 0.11). The higher the HbA1c, and the younger the participant at baseline, the greater was the reduction in HbA1c. The interventions had no effect on secondary outcomes such as depression and quality of life. The economic evaluation was inconclusive. Both interventions were associated with increased health care costs than for usual care alone. There was no significant difference in social costs. Cost effectiveness ratios, up to one year, varied considerably according to whether QALY estimates were based on EQ-5D or SF-36 and whether imputed or complete data were used in the analyses. CONCLUSIONS A combination of MET and CBT may be useful for patients with persistent sub-optimal diabetic control. MET alone appears less effective than usual care. Economic evaluation was inconclusive. TRIAL REGISTRATION Current Controlled Trials ISRCTN77044517.

Increased brain atrophy rates in cognitively normal older adults with low cerebrospinal fluid Aβ1-42

To identify cognitively normal individuals at risk of Alzheimer disease (AD) based on cerebrospinal fluid (CSF) Aβ1‐42, and to determine rates of cerebral atrophy.

Robust atrophy rate measurement in Alzheimer's disease using multi-site serial MRI: Tissue-specific intensity normalization and parameter selection

We describe an improved method of measuring brain atrophy rates from serial MRI for multi-site imaging studies of Alzheimers disease (AD). The method (referred to as KN-BSI) improves an existing brain atrophy measurement technique-the boundary shift integral (classic-BSI), by performing tissue-specific intensity normalization and parameter selection. We applied KN-BSI to measure brain atrophy rates of 200 normal and 141 AD subjects using baseline and 1-year MRI scans downloaded from the Alzheimers Disease Neuroimaging Initiative database. Baseline and repeat images were reviewed as pairs by expert raters and given quality scores. Including all image pairs, regardless of quality score, mean KN-BSI atrophy rates were 0.09% higher (95% CI 0.03% to 0.16%, p=0.007) than classic-BSI rates in controls and 0.07% higher (-0.01% to 0.16%, p=0.07) higher in ADs. The SD of the KN-BSI rates was 22% lower (15% to 29%, p<0.001) in controls and 13% lower (6% to 20%, p=0.001) in ADs, compared to classic-BSI. Using these results, the estimated sample size (needed per treatment arm) for a hypothetical trial of a treatment for AD (80% power, 5% significance to detect a 25% reduction in atrophy rate) would be reduced from 120 to 81 (a 32% reduction, 95% CI=18% to 45%, p<0.001) when using KN-BSI instead of classic-BSI. We concluded that KN-BSI offers more robust brain atrophy measurement than classic-BSI and substantially reduces sample sizes needed in clinical trials.

Brain MAPS: An automated, accurate and robust brain extraction technique using a template library.

Whole brain extraction is an important pre-processing step in neuroimage analysis. Manual or semi-automated brain delineations are labour-intensive and thus not desirable in large studies, meaning that automated techniques are preferable. The accuracy and robustness of automated methods are crucial because human expertise may be required to correct any suboptimal results, which can be very time consuming. We compared the accuracy of four automated brain extraction methods: Brain Extraction Tool (BET), Brain Surface Extractor (BSE), Hybrid Watershed Algorithm (HWA) and a Multi-Atlas Propagation and Segmentation (MAPS) technique we have previously developed for hippocampal segmentation. The four methods were applied to extract whole brains from 682 1.5T and 157 3T T(1)-weighted MR baseline images from the Alzheimers Disease Neuroimaging Initiative database. Semi-automated brain segmentations with manual editing and checking were used as the gold-standard to compare with the results. The median Jaccard index of MAPS was higher than HWA, BET and BSE in 1.5T and 3T scans (p<0.05, all tests), and the 1st to 99th centile range of the Jaccard index of MAPS was smaller than HWA, BET and BSE in 1.5T and 3T scans ( p<0.05, all tests). HWA and MAPS were found to be best at including all brain tissues (median false negative rate ≤0.010% for 1.5T scans and ≤0.019% for 3T scans, both methods). The median Jaccard index of MAPS were similar in both 1.5T and 3T scans, whereas those of BET, BSE and HWA were higher in 1.5T scans than 3T scans (p<0.05, all tests). We found that the diagnostic group had a small effect on the median Jaccard index of all four methods. In conclusion, MAPS had relatively high accuracy and low variability compared to HWA, BET and BSE in MR scans with and without atrophy.

The structural neuroanatomy of music emotion recognition: Evidence from frontotemporal lobar degeneration

Despite growing clinical and neurobiological interest in the brain mechanisms that process emotion in music, these mechanisms remain incompletely understood. Patients with frontotemporal lobar degeneration (FTLD) frequently exhibit clinical syndromes that illustrate the effects of breakdown in emotional and social functioning. Here we investigated the neuroanatomical substrate for recognition of musical emotion in a cohort of 26 patients with FTLD (16 with behavioural variant frontotemporal dementia, bvFTD, 10 with semantic dementia, SemD) using voxel-based morphometry. On neuropsychological evaluation, patients with FTLD showed deficient recognition of canonical emotions (happiness, sadness, anger and fear) from music as well as faces and voices compared with healthy control subjects. Impaired recognition of emotions from music was specifically associated with grey matter loss in a distributed cerebral network including insula, orbitofrontal cortex, anterior cingulate and medial prefrontal cortex, anterior temporal and more posterior temporal and parietal cortices, amygdala and the subcortical mesolimbic system. This network constitutes an essential brain substrate for recognition of musical emotion that overlaps with brain regions previously implicated in coding emotional value, behavioural context, conceptual knowledge and theory of mind. Musical emotion recognition may probe the interface of these processes, delineating a profile of brain damage that is essential for the abstraction of complex social emotions.