John J. Bartko

The intraclass correlation coefficient as a measure of reliability.

A procedure for estimating the reliability of sets of ratings in terms of the intraclass correlation coefficient is discussed. The procedure is based upon the analysis of variance and the estimation of variance components. For the one-way classification the intraclass correlation coefficient defined as the ratio of variances can be interpreted as a correlation coefficient. Caution, however, is urged in the application of the definition to a two-way model, i.e., one in which between-rater variance is removed. It is maintained that the frequent use of the standard definition of the one-way intraclass correlation coefficient applied to the two-way classification is not a proper procedure if in fact the coefficient is to be interpreted as a correlation coefficient. Definitions for reliability obtained from the two-way models are given which can legitimately be considered correlation coefficients.

On the methods and theory of reliability.

This paper reviews the most frequently used and misused reliability measures appearing in the mental health literature. We illustrate the various types of data sets on which reliability is assessed (i.e., two raters, more than two raters, and varying numbers of raters with dichotomous, polychotomous, and quantitative data). Reliability statistics appropriate for each data format are presented, and their pros and cons illustrated. Inadequacies of some methods are highlighted. The meaning of different levels of reliability obtained with various statistics is discussed. This critique is intended for the reading professional and the investigator who has an occasional need of reliability assessment. Statistical expertise is not required and theoretical material is referenced for the interested reader. Necessary formulas for computations are presented in the appendices. A summary table of some suitable reliability measures is presented.

Flexible System for the Diagnosis of Schizophrenia: Report from the WHO International Pilot Study of Schizophrenia

Behavioral data on a large patient group were collected by investigators from nine countries in the International Pilot Study of Schizophrenia, sponsored by the World Health Organization. The data on half the group were analyzed to derive a system of 12 signs and symptoms for the identification of schizophrenia, as this disorder is diagnosed in many centers throughout the world. The findings were replicated with the other half of the patient group. The criteria constitute an operational method for identifying patients who would be commonly considered schizophrenic in many centers.

Toxoplasma gondii and Other Risk Factors for Schizophrenia: An Update

The failure to find genes of major effect in schizophrenia has refocused attention on nongenetic, including infectious factors. In a previous study, antibodies to Toxoplasma gondii were found to be elevated in 23 studies of schizophrenia (OR 2.73; 95% CI 2.10-3.60). The current study replicates this finding with 15 additional studies (OR 2.71; 95% CI 1.93-3.80) and compares this with other identified schizophrenia risk factors. The highest risk factors are having an affected mother (relative risks [RR] 9.31; 95% CI 7.24-11.96), father (RR 7.20; 95% CI 5.10-10.16), or sibling (RR 6.99; 95% CI 5.38-9.08) or being the offspring of immigrants from selected countries (RR 4.5; 95% CI 1.5-13.1). Intermediate risk factors, in addition to infection with T. gondii, include being an immigrant from and to selected countries (RR 2.7; 95% CI 2.3-3.2), being born in (RR 2.24; 95% CI 1.92-2.61) or raised in (RR 2.75; 95% CI 2.31-3.28) an urban area, cannabis use (OR 2.10-2.93; 95% CI 1.08-6.13), having minor physical anomalies (OR 2.23; 95% CI 1.42-3.58), or having a father 55 or older (OR 2.21-5.92; 95% CI 1.46-17.02). Low-risk factors include a history of traumatic brain injury (OR 1.65; 95% CI 1.17-2.32), sex abuse in childhood (OR 1.46; 95% CI 0.84-2.52), obstetrical complications (OR 1.29-1.38; 95% CI 1.00-1.84), having a father 45 or older (OR 1.21-1.66; 95% CI 1.09-2.01), specific genetic polymorphisms (OR 1.09-1.24; 95% CI 1.06-1.45), birth seasonality (OR 1.07-1.95; 95% CI 1.05-2.91), maternal exposure to influenza (RR 1.05; 95% CI 0.98-1.12), or prenatal stress (RR 0.98-1.00; 95% CI 0.85-1.16).

Education and change in cognitive function. The Epidemiologic Catchment Area Study.

The association between educational attainment and decline in cognitive function over an interval of 1 year was examined for 14,883 subjects 18 years and older in the National Institute of Mental Health Epidemiologic Catchment Area Study. Cognitive function was assessed at both time points by the Mini-Mental State Examination (MMSE); cognitive decline was coded as a dichotomous variable and was defined as 1 if the subjects score had declined 3 or more points from the baseline MMSE score at the 1-year follow-up interview and as 0 otherwise. The association between educational attainment and decline in cognitive function over 1 year was examined in logistic regression models that were stratified by age group (< 65 years, > or = 65 years) and by baseline MMSE level (MMSE > 23, MMSE < or = 23). Covariates included age, baseline MMSE score, ethnicity, residence, lifetime diagnosis of abuse of alcohol or other drugs, and gender. In those with baseline MMSE > 23, education was a significant predictor of cognitive decline, not only in the elderly but also in younger subjects. Among those with baseline MMSE < or = 23, education was not a significant predictor of cognitive decline. The fact that education provides protection against cognitive decline even in those younger than 65 years, in whom the prevalence and incidence of dementia are very low, would seem to indicate that education or its correlates provides protection against processes other than dementia that might produce a decline in test performance in young persons.

Cerebrospinal fluid β-amyloid1–42 and tau in control subjects at risk for Alzheimer’s disease: The effect of APOE ε4 allele

Background Cerebrospinal fluid (CSF) measures of β-amyloid 1–42 and tau are linked with the known neuropathology of Alzheimers disease (AD). Numerous lines of evidence have also suggested that individuals with at least one APOE e4 allele on chromosome 19 are at increased risk of developing AD. We tested these CSF markers in groups of subjects with AD and healthy older control subjects, using the absence or presence of the APOE e4 allele as a predictive variable in the search for possible prognostic biomarkers of AD. Methods We assessed the levels of β-amyloid 1–42 and total tau in the CSF of 292 subjects (142 control subjects and 150 subjects with mild-to-moderate AD), who were research participants at the National Institute of Mental Health. The group of control subjects was enriched with a high percentage of subjects with a positive family history of AD. All subjects underwent extensive global cognitive testing. Results When divided according to the absence or presence of the APOE e4 allele, the control subjects with at least one e4 allele had significantly lower CSF β-amyloid 1–42 but not tau levels than control subjects without an APOE e4 allele ( p 1–42 and higher CSF tau levels than the normal control group ( p Conclusions The association of APOE e4 allele and lower, more AD-like levels of CSF β-amyloid 1–42 in older control subjects is consistent with previous studies showing possible neuroimaging and cognitive abnormalities with e4 carriers and suggests that CSF β-amyloid 1–42 decreases might represent an early biomarker of AD. Longitudinal follow-up is of course required to verify whether this biomarker is indeed predictive of clinical conversion to AD.

Cortical correlates of neuromotor development in healthy children

OBJECTIVE To examine the relationship between acquisition of fine motor skills in childhood and development of the motor cortex. METHODS We measured finger tapping speed and mirror movements in 43 healthy right-handed subjects (6-26 years of age). While recording surface electromyographic activity from right and left first dorsal interosseus, we delivered focal transcranial magnetic stimulation (TMS) over the hand areas of each motor cortex. We measured motor evoked potential (MEP) threshold, and ipsilateral (iSP) and contralateral (CSP) silent periods. RESULTS As children got older, finger speeds got faster, MEP threshold decreased, iSP duration increased and latency decreased. Finger tapping speed got faster as motor thresholds and iSP latency decreased, but was unrelated to CSP duration. In all subjects right hemisphere MEP thresholds were higher than those on the left and duration of right hemisphere CSP was longer than that on the left. Children under 10 years of age had higher left hand mirror movement scores, and fewer left hemisphere iSPs which were of longer duration. CONCLUSIONS Maturation of finger tapping skills is closely related to developmental changes in the motor threshold and iSP latency. Studies are warranted to explore the relationship between these measures and other neuromotor skills in children with motor disorders. SIGNIFICANCE TMS can provide important insights into certain functional aspects of neurodevelopment in children.

Penny-wise and pound-foolish: the impact of measurement error on sample size requirements in clinical trials.

BACKGROUND Clinical research studies must compensate for measurement error by increasing the number of subjects that are studied, thereby increasing the financial costs of research and exposing greater numbers of subjects to study risks. In this article, we model the relationship between reliability and sample-size requirements and consider the potential tangible cost savings resulting from the decreased number of subjects needed when reliability of raters is improved or multiple ratings are used. METHODS Standard methods are used to model reliability based on the intraclass correlation coefficient (R) and to perform power calculations. The impact of multiple raters on reliability for a given baseline level of reliability is modeled according to the Spearman Brown formula. RESULTS Our models demonstrate that meaningful reductions in sample size requirements are gained from improvements in reliability. For example, improving reliability from R = .7 to R = .9 will decreases sample size requirements by 22%. Reliability is improved by training and by the use of the mean of multiple ratings. For example, if the reliability of a single rating is 0.7, the reliability of the mean of two ratings will be 0.8. CONCLUSIONS The costs to improve reliability either through rater training efforts or use of the mean of multiple ratings is cost effective because of the consequent reduction in number of subjects needed. Efforts to improve reliability and thus reduce subject requirements in a study also may lead to fewer patients bearing the burden of research participation and to a shortening of the duration of studies.

Automatic and effortful processing in attention deficit/hyperactivity disorder.

Twenty-five boys with Attention Deficit/Hyperactivity Disorder and 23 age-matched controls were compared on verbal memory tasks differentiating automatic versus effortful information processing. Automatic processing tasks included the recognition of new or old words in a list and the recognition of frequency of occurrence of words in a list. Effortful tasks included free recall of lists of both related and unrelated words. Hyperactive boys did not differ from controls in automatic processing capabilities but demonstrated significantly poorer effortful processing. Intercorrelations of the variables revealed high correlations between scores on effortful measures and also raise questions about the purity of automaticity in some tasks employed. Stepwise discriminant analysis demonstrated that free recall of related words (an effortful task) best discriminated between groups. Effort-related processing in hyperactive and normal children is discussed in relation to variables of motivation, affect, arousal, and other higher-order cognitive processes.

Subjective Reactions of Children to Single-Pulse Transcranial Magnetic Stimulation

Single-pulse transcranial magnetic stimulation is a useful tool to investigate cortical function in childhood neuropsychiatric disorders. Magnetic stimulation is associated with a shock-like sensation that is considered painless in adults. Little is known about how children perceive the procedure. We used a self-report questionnaire to assess childrens subjective experience with transcranial magnetic stimulation. Normal children and children with attention-deficit hyperactivity disorder (ADHD) underwent transcranial magnetic stimulation in a study of cortical function in ADHD. Subjects were asked to rate transcranial magnetic stimulation on a 1 to 10 scale (most disagreeable = 1, most enjoyable = 10) and to rank it among common childhood events. Thirty-eight subjects completed transcranial magnetic stimulation; 34 said that they would repeat it. The overall rating for transcranial magnetic stimulation was 6.13, and transcranial magnetic stimulation was ranked fourth highest among the common childhood events. These results suggest that although a few children find transcranial magnetic stimulation uncomfortable, most consider transcranial magnetic stimulation painless. Further studies are necessary to confirm these findings. (J Child Neurol 2001;16:891-894).