Domenic V. Cicchetti

Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.

In the context of the development of prototypic assessment instruments in the areas of cognition, personality, and adaptive functioning, the issues of standardization, norming procedures, and the important psychometrics of test reliability and validity are evaluated critically. Criteria, guidelines, and simple rules of thumb are provided to assist the clinician faced with the challenge of choosing an appropriate test instrument for a given psychological assessment. Clinicians are often faced with the critical challenge of choosing the most appropriate available test instrument for a given psychological assessment of a child, adolescent, or adult of a particular age, gender, and class of disability. It is the purpose of this report to provide some criteria, guidelines, or simple rules of thumb to aid in this complex scientific decision. As such, it draws upon my experience with issues of test development, standardization, norming procedures, and important psychometrics, namely, test reliability and validity. As I and my colleagues noted in an earlier publication, the major areas of psychological functioning, in the normal development of infants, children, adolescents, adults, and elderly people, include cognitive, academic, personality, and adaptive behaviors (Sparrow, Fletcher, & Cicchetti, 1985). As such, the major examples or applications discussed in this article derive primarily, although not exclusively, from these several areas of human functioning.

High agreement but low kappa: I. The problems of two paradoxes.

In a fourfold table showing binary agreement of two observers, the observed proportion of agreement, p0, can be paradoxically altered by the chance-corrected ratio that creates kappa as an index of concordance. In one paradox, a high value of p0 can be drastically lowered by a substantial imbalance in the tables marginal totals either vertically or horizontally. In the second pardox, kappa will be higher with an asymmetrical rather than symmetrical imbalanced in marginal totals, and with imperfect rather than perfect symmetry in the imbalance. An adjustment that substitutes kappa max for kappa does not repair either problem, and seems to make the second one worse.

Children's Yale-Brown Obsessive Compulsive Scale: Reliability and Validity

OBJECTIVE To evaluate the reliability and validity of a semistructured measure of obsessive-compulsive symptom severity in children and adolescents with obsessive-compulsive disorder (OCD). METHOD Sixty-five children with OCD (25 girls and 40 boys, aged 8 to 17 years) were assessed with the Childrens Yale-Brown Obsessive Compulsive Scale (CY-BOCS). Interrater agreement was assessed by four raters in a subsample (n = 24). Discriminant and convergent validity were assessed by comparing CY-BOCS scores to self-ratings of depression, anxiety, and obsessive-compulsive symptoms. RESULTS Internal consistency was high, measuring .87 for the 10 items. The intraclass correlations for the CY-BOCS Total, Obsession, and Compulsion scores were .84, .91, and .68, suggesting good to excellent interrater agreement for subscale and total scores. The CY-BOCS Total score showed a significantly higher correlation with a self-report of obsessive-compulsive symptoms (r = .62 for the Leyton survey) compared with the Childrens Depression Inventory (r = .34) and the Childrens Manifest Anxiety Scale (r = .37) (p = .02 and .05, respectively). CONCLUSIONS The CY-BOCS yields reliable and valid subscale and total scores for obsessive-compulsive symptom severity in children and adolescents with OCD. Reliability and validity appear to be influenced by age of the child and the hazards associated with integrating data from parental and patient sources.

A normed study of face recognition in autism and related disorders.

Although the interpretation of studies of face recognition in older children, adolescents, and adults with autism is complicated by the fact that participating samples and adopted methodologies vary significantly, there is nevertheless strong evidence indicating processing peculiarities even when task performance is not deficient. Much less is known about face recognition abilities in younger children with autism. This study employed a well-normed task of face recognition to measure this ability in 102 young children with autism, pervasive developmental disorder not otherwise specified (PDDNOS), and non-PDD disorders (mental retardation and language disorders) matched on chronological age and nonverbal mental age, and in a subsample of 51 children divided equally in the same three groups matched on chronological age and verbal mental age. There were pronounced deficits of face recognition in the autistic group relative to the other nonverbally matched and verbally matched groups. Performance on two comparison tasks did not reveal significant differences when verbal ability was adequately controlled. We concluded that young children with autism have face recognition deficits that cannot be attributed to overall cognitive abilities or task demands. In contrast to controls, there was a lower correlation between performance on face recognition and nonverbal intelligence, suggesting that in autism face recognition is less correlated with general cognitive capacity. Contrary to our expectation, children with PDDNOS did not show face recognition deficits.

The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation

The reliability of peer review of scientific documents and the evaluative criteria scientists use to judge the work of their peers are critically reexamined with special attention to the consistently low levels of reliability that have been reported. Referees of grant proposals agree much more about what is unworthy of support than about what does have scientific value. In the case of manuscript submissions this seems to depend on whether a discipline (or subfield) is general and diffuse (e.g., cross-disciplinary physics, general fields of medicine, cultural anthropology, social psychology) or specific and focused (e.g., nuclear physics, medical specialty areas, physical anthropology, and behavioral neuroscience). In the former there is also much more agreement on rejection than acceptance, but in the latter both the wide differential in manuscript rejection rates and the high correlation between referee recommendations and editorial decisions suggests that reviewers and editors agree more on acceptance than on rejection. Several suggestions are made for improving the reliability and quality of peer review. Further research is needed, especially in the physical sciences.

Social deficits in autism: an operational approach using the Vineland Adaptive Behavior Scales.

Abstract Although social factors are commonly viewed as an important defining characteristics of the autistic syndrome, earlier research studies have not systematically evaluated social dysfunction in autistic individuals. In this study, the Vineland Adaptive Behavior Scales were used to assess adaptive behavior in a group of autistic and a group of nonautistic, developmentally disabled, individuals. As compared to nonautistic subjects, the autistic group exhibited significantly greater deficits in adaptive social behaviors. The result suggests the utility of a well-standardized, normative assessment instrument for documenting autistic social dysfunction.

The positive and negative syndrome scale and the brief psychiatric rating scale : reliability, comparability, and predictive validity

In a psychiatric rehabilitation study, 154 concurrent ratings were performed using the 30-item Positive and Negative Syndrome Scale (PANSS) and the 18-item Brief Psychiatric Rating Scale (BPRS). Although both instruments had excellent interrater reliability, the PANSS was consistently better: on the 18 symptom items the two instruments share, the PANSS had higher intraclass rs on 14; for the syndromes, the PANSS was higher than the BPRS on positive, negative, and total. Weighted Kappas comparing shared items revealed that most were not interchangeable, with only three coefficients in the excellent range. However, syndrome scale scores were very highly correlated and resulted in similar classification for negative schizophrenia. Ten of the 12 items of the PANSS not included in the BPRS had low zero-order correlations with BPRS items, which suggests that they measure symptoms distinct from those measured by the BPRS and should add to clinical predictive power. This proved true in our study of rehabilitation of patients with schizophrenia PANSS symptom ratings explained up to 55% of the variance on seven measures of work performance, whereas the BPRS had lower predictive power on six of the seven measures. We concluded that the PANSS may be superior to the BPRS in clinical research on schizophrenia and that most BPRS items are not interchangeable with identically named PANSS items.

The effect of number of rating scale categories on levels of interrater reliability: A monte carlo investigation

A computer simulation study was designed to in vestigate the extent to which the interrater reliability of a clinical scale is affected by the number of cate gories or scale points (2, 3, 4, ... ,100). Results in dicate that reliability increases steadily up to 7 scale points, beyond which no substantial increases occur, even when the number of scale poirits is increased to as many as 100. These findings hold under the follow ing conditions: (1) The research investigator has insuf ficient a priori knowledge to use as a reliable guide line for deciding on an appropriate number of scale points to employ, and (2) the dichotomous and ordinal categories being considered all have an underlying metric or continuous scale format.

An Evaluation of the Autism Behavior Checklist.

The Autism Behavior Checklist (ABC), an assessment instrument for autistic individuals, was evaluated in a group of 157 subjects, 94 clinically autistic and 63 nonautistic. The two groups differed significantly in ratings of pathology. Both false positive and false negative diagnostic classifications were made when the results of the checklist were compared with clinical diagnosis. Effects of developmental level and age were observed. The ABC appears to have merit as a screening instrument, though results of the checklist alone cannot be taken as establishing a diagnosis of autism. Important issues of reliability and validity remain to be addressed.

The Role of Adaptive Behavior in Autism Spectrum Disorders: Implications for Functional Outcome

The relationship between adaptive functioning and autism symptomatology was examined in 1,089 verbal youths with ASD examining results on Vineland-II, IQ, and measures of ASD severity. Strong positive relationships were found between Vineland subscales and IQ. Vineland Composite was negatively associated with age. IQ accounted a significant amount of the variance in overall adaptive skills (55%) beyond age and ASD severity. Individuals with ASD demonstrated significant adaptive deficits and negligible associations were found between the level of autism symptomatology and adaptive behavior. The results indicate that IQ is a strong predictor of adaptive behavior, the gap between IQ and adaptive impairments decreases in lower functioning individuals with ASD, and older individuals have a greater gap between IQ and adaptive skills.