John C. Caruso

Reliability generalization of the NEO Personality Scales.

A reliability generalization of 51 samples employing one of the NEO personality scales was conducted. Reliability generalization is a meta-analytic method for examining the variability in the reliability of scores by determining which sample characteristics are related to differences in score reliability. It was found that there was a large amount of variability in the reliability of NEO scores, both between and within personality domains. The sample characteristics that are related to score reliability were dependent on NEO domain. Agreeableness scores appear to be the weakest of the domains assessed by the NEO scales in terms of reliability, particularly in clinical samples, for male-only samples, and when temporal consistency was the criterion for reliability. The reliability of Openness to Experience scores was low when the NEO-Five Factor Inventory was used. The advantages of conceptualizing reliability as a property of scores, and not tests, are discussed.

Reliability generalization: Moving toward improved understanding and use of score reliability

Reliability generalization (RG) is a measurement meta-analytic method used to explore the variability in score reliability estimates and to characterize the possible sources of this variance. This article briefly summarizes some RG considerations. Included is a description of how reliability confidence intervals might be portrayed graphically. The article includes tabulations across various RG studies, including how frequently authors (a) report score reliabilities for their own data, (b) conduct reliability induction, or (c) do not even mention reliability.

Reliability of Scores from the Eysenck Personality Questionnaire: A Reliability Generalization Study

A reliability generalization study was conducted on data from 69 samples found in 44 studies that employed the Psychoticism (P), Extraversion (E), Neuroticism (N), and Lie (L) scales of the Eysenck Personality Questionnaire (EPQ) or EPQ-Revised. The reliability of the scores varied considerably between scales, with P scores tending to have the lowest reliability. Hierarchical regression analyses revealed that a larger standard deviation of scores was associated with higher score reliability for all four EPQ scales. More variability in age was associated with higher score reliability for the P scale and the L scale. Samples composed of students provided scores with higher reliability than those composed of other types of individuals for the P scale. Several other potential predictors (form, language of administration, average score, average age, gender composition, and number of items per scale) were not significantly related to score reliability.

An Examination of the Factor Structure of the Alcohol Use Disorders Identification Test in Two High-Risk Samples

The Alcohol Use Disorders Identification Test (AUDIT) was examined by employing confirmatory factor analytic techniques to data from two samples collected 1998–1999: college students (n = 465) and court-referred, substance use treatment outpatients (clinical sample; n = 135). Despite the fact that the AUDIT was originally designed as a three-factor measure (consumption, dependence, and consequences), previous studies have lent support to one- and two-factor models. The results of this study support a two-factor model (alcohol consumption and dependence/consequences) in both samples. As further evidence that the two-factor model is appropriate, a psychometric evaluation suggested that the AUDIT generated reliable scores in both groups when used as either a one- or two-factor measure, but not when three scores are derived in the student sample.

A Reliability Induction and Reliability Generalization Study of the Cage Questionnaire

The CAGE is a commonly used alcohol screening instrument. Although considerable work has been done on the validity of CAGE scores, relatively little information is available on their reliability. Reliability induction and generalization studies were performed for the CAGE. Of the 259 studies available for analysis, only 19 (7.3%) contained reliability information for the sample scores. Thirteen (5.0%) and 227 (87.6%) articles made what are designated as reliability induction by report and reliability induction by omission. The median internal consistency reliability across 22 samples was .74, with a range of .52 to .90. Sample age was the only identified sample characteristic that demonstrated a statistically significant relationship with CAGE score reliability.

Reliability Generalization of the Alcohol Use Disorders Identification Test

The present study evaluated the reliability of scores from the Alcohol Use Disorders Identification Test (AUDIT) in a reliability generalization study. This reliability generalization had two primary goals: (a) to characterize the typical reliability of scores for the AUDIT, and (b) to examine factors that may be related to the reliability of AUDIT scores. The median internal consistency reliability across 24 samples was .81, with a range of .59 to .91. Results suggest that the AUDIT is capable of generating generally reliable scores across some varied sample conditions. After controlling for score variability, no sample characteristic was a statistically significantly predictor of score reliability and effects were small. Only 17 of 104 empirical journal articles contained adequate psychometric information to be included in the present study.

An examination of the reliability of scores from Zuckerman's sensation seeking scales, form V

A reliability generalization (RG) was conducted on Zuckerman’s Sensation Seeking Scale, Form V (SSS-V). Two hundred and forty-four empirical articles on the SSS-V were reviewed spanning a 20-year period but fewer than 9% provided useable reliability estimates. One hundred and thirteen reliability coefficients were collected from 21 published studies for the five scales and subscales of SSS-V. The reliability of scores was marginal for four of the five scales and low for the Boredom Susceptibility scale (median = .61). Mean age of the respondents had a statistically significantly relationship with score reliability, with older respondents’scores having higher reliability for three of the five scales. Unfortunately, the infrequent reporting of basic descriptive information hindered further examination of factors that may have been related to score reliability. Although certain descriptive information (e.g., score reliability estimates, score SDs) may not bethefocus of substantivestudies, theaccumulation of knowledgeover time requires more complete data reporting in all research, even nonmeasurement studies.

Reliability generalization of the Junior Eysenck Personality Questionnaire

Abstract A reliability generalization was conducted on the Psychoticism (P), Extraversion (E), Neuroticism (N) and Lie (L) scales of the Junior Eysenck Personality Questionnaire (J-EPQ). Twenty-three studies provided data on 44 samples of children who had been administered the J-EPQ. Score reliability was found to vary significantly both between and within scales. N and L provided the most reliable scores (with median reliabilities of 0.80 and 0.79 respectively) followed by E (median reliability=0.73) and P (median reliability=0.68). Scale length was the best predictor of score reliability, but sample gender makeup, language of administration, and the amount of variation in the ages of children in each sample were also significant predictors of reliability for various J-EPQ scales. The results highlight the importance of considering reliability to be a property of scores for a particular group, as opposed to a property of a test generally.

Reliability Generalization of the Revised Children's Manifest Anxiety Scale

Areliability generalization of the Revised Childrens Manifest Anxiety Scale (RCMAS) was conducted using the normative sample. The RCMAS consists of a Total Anxiety scale as well as four subscales. Results suggest that the Total Anxiety scores are typically reliable (median • across 48 samples = .81). Subscale scores were less reliable: The median • coefficients were .61 for the Physiological subscale, .63 for the Concentration subscale, .77 for theWorry& Oversensitivity subscale, and .72 for the Lie subscale. Hierarchical regression analyses demonstrated that score variability, age, race, and mean score were statistically significant predictors of reliability across scales although effects were variable and often, but not always, small. The predictive power of score variability is consistent with psychometric theory, but focused psychometric analyses are needed to isolate the unique and interactive impact of age, race, and mean score on RCMAS scale score reliability estimates, especially the Lie subscale.

The Factor Structure of the WAIS-R: Replicability Across Age-Groups

This article examines the factor structure of the Wechsler Adult Intelligence Scale- Revised across nine age groups using several methods of factor analysis, including reliable component analysis (RCA). RCA defines orthogonal components which have maximum reliability, and has several desirable properties which are discussed. Although the one- factor model (General Intelligence, or g), and the two-factor model (Verbal and Performance) of the WAIS-R are fairly well established. no such consensus has been reached regarding the three-factor model (Verbal Comprehension. Perceptual Organization, and Freedom from Distractibility). In the present study, g and Verbal and Performance factors were consistent across age groups for most methods of extraction, although somewhat different from the usual division. The three-factor model, however, was not consistently identified across age groups by any method, particularly with respect to Freedom from Distractibility. Meaningful interpretation of scores on this factor is therefore tenuous. RCA performed well, relative to most other methods, in identifying factors consistently across age groups and can provide useful and unique information.