Alan L. Shields

Equivalence of Electronic and Paper-and-Pencil Administration of Patient-Reported Outcome Measures: A Meta-Analytic Review

OBJECTIVES Patient-reported outcomes (PROs; self-report assessments) are increasingly important in evaluating medical care and treatment efficacy. Electronic administration of PROs via computer is becoming widespread. This article reviews the literature addressing whether computer-administered tests are equivalent to their paper-and-pencil forms. METHODS Meta-analysis was used to synthesize 65 studies that directly assessed the equivalence of computer versus paper versions of PROs used in clinical trials. A total of 46 unique studies, evaluating 278 scales, provided sufficient detail to allow quantitative analysis. RESULTS Among 233 direct comparisons, the average mean difference between modes averaged 0.2% of the scale range (e.g., 0.02 points on a 10-point scale), and 93% were within +/-5% of the scale range. Among 207 correlation coefficients between paper and computer instruments (typically intraclass correlation coefficients), the average weighted correlation was 0.90; 94% of correlations were at least 0.75. Because the cross-mode correlation (paper vs. computer) is also a test-retest correlation, with potential variation because of retest, we compared it to the within-mode (paper vs. paper) test-retest correlation. In four comparisons that evaluated both, the average cross-mode paper-to-computer correlation was almost identical to the within-mode correlation for readministration of a paper measure (0.88 vs. 0.91). CONCLUSIONS Extensive evidence indicates that paper- and computer-administered PROs are equivalent.

A Reliability Induction and Reliability Generalization Study of the Cage Questionnaire

The CAGE is a commonly used alcohol screening instrument. Although considerable work has been done on the validity of CAGE scores, relatively little information is available on their reliability. Reliability induction and generalization studies were performed for the CAGE. Of the 259 studies available for analysis, only 19 (7.3%) contained reliability information for the sample scores. Thirteen (5.0%) and 227 (87.6%) articles made what are designated as reliability induction by report and reliability induction by omission. The median internal consistency reliability across 22 samples was .74, with a range of .52 to .90. Sample age was the only identified sample characteristic that demonstrated a statistically significant relationship with CAGE score reliability.

Reliability Generalization of the Alcohol Use Disorders Identification Test

The present study evaluated the reliability of scores from the Alcohol Use Disorders Identification Test (AUDIT) in a reliability generalization study. This reliability generalization had two primary goals: (a) to characterize the typical reliability of scores for the AUDIT, and (b) to examine factors that may be related to the reliability of AUDIT scores. The median internal consistency reliability across 24 samples was .81, with a range of .59 to .91. Results suggest that the AUDIT is capable of generating generally reliable scores across some varied sample conditions. After controlling for score variability, no sample characteristic was a statistically significantly predictor of score reliability and effects were small. Only 17 of 104 empirical journal articles contained adequate psychometric information to be included in the present study.

The Michigan Alcoholism Screening Test and Its Shortened Form: A Meta-Analytic Inquiry Into Score Reliability

Meta-analytic methods provide a framework around which an inquiry into MAST and SMAST score reliability was completed. Of the 470 measurement opportunities observed between 1971 and 2005, 62 (13.2%) were coupled with accurate reliability information. Weighted reliability estimates centered on. 80 suggesting that the MAST and SMAST generally produce scores of similar and adequate reliability for most research purposes. However, the variability of internal consistency estimates shows that at times these tools will not produce reliable scores, particularly among female and nonclinical respondents. Multiple regression equations provide practical guidelines to improve reliability estimates for the future use of these instruments.

Score Reliability of Adolescent Alcohol Screening Measures: A Meta-Analytic Inquiry

ABSTRACT This study describes the reliability reporting practices in empirical studies using eight adolescent alcohol screening tools and characterizes and explores variability in internal consistency estimates across samples. Of 119 observed administrations of these instruments, 40 (34%) reported usable reliability information. The Personal Experience Screening Questionnaire—Problem Severity scale generated average reliability estimates exceeding 0.90 (95% CI = 0.90–0.96) and the Adolescent Alcohol Involvement Scale generated average score reliability estimates below 0.80 (95% CI = 0.67–0.85). Average reliability estimates of the remaining instruments were distributed between these extremes. Sample characteristics were identified as potentially important predictors of variability in the reliability estimates of all the instruments and all instruments under evaluation generated more reliable scores in clinical settings (M = 0.89) as opposed to nonclinical settings (M = 0.82; r effect size (38) = 0.29, p < .10). Clinicians facing instrument selection decisions can use these data to guide their choices and researchers evaluating the performance of these instruments can use these data to inform their future studies.

Reliability Generalization of the Revised Children's Manifest Anxiety Scale

Areliability generalization of the Revised Childrens Manifest Anxiety Scale (RCMAS) was conducted using the normative sample. The RCMAS consists of a Total Anxiety scale as well as four subscales. Results suggest that the Total Anxiety scores are typically reliable (median • across 48 samples = .81). Subscale scores were less reliable: The median • coefficients were .61 for the Physiological subscale, .63 for the Concentration subscale, .77 for theWorry& Oversensitivity subscale, and .72 for the Lie subscale. Hierarchical regression analyses demonstrated that score variability, age, race, and mean score were statistically significant predictors of reliability across scales although effects were variable and often, but not always, small. The predictive power of score variability is consistent with psychometric theory, but focused psychometric analyses are needed to isolate the unique and interactive impact of age, race, and mean score on RCMAS scale score reliability estimates, especially the Lie subscale.

Integration of alcohol use disorders identification and management in the tuberculosis programme in Tomsk Oblast, Russia

Alcohol use disorders (AUDs) among tuberculosis (TB) patients are associated with nonadherence and poor treatment outcomes. We developed a multidisciplinary model to manage AUDs among TB patients in Tomsk, Russia. First, we assessed current standards of care through stakeholder meetings and ethnographic work. The Alcohol Use Disorders Identification Test (AUDIT) was incorporated into routine assessment of all patients starting TB treatment. We established treatment algorithms based on AUDIT scores. We then hired specialists and addressed licensing requirements to provide on-site addictions care. Our experience offers a successful model in the management of co-occurring AUDs among patients with chronic medical problems.

Integrated Management of Physician-delivered Alcohol Care for Tuberculosis Patients: Design and Implementation.

BACKGROUND While the integration of alcohol screening, treatment, and referral in primary care and other medical settings in the U.S. and worldwide has been recognized as a key health care priority, it is not routinely done. In spite of the high co-occurrence and excess mortality associated with alcohol use disorders (AUDs) among individuals with tuberculosis (TB), there are no studies evaluating effectiveness of integrating alcohol care into routine treatment for this disorder. METHODS We designed and implemented a randomized controlled trial (RCT) to determine the effectiveness of integrating pharmacotherapy and behavioral treatments for AUDs into routine medical care for TB in the Tomsk Oblast Tuberculosis Service (TOTBS) in Tomsk, Russia. Eligible patients are diagnosed with alcohol abuse or dependence, are newly diagnosed with TB, and initiating treatment in the TOTBS with Directly Observed Therapy-Short Course (DOTS) for TB. Utilizing a factorial design, the Integrated Management of Physician-delivered Alcohol Care for Tuberculosis Patients (IMPACT) study randomizes eligible patients who sign informed consent into 1 of 4 study arms: (1) Oral Naltrexone + Brief Behavioral Compliance Enhancement Therapy (BBCET) + treatment as usual (TAU), (2) Brief Counseling Intervention (BCI) + TAU, (3) Naltrexone + BBCET + BCI + TAU, or (4) TAU alone. RESULTS Utilizing an iterative, collaborative approach, a multi-disciplinary U.S. and Russian team has implemented a model of alcohol management that is culturally appropriate to the patient and TB physician community in Russia. Implementation to date has achieved the integration of routine alcohol screening into TB care in Tomsk; an ethnographic assessment of knowledge, attitudes, and practices of AUD management among TB physicians in Tomsk; translation and cultural adaptation of the BCI to Russia and the TB setting; and training and certification of TB physicians to deliver oral naltrexone and brief counseling interventions for alcohol abuse and dependence as part of routine TB care. The study is successfully enrolling eligible subjects in the RCT to evaluate the relationship of integrating effective pharmacotherapy and brief behavioral intervention on TB and alcohol outcomes, as well as reduction in HIV risk behaviors. CONCLUSIONS The IMPACT study utilizes an innovative approach to adapt 2 effective therapies for treatment of alcohol use disorders to the TB clinical services setting in the Tomsk Oblast, Siberia, Russia, and to train TB physicians to deliver state of the art alcohol pharmacotherapy and behavioral treatments as an integrated part of routine TB care. The proposed treatment strategy could be applied elsewhere in Russia and in other settings where TB control is jeopardized by AUDs. If demonstrated to be effective, this model of integrating alcohol interventions into routine TB care has the potential for expanded applicability to other chronic co-occurring infectious and other medical conditions seen in medical care settings.

Performance of the alcohol use disorders identification test among tuberculosis patients in Russia.

The alcohol use disorders identification test (AUDIT), a screening instrument to identify individuals at risk of alcohol use-related problems, has not been validated in a Russian primary care population. We assessed the reliability, factor structure, sensitivity, and specificity of AUDIT scores among 254 subjects initiating tuberculosis treatment from 2005 to 2007 in Tomsk City. Our findings support the use of the AUDIT as a screening instrument among Russian individuals seeking primary care. We discuss implications, limitations, and future research.

Assessing Motivation to Change Among Problem Drinkers with and without Co-occurring Major Depression

Abstract The University of Rhode Island Change Assessment Scale (URICA) is a widely used measure of readiness to change. To evaluate the URICAs ability to discriminate among alcohol abusers with and without co-occurring major depression, the authors administered it to 193 outpatients court-referred for alcohol treatment. Estimates of internal consistency suggest that scoring the URICA using its traditional factors, as well as using the newer Readiness to Change index, produced variable yet adequately reliable scores. Further, the URICA detected statistically significant differences in motivation to change an alcohol problem between an alcohol use disorder group (AD; n = 131) and an alcohol use disorder with co-occurring depression group (AD/D; n = 62) with the AD/D group showing greater readiness to change. For the AD/D group, separate URICAs were given for alcohol use and depressed mood. Confirming previous findings, results suggest the URICA may lack sensitivity to discriminate among two simultaneously occurring psychological disorders.