Stacey A. McCaffrey

Comparing Fit and Reliability Estimates of a Psychological Instrument using Second-Order CFA, Bifactor, and Essentially Tau-Equivalent (Coefficient Alpha) Models via AMOS 22:

Estimation of composite reliability within a hierarchical modeling framework has recently become of particular interest given the growing recognition that the underlying assumptions of coefficient alpha are often untenable. Unfortunately, coefficient alpha remains the prominent estimate of reliability when estimating total scores from a scale with a hierarchical structure, in part because there are few published articles that provide a step-by-step demonstration of how to estimate reliability within the context of structural equation modeling. Using AMOS 22 to analyze simulated and Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV) summary data, the authors demonstrate how to compare the fit and reliability estimates of a (a) second-order confirmatory factor analytic (CFA) model, (b) bifactor model, and (c) essentially tau-equivalent model, which conforms to the stringent assumptions underlying coefficient alpha. The variance–covariance matrices generated from the simulated data as well as the WAIS-IV data are provided to allow for replication of results.

Development and Validation of an Eight-Item Brief Form of the SOAPP-R (SOAPP-8)

Objective Although the Screener and Opioid Assessment for Patients with Pain-Revised (SOAPP-R) has become a widely used screener for aberrant opioid-related behavior in adults, the length of the instrument may limit its utility. The purpose of the current study was to develop a short form of the SOAPP-R by retaining as few items as possible while maximizing predictive accuracy. Methods Participants (N = 555), recruited from pain clinics, completed the 24-item SOAPP-R and participated in a five-month follow-up visit to evaluate aberrant drug-related behaviors. Opioid aberrant-related behavior was determined through self-report, physician report, and urine toxicology screen. The optimal subset of SOAPP-R items to predict aberrant opioid-related behavior was identified empirically by employing the LASSO selection method and the leave-one-out cross-validation (LOOCV) method offered in the GLMSELECT procedure in SAS 9.4 in conjunction with content expertise. Results Eight items were identified before the selection method stopped. The receiver operating characteristic curve generated from the predicted probabilities from the model produced an area under the curve (AUC) value greater than the AUC value produced by the 24-item SOAPP-R total score and yielded a sensitivity of 0.74 and a specificity of 0.66. Conclusions These results provide strong preliminary support for the SOAPP-8 as a brief screening tool of aberrant opioid-related behavior in chronic pain patients.

Psychometric evaluation of the PainCAS Interference with Daily Activities, Psychological/Emotional Distress, and Pain scales

PurposeThe PainCAS is a web-based clinical tool for assessing and tracking pain and opioid risk in chronic pain patients. Despite evidence for its utility within the clinical setting, the PainCAS scales have never been subject to psychometric evaluation. The current study is the first to evaluate the psychometric properties of the PainCAS Interference with Daily Activities, Psychological/Emotional Distress, and Pain scales.MethodsPatients (N = 4797) from treatment centers and hospitals in 16 different states completed the PainCAS as part of routine clinical assessment. A subsample (n = 73) from two hospital-based treatment centers also completed comparator measures. Rasch Rating Scale Models were employed to evaluate the Interference with Daily Activities and Psychological/Emotional Distress scales, and empirical evaluation included assessment of dimensionality, discrimination, item fit, reliability, information, and person-to-item targeting. Additionally, convergent and discriminant validity were evaluated through classical test theory approaches. Convergent validity of the Pain scales was evaluated through correlations with corresponding comparator items.ResultsOne Interference with Daily Activities item was removed due to poor functioning and discrimination. The retained items from the Interference with Daily Activities and Psychological/Emotional Distress scales conformed to unidimensional Rasch measurement models, yielding satisfactory item fit, reliability, precision, and coverage. Further, results provided support for the convergent and discriminant validity of these two scales. Convergent validity between the PainCAS Pain and BPI Pain items was also strong.ConclusionTaken together, results provide strong psychometric support for these PainCAS Pain scales. Strengths and limitations of the current study are discussed.

A computer adaptive testing version of the Addiction Severity Index—Multimedia Version (ASI–MV): The Addiction Severity CAT.

The purpose of this study was to develop and validate a computer adaptive testing (CAT) version of the Addiction Severity Index—Multimedia Version (ASI–MV), the Addiction Severity CAT. This goal was accomplished in 4 steps. First, new candidate items for Addiction Severity CAT domains were evaluated after brainstorming sessions with experts in substance abuse treatment. Next, this new item bank was psychometrically evaluated on a large nonclinical (n = 4,419) and substance abuse treatment (n = 845) sample. Based on these results, final items were selected and calibrated for the creation of the Addiction Severity CAT algorithms. Once the algorithms were developed for the entire assessment, a fully functioning prototype of an Addiction Severity CAT was created. CAT simulations were conducted, and optimal termination criteria were selected for the Addiction Severity CAT algorithms. Finally, construct validity of the CAT algorithms was evaluated by examining convergent and discriminant validity and sensitivity to change. The Addiction Severity CAT was determined to be valid, sensitive to change, and reliable. Further, the Addiction Severity CAT’s time of completion was found to be significantly less than the average time of completion for the ASI–MV composite scores. This study represents the initial validation of an Addiction Severity CAT based on item response theory, and further exploration of the Addiction Severity CAT is needed.

Behavioral Science Research Methods

This chapter reviews foundational descriptive, quasi-experimental, and experimental research design strategies, while highlighting each of their strengths and weaknesses. Clinical applications of each design are presented, and their unique vulnerabilities to validity threats are discussed. After reading this chapter, the reader will have a firm grasp of the various foundational research design strategies and will have the skills to develop a strong study that is resistant to internal validity threats. Perhaps more importantly, information presented in this chapter should alert the reader to various flaws that are inherent in research, boosting the reader’s critical thinking skills that are necessary to function effectively in nearly any field of study.