Robert G. Orwin

Effects of the National Youth Anti-Drug Media Campaign on youths.

OBJECTIVES We examined the cognitive and behavioral effects of the National Youth Anti-Drug Media Campaign on youths aged 12.5 to 18 years and report core evaluation results. METHODS From September 1999 to June 2004, 3 nationally representative cohorts of US youths aged 9 to 18 years were surveyed at home 4 times. Sample size ranged from 8117 in the first to 5126 in the fourth round (65% first-round response rate, with 86%-93% of still eligible youths interviewed subsequently). Main outcomes were self-reported lifetime, past-year, and past-30-day marijuana use and related cognitions. RESULTS Most analyses showed no effects from the campaign. At one round, however, more ad exposure predicted less intention to avoid marijuana use (gamma = -0.07; 95% confidence interval [CI] = -0.13, -0.01) and weaker antidrug social norms (gamma = -0.05; 95% CI = -0.08, -0.02) at the subsequent round. Exposure at round 3 predicted marijuana initiation at round 4 (gamma = 0.11; 95% CI = 0.00, 0.22). CONCLUSIONS Through June 2004, the campaign is unlikely to have had favorable effects on youths and may have had delayed unfavorable effects. The evaluation challenges the usefulness of the campaign.

Marijuana Initiation Among American Youth and Its Risks as Dynamic Processes: Prospective Findings from a National Longitudinal Study

The aim of this study was threefold: (1) to prospectively estimate population incidence rates of marijuana use from late childhood to adolescence, (2) to identify important risk factors, and (3) to examine and demonstrate the dynamic nature of risk factors of marijuana initiation, that is, the degree to which influences change as youth age. The longitudinal data from seven nationally representative age cohorts (aged 10–16 years) of marijuana never-users (N = 4,607) and their parents were used. These data were collected during 1999–2004 under the National Institute on Drug Abuse (NIDA) contract N01DA-8-5063, USA, using the National Survey of Parents and Youth. This survey was designed in part to measure changes in drug-related beliefs, attitudes, and behaviors in American youth and their parents. A series of lagged logistic regression analyses were performed with a cohort-sequential design. Multiple imputation was used to handle missing data and longitudinal replicate weights were incorporated into the analyses. Results showed that the population incidence rates of marijuana use increased from 1.30% to 16.29% from age 11 to 16 years and then appeared to level off. A sharp increase was found during ages 13–15 years. Among six identified important factors, alcohol and/or tobacco use and marijuana offers appeared to be the most important risk factors across ages and age cohorts. Consistent with hypotheses, parental influence and peer influence varied as youth age. Both parental influence and peer influence had significant effects during early adolescence and peer influence continued to middle adolescence. Parental monitoring functioned as a protective factor against peer influence on marijuana initiation, but the effect vanishes during late adolescence. Results provide some empirical evidence of a shift from parental influence to peer influence. The studys limitations are noted.

Gender-Sensitive Substance Abuse Treatment and Arrest Outcomes for Women ☆

The present study links an empirically-developed quantitative measure of gender-sensitive (GS) substance abuse treatment to arrest outcomes among 5109 substance abusing women in mixed-gender short-term residential programs in Washington State. Frailty models of survival analysis and three-level hierarchical linear models were conducted to test the beneficial effects of GS treatment on decreasing criminal justice involvement. Propensity scores were used to control for the pre-existing differences among women due to the quasi-experimental nature of the study. Mens arrest outcomes were used to control for confounding at the program level. Results show that women in more GS treatment programs had a lower risk of drug-related arrests, and women in more GS treatment programs who also completed treatment had a significant reduction in overall arrests from 2 years before- to 2 years after treatment, above and beyond the reduction in arrests due to treatment alone. Implications and directions for future research are discussed.

Transitions through homelessness and factors that predict them: residential outcomes in the Chicago Target Cities treatment sample

Abstract The course of homelessness was examined among adult substance abusers entering treatment in the Chicago Target Cities treatment sample. The research objectives were (1) examine client movement in and out of homelessness over for 2 years post entry into the index treatment episode, and (2) determine the treatment and non-treatment factors that predict achieving and sustaining residential stability. The sample, recruited from treatment programs on Chicagos West Side, was 59% female and predominantly African-American (87%), unemployed (86%), and unmarried (90%). Approximately one-third self-presented as homeless. Descriptive results showed that 73% of initially homeless clients had improved their residential status at 6 months, and 60% were stably housed at 24 months. By contrast, 28% of initially housed clients were not stably housed at 6 months (the majority of these had gone into residential treatment), and only 16% were homeless at 24 months. Sample-wide, homelessness was reduced by 37% between baseline and the 2-year follow-up. The high percentage of homeless substance abusers that achieved and maintained stable housing is consistent with a conclusion that treatment reduced homelessness in the Chicago Target Cities sample. Alternative explanations cannot be entirely ruled out, but are insufficient to nullify the general conclusion. Factors discriminating homeless and housed clients at baseline were consistent with prior literature. Several treatment and non-treatment factors predicted 6- and 24-month housing outcomes in conditional logistic regression models, although the significance and direction of effect estimates varied across conditions. The most consistent predictors were crack as the primary substance, which appears to be a persistent risk factor for becoming and remaining homeless, and whether or not the participant reported that persons were dependent on him/her for food/shelter, which appears to be a persistent protective factor for achieving housing and preventing homelessness. Implications for program planning are discussed.

Assessing state substance abuse prevention infrastructure through the lens of CSAP's Strategic Prevention Framework.

Although the organizational structures and operating procedures of state substance abuse prevention systems vary substantially across states, there is scant empirical research regarding approaches for rigorous assessment of system attributes and which attributes are most conducive to overall effectiveness. As one component of the national cross-site evaluation of the SPF State Incentive Grant Program (SPF SIG), an instrument was developed to assess state substance abuse prevention system infrastructure in order to measure infrastructure change and examine the role of state infrastructure in achieving prevention-related outcomes. In this paper we describe the development of this instrument and summarize findings from its baseline administration. As expected, states and territories were found to vary substantially with respect seven key characteristics, or domains, of state prevention infrastructure. Across the six domains that were assessed using numeric ratings, states scored highest on data systems and lowest on strategic planning. Positive intercorrelations were observed among these domains, indicating that states with high capacity on one domain generally have relatively high capacity on other domains as well. The findings also suggest that state prevention infrastructure development is linked to both funding from state government and the presence of a state interagency coordinating body with decision-making authority. The methodology and baseline findings presented will be used to inform the ongoing national cross-site evaluation of the SPF SIG and may provide useful information to guide further research on state substance abuse prevention infrastructure.

Measurement of gender-sensitive treatment for women in mixed-gender substance abuse treatment programs

BACKGROUND Gender-sensitive (GS) substance abuse treatment services have emerged in response to the multidimensional profile of problems that women display upon admission to substance abuse treatment. The present study examines the extent to which treatment programs vary in GS programming for women in real-world mixed-gender treatment settings, where most women are treated. METHODS Data were collected through site visits using semi-structured interviews with program directors, clinical directors, and counselors in 13 mixed-gender treatment programs from Washington State. Rasch modeling techniques were used to analyze the data. RESULTS Naturally occurring variation was revealed within and across the treatment programs, and demonstrated that reliable measures of three GS domains (Grella, 2008) can be constructed despite a small number of programs. CONCLUSIONS This is the first study to quantify GS treatment for substance abusing women. The identified treatment services and practices and the way they clustered together to form scales have practical implications for researchers, service providers, clinicians, and policy makers. The scales can be used to study treatment outcomes and to evaluate the effectiveness, cost-effectiveness, and cost-benefit of GS programming for women.

Effects of the Strategic Prevention Framework State Incentives Grant (SPF SIG) on State Prevention Infrastructure in 26 States

The Strategic Prevention Framework State Incentive Grant (SPF SIG) program is a national public health initiative sponsored by the U.S. Substance Abuse and Mental Health Services Administration’s Center for Substance Abuse Prevention to prevent substance abuse and its consequences. State grantees used a data-driven planning model to allocate resources to 450 communities, which in turn launched over 2,200 intervention strategies to target prevention priorities in their respective populations. An additional goal was to build prevention capacity and infrastructure at the state and community levels. This paper addresses whether the state infrastructure goal was achieved, and what contextual and implementation factors were associated with success. The findings are consistent with claims that, overall, the SPF SIG program met its goal of increasing prevention capacity and infrastructure across multiple infrastructure domains, though the mediating effects of implementation were evident only in the evaluation/monitoring domain. The results also show that an initiative like the SPF SIG, which could easily have been compartmentalized within the states, has the potential to permeate more broadly throughout state prevention systems.

Welfare Reform and Addiction: A Priori Hypotheses, Post Hoc Explorations, and Assisted Sensemaking in Evaluating the Effects of Terminating Benefits for Chronic Substance Abusers.

The passage of the Contract with America Advancement Act terminated the Social Security Administration’s Disability Insurance and Supplemental Security Income benefits for persons diagnosed with drug or alcohol addiction (also known as DA&A recipients), effective January 1, 1997. From an evaluation standpoint, the law created the opportunity for a “natural” experiment, specifically, an interrupted time series quasi-experiment. Through the availability of person-level data on an entire state population, the present study combines the design tradition of interrupted time series with the analytic tradition of growth curve modeling to create an “enriched” interrupted time series with greater flexibility and explanatory power than the standard design based on aggregate data. Additional techniques from the contemporary evaluation toolkit include: use of propensity scoring for matching and covariance adjustment, multilevel modeling, alternate estimates of the counter-factual, and sensitivity analyses to empirically test the plausibility of validity threats. The study assesses the net impact of the law on labor force participation and criminal behavior among former DA&A recipients in Washington State, using person-level data on employment and arrests from 2 years before implementation to 3 years after. Hypothesis tests showed unambiguously that the law increased employment but did not increase arrests. Post hoc exploratory analyses—framed by Mark et al. [Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco: Jossey Bass] theory of evaluation as assisted sensemaking with particular attention to generative mechanisms and social betterment—suggest a more nuanced interpretation. The combination of a priori and post hoc analyses confirmed that, as a result of the law, there were fewer DA&As on public supports, more DA&As in the workforce, and no evidence of more criminals menacing the public. Analyses also confirmed that a segment of the population, already poor, was made destitute.

Women’s Employment Outcomes Following Gender-Sensitive Substance Abuse Treatment

Employment problems are common among low-income, substance abusing women. The present study links an empirically developed quantitative measure of gender-sensitive (GS) substance abuse treatment to employment outcomes among substance abusing women (N = 5,109) treated in 13 mixed-gender intensive inpatient programs (IIPs) in the Washington State. Hierarchical linear models were used to test the relationship between GS treatment and subsequent employment. Propensity scores and receipt of public assistance were used to control for the preexisting differences among women. Men’s employment outcomes were used to control for potential confounding at the program level. The study found that women treated in more (vs. less) GS treatment programs were more likely to be employed 12 months after treatment admission, though not for the hypothesized 24 months. Treatment completion did not affect the relationship between GS treatment and employment. Findings point to recent progress in tailoring generic substance abuse treatment to women’s needs.

Robert Hornik and Robert Orwin on Stephen Magura's “Failure of Intervention or Failure of Evaluation: A Meta-Evaluation of the National Youth Anti-Drug Media Campaign Evaluation”

Each of Dr. Magura’s specific critiques is right in the abstract, and in our view deeply wrong in the practice of evaluation. Our comments here will be both specific—direct responses to the elements of his critique, and general—a statement of contrasting philosophies of evaluation. 1. The basic design concern. Dr. Magura believes that our use of a longitudinal panel observational design was problematic and perhaps unethical. We should have been doing some form of cluster-randomized design, instead. His argument is that our results were guaranteed to be useless, because we could never be sure that all confounders (measured and unmeasured) were balanced between more and less exposed youth. Our counter-argument is as follows: (a) we had no choice—none of the relevant political actors had any interest in spending