Rebecca Kleinman

External Validity: The Next Step for Systematic Reviews?

Background: Systematic reviews—which identify, assess, and summarize existing research—are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions. Objectives: In this article, we focus on how systematic reviews address external validity. Methods: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research. Results: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics—such as sample demographics or settings—are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married. Conclusion: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Preventing unemployment and disability benefit receipt among people with mental illness: Evidence review and policy significance.

Objective: We identify effective services to assist 3 groups of people with mental illnesses become or remain employed and prevent dependence on disability cash benefits: (a) individuals, including youth, who are experiencing an initial episode of psychosis; (b) employed individuals at risk of losing jobs due to mental illness; and (c) individuals who are or may become long-term clients of mental health services and are likely to apply for disability benefits. Method: We searched for articles published between 1992 and 2015 using key word terminology related to employment support services and each subgroup, and prioritized articles by study design. Results: The individual placement and support model of supported employment is more effective than traditional vocational programs in helping people with serious mental illnesses who are engaged in treatment or receiving disability benefits obtain competitive employment. Some early intervention programs effectively serve people who experience a first episode of mental illness, but more research is needed to demonstrate long-term outcomes. Less is known about the effectiveness of employment interventions in preventing unemployment and use of disability benefits among individuals at risk for job loss or long-term mental illness. Conclusions and Implications for Practice: States can fund employment supports to help prevent the need for disability benefit receipt by creatively combining federal sources, but the funding picture is imperfect. Medicaid expansion and other provisions of the Affordable Care Act may fund employment supports and assist in reducing dependence on disability benefits.

State Strategies for Coordinating Medicaid and Housing Services.

Objective: This article reports findings from case studies of 4 states (Illinois, Louisiana, Massachusetts, and Tennessee) that used different approaches to coordinate Medicaid services with temporary or permanent housing supports for individuals with psychiatric disabilities. Method: Data were collected through document review, telephone interviews with state officials and managed care organizations, and site visits to behavioral health and housing providers, and consumer organizations. Qualitative analyses focused on identifying key features of each state’s approach, including the strengths and limitations from multiple perspectives. Results: All 4 states facilitated partnerships between behavioral health and housing providers. Each state used managed care strategies to some degree and identified opportunities to use Medicaid to finance the coordination of services with housing providers. These financing strategies included using flexible case rates to fund community support workers; using a 1915(i) state plan amendment to fund intensive Medicaid behavioral health services for those in permanent supportive housing; funding new local entities to support local partnerships between health and housing organizations; and creating a Medicaid supportive housing benefit. Conclusions and Implications for Practice: These 4 states took advantage of the flexibility that Medicaid offers to implement different service models in an effort to improve the coordination of behavioral health services and housing. The strategies used in these states may be useful to other states and communities seeking to strengthen coordination of care for individuals who require housing support.