Paul J. Margolies

Use of Learning Collaboratives by the Center for Practice Innovations to Bring IPS to Scale in New York State

This column focuses on use of learning collaboratives by the Center for Practice Innovations to help programs implement the evidence-based individual placement and support model of supported employment in New York State. These learning collaboratives use fidelity and performance indicator data to drive the development of program-specific individualized quality improvement plans. As of 2014, 59 (69%) of 86 eligible programs have joined the initiative. Programs are achieving employment outcomes for consumers on par with national benchmarks, along with improved fidelity.

Best Practices: Wellness Self-Management: An Adaptation of the Illness Management and Recovery Program in New York State

Wellness Self-Management (WSM) is a recovery-oriented, curriculum-based practice designed to help adults with serious mental health problems make informed decisions and take action to manage symptoms and improve their quality of life. WSM is an adaptation of the illness management and recovery program, a nationally recognized best practice. WSM uses comprehensive personal workbooks for group facilitators and consumers and employs a structured and easy-to-implement group facilitation framework. Currently, more than 100 adult mental health agencies are implementing WSM in New York State. The authors describe the development and key features of WSM and an initiative to promote widespread adoption and sustainability.

Distance Training and Implementation Supports to Scale Up Integrated Treatment for People With Co-occurring Mental Health and Substance Use Disorders

Objective: Integrated, evidence-based treatment for co-occurring severe mental illness and substance use disorder (co-occurring disorders) reduces substance use, hospitalization, homelessness, incarceration, and treatment costs and promotes individuals’ recovery, independent living, and employment. This report describes and characterizes early uptake of New York States Web-based training and distance implementation supports to promote statewide dissemination of integrated, evidence-based co-occurring disorders services. Methods: Thirty-five half-hour online modules provide training on integrated evidence-based co-occurring disorders treatment. Distance supports for implementation include monthly webinars, phone calls, and virtual learning communities. The authors calculated the proportion of programs with one or more practitioners who accessed the Web-based resources and the cumulative number of modules completed as an indicator of the reach of the Web-based approach to scaling up an innovation. They also calculated the number and proportion of staff per program who accessed the Web-based resources as an indicator of critical mass that may be needed for sustainability. Results: Of 369 programs participating in the initiative, 232 (63%) had at least one staff member accessing the online modules; 30% of programs had one-third or more of their staff accessing the online modules, and total attendance at webinars was close to 700. Staff who started taking the training averaged a little more than 6 modules, with 8,862 modules completed across staff to date. Conclusions: Early uptake of the New York State initiative suggests that distance learning and supports may be a feasible vehicle to train practitioners in large treatment delivery systems.

Is There a Role for Fidelity Self-Assessment in the Individual Placement and Support Model of Supported Employment?

OBJECTIVE Fidelity assessments help ensure that evidence-based practices are implemented properly. Although assessments are typically conducted by independent raters, some programs have implemented self-assessments because of resource constraints. Self-assessments were compared with independent assessments of programs implementing individual placement and support supported employment. METHODS Eleven community-based outpatient programs in New York State completed both self- and independent assessments. Intraclass correlation coefficients and paired t tests were used to compare scores from self- and independent assessments. RESULTS For both assessment methods, mean scores for all programs were within the range of fair fidelity. Self- and independent assessment total scores were not significantly different; however, significant differences were found on some scale items in this small sample. CONCLUSIONS Self-assessment may be valid for examining a programs overall functioning and useful when resource constraints prevent independent assessment. Independent assessors may be able to identify nuances, particularly on individual assessment items, that can point to areas for program improvement.

Using Distance Technologies to Facilitate a Learning Collaborative to Implement Stagewise Treatment

OBJECTIVE This report describes experiences and outcomes of an online learning collaborative focused on implementation of stagewise treatment. METHODS Eleven participating programs convened online monthly for a year. Between meetings, program staff created an implementation plan and programs collected performance indicator data, including assessment of staff knowledge of integrated treatment for people with co-occurring disorders, whether a persons current stage of treatment was documented in his or her chart, and whether the treatments were appropriate for the stage of treatment. Descriptive statistics were used to characterize performance indicators and feedback. Wilcoxon matched-pairs signed-rank tests examined changes in performance indicators over time. RESULTS Program staff generally demonstrated significant improvements in performance indicators over time and rated the distance learning collaborative favorably. CONCLUSIONS Distance learning collaboratives can be structured to provide opportunities for program staff to interact and learn from one another and to implement and sustain changes.

An Online Training Module on the Cultural Formulation Interview: The Case of New York State

Professional organizations and government guidelines recommend cultural competence training for providers, but the lack of a standardized cultural assessment has hindered research. Studies with the DSM-5 Cultural Formulation Interview (CFI) suggest that active learning during training improves perceptions of the CFIs usefulness as a cultural competence tool. This column reports demographic characteristics and evaluation scores among 423 providers who completed an online CFI training module developed through the New York State Office of Mental Health. Both the module, which uses the principle of active learning, and the CFI were associated with strong favorability ratings.

Using Instructional Design, Analyze, Design, Develop, Implement, and Evaluate, to Develop e-Learning Modules to Disseminate Supported Employment for Community Behavioral Health Treatment Programs in New York State

Background Implementation science lacks a systematic approach to the development of learning strategies for online training in evidence-based practices (EBPs) that takes the context of real-world practice into account. The field of instructional design offers ecologically valid and systematic processes to develop learning strategies for workforce development and performance support. Objective This report describes the application of an instructional design framework—Analyze, Design, Develop, Implement, and Evaluate (ADDIE) model—in the development and evaluation of e-learning modules as one strategy among a multifaceted approach to the implementation of individual placement and support (IPS), a model of supported employment for community behavioral health treatment programs, in New York State. Methods We applied quantitative and qualitative methods to develop and evaluate three IPS e-learning modules. Throughout the ADDIE process, we conducted formative and summative evaluations and identified determinants of implementation using the Consolidated Framework for Implementation Research (CFIR). Formative evaluations consisted of qualitative feedback received from recipients and providers during early pilot work. The summative evaluation consisted of levels 1 and 2 (reaction to the training, self-reported knowledge, and practice change) quantitative and qualitative data and was guided by the Kirkpatrick model for training evaluation. Results Formative evaluation with key stakeholders identified a range of learning needs that informed the development of a pilot training program in IPS. Feedback on this pilot training program informed the design document of three e-learning modules on IPS: Introduction to IPS, IPS Job development, and Using the IPS Employment Resource Book. Each module was developed iteratively and provided an assessment of learning needs that informed successive modules. All modules were disseminated and evaluated through a learning management system. Summative evaluation revealed that learners rated the modules positively, and self-report of knowledge acquisition was high (mean range: 4.4–4.6 out of 5). About half of learners indicated that they would change their practice after watching the modules (range: 48–51%). All learners who completed the level 1 evaluation demonstrated 80% or better mastery of knowledge on the level 2 evaluation embedded in each module. The CFIR was used to identify implementation barriers and facilitators among the evaluation data which facilitated planning for subsequent implementation support activities in the IPS initiative. Conclusion Instructional design approaches such as ADDIE may offer implementation scientists and practitioners a flexible and systematic approach for the development of e-learning modules as a single component or one strategy in a multifaceted approach for training in EBPs.

Decisional Informatics for Psychosocial Rehabilitation: A Feasibility Pilot on Tailored and Fluid Treatment Algorithms for Serious Mental Illness

Abstract This study introduces a computerized clinical decision-support tool, the Fluid Outpatient Rehabilitation Treatment (FORT), that incorporates individual and ever-evolving patient needs to guide clinicians in developing and updating treatment decisions in real-time. In this proof-of-concept feasibility pilot, FORT was compared against traditional treatment planning using similar behavioral therapies in 52 adults with severe mental illness attending community-based day treatment. At posttreatment and follow-up, group differences and moderate-to-large effect sizes favoring FORT were detected in social function, work readiness, self-esteem, working memory, processing speed, and mental flexibility. Of participants who identified obtaining a General Education Diploma as their goal, 73% in FORT passed the examination compared with 18% in traditional treatment planning. FORT was also associated with higher agency cost-effectiveness and a better average benefit-cost ratio, even when considering diagnosis, baseline symptoms, and education. Although the comparison groups were not completely equivalent, the findings suggest computerized decision support systems that collaborate with human decision-makers to personalize psychiatric rehabilitation and address critical decisions may have a role in improving treatment effectiveness and efficiency.

Using incentives for training participation.

TOPIC This column describes how public partners can help incentivize participation in training. Specifically, a state mental health agency and its implementation center applied financial and nonfinancial incentives to encourage participation in training and implementation supports. PURPOSE Although training is not sufficient to change practice, it is a necessary first step in implementing evidence-based treatments. Finding ways to incentivize participation, particularly strategies with minimal resource involvement, is important for the psychiatric rehabilitation workforce and cash-strapped public systems. SOURCES USED This description draws from published material and experiences from New York State. CONCLUSIONS AND IMPLICATIONS FOR PRACTICE Engaging public partners to incentivize training can significantly increase participation in training. Incentive programs exist that do not require additional funding-an important consideration, given the fiscal climate for most public payers.

Behavioral and Physical Health Self-Management

In addition to its traditional focus on acute illnesses and injuries, our healthcare system has increasingly focused on enduring, serious conditions (Wagner et al. 2001). With this shift in focus has come a shift in the role for users of these services. For acute problems, there is little need to engage patients in having the knowledge and skills to successfully self-manage persisting conditions. This changes considerably when the focus shifts to enduring conditions. The term chronic is customarily used in the medical community to describe enduring conditions, while the behavioral health system typically refers to serious and persisting conditions. Both terms are used throughout this chapter to describe those nonacute conditions that are long lasting, associated with functional difficulties, and require ongoing care.