Brenda M. Joly

Train-the-trainer as an educational model in public health preparedness.

Train-the-trainer (TTT) is a widely acknowledged educational model across a number of disciplines, including public health preparedness. However, many questions arise about its proper use, its efficacy, and its optimal role in preparedness education. After careful consideration of the literature and past anecdotal experiences with this educational model, TTT was selected as the preferred and sustainable method for public health preparedness in Maine. Upon completion of Year 1 of the Maine TTT program, our evaluation results revealed that TTT was successful and well received by participants. It is evident that TTT is potentially a valuable tool in public health preparedness, but it should be noted that there is no clear prescription for implementing TTT. Institutions and communities adopting TTT as an educational model must be flexible and willing to revise the training plan as challenges are encountered. Future research is required to help clarify optimal ways of using TTT to promote public health preparedness in the 21st century.

Measuring Quality Improvement in Public Health The Development and Psychometric Testing of a QI Maturity Tool

There is growing interest and investment in improving the quality of public health services and outcomes. Following the lead of other sectors, efforts are underway to introduce systematic quality improvement (QI) tools and approaches to state and local public health agencies. Little is known, however, about how to describe and reliably measure the level of QI maturity within a public health agency. The authors describe the development of a QI Maturity Tool using research from the fields of organizational design, psychology, health care, and complexity theory. The 37-item assessment tool is based on four quality domains derived from the literature: (a) organizational culture, (b) capacity and competency, (c) practice, and (d) alignment and spread. The tool was designed to identify features of an organization that may be enhancing or impeding QI; monitor the impact of efforts to create a more favorable environment for QI; and define potential cohorts of public health agencies for evaluation purposes. The article presents initial steps in testing and validating the QI Maturity Tool including: (a) developing a theoretical framework, (b) assuring face and content validity, (c) determining the tool’s reliability based on estimates of internal consistency, (d) assessing the dimensionality, and (f) determining the construct validity of the instrument. The authors conclude that there is preliminary evidence that the QI Maturity Tool is a promising instrument. Further work is underway to explore whether self-reported survey results align with an agency’s actions and the products of their QI efforts.

Quality improvement learning collaboratives in public health: findings from a multisite case study.

OBJECTIVES This study was designed to (1) test a theory on the key drivers of a successful mini-collaborative, (2) describe the application of quality improvement (QI) approaches and techniques among mini-collaborative participants, and (3) identify key attributes that affect the spread and sustainability of QI efforts within a local public health agency. METHODS A multisite case study methodology was used to evaluate a subset of mini-collaboratives and their participating local health departments that were working in specific target areas. Data were collected during semistructured interviews and while observing mini-collaborative meetings. Documentation reviews were also conducted. We used standard techniques to code the data on the basis of themes and connections between themes. RESULTS The findings provide early evidence that support our case study theory on the drivers of a successful mini-collaborative including advanced planning, the selection of faculty, timely training and technical assistance, the role of senior leaders, the application of evidence-based practices, the use of an improvement model, evaluation efforts, communication, the availability of resources, target selection, and prior experience with and application of QI. While the case studies provided limited evidence of sustainability and broad spread of QI within participating states, mini-collaboratives appear to have had a major impact and will likely influence the work of participating local health departments going forward. CONCLUSIONS Our findings suggest that the mini-collaboratives served as a catalyst for engaging local health departments in the theory and practice of QI.

Assessing quality improvement in local health departments: results from the Multi-State Learning Collaborative.

OBJECTIVE This study examined changes in quality improvement: (1) over a 3-year period, (2) among mini-collaborative participants, and (3) among agencies that were classified in the lowest QI performance quartile. METHODS : A QI Maturity Tool was administered to all local health departments in the Multi-State Learning Collaborative states. Factorial ANOVA was performed to determine differences in composite factor scores and interaction effects were explored. RESULTS : The results revealed a significant increase in the percent of agencies that reported ever implementing a formal QI process from 2009 to 2011 and agencies that participated in a mini-collaborative were more likely than their non-mini-collaborative counterparts to report QI implementation. The findings also suggested significant changes in QI capacity and competency as well as alignment and spread over the course of the grant, among mini-collaborative participants, and among agencies that were originally identified in the lowest QI maturity quartile. No significant changes in organizational culture were revealed by year or among mini-collaborative participants. However, agencies in the lowest quartile saw significant changes in organizational culture during the project period. CONCLUSIONS Significant changes in specific QI domains are possible to detect during a 3-year period. Our research gives us confidence that the QI Maturity Tool measures the right domains and is an important contribution to quantifying the adoption and spread of QI in public health. However, further refinement is needed to better standardize language and definitions of the component parts of a QI system.

Effectiveness of public health quality improvement training approaches: Application, application, application

CONTEXT Quality improvement (QI) has been identified as a key strategy to improve the performance of state and local public health agencies. Quality improvement training effectiveness has received little attention in the literature. OBJECTIVES To evaluate the effectiveness of 3 QI training types: webinars, workshops, and demonstration site activities on improving participant knowledge, skill, and ability to conduct QI through a questionnaire conducted after training participation. DESIGN We used a natural experimental design hypothesizing that demonstration site participants would have the greatest gains on outcomes of interest compared with webinar and workshop participants. Bivariate and multivariate models were used to examine outcome differences between questionnaire respondents who participated in various training types. PARTICIPANTS Local health department employees who participated in the 3 training strategies. MAIN OUTCOME MEASURES Measures included knowledge and skill gain, skill application, QI receptivity, and ability to successfully participate in a QI project. RESULTS Two hundred eighty-four unique individuals who work in 143 health departments completed the questionnaire for a 59% response rate. The majority of these health departments serve midsize populations. Demonstration site respondents had significantly greater gains in knowledge and skills, skill application, and ability to successfully participate in a QI project. Webcast training participants had significantly higher QI receptivity in adjusted models. Respondents who participated in both webcast and demonstration site trainings had higher mean scores on all outcomes when compared with demonstration site single training participants, these differences were significant in unadjusted models. CONCLUSION Our findings suggest that QI training for public health agency employees should include both didactic training on QI content and opportunities for QI application. Future research should examine if this approach can effectively increase successful participation in QI projects for staff in LHDs of all sizes.

Driving a public health culture of quality: how far down the highway have local health departments traveled?

CONTEXT There has been an extensive investment in building public health organizational capacity to improve performance and prepare for accreditation. An evolving perspective has focused not only on the practice of quality improvement (QI) within the health department but also upon the extent the culture of QI is embraced within the agency. OBJECTIVE No studies have examined the current national baseline of QI culture implementation, nor estimated the degree of QI sophistication local health departments (LHDs) have attained. We attempt to fill this void by aligning the findings from the QI module of the National Association of County & City Health Officials (NACCHO) 2010 Profile of LHDs against the constructs defined by the QI Maturity Tool and the NACCHO QI Roadmap (Roadmap to a Culture of Quality Improvement). DESIGN Specific questions regarding QI activities from the 2010 Profile Study QI module were used to assign responding LHDs to stages within the Roadmap. We also used data from the QI Maturity Tool administered to all LHDs in the 16 participating Multi-State Learning Collaborative states in 2010 and 2011. On the basis of this matched set, we applied the summative domain scores algorithm, classified agencies into 1 of 5 groups, compared our findings with those of the NACCHO survey, and aligned our categories to those of the Roadmap. RESULTS Nearly 80% of LHDs classified using the NACCHO Profile data were assigned to group 3 or 4 versus 48% using the QI Maturity Tool. Results from the cross-tabulations of the matched data set between the QI Maturity Tool classifications and the NACCHO Profile classifications revealed exact alignment 30% of the time. Forty-nine of 163 agencies were classified in the same grouping in both schemata. In addition, 84% of the agencies were classified within 1 neighboring category. CONCLUSIONS The results revealed that half, if not most, LHDs fall within the middle categories of QI maturity and sophistication, regardless of which classification system was deployed.

Using the QI Maturity Tool to Classify Agencies Along a Continuum

Major investments have been made to encourage health departments to implement quality improvement (QI) efforts. Yet, there are few empirically tested tools for public health agencies that assess these efforts and classify health departments along a QI continuum. This paper presents a new classification scheme for measuring QI Maturity in public health agencies based on a validated tool. The findings can be used to establish benchmarks, make comparisons and conduct future research linking QI and population health outcomes.

Evaluating the Multi-State Learning Collaborative

The third phase of the Multi-State Learning Collaborative (MLC) is a 3-year Robert Wood Johnson Foundation initiative designed to build quality improvement capacity within state and local health departments. This commentary provides a brief overview of the MLC, a methodological description of its evaluation, and preliminary findings. Major evaluation data sources include quarterly reports, surveys, case studies, and key informant interviews. Preliminary survey data and quarterly reports from year 1 provide some early evidence that states are progressing toward MLC goals. Approximately 27 percent of local health departments in the 16 participant states report that they have begun to prepare for national accreditation and approximately 39 percent report implementation of at least one quality improvement project within the past 12 months. Ongoing data collection efforts are underway to more fully address the evaluation questions.

Early Detection, Intervention and Prevention of Psychosis Program: Community Outreach and Early Identification at Six U.S. Sites

OBJECTIVE This study assessed the effects of a community outreach and education model implemented as part of the Early Detection, Intervention and Prevention of Psychosis Program (EDIPPP), a national multisite study in six U.S. regions. METHODS EDIPPPs model was designed to generate rapid referrals of youths at clinical high risk of psychosis by creating a network of professionals and community members trained to identify signs of early psychosis. Qualitative and quantitative data were gathered through an evaluation of outreach efforts at five sites over a two-year period and through interviews with staff at all six sites. All outreach activities to groups (educational, medical, and mental health professionals; community groups; media; youth and parent groups; and multicultural communities) were counted for the six sites to determine correlations with total referrals and enrollments. RESULTS During the study period (May 2007-May 2010), 848 formal presentations were made to 22,840 attendees and 145 informal presentations were made to 11,528 attendees at all six sites. These presentations led to 1,652 phone referrals. A total of 520 (31%) of these individuals were offered in-person orientation, and 392 (75%) of those were assessed for eligibility. A total of 337 individuals (86% of those assessed) met criteria for assignment to the EDIPPP study. CONCLUSIONS EDIPPPs outreach and education model demonstrated the effectiveness of following a protocol-defined outreach strategy combined with flexibility to reach culturally diverse audiences or initially inaccessible systems. All EDIPPP sites yielded appropriate referrals of youths at risk of psychosis.

Evaluation as a Critical Factor of Success in Local Public Health Accreditation Programs

This article presents the variety of approaches used to conduct evaluations of performance improvement or accreditation systems, while illustrating the complexity of conducting evaluations to inform local public health practice. We, in addition, hope to inform the Exploring Accreditation Program about relevant experiences involving accreditation and performance assessment processes, specifically evaluation, as it debates and discusses a national voluntary model. A background of each state is given. To further explore these issues, interviews were conducted with each states evaluator to gain more in-depth information on the many different evaluation strategies and approaches used. On the basis of the interviews, the authors provide several overall themes, which suggest that evaluation is a critical tool and success factor for performance assessment or accreditation programs.