Lisle Hites

Evaluating policy, systems, and environmental change interventions: Lessons learned from CDC's prevention research centers

Introduction The field of public health is increasingly implementing initiatives intended to make policies, systems, and environments (PSEs) more supportive of healthy behaviors, even though the evidence for many of these strategies is only emerging. Our objective was 3-fold: 1) to describe evaluations of PSE-change programs in which the evaluators followed the steps of the Centers for Disease Control and Prevention’s (CDC’s) Framework for Program Evaluation in Public Health, 2) to share the resulting lessons learned, and 3) to assist future evaluators of PSE-change programs with their evaluation design decisions. Methods Seven Prevention Research Centers (PRCs) applied CDC’s framework to evaluate their own PSE-change initiatives. The PRCs followed each step of the framework: 1) engage stakeholders, 2) describe program, 3) focus evaluation design, 4) gather credible evidence, 5) justify conclusions, and 6) ensure use and share lessons learned. Results Evaluation stakeholders represented a range of sectors, including public health departments, partner organizations, and community members. Public health departments were the primary stakeholders for 4 of the 7 evaluations. Four PRCs used logic models to describe the initiatives being evaluated. Their evaluations typically included both process and outcome questions and used mixed methods. Evaluation findings most commonly focused on contextual factors influencing change (process) and the adoption or implementation of PSE-change strategies (outcome). Evaluators shared lessons learned through various channels to reach local stakeholders and broader public health audiences. Conclusion Framework for Program Evaluation in Public Health is applicable to evaluations of PSE-change initiatives. Using this framework to guide such evaluations builds practice-based evidence for strategies that are increasingly being used to promote healthful behaviors.

Expanding the public health emergency preparedness competency set to meet specialized local and evolving national needs: a needs assessment and training approach.

This article presents an approach to competency development created at the South Central Center for Public Health Preparedness to expand and refine existing competency sets for public health emergency responders. The technique uses course content generated by subject matter experts to validate and expand existing national competency sets. New competencies based on local needs were identified that provide coverage of subject matter appropriate to local public health emergency responders beyond the focus of existing national competency sets.

The South Central Center for Public Health Preparedness Training System Model: A Comprehensive Approach

The South Central Center for Public Health Preparedness (SCCPHP) is a collaboration among the schools of public health at Tulane University and the University of Alabama at Birmingham and the state health departments in Alabama, Arkansas, Louisiana, and Mississippi. The SCCPHP provides competency-based training via distant delivery methods to prepare public health workers to plan for and rapidly respond to public health threats and emergency events. This article presents the training system model used by the SCCPHP to assess, design, develop, implement, and evaluate training that is both competency driven and practice based. The SCCPHP training system model ensures a standardized process is used across public health occupations and geographic regions, while allowing for tailoring of the content to meet the specific training needs of the workforce in the respective state and local health departments. Further, the SCCPHP training system model provides evidence of the reciprocal nature between research and practice needed to advance the area of emergency preparedness training and workforce development initiatives in public health.

Emergency preparedness training of tribal Community Health Representatives

This study describes the development and evaluation of online Public Health Emergency Preparedness (PHEP) training adapted to the learning styles and needs of tribal Community Health Representatives (CHRs). Working through a university-tribal community college partnership, the Arizona Center for Public Health Preparedness at the University of Arizona and Diné College of the Navajo Nation delivered a blended online and face-to-face public health preparedness certificate program based on core public health emergency preparedness competencies. This program was carefully adapted to meet the environmental and learning needs of the tribal CHRs. The certificate program was subsequently evaluated via a scenario-based decision-making methodology. Significant improvements in five of six competency areas were documented by comparison of pre- and post-certificate training testing. Based on statistical support for this pedagogical approach the cultural adaptations utilized in delivery of the certificate program appear to be effective for PHEP American Indian education.

Clinical Nurse Leader Integrated Care Delivery to Improve Care Quality: Factors Influencing Perceived Success

PURPOSE Clinical nurse leader(TM) (CNL)-integrated care delivery is a new model for organizing masters-level nursing clinical leadership at the microsystem level. While there is growing evidence of improved patient care quality and safety outcomes associated with CNL practice, organizational and implementation characteristics that influence CNL success are not well characterized. The purpose of this study was to identify organization and implementation factors associated with perceived success of CNL integration into microsystem care delivery models. METHODS A survey was developed and administered to a nationwide sample of certified CNLs and managers, leaders, educators, clinicians, and change agents involved in planning or integrating CNLs into a health systems nursing care delivery model. Items addressed organizational and implementation characteristics and perceived level of CNL initiative success. Generalized linear modeling was used to analyze data. RESULTS The final sample included 585 respondents. The final model accounted for 35% of variance in perceived CNL initiative success, and included five variables: phase of CNL initiative, CNL practice consistency, CNL instructor or preceptor involvement, CNL reporting structure, and CNL setting ownership status. CONCLUSIONS CNL initiative success is associated with modifiable organizational and implementation factors. CLINICAL RELEVANCE Study findings can be used to inform the development of successful implementation strategies for CNL practice integration into care delivery models to improve care quality outcomes.

Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one‐fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

Evaluation of Team-Based Care in an Urban Free Clinic Setting.

This article reports the experiences of a school of nursing, academic health center, and community-based organization working via an interprofessional collaborative practice model to meet the mutual goal of serving the health care needs of an indigent, largely minority population in Birmingham, Alabama. The population suffers disproportionately from chronic health problems including diabetes, obesity, cardiovascular disease, asthma, and mental health disorders. The program emphasizes diabetes management because the academic health center recognized the need for transitional and primary care, including mental health services, for the increasing numbers of uninsured patients with diabetes and its comorbidities. Half of the clinicians involved in this project had no prior experience with interprofessional collaborative practice, and there was confusion regarding the roles of team members from the partnering institutions. Activities involving care coordination consistently received low scores on weekly rating scales leading to the creation of positions for a nurse care manager and pharmaceutical patient assistance program coordinator. Conversely, shared decision making and cooperation ratings were consistently high. Evaluation identified the need for reliable, accessible data and data analysis to target clinically effective interventions and care coordination and to assess cost effectiveness. The strengths, challenges, lessons learned, and next steps required for sustainability of this alignment are discussed.

Bringing Community and Academic Scholars Together to Facilitate and Conduct Authentic Community Based Participatory Research: Project UNITED

Cultural competency, trust, and research literacy can affect the planning and implementation of sustainable community-based participatory research (CBPR). The purpose of this manuscript is to highlight: (1) the development of a CBPR pilot grant request for application; and (2) a comprehensive program supporting CBPR obesity-related grant proposals facilitated by activities designed to promote scholarly collaborations between academic researchers and the community. After a competitive application process, academic researchers and non-academic community leaders were selected to participate in activities where the final culminating project was the submission of a collaborative obesity-related CBPR grant application. Teams were comprised of a mix of academic researchers and non-academic community leaders, and each team submitted an application addressing obesity-disparities among rural predominantly African American communities in the US Deep South. Among four collaborative teams, three (75%) successfully submitted a grant application to fund an intervention addressing rural and minority obesity disparities. Among the three submitted grant applications, one was successfully funded by an internal CBPR grant, and another was funded by an institutional seed funding grant. Preliminary findings suggest that the collaborative activities were successful in developing productive scholarly relationships between researchers and community leaders. Future research will seek to understand the full-context of our findings.

Impact of elective versus required medical school research experiences on career outcomes

Many US medical schools have added a scholarly or research requirement as a potential intervention to increase the number of medical students choosing to become academic physicians and physician scientists. We designed a retrospective qualitative survey study to evaluate the impact of medical school research at the University of Alabama at Birmingham (UAB) on career choices. A survey tool was developed consisting of 74 possible questions with built-in skip patterns to customize the survey to each participant. The survey was administered using the web-based program Qualtrics to UAB School of Medicine alumni graduating between 2000 and 2014. Alumni were contacted 3 times at 2-week intervals during the year 2015, resulting in 168 completed surveys (11.5% response rate). MD/PhD graduates were excluded from the study. Most respondents completed elective research, typically for reasons relating to career advancement. 24 per cent said medical school research increased their desire for research involvement in the future, a response that positively correlated with mentorship level and publication success. Although completion of medical school research was positively correlated with current research involvement, the strongest predictor for a physician scientist career was pre-existing passion for research (p=0.008). In contrast, students motivated primarily by curricular requirement were less likely to pursue additional research opportunities. Positive medical school research experiences were associated with increased postgraduate research in our study. However, we also identified a strong relationship between current research activity and passion for research, which may predate medical school.

Planning and executing complex large-scale exercises.

INTRODUCTION Increasingly, public health departments are designing and engaging in complex operations-based full-scale exercises to test multiple public health preparedness response functions. The Department of Homeland Securitys Homeland Security Exercise and Evaluation Program (HSEEP) supplies benchmark guidelines that provide a framework for both the design and the evaluation of drills and exercises; however, the HSEEP framework does not seem to have been designed to manage the development and evaluation of multiple, operations-based, parallel exercises combined into 1 complex large-scale event. METHODS Lessons learned from the planning of the Mississippi State Department of Health Emergency Support Function--8 involvement in National Level Exercise 2011 were used to develop an expanded exercise planning model that is HSEEP compliant but accounts for increased exercise complexity and is more functional for public health. RESULTS The Expanded HSEEP (E-HSEEP) model was developed through changes in the HSEEP exercise planning process in areas of Exercise Plan, Controller/Evaluator Handbook, Evaluation Plan, and After Action Report and Improvement Plan development. The E-HSEEP model was tested and refined during the planning and evaluation of Mississippis State-level Emergency Support Function-8 exercises in 2012 and 2013. DISCUSSION As a result of using the E-HSEEP model, Mississippi State Department of Health was able to capture strengths, lessons learned, and areas for improvement, and identify microlevel issues that may have been missed using the traditional HSEEP framework. NEXT STEPS The South Central Preparedness and Emergency Response Learning Center is working to create an Excel-based E-HSEEP tool that will allow practice partners to build a database to track corrective actions and conduct many different types of analyses and comparisons.