Kelli Johnson

Research on Evaluation Use A Review of the Empirical Literature From 1986 to 2005

This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted since that time. The literature review located 41 empirical studies of evaluation use conducted between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood framework allowed a comparison over time. After initially grouping these studies according to Cousins and Leithwood’s two categories and twelve characteristics, one additional category and one new characteristic were added to their framework. The new category is stakeholder involvement, and the new characteristic is evaluator competence (under the category of evaluation implementation). Findings point to the importance of stakeholder involvement in facilitating evaluation use and suggest that engagement, interaction, and communication between evaluation clients and evaluators is critical to the meaningful use of evaluations.

Expert panel reviews of research centers: The site visit process

Site visits are used extensively in a variety of settings within the evaluation community. They are especially common in making summative value decisions about the quality and worth of research programs/centers. However, there has been little empirical research and guidance about how to appropriately conduct evaluative site visits of research centers. We review the processes of two site visit examples using an expert panel review: (1) a process to evaluate four university research centers and (2) a process to review a federally sponsored research center. A set of 14 categories describing the expert panel review process was obtained through content analysis and participant observation. Most categories were addressed differently through the two processes highlighting the need for more research about the most effective processes to use within different contexts. Decisions about how to structure site visits appear to depend on the research context, practical considerations, the level at which the review is being conducted and the intended impact of the report. Future research pertaining to the selection of site visitors, the autonomy of the visitors in data collection and report writing, and the amount and type of information provided would be particularly valuable.

Improving geriatric transitional care through inter-professional care teams

OBJECTIVES The aim of this study was to examine the impact of the use of an inter-professional care team on patient length of stay and payer charges in a geriatric transitional care unit. METHODS An analysis of de-identified administrative records for transitional care patients for the 12-month period (2003-2004) cared for by the inter-professional team (n = 163) and cared for by traditional single provider care model (n = 176) was carried out. We conducted logistic regression on length of stay and charges controlling for patient demographics and acuity levels. RESULTS The inter-professional care team patients had significantly shorter lengths of stay, fewer patient days and lower total charges. Patient diagnosis and acuity were similar across groups. CONCLUSION This study provides empirical evidence of the impact of an inter-professional care model in providing cost-effective transitional care in a nursing home setting. Evidence of shorter lengths of stay, shorter patient days and lower charges suggests benefit in the development and financing of inter-professional care teams for transitional care services.

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals. Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.

Legal Authority for Infectious Disease Reporting in the United States: Case Study of the 2009 H1N1 Influenza Pandemic

Tracking of infectious diseases is a public health core function essential to disease prevention and control. Each state mandates reporting of certain infectious diseases to public health authorities. These laws vary by state, and the variation could affect the ability to collect critical information. The 2009 H1N1 influenza pandemic served as a case study to examine the legal authority in the 50 states; Washington, DC; and New York City for mandatory infectious disease reporting, particularly for influenza and new or emerging infectious diseases. Our study showed reporting laws to be generally present and functioning well; nevertheless, jurisdictions should be mindful of their mandated parameters and review the robustness of their laws before they face a new or emerging disease outbreak.