Carolyn J. Heinrich

Outcomes–Based Performance Management in the Public Sector: Implications for Government Accountability and Effectiveness

Requirements for outcomes–based performance management are increasing performance–evaluation activities at all government levels. Research on public–sector performance management, however, points to problems in the design and management of these systems and questions their effectiveness as policy tools for increasing governmental accountability. In this article, I analyze experimental data and the performance–management experiences of federal job–training programs to estimate the influence of public management and system–design factors on program outcomes and impacts. I assess whether relying on administrative data to measure program (rather than impacts) produces information that might misdirect program managers in their performance–management activities. While the results of empirical analyses confirm that the use of administrative data in performance management is unlikely to produce accurate estimates of true program impacts, they also suggest these data can still generate useful information for public managers about policy levers that can be manipulated to improve organizational performance.

Governance and Performance: New Perspectives

Drawing on recent advances in the social sciences, this volume shows how rigorous, theory-based empirical research can help improve the management of public policies and programs - and how better governance can lead to better performance. These original essays demonstrate how better data and improved statistical techniques have allowed researchers to construct more complex models of governance processes and thereby assess the effects of many variables on policy and program outcomes. They present useful research results that illuminate such issues as automatic grade advancement in public schools, management of federally-funded job-training programs, reducing welfare caseloads, and management of welfare-to-work programs. Illustrating a range of theoretical and methodological possibilities, this book shows how more sophisticated research in public management can help improve government performance.

Performance-Based Contracting in Social Welfare Programs:

The Wisconsin Works (W-2) program changed the administrative structure for social welfare services delivery from county government administration to one that includes performance-based contracting and private sector management of programs. We present a theoretical discussion of government contracting and the implications of alternative contract structures for service provider behavior and performance. We then analyze W-2 contract design and management across four contract periods (1997-2005) as the state transitioned to a performance-based contracting regime. We find that the state made rational changes in the W-2 contract specifications and performance measures that should have improved contract efficiency and effectiveness over time, and service providers responded to these changes in allocating effort toward the achievement of priority performance goals. At the same time, deficiencies in program administration and contract management contributed to some contract and performance failures and setbacks.

Evidence-Based Policy and Performance Management Challenges and Prospects in Two Parallel Movements

Both the evidence-based policy and performance management movements aim to improve government effectiveness by developing and utilizing a more rigorous base of information and scientific evidence to guide decisions about program design, funding, implementation, and management. In practice, however, differences and tensions between these movements—such as their methods and standards for assembling and analyzing data, and the strategic timing and use of this information to influence policy and hold public managers accountable for performance—could limit their success. Using cases and empirical studies, this article considers questions about what should count as evidence, how it should be communicated, who should judge the quality and reliability of evidence and performance information, and how to achieve a balance between processes that produce rigorous information for decision making and those that foster democratic governance and accountability. Recommendations are made for improving government effectiveness by using more rigorous information in decision making, along with acknowledgment of the limitations and risks associated with such efforts.

The Effects of Welfare-to-Work Program Activities on Labor Market Outcomes

Studies examining welfare‐to‐work program effectiveness present mixed and sometimes discrepant findings, partly due to research design, data, and methodological limitations. Using administrative data on Missouri and North Carolina welfare recipients, we substantially improve on past estimation approaches to identify the distinct effects of each state’s welfare‐to‐work subprograms—assessment, job search assistance and job readiness training, and more intensive programs designed to augment human capital. More intensive training is associated with greater initial earnings losses but also greater long‐run earnings gains. The negative program impacts we observe in quarters immediately following participation turn positive by the second year after participation.

Supplemental Education Services under No Child Left Behind: Who Signs up, and What Do They Gain?.

Schools that have not made adequate yearly progress in increasing student academic achievement are required, under No Child Left Behind (NCLB), to offer children in low-income families the opportunity to receive supplemental educational services (SES). In research conducted in Milwaukee Public Schools, the authors explore whether parents and students are aware of their eligibility and options for extra tutoring under NCLB, and who among eligible students registers for SES. Using the best information available to school districts, the authors estimate the effects of SES in increasing students’ reading and math achievement. Their nonexperimental estimates suggest no average effects of SES attendance on student achievement gains. They use qualitative research to explore possible explanations for the lack of observed effects.

Improving the Organization, Management, and Outcomes of Substance Abuse Treatment Programs

Objective: This study presents an organizing framework for empirical analyses of substance abuse treatment program effectiveness, based on relevant theories of organization and public management and the body of substance abuse treatment studies, and is applied in analyses of treatment program data. Method: We use descriptive analyses of data from the National Treatment Improvement Evaluation Study that were collected from 519 substance abuse treatment service delivery units and 6593 patients to identify instruments of policy and management that might either frustrate or facilitate the implementation of “best practices” in substance abuse treatment. Results: The analyses show statistically significant and substantively interesting relationships among measures of organizational structure and mission, financial management (e.g., revenues per patient and revenue sources), human resources management (e.g., staffing levels, the use of case managers, etc.), and measures of service technology (e.g., the provision of supportive services, counseling intensity, etc.). Conclusion: Researchers should strive to measure and account for the significant interactions among structural, management, and service technology variables in substance abuse treatment programs and the impact of these variables, mediated through patient characteristics and pre-treatment histories, on treatment outcomes.

Setting the Standard in Performance Measurement Systems

ABSTRACT A fundamental challenge in the design of performance measurement and incentive systems is the establishment of appropriate benchmark levels of performance, also known as performance standards. Drawing from the information economics, contract theory and public administration literatures, we derive theoretical implications for the construction of performance standards. We then assess alternative methods that are commonly used to construct performance standards and consider their application in performance measurement systems in public programs. We draw out important lessons for the establishment of performance benchmarks and other implications for performance standards system design in public organizations.

Do Public Employment and Training Programs Work

We estimate impacts on earnings and employment of the two primary adult workforce support and training programs under the U.S. Workforce Investment Act (WIA) using administrative data on 160,000 participants from 12 states for up to four years following program entry. We find that participants in the WIA Adult program, who typically enter with poor work histories, realize improved employment levels and increased average quarterly earnings of several hundred dollars. Earnings gains for Dislocated Worker program participants are appreciably smaller, although these participants do experience employment gains.JEL codesI38, J08, J24

Modeling Trajectories in Social Program Outcomes for Performance Accountability.

Government and public focus on accountability for program outcomes, combined with practical and ethical constraints on experimental designs, make nonexperimental studies of social programs an increasingly common approach to producing information on program performance. In this paper, we compare the effectiveness of alternative nonexperimental evaluation methods used with longitudinal data to produce information about social program outcomes. The analysis is applied in an evaluation of a demonstration program designed to help young parents and their partners make the transition from receiving public transfer income to earned income. Empirical findings indicate that modeling outcome trajectories using multilevel methods generates more complete information about the nature of the program effects, relative to standard econometric alternatives as commonly applied in evaluations. With increased access to sources of administrative and management information system data covering multiple time points, we urge researchers to help social policy and program decision-makers appropriately model and produce more accurate information on program outcomes.