Joanne G. Carman

Evaluation Practice Among Community-Based Organizations: Research Into the Reality

Increasingly, government, foundations, and others are asking community-based organizations for more evaluation information. Although the demand for this information may be increasing, the field knows very little about the way organizations are responding to this demand. Using data collected from interviews and a mail survey of community-based organizations, this article addresses five research questions: (a) What types of activities do community-based organizations use to evaluate their programs? (b) What types of data do they collect? (c) How do they collect data? (d) Who has the primary responsibility for conducting evaluation? and (e) Where does the funding come from? The answers to these questions are intended to serve as the basis for future discussions about building evaluation capacity and improving community-based services.

Evaluation Capacity and Nonprofit Organizations: Is the Glass Half-Empty or Half-Full?

In this article, we explore the evaluation capacity of today’s nonprofit organizations. We report the findings of a cluster analysis that suggest that when it comes to evaluation, there are three types of nonprofit organizations. The first type of nonprofit organization is one that, by most accounts, is satisfied with their evaluation efforts. Although these organizations report that they struggle with not having as much time as they would like to devote to evaluation, they are fairly satisfied with their levels of evaluation expertise and report having few problems with the implementation of evaluation systems. The second type of organization has some struggles with evaluation. These organizations report having internal support for evaluation from management, the board and staff, and some capacity to implement an evaluation system, yet they struggle with evaluation design issues, data collection, and resources for evaluation. The third type of organization is one that is struggling across the board. These organizations report having substantial implementation challenges, in terms of lacking basic resources (i.e., staff, funding, time), lacking evaluation expertise, and they report having little support for evaluation from funders, the board, management, and staff. We conclude by exploring the implications of these findings.

The Accountability Movement: What's Wrong With This Theory of Change?

In the field of program evaluation, one of the first things evaluators often do to evaluate a program is to identify the “theory of change” behind the program and create a “logic model” to articulate why the program is expected to work. This allows evaluators to make an assessment about the expectations and feasibility of the program, as well as assess program implementation. In this article, the author uses the theory of change approach to evaluation and creates a logic model for the “accountability movement” or the tendency for funders to require nonprofit organizations to conduct evaluations and report performance information. The model is tested with empirical data, finding that the theory behind the accountability movement suffers from four design and implementation flaws well-known to evaluators: (a) unstandardized treatment, (b) confusion among targets, (c) coverage, and (d) intensity issues. The article concludes by offering insights and recommendations for improving accountability practices.

Nonprofits, Funders, and Evaluation Accountability in Action

This article examines the extent to which different types of funders are asking nonprofit organizations for evaluation and performance measurement data, and describes the many ways in which nonprofit organizations are responding to these requests. The picture that emerges is one that is decidedly mixed, illustrating a range of behaviors that challenges the current perception that most, if not all, funders are asking nonprofit organizations for more evaluation and performance measurement data. The data collected during this study show that only those nonprofit organizations that receive considerable funding from the federal government and the United Way are engaging in program evaluation and performance measurement, compared to nonprofit organizations that receive more funding from state and local governments, foundations, and other sources. Furthermore, the extent to which nonprofit organizations are subjected to external monitoring and descriptive reporting requirements also varies according to the type and amount of funding.

Understanding Evaluation in Nonprofit Organizations

This study uses organizational theory to explore why nonprofit organizations conduct evaluations and how they use evaluation information. According to rational choice theory, managers of nonprofit organizations will conduct evaluations to make sure they are making good decisions about how to improve their services and achieve their goals. Yet, according to data gathered about the experiences of 31 nonprofit organizations, this assumption about the evaluation practices of nonprofits rarely holds true. Rather, other organizational theories, such as agency theory, institutional theory, resource dependence theory, and stewardship theory, which make different assumptions about how organizations operate and how managers behave, provide a more accurate description of why nonprofits engage in evaluation and how they use evaluation information. The findings from this study are important because they help to identify strategies that would encourage nonprofit organizations to use evaluation in more productive and meaningful ways.

Using System Dynamics as an Evaluation Tool Experience From a Demonstration Program

Evaluators are often faced with many challenges in the design and implementation of a programs evaluation. Because programs are entangled in complex networks of structures and stakeholders, they can be challenging to understand, and they often pose issues of competing and conflicting goals. However, by using a systems mapping approach to evaluation design, evaluators can identify and diagram the relationships between different variables and learn about their potential impact and unintended consequences on a programs processes. In this article, we describe how using a systems approach and causal loop mapping approach during an evaluation of a demonstration program for people with developmental disabilities helped evaluators to better understand the programs implementation and the variation in the programs outcomes.

Founding New Nonprofit Organizations: Syndrome or Symptom?

In this study, the authors explore how the dynamics associated with the founding of new nonprofit organizations, the characteristics of the founders, and the developmental life cycles of nonprofit organizations contribute to the seemingly fragmented landscape of the nonprofit sector. Based on data collected from interviews with 31 nonprofit organizations, we find that new nonprofits are being created by passionate, entrepreneurial individuals who hope to make a difference in the community.  Although these organizations are typically small, with few staff members and small budgets, the extent to which they rely on volunteers and are connected to the broader community varies considerably. Moreover, many founders had little experience volunteering or working in the sector. The findings from this study have important implications for the professional development of nonprofit staff, leaders, and volunteers, and they shed new light on how we think about and describe the founders of nonprofit organizations.

Nonprofits and Accreditation: Exploring the Implications for Accountability

In the nonprofit sector of the United States, particularly in the health and human service subsectors, accreditation has emerged as an accountability tool, with some government agencies requiring that service providers be accredited. Using data gathered through interviews with nonprofit executives from organizations currently accredited by the Council on Accreditation, this article explores why some nonprofit organizations choose to become accredited, the benefits associated with accreditation, and the challenges or drawbacks. The article also examines how the performance and quality improvement standards associated with the accreditation process can serve as tools to institutionalize the process of gathering and using performance measurement data in nonprofit organizations.

Evaluation in an Era of Accountability: Unexpected Opportunities--A Reply to Jill Chouinard.

In “The Case for Participatory Evaluation in an Era of Accountability” (this issue), Jill Chouinard highlights seven tensions between the participatory and technocratic approaches to evaluation. She suggests that there is a need to create opportunities for more participatory approaches to evaluation. In this essay, I suggest that the era of accountability has actually created unexpected opportunities for participatory and meaningful evaluation. These include: encouraging organizations to use evidence-based practices and be more deliberate about program strategy; identifying common outcomes and indicators for program managers to adopt and use; and using accountability mandates to create opportunities for capacity building and organizational learning. I also describe how evaluators can use the current focus on outcomes and impact to create opportunities to have broader discussions about the benefits of other types of evaluation.