Leanne M. Kallemeyn

Cross-Continental Reflections on Evaluation Practice: Methods, Use, and Valuing.

The evaluation theory tree typology reflects the following three components of evaluation practice: (a) methods, (b) use, and (c) valuing. The purpose of this study was to explore how evaluation practice is conceived as reflected in articles published in the American Journal of Evaluation (AJE) and Evaluation, a journal supported by the European Evaluation Society. A key finding from this international comparison suggests that evaluation practice as reflected in AJE and Evaluation both emphasize methods, in comparison to use and valuing. This article concludes using Peter Dahler-Larsen’s discussion on evaluation societies, among other sources, to examine the audit society, which might account for the trends in our findings. EvalPartners, a global community of evaluators, has declared 2015 as the international year of evaluation. These findings regarding cross-continental trends in evaluation are relevant for engaging in a global dialogue on evaluation practice.

Perceived Purposes of the Head Start National Reporting System at the Local Program Level: A Case Study

The Head Start National Reporting System (NRS) exemplified an assessment system where significant ambiguity existed regarding its purpose(s). This article considers the perceived purposes of the NRS from a local program perspective through observations and interviews with staff members during the 2004–2005 school year. This study found that a local program primarily viewed the NRS as providing accountability at the federal level. In comparison, the NRS provided limited accountability at the local level and was not used to help guide program development. The researchers also documented lessons learned through a collaborative research process that considered how assessment systems may be used for the purposes of development, in addition to accountability, at the local program level.

Case study of an evaluation coaching model: Exploring the role of the evaluator

This study examined the role of the external evaluator as a coach. More specifically, using an evaluative inquiry framework (Preskill & Torres, 1999a; Preskill & Torres, 1999b), it explored the types of coaching that an evaluator employed to promote individual, team and organizational learning. The study demonstrated that evaluation coaching provided a viable means for an organization with a limited budget to conduct evaluations through support of a coach. It also demonstrated how the coaching processes supported the development of evaluation capacity within the organization. By examining coaching models outside of the field of evaluation, this study identified two forms of coaching--results coaching and developmental coaching--that promoted evaluation capacity building and have not been previously discussed in the evaluation literature.

School-level organizational routines for learning: supporting data use

Purpose – The purpose of this paper is to use an extreme case to identify and describe the nature of routines that might support processes and outcomes of data use, drawing from a framework developed by Coburn and Turner (2012a). Design/methodology/approach – The author conducted a four-month case study (Stake, 1995) of an elementary school in a large urban school district that had implemented balanced score cards. The author identified a school that had strong qualities to support data use, including leadership and information systems. Findings – Two school-level organizational routines facilitated teachers’ data use: collaborative teams and processes of inquiry. These routines stored knowledge about the types of data teachers ought to notice, and to a lesser extent, how they ought to interpret data and construct implications for practice. These routines also provided opportunities for single and double-loop learning (Argyris and Schon, 1996) and might contribute to improvements in student learning. This...

Lessons Learned about Perceived Purposes of the Head Start National Reporting System at the Local Program Level.

Two purposes for early childhood assessment systems are informing program development and providing accountability. Based on findings from a case study of a local Head Start program implementing the National Reporting System (NRS) in 2004–2005, we provide 5 lessons learned from experiences with the NRS that may have implications for other assessment and accountability systems in early childhood programs. The lessons include (a) be intentional about the purposes of accountability and/or development, (b) design assessment and accountability systems so that the purposes fit with both the local and federal levels, (c) attempt to take ownership for assessment and accountability systems at the local level, (d) contextualize the assessed child outcomes, and (e) link program processes with child outcomes.

Methodological Changes and Respecting Stakeholder Dignity.

The purpose of an evaluation, its methodology and design, and the data that evaluators generate represent programs and stakeholders in particular ways. In the evaluation described in this article, I faced making extensive methodological changes to an evaluation. In the process, I considered and reflected on the implications of these changes for the stakeholders involved in the program and the evaluation. What was my responsibility? To help me answer this question, I read through and reflected on the Guiding Principles for Evaluators (2004) by the American Evaluation Association (AEA) at critical and periodic points during the evaluation, using them to ‘‘sensitize [myself] to critical domains relevant to [my] work’’ (Morris, 2008, p. 8). In the case example presented below, I demonstrate when the guiding principles were helpful and not helpful in navigating the methodological changes. I identify some of the guiding principles that were beneficial at particular points in the process. I conclude with a discussion of how Guiding Principle Respect for People D.4 was particularly salient during the entire evaluation. ‘‘Knowing that evaluations may negatively affect the interests of some stakeholders, evaluators should conduct the evaluation and communicate its results in a way that clearly respects the stakeholders’ dignity and self-worth.’’ The questions for me were: a) What type of evaluation do I conduct to respect ‘‘stakeholders’ dignity and self-worth’’? b) Which stakeholders? Although professional ethics, such as the guiding principles that will be the focus in this discussion, provide one influence on ethical decision making, another influence is an evaluator’s tendency toward a particular approach to doing ethics (Simons, 2006). I embrace an ethics of care (refer to Christians, 2000; Gilligan, 1982; Noddings, 1984), which closely aligns with participatory and democratic approaches to evaluation (e.g., House & Howe, 1999).