Daniela C. Schröter

A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009

Although the general conceptual basis appeared far earlier, theory-driven evaluation came to prominence only a few decades ago with the appearance of Chen’s 1990 book Theory-Driven Evaluations. Since that time, the approach has attracted many supporters as well as detractors. In this paper, 45 cases of theory-driven evaluations, published over a twenty-year period, are systematically examined to ascertain how closely theory-driven evaluation practices comport with the key tenants of theory-driven evaluation as described and prescribed by prominent theoretical writers. Evidence derived from this review to repudiate or substantiate many of the claims put forth both by critics of and advocates for theory-driven forms of evaluation are presented and an agenda for future research on the approach is recommended.

Evaluator competencies: What's taught versus what's sought

This article explores the overlaps and disconnects between the competencies evaluators acquire during graduate school and those required and desired by employers. To investigate this relationship, two different surveys are administered, one for job seekers and the other for employers; 205 postings in the American Evaluation Association job bank were analyzed. The findings suggest that employers, job seekers, and job posters generally agree on the importance of some competencies, such as quantitative analyses and data management. However, some skills desired by employers, such as interpersonal, project management, and presentation skills, differ from skills that job seekers acquire in graduate school. Opportunities for additional experiences in real-world evaluation settings could fill these gaps. Implications for training and future research on training in evaluation are discussed.

Adding a Time-Series Design Element to the Success Case Method to Improve Methodological Rigor An Application for Nonprofit Program Evaluation

Brinkerhoffs Success Case Method (SCM) was developed with the specific purpose of assessing the impact of organizational interventions (e.g., training and coaching) on business goals by analyzing extreme groups using case study techniques and storytelling. As an efficient and cost-effective method of evaluative inquiry, SCM is attractive in other contexts as well, although few examples of such uses are to be found in the published literature. However, modifications of the SCM concept and design are sometimes necessary for implementing the approach in nonprofit environments where business goals are not necessarily an explicit objective. This method note demonstrates how SCM was modified and extended to a social service context, in which the program evaluated was aimed at reducing chronic homelessness and unemployment. Modifications included defining success in a nonprofit setting and adding a time-series element to the design features of traditional SCM to increase methodological rigor.

Does Research on Evaluation Matter? Findings From a Survey of American Evaluation Association Members and Prominent Evaluation Theorists and Scholars

Research on evaluation theories, methods, and practices has increased considerably in the past decade. Even so, little is known about whether published findings from research on evaluation are read by evaluators and whether such findings influence evaluators’ thinking about evaluation or their evaluation practice. To address these questions, and others, a random sample of American Evaluation Association (AEA) members and a purposive sample of prominent evaluation theorists and scholars were surveyed. A majority of AEA members (80.95% ± 7.60%) and sampled theorists and scholars (84.21%) regularly read research on evaluation and indicate that research on evaluation has influenced their thinking about evaluation and their evaluation practice (97.00% ± 3.38% and 94.00% ± 4.79%, for AEA members, and 100% and 100%, for prominent theorists and scholars, respectively).

A Decade of Research on Evaluation: A Systematic Review of Research on Evaluation Published Between 2005 and 2014

Although investigations into evaluation theories, methods, and practices have been occurring since the late 1970s, research on evaluation (RoE) has seemingly increased in the past decade. In this review, 257 studies published in 14 evaluation-focused journals over a 10-year period (between 2005 and 2014) were identified as RoE and then classified according to Henry and Mark’s and Mark’s taxonomies of RoE. The majority of RoE published in evaluation-focused journals consists of descriptive studies and, to a lesser extent, comparative studies of evaluation practices. Few investigations, however, address questions pertaining to values or valuing in evaluation, ethics, or evaluation consequences. Gaps in and an agenda for future RoE are discussed.

A Mixed Methods Study of Some of the Factors Associated With Successful School Strategies for Native Hawaiian Students in the State of Hawai’i

A concurrent mixed methods design, giving equal priority to both quantitative and qualitative methods, was used to investigate some of the factors that influence academic achievement of Native Hawaiian students in public schools in the state of Hawai’i. School-level instructional strategies, curricula, policies, and other variables that distinguish more successful schools from less successful schools were a primary focus of the study. Successful schools are those having collaborative school governance structures, linking decision structures to data, having a well-established and dedicated teaching force, having focused learning communities, having engaged leaders, share accountability for their students’ learning, are committed to continuous learning, and that provide effective supplementary and after-school programming.