Thomas Archibald

Defining and Teaching Evaluative Thinking: Insights From Research on Critical Thinking

Evaluative thinking (ET) is an increasingly important topic in the field of evaluation, particularly among people involved in evaluation capacity building (ECB). Yet it is a construct in need of clarification, especially if it is to be meaningfully discussed, promoted, and researched. To that end, we propose that ET is essentially critical thinking applied to contexts of evaluation. We argue that ECB, and the field of evaluation more generally, would benefit from an explicit and transparent appropriation of well-established concepts and teaching strategies derived from the long history of work on critical thinking. In this article, based on previous work in the fields of education, cognitive science, and critical thinking, as well as on our experience as ECB practitioners, we propose several guiding principles and specific strategies for teaching ET that draw directly from research on the teaching of critical thinking.

Relationships Between Quantitative Measures of Evaluation Plan and Program Model Quality and a Qualitative Measure of Participant Perceptions of an Evaluation Capacity Building Approach

Despite a heightened emphasis on building evaluation capacity and evaluation quality, there is a lack of tools available to identify high-quality evaluation. In the context of testing the Systems Evaluation Protocol (SEP), quantitative rubrics were designed and tested to assess the quality of evaluation plans and models. Interview data were also collected and analyzed using a priori codes. A mixed methods approach was used to synthesize quantitative and qualitative data and explore trends. Consistencies between data types were found for attitude and capacity, and disconnects were found for knowledge, cyberinfrastructure, time, and quality. This approach to data integration represents a novel way to tap the generative potential of divergence that arises when different methods produce contradictory results.

Using a Participatory Approach to Investigate a Leadership Program's Theory of Change.

The linkages between interventions and outcomes of leadership initiatives have been insufficiently studied. To better understand these links, the Virginia agricultural leadership program conducted a pathway mapping session to investigate the program’s theory of change. In this novel process, program participants engaged in collaborative brainstorming to identify outcomes and connect them to program activities. This process helped participants reflect on their learning and helped program staff identify the mechanisms (i.e., learning activities) that led to desired outcomes. The group produced a visual model that represents the program’s theory of change depicting how the program develops capacity for leadership. Using program participants to identify the connections between pedagogy, practice, and outcomes can inform the evidencebase and development of evaluation plans for leadership development programs.

Factors affecting evaluation culture within a non-formal educational organization

While research has been done on many aspects of evaluation within a variety of contexts and organizations, there is a lack of research surrounding the culture of evaluation. This study set out to examine this evaluative culture in one of the worlds largest non-formal educational organizations through the use of an online survey and quantitative methodology. A path model was developed to examine the factors affecting evaluation culture. Results show perception regarding evaluation, program area, college major, location, training in evaluation, degree level, and years of experience explained 28% of the variance within evaluation culture. Results also found that the culture of evaluation is greatly impacted by leadership. By taking a closer look at the evaluation culture of a large non-formal educational organization, much can be learned about how to better develop and support evaluative work in other similar organizations and programs.