Brad Astbury

Unpacking Black Boxes: Mechanisms and Theory Building in Evaluation:

There is growing interest in the concept of ‘‘mechanism’’ across many areas of the social sciences. In the field of program and policy evaluation, a number of scholars have also emphasized the importance of causal mechanisms for explaining how and why programs work. However, there appears to be some ambiguity about the meaning and uses of mechanism-based thinking in both the social science and evaluation literature. In this article we attempt to clarify what is meant by mechanisms in the context of program evaluation by identifying three main characteristics of mechanisms and outlining a possible typology of mechanisms. A number of theoretical and practical implications for evaluators are also discussed, along with some precautions to consider when investigating mechanisms that might plausibly account for program outcomes.

Some reflections on Pawson’s Science of Evaluation: A Realist Manifesto:

This article examines the evolution of Ray Pawson’s realist theory of evaluation, with a particular focus on his most recent book, Science of Evaluation: A Realist Manifesto. It is not a substitute for reading the original text. Reviews always say less about a book than the book in question. The goal is to provide a broader context for interpretation and an invitation to consider critically the practical import of Pawson’s grand ambitions for a new evaluation science. Like previous writings, this latest call-to-arms will appeal to some quarters of the evaluation community and dismay others. Regardless, evaluators should not remain indifferent. Pawson presents a vigorous, if at times irreverent, argument for advancing evaluation as a realist enterprise. Science of Evaluation deserves to be read and discussed widely.

Systematic development and implementation of interventions to OPtimise Health Literacy and Access (Ophelia)

BackgroundThe need for healthcare strengthening to enhance equity is critical, requiring systematic approaches that focus on those experiencing lesser access and outcomes. This project developed and tested the Ophelia (OPtimising HEalth LIteracy and Access) approach for co-design of interventions to improve health literacy and equity of access. Eight principles guided this development: Outcomes focused; Equity driven, Needs diagnosis, Co-design, Driven by local wisdom, Sustainable, Responsive and Systematically applied. We report the application of the Ophelia process where proof-of-concept was defined as successful application of the principles.MethodsNine sites were briefed on the aims of the project around health literacy, co-design and quality improvement. The sites were rural/metropolitan, small/large hospitals, community health centres or municipalities. Each site identified their own priorities for improvement; collected health literacy data using the Health Literacy Questionnaire (HLQ) within the identified priority groups; engaged staff in co-design workshops to generate ideas for improvement; developed program-logic models; and implemented their projects using Plan-Do-Study-Act (PDSA) cycles. Evaluation included assessment of impacts on organisations, practitioners and service users, and whether the principles were applied.ResultsSites undertook co-design workshops involving discussion of service user needs informed by HLQ (n = 813) and interview data. Sites generated between 21 and 78 intervention ideas and then planned their selected interventions through program-logic models. Sites successfully implemented interventions and refined them progressively with PDSA cycles. Interventions generally involved one of four pathways: development of clinician skills and resources for health literacy, engagement of community volunteers to disseminate health promotion messages, direct impact on consumers’ health literacy, and redesign of existing services. Evidence of application of the principles was found in all sites.ConclusionsThe Ophelia approach guided identification of health literacy issues at each participating site and the development and implementation of locally appropriate solutions. The eight principles provided a framework that allowed flexible application of the Ophelia approach and generation of a diverse set of interventions. Changes were observed at organisational, staff, and community member levels. The Ophelia approach can be used to generate health service improvements that enhance health outcomes and address inequity of access to healthcare.

Using Theory in Criminal Justice Evaluation

This chapter examines the nature and role of theory in criminal justice evaluation. A distinction between theories of and theories for evaluation is offered to clarify what is meant by ‘theory’ in the context of contemporary evaluation practice. Theories of evaluation provide a set of prescriptions and principles that can be used to guide the design, conduct and use of evaluation. Theories for evaluation include programme theory and the application of social science theory to understand how and why criminal justice interventions work to generate desired outcomes. The fundamental features of these three types of theory are discussed in detail, with a particular focus on demonstrating their combined value and utility for informing and improving the practice of criminal justice evaluation.

Problems of Implementing Offender Programs in the Community

ABSTRACT Rehabilitation of offenders is, at present, an important focus among many correctional departments. A substantial body of international research literature now exists to guide the design and development of new programs that aim to reduce re-offending. However, successful implementation of these programs has been challenging for many correctional authorities. Drawing on the experience of a community correctional agency in Australia, this paper identifies and examines a number of barriers to successful delivery of community-based offender rehabilitation programs and services. The findings suggest that basing interventions on scientific knowledge about “what works” in offender rehabilitation is necessary but not sufficient for effective programs and services. More careful attention needs to be paid to how correctional authorities can take this research and implement it in practice.

Reframing how evaluators think and act: New insights from Ernest House:

This essay examines some of the core concerns of House’s approach to evaluation, with specific reference to his latest book Evaluating: Values, Biases and Practical Wisdom. This is a difficult book to characterise in a few pithy statements. Part memoir, part evaluation novella and part scholarly textbook. Arguably, this is one of the most unique discourses on evaluation in the past decade. Anyone who has conducted evaluations knows there are always myriad choices. Determining what actions to take and who gets to decide is never easy. House provides a rich repository of wise counsel. After reading this book evaluators will find themselves pausing longer to think and reflect on everyday decisions. I conclude by offering some suggestions about areas where House’s work could benefit from further development.

From Evaluation Theory to Tests of Evaluation Theory

Compared to other fields of inquiry, evaluation is a relatively young discipline. Several methods for conducting evaluation, particularly in education and curriculum assessment, were advanced by scholars such as Ralph Tyler in the 1930s and 1940s. However, it was not until the 1960s that evaluation experienced a significant boom. Under the Kennedy and Johnson administrations, social scientists from a variety of disciplinary backgrounds were called upon to conduct evaluations of large-scale reforms designed to ameliorate the USA’s most pressing social problems. Few theoretical writings on evaluation were available to guide practice. Ernie House recalls that when he first entered the field of evaluation he ‘tossed all the papers I could find about the topic into a small cardboard box in the corner of my office and read them in one month’ (House, 1990, p. 24). With necessity being the mother of all invention, evaluators set to work on developing new ways of conceptualizing evaluation.1