Karol Olejniczak

Evaluation units as knowledge brokers: Testing and calibrating an innovative framework

Evaluation units, located within public institutions, are important actors responsible for the production and dissemination of evaluative knowledge in complex programming and institutional settings. The current evaluation literature does not adequately explain their role in fostering better evaluation use. The article offers an empirically tested framework for the analysis of the role of evaluation units as knowledge brokers. It is based on a systematic, interdisciplinary literature review and empirical research on evaluation units in Poland within the context of the European Union Cohesion Policy, with complementary evidence from the US federal government and international organizations. In the proposed framework, evaluation units are to perform six types of brokering activities: identifying knowledge users’ needs, acquiring credible knowledge, feeding it to users, building networks between producers and users, accumulating knowledge over time and promoting an evidence-based culture. This framework transforms evaluation units from mere buyers of expertise and producers of isolated reports into animators of reflexive social learning that steer streams of knowledge to decision makers.

Mechanisms Shaping an Evaluation System—A Case Study of Poland 1999–2010

A criticism of evaluation practice in regional policy in Central and Eastern Europe (CEE) countries is that it stresses its function in terms of accountability and formal reporting on progress, rather than its role in policy or institutional learning. This article aims to test this critique through the use of ‘system thinking’ and focuses on the case of regional policy in Poland, where evaluation practice has developed dynamically since 2000. It contributes to the literature by developing an analytical framework that sets out the role of evaluation in regional policy learning and identifies key factors and mechanisms that determine the development of an effective evaluation system for regional policy. The analysis draws on all Polish evaluation studies conducted between 1999 and 2010, interviews with civil servants and experts involved in the implementation of cohesion policy programmes, as well as secondary data from earlier studies. The essay identifies key mechanisms and factors that determine the main function of evaluation as a learning tool that produces and utilises knowledge in the decision-making process. These main elements are: growth in funding of the policy; stability of institutions; motivators; and system architecture.

The Role of Analysts in Public Agencies: Toward an Empirically Grounded Typology

This chapter focuses on the analytical capabilities of individuals in central government public organizations. Drawing upon the results of a large study undertaken for the Polish government that aimed to identify and improve the analytical capacity of central government personnel across all policy domains, the chapter provides a definition and a multi-dimensional typology of analysts in government. The typology is accompanied by a practical toolbox for identifying and grouping analysts, as tested in the Polish government.

The Game of Knowledge Brokering: A New Method for Increasing Evaluation Use.

Knowledge brokering is a promising practice for addressing the challenge of using research evidence, including evaluation findings, in policy implementation. For public policy practitioners, it means playing the role of an intermediary who steers the flow of knowledge between producers (researchers) and users (decision makers). It requires a set of specific skills that can be learnt most effectively by experience. The article explains how to develop knowledge brokering skills through experiential learning in a risk-free environment. It reports on the application of an innovative learning method—a game-based workshop. The article introduces the conceptual framework for designing game-based learning. Then, it demonstrates how this framework was applied in practice of teaching knowledge brokering. In conclusion, article discusses the first lessons of game application in training public policy practitioners. It concludes that game-based workshop is a promising method for learning about knowledge brokering strategies that increase evaluation use.

Regulation crash-test: applying serious games to policy design

Abstract Successful policy solutions rely on policy addressees responding in certain ways. Policy designers need an analytical method that allows them to anticipate impact of a new intervention, while taking into account bounded rationality of policy actors and sociopolitical complexity. The article proposes using serious games at the stage of policy formulation to test the architecture of a new regulation in a safe environment. It provides a blueprint for using games in policy design, consisting of conceptual framework, design procedure, and techniques for strengthening game validity. The application is illustrated with an example of a draft regulation on rural transport in Poland. The case study points out three advantages of game method: (1) revealing mechanisms triggered by the architecture of regulation, meaning actors’ initial assumptions, decisions, and feedback loops created by actors’ responses, (2) demonstrating the consequences of mechanisms over time, that in real life would occur with a long delay, and (3) creating a risk-free environment where policy actors can verify their assumptions and experiment with ways of interpreting and responding to new regulation. The article concludes that serious games are a promising method for anticipating impact of complex policy regulation.

Knowledge Brokers in Action: A Game-Based Approach for Strengthening Evidence-Based Policies

Public policies need research results in order to effectively address the complex socioeconomic challenges (so-called evidence-based policies). However there is a clear gap between producing scientific expertise and using it in public decision-making. This “know-do” gap is common in all policy areas. Knowledge brokering is a new and promising practice for tackling the challenge of evidence use. It means that selected civil servants play the role of intermediaries who steer the flow of knowledge between its producers (experts and researchers) and users (decision-makers and public managers). Knowledge brokering requires a specific combination of skills that can be learned effectively only by experience. However this is very challenging in the public sector. Experiential learning requires learning from own actions – often own mistakes, while public institutions tend to avoid risk and are naturally concerned with the costs of potential errors. Therefore, a special approach is required to teach civil servants.

Towards Behaviorally Informed Public Interventions

Abstract Purpose: This article informs readers about the theoretical and practical origins of the behaviorally informed interventions (BIPI), analyzes examples of the BIPI from different policy sectors and strategies they offer for policy and regulatory design, and discusses applications and implications of BIPI for public interventions Methodology: This paper is based on a review of literature, as well as an inspection of administrative practices in OECD countries. It encompasses a systematic analysis of scientific papers from the SCOPUS database and a query carried out at the library of George Washington University. Findings: The traditional approach to public policy research is based on rational choice theory. It offers limited support, because by assuming perfect rationality of policy decisions, it overlooks existence of systematic errors and biases of human decision-making. The authors argue that behaviorally informed public interventions (BIPI) might contribute to improving the effectiveness of a number of public measures - regulation, projects, programs, and even entire policies. Practical implications: The behavioral approach allows decision-makers to better understand the decisions and behaviors of citizens, as well as to design more effective interventions with minimum effort by adapting the existing solutions to real decision mechanisms of citizens. Originality: By combining the concepts of traditional approach with the growing behavioral approach, the authors aim to propose a new theoretical framework (BIPI) to be used as a tool for policy design, delivery and evaluation