Gretchen Jordan

Logic models: a tool for telling your programs performance story

Abstract Program managers across private and public sectors are being asked to describe and evaluate their programs in new ways. People want managers to present a logical argument for how and why the program is addressing a specific customer need and how measurement and evaluation will assess and improve program effectiveness. Managers do not have clear and logically consistent methods to help them with this task. This paper describes a Logic Model process, a tool used by program evaluators, in enough detail that managers can use it to develop and tell the performance story for their program. The Logic Model describes the logical linkages among program resources, activities, outputs, customers reached, and short, intermediate and longer term outcomes. Once this model of expected performance is produced, critical measurement areas can be identified.

New directions in the use of network analysis in research and product development evaluation

In recent years, the use of social network analysis (SNA) has received increased attention in R&D evaluation. While SNA provides insights into communication and knowledge flows, its efficacy in evaluation methodology remains unclear. As Rogers et al (2001) discuss, the applicability of SNA in the evaluation of R&D is marked by several shortcomings, such as the weakness of understanding the content of ties and the inability to identify a generalizable concept of network effectiveness. This paper explores these issues through a discussion of two recent studies of social networks in R&D and concludes with an assessment of the results of these studies using the challenges outlined by Rogers et al. Copyright , Beech Tree Publishing.

A theory-based logic model for innovation policy and evaluation

Current policy and program rationale, objectives, and evaluation use a fragmented picture of the innovation process. This presents a challenge since in the United States officials in both the executive and legislative branches of government see innovation, whether that be new products or processes or business models, as the solution to many of the problems the country faces. The logic model is a popular tool for developing and describing the rationale for a policy or program and its context. This article sets out to describe generic logic models of both the R&D process and the diffusion process, building on existing theory-based frameworks. Then a combined, theory-based logic model for the innovation process is presented. Examples of the elements of the logic, each a possible leverage point or intervention, are provided, along with a discussion of how this comprehensive but simple model might be useful for both evaluation and policy development. Copyright , Beech Tree Publishing.

A theory-based innovation systems framework for evaluating diverse portfolios of research, part two: Macro indicators and policy interventions

This framework for multi-level evaluation of scientific research is a bridge between social science theory and the provision of effective feedbacks to governments so they can overcome systemic blockages to innovation and successful outcomes of research policy. Starting with the idea of innovation network theory and organizational theory involved in the research environment survey, a small set of indicators is suggested at micro, meso, and macro levels. Data from this integrated set of indicators can identify the blockages and suggest corrections. This paper concentrates on the macro-level indicators. Three familiar kinds of government policy lever — capital, capabilities, and coordination modes — are discussed. However, the discussion of ways in which these interventions can correct blockages is far more complex than has previously been acknowledged in the evaluation literature. The proposed framework is an important step for evaluators and policy-makers to develop research, technology and development investment portfolios and strategies more effectively. Copyright , Beech Tree Publishing.

Impact Evaluation Framework For Technology Deployment Programs

An approach for quantifying retrospective energy savings, clean energy advances, and market effects.

Using systems theory and logic models to define integrated outcomes and performance measures in multi-program settings

This paper describes the application of diffusion of innovations theory (Rogers, 2003) to four domains representing the typical partners and target audiences of deployment programs in order to systematically define program outcomes. Diffusion of innovations is a broad systems theory that describes how technologies and practices are diffused and adopted. The theory is integrated with logical modeling to provide a powerful new tool for defining outcomes. The domains, the theory, the use of the theory to develop researchable issues and measures, and examples of its application to an energy-efficiency agency are provided. While this paper focuses on an energy-efficiency agency, with only slight modification the approach can apply to other types of deployment agencies such as education, human service, health, or agriculture. Copyright , Beech Tree Publishing.

Measuring radical innovation in real time

As Leifer et al. have argued, the management of radical innovation projects necessitates different strategies and practices, including different ways of measuring performance. This paper discusses our efforts to explore indicators of progress for radical innovation in basic and applied research and technology development at both the project and programme levels. The paper provides examples of our work with scientific research projects in two research organisations to develop real-time indicators of the research progress. Finally, it is argued that the continued development of similar indicators can play an important role in managing research to achieve radical innovation.

Measuring the performance of research and technology programs: A balanced scorecard approach

Research programs, like other government programs, are now being requested to demonstrate relevance and value added to national social and economic needs. Complexity, unpredictability and other factors make it difficult to define specific performance measures for R&D programs. This paper describes the performance measurement efforts of one technology development program within the U.S. Department of Energy and proposes a strategy for applying this balanced scorecard approach to a fundamental research organization. Simple logical models of the inputs, activities, outcomes and long term results of R&D programs are proposed. A critical few measures of performance that answer questions from multiple audiences are then chosen across this performance spectrum.

Estimating the benefits of government-sponsored energy R&D

A National Research Council (NRC) committee recently conducted a retrospective study of the benefits of some of the energy efficiency and fossil energy programs in the US Department of Energy (US DOE). A subsequent conference discussed ways of adapting and refining the NRC framework for possible use by US DOE offices to help plan and manage their R&D. A framework that emerged from the conference is depicted by a matrix that categorizes different types of benefits. The rows of the matrix reflect the goals of the department, and the columns reflect the time-frame and level of certainty. The conference suggested many approaches for estimating the benefits within each category and also noted many challenges in making these estimates. Copyright , Beech Tree Publishing.

R&D Integration: How to Build a Diverse and Integrated Knowledge Community

Stokes advocated the benefits of uniting basic and applied research as a way to facilitate research breakthroughs. Recently, the U.S. Department of Energy launched an initiative designed to foster better integration in research and technology development (R&D), such as the concurrent application of scientific and engineering knowledge. This paper suggests that in basic and applied research- two arenas in the production of knowledge- there are difficulties in integrating them because of two somewhat disparate barriers: (1) cognitive distance among the researchers and (2) structural differentiation in the idea innovation network. This paper discusses these two barriers in greater depth and explains why these barriers are increasing. The larger issue is to build a diverse and integrated knowledge community via the following kinds of mechanisms: complex charters, visionary team leadership, recruitment from diverse sources, multiple team and network integration mechanisms, and diverse sources of funding. These ideas emerge from not only the recent literature but more critically from a case study of a transformational research organization that built an international knowledge community in biomedicine, the Institut Pasteur.