David Roessner

Evaluating technology programs: tools and methods

Abstract This article reviews the analytical tools, methods, and designs being used to evaluate public programs intended to stimulate technological advance. The review is organized around broad policy categories rather than particular types of policy intervention, because methods used are rarely chosen independent of context. The categories addressed include publicly-supported research conducted in universities and public sector research organizations; evaluations of linkages, especially those programs seeking to promote academic-industrial and public-private partnerships; and the evaluation of diffusion and industrial extension programs. The internal evaluation procedures of science such as peer review and bibliometrics are not covered, nor are methods used to projects and individuals ex ante. Among the conclusion is the observation that evaluation work has had less of an impact in the literature that it deserves, in part because much of the most detailed and valuable work is not easily obtainable. A second conclusion is that program evaluations and performance reviews, which have distinctive objectives, measures, and tools, are becoming entangled, with the lines between the two becoming blurred. Finally, new approaches to measuring the payoffs from research that focus on linkages between knowledge producers and users, and on the characteristics of research networks, appear promising as the limitations of the production function and related methods have become apparent.

How interdisciplinary is a given body of research

This article presents results to date produced by a team charged with evaluating the National Academies Keck Futures Initiative, a 15-year US

Quantitative and qualitative methods and measures in the evaluation of research

40 million program to facilitate interdisciplinary research in the United States. The team has developed and tested promising quantitative measures of the integration (I) and specialization (S) of research outputs, the former essential to evaluating the impact of the program. Both measures are based on Thomson-ISI Web of Knowledge subject categories. ‘I’ measures the cognitive distance (dispersion) among the subject categories of journals cited in a body of research. ‘S’ measures the spread of subject categories in which a body of research is published. Pilot results for samples from researchers drawn from 22 diverse subject categories show what appears to be a surprisingly high level of interdisciplinarity. Correlations between integration and the degree of co-authorship of selected bodies of research show a low degree of association. Copyright , Beech Tree Publishing.

Validating indicators of interdisciplinarity: linking bibliometric measures to studies of engineering research labs

This paper proposes that the choice of quantitative versus qualitative measures in research evaluation is a false one, especially for evaluators isolated from the real world. This choice, sometimes substantially client-driven, should be tempered by professional judgment. It is sometimes easier to develop quantitative ‘indicators’ of performance than to work out what the program has to accomplish. As a result legislators and others increasingly ask public agencies for quantitative measures of research performance, and in so doing generate all kinds of mischief. Unfortunately, the fallacy of misplaced concreteness in research evaluation is still alive if not necessarily well. Copyright , Beech Tree Publishing.

Practical research proposal and publication profiling

This article examines the extent to which specific features of interdisciplinary research are accurately reflected in selected bibliometric measures of scholarly publications over time. To test the validity of these measures, we compare knowledge of research processes and impact based on ethnographic studies of a well-established researcher’s laboratory, together with personal interview data, against bibliometric indicators of cognitive integration, diffusion, and impact represented in the entire portfolio of papers produced by this researcher over time.

Politics and the political setting as an influence on evaluation activities: National research and technology policy programs in the United States and Canada

We present means to enable profiling to situate research activities (such as proposals or publications) within research domains, identify cognitive and social networks, and track knowledge transfers. Here we profile a set of 123 National Science Foundation educational research awards. We investigate NSF award summaries, proposal references, publications by the funded researchers, and citations to those publications. Our integration measure and science overlay maps show that these ROLE/REESE programs have facilitated interdisciplinary research. NSF program managers were able to associate highly cited researchers into key contributing communities using the co-citation maps. We explore ‘before vs. after’ research patterns to examine whether this research support altered teaming and cross-disciplinary exchanges. Evaluators can use research profiling to understand the role of particular research endeavors and to study community evolution. Copyright , Beech Tree Publishing.

Use of Quantitative Methods to Support Research Decisions in Business and Government

Program evaluation activities in the United States, after a long history of decentralized, uncoordinated activity, have taken a new turn. It is called performance measurement. A portion of this history, and the recent changes, may be explained by characteristics of the political environment. In contrast, Canada has a long history of centralized, coordinated evaluation of its federal programs. In this paper, we identify particular attributes of the Canadian and the, U.S. political systems that we posit are related to each nations respective evaluation system. Specifically, we address the following factors differentiating the evaluation experience in both countries as being a function of (1) the level of centralization; (2) legislative history; (3) legislative precedent; and (4) political support for evaluation. Using case studies of research and development program evaluations in both countries, we examine the forces at work that shape the design and implementation of evaluation programs. The paper concludes with a discussion of the implications of recent changes in both systems for their respective evaluation systems and processes.

Comment on Tarmo Lemola’s Article

Modern industrial nations expend about 2.3% of their gross national products for research and development. In the U.S., concern over the nation’s economic performance has been linked to the size and composition of expenditures for research. At issue is not whether the U.S. should make such expenditures, but how large they should be; how much should be devoted to basic research, applied research, development, and to other phases of knowledge generation and application; how research resources such as money and manpower should be allocated among scientific fields, industries, and areas of national concern (defense, energy, health); and, more broadly, what the relative roles of government and the private sector should be in supporting the national research effort.

The economic impact of licensed commercialized inventions originating in university research

Max Jakobson, in the beginning of his little book Finland: Myth and Reality, notes that Finland pays a price for her policy of neutrality: the rest of the world has little incentive to follow Finnish affairs; their knowledge of the country tends to be superficial and fragmentary. “As a result, Finland is forever at the mercy of the itinerant columnist who after lunch and cocktails in Helsinki is ready to pronounce himself upon the fate of the Finnish people” (p. 8).