Claire Donovan

State of the art in assessing research impact: introduction to a special issue

Papers in this special issue were developed at an international workshop on ‘State of the Art in Assessing Research Impact’, hosted by the Health Economics Research Group at Brunei University. The workshop debated what constitutes state-of-the-art methods for assessing the ‘impact’ (or broader societal returns) of research. Metrics-only approaches employing economic data and science, technology and innovation indicators were found to be behind the times: best practice combines narratives with relevant qualitative and quantitative indicators to gauge broader social, environmental, cultural and economic public value. Limited consultation between policy-makers and the research evaluation community has led to a lack of policy-learning from international developments. Little engagement between research evaluation specialists and the academic community has cast ‘impact’ as the height of philistinism: yet ‘impact’ is a strong weapon for making an evidence-based case to governments and research funders for enhanced financial support, and ‘the state of the art’ is suited to the characteristics of all research fields (including the humanities, creative arts and social sciences) in their own terms. Copyright , Beech Tree Publishing.

The qualitative future of research evaluation

Science, technology and innovation (STI) policy aimed at technological advance, international competitiveness and wealth creation underpins the regulation of publicly funded research. Familiar quantitative evaluative ‘metrics’ fit snugly with these economic objectives. A re-imagined STI policy embraces wider intellectual, social, cultural, environmental and economic returns, using qualitative measures and processes to capture research outcomes. Copyright , Beech Tree Publishing.

The ‘Payback Framework’ explained

The Payback Framework, originally developed to examine the ‘impact’ or ‘payback’ of health services research, is explained. The Payback Framework is a research tool used to facilitate data collection and cross-case analysis by providing a common structure and so ensuring cognate information is recorded. It consists of a logic model representation of the complete research process, and a series of categories to classify the individual paybacks from research. Its multi-dimensional categorisation of benefits from research starts with more traditional academic benefits of knowledge production and research capacity-building, and then extends to wider benefits to society. Copyright , Beech Tree Publishing.

Evaluation of the impact of National Breast Cancer Foundation-funded research

Objective: To evaluate the impact of the National Breast Cancer Foundations (NBCFs) research investment.

Introduction: Future pathways for science policy and research assessment: metrics vs peer review, quality vs impact

The idea for this special issue arose from observing contrary developments in the design of national research assessment schemes in the UK and Australia during 2006 and 2007. Alternative pathways were being forged, determined, on the one hand, by the perceived relative merits of ‘metrics’ (quantitative measures of research performance) and peer judgement and, on the other hand, by the value attached to scientific excellence (‘quality’) versus usefulness (‘impact’). This special issue presents a broad range of provocative academic opinion on preferred future pathways for science policy and research assessment. It unpacks the apparent dichotomies of metrics vs peer review and quality vs impact, and considers the hazards of adopting research evaluation policies in isolation from wider developments in scientometrics (the science of research evaluation) and divorced from the practical experience of other nations (policy learning). Copyright , Beech Tree Publishing.

Testing novel quantitative indicators of research ‘quality’, esteem and ‘user engagement’: an economics pilot study

Applying ‘standard’ publication and citation measures to the social sciences is fast becoming an outmoded practice, yet we have still to develop credible quantitative alternatives to inform research evaluation exercises. This paper reports the outcomes of a comparative pilot study of five Australian economics departments which tested data produced using novel bibliometric, esteem, and ‘user engagement’ measures. The results were presented to a group of expert peers drawn from the economics groups studied. There was support for developing some novel bibliometric indicators and ‘user engagement’ measures, but esteem indicators were roundly rejected. Copyright , Beech Tree Publishing.

Gradgrinding the Social Sciences: The Politics of Metrics of Political Science

This article employs an interpretive approach, and in the light of contributions to this symposium by Butler and McAllister, and McLean et al., holds that metrics of research ‘quality’ are socially constructed and hence are as ‘subjective’ as peer review. Thus it rejects the use of stand-alone metrics as an ‘objective’ basis to inform funding allocations. Rather, the optimum method of ‘quality’ assessment is a panel-based exercise with expert judgement informed by a range of discipline-sensitive metrics and peer review of publications. The article maintains that the politics of metrics of political science conceals interests about the foundations of social scientific knowledge, and so the dispute over metrics and peer review is a metaphor for the conflicting epistemological preferences of UK political scientists. It is also argued that metrics-led assessment subjects political science to ‘Gradgrinding’ on two fronts: that political science departments amount to less than the sum of their parts, and the audit culture strips the discipline of its humanism.

Tracing the wider impacts of biomedical research: a literature search to develop a novel citation categorisation technique

There is an increasing need both to understand the translation of biomedical research into improved healthcare and to assess the range of wider impacts from health research such as improved health policies, health practices and healthcare. Conducting such assessments is complex and new methods are being sought. Our new approach involves several steps. First, we developed a qualitative citation analysis technique to apply to biomedical research in order to assess the contribution that individual papers made to further research. Second, using this method, we then proposed to trace the citations to the original research through a series of generations of citing papers. Third, we aimed eventually to assess the wider impacts of the various generations. This article describes our comprehensive literature search to inform the new technique. We searched various databases, specific bibliometrics journals and the bibliographies of key papers. After excluding irrelevant papers we reviewed those remaining for either general or specific details that could inform development of our new technique. Various characteristics of citations were identified that had been found to predict their importance to the citing paper including the citation’s location; number of citation occasions and whether the author(s) of the cited paper were named within the citing paper. We combined these objective characteristics with subjective approaches also identified from the literature search to develop a citation categorisation technique that would allow us to achieve the first of the steps above, i.e., being able routinely to assess the contribution that individual papers make to further research.

The Hidden Perils of Citation Counting for Australasian Political Science

In a recent article in Australian Journal of Political Science, Dale and Goldfinch present ‘standard’ journal-based publication and citation rankings of Australasian political science departments designed to complement what they characterise as the multidisciplinary, historical, qualitative and humanistic political science of the region. However, the ‘highly cited’ articles in their top-ranked political science department belong to quantitative psychology. Through unravelling why their study favours the opposite of that which it was meant to detect, this paper alerts political scientists to the hidden perils of accepting ‘standard’ Institute of Scientific Information-based approaches to citation counting as valid measures of research ‘quality’. It exposes the veiled bibliometric assumption that the ‘best’ social science is quantitative research, notes that incongruous citation scores may inform the distribution of block funding and departmental appointment processes, and warns against using ‘standard’ data to unintentionally self-police the future shape of Australasian political science.

Beyond the ‘Postmodern University’

As an institution, the “postmodern university” is central to the canon of todays research on higher education policy. Yet in this essay I argue that the postmodern university is a fiction that frames and inhibits our thinking about the future university. To understand why the postmodern university is a fiction, I first turn to grand theory and ask whether we can make sense of the notion of “post”-postmodernity. Second, I turn to the UK higher education sector and show that the postmodern university is a chimera, a modern artefact of competing instrumentalist, gothic, and postmodernist discourses. Third, I discuss competing visions of the future university and find that the progressive (yet modernist) agendas that re-imagine the public value of knowledge production, transmission, and contestation, are those that can move us beyond the palliative and panacea of the postmodern university.