John Irvine

Assessing basic research : Some partial indicators of scientific progress in radio astronomy

As the costs of certain types of scientific research have escalated, and as growth rates in overall national science budgets have declined, so the need for an explicit science policy has grown more urgent. In order to establish priorities between research groups competing for scarce funds, one of the most important pieces of information needed by science policymakers is an assessment of those groups’ recent scientific performance. This paper suggests a method for evaluating that performance. After reviewing the literature on scientific assessment, we argue that, while there are no simple measures of the contributions to scientific knowledge made by scientists, there are a number of ‘partial indicators’ that is, variables determined partly by the magnitude of the particular contributions, and partly by ‘other factors’. If the partial indicators are to yield reliable results, then the influence of these ‘other factors’ must be minimised. This is the aim of the method of ‘converging partial indicators’ proposed in this paper. We argue that the method overcomes many of the problems encountered in previous work on scientific assessment by incorporating the following elements: (1) the indicators are applied to research groups rather than individual scientists; (2) the indicators based on citations are seen as reflecting the impact, rather than the quality or importance, of the research work; (3) a range of indicators are employed, each of which focuses on different aspects of a group’s performance; (4) the indicators are applied to matched groups, comparing ‘like’ with ‘like’ as far as

Bibliometric profiles for British academic institutions: An experiment to develop research output indicators

In this paper, we report the results of an exploratory study commissioned by the Advisory Board for the Research Councils to produce bibliometric research profiles for academic and related institutions within the UK. The approach adopted is based on the methodology developed by CHI Research whereby publications from a given institution are weighted according to the influence of the journal in which they appear. Although certain technical limitations were encountered with the approach, the study nonetheless yielded potentially useful information on the comparative research output of British universities and polytechnics.

Basic Research in the East and West: A Comparison of the Scientific Performance of High-Energy Physics Accelerators

This paper presents the results of a study comparing the past scientific performance of high-energy physics accelerators in the Eastern bloc with that of their main Western counterparts. Output-evaluation indicators are used. After carefully examining the extent to which the output indicators used may be biased against science in the Eastern bloc, various conclusions are drawn about the relative contributions to science made by these accelerators. Where significant differences in performance are apparent, an attempt is made to identify the main factors responsible.

CERN: Past performance and future prospects: II. The scientific performance of the CERN accelerators

Abstract In a series of three papers, we attempt to evaluate the past scientific performance of the three main particle accelerators at the Geneva-based European Organization for Nuclear Research (CERN) over the period since 1960, and to assess the future prospects for CERN and its users during the next ten to fifteen years. We concerned ourselves in a previous paper (Paper 1 - Martin and Irvine [51] with the position of the CERN accelerators in world high-energy physics relative to those at other large laboratories working in the field. We dealt primarily with the period from 1969 to 1978, and attempted to establish how the experimental output from the three principal CERN accelerators, taken as a whole, compared with that from other major facilities. In undertaking this comparative evaluation, we drew on the method of “converging partial indicators” used in previous studies of three Big Science specialties. In contrast, this paper (Paper II) focuses in detail on the scientific performance of each of the CERN accelerators taken individually. In particular, it asks, first, how the outputs from the CERN 28 GeV (giga or billion electron-volts) Proton Synchrotron compare with those from a very similar 33 GeV American accelerator at Brookhaven National Laboratory over the past two decades? Second, how great have been the experimental achievements of the Intersecting Storage Rings in world terms? And, third, how do the outputs from the CERN 400 GeV Super Proton Synchrotron and from a rival American machine at Fermi National Accelerator Laboratory compare? Attempts are then made to identify the main factors responsible for determining the relative scientific performance of each CERN machine. These factors are of relevance to the subject of a third paper (Paper III - Martin and Irvine [52]) which sets out to assess the future prospects for CERN and in particular for LEP, the large electron-positron collider scheduled for completion in the latter part of the 1980s. What are the construction requirements (financial and technical) associated with LEP, and how easily will they be met? How does the scientific potential of LEP compare with that of other major accelerators under construction around the world? And, in the light of the previous record of the CERN accelerators, to what extent is this potential likely to be realized? The paper concludes with a discussion of the extent to which predictive techniques can be utilized in the formulation of scientific priorities, and of the problems in current science policy-making that such techniques might help address.

CERN: Past performance and future prospects: I. CERN's position in world high-energy physics

Abstract In a series of three papers, we attempt to evaluate the past scientific performance of the three main particle accelerators at the Geneva-based European Organization for Nuclear Research (CERN) over the period since 1960 and to assess the future prospects for CERN and its users over the next ten to fifteen years. We concern ourselves in this paper (paper I) with the position of the CERN accelerators in world high-energy physics relative to those at other large laboratories working in the field. We deal primarily with the period from 1969 to 1978, and attempt to establish how the experimental output from the three principal CERN accelerators, taken as a whole , compares with that from other major facilities. In undertaking this comparative evaluation, we draw on the method of “converging partial indicators” used in previous studies of three Big Science specialities. In contrast, the second paper (paper II Irvine amd Martin [12]) focuses in detail on the scientific performance of each of the CERN accelerators taken individually . In particular, it asks, first, how the outputs from the CERN 28 GeV (giga or billion electron-volts) Proton Synchrotron compare with those from a very similar 33 GeV American accelerator at Brookhaven National Laboratory over the past two decades. Second, how great have been the experimental achievements of the CERN Intersecting Storage Rings in world terms? And, third, how do the outputs from the CERN 400 GeV Super Proton Synchrotron and from a rival US machine at Fermi National Accelerator Laboratory compare? Attempts are then made to identify the main factors responsible for determining the relative scientific performance of each CERN machine. These factors are of relevance to the subject of a third paper (paper III - Martin and Irvine [20]) which sets out to assess the future prospects for CERN and in particular for LEP, the large electron positron collider scheduled for completion in the latter half of the 1980s. What are the construction requirements (financial and technical) associated with LEP, and how easily will they he met? How does the scientific potential of LEP compare with that of other major accelerators under construction around the world? And, in the light of the previous record of the CERN accelerators, to what extent is this potential likely to be realized? The paper concludes with a discussion of the extent to which predictive techniques can be utilized in the formulation of scientific priorities, and of the problems in current science policy-making that such techniques might help address.

Evaluating big science: CERN's past performance and future prospects

After explaining the reasons why science policy-makers face a growing need for more rigorous forms of research evaluation, we outline an approach combining bibliometric and peer-evaluation data that has been developed at the Science Policy Research Unit in the course of a programme of studies of Big Science specialties. The paper describes the results obtained when this ‘method of converging partial indicators’ is applied to compared the past research performance of the accelerators at CERN — the joint European Laboratory for Particle Physics — with that of the worlds other main accelerators. The paper concludes by demonstrating how, on the basis of an analysis of the factors that have structured research performance in the past, it is possible to arrive at a systematic set of conclusions about the future prospects for a major new research facility such as an accelerator.

Bibliometric analysis for Science Policy : an evaluation of the United Kingdom's research performance in ocean currents and protein crystallography

This paper presents the results of a study of Britains scientific performance in the fields of ocean currents and protein crystallography carried out for the Advisory Board for the Research Councils (ABRC). Using a range of publication and citation indicators, the study aimed to explore the potential value to science policy-making of low-cost scientometric approaches to research evaluation.

CERN: Past Performance and Future Prospects - III - CERN and the Future of World High-Energy Physics

In a series of three papers, we attempt to evaluate the past scientific performance of the three main particle accelerators at the Geneva-based European Organization for Nuclear Research (CERN) over the period since 1960, and to assess the future prospects for CERN and its users during the next ten to fifteen years. We concerned ourselves in the first paper (Paper I-Martin and Irvine [29]) with the position of the CERN accelerators in world high-energy physics relative to those at other large laboratories working in the field. We dealt primarily with the period from 1969 to 1978, and attempted to establish how the experimental output from the three principal CERN accelerators, taken as a whole, compared with that from other major facilities. In undertaking this comparative evaluation, we drew on the method of “converging partial indicators” used in previous studies of three Big Science specialties. In contrast, the second paper (Paper II - Irvine and Martin [24]) focused in detail on the scientific performance of each of the CERN accelerators taken individually. In particular, it asked, first, how the outputs from the CERN 28 GeV (giga or billion electron-volts) Proton Synchrotron compare with those from a very similar 33 GeV American accelerator at Brookhaven National Laboratory over the past two decades. Second, how great have been the experimental achievements of the Intersecting Storage Rings in world terms? And, third, how do the outputs from the CERN 400 GeV Super Proton Synchrotron and from a rival US machine at Fermi National Accelerator Laboratory compare? Attempts were then made to identify the main factors responsible for determining the relative scientific performance of each CERN machine. These factors are of relevance to the subject of this third paper (Paper III), which sets out to assess the future prospects for CERN and in particular for LEP, the large electron-positron collider scheduled for completion in the latter part of 1988. What are the construction requirements (financial and technical) associated with LEP, and how easily will they be met? How does the scientific potential of LEP compare with that of other major accelerators under construction or planned around the world? In the light of the previous record of the CERN accelerators, to what extent is this scientific potential likely to be realized? What spin-off is there likely to be from LEP to accelerator physics in general? Finally, how “flexible” is LEP -in other words, what is its potential for future development? The paper concludes with a discussion of the extent to which predictive techniques can be utilized in the formulation of scientific priorities, and of the problems in current science policy-making that such techniques might help address.

Assessing basic research: Reappraisal and update of an evaluation of four radio astronomy observatories

This paper critically reassesses the results of a previous evaluation of the research performance of four radio astronomy observatories over the period 1969?1978, completed by two of the authors in 1980. At the same time, the evaluation is updated by presenting bibliometric (publication and citation) data for the subsequent five-year period 1979?1983. While application of a more developed evaluation methodology is shown not to alter significantly the conclusions arrived at in the earlier study, there is evidence that the relative research performance of the four observatories has changed significantly since 1978.

Changing ways of life: Costs, benefits and strategies

Abstract The authors set out an approach to evaluating ways of life, and discuss different types of explanations which are used to account for the central problems that have arisen. Particular explanations are shown to be crucially linked to different strategies for achieving (or restricting) change. They then outline implications for policy and forecasting of this clash of theories and practices, arguing that such questions deserve greater consideration than they have achieved in the past.