Jonathan Adams

Early citation counts correlate with accumulated impact

SummaryThe present paper addresses the objective of developing forward indicators of research performance using bibliometric information on the UK science base. Most research indicators rely primarily on historical time series relating to inputs to, activity within and outputs from the research system. Policy makers wish to be able to monitor changing research profiles in a more timely fashion, the better to determine where new investment is having the greatest effect. Initial (e.g. 12 months from publication) citation counts might be useful as a forward indicator of the long-term (e.g. 10 years from publication) quality of research publications, but - although there is literature on citation-time functions - no study to evaluate this specifically has been carried out by Thomson ISI or any other analysts. Here, I describe the outcomes of a preliminary study to explore these citation relationships, drawing on the UK National Citation Report held by Evidence Ltd under licence from Thomson ISI for OST policy use. Annual citation counts typically peak at around the third year after publication. I show that there is a statistically highly significant correlation between initial (years 1-2) and later (years 3-10) citations in six research categories across the life and physical sciences. The relationship holds over a wide range of initial citation counts. Papers that attract more than a definable but field dependent threshold of citations in the initial period after publication are usually among the top 1% (the most highly cited papers) for their field and year. Some papers may take off slowly but can later join the high impact group. It is important to recognise that the statistical relationship is applicable to groups of publications. The citation profiles of individual articles may be quite different. Nonetheless, it seems reasonable to conclude that leading indicators of research excellence could be developed. This initial study should now be extended across a wider range fields to test the initial outcomes: earlier papers suggest the model holds in economics. Additional statistical tests should be applied to explore and model the relationship between initial, later and total citation counts and thus to create a general tool for policy application.

International collaboration clusters in Africa

Recent discussion about the increase in international research collaboration suggests a comprehensive global network centred around a group of core countries and driven by generic socio-economic factors where the global system influences all national and institutional outcomes. In counterpoint, we demonstrate that the collaboration pattern for countries in Africa is far from universal. Instead, it exhibits layers of internal clusters and external links that are explained not by monotypic global influences but by regional geography and, perhaps even more strongly, by history, culture and language. Analysis of these bottom-up, subjective, human factors is required in order to provide the fuller explanation useful for policy and management purposes.

Calibrating the zoom — a test of Zitt’s hypothesis

Bibliometric indicators are widely used to compare performance between units operating in different fields of science.For cross-field comparisons, article citation rates have to be normalised to baseline values because citation practices vary between fields, in respect of timing and volume. Baseline citation values vary according to the level at which articles are aggregated (journal, sub-field, field). Consequently, the normalised citation performance of each research unit will depend on the level of aggregation, or ‘zoom’, that was used when the baselines were calculated.Here, we calculate the citation performance of UK research units for each of three levels of article-aggregation. We then compare this with the grade awarded to that unit by external peer review. We find that the correlation between average normalised citation impact and peerreviewed grade does indeed vary according to the selected level of zoom.The possibility that the level of ‘zoom’ will affect our assessment of relative impact is an important insight. The fact that more than one view and hence more than one interpretation of performance might exist would need to be taken into account in any evaluation methodology. This is likely to be a serious challenge unless a reference indicator is available and will generally require any evaluation to be carried out at multiple levels for a reflective review.

Profiling citation impact: A new methodology

A methodology for creating bibliometric impact profiles is described. The advantages of such profiles as a management tool to supplement the reporting power of traditional average impact metrics are discussed. The impact profile for the UK as a whole reveals the extent to which the median and modal UK impact values differ from and are significantly below average impact. Only one-third of UK output for 1995-2004 is above world average impact although the UK’s average world-normalised impact is 1.24.Time-categorised impact profiles are used to test hypotheses about changing impact and confirm that the increase in average UK impact is due to real improvement rather than a reduction in low impact outputs.The impact profile methodology has been applied across disciplines as well as years and is shown to work well in all subject categories. It reveals substantial variations in performance between disciplines. The value of calculating the profile median and mode as well as the average impact are demonstrated. Finally, the methodology is applied to a specific data-set to compare the impact profile of the elite Laboratory of Molecular Biology (Cambridge) with the relevant UK average. This demonstrates an application of the methodology by identifying where the institute’s exceptional performance is located.The value of impact profiles lies in their role as an interpretive aid for non-specialists, not as a technical transformation of the data for scientometricians.

The negative effects of citing with a national orientation in terms of recognition: National and international citations in natural-sciences papers from Germany, the Netherlands, and the UK

Abstract Nations can be distinguished in terms of whether domestic or international research is cited. We analyzed the research output in the natural sciences of three leading European research economies (Germany, the Netherlands, and the UK) and ask where their researchers look for the knowledge that underpins their most highly-cited papers. Is one internationally oriented or is citation limited to national resources? Do the citation patterns reflect a growing differentiation between the domestic and international research enterprise? To evaluate change over time, we include natural-sciences papers published in the countries from three publication years: 2004, 2009, and 2014. The results show that articles co-authored by researchers from Germany or the Netherlands are less likely to be among the globally most highly-cited articles if they also cite “domestic” research (i.e. research authored by authors from the same country). To put this another way, less well-cited research is more likely to stand on domestic shoulders and research that becomes more highly-cited is more likely to stand on international shoulders. A possible reason for the results is that researchers “over-cite” the papers from their own country – lacking the focus on quality in citing. However, these differences between domestic and international shoulders are not visible for the UK.

Information and misinformation in bibliometric time-trend analysis

Abstract A diachronous time-series of bibliometric data (using all data available) suggests rising normalised citation impact (nci) for Germany and other G7 nations, while China suffers a decline in later years of any series. This is shown to be a consequence of the time-series, which has led to an erroneous interpretation of real trajectories. A synchronous series (using fixed time windows) based on the final year suggests a lower trajectory while a diachronous series tracking the fate of a single publication year reveals that nci progressively falls for Germany and the USA whereas it climbs for China. This has implications for research policy and for the interpretation of changes in the competitive research environment in the presence of dynamic growth. By analogy, this may extend to institutional as well as national comparisons. It has implications for analytical methodology, supporting prior suggestions that recent papers should be omitted from citation analysis.

Bilateral and Multilateral Coauthorship and Citation Impact: Patterns in UK and US International Collaboration

International collaboration makes up an increasing, high citation-impact share of research output, but the UK’s collaboration with key partners is threatened by its decision to leave the EU. Data show that about 85% of US and UK international collaboration is with only one or two partners, usually among other ‘leading’ research economies. Although highly multi-national research (10 or more authors) is growing more rapidly than total research output, it actually remains scarce (about 1% of all collaboration) among the established research economies. Analysis also shows that the ‘citation bonus’ contributed by international collaboration is in fact both specific and limited; it should therefore be interpreted with some care. For example, citation impact trends look different for two-country and multi-country collaborations involving the same countries. Impact also increases but then plateaus with increasing numbers of partners. Further, we find that massively multi-national papers are of such a different kind that we suggest they should be excluded from standard citation analysis.