Olesya Mryglod

Predicting results of the Research Excellence Framework using departmental h-index

We compare estimates for past institutional research performances coming from two bibliometric indicators to the results of the UK’s Research Assessment Exercise which last took place in 2008. We demonstrate that a version of the departmental h-index is better correlated with the actual results of that peer-review exercise than a competing metric known as the normalised citation-based indicator. We then determine the corresponding h-indices for 2008–2013, the period examined in the UK’s Research Excellence Framework (REF) 2014. We place herewith the resulting predictions on the arXiv in advance of the REF results being published (December 2014). These may be considered as unbiased predictions of relative performances in that exercise. We will revisit this paper after the REF results are available and comment on the reliability or otherwise of these bibliometrics as compared with peer review.

Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence

Many different measures are used to assess academic research excellence and these are subject to ongoing discussion and debate within the scientometric, university-management and policy-making communities internationally. One topic of continued importance is the extent to which citation-based indicators compare with peer-review-based evaluation. Here we analyse the correlations between values of a particular citation-based impact indicator and peer-review scores in several academic disciplines, from natural to social sciences and humanities. We perform the comparison for research groups rather than for individuals. We make comparisons on two levels. At an absolute level, we compare total impact and overall strength of the group as a whole. At a specific level, we compare academic impact and quality, normalised by the size of the group. We find very high correlations at the former level for some disciplines and poor correlations at the latter level for all disciplines. This means that, although the citation-based scores could help to describe research-group strength, in particular for the so-called hard sciences, they should not be used as a proxy for ranking or comparison of research groups. Moreover, the correlation between peer-evaluated and citation-based scores is weaker for soft sciences.

Absolute and specific measures of research group excellence

A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellence as performance per head. Using biology research institutions in the UK as a test case, we examine the correlations between peer review-based and citation-based measures of research excellence on these two scales. We find that citation-based indicators are very highly correlated with peer-evaluated measures of group strength, but are poorly correlated with group quality. Thus, and almost paradoxically, our analysis indicates that citation counts could possibly form a basis for deciding on, how to fund research institutions, but they should not be used as a basis for ranking them in terms of quality.

Interevent time distributions of human multi-level activity in a virtual world

Studying human behavior in virtual environments provides extraordinary opportunities for a quantitative analysis of social phenomena with levels of accuracy that approach those of the natural sciences. In this paper we use records of player activities in the massive multiplayer online game Pardus over 1238 consecutive days, and analyze dynamical features of sequences of actions of players. We build on previous work where temporal structures of human actions of the same type were quantified, and provide an empirical understanding of human actions of different types. This study of multi-level human activity can be seen as a dynamic counterpart of static multiplex network analysis. We show that the interevent time distributions of actions in the Pardus universe follow highly non-trivial distribution functions, from which we extract action-type specific characteristic “decay constants”. We discuss characteristic features of interevent time distributions, including periodic patterns on different time scales, bursty dynamics, and various functional forms on different time scales. We comment on gender differences of players in emotional actions, and find that while males and females act similarly when performing some positive actions, females are slightly faster for negative actions. We also observe effects on the age of players: more experienced players are generally faster in making decisions about engaging in and terminating enmity and friendship, respectively.

Quantifying the evolution of a scientific topic: reaction of the academic community to the Chornobyl disaster

We analyze the reaction of academic communities to a particular urgent topic which abruptly arises as a scientific problem. To this end, we have chosen the disaster that occurred in 1986 in Chornobyl (Chernobyl), Ukraine, considered as one of the most devastating nuclear power plant accidents in history. The academic response is evaluated using scientific-publication data concerning the disaster using the Scopus database to present the picture on an international scale and the bibliographic database “Ukrainika naukova” to consider it on a national level. We measured distributions of papers in different scientific fields, their growth rates and properties of co-authorship networks. Elements of descriptive statistics and tools of complex network theory are used to highlight the interdisciplinary as well as international effects. Our analysis allows comparison of contributions of the international community to Chornobyl-related research as well as integration of Ukraine in international research on this subject. Furthermore, the content analysis of titles and abstracts of the publications allowed detection of the most important terms used for description of Chornobyl-related problems.

Editorial process in scientific journals: analysis and modeling

The editorial handling of articles in scientific journals as a human activity process is considered. Using recently proposed approaches of human dynamics theory we examine the probability distributions of random variables reflecting the temporal characteristics of studied processes. The first part of this article contains our results of analysis of the real data about articles published in scientific journals. The second part is devoted to modeling of time-series connected with editorial work. The purpose of our study is to present new object that can be studied in terms of human dynamics theory and to corroborate the scientometrical application of the results obtained.

A scientists' view of scientometrics: Not everything that counts can be counted

Like it or not, attempts to evaluate and monitor the quality of academic research have become increasingly prevalent worldwide. Performance reviews range from at the level of individuals, through research groups and departments, to entire universities. Many of these are informed by, or functions of, simple scientometric indicators and the results of such exercises impact onto careers, funding and prestige. However, there is sometimes a failure to appreciate that scientometrics are, at best, very blunt instruments and their incorrect usage can be misleading. Rather than accepting the rise and fall of individuals and institutions on the basis of such imprecise measures, calls have been made for indicators be regularly scrutinised and for improvements to the evidence base in this area. It is thus incumbent upon the scientific community, especially the physics, complexity-science and scientometrics communities, to scrutinise metric indicators. Here, we review recent attempts to do this and show that some metrics in widespread use cannot be used as reliable indicators research quality.

Analyses of a Virtual World

We present an overview of a series of results obtained from the analysis of human behavior in a virtual environment. We focus on the massive multiplayer online game (MMOG) Pardus which has a worldwide participant base of more than 400,000 registered players. We provide evidence for striking statistical similarities between social structures and human-action dynamics in real and virtual worlds. In this sense MMOGs provide an extraordinary way for accurate and falsifiable studies of social phenomena. We further discuss possibilities to apply methods and concepts developed in the course of these studies to analyse oral and written narratives.

Correlations, connections, and physics in complex systems

It is with great pleasure that we present, in this issue, a series of papers dedicated to the 60th birthday of Professor Yurij Holovatch. The broad spectrum of topics treated by the contributions reflect Yurij’s comprehensive interests and his wide-ranging research. Thus, contributions range from papers on phasetransition theory to papers dealing with scientometric issues, from condensed matter theory to cosmological and astronomical problems. Yurij’s broad activities are not only reflected in the scope of his publications but also in his activities disseminating physics into other fields and initiating communications between scientists. For example, he founded the now-famous Ising lectures, which for the past two decades have been an important beacon for statistical physics in Europe. The associated review series Order Disorder and Criticality: Advanced Problems of Phase Transition Theory have been single-handedly edited by Yurij and published by World Scientific since 2004. By now, these contain 21 important and influential chapters describing the state of the art by 25 authorities in the field. In 2010 Yurij founded the Laboratory for Statistical Physics of Complex Systems at the Institute for Condensed Matter Physics in Lviv. There, he maintains regular activities with the members of his group, having seminars every week, even through modern communication tools when the majority of the members are abroad. Yurij co-founded the interdisciplinary seminar series Horizon of Science where problems at the cutting edge of modern science are discussed. A set of talks from these seminars were recently published (in Ukrainian). More recently, he initiated an exciting series of interdisciplinary workshops titled Physics Meets Humanities to demonstrate the applicability of the concepts and methods of science in social areas.