Rodrigo Costas

The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level

The relationship of the h-index with other bibliometric indicators at the micro level is analysed for Spanish CSIC scientists in Natural Resources, using publications downloaded from the Web of Science (1994–2004). Different activity and impact indicators were obtained to describe the research performance of scientists in different dimensions, being the h-index located through factor analysis in a quantitative dimension highly correlated with the absolute number of publications and citations. The need to include the remaining dimensions in the analysis of research performance of scientists and the risks of relying only on the h-index are stressed. The hypothesis that the achievement of some highly visible but intermediate-productive authors might be underestimated when compared with other scientists by means of the h-index is tested.

Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective

An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%–24% of the publications presenting some altmetric activity and concentrated on the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities, and the medical and life sciences show the highest presence of altmetrics, indicating their potential value and interest for these fields. The analysis of the relationships between altmetrics and citations confirms previous claims of positive correlations but is relatively weak, thus supporting the idea that altmetrics do not reflect the same kind of impact as citations. Also, altmetric counts do not always present a better filtering of highly‐cited publications than journal citation scores. Altmetric scores (particularly mentions in blogs) are able to identify highly‐cited publications with higher levels of precision than journal citation scores (JCS), but they have a lower level of recall. The value of altmetrics as a complementary tool of citation analysis is highlighted, although more research is suggested to disentangle the potential meaning and value of altmetric indicators for research evaluation.

Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns

A number of new metrics based on social media platforms—grouped under the term “altmetrics”—have recently been introduced as potential indicators of research impact. Despite their current popularity, there is a lack of information regarding the determinants of these metrics. Using publication and citation data from 1.3 million papers published in 2012 and covered in Thomson Reuters’ Web of Science as well as social media counts from Altmetric.com, this paper analyses the main patterns of five social media metrics as a function of document characteristics (i.e., discipline, document type, title length, number of pages and references) and collaborative practices and compares them to patterns known for citations. Results show that the presence of papers on social media is low, with 21.5% of papers receiving at least one tweet, 4.7% being shared on Facebook, 1.9% mentioned on blogs, 0.8% found on Google+ and 0.7% discussed in mainstream media. By contrast, 66.8% of papers have received at least one citation. Our findings show that both citations and social media metrics increase with the extent of collaboration and the length of the references list. On the other hand, while editorials and news items are seldom cited, it is these types of document that are the most popular on Twitter. Similarly, while longer papers typically attract more citations, an opposite trend is seen on social media platforms. Finally, contrary to what is observed for citations, it is papers in the Social Sciences and humanities that are the most often found on social media platforms. On the whole, these findings suggest that factors driving social media and citations are different. Therefore, social media metrics cannot actually be seen as alternatives to citations; at most, they may function as complements to other type of indicators.

Is g-index better than h-index? An exploratory study at the individual level

The ability of g-index and h-index to discriminate between different types of scientists (low producers, big producers, selective scientists and top scientists) is analysed in the area of Natural Resources at the Spanish CSIC (WoS, 1994–2004). Our results show that these indicators clearly differentiate low producers and top scientists, but do not discriminate between selective scientists and big producers. However, g-index is more sensitive than h-index in the assessment of selective scientists, since this type of scientist shows in average a higher g-index/h-index ratio and a better position in g-index rankings than in the h-index ones. Current research suggests that these indexes do not substitute each other but that they are complementary.

F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations

F1000 is a postpublication peer review service for biological and medical research. F1000 recommends important publications in the biomedical literature, and from this perspective F1000 could be an interesting tool for research evaluation. By linking the complete database of F1000 recommendations to the Web of Science bibliographic database, we are able to make a comprehensive comparison between F1000 recommendations and citations. We find that about 2% of the publications in the biomedical literature receive at least one F1000 recommendation. Recommended publications on average receive 1.30 recommendations, and more than 90% of the recommendations are given within half a year after a publication has appeared. There turns out to be a clear correlation between F1000 recommendations and citations. However, the correlation is relatively weak, at least weaker than the correlation between journal impact and citations. More research is needed to identify the main reasons for differences between recommendations and citations in assessing the impact of publications.

Self-citations at the meso and individual levels: effects of different calculation methods

This paper focuses on the study of self-citations at the meso and micro (individual) levels, on the basis of an analysis of the production (1994–2004) of individual researchers working at the Spanish CSIC in the areas of Biology and Biomedicine and Material Sciences. Two different types of self-citations are described: author self-citations (citations received from the author him/herself) and co-author self-citations (citations received from the researchers’ co-authors but without his/her participation). Self-citations do not play a decisive role in the high citation scores of documents either at the individual or at the meso level, which are mainly due to external citations. At micro-level, the percentage of self-citations does not change by professional rank or age, but differences in the relative weight of author and co-author self-citations have been found. The percentage of co-author self-citations tends to decrease with age and professional rank while the percentage of author self-citations shows the opposite trend. Suppressing author self-citations from citation counts to prevent overblown self-citation practices may result in a higher reduction of citation numbers of old scientists and, particularly, of those in the highest categories. Author and co-author self-citations provide valuable information on the scientific communication process, but external citations are the most relevant for evaluative purposes. As a final recommendation, studies considering self-citations at the individual level should make clear whether author or total self-citations are used as these can affect researchers differently.

Do age and professional rank influence the order of authorship in scientific publications? Some evidence from a micro-level perspective

Scientific authorship has important implications in science since it reflects the contribution to research of the different individual scientists and it is considered by evaluation committees in research assessment processes. This study analyses the order of authorship in the scientific output of 1,064 permanent scientists at the Spanish CSIC (WoS, 1994–2004). The influence of age, professional rank and bibliometric profile of scientists over the position of their names in the byline of publications is explored in three different research areas: Biology and Biomedicine, Materials Science and Natural Resources. There is a strong trend for signatures of younger researchers and those in the lower professional ranks to appear in the first position (junior signing pattern), while more veteran or highly-ranked ones, who tend to play supervisory functions in research, are proportionally more likely to sign in the last position (senior signing pattern). Professional rank and age have an effect on authorship order in the three fields analysed, but there are inter-field differences. Authorship patterns are especially marked in the most collaboration-intensive field (i.e. Biology and Biomedicine), where professional rank seems to be more significant than age in determining the role of scientists in research as seen through their authorship patterns, while age has a more significant effect in the least collaboration-intensive field (Natural Resources).

Approaching the “reward triangle”: General analysis of the presence of funding acknowledgments and “peer interactive communication” in scientific publications

Understanding the role of acknowledgments given by researchers in their publications has been a recurrent challenge in the bibliometric field, but relatively unexplored until now. This study presents a general bibliometric analysis on the new “funding acknowledgment” (FA) information available in the Web of Science. All publications covered by the database in 2009 have been analyzed. The presence and length of the FA text, as well as the presence of “peer interactive communication” in the acknowledgments, are related to impact indicators, distribution of papers by fields, countries of the authors, and collaboration level of the papers. It is observed that publications with FAs present a higher impact as compared with publications without them. There are also differences across countries and disciplines in the share of publications with FAs and the acknowledgment of peer interactive communication. China is the country with the highest share of publications acknowledging funding, while the presence of FAs in the humanities and social sciences is very low compared to the more basic disciplines. The presence of peer interactive communication in acknowledgments can be linked to countries that have a strong scientific tradition and are incorporated in scientific networks. Peer interactive communication is also common in the fields of humanities and social sciences and can be linked to lower levels of co-authorship. Observed patterns are explained and topics of future research are proposed.

The skewness of scientific productivity

This paper exploits a unique 2003–2011 large dataset, indexed by Thomson Reuters, consisting of 17.2 million disambiguated authors classified into 30 broad scientific fields, as well as the 48.2 million articles resulting from a multiplying strategy in which any article co-authored by two or more persons is wholly assigned as many times as necessary to each of them. The dataset is characterized by a large proportion of authors who have their oeuvre in several fields. We measure individual productivity in two ways that are uncorrelated: as the number of articles per person and as the mean citation per article per person in the 2003–2011 period. We analyze the shape of the two types of individual productivity distributions in each field using size- and scale-independent indicators. To assess the skewness of productivity distributions we use a robust index of skewness, as well as the Characteristic Scores and Scales approach. For productivity inequality, we use the coefficient of variation. In each field, we study two samples: the entire population, and what we call “successful authors”, namely, the subset of scientists whose productivity is above their field average. The main result is that, in spite of wide differences in production and citation practices across fields, the shape of field productivity distributions is very similar across fields. The parallelism of the results for the population as a whole and for the subset of successful authors, when productivity is measured as mean citation per article per person, reveals the fractal nature of the skewness of scientific productivity in this case. These results are essentially maintained when any article co-authored by two or more persons is fractionally assigned to each of them.

Meta-assessment of bias in science

Significance Science is said to be suffering a reproducibility crisis caused by many biases. How common are these problems, across the wide diversity of research fields? We probed for multiple bias-related patterns in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was on average relatively small. However, we consistently observed that small, early, highly cited studies published in peer-reviewed journals were likely to overestimate effects. We found little evidence that these biases were related to scientific productivity, and we found no difference between biases in male and female researchers. However, a scientist’s early-career status, isolation, and lack of scientific integrity might be significant risk factors for producing unreliable results. Numerous biases are believed to affect the scientific literature, but their actual prevalence across disciplines is unknown. To gain a comprehensive picture of the potential imprint of bias in science, we probed for the most commonly postulated bias-related patterns and risk factors, in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was overall relatively small. However, we consistently observed a significant risk of small, early, and highly cited studies to overestimate effects and of studies not published in peer-reviewed journals to underestimate them. We also found at least partial confirmation of previous evidence suggesting that US studies and early studies might report more extreme effects, although these effects were smaller and more heterogeneously distributed across meta-analyses and disciplines. Authors publishing at high rates and receiving many citations were, overall, not at greater risk of bias. However, effect sizes were likely to be overestimated by early-career researchers, those working in small or long-distance collaborations, and those responsible for scientific misconduct, supporting hypotheses that connect bias to situational factors, lack of mutual control, and individual integrity. Some of these patterns and risk factors might have modestly increased in intensity over time, particularly in the social sciences. Our findings suggest that, besides one being routinely cautious that published small, highly-cited, and earlier studies may yield inflated results, the feasibility and costs of interventions to attenuate biases in the literature might need to be discussed on a discipline-specific and topic-specific basis.