Jesper W. Schneider

Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods

This paper presents an empirical analysis of two different methodologies for calculating national citation indicators: whole counts and fractionalised counts. The aim of our study is to investigate the effect on relative citation indicators when citations to documents are fractionalised among the authoring countries. We have performed two analyses: a time series analysis of one country and a cross-sectional analysis of 23 countries. The results show that all countries’ relative citation indicators are lower when fractionalised counting is used. Further, the difference between whole and fractionalised counts is generally greatest for the countries with the highest proportion of internationally co-authored articles. In our view there are strong arguments in favour of using fractionalised counts to calculate relative citation indexes at the national level, rather than using whole counts, which is the most common practice today.

A review of the characteristics of 108 author-level bibliometric indicators

An increasing demand for bibliometric assessment of individuals has led to a growth of new bibliometric indicators as well as new variants or combinations of established ones. The aim of this review is to contribute with objective facts about the usefulness of bibliometric indicators of the effects of publication activity at the individual level. This paper reviews 108 indicators that can potentially be used to measure performance on individual author-level, and examines the complexity of their calculations in relation to what they are supposed to reflect and ease of end-user application. As such we provide a schematic overview of author-level indicators, where the indicators are broadly categorised into indicators of publication count, indicators that qualify output (on the level of the researcher and journal), indicators of the effect of output (effect as citations, citations normalized to field or the researcher’s body of work), indicators that rank the individual’s work and indicators of impact over time. Supported by an extensive appendix we present how the indicators are computed, the complexity of the mathematical calculation and demands to data-collection, their advantages and limitations as well as references to surrounding discussion in the bibliometric community. The Appendix supporting this study is available online as supplementary material.

Introduction to bibliometrics for construction and maintenance of thesauri: Methodical considerations

The paper introduces bibliometrics to the research area of knowledge organization – more precisely in relation to construction and maintenance of thesauri. As such, the paper reviews related work that has been of inspiration for the assembly of a semi‐automatic, bibliometric‐based, approach for construction and maintenance. Similarly, the paper discusses the methodical considerations behind the approach. Eventually, the semi‐automatic approach is used to verify the applicability of bibliometric methods as a supplement to construction and maintenance of thesauri. In the context of knowledge organization, the paper outlines two fundamental approaches to knowledge organization, that is, the manual intellectual approach and the automatic algorithmic approach. Bibliometric methods belong to the automatic algorithmic approach, though bibliometrics do have special characteristics that are substantially different from other methods within this approach.

Caveats for using statistical significance tests in research assessments

This article raises concerns about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators by Opthof and Leydesdorff (2010). Statistical significance tests are highly controversial and numerous criticisms have been leveled against their use. Based on examples from articles by proponents of the use statistical significance tests in research assessments, we address some of the numerous problems with such tests. The issues specifically discussed are the ritual practice of such tests, their dichotomous application in decision making, the difference between statistical and substantive significance, the implausibility of most null hypotheses, the crucial assumption of randomness, as well as the utility of standard errors and confidence intervals for inferential purposes. We argue that applying statistical significance tests and mechanically adhering to their results are highly problematic and detrimental to critical thinking. We claim that the use of such tests do not provide any advantages in relation to deciding whether differences between citation indicators are important or not. On the contrary their use may be harmful. Like many other critics, we generally believe that statistical significance tests are over- and misused in the empirical sciences including scientometrics and we encourage a reform on these matters.

Null hypothesis significance tests. A mix-up of two different theories: the basis for widespread confusion and numerous misinterpretations

Null hypothesis statistical significance tests (NHST) are widely used in quantitative research in the empirical sciences including scientometrics. Nevertheless, since their introduction nearly a century ago significance tests have been controversial. Many researchers are not aware of the numerous criticisms raised against NHST. As practiced, NHST has been characterized as a ‘null ritual’ that is overused and too often misapplied and misinterpreted. NHST is in fact a patchwork of two fundamentally different classical statistical testing models, often blended with some wishful quasi-Bayesian interpretations. This is undoubtedly a major reason why NHST is very often misunderstood. But NHST also has intrinsic logical problems and the epistemic range of the information provided by such tests is much more limited than most researchers recognize. In this article we introduce to the scientometric community the theoretical origins of NHST, which is mostly absent from standard statistical textbooks, and we discuss some of the most prevalent problems relating to the practice of NHST and trace these problems back to the mix-up of the two different theoretical origins. Finally, we illustrate some of the misunderstandings with examples from the scientometric literature and bring forward some modest recommendations for a more sound practice in quantitative data analysis.

A comparative study of first and all-author co-citation counting, and two different matrix generation approaches applied for author co-citation analyses

AimThe present article contributes to the current methodological debate concerning author co-citation analyses. (ACA) The study compares two different units of analyses, i.e. first- versus inclusive all-author co-citation counting, as well as two different matrix generation approaches, i.e. a conventional multivariate and the so-called Drexel approach, in order to investigate their influence upon mapping results. The aim of the present study is therefore to provide more methodological awareness and empirical evidence concerning author co-citation studies.MethodThe study is based on structured XML documents extracted from the IEEE collection. These data allow the construction of ad-hoc citation indexes, which enables us to carry out the hitherto largest all-author co-citation study. Four ACA are made, combining the different units of analyses with the different matrix generation approaches. The results are evaluated quantitatively by means of multidimensional scaling, factor analysis, Procrustes and Mantel statistics.ResultsThe results show that the inclusion of all cited authors can provide a better fit of data in two-dimensional mappings based on MDS, and that inclusive all-author co-citation counting may lead to stronger groupings in the maps. Further, the two matrix generation approaches produce maps that have some resemblances, but also many differences at the more detailed levels. The Drexel approach produces results that have noticeably lower stress values and are more concentrated into groupings. Finally, the study also demonstrates the importance of sparse matrices and their potential problems in connection with factor analysis.ConclusionWe can confirm that inclusive all-ACA produce more coherent groupings of authors, whereas the present study cannot clearly confirm previous findings that first-ACA identifies more specialties, though some vague indication is given. Most crucially, strong evidence is given to the determining effect that matrix generation approaches have on the mapping of author co-citation data and thus the interpretation of such maps. Evidence is provided for the seemingly advantages of the Drexel approach.

Verification of bibliometric methods' applicability for thesaurus construction

The doctoral dissertation work concerns the development and exploration of a semi-automatic thesaurus construction approach based on bibliometric methods.

Developing a methodology to assess the impact of research grant funding: A mixed methods approach

This paper discusses the development of a mixed methods approach to analyse research funding. Research policy has taken on an increasingly prominent role in the broader political scene, where research is seen as a critical factor in maintaining and improving growth, welfare and international competitiveness. This has motivated growing emphasis on the impacts of science funding, and how funding can best be designed to promote socio-economic progress. Meeting these demands for impact assessment involves a number of complex issues that are difficult to fully address in a single study or in the design of a single methodology. However, they point to some general principles that can be explored in methodological design. We draw on a recent evaluation of the impacts of research grant funding, discussing both key issues in developing a methodology for the analysis and subsequent results. The case of research grant funding, involving a complex mix of direct and intermediate effects that contribute to the overall impact of funding on research performance, illustrates the value of a mixed methods approach to provide a more robust and complete analysis of policy impacts. Reflections on the strengths and weaknesses of the methodology are used to examine refinements for future work.

Concept symbols revisited: Naming clusters by parsing and filtering of noun phrases from citation contexts of concept symbols

SummaryThe present study presents a semi-automatic method for parsing and filtering of noun phrases from citation contexts of concept symbols. The purpose of the method is to extract contextual, agreed upon, and pertinent noun phrases, to be used in visualization studies for naming clusters (concept groups) or concept symbols. The method is applied in a case study, which forms part of a larger dissertation work concerning the applicability of bibliometric methods for thesaurus construction. The case study is carried out within periodontology, a specialty area of dentistry. The result of the case study indicates that the method is able to identify highly important noun phrases, and that these phrases accurately describe their parent clusters. Hence, the method is able to reduce the labour intensive work of manual citation context analysis, though further refinements are still needed.The present study presents a semi-automatic method for parsing and filtering of noun phrases from citation contexts of concept symbols. The purpose of the method is to extract contextual, agreed upon, and pertinent noun phrases, to be used in visualization studies for naming clusters (concept groups) or concept symbols. The method is applied in a case study, which forms part of a larger dissertation work concerning the applicability of bibliometric methods for thesaurus construction. The case study is carried out within periodontology, a specialty area of dentistry. The result of the case study indicates that the method is able to identify highly important noun phrases, and that these phrases accurately describe their parent clusters. Hence, the method is able to reduce the labour intensive work of manual citation context analysis, though further refinements are still needed.

One and a half million medical papers reveal a link between author gender and attention to gender and sex analysis

Gender and sex analysis is increasingly recognized as a key factor in creating better medical research and health care1–7. Using a sample of more than 1.5 million medical research papers, our study examined the potential link between women’s participation in medical science and attention to gender-related and sex-related factors in disease-specific research. Adjusting for variations across countries, disease topics and medical research areas, we compared the participation of women authors in studies that do and do not involve gender and sex analysis. Overall, our results show a robust positive correlation between women’s authorship and the likelihood of a study including gender and sex analysis. These findings corroborate discussions of how women’s participation in medical science links to research outcomes, and show the mutual benefits of promoting both the scientific advancement of women and the integration of gender and sex analysis into medical research.Nielsen and colleagues’ analysis of a large database of medical research papers shows a correlation between women’s authorship and the likelihood of a study including gender and sex analysis.