Michael Ochsner

Citation analysis with microsoft academic

We explore if and how Microsoft Academic (MA) could be used for bibliometric analyses. First, we examine the Academic Knowledge API (AK API), an interface to access MA data, and compare it to Google Scholar (GS). Second, we perform a comparative citation analysis of researchers by normalizing data from MA and Scopus. We find that MA offers structured and rich metadata, which facilitates data retrieval, handling and processing. In addition, the AK API allows retrieving frequency distributions of citations. We consider these features to be a major advantage of MA over GS. However, we identify four main limitations regarding the available metadata. First, MA does not provide the document type of a publication. Second, the “fields of study” are dynamic, too specific and field hierarchies are incoherent. Third, some publications are assigned to incorrect years. Fourth, the metadata of some publications did not include all authors. Nevertheless, we show that an average-based indicator (i.e. the journal normalized citation score; JNCS) as well as a distribution-based indicator (i.e. percentile rank classes; PR classes) can be calculated with relative ease using MA. Hence, normalization of citation counts is feasible with MA. The citation analyses in MA and Scopus yield uniform results. The JNCS and the PR classes are similar in both databases, and, as a consequence, the evaluation of the researchers’ publication impact is congruent in MA and Scopus. Given the fast development in the last year, we postulate that MA has the potential to be used for full-fledged bibliometric analyses.

Humanities Scholars’ Conceptions of Research Quality

The assessment of research performance in the humanities is linked to the question of what humanities scholars perceive as ‘good research’. Even though scholars themselves evaluate research on a daily basis, e.g. while reading other scholars’ research, not much is known about the quality concepts scholars rely on in their judgment of research. This chapter presents a project funded by the Rectors’ Conference of the Swiss Universities, in which humanities scholars’ conceptions of research quality were investigated and translated into an approach to research evaluation in the humanities. The approach involves the scholars of a given discipline and seeks to identify agreed-upon concepts of quality. By applying the approach to three humanities disciplines, the project reveals both the opportunities and limitations of research quality assessment in the humanities: A research assessment by means of quality criteria presents opportunities to make visible and evaluate humanities research, while a quantitative assessment by means of indicators is very limited and is not accepted by scholars. However, indicators that are linked to the humanities scholars’ notions of quality can be used to support peers in the evaluation process (i.e. informed peer review).

Research Assessment in the Humanities: Introduction

Research assessments in the humanities are highly controversial. While citation-based research performance indicators are widely used in the natural and life sciences, quantitative measures for research performance meet strong opposition in the humanities. Since there are many problems connected to the use of bibliometrics in the humanities, new approaches have to be considered for the assessment of humanities research. Recently, concepts and methods for measuring research quality in the humanities have been developed in several countries. The edited volume ‘Research Assessment in the Humanities: Towards Criteria and Procedures’ analyses and discusses these recent developments in depth. It combines the presentation of state-of-the-art projects on research assessments in the humanities by humanities scholars themselves with a description of the evaluation of humanities research in practice presented by research funders. Bibliometric issues concerning humanities research complete the exhaustive analysis of humanities research assessment.

Quality criteria for sociology? What sociologists can learn from the project «developing and testing research quality criteria in the humanities»

Universities take an important role in the knowledge-society. For reasons of accountability to the public or in order to assure or enhance research quality, many universities implemented assessment procedures, often using bibliometric and other performance indicators. These procedures are mostly developed in a data-driven manner and not much is known about what the indicators in these procedures actually measure and how they affect behavior. Furthermore, the methods stem from the natural and life sciences and cannot be readily transferred to the social sciences and humanities. In this article, we present (i) quality criteria for research from the perspective of humanities scholars and how they can be transferred to sociology (ii) summarise the opportunities and limitations of the research rating of the German Council of Science and Humanities, and (iii) suggest that sociology as a discipline should develop a discipline-specific approach to research evaluation that takes into account the sociology scholars’ notions of quality and the disciplines’ research practices, that is bottom-up in nature, and uses both quantitative as well as qualitative data.