A.I.M. Jakaria Rahman

Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters

Discipline-specific research evaluation exercises are typically carried out by panels of peers, known as expert panels. To the best of our knowledge, no methods are available to measure overlap in expertise between an expert panel and the units under evaluation. This paper explores bibliometric approaches to determine this overlap, using two research evaluations of the departments of Chemistry (2009) and Physics (2010) of the University of Antwerp as a test case. We explore the usefulness of overlay mapping on a global map of science (with Web of Science subject categories) to gauge overlap of expertise and introduce a set of methods to determine an entitys barycenter according to its publication output. Barycenters can be calculated starting from a similarity matrix of subject categories (N dimensions) or from a visualization thereof (2 dimensions). We compare the results of the N-dimensional method with those of two 2-dimensional ones (Kamada–Kawai maps and VOS maps) and find that they yield very similar results. The distance between barycenters is used as an indicator of expertise overlap. The results reveal that there is some discrepancy between the panels and the groups’ publications in both the Chemistry and the Physics departments. The panels were not as diverse as the groups that were assessed. The match between the Chemistry panel and the Department was better than that between the Physics panel and the Department.

Measuring cognitive distance between publication portfolios

We study the problem of determining the cognitive distance between the publication portfolios of two units. In this article we provide a systematic overview of five different methods (a benchmark Euclidean distance approach, distance between barycenters in two and in three dimensions, distance between similarity-adapted publication vectors, and weighted cosine similarity) to determine cognitive distances using publication records. We present a theoretical comparison as well as a small empirical case study. Results of this case study are not conclusive, but we have, mainly on logical grounds, a small preference for the method based on similarity-adapted publication vectors.

Corrigendum to “Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters” [Journal of Informetrics 9 (4) (2015) 704–721]

In Rahman, Guns, Rousseau, and Engels (2015) we described several approaches to determine the cognitive distance between two units. One of these approaches was based on what we called barycenters in N dimensions. This note corrects this terminology and introduces the more adequate term ‘similarity-adapted publication vectors’.

Determining cognitive distance between publication portfolios of evaluators and evaluees in research evaluation: An exploration of informetric methods

This doctoral thesis develops informetric methods for determining cognitive distance between publication portfolios of evaluators and evaluees in research evaluation. In a discipline specific research evaluation, when an expert panel evaluates research groups, it is an open question how one can determine the extent to which the panel members are in a position to evaluate the research groups. This thesis contributes to the literature by proposing six different informetric approaches to measure the match between evaluators and evaluees using their publications as a representation of their expertise. An expert panel is specifically appointed for the research evaluation. Experts are typically selected in one of two ways: (1) straightforward selection: the person(s) in charge of the research evaluation has access to a list of acknowledged experts in specific fields, and limits its selection process to ensuring the experts’ independence regarding the program under evaluation; and (2) gradual selections: preferred profiles of experts are developed with respect to the specialization under scrutiny in the evaluation. Both ways leave some freedom for an “old boys’ network” to appoint someone without properly evaluating their qualifications. There are also other ways for expert selection, for example, inviting open application or the research groups that will be evaluated can propose their choice of experts. In research evaluation, an expert panel usually comprises independent specialists, each of which is recognized in at least one of the fields addressed by the unit under evaluation. The expertise of the panel members should be congruent with the research groups to ensure the quality and trustworthiness of the evaluation. All things being equal, panel members who are credible experts in the field are also most likely to provide valuable, relevant recommendations and suggestions that should lead to improved research quality. However, there was an absence of methods to determine the cognitive distance between evaluators and evaluees in research evaluation when we started working in July 2013. In this thesis, we develop and test informetric methods to identify the cognitive distances between the (members of) an expert panel on the one hand, and the (whole of the) units of assessment (typically research groups) on the other. More generally, we introduce a number of methods that allow measuring cognitive distances based on publication portfolios. In academia, publications are considered key indicators of expertise that help to identify qualified or similar experts to assign papers for review, and to form an expert panel. Our main objective is to propose informetric methods to identify panel members who have closely related expertise in the research domain of the research groups based on their publications profile. The main factor that we have taken into account is the cognitive distance between an expert panel and research groups. We consider the publication portfolio of the involved researchers to reflect the position of the unit in cognitive space and, hence, to determine cognitive distance. Expressed in general terms we measure cognitive distance between units based on how often they have published in the same or similar journals. Our investigations lead to the development of new methods of expert panel composition for the research evaluation exercises. We explore different ways of quantifying the cognitive distance between panel members and research groups publication profiles. We consider all the publications of the research groups (during the eight years preceding their evaluation) and panel members indexed in Web of Science (WoS). We pursue the investigation at two levels of aggregation: WoS subject categories (SCs) and journals. The aggregated citation relations among SCs or journals provide a matrix. From the matrix, one can construct a similarity matrix. From the similarity matrix, one can construct a global SCs or journal map in which similar SCs or journals are located more closely together. The maps can be visualized using a visualization program. During the visualization process, a multi-dimensional space is reduced to a projection in two dimensions. In this process, similar SCs or journals are positioned closer to each other. We propose three methods, namely the use of barycenters, of similarity-adapted publication vector (SAPV) and of weighted cosine similarity (WCS). We take into account the similarity between WoS SCs and between journals, either by incorporating a similarity matrix (in the case of SAPV and WCS) or a 2-dimensional base map derived from it (in the case of barycenters). We determine the coordinates of barycenters using a 2-dimensional base map based on the publication profiles of research groups and panel members, and calculate the Euclidean distances between the barycenters. We also identify SAPV using the similarity matrix and calculated the Euclidean distances between the SAPVs. Finally, we calculate WCS using the similarity matrix. The SAPV and WCS methods use a square N-dimensional similarity matrix. Here N is equivalent to 224 WoS SCs and 10,675 journals. We used the distance/similarity between panel members and research groups as an indicator of cognitive distance. Small differences in Euclidean distances (both between barycenters and SAPVs) or in cosine similarity values bear little meaning. For this reason, we employ a bootstrapping approach in order to determine a 95% confidence interval (CI) for each distance or similarity value. If two CIs do not overlap, difference between the values is statistically significant at the 0.05 level. Although it is possible for two values to have a statistically significant difference while having overlapping CIs, the difference is less likely to have practical meaning. Two levels of aggregation and three methods lead to six informetric approaches to quantify the cognitive distance. Our proposed approaches hold advantages over a simple comparison of publication portfolios. Our approaches quantify the cognitive distance between a research group and panel members. We also compare our proposed approaches. We examine which of the approaches best reflects the prior assignment of main assessor to each research group, how much influence the level of aggregation (journals and WoS SCs) plays, and how much the dimensionality matters. The results show that, regardless of the method used, the level of aggregation has only a minor influence, whereas the influence of the number of dimensions is substantial. The results also show that the number of dimensions plays a major role in the case of identifying shortest cognitive distance. While the SAPV and WCS methods agree at most of cases at both the levels of aggregation the barycenter approaches yield different results. We find that the barycenter approaches score highest at both levels of aggregation to identify the previously assigned main assessor. When it comes to uniquely identifying the main assessor, all methods score better at the journal level than at the WoS SC level. Our approaches, but of course not the numerical result, are independent of the similarity matrix or map used. All six approaches give the opportunity to assess the composition of the panel in terms of cognitive distance if one or more panel members are replaced and compare the relative contribution of each potential panel member to the panel fit as a whole, by observing the changes to the distance between the panel’s and the groups’. In addition, our approaches allow the panel composition authority to see in advance about the panel’s fit to the research groups that are going to be evaluated. Therefore, the concerned authority will have the opportunity to replace outliers among the panel members to make the panel fit well with the research groups to be evaluated. For example, the authority can find a best-fitting expert panel by replacing a more distant panel member with a potential panel member located closer to the groups.

Cognitive Distances between Evaluators and Evaluees in Research Evaluation: A Comparison between Three Informetric Methods at the Journal and Subject Category Aggregation Level

This article compares six informetric approaches to determine cognitive distances between the publications of panel members and those of research groups in discipline-specific research evaluation. We used data collected in the framework of six completed research evaluations from the period 2009-2014 at the University of Antwerp as a test case. We distinguish between two levels of aggregation – Web of Science subject categories and journals – and three methods: while the barycenter method (2-dimensional) is based on global maps of science, the similarity-adapted publication vector (SAPV) method and weighted cosine similarity (WCS) method (both in higher dimensions) use a full similarity matrix. In total, this leads to six different approaches, all of which are based on the publication profile of research groups and panel members. We use Euclidean distances between barycenters and SAPVs, as well as values of WCS between panel members and research groups as indicators of cognitive distance. We systematically compare how these six approaches are related. The results show that the level of aggregation has minor influence on determining cognitive distances, but dimensionality (two versus a high number of dimensions) has a greater influence. The SAPV and WCS methods agree in most cases at both levels of aggregation on which panel member has the closest cognitive distance to the group to be evaluated, whereas the barycenter approaches often differ. Comparing the results of the methods to the main assessor that was assigned to each research group, we find that the barycenter method usually scores better. However, the barycenter method is less discriminatory and suggests more potential evaluators, whereas SAPV and WCS are more precise.

Digital Information Literacy: A Case Study in Oslo Public Library

This paper examines the digital information literacy (DIL) of public library professionals in Norway and explores the ways to improve their skills as well as identify barriers to improvement. The case study method was used and semi-structured face-to-face interviews were conducted with twenty public library professionals. The knowledge sharing approach was visible among the staff, but the slow adaptation of technology, and organizational, personal, and technological barriers were hindering the DIL development. Online training modules, mapping the staff competencies, assessment of the staff needs, advanced and customized training programs, long-term strategies, and decentralized initiatives were suggested for the improvement of DIL.