Aidan Kelly

The use and abuse of journal quality lists

The ABS list is one of the most widely used journal quality lists available. It is pluralistic and comprehensive, and is widely seen as capturing the consensus in relation to journal ratings. As editors of the ABS list we are concerned at recent calls for a moratorium, or even the abolition of journal quality lists. These calls reinforce the elitist view that the tacit knowledge of where to publish should not be made explicit in lists.

Measuring the Value of Sociology? Some Notes on Performative Metricization in the Contemporary Academy

The performative co-construction of academic life through myriad metrics is now a global phenomenon as indicated by the plethora of university research or journal ranking systems and the publication of ‘league’ tables based on them. If these metrics are seen as actively constituting the social world, can an analysis of this ‘naturally occurring’ data reveal how these new technologies of value and measure are recursively defining the practices and subjects of university life? In the UK higher education sector, the otherwise mundane realities of academic life have come to be recursively lived through a succession of research assessment exercises (RAEs). Lived through not only in the RAEs themselves, but also through the managed incremental changes to the academic and organizational practices linked to the institutional imaginings of planning for, and anticipating the consequences of, the actual exercises. In the ‘planning for’ mode an increasing proportion of formerly sociology submissions have shifted into ‘social policy’. This is one instance of how institutional ‘game-playing’ in relation to the RAE enacts the social in quite fundamental ways. Planning an RAE 2008 submission in Sociology required anticipation of how a panel of 16 peers would evaluate 39 institutions by weighted, relative worth of: aggregated data from 1,267 individuals who, between them cited a total of 3,729 ‘outputs’; the detailed narrative and statistical data on the research environment; and a narrative account of academic ‘esteem’. This data provided such institutional variables as postgraduate student numbers, sources of student funding, and research income from various sources. To evaluate the ‘quality’ of outputs various measures of the ‘impact’ and/or ‘influence’ of journals, as developed from the Thomson-Reuters Journal Citation Reports, was linked to the data. An exploratory modelling exercise using these variables to predict RAE 2008 revealed that despite what we might like to think about the subtle nuances involved in peer review judgements, it turns out that a fairly astonishing 83 per cent of the variance in outcomes can be predicted by some fairly simple ‘shadow metrics’: quality of journals in the submission, research income per capita and scale of research activity. We conclude that measuring the value of sociology involves multiple mutual constructions of reality within which ever more nuanced data assemblages are increasingly implicated and that analysis of this data can make explicit some of the parameters of enactment within which we operate in the contemporary academy.

Accounting Journals and the ABS Guide: A Review of Evidence and Inference

This article reviews the evidence presented by those alleging bias in the rating of accounting journals in the ABS Guide to Journal Quality in Business and Management Studies. Drawing upon a wider range of citation metrics and international journal ranking schemes, the analysis presented covers all the accounting journals listed in the current 2010 version of the ABS Guide. The evidence of bias based on the mean RAE 2008 GPA scores for institutions and on the ‘world elite count’ as published in the guide is rejected. The simplistic comparison of the proportion of grade ‘4’ journals in accounting relative to other subject areas is rejected as evidence of bias and the preferred method reveals that the ABS Guide provides the accounting subject area with one more grade ‘4’ journal and seven more grade ‘3’ journals than might otherwise be expected. There is no evidence that the use of Thomson-Reuters citation data in the process of constructing the ABS Guide has been detrimental to accounting journals. The accounting journals in the ABS Guide have lower mean scores on almost all citation metrics when compared to other subject area journals with the same ABS grade. A detailed comparison of the ranking of accounting journals in seven international journal ranking lists fails to find evidence of bias against accounting journals and suggests that grade ‘3’ accounting journals have been favorably rated. In a detailed comparison of accounting and business history journals, there is little evidence from any source reviewed to suggest that Accounting History is of the same quality rank as Business History or that Accounting History clearly warrants a higher ABS Guide rating. The conclusion is that the evidence presented supports the view that the ABS Guide has not been biased in its rating of accounting journals.