Prestige of scholarly book publishers: an investigation into criteria, processes, and practices across countries
Prestige of scholarly book publishers – an investigation into criteria, processes, and practices across countries
Eleonora Dagienė Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands
Abstract
Numerous national research assessment policies set the goal of promoting ‘excellence’ and incentivise scholars to publish their research in the most prestigious journals or with the most prestigious book publishers. We investigate the practicalities of the assessment of book outputs based on the prestige of book publishers (Denmark, Finland, Flanders, Lithuania, Norway). Additionally, we test whether such assessments are transparent and yield consistent results. We show inconsistencies in the assessment of publishers, such as the same publisher being ranked as prestigious and not-so-prestigious in different countries or in different years in the same country. Likewise, we find that verification of compliance with the mandatory prerequisites is not always possible because of the lack of transparency. Our findings raise doubts about whether the assessment of books based on a judgement about their publisher yields acceptable outcomes. Currently used rankings of publishers focus on evaluating the gatekeeping role of publishers but do not assess their dissemination role. Our suggestion for future research is to develop approaches for assessing books which consider both quality control and the distribution of books (and their metadata) as measured by the importance of communication between researchers. That means that publishers should be transparent about the services they deliver in both areas, preferably at the level of individual books, so that there is no need to rely on general information about publishers.
Keywords : book assessment, rankings of publishers, prestigious publishers, research excellence, book publishers, UK REF
1. Introduction
For several decades, policymakers in many countries have incentivised researchers to publish their findings through the ‘most prestigious channels’ to promote ‘excellence’. The narratives of ‘excellence’ and ‘quality’ in academia (Lamont 2009; Moore et al. 2017), as well as the academic ‘prestige culture’ (Fyfe et al. 2017) and ‘quantified control’ (Burrows 2012) for funding allocation affect the perception of research book publishers’ ‘quality’ and ‘prestige’. Acquisition librarians were the first who asked researchers to determine the ‘quality’ of publishers (Lewis 2000; Metz & Stemmer 1996) to support the development of library collections. Academics were surveyed to identify ‘quality indicators’ for books and their publishers (Giménez-Toledo et al. 2013) so as to assist in creating a ranking of publishers. Apart from this traditional method, many attempts have emerged to quantify the assessment of books, evaluate their impact, and distinguish publishers: using libcitations (White et al. 2009; White & Zuccala 2018), book reviews (Zuccala & van Leeuwen 2011; Zuccala & Robinson-García 2019), or a set of digital indicators (Neville & Henry 2014). The newest qualitative and quantitative initiatives are often experimental (Giménez-Toledo et al. 2015), while efforts to be purely quantitative, in some cases, lack clearly stated policies (Basso et al. 2017; Williams & Galleron 2016). Almost two decades ago, Nederhof et al. (2001) constructed three indices ‘a quality weight, an (inter)national visibility weight, and a combined index’ for publishers assessing publishers in linguistics. Currently, ranking publishing channels and compiling lists of prestigious publishers are common practices for metric-based funding systems. The first thoroughly documented ranking of publication channels (journals, book series, and publishers), called the Norwegian model by its developer Sivertsen (2018), was implemented in 2005. It is a highly visible and extensively followed approach. In 2008 and 2012, respectively, Denmark and Finland put into practice similar models, including the identification of top-level publishers. Flanders (Belgium) takes a somewhat different approach by differentiating publishers according to their peer review practices (Giménez-Toledo et al. 2016). In Lithuania (the author’s home country), policymakers began using the term ‘prestigious publisher’ a quarter-century ago, but little is known about Lithuanian book assessment practices. Some of the procedure has been described as ‘an essentially bureaucratic decision on what is and what is not a book’ (Williams et al. 2018). There is an additional special element in this process in Lithuania compared with other countries. This is the use of a publisher points system to allocate funding: every fourteen pages of a book or a chapter in an edited volume earns points, and therefore funds, for Lithuanian institutions. This seems to be a unique feature of the Lithuanian system. Verleysen and Engels (2018), for instance, discuss weight ratios of publication types, but they do not suggest the possibility of taking into account the number of pages of a publication. There is one more distinctive Lithuanian feature in the assessment of book outputs in the sciences . Only books published by prestigious foreign publishers earn points (and funds)––and nothing is earned if a publisher is not prestigious. This has elevated publisher prestige to the utmost importance for Lithuanian institutions. Moreover, while formal national regulations define the notion of a prestigious publisher, the decision whether a specific publisher is prestigious or not depends strongly on the opinion of anonymous experts. Few empirical studies have investigated book assessment based on publisher judgements. Only a handful of papers examine the Norwegian model (see, e.g. Aagaard et al. 2015) or emphasise challenges in verifying the prerequisites for publishers of academic book outputs (Borghart 2013). Several papers flag the unexpected potential consequences of national performance-based funding systems on research practices (Aagaard 2015; Faggiolani & Solimine 2018; Hammarfelt & de Rijcke 2015; Rowlands & Wright 2019). These studies provide a background for more extensive research, particularly regarding the possible implications of assessing books based on their publishers’ prestige. Numerous policies on research assessment, with the goal of promoting ‘excellence,’ incentivise scholars to publish their best research in the most prestigious outlets; these policies are defined at national, regional, and institutional levels. Nevertheless, how such policies actually distinguish prestigious channels has not yet been examined. In this paper, we intend to answer the following research questions: We use the term ‘sciences’ to refer to all fields of research except for the social sciences and humanities. (1) What methods are employed in Lithuania to identify prestigious publishers in the assessment of book outputs, and how do these methods differ from the methods employed in other countries? (2) To what extent do assessments of book outputs based on the prestige of book publishers yield consistent results, both over time and between countries? (3) To what extent is it possible to verify whether book publishers meet the formal prerequisites of national assessments? Using a mixed methods approach, we will explore the different ways in which the prestige of publishers is determined. This will contribute to a deeper understanding of the complexity of the assessment of scholarly books. It will also identify the uncertainties of a process in which books are assessed based on their publisher’s prestige.
2. Research design
This paper presents a case study which employs a combination of qualitative and quantitative methods. A qualitative document analysis was performed to study two related phenomena: the assessment of scholarly book publishers and the methodologies and practices used to determine prestigious publishers . A qualitative analysis of research papers, edited volumes, reports, and grey literature was conducted to identify the regulations, rules, and practicalities related to the assessment of scholarly books and their publishers. We used a snowball method for gathering relevant literature starting from Sivertsen (2018). As an example, the documents, regulations, research papers and other information related to the Norwegian Publication Indicator––as the best-documented indicator followed by other countries––were obtained from its webpage ‘About NPI’ . Rankings of publishers and the publication points earned by Norwegian institutions were taken from the Norwegian Register for Scientific Journals, Series and Publishers . Lithuanian legal acts containing the methodologies for the formal evaluation of research produced by research and higher education institutions were obtained from an official database: the Register of Legal Acts of the Republic of Lithuania (TAR) . For an empirical investigation, we chose the Lithuanian book outputs submitted by institutions to the national metric-based funding systems from 2005 to 2016. We thoroughly examined bibliographic data on registered books assessed by anonymous panel experts. Then, we identified publishers that were awarded both the highest category (prestigious publisher) and at least one other category (not-so-prestigious or non-prestigious). Such discrepancies in judgements about publishers have been discussed within Lithuanian academia at all levels (departments, faculties, and universities) for years. The results are significant for researchers because institutions operate internal incentive schemes reallocating funds received from these annual assessments. However, we only studied issues related to publishers’ prestige. We did not analyse the incentive schemes at different institutional levels. The Norwegian Publication Indicator (NPI) https://npi.nsd.no/informasjon accessed 16 April 2020 The Norwegian Register for Scientific Journals, Series and Publishers https://dbh.nsd.uib.no/publiseringskanaler/Forside accessed 16 April 2020 Bibliographic information on Lithuanian book outputs was derived from databases managed by the Lithuanian Research Council: (1) Dynamics of Lithuanian Research Potential for outputs published from 2004 to 2008; and (2) Reports on Scientific, Arts and other Relevant Activities of Research and Higher Education Institutions for outputs published from 2009 to 2016. From a bibliographical perspective, the compiled records had various shortcomings. So, we enriched the primary bibliographical data with manual searches of the missing details in various catalogues (e.g. the National Bibliographic Database by Martynas Mazvydas, National Library of Lithuania , the Lithuanian Academic Electronic Library , WorldCat catalogue ) and on the web. ISBNs are mandatory prerequisites for scholarly books in almost all countries, including Lithuania, and registrants of ISBNs can be presumed to be responsible for the content they make publicly available. Thus, in the analysed dataset, we recognised the ISBN registrants as the publishers. We chose the Global Register of Publishers (GRP) as a primary, reliable, and free resource about registrants of ISBNs created by the International ISBN Agency . Thus, alongside the enriched bibliographical information for book outputs gathered from the Lithuanian Research Council, we derived further data about publishers of those books from the GRP. A thorough investigation of experts’ comments on 4135 records (Dagienė et al. 2019) revealed that 50 records were conference proceedings published as edited volumes. We excluded these records from the dataset. The final dataset of Lithuanian book outputs (books and edited volumes and no conference proceedings) analysed in this study reflects institutional submissions of 4085 unique titles having ISBN codes published from 2004 to 2016. The experts positively assessed 3712 (out of 4085) reported book outputs and scored them according to their publisher’s prestige. In these cases, the publisher was classified as prestigious or not-so-prestigious. The panels rejected the other 373 titles as inappropriate mostly because the publisher was considered non-prestigious. The analysis in this paper focuses on publishers for which the experts were not consistent, i.e. publishers which some experts classified as prestigious, while other experts considered them to be not-so-prestigious or even not prestigious at all.
3. Defining prestigious book publishers
Numerous research papers confirm that modern research evaluation systems increase the pressure on researchers to publish more and reinforce their ‘publish or perish’ habits, which significantly changes the publishing patterns of both journal papers and scholarly books (Broz & Stöckelová 2018; Butler 2003; Elton 2000; Good et al. 2015; Moed 2008; Osuna et al. 2011; De Rijcke et al. 2016). Some studies show that scholars adjust their behaviour in Mokslo ir studijų institucijų mokslinės, meninės ir su jomis susijusios kitos veiklos ataskaita https://mokslas.lmt.lt/INSTITUCIJOS/ accessed 16 April 2020 National bibliographic database. Martynas Mazvydas National Library of Lithuania https://nbdb.libis.lt/showCustomPage.do?showByIdentificator=complexSearch accessed 16 April 2020 The Global Register of Publishers https://grp.isbn-international.org/ accessed 16 April 2020 response to these research assessments’ requirements, especially when the number of publications is explicitly linked to their research funding. In a qualitative investigation of literature, we found that the regulations surrounding research assessment best reflect the policymakers and scholarly community’s perceptions of the ‘quality’ of book outputs as well as the ‘prestige’ of publishers producing research outcomes. In recently announced regulations on research assessment, countries approaching qualitative assessment do not emphasise book publishers’ prestige. Meanwhile, countries having metric-based funding systems rank publishers and score submitted book outputs according to the levels awarded by experts to publishers.
In countries having peer review based systems.
In the UK, experts assess the quality of research outputs (and books as well) by reading the actual submitted books which institutions select as their best outcomes. The UK Research Excellence Framework (REF) clearly states that these are required as physical books (Rosenberg 2015). According to the REF policies, panel reviewers evaluate three distinct elements for each submission: the quality, the impact beyond academia, and the environment that supports research . Also, there is a statement: “53. No sub-panel will make use of journal impact factors, rankings or lists, or the perceived standing of the publisher, in assessing the quality of research outputs.” (REF2014 2012) Nevertheless, there are several reports with widespread scope commissioned by the UK’s Higher Education Funding Council for England (HEFCE) that investigate metrics and possible changes in the assessment process. The Metric Tide (Wilsdon et al. 2015) discusses book-based indicators, among other metrics, and Crossick (2015) examines the issues around open access for monographs. The latter relates to the intention of policymakers to mandate open access monographs as book outputs in the REF in 2027 (Lockett 2018). There are more independent reports on the REF2014 results. In one, Tanner (2016) provides a thorough analysis of publishing data on books submitted across the arts and humanities––the experts assessed 8,513 books produced by 1,180 unique publishers. It worth emphasising that 60 per cent of the publishers produced only a single submitted book. Tanner’s conclusions include: “As far as can be ascertained from the available data, attempting to assess books through a purely quantitative method would be nigh on impossible to do fairly or equitably. […] This study adds further evidence to the sense that bibliometrics remain a very unhelpful means of analysing books for research excellence.” (Tanner 2016) A general independent review on REF2014 results, widely known as the Stern Review (Stern 2016), includes a recommendation supporting the current peer review based assessment and emphasises that if the metrics are provided to support the assessment, they should be used transparently. Similarly to the UK, currently, it looks like the status of publishers is not a decisive factor in France (Williams & Galleron 2016) or Italy (Basso et al. 2017; Faggiolani & Solimine 2018). Nevertheless, an Italian study has been conducted investigating the possibilities of employing quantitative metrics in the assessment of books (Basso et al. 2017). Researchers conclude that classifying publishers is fraught with difficulty (Williams et al. 2018) and suggest surveying researchers, as was done in Spain (see Sub-section 3.1). Nevertheless, based on a thorough systematic review, Giménez-Toledo et al. (2009) conclude that although there is no simple way to determine the ‘prestige’ of publishers, this is predominant in the research assessment. In countries with metric-based assessment funding system s, the importance of publishers’ prestige varies. While the system in some countries does not judge the publishers, in others, it ranks them into levels from a basic entry level to the most prestigious. In the Czech Republic, the actual publisher of book outputs was of no importance until 2013, when panel peer review evaluation was introduced (Broz & Stöckelová 2018); and, the current formal criteria do not mention the importance of publishers (Government of the Czech Republic 2018). In Poland, researchers can self-publish monographs which meet the formal criteria for metric-based assessment funding (Kulczycki 2018). There are countries with extensive experience in the rankings of publishers. Norway introduced the first and widely documented ranking of publishing channels in 2005 (Sivertsen 2018), Denmark implemented a similar ranking in 2008 , and Finland followed them with a national ranking launched in 2012 . More details of these rankings are presented in Sub-sections 3.2 and 3.3 below. Meanwhile, a slightly different publisher ranking system was established in Flanders, the Northern Dutch-speaking region of Belgium. The Flemish regulations do not mention the publisher’s prestige; they concentrate on the ‘guaranteed peer reviewed content’ (Verleysen & Engels 2013). The national Authoritative Panel, which consists of professors affiliated with Flemish universities with expertise covering the social sciences and humanities, is authorised to evaluate publications channels (journals and books) against the criteria stipulated in the regulations (Verleysen et al. 2014). This panel has found the most challenging aspect of its work is to verify the peer review procedures in the process of book output production. So, the Flemish list of publishers consists of two publisher types––those who handle peer review for all their published books and those who manage peer review for individual books or book series. To find out academics’ perceptions of what exactly determines the quality of publishers of monographs, about three thousand Spanish researchers were surveyed (Giménez-Toledo et al. 2013). Considerable variations were revealed within the criteria for the ‘prestige’ of publishers in different scientific fields. As Giménez-Toledo et al. noticed, some of the leading indicators recognised by researchers (peer review, an ongoing trajectory of publications, publishers’ monographs being in libraries, and in international databases) partially coincide with those indicated by the Spanish research evaluation agencies. However, half of the The BFI is an element of the performance-based model for distribution of the new block grants for research to universities. In: Ministry of Higher Education and Science, Denmark https://ufm.dk/en/research-and-innovation/statistics-and-analyses/bibliometric-research-indicator/bfi-rules-and-regulations accessed 25 April 2020. academics pointed out additional indicators of ‘prestige’, such as books having an adequate structure or publishers maintaining a presence in foreign bookstores. A Spanish information system on publishers entitled the Scholarly Publishers Indicators was created in 2012 as a result of this research. The ranking of publishers was updated after the second survey in 2014. The latest version of the Scholarly Publishers Indicators consists of (1) impressions of the Spanish researchers surveyed in 2014; (2) publishers indexed in the Book Citation Index (Clarivate Analytics) and (3) Scopus (Elsevier); as well as (4) the Finnish and (5) the Norwegian lists of publishers (Elea Giménez-Toledo et al. 2017). The Scholarly Publishers Indicators (in addition to the general impressions from the 2014 survey) allows selecting the most highly valued publishers in sixteen disciplines within the social sciences and humanities. Nevertheless, the Spanish Scholarly Publishers Indicators does not propose any precise meaning of publishers’ ‘prestige’. The Norwegian model, developed for indicator-based funding, incentivises researchers to publish in the most prestigious channels within their field of study (Sivertsen 2018). This model implies that prominent researchers designate which journals and book series that have met the entry requirements (level 1) are considered prestigious (level 2) in their particular area of sciences. Along with research papers, we investigated the regulations published on two separate portals within the Norwegian model. One was taken from the Norwegian Publication Indicator (henceforth referred to as the NPI) (‘About the Norwegian Publication Indicator’ 2020), and the other from the Norwegian Register for Scientific Journals, Series and Publishers (henceforth referred to as the Norwegian Register). Both have interfaces in English and provide extensive information on processes for publisher ranking. According to mandatory regulations declared in the Norwegian Register, to be registered at level 1 (which is the basic entry level), book publishers must submit for primary evaluation: (1) their ISBN prefix; (2) documentation of their scientific publishing programme (not the editorial board), (3) external peer review procedures (an explanation in a PDF file is enough), and (4) proof of their international or national authors (names and affiliations from the last two years). Figure 1 shows prerequisites in the Norwegian Registry for the basic level, level 1 and conditions for level 2. The Scholarly Publishers Indicators (SPI) http://ilia.cchs.csic.es/SPI/indexEn.html accessed 16 April 2020
Figure 1. Prerequisites and approval procedures for publication channels of the Norwegian Register
The prestigious level 2 is limited to channels––journals, book series, or book publishers––that issue ‘the most outstanding works by researchers from different countries.’ While it is unclear how to identify outstanding works, the guidelines state that for calibrating ‘prestige’––‘level 2 publication channels shall together constitute about one fifth (20 per cent) of the field’s total academic publications’ (Sivertsen 2018). However, according to the NPI, the Norwegian Register of academic book publishers is not divided into academic fields. The National Board of Scholarly Publishing ‘is responsible for the publisher rating levels and updates this annually, based on input from academic fields where book publishing is a central or frequent format for publishing research.’ Nevertheless, it seems that level 2 is still limited to leading channels only and has a 20 per cent field-based threshold. Denmark and Finland have implemented the Norwegian model with some adjustments. The main requirements for an entry level listing correspond with those set in Norway (e.g. peer review before publication). Also, as in Norway, local researchers on panels of experts decide which publishers deserve to be designated as prestigious at the highest level. In Denmark, the Bibliometric Research Indicator (BFI), announced in 2008, has two types of lists: one for journals, books, and conference series, and the other for publishers. These BFI lists were each divided into two levels from 2012 to 2017. Since 2018, these lists have three publication levels: level 1 (ordinary), level 2 (particularly distinguished), and level 3 (prestigious). Level 3 includes no book publishers but only the most prestigious journals, book series and conferences. The allocation of levels depends on the recommendations of researchers who serve on expert panels ( Guidelines for registering research for the Danish Bibliometric Research Indicator The National Board of Scholarly Publishing https://npi.nsd.no/organisering/npu?id=1109 accessed 16 April 2020 The Bibliometric Research Indicator (BFI) provides an overview of research publications from Danish universities. https://ufm.dk/en/research-and-innovation/statistics-and-analyses/bibliometric-research-indicator accessed 16 April 2020
The Finnish Publication Forum (henceforth referred to as the Finnish ranking) was launched in 2012 and currently has three levels for book publishers. From 2012 to 2014, book publishers were only distributed between two levels in a particular proportion. In essence, 90 per cent of book publishers were in level 1 and 10 per cent designated as level 2 (prestigious leading publishers). Level 3 was reserved for 25 per cent of level 2 journals and series (not book publishers). Since 2015, some book publishers have been awarded level 3. It worth noting that publishers and the book series they produce are ranked separately––which creates confusion in the rankings. Different countries deal with such issues differently. The Norwegian model allows differences in levels, e.g. a book series can be ranked as level 2, and its publisher ranked as level 1.On the contrary, the Finnish ranking determines that (even if ranked at level 1 before) a book publisher must be ranked at level 2 if one of its book series is assigned level 2. Despite this, the designers of the Finnish ranking warn: ‘the quality levels applied in the Publication Forum predict the average quality and impact of large publication volumes, but they are too arbitrary a tool for the evaluation of individual publications or researchers’ (Auranen & Pölönen 2012). Lithuania, being a small post-Soviet country, introduced a metric-based funding system in 2005, the year in which Norway launched its Norwegian Register. Since 2005, the Lithuanian regulations have defined ‘prestigious publishers’ as publishers which (1) continually release publications authored by national and international researchers, (2) distribute their products in many countries, (3) issue globally recognised journals and series of books, and (4) provide enough information about these achievements on their websites. According to the formal Lithuanian definition, prestigious publishers should fulfil all four listed above criteria. In 2006, designers of Lithuania’s national performance-based funding system changed some rules. They explained the aims of these amendments, declaring that they seek to incentivise research institutions to work efficiently, to raise their international competitiveness, and to comply with the needs of the state. Within this order for outputs in the sciences (and not for the social sciences and humanities), policymakers introduced the List of Globally Recognised Publishers. It comprised sixteen named book producers such as Elsevier Science Group, Springer Group, Oxford and Cambridge University Presses, and other similar publishers. We want to explicitly draw attention to the fact that publishers of book outputs in the sciences were assessed based on their journal publishing activities . The List of Globally Recognised Publishers concluded by stating that prestigious publishers of monographs are those publishers which issue at least five peer reviewed journals indexed in the Journal Citation Reports (Clarivate Analytics, the former Institute of Scientific Information). Officially, the list was revoked soon after, in 2010. Publication Forum The Ministry of Education and Science of the Republic of Lithuania, 12 April 2006 order No ISAK-685 (Valid from 7 May 2006, not valid from 19 July 2009) peer review assessment (for 20 per cent of institutional outputs) and formal metric-based assessment (for the remaining eligible pieces of research). The Research Council of Lithuania administers this ex-post evaluation. It assigns self-registered senior researchers (henceforth referred to as the experts) into the pool for two expert panels (one for the sciences and the other for the social sciences and humanities). The experts have not been named; they work anonymously. According to the Lithuanian regulations for the peer review part of the assessment, the experts evaluate the quality of research presented in a book. In contrast, in the formal metric-based part of the assessment, they should appraise only the publisher of a book and the bibliographic information. In the next section, we present a detailed analysis of the assessment of books in the Lithuanian system.
4. Inconsistencies in assessing the prestige of book publishers
In Lithuania, for research outputs in the sciences, only monographs and chapters in edited volumes produced by prestigious publishers are eligible for formal assessment and for being assigned to the only existing level, prestigious (otherwise, they are rejected and not scored). In contrast, for the outputs in the social sciences and humanities, scholarly books issued by any publishers are eligible, but books produced by prestigious publishers (level 2) earn several times more points than those at level 1. When setting out to investigate the Lithuanian case, we selected publishers for which experts did not agree on their prestige. In this way, we identified sixteen inconsistently assessed publishers (and thirteen self-published cases ) that issued multiple books (either in the sciences or in the social sciences and humanities). This means the same publisher was sometimes assessed by anonymous experts as prestigious, while in other cases it was assessed as non-prestigious. These publishers issued a total of 224 books submitted for evaluation by the Lithuanian Research Council expert panels. The experts designated 93 books as published by prestigious publishers. They also rejected 63 books published by the same publishers, thus allocating these publishers both to the highest and to the lowest categories. The remaining 68 books produced by the same publishers were submitted to the not-so-prestigious publisher category (allowed only for books in the social sciences and humanities). Table 1 shows the publishers ranked by experts as both prestigious and at least one other category, that is, not-so-prestigious (but sufficient for the social sciences and humanities) or Using the GRP, thirteen self-publishers were identified based on their ISBNs. According to the Lithuanian formal regulations, self-published works do not qualify for submission to metric-based assessments, neither do those having no ISBNs or misleading ISBNs.
Table 1. Publishers awarded by the Lithuanian experts to the highest category and at least one other category in the national research funding assessments from 2004 to 2016, also the levels and years the same publishers were awarded in the Norwegian Register.
The titles on publishers identified by ISBNs in the Global Register of Publishers (country ISBNs were registered) The number of books evaluated by the experts and categories publishers received within the years The scientific level of the publisher and the years from the Norwegian Register Prestigious publisher (for all sciences) ** Not-so-prestigious, lower category (for SSH, scored) Not prestigious, outcomes rejected (for the sciences) In Tech d.o.o. (Croatia) 16 | 2010 – – – – OmniScriptum GmbH & Co. KG (Germany) 20 26 18 – VDM Verlag Dr Müller* 5 | 2008 – – – level 1 | 2008 – – – – – - – Scholars’ Press* – – – – GlobeEdit* – – – Nova Science Publishers, Incorporated (the USA) 9 | 2009 – – – – – – – Authors or miscellaneous publishers (Austria, Australia, Estonia, Finland, Germany, Latvia, and Lithuania) 2 | 2009, 2011 9 | 2004 – – Studium Press, LLC (the USA) 1 | 2014 – – – – Herder-Institut e.V. (Germany) 1 | 2015 – – – Shaker Verlag (Germany) 1 | 2017*** – – Cambridge Scholars Publishing (the UK) 20 | 2010 – – – level 1 | 2006-2018 Hermann (France) 1 | 2017*** 1 | 2013 – level 1 | 2009 – * Publishers as they are named in the bibliographic data of books submitted to the assessment in Lithuania. ** The Lithuanian level ‘prestigious publishers’ is equal to level
By checking the GRP, data on registrants of ISBNs reveals quite a striking finding. It appears that the publisher listed in a library catalogue (which shows the bibliographical information of a book which librarians usually take from the book’s copyright page) and the registrant of the ISBN do not necessarily match. Apparent discrepancies between these are detailed below. The first finding reveals that some publications (whose copyright pages state that the publishers are universities or academic institutions) were actually identified by the GRP as self-published. According to the GRP, the registrants of those books are author-publishers (or miscellaneous publishers), who may have been assigned just one or two ISBNs each. Of these thirteen self-published books, seven were positively assessed, with two ranked as published by prestigious publishers. A second noteworthy finding was that a German-based company, OmniScriptum GmbH & Co KG, was indicated in the data as six separate publishers (compiled in the first column and marked with an asterisk in Table 1). Nevertheless, these imprints do not exist in the GRP, despite OmniScriptum having identified every imprint separately on its webpage. The Lithuanian experts awarded two imprints––Verlag Dr Müller (VDM) and Lambert Academic 2 Publishing (LAP)––the highest category from 2009 to 2011. Currently, both imprints are included in the Beall’s List of vanity presses . Thirdly, book outputs produced by LAP were both rejected as inappropriate and awarded the highest category, within the same years. An in-depth investigation of the disagreements among the Lithuanian experts revealed a quite complicated situation. In 2009, both panels (in the sciences and the social sciences and humanities) decided that LAP was a prestigious publisher and book outputs received maximum points. Nevertheless, after a year, in 2010, the panel in the sciences rejected the book outputs issued by LAP as produced by a non-prestigious publisher (and institutions received zero points). At the same time, the panellists in the social sciences and humanities awarded the books issued by LAP the highest category (and institutions received the maximum points). Then, in the assessment of 2011, both panels decided that LAP was a prestigious publisher, and six books submitted in the sciences as well as in the social sciences and humanities received the maximum points again. A turning point occurred in 2012 when the submission of monographs published by LAP doubled. Regrettably for the institutions (which were aware that the experts treated LAP as a prestigious publisher in the preceding year), the experts on both panels decided that LAP was not prestigious any more. So, book outputs in the sciences were rejected (meaning no points), while those in the social sciences and humanities were awarded level 1 (meaning fewer points). There are more inconsistencies in the experts’ decisions. The next example deals with the status of InTech d.o.o. The Lithuanian experts scored maximum points for sixteen outputs (in the sciences) published by InTech d.o.o. over 2010-11. However, from 2012 to 2016, the experts decided that 25 outputs produced by this publisher were inappropriate because a non-prestigious publisher issued them. Thus, these 25 were rejected, and institutions did not receive the points (and funding) they expected, even though they had been incentivised to publish with this book producer one year earlier. Nevertheless, Lithuanian institutions received their points for chapters (in four edited volumes published by the same InTech d.o.o.) as outputs in the social sciences and humanities at the lower category (not-so-prestigious publishers)––where any publisher is eligible. Significant changes in circumstances regarding a publisher’s prestige have surrounded the widely known UK-based publisher Cambridge Scholars Publishing (formerly Cambridge Scholars Press Ltd. ). It is interesting to note that the Lithuanian experts designated twenty of its titles as being produced by a prestigious publisher from 2010 to 2016. At the same time, four books were classified as issued by a not-so-prestigious one (so institutions received fewer points). In 2018, the Lithuanian experts awarded Cambridge Scholars Publishing the highest level, deeming it a prestigious publisher in the humanities. There is significant controversy around Cambridge Scholars Publishing. In France, when describing the inevitable confusion about some publishing houses’ value, interviewed researchers mentioned Cambridge Scholars Publishing: The Beall’s List of vanity presses.
What is vanity press ? https://beallslist.net/vanity-press/ In Norway, Cambridge Scholars Publishing had level 1 status, and it was ninth on the list of top ten publishers, covering 25 per cent of all scholarly book outputs published in international languages in the social sciences and humanities between 2005 and 2009 (Sivertsen & Larsen 2012). However, Cambridge Scholars Publishing received level 0 status in 2019, although Norwegian scholars still publish their works with this publisher, which is confirmed by significant numbers of production points registered in the Norwegian Publication Indicator . In the Danish BFI lists, Cambridge Scholars Publishing first appeared in 2011 (BFI lists had no levels for book publishers at that time), then it became a level 1 publisher over 2012-13, and has disappeared from the BFI lists since 2014. In Finland, researchers on panels also assigned Cambridge Scholars Publishing to the basic level 1 status. In the Flemish database (VABB-SHW), this book producer is indicated as a level 0 publisher, having only some ISBN titles peer reviewed. Another striking instance occurred in the Lithuanian data when a publisher became prestigious in the sciences category within a year. In 2017, the unnamed experts on the sciences panel rejected a monograph published (in 2016) by Germany-based publisher Shaker Verlag on the basis that the publisher was not prestigious. In 2018, the experts (we do not know if these were the same anonymous experts) selected Shaker Verlag as prestigious in the sciences. In Norway, Finland, and Denmark, this book producer has a level 1 publisher status. In the Flemish database, Shaker has a level 0 publisher status. Since 2018, the Research Council of Lithuania has distributed three separate lists of book publishers determined as prestigious in the sciences (nine publishers), the social sciences (eleven publishers) , and the humanities (twenty-three publishers) on its website. However, this does not explain if books produced by these prestigious publishers in subsequent years would receive maximum points as well––uncertainties for the submitting institutions still exist. Additionally, the results presented in the last column of Table 1 indicate if the publisher was ranked in the Norwegian Register and if so, the years it happened. Level 0 means that REF2014, Research outputs (REF2) https://webarchive.nationalarchives.gov.uk/20170302140351/http://results.ref.ac.uk/DownloadSubmissions/ByForm/REF2 accessed 20 April 2020 Cambridge Scholars Publishing in The Norwegian Publication Indicator https://npi.nsd.no/forlagoversikt/forlaginfo/faglig?forlagId=19631
5. Verifying whether book publishers meet basic assessment prerequisites
We reviewed the above-analysed international publishers listed in Table 1 to see whether we could verify that publishers meet the minimum requirements for entry into the Norwegian Register and the Lithuanian formal definition of prestigious publishers set in the regulations. The importance of making the right decision regarding book publishers is more prominent for scholars in countries approaching metric-based assessment systems (e.g. Norway or Lithuania) than in countries having peer review evaluation (e.g. the UK). As an example, if Lithuanian researchers published with the ‘wrong’ publishers, institutions would not earn funding for research (currently this system is used to allocate almost half of the governmental funds for research). Table 2 shows data on publishers anyone could collect from various sources. The first column shows how transparent and identifiable publishers are (to be discussed in Sub-section 5.1). The middle columns give the minimum necessary prerequisites for inclusion into the Norwegian, Finnish, and Danish rankings of publishing channels (to be discussed in Sub-section 5.2). The last section offers some insight into how the publishers fit the Lithuanian explanation of ‘prestige’ (to be discussed in Sub-section 5.3). To be exact, we assumed that if publishers are international, they must provide information about the distribution of their books, their policies, and their authors on their websites in internationally understandable languages. 5
Table 2. Publishers which Lithuanian panel experts awarded the highest and at least one other category compared with the metric-based assessments by various other countries (data as of 19 April 2019)
The transparency of the publishers The minimum requirements for entry to the national registries, and the levels publishers awarded by 2019 The Lithuanian description of prestigious publishers ‘Registrant name’ in the Global Register of Publishers | ‘Publisher’ on the website (if different) | a country of ISBNs, declared, year established Policy on peer review of books Editorial/ Advisory Board Author-ship
Publisher’s level in the national system in
Publish book series or journals Languages of the content provided on the website Norway Finland Denmark Flanders 1 2 3 4 5 6 7 8 9 10
In Tech d.o.o. (Croatia, 2007) = IntechOpen (the UK, 2017 ) yes | COPE* yes Int’l level 1 2007 – level 0 – – book series English OmniScriptum GmbH & Co. KG (18 academic brands, Germany, 2002)
VDM Verlag – – Int’l level 1 2008-10** level 1 2012-14 – – ? English LAP Lambert Academic Publishing – – Int’l level 0 level 0 – – ? English Nova Science Publishers**** (the USA, 1985) yes ? *****
Int’l level 1 2005-17 level 1 2012-19 level 1*** 2008-19
ISBN-selection journals, book series English
Peter Lang (the USA, Switzerland, 1970) yes – Int’l level 1 2004-19 level 1 2012-19 level 1 2008-19
ISBN-selection journals, book series English
Studium Press (the USA, India, 1980) – – ? level 0 – – – book series English Begell House (the USA, 1991) – ? Int’l level 1 2004-19 level 1 2012 – level 1 2008-19 – journals, book series English, six more languages Herder-Institut (Germany, 1990) – – ? level 0 level 1 2014-19 – – journal, book series German, English IGI Global**** (the USA, 1998) yes
COPE ?
Int’l level 1 2007-19 level 1 2012-15 and 2018-19 level 1 2008-19 to employ peer review for all books journal, book series English, Chinese
Shaker Verlag (Germany, 1986) – – Int’l level 1 2005-19 level 1 2012-19 level 1 2008-19
ISBN-selection book series German, English, Dutch
Cambridge Scholars Publishing (the UK, 2001) – ? Int’l level 1 2006-18 level 1 2012-19 level 1 2011-13
ISBN-selection book series English
Hermann (France, 1876) – – ? level 1 2009-19 level 1 2015-19 level 1 2008-19 – book series French * Publisher is a member of the Committee on Publication Ethics (COPE). ** as VDM Verlag Dr Müller Aktiengesellschaft & Co. KG. in the Norwegian Register. VDM Verlag relaunched as OmniScriptum in 2013. *** The BFI list of publishers in Denmark had no levels for book publishers from 2008 to 2010. Two levels (1 and 2) for book publishers have launched since 2012. **** Publishers included into the Beall’s List of vanity presses https://beallslist.net/vanity-press/ assessed 9 June 2020. ***** ‘?’ means that we were not abl e to find information on authorship or about an editorial/advisory board. To interrogate the data further, we chose the Principles of Transparency and Best Practice in Scholarly Publishing developed by well-known scholarly organisations as the primary standard. As an example, in a study on scholarly journals’ compliance with this standard (Choi et al. 2019), the sixteen principles of the standard were sub-divided into 33 items in four different categories: (1) basic journal information, (2) publication ethics information, (3) copyright and archiving information, (4) profit model information. For the analysis presented in this sub-section, we adjusted the items proposed in the category ‘basic journal information’. ISBNs and publishers . The regulations mandate ISBNs for book outputs in Norway, Finland, Denmark, Flanders, and Lithuania. According to the International ISBN Agency, it is always Principles of Transparency and Best Practice in Scholarly Publishing. Developed by the Committee on Publication Ethics (COPE), the Directory of Open Access Journals (DOAJ), the Open Access Scholarly Publishers Association (OASPA), and the World Association of Medical Editors (WAME). The third version published on 15 January 2018 http://wame.org/principles-of-transparency-and-best-practice-in-scholarly-publishing accessed 16 May 2020. . Thus, we examined the transparency of publishers, comparing the ‘registrant name’ in the GRP with the ‘publisher’ as it appears on the website it owns. We supposed that the publishers are transparent for authors, readers, and evaluators when this information matches. The data show that two (out of eleven) publishers have a different presence on their webpages and the GRP: In Tech d.o.o. (named IntechOpen on its website) and OmniScriptum (which has numerous brands recognised on its website). The publisher In Tech d.o.o. would look inconsistent for some researchers and the panel experts who assess its book outputs because searches in the GRP (by the prefix ‘978-953-307’ of Lithuanian outputs’ ISBNs) produce the publisher In Tech d.o.o. based in Croatia. However, the URL (provided on the GRP) directs users to the IntechOpen website , which declares only its headquarters in the UK (‘About IntechOpen’). Nevertheless, the ‘Contacts’ page reveals that IntechOpen has two offices: In Tech d.o.o. in Croatia (registered in 2007 ) and IntechOpen Limited in the UK (registered in 2017 ). Despite its achievements and membership of COPE and OASPA (both being developers of the principles of transparency), IntechOpen stands as a level 1 publisher in Norway, a level 0 in Finland, and has no level in Denmark and Flanders. Researchers and assessment panel experts would have more doubts regarding the publisher OmniScriptum. In the Lithuanian dataset, OmniScriptum consists of six imprints. However, only two of them (those awarded the prestigious category in Lithuania and listed in the Norwegian Register) are included in Table 2. At its origins, it was VDM Verlag launched in 2002 and relaunched as ‘OmniScriptum’ in 2013. Lambert Academic Publishing (LAP) is another brand of OmniScriptum, about which searches on the internet reveal claims it is a predator, vanity press , or at least questionable (Broz & Stöckelová 2018). Currently, OmniScriptum openly declares its policies and business models on its website: ‘Yes, we are aware of the criticism towards OmniScriptum that can be found on the web. […] Our company has changed tremendously in the last years. We have changed our business (no more Wikipedia since ages), we have changed our publishing terms, we have even changed our name. Just to clarify – we are OmniScriptum! […] Meanwhile our publishing group incorporates more than 45 imprints’. The criteria for inclusion of new scientific publication channels into the Norwegian Register are like those required in Finland, Denmark, and Flanders. Book publishers should have The Beall’s List of vanity presses.
What is vanity press ? https://beallslist.net/vanity-press/ In the Norwegian Register, the procedures for processing new submissions include: “... New scientific publication channels can be submitted continuously. ... Submissions from commercial publishers will not be considered.” https://dbh.nsd.uib.no/publiseringskanaler/OmProsedyrer Accessed on 20 April 2020. ––necessary procedures for external peer review in book publishing ––is essential, as it usually takes place in journal publishing. However, independent academic book publishers operate differently (Derricourt 2012); this is why we looked for the policies on peer review practices on the publishers’ websites (Table 2, column 2). We found that only four publishers make publicly available their statements or descriptions about their peer review procedures, the main requirement for publishers accepted for the entry level into these four publisher assessment systems. The second prerequisite ––a required advisory board of academics ––is declared as a list of people only on the IntechOpen website (Table 2, column 3). The symbol ‘–’ means that we did not find any advisory board on the publisher’s website. The symbol ‘?’ means that publishers do not publish who is on their advisory board; instead, they list authors, editors, and reviewers (in some cases) in one general list (e.g. Nova Science Publishers, Begell House). Alternatively, Cambridge Scholars Publishing lists 130 boards in the physical sciences and 102 boards in the social sciences on its website. It is challenging to conduct the assessment of the presence of an editorial or advisory board or scientific committee on the websites of book publishers because of the numerous practices book publishers have in place. The third prerequisite ––an international or at least a national authorship the levels the publishers gained in the national systems to compare our findings on the fulfilment of compulsory requirements with the levels the publishers were ranked at (Table 2, columns 5-8). There are some disparities in the levels of IntechOpen and the Herder Institute in their rankings in Norway and Finland, and it is not 8 clear if these publishers had actually been considered and had not received any level in Denmark. However, LAP and Studium Press were not approved unanimously in all countries. The results suggest that some publishers (e.g. Begell House, Cambridge Scholars Publishing, Hermann, or Shaker Verlag) have no verifiable mandatory prerequisites in place. Unfortunately, in some cases, we were not able to find a straightforward way to verify if the book publisher complies with the minimum necessary prerequisites; thus transparent verification of compliance with the prerequisites is not possible. Notwithstanding some publishers without particularly high results in other countries were designated as prestigious in Lithuania, which prompts examination of the formal national definition of prestigious publishers.
The Lithuanian regulations of the formal assessment define prestigious publishers as publishers which continually (1) release publications authored by national and international researchers, and (2) distribute their products in many countries. Moreover, prestigious publishers are classified as such when they (3) issue globally recognised journals and series of books, and (4) provide sufficient information about these achievements on their websites. The first feature–– international authorship ––is a similar prerequisite for level 2 or level 3 publishers in the above-discussed national rankings (in Norway, Denmark, and Finland) which define this as less than two thirds of authors from the same country (Table 2, column 2). Nonetheless, Lithuanian policymakers do not specify that authorship should be ‘international’. The second attribute–– distribution of books in many countries ––does not make publishers unique because currently many publishers (and all we investigated in this study) distribute books they produce through their own websites, Amazon, or other vendors. Hence, we did not include this piece of additional information in the table. The third quality–– issue globally recognised journals and series of books ––is somewhat ambiguous. It seems strange to judge book publishers according to their journal activities, as well as to decide whether their journals and book series are globally recognised (because the formal regulation does not explain how to measure the level of recognition). Likewise, uncertainty is left regarding publishers which issue only book series (and no journals) as to whether such publishers can be prestigious or not. Therefore, the results compiled in column 9 (Table 2) suggest that the production of a book series would be enough for publishers to be awarded the prestigious rank. Notably, LAP (an imprint of OmniScriptum) announces on its website that its main targets are theses and dissertations (and we did not recognise series on its website). Nonetheless, the experts scored books they produced as those published by prestigious publishers, and with such a decision, they have created incentives for researchers to publish with this publisher. The concluding requirement–– provide sufficient information about all achievements on their website s––seems rational because achievements (if the formal regulations specify them)
6. Discussion and conclusions
The document analysis revealed that the evaluation of book outputs is debated at length in research papers studying national practices or examining indicators for the assessment of books. As an example, in the UK, which recently has a qualitative peer review assessment, researchers debate the benefits of a metrics-based approach versus a peer review approach (Allen & Heath 2013). Meanwhile, policymakers, exploring ways to extend the possibilities for evaluating books, have introduced some pioneering prerequisites such as open access for monographs, which have already been widely discussed (Crossick 2015; Lockett 2018). At the same time, in countries using a quantitative assessment system, researchers warn for the effects of quantitative research assessment on research practice, for instance, because some institutions reward individual researchers using metrics that were originally intended to be used only at the institutional level (Aagaard 2015; Hammarfelt & de Rijcke 2015)(Hammarfelt & de Rijcke 2015). Additionally, Rowlands and Wright (2019) pose an unresolved question: ‘why was there so little resistance to the BFI [publishers’ lists in Denmark] despite it being universally disliked among almost all those who knew about it’? Intriguingly, Mouritzen and Opstrup (2020) explain the Danish Bibliometric Research Indicator with models of gaming. Our findings show that no matter whether countries employ qualitative peer review or quantitative metrics-based assessment, they still use experts’ knowledge in the assessment of book outputs. In a qualitative assessment, experts assess the submitted books individually, while in a metrics-based evaluation, empowered researchers select the most prestigious book publishers from all publishers that are considered to meet certain basic entry requirements. As our examination of various national assessment systems reveals, rankings of publishers rely on three main prerequisites––external peer review procedures, a scientific publishing 0 programme or advisory board, and national or international authorship. However, as our findings show, rankings do not disclose the details of their approval procedures, and neither do book publishers make the relevant information publicly available. Hence, there is little transparency in the process of determining whether publishers meet the minimum requirements for entry into a national register. Furthermore, our results show that experts in different countries may have contradictory opinions on the prestige of a publisher. The same publisher may be ranked differently in the Danish, Finnish, Norwegian, and Lithuanian registers. Also, within Lithuania, the same publisher may be ranked as prestigious in one year and as satisfactory or even ineligible in the next year. These findings indicate that it is difficult to reach a common understanding of what it means to be a prestigious publisher. This raises doubts about whether assessments of books based on a judgement about their publisher yield acceptable outcomes. It strikes us that neither quantitative nor qualitative assessment approaches stress the importance of the dissemination of research published in books. Rankings of book publishers are focused on the gatekeeping and content quality expected from publishers and do not assess how publishers contribute to the dissemination of academic research and scholarship. As an exception, the Lithuanian regulations mention dissemination as a feature of prestigious publishers, but there is no further explanation of what is meant or required. By dissemination, we mean the efforts publishers make for an innovative distribution of books (e.g. digital formats, completeness of metadata in catalogues, among other aspects) to ensure a wide dissemination and lasting impact for research presented in the books they produced. Future research may focus on developing improved approaches for assessing books. Our suggestion would be to start from the idea that there are two core roles of book publishers–– gatekeeping (or quality control) and dissemination ––and that there is a need for publishers to be transparent about the services they deliver in each of those areas. In other words, to be a publisher that is regarded as very important for communication between researchers. Publishers may decide how much they want to offer in each of these two areas. Presumably, different publishers will make different choices in that respect. If publishers are transparent and indicate what they offer in terms of quality control and dissemination, national assessment systems can make use of this information. Ideally, the information is provided at the level of individual books, so that there is no need to rely on general information about publishers.
Acknowledgements
The cleansing of the primary bibliographical data on Lithuanian book outputs, which consisted of many missing or incorrect ISBNs codes, would have been not successful without the generous assistance of Aldona Barodicaitė, head of the Lithuanian ISBN and ISMN Agency. The finishing touches were added by Stella Griffiths, executive director of the International ISBN Agency, who helped to solve several misleading cases and explained the subtleties of the ISBN world. This increased my confidence in analysing the final empirical data and has prompted new research topics. I am grateful for the exceptional support of my supervisor, Ludo Waltman. His knowledge and exacting attention to this research project have been an inspiration and have improved this study in innumerable ways. Moreover, his generosity and expertise saved me from many 1 mistakes. I am also grateful to Julie Martyn, a professional language editor at Grammarfun, who kept this paper on track from the first draft.
References
Aagaard, K. (2015). ‘How incentives trickle down: Local use of a national bibliometric indicator system’,
Science and Public Policy , 42/5: 725–37. DOI: 10.1093/scipol/scu087 Aagaard, K., Bloch, C., & Schneider, J. W. (2015). ‘Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator’,
Research Evaluation , 24/2: 106–17. DOI: 10.1093/reseval/rvv003 ‘About the Norwegian Publication Indicator’. (2020). . Retrieved April 23, 2020, from The British Journal of Politics and International Relations , 15/1: 147–62. DOI: 10.1111/1467-856X.12006 Auranen, O., & Pölönen, J. (2012). Classification of scientific publication channels: Final report of the Publication Forum project (2010–2012) . Federation of Finnish Learned Societies . Basso, A., Galleron, I., Lippiello, T., & Geoffrey, W. (2017). The role of books in non-bibliometric areas (ROBINBA) . Borghart, P. (2013). ‘A label for peer-reviewed books? Some critical reflections’, Learned Publishing , 26/3: 167–71. DOI: 10.1087/20130303 Broz, L., & Stöckelová, T. (2018). ‘The culture of orphaned texts’, Aslib Journal of Information Management , 70/6: 623–42. DOI: 10.1108/AJIM-03-2018-0063 Burrows, R. (2012). ‘Living with the H-Index? Metric assemblages in the contemporary academy’, The Sociological Review , 60/2: 355–72. DOI: 10.1111/j.1467-954X.2012.02077.x Butler, L. (2003). ‘Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts’, Research Policy , 32/1: 143–55. DOI: 10.1016/S0048-7333(02)00007-0 Choi, H. W., Choi, Y. J., & Kim, S. (2019). ‘Compliance of “Principles of transparency and best practice in scholarly publishing” in academic society published journals’, Science Editing , 6/2: 112–21. DOI: 10.6087/kcse.171 Crossick, G. (2015). Monographs and Open Access A report to HEFCE ISSI 2019: 17th International Conference of the International Society for Scientometrics and Informetrics, 2-5 September, Rome, Italy , pp. 2716–7. Leuven: International Society for Scientometrics and Informetrics. Derricourt, R. (2012). ‘Peer review: fetishes, fallacies, and perceptions’, Journal of Scholarly Publishing , 43/2: 137–47. DOI: 10.3138/jsp.43.2.137 Elton, L. (2000). ‘The UK Research Assessment Exercise: unintended consequences’, Higher Education Quarterly , 54/3: 274–83. Blackwell Publishers Ltd. DOI: 10.1111/1468-2273.00160 Faggiolani, C., & Solimine, G. (2018). ‘Mapping the role of the book in evaluation at the individual and department level in Italian SSH. A multisource analysis’. The Evaluation of Research in Social Sciences and Humanities , pp. 33–53. Springer International Publishing: Cham. DOI: 10.1007/978-3-319-68554-0_2 Fyfe, A., Coate, K., Curry, S., Lawson, S., Moxham, N., & Røstvik, C. M. (2017). ‘Untangling academic publishing: a history of the relationship between commercial interests, academic prestige and the circulation of research’, Zenodo , May: 24. DOI: 10.5281/zenodo.546100 Giménez-Toledo, E., Tejada-Artigas, C., & Manana-Rodriguez, J. (2013). ‘Evaluation of scientific books’ publishers in social sciences and humanities: Results of a survey’, Research Evaluation , 22/1: 64–77. DOI: 10.1093/reseval/rvs036 Giménez-Toledo, Elea, Mañana-Rodríguez, J., Engels, T. C. E., Ingwersen, P., Pölönen, J., Sivertsen, G., Verleysen, F. T., et al. (2016). ‘Taking scholarly books into account: current developments in five European countries’, Scientometrics , 107/2: 685–99. DOI: 10.1007/s11192-016-1886-5 Giménez-Toledo, Elea, Mañana-Rodríguez, J., & Sivertsen, G. (2017). ‘Scholarly book publishing: Its information sources for evaluation in the social sciences and humanities’, Research Evaluation , 26/2: 91–101. Oxford University Press. DOI: 10.1093/reseval/rvx007 Giménez-Toledo, Elea, Mañana-Rodríguez, J., & Tejada-Artigas, C.-M. (2015). ‘Review of national and international initiatives on books and book publishers assessment’, El Profesional de la Información , 24/6: 705. DOI: 10.3145/epi.2015.nov.02 Giménez-Toledo, Elea, & Román-Román, A. (2009). ‘Assessment of humanities and social sciences monographs through their publishers: a review and a study towards a model of evaluation’, Research Evaluation , 18/3: 201–13. DOI: 10.3152/095820209X471986 Good, B., Vermeulen, N., Tiefenthaler, B., & Arnold, E. (2015). ‘Counting quality? The Czech performance-based research funding system’, Research Evaluation , 24/2: 91–105. DOI: 10.1093/reseval/rvu035 Government of the Czech Republic. (2018). Methodology for evaluating research organisations and research , development and innovation purpose-tied aid programmes approved under Czech Government Resolution No . 107 of 8 February 2017 . , p. 45. Prague: Office of the Government of the Czech Republic. Guidelines for registering research for the Danish Bibliometric Research Indicator . (2019). . Hammarfelt, B., & de Rijcke, S. (2015). ‘Accountability in context: effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University’, Research Evaluation , 24/1: 63–77. DOI: 10.1093/reseval/rvu029 Kulczycki, E. (2018). ‘The diversity of monographs: changing landscape of book evaluation in Poland’, Aslib Journal of Information Management , 70/6: 608–22. DOI: 10.1108/AJIM-03-2018-0062 Lamont, M. (2009). How Professors Think . How Professors Think . Harvard University Press. DOI: 10.4159/9780674054158 Lewis, J. S. (2000). ‘An Assessment of publisher quality by political science librarians’, College & Research Libraries , 61/4: 313–23. DOI: 10.5860/crl.61.4.313 Lockett, A. (2018). ‘Monographs on the move?: a view on “decoupling” and other prospects’, Insights the UKSG journal , 31. DOI: 10.1629/uksg.435 Metz, P., & Stemmer, J. (1996). ‘A Reputational Study of Academic Publishers’, College & Research Libraries , 57/3: 234–47. DOI: 10.5860/crl_57_03_234 Moed, H. F. (2008). ‘UK Research Assessment Exercises: Informed judgments on research quality or quantity?’, Scientometrics , 74/1: 153–61. Springer Netherlands. DOI: 3 10.1007/s11192-008-0108-1 Moore, S., Neylon, C., Paul Eve, M., Paul O’Donnell, D., & Pattinson, D. (2017). ‘“Excellence R Us”: university research and the fetishisation of excellence’, Palgrave Communications , 3/1: 16105. Nature Publishing Group. DOI: 10.1057/palcomms.2016.105 Mouritzen, P. E., & Opstrup, N. (2020). Performance management at universities: the Danish Bibliometric Research Indicator at work . Palgrave Macmillan UK. Nederhof, A. J., Luwel, M., & Moed, H. F. (2001). ‘Assessing the quality of scholarly journals in linguistics: an alternative to citation-based journal impact factors’, Scientometrics , 51/1: 241–65. DOI: 10.1023/A:1010533232688 Neville, T. M., & Henry, D. B. (2014). ‘Evaluating scholarly book publishers—a case study in the field of journalism’, The Journal of Academic Librarianship , 40/3–4: 379–87. Elsevier Inc. DOI: 10.1016/j.acalib.2014.05.005 Osuna, C., Cruz-Castro, L., & Sanz-Menéndez, L. (2011). ‘Overturning some assumptions about the effects of evaluation systems on publication performance’, Scientometrics , 86/3: 575–92. DOI: 10.1007/s11192-010-0312-7 REF2014. (2012). Panel criteria and working methods Contents . De Rijcke, S., Wouters, P. F., Rushforth, A. D., Franssen, T. P., & Hammarfelt, B. (2016). ‘Evaluation practices and effects of indicator use-a literature review’, Research Evaluation , 25/2: 161–9. DOI: 10.1093/reseval/rvv038 Rosenberg, G. (2015). Research Excellence Framework 2014: Manager’s report Studies in Higher Education , 0/0: 1–15. Taylor & Francis. DOI: 10.1080/03075079.2019.1706077 Sivertsen, G. (2018). ‘The Norwegian Model in Norway’, Journal of Data and Information Science , 3/4: 2–18. DOI: 10.2478/jdis-2018-0017 Sivertsen, G., & Larsen, B. (2012). ‘Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential’, Scientometrics , 91/2: 567–75. DOI: 10.1007/s11192-011-0615-3 Stern, N. (2016). Building on Success and Learning from Experience. An Independent Review of the Research Excellence Framework. Tanner, S. G. (2016). An analysis of the Arts and Humanities submitted research outputs to the REF2014 with a focus on academic books (unknown). An Academic Book of the Future . King’s College London. Retrieved from Journal of the American Society for Information Science and Technology , 64/2: 428–30. DOI: 10.1002/asi.22836 ——. (2018). ‘How arbitrary are the weights assigned to books in performance-based research funding? An empirical assessment of the weight and size of monographs in Flanders’, Aslib Journal of Information Management , 70/6: 660–72. DOI: 10.1108/AJIM-05-2018-0110 Verleysen, F. T., Ghesquiere, P., & Engels, T. C. E. (2014). ‘The objectives, design and selection process of the Flemish Academic Bibliographic Database for the Social Sciences and Humanities (VABB-SHW)’. Bibliometrics Use and Abuse in the Review of Research Performance , Vol. c, pp. 117–27. White, H. D., Boell, S. K., Yu, H., Davis, M., Wilson, C. S., & Cole, F. T. H. (2009). 4 ‘Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences’, Journal of the American Society for Information Science and Technology , 60/6: 1083–96. DOI: 10.1002/asi.21045 White, H. D., & Zuccala, A. A. (2018). ‘Libcitations, WorldCat, cultural impact, and fame’, Journal of the Association for Information Science and Technology , 69/12: 1502–12. Wiley-Blackwell. DOI: 10.1002/asi.24064 Williams, G., Basso, A., Galleron, I., & Lippiello, T. (2018). ‘More, less or better: the problem of evaluating books in SSH research’. The Evaluation of Research in Social Sciences and Humanities , pp. 133–58. Springer International Publishing: Cham. DOI: 10.1007/978-3-319-68554-0_6 Williams, G., & Galleron, I. (2016). ‘Bottom Up from the Bottom: A New Outlook on Research Evaluation for the SSH in France’. Research Assessment in the Humanities , pp. 181–98. Springer International Publishing: Cham. DOI: 10.1007/978-3-319-29016-4_14 Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., et al. (2015). Metric Tide: report of the independent review of the role of metrics in research assessment and management . Higher Education Funding Council for England . DOI: 10.13140/RG.2.1.4929.1363 Zuccala, A., & van Leeuwen, T. (2011). ‘Book reviews in humanities research evaluations’, Journal of the American Society for Information Science and Technology , 62/10: 1979–91. Wiley-Blackwell. DOI: 10.1002/asi.21588 Zuccala, A., & Robinson-García, N. (2019). ‘Reviewing, Indicating, and Counting Books for Modern Research Evaluation Systems’. Springer Handbook of Science and Technology Indicators , pp. 715–28. DOI: 10.1007/978-3-030-02511-3_27, pp. 715–28. DOI: 10.1007/978-3-030-02511-3_27