Jochen Gläser

The Social Construction Of Bibliometric Evaluations

While national systems of research evaluation vary in many dimensions, they all need to rely on very few methods of evaluating research performance. These methods constitute a crucial interface between the science system and science policy through which information about research is translated into strategic knowledge for policy decisions. They therefore merit specific attention.

Evaluation Without Evaluators

The Australian research evaluation system (RES) is unique in its exclusive reliance on a funding formula. For each university, statistics on income from competitive research grants, numbers of publications, numbers of current research students (Masters and PhD students), and timely completions of Masters and PhD studies are collected and used to calculate the allocation of state funds without any further consideration.

Integrating scientometric indicators into sociological studies: methodical and methodological problems

This article discusses the methodological problems of integrating scientometric methods into a qualitative study. Integrative attempts of this kind are poorly supported by the methodologies of both the sociology of science and scientometrics. Therefore it was necessary to develop a project-specific methodological approach that linked scientometric methods to theoretical considerations. The methodological approach is presented and used to discuss general methodological problems concerning the relation between (qualitative) theory and scientometric methods. This discussion enables some conclusions to be drawn as to the relations that exist between scientometrics and the sociology of science.

Why are the Most Influential Books in Australian Sociology not Necessarily the Most Highly Cited Ones

The vote by TASA members on the ‘Most Influential Book in Australian Sociology’ (MIBAS) provides an opportunity to compare quantitative indicators based on citations with the peer review represented by the MIBAS votes. While it is generally agreed that citations are only a partial indicator of such a complex phenomenon as quality, citations are regarded as a reflection of a publication’s, author’s or research group’s ‘impact’ or influence. This assumption is explored by applying several interpretations and operationalizations of the concept of ‘influence’, by discussing their validity, and by comparing the respective citation rankings to the MIBAS poll. The major lessons of the comparison are that citation-based indicators should not be applied in diachronic comparisons, for evaluating publications on nationally specific topics, or in fields in which books are an important part of the research output.

Intraorganisational evaluation: are there ‘least evaluable units'?

An increasing number of universities and research organisations are introducing internal evaluations which are often based on quantitative indicators. However, it is likely that a ‘least evaluable unit’ (LEU) exists in every research organisation, and that below this level many standard quantitative performance indicators no longer provide a valid measure of performance. In this paper, the LEU of a research organisation is identified by analysing retrospective performance evaluations at different levels of aggregation and enhancing their interpretation with the detailed knowledge of the organisations senior manager. The main obstacles to further disaggregation below the LEU are that indicators lose their statistical validity because of low numbers of publications and that the performance of subunits cannot be independently measured. The latter phenomenon is heightened at the level of scientists because work roles emerged that further clouded the application of performance measures to individuals. Copyright , Beech Tree Publishing.

Advantages and dangers of ‘remote’ peer evaluation

Under conditions of an increasing scarcity of reviewer time, a ‘remote peer review’ of research organisations — conducted without meetings between assessors or with the assessed academics at their institution — might be an easier and cheaper solution. This paper explores the impact of ‘remoteness’ on the practices of reviewers by analysing the recent Quality Review of the Australian National University. A taylorisation of the review process was observed that split the roles of designing the process, assessing the research, synthesising results, and taking responsibility for the outcome of the assessment. This taylorisation of the assessment process, the heterogeneity of individual assessment practices, and the low availability of publications in electronic format led to the conclusion that further organisational innovations are necessary in order to make remote peer reviews feasible. Copyright , Beech Tree Publishing.

On Interviewing “Good” and “Bad” Experts

The success of interview-based investigations considerably depends on the “quality” of the interviewees, that is on the extent to which they meet our expectations in the interview situation. We expect interviewees to understand which information we need, to provide this information in extensive, complete and detailed responses, and to adjust their communication to our steering of the conversation. We also hope to meet respondents which reflect on their own social situation and who are able to provide information about their perceptions, social relations and motives.1

The Social Order of Open Source Software Production

This article contributes to the sociological understanding of open source software OSS production by identifying the social mechanism that creates social order in OSS communities. OSS communities are identified as production communities whose mode of production employs autonomous decentralized decision making on contributions and autonomous production of contributions while maintaining the necessary order by adjustment to the common subject matter of work. Thus, OSS communities belong to the same type of collective production system as scientific communities. Both consist of members who not only work on a common product, but are also aware of this collective work and adjust their actions accordingly. Membership is based on the self-perception of working with the communitys subject matter software or respectively scientific knowledge. The major differences between the two are due to the different subject matters of work. Production communities are compared to the previously known collective production systems, namely, markets, organizations, and networks. They have a competitive advantage in the production under complete uncertainty, that is, when neither the nature of a problem, nor the way in which it can be solved, nor the skills required for its solution are known in advance.

Path Dependence and Policy Steering in the Social Sciences: The Varied Impact of International Large Scale Student Assessment on the Educational Sciences in Four European Countries

The aim of this article is to explain commonalities and differences in the responses of four national educational science communities to the same external stimulus, namely international comparative large scale student assessments that offered vastly improved comparability of national results from the beginning of the 1990s. The comparison shows the epistemic traditions of educational research in the four countries and properties of the data produced by the international comparative studies to be the central explanatory factors for commonalities and differences of responses to the new studies.

Where to Go for a Change: The Impact of Authority Structures in Universities and Public Research Institutes on Changes of Research Practices

In this article, we analyse how variations in organisational conditions for research affect researchers’ opportunities for changing individual-level or group-level research programmes. We contrast three innovations that were developed in universities and public research institutes in Germany and the Netherlands, which enables comparisons both between organisational settings and between properties of innovations. Comparing the development of three innovations in the two types of organisations enables the identification of links between patterns of authority sharing at these organisations and the opportunities to develop innovations. On this basis, the distribution of opportunities to change research practices among researchers in the two countries can be established.