Torben Schubert

Are international co-publications an indicator for quality of scientific research?

This article deals with the role of internationally co-authored papers (co-publications). Specifically, we compare, within a data-set of German research units, citation and co-publication indicators as a proxy for the unobserved quality dimension of scientific research. In that course we will also deal with the question whether both citations and co-publications are considerably related. Our results suggest that, although there is a strong partial correlation between citations and co-publications within a multivariate setting, we cannot use reasonably normalised co-publication indicators as an alternative proxy for quality. Thus, concerning quality assessment, there remains a primer on citation analysis.

Drittmittel als Performanzindikator der Wissenschaftlichen Forschung

ZusammenfassungRessourcen innerhalb und zwischen Universitäten und in der außeruniversitären Forschung werden angesichts der Finanzknappheit im Wissenschaftssystem zunehmend in Abhängigkeit von Forschungsleistungen verteilt. Hiermit bekommen Evaluationsverfahren und die Messung von Forschungsleistungen einen neuen Stellenwert. Drittmitteleinkommen als relativ einfach zu erhebender Performanzindikator spielt in allen neuen leistungsbasierten Verteilungsverfahren eine große Rolle. Ziel dieses Beitrags ist es, die Aussagekraft von Drittmitteln für die Bewertung von Forschungsleistungen zu untersuchen. Auf der Grundlage von quantitativen und qualitativen Daten zu Forschungsgruppen aus drei sehr unterschiedlichen Forschungsfeldern wird der Zusammenhang von Strukturindikatoren, Drittmitteleinkommen als weiterem Inputfaktor und dem Output der wissenschaftlichen Produktion betrachtet. Hierbei zeigt sich, dass der Einsatz von Drittmitteln als Performanzindikator nur unterhalb eines von disziplinspezifischen Produktionsbedingungen bestimmten Schwellenwertes sinnvoll ist. Ferner ergibt die weitergehende Analyse, dass Forschungsperformanz mehrdimensional ist, Publikationen also nicht der einzig sinnvolle Indikator für Performanz sein können. Schließlich zeigt sich, dass die Bedingungen wissenschaftlicher Produktion zwischen den Fachgebieten so unterschiedlich sind, dass Performanzvergleiche nur innerhalb homogener Forschungsfelder sinnvoll einsetzbar sind. Abschließend werden die forschungspolitischen Implikationen für Forschungsfinanzierung und Forschungsevaluation diskutiert sowie weiterer Forschungsbedarf aufgezeigt.AbstractResources within and between higher education and research institutions are increasingly allocated according to scientific performance. Evaluation exercises and the measurement of research performance take on a new role in this context. Third party research income is a performance indicator which is rather easy to measure and is used in most of the new performance-based evaluation procedures. This paper sets out to scrutinize the meaning and validity of third party research income. We studied research teams from three different research fields with a mixed quantitative / qualitative approach. The focus is on the causal relations between institutional / input indicators, third party research income as another input indicator, and a variety of output indicators of scientific production. An important result is that third party money has a positive effect on performance only below a certain and discipline-specific threshold. Further analysis shows that output performance is to a great extent field-specific. The context conditions for scientific production differ greatly, so that comparative performance assessments are only valid within homogeneous research fields. Another important result is that output performance is multidimensional and cannot be measured by bibliometric indicators only. These findings have implications for the rationality of the evaluation assessment exercises as well as for the funding decisions of science foundations, ministries of science and — increasingly, the heads of universities and departments. These topics are discussed in the last chapter as well as the implications for further research.

Drittmittel als Performanzindikator der Wissenschaftlichen Forschung zum Einfluss von Rahmenbedingungen auf Forschungsleistung

ZusammenfassungRessourcen innerhalb und zwischen Universitäten und in der außeruniversitären Forschung werden angesichts der Finanzknappheit im Wissenschaftssystem zunehmend in Abhängigkeit von Forschungsleistungen verteilt. Hiermit bekommen Evaluationsverfahren und die Messung von Forschungsleistungen einen neuen Stellenwert. Drittmitteleinkommen als relativ einfach zu erhebender Performanzindikator spielt in allen neuen leistungsbasierten Verteilungsverfahren eine große Rolle. Ziel dieses Beitrags ist es, die Aussagekraft von Drittmitteln für die Bewertung von Forschungsleistungen zu untersuchen. Auf der Grundlage von quantitativen und qualitativen Daten zu Forschungsgruppen aus drei sehr unterschiedlichen Forschungsfeldern wird der Zusammenhang von Strukturindikatoren, Drittmitteleinkommen als weiterem Inputfaktor und dem Output der wissenschaftlichen Produktion betrachtet. Hierbei zeigt sich, dass der Einsatz von Drittmitteln als Performanzindikator nur unterhalb eines von disziplinspezifischen Produktionsbedingungen bestimmten Schwellenwertes sinnvoll ist. Ferner ergibt die weitergehende Analyse, dass Forschungsperformanz mehrdimensional ist, Publikationen also nicht der einzig sinnvolle Indikator für Performanz sein können. Schließlich zeigt sich, dass die Bedingungen wissenschaftlicher Produktion zwischen den Fachgebieten so unterschiedlich sind, dass Performanzvergleiche nur innerhalb homogener Forschungsfelder sinnvoll einsetzbar sind. Abschließend werden die forschungspolitischen Implikationen für Forschungsfinanzierung und Forschungsevaluation diskutiert sowie weiterer Forschungsbedarf aufgezeigt.AbstractResources within and between higher education and research institutions are increasingly allocated according to scientific performance. Evaluation exercises and the measurement of research performance take on a new role in this context. Third party research income is a performance indicator which is rather easy to measure and is used in most of the new performance-based evaluation procedures. This paper sets out to scrutinize the meaning and validity of third party research income. We studied research teams from three different research fields with a mixed quantitative / qualitative approach. The focus is on the causal relations between institutional / input indicators, third party research income as another input indicator, and a variety of output indicators of scientific production. An important result is that third party money has a positive effect on performance only below a certain and discipline-specific threshold. Further analysis shows that output performance is to a great extent field-specific. The context conditions for scientific production differ greatly, so that comparative performance assessments are only valid within homogeneous research fields. Another important result is that output performance is multidimensional and cannot be measured by bibliometric indicators only. These findings have implications for the rationality of the evaluation assessment exercises as well as for the funding decisions of science foundations, ministries of science and — increasingly, the heads of universities and departments. These topics are discussed in the last chapter as well as the implications for further research.

How to use indicators to measure scientific performance : A balanced approach

Scientific performance should not be measured by a one-dimensional metric such as publication, since it is a multi-dimensional phenomenon. A quantitative analysis of the activities of research groups in three scientific fields demonstrates in particular the importance of sufficient numbers of PhD graduates and of contributions to the infrastructure of the scientific community, in terms of editorships or memberships of boards, etc. The results of a quantitative analysis are largely confirmed by a parallel qualitative investigation; however, both approaches complement each other by highlighting different aspects. For example, the qualitative approach conveys explicitly the demand structure for intermediary and final outputs in the qualitative approach that interlinks the activities of different research units. The results show that it is important for science policy to set appropriate incentives for all dimensions of scientific activities, i.e. not publication output exclusively, as this entails considerable hazard of distortion, endangering the sustainability of scientific research. Copyright , Beech Tree Publishing.

Can the centre–periphery model explain patterns of international scientific collaboration among threshold and industrialised countries? The case of South Africa and Germany

As scientific collaboration is a phenomenon that is becoming increasingly important, studies on scientific collaboration are numerous. Despite the proliferation of studies on various dimensions of collaboration, there is still a dearth of analyses on the effects, motives and modes of collaboration in the context of developing countries. Adopting Wallerstein’s world-system theory, this paper makes use of bibliometric data in an attempt to understand the pattern of collaboration that emerges between South Africa and Germany. The key argument is that we can expect the collaborative relationship between South Africa and Germany to be one that is shaped by a centre–periphery pattern. The analyses show that a theory of scientific collaboration building on the notion of marginality and centre–periphery can explain many facets of South African–German collaboration, where South Africa is a semi-peripheral region, a centre for the periphery, and a periphery for the centre.

Sustainability of incentives for excellent research - The German case

The state authorities in Germany used to fund public sector research without controlling the performance of the research units. This has changed during past decade, where the dominant mechanism by which formerly unconditional state funds are allocated nowadays is indicator-based performance measurement. The indicator sets used to measure the research-related performance in the German public science sector are usually very narrow, often consisting exclusively of finished doctoral theses and third-party funds. Using a unique dataset of 473 German research units from astrophysics, nanotechnology, economics and biotechnology, this paper outlines principles for the construction of sensible indicator sets for the performance measurement of scientific research groups. It is argued that scientific production is multidimensional. Thus one-sided indicator sets that fail to cover the relevant output dimensions give rise to incentives that will ultimately lower the performance of the science sector in total. Indicator sets should strive for sustainable incentives, which can be guaranteed if the sets are broad enough. As a starting point it is shown that the very common performance indicator ‘acquired third-party funds’ may affect research efficiency negatively, especially if the level of third-party funds is already very high. Therefore, we conclude that third-party funds should be used with great care, if at all.

Is the university model an organizational necessity? Scale and agglomeration effects in science

In this paper we argue that the emergence of the dominant model of university organization, which is characterized by a large agglomeration of many (often loosely affiliated) small research groups, might have an economic explanation that relates to the features of the scientific production process. In particular, we argue that there are decreasing returns to scale on the level of the individual research groups, which prevent them from becoming to large, while we argue for positive agglomeration effects on the supra-research-group-level inside the university. As a consequence an efficient university organization would precisely consist of tying together many small individual research groups without merging them. Basing our empirical analysis on a multilevel dataset for German research institutes from four disciplines we are able to find strong support for the presence of these effects. This suggests that the emergence of the dominant model of university organization may also be the result of these particular features of the production process, where the least we can say is that this model is under the given circumstances highly efficient.

Assessing the value of patent portfolios: an international country comparison

Patent counts have been extensionally used to measure the innovative capacities of countries. However, since economic values of patents may differ, simple patent counts may give misleading rankings, if the patents of one country are on average more valuable than those of another. In the literature several methods have been proposed, which shall adjust for these differences. However, often these do not possess a solid economic micro-foundation and therefore are often ad-hoc and arbitrary procedures. In this paper, we intend to present an adjustment method that is based on the analysis of renewal decisions. The method builds on the theoretical model used in Schankerman and Pakes (1986) and Besson (2008) but goes beyond both approaches in that it recovers the important long tail of the value distribution. It also transfers Besson’s (2008) econometric methodology (applicable to the organisational structures of the US Patent and Trademark Office) also to the European Patent Office which is necessary, since each application here may split up into several national patent documents. The analysis is performed for 22 countries. Exemplarily, we find that in the cohort of 1986 patent applications, Danish patents are about 60% more valuable than the average patent. German patents are a bit below average. Japanese patents are of least value. In the cohort of 1996, Danish patents lose some of their lead but are still more valuable than the average. While German are a bit above average, Japanese patents even fall further behind (possibly due to the economic downturn in since the mid of 1990ies).

Are there scale economies in scientific production? On the topic of locally increasing returns to scale

In this paper the question of returns to scale in scientific production is analysed using non-parametric techniques of multidimensional efficiency measurement. Based on survey data for German research groups from three scientific fields, it is shown that the multidimensional production possibility sets are weakly non-convex and locally strictly non-convex. This suggests that the production functions for the groups in the sample are characterised by increasing returns to scale in some regions and at least constant returns to scale otherwise. This has two implications for the organisation of scientific research: first, the size of at least some groups in our sample is suboptimal and they would benefit from growth. Second, greater specialisation in certain tasks in science (e.g. transfer-oriented groups vs. research-oriented groups) would increase the output of the overall system.

Is there a European university model? New evidence on national path dependence and structural convergence

For the first time, data on individual European higher education institutions (rather than data aggregated at the country level) is used in order to examine a wide range of issues that are both theoretically challenging and relevant from policy-making and societal perspectives. The contributors integrate statistics on universities and colleges with other sources of information such as patents, start-up firms and bibliometric data, and employ rigorous empirical methods to address a range of key questions, including: what is the role of non-university tertiary education, such as vocational training? How important is the private sector? Are European universities internationalized? Are they efficient from the point of view of costs and educational output? Are there pure research universities in Europe? How do universities contribute to economic growth?