Michaela Saisana

Handbook on Constructing Composite Indicators

This Handbook aims to provide a guide for constructing and using composite indicators for policy makers, academics, the media and other interested parties. While there are several types of composite indicators, this Handbook is concerned with those which compare and rank country performance in areas such as industrial competitiveness, sustainable development, globalisation and innovation. The Handbook aims to contribute to a better understanding of the complexity of composite indicators and to an improvement of the techniques currently used to build them. In particular, it contains a set of technical guidelines that can help constructors of composite indicators to improve the quality of their outputs. It has been prepared jointly by the OECD (the Statistics Directorate and the Directorate for Science, Technology and Industry) and the Applied Statistics and Econometrics Unit of the Joint Research Centre of the European Commission in Ispra, Italy. Primary authors from the JRC are Michela Nardo, Michaela Saisana, Andrea Saltelli and Stefano Tarantola. Primary authors from the OECD are Anders Hoffmann and Enrico Giovannini. Editorial assistance was provided by Candice Stevens, Gunseli Baygan and Karsten Olsen. The research is partly funded by the European Commission, Research Directorate, under the project KEI (Knowledge Economy Indicators), Contract FP6 No. 502529. In the OECD context, the work has benefitted from a grant from the Danish government. The views expressed are those of the authors and should not be regarded as stating an official position of either the European Commission or the OECD. Ce Manuel a pour objectif de procurer aux responsables politiques, universitaires, medias et autres parties concernees un guide sur la facon d’elaborer et d’utiliser des indicateurs composites. Si il existe plusieurs types d’indicateurs composites, ce Manuel interesse ceux qui comparent et classent la performance d’un pays dans des domaines comme la competitivite industrielle, le developpement durable, la mondialisation et les innovations. Le Manuel a pour objectif de contribuer a une meilleure comprehension de la complexite des indicateurs composites et a une amelioration des techniques actuellement utilisees pour les elaborer. En particulier, il contient une serie de lignes directrices techniques qui peuvent aider les concepteurs d’indicateurs composites a ameliorer la qualite de leurs productions. Il a ete conjointement prepare par l’OCDE (la Direction des statistiques et la Direction de la science, de la technologie et de l’industrie) et la cellule des Statistiques appliquees et de l’econometrie du Centre commun de recherche (CCR) de la Commission europeenne a Ispra en Italie. Les auteurs originaux du CCR sont Michela Nardo, Michaela Saisana, Andrea Saltelli et Stefano Tarantola. Les auteurs originaux de l’OCDE sont Anders Hoffmann et Enrico Giovannini. L’assistance editoriale a ete assuree par Candice Stevens, Gunseli Baygan et Karsten Olsen. Les recherches sont partiellement financees par la Direction des recherches de la Commission europeenne, pour le projet KEI (Knowledge Economy Indicators), Contrat FP6 no. 502529. Pour ce qui est de l’OCDE, le travail a beneficie d’une subvention du gouvernement danois. Les points de vue exprimes sont ceux des auteurs et ils ne doivent pas etre consideres comme l’expression d’une position officielle de la Commission europeenne ou de l’OCDE.

Ratings and rankings: Voodoo or Science?

Summary.  Composite indicators aggregate a set of variables by using weights which are understood to reflect the variables’ importance in the index. We propose to measure the importance of a given variable within existing composite indicators via Karl Pearsons ‘correlation ratio’; we call this measure the ‘main effect’. Because socio-economic variables are heteroscedastic and correlated, relative nominal weights are hardly ever found to match relative main effects; we propose to summarize their discrepancy with a divergence measure. We discuss to what extent the mapping from nominal weights to main effects can be inverted. This analysis is applied to six composite indicators, including the human development index and two popular league tables of university performance. It is found that in many cases the declared importance of single indicators and their main effect are very different, and that the data correlation structure often prevents developers from obtaining the stated importance, even when modifying the nominal weights in the set of non-negative numbers with unit sum.

Uncertainty and Sensitivity Analysis of the 2010 Environmental Performance Index

Abstract An assessment of the robustness of the 2010 Environmental Performance Index (EPI) ranks requires the evaluation of uncertainties underlying the index and the sensitivity of the country rankings to the methodological choices made during the development of the Index. To test this robustness, the Yale and Columbia University have continued their partnership with the Joint Research Centre (JRC) of the European Commission in Ispra, Italy. This JRC report shows that although the theoretical framework and the indicators for the EPI were carefully chosen by experts, the issue of weighting is crucial to obtain a robust performance index. The current weighting and normalization schemes result in an EPI that is dominated by very few indicators while having an almost random association with several other underlying indicators. With respect to the five main assumptions tested in the uncertainty and sensitivity analysis, the country ranks are relatively reliable for 109 countries, while any conclusion on the ranks for the remaining countries should be made with great caution. An equal weighting approach or factor analysis-derived weights at the indicator level, as opposed to the current weighting scheme greatly influences the ranks. Thus, the choice of the weights must be evaluated according to the EPI’s analytical rationale, policy relevance, and implied value judgments. If the objective of EPI is to promote action on all policies categories more work would be needed to ensure that all policy fields have an impact on the aggregated EPI or, alternatively, policy categories should be given more emphasis than the aggregated measure. The 2010 EPI is developed for 163 countries and is based on twenty five indicators grouped in ten policy categories: Environmental burden of disease, Air pollution (effects on humans), Water (effects on humans), Air Pollution (effects on ecosystem), Water (effects on ecosystem), Biodiversity & Habitat, Forestry, Fisheries, Agriculture and Climate Change. The EPI ranking is assessed by evaluating how sensitive the country ranks are to the assumptions made on the index structure and the aggregation of the 25 underlying indicators. The assumptions tested by the JRC-IPSC are: • measurement error of the raw data, • EPI structure – grouping at policy categories, • weights assigned to the indicators and/or to the policy categories, • aggregation function at the policy or at the objectives level, and • number of indicators or policy categories.

A new estimator for sensitivity analysis of model output: An application to the e-business readiness composite indicator

Abstract In this paper we propose and test a generalisation of the method originally proposed by Sobol’, and recently extended by Saltelli, to estimate the first-order and total effect sensitivity indices. Exploiting the symmetries and the dualities of the formulas, we obtain additional estimates of first-order and total indices at no extra computational cost. We test the technique on a case study involving the construction of a composite indicator of e-business readiness, which is part of the initiative “e-Readiness of European enterprises” of the European Commission “e-Europe 2005” action plan. The method is used to assess the contribution of uncertainties in (a) the weights of the component indicators and (b) the imputation of missing data on the composite indicator values for several European countries.

Measuring uncertainties in composite indicators of sustainability

Composite indicators are very common in economic and business statistics for benchmarking the relative progress of countries in a variety of policy domains such as industrial competitiveness, sustainable development, globalisation and innovation. This article provides an analysis of the key uncertainties underlying the mathematical modelling used to construct composite indicators within the broader context of sustainability measurement. A case study of the Environmental Sustainability Index illustrates how these uncertainties can be modelled and estimated.

Rankings and Ratings: Instructions for Use

Multidimensional measures (composite indicators, indices, ratings, league tables) can effectively underpin the development of data-driven narratives in support to policy. A controversy surrounds the use of these measures. We review some good and bad practices from the recent literature. We then discuss briefly a decalogue to develop a multidimensional measure. We argue in favor of a multi-modeling approach to represent different scenarios in the construction of an aggregate measure prior to drawing recommendations for policy making. Finally, we try to establish a link between the analytic use of well-designed aggregate measures and the development of a robust culture of evaluation of policies based on evidence. An application of these concepts and tools to the Rule of Law index developed by the World Justice Project is given.

Weights and Importance in Composite Indicators: Closing the Gap

Highlights • Composite indicators are widely used in sustainable development and elsewhere.• The effect of weights used in aggregating indicators is complex.• Three tools are presented which help developers and users to investigate effects of weights.• Case studies related to sustainable development demonstrate the benefits.

Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators

Composite indicators aggregate multi-dimensional processes into simplified concepts often aiming at underpinning the development of data-driven narratives for policy consumption. Due to methodological issues, doubts are often raised about the robustness of the composite indicators and the significance of the associated policy messages. In this paper we use expert panel information (derived from budget allocation and analytic hierarchy process) on the relative importance of the underlying indicators included in a composite indicator and run in tandem uncertainty and sensitivity analysis to gain useful insights during the process of composite indicators building. We discuss the extent to which variance-based sensitivity analysis may increase transparency or make policy inference more defensible by using the United Nation’s Technology Achievement Index as an illustration.

The Multidimensional Poverty Assessment Tool (MPAT): Robustness issues and Critical assessment

The Multidimensional Poverty Assessment Tool (MPAT) was developed by the UN International Fund for Agricultural Development with a view to assess local-level poverty in rural regions around the globe. The MPAT is a survey-based thematic indicator of ten dimensions, from Food & Nutrition Security to Domestic Water Supply, Health & Healthcare, to Gender Equality. The aims of this validation report are: (a) to spot eventual conceptual and methodological shortcomings in the MPAT, (b) to identify suitable aggregation methods for the survey items, (c) to assess the internal consistency of the MPAT conceptual framework, and finally, (d) to offer snapshots of the MPAT results. The results show that the MPAT, upon some improvements throughout the entire development, would pass the “statistical” filters of index quality, and it could thus be reliably used to identify weaknesses and possible remedial actions, prioritize villages or even households with relatively low levels of rural poverty, and ultimately monitor and evaluate policy effectiveness. The analysis undertaken in this work provides no guarantee of the true ability of the MPAT to describe rural poverty world wide. Yet, it provides enough evidence that the MPAT cannot easily be falsified by methodological cunning. How to obtain EU publications Our priced publications are available from EU Bookshop (http://bookshop.europa.eu), where you can place an order with the sales agent of your choice. The Publications Office has a worldwide network of sales agents. You can obtain their contact details by sending a fax to (352) 29 29-42758. The mission of the JRC is to provide customer-driven scientific and technical support for the conception, development, implementation and monitoring of EU policies. As a service of the European Commission, the JRC functions as a reference centre of science and technology for the Union. Close to the policy-making process, it serves the common interest of the Member States, while being independent of special interests, whether private or national. LB -N A -2310 EN -C