Giancarlo Manzi

An imputation method for categorical variables with application to nonlinear principal component analysis

The problem of missing data in building multidimensional composite indicators is a delicate problem which is often underrated. An imputation method particularly suitable for categorical data is proposed. This method is discussed in detail in the framework of nonlinear principal component analysis and compared to other missing data treatments which are commonly used in this analysis. Its performance vs. these other methods is evaluated throughout a simulation procedure performed on both an artificial case, varying the experimental conditions, and a real case. The proposed procedure is implemented using R.

Citizens evaluate public services: a critical overview of statistical methods for analysing user satisfaction

Public enterprises may be unaware of their performance in providing services. In situations where citizens cannot switch to other providers or reduce the use of the service, the evaluation of users’ satisfaction becomes a very important topic. At the same time, this is a tricky task, given the particular nature of this variable. Appropriate statistical methods to assess and explain the level of satisfaction are useful tools to face these issues. In this paper, we analyse some of these methods and their potential in giving advice to public managers to improve citizens’ satisfaction.

Modelling bias in combining small area prevalence estimates from multiple surveys

Summary Combining information from multiple surveys can improve the quality of small area estimates. Customary approaches, such as the multiple-frame and statistical matching methods, require individual level data, whereas in practice often only multiple aggregate estimates are available. Commercial surveys usually produce such estimates without clear description of the methodology that is used. In this context, bias modelling is crucial, and we propose a series of Bayesian hierarchical models which allow for additive biases. Some of these models can also be fitted in a classical context, by using a mixed effects framework. We apply these methods to obtain estimates of smoking prevalence in local authorities across the east of England from seven surveys. All the surveys provide smoking prevalence estimates and confidence intervals at the local authority level, but they vary by time, sample size and transparency of methodology. Our models adjust for the biases in commercial surveys but incorporate information from all the sources to provide more accurate and precise estimates.

Are they telling the truth? Revealing hidden traits of satisfaction with a public bike-sharing service

ABSTRACT Public bike-sharing systems (BSSs) are an emerging mode of transportation introduced by municipalities to solve congestion problems in metropolitan areas, especially when integrated with other types of transportation. In the last years, the number of public bike-sharing services has been constantly on the rise all over the world, and generally the overall satisfaction with them is high. However, satisfaction with public services is driven by mechanisms that can differ from those in the private sector. It is important to establish to what extent a high satisfaction is genuine or simply ephemeral. Even “old” public services (like public transportation) become “gold” when accompanied by the introduction of new technologies. In this paper we analyze this phenomenon using data from a satisfaction web-survey conducted among customers of the public BSS “BikeMi” in Milan, Italy, in a period when mobile technologies have been introduced to speed up the service. On analyzing the responses to satisfaction questions using simple summary statistics, the level of satisfaction resulted very high. However, our aim was to look for potential “darker” sides of the service by detecting possible hidden satisfaction components. For this purpose, we used the Nonlinear Principal Components Analysis, which is particularly powerful in this context. A simple textual analysis was also performed as a validating test. Results from our analysis indicated that satisfaction is flawed by a set of factors like the mechanics of the bikes, the picking and dropping system, and the apps used to organize the service. Less concern was detected for more general aspects of the service.

A sequential distance-based approach for imputing missing data: Forward Imputation

Missing data recurrently affect datasets in almost every field of quantitative research. The subject is vast and complex and has originated a literature rich in very different approaches to the problem. Within an exploratory framework, distance-based methods such as nearest-neighbour imputation (NNI), or procedures involving multivariate data analysis (MVDA) techniques seem to treat the problem properly. In NNI, the metric and the number of donors can be chosen at will. MVDA-based procedures expressly account for variable associations. The new approach proposed here, called Forward Imputation, ideally meets these features. It is designed as a sequential procedure that imputes missing data in a step-by-step process involving subsets of units according to their “completeness rate”. Two methods within this context are developed for the imputation of quantitative data. One applies NNI with the Mahalanobis distance, the other combines NNI and principal component analysis. Statistical properties of the two methods are discussed, and their performance is assessed, also in comparison with alternative imputation methods. To this purpose, a simulation study in the presence of different data patterns along with an application to real data are carried out, and practical hints for users are also provided.

Algorithmic-Type Imputation Techniques with Different Data Structures: Alternative Approaches in Comparison

In recent years, with the spread availability of large datasets from multiple sources, increasing attention has been devoted to the treatment of missing information. Recent approaches have paved the way to the development of new powerful algorithmic techniques, in which imputation is performed through computer-intensive procedures. Although most of these approaches are attractive for many reasons, less attention has been paid to the problem of which method should be preferred according to the data structure at hand. This work addresses the problem by comparing the two methods missForest and IPCA with a new method we developed within the forward imputation approach. We carried out comparisons by considering different data patterns with varying skewness and correlation of variables, in order to ascertain in which situations a given method produces more satisfying results.

A simulation comparison of imputation methods for quantitative data in the presence of multiple data patterns

ABSTRACT An extensive investigation via simulation is carried out with the aim of comparing three nonparametric, single imputation methods in the presence of multiple data patterns. The ultimate goal is to provide useful hints for users needing to quickly pick the most effective imputation method among the following: Forward Imputation (), considered in the two variants of with the principal component analysis (PCA), which alternates the use of PCA and the Nearest-Neighbour Imputation (NNI) method in a forward, sequential procedure, and with the Mahalanobis distance, which involves the use of the Mahalanobis distance when performing NNI; the iterative PCA technique, which imputes missing values simultaneously via PCA; the method, which is based on random forests and is developed for mixed-type data. The performance of these methods is compared under several data patterns characterized by different levels of kurtosis or skewness and correlation structures.

Bootstrapping probability-proportional-to-size samples via calibrated empirical population

A collection of six novel bootstrap algorithms, applied to probability-proportional-to-size samples, is explored for variance estimation, confidence interval and p-value production. Developed according to bootstrap fundamentals such as the mimicking principle and the plug-in rule, these algorithms make use of an empirical bootstrap population informed by sampled units each with assigned weight. Starting from the natural choice of Horvitz–Thompson (HT)-type weights, improvements based on calibration to known population features are fostered. Focusing on the population total as the parameter to be estimated and on the distribution of the HT estimator as the target of bootstrap estimation, simulation results are presented with the twofold objective of checking practical implementation and of investigating the statistical properties of the bootstrap estimates supplied by the algorithms explored.

Fifty Years of Business Confidence Surveys on Manufacturing Sector

In this work the evolution of the Italian Business Confidence Survey on manufacturing sector is presented starting from the preliminary European project for harmonized statistics launched in the late fifties of the last century. Survey changes are described, focusing in particular on the so-called confidence indicator. The continuing increase of statistical accuracy in sampling is recalled, from the initial purposive sample and controls, up to the present state of the art. Specific attention is devoted to the role of administrative archives in the sampling plan. Emphasis is also given to the increasing use of computer simulation in assessing the validity of the estimates. The role of cyclical analysis is finally highlighted with regard to two aspects: (1) the business confidence has not a corresponding variable in the economic system—the validation can only be performed in comparison with correlated variables (e.g. IP, GDP); (2) confidence shows forecasting capability for the economic system.

Handling Missing Data in Presence of Categorical Variables: a New Imputation Procedure

In this paper we propose a new method to deal with missingness in categorical data. The new proposal is a forward imputation procedure and is presented in the context of the Nonlinear Principal Component Analysis, used to obtain indicators from a large dataset. However, this procedure can be easily adopted in other contexts, and when other multivariate techniques are used. We discuss the statistical features of our imputation technique in connection with other treatment methods which are popular among Nonlinear Principal Component Analysis users. The performance of our method is then compared to the other methods through a simulation study which involves the application to a real dataset extracted from the Euro-barometer survey. Missing data are created in the original data matrix and then the comparison is performed in terms of how close the Nonlinear Principal Component Analysis outcomes from missing data treatment methods are to the ones obtained from the original data. The new procedure is seen to provide better results than the other methods under the different conditions considered.