Barry Schouten

Disentangling mode-specific selection and measurement bias in social surveys

A large-scale mixed-mode experiment linked to the Dutch Crime Victimization Survey was conducted in 2011. The experiment consisted of two waves; one wave with random assignment to one of the modes web, paper, telephone and face-to-face, and one follow-up wave to the full sample with interviewer modes only. The objective of the experiment is to estimate total mode effects and more specifically the corresponding mode effect components arising from undercoverage, nonresponse and measurement. In this paper, mode-specific selection and measurement bias are defined, and estimators for the bias terms based on the experimental design are introduced and discussed. The proposed estimators are applied to a number of key survey variables from the Labour Force Survey and the Crime Victimization Survey.

Remote access systems for statistical analysis of microdata

Statistical Agencies manage huge amounts of microdata. The main task of these agencies is to provide a variety of users with general information about for instance the population and the economy. However, in some cases users request additional, more specific information. Many agencies have therefore set up facilities that enable selected users to obtain tailor-made statistical information.A remote access system is an example of such a facility where users can submit queries for statistical information from their own computer. These queries are handled by the statistical agency and the generated, possibly confidentialised, output is returned to the user. This way the agency still keeps control over its own data while the user does not need to make frequent visits to the agency.For some years, the Luxembourg Income Study (LIS) and Luxembourg Employment Study (LES) have made use of an advanced remote access system. At Statistics Netherlands and at other statistical institutes recently the need for a similar system has been expressed. In this article, we discuss the characteristics, limitations and desired properties of a remote access system. We illustrate the discussion by the system used at LIS/LES.

Optimal resource allocation in survey designs

Resource allocation is a relatively new research area in survey designs and has not been fully addressed in the literature. Recently, the declining participation rates and increasing survey costs have steered research interests towards resource planning. Survey organizations across the world are considering the development of new mathematical models in order to improve the quality of survey results while taking into account optimal resource planning. In this paper, we address the problem of resource allocation in survey designs and we discuss its impact on the quality of the survey results. We propose a novel method in which the optimal allocation of survey resources is determined such that the quality of survey results, i.e., the survey response rate, is maximized. We demonstrate the effectiveness of our method by extensive numerical experiments.

Measuring Representativeness of Short-Term Business Statistics

Abstract Short-term statistics (STS) are important early indicators of economic activity. The statistics are obligatory for all EU countries and also serve as input to national accounts. In most countries, short-term Statistics are based on business surveys. However, in recent years a number of countries have gradually replaced their business surveys with business VAT registry data. An important question is whether these surveys and registries are representative of the populations and whether representativity is stable in time. We apply R-indicators and partial R-indicators to measure the representativity of both kinds of data sources. We find large differences between different months of the year and between the two data sources. We discuss dual frame approaches that optimize the accuracy of STS statistics.

Weighting to Adjust for Non-observation Errors in Telephone Surveys

For its social and demographic surveys, Statistics Netherlands favours CAPI over cheaper modes that employ web or telephone. Due to the persuasive power and assistance of interviewers visiting selected persons or households, nonresponse in CAPI surveys is relatively low and data quality is high. However, the costs of this mode of interviewing are relatively high. A large group of trained interviewers is required. Although the Netherlands is a small country, travel costs make up a considerable proportion of the total costs.

Responsive and Adaptive Design for Survey Optimization

Abstract We discuss an evidence-based approach to guiding real-time design decisions during the course of survey data collection. We call it responsive and adaptive design (RAD), a scientific framework driven by cost-quality tradeoff analysis and optimization that enables the most efficient production of high-quality data. The notion of RAD is not new; nor is it a silver bullet to resolve all the difficulties of complex survey design and challenges. RAD embraces precedents and variants of responsive design and adaptive design that survey designers and researchers have practiced over decades. In this paper, we present the four pillars of RAD: survey process data and auxiliary information, design features and interventions, explicit quality and cost metrics, and a quality-cost optimization tailored to survey strata. We discuss how these building blocks of RAD are addressed by articles published in the 2017 JOS special issue and this special section. It is a tale of the three perspectives filling in each other. We carry over each of these three perspectives to articulate the remaining challenges and opportunities for the advancement of RAD. We recommend several RAD ideas for future research, including survey-assisted population modeling, rigorous optimization strategies, and total survey cost modeling.

Can Survey Item Characteristics Relevant to Measurement Error Be Coded Reliably? A Case Study on 11 Dutch General Population Surveys

Item characteristics can have a significant effect on survey data quality and may be associated with measurement error. Literature on data quality and measurement error is often inconclusive. This could be because item characteristics used for detecting measurement error are not coded unambiguously. In our study, we use a systematic coding procedure with multiple coders to investigate the extent to which the coding of item characteristics could be done reliably. For this purpose, we constructed an item characteristics scheme that is based on typologies of characteristics. High intercoder reliability indicates a clear relation between item characteristic, item content, and measurement error. Our results show that intercoder reliability is often low, especially for item characteristics that are hard to code due to subjectivity. Low intercoder reliability complicates comparisons between studies about item characteristics and measurement error. We give suggestions for coping with low intercoder reliability.

Robustness of Adaptive Survey Designs to Inaccuracy of Design Parameters

Abstract Adaptive survey designs (ASDs) optimize design features, given 1) the interactions between the design features and characteristics of sampling units and 2) a set of constraints, such as a budget and a minimum number of respondents. Estimation of the interactions is subject to both random and systematic error. In this article, we propose and evaluate four viewpoints to assess robustness of ASDs to inaccuracy of design parameter estimates: the effect of both imprecision and bias on both ASD structure and ASD performance. We additionally propose three distance measures to compare the structure of ASDs. The methodology is illustrated using a simple simulation study and a more complex but realistic case study on the Dutch Travel Survey. The proposed methodology can be applied to other ASD optimization problems. In our simulation study and case study, the ASD was fairly robust to imprecision, but not to realistic dynamics in the design parameters. To deal with the sensitivity of ASDs to changing design parameters, we recommend to learn and update the design parameters.

JOS Special Issue on Responsive and Adaptive Survey Design: Looking Back to See Forward – Editorial

A rapidly changing survey environment today requires a nimble, flexible design that leverages multiple data sources, frequently collects data in multiple modes, produces high quality data, and optimizes cost allocation around the survey life cycle. In this environment, surveys are ever more challenging and complex to implement. It was in response to these challenges that responsive and adaptive designs were born. This special issue includes twelve articles that give a glimpse of how these concepts are being used in diverse settings. Three phenomena are worth noting. First, evidence-driven policy makers as well as survey researchers have renewed their attention to administrative records, as a key additional data source to enhance survey data quality and minimize survey cost (Chun 2009; Nordbotten 2010; Chun and Scheuren 2011; Chun and Larsen forthcoming). Big data, sometimes called organic data, also create new opportunities when linked to survey data. Second, the computerization of survey data collection has made accessible real-time analysis of paradata, or process data. Paradata have proven useful for managing existing surveys and designing future ones (Couper 1998; Chun and Kwanisai 2010; Couper and Kreuter 2012). Third, methods from fields as diverse as machine learning, operations research, and Bayesian statistics have been found to be useful for the advancement of response propensity modeling and in the design of creative surveys (Early, Mankoff and Fienberg in this special issue; Calinescu et al. 2013; Little 2006). Responsive and adaptive survey designs are evolving in this dynamic environment. The JOS is publishing this special issue in memory of Professor Stephen Fienberg, University Professor of Statistics and Social Science at Carnegie Mellon University, who