Olena Kaminska

Survey-Based Cross-Country Comparisons Where Countries Vary in Sample Design: Issues and Solutions

Abstract In multi-national surveys, different countries usually implement different sample designs. The sample designs affect the variance of estimates of differences between countries. When making such estimates, analysts often fail to take sufficient account of sample design. This failure occurs sometimes because variables indicating stratification, clustering, or weighting are unavailable, partially available, or in a form that is unsuitable for cross-national analysis. In this article, we demonstrate how complex sample design should be taken into account when estimating differences between countries, and we provide practical guidance to analysts and to data producers on how to deal with partial or inappropriately-coded sample design indicator variables. Using EU-SILC as a case study, we evaluate the inverse misspecification effect (imeff ) that results from ignoring clustering or stratification, or both in a between-country comparison where countries’ sample designs differ. We present imeff for estimates of between-country differences in a number of demographic and economic variables for 19 European Union Member States. We assess the magnitude of imeff and the associated impact on standard error estimates. Our empirical findings illustrate that it is important for data producers to supply appropriate sample design indicators and for analysts to use them.

Panel Attrition: How Important is Interviewer Continuity?

Abstract We assess whether the probability of a sample member cooperating at a particular wave of a panel survey is greater if the same interviewer is deployed as at the previous wave. Previous research on this topic mainly uses nonexperimental data. Consequently, a) interviewer change is generally nonrandom, and b) continuing interviewers are more experienced by the time of the next wave. Our study is based on a balanced experiment in which both interviewer continuity and experience are controlled. Multilevel multiple membership models are used to explore the effects of interviewer continuity on refusal rate as well as interactions of interviewer continuity with other variables. We find that continuity reduces refusal propensity for younger respondents but not for older respondents, and that this effect depends on the age of the interviewer. This supports the notion that interviewer continuity may be beneficial in some situations, but not necessarily in others.

Eye-tracking Social Desirability Bias

Eye tracking is now a common technique studying the moment-by-moment cognition of those processing visual information. Yet this technique has rarely been applied to different survey modes. Our paper uses an innovative method of real-world eye tracking to look at attention to sensitive questions and response scale points, in Web, face-to-face and paper-and-pencil self-administered (SAQ) modes. We link gaze duration to responses in order to understand how respondents arrive at socially desirable or undesirable answers. Our novel technique sheds light on how social desirability biases arise from deliberate misreporting and/or satisficing, and how these vary across modes.

The implications of alternative allocation criteria in adaptive design for panel surveys

Abstract Adaptive survey designs can be used to allocate sample elements to alternative data collection protocols in order to achieve a desired balance between some quality measure and survey costs. We compare four alternative methods for allocating sample elements to one of two data collection protocols. The methods differ in terms of the quality measure that they aim to optimize: response rate, R-indicator, coefficient of variation of the participation propensities, or effective sample size. Costs are also compared for a range of sample sizes. The data collection protocols considered are CAPI single-mode and web-CAPI sequential mixed-mode. We use data from a large experiment with random allocation to one of these two protocols. For each allocation method we predict outcomes in terms of several quality measures and costs. Although allocating the whole sample to single-mode CAPI produces a higher response rate than allocating the whole sample to the mixed-mode protocol, we find that two of the targeted allocations achieve a better response rate than single-mode CAPI at a lower cost. We also find that all four of the targeted designs out-perform both single-protocol designs in terms of representativity and effective sample size. For all but the smallest sample sizes, the adaptive designs bring cost savings relative to CAPI-only, though these are fairly modest in magnitude.