Tineke de Jonge

Homogenizing responses to different survey questions on the same topic: Proposal of a scale homogenization method using a reference distribution

AbstractSurvey data are often used for comparison purposes, such as comparisons across nations or comparisons over time. To be effective, this would require equivalent questions and equivalent responses options to the questions. Yet there is a lot of variation in the response scales used, which, for example, differ in the number of response options used and the labeling of these options. This is the case in happiness research, and as a result most of the research data in this field is incomparable. Several methods have been proposed to transform ratings on verbal response scales to a common numerical scale, typically ranging from 0 to 10. In this paper we give an overview of the progress made in those Scale Homogenization methods over time. We describe two early methods: Linear Stretch and the Semantic Judgement of Fixed Word Value Method. Next we discuss the Semantic Judgement of Word Value in Context Method in more detail. Based on these discussions we propose a new Reference Distribution Method. We apply the Semantic Judgement of Word Value in Context and the Reference Distribution Methods to data on happiness in The Netherlands for the years 1989–2009. We show that the Reference Distribution Method produces comparable time series on different questions and that it allows discontinuities in data to be corrected.

The Reference Distribution Method

With the Reference Distribution Method an attempt is made to deal with the fact that, for a given year and a given population, one would expect the distribution means after scale transformation for similar questions about a certain topic asked in different representative surveys to be approximately the same irrespective of the primary response scales used. This provides the basis for another approach to transformation. The Reference Distribution Method for making survey data comparable builds heavily on the Scale Interval Method. Basically the two methods are identical except that in the Reference Distribution Method the boundaries between the response options of the primary scale are derived from a reference distribution instead of being derived from the assessments by judges by means of the Scale Interval Recorder.

Stability of the Boundaries Between Response Options for Different Countries

When the Continuum Approach is applied to a time series of a survey which has remained unchanged over time, the boundaries between the response options are assumed to remain unchanged over time. We inspect whether this assumption is realistic using time series of survey items taken from the Eurobarometer for different parts of Europe and on the topics life satisfaction and satisfaction with the way the democracy works in the country. We conclude that the research question can be answered affirmatively.

Combining and Pooling of Time Series on Life Satisfaction in the USA, Japan, The Netherlands and Spain

We applied the Reference Distribution Method to pool time series on life satisfaction in the USA, Japan, The Netherlands and Spain, using results from the World Values Survey to derive reference distributions from. This resulted in consistent time series spanning almost 60 years for Japan, 40 years for The Netherlands and 35 years for the USA and Spain. Life satisfaction in Japan and The Netherlands was almost equal in the eighties, but at present differs more than one point in favor of The Netherlands. Life satisfaction in Spain reached its lowest value of 6.0 in 2012, but has increased since then to 6.3 in 2015.

Diversity in Survey Items and the Comparability Problem

This chapter starts with an introduction to the incomparability problem and an overview of the diversity in survey items. This is followed by a description of the problem of incomparability of the time series on life satisfaction in the USA, Japan and The Netherlands to illustrate the problem. Next two conventional methods for scale transformation are described: the Linear Stretch Method and the Semantic Judgment of Fixed Word Value Method. We explain why they fall short to solve the comparability problem and conclude that these shortcomings require further investigations and innovative solutions to solve them.

The Labeling of Anchor Points and the Occurrence of Zero-Width Intervals

The labeling of the anchor points of a response scale may tempt the judges participating in scale intervals studies to assign zero-width intervals to these points by choosing the upper and lower bound equal to an extreme of the continuum. The phenomenon of assigning zero-width intervals occurs more frequently when extreme wording is used for an anchor point label and when the response scale consists of more than five response options. Although all-inclusive response scales would encompass the full spectrum of possible experiences, there uses is likely to make anchor points redundant for part of the respondents using such a response scale.

The Continuum Approach

The comparability problem is partly due to the variety of response scales that has developed over time in the field of survey research and is caused by the use of discrete scales. When conventional scale transformation methods are used or the Scale Interval Method to transform a discrete primary scale, the resulting secondary scale is still discrete, and although the use of discrete scales in survey research is often practically motivated, a more valid approach is to consider the existence of a latent continuous variable underlying the survey variable, the distribution of which is estimated using the survey item and the response to it. The Continuum Approach was developed with the notion that happiness should be treated as a continuous variable. In the Continuum Approach the shape parameters α and β of the best fit beta distribution are estimated on a basis of cumulative frequencies and the values on the continuum from 0 to 10 of the boundaries between the response options of the primary scale. The mean μ based on the parameters of this best fit beta distribution is then considered to be an estimator of the mean happiness in a population.

Methods to Increase the Comparability in Cross-National Surveys, Highlight on the Scale Interval Method and the Reference Distribution Method

The Scale Interval Method is a new method to investigate which intervals on a continuum from 0 to 10 are assigned to verbally labeled response options when asked in different questions and languages. The method is very useful for getting insight in the extent to which the interpretation of response options depends on language, culture, and the context of the scale. The Reference Distribution Method is a new method to make the responses to different survey questions on the same topic comparable. The method is based on the idea that, for a given year and a given population, the distribution means after scale transformation for similar questions about happiness asked in different representative surveys should be approximately the same irrespective of the primary response scales used. In this method, the boundaries between the response options are derived from a reference distribution. The method can be applied to combine time series from different surveys on the same topic which span different periods of time into one long time series and to bring the responses to survey questions on the same topic using different response scales to a comparable level.

The Happiness Scale Interval Study

The Happiness Scale Interval Study, is designed to be used for survey questions on happiness using verbal response options. It is used to determine what degrees of happiness are denoted by such terms when used in particular questions and different languages. The method uses a web-based Scale Interval Recorder. First a description is given of this Scale Interval Recorder. Next, the differences with conventional scale transformation methods are discussed. This is followed by an exemplar application of the conventional methods and the Scale Interval Method to existing survey data. The results of these applications are mutually compared from the perspective of the comparability problem.

Analysis of Differences in Trends Among the Satisfied Few and the Dissatisfied Few

Scholars often want to know what the interrelations of the means in subgroups of the population are, for example from the perspective of inequality or marginalization. When using a discrete response scale with a limited number of response options it will be a difficult task to derive this information from the survey results. This difficulty can be overcome if the Reference Distribution Method is applied to these survey results. Once a continuous distribution has been found to estimate a population mean for life satisfaction over time, it is also possible to estimate the means for subgroups of a population such as the least satisfied and the most satisfied part. We use the topics satisfaction with life and satisfaction with how democracy works in Western Europe, Southern Europe and Eastern Europe to demonstrate how the Reference Distribution Method can be used for this purpose.