Confirming the continued representativeness of an online/telephone panel using equivalence testing

Abstract

Decreasing response rates to traditional survey methods, like face-to-face and telephone interviews, have led survey practitioners around the world to seek new ways of conducting surveys in recent years.” The COVID-19 pandemic exacerbated this problem because it made conducting face-to-face interviews even more difficult than before. For example, it made conducting face-to-face surveys infeasible in 2020 in South Korea, and so the Korean Academic Multimode Open Survey (KAMOS) was unable to conduct a planned face-to-face survey to recruit new panel members. The entire 8,514-member panel, established via two-stage probability-based sampling from 2016 to 2019, was invited to take three online/telephone surveys in 2020. Of these panel members, 1,352 responded to at least one survey in 2020. To test to what extent the panel remained representative of the adult South Korean population, we compared the two groups of panel members: Those who responded to at least one survey in 2020 and those who did not. After weighting both groups on the basis of age, sex, and geographical area, we analyzed their responses to some of the questions that were asked during multiple rounds of the face-to-face panel-recruiting interviews. Using Cohen’s d for survey items that could be analyzed numerically and Cramer’s V for categorical items, we were able to conclude that the respondents to the 2020 surveys were equivalent to the non-respondents in terms of both demographics and in the answers they originally gave to substantive questions on a variety of topics related to social science or public opinion research, including questions about quality of life, societal issue, and politics (Cohen s d items <0.2, 95% CI;Cramer’s V items <0.1, 95% CI). This analysis may provide a model for others who wish to test the continued representativeness of their panel or who would like to use a different survey mode or change some other aspect of their methodology and test whether it is equivalent to their former methodology. Our success in building a panel that retained its representativeness may be useful to those in other countries where face-to-face surveys had previously been the norm but are becoming increasingly difficult to conduct. © 2021, Center for Asian Public Opinion Research and Collaboration Initiative. All rights reserved.