Jolene D. Smyth

Using the Internet to Survey Small Towns and Communities: Limitations and Possibilities in the Early 21st Century

Researchers who are interested in small towns and rural communities in the United States often find that they need to conduct their own sample surveys because many large national surveys, such as the American Community Survey, do not collect enough representative responses to make precise estimates. In collecting their own survey data, researchers face a number of challenges, such as sampling and coverage limitations. This article summarizes those challenges and tests mail and Internet methodologies for collecting data in small towns and rural communities using the U.S. Postal Service’s Delivery Sequence File as a sample frame. Findings indicate that the Delivery Sequence File can be used to sample households in rural locations by sending them invitations via postal mail to respond to either paper-and-pencil or Internet surveys. Although the mail methodology is quite successful, the results for the Internet suggest that Web surveys alone exclude potentially important segments of the population of small towns and rural communities. However, Web surveys supplemented with postal questionnaires produce results quite similar to those of mail-only surveys, representing a possible cost savings for researchers who have access to Web survey capabilities.

The Effects of Item Saliency and Question Design on Measurement Error in a Self-Administered Survey

Recent survey design research has shown that small changes in the structure and visual layout of questions can affect respondents’ answers, but the results are not always consistent across studies. One possible reason for some of the inconsistency may be differences in the item saliency of the questions used in the experiments. In this article, the authors examine how item saliency might influence visual design effects. The authors report the results of three experimental alterations in question format and visual design using data from a 2005 random sample mail survey of 1,315 households. The results suggest that the saliency of the questions has effects both independent of and in concert with the layout of the questions. The implications for survey design are discussed.

Accuracy of Within-household Selection in Web and Mail Surveys of the General Population

Household surveys are moving from interviewer-administered modes to self-administered modes for data collection, but many households do not accurately follow within-household selection procedures in mail surveys. In this article, we examine accuracy of within-household selection using an oldest adult/youngest adult method in web, mail, and mixed-mode surveys. The frame for this study comes from a telephone survey conducted with Nebraska residents in which the oldest adult/youngest adult method is used to select the initial respondent. One year later, these telephone participants are followed up using identical household selection methods. This article examines characteristics of people who followed the selection procedures compared to those who did not.

Effects of Smiley Face Scales on Visual Processing of Satisfaction Questions in Web Surveys

Web surveys permit researchers to use graphic or symbolic elements alongside the text of response options to help respondents process the categories. Smiley faces are one example used to communicate positive and negative domains. How respondents visually process these smiley faces, including whether they detract from the question’s text, is understudied. We report the results of two eye-tracking experiments in which satisfaction questions were asked with and without smiley faces. Respondents to the questions with smiley faces spent less time reading the question stem and response option text than respondents to the questions without smiley faces, but the response distributions did not differ by version. We also find support that lower literacy respondents rely more on the smiley faces than higher literacy respondents.

Using a Calendar and Explanatory Instructions to Aid Within-household Selection in Mail Surveys

Although researchers can easily select probability samples of addresses using the U.S. Postal Service’s Delivery Sequence File, randomly selecting respondents within households for surveys remains challenging. Researchers often place within-household selection instructions, such as the next or last birthday methods, in survey cover letters to select respondents. Studies show that up to 30% of selections are inaccurate and samples tend not to represent key demographics. This article tests two design elements—a calendar and explanatory wording of selection instructions in cover letters—to aid and motivate households to carry out selection procedures accurately. We empirically examine these elements in two mail surveys of Nebraskans—the 2012 Nebraska Annual Social Indicators Survey and 2013 Nebraska Trees and Forest Survey. We find that neither the calendar nor the explanatory wording adversely affected response rates but that neither improved the representativeness of the completed samples and that the calendar actually reduced selection accuracy.

Drawing on LGB Identity to Encourage Participation and Disclosure of Sexual Orientation in Surveys

ABSTRACT This paper reports an experiment that tested how three survey cover designs – images of traditional families and individuals displaying themselves in typical gender ways; images of lesbian, gay, and bisexual (LGB) and heterosexual individuals and families; and no cover images – affected LGB people’s participation and disclosure of LGB identity and non-LGB people’s participation. Analyses showed the LGB-inclusive cover led to significantly more LGB respondents than the other designs, without significantly affecting the demographic, political, and religious makeup of the completed sample. We discuss what these findings mean for addressing two challenges: getting LGB people to respond to surveys and to disclose their LGB identity.

Item Location, the Interviewer–respondent Interaction, and Responses to Battery Questions in Telephone Surveys:

Survey researchers often ask a series of attitudinal questions with a common question stem and response options, known as battery questions. Interviewers have substantial latitude in deciding how to administer these items, including whether to reread the common question stem on items after the first one or to probe respondents’ answers. Despite the ubiquity of use of these items, there is virtually no research on whether respondent and interviewer behaviors on battery questions differ over items in a battery or whether interview behaviors are associated with answers to these questions. This article uses a nationally representative telephone survey with audio-recorded interviews and randomized placement of items within four different batteries to examine interviewer and respondent behaviors and respondent answers in battery questions. Using cross-classified random-effects models, the authors find strong evidence that there is more interviewer–respondent interaction on items asked earlier in the battery. In addition, interviewer and respondent behaviors are associated with both substantive and nonsubstantive answers provided to battery items, especially if the interviewer decided to reread or probe with the response options. These results suggest that survey designers should follow recommendations to randomize battery items and consider the importance of standardization of question administration when designing battery questions.

Comparing survey ranking question formats in mail surveys

Although questions that ask respondents to rank-order a list of items can be analytically valuable, responding to ranking questions typically requires a good deal of cognitive effort. This is especially true in mail questionnaires where the advantages of electronic response formats available in web surveys are inaccessible. In this article, we examine two alternative formats for ranking questions in mail surveys. Using a nationally representative mail survey of U.S. adults, this article experimentally compares ranking formats in which respondents write numbers in boxes versus selecting items for the most and second most important issues using a grid layout. Respondents to the numbering format were more likely to provide usable data, although one-third of respondents in this format still did not follow instructions correctly. Substantive responses differed somewhat across formats. Less educated respondents had difficulty with both formats, resulting in substantively different conclusions about preferences across formats for this group. A numbering format is more effective than a most–second most grid format for collecting ranking data in mail surveys.

Understanding the Human Condition through Survey Informatics

Survey informatics leverages two separate but vital areas of research--survey methodology and computer science and engineering--to advance the state of the art in each and improve our understanding of the human experience.