Mick P. Couper

Psychological research online: report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet.

As the Internet has changed communication, commerce, and the distribution of information, so too it is changing psychological research. Psychologists can observe new or rare phenomena online and can do research on traditional psychological topics more efficiently, enabling them to expand the scale and scope of their research. Yet these opportunities entail risk both to research quality and to human subjects. Internet research is inherently no more risky than traditional observational, survey, or experimental methods. Yet the risks and safeguards against them will differ from those characterizing traditional research and will themselves change over time. This article describes some benefits and challenges of conducting psychological research via the Internet and offers recommendations to both researchers and institutional review boards for dealing with them. ((c) 2004 APA, all rights reserved)

Web Survey Design and Administration

Many claims are being made about the advantages of conducting surveys on the Web. However, there has been little research on the effects of format or design on the levels of unit and item response or on data quality. In a study conducted at the University of Michigan, a number of experiments were added to a survey of the student population to assess the impact of design features on resulting data quality. A sample of 1,602 students was sent an e-mail invitation to participate in a Web survey on attitudes toward affirmative action. Three experiments on design approaches were added to the survey application. One experiment varied whether respondents were reminded of their progress through the instrument. In a second experiment, one version presented several related items on one screen, while the other version presented one question per screen. In a third experiment, for one series of questions a random half of the sample clicked radio buttons to indicate their answers, while the other half entered a numeric response in a box. This article discusses the overall implementation and outcome of the survey, and it describes the results of the imbedded design experiments.

UNDERSTANDING THE DECISION TO PARTICIPATE IN A SURVEY

The lack of full participation in sample surveys threatens the inferential value of the survey method. We review a set of conceptual developments and experimental findings that appear to be informative about causes of survey participation; offer an integration of that work with findings from the more traditional statistical and survey methodological literature on nonresponse; and, given the theoretical structure, deduce potentially promising paths of research toward the understanding of survey participation.

Web surveys: perceptions of burden

Web surveys appear to be attaining lower response rates than equivalent mail surveys. One reason may be that there is currently little information on effective strategies for increasing response to Internet-based surveys. Web users are becoming more impatient with high-burden Web interactions. The authors examined the decision to respond to a Web survey by embedding a series of experiments in a survey of students at the University of Michigan. A sample of over 4,500 students was sent an e-mail invitation to participate in a Web survey on affirmative action policies. Methodological experiments included using a progress indicator, automating password entry, varying the timing of reminder notices to nonrespondents, and using a prenotification report of the anticipated survey length. Each of these experiments was designed to vary the burden (perceived or real) of the survey request. Results of these experiments are presented.

The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial.

Background Web-based programs for health promotion, disease prevention, and disease management often experience high rates of attrition. There are 3 questions which are particularly relevant to this issue. First, does engagement with program content predict long-term outcomes? Second, which users are most likely to drop out or disengage from the program? Third, do particular intervention strategies enhance engagement? Objective To determine: (1) whether engagement (defined by the number of Web sections opened) in a Web-based smoking cessation intervention predicts 6-month abstinence, (2) whether particular sociodemographic and psychographic groups are more likely to have lower engagement, and (3) whether particular components of a Web-based smoking cessation program influence engagement. Methods A randomized trial of 1866 smokers was used to examine the efficacy of 5 different treatment components of a Web-based smoking cessation intervention. The components were: high- versus low-personalized message source, high- versus low-tailored outcome expectation, efficacy expectation, and success story messages. Moreover, the timing of exposure to these sections was manipulated, with participants randomized to either a single unified Web program with all sections available at once, or sequential exposure to each section over a 5-week period of time. Participants from 2 large health plans enrolled to receive the online behavioral smoking cessation program and a free course of nicotine replacement therapy (patch). The program included: an introduction section, a section focusing on outcome expectations, 2 sections focusing on efficacy expectations, and a section with a narrative success story (5 sections altogether, each with multiple screens). Most of the analyses were conducted with a stratification of the 2 exposure types. Measures included: sociodemographic and psychosocial characteristics, Web sections opened, perceived message relevance, and smoking cessation 6-months following quit date. Results The total number of Web sections opened was related to subsequent smoking cessation. Participants who were younger, were male, or had less formal education were more likely to disengage from the Web-based cessation program, particularly when the program sections were delivered sequentially over time. More personalized source and high-depth tailored self-efficacy components were related to a greater number of Web sections opened. A path analysis model suggested that the impact of high-depth message tailoring on engagement in the sequentially delivered Web program was mediated by perceived message relevance. Conclusions Results of this study suggest that one of the mechanisms underlying the impact of Web-based smoking cessation interventions is engagement with the program. The source of the message, the degree of message tailoring, and the timing of exposure appear to influence Web-based program engagement.

Technology Trends in Survey Data Collection

This article reviews recent and emerging technology developments in survey research. Recent developments include computer-assisted self-interviewing (CASI) methods using audio and/or video, automated telephone interviewing systems (interactive voice response), and the World Wide Web. These developments are already having a profound effect on survey data collection. Newer technological challenges include wireless applications (mobile web) and portable digital devices. These technologies offer many opportunities to expand the way we think of survey data collection, increasing the ways we can interact with survey respondents and expanding the range of stimulus material that can be used. The implications of many of these new developments for survey data quality are yet to be fully understood. This article reviews the state of the field with regard to emerging data collection technologies, and their implications for survey research.

Selection Bias in Web Surveys and the Use of Propensity Scores

Web surveys are a popular survey mode, but the subpopulation with Internet access may not represent the population of interest. The authors investigate whether adjusting using weights or matching on a small set of variables makes the distributions of target variables representative of the population. This application has a rich sampling design; the Internet sample is part of an existing probability sample, the Health and Retirement Study, that is representative of the U.S. population aged 50 and older. For the dichotomous variables investigated, the adjustment helps. On average, the sample means in the Internet access sample differ by 6.5 percent before and 3.7 percent after adjustment. Still, a large number of adjusted estimates remain significantly different from their target estimates based on the complete sample. This casts doubt on the common procedure to use only a few variables to correct for the selectivity of convenience samples.

Decision-making processes for breast, colorectal, and prostate cancer screening: the DECISIONS survey.

Background Patients should understand the risks and benefits of cancer screening in order to make informed screening decisions. Objectives To evaluate the extent of informed decision making in patient-provider discussions for colorectal (CRC), breast (BrCa), and prostate (PCa) cancer screening. Setting National sample of US adults identified by random-digit dialing. Design Cross-sectional survey conducted between November 2006 and May 2007. Participants English-speaking US adults aged 50 y and older who had discussed cancer screening with a health care provider within the previous 2 y. Measurements Cancer screening survey modules that asked about demographic characteristics, cancer knowledge, the importance of various sources of information, and self-reported cancer screening decision-making processes. Results Overall, 1082 participants completed 1 or more of the 3 cancer modules. Although participants generally considered themselves well informed about screening tests, half or more could not correctly answer even 1 open-ended knowledge question for any given module. Participants consistently overestimated risks for being diagnosed with and dying from each cancer and overestimated the positive predictive values of prostate-specific antigen tests and mammography. Providers were the most highly rated information source, usually initiated screening discussions (64%–84%), and often recommended screening (73%–90%). However, participants reported that providers elicited their screening preferences in only 31% (CRC women) to 57% (PCa) of discussions. Although more than 90% of the discussions addressed the pros of screening, only 19% (BrCa) to 30% (PCa) addressed the cons of screening. Limitations Recall bias is possible because screening process reports were not independently validated. Conclusions Cancer screening decisions reported by patients who discussed screening with their health care providers consistently failed to meet criteria for being informed. Given the high ratings for provider information and frequent recommendations for screening, providers have important opportunities to ensure that informed decision making occurs for cancer screening decisions.

What they see is what we get: response options for web surveys

Several alternative response formats are available to the web survey designer, but the choice of format is often made with little consideration of measurement error. The authors experimentally explore three common response formats used in web surveys: a series of radio buttons, a drop box with none of the options initially displayed until the respondent clicks on the box, and a scrollable drop box with some of the options initially visible, requiring the respondent to scroll to see the remainder of the options. The authors reversed the order of the response options for half the sample. The authors find evidence of response order effects but stronger evidence that visible response options are endorsed more frequently, suggesting that visibility may be a more powerful effect than primacy in web surveys. The results suggest that the response format used in web surveys does affect the choices made by respondents.

Prostate cancer screening decisions: Results from the National Survey of Medical Decisions (DECISIONS study)

BACKGROUND Guidelines recommend informing patients about the risks and benefits of prostate cancer screening. We evaluated the medical decision-making process for prostate-specific antigen (PSA) testing. METHODS We conducted a telephone survey of a randomly selected national sample of 3010 English-speaking US adults 40 years and older. Included in the survey were 375 men who had either undergone or discussed (with health care providers [HCPs]) PSA testing in the previous 2 years. We asked subjects about sociodemographic characteristics, prostate cancer screening discussion features, prostate cancer knowledge, and the importance of various decision factors and sources of information. RESULTS Overall, 69.9% of subjects discussed screening before making a testing decision, including 14.4% who were not tested. Health care providers most often (64.6%) raised the idea of screening, and 73.4% recommended PSA testing. Health care providers emphasized the pros of testing in 71.4% of discussions but infrequently addressed the cons (32.0%). Although 58.0% of subjects felt well-informed about PSA testing, 47.8% failed to correctly answer any of the 3 knowledge questions. Only 54.8% of subjects reported being asked for their screening preferences. An HCP recommendation (odds ratio, 2.67; 95% confidence interval, 1.08-6.58) was the only discussion characteristic associated with testing. Valuing HCP information was also associated with testing (odds ratio, 1.26; 95% confidence interval, 1.04-1.54). CONCLUSIONS Recommendations and information from HCPs strongly influenced testing decisions. However, most prostate cancer screening decisions did not meet criteria for shared decision making because subjects did not receive balanced discussions of decision consequences, had limited knowledge, and were not routinely asked for their preferences.