Anja S. Göritz

The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panels

Two incentive experiments were conducted in different online access panels. Experiment 1 was carried out in a commercial market research panel. It examined whether three different types of promised incentives (redeemable bonus points, money lottery and gift lottery), four different amounts of bonus points or raffled money, and two different denominations of raffled money influenced response quantity, sample composition, response quality and survey outcome. Type of incentive and number of bonus points mildly influenced dropout and sample composition. Moreover, response was higher with bonus points than with the two types of lotteries. Response quality and survey outcome were not affected. Experiment 2 was conducted in a non-profit panel, which holds one half selfselected and one half non-self-selected participants. Incentives were two different amounts of raffled money in two different denominations. Response, dropout, response quality, survey outcome and sample composition were not affected. Based on a cost-benefit analysis, recommendations for employing incentives in online access panels are given.

Cash lotteries as incentives in online panels

Six incentive experiments were conducted in a nonprofit online panel. In each experiment, the incentive offered for participation was a cash lottery. The control group was not offered any incentive. The cash lottery was offered in two versions: Either the total payout of the lottery was mentioned, or the lottery was split into multiple prizes. Dependent measures included response and retention rates. The results of the six individual experiments were meta-analytically summarized. Cash lotteries relative to no incentives did not reliably increase response or retention; neither did it make a significant difference if one large prize or multiple smaller prizes were raffled.

Recruitment for online access panels.

This paper describes a German study which compared eight ways of recruiting members for an online access panel. Two thousand respondents, divided into four groups of 500, were invited to sign up with the panel via email, fax, flier or letter. Half of each sample’s invitations offered a cash lottery, into which new panellists would be entered, whereas the other half of the invitations did not offer a lottery. Overall, email was the most successful means of solicitation, followed by flier and fax, which were equally efficient. Very few panellists were recruited via letter. The lottery was effective only with fliers. The composition of the recruited samples differed according to solicitation method. Fax-recruited individuals were older than those recruited by flier and email. Panellists recruited via email had been using the internet longer than flier- and fax-recruited panellists and they used the internet more often than those recruited via fax. After their recruitment, panellists were followed up in the first two studies run in the panel. The probability of their taking part in these studies and of completing these questionnaires was independent of the method by which they had been recruited.During the past decade, data collection via the world wide web has been established in theory and practice and various studies have shown that web-delivered research can yield valid results (e.g. Gritz & Schumacher 2000; McGraw et al. 2000; Gritz 2002). With regard to the recruitment of participants for web-based studies one can distinguish two forms. In ad hoc recruitment, participants are solicited on the basis of a stand-alone study, for example through search engines, banners, links, newsletters, news groups, mailing lists, word of mouth, or via offline media. Ad hoc recruitment is often expensive and its success is unpredictable.

Using the World-Wide Web to obtain large-scale word norms: 190,212 ratings on a set of 2,654 German nouns

This article presents a new database of 2,654 German nouns rated by a sample of 3,907 subjects on three psycholinguistic attributes: concreteness, valence, and arousal. As a new means of data collection in the field of psycholinguistic research, all ratings were obtained via the Internet, using a tailored Web application. Analysis of the obtained word norms showed good agreement with two existing norm sets. A cluster analysis revealed a plausible set of four classes of nouns: abstract concepts, aversive events, pleasant activities, and physical objects. In an additional application example, we demonstrate the usefulness of the database for creating parallel word lists whose elements match as closely as possible. The complete database is available for free from ftp://ftp.uni-duesseldorf.de/pub/psycho/lahl/WWN. Moreover, the Web application used for data collection is inherently capable of collecting word norms in any language and is going to be released for public use as well.

Lotteries as Incentives in Longitudinal Web Studies

A longitudinal experiment examined the influence of a lottery of gift certificates on response and retention in a four-wave study conducted in an online panel. Independent of the lottery, people who responded in a given wave were more likely to respond in the next wave. This process was characterized to follow a first-order Markov chain. There was a direct positive effect of the lottery on response only at the first wave of the study. However, mediated by the Markov process, the positive effect of the lottery on response at the first wave was carried over into later waves. The lottery did not have any effect on retention. Furthermore, it was found that retention at a given wave is a reliable predictor for response at the next wave. This information could be used by survey managers to diagnose and act on any impending nonresponse.

Individual payments as a longer-term incentive in online panels.

Does it pay to pay online panel members? A three-wave longitudinal experiment was conducted with an online panel to examine whether per person payments, paid through an online intermediary, influence response and retention rates. In the payment condition, participants were promised payment for participation at each wave, whereas control participants were not offered any payment. The promise of a payment had a negative effect on response in Wave 1, but a positive effect on response in Wave 2. Payment had no significant effect on retention. Completing a given wave was an indicator for responding to a subsequent invitation.

Web-based mood induction

This article introduces the methodology of Web-based mood induction and it presents empirical evidence for its feasibility. The Velten procedure, autobiographical recall, and mood-suggestive photographs were tested for their effectiveness in evoking either a positive or a negative mood when compared to an untreated control group. The Velten technique and the photographs successfully deteriorated mood, but were unsuccessful in generating a positive mood. Autobiographical recall failed to induce either positive or negative mood. The effectiveness of these on-line techniques is compared with the effectiveness of similar off-line mood induction procedures.

Generic HTML Form Processor: A versatile PHP script to save Web-collected data into a MySQL database

The customizable PHP script Generic HTML Form Processor is intended to assist researchers and students in quickly setting up surveys and experiments that can be administered via the Web. This script relieves researchers from the burdens of writing new CGI scripts and building databases for each Web study. Generic HTML Form Processor processes any syntactically correct HTML form input and saves it into a dynamically created open-source database. We describe five modes for usage of the script that allow increasing functionality but require increasing levels of knowledge of PHP and Web servers: The first two modes require no previous knowledge, and the fifth requires PHP programming expertise. Use of Generic HTML Form Processor is free for academic purposes, and its Web address is www.goeritz.net/brmic.

Building and managing an online panel with phpPanelAdmin

Online data collection within social and behavioral research is burgeoning. For surveying individuals online, researchers are increasingly relying on online panels and similar subject pools. phpPanelAdmin is a Web-based platform that is intended to assist researchers in quickly setting up and managing an online panel. phpPanelAdmin solves many problems that can arise when one is collecting data in online panels. As an open-source program placed under GNU General Public License, phpPanelAdmin (www.goeritz.net/panelware) is not only free of charge but free of restrictions on its use, modification, and distribution. Building a new online panel—which before required weeks and special skills—is now possible within minutes and at no cost. Furthermore, phpPanelAdmin can be connected to an existing online panel or even be used for managing offline subject pools. Functions include creating a panel sign-up form; automatically unfolding the panel database; searching for panelists; viewing, modifying, and deleting panelists’ data; deleting and cloning variables; drawing samples; sending e-mail to panelists; exporting data; displaying panel statistics at a glance; identifying duplicates; creating and managing e-mail templates; executing user-defined SQL statements; and obtaining help.

The high-hurdle technique put to the test: Failure to find evidence that increasing loading times enhances data quality in Web-based studies

Two Web-based experiments examined the usefulness of artificially delaying the loading of the first page of the study. The idea pursued in this technique is to filter out less-motivated respondents through a higher respondent burden in the form of waiting time. Participants who remain in the study despite having had to wait for the first page of the study to appear on the screen are expected to be more highly motivated, and thus to produce data of higher quality. In both experiments, as expected, the longer the loading time, the lower the likelihood of people responding to the study. However, contrary to expectation, the dropout rate and quality of data were independent of the loading time. Therefore, artificially delaying the loading of the first page of the study is counterproductive.