Cali M. Ellis

Learning Political Science with Prediction Markets: An Experimental Study

Prediction markets are designed to aggregate the information of many individuals to forecast future events.Thesemarkets provide participants with an incentive to seek information and a forum for interaction, making markets a promising tool to motivate student learning. We carried out a quasi-experiment in an introductory political science class to study the effect of prediction markets on student engagement with the course topics. Although we found no significant improvement in students’ enthusiasm or extent of topical reading, we did find that those students whowere already reading broadly at the course start were more likely to trade actively in the markets. These findings indicate that prediction markets may be most successful as an education tool in settings, like graduate education, where individuals are already knowledgeable about the topics of the market, instead of an introductory learning context. Prediction markets (also known as information markets or decision markets) are designed to aggregate the informationofmany individuals to forecast future events. Such markets have been used in a variety of contexts. For example, the Iowa Electronic Market forecasts political and economic events including election outcomes, and the Defense Advanced Research Projects Agency’s (DARPA) proposed Policy Analysis Market was intended to forecast world political, economic, and military actions. The motivation for using markets for forecasting is that traders reveal some privately held information through the trades they carry out, and themarket price aggregates the informationof uncoordinated traders through a “wisdomof crowds” effect.Thesemarkets have been shown to closely approximate the predictive capabilities of public polls and expert opinions (Wolfers andZitzewitz 2004).They have been studied for their effect on public policy choices (Hahn and Tetlock 2003), predictive abilities (Servan-Schreiber et al. 2004), and accuracy in the face of severely limited information (Forsythe et al. 1992). Prediction markets, however, have not been studied for their effect on traders themselves. Pedagogically, these markets are part of a larger trend in education toward including interactive and technological resources in the classroom (Buckley, Garvey, andMcGrath 2011). Among the key advantages of predictionmarkets, researchers have noted that they provide incentives to motivate traders to “ferret out accurate information” and “not amplify individual errors, but eliminate them” (Sunstein 2006). These strengths alignwell with our goals as instructors: wewant to train our students to search for relevant information and critically analyze received information. Prediction markets can be a useful in-class teaching tool for a variety of political science subfields, including American elections and campaigns (Abramson 2010) and public policy (Wolfers andZitzewitz 2004). In the Iowa ElectronicMarkets, for example, the greatest volume of trading occurs prior to the presidential primary and general elections, and instructors may be interested in using prediction markets in the classroom as early as the fall term of 2011 (i.e., the time leading up to the first primaries of 2012 in Iowa and New Hampshire). Prediction markets also provide interesting information for professors of comparative politics about a variety of international elections (for example, InTrade hosted a market about the presidential elections of Brazil in 2010). Yet, until now, we are unaware of any controlled studies of the use of prediction markets as learning tools. We carried out a quasiexperiment in an introductory political science class at a large midwestern university to study the effect of prediction markets on student engagement with the course material. We used a prediction market website, built around the open-source Zocalo software, and created markets relevant to the topic of the course (world politics, for the study reported here). We intentionally pickedmarkets that were tangential rather than CaliMortenson Ellis is a PhD candidate in public policy and political science at the University of Michigan, Ann Arbor. She can be reached at cmortens@umich.edu. Rahul Sami is an associate professor in the School of Information, University of Michigan, Ann Arbor. He can be reached at rsami@umich.edu. The Teache r doi:10.1017/S1049096511002113 PS • April 2012 277 central to the class syllabus because we did not want to give a subset of students an advantage in class performance. We conducted surveys of the entire class at the start and end of the course, with questions intended to measure their engagement with the course topic as well as their knowledge of specific topics. We selected a random group of students to be invited to trade in the market, tracked their individual participation, and provided small monetary rewards based on performance. The results on the effects of prediction markets on student enthusiasm were disappointing: We detected no significant improvement in students’ enthusiasm for the topics of the course or extent of extra reading among students who traded in the markets relative to the control group. However, a deeper analysis revealed that, among the students who were eligible to trade, those who chose to trade actively had a significantly higher prior level of reading in the area. In other words, the students who were already reading broadly at the start of the course were more likely to trade actively in the market. Active traders self-reported a high level of satisfaction with the prediction markets. Our findings are consistent with those of Whitton (2007), who cautions that not all students are receptive to interactive technology in the classroom, warning that, “the sole reason for using [computerbased games] should not be because they are perceived to be motivational.” Our results suggest that instructors may benefit from using prediction markets to engage students with selected topics, but should do cautiously: this tool is enjoyable and useful for some students, but it does not appear to motivate the entire spectrum of students. The remainder of this article is structured as follows: First, we outline observations from our earlier pilot experiments that led to conclusions about how to motivate participation in markets. Second, we describe our experimental methodology. Third, we present descriptive statistics of our student population. Next, we provide reports on our data analysis procedures and the results of this analysis. Finally, we discuss the insights yielded by this analysis, the limitations of this study, and interesting directions for future research. TWO-PHASE PILOT STUDY A pilot study was carried out in the spring semester of 2009 to gather baseline data about participation in and response to decision markets in a pedagogical setting. This pilot used customized online predictionmarkets in a large undergraduate course onworld politics. Our original primary hypothesis was that participation in prediction markets would increase students’ enthusiasm for the topic of world politics. We supposed that students would research the market topics to increase their information advantage and the likelihood of receiving cash rewards. In our pilot phase, 166 studentswere randomly divided into treatment (trader) and control (nontrader) groups. Presurveys and postsurveys were conducted with both groups at semester start and finish. Treated students (traders) were allowed to trade in the market with virtual points. Surveys at the beginning and end of the semester evaluated student enthusiasm for the subject matter. Unfortunately, we only obtained a 7% compliance rate among the treated (6 traders), despite 92% awareness of the potential for cash prizes. We also found underreporting of treatment status by more than half (i.e., students assigned to be traders did not report that they knew they were in the treatment group; 41 reported treatment, 83 actually treated). Low compliance among the treatment group, a feature typical of experimental studies of predictionmarkets (Forsythe et al. 1992;Wolfers and Zitzewitz 2004), limited our ability to make inferences from the pilot study. As a result, in fall 2009, we performed a nonrandomized study in a smaller (N 22) elective upperlevel undergraduate course that focused on war in international relations. We changed the interface and markets, implemented a different incentive structure, and administeredmoredetailed and indepth surveys to understand what students claim would be motivating to them. We collected unique identifiers to create a panel dataset including both survey responses and trading behavior.We also developed a new quiz about topics related exclusively to the markets to determine changes in knowledge levels at the subject level. Finally, we introduced students to the prediction market experiment using a live in-class demonstration, during which they were able to ask the researchers questions. Both phases of the pilot study informed the design of our final experiment, which is reported later in this article.

Introducing the LEAD Data Set

The Leader Experience and Attribute Descriptions (LEAD) data set provides a rich source of new information about the personal lives and experiences of over 2,000 state leaders from 1875–2004. For the first time, we can combine insights from psychology and human development with large-N data on interstate conflict for a new theory of leadership and interstate relations. The data set provides details about military experiences, childhood, education, personal and family life, and occupational history before leaders assumed power. The data are available in leader-year format and are compatible with existing tools for analysis such as EUGene (Bennett and Stam 2000). This research note discusses the motivation for the creation of the LEAD data set and discusses the coding decisions for most of the key variables. We provide a series of descriptive statistical illustrations of the data and illustrate the depth of the available information with cases from Latin American leaders, showing the durability of these personal experiences across space and time.

Leader Risk across Geography and Time

How important are leader attributes, compared with the predictions of system-level theories, when it comes to the outbreak of international conflict? The previous chapter began the process of systematically examining the importance of leaders by describing the creation of an index of leader risk that measures, based on the background experiences of leaders, their propensity to start international conflicts. This chapter compares the Leader Risk Index to more traditional explanations for warfare. It maps conflict across the world, as well as across specific regions, from 1875 to 2001, to highlight the importance of understanding leader risk. Using advanced statistical models, the results in this chapter show that while the effects of leaders are almost certainly influenced by the relative power of their countries and the domestic political contexts in which they operate, leaders still exercise independent influence over international politics. Mapping Leader Risk The empirical literature on war outbreak and escalation provides strong evidence for the importance of analysis at the regional level. International relations theory acknowledges the fundamental importance of geography as a risk factor either mitigating or exacerbating the risks of interstate war. John Mearsheimer notes the “stopping power of water,” for example, that makes it difficult to project power across oceans even for the most powerful countries in the world. Mountains, rivers, and swamps make for relatively stable national borders because they greatly increase the difficulty of projecting military power. States such as Poland, with borders lacking natural defenses, have been among the most frequently conquered. One of the strongest findings from empirical research on international conflict, not surprisingly, is that contiguous neighbors are more likely to fight, whether the disputes are related to territory, revenge for past grievances, or ethnic conflict. In attempting to understand the role of leaders in shaping the distribution of international conflict more generally, one approach is to look at particular regions and the distribution, or lack thereof, of conflict-prone leaders. Figure 3.1 shows a global risk map for leaders in office in 1989, at the end of the Cold War. In this map and the ones that follow, the color corresponds to the risk level associated with each countrys leader. Compare the leader-risk data in Figure 3.1 with the same map based solely on the same “system” level models of international conflict used in Chapter 2, in Figure 3.2.

Prediction markets for education: an experimental study

In this letter, we report the results of a quasi-experimental study of prediction markets as a pedagogical tool in an undergraduate setting.