On the Meaning of Survey Reports of Roll‐Call “Votes”

Abstract

Abstract: Contemporary efforts to evaluate the quality of representation often use survey measures of how citizens say they would vote on legislation to compare citizen preferences to what elected officials do in office. These comparisons often suggest poor representation. We argue here that this common design is unlikely to effectively evaluate representation because responses to survey questions, even on roll call votes, differ in important aspects from voting in legislatures. This leads to systematic measurement error that undermines the key assumption that survey responses measure preferences in the same policy space as legislative behavior. Results from two survey experiments show that providing information readily available to legislators though not readily available to survey respondents materially changes respondents’ expressed preferences on roll call votes. Expressed policy positions in this “informed” survey condition indicate survey respondents are both more extreme than commonly estimated and more closely matched to legislator behavior in their preferred party. Information increases party splits among respondents by 40 and 60 percent. We also show that respondents appear aware of their own lack of knowledge in evaluating roll call policy votes and that the size of the treatment effect of information decreases in the confidence judging policy in that area.