SOBA: Secrecy-preserving Observable Ballot-level Audit
Josh Benaloh, Douglas Jones, Eric Lazarus, Mark Lindeman, Philip B. Stark
aa r X i v : . [ s t a t . A P ] J u l SOBA: Secrecy-preserving Observable Ballot-level Audit
Josh Benaloh, Microsoft ResearchDouglas Jones, Department of Computer Science, University of IowaEric L. Lazarus, DecisionSmithMark LindemanPhilip B. Stark, Department of Statistics, University of California, Berkeley
Abstract
SOBA is an approach to election verification that pro-vides observers with justifiably high confidence that thereported results of an election are consistent with an audittrail (“ballots”), which can be paper or electronic. SOBAcombines three ideas: (1) publishing cast vote records(CVRs) separately for each contest, so that anyone canverify that each reported contest outcome is correct, ifthe CVRs reflect voters’ intentions with sufficient accu-racy; (2) shrouding a mapping between ballots and theCVRs for those ballots to prevent the loss of privacy thatcould occur otherwise; (3) assessing the accuracy withwhich the CVRs reflect voters’ intentions for a collectionof contests while simultaneously assessing the integrityof the shrouded mapping between ballots and CVRs bycomparing randomly selected ballots to the CVRs thatpurport to represent them. Step (1) is related to workby the Humboldt County Election Transparency Project,but publishing CVRs separately for individual contestsrather than images of entire ballots preserves privacy.Step (2) requires a cryptographic commitment from elec-tions officials. Observers participate in step (3), whichrelies on the “super-simple simultaneous single-ballotrisk-limiting audit.” Step (3) is designed to reveal rel-atively few ballots if the shrouded mapping is proper andthe CVRs accurately reflect voter intent. But if the re-ported outcomes of the contests differ from the outcomesthat a full hand count would show, step (3) is guaranteedto have a large chance of requiring all the ballots to becounted by hand, thereby limiting the risk that an incor-rect outcome will become official and final.
The majority of Americans now vote electronically, ei-ther on machine-counted paper ballots or on DirectRecording Electronic (DRE) machines. Electronic vot-ing offers advantages over hand counts and lever ma-chines, but it poses challenges for determining whether votes were recorded and counted correctly. A wide rangeof security vulnerabilities and other flaws have been doc-umented in contemporary voting equipment. The 2007“Top-to-Bottom Review” of the systems used in Califor-nia found that all the systems had “serious design flaws”and “specific vulnerabilities, which attackers could ex-ploit to affect election outcomes” [Bowen, 2007]. Whilesome of these vulnerabilities can be mitigated, the under-lying verification challenge is formidable. As Rivest andWack comment, “complexity is the enemy of security,”and demonstrating that any complex system is free offaults may be impossible or infeasible [Rivest and Wack,2006].Electronic voting systems have failed in real elec-tions. In the 2004 general election in Carteret County,North Carolina, over 4,000 votes were lost irretriev-ably due to a programming error that affected UniLectPatriot voting machines, casting doubt on a statewideelection outcome [Bonner, 2004]. More controversially,in the 2006 general election, ES&S iVotronic DREsin Sarasota County, Florida did not record a vote forU.S. House for about 15% of voters—far more than canplausibly be attributed to intentional undervoting. Inad-vertent undervotes were probably decisive in that con-test [Ash and Lamperti, 2008; Mebane and Dill, 2007].Hypotheses explaining these undervotes include voterconfusion caused by poor ballot layout [Frisina et al.,2008] and machine failure [Garber, 2008; Mebane,2009]. Unfortunately, the forensic evidence generatedby the voting systems was inadequate to determine thecause of the undervotes or the intentions of the voters.Voter-marked paper ballots provide a clearer recordof what voters did and more evidence about voter in-tent, but by themselves do not solve the election verifica-tion problem. In 2005, Harri Hursti repeatedly demon-strated the ability to “hack” optical scan counts whengiven access to a memory card [Zetter, 2005]. In a June2006 primary election in Pottawattamie County, Iowa,incorrectly configured optical scanners miscounted ab-1entee ballots in every contest, altering two outcomes.The county auditor ordered a hand recount, which cor-rected the errors [Flaherty, 2006]. Similar errors in otherelections may have altered outcomes without ever beingdetected. Even when scanners work correctly, their re-sults may differ materially from voter intent. Considerthe 2006 U.S. Senate contest in Minnesota, where AlFranken beat Norm Coleman in a hand recount largelybecause of ballots where the human interpretation dif-fered from the machine interpretation. Computerized election equipment cannot be infallible,so Rivest and Wack [2006] and Rivest [2008] suggestthat voting systems should be software-independent. Avoting system is software-independent “if an undetectedchange or error in its software cannot cause an unde-tectable change or error in an [apparent] election out-come.” This idea can be generalized to define indepen-dence from hardware and from elections personnel, lead-ing to so-called end-to-end verifiable election technolo-gies. However, end-to-end technology may require fun-damental changes in current voting processes.The outcome of a contest is the set of winners, not theexact vote counts. The apparent outcome of a contestis the winner or winners according to the voting system.The correct outcome of a contest is the winner or win-ners that a full hand count of the “audit trail” would find.The audit trail is assumed to be an indelible record ofhow voters cast their votes. It might consist of a com-bination of voter-marked paper ballots, voter receipts, avoter-verifiable paper audit trail (VVPAT), and suitableelectronic records.This definition of “correct” is generally a matter oflaw. It does not necessarily imply that the audit trailis inviolate (nor that the outcome according to the audittrail is the same as the outcome according to how votersoriginally cast their ballots); that there is no controversyabout which records in the audit trail reflect valid votes;that human observers agree on the interpretation of theaudit trail; that the actual hand counting is accurate; northat repeating the hand count would give the same an-swer. If there is no audit trail, defining what it means forthe apparent outcome to be correct requires hypotheticalcounterfactuals—but for the fault in the voting system,what would the outcome have been?Software independence means that errors that causeapparent outcomes to be wrong leave traces in the audittrail. But software independence does not guarantee anyof the following: The 2000 presidential election may have been decided by differ-ences between the machine interpretation of certain Florida optical scanballots and the likely human interpretation [Keating, 2002].
1. that no such traces will occur if the apparent out-come is correct
2. that those traces will be noticed or acted upon3. that the cost of looking through the audit trail forthose traces is affordable4. that, in principle, there is a way to correct the appar-ent outcome without holding another election5. that, in practice, the audit trail was preserved andprotected well enough to determine the outcome ac-cording to how the voters originally cast their bal-lotsThe penultimate property is guaranteed by strong soft-ware independence. Rivest and Wack [2006] and Rivest[2008] define a voting system to be strongly software-independent if an undetected change or error in its soft-ware cannot cause an undetectable change or error in an[apparent] election outcome, and moreover, a detectedchange or error in an [apparent] election outcome (due tochange or error in the software) can be corrected withoutre-running the election. Having an audit trail does notguarantee that anyone will dig through it to see whetherthere is a problem or to correct the outcome if the out-come is wrong. Strong software independence does notcorrect anything, but it is an essential ingredient for asystem to be self-correcting.
Compliance audits can be used to assess whether thelast property listed above holds: Given that the electionused a strongly software-independent voting system, didit adhere to procedures that should keep the audit trailsufficiently accurate to reconstruct the outcome accord-ing to how voters cast their ballots? Strong evidence thatsuch procedures were followed is strong evidence thatthe legally correct outcome—what a full hand count ofthe audit trail would show—is the same as the outcomeaccording to how the voters originally cast their ballots.As we discuss below in section 4, we believe that com-pliance audits should always be required: If the electionfails the compliance audit, there is no assurance thateven a full hand count of the audit trail would show theoutcome according to how the voters really voted. Be-low, we assume that the election has passed a complianceaudit. False alarms are possible. An analogy is that if a tamper-evidentseal shows that a package has been opened, it does not follow that thepackage contents have been altered. “Failure” means failure to find strong evidence that such proce-dures were followed, rather than finding evidence that such procedureswere not followed. .2 Vote tabulation audits Vote tabulation audits compare reported vote subtotalsfor subsets of ballots (“audit units”) with hand counts ofthe votes for each of those subsets. Audit units have to besubsets for which the voting system reports vote subto-tals. Most present U.S. audits use audit units that consistof all the ballots cast in individual precincts or all the bal-lots tabulated on individual voting machines. Generally,audit laws do not have provisions that would lead to cor-recting incorrect electoral outcomes [Hall et al., 2009]. A risk-limiting post-election audit uses the au-dit trail to guarantee that there is a large, pre-specified probability that the audit will correct theapparent outcome if the apparent outcome is wrong.Risk-limiting audits are widely considered best prac-tice [Lindeman et al., 2008]. Risk-limiting audits havebeen endorsed by the American Statistical Associa-tion [American Statistical Association, 2010], the Bren-nan Center for Justice, Common Cause, the League ofWomen Voters, and Verified Voting, among others. Cal-ifornia AB 2023 (2010), requires a pilot of risk-limitingaudits in 2011 [Salda˜na, 2010]. Colorado RevisedStatutes § Our goal in this work is to sketch a personally verifiableprivacy-preserving P -resilient canvass framework. Wemust first say what this means. For instance, under New York law, each county determines inde-pendently whether its audit in a particular contest must be expanded.This provision means that a correct outcome might be changed to anincorrect outcome even if the conduct of the audit is formally flawless. A canvass framework consists of the vote-tabulationsystem together with other human, hardware, software,and procedural components of the canvass, includingcompliance and vote-tabulation audits. A canvass frame-work is resilient with probability P or P-resilient if theprobability that the outcome it gives is the correct out-come is at least P , even if its software has an error, short-coming, or undetected change. Resilience means thatthe framework tends to recover from faults. If a can-vass framework is P -resilient, either the outcome it giveswhen all is said and done is correct, or something oc-curred that had probability less than 1 − P . The canvassframework that results from performing a risk-limitingaudit on a strongly software-independent voting systemthat passes a compliance audit is P -resilient, with P equalto 100% minus the risk limit. If the system fails the com-pliance audit, the framework should not declare any out-come. Instead, the election should be re-run.Even if a canvass framework is P -resilient, in prac-tice the public might not trust the system unless theycan observe crucial steps, especially the audit. The mereright or opportunity to observe the audit will not engen-der much trust if—as a practical matter—no single per-son or small group could observe all the steps that areessential to ensuring the accuracy of the final result. Forinstance, if a vote-tabulation audit takes ten teams of au-ditors working in separate offices four days to complete,it would take a large team of independent observers—with lots of free time and long attention spans—to verifythat the audit was carried out correctly. The longer anaudit takes and the more people required to carry out theaudit, the more opportunities there are to damage the au-dit trail, and the harder it is for an observer to be satisfiedthat the audit has been conducted correctly.We define a canvass framework to be personally verifi-able P-resilient if it is P -resilient and a single individualcould, as a practical matter, observe enough of the pro-cess to have convincing evidence that the canvass frame-work is in fact P -resilient.The transparency required for a canvass framework tobe personally verifiable can impact privacy. For instance,publishing images of all the ballots cast in an election might give the individuals compelling evidence that thevote tabulation system found the correct outcome, sincethe images allow people to count the votes themselves— As discussed in section 4, to be P -resilient, a canvass frameworkshould refrain from giving any outcome at all if some preconditions arenot met. The probability comes from the overall voting system, in our casefrom the fact that the audit relies on a random sample. The probabil-ity does not come from treating votes, voters, or election outcomes asrandom, for instance. There also needs to be proof that the images are sufficiently com-plete and accurate to determine the correct outcome.
3t least to the extent that voter intent is unambiguous. But publishing ballot images can facilitate vote-sellingand coercion and can compromise privacy, because vot-ers can deliberately or accidentally reveal their identitiesthrough marks on the ballots including idiosyncrasies ofhow individuals fill in bubbles [Calandrino et al., 2011]or even the fiber structure of the paper on which the bal-lot is printed [Calandrino et al., 2009]. A lesser but substantial degree of transparency isconferred by publishing cast vote records (CVRs) enabling anyone to verify that the contest outcomesare correct—if the CVRs are accurate. However,as Popoveniuc and Stanton [2007] and Rescorla [2009]point out, publishing CVRs also can aid vote-selling orcoercion because of the potential for pattern voting. Onetypical sample ballot (from Tulsa, Oklahoma) contains18 contests with over 589,000 possible combinations ifa voter votes in every contest, or over 688 million com-binations allowing for undervotes. Thus, a voter couldbe instructed to vote for the preferred candidate in onecontest, and to cast a series of other votes that would al-most certainly (especially within a precinct), confirm thevoter’s identity if all of the voter’s selections were pub-lished. Hence, publishing whole-ballot CVRs for largenumbers of ballots improves transparency but can sacri-fice privacy.When there is not strong evidence that the apparentoutcome is correct, risk-limiting audits can require ex-amining the entire audit trail, potentially exposing all theballots to public scrutiny. If the apparent outcome iswrong, such exposure is necessary in order to correctthe outcome. Therefore, if a risk-limiting audit is tobe personally verifiable, there may be occasions wherecompromising privacy is unavoidable. But minimizingthe number of ballots or whole-ballot CVRs that areroutinely exposed helps protect privacy, impeding vote- Verification methods like Humboldt County Election TransparencyProject (see below) involve publishing digital images of all the ballots. There are arguments that images of ballots should be publishedanyway—that transparency is more important than privacy. In juris-dictions that permit voting by mail, there is an opportunity to confirmhow someone votes for the purpose of vote-selling or coercion; indeed,someone could fill out another’s ballot. Whether publishing images ofballots would change the rate of vote-selling or coercion substantiallyis the subject of some debate. In the 2002 FEC Voting System Stan-dards [Federal Election Commission, 2002], these werecalled “ballot images”; however, the term CVR has beenused in more recent EAC Voluntary Voting System Guide-lines [Election Assistance Commission, 2005]; we prefer the latterterm because it does not suggest an actual image but rather a record ofthe interpretation of the system’s interpretation of the ballot. And whatmatters is the system’s interpretation of the ballot as a set of votes. One could have a risk-limiting audit that, if it had not terminatedafter some fraction of the ballots had been examined, triggered a handcount of the remaining ballots, but did not allow the public to observethat hand count. But then why should the public trust that the handcount was accurate? selling and coercion.We define a canvass framework to be personally ver-ifiable privacy-preserving P-resilient if it is personallyverifiable P -resilient and it does not sacrifice privacy un-necessarily. Neither personally verifiable nor privacy-preserving is a mathematically precise characteristic,while P -resilience is.The contribution of the present work is to sketch apersonally verifiable privacy-preserving P -resilient vot-ing system. We assume, as a foundation for buildingthis system, that we are starting with a strongly software-independent voting system with an audit trail that corre-sponds to individual ballots. Moreover, we assume that acompliance audit has determined that the audit trail gen-erated by the system is sufficiently trustworthy to reflectthe correct outcomes of the contests. We augment thesystem with procedures and data structures that make itpossible for an individual observer to gain compellingevidence that either the outcomes are correct, or some-thing very unlikely occurred—that is, that the overallcanvass framework is P -resilient. Unless some of theapparent outcomes are wrong or a margin is extremelysmall, gathering that evidence will generally involve ex-posing only a tiny percentage of ballots and whole-ballotCVRs.In essence, our method adds a special risk-limitingaudit to a strongly software-independent voting system(one that has had a compliance audit to ensure that itsaudit trail is intact). Since one person cannot be in twoplaces at the same time, the procedure cannot be person-ally verifiable if it involves auditing a multi-jurisdictionalcontest in different jurisdictions simultaneously; it wouldthen be necessary to trust confederates to observe what ishappening elsewhere. The next few sections outline ele-ments of this risk-limiting audit. One key to keeping the process personally verifiable (bykeeping amount of observation required low) and to pro-tecting privacy (by exposing as few ballots as possibleto observers) is to audit the record at the level of indi-vidual ballots, rather than large batches of ballots suchas precincts. The fewer ballots there are in each auditunit, the smaller the expected counting burden for risk-limiting audits tends to be—when the electoral outcomeis correct (see, e.g., [Stark, 2009a, 2010a,b]). A vote-tabulation audit based on checking the CVRs of individ-ual ballots against a human interpretation of those ballotsis often called a “ballot-level audit,” a “single-ballot au-dit,” or a “ballot-based audit.” Because they reduce thetime it takes to audit and the number of ballots involved,ballot-level risk-limiting audits are especially amenableto personal verification.4allot-level audits are extremely efficient statistically,but they are not simple to implement using current vot-ing systems. To perform a ballot-level audit, there mustbe a way to identify each ballot uniquely, for instance,a serial number on a paper ballot, or identifying the bal-lot by its location: “the 17th ballot in deck 152 scannedby scanner C,” for instance. There must also be a wayto match each ballot to its CVR. Some commercial vot-ing systems do not generate or do not store CVRs forindividual ballots. Other voting systems record individ-ual CVRs, but are designed make it difficult or impos-sible to match individual CVRs to the ballots they pur-port to represent. In some cases, audit trails have identi-fiers that can be used to find the corresponding CVRs;this method was used for part of a 2008 audit in Ea-gle County, Colorado [Branscomb, 2008] and a ballot-level risk-limiting audit in Orange County, California, in2011 [P.B. Stark, personal communication, 2011]. How-ever, to protect privacy, most paper ballots do not haveidentification numbers. In a 2009 pilot ballot-level auditin Yolo County, California, Stark [2009c] exploited thefact that the CVRs and the physical ballots were in thesame order. The scanned images associated with eachCVR in the audit sample were compared with the physi-cal ballots to check the accuracy of the CVRs.Calandrino et al. [2007] describe an approach to elec-tion verification that involves imprinting ballots withidentification numbers and scanning the ballots with a“parallel” system in addition to the system of record.The parallel system derives its own CVRs, from whichthe apparent contest outcome can be determined inde-pendently. The accuracy of the unofficial CVRs and ofthe imprinting process is then assessed by a ballot-levelaudit.Since 2008, the Humboldt County Election Trans-parency Project (Humboldt County ETP) has experi-mented with publishing ballot images and independentlytabulating CVRs extracted from those images. Usingcommercially available equipment, Humboldt CountyETP rescans paper ballots after embossing them withserial numbers. Then, open-source software is used toform CVRs from the digital images. Humboldt CountyETP has processed ballots for six elections and publishedscanned ballot images as well as its version of the CVRsfor some of them. The results based on their re-scans If an identifier is printed on paper ballots, the printing should occurafter the voter casts his or her vote and the ballots are co-mingled. Ifthe identifier is printed before the voter casts his or her vote, privacycould be compromised. Optical-scan ballots as well as DRE paper audit trails can haveidentifiers. For instance, in Boulder County, Colorado, the Hart BallotNow system is configured to print unique identifiers and bar codes oneach ballot. In Orange County, California, ballots for the Hart BallotNow system have non-unique identifiers and bar codes (numbered 1–2500, then repeating). generally have agreed well with the original results, withone important exception: The Humboldt County ETPanalysis of the November 2008 election uncovered a de-fect in the election management software that led the re-sults of an entire ballot batch to be silently discarded!The Clear Ballot Group, inspired in part by HumboldtCounty ETP, is developing a system that, in its words,could permit election outcomes to be “thoroughly andtransparently verified within 36–48 hours after the pollsclose.” Neither the Humboldt County ETP nor ClearBallot Group currently incorporate risk-limiting audits, but the parallel scans their systems perform facilitateballot-level risk-limiting audits, along the general linesproposed by Calandrino et al. [2007]. If the system ofrecord and the parallel system agree on the set of win-ners, a risk-limiting audit of the parallel system transi-tively confirms the outcome according to the system ofrecord. The method we propose here presupposes that CVRs areavailable, either from the system of record or from a par-allel system. It publishes all the data contained in theCVRs in a form that (1) still permits all observers tocheck the contest outcomes on the assumption that theCVRs are accurate, (2) does not compromise privacy, and(3) enables the CVRs to be checked against the audit trailwhile minimizing the loss of privacy.In SOBA, election officials make a cryptographiccommitment to the full set of CVRs by publishing theCVRs separately for each contest, disaggregating the bal-lots (we call these contest-CVRs or CCVRs in contrast towhole-ballot CVRs), and a shrouded link between eachCCVR and the ballot it purports to represent. Splittingthe CVRs into CCVRs and obfuscating the identity of theballot from which each CCVR comes eliminates some ofthe information required to identify a voter’s ballot styleor to use pattern voting to signal the voter’s identity. This makes the procedure privacy-preserving. But it re-tains enough information for any observer to check that Clear Ballot Group is adding support for risk-limiting audits totheir software [L. Moore, personal communication, 2011]. This is true as long as the systems agree on the set of winners,even if they disagree about vote totals or margins. For instance, supposecandidate A defeats candidate B by one percentage point in the originalreturns, and by ten points according to the parallel system. Such a largediscrepancy might justify close scrutiny, but a risk-limiting audit of theresults of the parallel system would still provide strong evidence that Adefeated B, or would lead to a full hand count to set the record straight. See http://en.wikipedia.org/wiki/Commitment_scheme .Cryptographic commitments have two important properties, the bind-ing property and the hiding property, discussed in section 3.2. Of course, if there is a contest in which few voters are eligible tovote, eligibility itself is a signal. ) that observerscan apply to the published CCVRs to calculate the cor-rect outcome of every contest—provided the CCVRs re-flect the ballots (more generally, audit trail) accuratelyenough. This is part of making the procedure personallyverifiable. Loosely speaking, the required level of accu-racy depends on the number of CVRs that must have er-rors for the apparent outcome to be wrong: The fewerballots that need to be changed to affect the outcome,the larger the sample generally will need to be to attaina given level of confidence that the apparent outcome iscorrect.The CCVRs might fail to be sufficiently accurate be-cause • At least one CCVR and the ballot it purports to rep-resent do not match because human and machine in-terpretations of voter intent differ (for instance, be-cause the voter marked the ballot improperly). Thisis a failure of the generation of CCVRs. • At least one CCVR does not in fact correspond toany ballot. It is an “orphan.” This is a failure of themapping between ballots and CCVRs. • More than one CCVR for the same contest ismapped to the same ballot. It is a “multiple.” Thisis also a failure of the mapping between ballots andCCVRs. • There is no CCVR corresponding to some votingopportunity on a ballot.A failure of the mapping might be the more distressingsource of error, since it is a failure on the part of theelection official, but we must ensure (statistically) that—together—all sources of error did not combine to causethe outcome to be wrong. SOBA uses a risk-limiting au-dit to assess statistically whether the winners accordingto the full audit trail differs from the winners accordingto the CCVRs, for all contests under audit, taking into ac-count all sources of error. If the outcome according to theCCVRs is incorrect, the audit is very likely to proceed toa full hand count of the audit trail, thereby revealing thecorrect outcome. This provides P -resilience.To make the risk-limiting audit possible, elections of-ficials are required to publish another file, the ballot style For first-past-the-post contests, the winner algorithm just findswho has the most votes. Other voting schemes, such as instant-runoffvoting (IRV) or ranked choice voting (RCV), have more complicatedwinner algorithms. In plurality voting, this is the margin or the set of margins betweeneach (winner, loser) pair. Defining the margins for IRV and calculatingthem for a given set of reported results is not simple. See Cary [2011];Magrino et al. [2011]. file , which contains ballot identifiers and lists the con-tests each of those ballots contains. It does not containthe voters’ selections.The risk-limiting technique we propose is the super-simple simultaneous single-ballot risk-limiting au-dit [Stark, 2010b]. It is not the most efficient ballot-level audit, but the calculations it requires can be doneby hand, increasing transparency. It involves drawingballots at random with equal probability; some more ef-ficient audits require using different probabilities for dif-ferent ballots, which is harder to implement and to ex-plain to the public. Moreover, this technique allows acollection of contests to be audited simultaneously usingthe same sample of ballots. That can reduce the numberof randomly selected ballots that must be located, inter-preted, and compared with CVRs, decreasing the costand time required for the audit and thereby increasingtransparency.The following subsections give more technical detail.
We assume that the audit trail consists of one record perballot cast. There are C contests we wish to assess. Thecontests might be simple measures, measures requiringa super-majority, multi-candidate contests, or contests ofthe form “vote for up to W candidates.” We refer torecords in the audit trail as “ballots.” A ballot may be anactual voter-marked paper ballot, a voter-verifiable paperaudit trail (VVPAT), or a suitable electronic record.There are N ballots in the audit trail that each con-tain one or more of the C contests. Each ballot can bethought of as a list of pairs, one pair for each contest onthat ballot. Each pair identifies a contest and the voter’sselection(s) in that contest, which might be an undervoteor a vote for one or more candidates or positions. Ex-amining a ballot by hand reveals all the voter’s selectionson that ballot; we assume that there is no ambiguity ininterpreting each voter’s intentions from the audit trail.Before the audit starts, the voting system must reportresults for each of the C contests. The report for contest c gives N c , the total number of ballots cast in contest c (including undervotes and spoiled ballots), as well as thenumber of valid votes for each position or candidate incontest c . Let M ≡ N + N + · · · + N C denote the totalnumber of voting opportunities on the N ballots. We as-sume that the compliance audit assures us (e.g., throughballot accounting) that the reported values of N c are accu-rate, and that the audit trail is trustworthy. In the presentwork, we do not consider attacks on the audit trail. We do not specifically consider instant-runoff voting or ranked-choice voting here. Risk-limiting methods can be extended to suchvoting methods, but the details are complex. N lines in the file, and the N ballot identifiers shouldbe unique. Because the ballot style file is published, indi-vidual can check this for themselves. Moreover, individ-uals can check whether the number of lines in the ballotstyle file that list contest c equals N c , the total number ofballots the system reports were cast in contest c .Before the audit starts, the voting system or a paral-lel system has produced a CVR for each ballot. Theseare not published as whole-ballot CVRs. Rather, theCVRs are split by contest to make contest-specific CVRs(CCVRs) that contain voters’ selections in only one con-test. Each whole-ballot CVR is (supposed to be) splitinto as many CCVRs as there are contests on the ballot.The CCVRs for the contests are published in C files,one for each contest. The CCVR file for contest c shouldcontain N c lines; because this file is published, individu-als can check this for themselves. Each line in the CCVRfile for contest c lists a voter’s selection and a shroudedversion of the identifier of the ballot that the selectionis supposed to represent. The order of the lines in eachof the C CCVR files should by shuffled (preferably us-ing random permutations) so that whole CVRs cannot bereassembled without knowing secret information. The public can confirm whether the contest outcomesaccording to the CCVR files match the voting system’sreported outcomes. If they do not match, there shouldbe a full hand count of any contests with discrepant out-comes. We assume henceforth that the outcomes domatch, but we do not assume the exact vote totals ac-cording to the CCVR files match the reported vote totals.The data include one more file that is not published,the lookup file . The lookup file contains M lines, onefor each voting opportunity on each ballot. Each line hasthree entries: a shrouded ballot identifier, the correspond-ing unshrouded ballot identifier, and a number (“salt”)that is used in computing the shrouded identifier fromthe unshrouded identifier using a cryptographic commit-ment function, as described below. (For a review of usesfor cryptography in voting, see Adida [2006].)The salt on the j th line of the file is denoted u j . Eachline corresponds to a (ballot, contest) pair: We can thinkof u j as being u ic , the salt used to shroud the identity ofballot b i in the CCVR file for contest c . The electionofficial will use this file to convince observers that everyselection on every ballot corresponds to exactly one entry For example, each CCVR file could be sorted in order of theshrouded ballot identifier. in a CCVR file, and vice versa.
The method of shrouding ballot identifiers is crucial tothe approach. SOBA requires election officials to cryp-tographically commit to the value of the ballot identifierthat goes with each CCVR. A cryptographic commitmentensures that the ballot identifier is secret but indelible:The election official can, in effect, prove to observers thata shrouded identifier corresponds to a unique unshroudedidentifier, but nobody can figure out which unshroudedidentifier corresponds to a given shrouded identifier with-out secret information.The next few paragraphs describe a suggested instanti-ation of the cryptographic commitment. We assume thatballot identifiers all have the same length. If necessary,this can be achieved by padding identifiers with leadingzeros. The commitment function H () must be disclosedpublicly and fixed for the duration of the election.Each commitment represents a claim about a voter’sselection(s) on a given ballot in a given contest. Foreach set of selections that any voter made in each con-test, including undervotes and votes for more than onecandidate, the election official will create a set of com-mitments. Each commitment designates the ballot iden-tifier of a ballot that the election official claims containsthat set of selections in that contest. To commit to theballot identifier b , the election official selects a secret“salt” value u and computes the commitment value y = H ( b , u ) . At a later stage, the official can open thecommitment by revealing u and b : Then anyone can ver-ify that the value y revealed earlier is indeed equal to H ( b , u ) .Loosely speaking, a commitment function must havetwo properties, the binding property and the hidingproperty . The binding property makes it infeasible forthe official to find any pair ( b ′ , u ′ ) = ( b , u ) for which H ( b ′ , u ′ ) = H ( b , u ) . This provides integrity by helping toensure that election officials cannot contrive to have morethan one CCVR for a given contest claim to come fromthe same ballot. The binding property is crucial for P -resilience; indeed, the proof of P -resilience requires onlythat the commitment have the binding property and that { N c } Cc = are known.The hiding property makes it infeasible for anyonewith access only to the shrouded values H ( b , u ) to learnanything about which ballot is involved in each commit-ment. This provides privacy by helping to ensure that To protect voter privacy, it must be infeasible to guess the salts:Each salt should contain many random or pseudo-random bits. For thecommitment to be effective, the length of all salt values should be fixedand equal. See section 4. See step 7 of the proof in section 3.4. b may appear in multiplecommitments since a separate commitment is generatedfor each candidate selection on each ballot. The hidingproperty ensures that those collections of commitmentsdo not together reveal the value of any b . This is crucialfor the method to be privacy-preserving.An HMAC (as described in Federal Information Pro-cessing Standard Publication 198) with a secure hashfunction such as SHA-256 (described in Federal Infor-mation Processing Standard Publication 180-2) can beused to instantiate the commitment function. However,since each of the parameters of the commitment functionis of fixed length it is more efficient to simply use a cryp-tographic hash function such as SHA-256 directly. Thelength of the ballot identifiers does not matter, as long asall ballot identifiers in the election have the same length.We recommend that all salt values have equal length, ofat least 128 bits. Our results do not depend on the partic-ular commitment function chosen, as long as it has boththe binding and hiding properties. We now describe how to perform a risk-limiting auditthat simultaneously checks the accuracy of the CCVRs,whether each CCVR entry comes from exactly one bal-lot, and whether every voting opportunity on every ballotis reflected in the correct CCVR file.
The first three steps check the consistency of the CCVRswith the reported results and the uniqueness of theshrouded identifiers.1. Verify that, for each contest c , there are N c entriesin the CCVR file for contest c .2. Verify that, for each contest c , the CCVR file showsthe same outcome as the reported outcome.3. Verify that the M = N + · · · + N C shrouded ballotidentifiers in all C CCVR files are unique.If step 2 shows a different outcome for one or more con-tests, those contests (at least) should be completely handcounted.Steps 4 and 5 check the logical consistency of the bal-lot style file with the reported results.4. Verify that, for each contest c , there are N c entriesin the ballot style file that list the contest. Menezes et al. [1996] offers a thorough treatment of hash func-tions and their use for commitments in applications such as digital sig-natures.
5. Verify that the ballot identifiers in the ballot stylefile are unique.If steps 1, 3, 4, or 5 fail, there has been an error or mis-representation. The election official needs to correct allsuch problems before the audit can start.The remaining steps comprise the statistical portion ofthe risk-limiting audit, which checks whether the CCVRsand the mapping from ballots to CCVRs is accurateenough to determine the correct winner.6. Set the audit parameters:(a) Choose the risk limit a .(b) Choose the maximum number of samples D to draw; if there is not strong evidence that theoutcomes are correct after D draws, the entireaudit trail will be counted by hand.(c) Choose the “error bound inflator” g > l ∈ ( , ) for the super-simple simultaneous method [Stark, 2010b]( g = .
01 and l = . r = − log a g + l log ( − g ) . (1)(e) For each of the C contests, calculate the mar-gin of victory m c in votes from the CCVRs forcontest c . (f) Calculate the diluted margin m : the smallestvalue of m c / N among the C contests. (g) Calculate the initial sample size n = ⌈ r / m ⌉ .(h) Select a seed s for a pseudo-random numbergenerator (PRNG). Observers and electionofficials could contribute input values to s or s could be generated by an observable, mechan-ical source of randomness such as rolls of a10-sided die. The seed should be selected onlyonce.7. Draw the initial sample by finding n pseudo-random numbers between 1 and N and audit the cor-responding ballots: This would be replaced by a different calculation for IRV or RCVcontests. See, e.g., Magrino et al. [2011]; Cary [2011]. The diluted margin controls the sample size. If contest c has thesmallest value of m c / N and N c is rather smaller than N , it can be moreefficient to audit contest c separately rather than auditing all C contestssimultaneously. The code for the PRNG algorithm should be published so that itcan be checked and so that, given the seed s , observers can reproducethe sequence of pseudo-random numbers. The PRNG should producenumbers that are statistically indistinguishable from independent ran-dom numbers uniformly distributed between 0 and 1 (i.e., have large p -values) for sample sizes up to millions for a reasonable battery oftests of randomness, such as the Diehard tests. s to generate n pseudo-random numbers, r , r , . . . , r n .(b) Let ℓ j ≡ ⌈ Nr j ⌉ , j = , . . . , n . This list mightcontain repeated values. If so, the tests be-low only need to be performed once for eachvalue, but the results count as many times asthe value occurs in the list. (c) Find rows ℓ , . . . , ℓ n in the ballot style file.(d) Retrieve the ballots b ℓ j in the audit trail iden-tified by those rows in the ballot style file. Ifthere is no ballot with identifier b ℓ j , pretend instep 7(g) below that the ballot showed a votefor the runner-up in every contest listed in thatrow of the ballot style file.(e) Determine whether each ballot shows thesame contests as its corresponding entry in theballot style file. If there are any contests on theballot that are not in the ballot style file entry,pretend in step 7(g) below that the CCVR forthat (ballot, contest) pair showed a vote for theapparent winner of the contest. If there are anycontests in the ballot style file entry that arenot on the ballot, pretend in step 7(g) belowthat the ballot showed a vote for the apparentrunner-up for that contest.(f) For each ballot b ℓ j in the sample, the electionofficial reveals the value of u ℓ j c for each con-test c on the ballot.(g) For each ballot in the sample, for each conteston that ballot, observers calculate H ( b ℓ j , u ℓ j c ) and find the entry in the CCVR file for con-test c that has that shrouded identifier. If theshrouded identifier is not in the CCVR file,pretend that the CCVR file showed that thevoter had selected the apparent winner of con-test c . Compare the voter’s selection(s) ac-cording to the CCVR file to the voter’s selec-tion(s) according to a human reading of ballot b ℓ j . Find e ℓ j , the largest number of votes bywhich any CCVR for ballot b ℓ j overstated themargin between any (winner, loser) pair in anycontest on ballot b ℓ j . This number will be be-tween − + e ℓ j = lm n have e ℓ j =
1, the audit stops. (In thiscalculation, the value of e ℓ j should be counted asmany times as ℓ j occurs in the sample.)9. Otherwise, calculate the Kaplan-Markov P -value, P KM according to equation (9) in Stark [2009d,c, The auditing method relies on sampling with replacement to limitthe risk. If P KM is less than a , the audit stops. If P KM is greater than a , the sample is expanded: An-other random number r j is generated and steps 7(c)–(g) are repeated. The value of P KM is updated toinclude the overstatement errors found in the newdraw. This continues until either P KM ≤ a or therehave been D draws. In the latter case, all remainingballots are counted by hand, revealing the true out-come.The next section establishes that this procedure in factgives a risk-limiting audit. If the ballot style file is correct and entries in the CCVRfiles are mapped properly to voting opportunities on ac-tual ballots, the only potential source of error is thatCCVR entries do not accurately reflect the voters’ selec-tions according to a human reading of the ballot. If thatis the case, this is an “ordinary” risk-limiting audit, andthe proof in Stark [2010b] that the super-simple simulta-neous method is risk-limiting applies directly.Suppose therefore that the ballot style file or the map-ping between ballots and CCVRs is faulty. Recall thatthe super-simple simultaneous method assumes that noballot can overstate any margin by more than 2 g votes,where g >
1. There are seven cases to consider.1. The ballot style file has more than one entry thatcorresponds to the same actual ballot, or more thanone actual ballot corresponds to the same entry inthe ballot style file. These faults are precluded bythe uniqueness of the ballot identifiers and of therecipes for locating the actual ballot with each iden-tifier.2. More than one ballot identifier corresponds to thesame shrouded entry (for different values of u ). Thisis precluded by the binding property of H . We consider only plurality voting here: IRV is more complicated.For each contest c , let W c be the indices of the apparent winners of thecontest and let L c be the indices of the apparent losers of the contest.If w ∈ W c and x ∈ L c , let V wx be the margin in votes between candidate w and candidate x according to the CCVR file for contest c . For eachcandidate k on ballot ℓ , let v ℓ k denote the number of votes for candidate k on ballot ℓ according to the CCVR file and let a ℓ k denote the numberof votes on ballot ℓ for candidate k according to a human reading ofballot ℓ . Let e ℓ ≡ max c max w ∈ W c , x ∈ L c ( v ℓ w − a ℓ w − v ℓ x + a ℓ x ) / V wx . (2)Then P KM ≡ n (cid:213) j = − / U − e ℓ j g / V . (3) Overstatements are calculated as step 7 above, including, in partic-ular, steps 7(e) and 7(g), which say how to treat failures to find ballotsor contests.
9. The ballot style file contains identifiers that do notcorrespond to actual ballots, or claims that a ballotcontains a contest that it does not actually contain.The biggest effect this could have on an apparentcontest outcome is if the ballot that entry is sup-posed to match showed a vote for the runner-up inevery missing contest, which is no greater than atwo-vote change to any margin. Because the au-dit samples entries of the ballot style file with equalprobability, this kind of error in an entry is just aslikely to be revealed as any other. If such a ballotstyle file entry is selected for audit, steps 7(d) and7(e) treat it this worst-case way.4. The ballot style file claims that a ballot does notcontain a contest that it does contain. The biggesteffect this could have on an apparent contest out-come is if the CCVR for that contest showed a votefor the apparent winner, which cannot change themargin by more than two votes, so the error-boundassumptions are satisfied. Because the audit sam-ples entries of the ballot style file with equal proba-bility, this kind of error in an entry is just as likelyto be revealed as any other. If such a ballot stylefile entry is selected for audit, step 7(e) treats it thisworst-case way.5. There are ballots whose identifiers do not appear inthe ballot style file. Since there are the same numberof ballots as entries in the ballot style file and theballot identifiers in the ballot style file are unique,there must be ballot identifiers in the ballot style filethat do not match any ballot. Hence, case (3) holds.6. There are CCVRs for which the shrouded ballotidentifier is not the identifier of any ballot. If theshrouded identifier matches an identifier in the bal-lot style file, we are in case (3). Suppose thereforethat the shrouded identifier does not match any inthe ballot style file. Suppose this happens for con-test c . The preliminary checks show that the bal-lot style file has exactly N c entries for contest c andthat there are exactly N c entries in the CCVR filefor contest c . Therefore, if there is such a CCVR,one of the ballot style file entries that lists contest c has an identifier that does not occur in shroudedform in the CCVR file for that contest. The largesteffect this could have on contest c is if the “substi-tuted” CCVR entry reported a vote for the apparentwinner; this cannot overstate the margin by morethan two votes, so the audit’s error-bound assump-tion still holds. Because the audit samples entries ofthe ballot style file with equal probability, this kindof error in a ballot style file entry is just as likelyto be revealed as any other. If such a ballot style file entry is selected for audit, step 7(e) treats it thisworst-case way.7. The same ballot identifier appears in shrouded formmore than once in a single CCVR file. As in theprevious case, we know there are N c entries in theCCVR file for contest c and N c entries in the bal-lot style file that include contest c ; moreover, theidentifiers in the ballot style file are unique. Hence,there must be at least one entry in the ballot stylefile that lists contest c for which the ballot identifierdoes not appear in shrouded form in the CCVR file.We are therefore in case (6). Others have proposed election verification methodsthat involve a cryptographic commitment by elec-tions officials to a mapping between ballots andCVRs [E.K. Rescorla, personal communication, 2011;R.L. Rivest, personal communication, 2009; D. Wallach,personal communication, 2010; see also Adida [2006]].However, we believe SOBA is the first method that re-quires only one commitment and that uses a risk-limitingaudit to check whether the mapping is accurate enoughto determine the correct winner.We have said little about the requirement for a com-pliance audit. In part, this is a definitional issue: Even ifthe audit trail is known to have been compromised, it isour understanding that in many states, a full hand countof the audit trail would still be the “correct” outcome, asa matter of law. Hence, an audit to assess whether theaudit trail was protected and preserved adequately for itto reflect the outcome according to how the voters casttheir ballots is legally superfluous. We consider this ashortcoming of current audit and recount laws. More-over, we doubt that any system can be P -resilient unlessthe election and the data it generates satisfies particularconditions. For instance, risk-limiting audits generallyassume that the number of ballots cast in all in each con-test is known. Such conditions should be checked.We would advocate carrying out a compliance au-dit to assess whether the procedures as followed in theelection give reasonable assurance that the audit trail istrustworthy—sufficiently accurate to reflect the outcomeaccording to how voters cast their ballots—and to assesswhether any other preconditions of the risk-limiting au-dit hold. The compliance audit should evaluate whetherthere is strong evidence that the chain of custody of theballots is intact, or whether it is plausible that ballotswere lost, “found,” altered, or substituted. The compli-ance audit should confirm the values of { N c } by bal-lot accounting: confirming that the number of ballotsprinted equals the number returned voted, unvoted, and10poiled, for each ballot type.If the election passes the compliance audit, a risk-limiting audit can then assess the accuracy of the reportedresult and would have a large chance of correcting the ap-parent outcome if it is wrong (by examining the full audittrail). But if the election fails the compliance audit—thatis, if we lack strong evidence that the audit trail is reli-able and that the preconditions for the risk-limiting auditare met—a P -resilient election framework should not de-clare any outcome at all.For the method to be P -resilient, H must be bindingand we must know { N c } . Because the election officialdiscloses H and the (fixed) length of the ballot identi-fiers, we can determine whether H is binding. For themethod to be privacy-preserving, H must have the hidingproperty, which will depend on how the salts are chosenand how the CCVR files are organized. If the salts canbe discovered, inferred, or guessed, or if observers haveanother way to reassemble whole-ballot CVRs from theCCVRs (for instance, if the CCVRs are in the same bal-lot order across contests), voter privacy can be compro-mised. SOBA makes possible a personally verifiable privacy-preserving P -resilient canvass framework. It allows indi-viduals to obtain strong firsthand evidence that appar-ent election outcomes either are correct in the first place,or are corrected by a risk-limiting audit before becomingfinal, without unnecessary compromises to privacy. Af-ter the procedure is complete, either all the outcomes arecorrect or an event with probability less than 1 − P hasoccurred. The published data structures allow the pub-lic to check the consistency of the apparent outcomes butdo not allow whole-ballot cast vote records to be recon-structed, thereby preserving privacy. When all the appar-ent contest outcomes are correct, gathering the evidencethat the outcomes are right typically will require expos-ing only a small fraction of ballots to observers, protect-ing privacy. But the data structures and auditing protocolensure that if the apparent outcome of one or more of thecontests is wrong, there is a large chance of a full handcount of the audit trail to set the record straight. This work was supported in part by NSF Grant CNS-05243 (ACCURATE). We are grateful to Poorvi Vora For multi-jurisdictional contests, it might not be possible to con-duct an audit in a single place and time. If the audit step takes placein pieces in separate jurisdictions simultaneously, firsthand knowledgemight be impossible; one might need to trust observers in other loca-tions. for shepherding the paper and to anonymous referees forhelpful comments. We are grateful to Joseph LorenzoHall, David Jefferson, Neal McBurnett, Dan Reardon,Ronald L. Rivest, and Emily Shen for helpful conversa-tions and comments on earlier drafts.
References
Adida, B. (2006).
Advances in Cryptographic VotingSystems . PhD thesis, Massachusetts Institute of Tech-nology.American Statistical Association (2010).American Statistical Association state-ment on risk-limiting post-election audits. .Ash, A. and Lamperti, J. (2008). Florida 2006: CanStatistics tell us who won Congressional District 13?
Chance , 21(2):18–27.Bonner, L. (2004). New statewide election possible:Board may reconsider ag commissioner race.
TheNews & Observer (Raleigh, NC), republished at Vot-ersUnite.org . Retrieved February 25, 2011.Bowen, D. (2007). Withdrawal of approvalof Diebold Elections Systems, Inc., GEMS1.18.24/AccuVote-TSX/AccuVote-OS DRE & Opti-cal Scan voting System (October 25, 2007 revision). .Retrieved February 22, 2011.Branscomb, H. (2008). Audit report to sat-isfy Colorado Revised Statutes, Eagle County,Colorado Nov 4, 2008 General Election. ,Retrieved March 6, 2011.Calandrino, J., Clarkson, W., and Felten, E. (2009).Some consequences of paper fingerprinting for elec-tions. In
Proceedings of the 2009 Electronic Vot-ing Technology Workshop / Workshop on TrustworthyElections (EVT/WOTE ’09) . USENIX.Calandrino, J., Clarkson, W., and Fel-ten, E. (2011). Bubble trouble: Off-line de-anonymization of bubble forms. .Calandrino, J., Halderman, J., and Felten, E. (2007).Machine-assisted election auditing. In
Proceedingsof the 2007 USENIX/ACCURATE Electronic VotingTechnology Workshop (EVT 07) . USENIX.Cary, D. (2011). Estimating the margin of victory forinstant-runoff voting. In
Proceedings of the 2011 lectronic Voting Technology Workshop / Workshop onTrustworthy Elections (EVT/WOTE ’11) . USENIX.Checkoway, S., Sarwate, A., and Shacham, H. (2010).Single-ballot risk-limiting audits using convexoptimization. In Proceedings of the 2010 Elec-tronic Voting Technology Workshop / Workshop onTrustworthy Elections (EVT/WOTE ’10) . USENIX. .Retrieved April 20, 2011.Election Assistance Commission (2005).
Volun-tary Voting System Guidelines: Volume I –Voting System Performance Guidelines . Elec-tion Assistance Commission. Internet Archive: .Federal Election Commission (2002).
Vor-ing System Performance and Test Standards:Volume I – Performance Standards . Fed-eral Election Commission. Internet Archive: .Flaherty, S. (2006). In an age of computerized voting,is it possible to maintain voting integrity?
IowaCity Press-Citizen . Republished by VoteTrustUSA, .Retrieved February 25, 2011.Frisina, L., Herron, M., Honaker, J., and Lewis, J.(2008). Ballot formats, touchscreens, and undervotes:A study of the 2006 midterm elections in Florida.
Election Law Journal , 7(1):25–47.Garber, K. (2008). Lost votes in Floridas 2006general election: A look at extraordinary un-dervote rates on the ES&S iVotronic, part 2. .Retrieved February 22, 2011.Hall, J. L., Miratrix, L. W., Stark, P. B., Briones,M., Ginnold, E., Oakley, F., Peaden, M., Pellerin,G., Stanionis, T., and Webber, T. (2009). Im-plementing risk-limiting post-election audits in Cal-ifornia. In
Proc. 2009 Electronic Voting Technol-ogy Workshop/Workshop on Trustworthy Elections(EVT/WOTE ’09) , Montreal, Canada. USENIX.Keating, D. (2002). Democracy counts: The MediaConsortium Florida Ballot Project. In . American Political Science Association.
Retrieved March 1, 2011.Lindeman, M., Halvorson, M., Smith, P., Garland,L., Addona, V., and McCrea, D. (2008). Prin-ciples and best practices for post-election audits. .Retrieved April 20, 2011.Magrino, T., Rivest, R., Shen, E., and Wagner, D. (2011).Computing the margin of victory in IRV elections. In
Proceedings of the 2011 Electronic Voting Technol-ogy Workshop / Workshop on Trustworthy Elections(EVT/WOTE ’11) . USENIX.Mebane, W. (2009). Machine errors andundervotes in Florida 2006 revisited. .Retrieved February 22, 2011.Mebane, W. and Dill, D. (2007). Factors associ-ated with the excessive CD-13 undervote in the2006 General Election in Sarasota County, Florida. .Retrieved February 22, 2011.Menezes, A. J., Vanstone, S. A., and Oorschot, P. C. V.(1996).
Handbook of Applied Cryptography . CRCPress, Inc., Boca Raton, FL, USA, 1st edition.Miratrix, L. and Stark, P. (2009). The trinomial boundfor post-election audits.
IEEE Transactions on Infor-mation Forensics and Security , 4:974–981.Popoveniuc, S. and Stanton, J. (2007). Undervote andpattern voting: Vulnerability and a mitigation tech-nique. In
Pre-Proceedings of the 2007 IAVoSS Work-shop on Trustworthy Elections (WOTE ’09) .Rescorla, E. (2009). Understanding the securityproperties of ballot-based verification tech-niques. In
Proceedings of the 2010 Elec-tronic Voting Technology Workshop / Workshopon Trustworthy Elections (EVT/WOTE ’10) . Retrieved March 6, 2011.Rivest, R. (2008). On the notion of ‘software inde-pendence’ in voting systems.
Phil. Trans. R. Soc. A ,366(1881):3759–3767.Rivest, R. and Wack, J. (2006). On the no-tion of “software independence” in voting sys-tems (draft version of july 28, 2006). Tech-nical report, Information Technology Laboratory,National Institute of Standards and Technology. http://vote.nist.gov/SI-in-voting.pdf
Re-trieved April 20, 2011.Salda˜na, L. (2010). California Assembly Bill 2023.
Retrieved April 20, 2011.12tark, P. (2008a). Conservative statistical post-electionaudits.
Ann. Appl. Stat. , 2:550–581.Stark, P. (2008b). A sharper discrepancy measure forpost-election audits.
Ann. Appl. Stat. , 2:982–985.Stark, P. (2009a). Auditing a collection of races simulta-neously. Technical report, arXiv.org.Stark, P. (2009b). CAST: Canvass audits by samplingand testing.
IEEE Transactions on Information Foren-sics and Security, Special Issue on Electronic Voting ,4:708–717.Stark, P. (2009c). Efficient post-election auditsof multiple contests: 2009 California tests.http://ssrn.com/abstract=1443314. 2009 Confer-ence on Empirical Legal Studies.Stark, P. (2009d). Risk-limiting post-election audits: P -values from common probability inequalities. IEEETransactions on Information Forensics and Security ,4:1005–1014.Stark, P. (2010a). Risk-limiting vote-tabulation audits:The importance of cluster size.
Chance , 23(3):9–12.Stark, P. (2010b). Super-simple simultaneous single-ballot risk-limiting audits. In
Proceedings of the 2010Electronic Voting Technology Workshop / Workshopon Trustworthy Elections (EVT/WOTE ’10) . USENIX. .Retrieved April 20, 2011.Zetter, K. (2005). Diebold hack hintsat wider flaws.
Wired , December.