William C. Thompson

The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion

Introduction: The Requirements of Kumho Tire Co. v. Carmichael ........... 3 I. Observer Effects .................................................................... 6 A. Evolution of the Awareness of Observer Effects ....................... 6 B. What This Article Is Not About (Honesty and Observer Effects) ...................................................................................... 10 C. The Psychology of Observer Effects ......................................... 12

The Effects of Death Qualification on Jurors' Predisposition to Convict and on the Quality of Deliberation*

This study provides a straightforward test of the proposition that people who are permitted to serve on juries in capital cases (death-qualified jurors) are more likely to convict a defendant than are people who are excluded from serving on capital juries due to their unwillingness to impose the death penalty (excludable jurors). A sample of 288 subjects classified as death-qualified or excludable under theWitherspoon standard watched a 2 1/2-hour videotape of a simulated homicide trial including the judges instructions, and gave an initial verdict. Death-qualified subjects were significantly more likely than excludable subjects to vote guilty, both on the initial ballot and after an hours deliberation in 12-person juries. Nine juries were composed entirely of death-qualified subjects (death-qualified juries), while 10 contained from 2 to 4 excludable subjects (mixed juries). On postdeliberation measures, with initial death-penalty attitudes controlled, subjects who had served on the mixed juries were generally more critical of the witnesses, less satisfied with their juries, and better able to remember the evidence than subjects from the death-qualified juries, suggesting that diversity may improve the vigor, thoroughness, and accuracy of the jurys deliberations.

Sequential Unmasking: A Means of Minimizing Observer Effects in Forensic DNA Interpretation

Sir: Observer effects are rooted in the universal human tendency to interpret data in a manner consistent with one’s expectations (1). This tendency is particularly likely to distort the results of a scientific test when the underlying data are ambiguous and the scientist is exposed to domain-irrelevant information that engages emotions or desires (2). Despite impressions to the contrary, forensic DNA analysts often must resolve ambiguities, particularly when interpreting difficult evidence samples such as those that contain mixtures of DNA from two or more individuals, degraded or inhibited DNA, or limited quantities of DNA template. The full potential of forensic DNA testing can only be realized if observer effects are minimized. We met on December 1 and 2, 2007 in Washington, D.C. to discuss the implications of observer effects in forensic DNA testing and ways to minimize them. The interpretation of an evidentiary DNA profile should not be influenced by information about a suspect’s DNA profile (3–6). Each item of evidence must be interpreted independently of other items of evidence or reference samples. Yet forensic analysts are commonly aware of submitted reference profiles when interpreting DNA test results, creating the opportunity for a confirmatory bias, despite the best intentions of the analyst. Furthermore, analysts are sometimes exposed to information about the suspects, such as their history or motives, eyewitness identifications, presence or absence of a confession, and the like. Such information should have no bearing on how the results of a DNA test are interpreted, yet may compound an unintentional confirmatory bias. This bias can result in false inclusions under not uncommon conditions of ambiguity encountered in actual casework. It can also render currently used frequency statistics or likelihood ratios misleading. These problems can be minimized by preventing analysts from knowing the profile of submitted references (i.e., known samples) when interpreting testing results from evidentiary (i.e., unknown or questioned) samples. The necessary filtering or masking of submitted reference profiles can be accomplished in several ways, perhaps most easily by sequencing the laboratory workflow such that evidentiary samples are interpreted, and the interpretation is fully documented, before reference samples are compared. A simple protocol would dictate a separation of tasks between a qualified individual familiar with case information (a case manager) and an analyst from whom domain-irrelevant information is masked. Such a protocol would have the following steps. First, the analyst interprets the results of testing on the evidentiary samples. In this initial interpretation, the analyst would perform the following:

The death-qualified jury and the defense of insanity

We predicted that people who are excluded from serving on juries in capital cases due to their opposition to the death penalty (excludable subjects) tend to place a greater value on the preservation of due process guarantees than on efficient crime control, and therefore are more likely to accept an insanity defense in criminal cases than are people who are permitted to serve on capital juries (death-qualified subjects). Subjects who had previously been classified as death-qualified or excludable read four summaries of cases in which the defendant entered a plea of insanity, and made judgments of guilt or innocence. In the two cases involving nonorganic disorders (schizophrenia), death-qualified subjects were significantly more likely than excludable subjects to vote guilty; in the two cases involving organic disorders (mental retardation and psychomotor epilepsy), there were no differences between the two groups. In addition, excludable subjects gave significantly higher estimates than death-qualified subjects of the proportion of defendants pleading insanity who “really are” insane.

What Role Should Investigative Facts Play in the Evaluation of Scientific Evidence

Concern about contextual bias has led some authorities to recommend that forensic scientists know as little as possible about the facts of the underlying case when interpreting physical evidence; but concern about contextual ignorance has led other authorities to recommend, to the contrary, that forensic scientists know as much as possible in order to frame questions properly. This article recommends a case manager model that addresses both concerns. This article also responds to standard objections to the use of blind procedures in forensic science, explaining why contextual bias cannot be conquered through willpower; why use of domain-irrelevant contextual facts undermines the value of forensic evidence; how a well-known cognitive illusion (the ‘introspection illusion’) can mislead forensic scientists into thinking they can control their biases, when they cannot; and how a paradoxical feature of forensic inference (the ‘criminalists paradox’) can mislead analysts into thinking they should rely on contextual facts, when they should not.

Subjective interpretation, laboratory error and the value of forensic DNA evidence: three case studies

This article discusses two factors that may profoundly affect the value of DNA evidence for proving that two samples have a common source: uncertainty about the interpretation of test results and the possibility of laboratory error. Three case studies are presented to illustrate the importance of the analyst’s subjective judgments in interpreting some RFLP-based forensic DNA tests. In each case, the likelihood ratio describing the value of DNA evidence is shown to be dramatically reduced by uncertainty about the scoring of bands and the possibility of laboratory error. The article concludes that statistical estimates of the frequency of matching genotypes can be a misleading index of the value of DNA evidence, and that more adequate indices are needed. It also argues that forensic laboratories should comply with the National Research Council’s recommendation that forensic test results be scored in a blind or objective manner.

Time for DNA Disclosure

The legislation that established the U.S. National DNA Index System (NDIS) in 1994 explicitly anticipated that database records would be available for purposes of research and quality control “if personally identifiable information is removed” [42 U.S.C. Sec 14132(b)(3)(D)]. However, the Federal

Lay understanding of forensic statistics: Evaluation of random match probabilities, likelihood ratios, and verbal equivalents.

Forensic scientists have come under increasing pressure to quantify the strength of their evidence, but it is not clear which of several possible formats for presenting quantitative conclusions will be easiest for lay people, such as jurors, to understand. This experiment examined the way that people recruited from Amazons Mechanical Turk (n = 541) responded to 2 types of forensic evidence--a DNA comparison and a shoeprint comparison--when an expert explained the strength of this evidence 3 different ways: using random match probabilities (RMPs), likelihood ratios (LRs), or verbal equivalents of likelihood ratios (VEs). We found that verdicts were sensitive to the strength of DNA evidence regardless of how the expert explained it, but verdicts were sensitive to the strength of shoeprint evidence only when the expert used RMPs. The weight given to DNA evidence was consistent with the predictions of a Bayesian network model that incorporated the perceived risk of a false match from 3 causes (coincidence, a laboratory error, and a frame-up), but shoeprint evidence was undervalued relative to the same Bayesian model. Fallacious interpretations of the experts testimony (consistent with the source probability error and the defense attorneys fallacy) were common and were associated with the weight given to the evidence and verdicts. The findings indicate that perceptions of forensic science evidence are shaped by prior beliefs and expectations as well as expert testimony and consequently that the best way to characterize and explain forensic evidence may vary across forensic disciplines.

The role of prior probability in forensic assessments

As the importance of forensic science in the legal system has grown, debate has arisen about the way forensic scientists should characterize their findings in order to communicate most effectively with legal fact-finders. This article will focus on one aspect of that debate: the framing of conclusions involving elements of probability. In particular, we will examine the contentious issue of whether forensic scientists, when asked to provide evidence that will be used to evaluate various competing propositions about physical evidence, should consider the prior probabilities that those propositions are true. Disputes about this issue have arisen in a number of contexts and recent examples suggest that opinions still diverge (e.g., Budowle et al., 2011; Biedermann et al., 2012). In this comment, we will argue that a reasoned approach to this issue depends on the role that forensic scientists are expected to play in the legal system.

Comments on the review of low copy number testing

Dear Sir,A challenge to the reliability of low copy number (LCN)DNA profiling in the trial of Sean Hoey in Belfast CrownCourt in Northern Ireland (R v Hoey [2007] NICC 49, 20December, 2007) prompted the UK’s new Forensic ScienceRegulator (Andrew Rennison) to commission a review oflow template DNA profiling techniques. That review [2],conducted by Professor Brian Caddy (with the assistance ofDr. Adrian Linacre and Dr. Graham Taylor) was released on12 April, 2008 and concluded that LCN DNA profiling is“robust” and “fit for purpose.” Yet, the review accepts thatthe evidence presented in Sean Hoey’s trial was insufficientto establish the validity of the technique. It also enumerates21 recommendations for specific improvements that shouldbe undertaken to improve the methodology, including suchbasic steps as the development of a consensus on theinterpretation of test results and efforts to establish “bestpractices” for interpretation.We believe the conclusions of the review are inconsistentwith its recommendations in a number of respects. Forexample, it is difficult to see how a forensic techniquecould be deemed adequately validated for use in thecourtroom when there is not yet a consensus on how itsresults should be interpreted. The review thus raisesimportant issues about what it means for a forensic sciencetechnique to be validated. It also establishes grounds forconcern about the way that LCN DNA test results havebeen interpreted in earlier cases.We are concerned that the review team relied only oninput regarding the merits of LCN approaches fromorganizations that are dedicated to promoting its use bylaw enforcement. Consultation with known critics of thetechnique (or even a review of their published works)would have provided the reviewers with a broaderperspective of what work remains to be done before theapproach can become generally accepted within theinternational scientific community. There are in fact thingsabout LCN approaches upon which the reviewers andcritics do agree. For instance, caution that “[p]ublicizing thepotential of the application of LCN typing withoutdescribing its limitations may cause misunderstanding” [1]which is consistent with the review’s recommendations 1,3, and 13. But given the conclusion that “[t]he methodcannot be used for exculpatory purposes” [1], the review’sultimate conclusion that LCN testing is “fit for purpose”leaves the important but unanswered question of “what isthat purpose?”