D. Michael Risinger

The Daubert/Kumho Implications of Observer Effects in Forensic Science: Hidden Problems of Expectation and Suggestion

Introduction: The Requirements of Kumho Tire Co. v. Carmichael ........... 3 I. Observer Effects .................................................................... 6 A. Evolution of the Awareness of Observer Effects ....................... 6 B. What This Article Is Not About (Honesty and Observer Effects) ...................................................................................... 10 C. The Psychology of Observer Effects ......................................... 12

Sequential Unmasking: A Means of Minimizing Observer Effects in Forensic DNA Interpretation

Sir: Observer effects are rooted in the universal human tendency to interpret data in a manner consistent with one’s expectations (1). This tendency is particularly likely to distort the results of a scientific test when the underlying data are ambiguous and the scientist is exposed to domain-irrelevant information that engages emotions or desires (2). Despite impressions to the contrary, forensic DNA analysts often must resolve ambiguities, particularly when interpreting difficult evidence samples such as those that contain mixtures of DNA from two or more individuals, degraded or inhibited DNA, or limited quantities of DNA template. The full potential of forensic DNA testing can only be realized if observer effects are minimized. We met on December 1 and 2, 2007 in Washington, D.C. to discuss the implications of observer effects in forensic DNA testing and ways to minimize them. The interpretation of an evidentiary DNA profile should not be influenced by information about a suspect’s DNA profile (3–6). Each item of evidence must be interpreted independently of other items of evidence or reference samples. Yet forensic analysts are commonly aware of submitted reference profiles when interpreting DNA test results, creating the opportunity for a confirmatory bias, despite the best intentions of the analyst. Furthermore, analysts are sometimes exposed to information about the suspects, such as their history or motives, eyewitness identifications, presence or absence of a confession, and the like. Such information should have no bearing on how the results of a DNA test are interpreted, yet may compound an unintentional confirmatory bias. This bias can result in false inclusions under not uncommon conditions of ambiguity encountered in actual casework. It can also render currently used frequency statistics or likelihood ratios misleading. These problems can be minimized by preventing analysts from knowing the profile of submitted references (i.e., known samples) when interpreting testing results from evidentiary (i.e., unknown or questioned) samples. The necessary filtering or masking of submitted reference profiles can be accomplished in several ways, perhaps most easily by sequencing the laboratory workflow such that evidentiary samples are interpreted, and the interpretation is fully documented, before reference samples are compared. A simple protocol would dictate a separation of tasks between a qualified individual familiar with case information (a case manager) and an analyst from whom domain-irrelevant information is masked. Such a protocol would have the following steps. First, the analyst interprets the results of testing on the evidentiary samples. In this initial interpretation, the analyst would perform the following:

Regarding Champod, editorial: “Research focused mainly on bias will paralyse forensic science”

Dear Dr. Barron, Regarding Champod, editorial: “Research focused mainly on bias will paralyse forensic science.” In 2009, a report of the (U.S.) National Research Council declared that “[t]he forensic science disciplines are just beginning to become aware of contextual bias and the dangers it poses” [1]. The report called for additional research and discussion of how best to address this problem. Since that time, the literature on the topic of contextual bias in forensic science has begun to expand, and some laboratories are beginning to change procedures to address the problem. In his recent editorial in Science and Justice, Christophe Champod suggests that this trend has gone too far and threatens to “paralyse forensic science” [2]. We think his arguments are significantly overstated and deserve forceful refutation, lest they stand in the way of meaningful progress on this important issue. Dr. Champod opens by acknowledging that forensic scientists are vulnerable to bias. He says that he does not “want to minimize the importance of [research on this issue] and how it contributes to a better management of forensic science…” He continues by asking “...but should research remain focused on processes, or should we not move on to the basic understanding of the forensic traces?” He then comments on risks of “being focused on bias only.” By framing the matter in this way, Dr. Champod creates a false dichotomy, and implies facts about the current state of funding and research that are simply not the case. He seems to be saying that currently all or most research funding and publication is directed toward problems of bias, and little or none toward “basic understanding of the forensic traces.” Dr. Champod should know this is not the case, however, since (among other things) he is a co-author of a marvelous recently-released empirical study on fingerprint analysis funded by the (U.S.) National Institute of Justice [3]. Any perusal of NIJ grants, or the contents of leading forensic science journals, would not support Dr. Champod’s apparent view of the current research world. It would of course be a mistake for all of the available funding for research on forensic science topics to be devoted to the potential effects of bias, but again, this is neither the case currently nor is it in our opinion likely to become the case in the future. To discuss the risks of focusing “on bias only,” is simply raising a straw man when no one, not even the most ardent supporter of sequential unmasking or other approaches to the control of biasing information in forensic science practice, suggests focusing research “on bias only.” That said, we do believe that the research record both in forensic science and in a variety of other scientific areas has reached a point that clearly establishes the pressing need for all forensic areas to address the problem of contextual bias. As Andrew Rennison, who was then the forensic science regulator for England and Wales, told the plenary session of the American Academy of Forensic Science in February, “we don’t need more research on this issue, what we need is action.” This is not to say that further research on bias and its effects is not valuable, and

Commentary on: Thornton JI. Letter to the editor-a rejection of "working blind" as a cure for contextual bias. J Forensic Sci 2010;55(6):1663

Sir, In a recent letter (1) on the subject of contextual bias, Dr. John Thornton criticized what he called the ‘‘working blind’’ approach. According to Thornton, some commentators (he does not say who) have suggested that forensic scientists should know nothing about the case they are working on ‘‘apart from that which is absolutely necessary to conduct the indicated analysis and examination.’’ This ‘‘blind’’ approach is dangerous, Thornton argues, because forensic scientists need to know the facts of a case to make reasonable judgments about what specimens to test and how to test them. Thornton’s argument is correct, but he is attacking a straw man. As far as we know, no one has suggested that the individuals who decide what specimens to collect at a crime scene, or what analyses and examinations to perform on those specimens, should be blind to the facts of the case. What we, and others, have proposed is that individuals be blind to unnecessary contextual information when performing analytical tests and when making interpretations that require subjective judgment (2–5). One obvious way for forensic scientists to be ‘‘blind’’ during the analytical and interpretational phases of their work is to separate functions in the laboratory. Under what has been called the case manager approach (2–5), there would be two possible roles that a forensic scientist could perform. The case manager would ‘‘communicate with police officers and detectives, participate in decisions about what specimens to collect at crime scenes and how to test those specimens, and manage the flow of work to the laboratory’’ (5). The analyst would perform analytical tests and comparisons on specimens submitted to the laboratory in accordance with the instructions of the case manager. Under this model, the analyst can be blind to unnecessary contextual facts, while the case manager remains fully informed. A well-trained examiner could perform either role on different cases. The roles could be rotated among laboratory examiners to allow the laboratory access to the full breadth of expertise available; this would also allow the examiners to acquire and maintain a diversity of skills. Some of us have proposed a procedure called sequential unmasking as a means of minimizing contextual bias (6–8). Thornton mentions sequential unmasking but has not described it correctly. The purpose of sequential unmasking is not to provide analysts an opportunity to ‘‘determine whether tests that they have already run have been appropriate’’ (1). The purpose of sequential unmasking is to protect analysts from being biased unintentionally by information irrelevant to the exercise of their expertise or information that may have avoidable biasing effects if seen too early in the process of analysis. As an illustration, we presented a protocol that would prevent a DNA analyst from being influenced inappropriately by knowledge of reference profiles while making critical subjective judgments about the interpretation of evidentiary profiles. Aspects of this particular sequential unmasking approach have already been adopted by some laboratories in the U.S. in accordance with 2010 SWGDAM guideline 3.6.1, which states: ‘‘to the extent possible, DNA typing results from evidentiary samples are interpreted before comparison with any known samples, other than those of assumed contributors’’ (http://www.fbi.gov/about-us/lab/codis/swgdaminterpretation-guidelines). However, the approach is by no means limited to DNA. We believe similar sequential unmasking protocols can and should be developed for other forensic science disciplines. Sequential unmasking is not a call for uninformed decision making. We believe that analysts should have access to whatever information is actually necessary to conduct a thorough and appropriate analysis at whatever point that information becomes necessary. We recognize that difficult decisions will need to be made about what information is domain relevant and about when and how to ‘‘unmask’’ information that, while relevant, also has biasing potential. We believe that forensic scientists should be actively discussing these questions, rather than arguing that such a discussion is unnecessary. Calls for greater use of blind procedures to increase scientific rigor in forensic testing have indeed become more common in recent years. We were pleased that Dr. Thornton reported encountering such calls ‘‘everywhere we now turn,’’ although we were disappointed that a scientist with his distinguished record of contributions to the field remains unpersuaded of their value. The only argument Thornton offers in opposition is the mistaken claim that forensic scientists can ‘‘vanquish’’ bias by force of will. As he put it: ‘‘I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort’’ (1). Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity. Cognitive scientists and psychologists who study the operation of the human mind in judgmental tasks have shown repeatedly that people lack conscious awareness of factors that influence them (9–16). People often believe they were influenced by factors that did not affect their judgments and believe they were not influenced by factors that did affect their judgments. This research has a clear implication for the present discussion: contextual bias cannot be conquered by force of will because people are not consciously aware of the extent to which they are influenced by contextual factors. The inevitability of contextual bias is recognized and accepted in most scientific fields. Imagine the reaction in the medical community if a medical researcher claimed that he need not use blind procedures in his clinical trials because he is a person of integrity who will not allow himself to be biased. The claim would not only be rejected, but it would also likely invoke ridicule from professional colleagues. Forensic scientists who claim to be able to avoid contextual bias through force of will are making a claim contrary to well-established scientific facts concerning human judgment. If science is to progress, erroneous statements of this type must be rebutted forcefully even when (perhaps especially when) they are made by respected, senior scientists.

Bayes Wars Redivivus - An Exchange

An electronic exchange among 10 evidence scholars that began with a discussion of the restyled Federal Rules and grew into a significant restatement of debates in evidentiary scholarship over the last 50 years, touching on relevance, probative value, inference, Bayesianism and the foundations of evidence, with an introduction by Michael Risinger.

Forensic bitemark identification: weak foundations, exaggerated claims

Abstract Several forensic sciences, especially of the pattern-matching kind, are increasingly seen to lack the scientific foundation needed to justify continuing admission as trial evidence. Indeed, several have been abolished in the recent past. A likely next candidate for elimination is bitemark identification. A number of DNA exonerations have occurred in recent years for individuals convicted based on erroneous bitemark identifications. Intense scientific and legal scrutiny has resulted. An important National Academies review found little scientific support for the field. The Texas Forensic Science Commission recently recommended a moratorium on the admission of bitemark expert testimony. The California Supreme Court has a case before it that could start a national dismantling of forensic odontology. This article describes the (legal) basis for the rise of bitemark identification and the (scientific) basis for its impending fall. The article explains the general logic of forensic identification, the claims of bitemark identification, and reviews relevant empirical research on bitemark identification—highlighting both the lack of research and the lack of support provided by what research does exist. The rise and possible fall of bitemark identification evidence has broader implications—highlighting the weak scientific culture of forensic science and the laws difficulty in evaluating and responding to unreliable and unscientific evidence.

Innocents Convicted: An Empirically Justified Wrongful Conviction Rate

That would make the error rate [in felony convictions] .027 percent - or to put it another way, a success rate of 99.973 percent. - Justice Antonin Scalia, concurring in Kansas v. Marsh, June 29, 2006 (quoting Joshua Marquis) The news about the astounding accuracy of felony convictions in the United States, delivered by Justice Scalia and Joshua Marquis in the passage set out epigrammatically above, would be cause for rejoicing if it were true. Imagine. Only 27 factually wrong felony convictions out of every 100,000! Unfortunately, it is not true, as the empirical data analyzed in this article demonstrates. To a great extent, those who believe that our criminal justice system rarely convicts the factually innocent and those who believe such miscarriages are rife have generally talked past each other for want of any empirically-justified factual innocence wrongful conviction rate. This article remedies at least a part of this problem by establishing the first such empirically justified wrongful conviction rate ever for a significant universe of real world serious crimes: capital rape-murders in the 1980s. Using DNA exonerations for capital rape-murders from 1982 through 1989 as a numerator, and a 406-member sample of the 2235 capital sentences imposed during this period, this article shows that 21.45%, or around 479 of those, were cases of capital rape murder. Data supplied by the Innocence Project of Cardozo Law School and newly developed for this article show that only 67% of those cases would be expected to yield usable DNA for analysis. Combining these figures and dividing the numerator by the resulting denominator, a minimum factually wrongful conviction rate for capital rape-murder in the 1980s emerges: 3.3%. The article goes on to consider the likely ceiling accompanying this 3.3% floor, arriving at a slightly softer number for the maximum factual error rate of around 5%. The article then goes on to analyze the implications of a factual error rate of 3.3%-5% for both those who currently claim errors are extremely rare, and those who claim they are extremely common. Extension of the 3.3%-5% to other capital and non-capital categories of crime is discussed, and standards of moral duty to support system reform in the light of such error rates is considered at length.

The Emerging Role of Innocence Lawyer and the Need for Role-Differentiated Standards of Professional Conduct

There is growing awareness that a new lawyer role is emerging in American criminal litigation, separating itself from the traditional criminal defense role in ways that raise new issues of professional responsibility and ethics, both for the new role and for the traditional role of the criminal defense attorney. For want of a better term we may style this new role “innocence lawyer.” This new role is, at least for now, largely limited to attempting to aid innocent persons already convicted of a crime. It differs from the normal role of a lawyer representing a criminal defendant, or even representing an ordinary convicted person post-trial. Traditionally, lawyers at the post-conviction stage, whether on appeal or in collateral proceedings, carried forward the standard assumptions of the criminal defense role that emerged in the late 18th century and were largely solidified by the middle of the 19th century. We may label two of the main elements of those assumptions the “cab rank” principle and the “I don’t care if you are innocent” principle. The “cab rank” principle says that in general a criminal defense attorney is obliged, or at least encouraged and allowed, to accept as a client whoever comes to them (at least if they are not already overcommitted and the person can pay for their services). The “I don’t care if you are innocent” principle says that everyone, guilty or innocent, is entitled to put the government to its proof, and that while there are some constraints on the affirmative actions that may be taken to obtain an acquittal, the criminal defense lawyer is there to aggressively test and contest the evidence proffered by the government, and to insure that all constitutional limitations on the prosecution are respected regardless of their impact on the accuracy of the verdict of guilt, and regardless of underlying guilt or innocence. While each of the two principles may serve slightly different purposes, they are not unconnected. Together they work to create a strong “anti-signaling” ethic, in which the criminal defense attorney is prevented from signaling to anyone by any means that a client is guilty, and by necessary implication, affirmatively discouraged, and perhaps even prevented, from effectively signaling that there is real reason to believe that the client is in fact factually innocent of the crime, as opposed to being formally “presumed innocent.” The role of the innocence lawyer is significantly different. Working almost exclusively post-trial, the innocence lawyer is committed to finding those who are factually innocent of the crime for which they have been convicted, and to obtaining their freedom. This entails an ethical obligation to critically investigate and examine all available information, and in general to go forward only with those cases where factual innocence is clear or highly likely. This inherently involves signaling of a well-warranted belief in actual innocence, or at least the gross unsafety of the verdict in regard to actual guilt. Indeed, if the innocence lawyer is doing the right job of screening, the client is entitled to the benefit of such a signal, which is the product of the examination of many cases and the rejection of most, and countless hours of reinvestigation.

Commentary on: Thornton JI. Letter to the editor-a rejection of “working blind” as a cure for contextual bias. J Forensic Sci 2010;55(6):1663: LETTER TO THE EDITOR

Sir, In a recent letter (1) on the subject of contextual bias, Dr. John Thornton criticized what he called the ‘‘working blind’’ approach. According to Thornton, some commentators (he does not say who) have suggested that forensic scientists should know nothing about the case they are working on ‘‘apart from that which is absolutely necessary to conduct the indicated analysis and examination.’’ This ‘‘blind’’ approach is dangerous, Thornton argues, because forensic scientists need to know the facts of a case to make reasonable judgments about what specimens to test and how to test them. Thornton’s argument is correct, but he is attacking a straw man. As far as we know, no one has suggested that the individuals who decide what specimens to collect at a crime scene, or what analyses and examinations to perform on those specimens, should be blind to the facts of the case. What we, and others, have proposed is that individuals be blind to unnecessary contextual information when performing analytical tests and when making interpretations that require subjective judgment (2–5). One obvious way for forensic scientists to be ‘‘blind’’ during the analytical and interpretational phases of their work is to separate functions in the laboratory. Under what has been called the case manager approach (2–5), there would be two possible roles that a forensic scientist could perform. The case manager would ‘‘communicate with police officers and detectives, participate in decisions about what specimens to collect at crime scenes and how to test those specimens, and manage the flow of work to the laboratory’’ (5). The analyst would perform analytical tests and comparisons on specimens submitted to the laboratory in accordance with the instructions of the case manager. Under this model, the analyst can be blind to unnecessary contextual facts, while the case manager remains fully informed. A well-trained examiner could perform either role on different cases. The roles could be rotated among laboratory examiners to allow the laboratory access to the full breadth of expertise available; this would also allow the examiners to acquire and maintain a diversity of skills. Some of us have proposed a procedure called sequential unmasking as a means of minimizing contextual bias (6–8). Thornton mentions sequential unmasking but has not described it correctly. The purpose of sequential unmasking is not to provide analysts an opportunity to ‘‘determine whether tests that they have already run have been appropriate’’ (1). The purpose of sequential unmasking is to protect analysts from being biased unintentionally by information irrelevant to the exercise of their expertise or information that may have avoidable biasing effects if seen too early in the process of analysis. As an illustration, we presented a protocol that would prevent a DNA analyst from being influenced inappropriately by knowledge of reference profiles while making critical subjective judgments about the interpretation of evidentiary profiles. Aspects of this particular sequential unmasking approach have already been adopted by some laboratories in the U.S. in accordance with 2010 SWGDAM guideline 3.6.1, which states: ‘‘to the extent possible, DNA typing results from evidentiary samples are interpreted before comparison with any known samples, other than those of assumed contributors’’ (http://www.fbi.gov/about-us/lab/codis/swgdaminterpretation-guidelines). However, the approach is by no means limited to DNA. We believe similar sequential unmasking protocols can and should be developed for other forensic science disciplines. Sequential unmasking is not a call for uninformed decision making. We believe that analysts should have access to whatever information is actually necessary to conduct a thorough and appropriate analysis at whatever point that information becomes necessary. We recognize that difficult decisions will need to be made about what information is domain relevant and about when and how to ‘‘unmask’’ information that, while relevant, also has biasing potential. We believe that forensic scientists should be actively discussing these questions, rather than arguing that such a discussion is unnecessary. Calls for greater use of blind procedures to increase scientific rigor in forensic testing have indeed become more common in recent years. We were pleased that Dr. Thornton reported encountering such calls ‘‘everywhere we now turn,’’ although we were disappointed that a scientist with his distinguished record of contributions to the field remains unpersuaded of their value. The only argument Thornton offers in opposition is the mistaken claim that forensic scientists can ‘‘vanquish’’ bias by force of will. As he put it: ‘‘I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort’’ (1). Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity. Cognitive scientists and psychologists who study the operation of the human mind in judgmental tasks have shown repeatedly that people lack conscious awareness of factors that influence them (9–16). People often believe they were influenced by factors that did not affect their judgments and believe they were not influenced by factors that did affect their judgments. This research has a clear implication for the present discussion: contextual bias cannot be conquered by force of will because people are not consciously aware of the extent to which they are influenced by contextual factors. The inevitability of contextual bias is recognized and accepted in most scientific fields. Imagine the reaction in the medical community if a medical researcher claimed that he need not use blind procedures in his clinical trials because he is a person of integrity who will not allow himself to be biased. The claim would not only be rejected, but it would also likely invoke ridicule from professional colleagues. Forensic scientists who claim to be able to avoid contextual bias through force of will are making a claim contrary to well-established scientific facts concerning human judgment. If science is to progress, erroneous statements of this type must be rebutted forcefully even when (perhaps especially when) they are made by respected, senior scientists.