Keith Inman

Sequential Unmasking: A Means of Minimizing Observer Effects in Forensic DNA Interpretation

Sir: Observer effects are rooted in the universal human tendency to interpret data in a manner consistent with one’s expectations (1). This tendency is particularly likely to distort the results of a scientific test when the underlying data are ambiguous and the scientist is exposed to domain-irrelevant information that engages emotions or desires (2). Despite impressions to the contrary, forensic DNA analysts often must resolve ambiguities, particularly when interpreting difficult evidence samples such as those that contain mixtures of DNA from two or more individuals, degraded or inhibited DNA, or limited quantities of DNA template. The full potential of forensic DNA testing can only be realized if observer effects are minimized. We met on December 1 and 2, 2007 in Washington, D.C. to discuss the implications of observer effects in forensic DNA testing and ways to minimize them. The interpretation of an evidentiary DNA profile should not be influenced by information about a suspect’s DNA profile (3–6). Each item of evidence must be interpreted independently of other items of evidence or reference samples. Yet forensic analysts are commonly aware of submitted reference profiles when interpreting DNA test results, creating the opportunity for a confirmatory bias, despite the best intentions of the analyst. Furthermore, analysts are sometimes exposed to information about the suspects, such as their history or motives, eyewitness identifications, presence or absence of a confession, and the like. Such information should have no bearing on how the results of a DNA test are interpreted, yet may compound an unintentional confirmatory bias. This bias can result in false inclusions under not uncommon conditions of ambiguity encountered in actual casework. It can also render currently used frequency statistics or likelihood ratios misleading. These problems can be minimized by preventing analysts from knowing the profile of submitted references (i.e., known samples) when interpreting testing results from evidentiary (i.e., unknown or questioned) samples. The necessary filtering or masking of submitted reference profiles can be accomplished in several ways, perhaps most easily by sequencing the laboratory workflow such that evidentiary samples are interpreted, and the interpretation is fully documented, before reference samples are compared. A simple protocol would dictate a separation of tasks between a qualified individual familiar with case information (a case manager) and an analyst from whom domain-irrelevant information is masked. Such a protocol would have the following steps. First, the analyst interprets the results of testing on the evidentiary samples. In this initial interpretation, the analyst would perform the following:

DNA Based Identification

No single biometric technique provides an optimal identification method in all cases. However, depending on the biological system, a particular biometric may stand out as the preferred method of identification. DNA profiling clearly emerges as the most powerful and reliable system for measuring genetic traits. DNA typing provides valuable information in such diverse applications as medical science, environmental science, historical research, and, of course, forensic science. In this chapter, we review the most common genetic typing systems and the laboratory techniques employed to analyze the markers. We discuss the precautions that must be taken in collecting samples and the consequences of analyzing non-optimal material. Of particular interest from a societal standpoint is the creation of DNA databanks and the privacy issues associated with them. Finally, when two samples appear to be indistinguishable by the tests conducted, the significance of the association must then be determined. The methods by which the rarity of a genetic profile is determined provide the DNA typing community with it’s most fertile ground for debate. We review the major elements of the discussion here. With the advent of solid state and automated methods for DNA typing, the technique will soon become not only indispensable, but practical.

Commentary on: Thornton JI. Letter to the editor-a rejection of “working blind” as a cure for contextual bias. J Forensic Sci 2010;55(6):1663: LETTER TO THE EDITOR

Sir, In a recent letter (1) on the subject of contextual bias, Dr. John Thornton criticized what he called the ‘‘working blind’’ approach. According to Thornton, some commentators (he does not say who) have suggested that forensic scientists should know nothing about the case they are working on ‘‘apart from that which is absolutely necessary to conduct the indicated analysis and examination.’’ This ‘‘blind’’ approach is dangerous, Thornton argues, because forensic scientists need to know the facts of a case to make reasonable judgments about what specimens to test and how to test them. Thornton’s argument is correct, but he is attacking a straw man. As far as we know, no one has suggested that the individuals who decide what specimens to collect at a crime scene, or what analyses and examinations to perform on those specimens, should be blind to the facts of the case. What we, and others, have proposed is that individuals be blind to unnecessary contextual information when performing analytical tests and when making interpretations that require subjective judgment (2–5). One obvious way for forensic scientists to be ‘‘blind’’ during the analytical and interpretational phases of their work is to separate functions in the laboratory. Under what has been called the case manager approach (2–5), there would be two possible roles that a forensic scientist could perform. The case manager would ‘‘communicate with police officers and detectives, participate in decisions about what specimens to collect at crime scenes and how to test those specimens, and manage the flow of work to the laboratory’’ (5). The analyst would perform analytical tests and comparisons on specimens submitted to the laboratory in accordance with the instructions of the case manager. Under this model, the analyst can be blind to unnecessary contextual facts, while the case manager remains fully informed. A well-trained examiner could perform either role on different cases. The roles could be rotated among laboratory examiners to allow the laboratory access to the full breadth of expertise available; this would also allow the examiners to acquire and maintain a diversity of skills. Some of us have proposed a procedure called sequential unmasking as a means of minimizing contextual bias (6–8). Thornton mentions sequential unmasking but has not described it correctly. The purpose of sequential unmasking is not to provide analysts an opportunity to ‘‘determine whether tests that they have already run have been appropriate’’ (1). The purpose of sequential unmasking is to protect analysts from being biased unintentionally by information irrelevant to the exercise of their expertise or information that may have avoidable biasing effects if seen too early in the process of analysis. As an illustration, we presented a protocol that would prevent a DNA analyst from being influenced inappropriately by knowledge of reference profiles while making critical subjective judgments about the interpretation of evidentiary profiles. Aspects of this particular sequential unmasking approach have already been adopted by some laboratories in the U.S. in accordance with 2010 SWGDAM guideline 3.6.1, which states: ‘‘to the extent possible, DNA typing results from evidentiary samples are interpreted before comparison with any known samples, other than those of assumed contributors’’ (http://www.fbi.gov/about-us/lab/codis/swgdaminterpretation-guidelines). However, the approach is by no means limited to DNA. We believe similar sequential unmasking protocols can and should be developed for other forensic science disciplines. Sequential unmasking is not a call for uninformed decision making. We believe that analysts should have access to whatever information is actually necessary to conduct a thorough and appropriate analysis at whatever point that information becomes necessary. We recognize that difficult decisions will need to be made about what information is domain relevant and about when and how to ‘‘unmask’’ information that, while relevant, also has biasing potential. We believe that forensic scientists should be actively discussing these questions, rather than arguing that such a discussion is unnecessary. Calls for greater use of blind procedures to increase scientific rigor in forensic testing have indeed become more common in recent years. We were pleased that Dr. Thornton reported encountering such calls ‘‘everywhere we now turn,’’ although we were disappointed that a scientist with his distinguished record of contributions to the field remains unpersuaded of their value. The only argument Thornton offers in opposition is the mistaken claim that forensic scientists can ‘‘vanquish’’ bias by force of will. As he put it: ‘‘I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort’’ (1). Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity. Cognitive scientists and psychologists who study the operation of the human mind in judgmental tasks have shown repeatedly that people lack conscious awareness of factors that influence them (9–16). People often believe they were influenced by factors that did not affect their judgments and believe they were not influenced by factors that did affect their judgments. This research has a clear implication for the present discussion: contextual bias cannot be conquered by force of will because people are not consciously aware of the extent to which they are influenced by contextual factors. The inevitability of contextual bias is recognized and accepted in most scientific fields. Imagine the reaction in the medical community if a medical researcher claimed that he need not use blind procedures in his clinical trials because he is a person of integrity who will not allow himself to be biased. The claim would not only be rejected, but it would also likely invoke ridicule from professional colleagues. Forensic scientists who claim to be able to avoid contextual bias through force of will are making a claim contrary to well-established scientific facts concerning human judgment. If science is to progress, erroneous statements of this type must be rebutted forcefully even when (perhaps especially when) they are made by respected, senior scientists.