Jason R. Gilder

Sequential Unmasking: A Means of Minimizing Observer Effects in Forensic DNA Interpretation

Sir: Observer effects are rooted in the universal human tendency to interpret data in a manner consistent with one’s expectations (1). This tendency is particularly likely to distort the results of a scientific test when the underlying data are ambiguous and the scientist is exposed to domain-irrelevant information that engages emotions or desires (2). Despite impressions to the contrary, forensic DNA analysts often must resolve ambiguities, particularly when interpreting difficult evidence samples such as those that contain mixtures of DNA from two or more individuals, degraded or inhibited DNA, or limited quantities of DNA template. The full potential of forensic DNA testing can only be realized if observer effects are minimized. We met on December 1 and 2, 2007 in Washington, D.C. to discuss the implications of observer effects in forensic DNA testing and ways to minimize them. The interpretation of an evidentiary DNA profile should not be influenced by information about a suspect’s DNA profile (3–6). Each item of evidence must be interpreted independently of other items of evidence or reference samples. Yet forensic analysts are commonly aware of submitted reference profiles when interpreting DNA test results, creating the opportunity for a confirmatory bias, despite the best intentions of the analyst. Furthermore, analysts are sometimes exposed to information about the suspects, such as their history or motives, eyewitness identifications, presence or absence of a confession, and the like. Such information should have no bearing on how the results of a DNA test are interpreted, yet may compound an unintentional confirmatory bias. This bias can result in false inclusions under not uncommon conditions of ambiguity encountered in actual casework. It can also render currently used frequency statistics or likelihood ratios misleading. These problems can be minimized by preventing analysts from knowing the profile of submitted references (i.e., known samples) when interpreting testing results from evidentiary (i.e., unknown or questioned) samples. The necessary filtering or masking of submitted reference profiles can be accomplished in several ways, perhaps most easily by sequencing the laboratory workflow such that evidentiary samples are interpreted, and the interpretation is fully documented, before reference samples are compared. A simple protocol would dictate a separation of tasks between a qualified individual familiar with case information (a case manager) and an analyst from whom domain-irrelevant information is masked. Such a protocol would have the following steps. First, the analyst interprets the results of testing on the evidentiary samples. In this initial interpretation, the analyst would perform the following:

Run-Specific Limits of Detection and Quantitation for STR-based DNA Testing

ABSTRACT: STR‐based DNA profiling is an exceptionally sensitive analytical technique that is often used to obtain results at the very limits of its sensitivity. The challenge of reliably distinguishing between signal and noise in such situations is one that has been rigorously addressed in numerous other analytical disciplines. However, an inability to determine accurately the height of electropherogram baselines has caused forensic DNA profiling laboratories to utilize alternative approaches. Minimum thresholds established during laboratory validation studies have become the de facto standard for distinguishing between reliable signal and noise/technical artifacts. These minimum peak height thresholds generally fail to consider variability in the sensitivity of instruments, reagents, and the skill of human analysts involved in the DNA profiling process over the course of time. Software (BatchExtract) made publicly available by the National Center for Biotechnology Information now provides an alternative means of establishing limits of detection and quantitation that is more consistent with those used in other analytical disciplines. We have used that software to determine the height of each data collection point for each dye along a control samples electropherogram trace. These values were then used to determine a limit of detection (the average amount of background noise plus three standard deviations) and a limit of quantitation (the average amount of background noise plus 10 standard deviations) for each control sample. Analyses of the electropherogram data associated with the positive, negative, and reagent blank controls included in 50 different capillary electrophoresis runs validate that this approach could be used to determine run‐specific thresholds objectively for use in forensic DNA casework.

Time for DNA Disclosure

The legislation that established the U.S. National DNA Index System (NDIS) in 1994 explicitly anticipated that database records would be available for purposes of research and quality control “if personally identifiable information is removed” [42 U.S.C. Sec 14132(b)(3)(D)]. However, the Federal

Magnitude-dependent variation in peak height balance at heterozygous STR loci

When the smaller of two peaks at an STR locus is less than 70% the magnitude of the larger peak at that locus, the disparity is typically taken to be an indication that there is more than one contributor of template DNA to the sample being tested. An analysis of 1,763 heterozygous allele pairs suggests that a peak height imbalance threshold that varies with the magnitude of the peaks being evaluated at a locus is superior to a fixed threshold. Identifying samples that are likely to be mixtures and those that are likely to have arisen from a single source is accomplished more reliably when a statistically based, magnitude-dependent peak height imbalance threshold is used. The amelogenin locus was found to behave in a similar fashion and was also found to have no systematic bias that favored the amplification of Y or X alleles.

Comments on the review of low copy number testing

Dear Sir,A challenge to the reliability of low copy number (LCN)DNA profiling in the trial of Sean Hoey in Belfast CrownCourt in Northern Ireland (R v Hoey [2007] NICC 49, 20December, 2007) prompted the UK’s new Forensic ScienceRegulator (Andrew Rennison) to commission a review oflow template DNA profiling techniques. That review [2],conducted by Professor Brian Caddy (with the assistance ofDr. Adrian Linacre and Dr. Graham Taylor) was released on12 April, 2008 and concluded that LCN DNA profiling is“robust” and “fit for purpose.” Yet, the review accepts thatthe evidence presented in Sean Hoey’s trial was insufficientto establish the validity of the technique. It also enumerates21 recommendations for specific improvements that shouldbe undertaken to improve the methodology, including suchbasic steps as the development of a consensus on theinterpretation of test results and efforts to establish “bestpractices” for interpretation.We believe the conclusions of the review are inconsistentwith its recommendations in a number of respects. Forexample, it is difficult to see how a forensic techniquecould be deemed adequately validated for use in thecourtroom when there is not yet a consensus on how itsresults should be interpreted. The review thus raisesimportant issues about what it means for a forensic sciencetechnique to be validated. It also establishes grounds forconcern about the way that LCN DNA test results havebeen interpreted in earlier cases.We are concerned that the review team relied only oninput regarding the merits of LCN approaches fromorganizations that are dedicated to promoting its use bylaw enforcement. Consultation with known critics of thetechnique (or even a review of their published works)would have provided the reviewers with a broaderperspective of what work remains to be done before theapproach can become generally accepted within theinternational scientific community. There are in fact thingsabout LCN approaches upon which the reviewers andcritics do agree. For instance, caution that “[p]ublicizing thepotential of the application of LCN typing withoutdescribing its limitations may cause misunderstanding” [1]which is consistent with the review’s recommendations 1,3, and 13. But given the conclusion that “[t]he methodcannot be used for exculpatory purposes” [1], the review’sultimate conclusion that LCN testing is “fit for purpose”leaves the important but unanswered question of “what isthat purpose?”

Commentary on: Thornton JI. Letter to the editor-a rejection of "working blind" as a cure for contextual bias. J Forensic Sci 2010;55(6):1663

Sir, In a recent letter (1) on the subject of contextual bias, Dr. John Thornton criticized what he called the ‘‘working blind’’ approach. According to Thornton, some commentators (he does not say who) have suggested that forensic scientists should know nothing about the case they are working on ‘‘apart from that which is absolutely necessary to conduct the indicated analysis and examination.’’ This ‘‘blind’’ approach is dangerous, Thornton argues, because forensic scientists need to know the facts of a case to make reasonable judgments about what specimens to test and how to test them. Thornton’s argument is correct, but he is attacking a straw man. As far as we know, no one has suggested that the individuals who decide what specimens to collect at a crime scene, or what analyses and examinations to perform on those specimens, should be blind to the facts of the case. What we, and others, have proposed is that individuals be blind to unnecessary contextual information when performing analytical tests and when making interpretations that require subjective judgment (2–5). One obvious way for forensic scientists to be ‘‘blind’’ during the analytical and interpretational phases of their work is to separate functions in the laboratory. Under what has been called the case manager approach (2–5), there would be two possible roles that a forensic scientist could perform. The case manager would ‘‘communicate with police officers and detectives, participate in decisions about what specimens to collect at crime scenes and how to test those specimens, and manage the flow of work to the laboratory’’ (5). The analyst would perform analytical tests and comparisons on specimens submitted to the laboratory in accordance with the instructions of the case manager. Under this model, the analyst can be blind to unnecessary contextual facts, while the case manager remains fully informed. A well-trained examiner could perform either role on different cases. The roles could be rotated among laboratory examiners to allow the laboratory access to the full breadth of expertise available; this would also allow the examiners to acquire and maintain a diversity of skills. Some of us have proposed a procedure called sequential unmasking as a means of minimizing contextual bias (6–8). Thornton mentions sequential unmasking but has not described it correctly. The purpose of sequential unmasking is not to provide analysts an opportunity to ‘‘determine whether tests that they have already run have been appropriate’’ (1). The purpose of sequential unmasking is to protect analysts from being biased unintentionally by information irrelevant to the exercise of their expertise or information that may have avoidable biasing effects if seen too early in the process of analysis. As an illustration, we presented a protocol that would prevent a DNA analyst from being influenced inappropriately by knowledge of reference profiles while making critical subjective judgments about the interpretation of evidentiary profiles. Aspects of this particular sequential unmasking approach have already been adopted by some laboratories in the U.S. in accordance with 2010 SWGDAM guideline 3.6.1, which states: ‘‘to the extent possible, DNA typing results from evidentiary samples are interpreted before comparison with any known samples, other than those of assumed contributors’’ (http://www.fbi.gov/about-us/lab/codis/swgdaminterpretation-guidelines). However, the approach is by no means limited to DNA. We believe similar sequential unmasking protocols can and should be developed for other forensic science disciplines. Sequential unmasking is not a call for uninformed decision making. We believe that analysts should have access to whatever information is actually necessary to conduct a thorough and appropriate analysis at whatever point that information becomes necessary. We recognize that difficult decisions will need to be made about what information is domain relevant and about when and how to ‘‘unmask’’ information that, while relevant, also has biasing potential. We believe that forensic scientists should be actively discussing these questions, rather than arguing that such a discussion is unnecessary. Calls for greater use of blind procedures to increase scientific rigor in forensic testing have indeed become more common in recent years. We were pleased that Dr. Thornton reported encountering such calls ‘‘everywhere we now turn,’’ although we were disappointed that a scientist with his distinguished record of contributions to the field remains unpersuaded of their value. The only argument Thornton offers in opposition is the mistaken claim that forensic scientists can ‘‘vanquish’’ bias by force of will. As he put it: ‘‘I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort’’ (1). Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity. Cognitive scientists and psychologists who study the operation of the human mind in judgmental tasks have shown repeatedly that people lack conscious awareness of factors that influence them (9–16). People often believe they were influenced by factors that did not affect their judgments and believe they were not influenced by factors that did affect their judgments. This research has a clear implication for the present discussion: contextual bias cannot be conquered by force of will because people are not consciously aware of the extent to which they are influenced by contextual factors. The inevitability of contextual bias is recognized and accepted in most scientific fields. Imagine the reaction in the medical community if a medical researcher claimed that he need not use blind procedures in his clinical trials because he is a person of integrity who will not allow himself to be biased. The claim would not only be rejected, but it would also likely invoke ridicule from professional colleagues. Forensic scientists who claim to be able to avoid contextual bias through force of will are making a claim contrary to well-established scientific facts concerning human judgment. If science is to progress, erroneous statements of this type must be rebutted forcefully even when (perhaps especially when) they are made by respected, senior scientists.

A versatile tool for student projects: an ASM programming language for the Lego mindstorm

Finding new and exciting senior design projects for undergraduate computer engineers can be challenging for both students and faculty advisors. The challenge is even more pronounced when attempting to construct interdisciplinary design projects to better meet ABET 2000 criteria and to better prepare graduating students for careers as engineers. We present an interdisciplinary, honors senior design project incorporating robotics control, algorithmic state machine design, reverse-engineering, assembly programming, and language, and basic compiler design. We believe this two-quarter project was extremely successful, and can be used as a model for similar design projects at other undergraduate institutions.

Systematic Differences in Electropherogram Peak Heights Reported by Different Versions of the GeneScan Software

DNA profiling using STRs on the 310 and 3100 Genetic Analyzers routinely generates electropherograms that are analyzed with the GeneScan software available from the instruments manufacturer, Applied Biosystems. Users have been able to choose from three different smoothing options that have been known to result in significant differences in the peak heights that are reported. Improvements in the underlying algorithm of the most recent version of the software also result in significant and somewhat predictable differences in peak height values. Laboratories that have performed validation studies using older versions of GeneScan should either reanalyze the data generated in those validation studies with the newest version of the software or otherwise take into consideration the systematically higher peak height values obtained as they begin following the recommendation of the manufacturer and use the new algorithm.

PocketMol: a molecular visualization tool for the Pocket PC

Molecular visualization programs are available on many platforms. They allow a user to visualize and manipulate molecular structures. PocketMol provides the same functionality on a Pocket PC handheld computer. Using standard protein data bank (pdb) files, the user can move, rotate, and scale a protein to explore its structure and function. The user can choose from a standard backbone view or a simplified view using only alpha carbon atoms. PocketMolGX uses the Microsoft Game API to provide fast animation that is quite smooth. PocketMol is designed as an aid for those wishing to explore or demonstrate protein structures without the availability of a full-size computer.

Commentary on: Thornton JI. Letter to the editor-a rejection of “working blind” as a cure for contextual bias. J Forensic Sci 2010;55(6):1663: LETTER TO THE EDITOR

Sir, In a recent letter (1) on the subject of contextual bias, Dr. John Thornton criticized what he called the ‘‘working blind’’ approach. According to Thornton, some commentators (he does not say who) have suggested that forensic scientists should know nothing about the case they are working on ‘‘apart from that which is absolutely necessary to conduct the indicated analysis and examination.’’ This ‘‘blind’’ approach is dangerous, Thornton argues, because forensic scientists need to know the facts of a case to make reasonable judgments about what specimens to test and how to test them. Thornton’s argument is correct, but he is attacking a straw man. As far as we know, no one has suggested that the individuals who decide what specimens to collect at a crime scene, or what analyses and examinations to perform on those specimens, should be blind to the facts of the case. What we, and others, have proposed is that individuals be blind to unnecessary contextual information when performing analytical tests and when making interpretations that require subjective judgment (2–5). One obvious way for forensic scientists to be ‘‘blind’’ during the analytical and interpretational phases of their work is to separate functions in the laboratory. Under what has been called the case manager approach (2–5), there would be two possible roles that a forensic scientist could perform. The case manager would ‘‘communicate with police officers and detectives, participate in decisions about what specimens to collect at crime scenes and how to test those specimens, and manage the flow of work to the laboratory’’ (5). The analyst would perform analytical tests and comparisons on specimens submitted to the laboratory in accordance with the instructions of the case manager. Under this model, the analyst can be blind to unnecessary contextual facts, while the case manager remains fully informed. A well-trained examiner could perform either role on different cases. The roles could be rotated among laboratory examiners to allow the laboratory access to the full breadth of expertise available; this would also allow the examiners to acquire and maintain a diversity of skills. Some of us have proposed a procedure called sequential unmasking as a means of minimizing contextual bias (6–8). Thornton mentions sequential unmasking but has not described it correctly. The purpose of sequential unmasking is not to provide analysts an opportunity to ‘‘determine whether tests that they have already run have been appropriate’’ (1). The purpose of sequential unmasking is to protect analysts from being biased unintentionally by information irrelevant to the exercise of their expertise or information that may have avoidable biasing effects if seen too early in the process of analysis. As an illustration, we presented a protocol that would prevent a DNA analyst from being influenced inappropriately by knowledge of reference profiles while making critical subjective judgments about the interpretation of evidentiary profiles. Aspects of this particular sequential unmasking approach have already been adopted by some laboratories in the U.S. in accordance with 2010 SWGDAM guideline 3.6.1, which states: ‘‘to the extent possible, DNA typing results from evidentiary samples are interpreted before comparison with any known samples, other than those of assumed contributors’’ (http://www.fbi.gov/about-us/lab/codis/swgdaminterpretation-guidelines). However, the approach is by no means limited to DNA. We believe similar sequential unmasking protocols can and should be developed for other forensic science disciplines. Sequential unmasking is not a call for uninformed decision making. We believe that analysts should have access to whatever information is actually necessary to conduct a thorough and appropriate analysis at whatever point that information becomes necessary. We recognize that difficult decisions will need to be made about what information is domain relevant and about when and how to ‘‘unmask’’ information that, while relevant, also has biasing potential. We believe that forensic scientists should be actively discussing these questions, rather than arguing that such a discussion is unnecessary. Calls for greater use of blind procedures to increase scientific rigor in forensic testing have indeed become more common in recent years. We were pleased that Dr. Thornton reported encountering such calls ‘‘everywhere we now turn,’’ although we were disappointed that a scientist with his distinguished record of contributions to the field remains unpersuaded of their value. The only argument Thornton offers in opposition is the mistaken claim that forensic scientists can ‘‘vanquish’’ bias by force of will. As he put it: ‘‘I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort’’ (1). Let us be clear. We are not ‘‘insinuating’’ that forensic scientists lack this intellectual capacity; we are asserting that it is a proven and well-accepted scientific fact that all human beings, including forensic scientists, lack this capacity. Cognitive scientists and psychologists who study the operation of the human mind in judgmental tasks have shown repeatedly that people lack conscious awareness of factors that influence them (9–16). People often believe they were influenced by factors that did not affect their judgments and believe they were not influenced by factors that did affect their judgments. This research has a clear implication for the present discussion: contextual bias cannot be conquered by force of will because people are not consciously aware of the extent to which they are influenced by contextual factors. The inevitability of contextual bias is recognized and accepted in most scientific fields. Imagine the reaction in the medical community if a medical researcher claimed that he need not use blind procedures in his clinical trials because he is a person of integrity who will not allow himself to be biased. The claim would not only be rejected, but it would also likely invoke ridicule from professional colleagues. Forensic scientists who claim to be able to avoid contextual bias through force of will are making a claim contrary to well-established scientific facts concerning human judgment. If science is to progress, erroneous statements of this type must be rebutted forcefully even when (perhaps especially when) they are made by respected, senior scientists.