Subjectivity and bias in forensic DNA mixture interpretation

Science and Justice 51 (2011) 204–208 Contents lists available at SciVerse ScienceDirect Science and Justice j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / s c i j u s Subjectivity and bias in forensic DNA mixture interpretation☆ Itiel E. Dror a, b,⁎, Greg Hampikian c a b c Institute of Cognitive Neuroscience, University College London (UCL), London, UK Cognitive Consultants International (CCI), London, UK Departments of Biology and Criminal Justice, Boise State University, USA a r t i c l e i n f o Article history: Received 30 July 2010 Received in revised form 30 July 2011 Accepted 4 August 2011 Keywords: Human cognition Bias Forensic decision making Contextual influences DNA interpretation a b s t r a c t The objectivity of forensic science decision making has received increased attention and scrutiny. However, there are only a few published studies experimentally addressing the potential for contextual bias. Because of the esteem of DNA evidence, it is important to study and assess the impact of subjectivity and bias on DNA mixture interpretation. The study reported here presents empirical data suggesting that DNA mixture interpretation is subjective. When 17 North American expert DNA examiners were asked for their interpretation of data from an adjudicated criminal case in that jurisdiction, they produced inconsistent interpretations. Furthermore, the majority of 'context free' experts disagreed with the laboratory's pre-trial conclusions, suggesting that the extraneous context of the criminal case may have influenced the interpretation of the DNA evidence, thereby showing a biasing effect of contextual information in DNA mixture interpretation. © 2011 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved. Seeking and interpreting information in a biased way so that it fits existing beliefs, expectation, hope, or motivation is a result of how we reason and is widespread [1]. The potential for such biases in forensic science disciplines has been suggested before [2,3], and has now been highlighted by the National Academy of Science (NAS) report on Strengthening Forensic Science in the United States: A Path Forward [4]. It directly discusses “the potential for bias and error in human observers” (p. 8), and states that “the extent to which practitioners in a particular forensic discipline rely on human interpretation that could be tainted by error, [or] the threat of bias . . . . [is] significant” (p. 9). Indeed, empirical research supports the effects of bias in some forensic disciplines; for example, in fingerprinting, the same forensic experts may arrive at different conclusions when identical evidence is presented within different extraneous contexts (e.g., whether the detective believes the suspect is guilty, or the suspect confessed) [5–8]. However, in contrast to other forensic disciplines, DNA is regarded as the gold standard of forensic science [9]. DNA has been held as objective and immune to subjectivity and bias; “In the past several years, it has become commonplace in the courts, in the media, and in much of the technical literature, to contrast the scientific and objective evidence supplied by DNA profiling, with the experiential or subjective opinions given by traditional forensic experts” [9] (p. 97). Indeed, even the NAS ☆ One sentence summary: DNA mixture interpretation is subjective and may be susceptible to bias by extraneous context, as evidenced by conflicting conclusions concerning the inclusion or exclusion of suspects. ⁎ Corresponding author. E-mail address: i.dror@ucl.ac.uk (I.E. Dror). URL: http://www.cci-hq.com (I.E. Dror). distinguishes between “forensic science disciplines [that] are laboratory based (e.g., nuclear and mitochondrial DNA analysis, toxicology and drug analysis)” [4] (p. 38), and other forensic disciplines that are “based on expert interpretation of observed patterns (e.g., fingerprints, writing samples, toolmarks, bite marks, and specimens such as hair)” [4] (p. 38). If correct, then DNA analyses should be consistent and not affected by domain irrelevant contextual circumstances. It seems, however, that at least in complex situations (such as with DNA mixtures) DNA does require and rely on human examiners making a variety of subjective judgements that are susceptible to bias. Indeed, in contrast to the view that DNA is objective, some have proposed that DNA analysis interpretations may be subjective and may even be influenced by a variety of factors [10,11]. However, such claims – both for the subjectivity or for the objectivity – of DNA analysis have rarely been examined and tested through empirical research. To investigate the subjectivity and biasability of mixture DNA analysis we observed and compared the conclusions on identical DNA evidence that was presented within and between different extraneous contextual information. To properly investigate this issue, it was critical to: 1. conduct the study with qualified DNA expert analysts who conduct real casework in accredited laboratories, and 2. that the examiners genuinely believed the contextual information, as contrived context within an experimental setup does not have the effect or impact as that of genuinely believed real context [8]. To achieve these goals we used mixture DNA analysis from a real adjudicated criminal case, using records obtained through the Georgia Freedom of Information Act. The case we chose provided us with analysis within extraneous context. We then took the same DNA evidence and presented it to 17 independent North American DNA expert analysts, but without the potentially biasing contextual case 1355-0306/$ – see front matter © 2011 Forensic Science Society. Published by Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.scijus.2011.08.004 I.E. Dror, G. Hampikian / Science and Justice 51 (2011) 204–208 information. First we compared the consistency in interpretation and conclusion within those 17 examiners to assess subjectivity in DNA analysis. Then we compared between them and those who examined the DNA mixture within the extraneous context of the criminal case to assess biasability in DNA analysis. The DNA evidence related to a gang rape case in which one of the assailants testified against the other suspects in return for a lesser sentence as part of his cooperation in a plea bargain deal. However, those identified through the plea bargain denied any involvement in the rape. The mixture DNA from the sexual assault was examined by experts in the real criminal case, and their analysis and conclusions were that the suspects that were identified by the cooperative assailant could not be excluded from being contributors to the mixture. The establishment of this corroborating fact was essential to the prosecution of the suspects who claimed innocence. Under the law of that state where this act occurred, the testimony of the admitted rapist would not be admitted without corroborating evidence. Therefore the DNA conclusions were critical to prosecution. If the suspects were excluded by DNA, or even if the DNA was “inconclusive”, the incriminating testimony of the admitted rapist would most likely not be allowed. As potentially biasing as this domain irrelevant context was, if DNA was totally objective it should not have affected their analysis. In this study we took the original materials used by the DNA examiners that concluded that the suspect cannot be excluded, and presented them to 17 other DNA examiners, 'context free.' These 17 DNA examiners were all expert DNA analysts who were working casework in an accredited governmental laboratory in North America. Fourteen were female and three were male; their mean age was 40.7 (SD = 5.86), and their mean years of experience conducting DNA analysis was 8.9 (SD = 3.96). Two examiners had a BSc, 12 had a MSc (either in biology or forensic science), and 2 had PhDs (one participant did not provide information on their level of education). We asked the 17 independent DNA examiners to examine the DNA mixture along with DNA profiles of the victim and three suspects (Table 1) (one of the suspects, suspect 3, was the point of interest, as he was determined as ‘cannot be excluded’ by the DNA examiners who examined his DNA within the potentially biasing context). The evidence presented to them was comprised from the electropherograms (Figs. 1 and 2) available to the original examiners, and included the Vaginal Sperm Fraction (Profiler+) and Vaginal Sperm fraction (CoFiler). They were also provided with the relevant contextual information that was provided to the original examiners, such as the concentration of DNA in the sperm fraction extract, the DNA amplification conditions, and capillary electrophoresis injection times. Each of the 17 DNA examiners independently examined the evidence, and gave one of three conclusions for each of the suspects: ‘cannot be excluded’, ‘excluded’, or ‘inconclusive.’ In regard to suspect three, the results obtained from the 17 independent DNA examiners varied. One examiner concluded that the suspect ‘cannot be excluded’, 4 examiners concluded ‘inconclusive’, and Table 1 Suspect 3 portion of the allele chart. Locus S3 D3 vWA FGA D8 D21 D18 D5 D13 D7 D3 D16 THO1 TPOX CSF1PO 14, 17 17, 18 22, 24 14, 15 28, 28 13, 18 12, 13 10, 14 9, 10 No data 9, 13 7, 8 9, 9 11, 11 205 12 examiners concluded ‘exclude.’ The results are revealing in two respects: First, the fact that the 17 DNA examiners were not consistent in their conclusions, by itself, suggests that there is an element of subjectivity in DNA interpretation. If it was totally objective, then all the examiners would have reached the same conclusion, especially since they all work in the same laboratory and follow the same interpretation guidelines. The observed inconsistencies within the 17 examiners who conducted their analysis on the identical evidence, ‘context free,’ demonstrated subjectivity in DNA mixture analysis, which may reflect individual differences (e.g., training, experience, personality, and motivation). It is interesting that even using the ‘gold standard’ [9] DNA, different examiners reach conflicting conclusions based on identical evidentiary data. Second, comparing the data between examiners, those from the context free condition to those who were exposed to the extraneous context condition, it is possible that the domain irrelevant information may have biased their interpretation. The DNA analysts who concluded that the suspect cannot be excluded within the biasing context of the criminal case, are in sharp contrast to the vast majority of examiners who examined the same evidence without this biasing context. Only 1 (out of 17) gave the same conclusion as the original analysts, 16 other examiners reached a different and conflicting conclusion (either ‘exclude’, 12 14 examiners, or ‘inconclusive’, 4 examiners). Thus, the extraneous context appears to have influenced the interpretation of the DNA mixture, however, it is always hard to draw scientific conclusions when dealing with methodologies involving real casework. It must be emphasized, however, that these effects were observed for a DNA mixture analysis. Previous research in forensic identification suggests that contextual influences are most powerful when the evidence is ambiguous, complex, and a ‘hard call’ [8]. When the data is clear and decisions are simple, then the power of context is diminished. Gill has been quoted to say that “If you show 10 colleagues a mixture, you will probably end up with 10 different answers” [12]. The difficulties and challenges presented by complex DNA mixture have been the focus of several discussions [13–21], and are an important component of ‘expert systems’ and statistical computing that try to more objectively deconvolute and interpret DNA mixtures [22,23]. The study reported here, the first experimental study exploring DNA interpretation, demonstrates that DNA mixture interpretation has subjective elements and may be susceptible to bias and other contextual influences. Minimizing such potential effects is important, and may include specific training on bias issues, as well as procedures and best practices especially designed to limit contextual influences (such as sequential unmasking [24]). This study also demonstrates that all types of DNA analysis should not be lumped together as the “gold standard.” It is true, that in contrast with many areas of forensic science [25], identity testing using DNA has progressed to the point of general acceptance when complete profiles are obtained from a single DNA contributor [26]. Consistent with this level of acceptance in the scientific community, the courts in the United States and elsewhere equate identity with DNA profiles that include complete allelic data from 13 or more of the standard short tandem repeat loci (STRs). However, in cases where low numbers of template molecules are amplified [27], or where complex mixtures are analyzed, subjective conclusions are made by analysts. This is evidenced by our experiment and the case we discuss, however, one cannot estimate its magnitude and impact without more empirical studies. The great degree of variability in laboratory methods regarding DNA mixtures has been the subject of concern in the DNA community, and the Scientific Working Group on DNA Analysis Methods (SWGDAM). It is also important to note that while some laboratories in North America still report qualitative results such as “cannot exclude” without quantitative measure, the 2010 SWGDAM guidelines state that “The laboratory must perform statistical analysis in support of any inclusion that is determined to be relevant in the context of a case, irrespective of the number of alleles detected and the quantitative value of the statistical analysis.” [28] 206 I.E. Dror, G. Hampikian / Science and Justice 51 (2011) 204–208 D3 AMEL D16 THO1 TPOX CSF1PO D7 Fig. 1. Sperm fraction electropherogram from victim's vaginal swab, after amplification with CoFiler (ABI). This electropherogram was given to analysts for interpretation. Genetic loci are indicated in boxes above alleles. These guidelines however are not binding, and are not required for The American Society of Crime Laboratory Directors Laboratory Accreditation Board (ASCLD/LAB) accreditation. Outside of North America, the International Society for Forensic Genetics (ISFG) DNA commission recommendations on the interpretation of mixtures strongly supports the use of likelihood ratios [16], and this approach is beginning to gain ground in North America. It is also important to note that while this is the first published empirical study of potential DNA bias, Butler of the NIST laboratories has conducted extensive studies of mixture analysis over several years, wherein he supplies a large number of volunteer laboratories identical DNA mixture data and asks for their analysis. The results of these excellent studies have been presented at conferences and are available at the NIST webpages [29], but have never been published in a peer-reviewed journal. An interesting and perhaps the most critical point for this paper is that Butler's research findings show that inclusion statistics for the same profiles (using the same data) varied over 10 logs, that is from 1 in 434,600 to 1.18 × 10 15, using the exact same electropherograms [29]. Therefore, although the use of statistics is paramount, it does not resolve the issue of subjectivity and potential bias, the topic of this study. 207 I.E. Dror, G. Hampikian / Science and Justice 51 (2011) 204–208 D3 AMEL VWA D8 FGA D21 D5 D18 D13 D7 Fig. 2. Sperm fraction electropherogram from victim's vaginal swab, after amplification with Profiler Plus (ABI). This electropherogram was given to analysts for interpretation. Genetic loci are indicated in boxes above allele. The work presented here is a step in addressing the subjectivity and potential for bias in DNA mixture interpretation. It is clear that additional and follow-up studies are called for. However, acknowledging the role of the human examiner, understanding the role (and weaknesses) of human cognition in making forensic comparisons (including DNA mixtures), is an important step in correctly conceptualizing forensic science and finding ways for improvements [30]. Appendix A. Supplementary data Supplementary data to this article can be found online at doi:10. 1016/j.scijus.2011.08.004. Additional information can be found at: www.cci-hq.com. References [1] R.S. Nickerson, Confirmation bias: a ubiquitous phenomenon in many guises, Review of General Psychology 2 (2) (1998) 175–220. [2] D.M. Risinger, M.J. Saks, W.C. Thompson, R. Rosenthal, The Daubert/Kumho implications of observer effects in forensic science: hidden problems of expectation and suggestion, California Law Review 90 (1) (2002) 1–56. [3] I.E. Dror, A. Péron, S. Hind, D. Charlton, When emotions get the better of us: the effect of contextual top-down processing on matching fingerprints, Applied Cognitive Psychology 19 (6) (2005) 799–809. [4] NAS, Strengthening forensic science in the United States: a path forward, National Academy of Sciences, Washington D.C, 2009. [5] I.E. Dror, D. Charlton, Why experts make errors, Journal of Forensic Identification 56 (4) (2006) 600–616. [6] I.E. Dror, D. Charlton, A.E. Péron, Contextual information renders experts vulnerable to make erroneous identifications, Forensic Science International 156 (1) (2006) 74–78. [7] I.E. Dror, R. Rosenthal, Meta-analytically quantifying the reliability and biasability of fingerprint experts' decision making, Journal of Forensic Sciences 53 (4) (2008) 900–903. [8] I.E. Dror, S. Cole, The vision in ‘blind’ justice: expert perception, judgment and visual cognition in forensic pattern recognition, Psychonomic Bulletin & Review 17 (2) (2010) 161–167. [9] M. Lynch, God's signature: DNA profiling, the new gold standard in forensic science, Endeavour 27 (2) (2003) 93–97. [10] W.C. Thompson, Subjective interpretation, laboratory error and the value of DNA evidence: three case studies, Genetica 96 (1995) 153–168. [11] W.C. Thompson, Painting the target around the matching profile: the Texas sharpshooter fallacy in forensic DNA interpretation, Law, Probability and Risk 8 (3) (2009) 257–276. [12] M. Dolan, J. Felch, The peril of DNA: it's not perfect: DNA: genes as evidence, LA Times, December 26 (2008) 2008. [13] B. Budowle, A.J. Onorato, T.F. Callaghan, A.D. Manna, A.M. Gross, R.A. Guerrieri, J.C. Luttman, D.L. McClure, Mixture interpretation: defining the relevant features for guidelines for the assessment of mixed dNA profiles in forensic casework, Journal of Forensic Sciences 54 (4) (2009) 810–821. [14] P.M. Schneider, R. Fimmers, W. Keil, G. Molsberger, D. Patzelt, W. Pflug, T. Rothamel, H. Schmitter, H. Schneider, B. Brinkmann, The German Stain Commission: recommendations for the interpretation of mixed stains, International Journal of Legal Medicine 123 (1) (2009) 1–5. [15] J.S. Buckleton, J.M. Curran, A discussion of the merits of random man not excluded and likelihood ratios, Forensic Science International: Genetics 2 (2008) 343–348. [16] P. Gill, C.H. Brenner, J.S. Buckleton, A. Carracedo, M. Krawczak, W.R. Mayr, N. Morling, M. Prinz, P.M. Schneider, B.S. Weir, DNA commission of the International Society of Forensic Genetics: recommendations on the interpretation of mixtures, Forensic Science International 160 (2006) 90–101. [17] I.W. Evett, P. Gill, J.A. Lambert, Taking account of peak areas when interpreting mixed DNA profiles, Journal of Forensic Sciences 43 (1) (1998) 62–69. 208 I.E. Dror, G. Hampikian / Science and Justice 51 (2011) 204–208 [18] T.M. Clayton, J.P. Whitaker, R. Sparkes, P. Gill, Analysis and interpretation of mixed forensic stains using DNA STR profiling, Forensic Science International 91 (1) (1998) 55–70. [19] P. Gill, R. Sparkes, R. Pinchin, T. Clayton, J. Whitaker, J. Buckleton, Interpreting simple STR mixtures using allele peak areas, Forensic Science International 91 (1998) 41–53. [20] B.S. Weir, C.M. Triggs, L. Starling, K.A.J. Stowell, J. Buckleton, Interpreting DNA mixtures, Journal of Forensic Sciences 42 (1997) 213–222. [21] I.W. Evett, C. Buffery, G. Willot, D.A. Stoney, A guide to interpreting single locus profiles of DNA mixtures in forensic cases, Journal of Forensic Sciences 31 (1991) 41–47. [22] R.G. Cowell, S.L. Lauritzen, J. Mortera, Probabilistic expert systems for handling artifacts in complex DNA mixtures, Forensic Science International: Genetics 5 (3) (2011) 202–209. [23] Perlin, M.W, Legler, M.M., Spencer, C.E., Smith, J.L. Allan, W.P., Belrose, J.L., & Duceman, B. W. (in press). Validating TrueAllele® DNA mixture interpretation. Journal of Forensic Sciences. (Also presented at the 2010 American Academy for Forensic Science 62nd [24] [25] [26] [27] [28] [29] [30] Annual Scientific Meeting “Casework validation of Genetic Calculator mixture interpretation”). D.E. Krane, S. Ford, J. Gilder, K. Inman, A. Jamieson, R. Koppl, I. Kornfield, D.M. Risinger, N. Rudin, M.S. Taylor, W.C. Thompson, Sequential unmasking: a means of minimizing observer effects in forensic DNA interpretation, Journal of Forensic Sciences 53 (4) (2008) 1006–1007. M.J. Saks, J.J. Koehler, Identification science the coming paradigm shift in forensic, Science 309 (2005) 892–895. J.M. Butler, Forensic DNA Typing, 2nd Edition, 2005 Elsevier Academic Press, 2005. N. Gilbert, DNA's identity crisis, Nature 464 (2010) 347–348. SWGDAM, 4.1 Interpretation guidelines for autosomal STR typing by forensic DNA testing laboratories(available at, http://www.fbi.gov/about-us/lab/codis/ swgdam-interpretation-guidelines. J. Butler, American Academy of Forensic Sciences annual meeting(available at, http://www.cstl.nist.gov/strbase/pub_pres/AAFS2006_mixtures.pdf 2006. I. E. Dror, Cognitive bias in forensic science, in McGraw-Hill 2012 Yearbook of Science & Technology, McGraw-Hill publishing, in press.