Statistics in Court - The Ultimate Communication Challenge

ICOTS-7, 2006: Chaseling STATISTICS IN COURT: THE ULTIMATE COMMUNICATION CHALLENGE Janet Chaseling Griffith University, Australia j.chaseling@griffith.edu.au Statistical teachers and consultants must target their explanations to meet the level of their audience. When statistics are placed within a legal court setting, three audiences must be addressed; the barrister, the judge and the jury. In the adversarial system the statistical expert will present evidence in chief and then be subjected to cross examination. In a pre-trial (voir dire) a judge will determine whether or not he will allow evidence to be presented before the jury; he/she may consider it to be too confusing for his/her jury or he may be concerned at the prejudicial weight of small probabilities. Drawing on real experiences, suggestions are made of how teaching and communication in a court setting can best be implemented. Reference is also made to the different models of deliberation practised by jurors. INTRODUCTION Teaching statistics involves communication. Good statistical teachers and consultants target their explanations to meet the level of their audience. In everyday experiences, communicating statistical content occurs at different levels: to peers in professional situations (journals, seminars); to clients who need advice on the application of statistical methods to assist in the interpretation of their research; to students who may or may not need statistics but who need to pass the course; and to the public who need statistical awareness as an aid in decision making. A rather different situation arises when statistics are placed within a legal court setting. In a court, the work of the statistician is judged not by peers or by students seeking to learn, but by legal experts and a jury of laypersons. In the adversarial legal system as practised in Australia, the UK and the USA, a statistical expert may be called upon to present an interpretation of statistical issues pertaining to a range of evidential types including DNA, drugs, chemicals, discrimination and standards violation. Cases may involve murder, rape, aggravated assault, break and enter, compensation, environmental hazard or fraud. Within the adversarial system the statistical expert will present evidence in chief led by their own barrister, and then be subjected to cross examination by an opposing barrister. These deliveries of evidence will take place before a judge and usually a jury. The statistical issue involved commonly relates to some form of inference arising from comparison with a database, and an associated profile match; an interval estimate of a probability is often required with corresponding explanation of why a definitive answer is not possible. In some cases, evidence is given in a pre-trial (voir dire) in which a judge will determine whether or not he will allow the evidence at issue to be presented before the jury. A judge may disallow evidence if he/she considers it to be too confusing for his/her jury, or if he/she feels that the small (or large) numbers and probabilities have too great a prejudicial weight. THE AUDIENCES Three main audiences must be addressed in court statistical communication; at different stages of the process the statistician must target their communication to the barrister, the judge and the jury. In addition, some form of communication will usually be needed with the relevant forensic expert; this will take place prior to any court proceedings but is vital for the consistency of evidence and to avoid confusing a jury. Most forensic experts recognise their limitations with respect to statistical issues, and will state that the more technical statistical areas are beyond their level of expertise. None-the-less, it is inevitable that some component of statistical interpretation will be included in their evidence. If any quantitative aspect of this information is not identical with that presented by the statistical expert, there will be confusion. Many forensic experts will have attended statistical courses specifically designed to provide the basic statistical concepts for the relevant area of forensic science. Such courses will also attempt to help the forensic expert 1 ICOTS-7, 2006: Chaseling identify their limitations on the likely statistical issues. It is important that issues relating to the recognition of knowledge limitations be included in the curriculum of these specialised courses. Nothing can be assumed about a judge’s attitude to, and competence in, the statistical aspects of the evidence. In Australian courts I have encountered judges who understood and valued the statistical component and were very keen to facilitate understanding of the statistical evidence for the jury. The published judgement by the judge in the case of R v Jarrett (1994) is an excellent set of statistical teaching notes. Other judges have been concerned about the statistics and have been reluctant to allow it before their jury for fear of causing confusion. Some judges clearly have little understanding of basic probability as is seen in the quote from the case of R v Abbott (1996) as given in Chaseling (2000, p. 112): … the expert witness quoted … one in 23 billion. After a brief guffaw, the judge said to the expert: ‘That’s more than the population of the world, it’s meaningless.’ In the case of R v Mitchell (1997) the judge was extremely hostile to the statistical expert, refusing to allow any form of visual aids and attacking the use of the ‘Law of Large Numbers’ with the statement that a statistician has no right to quote laws, that is the prerogative of the legal fraternity. Mann (2000) discusses the positive aspects of working with legal issues. In particular he emphasises that, in general, barristers are intelligent with good skills in logical analysis. This can be a two-edged sword in that the barrister is also required to prepare the case that is best for their client, regardless of what the statistical expert may wish to present. The statistical expert must anticipate the direction in which their evidence may be led by a skilled barrister, and be prepared to alter the way they present their argument in midstream whilst maintaining clarity and integrity. Members of the jury will, in general, have only the knowledge of statistics from basic schooling and from everyday experience. Unfortunately, the general public usually has a poor attitude to statistics. As reported in Chaseling (2000) the jury may well consist of people who suffer from ‘mathphobia.’ A statistical expert will often hear mutterings as they pass the jury on the way to the witness box, with comments such as: ‘you can prove anything with statistics,’ ‘I hated maths at school,’ and ‘statistics is boring.’ Thus, the statistical expert must overcome these negative attitudes before they even begin to present their evidence in an understandable manner. The jury is the class, and the statistical expert the teacher who must ‘read’ that class’ reponse, and ensure understanding. Unlike a ‘normal’ class in which a student is experiencing a continuum of learning, the jury will encounter the statistical material as a one-off event. And, unlike a normal class where a misunderstanding will perhaps result in some temporary confusion or, at worst, lead to a lower grade, it is vitally important that the jury fully understand all the statistical material in this one-off encounter. JURY DECISION MAKING MODELS Teachers recognise that students learn in different ways and at different speeds; using a variety of teaching methods is a means of overcoming this problem. The models used by jurors to assimilate evidence and reach a decision vary considerably and have been studied by numerous researchers. The various contributing authors in Hastie (1994) discuss four distinctive models of deliberation practised by jurors. The first of these models is a formal, Bayesian type approach to evidence, in which successive pieces of information are combined through a multiplicative approach such as occurs in a formal Bayesian analysis. A second model is equated to a multiple linear regression where the different pieces of evidence add to the overall decision. The third is an interesting and very different model which is described as a ‘stochastic process model’ in which: … a Poisson random process is defined to describe the distribution of “stopping points” in time at which a juror’s thought process terminates to yield a final strength of belief about guilt or innocence.(Hastie, 1994, p. 20) For a juror using this model of decision making, a single piece of evidence is of such strength that it determines the final decision, and no further evidence will change that decision. In the case of DNA profiling, for example, the very small probabilities associated with a multi-locus profile can represent extremely strongly weighted evidence. This in some ways contributes to the judges’ concerns in determining whether or not to allow the statistical evidence before their jury. 2 ICOTS-7, 2006: Chaseling The final model described by Hastie (1994) is that of the cognitive approach in which a juror imposes a narrative structure onto the evidence in the form of a story. This process has been shown to have limits on the amount of information that can be absorbed simultaneously, suggesting that it is difficult for a juror to assimilate material from a barrister and a witness simultaneously. Clearly it cannot be assumed that all jurors use the evidence in the same way, nor do they necessarily consider the evidence in a linear one-dimensional way. In the face of these very different learning models, the statistical expert needs to consider how best to present the statistical content in the single opportunity available. It is unlikely that there will be the option of a variety of teaching media; there may be no visual aids. COMMUNICATION ISSUES Numerous authors including Aitken (1995), Evett and Weir (1998) and Balding (2005), have written text books to assist forensic scientists in the interpretation of statistical issues in the court, particularly in the area of DNA evidence. However, little attention is given by these authors to the issue of how best to present the material for the jury. What is the best way to explain basic statistical issues to a lay jury; how much theory (if any) can be included? How and what visual aids should be used? What approach should be taken if no visual aids are allowed? What words are likely to be misinterpreted by a jury? Chaseling et al. (2000) report research in which 900 people were asked to say on a percentage basis from 50% to 99%, what they understood by the phrase ‘most of the time.’ Twelve percent of people nominated 50%, 28% nominated 75% with 19% and 12% responding with 95% and 99%, respectively. Clearly, such a simple word as ‘most’ has the potential for considerable confusion. Of great importance is knowledge of analogies which are likely to convey the statistical concepts in a way the jury will understand. Many, if not all, of the relevant statistical issues are imbedded in basic logic and are reflected in real life scenarios. The good statistical expert witness needs a suite of suitable analogies on which to draw when required. Chaseling (2000) gives several real life analogies which have proved valuable in Australian courts. Mann (2000) provides an excellent review of the difficulties faced by the statistical expert witness, but he stops short of any specific advice on how best to deal with communication issues. A number of issues important for the statistical expert witness are also discussed in Aitken (2000). The way in which probabilities are reported is another area of concern and confusion. Forensic scientists have traditionally used the format ‘one in so many’ rather than a classical expression of probability. This has led to problems when the concept of probability is not understood. When is a probability effectively zero? This argument has been raised on many occasions and some experts feel that there must come a time when DNA profiles are given equal status with dermal fingerprints and assigned a rating of uniqueness. In their research, Chaseling et al. (2000) found that 55% of the 900 people they surveyed felt that something was unique if the chance of there being another identical item was one in one billion. Another 37% felt that uniqueness was achieved when the chance was one in ten billion. The probabilities associated with multi-locus DNA profiles in Australia are now often in the order of one in one quintillion (R v Murdoch, 2005), yet still courts are requiring this number rather than accepting the equivalent of a dermal fingerprint match. THE GAME Under the adversarial legal system, the process in the court is an orchestrated performance conducted by the barristers for the benefit of judge and jury. The statistical expert witness is one of the cast who has a role to play as determined by the director. Unlike a lecture or seminar to peers, the statistical expert is unlikely to be able to complete a reasoned, prepared argument during the presentation of their evidence. Explanations will inevitably be ‘cut short’ as soon as the barrister hears the point he wishes made. The system does not allow the expert statistician to add further explanation if they feel the information they have conveyed has been misunderstood. During a lecture or seminar, a good communicator reads their audience and if necessary can clarify content when it is obvious that the class has become ‘lost.’ Attempts will be made to ‘rattle’ the expert, to push them into making ill reasoned 3 ICOTS-7, 2006: Chaseling comments under pressure. All statements made from the witness box are recorded and are available for use by opposing counsel in the current trial and for any future trial. Retraction and/or clarification are not options. The statistical expert must also accept the decision of a barrister as to what evidence will be presented; a particular approach may seem imperative to the statistician for clear understanding, however, if the barrister does not see it this way then it is the statistical expert who must change. The timing of the statistical evidence is also controlled by the barrister. The statistical expert may also be asked to assist in the cross examination of another statistical expert, not at the coal face in front of the witness box, but as advisor from behind the barrister’s bench. This can lead to an adversarial situation between two statisticians who can communicate only through the barrister who is a third party. Such interchanges are witnessed and absorbed by the remaining court audiences, the judge and jury; these potentially heated and technical communications will form part of the deliberation process of the lay jurors. How can such a communication process be best managed? Each statistical expert has an obligation to provide complete and accurate information which may require disagreement with an opposing expert. Is such disagreement to be left unaired? How does an expert provide sufficient guidance to a barrister to ensure that not only are the pertinent questions asked, but that when the response is given, the barrister understands it and can determine its value to his case? I recall several cases when I have been asked questions which were clearly written in haste and which I could not understand. After I stated several times that I did not understand the question, the barrister admitted that he also did not understand it and the question was withdrawn. In another case, the Judge was able to assist the barrister by rephrasing the question. The uncertainty associated with all statistical conclusions and the commonly seen fear of statistics within the public, make the statistical component of any evidence ‘fair game’ for an attack by the opposing party. Opposing experts do not even need to carry out any analyses; all they need do is invoke some level of confusion or doubt. A barrister will try all ways possible to ‘muddy the waters’ of the statistical evidence. If the evidence cannot be shaken then can the expert be shown to be unreliable? Are the expert’s qualifications sound and relevant? What statements has the expert made in previous cases under cross examination that may conflict with the well prepared evidence in chief of the current case? THE STATISTICAL AND NON-STATISTICAL ISSUES The statistical issues are often not theoretically complex, involving basic concepts of inference including confidence levels, variation and errors of Type I, II and III. However, without the ability to give clear explanation and overcome the underlying fear and doubt in statistics, the expert will be vulnerable in cross examination; their evidence may not even be allowed before the jury. A number of commonly cited issues are regularly and incorrectly attributed to ‘statistical mystique.’ As stated by Chaseling (2000, p. 115): …the errors referred to as the prosecutor’s and the defender’s fallacies, in which valid statements are incorrectly reversed, have no more basis in statistics than they do with, for example, English grammar. An expert witness can only report outcomes concerning the evidence; these may be conditioned on the guilt or innocence of the suspect allowing a likelihood ratio to be constructed relating to the evidence. A prosecution barrister will attempt to change such statements into outcomes about the guilt, conditioned on the evidence; a flawed argument of the form of a transposed conditional or confirmation of the consequence, which lies not in statistical argument but rather in formal logic. Argument frequently arises as to which population and corresponding database is appropriate for the case at issue. For example, should a white Caucasian database be used to estimate the DNA profile probability if a rape has occurred in a small aboriginal community in outback Australia when the victim did not see their attacker? Does this decision change if the accused is an aboriginal? What if the rape occurred during a large festival frequented by many tourists? Can an Australian database of mixed races be used when a witness claims the murderer 4 ICOTS-7, 2006: Chaseling was of Middle Eastern descent? The decision as to which population is relevant is not a statistical issue. It depends on the other evidence available and the possible scenarios that the barrister wishes to use. None-the-less, this issue is almost always seen as part of the statistical portfolio, and raised with the statistical expert whose response must be that the database used is the result of discussion with the barrister and/or direction from the court. Balding (2000) discusses the issue of which population. Another argument which arises when probabilities are quoted for multivariate data such as is seen in various forms of profile evidence (for example, DNA, drugs, oil), is that of statistical independence versus independence in the underlying process (biological, physical or chemical). It is difficult to explain that statistical independence is a feature of the data and that it is through formal statistical tests on data that researchers are able to validate proposed models of independence in the underlying process. For example, in DNA profiling the small probabilities for a multi-locus profile are derived by multiplication of probabilities obtained for the two alleles at each of the individual loci. Clearly this requires independence between the alleles at each locus and between the loci. Objections are frequently raised through reference to various genetic ‘laws,’ with statements that you cannot use the multiplication rule because the loci are not in Hardy Weinberg equilibrium and there is linkage disequilibrium. Genetics researchers in areas other than forensic science acknowledge the role played by statistics in testing for Hardy Weinberg and other genetic dependencies. However, in forensic science this seems unrecognised and no amount of explanation in Australian courts seems to be able to change this. In the case of R v Jarrett (1994), an expert witness stated that the assumption of independence needed for the multiplication of individual allele probabilities to obtain the combined probability for the multilocus profile, was a genetics issue and nothing to do with statistics. CONCLUDING COMMENTS Communicating statistical evidence to a court is difficult and requires different skills from those used by a statistician in teaching and communicating in everyday activities. More research is needed to identify useful anecdotes to aid in the explanation of statistical concepts to a jury of lay people who often have preconceived worries and fears about statistics in general, and who may be using different models of decision making to reach their verdict. The statistical community needs to find answers to the question of why statistical evidence is treated so differently. Other technical evidence such as extraction of DNA or chemical profiling of oil samples, does not receive the same adverse attention. Why is the attitude to statistical evidence different? Is it solely a product of the often negative attitudes the public has to statistics (as witnessed by any statistician who reveals their occupation at a dinner party)? Or is the legal fraternity really afraid of the weight of evidence it provides? How much of the problem relates to issues of probability and how much is caused by the grey concepts of inference? Maybe the statistical expert witness should make use of the strategies adopted by the great salesman Casanova who in the 18th century was able to sell the idea of a state lottery to the sceptical Finance Ministers in the Italian Government (Stigler, 2005)! REFERENCES Aitken, G. C. (1995). Statistics and the Evaluation of Evidence for Forensic Scientists. Chichester: John Wiley and Sons. Aitken, G. c. (2000). Interpretation of evidence and sample size determination. In J. L. Gastwirth (Ed.), Statistical Science in the Courtroom, New York: Springer-Verlag. Balding, D. J. (2000). Interpreting DNA evidence: can probability theory help? In J. L. Gastwirth (Ed.), Statistical Science in the Courtroom, New York: Springer-Verlag. Balding, D. J. (2005). Weight-of-evidence for Forensic DNA Profiles. Chichester: John Wiley. Chaseling, J. (2000). DNA statistics under trial in the Australian adversarial system. In J. L. Gastwirth (Ed.), Statistical Science in the Courtroom, New York: Springer-Verlag. 5 ICOTS-7, 2006: Chaseling Chaseling, J. N. Nic Daeid and A. Linacre (2000). DNA Profiles, Statistics and the Meaning of Uniqueness. CRIMTRAC 15th International Symposium on Forensic Science, Gold Coast, Queensland, Australia. Evett, I. W. and Weir, B. S. (1998). Interpreting DNA Evidence. Sunderland, MA: Sinauer. Hastie, R. (1994). Introduction. In R. Hastie (Ed.), Inside the Juror: The Psychology of Juror Decision Making. Cambridge: Cambridge University Press. Mann, C. R. (2000). Statistical consulting in the legal environment. In J. L. Gastwirth (Ed.), Statistical Science in the Courtroom, New York: Springer-Verlag. R v Abbott (1996). Supreme Court of Queensland, Brisbane, Australia. R v Jarrett (1994). 62SASR 443, Supreme Court of South Australia, Adelaide, Australia. R v Mitchell (1997). ACTR 48, Federal Supreme Court, Australian Capital Territory, Canberra, Australia. R v Murdoch (2005). Darwin Supreme Court, Northern Territory, Australia. Stigler, S. (2005). Casanova not just a lover. Interviewed on ABC Radio PM, Australia, 8th June. http://www.abc.net.au/pm/content/2005/s1387912.htm. 6