Reasoning about statistical evidence in law
Anne Hsu, Queen Mary University of London
Statistical reasoning plays an increasingly important role in courtroom evaluations. Probability theory prescribes correct, bounded solutions for statistical problems. However, most legal practitioners (jurors, judges, and lawyers) are not trained to maneuver even the most basic statistical formula. The consequent misuse of statistical evidence has resulted in frequent miscarriages of justice, both in the UK and worldwide. The mistakes made are common and systematic, and have been encapsulated in a handful of widely documented fallacies and other misunderstandings. However, even when law practitioners are aware of these fallacies, they still struggle to apply the correct statistical methods and the gross mishandling of statistics continues to rampage throughout legal practice. Furthermore, there is even widespread contention and confusion about what statistics and probabilities even mean, and what they are able to represent. This has led some people to wonder whether statistics should be allowed in the courtroom at all.
In this talk, I provide a general introduction to how statistical theory and probabilities can be used to represent a wide variety of uncertainty, including random processes and human beliefs, both of which are relevant to legal trials. I discuss how psychology researchers investigate questions about whether human judgments and decisions are capable of being ‘rational’ about uncertain scenarios. I review the literature on the types of statistical mistakes that people have been found to regularly make in the domain of legal evidence. I then discuss the avenues to which reasoning about legal statistics might be improved.
An example of statistics in the courtroom can be seen in a case of a random match probability. This is where it has been possible to determine the profile X of some forensic trace found at a crime scene, where a trace could be a type of print (such as a fingerprint, footprint, shoeprint, or even a CCTV image), or actual human tissue (blood, semen, hair etc), or fragment of their clothing. The trace found at the scene is called the source trace. In addition, we assume that there is a defendant from whom it is possible to get a similar type of trace, referred to as the target trace. The evidence E of a forensic match is the observation that the profile of the target is the same as the profile of the source trace, i.e. the same set of instantiated parameters X. In such situations we wish to know what this evidence tells us about the hypothesis H: “The source trace belongs to the defendant”. There is much unnecessary confusion about the meaning of a forensic match, and hence about what we can infer about H from E. The most common confusion concerns fingerprint and DNA ‘matches’ which are often wrongly assumed to be synonymous with ‘identification’, i.e. it is wrongly assumed that the ‘match’ implies that H must be true. In probabilistic terms the question we are asking is: How does the probability of H change after we observe the evidence E`? Formally, our belief about H before seeing the evidence (prior probability) is written P(H) and our belief about H after seeing the evidence (posterior probability) is written P(H | E). In practice, if we assume that the profile testing is always perfectly accurate and that the entire process is carried out without errors or malicious intent, then it turns out that all we actually need in principle to determine the impact of E on H are the following two probabilities (although note that these may be extremely difficult to obtain in practice):
- The probability of E given H (the ‘Prosecution likelihood’) written P(E | H): This is the probability that we would find the defendant’s trace profile matching the source profile if the source trace belongs to the defendant.
- The probability of E given not H (the ‘Defence likelihood’) written P(E | not H): This is the probability that we would find the defendant’s trace profile matching the source profile if the source trace does not belong to the defendant. The defence likelihood is also sometimes referred to as the ‘random match probability’.
Intuitively, the smaller the defence likelihood is relative to the prosecution likelihood, the greater the ‘probative value’ of the evidence in favour of the prosecution. Hence, a commonly used measure of the impact of evidence is the likelihood ratio, the prosecution likelihood divided by the defence likelihood. The (only) formal explanation for the probative value of the likelihood ratio relies on Bayes’ theorem, a central theorem of probability theory. Specifically, Bayes’ theorem provides the following formula: Posterior odds of H = Likelihood ratio × Prior odds of H.
In realistic cases, the correct translation of statistical evidence into courtroom judgments is a non-trivial task: Even within the simplest scenarios, statistical evidence can require calculations that readily incur erroneous reasoning in a non-statistician. Because of the complex nature of statistical evidence, the legal system rightly does not formally require jurors, judges, and lawyers to make complex statistical computations: there are designated experts who provide statistical values that are the results of complex calculations. Nevertheless, it is essential that jurors, judges, and lawyers know how to correctly make inferences and judgments based on these statistical values provided by experts. This is because the values provided by the experts are almost never direct probabilities of how much guilt or innocence is likely given the evidence, also known as posterior probabilities. (Indeed it has been previously argued that experts should never provide such posterior probabilities). Instead, the reported values usually pertain to the likelihood of observing a piece of evidence under various hypothesized scenarios (e.g., the random match probability of a piece of evidence), which need to be combined with other evidence and prior assumptions in order to be translated into a judgment of guilt vs. innocence. This is a non-trivial task. Thus, in an age of ever increasing availability of statistical evidence, jurors and lawyers cannot escape the need for requisite statistical knowledge.
The challenges facing application of statistical evidence in the courtroom are many. Much research has shown that human intuition performs poorly at even simple statistical reasoning tasks. These challenges are compounded by the fact that the statistics applied to legal evidence often reside in more complicated contexts, e.g., they may span a range of estimated values, enumerate the possibility of errors, and need to be evaluated alongside other evidence, which may be “traditionally non-statistical” (e.g., the probability of a false alibi). As expected, there have been many reports of erroneous reasoning about statistical legal evidence by both the lay-people who form the juries as well as legal professionals. Thus, understandably, the use of statistical evidence in the courtroom remains a contentious issue. The most important question remains unresolved: How can we best ensure appropriate application of statistical evidence in the courtroom?