You Are Probably Familiar with the Quote There Are Three Types of Lies - Lies, Damn Lies

April 26, 2006

STATISTICS AND THE LAW: SOME EXAMPLES AND THOUGHTS* by Dennis Gilliland, Co-Director, CSTAT

Statistics as the science of data plays a role in virtually every human activity including Law. We take this opportunity to give examples of the varied roles of statistics and probability in legal settings and to offer comments on expert testimony.

You are probably familiar with the quote "There are three types of lies - lies, damn lies, and statistics.” You may also be familiar with the declining prestige of the legal profession (The Harris Poll® #69, September 8, 2005). The union of statistics and law may be a marriage made in heaven or a playground for deviltry.

Some examples of statistics and probability in laws and regulations. Federal and state agencies operationalize the intent of laws through administrative action and rules. These can be far reaching and detailed descriptions of statistical methods for testing and monitoring of the environment, product quality, safety, or even truth in advertising. For example, the Michigan Public Service Commission requires a utility to employ ASTM sampling and test methodologies to monitor the accuracy of its electricity meters. The Michigan Department of Environmental Quality requires a paper mill to conduct specified tests of the effects of its effluent on water quality, including the effect on the taste of fish. Before voiding the result of a contested election, a court may require a probability, more than a possibility, of a reversed election result but for the irregularities.

The Michigan Lottery Act 432.9 in Section (1) states that “The commissioner shall initiate, establish and operate a state lottery…” In Section (2) the Act prohibits a lottery based on “skill.” Rules employ the word random, and it is the ambiguous meaning of this word that the state later used to defend itself following an error. The December 30, 1981 lottery to select five finalists for a $1,000,000 drawing from a set 11,561 semi-finalists is described in Gilliland, Experiences in Statistics (Kendall-Hunt, 1990, pp. 15 -17). The state executed a system that gave about sixty times more probability of selection to some semi-finalists than other semi-finalists. Later the commissioner of the lottery was quoted as saying “It is my belief that the drawing was fairly conducted and that each of the contestants had a fair chance of winning.” In Bad Or, the Dumbing of America (Random House Value Publishing, 1993, p. 194), Paul Fussell states that “…..law is about little more than language precisely construed and effectively deployed.” We might conclude in this lottery example that the lottery rules were neither precisely construed nor effectively deployed.

Role of statistics in expert testimony. Since testimony often involves data and information, it is not surprising that statistics and statistical inference show-up in court testimony. Statistical testimony and those that proffer it should be judged against the accepted practice in the professions (the Frye Rule of evidence resurrected) and the code of ethics of the American Statistical Association (see www.amstat.org/profession/ ). Under this code, an expert leaving out data that are unfavorable to his client would be in violation of Section C. Responsibilities in Publications and Testimony, Part 5. Account for all data considered in a study and explain the sample(s) actually used. (We expect that a future issue of the CSTAT NEWS, Center for Statistical Training and Consulting, Michigan State University, will feature an article on ethics in statistical analyses by Professor Francis Dane, Finkbeiner Endowed Chair of Ethics, Saginaw Valley State University. www.cstat.msu.edu )

Peter W. Huber in Galileo’s Revenge; Junk Science in the Courtroom (Basic Books, 1991, p. 17) bemoans the loss of Frye. Huber documents damage resulting from the let-it-all-in approach to expert testimony. (See Gilliland, The practice of statistics in legal proceedings, Proc. Social Statistics Section, ASA 2001 Annual Meeting, Atlanta, 1 – 6 for three specific examples of questionable testimony.)

The readers might be interested in an open letter dated 23 January 2002 from the Royal Statistical Society to the Lord Chancellor regarding the use of statistical evidence in court cases (www.therss.org.uk/archive/evidence/sc_letter.htlm). An unnamed expert had done one of those weird probability calculations reminiscent of the Collins case. The egregious testimony was let in though not tied to anything remotely sensible nor part of the science of statistics and probability.

Attorneys may not favor the notion of a neutral expert, one anchored in the sea of science and reviewed research. Paul Meier in Damned Liars and Expert Witnesses (J. Am. Statist. Assoc, 81, 1986, 269-276) gives us this quote: “John C. Shepherd of St. Louis, a distinguished trial lawyer, who was President of the American Bar Association in 1984-1985, spoke to a conference for lawyers on relations with the expert witnesses (Shepherd, 1973), and this is what he said: Many people are convinced that the expert who really persuades a jury is the independent, objective, nonarticulate type.... I disagree. I would go into a lawsuit with an objective, uncommitted, independent expert about as willingly as I would occupy a foxhole with a couple of non-combatant soldiers (pp. 21-22)."

Skepticism regarding the marriage of statistics and law will continue until the statistics profession accepts professional standards and publicizes expert testimony. Until then, we can expect some statisticians to blithely pick and choose procedures to please their sponsors or to get statistical significance to please an editor. We end with the quote of remarks attributed to Justice Justin J. Dewey by Arnold R. Brown in Lizzie Borden; The Legend, the Truth, the Final Chapter (Rutledge Hill Press, 1991, p. 277). “Expert testimony constitutes a class of evidence which the law requires you to subject to great scrutiny. It is a matter of frequent observation to see experts of good standing expressing conflicting and irreconcilable views upon questions arising in a trial. They sometimes manifest a strong bias or partisan spirit in favor of the party employing them. They often exhibit a disposition to put forth theories rather than to verify or establish or illustrate facts."