2006]the power and the glory1

Modeling Law Review Impact Factors as an Exponential Distribution

Jim Chen[*]

Of all the miracles available for inspection, none is more striking than the fact that the real world may be understood in terms of the real numbers, time and space and flesh and blood and dense primitive throbbings sustained somehow and brought to life by a network of secret mathematical nerves, the juxtaposition of the two, throbbings on the one hand, those numbers on the other, unsuspected and utterly surprising, almost as if some somber mechanical puppet proved capable of articulated animation by means of a distant sneeze or sigh.

David Berlinski, A Tour of the Calculus (1996)*[*]

Any sufficiently advanced technology is indistinguishable from magic.

Arthur C. Clarke, Profiles of the Future (1962)**[*]

I. A Bibliometric Manifesto

Truth, though elusive, sometimes allows itself to be manifested by mathematical means. Some things, in other words, can be measured and articulated through numbers. Peer-based evaluations of educational quality are among those things. This article proposes some steps toward quantifying the otherwise intangible sense of academic quality. Specifically, this article argues that reputational differences among law schools can and should be rigorously analyzed according to quantitative tools. Law review impact factors and citation rates are among the most salient and least manipulable sources of evidence bearing on the impact of law schools as research institutions. This bibliometric data should therefore figure prominently in any effort to gauge differences in prestige and influence within legal education.

This article proposes the unapologetic embrace of bibliometrics as a fundamental tool of academic assessment. No less than postmodern criticism,[1] bibliometric thinking should come naturally to legal academics. As social scientists who have nurtured “something like a third culture” between science and literature in order to improve the circumstances under which real “human beings are living,”[2] legal academics enjoy a special opportunity to unite the literary culture’s “canon of works and expressive techniques” with the scientific culture’s “guiding principles of quantitative thought and strict logic.”[3]

More than one century after Oliver Wendell Holmes declared that “the man of the future is the man of statistics and the master of economics,”[4] and nearly two decades after Richard Posner celebrated the decline of law as an autonomous discipline,[5] the legitimacy ─ perhaps even the primacy ─ of empiricism and quantitative analysis in law lies beyond dispute. This project demands (and secures) a considerable commitment to improving the data and tools available to quantitatively inclined scholars. The “absence of reliable data” lamentably circumscribes the ability of researchers to explain the workings of institutions such as the Supreme Court.[6] Having applied mathematical tools to their study of other institutions, legal scholars should now proceed without hesitation to train these instruments on themselves.

The mathematical instinct comes to us at play as well as at work. Within an country whose “legal culture” and perhaps even its very commitment to the “rule of law” arises from its “national past time” [sic],[7] baseball combines mathematical rigor with a respect for tradition.[8] Just as sabermetrics represents “the search for objective knowledge about baseball,”[9] bibliometrics represents the quest to quantify texts, information, and the academic pursuit of truth.[10] As with baseball and sabermetrics, the preeminence of mathematics transforms bibliometrics into a hopeful, uplifting enterprise. Baseball as “a ritual . . . of hope” traces its inspirational power to “the clarity of the sport ─ a kind of mathematical absoluteness that spills over into moral absoluteness.”[11]

Bibliometrics does differ from its sabermetric counterpart in one crucial respect. Whereas law aspires to define itself as the grand “enterprise of subjecting human conduct to the governance of rules,”[12] baseball affects nothing besides the happiness of devoted individuals who play or follow “a game with increasingly heightened anticipation of increasingly limited action.”[13] “Sports,” after all, “are entertainment.” They “do not often change our world; rather they serve as a distraction from our world.”[14] Statistical evaluation of baseball is fun precisely because it is frivolous. As Bill James, the founder of modern sabermetrics, has presciently observed, sabermetric “[a]nalysis is fascinating exactly because nothing is at stake ─ which allows a clearer view of certain issues being played out within a game.”[15] Though law takes itself seriously and legal education even more so, we can dedicate ourselves to quantitative rigor without forgetting to have fun. Failing to find the pleasure in serious academic work, even when it is focused on the academy itself, represents one of the most common ways in which a generally humorless legal professoriate misses the “play of intelligence.”[16]

The playfulness rightly associated with baseball and sabermetrics should infuse legal academics’ efforts to engage in the quantitative evaluation of their own profession. As much as academics rightly despise rankings, higher education demands comparative evaluations of institutions and of individual researchers. “[T]he only coin worth having” in academia remains the “applause” of our peers.[17] The dominant source of law school rankings is the U.S. News and World Report’s annual guide to graduate schools. Within a discipline marked by the absence of objective criteria and, in some quarters, by an ideological commitment to disagreement as a measure of personal virtue, dissatisfaction with the U.S. News rankings may be the lone point on which all members of the legal academy agree.

As though united by U.S. News as a common enemy,[18] law professors have risen to the challenge posed by the dismal project of evaluating themselves and their employers. For the better part of the last decade, leading scholars have devoted considerable effort to the project of quantifying the reputation and impact of American law schools.[19] In 2006 the Indiana Law Journal devoted an entire issue to dissecting the practice of law school rankings.[20] In a celebrated review essay on Michael Lewis’s best-selling report on the business of baseball,[21] Paul Caron and Rafael Gely suggested that law schools might do well to apply sabermetric lessons to their own managerial problems.[22] This article now proposes its own set of contributions to the burgeoning game of “Moneylaw.”

This paper endeavors to add one more tool to the burgeoning toolkit for improving the quantitative assessment of legal education in the United States. Since humans measure their happiness in relative rather than absolute terms, comparisons between individuals and institutions will necessarily emerge as part of the hierarchical structuring of all human relations. As Karl Marx observed in The German Ideology, human society begins with the production of means to satisfy the need for physical sustenance.[23] If cooperation to secure the production of means represents the first step toward civilization, however, conflict to control those means surely represents the second.[24] Insofar as the competitive treadmill on which all humanity walks condemns us to choose some measure of differences in educational quality and reputation, we should discharge this duty with some degree of competence.

“Law schools don’t have football teams,” proclaims a sign in the offices of the Harvard Law Review. “They have law reviews.” The exceptional institution of the student-edited law review ─ an anomaly, even an embarrassment, within an academic universe in which the professors of every other discipline get to edit the journals and the students are made to teach class ─ offers one way of gauging academic reputation without relying on surveys that rely almost entirely on popularity contests, readily (and routinely) falsified employment data for recent graduates, and affirmative inefficiency in law school spending. To the extent that hiring and promotion in legal academia still depends on article placement and citations within law reviews, measuring relative influence among law reviews offers one way of assessing the prestige of the law schools that publish these journals.

As the total information load in the world explodes,[25] “documentary chaos” threatens to overwhelm scholars, librarians, and others who attempt to swallow this seemingly unstoppable stream of information.[26] What information science now understands as “Bradford’s Law” predicts that “journal scatter,” or the concentration of influence within a nucleus of core journals, will emerge as the only way of bringing discipline to documentary chaos.[27] As a result, a “surprisingly small number of journals generate the majority of both what is cited and what is published.”[28] To the extent “that publication and citation patterns in the scientific literature are highly skewed,” a small core of journals will dominate each scientific discipline.[29]

To measure the influence of journals within their disciplines, most scientists rely on an admittedly flawed metric, the impact factor.[30] Eugene Garfield invented the impact factor in 1955 as part of an exercise to determine which journals should be documented for purposes of tracking scientific citations.[31] Despite frequent (and justifiable) criticism,[32] the impact factor remains the dominant statistic of bibliometrics. As computed by Garfield, the impact factor for a journal in year y represents all citations to that journal in years y ─ 1 and y ─ 2, divided by the total number of articles published in that journal in years y ─ 1 and y ─ 2.[33] Using impact factor rather than total citations equalizes the playing field between large and small journals by according more weight to a journal with fewer but more influential articles than a competitor that has built its reputation through the sheer number of articles it publishes.[34]

Legal scholars have already begun examining impact factors among law reviews as a measure of influence among these journals and the schools that publish them. John Doyle, director of the law library at Washington and Lee University, has compiled citation statistics for a wide range of English-language law journals, including total citations in law journals, total citations in judicial opinions, and journal impact factors.[35] Doyle’s impact factor calculation differs from Garfield’s in that the Doyle measure encompasses seven rather than two years of publication data. In every other respect it represents a straightforward application of the traditional impact factor to law journals. Ronen Perry of the University of Haifa has developed a measure of law school prestige based on the law review citation statistics reported in the Doyle database.[36]

This article evaluates the underlying mathematics of impact factors among law journals as reported by the Doyle database. It presumes that the reader has access to the Doyle database and can generate a ranked list of 901 journals that are (1) not exclusively online and (2) not “unranked.” I have discovered that law journal impact factors follow the sort of stretched exponential distribution that characterizes many “right-skewed” distributions found in the social and natural sciences. Indeed, a simple exponential distribution ─ that is, a stretched exponential distribution with an exponent of 1 ─ suffices to describe the probability density function of impact factors among law reviews. Mindful of physicist Hermann Weyl’s admonition that any necessary choice between truth and beauty should favor beauty,[37] I freely admit to sacrificing some marginal improvement in the descriptive accuracy of my model in order to develop the elegant mathematics of the simple exponential distribution. Further elaboration of this model of law review impact factors as an exponential distribution allows us to calculate the Gini coefficient of a stylized legal literature in which each journal’s influence is expressed by its impact factor. The striking result of this “inequality” computation is that the Gini coefficient of the legal literature modeled according to a simple exponential distribution is exactly 1/2, an outcome that is determined analytically rather than empirically. I conclude that modeling law review impact factors according to an exponential distribution gives rise to a powerful mathematical tool for assessing influence among law journals and law schools.

II. Modeling Right-Skewed Distributions: Power Laws, Stretched Exponentials, and Simple Exponential Distributions

Many statistical distributions revolve around a typical size or “scale,” evocative of the so-called “bell curve” that characterizes the standard Gaussian distribution. The histogram of speeds on a highway at any one time, for instance, would cluster near the average speed of cars on that highway. Many other histograms, however, would be significantly right-skewed in the sense that the distribution consists largely of fairly small items but also contains a small number of much larger items. One of the oldest of these right-skewed distributions is Zipf’s law, which describes the rank-frequency of words in natural languages.[38] Earthquakes,[39] cities,[40] meteorites,[41] personal incomes,[42] and for-profit businesses,[43] pages on the World Wide Web,[44] and legal precedents[45] have also been demonstrated to follow right-skewed distributions.[46]

Much of the current literature on right-skewed distributions treats these phenomena as expressions of power laws, relationships in which one quantity can be expressed as some power of another.[47] In practical terms, replotting a histogram by taking the natural logarithm of the horizontal and vertical axes can reveal a power law distribution at work. If the histogram, thus replotted, “follows quite closely a straight line,” this “remarkable pattern” provides the most vivid visual evidence of a power law.[48] In formal terms, the appearance of a straight line on a log-log plot suggests that ln p(x) = ─ α ln x + c, where α and c are constants.[49] Taking the exponential of both sides of this equation allows us to re-express the probability density function p(x) in simple terms:

p(x) = Cx─α

with C = ec.[50] Of the two constants in a typical power law relationship, the exponent α is more interesting, since the requirement that the cumulative distribution function equal 1 will dictate the value of c once α is computed.[51]

A brief methodological elaboration is warranted. The power law is so seductively elegant that it dominates the literature on complex adaptive systems. It suffers from the assumption that the fractal, “scale-free” properties of a power law distribution ─ as evidenced by the linear appearance of the log-log plot ─ continue infinitely. Because real-world phenomena occur within finite systems, analysts can scarcely afford to live within an “Asymptopia” where computer simulations, temperatures, gravity, and the like can be extended and observed infinitely.[52] The appearance of a curve in a log-log curve suggests that a slightly different model, that of the stretched exponential, is more appropriate.[53] The signature characteristic of a stretched exponential is that its histogram is a linear function of the natural logarithm of rank n:[54]

Ync = ─ a ln n + b

The stretched exponential therefore bears a strong resemblance to the power law. In order to convert a stretched exponential to a linear model, the independent variable (x-axis) alone is plotted on a logarithmic scale. Yn, the histogram, should then be raised to some power c, with c  1. The exponent c expresses the degree of curvature that can be seen in a log-log plot.[55] As c approaches 1, a stretched exponential model approaches an ordinary exponential distribution.[56] Many right-skewed distributions that are more appropriately modeled according to a stretched exponential may nevertheless be modeled with a power law for multiple orders of magnitude of Euler’s constant.[57]

This methodological debate is pertinent to bibliometrics, the science of measuring citations in academic literature. At least one pair of proponents of stretched exponentials favors this finite model over power laws for evaluating citation statistics among physicists.[58] Another study of citation patterns within the broader catalog of the Institute for Scientific Information yields a power law relationship.[59] My empirical evaluation of law review impact factors reveals that a simple exponential distribution ─ that is, a stretched exponential distribution with an exponent of 1 ─ comes so close to describing the observed data that any marginal improvement in descriptive accuracy attributable to the adoption of a more elaborate stretched exponential model is simply not worth the additional mathematical complexity.

III. Law Review Impact Factors Follow an Exponential Distribution

A. The Basic Relationship Between Impact Factor and Journal Rank

Law review impact factor among law journals, as measured by John Doyle’s database on the Washington & Lee University School of Law’s website, range from a high of 12.1 for the Yale Law Journal to 0.1 for the Temple Journal of Law, Science & Environmental Law and 118 other journals. In addition to 781 journals with measurably positive impact factors, another 120 journals report an impact factor rounded down to 0. Graph (1) depicts the basic histogram of observed impact factors as reported by the Doyle database:

1 Law journal impact factors as observed within the Doyle database

It is possible to project, with very satisfying accuracy, the impact factor of any given law journal as the product of the average impact factor in the survey (empirically determined to be 2.0) and the difference of the natural logarithm of the total number of “rankable” journals (a stylized number determined to be 782, representing the total number of journals with a positive impact factor, 781, plus 1) and the natural logarithm of the journal’s rank. The following histogram, as plotted in graph (2), depicts this model’s projected impact factor as a function of journal rank:

2 Projected law review impact factors

Combining these two historgrams reveals the closeness of fit between the projected and observed impact factors:

3 The combined histograms of observed and projected impact factors