A Rejoinder to Nicolaisen’s (2016) Refutation of Hjørland’s Relevance Definition
By Birger Hjørland
Royal School of Library and Information Science, University of Copenhagen,
DK-2300 Copenhagen S, Denmark. Email:
Abstract:Dr.Nicolaisen(2016) has claimed a refutation of the definition of relevance as provided by the present author. This present paper examinesNicolaisen’sarguments and finds that Nicolaisenhas failed to consider the differences between defining the concept of relevance and the measurement instances of it.His arguments are, therefore, misdirected and irrelevant. The epistemological problems as discussed by Nicolaisenare important, but not in relation to the task of defining the concept of relevance. Furthermore, his “refutation” resembles the well-known mythical “proof” that bumblebees cannot fly. As a result, it is concluded that the relevance definition under discussion is still valid and it is the most fruitful one that has been suggested so far.
- Introduction
Scholarly debates and critiques are necessary for the advancement of knowledge and should always be considered welcome. It is more important to contribute to progress and the elimination of failures than to protect the researchers from criticism. It cannot be overrated that scientific and scholarly progress is basedupon arguments in what Charles Sanders Peirce called a “community of inquirers” (Atkin, 2010).It should be clear from my bibliography that criticism and debate are genres that I give a high priority to in my writings. Therefore, I also consider it fruitful that Dr.Nicolaisen has chosen to examine and to discuss the definition of relevance, originally published in Hjørland andSejer Christensen (2002).To receive serious critical discussions of one’s work should be considered as a gift and they are reallyan important matter that appears much too seldom in our field. I will also add that the epistemological problems as addressed by Dr.Nicolaisenseem important (and few peoplehaveemphasised the importance of the theory of knowledge in relation to the problems of information science more thanI).
However, this paper will argue that Nicolaisen’s arguments againstmy relevance definitionare invalid andthat it is,therefore,not refuted. In addition, by not considering the aspects of the definition (e.g. its aim at moving the understanding of relevance, from the psychological, cognitive and individual understanding, to the sphere of social epistemology), Nicolaisen’s refusal may well support the less fruitful understandings that so far havebeen the dominating conception of relevance.
- The Definition Explained and Exemplified
The definition, which I have suggested, and which is the object of Nicolaisen’s criticism, is this:
"Something (A) is relevant to a task (T) if it increases the likelihood of accomplishing the goal (G), which is implied by T." (HjørlandSejer Christensen,2002, p. 964).
In Hjørland (2010, p.227), I used scurvy as an example. I will return to that, but first, I want to introduce two other exampleswith more adequate empirical data. The first example is a contemporary controversy in medicine concerning mammography. Is it relevant for women aged 50+ to have a regular screening? The way that this is decided in evidence-based medicine (EBM) is usually by considering randomly controlled clinical trials(RCCTs). I shall return to some of the methodological and epistemological problems below. So far, I will just say that according to medical science, the criterion for whether mammography is relevant or not is whether it reduces the likelihood or the probability that women receiving a mammography will die due to cancer (based upon survival statistics,e.g. within a five-year period after the diagnosis). This example fits very nicely with my definition: Something (here, mammography) is relevant to a task (the early detection of breast cancer) if it increases the likelihood of accomplishing the goal (here, preventing a death by cancer).
The other example concerns the former Minister and the Chief Burgomaster in Copenhagen, RittBjerregaard, who has been very open about her cancer illness. After an operation in her rectum, she was offered preventive chemotherapy, which she refuted with the following argument:
“Does that mean, that if I agree to this therapy, then I can be sure that the cancer will not return”, I asked the doctor. “No, it does not”,was the answer. “There is a 20% risk for a new cancer and if the preventive chemotherapy is followed correctly, that risk can be reduced by 5%.” I went home and I thought about that. I should have chemotherapy for half a year and the stomycould not be returned to normal until a month after the therapy was finished. I could imagine that I had to be a patient for the rest of 2015 and I would not agree to that. I would have some good times in the years that I have left.” (Madsen, 2016; translated by BH)
In this example, A is the chemotherapy, the task T is to cure the cancer, and G is the goal to prevent the spreading of the cancer. In this case, there are two different and conflicting views of the goal: G1: simply reducing the probability of the cancer recurring and G2: to obtain a reasonable quality of life.
We see that the probability of getting a new cancer was, according to Bjerregaard, assumed by the doctors to be reducedby 5% with the new chemotherapy. If the goal alonewas to survive, this would have been a relevant cure. But because the cure reduced her quality of life,Bjerregaard did not consider it relevant in her case. We see that if we accept the information, upon which Bjerregaard based her decisions, then the relevance formula fits this example very well. Ifthe goal can be clearly decided (e.g. if the goal is a survival at all costs), then the relationship between the goal and the therapy is a logical implication (and the relevance is logically deduced). If, on the other hand, the goal is unclear (how much should a quality of life count in relation to a reduced risk?), then the goal has to be clarified first (which is more of a psychological problem).
In these two examples,I do not consider how medical science has measured the reduced risk of dying from cancer (I will return to that, although I claim that it has no bearing on my definition). All I am saying is that such estimationsor measurements are made all of the time (in medicine and elsewhere) and that they are used in determining relevant actions. There are always methodological and epistemological problems that are associated with doing research, but although researchers may disagree whether mammography or preventive chemotherapyreduces the risk of dying, they nevertheless agree that this is the right criterion. The methodological and epistemological questions when determining probabilities does not invalidate my definition of relevance.
- Examining Dr.Nicolaisen’sArguments
3.1Nicolaisen’s arguments against my definition all fail because he has confused my definition of what “relevance” means with the methodological and epistemological problems in measuring and“interpreting” probabilities. Already in his introduction, Nicolaisenhas made the false statement that I have claimed to have solved the problems related to the measurement of relevance (without reference to where I should have said this).It is simply untrue that I have claimed to solve the problems concerning the measurement of relevance. I just claim to have provided a fruitful definition of relevance.
3.2Nicolaisen introduces two options: the so-called “logical theory” and the “subjective theory”. He has found that my definition fails, irrespective of which of these theories are chosen (although he does find that my relevance definition will be supported by the subjective theory, in cases of absolute consensus, which, however, is a very seldom phenomenon). Nicolaisenhas not considered more thanthese two options as being possible. My first comment is that probability estimations are common in much of science and that our understanding has greatly increased in the 20th Century. It is unclear how the two options presented by Nicolaisenare related to actual scientific methodology.
If we look at the mammography case, according to Goodman (2002),there is no agreement or consensus whether mammography reduces the risk of dying from breast-cancer. Two systematic reviews at that time have produced opposing conclusions:
- Olsen and Gøtzsche (2001) found “no reliable evidence that mammographic screening reduces the overall mortality.”
- Humphrey et al., (2002), on the other hand, found that “mammography reduced the breast cancer mortality rates among women of 40 to 74 years of age.”
The first thing to observe is that although the two studies are in a disagreement on whether mammography is effective or not, they nonetheless agree about what counts as a relevant action criterion, thus supporting my definition of relevance. Goodmandiscussed these opposing findings and wrote:
“But a closer look at this controversy […] shows that its focus hasshifted in a way that poses a dilemma, not only for womenand their doctors, but for evidence-basedmedicine itself.The debate in the 1990s was mainly about the advisabilityof screening for women younger than 50 years of age;for older women, the benefits of mammography werethought to be certain.”(Goodman, 2002).
I shall not review further studies about mammography here (Olsen and Gøtzsche have since modified their view, but this does not change the principles that are being discussed here). I willjust consider Goodman’s point of view, who considered the disagreement in the two systematic reviews about mammography. In the case of disagreements on “a factual level” (in the case of the effects of mammography) by scientists, according to Laudan (1984), they try to reach an agreement by considering “the methodological level” (in thiscase,of how the reduced risk is measured). (Disagreements on a methodological level may be caused by disagreements on “an axiological level”, but such disagreements about values cannot be solved by an appeal to a higher level). What Goodman (2002) said was that Olsen and Gøtzsche (2001) did not only bring forth a disagreement on a factual level, but also on amethodological level (problems related to the axiological level were considered in the Bjerregaard case above).
In medical science, there is an agreement about the meaning of the sentence “mammography is a relevant precaution”, and therefore, about the meaning of “relevance”, which correspondsto my definition of relevance. Both of the parties in the controversy agree that mammography is relevant if it reduces the probability of dyingthat is caused by breast cancer. What is important for information science is how we index and retrieve the information, so that the relevant studies may be distinguished from the non-relevant. Hjørland (2011) criticised EBM, as being “too narrow, too formalist, and too mechanical an approach, on which to base scientific and scholarly documentation”. This corresponds to Sadegh-Zadeh’s (2015, p. 386-389) criticism as well,when referring to Goodman’s (2002) view that studies cannot just be selected by formal criteria. Different views (e.g. Olsen and Gøtzsche, 2001 and Humphrey et al., 2002), used different subjective criteria about which studies to include in their overall evaluation. There are other ways by which scientific methodology still have “open” methodological problems in this respect (see Parascandola, 2004, 2011), but if medical science is able to calculate the probability that a certain step will decrease an illness, then that step is relevant in order to combat that disease. Such calculations are made all of the time and they are neither based on “the subjective theory”,norare they based on purely logical deductions (see the next section). There is,thus,a serious gab between Nicolaisen’s philosophical analyses and actual scientific methodology.
3.3The reference in Section 3 to “probabilistic logic” byNicolaisen (2016) is mostly about inductive logic. It is concluded that the probability of any universal statement [based on induction from a limited number of observations] is zero. It is not clear from Section 3how this is relevant to the discussion. The only possibility that I can see is that Nicolaisenhas argued about the way relevance is determined (for example, in clinical trials and in EBM). Actually, in the next section (4.1),Nicolaisen wrote:
“But without the full insight of Laplace’s Demon, how do we then warrant the assumption that ascorbic acid increases the probability of curing scurvy? Arguing that, we know from experience that many scurvy patients have been cured by this treatment, and thus, that this experience raises the probability of treating the next scurvy patient by the same treatment, commits the fallacy of making universal generalizations from a limited number of observations. Actually, if we reason like this, according to Hjørland’s relevance definition, ascorbic acid should be seen as a non-relevant treatment! Why? Because, as we have learned from Chalmers in the preceding section, that by dividing a finite number of observations with an infinite number, equals zero. According to Hjørland’s definition of relevance, something (A) is relevant to a task (T), if it increases the probability of accomplishing the goal (G), which is implied by T. It follows logically, that the probability of the treatment would be zero, the probability of treatment is not increased, and that the treatment is thus, not relevant.”
This quotationfrom Nicolaisenreminds me of the old story of those scientists who are supposed to have claimed that bumblebees cannot fly.
“Supposedly someone did a back of an envelope calculation, taking the weight of a bumblebee and its wings area into account, and worked out that if it only flies at a couple of metres per second, the wings wouldn’t produce enough lift to hold the bee up” (Institute of Physics, 2009 ).
But of course science is able to explain why bumblebees can fly and that Vitamin C has a high probability of being able to cure scurvy. The first trial to indicate Vitamin C’s effect on scurvy was Lind (1753), but this study did not fulfil today’s methodological standards. I have not identified an updated systematic review of a scurvy treatmentand I have not made much of an effort, as I do not consider that that is important for the argument.By principle, this can be donein the same way as for any other medical intervention and the probability of a cure can be assessed in the same way.I am not sure whether Nicolaisenhas intended to criticise the methodology of RCCTs,as to establishing the fallacy of making universal generalisations from a limited number of observations?This would indeed be an attack of the queen of medical methodologies and should rather be addressed to that particular community.
Although RCCTs and EBMsare not without problems (see Hjørland, 2011; Goodman, 1999a+b, 2002, Sadegh-Zadeh, 2015, p. 386-389),the induction problemas described by Nicolaisen is not one of them. The best evidence is mostly considered to be information from RCCTs:
“In order for the doctor and patient to determine the expected values of treatment alternatives and to make a therapeutic decision, they must know what the therapeutic efficacy of those treatments are. Probabilistic statements of the form “in a patient with acute appendicitis, the probability of a cure on the condition that she receives an appendectomy, is 0.98” are simple examples of the knowledge required. Therapeutic efficacy is tested in so-called randomized, controlled clinical trials, or RCCTs for short. An RCCT is a genuine, scientific experiment in the proper sense of this term. It is a well-designed investigation consisting of specified intervention in, and manipulation of, some condition to determine the effect of the intervention and manipulation. More specifically, it constitutes a systematic, prospective study of the efficacy of an intervention in human affairs designed to prevent, cure, or ameliorate a malady.” (Sadegh-Zadeh, 2015, p. 377; italics in original).
An example:
From Sadegh-Zadeh(2015, p. 380, Table 9)“This 2 × 2 contingency table demonstrates the results of an RCCT in 250 patients with peptic ulcer disease. Patients in the treatment group received antibiotics (metronidazole, amoxycillin, and clarithromycin), while patients in the control group didn’t receive any therapy”
Cured / Not-Cured / All
Treatment A / 230 / 20 / 250
No treatment / 30 / 220 / 250
(When a placebo was the alternative to no treatment, the following results occurred)
Treatment B (placebo) / 80 / 170 / 250Sadegh-Zadeh(2015, p. 382, Table 10)
The relationship to the probability/likelihood was establishedin this way:
“According to this table [9], the proportion of the cured in the treatment group is230:250 = 0.92 and in the control group itis230:250=0.12. If we consider these numbers as estimates of probabilities in the long run, we obtain the following conditional probabilities, where X is the population of patients with peptic ulcer disease; A is the application of the therapeuticum A; and C means ‘cured’:
P(C|X∩ A) = 0.92
P(C|X) = 0.12
Obviously, it is more likely for a patient to be cured by treatment A thanwithout it.” (Sadegh-Zadeh, 2015, p. 380-381)
Thus, medical science does not estimate probabilities by making universal claims from a pool of observations, but by randomly and double-blindly dividing the patients into experimental and control groups, respectively, treating only the experimental groups, and then for example,comparing the statistical patterns for a 5-year survival in the two groups.
There are some specific problems with scurvy. In Hjørland(2010), scurvy was chosen as an example in order to convince those readers who belong to the dominating cognitive school of relevance research. I considered this a case of “established knowledge”, and thus,it was easy to distinguish from the relevance that was based on examining the user’s individual beliefs. Now, facing arguments based on a probabilistic philosophy, I have introduced the mammography and Bjerregaard cases, because in these examples, empirical findings and the methodological discussions from medical research are available. Another problem with the scurvy disease is that the blood level of vitamin Cis sometimes used in the diagnosis. There may, therefore, be a circular relationship between the definition of the disease and its cure (In the case of full circularity,anincrement of a patient’s level of Vitamin C in the blood,will by definition, cure the patient.The probability is then 1, not zero). A third problem is that the mechanisms behind scurvy are probably well understood today.(Today, the illnessis seldom in developed countries,except when people suffer from malnutrition. Perhaps,because of these two reasons,clinical trials are considered relatively unnecessary?). There is a difference between basing knowledge on clinical trials and basing it on research into fundamental mechanisms. This last difference implies a new challenge. How do we evaluate the probabilities of our actions when they are based on a deep understanding about underlying mechanismsthat are based upon many different experiencesthat have been cumulated over long periods of time?How do we evaluate “established knowledge”? Perhaps,the question is not relevant, because we act on our knowledge, until we have reasons to doubt it?(Medical science has considered mammography relevant, until questioned by Gøtzscheand others, i.e. a case related toPopperian’sfalsificationism).