What Is Information?

WHAT IS INFORMATION?

Index

Information according to Cybernetics

Information according to Shannon

The Contradiction between Shannon and Wiener

Meaning in a Cybernetic Sense

Meaning in Shannon`s Sense

Information or Knowledge or Life?

Information via Massmedia

Information Complexity and Overload

Information has No Value and No Meaning

Information in Etymology

The word information is derived from Latin informare which means "give form to". The etymology thus connotes an imposition of structure upon some indeterminate mass. Allén & Selander (1985) have analysed how the word is used in Swedish language and find that this is probably the most widely used meaning of the word. Most people tend to think of information as disjointed little bundles of "facts". In the Oxford definition of the word it is connected both to knowledge and communication.

Knowledge communicated concerning some particular fact, subject or event; that of which one is apprised or told; intelligence, news.

The way the word information is used can refer to both "facts" in themselves and the transmission of the facts.

Information according to Cybernetics

The double notions of information as both facts and communication are also inherent in one of the foundations of information theory: cybernetics introduced by Norbert Wiener (1948). The cybernetic theory was derived from the new findings in the 1930s and 1940s regarding the role of bioelectric signals in biological systems, including the human being. The full title was: Cybernetics or Control and Communication in the Animal and the Machine. Cybernetics was thus attached to biology from the beginning.

Wiener introduces the concepts, amount of information, entropy, feedback and background noise as essential characteristics of how the human brain functions.

From Wiener (1948) p. 18:

The notion of the amount of information attaches itself very naturally to a classical notion in statistical mechanics: that of entropy. Just as the amount of information in a system is a measure of its degree of organisation, so the entropy of a system is a measure of its degree of disorganisation.

Wiener coins the label of a whole new science:

We have decided do call the entire field of control and communication theory, whether in machine of animal by the name Cybernetics, which we form from the Greek steersman.

And declares his philosophical heritage:

If I were to choose a patron for cybernetics... I should have to choose Leibnitz.

What is information and how is it measured? Wiener defines it as a probability:

One of the simplest, most unitary forms of information is the recording of choice between two equally probable simple alternatives, one or the other is bound to happen - a choice, for example, between heads and tails in the tossing of a coin. We shall call a single choice of this sort a decision. If we then ask for the amount of information in the perfectly precise measurement of a quantity known to lie between A and B, which may with uniform a priori probability lie anywhere in this range, we shall see that if we put A = 0 and B = 1, and represent the quantity in the binary scale (0 or 1), then the number of choices made and the consequent amount of information is infinite.

Wiener describes the amount of information mathematically as an integral, i.e. an area of probability measurements (p.76). Wiener says the formula means:

The quantity that we here define as amount of information is the negative of the quantity usually defined as entropy in similar situations. (My bold)

Wiener`s view of information is thus that it contains a structure that has a meaning.

It will be seen that the processes which lose information are, as we should expect, closely analogous to the processes which gain entropy.

Information is from its conception attached to issues of decisions, communication and control, by Wiener. System theorists build further on this concept and see information as something that is used by a mechanism or organism, a system which is seen as a "black box", for steering the system towards a predefined goal. The goal is compared with the actual performance and signals are sent back to the sender if the performance deviates from the norm. This concept of negative feedback has proven to be a powerful tool in most control mechanisms, relays etc.

Information according to Shannon

The other scientist connected with information theory is Claude Shannon. He was a contemporary of Wiener and as an AT&T mathematician he was primarily interested in the limitations of a channel in transferring signals and the cost of information transfer via a telephone line. He developed a mathematical theory for such communication in The Mathematical Theory of Communication, (Shannon & Weaver 1959). Shannon defines information as a purely quantitative measure of communicative exchanges.

Weaver (in Shannon & Weaver 1959), links Shannon`s mathematical theory to the second law of thermodynamics and states that it is the entropy of the underlying stochastic process in the information source that determines the rate of information generation (p.103):

The quantity which uniquely meets the natural requirements that one sets up for "information" turns out to be exactly that which is known in thermodynamics as entropy.

Shannon defines the amount of information as the negative of the logarithm of a sum of probabilities. The minus sign in this formula means the opposite of Wiener`s minus sign. It is there because the amount of information according to Shannon is equal to entropy.

For an information theorist based on Shannon it does not matter whether we are communicating a fact, a judgement or just nonsense. Everything we transmit over a telephone line is "information". The message "I feel fine" is information, but "ff eeI efni" is an equal amount of information.

Shannon is said to have been unhappy with the word "information" in his theory. He was advised to use the word "entropy" instead, but entropy was a concept too difficult to communicate so he remained with the word. Since his theory concerns only transmission of signals, Langefors (1968) suggested that a better term for Shannon´s information theory would therefore perhaps be "signal transmission theory".

But Shannon`s "information" is not even a signal (p.100):

If one is confronted with a very elementary situation where he has to choose on of two alternative messages, then it is arbitrarily said that the information, associated with this situation, is unity. Note that it is misleading (although often convenient) to say that one or the other message conveys unit information. The concept of information applies not to the individual messages (as the concept of meaning would), but rather to the situation as a whole, the unit information indicating that in this situation one has a freedom of choice, in selecting a message, which it is convenient to regard as a standard or unit amount.

The contradiction

Weaver, explaining Shannon`s theory in the same book:

Information is a measure of one´s freedom of choice in selecting a message. The greater this freedom of choice, the greater the information, the greater is the uncertainty that the message actually selected is some particular one. Greater freedom of choice, greater uncertainty greater information go hand in hand.

There is thus one large - and confusing - difference between Shannon and Wiener. Whereas Wiener sees information as negative entropy, i.e. a "structured piece of the world", Shannon`s information is the same as (positive) entropy. This makes Shannon`s "information" the opposite of Wiener`s "information".

How can something be interpreted as both positive entropy and negative entropy at the same time? The confusion is unfortunately fuelled by other authors. The systems theorist James G. Miller writes in Living Systems (p.13): It was noted by Wiener and by Shannon that the statistical measure for the negative of entropy is the same as that for information.

Miller also quotes (p. 43) Shannon`s formula but omits Shannon`s minus sign. Since entropy is defined by Shannon as a negative amount, a positive amount should be the same as negative entropy, i.e structure. It seems that Miller makes a misinterpretation of Shannon.

Meaning and the Observer

Meaning in a Cybernetic Sense

There are many meanings about what meaning2 is. I try to approach an understanding by distinguishing between living systems or natural objects versus manmade.

James G. Miller defines in his work Living Systems (p.39) goals for living systems like this:

By the information input of its charter or genetic input, or by changes in behaviour brought about by rewards and punishments from its suprasystem, a system develops a preferential hierarchy of values that gives rise to decision rules which determine its preference for one internal steady-state value rather than another. This is its purpose. A system may also have an external goal. It is not difficult to distinguish purposes from goals. I use the terms: an amoeba has the purpose of maintaining adequate energy levels, and therefore it has the goal of ingesting (= swallow) a bacterium.

As I interpret the cybernetic view, the signals in a system thus contain "information" - which have some meaning for the purpose of the particular system. Someone or a system outside the system may define a goal, but the meaning of the information that is sent/received within the system does not necessarily have a meaning outside the system. The information in the feedback loop has a meaning only in relation to the purpose. The goal of the subsystem is determined by a system on a higher level. The brain itself can be seen as such a suprasystem as long as it performs functions of temperature regulator etc.

The signals controlling the muscle thus have no meaning outside the system of the muscle although the goal of the muscle is determined by a suprasystem like the brain. The only thing that the suprasystem "cares about" is whether the muscle fulfils its purpose or not.

Suppose I interfere with some apparatus and with the intention to interpret the signals from one of my own muscles while it is fulfilling its purpose according to the goal of my suprasystem, the brain. I am able to impose several meanings on the signals caught by the apparatus but those meanings are outside both the system and the suprasystem. I would assume that it is the same with a non living system man made system like the computer. The signals controlling the computer programs have no meaning outside the computer, even if they originally are programmed by a human being.

"Meaning" in the cybernetic concept relates to a system of systems only. If the human being interferes with a purpose outside the system - it imposes an interpreted level of meaning outside the system.

Wiener`s "information" presumes an observer with a meaning of his/her own outside the system who determines the goal of the system. The observer may be another machine but in the end (or perhaps beginning) there must be a human being somewhere with an intention or purpose. The observer`s meaning is thus interrelated with the system`s meaning. The signals of the system therefore have a relation to a human meaning, even if it can be very distant.

Miller argues that a living system should in principle be the same. The difference is the observer, however. Who is the final observer with a purpose or a goal? What is the goal of the amoeba in the world or the muscle in the human body or what is the purpose of man? One might as systems theory does, see no purpose other than the living system maintaining itself (Miller p.39). It seems a very meaningless world to me but I can`t answer that question! It seems to me that we then enter the realms of philosophy or theology.

Wiener`s concept of information relates both to manmade systems and to living subsystems like the liver or even the brain as a tissue of neurons. These systems use signals in a way that cybernetic theory seems to explain.

But there is a difference between the brain tissue itself and how this tissue is used in reflection and interpretation. They reflect two different unrelated levels of meaning. Even if one assumes that the only purpose of mankind is to maintain life, it seems that a human being may from time interfere in a way that rocks the foundation of any system. Vamos (1990) argues in a convincing way that closed manmade systems are impossible.

Meaning in Shannon`s Sense

One of the conclusions from Shannon`s theory is that entropy contains more information than structure. It is a strange idea that goes against common sense. Let us first try to understand Shannon`s concept by seeing it in connection with human/human communication.

Shannon presumes something/someone outside the transmission chain with a message which corresponds to "information". However, it is not information that is transmitted, but signals. There is a sender and a receiver of these signals. The sender`s meaning must be interpreted by the receiver outside the transmission itself. For doing this, both sender and receiver must have something in common - at least a language, otherwise they will not understand each other. If someone receives a meaningless expression via the telephone line there thus exists a very large number of possible interpretations. The information exists as a potential3 which of course is very large in a meaningless expression. If the expression on the other hand is crystal clear and the sender and the receiver share exactly the same understanding then there is very little information transmitted, only the signals themselves.

The signals thus exist on another level than the information and the two have nothing to do with each other unless the code of meaning is shared from the beginning.

Let us assume a natural system or object like a stone. The stone is meaningless in itself. Even a stone can thus be seen as "containing" an infinite number of potential meanings. It is a very large amount of "information" in Shannon`s sense. The stone may be measured, weighed observed etc. by humans down to the atomic level. The number of interpretations from such observations is equally infinite.

We will also see that the "sender" of the signals is not the stone but the human being`s apparatus. It would be an inconceivable task to transmit over a telephone line the entire possible amounts of data which make up the movements of the atoms which in their turn make up the object we call "stone". The object-called-stone exists as a source of potential information, which is not there until some human being interprets it, i.e. gives it meaning.

The word "stone" on the other hand consists of only five symbols that take up very little channel capacity. It is a very reduced human interpretation of the object-called-stone as compared to the richness of the atomic movements. The notion "stone" is therefore on another level than the atomic movements that make up the object-called-stone. The meaning of the word "stone" (as well as the meaning of any signals or data from any measurements of object-called-stone) are both human constructions, which have nothing to do with the object-called-stone itself.

All meaning is interpreted outside the transmission of signals. "Information" according to Shannon, must therefore not be confused with meaning. Shannon`s information relates not so much to what you do say as to what you could say (or do not say). The problems of interpreting signals into a "message" are left outside Shannon`s definition. Not so with Wiener. He assumes some meaning at least for the system level.

In order to get around the problem of meaning a common method is to contrast the word information with the word data. See for instance Schoderbek & al (1975/85, p.152). Data is according to them seen as:

unstructured, uninformed facts so copiously given out by the computer. Data can be generated indefinitely; they can be stored, retrieved, updated and again filed. They are a marketable commodity . . . each year the cost for data acquisition grows on the erroneous assumption that data are information.

The usage of the word information is by these authors restricted to facts with meaning or evaluated data. Information is connected to the circumstances of the receiver or user, whereas data exist independent of user. Data are seen as unevaluated pieces or materials, whereas information refers to data evaluated for a particular problem.

It is tempting to see Shannon`s signals as "data" and the meaning of the signals as "information", but this would be wrong. Shannon`s information can not be transmitted, like the system theorists assume.

This distinction is problematic also because meaningful information for one user in a specific situation might be devoid of meaning for another user in another situation. What can be defined as information in one context becomes data in another. The same set of symbols might therefore be toggling between "data" and "information" depending on the circumstances.

A definition of this kind does not bring any further understanding. Other successors of Shannon have suggested mathematical theories which add "meaning" to his theory. One idea suggested by Brillouin 1956 (referred in Jumarie 1990) is to regard the amount of information as a function of the ratio of the number of possible answers before and after a communication has taken place. Information would then be the difference between the two.

It might be this that Gregory Bateson referred4 to when he made his famous statement:

Information is a difference that makes a difference.

Brillouin also came up with a paradox: Suppose that a lenghty piece of information is sent as a text. The last item of the text is a bit which tells the receiver that all the text before the bit is untrue. Has any information been transferred then? Brillouin suggests an addition to Shannon`s theory which would take care of this: "negative" information.