The Role of Theory and Data

The Role of Theory and Data

in Social Science Research

I. Some Background Information

Western scientific reasoning began with Aristotle (384 BC – 322 BC) who felt that we could develop generalizations of a phenomenon from particular instances of it. Aristotle was very much in favor of using an inductive method to discover the underlying nature of the world. Once such generalizations were sufficiently established, one could then use deduction to help weave these generalizations together to try and form a consistent view of reality. One problem with Aristotle is that he did not sufficiently pursue the issue of testing his theories and predictions against data. He took great liberty in proposing generalizations from the data, but he did not always try hard to verify these with experimentation.

Aristotle’s method of looking from particulars to generalizations reached its peak with Francis Bacon (1561-1625) who optimistically attempted to create a systematic method of scientific analysis based on the listing of characteristics in tables and then the seeking of patterns in the data.[1] In modern day terms, he was a data-miner. This method was subsequently refined and innovated by scientists during the Enlightenmentof the 18th century. I. Newton (1642-1727) had found that by making some simple theoretical hypotheses (perhaps derived from loose empirical observations), a mathematical model of the scientific phenomenon such as gravitation could be derived — one that further predicted unknown facts, which could then be verified by additional experimentation. Newton’s success set the world on fire. Suddenly everyone was trying to create mathematical models of physical phenomena to view what predictions or resulting effects could be generated.

Unfortunately, experimental predictions and effects were never consistently a unique number. Instead, when experiments were carried out, different numbers for the effects of a single theoretical modelwould emerge each time. Sometimes the results of the experiments were higher than the theory predicted and sometimes they were lower. Never were they exactly equal to the theoretical values all the time. There was always some apparently unexplained error. For Newton’s gravitational laws, the predictions were so close that these errors were not considered important. In other cases, these errors were large enough that they cast doubt on the reliability of the proposed law or generalization. How were scientists to determine if a generalization was really there?

It was at this time that probability and statistics began to be applied to experimentation. Formal methods of probability had been introduced much earlier by B. Pascal and P. Fermat around 1654. But, these models were usually applied to card games and other curiosities. Later, as mathematicians and physicists began to employ these tools, the probabilistic measurement of errors was created. Through these methods an observed error could be a random outcome and yet be judged either significant or insignificant by using probability. Mean absolute errors and standard errors began to be discussed by scientists (such as Karl Pearson) at the end of the 19th century when judging the fit of their theoretical predictions with real life observations. The theoretical and the empirical predictions could be evaluated and thus the models proposed could be evaluated. The basic scientific setup could be written as

Observed Effect = Theoretical Effect + Random Error

From this one would first propose a theoretical model that would determine specific theoretical effects given the model’s assumed factors at play. One would then experiment by gathering data and generate many observed effectsfor the same set of additional factors at play. This would in turn generate a set of realizations of the random error by the equation above. If these random terms were sufficiently small (or close to zero) the theoretical model would have passed the test — otherwise the model would be falsified. According to this process, the theoretical model would be created by brilliant insight, innovation, and intuition; the observed effects would be gathered by laborious experimentation; and the judgment of whether the observed errors were sufficiently close to zero would be determined by methods of probability and statistics. Most scientific work today follows this same pattern. The only difference is in judging whether the model is a good model or a bad model.

This is the modern form of scientific method.[2]The equation above captures the essence of what is called positivism. This is the belief that there is a stable reality out there which can be identified by data, logic, and a mathematical or formal model, that the reality crosses cultural borders, and that the results do not depend on the investigator. As Stephen Hawking has claimed

“ …a scientific theory is a mathematical model that describes and codifies the observations we make. A good theory will describe a large range of phenomena on the basis of a few simple postulates and will make definite predictions that can be tested…”[3]

II. Is Social Science a Real Science?

The modern wonders around us,following on the application of science, have caused social scientists to adopt similar methods of research to those in chemistry, physics, etc. However, there have been some important members of the scientific community that have criticized social science as being a “pseudoscience”. Most prominent among these critics was Richard Feynman.[4] Feynman felt that social science was too loose. Social scientists could not find anything stable and permanent, unlike physics with its immutable laws. In physics, the errors between theoretical and observed can be extremely small — hundreds of trillions of times smaller than errors found in economic models. Feynman knew that there were many factors being left out of these social science models. He recognized that our society changes over time and there are no real constants for social phenomena; constant like the speed of light or the charge on an electron. He also knew how hard it was to isolate all the factors in an experiment and he couldn’t believe that social scientists had done the work necessary to make a convincing case. For Feynman, social science was not real science; social science only paraded around like a real science.

Feynman’s criticisms were formidable and needed to be answered. But, how could social science be defended? The answer to this is to be found in the writings of J.M. Keynes during the 1920s and 1930s. Keynes rejected the idea that economics (and social science) was a science. Instead, he saw economics as a special branch of logic. Here are some revealing portions of his writings during the period

“The Theory of Economics does not furnish a body of settled conclusions immediately applicable to policy. It is a method rather than a doctrine, an apparatus of the mind, a technique of thinking which helps its possessor to draw correct conclusions.”

and

“In chemistry and physics and other natural sciences the object of experiment is to fill in the actual values of the various quantities and factors appearing in an equation or a formula; and the work when done is once and for all. In economics that is not the case, and to convert a model into a quantitative formula is to destroy its usefulness as an instrument of thought.”

and still more

“I also want to emphasize strongly the point about economics being a moral science. I mentioned before that it deals with introspection and with values. I might have added that it deals with motives, expectations, psychological uncertainties. One has to be constantly on guard against treating the material as constant and homogeneous in the same way that the material of the other sciences, in spite of its complexity, is constant and homogeneous. It is as though the fall of the apple to the ground depended on the apple's motives, on whether it is worth while falling to the ground, and whether the ground wanted the apple to fall, and on mistaken calculations on the part of the apple as to how far it was from the centre of the earth.”

Thus, Keynes defense of social science is to recognize that it is not really a science in the same sense that chemistry and physics are sciences. Instead, social science is a branch of logic which allows the practitioner a better opportunity to draw correct inferences. The structure of society is constantly changing and therefore we must be constantly looking to create new and better models of social phenomena. We cannot be happy with models that work now, since such models will soon prove to be inadequate. We will never have social laws that remain constant and which will be stable over time. We must use our logical principles to guide us in our policymaking.

III. How Should We Model Social Phenomena

During the 1950s a major methodological row arose over the nature of economic modeling. Two giants of economic theory, Paul Samuelson and Milton Friedman, disagreed strongly over what constituted a good social science model. Samuelson defended a descriptivist view of modeling, which said that the assumptions and structure of the model were merely a description of reality and therefore realistic assumptions and structure were essential. For him, a bad model was one that was a poor description. Models were like paintings of reality and therefore needed to be almost photo-perfect — think of the 19th century painter Gustave Courbet. Every detail is shown in his paintings. By contrast, Milton Friedman defended instrumentalism. For Friedman, a good model was one that performed well, in the sense of making good predictions, stimulating further research, and enjoying wide spread acceptance. Friedman was more like an impressionist painter, who sought to deny realism in order to evoke a particular mood that was also part of reality — think of the 19th century painter Claude Monet. A good model did not have to be realistic if it were consistently a good predictor. Friedman went so far as to say that seeking absolute realism could often hamper good research by making models too complicated and unfocused. For him, a model might even have assumptions that were patently false and yet still be useful. Samuelson was outraged that Friedman would promote a methodology that at times denied the realism of assumptions and structure. Choosing assumptions that are known to be false was a horrible mistake and totally unscientific. He branded this use of faulty assumptions in models the Friedman F-Twist.

Neither Samuelson nor Friedman was entirely successful in convincing scholars about the best way to create social science models. Ideally, models that are built have both realistic assumptions and are good predictors. However, from what we have learned about Keynes’ thinking, even models that do well at describing reality today and predicting the data in the past will probably become obsolete over time due to the constant change in society’s structure. This must be so since new and better products and social behavior tend to replace the old in what J. Schumpeter called creative destruction. Creative destruction occurs when new social institutions, techniques, innovations, products, or firms enter the marketplace and competitively displace their older and outdated counterparts. This ruthlessly competitive process creates new entities while destroying the older ones. Our models must always be flexible to handle such inevitable change.

Discussion Questions

#1. Why can science never be purely deductive?

#2. Is it right to say that society can be well modeled by mathematical models?

#3. What is a good scientific theory to Popper?

#4. Why are there so many different schools of economics?

#5. What was the argument between Samuelson and Friedman on modeling?

#6. Explain Feynman’s criticism of social science. Answer this criticism with the comments of Keynes.

#7. How does creative destruction influence modeling of social phenomena? Can we apply creative destruction to the labor market?

[1] Those of you who feel this is a useless and laborious method should remember that it was just this kind of listing of characteristics that led Mendeleev (1869) to discover the nature of the chemical periodic table found in modern chemistry texts. It worked very well in this case. But, most scientific phenomena have too many variables, constraints, and complicated nonlinear relations to be visible from a simple table. John Stuart Mill in his book A System of Logic (1843) uses many of the same techniques as Bacon for isolating the cause of a phenomenon.

[2] There is a huge literature on scientific methodology. In short, the equation I have shown you above is the foundation of all modern scientific work. That is, theory is compared to experiment. However, there is a problem, since a theory can often fail in its testing, but nevertheless remain a powerful force in science. The adherents to a theory often continue to use it and to try to save it by adapting it in certain ways. K. Popper is the name that is associated with strict falsification. He believed that a serious scientific theory must be formulated so that a clear test can be made of it. If it passes the experimental testing, then it increases our conviction that the model is good. If it fails, the theory must be judged wrong and it’s back to the drawing board. T. Kuhn challenged this view of falsification. Kuhn felt that scientific research involves commitment to a research paradigm or a stable foundation. Scientists do not easily throw away these paradigms. Instead, changes to the foundations occur in jumps (like Einstein’s changes to Newton). Kuhn did not think that a set of tests could cause scientists to abandon the foundations of their research. They would instead look for new ways of explaining their results within the stable paradigm. Today, most physical scientists adopt Popper’s method, while many social scientists tend to adopt Kuhn’s method. This is why we find so many different groupings in economics and sociology (Keynesians, monetarists, new classical, etc) and so few in the physical sciences. Popper and Kuhn have been discussed and extended by people such as Imre Lakatos and Paul Feyerbend, but these subsequent extensions have not been particularly informative or impactful.

[3] From the Universe in a Nutshell, p.31.

[4] See the 2 minute Youtube video of Feynman at