Thinking Critically and Evaluating Evidence

CHAPTER 2

SCIENTIFIC METHODS IN PSYCHOLOGY

THINKING CRITICALLY AND EVALUATING EVIDENCE

What’s the Evidence: Psychology is a Science...... 37

A Step Further:

1. The Goals of the Scientific Method...... 37
2. Retesting Claims of Telepathy...... 39

Promoting Discussion:

1. People Want to Believe in ESP...... 40
2. Familiar Examples and Parsimonious Explanations...... 41
3. Feeling the Future: Bem’s Meta-analytic Study...... 43

In-Class Demonstration/Exercise:

1. Testing the Validity of Astrology...... 43
2. A Double-Blind Test of Astrology...... 44
3. Validation with Fortune Cookies...... 45

Out-of-Class Activity: Hoax or Just Strange? A Web Evaluation Exercise...... 45

CONDUCTING PSYCHOLOGICAL RESEARCH

Turning on the Lights: Interpreting the Correlation Coefficient...... 47

In-Class Demonstration:

1. Determining the Direction of Correlations...... 48
2. Belief in Common-Sense Adages...... 49
3. Using Adages to Learn about Variables and Hypothesis Testing...... 49
4. Selecting the Proper Control Group...... 50
5. A Sweet Way to Demonstrate Experimental Method...... 51
6. Distinguishing Between Correlational and Experimental Studies...... 52

Out-of-Class Activity:

1. Practice in Operationalizing Concepts...... 53
2. Recognizing Independent and Dependent Variables...... 53
3. What Went Wrong in These Experiments?...... 54

A Step Further: A Methodological Challenge: The Hawthorne Effect...... 55

Psychology and Culture: Methodological Challenges of Cross-Cultural Research...55

What’s the Evidence: Correlation Does Not Equal Causation...... 56

Promoting Discussion: Create a Hypothesis...... 58

MEASURING AND ANALYZING RESULTS

A Step Further:

1. Lying with Statistics...... 58
2. Meta-analysis and the “File-Drawer Problem”...... 59

Psychology and Gender: Gender Differences: Practical vs. Statistical Significance..60

Promoting Discussion: Chartjunk Awareness...... 61

GENERAL RESOURCES

Chapter Outline...... 33

References...... 61

Video References...... 62

Internet Resources...... 62

Questions for Review...... 65

Handouts...... 67

CHAPTER OUTLINE

Scientific Methods in Psychology

Module 2.1: Thinking Critically and Evaluating Evidence

1)Evidence and Theory in Science:

a)Induction, the process of inferring a general principle from observations, is a central process in science by which facts, hypotheses, and theories are developed. Science derives from the Latin word for knowledge. Knowledge in psychology, like other sciences, is derived from careful observation and collection of information (data.)

b)A theory is not just a guess or a speculation; it is an explanation or model that fits many observations and makes valid predictions. A good theory is falsifiable. It makes sufficiently precise predictions so that we can at least speculate about the kind of evidence that would contradict the theory.

2)Steps for Gathering and Evaluating Evidence:

a)Scientistswant to know the evidence behind a given claim. The burden of proof is generally viewed as being on those who make scientific claims. We gain knowledge by testing hypotheses in order to evaluate the evidence. A hypothesis is a clear predictive statement about what will happen under a given set of conditions.

b)Research proceeds according to these steps: Hypothesis, Method, Results, and Interpretation.

3)Replicability:

a)A result is replicable if any person can obtain it by following the same research procedure. If researchers cannot find conditions under which a result can be replicated, then they cannot accept the result.

b)A meta-analysiscombines and analyzes the results of many studies as if they were all one very large study.

4)Criteria for Evaluating Scientific Theories:

a)Theoriesarecomprehensive explanations of observable events that lead to useful predictions. Good theories are simple or parsimonious—they are based on ideas that work and are founded on the fewest possible assumptions.

b)Parsimony and Degrees of Open-Mindedness: Parsimony in scientific theory refers to the quality of relying on the fewest and simplest assumptions.

c)Applying Parsimony: Clever Hans, the Amazing Horse: Clever Hans (owned by Mr. von Osten, a German mathematics teacher) was a horse that supposedly had the ability to count and perform other “intelligent” operations. Oskar Pfungst was skeptical. By careful testing and observation, Pfungst was able to offer a more parsimonious and accurate explanation, that Hans’s “abilities” were based on a perfectly ordinary ability of mammals: he was able to interpret subtle nonverbal cues given by humans.

d)Applying Parsimony: Extrasensory Perception:Extrasensory perception(ESP) is the supposed ability “...to acquire information without using any sense organ and without receiving any form of energy.”

i)Evidence for ESP comes in various forms. Anecdotes, or people’s reports of isolated events, are poor evidence. Professional psychics are most often magicians or similar stage performers, and are using simple tricks well known to other experienced entertainers. The ganzfeld procedure has been very thoroughly investigated. Some intriguing preliminary results have ultimately been found to be non-replicable.

ii)Psychologists remain skeptical of ESP research because of shoddy methods, lack of replicable results, and a lack of parsimony that have been associated with it.

Module 2.2: Conducting Psychological Research

1)General Principles of Psychological Research: Definitions of psychological terms—Psychologists develop operational definitions for concepts of interest. These are special definitions that permit a scientist to make measurements by specifying the operations (procedures) used to produce or measure something, usually by giving it a numerical value. An operational definition allows psychologists to study intangible concepts such as memory, assertiveness, and happiness by making them more concrete, countable, or observable.

a)Population Samples: Because it is usually not practical to try to study all individuals in a population (the entire set of individuals to be considered), researchers study a small subset, or sample. A convenience sample is a group chosen because it can be easily studied. This is only appropriate when we are studying principles that are recognized as being generally the same for all people. A representative sample is one that sufficiently resembles the entire population to which we wish to generalize our results in its percentage of males and females, blacks and whites, young and old, city dwellers and farmers, or other important characteristics. It is sometimes difficult to achieve a truly representative profile in all aspects that might affect the validity of the results. In most scientific research we would prefer to try to select a random sample, one in which every individual in the population has the same chance to be selected. Random sampling decreases the chance of systematic bias that would detract from the validity of our results. True random sampling, however, can be difficult to execute; often psychologists doing experimental research end up with samples composed of volunteers.Cross-cultural sampling is also desirable because it increases the chances that we can accurately generalize our results to all humans. This method involves selecting people from at least two cultures for study.

2)Observational Research Designs:

a)Naturalistic Observation: A naturalistic observation is an examination and description of the activities of people or nonhuman animals under more-or-less natural conditions. This design may yield interesting and enlightening results, but the procedure itself may also produce bias as the presence of the observer may change the behavior that is being studied.

b)Case Histories: A case history is a thorough description of the unusual person or condition. It often relies on naturalistic observation, but focuses on an individual. Case histories are well suited to exploring unusual behavioral conditions, but their generalizability is limited.

c)Surveys: A survey is a study of the prevalence of certain beliefs, attitudes, or behaviors based on people’s responses to specific questions. Validity of survey data depends on several factors, including sampling, the competence of those being interviewed, wording of questions, and other surveyor biases.

d)Correlational Studies: A correlational study investigates the relationship between two variables, both of which are outside the investigator’s control. A variable is any characteristic that can differ between observations or measurements. The correlation coefficient is a numerical estimate of the strength and direction of the relationship between two variables; the coefficient can range in value from +1 to −1. Correlations indicate the strength and direction of a relationship and allow for prediction of one variable, based on the other. Illusory correlations: An illusory correlation is an apparent relationship based on casual observations or unrelated or weakly related events. The widely held belief in “moon madness” is a myth that is in part sustained by the power of illusory correlation. Although correlations tell us about the strength of a relationship, they do not justify cause-and-effect conclusions because correlations do not tell us why the relationship exists. To determine causation, variables need to be manipulated directly using an experiment.

3)Experiments: An experiment is a study in which the investigator manipulates at least one variable while measuring at least one other variable.

a)Some Important Terms:

i)The independent variable is the one that the experimenter manipulates. The dependent variable is the item that the experimenter measures to see how it was affected by the change in the independent variable.

ii)The experimental group receives the treatment that the experiment is designed to test. The control group is a set of individuals treated the same way as the experimental group except for the treatment. A good experiment uses random assignment to groups—every subject has the same probability of being assigned to either the experimental or control group.

iii)Experimenter Bias and Blind Studies: Experimenter bias is the tendency of an experimenter to distort or misperceive the results of an experiment based on the expected outcome.A blind observer (one who is unaware of the actual hypothesis of the study) may be used to record results on behalf of the researcher. A placebo condition may be used in an experiment. The participants who receive the placebo think that they are getting a treatment but they are actually receiving an inert or sham version. In a single-blind study either the observer or the participants are unaware of who received the treatment; in a double-blind study neither knows. Demand characteristics are clues that tell subjects what results the experimenter hopes to find. They can set up self-fulfilling prophecies that can cause the results of the experiment to be biased.

4)Ethical Considerations in Research:

a)Ethical Concerns with Humans: An important principle is that experiments should only include experiences that people would typically agree to. Potential subjects are asked to give informed consent prior to participating. This signifies that they have been told what to expect and that they agree to continue. At universities and other research institutions, the details of the study must be submitted to an Institutional Review Board that either grants approval or demands revisions if the procedure seems to be potentially harmful.

b)Ethical Concerns with Nonhumans: Animal research is useful for understanding basic mental processes and behavior. A greater degree of control over the subjects is possible. Some people oppose such research because animals cannot give informed consent. Others believe that the potential benefit to humans provides sufficient justification. Both sides in this debate have raised reasonable arguments, and no compromise seems fully satisfactory.

Module 2.3: Measuring and Analyzing Results

1)Descriptive Statistics: Descriptive statistics are mathematical summaries of research results.

a)Measures of the Central Score: The mean is the sum of all the scores divided by the total number of scores. It is especially useful if the scores approximate a normal distribution, a symmetrical frequency of scores clustered around the mean. The mean can be misleading if the distribution is not normal. The median is the middle score of the distribution. It is less affected by extreme scores and so can be useful if our distribution is not normal. The mode is the most frequently occurring score. It is a useful measure of central scores for categorical data such as diagnostic categories and other qualitative (non-numeric) data.

b)Measures of Variation: To accurately describe a distribution we also need some measurement of how scores tend to vary from the mean. One such measure, the range, is the span from the highest to the lowest scores. It is not very useful because it only reflects the extremes. The standard deviation is a measurement of the amount of variation that is typical among scores in a normal distribution and provides a useful means of comparing scores on two different tests.

2)Evaluating Results: Inferential Statistics: An inferential statistic is a statement about a large group based on an inference drawn from a sample. The usual convention is that if the probability that a result occurred due to chance is less than 5%, (probability value of less than 5% or “p < .05”), the result is considered “statistically significant” or “statistically reliable,” that is, unlikely to have arisen by chance. The smaller the stated probability value is, the more impressive the result is considered to be. Usually p-values are presented along with confidence intervals. The 95% confidence interval is the range within which the researcher can state with 95% certainty where the true mean lies.

THINKING CRITICALLY AND EVALUATING EVIDENCE

WHAT’S THE EVIDENCE?

Psychology is a Science

Although the text does a very good job of presenting psychology as a science, students will still have trouble seeing it that way. People most associate psychology with therapy, counseling, and “helping people.” This perception of psychology is woefully incomplete. Students’ perceptions of psychology and psychologists can be revealed quite easily through a simple word-association test. Have students get out a sheet of paper,
and tell them to write down the first five words that pop to mind when they think “psychology”; have them do the same thing in response to the word “psychologist.” Then, repeat the procedure by having them associate to the word “science,” and finally, to the word “scientist.” The results will reveal a clear clinical bias in people’s perceptions of psychologists. Typically, the words “psychology” and “psychologist” will be associated with words like “therapy,” “crazy,” “help people,” “couch,” “Freud,” “mental disorders,” and similar terms. “Science” and “scientist” will lead to associations like “theory,” “experiment,” “laboratory,” “white coat,” “measurement,” as well as an occasional pejorative association like “geek” or “nerd.”

This word-association exercise is a great starting point for a discussion of psychology as a science. Take the list of word associations to “science” and “scientist” and discuss how they apply to psychology and psychologists (almost all of the generated terms will—except, of course, “geek” and “nerd”). To help students see psychology as a science just like biology, chemistry, or physics, emphasize the important general scientific principles in psychology—principles such as unbiased observation, precise measurement, theories, hypotheses, and replicability.

Disabusing students of the notion that psychology is simply the study of mental disorders and their treatment will serve as a useful vehicle for a discussion of what science is, and will be an excellent preparation for the rest of the Kalat text, which emphasizes the scientific nature of the field.

A STEP FURTHER

1.The Goals of the Scientific Method

Students often struggle to understand the conceptual difference between the “scientific method” and the “types” of research methods. One way to facilitate their understanding is to present the scientific method as having four goals (description, prediction, explanation and control). These goals are listed in Handout 2-1. It is important to stress that these goals are the same for any entity that can be studied via the scientific method (a chemical compound, a biological organism, or, in the case of psychology, behavior). Each goal can be understood in terms of the question that it answers about the entity under investigation. For psychology, each goal answers the following question:

(1)Description: What are the characteristics of the behavior?

(2)Prediction: How likely is it that the behavior will occur?

(3)Explanation: What causes the behavior?

(4)Control: Can I make the behavior happen/not happen?

After presenting this framework, explain how each research method attempts to achieve at least one of these goals. The research methods that satisfy the goal of description are naturalistic observation, case history, and survey. The research method that satisfies the goal of prediction is correlation. (The correlational methodology may start with one of the other observational methods, but it does not stop with simple description. It attempts to predict one observed variable based on another observed variable.) The research method that satisfies the goals of explanation and control is the experiment. Given that an experiment is the only methodology that can provide an explanation, it can be explained to students that the true experiment is the only methodology that can determine causation and that “correlation does not equal causation.”

This framework will also help students interpret the results of research presented in the popular press. Very often the specific methodology used by the researcher is not reported in brief television or newspaper clips. This is problematic because the methodology indicates the goal. This in turn determines how the results should be interpreted and used. For example, if the methodology was correlational, then it satisfies the prediction goal, but causation should not be inferred. Sometimes the words used to describe the results of a study will provide a hint about the methodology used and the appropriate use of the results, but writers and reporters for the popular press don’t always give details of how research is carried out (for example, calling studies “experiments” when those studies do not involve any manipulation of an independent variable.) Consequently, students have a difficult time identifying the goal and making a proper interpretation. The exercise on Handout 2-2 helps students become aware of these subtleties. Students should be instructed to identify the goal of the scientific method that each statement satisfies. The correct answer is given in parentheses.

Researchers spent six months watching toddlers in day care and home settings. They have concluded that normal children combine words in simple sentences by age two. (Naturalistic observation; description)

By comparing three groups of people, one getting no treatment, one getting traditional treatment, and one receiving newly developed virtual reality therapy, psychologists have concluded that you can overcome fear of spiders by using a virtual reality system to gradually raise your comfort level in the presence of spiders. (Experiment; explanation and control)