Lecture 8 Research Design
A lecture in India started with this quote," If I had one last day, I would spend it in statistics class.....it would seem so much longer." Thanks to Nicky Ozbek.
Experimental Design
Definition: The structuring of research so that relationships between dependent variables and independent variables can be examined taking into account effects of extraneous variables.
Some definitions
Dependent variable
The variable whose differences we seek to explain or predict or control.
Typically,a measure of behavior - responses to a questionnaire or some other behavior
Dependent variables can be quantitative variables or categorical.
Independent Variable
Variable which we believe will explain, predict, or control differences in the dependent variable.
Independent variables can be quantitative variables, e.g., conscientiousness as an explanation of differences in GPA, or categorical, e.g., as Lecture vs. Computerized training programs as determiners of training performance.
Extraneous variables
Variables we’re not interested in that have the potential to explain, predict, or control differences in the dependent variable.
A few examples of Relationships examined in recent research.
1. The relationship of injury severity in ATV accidents to whether or not the rider was wearing a helmet.
2. The relationship of reading ability toamount of training of preschool teachers.
3. Quionna Caldwell: The relationship oforganizational commitment to organizational efforts to promote diversity.
4. Niki Wild: Relationship of team effectiveness to team personality composition.
5. Brittany Sentell: Relationship of Likelihood of hiringto Nature of Background check information.
6. Jennifer Scroggins: Relationship of Task performancetoConscientiousness and leader characteristics.
Goals
1. To explain.
To determine “why”. Demonstration of a relationship may provide evidence for or against an explanation of a relationship.
Example . . .
Everyone knows that conscientiousness is related to performance in a variety of tasks. Why?
A possible explanation.
Conscientiousness determines amount of time spent studying. People with high conscientiousness feel internal psychological “pain” if they haven’t studied enough.
Amount of time spent studying determines course grades. That is, differences in time spent studying are explained by differences in conscientiousness.
Therefore, differences in conscientiousness lead to differences in performance.
This explanation might be phrased as: Time spent practicing/studying mediates the conscientiousness -> performance relationship.
2. To predict.
To determine “if”. Discovery of a relationship may allow us to predict future behavior, such as when selecting employees or students.
Example:
Formula score predicts average performance in the first year I/O core courses. Persons with higher formula scores get better average grades in those courses.
We don’t care why the formula scores predict, we just want to use them to make sure we get persons in our graduate program who are able to profit from the experience.
3. To control
To determine. The existence of a relationship may allow us to control behavior by manipulating the independent variable.
Example: Offering a quarter point extra credit for each class attended increases averageattendance throughout a semester.
Knowing this relationship, I can use it to get more students to attend my class, by offering a quarter point extra credit for attendance.
The importance of variation
Three requirements for a relation . . .
1. The independent variable must vary.
2. The dependent variable must vary.
3. The dependent variable must covary with the independent variable.
This highlights the fact that variability is a partof all research efforts.
Without variability in the variables examined, there can be no relationships.
We’ve already touched on this in our study of regression when we covered restriction of range and its effect on correlation coefficients.
Ways of maximizing variability in the independent variable
1) If you can pick values of the independent variable, make sure they are as different from each other as possible – employ extreme values.
For example, you’re studying faking and wish to create a condition that will induce faking, use a big incentive, not a small one (as we did).
2) Employ optimal values of the independent variable – when the extremes aren’t enough.
Performance vs. Motivation.
Pick a low value of M and an intermediate value.
3) Use heterogeneous samples if each person comes to the research with his/her value of the independent variable.
For example, in the study of the faking ability ~ cognitive ability relationship, a wide range of cognitive ability should be studied. Our first sample showed the strongest relationship of CA to faking ability, because it contained both undergraduate and graduate students. For that reason, the variability of CA scores was greater in the first sample.
r = .308r = .114r = .226
X Variance = 45.6041.0043.39.
Extraneous Variables - Bad Variation
Extraneous variables complicate the identification of relationships between dependent and independent variables.
Extraneous variables affect our independent variables, our dependent variables, and the relationships between them.
Differences between people in extraneous variables contaminate all aspect of the research process.
Control – Minimizing effects of extraneous variables
Effects of extraneous variables must be controlled.
Much of the study of experimental design is devoted to controllingfor effects of extraneous variables
Example of statistically controlling for effects of an extraneous variable – from Biderman, Nguyen, Cunningham, & Ghorbani (Journal of Research in Personality, 2011). Table 7.
Big Five scores, which are not intended to measure affect, are contaminated by an extraneous variable, the Affective State of the participant. When you remove (control for) the effects of the participant’s affective state, the correlations of Big Five variables with other variables change, sometimes dramatically.
Correlations of Big Five with Positive Affect measured using the PANAS (red -> significant)
EACSO
.335.229.228.323.346
Same correlations after controlling for the effects of the participant’s affective state
.179-.098.110.157.120
The issue here is this: What is the REAL relationship of Agreeableness to Positive Affect?
Is it + .229, significantly positive – indicating that agreeable people have more positive affect?
Or is it -.098, not significant – indicating that agreeable people have neither more nor less positive affect.
More on control later in this lecture.
New Topic: Groups vs. Correlational Research.
Groups research.
Groups of persons are observed in one research condition and their behavior is compared with groups of persons observed in some other research condition.
The independent variable is the variable whose values represent the groups. There will be just 2 or 3 or a few values of this variable. That variable is typically a nominal / categorical value – the values are just names of the groups.
Typically, the dependent variable is a summary of performance within each group, typically the mean.
We compare means of the different groups in the research.
Even though we’re comparing means, such comparisons can be viewed as a study of how mean values arerelated to group membership.
Examples:
a. One group taught statistics without a lab. A 2nd group is taught with a lab. The independent variable is Requirement of a Labwith values 0 for No and 1 for Yes. Performance in the 2nd group is significantly higher than performance in the first.
Mean Comparison Conclusion: There is a significant difference in mean performance between the No Lab and the Lab Required groups.
Relationship Conclusion: There is a relationship between learning of statistics and requirement of a lab.
b. Three groups of patients are given a drug. Dosage for Group 0 is 0. For Group 1 it is 1 mg. For Group 2 it is 2 mg. So the independent variable is Drug Dosage with values 0, 1, and 2. Probability of improvement is highest in Group 2, second highest in Group 1 and lowest in Group 0.
Mean Comparison Conclusion: Improvement was least in the Dosage = 0 group, higher in the Dosage = 1 group and highest in the Dosage = 2 group.
Relationship Conclusion: Improvementis related to Drug dose.
So, in groups research, we can phrase our conclusions in two ways – either in group-comparison terms or in relationship terms. Either is appropriate.
Correlational Research.
In correlational research, the focus is on the relationship of the dependent variable values of individual persons to the independent variable values of those same individual persons.
We examine the relationship between the dependent variable and the quantitative independent variable. We often measure the relationship using the Pearson r.
Much (a lot) (an incredible amount of) (too much?) psychological research is correlational.
Examples:
a. The relationship of faking ability to cognitive ability.
Results of three studies – Nguyen, Wrensen, and Damron
r = .308r = .114r = .226
b. Organizational commitment to organizational diversity efforts.
Results of QuionnaCaldwell study relating Commitment to perceived fairness and perceived inclusion.
Note that the correlation does not have to be super strong for the research to be legitimate.
Few correlations in psychology are.
New Topic: Random Selection vs. Random Assignment
Selection refers to the movement of people from the population to the sample of research participants. In psychology, very few studies use a random process for this step. Very little psychological research involves random samples.
Assignment refers to the movement of already selected participants into one of the conditions of the research. Note that conceptually, assignment to conditions occurs after selection to participate in the research. Much psychological research involves random assignment.
Schematically
Some Specific Research Designs involving group comparisons
The goal is to examine the relation between a dependent variable and levels of an independent variable, controlling for extraneous variables.
When the independent variable has two levels, I will often refer to Groups as the Treatment Group and the No Treatment Group. I do this because much research is conducted to compare the effectiveness of a new way of treating people and the old way of treating them. In such examples, the people who get the new way are in the Treatment Group. Those who get the old way are said to be in the No Treatment Group. Probably “No New Treatment Group would be more appropriate, but no one says that.
Designs that are not useful
1. One-Group Posttest-Only Design.
There is only one group. It is the Treatment Group.
A group given a TreatmentPosttest Observation taken
One group is given the experimental treatment; only a posttest is obtained.
Example –
1. An infomercial presents testimonials of users of a product. The use of the product is the “treatment”.
Desired Statement: Performance is related to whether or not the product is used.
Weakness:
Only 1 value of the independent variable is used. There is no variation in the independent variable.
So, no difference has been observed
Without variability in the amount of product used, we cannot observe the relation of performance to amount of product. So we have nothing.
2. The Posttest-Only Design with Nonequivalent Groups.
Group 1 given Treatment 1Posttest Observation taken
Group 2 (not equivalent) given Treatment 2Posttest Observation taken
Non equivalent means that there has been no attempt to insure that the groups are equal prior to the treatment.
Example
A. Pre-existing groups that have already received the treatments are used.
Group 1:Persons who already take Tylenol
Group 2:Persons who already take Aleve
We would compare their responses to survey questions about efficacy of the pain relievers.
Desired Statement: Pain relief is related to type of pain reliever.
B. Two pre-existing groups are used. Treatments are assigned to the pre-existing groups.
Group 1:Persons in building A given enriched jobs.
Group 2:Persons in building B given routinized jobs.
Much organizational field research is of type B. It is impossible to randomly assign treatments to individuals.
Desired Statement: Job satisfaction is related to jobenrichment.
Weakness in both A and B . ..
The extraneous variable of Pre-existing group differences is not controlled for.
The groups may have differed in characteristics other than the characteristic of interest before the research was begun.
So differences in the dependent variable could be related to pre-existing group differences and not at all to differences in pain reliever (Example A) or differences in job enrichment (Example B).
These are called Retrospective Designs.
In the medical literature, designs involving groups that were formed prior to the conduct of the research are called retrospective designs.
3. One-Group Pretest-Posttest Design.
Pretest Observation takenGroup given treatmentPosttest observation Taken
A pretest is taken. The group is given the treatment. A posttest is taken. Change from pretest to posttest is measured.
Also called Before-After designs.
Example:
Football Team is doing lousyA new coach is hired.Team does better.
Desired Statement:Performance is related to who was coaching.
Weakness
The measurement of extraneous variables related to time are not controlled for.
The differences in behavior could be related to time of occurrence from before to after treatment, to loss of key players or a weak quarterback, for example.
Groups Research Designs that are useful.
4. An acceptable design: Posttest-Only Randomized Groups Design.
Persons randomly assigned to Group 1 given treatment 1Posttest Observation Taken
Persons randomly assigned to Group 2 given Treatment 2Posttest Observation taken
Individual participants are randomly assigned to groups.
Only posttest measures are taken.
Desired Statement: Posttest behavior are related to type of treatment.
Advantage: The extraneous variable of pre-existing group differences has been controlled for through randomization.
5. A great design: Pretest-Posttest Randomized Groups Design
Persons randomly assigned to Group 1 given pretestGiven Treatment 1. Posttest Observation Taken
Persons randomly assigned to Group 2 given pretestGiven Treatment 2. Posttest Observation taken
Individual participants are randomly assigned to groups. Pretest and posttest measures are taken.
Desired statement: Posttest behavior is related to type of treatment.
Advantages:
Extraneous variable of pre-existing group differences is controlled for by random assignment.
If that weren’t enough, pre-existing group differences can also be measured and controlled for through the pre-test.
Finally, having the pretest allows a more powerful statistical test of differences in treatment effects.
You should use this design whenever you can to compare treatments given to two groups.
These two designs are often called Prospective Designs.
In the medical literature, research involving persons randomly assigned to groups prior to treatment are called prospective designs.
6. A salvage design: The Pretest-Posttest with Nonequivalent Groups Design
(Salvaging Design 1)
It is important to note that the pretest and posttest are the same instrument.
One of the most frequently employed designs in the social sciences.
Desired statement: Differences on the posttest are related to treatment differences.
Advantage
Extraneous variable of pre-existing group differences is controlled for using the pre-test if, especially if, the following ideal outcome across time occurs.
The ideal outcome:
If this pattern of results occurs –
a) no difference on the pretest, and
b) difference favoring the treatment group on the posttest,
most researchers would argue that it is evidence for a relationship between type of the posttest to the type of treatment controlling for the extraneous variable of pre-existing group differences.
7. Another salvage design - One Group Interrupted Time Series design.
(Salvaging Design 1)
This is an extension of the simple One-Group Pretest-Posttest Design (#3 above).
Multiple pretest observations are taken. Multiple posttest observations are taken.
The variation (or lack of variation) among the multiple pretest observations and among the multiple posttest observations serves as a standard against which the variation between pre- and posttest can be compared.
Variation in behavior is related to treatment differences controlling for other time-related differences. We know that change in the dependent variable is not just a function of time because of the lack of systematic change in the dependent variable prior to introduction of the treatment and the lack of systematic change after the treatment.
8. Adding factors, creating Factorial Designs
Factorial Designs are those involving two or more independent variables (called factors in that literature). Observations are taken at all combinations of levels of the several factors.
Example.
Factor A: Instruction Technique – Human Lecture vs. Computerized
Factor B: Type of Job – Clerical vs. Assembly Line
Factorial designs allow us to
1. Look at the relationship of the dependent variable to more than one factor at the same time – increasing our efficiency.
2. Look at the relationship of the dependent variable each factor when controlling for the other factor.
3. Look at the relationship of the dependent variable to the interaction of the two (or more) factors, something no other design allows us to do.
THE VALIDITIES
Suppose I am conducting research involving social influence - the extent to which our decisions are affected by the opinions of others. Suppose I have compared two conditions - one in which a person designated as a conservative endorses a liberal position and another in which a liberal endorses a liberal position.
The interest is in determining whether a conservative endorser of a liberal position has more influence than a liberal endorser of the same liberal position.
I compare behavior of persons in two groups - a liberal endorser group and a conservative endorser group. My hope is to use the information to control peoples’ voting.
The dependent variable is whether each participant chose the endorsed position or not – 1=Yes; 0=No.
The independent variable is the ideology of the endorser.
Fifty participants who describe themselves as independents are told that the document was written by a liberal senator from a northeastern state.
The other fifty participants are told that the document was written a conservative senator from a western plains state.
Participants are shown a document advocating government health care insurance. After reading the document, participants are asked whether or not they would support the government health care insurance advocated in the document.