CHAPTER EIGHT
Confidence Intervals, Effect Size, and Statistical Power
NOTE TO INSTRUCTORS
Students have a tendency to think that if something is statistically significant, the story is over and that’s all that a person needs to know. In other words, they frequently confuse “statistically significant” with “meaningful.” This chapter will help students recognize that this is not always the case. Aside from using the discussion questions and classroom exercises, present examples of studies that demonstrate a significant difference between groups but not a very meaningful one. It is also important to break students of the habit of using phrases such as “very significant” by discussing effect sizes. Although students might be tempted to describe an effect as “very significant,” emphasize that they should use effect sizes for this purpose instead.
OUTLINE OF RESOURCES
I.Confidence Intervals
Discussion Question 8-1 (p. 74)
Classroom Activity 8-1: Understanding Confidence Intervals (p. 74)
Discussion Question 8-2 (p. 75)
II.Effect Size and prep
Discussion Question 8-3 (p. 75)
Discussion Question 8-4 (p. 76)
III.Next Steps: prep
IV.Statistical Power
Discussion Question 8-5 (p. 77)
Discussion Question 8-6 (p. 77)
Classroom Activity 8-2: Working with Confidence Intervals and Effect Size (p. 77)
V.Next Steps: Meta-Analysis
Discussion Question 8-7 (p. 78)
Classroom Activity 8-3: Analyzing Meta-Analyses (p. 78)
Additional Reading (p. 78)
Online Resources (p. 79)
VI.Handouts
Handout 8-1: Classroom Activity: Working with Confidence Intervals and Effect Size (p. 80)
Handout 8-2: Analyzing Meta-Analyses (p. 82)
Chapter Guide
II.Confidence Intervals
1.A point estimate is a summary statistic from a sample that is just one number as an estimate of the population parameter.
2.Instead of using a point estimate, it is wiser to use an interval estimate, which is based on a sample statistic and provides a range of plausible values for the population parameter.
Discussion Question 8-1
What is the difference between a point estimate and an interval estimate?
Your students’ answers should include:
A point estimate is a summary statistic from a sample that is just one number as an estimate of the population parameter. Point estimates are useful for gauging the central tendency, but by themselves can be misleading.
An interval estimate is based on a sample statistic and provides a range of plausible values for the population parameter. Interval estimates are frequently used in media reports, particularly when reporting political polls.
Classroom Activity 8-1
Understanding Confidence Intervals
The following Web site provides a nice applet to help your students understand confidence intervals:
The applet simulates a known population mean and standard deviation and allows you to control the sample size, providing a graphical display of the resulting confidence intervals.
3.A confidence interval is an interval estimate that includes the mean we would expect a certain percentage of the time for the sample statistic were we to sample from the same population repeatedly.
4.With a confidence interval, we expect to find a mean in this interval 95% of the time that we conduct the same study (if our confidence level is 95%).
5.To calculate a confidence interval with a z test, we first draw a normal curve that has the sample mean in the center.
6.We then indicate the bounds of the confidence interval on either end and write the percentages under each segment of the curve.
7.Next, we look up the z statistics for the lower and upper ends of the confidence interval in the z table.
8.We then convert the z statistic to raw means for the lower and upper ends of the confidence interval. To do so, we first calculate the standard error as our measure of spread using the formula M = /. Then, with this standard error and the sample mean, we can calculate the raw mean at the upper and lower end of the confidence interval. For the lower end we use the formula: MLower = –z(M) + MSample. For the upper end, we use the formulat: MUpper = –z(M) + MSample.
9.Lastly, we should check our answer to ensure that each end of the confidence interval is exactly the same distance from the sample mean.
Discussion Question 8-2
How would you calculate a confidence interval with a z test?
Your students’ answers should include:
To calculate a confidence interval with a z test:
Draw a normal curve with a sample mean in the center.
Indicate the bounds of the confidence interval on either end and write the percentages under each segment of the curve.
Look up the z statistics for the lower and upper ends of the confidence interval in the z table.
Convert the z statistic to raw means for the lower and upper ends of the confidence interval. For the lower end, use the formula:
MLower = –z(M) + MSample. For the upper end, use the formula:
MUpper = z(M) + MSample.
Lastly, check the answer to ensure that each end of the confidence interval is exactly the same distance from the sample mean.
II.Effect Size and prep
1.Increasing the sample size can lead to an increased test statistic during hypothesis testing. In other words, it becomes progressively easier to declare statistical significance as we increase the sample size.
2An effect size indicates the size of a difference and is unaffected by sample size.
3.Effect size tells us how much two populations do not overlap. Two populations can overlap less if either their means are farther apart or the variation within each population is smaller.
Discussion Question 8-3
What is an effect size, and why would reporting it be useful?
Your students’ answers should include:
An effect size is a measure of the degree to which groups differ in the population on the dependent variable.
It is useful to report the effect size because it provides you with a standardized value of the degree to which two populations do not overlap and addresses the relative importance and generalizability of your sample statistics.
3.Cohen’s d is a measure of effect size that assesses the difference between two means in terms of standard deviation, not standard error.
4.The formula for Cohen’s dfor a z distribution is: d = (M – )/
5.A d of .2 is considered a small effect size, a d of .5 is considered a medium effect size, and a d of .8 is considered a large effect size.
6.The sign of the effect size does not matter.
Discussion Question 8-4
Imagine you obtain an effect size of –0.3. How would you interpret this number?
Your students’ answers should include:
If you obtained an effect size of –0.3, you would interpret this as a small effect size.
III.Next Steps: prep
1.Another method we can use in hypothesis testing is prep or the probability of replicating an effect given a particular population and sample size. It is interpreted as “This effect will replicate 100(prep)% of the time.”
2.To calculate prep, we first calculate the specific p value associated with our test statistic.
3.Next, using Excel we enter into one cell the formula
=NORMDIST(NORMSINV (1-P/SQRT(2))) where we substitute the actual p value for p.
IV.Statistical Power
1.Statistical power is a measure of our ability to reject the null hypothesis given that the null hypothesis is false. In other words, it is the probability that we will not make a Type II error, or the probability that we will reject the null hypothesis when we should reject the null hypothesis.
2.Our calculation of statistical power ranges from a probability of 0.00 to 1.00. Historically, statisticians have used a probability of .80 as the minimum for conducting a study.
3.There are three steps to calculating statistical power. In the first step, we determine the information needed to calculate statistical power, including the population mean, the population standard deviation, the hypothesized mean for the sample, the sample size, and the standard error based on this sample size.
4.In step two, we caculate the critical value in terms of the z distribution and in terms of the raw mean so that statistical power can be calculated.
5.In step three we calculate the statistical power or the percentage of the distribution of means for population 2 (the distribution centered on the hypothesized sample mean) that falls above the critical value.
Discussion Question 8-5
What is statistical power, and how would you calculate it?
Your students’ answers should include:
Statistical power is the probability of rejecting the null hypothesis when it is false.
You calculate statistical power in three steps. First determine the characteristics of the two populations. Next, calculate the raw mean value that determines your cutoff values. Finally, determine the percentage that falls above the raw mean and at the cutoff value using population 2.
6.There are five ways that we can increase the power of a statistical test. First, we can increase alpha. Second, we could turn a two-tailed hypothesis into a one-tailed hypothesis. Third, we could increase N. Fourth, we could exaggerate the levels of the independent variable. Lastly, we could decrease the standard deviation.
Discussion Question 8-6
What are ways that you could increase statistical power?
Your students’ answers should include:
Three ways that you could increase your statistical power are:
Adapt a more lenient alpha level.
Use a one-tailed test in place of a two-tailed test.
Increase the size of the sample.
Exaggerate the levels of the independent variable.
Decrease the standard deviation.
Classroom Activity 8-2
Working with Confidence Intervals and Effect Size
For this activity, you will need to have the class take a sample IQ test. You can find many examples of abbreviated IQ tests online ( is one such site). Have students anonymously submit their scores and compare the class data to data for the general population (population mean = 100, population standard deviation = 15). Using these data:
Have students calculate the confidence interval for the analysis.
Have students calculate the effect size.
Use Handout 8-1, found at the end of this chapter, to complete the activity.
V.Next Steps: Meta-Analysis
1.A meta-analysis is a study that involves the calculation of a mean effect size from the individual effect sizes of many studies.
2.A meta-analysis can provide added statistical power by considering many studies at once. In addition, a meta-analysis can help to resolve debates fueled by contradictory research findings.
Discussion Question 8-7
What is a meta-analysis and why is it useful?
Your students’ answers should include:
A study that involves the calculation of a mean effect size from the individual effect sizes of many studies.
It is useful because it considers many studies at once and helps to resolve debates fueled by contradictory research findings.
3.The first step in a meta-analysis is to choose the topic and make a list of criteria for which studies will be included.
4.Our next step is to gather every study that can be found on a given topic and calculate an effect size for every study that was found.
5.Lastly, we calculate statistics—ideally, summary statistics, a hypothesis test, a confidence interval, and a visual display of the effect sizes.
6.A file-drawer analysis is a statistical calculation following a meta-analysis of the number of studies with null results that would have to exist so that a mean effect size is no longer statistically significant.
Classroom Activity 8-3
Analyzing Meta-Analyses
Directions: In this activity, have students find a meta-analysis within the psychological literature. You may want to point them in the right direction by suggesting journals that will typically publish meta-analyses such as Psychological Bulletin, Personality and Social Psychology Review, or Journal of Applied Psychology. Once students have found their meta-analysis, they should answer the questions in Handout 8-2.
Additional Readings
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum.
This is arguably the definitive source for power analysis. Many of the procedural guidelines for determining power that are useful in many types of research design are clearly laid out in this text.
Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London. Series A, 236, 333–380.
This is considered the seminal paper for confidence intervals.
Rosenthal, R. (1994). Parametric measures of effect size. In Cooper, H., and Hedges, L. V. (Eds.), The handbook of research synthesis (pp. 231–244). New York: Russell Sage Foundation.
A very readable account of many of the techniques for calculating effect sizes. The chapter also includes a lot of background information about these techniques and how to interpret them.
Online Resources
This is an excellent Web site with numerous statistical demonstrations that you can run in your classroom to help explain the concepts concretely: Here you will find demonstrations of effect size, goodness of fit, and power.
Math World is an excellent and extensive resource site, providing background information and succinct explanations for all of the statistical concepts covered in the textbook and beyond.
PLEASE NOTE: Due to formatting, the Handouts are only available in AdobePDF®.