STAT 405 - BIOSTATISTICS
Handout 4 – Power for Tests of a Single Binomial Proportion
Up to this point, we have reviewed the use of the binomial distribution for testing hypotheses concerning a proportion and for constructing confidence intervals. In this handout, we will consider the issues of power and sample size. Your text discusses power calculations based on the normal distribution in Section 7.10 of your text (we will also use the binomial distribution in this handout).
EXAMPLE: Recall the Cancer Study from the previous handout. We noted that 13 deaths occurred among 55-64 year-old male workers in a nuclear-power plant and that in 5 of them the cause of death was cancer. Based on vital-statistics reports, approximately 20% of all deaths were attributed to some form of cancer. Our hypothesis test did NOT provide evidence that the proportion of deaths from cancer in nuclear-power plant workers was greater than the proportion of deaths from cancer in men of comparable age in the general population (p-value = .0991).
Questions:
1. Is it possible that the cancer mortality rate is in fact greater for the study population than the general population, and our test simply failed to detect this? Explain.
2. Do you think the sample size of 13 is large enough to uncover a difference if one does in fact exist? Why or why not?
This is a question of power, which is the focus of this handout. First, let’s review some basics of hypothesis testing.
When testing for a difference,there are two possible outcomes: / Ideally, these outcomes would occur:
1. We find evidence that a difference exists. / 1. When a difference does in fact exist.
2. We fail to find evidence that a difference exists. / 2. When there is not a difference of practical significance.
However, we know that hypothesis tests are not error proof! The following table summarizes all possible scenarios in carrying out a hypothesis test (the probability of each is listed in parentheses):
Null is True / Alternative is TrueReject the Null
Fail to Reject the Null
Note that two types of errors exist:
v Type I Error: This occurs when we falsely reject the null hypothesis. That is, we reject the null when the null is true. The probability of committing this error is given by α.
Note that we control the Type I error rate by our selection of α prior to conducting an experiment.
v Type II Error: This occurs when we fail to reject the null when a particular alternative scenario is true. The probability of committing this error is given by β.
Our control of the Type II error rate depends on the sample size, and to some
degree on the test used (most powerful test – STAT 460).
The table shown above also describes scenarios in which we do not commit errors. For example, note that one possible scenario is to reject the null when the alternative is true. This is a good outcome, and it leads into the concept of power.
Power: By definition, the power of a test is the probability of correctly rejecting the null hypothesis if a particular alternative scenario is true. In other words, the power is the probability of rejecting the null hypothesis given that a specific difference does exist. This is calculated as 1- β.
Back to the Cancer Example:
Recall that we used a sample size of 13 in the study. How likely is the hypothesis test to detect that the cancer mortality rate is greater in the study population than in the general population, if this is in fact really the case? To answer this question, we must calculate the power of this test. First, consider our hypotheses:
H0: p = .20
Ha: p .20
To find the power, we need to find the probability of rejecting the null given that the alternative is true. However, note that our alternative hypothesis is very general—there are infinitely many scenarios in which the true proportion could be greater than 20%! It turns out that in order to calculate the power, we need to define a PARTICULAR alternative scenario. That is, we must specify the size of the difference that we want the test to detect.
Calculating the Power for the Binomial Test
The following list summarizes what is needed for a power calculation for a binomial test:
1. A specific alternative (i.e., the size of the difference we want to detect).
2. The sample size (i.e., the number of replications for each treatment).
3. The Type I error rate (α).
The power calculations for the Cancer example are shown below.
Power = P(reject the null hypothesis | a particular alternative scenario is true)
Step 1: / Pick a specific alternative. We will find the chance of finding a significant result (using a sample size of 13) if the true rate of cancer death for the 55-64 year old male nuclear-power plant workers is actually twice as high as that of the typical 55-64 year old male (40%).Step 2: / Find the values of the binomial random variable for which the null hypothesis will be rejected.
You can use SAS to find this “critical value”:
data Rejection_Region;
do x = 0 to 13;
cutoff = 1-cdf('binomial', x-1, .2, 13); output;
end;
proc print noobs; run;
Step 3: / Note that we will reject the null hypothesis if the number of cancer deaths is 6 or more. Now, find the probability of observing 6 OR MORE cancer deaths given that the true proportion of cancer deaths in the 55-64 male nuclear-power plant workers is actually 40%. Why?
Power = P(reject Ho | a particular alternative scenario is true)
The following graphic shows a binomial distribution with n=13 and p = .40.
We can use SAS to calculate the probability of 6 or more cancer deaths:
data CalcPower;
power=1-cdf('binomial', 5, .4, 13);
run;
proc print; run;
Interpretation: There is a 42.6% chance of finding a significant result, based on a sample size of 13, if the true rate of cancer deaths among 55-64 year old male nuclear-power plant workers is actually twice as high as that of typical 55-64 year old males.
Making the Power Calculations Simpler!
SAS PROC POWER can be used for the calculations. For the power of tests for a binomial proportion, you can use the following code:
proc power;
onesamplefreq test=exact
sides = 1
nullproportion = 0.20
proportion = 0.40
ntotal = 13
power = .;
run;
Compare this result to that based on normal-theory approximations using SAS:
proc power;
onesamplefreq test=z method=normal
sides = 1
nullproportion = 0.20
proportion = 0.40
ntotal = 13
power = .;
run;
Calculating the Power for the Binomial Test Using Normal-Theory Methods
Again, we will consider the power calculations for the Cancer Study example. The power based on normal-theory methods was already calculated on the previous page using PROC POWER; here, we will take a closer look into the calculations.
Step 1: / Pick a specific alternative. Once again, we will find the chance of finding a significant result (using a sample size of 13) if the true rate of cancer death for the 55-64 year old male nuclear-power plant workers is actually twice as high as that of the typical 55-64 year old male (40%).Step 2: / Using the normal approximation to the binomial distribution, find the values for which the null hypothesis will be rejected.
Recall the following for a binomial random variable, X:
v Mean =
v Standard Deviation =
We can derive the following for the sample proportion, X/n:
v Mean =
v Standard Deviation =
v Z-value =
Step 3: / Power = P(reject Ho | a particular alternative scenario is true)
Next, we find the probability of our observed Z-value falling in the rejection region given that the true proportion of cancer deaths in the 55-64 male nuclear-power plant workers is actually 40%.
That is, we must find P(p > .3825) using a normal distribution with
v Mean =
v Standard Deviation =
This probability is illustrated in the following graphic:
To find this normal approximation probability in R, you can use the following command:
> 1 - pnorm(.3825,mean=.4,sd=.1358)
[1] 0.5512682
Note that this matches the power calculated using PROC POWER.
Finding the necessary sample size
In many instances, researchers desire to find the necessary sample size to achieve a certain power BEFORE the study is carried out. PROC POWER makes this process very easy if you are willing to use normal-theory methods. For example, once again consider the Cancer Study. Suppose we want to achieve 80% power:
proc power;
onesamplefreq test=z method=normal
sides = 1
nullproportion = 0.20
proportion = 0.40
ntotal = .
power = .80;
run;
Next, calculate the power assuming this sample size using exact binomial probabilities in SAS and in JMP:
proc power;
onesamplefreq test=exact
sides = 1
nullproportion = 0.20
proportion = 0.40
ntotal = 29
power = .;
run;
Power for a Two-Sided Test
Suppose we wanted to show that the proportion of deaths from cancer in nuclear-power plant workers was DIFFERENT FROM the proportion of deaths from cancer in men of comparable age in the general population. The power calculations for this two-sided test are shown below.
Power = P(reject the null hypothesis | a particular alternative scenario is true)
Step 1: / Pick a specific alternative. We will find the chance of finding a significant difference (using a sample size of 13) if the true rate of cancer death for the 55-64 year old male nuclear-power plant workers is actually twice as high as that of the typical 55-64 year old male (40%).Step 2: / Find the values of the binomial random variable for which the null hypothesis will be rejected.
Find the “critical value”:
data Rejection_Region;
do x = 0 to 13;
cutoff = 1-cdf('binomial', x-1, .2, 13); output;
end;
proc print noobs; run;
Step 3: / Note that we will reject the null hypothesis if the number of cancer deaths exceeds 7. Now, find the probability of observing 7 OR MORE cancer deaths given that the true proportion of cancer deaths in the 55-64 male nuclear-power plant workers is actually 40%.
The following graphic shows a binomial distribution with n=13 and p = .40.
We can use SAS to calculate the probability of 7 or more cancer deaths:
data CalcPower;
power=1-cdf('binomial', 6, .4, 13);
run;
proc print noobs; run;
In SAS PROC POWER:
proc power;
onesamplefreq test=exact
nullproportion = 0.20
proportion = 0.40
sides = 2
ntotal = 13
power = .;
run;
The details for determining power using normal theory methods:
41