Bootstrap Example:
Data: 0, 2, 4, 10
Goal: CI for mean and median
Sample Mean = 4
Sample Median = 3
Number of Possible Bootstrap Samples = 256
Exact Boostrap Histograms:
95% CI for Mean = [1, 8] Uses 2.5% and 97.5% points which are between the
95% CI for Median = [1,10] 6th-7th and 250th-251st bootstrap values respectively
IN STATA: Bootstrap CIs for the mean
Bootstrap results Number of obs = 4
Replications = 2000
------
| Observed Bootstrap
| Coef. Bias Std. Err. [95% Conf. Interval]
------+------
Bootmean | 4 -.046 1.92 .23 7.77 (N)
| 1 8 (P)
| 1 8.5 (BC)
------
(N) normal confidence interval
(P) percentile confidence interval
(BC) bias-corrected confidence interval
IN STATA: Boostrap CIs for the median
Bootstrap results Number of obs = 4
Replications = 2000
------
| Observed Bootstrap
| Coef. Bias Std. Err. [95% Conf. Interval]
------+------
Bootmed | 3 .62 2.41 -1.73 7.73 (N)
| 0 10 (P)
| 1 10 (BC)
------
(N) normal confidence interval
(P) percentile confidence interval
(BC) bias-corrected confidence interval
Note that STATA gives 3 bootstrap confidence intervals. The first assumes the sample statistic is normally distributed so that the interval takes the form estimate ±2*BootSE. The second is the percentile interval we focused on in class. The third is a bias-corrected version. For the mean the normal confidence interval is not that different from the percentile and bias corrected intervals. However for the median it is very different and is by far the worst choice. As we can see from the histogram of all bootstrap samples the sample median has a very skewed distribution!
In this particular example, normality of the sample looks questionable (the final value of 10 is much higher than the others) and the sample size is too small for the central limit theorem to have kicked in to make the sample mean normal. Thus we may not want to use the standard normal theory CI for the mean. Of course for the median we do not even know a standard error formula. Thus in both cases a bootstrap CI may be a good choice. Note that in both cases the bootstrap CI is very wide, encompassing almost the whole range of the data set. This reflects the fact that the sample size is very small and we can not get very precise confidence intervals! The bootstrap does NOT save you from having small sample sizes.
Finally, in this problem because the sample size was so small it was possible to actually calculate the mean and median for every possible bootstrap sample. In general, however, the number of possible bootstrap samples will be too large and we will simply obtain a random selection of them. In the STATA printouts I had the computer pick 2000 bootstrap samples (generally a good choice of B for confidence intervals.) Since this is more than the number of possible bootstrap samples some of them just got repeated. Since STATA is randomly selecting bootstrap samples its results do not have to be the same as for my exact bootstrap distribution but with 2000 bootstrap samples I have a good approximation and the results do in fact match exactly for the percentile intervals.