Data Analysis Tutorial
(See teleassessment paper and DataAnalysis.doc for examples)
References
[1] Anderson, M, Whitcomb, P, DOE Simplified: Practical Tools for Effective Experimentation, Productivity Inc., 2000.
[2] Montgomery, D, Runger, G, Applied Statistics and Probability for Engineers, Wiley, 1994.
[3] Montgomery, D, Design and Analysis of Experiments, 5th ed, Wiley, 2001.
[4] Portney, LG, Watkins, MP, Foundations of Clinical Research: Applications to Practice, Appleton & Lange, 1993
[5] Glantz, SA, Primer of Biostatistics, 4th ed, McGraw Hill, 1997.
Web resources
Excel for one-way ANOVA
http://www.mnstate.edu/wasson/ed602excelss13.htm
http://www.sytsma.com/phad530/aovexel.html
http://www.isixsigma.com/library/content/c021111a.asp
Excel for independent t-test
http://www.mnstate.edu/wasson/ed602excelss11.htm
Excel for dependent t-test
http://www.mnstate.edu/wasson/ed602excelss12.htm
Power analysis
http://www.biostat.ucsf.edu/sampsize.html
Calculating effect size indices
http://web.uccs.edu/lbecker/Psy590/es.htm
G-Power freeware for power analysis
http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/
Engineering statistics handbook
http://www.itl.nist.gov/div898/handbook/index.htm
UMN Stats Consulting Service (exc!!)
http://www.stat.umn.edu/consulting/
Tutorials
Tutorial on Repeated Measures (Blocked) Design and Analysis
Refs: [4] 374-377, 387-390
[3] 126-133
[2] 660-667
[1] 34-35
For repeated measures design [4] (or blocked analysis [1]). Uses calculation method of [4] which is the same as [2] and [3]. See Table 20.4 in [4] for an example.
(For alternate method, see tutorial in Appendix and [1] Tables 2-3 and 2-4)
All calculations and tables created in Excel. See end of tutorial for letting built-in Excel tools do the hard work.
For example, see file repeatedmeasuresexample.xls
1. Determine what is a block. Blocks can be subjects or anything else where variation is expected. Every treatment must be conducted once and only once in every block.
2. Create data table with one row for each block and one column for each treatment. For example, see [4] Table 20.4. [2] and [3] have tables in transpose order.
3. Create column for sums of rows, Sum(Si) and create row for sums of columns, Sum(Xi)
4. Compute from the table. See [4] Table 20.4 for math formulas. See Excel example file for computation formulas
n = number of blocks
k = number of treatments
Sum(X) = sum of all data points
Sum(X^2) = sum of all squared data points
Sum(X)*2/nk =
Sum(colsum^2) =
Sum(rowsum^2) =
SST = Total sum of squares
SSTr = Treatment sum of squares
SSBl = Block sum of squares
SSE = Error sum of squares
DFTr = Treatment degrees of freedom = (k-1)
DFBl = Block degrees of freedom = (n-1)
DFE = Error degrees of freedom = (k-1)*(n-1)
MSTr = Treatment mean square = SSTr/DFTr
MSE = Error mean square = SSE/DFE
F = F stat = MSTr/MSE
Fcrit is from alpha=.05 F stat table with DFnum=DFTr, DFden=DFE
Or, rather than doing all this by hand, in Excel-speak, a repeated measures study can be analyzed with the “ANOVA: Two-factor without replication” tool. This is much simpler than the above and also gives exact values for F crit for any DF and gives prob values.
To use Excel:
Create data table with one row for each block and one column for each treatment. Label rows and columns. See bottom of file repeatedmeasuresexample.xls
Tools > Data Analysis… > ANOVA: Two-factor without replication (If Data Analysis is not available, Tools > Add Ins… > Analysis ToolPak)
Click in Input Range, then highlight data, including the header rows.
Check Labels
Click in Output Range, then in an empty cell to mark the top left of the results box.
Click OK
Because the treatments are the columns, the F value of interest is in the row marked “Columns”
The P value is the probability that the treatments are equal.
The Fcrit value is the 0.05 confidence level
Above the ANOVA, the Average column gives the mean for each treatment.
Interpret results: If F > Fcrit, treatment means are significantly different and can reject null hypothesis with 95% confidence.
If F > Crit F, treatment means are significant for P < 0.05
If treatment means are different, perform a multiple comparison test to determine which of the treatments are different.
Compute Scheffe comparison ([4] p 405)
Scheffe min sig difference =
k = number of treatments
Fcrit is from the Excel result or from a table
MSE is the MS value for the Error row
n = number of blocks = number of samples in each group.
For two treatments to be different, their means must differ by more than Scheffe.
Scheffe will avoid Type I errors, but has little power; it is a conservative measure. ([4] p. 417).
For good balance between power and Type I error, try Tukey’s HSD ([4] p. 401)
T HSD min sig diff =
MSE = mean square error
n = number of subjects in each group = number of blocks
q is from Table A.6 in [4] (p. 622) In the table r = number of treatments, DFe is error dof
To present comparisons, arrange means in ascending order and underline those that are not different ([4] p. 408).
Paired T-Test Tutorial
The paired t-test is run with the Excel “t-test: Paired Two Sample for Means” Data Analysis tool. Line up data in two side-by-side columns. The null hypothesis is that the two treatments are equal. The results will appear in the designated output range. There is a significant difference between treatments if the absolute value of “t Stat” is greater than “t Critical two-tail”, or if “P(T<=t) two-tail” is less than 0.05.
The paired two tailed t-test is the same as a repeated measures ANOVA, so instead of a paired t-test, can run the “ANOVA: Two-factor without replication” Data Analysis tool. The two treatments are different if for the row labeled Columns, F > F crit. The P-value is the same as the “P(T<=t) two-tail” in the paired t-test.
Tutorial on Power analysis for 2-tailed T-test
Following Appendix C in [1]. Also see [5] pp. 158-180, particularly Fig. 6-6 on p. 164.
You want to analyze the power of the study to detect a difference. What you would like to draw is a "power function" which is a plot of the power (00 to 100%) of a t test to detect a difference on the vertical axis against the difference on the horizontal axis (in the units of interest, e.g. Newton-meters for muscle torque).
Assume that comparisons are via 2-tailed T-test and that acceptance level is alpha=0.5.
Go to table C.2
Pick off the line for N, the number of subjects in each group for the study (22/2 = 11?). If different number of subjects in each group, compute N via Equation C.7.
For the sake of discussion, assume N=11.
Create an Excel table for this row with one column of D values and one column of Pwr values.
Plot (see enclosed)
For example, this power function plot says that for N=11, if the actual effect size index of the test was D=0.5, the power to detect the difference was 31%. Or, to detect a difference with power of 80%, the effect size index would have to be somewhere between 1.0 and 1.2.
"Effect size index" D is not a particularly useful number, so translate the D column into a column of effect sizes in the units you care about. Equation C.1 shows how to do this. For this you need to know s, the common std deviation. Calculate s using Eq. C.2 where s1 and s2 are the std deviations for each population (Eq 17.5 and 17.6). The new column of numbers is d*s and is the "effect size" which is the difference in means.
Create a new plot which is exactly what you want, a plot of the power of the test to pick out differences in the means.
For unpaired t-test, d =
d = effect size index
Xbar1- Xbar2 = diff in group means, or the difference you want to detect
s = pooled std deviation = sqrt(variance)
if variances not equal, s =
For paired t-test, must adjust the effect size index d. See [4] page 653. gPower cannot be used to automatically calculate effect size for paired t-tests
dbar = mean of difference scores, or the desired difference you want to detect
sd = std dev of difference scores
This formula is easy because you just create a column of data differences and compute the SD.
Another formula (Eq. C.4) is
where r = correlation coefficient for paired data and
Power calculations for ANOVA
See [4] p. 655
Effect size index = f
Eq. C.9 shows how to calculate f
Use pooled standard deviation (=SD=sigma) for s
K = number of treatments (groups)
Hopefully there are = sample sizes
For post hoc, Sm=diff to detect =difference from grand mean that would be useful to detect. So if you don’t care about any particular treatment, this is really the same as eq. C.1.
Calculations
Use gPower, a freeware program for power analysis instead of looking up numbers in the table. See next section.
Tutorial on gPower
gPower is a freeware program for power analysis. See References for where to download. Read the on-line tutorial, after you read Appendix C in [1] and the tutorial above.
For post-hoc analysis, use to determine the power of the test to detect a particular difference given the known deviations and known subject size.
Choose Tests > Means for t-test power analysis.
Post hoc, Two-tailed.
Calculate Effect size d from above tutorial
Alpha=.05
Sample size: n1, n2 are number in each group.
Push Calculate.
Looking for 80% or better power, the higher the power the better.
Choose Tests > F-test (ANOVA) for anova power analysis
Calculate effect size f from above tutorial. If you care about specific means to detect, use the Calculate Effect size tool in gPower or crunch eq. C.10
Alpha=.05
Groups=number of treatments
Total samples=treatments*number of samples per group=total number of data points.
For a more powerful Power Analysis program, try PASS from NCSS (http://www.ncss.com/pass.html) Expensive, but covers all designs. Prof. Richard DiFabio in PT has a copy.
Appendix
Blocked Design Tutorial using methods in [1]
Refs: [4] 374-377, 387-390
[3] 126-133
[2] 660-667
[1] 34-35
For repeated measures design [4] (or blocked analysis [1]). Uses calculation method of [1] which is simple to understand, with language of [4] which is more standard. See Table 20.4 in [4] and Tables 2-3 and 2-4 in [1] for example. Note: do not using this method, but using method described in tutorial in the main body above
5. Create data table with one row for each treatment and one column for each block where blocks can be subjects or anything else where variation is expected.
6. For mean use AVERAGE(), for variance use VAR().
7. Create row of Means to compute within block means.
8. Create column of Means to compute within treatment means.
9. Create column of Variance to compute within treatment variances
10. Compute grand mean (= mean of column of means)
11. Create row of Diff = within block mean – grand mean
12. Create new table where each data entry is data – Diff
13. Compute within treatment means and within treatment variances
F statistic is where MSt is mean square of treatment and MSe is mean square of error. MS = SS/df where SS = sum of squares and df = degrees of freedom. Formulas for SS are in Table 20.4, but [1] provides a simpler computation. MSe is also the variance of blocks pooled. Sqrt(MSe) is the pooled standard deviation ([1] p. 30).
14. Compute from the new table:
If F > Crit F, treatment means are significant for P < 0.05
11. Compute Fisher Least Significance Difference
See example at bottom of blocktestdata.xls.
Note: made file blocktestdata.xls that follows Table 2-3 example in [1] to verify methods.
For Table 2-3 without blocking, DF num = number of treatments minus 1 = 6-1=5. DF den is total number of observations (6*4=24) minus DF used to compute means = 6, DF=24-6=18. See [1] p30
For Table 2-4 in [1], DFnum = 6-1 = 5 (number of treatments minus 1). DFden=(n-1)*(k-1) (number of treatments minus 1 times number of blocks (subjects) minus 1) = 5*3=15. See [4] p. 390.
For Table 2-4 in [1], the mean of the within dot variance is sum(var)/(k-1) = sum(var)/5. In Table 2-3, it is calculated as sum(var)/k. This distinction is not clear from the text.
For blocktestdata.xls, used equation on p. 34 in [1] to compute F, and Tables in back of [1] to find F crit..
At bottom of blocktestdata.xls, used data from [4] Table 20.4, p. 390 to test if their method is same as equations for [1]. Note row/column reversal from book table. Get same answer for F.
StatsTutorial.doc Page 8 of 8 7/30/2004