Confidence Interval for Standardized Difference Between Means, Related Samples

Confidence Interval for Standardized Difference Between Means, Related Samples

The Point Estimate

Cohen defined dz as the mean difference score divided by the standard deviation of the difference scores. The “z” was added to note that the unit of analysis is no longer X (the one mean) or Y (the other mean) but their difference, Z. While I and others (Dunlap, Cortina, Vaslow, & Burke, 1996) think it is most often not a good idea to use dz as an effect size estimator, there may be cases where it is defensible. In such cases the confidence interval for dz could be obtained in exactly the same manner that one would obtain the confidence interval for d for a one sample test.

Why not use the standard deviation of the difference scores as the standardizer? Think about the  in the variance sum law: . Now suppose that you found a difference on a test of clerical accuracy between individuals tested after taking a placebo and individuals tested after taking a sedating antihistamine. The difference in means is 3. Within each group the standard deviation is 10. You have independent samples. The d is 3/10 = .3. Now suppose that you had correlated samples with the same difference in means and the same within-condition standard deviations. The correlation between samples will result in the standard deviation of the difference scores being less than the standard deviation within each group (again, remember the  in the variance sum law). Suppose it is reduced to a value of 7. If you were to use the standard deviation of the difference scores as the standardizer when estimating d, you would obtain a value of 3/7 = .43 – but the difference between the means is still just 3 points. Why should d = .3 with independent samples but .43 with correlated samples when the difference between means is the same in both cases and the within-conditions standard deviations the same as well?

Programs to compute the point estimate:

Excel
SAS
SPSS

The Confidence Interval

You cannot use my Conf_Interval-d2.sas program (or my similar SPSS program) to construct a confidence interval for d when the data are from correlated samples. With correlated samples the distributions are very complex, not following the noncentralt. You can construct an approximate confidence interval,, where SE is, but such a confidence interval is not very accurate unless you have rather large sample sizes. I I got this from Kline’s Beyond Significance Testing, Chapter 4. I do have an SAS program that uses this algorithm.

AlginaKesselman (2003) provided a new method for computing confidence intervals for the standardized difference between means. I shall illustrate that method here.

Run this SAS code:

optionspageno=min nodateformdlim='-';

******************************************************************************;

title'Experiment 2 of Karl''s Dissertation';

title2'Correlated t-tests, Visits to Mus Tunnel vs Rat Tunnel, Three Nursing Groups '; run;

dataMus; infile'C:\D\StatData\tunnel2.dat';

inputnurs $ 1-2 L1 3-5 L2 6-8 t1 9-11 t2 12-14v_mus15-16v_rat17-18;

v_diff=v_mus - v_rat;

procmeansmeanstddevnskewnesskurtosistprt;

varv_musV_ratv_diff;

run;

*****************************************************************************;

ProcCorrCov; varV_MusV_Rat; run;

Get this output

------

Experiment 2 of Karl's Dissertation 1

Correlated t-tests, Visits to Mus Tunnel vs Rat Tunnel, Three Nursing Groups

The MEANS Procedure

Variable Mean Std Dev N Skewness Kurtosis t Value Pr > |t|

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

v_mus 22.7291667 10.4428324 48 -0.2582662 -0.3846768 15.08 <.0001

v_rat 13.2916667 10.4656649 48 0.8021143 0.2278957 8.80 <.0001

v_diff 9.4375000 11.6343334 48 -0.0905790 -0.6141338 5.62 <.0001

ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

The CORR Procedure

Covariance Matrix, DF = 47

v_mus v_rat

v_mus 109.0527482 41.6125887

v_rat 41.6125887 109.5301418

Pearson Correlation Coefficients, N = 48

Prob > |r| under H0: Rho=0

v_mus v_rat

v_rat 0.38075 1.00000

0.0076

To obtain our point estimate of the standardized difference between mean, run this SAS code:

DataD;

M1 = 22.72917 ; SD1 = 10.44283;

M2 = 13.29167 ; SD2 = 10.46566;

d = (m1-m2) / SQRT(.5*(sd1*sd1+sd2*sd2)); run;

Procprint; vard; run;

Obtain this result:

Obs d

1 0.90274

Using the approximation method first presented, . A 95% CI is .90274(1.96)(.18567) = [.539, 1.267].

From James Algina’s webpage, I obtained a simple SAS program for the confidence interval. Here is it, with values from above.

*This program computes an approximate CI for the effect

size in a within-subjects design with two groups.

m2 and m1 are the means for the two groups

s1 and s2 are the standard deviations for the two groups

n1 and n2 are the sample sizes for the two groups

r is the correlation

prob is the confidence level;

data;

m1=22.7291667 ;

m2=13.2916667 ;

s1=10.4428324 ;

s2=10.4656649 ;

r= 0.38075 ;

n=48 ;

prob=.95 ;

v1=s1**2;

v2=s2**2;

s12=s1*s2*r;

se=sqrt((v1+v2-2*s12)/n);

pvar=(v1+v2)/2;

nchat=(m1-m2)/se;

es=(m1-m2)/(sqrt(pvar));

df=n-1;

ncu=TNONCT(nchat,df,(1-prob)/2);

ncl=TNONCT(nchat,df,1-(1-prob)/2);

ul=se*ncu/(sqrt(pvar));

ll=se*ncl/(sqrt(pvar));

output;

procprint;

title1'll is the lower limit and ul is the upper limit';

title2'of a confidence interval for the effect size';

varesllul ;

run;

quit;

Here is the result:

ll is the lower limit and ul is the upper limit 4

of a confidence interval for the effect size

Obs es ll ul

1 0.90274 0.53546 1.26275

The program presented by AlginaKeselman (2003) is available at another of Algina’s web pages . This program will compute confidence intervals for one or more standardized contrasts between related means, with or without a Bonferroni correction, and with or without pooling the variances across all groups. Here is code, modified to compare the two related means from above.

* This program is used with within-subjects designs. It computes

confidence intervals for effect size estimates. To use the program one

inputs at the top of the program:

m--a vector of means

v--a covariance matrix in lower diagonal form, with periods for

the upper elements

n--the sample size

prob--the confidence level prior to the Bonferroni adjustment

adjust--the number of contrats is a Bonferroni adjustment to the

confidence level is requested. Otherwise adjust is set

equal to 1

Bird--a switch that uses the variances of all variables to calculate

the denominator of the effect size as suggested by K. Bird

(Bird=1). Our suggestion is to use the variance of those

variables involved in the contrast to calculate the denominator

of the effect size (Bird=0)

In addition one inputs at the bottom of the program:

c--a vector of contrast weights

multiple contrasts can be entered. After each, type the code

run ci;

prociml;

m={ 22.7291713.29167};

v={109.0527482. ,

41.6125887109.5301418};

do ii = 1tonrow(v)-1;

dojj = ii+1tonrow(v);

v[ii,jj]=v[jj,ii];

end;

n=48 ;

Bird=1;

df=n-1;

cl=.95;

adjust=1;

prob=cl;

print'Vector of means:';

print m;

print'Covariance matrix:';

print v;

print'Sample size:';

print n;

print'Confidence level before Bonferroni adjustment:';

print cl;

cl=1-(1-prob)/adjust;

print'Confidence level with Bonferroni adjustment:';

print cl;

start CI;

pvar=0;

count=0;

if bird=1then do;

do mm=1tonrow(v);

if c[1,mm]^=0 then do;

pvar=pvar+v[mm,mm];

count=count+1;

end;

if bird=0then do;

do mm=1tonrow(v);

pvar=pvar+v[mm,mm];

count=count+1;

end;

pvar=pvar/count;

es=m*c`/(sqrt(pvar));

se=sqrt(c*v*c`/n);

nchat=m*c`/se;

ncu=TNONCT(nchat,df,(1-prob)/(2*adjust));

ncl=TNONCT(nchat,df,1-(1-prob)/(2*adjust));

ll=se*ncl/(sqrt(pvar));

ul=se*ncu/(sqrt(pvar));

print'Contrast vector';

print c;

print'Effect size:';

printes;

Print'Estimated noncentrality parameter';

printnchat;

Print'll is the lower limit of the CI and ul is the upper limit';

printllul;

finish;

c={1 -1};

run ci;

quit;

Here is the output:

ll is the lower limit and ul is the upper limit

of a confidence interval for the effect size

Vector of means:

22.72917 13.29167

Covariance matrix:

109.05275 41.612589

41.612589 109.53014

Sample size:

Confidence level before Bonferroni adjustment:

0.95

Confidence level with Bonferroni adjustment:

0.95

Contrast vector

1 -1

Effect size:

------

ll is the lower limit and ul is the upper limit

of a confidence interval for the effect size

0.9027425

Estimated noncentrality parameter

nchat

5.619997

ll is the lower limit of the CI and ul is the upper limit

ll ul

0.5354583 1.2627535

Notice that this produces the same CI produced by the shorter program.