Why Psychologists Should by Default Use Welch's T-Test Instead of Student's T-Test

ADDITIONAL FILE

Additional file to

Why Psychologists Should by Default Use Welch's t-test Instead of Student's t-test.

Marie Delacre, Université Libre de Bruxelles, Service of Analysis of the Data (SAD), Bruxelles, Belgium, e-mail:

Daniël Lakens, Eindhoven University of Technology, Human Technology Interaction Group, Eindhoven, Netherlands, email:

Christophe Leys, Université Libre de Bruxelles, Service of Analysis of the Data (SAD), Bruxelles, Belgium, e-mail:

Type 1 error rate

In order to estimate the Type 1 error rate for Student’s t-test and Welch’s t-test, we simulated 1,000,000 simulations of two samples generated under 20 conditions (yielding 20,000,000 simulations in total). In each condition, the first sample was generated from a population where and its sample size varied from 10 to 50 in a step of 10. The standard deviation of the second sample was a function of the SDR being respectively 0.01, 0.1, 10 and 100. The set of simulations was repeated nine times, varying the distribution underlying the data. We used R commands to generate data from different distributions.

-Two normal distributions (See Table A1.1): in order to assess the Type 1 error rate of both tests, under the normality assumption, data were generated by means of the function “rnorm” (from the package “stats”; “R: The Normal Distribution,” 2016).

-Two double exponential distributions (See Table A1.2): in order to assess the impact of high kurtosis on the Type 1 error rate of the three tests, data were generated by means of the function “rdoublex” (from the package “smoothmest”; “R: The double exponential (Laplace) distribution,” 2012).

-One normal and one double exponential distribution (See Table A1.3): in order to assess the impact of the unequal shape of distributions, in terms of kurtosis, on the Type 1 error rate, data were generated by means of the functions “rdoublex” and “rnorm”.

-Two uniform distributions (See Table A1.4): in order to assess the impact of low kurtosis on the Type 1 error rate of the three tests, data were generated by means of the function “runif” (from the package “stats”; “R: The Uniform Distribution,” 2016).

-One uniform and one normal distribution (See Table A1.5): in order to assess the impact of the unequal shape of distributions, in terms of kurtosis, on the Type 1 error rate, data were generated by means of the functions “runif” and “rnorm”.

-One uniform and one double exponential distribution (See Table A1.6): in order to assess the impact of the unequal shape of distributions, in terms of kurtosis, on the Type 1 error rate, data were generated by means of the functions “runif” and “rdoublex”.

-Two normal skewed distributions with positive skewness of 0.79 (See Table A1.7): in order to assess the impact of skewness on the Type 1 error rate, data were generated by means of the function “rsnorm” (from the package “fGarch”; “R: Skew Normal Distribution,” 2017). The normal skewed distribution was used because it is the only skewed distribution where the standard deviation ratio can vary without having an impact on skewness.

-One normal skewed distribution with negative skewness (-0.79) and one normal skewed distribution with positive skewness (+0.79;See Table A1.8): in order to assess the impact of unequal shapes, in terms of kurtosis, on the Type 1 error rate, when data are asymmetric, data were generated by means of the functions “rsnorm” with unequal skewness.

-One chi(2) and one normal skewed distribution with negative skewness of +0.79 (See Table A1.9): in order to assess the impact of high skewness and kurtosis on the Type 1 error rate, data were generated by means of the functions “rsnorm” with different skewness and from the function “rchisq”; “R: The (non-central) Chi-Squared Distribution,” 2016).

According to the definition of Bradley (1978), one consider that the alpha risk is “sufficiently close” to the nominal alpha risk if its value falls in the interval [0,025; 0,075] (Hayes & Cai, 2007). In the tables presented below, anytime the Type 1 error rate is out of this interval, we have added an “*” next to the value.

Simulations show that Student’s t-test is strongly robust to violations of the assumption of equal variances when sample sizes are the same between groups, even for big SDRs, as long as there are at least 10 subjects per groups and the distributions are symmetric. However, according to Gleason (2013): “the issue of being robust to departures from normality should not be confused with being the best statistic for the situation in question” (p.10). Even when Student’s Type 1 error rate is not critical, Welch’s Type 1 error rate is closer of the nominal alpha risk of 5%. This is particularly true with small sample sizes (when n1 = n2 = 10).

With moderately skewed distributions and big SDRs, Student’s t-test becomes too liberal while Welch’s t-test remains closer to the nominal Type 1 error rate when there are 10 subjects per group. With at least 20 subjects per group, both have an acceptable Type 1 error rate, but Welch’s t-test remains closer to the nominal Type 1 error rate.

With high levels of skewness and kurtosis, both tests become too liberal, when there are fewer than 30 subjects per group. With 30 subjects per group, Welch’s t-test have an acceptable Type 1 error rate while Student’s t-test still have a Type 1 error rate larger than 0.075. In all cases, Welch’s t-test remains closer to the nominal Type 1 error rate.

Power

In order to estimate the power for Student’s t-test and Welch’s t-test, we simulated 1,000,000 simulations of two samples generated under 20 conditions (yielding 20,000,000 simulations in total). In each condition, the first sample is generated from a population where and its sample size varies from 10 to 50 in a step of 10. The standard deviation of the second sample is a function of the SDRs being respectively 0.01, 0.1, 10 and 100. We included a mean difference δ = 1 (giving related Cohen’s effect sizes that varied from 0.29 to 0.71 which is realistic in the field of psychology). In the first step, the p-values of the two tests were extracted for each pair of samples and fromeach test, and in a second step, the percentage of p-value under the nominal alpha risk (5%) was computed for each test, giving the power.

In order to insure the reliability of our calculation method, we firstly used R commands to generate data from two normal distributions (See Table A1.1). Because computed power is very consistent with theoretical power, one can conclude that the method is reliable. The set of simulations was repeated nine times, varying the distribution underlying the data. We used R commands to generate data from different distributions.

-Two normal distributions (See Table A1.1)

-Two double exponential distributions (See Table A1.2)

-One normal and one double exponential distribution (See Table A1.3)

-Two uniform distributions (See Table A1.4)

-One uniform and one normal distribution (See Table A1.5)

-One uniform and one double exponential distribution (See Table A1.6)

-Two normal skewed distributions with positive skewness of 0.79 (See Table A1.7)

-One normal skewed distribution with negative skewness (-0.79) and one normal skewed distribution with positive skewness (+0.79;See Table A1.8)

-One chi(2) and one normal skewed distribution with negative skewness of +0.79 (See Table A1.9)

Simulations show that when sample sizes are equal between groups, Student’s power is better than Welch’s power, but very slightly. Even with extremely big SDRs (respectively 0.01,0.1,10 and 100) and small SDRs (10 subjects per group), the biggest advantage of Student’s t-test over Welch’s-t test is 5.37%, when the test is applied on two normal skewed distributions with unequal shapes. In all other cases, the difference between both tests is smaller (See Table A1.1 to A1.9).Moreover, the bigger the sample sizes are, the smaller are the gains of Student’s t-test in terms of power.

Conclusion

Even with equal sample sizes, Welch’s t-test is more robust than Student’s t-test in terms of alpha risk. On the other hand, the Student’s gain in power is very weak. In conclusion, we advise to always use Welch’s t-test, instead of Student’s t-test, particularly when distributions have moderate skewness.

Table A1.1

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.050 / 0.065 / 29.34 / 34.07
0.1 / 0.050 / 0.064 / 29.23 / 33.68
10 / 0.051 / 0.065 / 5.25 / 6.73
100 / 0.050 / 0.065 / 4.97 / 6.49
20 / 0.01 / 0.050 / 0.057 / 56.40 / 58.95
0.1 / 0.050 / 0.057 / 56.15 / 58.57
10 / 0.050 / 0.057 / 5.50 / 6.22
100 / 0.050 / 0.057 / 5.00 / 5.72
30 / 0.01 / 0.050 / 0.055 / 75.42 / 76.77
0.1 / 0.051 / 0.055 / 75.06 / 76.36
10 / 0.050 / 0.055 / 5.86 / 6.35
100 / 0.050 / 0.055 / 5.07 / 5.56
40 / 0.01 / 0.050 / 0.054 / 86.92 / 87.59
0.1 / 0.050 / 0.053 / 86.66 / 87.31
10 / 0.050 / 0.053 / 6.10 / 6.48
100 / 0.050 / 0.053 / 5.03 / 5.38
50 / 0.01 / 0.050 / 0.053 / 93.40 / 93.71
0.1 / 0.050 / 0.053 / 93.23 / 93.53
10 / 0.050 / 0.053 / 6.34 / 6.66
100 / 0.050 / 0.053 / 5.01 / 5.28

Table A1.2

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.042 / 0.057 / 20.70 / 24.50
0.1 / 0.042 / 0.056 / 20.60 / 24.10
10 / 0.042 / 0.056 / 4.40 / 5.90
100 / 0.042 / 0.057 / 4.20 / 5.70
20 / 0.01 / 0.046 / 0.054 / 35.90 / 38.10
0.1 / 0.047 / 0.054 / 35.70 / 37.80
10 / 0.047 / 0.054 / 5.00 / 5.70
100 / 0.046 / 0.054 / 4.60 / 5.40
30 / 0.01 / 0.048 / 0.052 / 49.10 / 50.70
0.1 / 0.048 / 0.052 / 49.00 / 50.50
10 / 0.048 / 0.052 / 5.20 / 5.70
100 / 0.048 / 0.053 / 4.80 / 5.30
40 / 0.01 / 0.048 / 0.052 / 60.30 / 61.40
0.1 / 0.048 / 0.052 / 60.00 / 61.00
10 / 0.049 / 0.052 / 5.40 / 5.80
100 / 0.049 / 0.052 / 4.80 / 5.20
50 / 0.01 / 0.049 / 0.052 / 69.70 / 70.60
0.1 / 0.049 / 0.052 / 69.30 / 70.00
10 / 0.049 / 0.051 / 5.60 / 5.90
100 / 0.049 / 0.052 / 4.90 / 5.20

Table A1.3

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.050 / 0.065 / 29.27 / 33.98
0.1 / 0.050 / 0.063 / 29.06 / 33.33
10 / 0.042 / 0.057 / 4.37 / 5.88
100 / 0.042 / 0.057 / 4.19 / 5.72
20 / 0.01 / 0.050 / 0.057 / 56.41 / 58.95
0.1 / 0.050 / 0.057 / 55.77 / 58.10
10 / 0.046 / 0.054 / 4.96 / 5.71
100 / 0.046 / 0.054 / 4.64 / 5.40
30 / 0.01 / 0.050 / 0.055 / 75.38 / 76.72
0.1 / 0.050 / 0.054 / 74.68 / 75.92
10 / 0.048 / 0.052 / 5.24 / 5.74
100 / 0.048 / 0.053 / 4.82 / 5.31
40 / 0.01 / 0.050 / 0.053 / 86.89 / 87.55
0.1 / 0.050 / 0.053 / 86.34 / 86.97
10 / 0.049 / 0.052 / 5.45 / 5.82
100 / 0.049 / 0.053 / 4.86 / 5.21
50 / 0.01 / 0.050 / 0.053 / 93.42 / 93.73
0.1 / 0.050 / 0.052 / 93.00 / 93.32
10 / 0.049 / 0.052 / 5.64 / 5.95
100 / 0.049 / 0.051 / 4.90 / 5.19

Table A1.4

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.054 / 0.068 / 26.09 / 30.77
0.1 / 0.054 / 0.068 / 25.96 / 30.45
10 / 0.054 / 0.068 / 5.60 / 6.96
100 / 0.055 / 0.069 / 5.39 / 6.79
20 / 0.01 / 0.051 / 0.058 / 54.78 / 57.45
0.1 / 0.052 / 0.058 / 54.47 / 57.05
10 / 0.052 / 0.059 / 5.61 / 6.33
100 / 0.051 / 0.058 / 5.14 / 5.85
30 / 0.01 / 0.051 / 0.055 / 75.09 / 76.48
0.1 / 0.051 / 0.055 / 74.71 / 76.04
10 / 0.051 / 0.055 / 5.84 / 6.32
100 / 0.051 / 0.055 / 5.08 / 5.55
40 / 0.01 / 0.051 / 0.054 / 87.14 / 87.83
0.1 / 0.051 / 0.054 / 86.88 / 87.54
10 / 0.051 / 0.054 / 6.11 / 6.49
100 / 0.051 / 0.054 / 5.09 / 5.44
50 / 0.01 / 0.051 / 0.053 / 93.72 / 94.04
0.1 / 0.051 / 0.053 / 93.50 / 93.80
10 / 0.050 / 0.053 / 6.37 / 6.68
100 / 0.050 / 0.053 / 5.06 / 5.34

Table A1.5

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.054 / 0.068 / 26.00 / 30.67
0.1 / 0.054 / 0.068 / 25.96 / 30.46
10 / 0.050 / 0.064 / 5.26 / 6.71
100 / 0.050 / 0.065 / 4.99 / 6.49
20 / 0.01 / 0.052 / 0.059 / 54.79 / 57.50
0.1 / 0.052 / 0.058 / 54.50 / 57.09
10 / 0.050 / 0.057 / 5.53 / 6.26
100 / 0.050 / 0.057 / 5.01 / 5.73
30 / 0.01 / 0.051 / 0.056 / 75.01 / 76.40
0.1 / 0.051 / 0.055 / 74.71 / 76.05
10 / 0.050 / 0.055 / 5.82 / 6.32
100 / 0.050 / 0.055 / 5.01 / 5.50
40 / 0.01 / 0.051 / 0.054 / 87.12 / 87.81
0.1 / 0.051 / 0.054 / 86.87 / 87.54
10 / 0.050 / 0.053 / 6.10 / 6.49
100 / 0.050 / 0.053 / 5.06 / 5.42
50 / 0.01 / 0.050 / 0.053 / 93.73 / 94.04
0.1 / 0.051 / 0.053 / 93.46 / 93.77
10 / 0.050 / 0.053 / 6.38 / 6.70
100 / 0.050 / 0.052 / 4.97 / 5.25

Table A1.6

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.054 / 0.068 / 25.93 / 30.59
0.1 / 0.054 / 0.067 / 25.92 / 30.21
10 / 0.042 / 0.057 / 4.40 / 5.87
100 / 0.042 / 0.057 / 4.21 / 5.72
20 / 0.01 / 0.052 / 0.058 / 54.74 / 57.41
0.1 / 0.052 / 0.058 / 54.20 / 56.68
10 / 0.046 / 0.053 / 4.96 / 5.72
100 / 0.046 / 0.054 / 4.62 / 5.34
30 / 0.01 / 0.051 / 0.055 / 75.10 / 76.48
0.1 / 0.051 / 0.055 / 74.32 / 75.63
10 / 0.048 / 0.052 / 5.21 / 5.71
100 / 0.048 / 0.053 / 4.80 / 5.29
40 / 0.01 / 0.051 / 0.054 / 87.18 / 87.85
0.1 / 0.051 / 0.054 / 86.50 / 87.14
10 / 0.049 / 0.052 / 5.43 / 5.80
100 / 0.049 / 0.052 / 4.87 / 5.22
50 / 0.01 / 0.050 / 0.053 / 93.72 / 94.04
0.1 / 0.050 / 0.053 / 93.32 / 93.62
10 / 0.048 / 0.051 / 5.63 / 5.94
100 / 0.054 / 0.068 / 4.87 / 5.16

Table A1.7

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.061 / 0.076* / 34.03 / 38.07
0.1 / 0.062 / 0.075 / 33.89 / 37.68
10 / 0.061 / 0.075 / 5.47 / 6.82
100 / 0.062 / 0.076* / 6.05 / 7.54
20 / 0.01 / 0.056 / 0.063 / 56.61 / 58.78
0.1 / 0.056 / 0.063 / 56.25 / 58.38
10 / 0.056 / 0.063 / 5.12 / 5.82
100 / 0.057 / 0.064 / 5.57 / 6.28
30 / 0.01 / 0.054 / 0.059 / 73.24 / 74.47
0.1 / 0.054 / 0.058 / 72.89 / 74.08
10 / 0.054 / 0.059 / 5.17 / 5.64
100 / 0.054 / 0.059 / 5.36 / 5.84
40 / 0.01 / 0.053 / 0.056 / 84.31 / 84.99
0.1 / 0.053 / 0.057 / 84.05 / 84.72
10 / 0.053 / 0.057 / 5.36 / 5.74
100 / 0.053 / 0.057 / 5.23 / 5.58
50 / 0.01 / 0.053 / 0.056 / 91.14 / 91.51
0.1 / 0.053 / 0.056 / 90.96 / 91.33
10 / 0.053 / 0.055 / 5.58 / 5.88
100 / 0.053 / 0.056 / 5.21 / 5.49

Table A1.8

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.062 / 0.076* / 23.24 / 28.37
0.1 / 0.062 / 0.076* / 5.44 / 6.80
10 / 0.061 / 0.075 / 6.08 / 7.56
100 / 0.062 / 0.077* / 56.44 / 59.47
20 / 0.01 / 0.057 / 0.064 / 55.98 / 58.91
0.1 / 0.057 / 0.063 / 5.12 / 5.81
10 / 0.056 / 0.062 / 5.56 / 6.27
100 / 0.057 / 0.064 / 78.58 / 79.98
30 / 0.01 / 0.055 / 0.059 / 78.05 / 79.43
0.1 / 0.054 / 0.059 / 5.18 / 5.66
10 / 0.054 / 0.059 / 5.35 / 5.83
100 / 0.055 / 0.059 / 90.42 / 91.01
40 / 0.01 / 0.053 / 0.057 / 89.98 / 90.56
0.1 / 0.053 / 0.057 / 5.34 / 5.71
10 / 0.053 / 0.057 / 5.26 / 5.61
100 / 0.053 / 0.057 / 95.91 / 96.14
50 / 0.01 / 0.053 / 0.055 / 95.71 / 95.95
0.1 / 0.053 / 0.056 / 5.56 / 5.86
10 / 0.053 / 0.056 / 5.20 / 5.48
100 / 0.052 / 0.055 / 23.23 / 28.60

Table A1.9

Welch’s t-test / Student’s t-test / Welch’s t-test / Student’s t-test
NOMINAL SIZE OF ALPHA RISK / POWER (%)
n1 = n2 / SDR / 5% / 5%
10 / 0.01 / 0.099* / 0.114* / 42.76 / 46.01
0.1 / 0.099* / 0.112* / 42.71 / 45.81
10 / 0.061 / 0.075 / 7.23 / 8.70
100 / 0.062 / 0.076* / 6.21 / 7.69
20 / 0.01 / 0.081* / 0.088* / 59.37 / 61.08
0.1 / 0.080* / 0.087* / 59.08 / 60.73
10 / 0.057 / 0.064 / 7.12 / 7.87
100 / 0.057 / 0.064 / 5.73 / 6.45
30 / 0.01 / 0.073 / 0.078* / 72.15 / 73.20
0.1 / 0.072 / 0.076* / 71.91 / 72.92
10 / 0.055 / 0.059 / 7.25 / 7.75
100 / 0.054 / 0.059 / 5.53 / 6.00
40 / 0.01 / 0.068 / 0.071 / 81.59 / 82.24
0.1 / 0.068 / 0.071 / 81.25 / 81.89
10 / 0.053 / 0.057 / 7.44 / 7.83
100 / 0.054 / 0.057 / 5.45 / 5.80
50 / 0.01 / 0.065 / 0.068 / 88.13 / 88.51
0.1 / 0.065 / 0.067 / 87.96 / 88.35
10 / 0.053 / 0.055 / 7.66 / 7.98
100 / 0.053 / 0.056 / 5.44 / 5.72

Standard deviations in samples and in populations

To check if the sample SD shows the same pattern as the population SD, we simulated 1,000,000 sets of samples for each sample size from 10 to 85 under 5 population SDs: 1.84, 0.92, 1.11, 1.32 and 1.63. Because the SD sampling distribution was quite close to a normal distribution, the mean and SD are good estimators of the quality of the estimation by the sample SD (see Table A2).

Table A2

POP. SD: / 0.92 / 1.11 / 1.32 / 1.63 / 1.84
N / Mean / SD / Mean / SD / Mean / SD / Mean / SD / Mean / SD
10 / 0.895 / 0.214 / 1.080 / 0.258 / 1.284 / 0.307 / 1.586 / 0.378 / 1.79 / 0.428
15 / 0.904 / 0.172 / 1.090 / 0.208 / 1.297 / 0.247 / 1.601 / 0.305 / 1.807 / 0.345
20 / 0.908 / 0.148 / 1.095 / 0.179 / 1.303 / 0.212 / 1.609 / 0.263 / 1.816 / 0.296
25 / 0.91 / 0.132 / 1.098 / 0.159 / 1.306 / 0.19 / 1.613 / 0.234 / 1.821 / 0.264
30 / 0.912 / 0.12 / 1.101 / 0.145 / 1.309 / 0.173 / 1.616 / 0.213 / 1.824 / 0.241
35 / 0.913 / 0.111 / 1.102 / 0.134 / 1.31 / 0.159 / 1.618 / 0.197 / 1.826 / 0.222
40 / 0.914 / 0.104 / 1.103 / 0.125 / 1.312 / 0.149 / 1.619 / 0.184 / 1.829 / 0.208
45 / 0.915 / 0.098 / 1.104 / 0.118 / 1.312 / 0.14 / 1.621 / 0.173 / 1.83 / 0.195
50 / 0.915 / 0.093 / 1.104 / 0.112 / 1.313 / 0.133 / 1.622 / 0.164 / 1.831 / 0.185
55 / 0.916 / 0.088 / 1.105 / 0.107 / 1.314 / 0.127 / 1.622 / 0.157 / 1.831 / 0.177
60 / 0.916 / 0.084 / 1.105 / 0.102 / 1.314 / 0.121 / 1.623 / 0.15 / 1.832 / 0.169
65 / 0.916 / 0.081 / 1.106 / 0.098 / 1.315 / 0.116 / 1.623 / 0.144 / 1.833 / 0.162
70 / 0.917 / 0.078 / 1.106 / 0.094 / 1.315 / 0.112 / 1.624 / 0.138 / 1.833 / 0.156
75 / 0.917 / 0.076 / 1.106 / 0.091 / 1.316 / 0.108 / 1.624 / 0.134 / 1.834 / 0.151
80 / 0.917 / 0.073 / 1.106 / 0.088 / 1.316 / 0.105 / 1.625 / 0.13 / 1.834 / 0.146
85 / 0.917 / 0.071 / 1.107 / 0.086 / 1.316 / 0.102 / 1.625 / 0.126 / 1.834 / 0.142

The smaller the sample size is, the further the average standard deviation is from the population standard deviation, and the bigger the dispersion around this average.

Type 1 error rate of Student’s t-test and Welch’s t-test when variances are unequal

Assuming a Type 1 error rate of 5% under the null, a test can yield either a significant result (p-value < 5%; or a “false positive” -FP) or a non-significant result (p-value > 5%; or a “true negative”-TN). The specificity is the relative frequency of effects detected as non-significant, under the null:

Specificity = =

The alpha risk is the complement of the specificity:

=

In order to estimate the Type 1 error rate for Student’s t-test, Welch’s t-test and Yuen’s t-test, we simulated 1,000,000 simulations of two samples generated under 64 different conditions (yielding 64,000,000 simulations in total). In each condition, the first sample is generated from a population where σ1=2, and its sample size varies from 10 to 40 in steps of 10. The standard deviation and the sample size of the second sample is a function of the sample sizes ratio (SSR = ; ranging from 0.5 to 2 in steps of 0.5) and the SDR (ranging from 0.5 to 2 in steps of .0.5). In a first step, the p-values of the three tests were extracted for each pair of samples and for each test, and in a second step, the percent of p-values under the nominal alpha risk (5%) was computed for each test.

The set of simulations was repeated nine times, varying the distributions underlying the data. We used R commands to generate data from different distributions.

-Two normal distributions (See Table A3.1): in order to assess the Type 1 error rate of the three tests, under the normality assumption, data were generated by means of the function “rnorm” (from the package “stats”; “R: The Normal Distribution,” 2016).

-Two double exponential distributions (See Table A3.2): in order to assess the impact of high kurtosis on the Type 1 error rate of the three tests, data were generated by means of the function “rdoublex” (from the package “smoothmest”; “R: The double exponential (Laplace) distribution,” 2012).

-One normal and one double exponential distribution (See Table A3.3): in order to assess the impact of the unequal shape of distributions, in terms of kurtosis, on the Type 1 error rate, data were generated by means of the functions “rdoublex” and “rnorm”.

-Two uniform distributions (See Table A3.4): in order to assess the impact of low kurtosis on the Type 1 error rate of the three tests, data were generated by means of the function “runif” (from the package “stats”; “R: The Uniform Distribution,” 2016).

-One uniform and one normal distribution (See Table A3.5): in order to assess the impact of the unequal shape of distributions, in terms of kurtosis, on the Type 1 error rate, data were generated by means of the functions “runif” and “rnorm”.

-One uniform and one double exponential distribution (See Table A3.6): in order to assess the impact of the unequal shape of distributions, in terms of kurtosis, on the Type 1 error rate, data were generated by means of the functions “runif” and “rdoublex”.

-Two normal skewed distributions with positive skewness of 0.79 (See Table A3.7): in order to assess the impact of skewness on the Type 1 error rate, data were generated by means of the function “rsnorm” (from the package “fGarch”; “R: Skew Normal Distribution,” 2017). The normal skewed was used because it is the only skewed distribution where the standard deviation ratio can vary without having an impact on skewness.

-One normal skewed distribution with negative skewness (-0.79) and one normal skewed distribution with positive skewness (+0.79;See Table A3.8): in order to assess the impact of unequal shapes, in terms of kurtosis, on theType 1 error rate, when data are asymmetric, data were generated by means of the functions “rsnorm” with unequal skewness.

-One chi(2) and one normal skewed distribution with negative skewness of +0.79 (See Table A3.9): in order to assess the impact of high skewness and kurtosis on the Type 1 error rate, data were generated by means of the functions “rsnorm” with different skewness and from the function “rchisq”; “R: The (non-central) Chi-Squared Distribution,” 2016).

Consistently with research conducted by Minitab statisticians available at show that the Type 1 error rate for Student’s t-test can differ noticeably from the nominal Type 1 error rate (i.e., 5%) when the groups have different variances. In light of the definition of Bradley (1978), one can consider the alpha risk “sufficiently close” to the nominal alpha risk if its value falls in the interval [0.025; 0.075] (Hayes & Cai, 2007). In the tables presented below, anytime the Type 1 error rate falls outside of this interval, we have added an “*” next to the value. Considering this norm, one can observe that when there is a positive correlation between sample size and standard deviation, the test is often too conservative and when there is a negative correlation between sample size and standard deviation, the test is often too liberal.

On the other hand, the Type 1 error rate of Welch’s t-test remains closer of the nominal size (i.e. 5%) in all the previously contemplated cases. Yuen’s t-test is not a good unconditional alternative, as we observe an unacceptable departure from the nominal alpha risk of 5% in several cases (See Table A3.1, A3.4, A3.7, A3.8 and A3.9). The Type 1 error rate of Yuen’s test is especially problematic when we are studying asymmetric distributions of unequal shapes (see Table A3.8 A3.9). Moreover, even when Yuen’s Type 1 error rate does not show a critical departure from the nominal alpha risk, Welch’s t-test is better at controlling the Type 1 error rate at the desired alpha level (See Table A3.2, A3.3, A3.5 and A3.6).

In order to estimate the Type 1 error rate for Welch’s t-test when there is extreme SDR and a very unbalanced design, we simulated 1,000,000 simulations of two samples generated under 60 different conditions (yielding 60,000,000 simulations in total). In each condition, the first sample is generated from a population where σ1=2, and its sample size varies from 10 to 30 in a step of 10. The standard deviation and the sample size of the second sample is a function of the sample sizes ratio SSR = being respectively 0.01, 0.1, 10 and 100 and SDRs ranging from 1 to 5 in steps of 1). Once again, the set of simulations was repeated nine times, varying the distributions underlying the data.

We found that even with very extreme SDRs and unbalanced designs, as long as there are at least 10 subjects per group (See Table A4; for more detail, see the appendix at Welch’s t-test remains very close to the nominal alpha level (i.e., 5%). The only exception is when at least one distribution has very high skewness and kurtosis, such as the chi-square distribution with two degrees of freedom. Under this distribution, one needs at least 30 subjects per groups to accurately control the alpha risk. With fewer than 30 subjects per groups, even Welch’s t-test becomes too liberal when there is a negative correlation between sample size and standard deviation (See Table A3.9).

Table A3.1

p-values
n2/n1 / SDR / n1 / n2 / sd1 / sd2 / Welch’s t-test / Student’s t-test / Yuen’s t-test
0.5 / 0.5 / 10 / 5 / 2 / 1 / 0.047 / 0.021* / 0.048
0.5 / 1 / 10 / 5 / 2 / 2 / 0.050 / 0.050 / 0.058
0.5 / 1.5 / 10 / 5 / 2 / 3 / 0.054 / 0.086* / 0.071
0.5 / 2 / 10 / 5 / 2 / 4 / 0.056 / 0.117* / 0.077*
1 / 0.5 / 10 / 10 / 2 / 1 / 0.050 / 0.055 / 0.055
1 / 1 / 10 / 10 / 2 / 2 / 0.049 / 0.050 / 0.050
1 / 1.5 / 10 / 10 / 2 / 3 / 0.049 / 0.052 / 0.052
1 / 2 / 10 / 10 / 2 / 4 / 0.050 / 0.054 / 0.055
1.5 / 0.5 / 10 / 15 / 2 / 1 / 0.051 / 0.087* / 0.059
1.5 / 1 / 10 / 15 / 2 / 2 / 0.050 / 0.050 / 0.052
1.5 / 1.5 / 10 / 15 / 2 / 3 / 0.049 / 0.036 / 0.050
1.5 / 2 / 10 / 15 / 2 / 4 / 0.049 / 0.030 / 0.051
2 / 0.5 / 10 / 20 / 2 / 1 / 0.051 / 0.114* / 0.061
2 / 1 / 10 / 20 / 2 / 2 / 0.050 / 0.050 / 0.054
2 / 1.5 / 10 / 20 / 2 / 3 / 0.050 / 0.027 / 0.051
2 / 2 / 10 / 20 / 2 / 4 / 0.050 / 0.019* / 0.051
0.5 / 0.5 / 20 / 10 / 2 / 1 / 0.049 / 0.019* / 0.050
0.5 / 1 / 20 / 10 / 2 / 2 / 0.050 / 0.050 / 0.054
0.5 / 1.5 / 20 / 10 / 2 / 3 / 0.051 / 0.086* / 0.059
0.5 / 2 / 20 / 10 / 2 / 4 / 0.051 / 0.114* / 0.061
1 / 0.5 / 20 / 20 / 2 / 1 / 0.050 / 0.053 / 0.053
1 / 1 / 20 / 20 / 2 / 2 / 0.049 / 0.050 / 0.051
1 / 1.5 / 20 / 20 / 2 / 3 / 0.050 / 0.051 / 0.052
1 / 2 / 20 / 20 / 2 / 4 / 0.050 / 0.052 / 0.053
1.5 / 0.5 / 20 / 30 / 2 / 1 / 0.050 / 0.084* / 0.054
1.5 / 1 / 20 / 30 / 2 / 2 / 0.050 / 0.050 / 0.051
1.5 / 1.5 / 20 / 30 / 2 / 3 / 0.050 / 0.035 / 0.051
1.5 / 2 / 20 / 30 / 2 / 4 / 0.050 / 0.029 / 0.051
2 / 0.5 / 20 / 40 / 2 / 1 / 0.050 / 0.112* / 0.055
2 / 1 / 20 / 40 / 2 / 2 / 0.050 / 0.050 / 0.052
2 / 1.5 / 20 / 40 / 2 / 3 / 0.050 / 0.027 / 0.051
2 / 2 / 20 / 40 / 2 / 4 / 0.050 / 0.018* / 0.050
0.5 / 0.5 / 30 / 15 / 2 / 1 / 0.049 / 0.018* / 0.051
0.5 / 1 / 30 / 15 / 2 / 2 / 0.050 / 0.050 / 0.053
0.5 / 1.5 / 30 / 15 / 2 / 3 / 0.050 / 0.085* / 0.055
0.5 / 2 / 30 / 15 / 2 / 4 / 0.050 / 0.113* / 0.056
1 / 0.5 / 30 / 30 / 2 / 1 / 0.050 / 0.052 / 0.052
1 / 1 / 30 / 30 / 2 / 2 / 0.050 / 0.050 / 0.051
1 / 1.5 / 30 / 30 / 2 / 3 / 0.050 / 0.050 / 0.051
1 / 2 / 30 / 30 / 2 / 4 / 0.050 / 0.051 / 0.051
1.5 / 0.5 / 30 / 45 / 2 / 1 / 0.050 / 0.083* / 0.052
1.5 / 1 / 30 / 45 / 2 / 2 / 0.050 / 0.050 / 0.051
1.5 / 1.5 / 30 / 45 / 2 / 3 / 0.050 / 0.035 / 0.051
1.5 / 2 / 30 / 45 / 2 / 4 / 0.050 / 0.028 / 0.051
2 / 0.5 / 30 / 60 / 2 / 1 / 0.050 / 0.111* / 0.052
2 / 1 / 30 / 60 / 2 / 2 / 0.050 / 0.050 / 0.051
2 / 1.5 / 30 / 60 / 2 / 3 / 0.050 / 0.026 / 0.051
2 / 2 / 30 / 60 / 2 / 4 / 0.050 / 0.017* / 0.050
0.5 / 0.5 / 40 / 20 / 2 / 1 / 0.050 / 0.018* / 0.051
0.5 / 1 / 40 / 20 / 2 / 2 / 0.050 / 0.050 / 0.052
0.5 / 1.5 / 40 / 20 / 2 / 3 / 0.051 / 0.086* / 0.054
0.5 / 2 / 40 / 20 / 2 / 4 / 0.051 / 0.111* / 0.054
1 / 0.5 / 40 / 40 / 2 / 1 / 0.050 / 0.052 / 0.051
1 / 1 / 40 / 40 / 2 / 2 / 0.050 / 0.050 / 0.051
1 / 1.5 / 40 / 40 / 2 / 3 / 0.050 / 0.050 / 0.051
1 / 2 / 40 / 40 / 2 / 4 / 0.050 / 0.051 / 0.051
1.5 / 0.5 / 40 / 60 / 2 / 1 / 0.050 / 0.083* / 0.052
1.5 / 1 / 40 / 60 / 2 / 2 / 0.050 / 0.050 / 0.051
1.5 / 1.5 / 40 / 60 / 2 / 3 / 0.050 / 0.034 / 0.051
1.5 / 2 / 40 / 60 / 2 / 4 / 0.050 / 0.028 / 0.051
2 / 0.5 / 40 / 80 / 2 / 1 / 0.050 / 0.110* / 0.052
2 / 1 / 40 / 80 / 2 / 2 / 0.050 / 0.050 / 0.051
2 / 1.5 / 40 / 80 / 2 / 3 / 0.050 / 0.026 / 0.050
2 / 2 / 40 / 80 / 2 / 4 / 0.050 / 0.017* / 0.050

Table A3.2

p-values
n2/n1 / SDR / n1 / n2 / sd1 / sd2 / Welch’s t-test / Student’s t-test / Yuen’s t-test
0.5 / 0.5 / 10 / 5 / 2 / 1 / 0.041 / 0.019* / 0.037
0.5 / 1 / 10 / 5 / 2 / 2 / 0.041 / 0.046 / 0.044
0.5 / 1.5 / 10 / 5 / 2 / 3 / 0.040 / 0.077 / 0.051
0.5 / 2 / 10 / 5 / 2 / 4 / 0.040 / 0.103* / 0.055
1 / 0.5 / 10 / 10 / 2 / 1 / 0.044 / 0.049 / 0.042
1 / 1 / 10 / 10 / 2 / 2 / 0.044 / 0.046 / 0.040
1 / 1.5 / 10 / 10 / 2 / 3 / 0.044 / 0.047 / 0.041
1 / 2 / 10 / 10 / 2 / 4 / 0.044 / 0.049 / 0.041
1.5 / 0.5 / 10 / 15 / 2 / 1 / 0.045 / 0.081* / 0.044
1.5 / 1 / 10 / 15 / 2 / 2 / 0.045 / 0.048 / 0.043
1.5 / 1.5 / 10 / 15 / 2 / 3 / 0.045 / 0.034 / 0.043
1.5 / 2 / 10 / 15 / 2 / 4 / 0.046 / 0.028* / 0.043
2 / 0.5 / 10 / 20 / 2 / 1 / 0.045 / 0.107* / 0.045
2 / 1 / 10 / 20 / 2 / 2 / 0.046 / 0.048 / 0.044
2 / 1.5 / 10 / 20 / 2 / 3 / 0.047 / 0.027 / 0.044
2 / 2 / 10 / 20 / 2 / 4 / 0.047 / 0.018* / 0.044
0.5 / 0.5 / 20 / 10 / 2 / 1 / 0.046 / 0.018* / 0.044
0.5 / 1 / 20 / 10 / 2 / 2 / 0.046 / 0.048 / 0.044
0.5 / 1.5 / 20 / 10 / 2 / 3 / 0.045 / 0.081* / 0.044
0.5 / 2 / 20 / 10 / 2 / 4 / 0.044 / 0.107* / 0.044
1 / 0.5 / 20 / 20 / 2 / 1 / 0.047 / 0.050 / 0.046
1 / 1 / 20 / 20 / 2 / 2 / 0.048 / 0.048 / 0.046
1 / 1.5 / 20 / 20 / 2 / 3 / 0.047 / 0.049 / 0.045
1 / 2 / 20 / 20 / 2 / 4 / 0.047 / 0.050 / 0.046
1.5 / 0.5 / 20 / 30 / 2 / 1 / 0.047 / 0.082* / 0.046
1.5 / 1 / 20 / 30 / 2 / 2 / 0.048 / 0.049 / 0.047
1.5 / 1.5 / 20 / 30 / 2 / 3 / 0.048 / 0.034 / 0.047
1.5 / 2 / 20 / 30 / 2 / 4 / 0.048 / 0.027 / 0.047
2 / 0.5 / 20 / 40 / 2 / 1 / 0.047 / 0.109* / 0.046
2 / 1 / 20 / 40 / 2 / 2 / 0.049 / 0.049 / 0.047
2 / 1.5 / 20 / 40 / 2 / 3 / 0.048 / 0.027 / 0.047
2 / 2 / 20 / 40 / 2 / 4 / 0.049 / 0.017* / 0.047
0.5 / 0.5 / 30 / 15 / 2 / 1 / 0.048 / 0.018* / 0.046
0.5 / 1 / 30 / 15 / 2 / 2 / 0.047 / 0.049 / 0.046
0.5 / 1.5 / 30 / 15 / 2 / 3 / 0.046 / 0.082* / 0.046
0.5 / 2 / 30 / 15 / 2 / 4 / 0.046 / 0.109* / 0.045
1 / 0.5 / 30 / 30 / 2 / 1 / 0.048 / 0.050 / 0.048
1 / 1 / 30 / 30 / 2 / 2 / 0.049 / 0.049 / 0.047
1 / 1.5 / 30 / 30 / 2 / 3 / 0.049 / 0.049 / 0.047
1 / 2 / 30 / 30 / 2 / 4 / 0.048 / 0.050 / 0.047
1.5 / 0.5 / 30 / 45 / 2 / 1 / 0.048 / 0.082* / 0.048
1.5 / 1 / 30 / 45 / 2 / 2 / 0.049 / 0.050 / 0.048
1.5 / 1.5 / 30 / 45 / 2 / 3 / 0.049 / 0.035 / 0.048
1.5 / 2 / 30 / 45 / 2 / 4 / 0.049 / 0.027 / 0.048
2 / 0.5 / 30 / 60 / 2 / 1 / 0.048 / 0.109* / 0.048
2 / 1 / 30 / 60 / 2 / 2 / 0.049 / 0.049 / 0.048
2 / 1.5 / 30 / 60 / 2 / 3 / 0.049 / 0.026 / 0.048
2 / 2 / 30 / 60 / 2 / 4 / 0.050 / 0.017* / 0.049
0.5 / 0.5 / 40 / 20 / 2 / 1 / 0.049 / 0.017* / 0.047
0.5 / 1 / 40 / 20 / 2 / 2 / 0.048 / 0.049 / 0.047
0.5 / 1.5 / 40 / 20 / 2 / 3 / 0.048 / 0.083* / 0.047
0.5 / 2 / 40 / 20 / 2 / 4 / 0.048 / 0.109* / 0.046
1 / 0.5 / 40 / 40 / 2 / 1 / 0.049 / 0.050 / 0.048
1 / 1 / 40 / 40 / 2 / 2 / 0.049 / 0.049 / 0.048
1 / 1.5 / 40 / 40 / 2 / 3 / 0.049 / 0.050 / 0.048
1 / 2 / 40 / 40 / 2 / 4 / 0.049 / 0.050 / 0.049
1.5 / 0.5 / 40 / 60 / 2 / 1 / 0.049 / 0.083* / 0.049
1.5 / 1 / 40 / 60 / 2 / 2 / 0.050 / 0.050 / 0.049
1.5 / 1.5 / 40 / 60 / 2 / 3 / 0.049 / 0.035 / 0.048
1.5 / 2 / 40 / 60 / 2 / 4 / 0.050 / 0.028 / 0.049
2 / 0.5 / 40 / 80 / 2 / 1 / 0.048 / 0.109* / 0.048
2 / 1 / 40 / 80 / 2 / 2 / 0.049 / 0.049 / 0.049
2 / 1.5 / 40 / 80 / 2 / 3 / 0.050 / 0.027 / 0.048
2 / 2 / 40 / 80 / 2 / 4 / 0.049 / 0.017* / 0.049

Table A3.3

p-values
n2/n1 / SDR / n1 / n2 / sd1 / sd2 / Welch’s t-test / Student’s t-test / Yuen’s t-test
0.5 / 0.5 / 10 / 5 / 2 / 1 / 0.043 / 0.028 / 0.045
0.5 / 1 / 10 / 5 / 2 / 2 / 0.041 / 0.066 / 0.047
0.5 / 1.5 / 10 / 5 / 2 / 3 / 0.041 / 0.103* / 0.053
0.5 / 2 / 10 / 5 / 2 / 4 / 0.040 / 0.131* / 0.056
1 / 0.5 / 10 / 10 / 2 / 1 / 0.049 / 0.052 / 0.053
1 / 1 / 10 / 10 / 2 / 2 / 0.045 / 0.048 / 0.045
1 / 1.5 / 10 / 10 / 2 / 3 / 0.044 / 0.049 / 0.043
1 / 2 / 10 / 10 / 2 / 4 / 0.044 / 0.051 / 0.043
1.5 / 0.5 / 10 / 15 / 2 / 1 / 0.050 / 0.070 / 0.058
1.5 / 1 / 10 / 15 / 2 / 2 / 0.047 / 0.038 / 0.049
1.5 / 1.5 / 10 / 15 / 2 / 3 / 0.047 / 0.028 / 0.046
1.5 / 2 / 10 / 15 / 2 / 4 / 0.046 / 0.023* / 0.044
2 / 0.5 / 10 / 20 / 2 / 1 / 0.051 / 0.084* / 0.060
2 / 1 / 10 / 20 / 2 / 2 / 0.049 / 0.031 / 0.053
2 / 1.5 / 10 / 20 / 2 / 3 / 0.048 / 0.017* / 0.048
2 / 2 / 10 / 20 / 2 / 4 / 0.047 / 0.012* / 0.046
0.5 / 0.5 / 20 / 10 / 2 / 1 / 0.047 / 0.028 / 0.048
0.5 / 1 / 20 / 10 / 2 / 2 / 0.046 / 0.073 / 0.045
0.5 / 1.5 / 20 / 10 / 2 / 3 / 0.045 / 0.111* / 0.045
0.5 / 2 / 20 / 10 / 2 / 4 / 0.044 / 0.135* / 0.045
1 / 0.5 / 20 / 20 / 2 / 1 / 0.049 / 0.050 / 0.052
1 / 1 / 20 / 20 / 2 / 2 / 0.048 / 0.049 / 0.048
1 / 1.5 / 20 / 20 / 2 / 3 / 0.048 / 0.050 / 0.047
1 / 2 / 20 / 20 / 2 / 4 / 0.047 / 0.051 / 0.047
1.5 / 0.5 / 20 / 30 / 2 / 1 / 0.050 / 0.068 / 0.053
1.5 / 1 / 20 / 30 / 2 / 2 / 0.049 / 0.037 / 0.050
1.5 / 1.5 / 20 / 30 / 2 / 3 / 0.049 / 0.027 / 0.048
1.5 / 2 / 20 / 30 / 2 / 4 / 0.048 / 0.023* / 0.048
2 / 0.5 / 20 / 40 / 2 / 1 / 0.050 / 0.081* / 0.054
2 / 1 / 20 / 40 / 2 / 2 / 0.049 / 0.030 / 0.051
2 / 1.5 / 20 / 40 / 2 / 3 / 0.049 / 0.016* / 0.049
2 / 2 / 20 / 40 / 2 / 4 / 0.049 / 0.011* / 0.049
0.5 / 0.5 / 30 / 15 / 2 / 1 / 0.049 / 0.028* / 0.048
0.5 / 1 / 30 / 15 / 2 / 2 / 0.047 / 0.075 / 0.047
0.5 / 1.5 / 30 / 15 / 2 / 3 / 0.046 / 0.112* / 0.046
0.5 / 2 / 30 / 15 / 2 / 4 / 0.046 / 0.135* / 0.045
1 / 0.5 / 30 / 30 / 2 / 1 / 0.050 / 0.050 / 0.051
1 / 1 / 30 / 30 / 2 / 2 / 0.049 / 0.049 / 0.049
1 / 1.5 / 30 / 30 / 2 / 3 / 0.048 / 0.050 / 0.048
1 / 2 / 30 / 30 / 2 / 4 / 0.049 / 0.051 / 0.048
1.5 / 0.5 / 30 / 45 / 2 / 1 / 0.050 / 0.068 / 0.052
1.5 / 1 / 30 / 45 / 2 / 2 / 0.050 / 0.037 / 0.050
1.5 / 1.5 / 30 / 45 / 2 / 3 / 0.049 / 0.027 / 0.049
1.5 / 2 / 30 / 45 / 2 / 4 / 0.049 / 0.023* / 0.048
2 / 0.5 / 30 / 60 / 2 / 1 / 0.050 / 0.081* / 0.052
2 / 1 / 30 / 60 / 2 / 2 / 0.050 / 0.030 / 0.051
2 / 1.5 / 30 / 60 / 2 / 3 / 0.050 / 0.016* / 0.049
2 / 2 / 30 / 60 / 2 / 4 / 0.050 / 0.011* / 0.049
0.5 / 0.5 / 40 / 20 / 2 / 1 / 0.049 / 0.028 / 0.050
0.5 / 1 / 40 / 20 / 2 / 2 / 0.048 / 0.076* / 0.048
0.5 / 1.5 / 40 / 20 / 2 / 3 / 0.047 / 0.113* / 0.047
0.5 / 2 / 40 / 20 / 2 / 4 / 0.047 / 0.135* / 0.047
1 / 0.5 / 40 / 40 / 2 / 1 / 0.050 / 0.051 / 0.051
1 / 1 / 40 / 40 / 2 / 2 / 0.049 / 0.050 / 0.049
1 / 1.5 / 40 / 40 / 2 / 3 / 0.049 / 0.050 / 0.049
1 / 2 / 40 / 40 / 2 / 4 / 0.049 / 0.051 / 0.048
1.5 / 0.5 / 40 / 60 / 2 / 1 / 0.050 / 0.067 / 0.052
1.5 / 1 / 40 / 60 / 2 / 2 / 0.050 / 0.037 / 0.050
1.5 / 1.5 / 40 / 60 / 2 / 3 / 0.050 / 0.027 / 0.049
1.5 / 2 / 40 / 60 / 2 / 4 / 0.049 / 0.022* / 0.049
2 / 0.5 / 40 / 80 / 2 / 1 / 0.050 / 0.081* / 0.051
2 / 1 / 40 / 80 / 2 / 2 / 0.050 / 0.029 / 0.051
2 / 1.5 / 40 / 80 / 2 / 3 / 0.050 / 0.016* / 0.050
2 / 2 / 40 / 80 / 2 / 4 / 0.050 / 0.011* / 0.050

Table A3.4