Which of Two Predictors Does the Better Job of Predicting a Dichotomous Outcome Variable?

Using ADDSC Scores to Predict Repeating a Grade

Using the Howell data set (described here), I am predicting repeating a grade from scores on an attention deficit/hyperactivity scale (ADDSC).

proclogistic; model repeat(event='repeated') = ADDSC / ctable; run;

Analysis of Maximum Likelihood Estimates
Parameter / DF / Estimate / Standard
Error / Wald
Chi-Square / PrChiSq
Intercept / 1 / -7.7037 / 1.9065 / 16.3283 / <.0001
addsc / 1 / 0.1016 / 0.0305 / 11.1253 / 0.0009
Classification Table
Prob
Level / Correct / Incorrect / Percentages
Event / Non-
Event / Event / Non-
Event / Correct / Sensi-
tivity / Speci-
ficity / False
POS / False
NEG
0.000 / 12 / 0 / 76 / 0 / 13.6 / 100.0 / 0.0 / 86.4 / .
0.020 / 12 / 10 / 66 / 0 / 25.0 / 100.0 / 13.2 / 84.6 / 0.0
0.040 / 12 / 22 / 54 / 0 / 38.6 / 100.0 / 28.9 / 81.8 / 0.0
0.060 / 10 / 33 / 43 / 2 / 48.9 / 83.3 / 43.4 / 81.1 / 5.7
0.080 / 10 / 47 / 29 / 2 / 64.8 / 83.3 / 61.8 / 74.4 / 4.1
0.100 / 10 / 53 / 23 / 2 / 71.6 / 83.3 / 69.7 / 69.7 / 3.6
0.120 / 9 / 56 / 20 / 3 / 73.9 / 75.0 / 73.7 / 69.0 / 5.1
0.140 / 9 / 57 / 19 / 3 / 75.0 / 75.0 / 75.0 / 67.9 / 5.0
0.160 / 9 / 60 / 16 / 3 / 78.4 / 75.0 / 78.9 / 64.0 / 4.8
0.180 / 8 / 63 / 13 / 4 / 80.7 / 66.7 / 82.9 / 61.9 / 6.0
0.200 / 8 / 63 / 13 / 4 / 80.7 / 66.7 / 82.9 / 61.9 / 6.0
0.220 / 8 / 66 / 10 / 4 / 84.1 / 66.7 / 86.8 / 55.6 / 5.7
0.240 / 6 / 67 / 9 / 6 / 83.0 / 50.0 / 88.2 / 60.0 / 8.2
0.260 / 6 / 69 / 7 / 6 / 85.2 / 50.0 / 90.8 / 53.8 / 8.0
0.280 / 6 / 70 / 6 / 6 / 86.4 / 50.0 / 92.1 / 50.0 / 7.9
0.300 / 5 / 70 / 6 / 7 / 85.2 / 41.7 / 92.1 / 54.5 / 9.1
0.320 / 3 / 70 / 6 / 9 / 83.0 / 25.0 / 92.1 / 66.7 / 11.4
0.340 / 2 / 70 / 6 / 10 / 81.8 / 16.7 / 92.1 / 75.0 / 12.5
0.360 / 2 / 70 / 6 / 10 / 81.8 / 16.7 / 92.1 / 75.0 / 12.5
0.380 / 2 / 71 / 5 / 10 / 83.0 / 16.7 / 93.4 / 71.4 / 12.3
0.400 / 2 / 71 / 5 / 10 / 83.0 / 16.7 / 93.4 / 71.4 / 12.3
0.420 / 2 / 71 / 5 / 10 / 83.0 / 16.7 / 93.4 / 71.4 / 12.3
0.440 / 1 / 72 / 4 / 11 / 83.0 / 8.3 / 94.7 / 80.0 / 13.3
0.460 / 1 / 72 / 4 / 11 / 83.0 / 8.3 / 94.7 / 80.0 / 13.3
0.480 / 1 / 72 / 4 / 11 / 83.0 / 8.3 / 94.7 / 80.0 / 13.3
0.500 / 1 / 73 / 3 / 11 / 84.1 / 8.3 / 96.1 / 75.0 / 13.1
0.520 / 1 / 73 / 3 / 11 / 84.1 / 8.3 / 96.1 / 75.0 / 13.1
0.540 / 1 / 73 / 3 / 11 / 84.1 / 8.3 / 96.1 / 75.0 / 13.1
0.560 / 1 / 74 / 2 / 11 / 85.2 / 8.3 / 97.4 / 66.7 / 12.9
0.580 / 1 / 74 / 2 / 11 / 85.2 / 8.3 / 97.4 / 66.7 / 12.9
0.600 / 1 / 74 / 2 / 11 / 85.2 / 8.3 / 97.4 / 66.7 / 12.9
0.620 / 1 / 75 / 1 / 11 / 86.4 / 8.3 / 98.7 / 50.0 / 12.8
0.640 / 1 / 75 / 1 / 11 / 86.4 / 8.3 / 98.7 / 50.0 / 12.8
0.660 / 1 / 75 / 1 / 11 / 86.4 / 8.3 / 98.7 / 50.0 / 12.8
0.680 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.700 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.720 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.740 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.760 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.780 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.800 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.820 / 0 / 76 / 0 / 12 / 86.4 / 0.0 / 100.0 / . / 13.6

I select .14 as the threshold for classification as going to repeat a grade. Cases with predicted probabilities of .14 or higher will be predicted to repeat a grade.

The logistic model is where Y-hat is the predicted probability of the event (repeating a grade) and X is the value of the predictor (ADDSC score). Solving for X, . For the prob = .14 threshold, odds = .14/(1-.14) = .163, so . That is, students with ADDSC of 58 or more will be predicted to repeat a grade.

Checking on my work, I “RECODE addsc (Lowest thru 57=0) (58 thru Highest=1) INTO Prediction.” Then I use crosstabs to produce the classification table:

repeat * Prediction (ADDSC)Crosstabulation
Count
Prediction / Total
.00 / 1.00
repeat / norepeat / 57 / 19 / 76
repeated / 3 / 9 / 12
Total / 60 / 28 / 88

Specificity = 9/12 = .75. Sensitivity = 57/76 = .75. False positive rate = 19/28 = .68. False negative rate = 3/57 = .05.

Total correct classifications = (57+9)/88 = .75. This matches up with the values in SAS’ ctable.

Using IQ Scores to Predict Repeating a Grade

Classification Table
Prob
Level / Correct / Incorrect / Percentages
Event / Non-
Event / Event / Non-
Event / Correct / Sensi-
tivity / Speci-
ficity / False
POS / False
NEG
0.000 / 12 / 0 / 76 / 0 / 13.6 / 100.0 / 0.0 / 86.4 / .
0.020 / 12 / 4 / 72 / 0 / 18.2 / 100.0 / 5.3 / 85.7 / 0.0
0.040 / 12 / 13 / 63 / 0 / 28.4 / 100.0 / 17.1 / 84.0 / 0.0
0.060 / 11 / 24 / 52 / 1 / 39.8 / 91.7 / 31.6 / 82.5 / 4.0
0.080 / 11 / 35 / 41 / 1 / 52.3 / 91.7 / 46.1 / 78.8 / 2.8
0.100 / 11 / 42 / 34 / 1 / 60.2 / 91.7 / 55.3 / 75.6 / 2.3
0.120 / 9 / 47 / 29 / 3 / 63.6 / 75.0 / 61.8 / 76.3 / 6.0
0.140 / 7 / 49 / 27 / 5 / 63.6 / 58.3 / 64.5 / 79.4 / 9.3
0.160 / 6 / 54 / 22 / 6 / 68.2 / 50.0 / 71.1 / 78.6 / 10.0
0.180 / 5 / 55 / 21 / 7 / 68.2 / 41.7 / 72.4 / 80.8 / 11.3
0.200 / 5 / 59 / 17 / 7 / 72.7 / 41.7 / 77.6 / 77.3 / 10.6
0.220 / 4 / 60 / 16 / 8 / 72.7 / 33.3 / 78.9 / 80.0 / 11.8
0.240 / 4 / 64 / 12 / 8 / 77.3 / 33.3 / 84.2 / 75.0 / 11.1
0.260 / 3 / 64 / 12 / 9 / 76.1 / 25.0 / 84.2 / 80.0 / 12.3
0.280 / 3 / 65 / 11 / 9 / 77.3 / 25.0 / 85.5 / 78.6 / 12.2
0.300 / 2 / 68 / 8 / 10 / 79.5 / 16.7 / 89.5 / 80.0 / 12.8
0.320 / 1 / 70 / 6 / 11 / 80.7 / 8.3 / 92.1 / 85.7 / 13.6
0.340 / 1 / 72 / 4 / 11 / 83.0 / 8.3 / 94.7 / 80.0 / 13.3
0.360 / 0 / 74 / 2 / 12 / 84.1 / 0.0 / 97.4 / 100.0 / 14.0
0.380 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.400 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.420 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.440 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.460 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.480 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.500 / 0 / 75 / 1 / 12 / 85.2 / 0.0 / 98.7 / 100.0 / 13.8
0.520 / 0 / 76 / 0 / 12 / 86.4 / 0.0 / 100.0 / . / 13.6

Sensitivity and specificity are most nearly equal for prob = .14.

Analysis of Maximum Likelihood Estimates
Parameter / DF / Estimate / Standard
Error / Wald
Chi-Square / PrChiSq
Intercept / 1 / 5.9488 / 3.0022 / 3.9265 / 0.0475
iq / 1 / -0.0812 / 0.0323 / 6.3295 / 0.0119

Cutoff for IQ = . Students with IQ less than 96 will be predicted to repeat a grade.

repeat * Prediction2 (IQ)Crosstabulation
Count
Prediction2 / Total
.00 / 1.00
repeat / norepeat / 47 / 29 / 76
repeated / 3 / 9 / 12
Total / 50 / 38 / 88

Given rounding of IQ to integer values, these counts match those in SAS’ ctable well enough.

Correct classifications = 47+9 = 56 out of 88 = .636.

Do the Two Predictors Differ Significantly?

With respect to total number of correct classifications, which was the better predictor, ADDSC or IQ?

IF (repeat-Prediction=0) correct=1.

EXECUTE.Is case correctly classified using ADDSC?

IF (repeat-Prediction2=0) correct2=1.

EXECUTE. Is case correctly classified using IQ?

RECODE correct correct2 (SYSMIS=0).

EXECUTE. Set missing values (incorrect classifications) to value 0.

CROSSTABS

/TABLES=correct2 BY correct

correct2 * correct Crosstabulation
Count
Correct (ADDSC) / Total
.00 / 1.00
correct2 (IQ) / .00 / 14 / 18 / 32
1.00 / 8 / 48 / 56
Total / 22 / 66 / 88

Construct McNemar Table

Using ADDSC
Correct Prediction (1) / Not (0) / Marginals
Using IQ / Correct Prediction (1) / 48 A / 8 B / 56
Not (0) / 18 C / 14 D / 32
Marginals / 66 / 22 / 88

Conduct McNemar’s Test by Hand

,p = .0776. This is using the correction for continuity.

, p = .0499. This is without the correction for continuity.

Exact Binomial Solution

B = 2*PROBBNML(.5, 26, 8);procprint; run;p = 0.0776

Using SAS to Conduct McNemar’s Test

Datahowell;

Input ADDSC IQ Count; Cards;

1 1 48

1 0 18

0 1 8

0 0 14

ProcFreq; Tables IQ*ADDSC / Agree; Weight Count; run;

McNemar's Test
Statistic (S) / 3.8462
DF / 1
Pr > S / 0.0499

Notice that SAS does not apply the correction for continuity.

Using SPSS to conduct McNemar’s Test

ADDSC * IQ Crosstabulation
Count
IQ / Total
.00 / 1.00
ADDSC / .00 / 14 / 8 / 22
1.00 / 18 / 48 / 66
Total / 32 / 56 / 88
Chi-Square Tests
Value / Exact Sig. (2-sided)
McNemar Test / .076a
N of Valid Cases / 88
a. Binomial distribution used.

SPSS does use the correction for continuity.

Using the Vassar App

The Vassar App uses the exact binomial solution.

Create ROC Curves with SPSS

ADDSC.

Case Processing Summary
repeat / Valid N (listwise)
Positivea / 12
Negative / 76
Larger values of the test result variable(s) indicate stronger evidence for a positive actual state.
a. The positive actual state is repeated.

Area Under the Curve
Test Result Variable(s): addsc
Area / Std. Errora / Asymptotic Sig.b / Asymptotic 95% Confidence Interval
Lower Bound / Upper Bound
.826 / .059 / .000 / .711 / .941
The test result variable(s): addsc has at least one tie between the positive actual state group and the negative actual state group. Statistics may be biased.
a. Under the nonparametric assumption
b. Null hypothesis: true area = 0.5
Coordinates of the Curve
Test Result Variable(s): addsc
Positive if Greater Than or Equal Toa / Sensitivity / 1 - Specificity
25.00 / 1.000 / 1.000
27.50 / 1.000 / .987
29.50 / 1.000 / .974
31.50 / 1.000 / .961
33.50 / 1.000 / .947
34.50 / 1.000 / .921
35.50 / 1.000 / .908
36.50 / 1.000 / .882
38.00 / 1.000 / .868
39.50 / 1.000 / .855
41.00 / 1.000 / .816
42.50 / 1.000 / .803
43.50 / 1.000 / .776
44.50 / 1.000 / .711
45.50 / 1.000 / .684
46.50 / 1.000 / .645
47.50 / 1.000 / .618
48.50 / 1.000 / .566
49.50 / .833 / .526
50.50 / .833 / .408
51.50 / .833 / .382
52.50 / .833 / .355
53.50 / .833 / .316
54.50 / .833 / .303
55.50 / .833 / .276
56.50 / .833 / .263
57.50 / .750 / .250
58.50 / .750 / .224
59.50 / .750 / .211
60.50 / .750 / .171
61.50 / .667 / .171
62.50 / .667 / .145
63.50 / .667 / .132
64.50 / .667 / .118
65.50 / .500 / .092
67.00 / .500 / .079
68.50 / .417 / .079
69.50 / .250 / .079
71.00 / .167 / .066
73.00 / .167 / .053
74.50 / .167 / .039
75.50 / .083 / .039
77.00 / .083 / .026
81.50 / .083 / .013
86.00 / .000 / .000
The test result variable(s): addsc has at least one tie between the positive actual state group and the negative actual state group.
a. The smallest cutoff value is the minimum observed test value minus 1, and the largest cutoff value is the maximum observed test value plus 1. All the other cutoff values are the averages of two consecutive ordered observed test values.

IQ. Because IQ is negatively associated with repeating a grade, I reflect it prior to constructing the ROC curve, forcing the curve above the diagonal. IQ150 = 150-IQ.

Case Processing Summary
repeat / Valid N (listwise)
Positivea / 12
Negative / 76

Area Under the Curve
Test Result Variable(s): IQ150
Area / Std. Errora / Asymptotic Sig.b / Asymptotic 95% Confidence Interval
Lower Bound / Upper Bound
.747 / .064 / .006 / .621 / .873
The test result variable(s): IQ150 has at least one tie between the positive actual state group and the negative actual state group. Statistics may be biased.
a. Under the nonparametric assumption
b. Null hypothesis: true area = 0.5

Area under the curve less than it was with the ADDSC predictor.

Coordinates of the Curve
Test Result Variable(s): IQ150
Positive if Greater Than or Equal Toa / Sensitivity / 1 - Specificity
12.0000 / 1.000 / 1.000
16.0000 / 1.000 / .987
20.5000 / 1.000 / .974
22.5000 / 1.000 / .961
26.0000 / 1.000 / .947
29.5000 / 1.000 / .934
31.0000 / 1.000 / .908
33.5000 / 1.000 / .868
35.5000 / 1.000 / .842
37.0000 / 1.000 / .829
38.5000 / 1.000 / .816
39.5000 / 1.000 / .763
40.5000 / 1.000 / .750
41.5000 / 1.000 / .711
42.5000 / .917 / .684
43.5000 / .917 / .645
44.5000 / .917 / .592
45.5000 / .917 / .553
46.5000 / .917 / .539
47.5000 / .917 / .513
48.5000 / .917 / .474
49.5000 / .917 / .447
50.5000 / .917 / .408
51.5000 / .917 / .395
52.5000 / .833 / .382
53.5000 / .750 / .382
54.5000 / .750 / .355
55.5000 / .583 / .303
56.5000 / .583 / .289
57.5000 / .500 / .276
58.5000 / .500 / .250
59.5000 / .417 / .224
60.5000 / .417 / .211
61.5000 / .417 / .184
63.0000 / .333 / .158
64.5000 / .250 / .145
65.5000 / .250 / .105
66.5000 / .250 / .079
67.5000 / .250 / .053
68.5000 / .167 / .026
70.0000 / .083 / .013
73.0000 / .000 / .013
76.0000 / .000 / .000
The test result variable(s): IQ150 has at least one tie between the positive actual state group and the negative actual state group.
a. The smallest cutoff value is the minimum observed test value minus 1, and the largest cutoff value is the maximum observed test value plus 1. All the other cutoff values are the averages of two consecutive ordered observed test values.

Karl L. Wuensch, 28-October-2017