Multicollinearity Exercise
Use the attached SAS output to answer the questions. [OPTIONAL: Copy the SAS program below into the SAS editor window and run it.] You do not need to submit any output, so there is no need to print anything..
1. Identify which variables are key participants in the most serious near linear dependency in the data. Hint: Look at the Variance Decomposition Proportions associated with the smallest eigenvalue of X’X.
2. Which variable has the “wrong sign” for its coefficient in this regression? Explain why its sign is wrong.
3. What is the smallest value of the ridge constant (k) that “fixes” the sign of the coefficient you named in #2?
4. What is the smallest value of the ridge constant (k) that reduces all VIF’s so that they are below the guideline of 10?
5. What is the smallest value of k that seems (in your opinion) to stabilize the coefficients?
6. If one principal component is removed, give the estimated coefficients for X1, X2, X3, X4. Does this fix the one with the “wrong” sign?
*******************************************************************
************ LAWSCHOOL ADMISSION DATA ******************
**************** PARTLY FROM PAGE 599 OF SMITH ***************
*******************************************************************;
**** DATA FOR 20 STUDENTS ******
Y IS THE LAW SCHOOL GPA
X1 IS THE UNDERGRADUATE SCHOOL GPA
X2 IS THE LMAT PERCENTILE
X3 IS A RATING OF THE UNDERGRADUATE SCHOOL QUALITY
X4 IS THE GRE SCORE;
DATA LAW; INPUT Y X1 X2 X3 X4 NO $; CARDS;
3.42 3.28 .96 6 1330 1
3.60 3.18 .97 7 1370 2
3.28 2.89 .93 5 1140 3
3.75 3.72 .99 8 1520 4
3.36 3.18 .95 6 1270 5
3.96 3.50 .98 8 1450 6
3.31 3.04 .94 5 1200 7
3.33 3.87 .95 5 1340 8
3.60 3.54 .96 7 1350 9
4.00 3.27 .99 10 1480 a
3.28 3.30 .95 5 1280 b
3.44 3.29 .91 7 1080 c
3.25 3.17 .93 5 1170 d
3.75 3.62 .97 8 1410 e
3.30 3.34 .96 5 1330 f
3.20 3.08 .90 4 1010 g
3.50 3.37 .96 6 1340 h
3.28 3.16 .94 5 1220 i
3.17 3.20 .95 4 1270 j
3.31 3.10 .94 5 1210 k
;
TITLE 'LAWSCHOOL ADMISSIONS DATA';
PROC CORR; VAR Y X1 X2 X3 X4;
PROC REG; MODEL Y=X1 X2 X3 X4 / COLLIN VIF;
PROC REG RIDGE = 0 TO .01 BY .001 OUTEST=B;
MODEL Y=X1 X2 X3 X4 ;
PROC PRINT;
PROC PLOT; PLOT (X1 X2 X3 X4) * _RIDGE_ / VREF=0 VPOS=25 HPOS=45;
PROC REG DATA = LAW RIDGE= 0 TO .01 BY .001 OUTEST=C OUTVIF;
MODEL Y=X1 X2 X3 X4;
PROC PRINT;
PROC REG DATA = LAW PCOMIT=1 2 3 OUTEST=C;
MODEL Y=X1 X2 X3 X4;
PROC PRINT;
run;
LAW SCHOOL ADMISSIONS DATAThe CORR Procedure
5 Variables: / Y X1 X2 X3 X4Simple Statistics
Variable / N / Mean / Std Dev / Sum / Minimum / Maximum
Y / 20 / 3.45450 / 0.24522 / 69.09000 / 3.17000 / 4.00000
X1 / 20 / 3.30500 / 0.24150 / 66.10000 / 2.89000 / 3.87000
X2 / 20 / 0.95150 / 0.02346 / 19.03000 / 0.90000 / 0.99000
X3 / 20 / 6.05000 / 1.57196 / 121.00000 / 4.00000 / 10.00000
X4 / 20 / 1289 / 131.15981 / 25770 / 1010 / 1520
Pearson Correlation Coefficients, N = 20
Prob > |r| under H0: Rho=0
Y / X1 / X2 / X3 / X4
Y / 1.00000
/ 0.47331
0.0350
/ 0.76094
<.0001
/ 0.95925
<.0001
/ 0.76574
<.0001
X1 / 0.47331
0.0350
/ 1.00000
/ 0.52911
0.0164
/ 0.42078
0.0647
/ 0.65377
0.0018
X2 / 0.76094
<.0001
/ 0.52911
0.0164
/ 1.00000
/ 0.69724
0.0006
/ 0.98781
<.0001
X3 / 0.95925
<.0001
/ 0.42078
0.0647
/ 0.69724
0.0006
/ 1.00000
/ 0.69983
0.0006
X4 / 0.76574
<.0001
/ 0.65377
0.0018
/ 0.98781
<.0001
/ 0.69983
0.0006
/ 1.00000
LAW SCHOOL ADMISSIONS DATA
The REG Procedure
Model: MODEL1
Dependent Variable: Y
Number of Observations Read / 20Number of Observations Used / 20
Analysis of Variance
Source / DF / Sum of
Squares / Mean
Square / F Value / PrF
Model / 4 / 1.07143 / 0.26786 / 56.54 / <.0001
Error / 15 / 0.07106 / 0.00474
Corrected Total / 19 / 1.14249
Root MSE / 0.06883 / R-Square / 0.9378
Dependent Mean / 3.45450 / Adj R-Sq / 0.9212
Coeff Var / 1.99243
Parameter Estimates
Variable / DF / Parameter
Estimate / Standard
Error / tValue / Pr|t| / Variance
Inflation
Intercept / 1 / -2.37864 / 24.38266 / -0.10 / 0.9236 / 0
X1 / 1 / 0.12572 / 0.64288 / 0.20 / 0.8476 / 96.67364
X2 / 1 / 6.05826 / 32.14872 / 0.19 / 0.8531 / 2280.94358
X3 / 1 / 0.12977 / 0.01417 / 9.16 / <.0001 / 1.99015
X4 / 1 / -0.00087848 / 0.00646 / -0.14 / 0.8937 / 2880.98384
Collinearity Diagnostics
Number / Eigenvalue / Condition
Index / Proportion of Variation
Intercept / X1 / X2 / X3 / X4
1 / 4.95347 / 1.00000 / 1.614169E-8 / 0.00000212 / 1.027655E-8 / 0.00120 / 1.375895E-7
2 / 0.04096 / 10.99716 / 0.00000115 / 0.00006334 / 4.377599E-7 / 0.56236 / 6.249609E-8
3 / 0.00348 / 37.70759 / 0.00003532 / 0.00216 / 0.00000613 / 0.31407 / 0.00041449
4 / 0.00209 / 48.66811 / 3.791043E-7 / 0.01504 / 0.00000729 / 0.11173 / 0.00052977
5 / 1.475688E-7 / 5793.71812 / 0.99996 / 0.98274 / 0.99999 / 0.01064 / 0.99906
LAW SCHOOL ADMISSIONS DATA
The REG Procedure
Model: MODEL1
Dependent Variable: Y
LAW SCHOOL ADMISSIONS DATAThe REG Procedure
Model: MODEL1
Dependent Variable: Y
LAW SCHOOL ADMISSIONS DATAMODEL / TYPE / DEPVAR / RIDGE / RMSE / Intercept / X1 / X2 / X3 / X4 / Y
1 / MODEL1 / PARMS / Y / . / 0.068829 / -2.37864 / 0.12572 / 6.05826 / 0.12977 / .000878480 / -1
2 / MODEL1 / RIDGE / Y / 0.000 / 0.068829 / -2.37864 / 0.12572 / 6.05826 / 0.12977 / .000878480 / -1
3 / MODEL1 / RIDGE / Y / 0.001 / 0.068871 / 0.89844 / 0.03989 / 1.73602 / 0.12932 / .000007758 / -1
4 / MODEL1 / RIDGE / Y / 0.002 / 0.068880 / 1.17880 / 0.03249 / 1.36524 / 0.12907 / 0.000068635 / -1
5 / MODEL1 / RIDGE / Y / 0.003 / 0.068886 / 1.28047 / 0.02977 / 1.23008 / 0.12883 / 0.000097650 / -1
6 / MODEL1 / RIDGE / Y / 0.004 / 0.068892 / 1.33140 / 0.02838 / 1.16182 / 0.12859 / 0.000113203 / -1
7 / MODEL1 / RIDGE / Y / 0.005 / 0.068899 / 1.36094 / 0.02755 / 1.12178 / 0.12836 / 0.000123073 / -1
8 / MODEL1 / RIDGE / Y / 0.006 / 0.068907 / 1.37947 / 0.02700 / 1.09627 / 0.12813 / 0.000130011 / -1
9 / MODEL1 / RIDGE / Y / 0.007 / 0.068915 / 1.39160 / 0.02663 / 1.07920 / 0.12790 / 0.000135238 / -1
10 / MODEL1 / RIDGE / Y / 0.008 / 0.068925 / 1.39968 / 0.02637 / 1.06748 / 0.12767 / 0.000139379 / -1
11 / MODEL1 / RIDGE / Y / 0.009 / 0.068935 / 1.40504 / 0.02617 / 1.05935 / 0.12744 / 0.000142787 / -1
12 / MODEL1 / RIDGE / Y / 0.010 / 0.068947 / 1.40850 / 0.02603 / 1.05373 / 0.12721 / 0.000145676 / -1
LAW SCHOOL ADMISSIONS DATA
Plot of X1*_RIDGE_. Legend: A = 1 obs, B = 2 obs, etc.
X1 ‚
‚
‚
‚
‚
0.15 ˆ
‚
‚
‚ A
‚
‚
0.10 ˆ
‚
‚
‚
‚
‚
0.05 ˆ
‚ A
‚ A A
‚ A A A A A A A
‚
‚
0.00 ˆƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ
‚
Šƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒ
0.000 0.002 0.004 0.006 0.008 0.010
Ridge regression control value
NOTE: 1 obs had missing values.
LAW SCHOOL ADMISSIONS DATA
Plot of X2*_RIDGE_. Legend: A = 1 obs, B = 2 obs, etc.
X2 ‚
‚
‚
‚
‚
6 ˆ A
‚
‚
‚
‚
‚
4 ˆ
‚
‚
‚
‚
‚
2 ˆ
‚ A
‚ A A
‚ A A A A A A A
‚
‚
0 ˆƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ
‚
Šƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒ
0.000 0.002 0.004 0.006 0.008 0.010
Ridge regression control value
NOTE: 1 obs had missing values.
LAW SCHOOL ADMISSIONS DATA
Plot of X3*_RIDGE_. Legend: A = 1 obs, B = 2 obs, etc.
X3 ‚
‚
‚
‚
‚
0.15 ˆ
‚
‚ A A
‚ A A A A A A A A A
‚
‚
0.10 ˆ
‚
‚
‚
‚
‚
0.05 ˆ
‚
‚
‚
‚
‚
0.00 ˆƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ
‚
Šƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒ
0.000 0.002 0.004 0.006 0.008 0.010
Ridge regression control value
NOTE: 1 obs had missing values.
LAW SCHOOL ADMISSIONS DATA
Plot of X4*_RIDGE_. Legend: A = 1 obs, B = 2 obs, etc.
X4 ‚
‚
‚
0.00025 ˆ
‚
‚ A A A A A A A A
‚ A
0 ˆƒƒƒƒƒAƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ
‚
‚
‚
-0.00025 ˆ
‚
‚
‚
-0.0005 ˆ
‚
‚
‚
-0.00075 ˆ
‚
‚ A
‚
-0.001 ˆ
‚
Šƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒˆƒƒƒ
0.000 0.002 0.004 0.006 0.008 0.010
Ridge regression control value
NOTE: 1 obs had missing values.
LAW SCHOOL ADMISSIONS DATA
MODEL / TYPE / DEP / RIDGE / RMSE / Intcpt / X1 / X2 / X3 / X4
1 / MODEL1 / PARMS / Y / . / . / 0.068829 / 2.37864 / 0.1257 / 6.06 / 0.12977 / -0.00
2 / MODEL1 / RIDGEVIF / Y / 0.000 / . / . / . / 96.6736 / 2280.94 / 1.99015 / 2880.98
3 / MODEL1 / RIDGE / Y / 0.000 / . / 0.068829 / 2.37864 / 0.1257 / 6.06 / 0.12977 / -0.00
4 / MODEL1 / RIDGEVIF / Y / 0.001 / . / . / . / 3.9053 / 59.05 / 1.95543 / 74.08
5 / MODEL1 / RIDGE / Y / 0.001 / . / 0.068871 / 0.89844 / 0.0399 / 1.74 / 0.12932 / -0.00
6 / MODEL1 / RIDGEVIF / Y / 0.002 / . / . / . / 2.1861 / 17.99 / 1.94548 / 22.21
7 / MODEL1 / RIDGE / Y / 0.002 / . / 0.068880 / 1.17880 / 0.0325 / 1.37 / 0.12907 / 0.00
8 / MODEL1 / RIDGEVIF / Y / 0.003 / . / . / . / 1.8012 / 8.89 / 1.93597 / 10.72
9 / MODEL1 / RIDGE / Y / 0.003 / . / 0.068886 / 1.28047 / 0.0298 / 1.23 / 0.12883 / 0.00
10 / MODEL1 / RIDGEVIF / Y / 0.004 / . / . / . / 1.6537 / 5.48 / 1.92660 / 6.41
11 / MODEL1 / RIDGE / Y / 0.004 / . / 0.068892 / 1.33140 / 0.0284 / 1.16 / 0.12859 / 0.00
12 / MODEL1 / RIDGEVIF / Y / 0.005 / . / . / . / 1.5803 / 3.84 / 1.91732 / 4.34
13 / MODEL1 / RIDGE / Y / 0.005 / . / 0.068899 / 1.36094 / 0.0275 / 1.12 / 0.12836 / 0.00
14 / MODEL1 / RIDGEVIF / Y / 0.006 / . / . / . / 1.5373 / 2.93 / 1.90811 / 3.19
15 / MODEL1 / RIDGE / Y / 0.006 / . / 0.068907 / 1.37947 / 0.0270 / 1.10 / 0.12813 / 0.00
16 / MODEL1 / RIDGEVIF / Y / 0.007 / . / . / . / 1.5090 / 2.37 / 1.89898 / 2.48
17 / MODEL1 / RIDGE / Y / 0.007 / . / 0.068915 / 1.39160 / 0.0266 / 1.08 / 0.12790 / 0.00
18 / MODEL1 / RIDGEVIF / Y / 0.008 / . / . / . / 1.4887 / 2.00 / 1.88992 / 2.02
19 / MODEL1 / RIDGE / Y / 0.008 / . / 0.068925 / 1.39968 / 0.0264 / 1.07 / 0.12767 / 0.00
20 / MODEL1 / RIDGEVIF / Y / 0.009 / . / . / . / 1.4731 / 1.74 / 1.88093 / 1.70
21 / MODEL1 / RIDGE / Y / 0.009 / . / 0.068935 / 1.40504 / 0.0262 / 1.06 / 0.12744 / 0.00
22 / MODEL1 / RIDGEVIF / Y / 0.010 / . / . / . / 1.4606 / 1.55 / 1.87201 / 1.47
23 / MODEL1 / RIDGE / Y / 0.010 / . / 0.068947 / 1.40850 / 0.0260 / 1.05 / 0.12721 / 0.00
LAW SCHOOL ADMISSIONS DATA
Obs / MODEL / TYPE / DEP / PCOMIT / RMSE / Intcpt / X1 / X2 / X3 / X4
1 / MODEL1 / PARMS / Y / . / 0.06883 / -2.3786 / 0.1257 / 6.0583 / 0.1298 / -.0008785
2 / MODEL1 / IPC / Y / . / 1 / 0.06670 / 1.5276 / 0.0235 / 0.9076 / 0.1295 / 0.0001569
3 / MODEL1 / IPC / Y / . / 2 / 0.10775 / -0.6633 / -0.1375 / 3.6579 / 0.0658 / 0.0005384
4 / MODEL1 / IPC / Y / . / 3 / 0.13095 / -0.7603 / 0.2087 / 2.7821 / 0.0357 / 0.0005138