1

9.4 Automatic Search Procedures for Model Selection

(a) Backward elimination:

Assume the model with all possible covariates is

.

Backward elimination procedure:

Step 1:

At the beginning, the original model is set to be

.

Then, the following r-1 tests are carried out, The lowest partial F-test value corresponding to or t-test value is compared with the preselected significance values and . One of two possible steps (step2a and step 2b) can be taken.

Step 2a:

If or , then can be deleted and the new original model is

.

Go back to step 1.

Step 2b:

If or , the original model is the model we should choose.

Example (continue):

Suppose the preselected significance level is Thus,

Step 1:

The original model is

.

corresponding to is the smallest partial F value.

Step 2a:

.

Thus, can be deleted. Go back to step 1.

Step 1:

The new original model is

.

corresponding to is the smallest partial F value.

Step 2a:

.

Thus, can be deleted. Go back to step 1.

Step 1:

The new original model is

.

corresponding to is the smallest partial F value.

Step 2b:

.

Thus,

,

is the selected model.

(b) Stepwise regression:

Stepwise regression procedure employs some statistical quantity, partial correlation, to add new covariate. We introduce partial correlation first.

Partial correlation:

Assume the model is

.

The partial correlation of and , denoted by

,

can be obtained as follows:

  1. Fit the model

obtain the residuals

.

Also, fit the model

obtain the residuals

.

2.

,

where

and .

Stepwise regression procedure:

The original model is . There are r-1 covariates, .

Step 1:

Select the variable most correlated Y, say , based on the correlation coefficient. Fit the model

and check if is significant. If not, then

,

isthe bestmodel. Otherwise, the new original model is

and go to step 2.

Step 2:

Examine the partial correlation . Find the covariate with largest value of partial correlation Then, fit

and obtain partial F-value, corresponding to and

corresponding to . Go to step 3.

Step 3:

The smallest partial F-value (one of and ) is compared with the preselected significance value. There are two possibilities:

(a)

If , then delete the covariate corresponding to . Go back to step 2. Note that if , then examine the partial correlation

.

(b)

If , then

,

is the new original model. Then, go back to step 2, but now examine the partial correlation .

The procedure will automatically stop when no variable in the new original model can be removed and all the next best candidate can not be retained in the new original model. Then, the new original model is our selected model.