All Roads Lead to F: the F - Ratio in Regression

All Roads Lead to F: the F - Ratio in Regression

Chi-Square Random Variables and the F Distribution

If the errors in a regression model are independentrandom variables, then , where is the Chi-square distribution with 1 degree of freedom (also written ). Similarly, . Then the F - Ratio, , that appears in the ANOVA table is the ratio of two independent chi-square distributions divided by their respective degrees of freedom. Under the model assumptions, the F - Ratio follows an F distribution with degrees of freedomand. There are various places in regression where F - Ratios make an appearance.

The F - Ratio for the Simple Linear Regression Model (ANOVA Table)

The hypothesis for the test in simple linear regression (SLR) takes the form;. As with all hypothesis tests, the test is conducted under the assumption that the null hypothesis,, is correct. Assuming the errors are iid, the following are all true under the null hypothesis:

Under the null hypothesis, the following are also true:

So, under the null hypothesis of the test we expect the F - Ratio to be about 1. Clearly, an F - Ratio very different from one is evidence for the alternative hypothesis (which is the working hypothesis of SLR). The question is: is the test left-, right-, or two-tailed. The answer is: it's right-tailed. The reason is that under the alternative hypothesis,, the following are true:

Notice that underthe expected mean square for the model is greater than, so the F - Ratio is expected to be greater than 1. Hence, we rejectin favor of, i.e., we accept the SLR model, for values of F significantly greater than 1. Although we could use a table of critical values of the F distribution withandto conduct the test at a fixed significance level,, we'll rely on the P-value in the ANOVA table of computer output.

Notice thatunder the alternative hypothesis. So we are more likely to be able to reject the null hypothesis (and conclude that there is a linear association between the X and Y variables) if either of the following are true:

, i.e., the magnitude of the slope of the line is large
. As discussed in previous notes, the uncertainty in fixing the true slope,, is inversely proportional to the square root of. The less uncertainty there is in fixing the true slope, the greater the power of any test of the slope.

The F -Ratio and Testing the Slope - Part II

In simple linear regression (SLR), there are actually two tests of the model. The first is the F-test of the model that appears in the ANOVA table discussed in the previous section. The second is the t-test of the slope coefficient that appears in the table of coefficients. The null and alternative hypotheses for this test are identical to those for the model,;. Since the t-test of the slope and the F-test of the model share hypotheses, we would like to think that they are related! It turns out that they are related in a very simple way. An F distribution withandis equivalent to the square of a t distribution with. Thus the square of the t test statistic of the slope equals the Ftest statistic of the model in simple regression. You should verify this for a few examples to convince yourself. Furthermore,the P-values for the two tests are always identical.

The F - Ratio as a Test for (Almost) Everything

One fine point that I've avoided discussing thus far is why we are justified in assuming that the two chi-square random variables in the F - Ratio in the ANOVA table are independent, which they must be for the ratio to have an F distribution. We will justify this in class with the aid of my usual picture of vectors in. In fact, we've already shown in class that the vectors involved in SSR and SSE,and, live in orthogonal vector subspaces of. This is the reason that their respective sums of squares are independent. Thus the two chi-square random variables formed from SSR and SSE must also be independent.

This leads us to a more general discussion of the role of F-tests in regression. We begin with restating the hypothesis test in simple linear regression (SLR) in a slightly different form that can be easily generalized to form a hypothesis test for a multiple linear regression (MLR) model.

F-test of the Model in Simple Linear Regression

We think of the test as choosing between two competing models:

the "Base" model , where is, in fact, just the unconditional mean of Y, i.e., the model is .
the "Full" model , i.e., the SLR model.

The null and alternative hypotheses for the test are:



Call the ith "residual" for the "Base" model. Then is the vector of these "residuals." Meanwhile, is the ith residual for the SLR "Full" model, and is the vector of residuals, as before. Finally, for completeness, we'll call the vector "between" the base and full models.

The vectors defined in the previous paragraph each lead to a sum of squares:

SST = , with
SSE = , with
SSR = , with , where we've used the relationships between SSR, SSE, and SST originating in the right triangle informed by the vectors,, and .

Then the F-test of the base versus full models uses the F - Ratio: , and we reject the null hypothesis in favor of accepting the SLR model for as before. We'll rely on the P-value for the test found in the ANOVA table in determining whether the observed value of the F - Ratio issignificant.

Recurring Example:For the Securicorp example in the multiple regression notes,the following represents the output for a simple linear regression of the dependent variable Sales, in thousands of dollars, on Advertizing, in hundreds of dollars:(I've boldfaced MSR, MSE, and the F - Ratio in the output for ease of identification.) Note: The value of 1.01185E6 reported for the regression mean square is shorthand for 1.01185 x 106 , or 1,011,850. You should verify that in the output.

Simple Regression - Sales vs. Ad

Dependent variable: Sales

Independent variable: Ad

Linear model: Y = a + b*X

Coefficients

Least Squares / Standard / T
Parameter / Estimate / Error / Statistic / P-Value
Intercept / -155.253 / 145.459 / -1.06733 / 0.2969
Slope / 2.76789 / 0.279929 / 9.88781 / 0.0000

Analysis of Variance

Source / Sum of Squares / Df / Mean Square / F-Ratio / P-Value
Model / 1.01185E6 / 1 / 1.01185E6 / 97.77 / 0.0000
Residual / 238036. / 23 / 10349.4
Total (Corr.) / 1.24988E6 / 24

F-test of the Model in Multiple Linear Regression

This generalizes to give anF-test of a multiple linear regression. We test the "Base" modelagainst the "Full" model . The hypotheses are:

, i.e., all slopes equal zero. Thus none of theare linearly related to.

at least one of theis not zero, i.e., some of theare linearly related to.

Note that the null hypothesis, , represents the "Base" model in the test, and the alternative hypothesis represents the "Full" multiple regression model. If we define as the vector of "residuals" from the "Base" model, as the vector of residuals from the "Full" model, and as the vector "between" the base and full models, then we get the sums of squares:

SST = , with
SSE = , with
SSR = , with , where SSR + SSE = SST as in Simple Regression

Then the F-test of the base versus full models uses the F - Ratio: . As in SLR, the test is right-tailed, i.e., is associated with rejecting the null hypothesis in favor of the "Full" model. Once again, we'll rely on the P-value in the ANOVA table to evaluate the results of the test.

Recurring Example:For the Securicorp example in the multiple regression notes,the output below represents a multiple regression model for the dependent variable Sales, in thousands of dollars, on the independent variables: (Statistics discussed in these notes have been boldfaced.)

Ad: Advertising, in hundreds of dollars
Bonus: Bonuses, in hundreds of dollars
West: The Western sales territory dummy variable
Midwest: the Midwestern sales territory dummy variable

Multiple Regression - Sales

Dependent variable: Sales

Independent variables:

Bonus

West

Midwest

Coefficients

Parameter / Estimate / Standard Error / T Statistic / P-Value
CONSTANT / 439.193 / 206.222 / 2.1297 / 0.0458
Ad / 1.36468 / 0.26179 / 5.21287 / 0.0000
Bonus / 0.96759 / 0.480814 / 2.0124 / 0.0578
West / -258.877 / 48.4038 / -5.34827 / 0.0000
Midwest / -210.456 / 37.4223 / -5.62382 / 0.0000

Analysis of Variance

Source / Sum of Squares / Df / Mean Square / F-Ratio / P-Value
Model / 1.18325E6 / 4 / 295812. / 88.79 / 0.0000
Residual / 66632.8 / 20 / 3331.64
Total (Corr.) / 1.24988E6 / 24

F-tests of the Slope Coefficients in Multiple Regression

Although the coefficients table in multiple regression output reports t-values used in t-tests of the marginal utility of the independent variables, we'll develop the corresponding F-tests instead. I do this not because the F-tests are superior, but rather so that we may develop a uniform treatment of all of the tests reported in the analysis window.

Without loss of generality, suppose that the variable whose marginal effect on the model we are testing is. Since the test involves considering the informationbrings to a model in which the other k - 1 variables are already present, it is equivalent to comparing the "Full" modelto the "Reduced" model in whichhas been deleted. The procedure is identical to that we've used several times already, but with sums of squares and degrees of freedom appropriate to the models being compared. (This is why I can get by with drawing the same pathetic picture for every F-test by simply relabeling the vectors.) The results below should, by now, look familiar (but pay attention to the new terminology and degrees of freedom).

The null and alternative hypotheses for the test are:

, which makes a claim about the marginal effect of in predicting Y in the model. It does not imply that might not be correlated to Y in a different model, for example, a simple linear regression.



If we define as the vector of residuals from the "Reduced" model, as the vector of residuals from the "Full" model, and as the vector "between" the two models, we get:

SSEReduced = , with , where SSEReduced is the sum of squared residuals for the Reduced model
SSEFull = , with ,where SSEFull is the sum of squared residuals for the Full model
SSR = , with , where SSR is sometimes called the "Extra" sum of squares because it's the extra portion of the regression sum of squares that is added whenis included in the model.

Then the F-test of the marginal effect that has on the model uses the F - Ratio: . As always, the test is right-tailed, i.e., is associated with rejecting the null hypothesis in favor of the "Full" model, i.e., of including in the model. Once again, we'll rely on the P-value in the ANOVA table to evaluate the results of the test.

The t-statistic found in the table of coefficients in the row corresponding to an independent variable is drawn from a t distribution with n - k - 1 degrees of freedom, and thus the P-value for a t-test of the slope of the variable will equal the P-value for an F-test of the slope in the preceding paragraph.

Recurring Example:For the Securicorp example, Suppose we wish to test the marginal contribution of the bonuses to the Full multiple regression model run in the previous section. The Reduced model omits the variable Bonus. Below, I've included the ANOVA tables for both models side-by-side for easy comparison.

Analysis of Variance - Full Model (Includes "Bonus")
Source / Sum of Squares / Df / Mean Square / F-Ratio / P-Value
Model / 1.18325E6 / 4 / 295812. / 88.79 / 0.0000
Residual / 66632.8 / 20 / 3331.64
Total (Corr.) / 1.24988E6 / 24
/ Analysis of Variance - Reduced Model (Excludes "Bonus")
Source / Sum of Squares / Df / Mean Square / F-Ratio / P-Value
Model / 1.16976E6 / 3 / 389919. / 102.19 / 0.0000
Residual / 80125.1 / 21 / 3815.48
Total (Corr.) / 1.24988E6 / 24

Then , where . Using my calculator, which has an F distribution utility (thank you TI), the P-value for this right-tailed test is 0.0578, which is the P-value reported for Bonus in the coefficients table for the full model. Notice also, that the square of the t statistic reported for the bonus variable equals the F statistic computed above, i.e., (2.0124)2 = 4.050

Another F'ing Test!

Just when you thought there couldn't possibly be another test!!! The test in the previous section can be extended to test any subset of independent variables in multiple regression. This would typically be done when the variables have some attribute in common, such as when a collection of m - 1 dummy variables are included to measure the effect of a categorical (qualitative) variable with m levels. The full model includes all of the variables being tested, while the reduced model omits them. If, without loss of generality, wesuppose that the set of p variables being tested are , then the null and alternative hypotheses for the test are:

, given that the effect of theotherk - p independent variables upon Y is already accounted for (in the reduced model).

 for some , i.e., at least one of the variables in the set being tested remains correlated to Y in the model even after conditioning on the effect of the otherof k - pvariables on the observed variation of the dependent variable.

Letting be the vector of residuals in the "Reduced" model, be the vector of residuals in the "Full" model, and be the vector "between" the two models:

SSEReduced = , with , where SSEReduced is the sum of squared residuals for the Reduced model. Remember:(the number of Betas that must be estimated in the model).
SSEFull = , with ,where SSEFull is the sum of squared residuals for the Full model
SSR = , with , where SSR is called the "Extra" sum of squares in the test.

Finally,the F-test for the set of p independent variables being tested uses the F - Ratio: . As always, the test is right-tailed, i.e., is associated with rejecting the null hypothesis in favor of the "Full" model, i.e., of including the set of p independent variables in the model. Once again, we'll rely on the P-value in the ANOVA table to evaluate the results of the test.

Note: The F-test described in the previous paragraph has no t-test analogue! (The relationship between the t and F distributions discussed earlier applies only to the case wherefor the F distribution, but here .)

Recurring Example:For the Securicorp example, Suppose we wish to test the marginal contribution of the regions to the Full multiple regression model run in the previous section. The Reduced model omits the variablesWest and Midwest. Below, I've included the ANOVA tables for both models side-by-side for easy comparison.

Analysis of Variance - Full Model (Includes West &Midwest)
Source / Sum of Squares / Df / Mean Square / F-Ratio / P-Value
Model / 1.18325E6 / 4 / 295812. / 88.79 / 0.0000
Residual / 66632.8 / 20 / 3331.64
Total (Corr.) / 1.24988E6 / 24
/ Analysis of Variance - Reduced Model (Excludes West &Midwest)
Source / Sum of Squares / Df / Mean Square / F-Ratio / P-Value
Model / 1.06722E6 / 2 / 533609. / 64.27 / 0.0000
Residual / 182665. / 22 / 8302.95
Total (Corr.) / 1.24988E6 / 24

Then , where . Using my calculator, the P-value for this right-tailed test is 0.00004, which confirms the contribution of West and Midwest to the full model. Of course, the astute among you may have surmised that since both West and Midwest received small P-values in the full model, the two of them taken together were bound to be significant in this test! But hey, it's only an example.