Learning Statistics with R - A tutorial for psychology students and other beginners, 2018a

16.5.1 The F test comparing two models
Let’s frame this in a slightly more abstract way. We’ll say that our full model can be written as an R
formula that contains several different terms, say Y ~ A + B + C + D. Our null model only contains some
subset of these terms, say Y ~ A + B. Some of these terms might be main effect terms, others might be
interaction terms. It really doesn’t matter. The only thing that matters here is that we want to treat
some of these terms as the “starting point” (i.e. the terms in the null model, A and B), and we want to
see if including the other terms (i.e., C and D) leads to a significant improvement in model performance,
over and above what could be achieved by a model that includes only A and B. In essence, we have null
and alternative hypotheses that look like this:
Hypothesis Correct model? R formula for correct model
Null M0 Y ~ A + B
Alternative M1 Y ~ A + B + C + D
Is there a way of making this comparison directly?
To answer this, let’s go back to fundamentals. As we saw in Chapter 14, theF -test is constructed
from two kinds of quantity: sums of squares (SS) and degrees of freedom (df). These two things define a
mean square value (MS = SS/df), and we obtain our F statistic by contrasting the MS value associated
with “the thing we’re interested in” (the model) with the MS value associated with “everything else”
(the residuals). What we want to do is figure out how to talk about the SS value that is associated with
the difference between two models. It’s actually not all that hard to do.
Let’s start with the fundamental rule that we used throughout the chapter on regression:
SS T “ SS M ` SS R
That is, the total sums of squares (i.e., the overall variability of the outcome variable) can be decomposed
into two parts: the variability associated with the model SS M , and the residual or leftover variability,
SS R . However, it’s kind of useful to rearrange this equation slightly, and say that the SS value associated
with a model is defined like this...
SS M “ SS T ´ SS R
Now, in our scenario, we have two models: the null model (M0) and the full model (M1):
SS M0 “ SS T ´ SS R0
SS M1 “ SS T ´ SS R1
Next, let’s think about what it is we actually care about here. What we’re interested in is the difference
between the full model and the null model. So, if we want to preserve the idea that what we’re doing
is an “analysis of the variance” (ANOVA) in the outcome variable, what we should do is define the SS
associated with the difference to be equal to the difference in the SS:
SS Δ “ SS M1 ´ SS M0
“ pSS T ´ SS R1 q´pSS T ´ SS R0 q
“ SS R0 ´ SS R1
Now that we have our degrees of freedom, we can calculate mean squares and F values in the usual
way. Specifically, we’re interested in the mean square for the difference between models, and the mean
square for the residuals associated with the full model (M1), which are given by
MS Δ “ SS Δ
df Δ
- 519 -