Theory of Statistics - George Mason University

208 3 Basic Statistical Theory
variables. Both the random variable of interest and any covariates are assumed
to be observable. There may also be unobservable variables, called “latent
variables”, associated with the observed data. Such variables are artifacts of
the statistical model and may or may not correspond to phenomena of interest.
When covariates are present our interest usually is in the conditional distribution
of Y , given X. For making statistical inferences, we generally assume
that the conditional distributions of Y1|X1, . . ., Yn|Xn are either conditionally
independent or at least conditionally exchangeable.
A statistic is any function T of the observables that does not involve any
unobservable values. We often use a subscript Tn to indicate the number of
observations, but usually a statistic is defined as some formula that applies to
a general number of observations (such as the sample mean). While we most
often work with statistics based on a random sample, that is, an iid set of
variables, or at least based on an exchangeable sample, we may have a statistic
that is a function of a general set of random variables, X1, . . ., Xn. We see
that if the random variables are exchangeable, then the statistic is symmetric,
in the sense that T(Xk1, . . ., Xkn) = T(X1, . . ., Xn) for any indices k1, . . ., kn
such that {k1, . . ., kn} = {1, . . ., n}.
Statistical Models
We assume that the sample arose from some distribution Pθ, which is a member
of some family of probability distributions P. The family of probability
distributions P is a statistical model. We fully specify the family P (it can be
a very large family), but we assume some aspects of Pθ are unknown. (If the
distribution Pθ that yielded the sample is fully known, while there may be
some interesting questions about probability, there are no interesting statistical
questions.) Our objective in statistical inference is to determine a specific
Pθ ∈ P, or some subfamily Pθ ⊆ P, that could likely have generated the
sample.
The distribution may also depend on other observable variables. In general,
we assume we have observations Y1, . . ., Yn on Y , together with associated observations
on any related variable X or x. We refer to the associated variables
as “covariates”. In this context, a statistic, which in our common use of the
term is a function that does not involve any unobserved values, may also
involve the observed covariates.
A general statistical model that includes covariates is
Y = f(x ; θ) + E, (3.5)
where Y and x are observable variables, f is some unknown function, θ is an
unknown parameter, and E is an unobservable random variable with unknown
distribution Pτ independent of other quantities in the model. In the usual
setup, Y is a scalar random random variable, and x is a p-vector. Given
independent observations (Y1, x1), . . ., (Yn, xn), we often use the notation Y
Theory of Statistics c○2000–2013 James E. Gentle