Theory of Statistics - George Mason University

4.2 Bayesian Analysis 327
even if they don’t think of Θ as a random variable. In the Bayesian approach
to testing and determining confidence sets, we do think of the parameter as a
random variable and so we can make statements about the probability of the
parameter taking on certain values.
If the parameter is a random variable, especially if it is a continuous random
variable, point estimation of the parameter or a test of an hypothesis that
a parameter takes a specific value when the parameter is modeled as a continuous
random variable does not make much sense. The idea of a point estimator
that formally minimizes the Bayes risk, however, remains viable. Going beyond
point estimation, the Bayesian paradigm provides a solid theoretical
infrastructure for other aspects of statistical inference, such as confidence intervals
and tests of hypotheses. The parameter random variable is different in
a fundamental way from the other random variable in the estimation problem:
the parameter random variable is not observable; the other random variable
is — that is, we can observe and record realizations of this random variable
of interest, and those observations constitute the sample, which is the basis
for the statistical inference.
4.2 Bayesian Analysis
The starting point in ordinary Bayesian inference is the conditional distribution
of the observable random variable. (In a frequentist approach, this is just
the distribution — not the “conditional” distribution.)
The prior density represents a probability distribution of the parameter
assumed a priori, that is, without the information provided by a random
sample. Bayesian inference depends on the conditional distribution of the
parameter, given data from the random variable of interest.
4.2.1 Theoretical Underpinnings
The relationships among the conditional, marginal, and joint distributions
can be stated formally in the “Bayes formula”. The simple relationship of
probabilities of events as in equations (1.227) and (1.228) allows us to express
a conditional probability in terms of the two marginal probabilities and the
conditional probability with the conditioning reversed;
Pr(A|B) = Pr(B|A)Pr(A)
. (4.11)
Pr(B)
This relationship expresses the basic approach in Bayesian statistical analysis.
Instead of probabilities of discrete events, however, we wish to utilize
relationships among probability densities.
We consider the random variable X with range X ⊆ IR d and Θ with range
Θ ⊆ IR k . We consider the product space X × Θ together with the product
Theory of Statistics c○2000–2013 James E. Gentle