3. Basic probability concepts

From: R.E. Strauss, Statistical Hypothesis Testing in the Biological Sciences, in preparation.
All rights reserved.
3. Basic probability concepts
It is evident that the primes are randomly distributed but, unfortunately, we
don't know what ‘random’ means. [R.C. Vaughan, 1990]
When there is no explanation, they give it a name, which immediately explains
everything. [H. Fabing and R. Mar, 1937]
Probability is a concept used to express the likelihood of an event. Qualitatively, adjectives such
as “remote”, “poor”, “weak”, “good” or “strong” are often used to indicate the relative likelihood
of an event. Quantitatively, probability is expressed as a fraction or decimal value between 0 and
1 or, equivalently, as a percentage between 0% and 100%. A probability of 0 implies that the
event cannot occur, while a probability of 1 implies that it is certain to occur.
Note that in the paragraph above I said that probability is used to express the likelihood of
an event, as if use of the term likelihood served to define the term probability. Actually, the
terms probability and likelihood are often used interchangeably (although likelihood also has a
more restricted technical definition in terms of conditional probabilities), and it is very difficult
to formulate an explicit definition for probability. This is seen in the following pairs of
definitions from several recent technical dictionaries:
HarperCollins Dictionary of Statistics:
Probability: a number between 0 and 1 which represents how likely some event is to
occur.
Likelihood: the probability of a specified outcome.
The Cambridge Dictionary of Statistics:
Probability: a measurement that denotes the likelihood that an event occurred simply
by chance.
Likelihood: the degree of certainty of an event occurring. Likelihood can be stated as a
probability.
Dictionary of Statistics Methodology:
Probability: When an event can occur in a finite number of discrete outcomes, the
probability of an event is the ratio of the number of ways in which the event can occur
to the total number of possibilities, assuming that each of them is equally likely.
1

Likelihood: A measurement of how often and probable an event might occur. It is
often used as a synonym for probability and frequency especially in a qualitative
context. See Probability.
The term probability actually has several different meanings, both colloquial and technical,
but in statistics it is used in two very different ways (Folks, 1981): (1) as an intrinsic property of
some system that does not depend on our knowledge of the system; and (2) as a measure of
personal belief in a statement about the system. The first usage implies that the probability
would exist even if humans weren’t here to estimate it. (If a tree falls in the forest with a certain
probability and no one is around to hear it fall, does the probability still exist) The second
usage implies that an assessment of the probability of an event might vary from person to person.
(If I believe that the probability of rain tomorrow is 20%, while you believe it to be 30%, do we
disagree) We will pursue these different meanings below, but for now it is important to stress
that they have led to sometimes heated controversy among statisticians, either from disagreement
over the meaning of the term or from failure to distinguish between the two senses of the term.
Although we generally regard concepts such as sampling and probability to be relatively
recent, the conception of equally likely events, independence of events, and the use of
probability in making decisions date back many centuries. Rabinovitch (1970) described an
early use of inferred probabilistic reasoning in the 12 th century by Maimonides, a theologian,
physician and philosopher. Arbuthnot (1710), etc.
Events
The term event is used rather loosely in discussing probability. Informally, we think of an event
as an occurrence of some sort, such as an historical event (e.g., the 1969 Apollo moon landing)
or physical phenomenon (e.g., aurora borealis). In statistics, we expand that usage to include
specific outcomes of probabilistic experiments, such as the head or tail of a flipped coin, the face
value of a card drawn from a deck, the number of individuals of a particular species in a sample
from a community, or whether a particular treated individual does or does not succumb to a
disease. By a probabilistic experiment we mean an experiment whose outcome is not predictable
in advance.
There are two senses in which the term event is commonly used in statistics. If we think
about drawing a card from a deck, event might be used to describe the act of drawing the card,
which is also called an experiment. Note that this is a trivial meaning for the term experiment,
compared to how it is used in science, but is commonly used in the statistics literature. Event
might also be used to describe the outcome of drawing the card, such as “three of spades”. This
ambiguity can be useful if the meaning is evident from the context, but can sometimes lead to
confusion. I will use event to indicate a collection (or set) of one or more outcomes of an
experiment, and will often use the synonym outcome when the event consists of a single
outcome. The flipping of a coin is an event, with the possible outcomes of ‘head’ or ‘tail’. If the
coin is fair, then (by definition) the two outcomes are equally likely.
These uses of event and outcome are consistent with the statistical concept of sampling from
a population. Flipping a fair coin 10 times is conceptually equivalent to drawing 10 outcomes
2

(observations or entities) from an infinite population of outcomes that are either ‘heads’ or
‘tails’, in equal proportions.
In estimating probabilities it is useful to classify events as simple or compound. A simple
event cannot be subdivided further into component events, while a compound event comprises
two or more simple events. Whether a particular event is simple or compound may depend on
the purpose or context of the experiment. For a card drawn from a deck, for example, the event
‘diamond’ is simple by this definition, but ‘queen of diamonds’ can be considered to be
compound because the card is both a ‘queen’ (one event) and a ‘diamond’ (second event).
Alternately, ‘queen of diamonds’ might be considered to be simple because it describes one
particular outcome out of 52. Similarly, for an organism sampled from a population, the event
‘female’ is simple, but ‘adult female’ can be considered to be either compound or simple,
depending on context. We will consider this distinction in more detail below.
Sample spaces
The set of all possible outcomes of an experiment is known as the sample space of the
experiment, denoted by S. The simplest sample spaces are those in which the outcomes are
discrete. If the experiment consists of flipping a coin, then S = { H,
T}
, where H means that the
outcome of the toss is a head, and T that it is a tail. If the experiment consists of tossing a die,
then the sample space is S = { 1, 2, 3, 4, 5, 6}
. If the experiment consists of flipping two coins
(either sequentially or simultaneously), the sample space is S = {( H, H) ,( H, T)( , T, H)( , T,
T)
}.
The sample space for throwing two dice is:
( 1,1 ), ( 1, 2 ), ( 1, 3 ), ( 1, 4 ), ( 1, 5 ), ( 1, 6)
( 2,1 ), ( 2, 2 ), ( 2,3 ), ( 2, 4 ), ( 2,5 ), ( 2,6)
( 3,1 ), ( 3, 2 ), ( 3,3 ), ( 3, 4 ), ( 3,5 ), ( 3, 6)
( 4,1 ), ( 4, 2 ), ( 4,3 ), ( 4, 4 ), ( 4,5 ), ( 4,6)
( 5,1 ), ( 5, 2 ), ( 5,3 ), ( 5, 4 ), ( 5,5 ), ( 5,6)
( 6,1 ), ( 6, 2 ), ( 6,3 ), ( 6, 4 ), ( 6,5 ), ( 6,6)
⎧
⎫
⎪
⎪
⎪
⎪
⎪
⎪
S = ⎨ ⎬
⎪
⎪
⎪
⎪
⎪
⎪
⎪⎩
⎪⎭
Similarly, if the experiment consists of forming a diploid condition at a Mendelian locus
with two alleles by sampling and combining two gametes, the sample space of the alleles is
S = A,
A and the sample space of the diploid locus is
{ 1 2}
{( 1, 1) ,( 1, 2) ,( 2, 1) ,( 2,
2)
}
S = A A A A A A A A .
Sample spaces can also be defined for continuous scales. If the experiment consists of
measuring the lifetime of an organism, then there are an infinite number of possible outcomes
S = 0 < time of death

Mutually exclusive events
In probability theory, a set of events { E E E }
1, 2, K ,
n
are said to be mutually exclusive if the
occurrence of any one of them automatically implies the non-occurrence of the remaining n − 1
events. In other words, two mutually exclusive events cannot both occur; at most one of the
events can occur. The events comprising a sample space are mutually exclusive.
A related concept is that of being collectively exhaustive, which means that at least one of
the events must occur.
Probability concepts
Probabilities are always expressed as fractional or decimal values, but there are important
distinctions in how those values are obtained. For simple events, probabilities may be derived
classically based on the concept of fairness, may be observed empirically, or may be assigned
subjectively. These distinctions have important implications for statistical analyses.
Classical probabilities
Some events are defined so narrowly that the probabilities of their outcomes can be deduced
logically. The simplest such case involves a sample space having a finite set of discrete events,
with each event having the same probability. Such probabilities can be called classical
probabilities. Few scientific situations are of this sort, but the derivation of classical
probabilities is worth examining because the predictability of simple processes may provide
insight into the more complicated situations that are so common in science.
Throwing dice is one of man’s oldest known probabilistic endeavors. Because it is a cube, a
modern die has six labeled faces and so a single throw of the die results in a face value from 1 to
6. The sample space is thus S = { 1, 2, 3, 4, 5, 6}
. What is the probability of obtaining, say, a 5
If all face values are equally likely to occur, then a 5 is one possible outcome out of six, and its
probability is therefore 1/6, or approximately 0.17. Note that to assign this probability we need
to assume that all face values are equally likely to occur. This is the strong assumption of
fairness, and in the case of a die is unlikely to be strictly true because of inhomogeneities in the
material of the die. However, it may be true enough for practical purposes, sufficiently true that
we are willing to live with the assumption that the die is fair.
If we define an event to be the throwing of two dice, then there are more possible outcomes.
Considering only the sum of the face values, there are 11 possible outcomes: 2 through 12. A
naïve dice player might assume that these 11 events are all equally likely; however, a craps
player who believed this would probably loose a lot of money. The probability of obtaining a 5,
for example, is not simply 1/11 (approximately 0.09) because there are a number of different
ways that this outcome can be generated. The throwing of two dice is a compound event, and the
possible outcomes (sums) are shown in the following table.
4

Value of first die
Value of second die 1 2 3 4 5 6
1 2 3 4 5 6 7
2 3 4 5 6 7 8
3 4 5 6 7 8 9
4 5 6 7 8 9 10
5 6 7 8 9 10 11
6 7 8 9 10 11 12
The table shows that there are 36 possible outcomes, four of which result in a sum of 5 for the
two dice. If the dice are fair then all 36 outcomes are equally likely to occur, and the probability
of a 5 on one throw of two fair die is 4/36, or approximately 0.11. The most likely outcomes
from throwing two dice simultaneously are sums of either 6 or 7; both outcomes have
probabilities of 6/36 = 1/6, or approximately 0.17. Since there is only one way to throw a sum of
12 (=6+6), its probability is 1/36, or approximately 0.03.
These examples suggest a general definition for classical probability:
The classical probability of an outcome is the number of ways that outcome can occur,
relative to the total number of ways that all outcomes of an event can be generated.
Or, more concisely,
Pr ( outcome)
number of ways outcome can occur
=
number of ways all outcomes can occur
where Pr( outcome ) is read as “the probability of the outcome”. If we call a particular outcome
A, this can be written as
nA ( )
Pr( A)
= .
n
The most common use of classical probabilities in statistics is to serve as null hypotheses.
For example, when we flip a fair coin, the probability of observing a head is the same as the
probability of a tail; both are ½. If we have reason to believe that the coin is not fair, the
assumption of fairness is translated into a null hypothesis of equal probability, which can then be
tested using empirical probabilities (described in the following section). In an ecological study
of predator behavior in which the predator has four potential prey species, an initial and
simplistic null hypothesis might be that the prey species are equally likely to be selected by the
predator, each with a probability of ¼. This can be tested using empirical probabilities.
Empirical probabilities
Empirical probabilities are values derived by sampling, and are based on the concept of
long-run or asymptotic relative frequency. The relative frequency of a particular outcome is the
5

proportion of that outcome that we observe in a sample of observations. The long-run relative
frequency is the proportion that we observe as the sample becomes very large. In most
circumstances, as the sample size increases toward infinity, the relative frequency of the outcome
in the sample will converge on its true value in the population. After estimating the long-run
relative frequency, we then turn the situation around and interpret the frequency as the
probability of observing the outcome of interest by sampling a single observation from the
population.
Say that we want to estimate the allele frequencies of individuals for the MN blood group in
a particular human population. The three possible genotypes are MM, MN, and NN. We could
sample a number of individuals (the more, the better for this purpose, as long as the sample is
representative of the population) and determine the genotype of each. From these we could in
turn estimate the frequencies of the alleles M and N. For example, the following table
summarizes the classic data of Race and Sanger (1962) on MN genotypes from a British
population.
Phenotype M MN N
Genotype MM MN NN Total
Number (absolute frequency) 363 634 282 1279
Relative frequency 0.284 0.496 0.220 1.000
Based on these data, the relative frequencies of the M and N alleles can be estimated as the
homozygote frequencies plus half of the heterozygote frequency:
1 1
M MM 2 MN
2
1 1
N NN 2 MN
2
( )
( )
p = f + f = 0.284 + 0.496 = 0.532
p = f + f = 0.220 + 0.496 = 0.468
where p indicates a relative frequency and f an absolute frequency (count). The samples sizes
are rather large, so we can assume (for now) that these estimates are reasonably precise. We can
now interpret the relative frequencies as probabilities. The p M value of 0.532 indicates that our
best estimate of the probability of drawing a single M allele from the gene pool is 53.2%. In
other words, for every 1000 alleles drawn from the gene pool, 532 will, on average, be M alleles.
Derivation of probabilities from relative frequencies is logically circular, of course, but still
valuable.
The strong assumption of fairness also applies to empirical probabilities: all entities are
assumed to be equally likely to occur. Thus empirical probabilities are based on logical
probabilities assigned to individual observations. For example, if we use a mark-recapture
method to estimate the number of fish living within a lake, the estimate will most certainly be
based on the strong assumption that every fish in the lake stands an equal chance of being
caught. A sample in which every individual has an equal probability of selection is a random
sample. We will see that this “assumption of randomness” is basic to many statistical
procedures. If the sample is not truly random in this sense, then the long-run relative frequency
of the outcome in the sample will not converge on its true value in the population. The relative
6

frequency (and thus estimated probability) of the outcome will then be biased, but we will be
unaware of the nature and extent of the bias. Minimizing bias is one of the main objectives of
sampling theory.
The classical and empirical definitions of probability are basically identical, but because the
total number of potential observations can be large, the terminology is modified slightly:
The empirical probability of an outcome is observed frequency of that outcome, relative
to the total number of observations.
Pr ( outcome)
observed frequency
=
total frequency
nA ( )
Pr( A)
=
n
The variable n in the case represents the number of observations rather than the number of
distinct outcomes. For obvious reasons this is known as the frequency (or frequentist) definition
of probability. As a definition, it is equally valid no matter what the observed and total
frequencies. However, as an estimate of some “true” population frequency, it is more precise for
larger than for smaller total frequencies.
For example, say that we are trying to estimate the sex ratio of a (biological) population of
field mice living in a particular large field, which we define to be our statistical population. The
sex ratio (expressed in this case as the ratio of males to total number of individuals) can be
viewed as the probability of randomly drawing a male from the population, assuming that all
individuals are equally likely to be caught. (Is this a reasonable assumption) We might
estimate this probability by sampling 10 individuals from the population and discovering that 6
are males, giving Pr( male ) = 6 /10 = 0.6 . Or we might sample 1000 individuals and discover
that 513 are males, giving Pr( male ) = 513/1000 = 0.513 . It is reasonable to conclude that the
second estimate is somehow better than the first, because it is larger. We will show later (when
discussing confidence intervals) in what sense the second estimate is better than the first.
One potential use of empirical probabilities is to test null hypotheses that specify explicit
values for population parameters. For example, our null hypothesis might be that the population
of field mice consists of half females and half males – i.e., that the probabilities of randomly
drawing a female or drawing a male from the population are both ½. We might posit these
probabilities simply because there are two classes of individuals, which are assumed to be
equally likely (the classical frequencies). Or, if we are more informed, we might posit the
probabilities because Fisher’s (1930) sex-ratio model predicts for most populations a stable
equilibrium with equal numbers of females and males. In either case, the null-hypothesis values
of ½ and ½ can be tested by randomly sampling individuals from the population and determining
the relative frequencies, which are then used as estimates of probabilities to compare against the
null-hypothesis values. This raises the issue of just how different the observed frequencies have
to be from the postulated values to reject the null hypothesis. We will return to this example
when discussing statistical tests of hypotheses.
7

A consequence of the frequency definition of probability is that, for a frequentist, probability
is defined as a state of nature that is independent of a person’s cognitions or beliefs. As a result,
if two frequentists disagree about the value of a probability, at least one of them must be wrong.
Subjective probabilities
Unlike classical and empirical probabilities, which are derived either from knowledge of
potential outcomes or from data, a subjective (or degree-of-belief or subjectivist) probability
describes an individual’s personal judgment about how likely a particular event is to occur. It is
based not on a precise computation, but rather on a reasonable assessment of the situation by a
knowledgeable person. Subjective probabilities can be based on qualitative factors, previous
experience in similar situations, and intuition. The subjectivist interpretation of probability
differs from the frequentist interpretation in two important ways. First, the subjectivist defines
probability as a belief an observer has about nature, rather than as a state of nature independent
of an observer. Second, subjectivist’s beliefs about the value of a probability may differ for
different observers.
For example, meteorologists often assign probabilities to particular weather conditions, such
as “The probability of rain tomorrow is 20%”. Although meteorological models are becoming
increasingly sophisticated and may be used to estimate frequentist probabilities by running
thousands of trials, traditionally a meteorologist would consider winds, pressure areas and fronts,
adjacent weather systems, and perhaps a decent amount of intuition to estimate the probability of
rain. The estimate would apply implicitly to a specific area of interest (domain). The probability
of rain on the surface of the earth is likely to be close to 1, and in my back yard it may be close
to 0, while still being 20% for the region in which I live.
Regardless of how the probability is estimated, we typically use the information subjectively
to make decisions about how to dress and whether to carry rain gear. If I’m walking, I might
decide to carry a coat and umbrella if the probability of rain is greater than 50%. If I’m walking
and wearing a suit, I might decide to carry a coat and umbrella if the probability of rain is greater
than 60%. If I’m driving, however, I might skip the coat and grab an umbrella only if the
probability is greater than 70% or 80%. My decision would be based on past experience. A
person more concerned about getting wet might carry an umbrella whenever the probability of
rain is greater than 30%.
As an example of how the assessment of a subjective probability might differ for different
individuals, consider the case of three people who have different information about the outcome
of a specific toss of a fair coin. Mary, who saw the flip of the coin come up heads, would say the
probability that the coin landed heads is 1. Javier, who didn’t see the outcome of the coin flip,
might say that the probability that the coin landed heads is ½. And Xiao, who didn’t see the coin
flip but did see the look on Mary’s face immediately following the flip, might say that the
probability that the coin landed heads is ¾. Under the subjectivist definition of probability all
three individuals would be equally correct as long as their beliefs were consistent with the
axioms of probability.
Although their quality depends on the experience of the person producing them, subjective
probabilities can be perfectly acceptable estimates of “true” probabilities. However, whether
8

subjective probabilities should be used in further statistical analyses is a subject of some
controversy. Statisticians of the frequentist (also called objectivist or classical) school of thought
reject their use, while those of the bayesian school accept them. We will discuss this contrast in
more detail below.
Compound events
Multiplication rule for simultaneous probabilities
Addition rule
Statistical independence
Counting rules: permutations and combinations
Conditional probabilities
Odds ratios
Likelihood and maximum likelihood
Bayesian probabilities
Bayes’ theorem
Random variables
9