Perceptual Coherence : Hearing and Seeing

98 Perceptual Coherence
Information, Redundancy, and Prior Probabilities
Information Theory and Redundancy
We can use the framework of information theory (Shannon & Weaver, 1949)
to quantify how much the neural response tells us about the stimulus. Our
interest lies in the mutual information between stimuli and neural responses.
Given the neural response, how much is the uncertainty reduced about which
stimulus actually occurred (which is the same reduction in uncertainty about
the resulting neural response given the stimulus, hence mutual)? I frame all
of these questions in terms of conditional probabilities: Given the neural response,
what are the probabilities of the possible stimuli as compared to the
probabilities of the same stimuli before observing the response?
In defining information, Shannon (1948) was guided by several commonsense
guidelines. 1 The first was that the uncertainty due to several independent
variables would be equal to the sum of the uncertainty due to each
variable individually. Independence means that knowing the values of one
variable does not allow you to predict the value of any other variable. The
probability of any joint outcome of two or more independent variables becomes
equal to the multiplication of the probabilities of each of the variables:
Pr(w, x, y, z, ...)= Pr(w)Pr(x)Pr(y)Pr(z) . . . (3.1)
To make the information from each variable add, we need to add the probabilities,
which can be accomplished by converting the probabilities into the
logarithms of the probabilities.
The second consideration was that the information of a single stimulus
should be proportional to its “surprise,” that is, its probability of occurrence.
Thus, events that occur with probability close to 1 should have no information
content, while events that occur with a low probability should
have high information content.
Shannon (1948) demonstrated that defining the information in terms of
the negative logarithm is the only function that satisfies both considerations. 2
Information =−log 2 Pr(x). (3.2)
If there are several possible outcomes, then the information from each
outcome should be equal to its surprise value multiplied by the probability
of that event. This leads to the averaged information for a stimulus distribution
or a response distribution:
1. I use the terms information and uncertainty interchangeably. The uncertainty of an
event equals its information value.
2. Traditionally, the logarithm is set to base 2 to make the information measure equivalent
to the number of bits. In fact, the choice of base is completely arbitrary because simply multiplying
by a constant converts between any two bases.