Perceptual Coherence : Hearing and Seeing

422 Perceptual Coherence
within the same modality. This is the correspondence problem: the ability
to track individual fireflies, detect first- and second-degree motion, perceive
the same color in different illuminations, recognize the same face at
different ages, and predict what an instrument or singer will sound like at a
different pitch. The sensory world is composed of objects with correlated
redundant features, and those objects change in predictable ways. The coherence
of the perceptual world depends on our ability to detect that redundancy
and predictability. If the sensory world were not predictable, then the
statistics in the sensory world would be meaningless, and perceiving as we
know it would not even exist.
These first three principles deal with object perception within a single
sense, and the argument here is that there is little difference between auditory
and visual object perception. The fourth principle proposed by Griffiths
and Warren (2004) is that object analysis deals with cross-modal
information, information that fuses auditory and visual information about
the same object. The outcomes described in chapter 9 make it clear that
perceivers normally assume that auditory and visual stimulation comes
from a single source (i.e., the unity assumption) and that people often make
illusionary judgments based on that assumption. The McGurk effect is one
such illusion. Auditory and visual information is not processed independently,
and in some cases I believe that joint processing is obligatory (e.g.,
temporal ventriloquism).
I think that the very existence of illusions makes a stronger point. Nearly
all of our perceptual experiences involve more than one sense, and the “reasonableness”
of the illusions requires the substitutability of the auditory for
the visual information and vice versa. In order to be substitutable, both
types of information must exist within a common frame of reference, and
the perceiver must infer that all of the sensory stimulation arises from the
same object. The concept of substitutability has been investigated most extensively
in speech perception, where any articulatory gesture simultaneously
produces visual facial movements and auditory speech sounds. As
discussed previously, visual movement information can dramatically improve
speech intelligibility from one speaker, particularly when the speech
signal is buried in noise (Sumby & Pollack, 1954). Lachs and Pisoni (2004)
demonstrated that observers could match visual facial movements to
speech sounds of speakers. The observers could not make the match if the
faces were static or if the speech sounds were reversed in time. Thus, it is
the correlation across time between the auditory and visual information
that allows observers to determine if the speech sounds came from the
same speaker whose face is moving.
Nelken (2004) made a different argument for the primacy of object perception
in the auditory system. Nelken summarized neural results which
demonstrate that auditory neurons in A1 match their responses to the time