Perceptual Coherence : Hearing and Seeing

was high, then the resolution of the other signal did not affect the judgments.
But if the resolution of either signal was intermediate, the resolution
of the other signal affected the bias. Increasing the auditory resolution
led to a stronger auditory bias and vice versa. There was a reciprocal shift
based on the reliability of each signal.
It is unknown how observers estimate the variability of the sensory
information. Do observers have to learn in conscious fashion, or would the
spread of excitation across a population of cells derive the variability automatically?
As argued in chapter 6, adaptation to changes in sensory arrays
are multifaceted. Fairhall et al. (2001) found that adaptation to changes
in the input variability occurred within 1 s for the fly and that speed suggests
that updating occurs automatically. Nonetheless, observers still can
attend to one modality voluntarily and thereby override any biasing toward
the modality with the greater precision.
McGurk Effect
Auditory and Visual Segmentation 417
Speaking brings forth both visual and acoustic information concerning the
articulatory production of speech sounds. We have argued that people will
normally assume that the auditory and visual information represent the
same activity or event (e.g., the sight of a swinging hammer and the sound
of its impact). Moreover, we have argued when there is conflict, the most
sensitive modality will dominate the percept.
When we come to speech, the role of vision is less clear. The visual
information comes from three sources: (1) lip modulation; (2) maximum
lip velocity; and (3) maximum lip amplitude (Summerfield, 1991). Lip reading
is very difficult. Summerfield (1991) estimated that there are about 12
distinct visual configurations, so that about 63% of speech sounds are invisible
to sight. Nonetheless, visual articulation information can be very
helpful in difficult, noisy conditions. Improvements as much as 50% have
been reported (Sumby & Pollack, 1954). Furthermore, Munhall, Jones,
Callans, Kuratate, and Vatikiotis-Bateson (2004) found that rhythmic head
movements were correlated with the pitch (fundamental frequency) and
amplitude of the speaker’s voice and that visual information could improve
performance by 100% over that possible using auditory information only.
The large improvement with visual input argues that speech perception is
inherently multimodal and that the perceptual goal is identifying the articulatory
gestures that underlie both the auditory and visual outputs (Rosenblum,
2004).
A striking illusion that has been used to study temporal and spatial ventriloquism
has been termed the McGurk effect (MacDonald & McGurk,
1978; McGurk & MacDonald, 1976). Participants are shown a person saying
simple consonant-vowel syllables coupled with an acoustic recording