Perceptual Coherence : Hearing and Seeing

Jackson (1953) found that the sound of a steam kettle coupled with the
sight of a spatially offset but silent puff of steam produced a greater bias
than the sound of a bell coupled with the sight of a spatially offset but inert
bell. However, the results using familiar stimuli may result from observers
responding on the basis of what they know about the objects, rather than
what they perceive.
Perceived Onset Synchrony
Auditory and Visual Segmentation 407
The critical property for the perception of one or more auditory streams
was onset synchrony. Onset asynchronies as short as 30 ms were sufficient
to lead to the perception of two complex sounds. On that basis, we
might expect that perceived onset synchrony of the auditory and visual
input would be a critical property for perceiving one object. Lewkowicz
(2000) summarized developmental results that propose that all types of
multimodal temporal perception (e.g., temporal rate and rhythm) emerge in
hierarchical fashion from the fundamental property of temporal synchrony.
The perception of synchrony between auditory and visual inputs poses a
more complicated problem than that between two auditory or two visual
events because the speed of light is so much faster than the speed of sound.
Even at 10 m, a light stimulus will reach the observer 30 ms before a sound
stimulus. Any difference in arrival time is compensated to some degree
by the faster mechanical processing of the sound at the cochlea. In cats, it
takes about 13 ms for an auditory stimulus to activate neurons in the superior
colliculus that receive inputs from different modalities, as opposed to
about 65–100 ms for a visual stimulus to activate the same neurons due
to the slower chemical processing of light energy at the retina (Stein &
Meredith, 1993). In humans, the P1 evoked response potential occurs about
75 ms after onset for auditory stimuli and about 100 ms after onset for
visual stimuli (Andreassi & Greco, 1975). Balancing the speed and processing
differences, it is estimated that the onset of the neural signals will
be synchronous only when the object is roughly 10 m away (Poppel, 1988).
Although there is some controversy about whether the perception of
synchrony is based on the excitation of individual multimodal cells or the
simultaneous excitation of unimodal cells, the underlying conception is
that of a temporal integration window. If the activity patterns resulting from
the two inputs overlap within such a window, synchrony will be perceived.
It is important to keep two things in mind. First, it is the activity pattern due
to the inputs that is critical, not the physical onsets or the latencies to activate
the neurons. The window can be quite long. Second, do not think that
there are discrete nonoverlapping windows. There must be “rolling” overlapping
windows (e.g., 0–200, 5–205, 10–210 ms, etc.). If the output from
the integration windows depends on the amount of simultaneous activation,