Sissy - Elise van den Hoven

Figure 1. Screenshot from Partial Reflections 3, showing the
simulated physical sculpture responding to sounds played on
an acoustic instrument.
Physical modelling techniques have a long history in
sound synthesis (Smith, 2004). Traditionally the
approach has been to create high-fidelity models of the
sound producing mechanisms of real-world musical
instruments in order to produce more realistic synthesised
sounds. One could say that rather than trying to build a
violin sound, the idea is to create a simulated violin. If
the simulation is accurate the sound it produces will be
realistic.
Another, less commonly applied approach, is to use
physical models as a kind of interface layer between the
gestures of the performer and the sounds and/or visuals
produced by the computer. This is the approach used in
the Partial Reflections and Touching Dialogue works.
The primary reason for using physical models as a kind of
intermediate mapping layer between the sounds produced
acoustically by the performer and the computer generated
sounds and visuals was because we were hoping to create
an “instantly knowable, indefinitely masterable interface”
(Levin 2000, p. 56). The musicians who participated in
the design process found that the physical model
interaction paradigm was intuitively understandable and
controllable but provided sufficiently rich and complex
audiovisual responses to allow the discovery and
exploration of new musical-visual material during
performance.
Physical modelling techniques have potential to create
and control sounds that provide a higher degree of
engagement for both performer and audience. Leman
argues that there is evidence that “listening focuses on the
moving source of a sound rather than on the sound itself”
(Leman, 2007 p.236). In other words, when we hear
music, we perceive it in terms of physical actions that we
associate with such sounds. These need not necessarily
be the physical actions that actually cause the sounds, but
actions that we somehow associate with them based on
past experiences.
He proposes a model of musical communication based on
the encoding and decoding of biomechanical energy in
sound. In this model, the performer realises musical
goals by physically manipulating an instrument, which
translates the performer's physical energy into sound.
14
The listener, at least partially through a process of
associating sounds with physical actions, makes sense of
the sound. This is not to say that the listener's
understanding of the music will be identical to that of the
performer's, but rather that the listener will make sense of
the sound in their own action-related terms. The
implication is that instruments, which facilitate a more
direct connection between the physical actions of
performers and generated sounds, are more likely to
facilitate musical communication at this gestural level.
Modes of Interaction
During 2007 and 2008 a series of user studies examining
musicians’ experiences with the Partial Reflections sound
sculptures were conducted (Johnston et al, 2008,
Johnston, 2009). The key issue that arose was that of
modes of interaction.
It was observed that the musicians’ interactions with the
virtual instruments could be classified into three modes:
instrumental, ornamental and conversational.
When approaching a virtual instrument ‘instrumentally’,
musicians sought detailed control over all aspects of its
operation. They wanted the response of the virtual
instrument to be consistent and reliable so that they could
guarantee that they could produce particular musical
effects on demand. When interacting in this mode,
musicians seemed to see the virtual instruments as
extensions of their acoustic instruments. For these
extensions to be effective, the link between acoustic and
virtual instruments had to be clear and consistent.
When musicians used a virtual instrument as an
‘ornament’, they surrendered detailed control of the
generated sound and visuals to the computer, allowing it
to create audio-visual layers or effects that were added to
their sound. A characteristic of ornamental mode is that
the musicians did not actively seek to alter the behaviour
or sound of the virtual instrument. Rather, they expected
that it would do something that complemented or
augmented their sound without requiring direction from
them.
While it was not always the case, it was observed that the
ornamental mode of interaction was sometimes a fallback
position when instrumental and conversational
modes were unsuccessful. While some musicians were
happy to sit back and allow the virtual instrument to
provide a kind of background ‘audiovisual wallpaper’ that
they could play counterpoint to, others found this
frustrating, ending up in an ornamental mode of
interaction only because their attempts at controlling or
conversing with the virtual instrument failed.
In the conversational mode of interaction, musicians
engaged in a kind of musical conversation with the virtual
instrument as if it were another musician. This mode is in
a sense a state where the musician rapidly shifts between
instrumental and ornamental modes, seizing the initiative
for a time to steer the conversation in a particular
direction, then relinquishing control and allowing the
virtual instrument to talk back and alter the musical
trajectory in its own way. Thus each of the three modes of