QWERTY Caster - Universitat Pompeu Fabra

Interpretació amb 

Mitjans Electrònics 

Especialitat de Sonologia 

Sessió 1 

L’Ordinador com a sintetitzador 

(o com a “instrument tradicional”) 

Prof. Sergi Jordà 

sergi.jorda@iua.upf.edu 

Sergi Jordà, Music Technology Group 

IUA, UPF – Barcelona, 2008

Part I: Digital vs. Acoustic Instruments 



Traditional Music Instruments / Construction 

Most acoustic instruments consist of 

an excitation source that can oscillate in different 

ways under the control of the performer 

a resonating system that couples the vibrations of 

the oscillator to the surrounding air 

Parts of the performer’s body generate the energy for 

the excitation of the instrument, interacting through 

the instrument’s control interface 



Traditional Music Instruments / Restrictions 

1. Limited by physical constraints 

2. Restricted control of their output 

3. Human excitation energy dependent (in acoustic instr.) 

4. Human control dependent 



Traditional Music Instruments / Restrictions 

 

 

 

 

The possibilities & limitations of acoustic instruments concerning 

both mechanical inputs & control, and sonic output, are hardwired 

by physical constraints. Acoustic instruments tend to impose by 

construction their own playability rules, which allow listeners to 

infer the type and form of the gesture from the generated sound 

No instrument allows for a complete control over its whole sonic 

richness. Microevolutions of their parameters are often 

uncontrollable and subject to a combination of (a) complex 

correlations (b) structural and morphological constraints and (c) 

additional random -or ungraspable- factors 

With few exceptions (as it is the case of the organs) the excitation 

energy needed for the instrument to sound has to be provided by 

the performer (blown, struck, plucked or rubbed) 

With few exceptions, not only energy but also all type of temporal 

control variations or modulations of parameters (e.g. vibrato or 

tremolo) have to be explicitly and permanently addressed 

physically by the performer 



Gesture Types in Music Making 

Selection 

Finger on a fret, selecting a soundbank … 

They are previous to sound production 

Decision 

Fretting a string, hitting a key … 

They bring the energy 

They make a sound with instant parameters 

Modulation 

Moving a pedal, a mod. wheel 

Changes sound parameters over time 



Digital Music Instruments 

Controller/Generator clearly separated (exchangeable?) 

Inputs 

no clear distinction between excitation and control 

everything is possible (??) 

blow, strike, pluck, rub, bow … + 

click, double-click, type, point, slide, twirl, drag 

and drop … + 

?????? (we will study sensors in another Module) 

Outputs: everything is possible (??) 



Outputs: everything is possible (??) 

 

 

 

 

Any parameter variation can now be implemented as desired; 

continuously, discreetly, or as a combination of both 

Timbre does not have to be left out anymore. Any imaginable 

timbre can be rendered theoretically; it can change abruptly or 

evolve continuously, and even smooth transitions between any set 

of distinct timbres can be achieved (e.g. Grey 1975; Wessel 1979; 

Wessel et al. 1998) 

Small periodic changes can still be applied to anything, with 

unlimited and precise control over all of the oscillation parameters 

(frequency, amplitude and shape of the oscillation), and these 

modulations do not have to be permanently addressed by the 

performer anymore since separate threads in the machine are 

taking charge of them 

The performer, who no longer needs to control all the smaller 

details, can control instead the processes that control these details 



AUTOMATED CONTROL 

PROCESSES 

Hammond+Leslie 

Analog modular 

synthesizer 

NON-HUMAN 

EXCITATION 

ENERGY 



From the “do-everything-yourself” 

to the “one-button-does-it-all” 

We will start by studying the left 

paradigm, i.e. when computers 

behave like traditional instruments, 

leaving everything to the musician 

responsibility… 




IUA, UPF – Barcelona, 2008 

Part II: Music Controllers

Traditional Music Instruments restrictions / 

Revisited 

Limited by physical constraints 

Restricted control of their output 

Human excitation energy dependent (in acoustic instr.) 

Human control dependent 

Even when we want to remain closer to the “acoustic 

instrument model”, point 1 does not make much sense 

(unless we want to make digital clones of acoustic 

instruments), and point 3 is not easy to achieve 



Controllers’ Summary (1/2) THE EASY SIDE 

(from Jordà 2007) 

Music controllers can preserve traditional playing 

modes, permitting us to blow, strike, pluck, rub or bow 

our ‘computers’; new traditionalists in turn, may 

prefer to continue clicking, double-clicking, typing, 

pointing, sliding, twirling or dragging and dropping 

them. The decision is up to everyone. With the 

appropriate sensors, new digital instruments can also 

be caressed, squeezed, kissed, licked, danced, hummed 

or sung. They can even disappear or dematerialise 

while responding to our movements, our muscle tension 

or our facial expressions. 



Controllers’ Summary (2/2) DIFFICULT SIDE!!! 

(from Jordà 2007) 

With the flexibility offered by MIDI, any controller can certainly be combined with 

any sound- and music-producing device. Still, each choice is critical. As pointed 

out by Joel Ryan, improviser, leading researcher in the NIME field and technical 

director of the Dutch laboratory STEIM, ‘a horizontal slider, a rotary knob, a 

sensor that measures the pressure under one finger, an accelerometer which can 

measure tilt and respond to rapid movements, a sonar or an infrared system that 

can detect the distance between two points, each have their idiosyncratic 

properties’ (Ryan 1991). 

Any input device can become a good or a bad choice depending on the context, 

the parameter to control, or the performer who will be using it. Just as the 

automotive engineer chooses a steering wheel over left/right incrementing 

buttons, ‘we should not hand a musician a butterfly net when a pitchfork is 

required’ (Puckette and Settel 1993). The challenge remains how to integrate and 

transform this apparatus into coherently designed, meaningful musical 

experiences with emotional depth. 

It is in fact extremely hard to design highly sophisticated control interfaces 

without a profound prior knowledge of how the sound or music generators will 

proceed; a parallel design process will surely be more enriching than buying the 

ultimate controller for plugging into any custom software. 



Interaction Coherence ? 

The disappearance of physical constraints, should be 

replaced with deep musical, metaphorical, ergonomic … 

coherence 

Seeking coherence is somewhat in the antipodes of the Theremin 

limited success, which is best known for its “magic” way of playing 



In Common? 



Timbre navigation 

The voice, the didgeridoo, brass with cups…. 

Not many acoustic instruments allow for a continuous 

modification/transformation of their resonant bodies… 

Not many acoustic instruments allow thus for continuous 

and (reasonable uncorrelated) timbre navigation 

Continuous timbre navigation (more than an infinite 

timbre palette) is arguably one of the assets of 

“traditional” computer based music instruments 



Music Controllers / Prehistory 

Except perhaps for keyboards, the concept of music 

controller is alien to acoustic music instruments 

The theremin, one of the first and still more famous, 

electronic instruments (famous because of the way it is 

played -- i.e. without touching it), is a full instrument, 

not a controller 

Clara Rockmore performs Saint-Saëns’ "The Swan" 

Canadian musician and researcher Hugh LeCaine, was 

between 1940s-70s a pioneer in the field of electric and 

electronic instruments 

electronic sackbut (1945-73) 



Hugh Le Caine’s electronic sackbut (1945-1973) 



Hugh Le Caine’s electronic sackbut 

Musical examples 

1948 “clarinet” (Rapshody in Blue) 

1948 “mutted trombone” (playing with formants) 

1948 “brass to clarinet” (Blues) 

1948 “strings” 

Bill Farrow plays “brass-like” 

Mal Clark plays “brass-like” 

From “Hugh Le Caine - Compositions Demostrations 

1946-1974 » 



Music controllers & MIDI 

 

 

 

 

Despite MIDI being extremely key-oriented, MIDI also made simpler 

the creation of new controllers (early 80s) 

Wanderley (2001) classifies 

instrument-like controllers (imitate) / Carl Wind Controller video 

extended controllers/hyperinstruments (expand) 

alternative controllers 

Commercial controllers tend to imitate trad. Instruments (at least 

until the mid-2000s) 

Paradiso (1997) further classifies alternative controllers into: 

Batons 

Non-contact (incl. Theremin-like, cvision, ultrasound…) 

Wearable 



Some Music Controllers… 

 

 

 

 

 

Michel Waisvisz’ The Hands (left up) 

BioMuse (Brainwave detector!) (left mid) 

Donald Buchla’s The thunder (left down) 

Sergi Jordà’s LowTech-QWERTY Caster 

(cent down) 

Max Mathews’ Radio Baton (right down) 

Videodetection with Very Nervous System, 

David Rockeby (right up) 



Sensing data 

Sensors and microcontrollers or ready-made analog2MIDI 

devices are widely available 

All types of external parameters (light, temperature …) 

All types of “external-conscious” human activity 

(pressure, breath, joint movements, face-expression …) 

as well as “internal-unconscious” [EMG (muscle 

activity), EOG (eye movement), EKG (heart), EEG 

(brainwave)], can be measured 



Other devices as Controllers 

Joysticks and gamepads 

DataGloves 

Graphic tablets 

Microphones (audio2MIDI, audio analysis…) 

…. 

More info: 

New Interfaces for Musical Expression (NIME) (Wikipedia 

, NIME Website, YouTube) 



Part III: Design Issues 

Jeff Pressing’s list (1990) 

Dimensionality 

Human Bandwidth 

Dimension Coupling 

Spatial vs. Temporal Multiplexing 



Dessign Issues (Pressing 1990) 

for a realtime interaction music interface x 

1. Physical variables carrying the information (e.g. position, 

pressure, velocity, acceleration, etc.) 

2. Dimensionality of control or degrees of freedom 

3. Control modality (i.e. discrete or continuous, and its digital 

quantization) 

4. Design appropriateness: efficiency, ergonomics, motor and 

cognitive loads, degree of sensory reinforcement and redundancy, 

appropriateness of gesture to expression 

5. Historical foundations: using an existing technique, modifying or 

creating a new one 

6. Psychological nature of control: perceived naturalness of the link 

between action and response, exploratory or goal-oriented, etc. 

7. Literalness of control (one-to-one, one-to-many, many-to-one, 

unpredictability, response delay, time dependence) 



Dimensionality / Degrees of Freedom (DOF) 

A dot in space is determined by 3 coordinates 

A solid object by 6 

HMD 



Degrees of Freedom / [multi]dimensionality 

A mouse can be considered one or two-dimensional 

(depending of the mapping) 

Most joysticks have 3 or more simultaneous control 

dimensions 

A data glove typically delivers 11 continuous control 

parameters in each hand! 



The 11 dimensions of a Data Glove 

 

 

 

5 dimensions (one for each finger) 

3 for the position of the hand in 

space (x, y, z) 

3 for the orientation of the hand in 

space (pitch, yaw, roll) 

(alternatively it can 

also be seen as the 

position of 2 points 

of a rigid object in 

space 

6 coordinates) 

The cheap P5 data glove 



How to profit from 11 dimensions? 

Yet, traditional instruments, tend to have a lower 

dimensionality 

e.g. reed instrument: pitch, breath pressure, reed 

pressure (not considering growl, unstandard 

fingering….) 



The hyperhuman controller ! 

From the strictly physical point of view the dimensionality of 

control possible for human beings could easily exceed 40 degrees 

of freedom (Pressing 1990) 

If we only consider articulations, one could easily count 3 

degrees of freedom for each hand and foot, 3 additional ones 

for each hand finger, 8 more for arms (elbows+shoulders) and 

legs, 2 for the head, plus at least 1 more for breath pressure, 

which gives a total of 53! [(3x4) + (3x10) + 8 + 2 + 1 = 53]. 

And that without taking into account other possible 

physiological controllable parameters such as muscle tension 

(e.g. Tanaka 2000), face expression (e.g. Lyons et al. 2003), 

eye-tracking (e.g. Hornof & Sato 2004), or the absolute position 

of the whole body in space (3 more parameters) 

But this is much more than any human can handle! 



Human control 

bandwidth 

How much a human is 

able to control 

meaningfully ? 



Human control bandwidth (2) 

 

 

How much control is humanly possible? The limitations must be of 

cognitive nature (Fitts 1954; Miller 1956; Fitts & Posner 1967; 

Pressing 1988; Cook 2001), and they are hard to evaluate 

Most of this research has been done in the field of music 

instruments and performance, arguably the more 

sophisticated/rich control devices invented 

To bring into play the full bandwidth of communication there 

seems to be no substitute, for mammals at least, than the 

playing of music live (Bischoff et al. 1978) 

 

First, although trad. music instruments are not digital, we have to 

consider that each dimension has a resolution (in bits) e.g. MIDI 

offers a 7-bit resolution/parameter, but 8, 12, 16 bits … are also 

common 



Human bandwidth, according to Bob Moog 

 

 

 

 

A flute player, is able to control amplitude with a 6-bit resolution and 

with a temporal resolution of about 100 Hz 

A drummer can play with a maximum frequency of 10 Hz controlling 

three parameters and an approximate resolution of 4 bit/parameter. 

That gives us 600 bits/sec (without considering pitch) and 120 bits/sec 

respectively. 

Moog estimates that the 

maximum meaningful 

information a skilled 

musician is able to 

generate is about 1000 

bits/sec (Moog 2004). 



Continuous / discrete 

 

 

 

 

Not all dimensions have to be continuous 

For example, in many trad. instrument, pitch choice is discrete 

(keyboard, guitar, wind…) (even if in the guitar and in wind 

instruments, pitch can depend on other factors, and therefore the 

output pitch can be more “continuous”, its initial choice is still 

discrete) 

In the piano, for ex., each event has a dimensionality of 2 (one 

discrete – key, the other continuous – velocity), but several 

simultaneous events are possible 

As in the analog world, in the digital world, discrete selections can 

be addressed in diff. ways (they can have 2 or more states, they 

can toggle or not, they can exclude previous choices – radio 

buttons …) 



Dimensions: coupled (L) / decoupled (R) 

What’s better ? 

What for !?!? 



Dimension coupling 

“To draw freehand, one prefers a mouse; to draw straight vertical 

and horizontal lines, the humble Etch-a-Sketch is better” (Buxton 

1986). 

The simultaneous manipulation of the resonance frequency and the 

Q factor of a filter, is perfectly controlled with a two-dimensional 

joystick. 

NB. We are here considering input coupling, not output correlation 

which we will discuss later. However, it seems that output 

correlation in trad. Instruments is one of the keys for 

“expressivity” (e.g. loudness & pitch in wind instruments, voice…) 



Multiplexing 

spatial/temporal 

 

 

 

In temporal multiplexing 

a device controls diff. 

functions at diff. 

moments 

The mouse is a timemultiplexed 

device and 

WIMP interaction tends 

to be very TM (although 

displays are more 

spatially multiplexed) 

In spatial multiplexing, 

each function has a 

dedicated device 



y and Vertegaal SenSorg cockpit ! 



Spatial Multiplexing / Dedicated Devices 

Analog devices (incl. acoustic instrument) tend to be space 

multiplexed 



Hold / return / bypass /skip 

 

 

 

Return or hold refers to whether a parameter returns (snaps back) 

to a nominal value (as the pitchbend wheel in most MIDI keyboards) 

or keeps it position upon release (as the modulation wheel in most 

MIDI keyboards). 

Bypass can be seen as the combination of return and hold, or as 

the quick alternation between two values, one of them being 

typically a zero effect position. 

Skip reflects the ability to have direct access to arbitrary values. 

Most controllers do not allow this and are forced to traverse 

intermediate values instead. An example of a controller able to 

skip is the ribbon present in several analog synthesizers from the 

1960s and 1970s (Pressing 1990). 



Part IV: Control Examples 

Traditional Instruments 

The Theremin 

The Joystick 

Bimanual input 



Acoustic Musical Instruments Examples 

 

 

 

 

A string can be accessed at any point of its length (and it can be 

accessed without perceivable sound, i.e. without the needed 

energy) 

A trombone slide has to cover all the way (although it doesn’t 

sound either if it’s not blown) 

In the piano and percussion instruments, energy & control is given 

with the same action 

Most acoustic instrument don’t have hold capabilities; but they 

often go back (bypass) to a default state (open strings, open 

tube…) 



The Theremin: a strange case study 

According to the previous criteria, the Theremin can be seen as one of 

the worst musical interaction designs 

It has only 2-dimensions (pitch/loudness), and totally uncoupled 

It lacks all hold/bypass/skip facilities, which make very difficult 

sharp attacks and discontinuities both in pitch & in amplitude 

 

It completely lacks any type of haptic feedback (shown that is it 

much easier to play it with rubber bands on each hand…) 

Possible reasons of success? 

Magic? 

Simple, identifiable yet full of references (voice, strings…) tone? 

Difficulty of playing? 



The Joystick 


IUA, UPF – Barcelona, 2008 

 

 

 

Good example for a microcontrol 

(e.g. timber nuances) 

2 or 3D very coupled / Return 

Easy use of time-multiplexing, 

with the various buttons (e.g. for 

alternating between diff. timbre 

parameters)

Some Joystick (and other bimanual) examples 

 

Daniel Arfib, J.M. Couturier, L. Kessous videos. 



The Voicer (Kessus, Arfib) 

Amplitude with 

pen pressure 



Loïc Kessous’ Gloves 



Some Joystick (and other bimanual), 

but not so low-level examples 

Michel Waisvisz, gloves 

Clay Chaplin, gloves (minute 1:00-2:00) 

The Suicide voice, gloves + voice control 

Controller’s band, gloves + joystick 



Part IV: Conclusions & Next Steps 



AUTOMATED CONTROL 

PROCESSES 

Hammond+Leslie 

Analog modular 

synthesizer 

NON-HUMAN 

EXCITATION 

ENERGY 



The 

Multithreaded vs. 

the Traditional 

Model 



Multithreaded instruments & shared control 

(Jordà 2005) 

 

 

 

 

In traditional instruments the performer is responsible for 

controlling every smallest detail, leaving nothing to the instrument 

responsibility 

One of the best assets of new digital instruments is the possibility 

to run several multiple and parallel musical processes in a shared 

control between the instrument and the performer 

These instruments’ ‘intelligence’ may be partially responsible for 

one or more musical processes, the control of which may be 

entirely left to the instrument’s responsibility or may be shared in 

different ways with the performer 

When performers delegate some control to the instrument, their 

role is also approaching that of composers, who do have to delegate 

control of their work to instrumentalists 



Some multithread & shared control corollaries 

These systems (instruments) tend to 

surpass the one gesture to one acoustic event’ paradigm 

go beyond the sound and note control level 

run multiple and parallel musical processes in a shared 

control between the instrument and the performer 

the possibility to have multiple performers seems as a 

logical and promising extension 



Dimensionality and Multiplexing Problems (1/2) 

(Jordà 2005) 

1. Analog devices (incl. acoustic instrument) tend to be space 

multiplexed 

2. Traditional instrument tend to have a low dimensionality 

3. Digital controllers can have a higher dimensionality 

4. Interactive music systems tend to have a still higher (output) 

dimensionality (i.e. many parameters to control, not only related to 

sound, but possibly to “form”…) 

5. They thus tend to need some temporal multiplexing (e.g. secondary 

buttons that change the behavior of primary controllers) 



Dimensionality and Multiplexing Problems (2/2) 

(Jordà 2005) 

 

 

 

 

 

Many new controllers have additional trouble dealing with 

hold/bypass/skip issues and thus need extreme time multiplexing 

Exosqueletons, bodysuites, wearables, CVision body tracking 

systems… represent the summit of this problem… 

They force the performer to work constantly on all its dimensions 

(they don’t have off-states) 

They don’t hold, skip (direct access), bypass or even mute, 

without the help of additional “secondary buttons” (often in the 

form of foot pedals) 

These devices may look cool & futuristic; they can work as 

synthesizers controllers, but I claim that they are the worst 

friends of interactive music systems 



“Air” Music 

 

 

Short BBC Theremin Documentary (on “air playing”) 

Dance Tensor 



Some Music Controllers… 

 

 

 

 

 

Michel Waisvisz’ The Hands (left up) 

BioMuse (Brainwave detector!) (left mid) 

Donald Buchla’s The thunder (left down) 

Sergi Jordà’s LowTech-QWERTY Caster 

(cent down) 

Max Mathews’ Radio Baton (right down) 

Videodetection with Very Nervous System, 

David Rockeby (right up) 



“Traditional” Controllers !! 

Most of the music controllers being developed prolong the traditional 

instrument paradigm. Trying perhaps to exorcise forty years of tape 

music, researchers in the field of new musical interfaces tend to 

conceive new musical instruments highly inspired by traditional ones, 

most often designed to be ‘worn’ and played all the time, and offering 

continuous, synchronous and precise control over a few dimensions. 

An intimate, sensitive and not necessarily highly dimensional interface 

of this kind (i.e. more like a violin bow, a mouthpiece or a joystick, 

than like a piano) will be ideally suited for direct microcontrol (i.e. 

sound, timbre, articulation). However, for macrostructural, indirect or 

higher level control, a non-wearable interface distributed in space and 

allowing intermittent access (i.e. more like a piano or a drum) should 

be undeniably preferred (Jordà 2005). 



Personal Guidelines (Jordà 2005) 

Space multiplexing is preferred for real-time interaction 

Always on controls (e.g. position in space) should be 

avoided, or used with very simple and easily accessible 

secondary buttons (scarce use of time-multiplexing) 

Return & low-dim control is preferred for micro-control 

(e.g. timber nuances) (conceptually closer to acoustic 

instruments) 

Hold & high-dim control is preferred for macrostructural 

control (typical of Interactive Music Systems) 



Recommended Readings 

Arfib, D., Couturier, J.M., Kessous, L., and Verfaille, V. (2002). 

Strategies of mapping between gesture data and synthesis model 

parameters using perceptual spaces. Organised Sound 7, 127-144. 

Jordà, S. (2005). Digital Lutherie: Crafting musical computers for new 

musics’ performance and improvisation. Ph.D. dissertation, Dept. of 

Computer Engineering, Universitat Pompeu Fabra, Barcelona (chaps. 2, 

4, 5) 

 

 

 

Jordà, S. (2007). Interactivity and Live Computer Music, in "The 

Cambridge Companion to Electronic Music", Edited by Nick Collins and 

Julio d’Escrivan. Cambridge University Press, UK. 

Pressing, J. (1990). Cybernetic Issues in Interactive Performance 

Systems. Computer Music Journal, 14(1), 12-25. 

Ryan, J. (1991). Some Remarks on Musical Instrument Design at STEIM. 

Contemporary Music Review, 6(1), 3-17.

QWERTY Caster - Universitat Pompeu Fabra

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?