NUI Galway – UL Alliance First Annual ENGINEERING AND - ARAN ...

A Computer-Vision Based Interface for the Control of Parameterised Music
Effects
Abstract
It is increasingly common for musicians to
route the signal from their instrument through a MIDI
interface to be processed digitally by commerciallyavailable
software rather than by traditional analog
effects units and amplifiers. This allows for greater
sonic possibilities and control.
This project investigates the potential of
utilizing computer vision techniques in the task of
controlling this processing, i.e. parameterised music
effects. The proposed system would create a flexible
and intuitive interface, enabling the control of any
processing being applied to an instruments signal in
real-time by using head movements as an interface to
generate MIDI control signals.
Introduction
At present, the parameters of music effects are
controlled either through the knobs, sliders and faders
found on MIDI interfaces or through floor-based units.
Current MIDI interfaces have the limitation
that in order to change a parameter, one must do so by
hand. This has the implication that one hand is
essentially removed from the instrument being played.
Floor-based units also have knobs enabling the
control of parameters. More tailored units enable the
control of a single parameter through a foot-pedal that
moves along one-axis (i.e. rocked back and forth from
heel to toe).
A computer-vision approach based on head
movements would enable the control of a number of
effects over three axis of motion – pitch, roll, and yaw –
while enabling both hands to be engaged in playing the
instrument at the same time.
Problems
Head tracking poses two distinct problems: 1)
the detection of the head in an initial frame and 2)
tracking the head, or features within the face, in all
subsequent frames. The human face has, by its nature, a
high degree of variability in its appearance. This makes
face detection a difficult problem in computer vision in
comparison to other problems, where the appearance of
the object to be detected and the location of the camera
may be known in advance.
K. Stephens
Information Technology, NUI Galway
Supervisor: Dr. S. Redfern
k.stephens2@nuigalway.ie
6
Although a number of head-tracking systems
already exist, few are examined under or designed for
all the conditions that such as system would be required
to operate in. Namely:
Approach
• Low latency, ensuring the high level of
responsiveness that would be required by a
musician.
• Low-level ambient lighting
• High levels of dynamic lighting
• Complex backgrounds where motion
independent of the focus head is present
• Handle partial/full occlusions of the face
gracefully & scaling of face as performer
moves in relation to camera
• Intuitive to use
As part of this project, a feature-based tracking
algorithm is developed. A successful system should be
one that is identity-independent i.e. can automatically
detect features without any prior user-focused feature
training; and, one which is capable of doing so
efficiently and robustly.
Feature-based tracking, involves matching
local interest-points between subsequent frames to
update the tracking parameters. Because local feature
matching does not depend on training data, it is less
sensitive to illumination and object appearance. This is
in contrast to appearance-based approaches, such as 2D
Active Appearance Models (AAM)[1] which, although
more stable, can have difficulty generalising to
unseen/untrained images.
The system is developed in C++ with the
OpenCV library, a library developed by Intel, optimised
for Intel architecture and focused on real-time image
processing.
References
[1] T.F.Cootes, G.J. Edwards and C.J.Taylor. "Active
Appearance Models", IEEE Pattern Analysis and Machine
Intelligence, Vol.23, No.6, pp.681-685, 2001