Real-World Speech Recognition - Haskins Laboratories

NSF SBE Grand Challenge White Paper: Real-World Speech Recognition Philip Rubin
Real-World Speech Recognition
NSF SBE “Grand Challenge” White Paper
by Philip Rubin, Haskins Laboratories, Sep. 24, 2010
Abstract
Speech recognition would seem, to many, to be a scientific/technical
problem that has been solved. Inexpensive recognition systems are
commonly available for personal computers and mobile devices. Why then
is the use of such a potentially enabling technology not as ubiquitous as past
predictions would have led us to believe? Note that I have typed this into
my computer, not spoken to it. One rarely sees people talking to their
computers, unless they are skyping, although the recognition (“talking
typewriter”) technology supposedly has been mastered. Unfortunately,
recognition performance is severely limited by real-world constraints.
Ambient noise, variability in the clarity of a speaker’s voice due to age,
speaker style, infirmity, and a host of other conditions limit the practical and
reliable use of speech interfaces. Speech is more informal and capricious
than algorithmic approaches are designed to handle. In addition, we help
disambiguate such ephemeral information by using as many contextual,
communicative cues as are available to us, including facial information,
gesture, indications of emotion, and situational indicators. The challenge is
to mount a sustained, focused effort to develop recognition systems (speech,
gesture, facial information, emotion, semantic, etc.) that work reliably in
real-world conditions, from the workplace to the battlefield.
Philip Rubin, Ph.D.
Haskins Laboratories
300 George St., Suite 900
New Haven, CT 06511
www.haskins.yale.edu
rubin@haskins.yale.edu
1

NSF SBE Grand Challenge White Paper: Real-World Speech Recognition Philip Rubin
Real-World Speech Recognition
Speech is a convenient, ubiquitous, and easily learned form of
communication for most. Reading is more difficult for many — its mastery
can be challenging, with such difficulties being reflected in the high profile
struggles of our educational system. The centrality of speech to the human
enterprise is considerable. It consumes our attention. Even in this era of
texting and tweeting, the urge to gab is boundless – we swim in a sea of
mobile, vocal communication. We are drawn to the human voice and seek it
out in increasingly varied forms: movies, television, theatre, YouTube
videos, live poetry readings, website multimedia, conferences, etc. Spoken
language creates social, cultural, and geographic bonds, and demarks our
differences, from generational to political even as it allows us to
communicate. We can speak around corners and, through technological
advances, around the world, often at minimal or no expense. The potential
uses of voice, gesture and facial information in assistive and educational
environments are numerous and excite our imaginations (Bickmore and
Cassell, 2005). There are considerable opportunities for enhancing
technologies related to these areas, including improving human-machine
interfaces. I could have spoken into my mobile phone as it automatically
transcribed the intended text (often referred to as “the talking typewriter” in
the olden days) instead of typing as I have always done.
Speech recognition would seem, to many, to be a scientific/technical
problem that has been solved. Inexpensive recognition systems are
commonly available for personal computers and mobile devices. Why then
is the use of such a potentially enabling technology not as ubiquitous as
predictions would lead us to believe? Unfortunately, recognition
2

NSF SBE Grand Challenge White Paper: Real-World Speech Recognition Philip Rubin
performance is severely limited by real-world constraints. Ambient and
background noise, variability in the clarity of a speaker’s voice due to age,
the importance of high-quality unidirectional microphones, speaker style and
accent, dialectal and regional differences, multilingualism, infirmity, and a
host of other factors limit the practical and reliable use of speech interfaces.
Speech is more informal and capricious then algorithmic approaches are
designed to handle. In addition, we help disambiguate such ephemeral
information by using as many contextual, communicative cues as are
potentially available to us, including facial information, gesture, indications
of emotion, and situational indicators.
The use of state-of-the art signal processing, techniques such as
Markovian modeling, other statistical and predictive methods, and a variety
of computational and engineering techniques have led to considerable
advances in recognition performance in the past decade. Such developments
show promise for the future. However, they have reached the limits of their
performance. The ability to extract linguistic categories, rather than
statistical approximations to them, must be informed by the realities and
complexities of the real world if we are to be able to improve our success
rate. This encourages considerations of embodied and situated aspects of
systems. Our physiological form and the nature of the world that we inhabit
-- natural, physical, social and cultural -- shape our behavior. Linguistic and
cognitive contributions to current recognition systems are not as significant
as are, for example, statistical methodologies. Yet, if the knowledge needed
to make improvements is linguistic and/or cognitive, those areas must be
critical to the development of future systems. In addition, the approaches
that we use when studying, modeling, and building such systems need to
give greater attention to what can be very difficult issues: temporality,
3

NSF SBE Grand Challenge White Paper: Real-World Speech Recognition Philip Rubin
dynamics, and complexity. Our scientific approaches must take such issues
seriously.
The scientific/technological difficulties in this enterprise require both
increased attention to scientific fundamentals and a multidisciplinary
approach that brings together biologists, computer scientists, educators,
engineers, linguistics, neuroscientists, psychologists, physicists, and social
scientists. The scientific questions are numerous and difficult. Critical areas
of importance include:
• the physiology of speech production
• neural representations of speech and language and the control of
motor behavior
• computational modeling of language, speech, and the mental lexicon
• development and use of realistic embodied conversational agents
• physical/physiological modeling of sound production
• a deeper understanding of the physics of sound/gesture production
• rich techniques for auditory/visual scene analysis and parsing
• cognitive, emotional, cultural and social aspects of language
understanding and use
Attacking such difficult issues will also require investments in
infrastructure and substantial advances in tool development, such as:
• computational models of speech production that are open source and
modular, and support comparison and addition of existing and
future articulatory and aerodynamic models, include rich
temporal controls to explore the evolution of
physiological/linguistic events over time
• multimodal, large-scale databases that provide for the display,
analysis and archiving of physiological, video, audio,
4

NSF SBE Grand Challenge White Paper: Real-World Speech Recognition Philip Rubin
neuroimaging, and other data, and that support automatic and
expert markup and annotation
• ontologies that represent linguistic, cognitive, and other knowledge
• statistical/computational techniques for characterizing and analyzing
temporal aspects of signals and significant events
In a way, speech is more of a dance than it is a sterile enumeration of
numbers, rules, or lists of features. When we talk, we usually engage in an
active dialogue that spans time. How we act, how we move, and what we say
are determined, in part, by the interactive, dynamically changing relationship
with our partner. As with dance, our speech also conveys our personality,
our heritage, our spontaneity, and our emotions. To deal with this degree of
richness, computational recognition systems should embody approaches that
are ecologically valid, taking into accounts the actions, goals, characteristics,
and other differences of the individuals engaged in conversation. Moreover,
the vagaries and realities of language use, the changing environments in
which language is used, and the diverse and momentarily changing aspects
of those who use it must be central concerns. Where possible, the
relationship between speech and the physical system that produces it,
particularly as it evolves over time, should be understood and used to
constrain and simplify the recognition process (Hogden, et al., 2007;
Iskarous, et al., 2010).
In summary, then, the challenge is to mount a sustained, focused
effort to develop recognition systems (speech, gesture, facial information,
emotion, semantic, etc.) that work reliably in real-world conditions, from the
workplace to the battlefield.
5

NSF SBE Grand Challenge White Paper: Real-World Speech Recognition Philip Rubin
References
Bickmore, Timothy & Cassell, Justine. (2005) “Social Dialogue with Embodied
Conversational Agents” In J. van Kuppevelt, L. Dybkjaer, & N. Bernsen (eds.),
Advances in Natural, Multimodal Dialogue Systems. New York: Kluwer Academic.
Hogden, J., Rubin, P., McDermott, E., Katagiri, S., & Goldstein, L. (2007). Inverting
mappings from smooth paths through R n to paths through R m : A technique applied
to recovering articulation from acoustics. Speech Communication, May 2007,
Volume 49, Issue 5, 361-383.
Iskarous, Khalil, Nam, Hosung, and Whalen, D. H. (2010). Perception of articulatory
dynamics from acoustic signatures. Journal of the Acoustical Society of America,
June 2010, 127 (6), 3717-3727.
Real-World Speech
Recognition by Philip Rubin is licensed under
a Creative
Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
6