2011ICAD-Walker-Keyn.. - Georgia Tech Sonification Lab - Georgia ...

Sonification and Auditory Displays as
Assistive Technology
(Some Sonification Lab Examples)
Bruce N. Walker
Sonification Lab
Georgia Institute of Technology
bruce.walker@psych.gatech.edu
ICAD 2011

Sonification & Auditory Display
Information display by means of sound
Great potential for users who cannot look
or cannot see (and others!)
Still very little deployment of audio as
assistive technology
(despite great work by many ICAD researchers!)

Alarms, Alerts, & Warnings
Taxonomy of Sounds
Object, Item, & Status Indicators
Auditory Menus
Data Representation & Exploration
3D Audio Displays & Spatial
Representations
Soundscapes & Ambient Audio
Arts & Entertainment

Assistive Technology and the
Hierarchy of Human Needs
Abraham Maslow (1943).
A Theory of Human Motivation
Assistive Need more Technology focus on
tends Aesthetic to focus and more
on Cognitive the “lower” needs

Disabilities, Aging Population
Persons with disabilities living
in the community more often
Correlations of age and
disability
Vision loss is prevalent
Mobility impairments, of many
kinds
Cognitive impairments, TBI
Cognitive and Aesthetic needs
remain to be fulfilled

Auditory AT in Sonification Lab
An example of the variety of possibilities…
Wayfinding
Auditory Interfaces (Menus)
Formal Education (Graphs, Stats)
Informal Education
New Work: Driving
…

Wayfinding: SWAN System
System for Wearable Audio
Navigation
Wayfinding for those who
cannot look or cannot see

SWAN Auditory Display
Navigation Beacons
Spatialized audio beacons form a path
which can be followed
Objects & obstacles
e.g., a desk in the hall; phone booth
Surface Transitions
e.g., sidewalk to grass; start of stairway
Location
e.g., lecture hall; intersection; office
Trailhead (P 0 )
Spatialized
Transition Path
Pivot Point
audio earcon
Best Path (Track)
Point where "next beacon"
becomes audible
Annotations
e.g., “Puddle here whenever it rains”
e.g., “Ramp on left side of entrance” Recorded
speech or
TTS

Success with “Good” Beacons

Bad Beacons (!)
Sound design matters!!

Hardware: Bone Phones
Bone conduction audio
Discrete
Ears open (or plugged)
Stereo separation

Software: Auditory Menus
Computers, phones, etc., have a
menu of functions or options
Simple or complex
1-Dimensional or hierarchical

Visual Menus are Sophisticated
Menu
Title
Menu
Item
(grayed
out)
Divider
Menu
Item
Shortcut
Icon
Submenu
Dialog
Box

Auditory Menus For Accessibility
Use sound to supplement or replace the
visual menu
Can’t look or Can’t see situations
Designers must:
Render as much or more info using sound
Optimize performance (menu usage)
Optimize subjective evaluation
Universal design, accessibility

Typical Auditory Menus = TTS
Text-to-Speech (TTS)
Renders text contents only
Missed opportunity for
rendering other details
Slow
Low ratings
Allegra Seidner
Print Preview

TTS Enhancements
Change TTS to “render” available vs.
unavailable items
Normal vs. whisper
Normal vs. quieter
Male vs. female
Performance & preference
Whisper, same voice
Print Preview
Print Preview

Spearcons
Brief sounds used to represent
the menu item, before the TTS
Generated from TTS
Performance & preference
Faster, more accurate
Easier to learn
Preferred over earcons,
auditory icons, TTS-only
Allegra Seidner
Deborah Joseph

Represent:
Size of menu
Location within it
Performance & preference
Faster navigation
Better awareness of menu size
Better situational awareness
Auditory Scrollbars

Speech-based index
Letter sounds (“A”, “B”, etc.)
Provides location in alphabetical list
Like tabs on a reference book
Performance & preference
Especially in long lists
Spindex
Basic Type
Attenuated Type
Decreased Type
Minimal Type

Formal Education for the Blind
Special vs. integrated (“mainstream”) schools
Regular vs. adapted curriculum
Low tech “still” rules
Braille
Push-pins
Thermaform
Audio can help!

Graphs in STEM Education
Math & science curricula are highly visual
Blind students are shut out of STEM

Auditory/Multimodal Graphs
Auditory graph
Visual graph
Various Exports
MIDI, WAV
Jpg, gif
QT movie

Accessible Graphing Engine
Java-based multimodal graphing engine
Powerful methods for rendering
equations, data sets
Multimodal output
Text description
Extended exporting
Flexible user interfaces

Auditory Graph Design Issues
Mapping
Data dimension --> Display
• e.g., Dollars --> Pitch
Polarity
Increasing pitch = increasing or decreasing $
Scaling
Double the pitch = double the $
Context
Equivalent to tick marks, axes, trend lines
Interaction techniques
Training Issues
Creating; interpreting; auditory graphs in pedagogy

Sonification Lab Research in STEM
Multimodal graphs software
Multimodal equations
Assistive technology for STEM
Accessible exams
Non-visual assessment of
cognitive capabilities

Low-N samples
Distributed and
mainstreamed students
Strict curriculum
Challenges to Research
High-stakes testing/assessment
Low impact of deployment

Mwangaza Project:
Accessible STEM in Kenya
Deploy computers & training
Accessible materials for learning and working in
STEM
Training and curriculum development for
teachers in accessible STEM
Development of accessible tests
Auditory display for all of these!

Mwangaza Takeaway Lessons
Very low baseline for education
Very low resources
Blind have even lower expectations
Lack of tools and techniques for teachers
Adapted curriculum hobbles aspirations
STEM largely unattainable, at present
Auditory tools can play an important role

Informal Ed: Dynamic Exhibits
Informal Learning Environments (ILEs)
Physical access
Informational access
Emotional access
PLAY /
EXPERIENCE
MOVE
WORK

The Visual Experience…

(Typical) Music & Sounds in Aquaria
Background / mood music
Audio tours (static, recorded)
Do not convey dynamics of an exhibit
(What is happening, what is moving or
changing, what are the fish doing)

New: Music from Critter Dynamics
Track the creatures (or other moving
bits)
Use the location, derivatives (velocity,
acceleration), and other global
variables to drive a dynamic, real-time
sound track
Background and “celebrity” fish

Expert Composers/Performers
For inspiration…

…and… Dynamic Narration
Docent Studies at Aquarium
Serving the needs of impaired or special
visitor populations
Salience Studies
Development of dynamic descriptions of
aquarium suitable for visitors with vision
loss

Driving and Special Needs
Millions with disabilities need to drive
How can we help them with this task
Carefully designed auditory displays can help
navigation, driving, even emotion regulation

Driving Assistive Audio: IVAT
In-Vehicle Assistive Technology system
Use auditory display as much as possible,
within human factors constraints

Assistive Technology and the
Hierarchy of Human Needs
…
Accessible Aquarium
Auditory Menus
Spindex
Spearcons
Auditory Graphs
Audio Abacus
STEM Education
Driving/IVAT
SWAN System
…

Summary
Many ways to apply auditory displays and
sonification to assistive technology
Design considerations
User-centered, and participatory design
Interdisciplinary development
Careful evaluation
Note: Many great audio AT projects &
researchers in ICAD!!
We are collectively just scratching the surface…
…but it is looking (and sounding) great!

Thank you!
Questions
Bruce.Walker@psych.gatech.edu