Human Sensing

Human Sensing
Prof. Joe Paradiso
Responsive Environments Group, MIT Media Lab
http://www.media.mit.edu/resenv
NSF - 1/07

1/07
Outline
JAP
• Motivation
• Wearable Wireless Sensing
– Interactive Media/HCI
– Medical
–Social
• Infrastructure Human Sensing
2

1/07
Sensing as Commodity
JAP
• Sensors are now becoming a commodity, and soon can
easily be designed into most any device.
– Rather than omitting them from a cost/complexity viewpoint, it
begins to make more sense to just include them if there’s any
suspicion that they could be needed.
– This causes a shift in how sensors are used – rather than rely on
only 1 or 2 sensors made a priori to measure particular quantities,
many sensors will be used that don’t necessarily exactly measure
the quantity of interest (especially as applications will become
more general and evolve over time).
3

1/07
JAP
Spring 07
In Aachen
200
Attendees -
Bartos filled
4

1/07
The Prototype Sensor Shoe: 1997
JAP
5

Expressive Footwear
17 Data Channels
• 2-axis tilt sensor
• 3-axis compass
• 1-axis rate gyro
• 3-axis shock sensor
• Height sensor (EFS)
• Sonar receiver
• 1 PVDF strip (sole)
• 3 FSR pressure tabs (sole)
• Bend sensor (sole)
• 3 Volt Battery Reference
• Battery low detect
• 20 kb/sec wireless
• 413 & 433 MHZ
• PIC 16C711
• 50 Hz updates from each foot.
• ~ 50 mA draw
- Half day or more of life

1/07
Modular Wireless Sensor Stack - 2001
JAP
Compact Configurable
Wireless Sensor Node
7

New Modules for Sensor Stack
Ari Benbasat
• Environmental Board
– VGA camera, PIR, IR,
Visible light, microphone
• Active/Passive IMU
– 3 gyros, 3 accelerometers,
3 passive tilt switches
• Data Logger
– 1 Gb of memory

The Wearable Gait Laboratory
Stacy Morris

1/07
“Shuffle Index” vs. “Stride Excitement” via CART
JAP
Minimum Foot Inclination (deg)
Minimum pitch [degrees]
-15
-20
-25
-30
-35
NL F Y
NL F
PD F
-40
0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 0.03
Variation in Insole Force [normalized by bodyweight]
10

Scaling to several dancers…
Capacitive proximity to 50 cm
6-axis IMU - 1 Mbps TDMA radio
100 Hz Full State Updates from 25 nodes
High Speed Sensor Fusion
Vocabulary of features

Mid-March Diversion to Ft. Meyers, FL

Red Sox Spring Training - March 2006
Biomotion measurement of a Red Sox Pitcher
Modified sensors for high range and high sample rates
• 1kHz sampling with data logged to flash memory
• Torso - Single IMU with 10g accelerometer, 1500 deg/sec gyro
• Upper Arm - Double IMU
• 10g accel, 1500 deg/sec gyro
• 70g accel (ADXL78), 10000 deg/sec gyro
• Wrist - Double IMU
• 10g accel, 1500 deg/sec gyro
• 120g accel (ADXL193), 10000 deg/sec gyro
• Hand - Single IMU with 120g accelerometer, 10000 deg/sec gyro

Preliminary Results - Red Sox Spring Training
• Acceleration at the wrist
peaks well above 100g
• Most of this acceleration
occurs in a 30ms window
• Equates to 30 samples for
the modified inertial system,
but only 5 frames on a
180Hz video capture system
QuickTime and a
YUV420 codec decompressor
are needed to see this picture.
Acceleration Phase of the Pitch Above Captured at Three Critical Locations - Hand, Wrist, and Biceps
>100 g’s!!

The Disposable Wireless Sensors
• Very simple “featherweight” motion sensor
– Cantilevered PVDF piezo strip with proof mass
– Activates CMOS dual monostable when jerked
– Sends brief (50 µs) pulse of 300 MHz RF
– 100 ms dead timer prevents multipulsing
– Can zone to within ~10 meters via amplitude
– Ultra low power – battery lasts up to shelf life
– Extremely cheap – e.g., under $1.00 in large quantity

16
Life Microscope: Wristband Node
Kazuo Yano et al - Hitachi Research, Tokyo
Size 6×4×1.5cm
Weight 50g include battery
Battery 1 Day internal battery
4 Days external battery
(※Sensing every 2 minutes)
Wireless IEEE802.15.4 Range 30m
Sensor Pulse, Acceleration,
Temperature
(C) Hitachi Ltd. 2006

17
Life Microscope: Data
Activity (/min) Pulse (/min)
Pulse & Activity Monitoring Result
140
100
50
0
400
200
0
0 3 6 9 12 15 18 21
Time [Hour]
24
(C) Hitachi Ltd. 2006

18
Day (month/day)
3/20
3/21
3/22
3/23
3/24
3/25
3/26
3/27
3/28
Age 46, Male
3/29
3/30
Activity Map
4/1
(accelerometer)
4/2
4/3
4/4
4/5
4/6
4/7
4/8
4/9
4/10
0 3 6 9 12 15 18 21
Time [Hour]
24
(C) Hitachi Ltd. 2006

19
Understand Real Human Behaviors
“Satellite Image of My Life”
Human-behavior knowledge & new services
Vacation Season
Travel Abroad
(C) Hitachi Ltd. 2006

1/07
Predicting User Comfort for Utility Control
JAP
Mark Feldmeier
Humidity
Temp.
Comfort
• User wore temperature and humidity sensors at 5 locations for 4
days (December 2007)
– Finger (rings), wrist (watches), neck (necklaces), chest (necklaces), shirt
exterior (pendants)
• Pager motor vibrated every 5 minutes to prompt user to key in
HOT/COLD/NEUTRAL
• 85% accurate in predicting comfort with all sensors
– 83% accurate just w. wrist temperature, chest temperature, chest humidity
20

The UberBadge
Mediate Group Interaction & Behavior Modeling
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
Mat Laibowitz
Responsive Environments
Group
• 16-bit MCU w/ 64kb flash, 2k
RAM, and GCC support
• 45-LED Display intended to be read
at distance
• IR communications
• RF communications with second
processor to handle MAC
• Up to 256MB of data memory
• Audio input and output with
onboard microphone
• Onboard accelerometer
• Pager motor for vibratory feedback
• Multiple ports for expansion
– Accepts Sensor Stack Modules
• Optional LCD display

1/07
120 Badges Fabricated
JAP
• 100 UbER-Badges dispensed at the Media Lab’s Things
That Think (TTT) Consortium Meeting on May 3, 2005
• Facilitate interaction between research sponsor attendees
and Media Lab
• Take data for analyzing human dynamics
• Day broken up into talks, coffee/lunch breaks, 3-hr Open
House
22

1/07
Broadcast Messages
Information on your badge is mainly for other people, not you!
JAP
23

1/07
Audience Voting in Auditorium
JAP
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
Yes
63%
No
37%
• Red = No, Green = Yes
24

1/07
Exchange Business Card, Bookmark Demos
JAP
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
25

1/07
Bookmark data posted on website
JAP
26

1/07
Bookmark data posted on website
JAP
27

1/07
Badge Accelerometer & Audio Data
JAP
28

1/07
Auditorium Fidgeting
JAP
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
• Medium-good correlation with length of talk
• “Resets” with every presentation
29

1/07
Interest Meter & Group Dynamics
JAP
Badge-Badge
80%
Accuracy
Badge-Demo
Interested
Interested
Uninterested
Uninterested
Classification Value = f (voice, motion, face-face time…)
Collaboration with Human Dynamics Group
30

1/07
Affiliated Wearers
JAP
Correlated Motion (accels)
Face-Face Time (IR)
Affiliated people tend to spend more time face-face
and move together!
Accuracy of inferred affiliation: 93%
31

1/07
Affiliated Wearers from Energy Only
JAP
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
32

1/07
Timekeeping for Talks
JAP
T – 3 mins T – 2 mins T – 1.5 mins T – 1 mins T – 30 sec
Out of time!!
• 24 talks in the morning (research updates)
• 5 Minute time limit on each!
• Audience badges flashed time queues
• We didn’t run over (first time ever…)!!
34

Block Diagram & Working
Prototype
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.

The SocioScope Platform(s)
• Location: GPS, tower ID
• Proximity: Bluetooth, , IR
• Movement and body language:
accelerometers
• Social Signals: voice, movement
features
Sandy Pentland, Human Dynamics Group

State Model of Human Behavior
ps s −
1 1
scale
cluster
cluster
( | )
2 2
t t 1
p s
p( st
| st
− 1)
0 0
p st
st
− 1
( | )
2 1
(
t
| st)
p s
1 0
(
t
| st)
p x s
0
(
t| t)
Time CellTower IDs Time CellTower IDs Time CellTower IDs Time CellTower IDs
time
• Lots of different observations on different scales
• Eigenvector analysis provides good prediction!

Friends:
Self-Report vs. Automatic
Self-Report Friendship
1-Day Proximity

330,000 hours for 94 people
Accuracy of social structure identification:
• 95% for close circle of friends
• 96% for work group
• 98% for boss
Predict work group satisfaction: r=0.57

1/07
Personal multimodal Dosimeter
JAP
• Measure items of health/environment
interest on wearable platform
– Gas, chemicals, particulates, radiation, EM
field, acoustic environment, UV, temperature,
location…
– Grass roots motivation (people vs
establishment)
– Mainly media artists now, but evolving
• Simple products beginning
• Tad Hirsh, MIT ML
40

1/07
MIT PlaceLab
JAP
QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
41

1/07
Bootstrapping a Ubiquitous Sensor Infrastructure
JAP
• Sensor networks are the foot soldiers at the front
lines of ubiquitous computing
• At this point, few if any customers will buy an
ensemble of “UbiComp” sensors
• They will aggregate from established devices
– Home security, appliances, utility devices, entertainment…
Just as the web sprouted from a networked ensemble of
personal computers, true “ubicomp” will arise from an
armada of networked devices installed for other purposes.
42

1/07
Power Strips are Everywhere
JAP
• Needed in Homes, offices, especially the Media Lab!
• Sensors are becoming commodity items
– Cost of adding sensors to a design is becoming incremental
• Power strips are ideal base platforms for hosting a
sensor network
– Ready access to power
– Power line can be a network port
– Can monitor the status of devices that are plugged in
43

1/07
PlugPoint – Power Strips as the backbone of a UbiComp Sensor Infrastructure
J. Lifton, M. Feldmeier, Y. Ono (Ricoh)
JAP
Power Line provides
energy & comm
Monitor current profiles,
Switch individual sockets
Hosts basic sensors (mic,
light, motion)
Expansion Port for others
Hub for wireless sensor
network
44

Army of Plugs
QuickTime and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
30 ON MEDIA LAB THIRD FLOOR

SecondLab - Binding real sensor data to virtual worlds
Third floor of ML built in
Second Life
ResEnv Lab rendered in
detail - other areas currently
derived from map
QuickTime and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Sensor data piped in and
interpreted as real-time
graphic phenomena
Simple sensor apparitions to explore basic ideas
- Energy use height of fire
- Activity (sound, motion) whirlwinds
- Active regions higher walls
- “Ghost” face individual presence
Josh Lifton - Building a virtual Media Lab in SecondLife

1/07
Mobile User Interfaces for Sensor Networks
JAP
Browsing, querying,
navigating through
sensor net data
What are the interface
affordances, displays,
interaction
modalities?
Privacy & Security?
Mobile platform
inside of network vs
fixed platform
outside?
McLuhan Extension
of human senses
47

1/07
Nokia 770 Sensor Net Tricorder - Mk. I
JAP
48

1/07
Conclusions
JAP
• Sensing, computation, and communication become tightly integrated and
commonly embedded
• Low power and energy scavenging enable active nodes to be embedded and
“forgotten”
• The Ubicomp infrastructure permiates our cities, our dwellings, our objects, our
clothing, and eventually our bodies
– Pervasive-Wearable-Implantable
• The 5 human senses locked into our body are augmented by interfaces into
ubiquitous sensor network data
– Marshall McLuhan for real
– Interface devices now - implantables some day
– Omniscience…
• This infrastructure mediates everything
– Collaboration, business, social interaction…
– Context engines filter, represent, and manifest information
• Google for reality
• Brave New World
– Privacy, security, promises vs. perils…
49