Human Sensing

Human Sensing 

Prof. Joe Paradiso 

Responsive Environments Group, MIT Media Lab 

http://www.media.mit.edu/resenv 

NSF - 1/07

1/07 

Outline 

JAP 

• Motivation 

• Wearable Wireless Sensing 

– Interactive Media/HCI 

– Medical 

–Social 

• Infrastructure Human Sensing 

2

1/07 

Sensing as Commodity 

JAP 

• Sensors are now becoming a commodity, and soon can 

easily be designed into most any device. 

– Rather than omitting them from a cost/complexity viewpoint, it 

begins to make more sense to just include them if there’s any 

suspicion that they could be needed. 

– This causes a shift in how sensors are used – rather than rely on 

only 1 or 2 sensors made a priori to measure particular quantities, 

many sensors will be used that don’t necessarily exactly measure 

the quantity of interest (especially as applications will become 

more general and evolve over time). 

3

1/07 

JAP 

Spring 07 

In Aachen 

200 

Attendees - 

Bartos filled 

4

1/07 

The Prototype Sensor Shoe: 1997 

JAP 

5

Expressive Footwear 

17 Data Channels 

• 2-axis tilt sensor 

• 3-axis compass 

• 1-axis rate gyro 

• 3-axis shock sensor 

• Height sensor (EFS) 

• Sonar receiver 

• 1 PVDF strip (sole) 

• 3 FSR pressure tabs (sole) 

• Bend sensor (sole) 

• 3 Volt Battery Reference 

• Battery low detect 

• 20 kb/sec wireless 

• 413 & 433 MHZ 

• PIC 16C711 

• 50 Hz updates from each foot. 

• ~ 50 mA draw 

- Half day or more of life

1/07 

Modular Wireless Sensor Stack - 2001 

JAP 

Compact Configurable 

Wireless Sensor Node 

7

New Modules for Sensor Stack 

Ari Benbasat 

• Environmental Board 

– VGA camera, PIR, IR, 

Visible light, microphone 

• Active/Passive IMU 

– 3 gyros, 3 accelerometers, 

3 passive tilt switches 

• Data Logger 

– 1 Gb of memory

The Wearable Gait Laboratory 

Stacy Morris

1/07 

“Shuffle Index” vs. “Stride Excitement” via CART 

JAP 

Minimum Foot Inclination (deg) 

Minimum pitch [degrees] 

-15 

-20 

-25 

-30 

-35 

NL F Y 

NL F 

PD F 

-40 

0.012 0.014 0.016 0.018 0.02 0.022 0.024 0.026 0.028 0.03 

Variation in Insole Force [normalized by bodyweight] 

10

Scaling to several dancers… 

Capacitive proximity to 50 cm 

6-axis IMU - 1 Mbps TDMA radio 

100 Hz Full State Updates from 25 nodes 

High Speed Sensor Fusion 

Vocabulary of features

Mid-March Diversion to Ft. Meyers, FL

Red Sox Spring Training - March 2006 

Biomotion measurement of a Red Sox Pitcher 

Modified sensors for high range and high sample rates 

• 1kHz sampling with data logged to flash memory 

• Torso - Single IMU with 10g accelerometer, 1500 deg/sec gyro 

• Upper Arm - Double IMU 

• 10g accel, 1500 deg/sec gyro 

• 70g accel (ADXL78), 10000 deg/sec gyro 

• Wrist - Double IMU 

• 10g accel, 1500 deg/sec gyro 

• 120g accel (ADXL193), 10000 deg/sec gyro 

• Hand - Single IMU with 120g accelerometer, 10000 deg/sec gyro

Preliminary Results - Red Sox Spring Training 

• Acceleration at the wrist 

peaks well above 100g 

• Most of this acceleration 

occurs in a 30ms window 

• Equates to 30 samples for 

the modified inertial system, 

but only 5 frames on a 

180Hz video capture system 

QuickTime and a 

YUV420 codec decompressor 

are needed to see this picture. 

Acceleration Phase of the Pitch Above Captured at Three Critical Locations - Hand, Wrist, and Biceps 

>100 g’s!!

The Disposable Wireless Sensors 

• Very simple “featherweight” motion sensor 

– Cantilevered PVDF piezo strip with proof mass 

– Activates CMOS dual monostable when jerked 

– Sends brief (50 µs) pulse of 300 MHz RF 

– 100 ms dead timer prevents multipulsing 

– Can zone to within ~10 meters via amplitude 

– Ultra low power – battery lasts up to shelf life 

– Extremely cheap – e.g., under $1.00 in large quantity

16 

Life Microscope: Wristband Node 

Kazuo Yano et al - Hitachi Research, Tokyo 

Size 6×4×1.5cm 

Weight 50g include battery 

Battery 1 Day internal battery 

4 Days external battery 

(※Sensing every 2 minutes) 

Wireless IEEE802.15.4 Range 30m 

Sensor Pulse, Acceleration, 

Temperature 

(C) Hitachi Ltd. 2006

17 

Life Microscope: Data 

Activity (/min) Pulse (/min) 

Pulse & Activity Monitoring Result 

140 

100 

50 

0 

400 

200 

0 

0 3 6 9 12 15 18 21 

Time [Hour] 

24 


18 

Day (month/day) 

3/20 

3/21 

3/22 

3/23 

3/24 

3/25 

3/26 

3/27 

3/28 

Age 46, Male 

3/29 

3/30 

Activity Map 

4/1 

(accelerometer) 

4/2 

4/3 

4/4 

4/5 

4/6 

4/7 

4/8 

4/9 

4/10 

0 3 6 9 12 15 18 21 

Time [Hour] 

24 


19 

Understand Real Human Behaviors 

“Satellite Image of My Life” 

Human-behavior knowledge & new services 

Vacation Season 

Travel Abroad 


1/07 

Predicting User Comfort for Utility Control 

JAP 

Mark Feldmeier 

Humidity 

Temp. 

Comfort 

• User wore temperature and humidity sensors at 5 locations for 4 

days (December 2007) 

– Finger (rings), wrist (watches), neck (necklaces), chest (necklaces), shirt 

exterior (pendants) 

• Pager motor vibrated every 5 minutes to prompt user to key in 

HOT/COLD/NEUTRAL 

• 85% accurate in predicting comfort with all sensors 

– 83% accurate just w. wrist temperature, chest temperature, chest humidity 

20

The UberBadge 

Mediate Group Interaction & Behavior Modeling 


TIFF (LZW) decompressor 


Mat Laibowitz 

Responsive Environments 

Group 

• 16-bit MCU w/ 64kb flash, 2k 

RAM, and GCC support 

• 45-LED Display intended to be read 

at distance 

• IR communications 

• RF communications with second 

processor to handle MAC 

• Up to 256MB of data memory 

• Audio input and output with 

onboard microphone 

• Onboard accelerometer 

• Pager motor for vibratory feedback 

• Multiple ports for expansion 

– Accepts Sensor Stack Modules 

• Optional LCD display

1/07 

120 Badges Fabricated 

JAP 

• 100 UbER-Badges dispensed at the Media Lab’s Things 

That Think (TTT) Consortium Meeting on May 3, 2005 

• Facilitate interaction between research sponsor attendees 

and Media Lab 

• Take data for analyzing human dynamics 

• Day broken up into talks, coffee/lunch breaks, 3-hr Open 

House 

22

1/07 

Broadcast Messages 

Information on your badge is mainly for other people, not you! 

JAP 

23

1/07 

Audience Voting in Auditorium 

JAP 




Yes 

63% 

No 

37% 

• Red = No, Green = Yes 

24

1/07 

Exchange Business Card, Bookmark Demos 

JAP 




25

1/07 

Bookmark data posted on website 

JAP 

26

1/07 

Bookmark data posted on website 

JAP 

27

1/07 

Badge Accelerometer & Audio Data 

JAP 

28

1/07 

Auditorium Fidgeting 

JAP 




• Medium-good correlation with length of talk 

• “Resets” with every presentation 

29

1/07 

Interest Meter & Group Dynamics 

JAP 

Badge-Badge 

80% 

Accuracy 

Badge-Demo 

Interested 

Interested 

Uninterested 

Uninterested 

Classification Value = f (voice, motion, face-face time…) 

Collaboration with Human Dynamics Group 

30

1/07 

Affiliated Wearers 

JAP 

Correlated Motion (accels) 

Face-Face Time (IR) 

Affiliated people tend to spend more time face-face 

and move together! 

Accuracy of inferred affiliation: 93% 

31

1/07 

Affiliated Wearers from Energy Only 

JAP 




32

1/07 

Timekeeping for Talks 

JAP 

T – 3 mins T – 2 mins T – 1.5 mins T – 1 mins T – 30 sec 

Out of time!! 

• 24 talks in the morning (research updates) 

• 5 Minute time limit on each! 

• Audience badges flashed time queues 

• We didn’t run over (first time ever…)!! 

34

Block Diagram & Working 

Prototype 



are needed to see this picture.

The SocioScope Platform(s) 

• Location: GPS, tower ID 

• Proximity: Bluetooth, , IR 

• Movement and body language: 

accelerometers 

• Social Signals: voice, movement 

features 

Sandy Pentland, Human Dynamics Group

State Model of Human Behavior 

ps s − 

1 1 

scale 

cluster 

cluster 

( | ) 

2 2 

t t 1 

p s 

p( st 

| st 

− 1) 

0 0 

p st 

st 

− 1 

( | ) 

2 1 

( 

t 

| st) 

p s 

1 0 

( 

t 

| st) 

p x s 

0 

( 

t| t) 

Time CellTower IDs Time CellTower IDs Time CellTower IDs Time CellTower IDs 

time 

• Lots of different observations on different scales 

• Eigenvector analysis provides good prediction!

Friends: 

Self-Report vs. Automatic 

Self-Report Friendship 

1-Day Proximity

330,000 hours for 94 people 

Accuracy of social structure identification: 

• 95% for close circle of friends 

• 96% for work group 

• 98% for boss 

Predict work group satisfaction: r=0.57

1/07 

Personal multimodal Dosimeter 

JAP 

• Measure items of health/environment 

interest on wearable platform 

– Gas, chemicals, particulates, radiation, EM 

field, acoustic environment, UV, temperature, 

location… 

– Grass roots motivation (people vs 

establishment) 

– Mainly media artists now, but evolving 

• Simple products beginning 

• Tad Hirsh, MIT ML 

40

1/07 

MIT PlaceLab 

JAP 




41

1/07 

Bootstrapping a Ubiquitous Sensor Infrastructure 

JAP 

• Sensor networks are the foot soldiers at the front 

lines of ubiquitous computing 

• At this point, few if any customers will buy an 

ensemble of “UbiComp” sensors 

• They will aggregate from established devices 

– Home security, appliances, utility devices, entertainment… 

Just as the web sprouted from a networked ensemble of 

personal computers, true “ubicomp” will arise from an 

armada of networked devices installed for other purposes. 

42

1/07 

Power Strips are Everywhere 

JAP 

• Needed in Homes, offices, especially the Media Lab! 

• Sensors are becoming commodity items 

– Cost of adding sensors to a design is becoming incremental 

• Power strips are ideal base platforms for hosting a 

sensor network 

– Ready access to power 

– Power line can be a network port 

– Can monitor the status of devices that are plugged in 

43

1/07 

PlugPoint – Power Strips as the backbone of a UbiComp Sensor Infrastructure 

J. Lifton, M. Feldmeier, Y. Ono (Ricoh) 

JAP 

Power Line provides 

energy & comm 

Monitor current profiles, 

Switch individual sockets 

Hosts basic sensors (mic, 

light, motion) 

Expansion Port for others 

Hub for wireless sensor 

network 

44

Army of Plugs 


TIFF (Uncompressed) decompressor 


30 ON MEDIA LAB THIRD FLOOR

SecondLab - Binding real sensor data to virtual worlds 

Third floor of ML built in 

Second Life 

ResEnv Lab rendered in 

detail - other areas currently 

derived from map 


TIFF (Uncompressed) decompressor 


Sensor data piped in and 

interpreted as real-time 

graphic phenomena 

Simple sensor apparitions to explore basic ideas 

- Energy use height of fire 

- Activity (sound, motion) whirlwinds 

- Active regions higher walls 

- “Ghost” face individual presence 

Josh Lifton - Building a virtual Media Lab in SecondLife

1/07 

Mobile User Interfaces for Sensor Networks 

JAP 

Browsing, querying, 

navigating through 

sensor net data 

What are the interface 

affordances, displays, 

interaction 

modalities? 

Privacy & Security? 

Mobile platform 

inside of network vs 

fixed platform 

outside? 

McLuhan Extension 

of human senses 

47

1/07 

Nokia 770 Sensor Net Tricorder - Mk. I 

JAP 

48

1/07 

Conclusions 

JAP 

• Sensing, computation, and communication become tightly integrated and 

commonly embedded 

• Low power and energy scavenging enable active nodes to be embedded and 

“forgotten” 

• The Ubicomp infrastructure permiates our cities, our dwellings, our objects, our 

clothing, and eventually our bodies 

– Pervasive-Wearable-Implantable 

• The 5 human senses locked into our body are augmented by interfaces into 

ubiquitous sensor network data 

– Marshall McLuhan for real 

– Interface devices now - implantables some day 

– Omniscience… 

• This infrastructure mediates everything 

– Collaboration, business, social interaction… 

– Context engines filter, represent, and manifest information 

• Google for reality 

• Brave New World 

– Privacy, security, promises vs. perils… 

49

Human Sensing

Create successful ePaper yourself

Delete template?

Save as template?