Drive-Safe Signal Processing & Advanced Information Technologies ...

Drive-Safe
Signal Processing & Advanced Information
Technologies for improving Driving Prudence &
Accident Reduction
Aytül Erçil
Sabancı University
aytulercil@sabanciuniv.edu

Justification of the Project
1. Every year, 3 million traffic accidents cause a total of
40000 deaths worldwide.
Turkey
Czech Republic
Greece*
France
Belgium
Austria
New Zealand
Japan
Germany
Denmark
Switzerland
Iceland
Australia
USA
Norway
Finland
Sweden
Great Britain
119,8
0 5 10 15 20 25 30 35 40
* 1998
Traffic deaths per 1 billion vehicle kilometres in 1999
BASt - U2 - 38/2001
Source: IRTAD

Justification of the Project
Driver error has been blamed as the primary cause for
approximately 80% of traffic accidents.

EXAMPLES:
U.S.: According to the figures given by US National Highway Traffic Safety
Administration, driver fatigue has resulted in 240,000 fatalities in the U.S.
In addition, it is also reported that sleep related accidents cost public and
private sector over $46 billion every year.
Australia: Cause of accidents : %30 driver fatigue/sleep
Finland: 26% of fatalities in traffic accidents are due to alcohol related
behavior

EXAMPLES (cont):
Canada (1998): 33% of the drivers who died in traffic accidents had an
alcohol level above the legal limit (BAC > 80 mg).
‘UNC Highway Safety Research Center’ finding (1999): A research
performed with 1403 drivers reported that the probability of having an
accident increases two-fold in drivers who slept 6-7 hours as compared to
the ones who slept eight or more hours. Staying awake for 24 hours is
considered to be equivalent to being legally drunk.
A driver who has an alcohol level twice the legal limit or higher is 50 times
more likely to be involved in a deadly accident.

Justification of the Project
1. In over 500.000
accidents in 2005 (in Turkey):
Injured: : 1231
23,985 people
Deceased: 3,215
people
Financial loss: 651,166,236 USD
Number of accidentsı
600.000
500.000
400.000
300.000
200.000
A. Kaza B. Ölü C. Yaralı D.
Maddi Hasar Miktarı (ABD $)
A
100.000
0
1980
1982
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004(*)

In Turkey, the ratio of highway usage in passenger/freight carriage age in
transportation ~% % 959
100
90
80
70
60
50
40
30
20
10
0
27,2
95
58,2
38,3
24
10,5
U.S.
22,5
12
7,3
4
Germany Turkey
0,8
0,2
Different modes in passanger transportation (%)
Highway %
Railway %
Sea %
Air %

Main Goal:
Project Goals
Using signal processing and communications technologies, create
conditions for prudent driving on highways and roadways with the purposes of
reducing accidents caused by driver behavior
To achieve these primary goals, critical data will be collected from multimodal
sensors (such as cameras, microphones and other sensors) to build a unique
databank on
• driver behaviour,
• driver-vehicle interaction,
• vehicle condition and
• transportation conditions
We will develop system and technologies for analyzing the data and automatically
determining potentially dangerous situations (such as driver fatigue, distraction,
drunk driving, and etc.). Based on the findings from these studies, we will
propose systems for warning the drivers and taking other precautionary
measures to avoid accidents once a dangerous situation is detected.

Current Funding Status:
• Turkish Development Agency funding of Drive-Safe
(August 2005-July. 2007)
• Japanese New Energy and Industrial Technology
Development Organization (NEDO) (October 2005-
September 2008)
• FP6 SPICE Project at Sabancı University (May 2005-
April 2008)
• EU FP6 Project at ITU (May 2005- April 2008).

Data Collection Vehicle

Sensors
Video Sensors:
3 day, 3 night cameras
NORPIX Video data acquisition system
Audio sensors:
Driver microphone
Lapel microphone
Visor microphone
Room microphone

Wheel angle sensor
Acceleration and speed sensor
GPS sensor
Alesis 24 canal data acquisition device

Behringer/UltraGain Pro8 Digital
ADA8000 ADC/DAC
(8-channel Input Signal Mixer Amplifiers to
convert 8 audio channels in Alesis)
8 sonar sensors
2D Laser scanner

GPRS SystemS
Brake pedal pressure sensor
(Custom made in Japan)
Electroencephalogram (EEG):

Data obtained from the CANbus
• tire angular speeds,
• steering wheel position and speed,
• engine rotational speed,
• vehicle longitudinal speed,
• vehicle yaw rate,
• turn signal states,
• clutch pedal position switches,
• brake pedal position switch,
• idle gear state and rear gear switch.

Red : reading signs (with OTAM
center)
Green: driving directions (with
OTAM center)
Blue: telephone banking
Start 11:00:46
Finish 11:45:13
Duration 0:44:27
Distance 27.3 km

Nagoya University Data Acquisition Vehicle
Video Images
Vehicle position
Sensor data

Simulator

3D modeling of the course

2 nd Data Acquisition car:
Ford Connect Minivan
Test Vehicle to be used with Simulator (Alternate):
Fiat STILO

Analysis of Nagoya University Data

An International Alliance for Advanced Studies on
In-Car Human Behavioral Signals: DATABASE
Large Database with more
than 800 drivers
- (99% Japanese
Speakers)
Real driving conditions
with Subjects driving on
public streets while
holding dialogues.
Multi-media recordings
Recorded data include
multi-channel audio,
multi-channel video,
vehicle-related
information (speed,
pedals, steering
handle etc..),
location.

Face Recognition from video images
Difficulties
• Wide pose and light variations
• low quality video
• small face images

Speech
Sampling: 16 kHz; 16-bit/sample; 12 channels
Video
MPEG-1; 29.97 frames per second; 3 channels
Driving
Signals
Acceleration, Accelerator Pedal Pressure, Brake Pedal
pressure, Steering Wheel Angle, Engine RPM,
Vehicle Speed: Each at 16 bit/sample and 1.0 kHz.
Location
Differential GPS: one reading per second

Driver Verification
Verification: Determine if the person is who they claim to be (Authentication)

Driver Verification
Verification: Determine if the person is who they claim to be (Authentication)
Error rates (%) when using fixed combination rules for
combining different modalities
Error rates (%) when using trainable combination methods
Individual performance results for different modalities

Analyzing driving signals
The experiments are carried out on two different subsets of
the CIAR dataset, one having 28 drivers and the other having
314 drivers.
Gauss mixture
number
2
4
29.25
28.49
34.04
34.04
B/BdB
BdB
46.80
47.34
64.89
60.96
A/AdA
AdA
22.87
24.46
26.06
25.00
E/EdE
EdE
9.04
13.82
11.17
14.05
S/SdS
SdS
6.38
7.10
8.51
9.04
T/TdT
TdT
Correct classification rates for
28 drivers (B: brake, A: gas, E:
motor speed, S: vehicle speed, T:
wheel angle, dX: derivative
features).
8
39.36
35.63
67.55
68.08
30.31
26.20
11.17
12.76
8.51
9.67
Gauss mixture
number
8 (Bayesian)
8 (weighted)
B+A
(1+2)
A+E
(2+1)
B+E
(2+1)
A+B+E
(3+2+1)
number Correct recognition rates for 28
drivers using combination of
64.36
63.29
44.68
68.61
different modalities (B: brake, A:
69.14
68.08
46.80
68.61
gas, E: motor speed, S: vehicle
speed, T: wheel angle, dX:
derivative features).

Face recognition with independent component based
super-resolution
Osman Gökhan Sezer
Yücel Altunbaşak
Aytül Erçil
Received best student paper award at VCIP 2006

Sub-band
band based SR
The goal is to combine face recognition systems with
super resolution algorithms.
Reduce the computational burden
To make face recognition more robust to noise and motion
estimation errors
w/o noise
Gaussian
(σ 2 = 50)
(i) Original
(ii) NN interpolation
(iii)Bilinear interpolation
(iv) Sub-band based SR
Eigenface
IC-face
Eigenface SR
IC-face
SR
80.88
80.88
94.12
94.12
57.35
52.94
88.23
92.65

Video Based Driver Fatique
Detection

Difficulties:

Driver Fatigue Symtomps :
• Facial Expression
Gaze direction
Head movement
Yawning
Closure of the eyes
• Driving Signals
• Voice Signals

• Facial Expression Analysis
Facial Action coding system

Animation data
Some frames from the Action Unit 1 (inner brow raiser) video
Some frames from the Action Unit 9 (nose wrinkler) video

Proposed Architecture
Subcomponents
Detecting Fatigue
Detecting Fatigue Classes
Action Unit Tracking
Action Unit Declaration
Feature Tracking
Feature Extraction
Data

Current Work
Spatial and temporal modeling of the relationships
Fatigue
Fatigue
Entire Face Behavior
Inattentive
Falling Asleep
Inattentive
Falling Asleep
Single AU’s or AU states
AU 61
AU 62
AU 51 AU 52
AU 61
AU 62
AU 51 AU 52
Partial Face Behavior
Pupil Motion
Gaze
Pupil Motion
Gaze
Sensing Channels
Eye Tracker
Gaze
Tracker
Eye Tracker
Gaze
Tracker
Features
Time n-1
Time n

Mouth Action Units Prediction Results:
After : 87%
Pred
AU25
AU26
AU27
True
AU25
32
2
0
AU26
4
3
0
AU27
1
0
13

Detecting Eye States
Two feature vectors :
1) Distance between lower and upper eyelid positions
2) First derivative of the distance between lower and upper eyelid
positions
Open Closing Closed
Opening

Future Work include:
Analyzing driving signals
Analyzing audio signals
Automated lane following

Active Passive Restraint Systems
Developing active and passive warning systems
Audio warning
Warning via wheel movements
Warning via the driver seat

Development of realistic cars
Car models with various degrees of freedom are being developed in i
ADAMS, MATLAB/Simulink, using real vehicle parameters; control
algorithms are being developed for automated lane following, prevention
of collision-skidding
skidding-rolling.

Lane Following Assistance
A preliminary system prepared in the simulator environment.
analyzes the success of the driver in following the lane and
intervenes if necessary with lane following assistance.
A supervisory control architecture with two levels was used.
The high level controller determines whether the driver needs
lane keeping assistance or not.
If assistance is required, the low level automated lane keeping
controller is switched on.

Lane Following Assistance
In the case of failure, an audio warning signal is first sent to
the driver. If the driver still does not correct his lane
following performance, the automated lane keeping
controller takes over and keeps sending audio warning
signals to the driver to correct his/her steering
performance.

Other Active Safety Methods
Other active safety systems that have been developed and are
being tested include yaw stability control, rollover avoidance
and safe following of preceding traffic using adaptive cruise
control and stop and go assistants.

Future Tasks
Reduce the vehicle speed to safe limits with
effective engine control and braking
Simulations are carried out
under sudden maneuvers
and road conditions with
varying friction
coefficients, controllers
are being developed to
increase the skidding
stability of the vehicle.

Thank you
A.Erçil: aytulercil@sabanciuniv.edu
www.drivesafeproject.org