Drive-Safe Signal Processing & Advanced Information Technologies ...
Drive-Safe Signal Processing & Advanced Information Technologies ...
Drive-Safe Signal Processing & Advanced Information Technologies ...
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>Drive</strong>-<strong>Safe</strong><br />
<strong>Signal</strong> <strong>Processing</strong> & <strong>Advanced</strong> <strong>Information</strong><br />
<strong>Technologies</strong> for improving Driving Prudence &<br />
Accident Reduction<br />
Aytül Erçil<br />
Sabancı University<br />
aytulercil@sabanciuniv.edu
Justification of the Project<br />
1. Every year, 3 million traffic accidents cause a total of<br />
40000 deaths worldwide.<br />
Turkey<br />
Czech Republic<br />
Greece*<br />
France<br />
Belgium<br />
Austria<br />
New Zealand<br />
Japan<br />
Germany<br />
Denmark<br />
Switzerland<br />
Iceland<br />
Australia<br />
USA<br />
Norway<br />
Finland<br />
Sweden<br />
Great Britain<br />
119,8<br />
0 5 10 15 20 25 30 35 40<br />
* 1998<br />
Traffic deaths per 1 billion vehicle kilometres in 1999<br />
BASt - U2 - 38/2001<br />
Source: IRTAD
Justification of the Project<br />
<strong>Drive</strong>r error has been blamed as the primary cause for<br />
approximately 80% of traffic accidents.
EXAMPLES:<br />
U.S.: According to the figures given by US National Highway Traffic <strong>Safe</strong>ty<br />
Administration, driver fatigue has resulted in 240,000 fatalities in the U.S.<br />
In addition, it is also reported that sleep related accidents cost public and<br />
private sector over $46 billion every year.<br />
Australia: Cause of accidents : %30 driver fatigue/sleep<br />
Finland: 26% of fatalities in traffic accidents are due to alcohol related<br />
behavior
EXAMPLES (cont):<br />
Canada (1998): 33% of the drivers who died in traffic accidents had an<br />
alcohol level above the legal limit (BAC > 80 mg).<br />
‘UNC Highway <strong>Safe</strong>ty Research Center’ finding (1999): A research<br />
performed with 1403 drivers reported that the probability of having an<br />
accident increases two-fold in drivers who slept 6-7 hours as compared to<br />
the ones who slept eight or more hours. Staying awake for 24 hours is<br />
considered to be equivalent to being legally drunk.<br />
<br />
A driver who has an alcohol level twice the legal limit or higher is 50 times<br />
more likely to be involved in a deadly accident.
Justification of the Project<br />
1. In over 500.000<br />
accidents in 2005 (in Turkey):<br />
Injured: : 1231<br />
23,985 people<br />
Deceased: 3,215<br />
people<br />
Financial loss: 651,166,236 USD<br />
Number of accidentsı<br />
600.000<br />
500.000<br />
400.000<br />
300.000<br />
200.000<br />
A. Kaza B. Ölü C. Yaralı D.<br />
Maddi Hasar Miktarı (ABD $)<br />
A<br />
100.000<br />
0<br />
1980<br />
1982<br />
1984<br />
1986<br />
1988<br />
1990<br />
1992<br />
1994<br />
1996<br />
1998<br />
2000<br />
2002<br />
2004(*)
In Turkey, the ratio of highway usage in passenger/freight carriage age in<br />
transportation ~% % 959<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
27,2<br />
95<br />
58,2<br />
38,3<br />
24<br />
10,5<br />
U.S.<br />
22,5<br />
12<br />
7,3<br />
4<br />
Germany Turkey<br />
0,8<br />
0,2<br />
Different modes in passanger transportation (%)<br />
Highway %<br />
Railway %<br />
Sea %<br />
Air %
Main Goal:<br />
Project Goals<br />
Using signal processing and communications technologies, create<br />
conditions for prudent driving on highways and roadways with the purposes of<br />
reducing accidents caused by driver behavior<br />
To achieve these primary goals, critical data will be collected from multimodal<br />
sensors (such as cameras, microphones and other sensors) to build a unique<br />
databank on<br />
• driver behaviour,<br />
• driver-vehicle interaction,<br />
• vehicle condition and<br />
• transportation conditions<br />
We will develop system and technologies for analyzing the data and automatically<br />
determining potentially dangerous situations (such as driver fatigue, distraction,<br />
drunk driving, and etc.). Based on the findings from these studies, we will<br />
propose systems for warning the drivers and taking other precautionary<br />
measures to avoid accidents once a dangerous situation is detected.
Current Funding Status:<br />
• Turkish Development Agency funding of <strong>Drive</strong>-<strong>Safe</strong><br />
(August 2005-July. 2007)<br />
• Japanese New Energy and Industrial Technology<br />
Development Organization (NEDO) (October 2005-<br />
September 2008)<br />
• FP6 SPICE Project at Sabancı University (May 2005-<br />
April 2008)<br />
• EU FP6 Project at ITU (May 2005- April 2008).
Data Collection Vehicle
Sensors<br />
Video Sensors:<br />
3 day, 3 night cameras<br />
NORPIX Video data acquisition system<br />
Audio sensors:<br />
<strong>Drive</strong>r microphone<br />
Lapel microphone<br />
Visor microphone<br />
Room microphone
Wheel angle sensor<br />
Acceleration and speed sensor<br />
GPS sensor<br />
Alesis 24 canal data acquisition device
Behringer/UltraGain Pro8 Digital<br />
ADA8000 ADC/DAC<br />
(8-channel Input <strong>Signal</strong> Mixer Amplifiers to<br />
convert 8 audio channels in Alesis)<br />
8 sonar sensors<br />
2D Laser scanner
GPRS SystemS<br />
Brake pedal pressure sensor<br />
(Custom made in Japan)<br />
Electroencephalogram (EEG):
Data obtained from the CANbus<br />
• tire angular speeds,<br />
• steering wheel position and speed,<br />
• engine rotational speed,<br />
• vehicle longitudinal speed,<br />
• vehicle yaw rate,<br />
• turn signal states,<br />
• clutch pedal position switches,<br />
• brake pedal position switch,<br />
• idle gear state and rear gear switch.
Red : reading signs (with OTAM<br />
center)<br />
Green: driving directions (with<br />
OTAM center)<br />
Blue: telephone banking<br />
Start 11:00:46<br />
Finish 11:45:13<br />
Duration 0:44:27<br />
Distance 27.3 km
Nagoya University Data Acquisition Vehicle<br />
Video Images<br />
Vehicle position<br />
Sensor data
Simulator
3D modeling of the course
2 nd Data Acquisition car:<br />
Ford Connect Minivan<br />
Test Vehicle to be used with Simulator (Alternate):<br />
Fiat STILO
Analysis of Nagoya University Data
An International Alliance for <strong>Advanced</strong> Studies on<br />
In-Car Human Behavioral <strong>Signal</strong>s: DATABASE<br />
<br />
<br />
<br />
Large Database with more<br />
than 800 drivers<br />
- (99% Japanese<br />
Speakers)<br />
Real driving conditions<br />
with Subjects driving on<br />
public streets while<br />
holding dialogues.<br />
Multi-media recordings<br />
Recorded data include<br />
multi-channel audio,<br />
multi-channel video,<br />
vehicle-related<br />
information (speed,<br />
pedals, steering<br />
handle etc..),<br />
location.
Face Recognition from video images<br />
Difficulties<br />
• Wide pose and light variations<br />
• low quality video<br />
• small face images
Speech<br />
Sampling: 16 kHz; 16-bit/sample; 12 channels<br />
Video<br />
MPEG-1; 29.97 frames per second; 3 channels<br />
Driving<br />
<strong>Signal</strong>s<br />
Acceleration, Accelerator Pedal Pressure, Brake Pedal<br />
pressure, Steering Wheel Angle, Engine RPM,<br />
Vehicle Speed: Each at 16 bit/sample and 1.0 kHz.<br />
Location<br />
Differential GPS: one reading per second
<strong>Drive</strong>r Verification<br />
Verification: Determine if the person is who they claim to be (Authentication)
<strong>Drive</strong>r Verification<br />
Verification: Determine if the person is who they claim to be (Authentication)<br />
Error rates (%) when using fixed combination rules for<br />
combining different modalities<br />
Error rates (%) when using trainable combination methods<br />
Individual performance results for different modalities
Analyzing driving signals<br />
The experiments are carried out on two different subsets of<br />
the CIAR dataset, one having 28 drivers and the other having<br />
314 drivers.<br />
Gauss mixture<br />
number<br />
2<br />
4<br />
29.25<br />
28.49<br />
34.04<br />
34.04<br />
B/BdB<br />
BdB<br />
46.80<br />
47.34<br />
64.89<br />
60.96<br />
A/AdA<br />
AdA<br />
22.87<br />
24.46<br />
26.06<br />
25.00<br />
E/EdE<br />
EdE<br />
9.04<br />
13.82<br />
11.17<br />
14.05<br />
S/SdS<br />
SdS<br />
6.38<br />
7.10<br />
8.51<br />
9.04<br />
T/TdT<br />
TdT<br />
Correct classification rates for<br />
28 drivers (B: brake, A: gas, E:<br />
motor speed, S: vehicle speed, T:<br />
wheel angle, dX: derivative<br />
features).<br />
8<br />
39.36<br />
35.63<br />
67.55<br />
68.08<br />
30.31<br />
26.20<br />
11.17<br />
12.76<br />
8.51<br />
9.67<br />
Gauss mixture<br />
number<br />
8 (Bayesian)<br />
8 (weighted)<br />
B+A<br />
(1+2)<br />
A+E<br />
(2+1)<br />
B+E<br />
(2+1)<br />
A+B+E<br />
(3+2+1)<br />
number Correct recognition rates for 28<br />
drivers using combination of<br />
64.36<br />
63.29<br />
44.68<br />
68.61<br />
different modalities (B: brake, A:<br />
69.14<br />
68.08<br />
46.80<br />
68.61<br />
gas, E: motor speed, S: vehicle<br />
speed, T: wheel angle, dX:<br />
derivative features).
Face recognition with independent component based<br />
super-resolution<br />
Osman Gökhan Sezer<br />
Yücel Altunbaşak<br />
Aytül Erçil<br />
Received best student paper award at VCIP 2006
Sub-band<br />
band based SR<br />
The goal is to combine face recognition systems with<br />
super resolution algorithms.<br />
<br />
<br />
Reduce the computational burden<br />
To make face recognition more robust to noise and motion<br />
estimation errors<br />
w/o noise<br />
Gaussian<br />
(σ 2 = 50)<br />
(i) Original<br />
(ii) NN interpolation<br />
(iii)Bilinear interpolation<br />
(iv) Sub-band based SR<br />
Eigenface<br />
IC-face<br />
Eigenface SR<br />
IC-face<br />
SR<br />
80.88<br />
80.88<br />
94.12<br />
94.12<br />
57.35<br />
52.94<br />
88.23<br />
92.65
Video Based <strong>Drive</strong>r Fatique<br />
Detection
Difficulties:
<strong>Drive</strong>r Fatigue Symtomps :<br />
• Facial Expression<br />
Gaze direction<br />
Head movement<br />
Yawning<br />
Closure of the eyes<br />
• Driving <strong>Signal</strong>s<br />
• Voice <strong>Signal</strong>s
• Facial Expression Analysis<br />
Facial Action coding system
Animation data<br />
Some frames from the Action Unit 1 (inner brow raiser) video<br />
Some frames from the Action Unit 9 (nose wrinkler) video
Proposed Architecture<br />
Subcomponents<br />
Detecting Fatigue<br />
Detecting Fatigue Classes<br />
Action Unit Tracking<br />
Action Unit Declaration<br />
Feature Tracking<br />
Feature Extraction<br />
Data
Current Work<br />
Spatial and temporal modeling of the relationships<br />
Fatigue<br />
Fatigue<br />
Entire Face Behavior<br />
Inattentive<br />
Falling Asleep<br />
Inattentive<br />
Falling Asleep<br />
Single AU’s or AU states<br />
AU 61<br />
AU 62<br />
AU 51 AU 52<br />
AU 61<br />
AU 62<br />
AU 51 AU 52<br />
Partial Face Behavior<br />
Pupil Motion<br />
Gaze<br />
Pupil Motion<br />
Gaze<br />
Sensing Channels<br />
Eye Tracker<br />
Gaze<br />
Tracker<br />
Eye Tracker<br />
Gaze<br />
Tracker<br />
Features<br />
Time n-1<br />
Time n
Mouth Action Units Prediction Results:<br />
After : 87%<br />
Pred<br />
AU25<br />
AU26<br />
AU27<br />
True<br />
AU25<br />
32<br />
2<br />
0<br />
AU26<br />
4<br />
3<br />
0<br />
AU27<br />
1<br />
0<br />
13
Detecting Eye States<br />
Two feature vectors :<br />
1) Distance between lower and upper eyelid positions<br />
2) First derivative of the distance between lower and upper eyelid<br />
positions<br />
Open Closing Closed<br />
Opening
Future Work include:<br />
Analyzing driving signals<br />
Analyzing audio signals<br />
Automated lane following
Active Passive Restraint Systems<br />
Developing active and passive warning systems<br />
<br />
<br />
<br />
Audio warning<br />
Warning via wheel movements<br />
Warning via the driver seat
Development of realistic cars<br />
<br />
Car models with various degrees of freedom are being developed in i<br />
ADAMS, MATLAB/Simulink, using real vehicle parameters; control<br />
algorithms are being developed for automated lane following, prevention<br />
of collision-skidding<br />
skidding-rolling.
Lane Following Assistance<br />
A preliminary system prepared in the simulator environment.<br />
analyzes the success of the driver in following the lane and<br />
intervenes if necessary with lane following assistance.<br />
A supervisory control architecture with two levels was used.<br />
The high level controller determines whether the driver needs<br />
lane keeping assistance or not.<br />
If assistance is required, the low level automated lane keeping<br />
controller is switched on.
Lane Following Assistance<br />
In the case of failure, an audio warning signal is first sent to<br />
the driver. If the driver still does not correct his lane<br />
following performance, the automated lane keeping<br />
controller takes over and keeps sending audio warning<br />
signals to the driver to correct his/her steering<br />
performance.
Other Active <strong>Safe</strong>ty Methods<br />
Other active safety systems that have been developed and are<br />
being tested include yaw stability control, rollover avoidance<br />
and safe following of preceding traffic using adaptive cruise<br />
control and stop and go assistants.
Future Tasks<br />
<br />
<br />
Reduce the vehicle speed to safe limits with<br />
effective engine control and braking<br />
Simulations are carried out<br />
under sudden maneuvers<br />
and road conditions with<br />
varying friction<br />
coefficients, controllers<br />
are being developed to<br />
increase the skidding<br />
stability of the vehicle.
Thank you<br />
A.Erçil: aytulercil@sabanciuniv.edu<br />
www.drivesafeproject.org