Sojka Corner Detection Algorithm
Sojka Corner Detection Algorithm
Sojka Corner Detection Algorithm
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
Screen Estimation for A Novel<br />
Pointing Device Based on <strong>Corner</strong><br />
<strong>Detection</strong> and Classification<br />
Qing Xia, Wei Huang<br />
July 30, 2004
TUDelft<br />
Overview<br />
• Part I: Problem Definition<br />
• Part II: Models and <strong>Algorithm</strong>s<br />
• Part III: System Implementation Design<br />
• Part IV: System Tests and Conclusions<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 2
TUDelft<br />
Part I: Problem Definition<br />
! Introduction and Problem Definition<br />
! Models and <strong>Algorithm</strong>s Overview<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 3
TUDelft<br />
Introduction and Problem<br />
Definition<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 4
TUDelft<br />
Introduction and Problem Definition<br />
Pointing devices<br />
•Microsoft X-Wand<br />
• Infrared pointing device<br />
• HeadMouse<br />
• CyberlinkMindMouse<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 5
TUDelft<br />
Introduction and Problem Definition<br />
UI-Wand project<br />
• Camera-Based Approach<br />
• Related Works<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 6
TUDelft<br />
Introduction and Problem Definition<br />
UI-Wand hardware components<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 7
TUDelft<br />
Introduction and Problem Definition<br />
Project goals<br />
• Screen positioning by screen corner detection<br />
• Gesture recognition<br />
• Robustness and stability<br />
• Develop utilities to evaluate the system<br />
• Real-time speed (10 frames/sec)<br />
• Development based on Vispirin<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 8
TUDelft<br />
Models and <strong>Algorithm</strong>s<br />
Overview<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 9
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Pointing Projection<br />
• How to calculate Ps?<br />
image coordinates<br />
x<br />
image coordinates<br />
x<br />
screen coordinates<br />
x<br />
screen coordinates<br />
x<br />
y<br />
y<br />
Pointing Point<br />
Pi(x,y)<br />
Ps(x,y)<br />
y<br />
y<br />
Pointing Point<br />
Pi(x,y)<br />
Ps ??<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 10
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Pointing Projection<br />
C1<br />
Wy<br />
Screen<br />
C2<br />
Cz<br />
Cx<br />
Cy<br />
UI-Wand<br />
Wx<br />
C4<br />
C3<br />
Pointing direction<br />
Wz<br />
image plane<br />
world point (x, y, z)<br />
pinhole (Px, Py, Pz)<br />
image point (Ix,Iy)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 11
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Pointing Projection<br />
• Problem is to estimate a new pointing device position<br />
π P,O<br />
: (x, y, z) " (I x<br />
,I y<br />
)<br />
(P'<br />
,<br />
O'<br />
)<br />
=<br />
argmin<br />
(P,<br />
O)<br />
4<br />
∑ ( π<br />
P , O<br />
( C<br />
i<br />
) − D<br />
i<br />
)<br />
i = 1<br />
2<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 12
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Screen corner detection<br />
• Direct corner detection (gradient of brightness)<br />
• Classification<br />
Scan<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 13
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Screen corner detection<br />
• All corners are candidates<br />
• Four screen corners are final winners<br />
Candidates<br />
Winners<br />
<strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong>s<br />
SOJKA SUSAN COMPASS<br />
Classification models<br />
RVM SVM KNN<br />
Filter <strong>Algorithm</strong><br />
Rectangle<br />
RVM: Relevance Vector Machine<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 14
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Main Modules<br />
• Screen corner detection<br />
• Pointing positioning<br />
• Tracking<br />
Video frames from UI-Wand<br />
Image Preprocessing<br />
• Gesture recognition<br />
Pointing Positioning<br />
Screen <strong>Corner</strong>s <strong>Detection</strong><br />
Pointing Positioning<br />
Drive<br />
Cursor<br />
Gesture Recognition<br />
Tracking<br />
Gesture Recognition<br />
Control<br />
Application<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 15
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
ROI Tracking<br />
• Detect screen corners only in ROIs<br />
x<br />
x<br />
y<br />
y<br />
P last<br />
ROI<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 16
TUDelft<br />
Models and <strong>Algorithm</strong>s Overview<br />
Gesture Recognition<br />
• Trace analysis<br />
• Gesture recognition by RVM<br />
Trace<br />
Gesture Label<br />
Tracking <strong>Algorithm</strong><br />
Gesture Recogntion<br />
CW on ROI<br />
CW<br />
Others<br />
Trace Analysis and RVM<br />
HMM<br />
CW: Candidates-Winners approach<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 17
TUDelft<br />
Part II: Models and <strong>Algorithm</strong>s<br />
! <strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
! RVM <strong>Corner</strong> Classification Model<br />
! Rectangle Filter<br />
! ROI Tracking Filter<br />
! RVM for Gesture Recognition<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 18
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong><br />
<strong>Algorithm</strong><br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 19
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong>s<br />
! Direct corner detectors<br />
! Color distribution based corner detectors<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 20
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Direct <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong>s<br />
! Beaudet’s detector [Bea78]<br />
! Kitchen and Rosenfeld’s detector [Kit82]<br />
! Harris and Stephens’ detector [Har88]<br />
! Deriche and Giraudon’s detector [Der93]<br />
! <strong>Sojka</strong> corner detector [Soj03]<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 21
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Other <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong>s<br />
! SUSAN corner detector [Smi97]<br />
! Compass operator [Ruz01]<br />
! A color-distribution-based detector [Son03]<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 22
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
SUSAN <strong>Corner</strong> Detector<br />
Boundary of mask<br />
Nucleus of mask<br />
Selection of mask where<br />
pixels have same<br />
brightness as nucleus<br />
(USAN)<br />
B<br />
B<br />
A<br />
C<br />
D<br />
E<br />
A<br />
C<br />
D<br />
E<br />
Dark area<br />
Light area<br />
Comments: Simple, Fast, Low accuracy<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 23
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong>s Comparison I<br />
70<br />
60<br />
Accuracy Comparison:<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
[Bea78] [Kit82] [Har88] [Der93] [Smi97] [Soj03]<br />
Accuracy Rate<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 24
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong>s Comparison II<br />
Location Error Comparison:<br />
2.5<br />
2<br />
1.5<br />
1<br />
0.5<br />
0<br />
[Bea78] [Kit82] [Har88] [Der93] [Smi97] [Soj03]<br />
Location Error<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 25
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong>s Comparison III<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
<strong>Detection</strong> Speed Comparison (in Millisecond):<br />
7<br />
18<br />
[Bea78] [Kit82] [Har88] [Smi97] [Soj03]<br />
35<br />
Running Times<br />
30<br />
78<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 26
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Comparisons Conclusion<br />
! The older direct corner detectors always have low detection<br />
accuracy rate<br />
! Most of existing corner detector can be used in real time<br />
detection tasks<br />
! The Color distribution based algorithms are only fit for still<br />
image analysis<br />
! <strong>Sojka</strong> corner detector has obvious excellent performance<br />
compared to the other detectors<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 27
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Make a Decision<br />
<strong>Sojka</strong> corner detector!<br />
! High accuracy<br />
! Low location error<br />
! Fast enough<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 28
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Main ideas:<br />
! Exploiting the probability information<br />
! Explicitly computing the corner angle<br />
! Computing apparency of the corner<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 29
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Main ideas of <strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Neighborhood of Q<br />
Area a<br />
Q<br />
Ω( Q)<br />
Area b<br />
α<br />
Area c<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 30
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> Model Definition I<br />
Brightness<br />
Edge location<br />
Perpendicular distance<br />
from the edge<br />
Let the derivative of brightness be positive everywhere with a single maximum at 0.<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 31
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> Model Definition II<br />
The edge is oriented by the rule that the higher brightness lies to the left.<br />
Lower brightness<br />
Higher brightness<br />
ϕ 1<br />
Edge orientation<br />
ϕ 2<br />
Edge orientation<br />
Higher brightness<br />
<strong>Corner</strong> axis<br />
Lower brightness<br />
<strong>Corner</strong> axis<br />
ϕ 2<br />
ϕ 1<br />
A convex corner model<br />
A concave corner model<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 32
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> Model Definition III<br />
Gradient direction vector:<br />
n<br />
i<br />
=<br />
( cosϕ , sinϕ<br />
)<br />
i<br />
i<br />
Convex corner:<br />
n<br />
1<br />
×n2<br />
><br />
0<br />
Concave corner:<br />
n<br />
1<br />
×n2<br />
<<br />
0<br />
b<br />
Brightness definition:<br />
( X )<br />
=<br />
⎧min<br />
⎨<br />
⎩max<br />
{ ψ ( n ⋅ ( X − C)<br />
),<br />
ψ ( n ⋅ ( X − C)<br />
)}<br />
1<br />
{ ψ ( n ⋅ ( X − C)<br />
),<br />
ψ ( n ⋅ ( X − C)<br />
)}<br />
1<br />
2<br />
2<br />
if<br />
n<br />
1<br />
× n<br />
2<br />
otherwise<br />
≥<br />
0<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 33
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> Model Examples<br />
Convex corner<br />
Concave corner<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 34
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Three Conditional Probabilities<br />
! Conditional probability<br />
! Conditional probability<br />
! Conditional probability<br />
P<br />
P<br />
P<br />
( BrQ ∆b( X ))<br />
( DirQ h( X ))<br />
( AngQ ∆ ( X ))<br />
ϕ<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 35
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Combined Probability<br />
P<br />
SG<br />
{ ( ) P( DirQ h( Y )) P( AngQ ∆ϕ( Y ))}<br />
( X ) = min P BrQ ∆b( Y )<br />
Y∈QX<br />
w<br />
−1<br />
( X ) = P ( X ) w ( r( X )) = A min p β ( ∆b( Y ))<br />
SG<br />
r<br />
{ ( ) p ( h( Y )) P( AngQ ∆ϕ<br />
( Y ))} w r( )<br />
( ).<br />
0 d<br />
d<br />
r<br />
X<br />
y∈QX<br />
Why<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 36
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Why? Because…<br />
X<br />
Y1<br />
ϕ( X )<br />
ϕ( Y1)<br />
ϕ( Y2)<br />
Y2<br />
Q<br />
<strong>Corner</strong> axis<br />
(a)<br />
(b)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 37
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> Point Decision I<br />
!<br />
! ---<br />
! ---<br />
∑ w( X<br />
i<br />
) ϕ( X ) ∑<br />
i<br />
w( X<br />
i<br />
) ϕ( X<br />
i<br />
)<br />
X i∈Ω<br />
2 X i∈Ω<br />
µ ϕ<br />
=<br />
σ ϕ =<br />
w( X ) ϕ<br />
w( X )<br />
Corr<br />
∑<br />
X<br />
i<br />
∈Ω<br />
i<br />
( ) ( )<br />
2<br />
Q = g Q ( Q)<br />
σ ϕ<br />
∑<br />
X<br />
i<br />
[ − µ ]<br />
Mean values that are computed for a corner candidate point<br />
Appar<br />
( Q) P ( ) ( ) ϕ( )<br />
SG<br />
X<br />
i<br />
g X<br />
i<br />
X<br />
i<br />
− µ<br />
ϕ<br />
= ∑<br />
X<br />
i<br />
∈Ω<br />
∈Ω<br />
i<br />
ϕ<br />
2<br />
Two functions for final decisions<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 38
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Corner</strong> Point Decision II<br />
! Theorem 1.<br />
2<br />
σ ϕ<br />
( Q)<br />
--- The function has its maximum just at then points<br />
lying on the axis of the corner.<br />
! Theorem 2.<br />
--- The function Corr(<br />
Q)<br />
has its maximum just at the corner<br />
point.<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 39
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Realization<br />
! Step 1: Computing the magnitude and the direction of the<br />
gradient of brightness.<br />
! Step 2: Selecting the candidates corner points.<br />
2<br />
! Step 3: Getting the value of µ<br />
ϕ<br />
( Q)<br />
σ<br />
ϕ<br />
( Q)<br />
Corr ( Q)<br />
Appar( Q)<br />
by value estimation.<br />
! Step 4: The candidates with local maximum Corr( Q)<br />
and at<br />
2<br />
which the value of σ ϕ<br />
( Q)<br />
Appar(<br />
Q)<br />
extending some thresholds<br />
are specified as corners.<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 40
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Tests<br />
! Accuracy tests (white screen and black screen)<br />
! Stability tests (rotated frames)<br />
! Sequence test (continuous frames)<br />
A PHIILIPS brilliance 109MP<br />
white computer screen<br />
A PHILIPS brilliance 180P2<br />
black LCD computer screen<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 41
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Test Results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 42
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Test Results Analysis I<br />
250<br />
200<br />
<strong>Corner</strong> detection number analysis<br />
150<br />
100<br />
50<br />
0<br />
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15<br />
White screen<br />
Black screen<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 43
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Test Results Analysis II<br />
100<br />
90<br />
80<br />
70<br />
60<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
Screen corner detection rate analysis<br />
1 2 3 4 5 6 7 8 9 101112131415<br />
White screen<br />
Black screen<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 44
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Test Results Analysis III<br />
2.5<br />
2<br />
Screen corner detection average variance analysis<br />
1.5<br />
1<br />
0.5<br />
0<br />
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15<br />
White screen<br />
Black screen<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 45
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Test Results Analysis IV<br />
Screen corner detection speed analysis<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 46
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
<strong>Algorithm</strong> Test Results Analysis V<br />
83<br />
82<br />
81<br />
80<br />
79<br />
78<br />
77<br />
76<br />
75<br />
Screen corner detection stability analysis<br />
0 6 9 12 22.5 45 90<br />
Black screen<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 47
TUDelft<br />
<strong>Sojka</strong> <strong>Corner</strong> <strong>Detection</strong> <strong>Algorithm</strong><br />
Test Results Conclusions<br />
Attentions:<br />
! Image quality affect the detection results<br />
! Parameters affect the detection results<br />
Good test result:<br />
! Accurate<br />
! Stable<br />
! Fast<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 48
TUDelft<br />
RVM <strong>Corner</strong> Classification<br />
Model<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 49
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Support Vector Machine<br />
• Finds the hyperplane leaving the<br />
largest possible fraction of points<br />
of the same class on the same side,<br />
while maximizing the distance of<br />
either class from the hyperplane.<br />
H1<br />
H<br />
H2<br />
• Disadvantages: No probability<br />
output; Support vectors grows with<br />
training set; Mercer’s condition;<br />
Distance=1/|w|<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 50
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• General model in supervised learning<br />
Samples: input vectors<br />
{ N<br />
x } n n=<br />
1<br />
and targets<br />
N<br />
{ tn}<br />
n=<br />
1<br />
M<br />
∑<br />
T<br />
y( x;<br />
w)<br />
= φ ( x)<br />
= w φ(x)<br />
i=<br />
1<br />
w i<br />
i<br />
where,<br />
φ<br />
( x) = ( φ ( x),<br />
φ ( x),...,<br />
φ ( x ) T<br />
w =<br />
1 2<br />
M<br />
)<br />
T<br />
( w1 , w2<br />
,..., wM<br />
)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 51
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Probabilistic formulation for regression case<br />
t<br />
n<br />
2<br />
= y( x ; w)<br />
+ ε<br />
p(<br />
t | x ) = N(<br />
t | y(<br />
x ), σ )<br />
n<br />
n<br />
, with noise:<br />
n<br />
n<br />
n<br />
likelihood:<br />
where,<br />
2<br />
2 N / 2 ⎧ 1<br />
p( t | w,<br />
σ ) = (2πσ<br />
) exp⎨−<br />
t − Φw<br />
2<br />
⎩ 2σ<br />
Φ<br />
φ<br />
=<br />
[<br />
2<br />
φ(<br />
x1),<br />
φ(<br />
x ), K,<br />
φ(<br />
x<br />
N<br />
)]<br />
− 2<br />
( x<br />
n<br />
) = [1, ( x<br />
n<br />
, x1),<br />
K ( x<br />
n<br />
, x<br />
2<br />
), K,<br />
K(<br />
x<br />
n<br />
, x<br />
N<br />
T<br />
K )]<br />
⎫<br />
⎬<br />
⎭<br />
T<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 52
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Hyperparameters to avoid over-fitting<br />
(no weight penalty term)<br />
N<br />
∏<br />
i=<br />
0<br />
N<br />
−<br />
p( w | α)<br />
= N(<br />
| 0, α<br />
1 ),<br />
∏<br />
i=<br />
0<br />
−2<br />
p(<br />
α ) = Gamma( α | a,<br />
b),<br />
p(<br />
σ ) = Gamma( β | c,<br />
d)<br />
i<br />
w i<br />
with,<br />
β ≡ σ<br />
2<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 53
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Prediction in regression case<br />
∫<br />
2<br />
2<br />
p( t* | t ) = p(<br />
t*<br />
| w,<br />
α,<br />
σ ) p(<br />
w,<br />
α,<br />
σ | t)<br />
dwdαdσ<br />
∫<br />
2<br />
2<br />
2<br />
p(<br />
t* | t,<br />
α<br />
MP<br />
, σ<br />
MP<br />
) = p(<br />
t*<br />
| w,<br />
σ<br />
MP<br />
) p(<br />
w | t,<br />
α<br />
MP<br />
, σ<br />
MP<br />
) dw<br />
2<br />
p(<br />
t* | t,<br />
α MP<br />
, σ ) = N(<br />
t*<br />
| y*<br />
, σ<br />
*<br />
MP<br />
2<br />
)<br />
2<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 54
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Prediction in regression case<br />
2<br />
p(<br />
t* | t,<br />
α MP<br />
, σ ) = N(<br />
t*<br />
| y*<br />
, σ<br />
*<br />
MP<br />
2<br />
)<br />
with,<br />
y = µ T φ( *<br />
)<br />
σ<br />
*<br />
x<br />
2 2<br />
*<br />
= σ<br />
MP<br />
+ x<br />
T<br />
φ( x*<br />
) Σφ(<br />
*)<br />
Σ<br />
µ<br />
−2<br />
T<br />
= ( σ Φ Φ + A<br />
−2<br />
= σ<br />
t<br />
ΣΦ t<br />
)<br />
−1<br />
A =<br />
diag(<br />
α , α1,<br />
K,<br />
α<br />
0 N<br />
)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 55
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Prediction in classification case<br />
Likelihood changes to:<br />
with,<br />
N<br />
∏<br />
tn<br />
P( t | w)<br />
= σ{<br />
y(<br />
x ; w)}<br />
[1 − σ{<br />
y(<br />
x ; w)}]<br />
n=<br />
1<br />
σ ( y)<br />
= 1 (1 + e<br />
− y<br />
n<br />
)<br />
n<br />
1−t<br />
n<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 56
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Prediction in classification case<br />
*<br />
w<br />
T<br />
MP<br />
x<br />
y =<br />
φ( *<br />
)<br />
Σ<br />
w<br />
T<br />
= ( Φ BΦ + A<br />
T<br />
MP<br />
= ΣΦ Bt<br />
)<br />
−1<br />
B = diag β , β , K,<br />
β )<br />
β<br />
n<br />
(<br />
1 2 N<br />
= σ{ y(<br />
x<br />
n<br />
)}[1 −σ{<br />
y(<br />
x<br />
n<br />
)}]<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 57
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Relevance Vector Machine Theory<br />
• Comparison between SVM and RVM<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 58
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Multi-class Classification (K classes)<br />
• Pair-wise, K*(K-1)/2 RVMs<br />
• One-versus-others, K RVMs<br />
C1<br />
C1<br />
C2<br />
C3<br />
C2<br />
C3<br />
(a)<br />
(b)<br />
Pair-wise is not more accurate than One-versus-others<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 59
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Multi-class Classification (K classes)<br />
• Five RVM models used in corner classification<br />
Left-Top vs Others<br />
RVM 1<br />
Right-Top vs Others<br />
RVM 2<br />
RVM<br />
Classifier<br />
Left-Bottom vs Others<br />
RVM 3<br />
Right-bottom vs Others<br />
RVM 4<br />
Non-<strong>Corner</strong> vs Others<br />
RVM 5<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 60
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Feature Extraction<br />
• Intensity values as features<br />
20 pixels<br />
RGB Color<br />
R:161 G:168 B:169<br />
Graylevel<br />
165<br />
20 pixels<br />
Dimension is too big and might be changed<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 61
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
DCT Coefficients as Feature Vectors<br />
• Independence with sample size<br />
• Dimension become much smaller<br />
u<br />
v<br />
0<br />
0 1 2 3 4 ... N<br />
4.96 1.36 -0.48 -0.14 ...<br />
1<br />
1.17 0.72 -0.19<br />
...<br />
2<br />
-0.47 -0.29<br />
...<br />
3<br />
-0.03<br />
...<br />
4<br />
...<br />
...<br />
M<br />
Feature Vector:<br />
[4.96, 1.36, -0.48, -0.14, 1.17, 0.72, -0.19, -0.47, -0.29, -0.03]<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 62
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Feature Extraction on Rotated Images<br />
• Rotate target window<br />
d P<br />
( α,<br />
β ) = d1 (1 −α)(1<br />
− β ) + d<br />
2α<br />
(1 − β ) + d3(1<br />
−α)<br />
β + d<br />
4αβ<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 63
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Sample Dataset Selection<br />
• Manually selected sample dataset<br />
• Synthetic sample datasets<br />
• Adaptive sample datasets<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 64
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Sample Dataset Selection<br />
• Sample dataset specification<br />
Sample Size<br />
20 x 20 pixels<br />
Number of Sample Class 5<br />
Number of Samples 80<br />
Samples Distribution 10/10/10/10/40<br />
• <strong>Corner</strong> samples:<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 65
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Sample Dataset Selection<br />
• Classification results<br />
RVM<br />
Error Analysis Utility Outputs<br />
KF<br />
RVN<br />
HCV<br />
MV<br />
GV<br />
CGV<br />
MUR<br />
CD<br />
WV<br />
L-2.0<br />
18.4<br />
7.26<br />
11.99<br />
5.62<br />
3.85<br />
93%<br />
98%<br />
5.71<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 66
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Sample Dataset Selection<br />
• Classification results<br />
1. Probability output is effective<br />
2. Misunderstanding Rate is high<br />
14<br />
12<br />
10<br />
8<br />
6<br />
4<br />
MV<br />
HCV<br />
GV<br />
CGV<br />
WV<br />
2<br />
0<br />
Detected Screen <strong>Corner</strong> Deviation<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 67
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Synthetic Sample Dataset<br />
• Generation<br />
Border<br />
P<br />
β<br />
α<br />
Content<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 68
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Synthetic Sample Dataset<br />
• Sample dataset specification<br />
Sample Size<br />
10 x 10 pixels<br />
Number of Sample Class 5<br />
Number of Samples 100<br />
Samples Distribution 10/10/10/10/60<br />
• <strong>Corner</strong> samples:<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 69
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Synthetic Sample Dataset<br />
• Classification results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 70
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Synthetic Sample Dataset<br />
• Classification results<br />
RVM<br />
Error Analysis Utility Outputs<br />
KF<br />
RVN<br />
HCV<br />
MV<br />
GV<br />
CGV<br />
MUR<br />
CD<br />
WV<br />
G-0.5<br />
4.4<br />
36.94<br />
5.38<br />
3.47<br />
2.57<br />
84.%<br />
100%<br />
6.65<br />
40<br />
35<br />
30<br />
25<br />
20<br />
15<br />
10<br />
5<br />
MV<br />
HCV<br />
GV<br />
CGV<br />
WV<br />
0<br />
Detected Screen <strong>Corner</strong> Deviation<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 71
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Adaptive Sample Dataset<br />
• Sample dataset specification<br />
Sample Size<br />
40 x 40 pixels<br />
Number of Sample Class 5<br />
Number of Samples 108<br />
Samples Distribution 10/10/10/10/68<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 72
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Adaptive Sample Dataset<br />
• Non-corner samples selection (three categories)<br />
Base: The samples that can be miss-classified<br />
Background: The samples far from corners<br />
Adaptive: The samples miss-classified before<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 73
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Adaptive Sample Dataset<br />
• Samples:<br />
• Classification results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 74
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Adaptive Sample Dataset<br />
• Classification results<br />
RVM<br />
Error Analysis Utility Outputs<br />
KF<br />
RVN<br />
HCV<br />
MV<br />
GV<br />
CGV<br />
MUR<br />
CD<br />
WV<br />
L-2.0<br />
15.6<br />
13.01<br />
7.56<br />
4.33<br />
2.84<br />
29%<br />
100%<br />
4.93<br />
14<br />
12<br />
10<br />
8<br />
6<br />
4<br />
MV<br />
HCV<br />
GV<br />
CGV<br />
WV<br />
2<br />
0<br />
Detected Screen <strong>Corner</strong> Deviation<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 75
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Sample datasets comparison<br />
40<br />
35<br />
30<br />
25<br />
20<br />
15<br />
10<br />
5<br />
0<br />
MS<br />
SYN<br />
ADAP<br />
100%<br />
90%<br />
80%<br />
70%<br />
60%<br />
50%<br />
40%<br />
30%<br />
20%<br />
10%<br />
0%<br />
MS<br />
SYN<br />
ADAP<br />
MV HCV GV CGV WV<br />
MUR<br />
20<br />
18<br />
16<br />
14<br />
12<br />
10<br />
8<br />
6<br />
4<br />
2<br />
0<br />
MS<br />
SYN<br />
ADAP<br />
RVN<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 76
TUDelft<br />
RVM <strong>Corner</strong> Classification Model<br />
Sample datasets comparison<br />
1. Manually selected dataset is not convenient for users<br />
2. Adaptive dataset makes classification slow<br />
3. Synthetic dataset is suitable for current use<br />
Feature extraction speed<br />
Target Size<br />
Speed (s/sec)<br />
10x10<br />
718<br />
20x20<br />
40x40<br />
199<br />
49<br />
So we used synthetic dataset in final system.<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 77
TUDelft<br />
Rectangle Filter<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 78
TUDelft<br />
Rectangle Filter<br />
Candidates-Winners Result By far<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 79
TUDelft<br />
Rectangle Filter<br />
Rectangle Filter with <strong>Corner</strong> Type Information<br />
• Select out the final screen corners<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 80
TUDelft<br />
Rectangle Filter<br />
Rectangle Filter Before RVM<br />
• Reduce candidates number<br />
15 candidates decreases to 9<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 81
TUDelft<br />
Rectangle Filter<br />
Results<br />
Less than 1 millionsecond per sample<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 82
TUDelft<br />
ROI Tracking<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 83
TUDelft<br />
ROI Tracking<br />
Purposes<br />
1. Accelerate detection speed<br />
2. Store the trace information<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 84
TUDelft<br />
ROI Tracking<br />
ROI Tracking Filter<br />
Cannot work well if move fast<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 85
TUDelft<br />
ROI Tracking<br />
ROI Prediction<br />
• Simple motion predication<br />
y<br />
MV1<br />
ROI<br />
mv =<br />
predicted<br />
mv<br />
1<br />
mv1<br />
/<br />
* mv<br />
2<br />
Pn-2<br />
MV2<br />
Pn-1<br />
Pn<br />
MVpredicated<br />
α<br />
predicted<br />
= α w<br />
1<br />
+ ∗ ( α1<br />
− α<br />
2<br />
)<br />
o<br />
ROI<br />
Pn+1<br />
x<br />
⎪⎧<br />
mv<br />
w = ⎨<br />
⎪⎩ mv<br />
2<br />
1<br />
/ mv<br />
/ mv<br />
1<br />
2<br />
,<br />
,<br />
if<br />
if<br />
mv<br />
mv<br />
2<br />
1<br />
≤<br />
≤<br />
mv<br />
mv<br />
1<br />
2<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 86
TUDelft<br />
ROI Tracking<br />
ROI Prediction Results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 87
TUDelft<br />
ROI Tracking<br />
Missing <strong>Corner</strong>s Prediction and Alignment<br />
• Type I missing<br />
Without screen information<br />
• Type II missing<br />
With screen detected in previous frames<br />
Type I missing seldom happens;<br />
Need to do more work on Type II missing<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 88
TUDelft<br />
ROI Tracking<br />
Type II Missing<br />
• Miss less than 4 corners<br />
• Miss 4 corners<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 89
TUDelft<br />
ROI Tracking<br />
Improvement Results<br />
Prediction without alignment<br />
Prediction with alignment<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 90
TUDelft<br />
Gesture Recognition<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 91
TUDelft<br />
Gesture Recognition<br />
Gesture Recognition Procedure<br />
Gesture Pattern<br />
gesture<br />
! Observations capturing<br />
Observation<br />
capturing<br />
observations<br />
A<br />
E D<br />
B<br />
F<br />
C<br />
! Feature extraction<br />
G<br />
H<br />
I<br />
Feature extraction<br />
A: < a1,b1,c1,d1 ><br />
! Classification<br />
B: < a2,b2,c2,d2 ><br />
C: < a3,b3,c3,d3 ><br />
...<br />
H: < a8,b8,c8,d8 ><br />
Classification<br />
I: < a9,b9,c9,d9 ><br />
gesture 1 gesture 2 gesture 3<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 92
TUDelft<br />
Gesture Recognition<br />
Gesture Recognition Technologies<br />
! HMM model for classification<br />
--- States transformation, probability<br />
! RVM model for classification<br />
--- Disjoint regions divided ,feature vectors<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 93
TUDelft<br />
Gesture Recognition<br />
HMM for Gesture Recognition I<br />
S1<br />
S2<br />
S3<br />
S4<br />
S5<br />
C1 C2 C3 C4 C5<br />
! Training phase<br />
! Classification phase<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 94
TUDelft<br />
Gesture Recognition<br />
HMM for Gesture Recognition II<br />
Phase 1:<br />
Step 1 Step 2<br />
Initial<br />
HMM<br />
model<br />
Extended Baum-<br />
Welch algorithm<br />
for parameters<br />
reestimation<br />
Updating HMM<br />
model<br />
Trained<br />
HMM<br />
model<br />
Phase 2:<br />
Using Viterbi algorithm for gesture recognition<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 95
TUDelft<br />
Gesture Recognition<br />
HMM for Gesture Recognition III<br />
! One HMM for each gesture (initial work)<br />
! Need well-trained HMM (cost time)<br />
! Speed up technology<br />
! More fitful for continuous gesture recognition<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 96
TUDelft<br />
Gesture Recognition<br />
New Idea for Gesture Recognition<br />
! We just apply simple gestures<br />
! Gesture is a pattern<br />
! We already have an existing classifier<br />
----RVM<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 97
TUDelft<br />
Gesture Recognition<br />
RVM for Gesture Recognition<br />
! Training phase<br />
! Classification phase<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 98
TUDelft<br />
Gesture Recognition<br />
RVM for Gesture Recognition<br />
2D screen<br />
corner<br />
points<br />
G e ttin g<br />
observations<br />
O b s e rv a tio n lis t<br />
analysis<br />
Observations<br />
re lo c a tio n<br />
Feature<br />
extraction<br />
Make training<br />
sam ples<br />
RVM training<br />
RVM m odel<br />
Classification<br />
Output label<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 99
TUDelft<br />
Gesture Recognition<br />
RVM Training Phase<br />
! Getting observations<br />
! Selecting candidate gesture traces<br />
! Interpolating candidates traces (relocation observations)<br />
! Extracting feature vectors<br />
! Training RVM (training samples generation)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 100
TUDelft<br />
Gesture Recognition<br />
RVM Training Phase I<br />
--- Getting observations<br />
! ROI for tracking screen corners<br />
! Four screen corners positioning<br />
! Geometric center of the screen corners<br />
! A sequence of positions as observations<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 101
TUDelft<br />
Gesture Recognition<br />
RVM Training Phase II<br />
---Selecting candidate gesture traces<br />
Real gesture<br />
MOVE<br />
HOLD<br />
HOLD<br />
HOLD<br />
MOVE_START<br />
MOVE<br />
MOVE<br />
MOVE_END<br />
HOLD<br />
HOLD<br />
HOLD<br />
<br />
<br />
<br />
Data flowing direction<br />
Position status distinguish<br />
! Observation list analysis<br />
--- List length (>=50 elements)<br />
--- Element states (MOVE, HOLD, MOVE_START, MOVE_END, INITIAL)<br />
-- Thresholds (MOVEMENT_SENSITIVITY, HOLDING_NUM_UNITTIME,<br />
MISSING_TOLERANCE)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 102
TUDelft<br />
Gesture Recognition<br />
RVM Training Phase III<br />
--- Interpolating candidate traces<br />
! Balanced observations distribution<br />
! Well-shaped gesture pattern<br />
! Keep the original trace shapes<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 103
TUDelft<br />
Gesture Recognition<br />
RVM Training Phase IV<br />
--- Extracting feature vectors<br />
Gesture trace<br />
B<br />
( x<br />
j<br />
, y j<br />
)<br />
Motion Vector of A->B<br />
x<br />
j<br />
− x , y<br />
i<br />
j<br />
−<br />
y<br />
i<br />
0 0<br />
A<br />
( x<br />
i<br />
, y i<br />
)<br />
Original 2D space<br />
Motion space<br />
! Motion vectors (direction and distance)<br />
! Without normalizing (keep velocity information)<br />
! 2*N-dimensional vectors (each for one gesture)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 104
TUDelft<br />
Gesture Recognition<br />
RVM Training Phase V<br />
---Training RVM model<br />
! Gesture traces generation (follows I,II,III,IV)<br />
! Non-gesture traces generation (see examples)<br />
! Training RVM (changes kernel function)<br />
K ( x , x ) ( x x + 1)<br />
m<br />
n<br />
=<br />
m n<br />
r<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 105
TUDelft<br />
Gesture Recognition<br />
Examples<br />
Short Middle Long<br />
Random straight lines<br />
Three different lengths<br />
Eight different main directions<br />
Random lines<br />
Selected main direction<br />
Random little reflection<br />
A random straight line non-gesture trace with observations<br />
a) b)<br />
Chosen length<br />
Chosen length<br />
Move direction<br />
Random main direction<br />
Random move direction<br />
Random reflection range<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 106
TUDelft<br />
Gesture Recognition<br />
RVM Classification Phase<br />
! Getting observations<br />
! Selecting candidate gesture traces<br />
! Interpolating candidate gesture traces<br />
! Feature extraction<br />
! Putting into RVM for classification<br />
--- Gesture type can be specified.<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 107
TUDelft<br />
Gesture Recognition<br />
RVM Gesture Recognition Tests I<br />
Left-Right Right-Left Cross<br />
Non-gesture<br />
Non-gesture<br />
Non-gesture<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 108
TUDelft<br />
Gesture Recognition<br />
RVM Gesture Recognition Tests II<br />
---Sequence 1<br />
25%<br />
46%<br />
50%<br />
50% 50%<br />
50%<br />
54%<br />
75%<br />
Left-Right Right-Left Cross Non<br />
20%<br />
80%<br />
Right<br />
Recognition<br />
Wrong<br />
recognition<br />
Total<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 109
TUDelft<br />
Gesture Recognition<br />
RVM Gesture Recognition Tests II<br />
---Sequence 2<br />
0%<br />
25%<br />
50%<br />
50%<br />
50%<br />
50%<br />
100%<br />
Left-Right Right-Left Cross Non<br />
75%<br />
27%<br />
Total<br />
73%<br />
Right<br />
Recognition<br />
Wrong<br />
recognition<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 110
TUDelft<br />
Gesture Recognition<br />
RVM Gesture Recognition Conclusions<br />
Problems:<br />
! Testing results are not so satisfied as our imagination<br />
--- Training sample qualities and quantities<br />
Future works:<br />
! Thresholding every gesture for recognition<br />
! Continuous real-time processing<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 111
TUDelft<br />
Part III: System Implementation<br />
Design<br />
! UI-Wand System Design<br />
! UI-Wand Utilities Design<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 112
TUDelft<br />
UI-Wand System Design<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 113
TUDelft<br />
UI-Wand System Design<br />
Visipirin Framework<br />
• Application class and <strong>Algorithm</strong> Interface<br />
<strong>Algorithm</strong><br />
*<br />
Application<br />
PIXELTYPE<br />
1<br />
<strong>Detection</strong><strong>Algorithm</strong><br />
Annotation<strong>Algorithm</strong><br />
FeatureExtraction<strong>Algorithm</strong><br />
Output<strong>Algorithm</strong><br />
Input<strong>Algorithm</strong><br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 114
TUDelft<br />
UI-Wand System Design<br />
Visipirin Framework<br />
• Application execution<br />
The command line format:<br />
(a) applicationname parameters ./theframesyouwanttorun/*<br />
(b) applicationname parameters.par ./theframesyouwanttorun/*<br />
The parameters have the following format:<br />
(a) Scopename=<strong>Algorithm</strong>Name<br />
(b) Scopename::ParameterName=ParameterValue<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 115
TUDelft<br />
UI-Wand System Design<br />
UI-Wand System Use Cases<br />
Synthetic Screen <strong>Corner</strong>s<br />
Dataset Generation<br />
Application<br />
Screen <strong>Corner</strong>s<br />
Training Application<br />
Adaptive Screen <strong>Corner</strong>s<br />
Dataset Collection<br />
Application<br />
Gestures Training<br />
Application<br />
Users<br />
Gestures Dataset<br />
Collection Application<br />
UI-Wand Application<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 116
TUDelft<br />
UI-Wand System Design<br />
UI-Wand Application Class Diagram<br />
Input<strong>Algorithm</strong><br />
P IX E L T Y P E<br />
F ile In p u tP P M<br />
1<br />
<strong>Detection</strong><strong>Algorithm</strong><br />
<strong>Sojka</strong><strong>Corner</strong>D etector<br />
1<br />
1<br />
P IX E L T Y P E<br />
<strong>Sojka</strong>C orner<strong>Detection</strong><br />
*<br />
FeatureExtraction<strong>Algorithm</strong><br />
AnnotationFeatures<br />
1<br />
P IX E L T Y P E<br />
*<br />
P IX E L T Y P E<br />
Classification<strong>Algorithm</strong><br />
R V M C la s s ifie r<br />
A p p lic a tio n<br />
1*<br />
R elevanceVectorM achine<br />
1<br />
1 1<br />
1<br />
P IX E L T Y P E<br />
UIW andApplication<br />
TraceAnalysis<strong>Algorithm</strong><br />
TraceInterpolation<br />
1<br />
O u tp u tA lg o rith m<br />
P IX E L T Y P E<br />
O u t p u t P P M<br />
1<br />
Annotation<strong>Algorithm</strong><br />
P IX E L T Y P E<br />
D raw Annotation<br />
1<br />
1<br />
<strong>Algorithm</strong><br />
ScreenC ornersFilter<br />
UIW andApplicationUtil<br />
1<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 117
TUDelft<br />
UI-Wand System Design<br />
UI-Wand Application Sequence Diagram<br />
UIWandApplication FileInputPPM <strong>Sojka</strong><strong>Corner</strong><strong>Detection</strong> AnnotationFeatures RVMClassifier1 Screen<strong>Corner</strong>Filter TraceInterpolation RVMClassifier2<br />
creatFrame<br />
init<br />
init<br />
corner<strong>Detection</strong><br />
rectangleFilterWithoutType<br />
cornersFeatureExtraction<br />
cornersClassification<br />
rectangleFilterWithType<br />
missingcornerpredication<br />
ROIpredication<br />
gestureFeatureExtraction<br />
gestureRecognition<br />
updateROIs<br />
updateAngle<br />
updateAngle<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 118
TUDelft<br />
UI-Wand Utilities Design<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 119
TUDelft<br />
UI-Wand Utilities Design<br />
UI-Wand Utilities Use Cases<br />
Reference <strong>Corner</strong>s<br />
Utility<br />
Visualization<br />
Utility<br />
User<br />
Error Analysis<br />
Utility<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 120
TUDelft<br />
UI-Wand Utilities Design<br />
Visualization Utility Class Diagram<br />
Application<br />
PIXELTYPE<br />
1<br />
PIXELTYPE<br />
ScanImageAppication<br />
1<br />
RelevanceVectorMachine<br />
1<br />
1<br />
1<br />
*<br />
1<br />
PIXELTYPE<br />
PIXELTYPE<br />
PIXELTYPE<br />
FileInputPPM<br />
WindowsFeatures<br />
1<br />
RVMClassifier<br />
Input<strong>Algorithm</strong><br />
FeatureExtraction<strong>Algorithm</strong><br />
Classification<strong>Algorithm</strong><br />
<strong>Algorithm</strong><br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 121
TUDelft<br />
UI-Wand Utilities Design<br />
Error Analysis GUI Utility<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 122
TUDelft<br />
UI-Wand Utilities Design<br />
Error Analysis Results Table<br />
• <strong>Corner</strong> detection analysis results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 123
TUDelft<br />
UI-Wand Utilities Design<br />
Error Analysis Results Table<br />
• RVM classification analysis results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 124
TUDelft<br />
UI-Wand Utilities Design<br />
Error Analysis Results Table<br />
• Complete application detection analysis results<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 125
TUDelft<br />
Part IV: System Tests and Future<br />
Works<br />
! System Tests<br />
! Conclusions and Future works<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 126
TUDelft<br />
System Tests<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 127
TUDelft<br />
System Tests<br />
System Test Conditions<br />
! Hardware conditions:<br />
--- Test machine (PIV-2.4G, 256M)<br />
--- Test monitor (PHILIPS brilliance 180P2<br />
black LCD computer screen)<br />
--- Devices (UI-Wand components)<br />
! Software conditions:<br />
--- Operation systems (Red Hat Linux, Windows XP)<br />
--- Error analysis tool (UI-Wand Error Analysis Utility)<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 128
TUDelft<br />
System Tests<br />
System Tests I<br />
! Off-line tests I<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 129
TUDelft<br />
System Tests<br />
System Tests II<br />
! Off-line tests II<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 130
TUDelft<br />
System Tests<br />
System Tests III<br />
! On-line tests<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 131
TUDelft<br />
System Tests<br />
System Tests Conclusions<br />
! High accuracy<br />
! Fast speed<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 132
TUDelft<br />
Conclusions and Future<br />
Works<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 133
TUDelft<br />
Conclusions and Future Works<br />
Conclusions<br />
• Detect screen corners by Candidates-Winners approach<br />
• Invent a model for gestures recognition.<br />
• The System is robust with some lighting changes.<br />
• It can run different applications.<br />
• It can work in a range of working space.<br />
• Its speed is faster than 10 frames/sec.<br />
• Developed evaluation utilities<br />
• The development is based on Vispirin, the existing framework<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 134
TUDelft<br />
Conclusions and Future Works<br />
Future Works<br />
• Current system improvement<br />
1. Parameters optimization<br />
2. Try Kalman filter to do the motion predication<br />
3. Do some research on color images<br />
4. Improve gesture recognition model<br />
• Invent a new structure<br />
Accelerate RVM Classification speed by using<br />
adaptive dataset, which should be a very<br />
promising way to go<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 135
TUDelft<br />
Questions?<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 136
TUDelft<br />
Thank you!<br />
Soon….<br />
Screen Estimation for A Novel Pointing Device Based on <strong>Corner</strong> <strong>Detection</strong> and Classification. Qing Xia, Wei Huang 137