Sojka Corner Detection Algorithm

Screen Estimation for A Novel 

Pointing Device Based on Corner 

Detection and Classification 

Qing Xia, Wei Huang 

July 30, 2004

TUDelft 

Overview 

• Part I: Problem Definition 

• Part II: Models and Algorithms 

• Part III: System Implementation Design 

• Part IV: System Tests and Conclusions 

Screen Estimation for A Novel Pointing Device Based on Corner Detection and Classification. Qing Xia, Wei Huang 2

TUDelft 

Part I: Problem Definition 

! Introduction and Problem Definition 

! Models and Algorithms Overview 


TUDelft 

Introduction and Problem 

Definition 


TUDelft 

Introduction and Problem Definition 

Pointing devices 

•Microsoft X-Wand 

• Infrared pointing device 

• HeadMouse 

• CyberlinkMindMouse 


TUDelft 


UI-Wand project 

• Camera-Based Approach 

• Related Works 


TUDelft 


UI-Wand hardware components 


TUDelft 


Project goals 

• Screen positioning by screen corner detection 

• Gesture recognition 

• Robustness and stability 

• Develop utilities to evaluate the system 

• Real-time speed (10 frames/sec) 

• Development based on Vispirin 


TUDelft 

Models and Algorithms 

Overview 


TUDelft 

Models and Algorithms Overview 

Pointing Projection 

• How to calculate Ps? 

image coordinates 

x 

image coordinates 

x 

screen coordinates 

x 

screen coordinates 

x 

y 

y 

Pointing Point 

Pi(x,y) 

Ps(x,y) 

y 

y 

Pointing Point 

Pi(x,y) 

Ps ?? 


TUDelft 



C1 

Wy 

Screen 

C2 

Cz 

Cx 

Cy 

UI-Wand 

Wx 

C4 

C3 

Pointing direction 

Wz 

image plane 

world point (x, y, z) 

pinhole (Px, Py, Pz) 

image point (Ix,Iy) 


TUDelft 



• Problem is to estimate a new pointing device position 

π P,O 

: (x, y, z) " (I x 

,I y 

) 

(P' 

, 

O' 

) 

= 

argmin 

(P, 

O) 

4 

∑ ( π 

P , O 

( C 

i 

) − D 

i 

) 

i = 1 

2 


TUDelft 


Screen corner detection 

• Direct corner detection (gradient of brightness) 

• Classification 

Scan 


TUDelft 


Screen corner detection 

• All corners are candidates 

• Four screen corners are final winners 

Candidates 

Winners 

Corner Detection Algorithms 

SOJKA SUSAN COMPASS 

Classification models 

RVM SVM KNN 

Filter Algorithm 

Rectangle 

RVM: Relevance Vector Machine 


TUDelft 


Main Modules 

• Screen corner detection 

• Pointing positioning 

• Tracking 

Video frames from UI-Wand 

Image Preprocessing 

• Gesture recognition 

Pointing Positioning 

Screen Corners Detection 

Pointing Positioning 

Drive 

Cursor 

Gesture Recognition 

Tracking 


Control 

Application 


TUDelft 


ROI Tracking 

• Detect screen corners only in ROIs 

x 

x 

y 

y 

P last 

ROI 


TUDelft 



• Trace analysis 

• Gesture recognition by RVM 

Trace 

Gesture Label 

Tracking Algorithm 

Gesture Recogntion 

CW on ROI 

CW 

Others 

Trace Analysis and RVM 

HMM 

CW: Candidates-Winners approach 


TUDelft 

Part II: Models and Algorithms 

! Sojka Corner Detection Algorithm 

! RVM Corner Classification Model 

! Rectangle Filter 

! ROI Tracking Filter 

! RVM for Gesture Recognition 


TUDelft 

Sojka Corner Detection 

Algorithm 


TUDelft 

Sojka Corner Detection Algorithm 

Corner Detection Algorithms 

! Direct corner detectors 

! Color distribution based corner detectors 


TUDelft 


Direct Corner Detection Algorithms 

! Beaudet’s detector [Bea78] 

! Kitchen and Rosenfeld’s detector [Kit82] 

! Harris and Stephens’ detector [Har88] 

! Deriche and Giraudon’s detector [Der93] 

! Sojka corner detector [Soj03] 


TUDelft 


Other Corner Detection Algorithms 

! SUSAN corner detector [Smi97] 

! Compass operator [Ruz01] 

! A color-distribution-based detector [Son03] 


TUDelft 


SUSAN Corner Detector 

Boundary of mask 

Nucleus of mask 

Selection of mask where 

pixels have same 

brightness as nucleus 

(USAN) 

B 

B 

A 

C 

D 

E 

A 

C 

D 

E 

Dark area 

Light area 

Comments: Simple, Fast, Low accuracy 


TUDelft 


Algorithms Comparison I 

70 

60 

Accuracy Comparison: 

50 

40 

30 

20 

10 

0 

[Bea78] [Kit82] [Har88] [Der93] [Smi97] [Soj03] 

Accuracy Rate 


TUDelft 


Algorithms Comparison II 

Location Error Comparison: 

2.5 

2 

1.5 

1 

0.5 

0 

[Bea78] [Kit82] [Har88] [Der93] [Smi97] [Soj03] 

Location Error 


TUDelft 


Algorithms Comparison III 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

Detection Speed Comparison (in Millisecond): 

7 

18 

[Bea78] [Kit82] [Har88] [Smi97] [Soj03] 

35 

Running Times 

30 

78 


TUDelft 


Comparisons Conclusion 

! The older direct corner detectors always have low detection 

accuracy rate 

! Most of existing corner detector can be used in real time 

detection tasks 

! The Color distribution based algorithms are only fit for still 

image analysis 

! Sojka corner detector has obvious excellent performance 

compared to the other detectors 


TUDelft 


Make a Decision 

Sojka corner detector! 

! High accuracy 

! Low location error 

! Fast enough 


TUDelft 



Main ideas: 

! Exploiting the probability information 

! Explicitly computing the corner angle 

! Computing apparency of the corner 


TUDelft 


Main ideas of Sojka Corner Detection Algorithm 

Neighborhood of Q 

Area a 

Q 

Ω( Q) 

Area b 

α 

Area c 


TUDelft 


Corner Model Definition I 

Brightness 

Edge location 

Perpendicular distance 

from the edge 

Let the derivative of brightness be positive everywhere with a single maximum at 0. 


TUDelft 


Corner Model Definition II 

The edge is oriented by the rule that the higher brightness lies to the left. 

Lower brightness 

Higher brightness 

ϕ 1 

Edge orientation 

ϕ 2 

Edge orientation 

Higher brightness 

Corner axis 

Lower brightness 


ϕ 2 

ϕ 1 

A convex corner model 

A concave corner model 


TUDelft 


Corner Model Definition III 

Gradient direction vector: 

n 

i 

= 

( cosϕ , sinϕ 

) 

i 

i 

Convex corner: 

n 

1 

×n2 

> 

0 

Concave corner: 

n 

1 

×n2 

< 

0 

b 

Brightness definition: 

( X ) 

= 

⎧min 

⎨ 

⎩max 

{ ψ ( n ⋅ ( X − C) 

), 

ψ ( n ⋅ ( X − C) 

)} 

1 

{ ψ ( n ⋅ ( X − C) 

), 

ψ ( n ⋅ ( X − C) 

)} 

1 

2 

2 

if 

n 

1 

× n 

2 

otherwise 

≥ 

0 


TUDelft 


Corner Model Examples 

Convex corner 

Concave corner 


TUDelft 


Three Conditional Probabilities 

! Conditional probability 



P 

P 

P 

( BrQ ∆b( X )) 

( DirQ h( X )) 

( AngQ ∆ ( X )) 

ϕ 


TUDelft 


Combined Probability 

P 

SG 

{ ( ) P( DirQ h( Y )) P( AngQ ∆ϕ( Y ))} 

( X ) = min P BrQ ∆b( Y ) 

Y∈QX 

w 

−1 

( X ) = P ( X ) w ( r( X )) = A min p β ( ∆b( Y )) 

SG 

r 

{ ( ) p ( h( Y )) P( AngQ ∆ϕ 

( Y ))} w r( ) 

( ). 

0 d 

d 

r 

X 

y∈QX 

Why 


TUDelft 


Why? Because… 

X 

Y1 

ϕ( X ) 

ϕ( Y1) 

ϕ( Y2) 

Y2 

Q 


(a) 

(b) 


TUDelft 


Corner Point Decision I 

! 

! --- 

! --- 

∑ w( X 

i 

) ϕ( X ) ∑ 

i 

w( X 

i 

) ϕ( X 

i 

) 

X i∈Ω 

2 X i∈Ω 

µ ϕ 

= 

σ ϕ = 

w( X ) ϕ 

w( X ) 

Corr 

∑ 

X 

i 

∈Ω 

i 

( ) ( ) 

2 

Q = g Q ( Q) 

σ ϕ 

∑ 

X 

i 

[ − µ ] 

Mean values that are computed for a corner candidate point 

Appar 

( Q) P ( ) ( ) ϕ( ) 

SG 

X 

i 

g X 

i 

X 

i 

− µ 

ϕ 

= ∑ 

X 

i 

∈Ω 

∈Ω 

i 

ϕ 

2 

Two functions for final decisions 


TUDelft 


Corner Point Decision II 

! Theorem 1. 

2 

σ ϕ 

( Q) 

--- The function has its maximum just at then points 

lying on the axis of the corner. 

! Theorem 2. 

--- The function Corr( 

Q) 

has its maximum just at the corner 

point. 


TUDelft 


Algorithm Realization 

! Step 1: Computing the magnitude and the direction of the 

gradient of brightness. 

! Step 2: Selecting the candidates corner points. 

2 

! Step 3: Getting the value of µ 

ϕ 

( Q) 

σ 

ϕ 

( Q) 

Corr ( Q) 

Appar( Q) 

by value estimation. 

! Step 4: The candidates with local maximum Corr( Q) 

and at 

2 

which the value of σ ϕ 

( Q) 

Appar( 

Q) 

extending some thresholds 

are specified as corners. 


TUDelft 


Algorithm Tests 

! Accuracy tests (white screen and black screen) 

! Stability tests (rotated frames) 

! Sequence test (continuous frames) 

A PHIILIPS brilliance 109MP 

white computer screen 

A PHILIPS brilliance 180P2 

black LCD computer screen 


TUDelft 


Algorithm Test Results 


TUDelft 


Algorithm Test Results Analysis I 

250 

200 

Corner detection number analysis 

150 

100 

50 

0 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 

White screen 

Black screen 


TUDelft 


Algorithm Test Results Analysis II 

100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

Screen corner detection rate analysis 

1 2 3 4 5 6 7 8 9 101112131415 

White screen 

Black screen 


TUDelft 


Algorithm Test Results Analysis III 

2.5 

2 

Screen corner detection average variance analysis 

1.5 

1 

0.5 

0 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 

White screen 

Black screen 


TUDelft 


Algorithm Test Results Analysis IV 

Screen corner detection speed analysis 


TUDelft 


Algorithm Test Results Analysis V 

83 

82 

81 

80 

79 

78 

77 

76 

75 

Screen corner detection stability analysis 

0 6 9 12 22.5 45 90 

Black screen 


TUDelft 


Test Results Conclusions 

Attentions: 

! Image quality affect the detection results 

! Parameters affect the detection results 

Good test result: 

! Accurate 

! Stable 

! Fast 


TUDelft 

RVM Corner Classification 

Model 


TUDelft 

RVM Corner Classification Model 

Support Vector Machine 

• Finds the hyperplane leaving the 

largest possible fraction of points 

of the same class on the same side, 

while maximizing the distance of 

either class from the hyperplane. 

H1 

H 

H2 

• Disadvantages: No probability 

output; Support vectors grows with 

training set; Mercer’s condition; 

Distance=1/|w| 


TUDelft 


Relevance Vector Machine Theory 

• General model in supervised learning 

Samples: input vectors 

{ N 

x } n n= 

1 

and targets 

N 

{ tn} 

n= 

1 

M 

∑ 

T 

y( x; 

w) 

= φ ( x) 

= w φ(x) 

i= 

1 

w i 

i 

where, 

φ 

( x) = ( φ ( x), 

φ ( x),..., 

φ ( x ) T 

w = 

1 2 

M 

) 

T 

( w1 , w2 

,..., wM 

) 


TUDelft 



• Probabilistic formulation for regression case 

t 

n 

2 

= y( x ; w) 

+ ε 

p( 

t | x ) = N( 

t | y( 

x ), σ ) 

n 

n 

, with noise: 

n 

n 

n 

likelihood: 

where, 

2 

2 N / 2 ⎧ 1 

p( t | w, 

σ ) = (2πσ 

) exp⎨− 

t − Φw 

2 

⎩ 2σ 

Φ 

φ 

= 

[ 

2 

φ( 

x1), 

φ( 

x ), K, 

φ( 

x 

N 

)] 

− 2 

( x 

n 

) = [1, ( x 

n 

, x1), 

K ( x 

n 

, x 

2 

), K, 

K( 

x 

n 

, x 

N 

T 

K )] 

⎫ 

⎬ 

⎭ 

T 


TUDelft 



• Hyperparameters to avoid over-fitting 

(no weight penalty term) 

N 

∏ 

i= 

0 

N 

− 

p( w | α) 

= N( 

| 0, α 

1 ), 

∏ 

i= 

0 

−2 

p( 

α ) = Gamma( α | a, 

b), 

p( 

σ ) = Gamma( β | c, 

d) 

i 

w i 

with, 

β ≡ σ 

2 


TUDelft 



• Prediction in regression case 

∫ 

2 

2 

p( t* | t ) = p( 

t* 

| w, 

α, 

σ ) p( 

w, 

α, 

σ | t) 

dwdαdσ 

∫ 

2 

2 

2 

p( 

t* | t, 

α 

MP 

, σ 

MP 

) = p( 

t* 

| w, 

σ 

MP 

) p( 

w | t, 

α 

MP 

, σ 

MP 

) dw 

2 

p( 

t* | t, 

α MP 

, σ ) = N( 

t* 

| y* 

, σ 

* 

MP 

2 

) 

2 


TUDelft 



• Prediction in regression case 

2 

p( 

t* | t, 

α MP 

, σ ) = N( 

t* 

| y* 

, σ 

* 

MP 

2 

) 

with, 

y = µ T φ( * 

) 

σ 

* 

x 

2 2 

* 

= σ 

MP 

+ x 

T 

φ( x* 

) Σφ( 

*) 

Σ 

µ 

−2 

T 

= ( σ Φ Φ + A 

−2 

= σ 

t 

ΣΦ t 

) 

−1 

A = 

diag( 

α , α1, 

K, 

α 

0 N 

) 


TUDelft 



• Prediction in classification case 

Likelihood changes to: 

with, 

N 

∏ 

tn 

P( t | w) 

= σ{ 

y( 

x ; w)} 

[1 − σ{ 

y( 

x ; w)}] 

n= 

1 

σ ( y) 

= 1 (1 + e 

− y 

n 

) 

n 

1−t 

n 


TUDelft 



• Prediction in classification case 

* 

w 

T 

MP 

x 

y = 

φ( * 

) 

Σ 

w 

T 

= ( Φ BΦ + A 

T 

MP 

= ΣΦ Bt 

) 

−1 

B = diag β , β , K, 

β ) 

β 

n 

( 

1 2 N 

= σ{ y( 

x 

n 

)}[1 −σ{ 

y( 

x 

n 

)}] 


TUDelft 



• Comparison between SVM and RVM 


TUDelft 


Multi-class Classification (K classes) 

• Pair-wise, K*(K-1)/2 RVMs 

• One-versus-others, K RVMs 

C1 

C1 

C2 

C3 

C2 

C3 

(a) 

(b) 

Pair-wise is not more accurate than One-versus-others 


TUDelft 


Multi-class Classification (K classes) 

• Five RVM models used in corner classification 

Left-Top vs Others 

RVM 1 

Right-Top vs Others 

RVM 2 

RVM 

Classifier 

Left-Bottom vs Others 

RVM 3 

Right-bottom vs Others 

RVM 4 

Non-Corner vs Others 

RVM 5 


TUDelft 


Feature Extraction 

• Intensity values as features 

20 pixels 

RGB Color 

R:161 G:168 B:169 

Graylevel 

165 

20 pixels 

Dimension is too big and might be changed 


TUDelft 


DCT Coefficients as Feature Vectors 

• Independence with sample size 

• Dimension become much smaller 

u 

v 

0 

0 1 2 3 4 ... N 

4.96 1.36 -0.48 -0.14 ... 

1 

1.17 0.72 -0.19 

... 

2 

-0.47 -0.29 

... 

3 

-0.03 

... 

4 

... 

... 

M 

Feature Vector: 

[4.96, 1.36, -0.48, -0.14, 1.17, 0.72, -0.19, -0.47, -0.29, -0.03] 


TUDelft 


Feature Extraction on Rotated Images 

• Rotate target window 

d P 

( α, 

β ) = d1 (1 −α)(1 

− β ) + d 

2α 

(1 − β ) + d3(1 

−α) 

β + d 

4αβ 


TUDelft 


Sample Dataset Selection 

• Manually selected sample dataset 

• Synthetic sample datasets 

• Adaptive sample datasets 


TUDelft 



• Sample dataset specification 

Sample Size 

20 x 20 pixels 

Number of Sample Class 5 

Number of Samples 80 

Samples Distribution 10/10/10/10/40 

• Corner samples: 


TUDelft 



• Classification results 

RVM 

Error Analysis Utility Outputs 

KF 

RVN 

HCV 

MV 

GV 

CGV 

MUR 

CD 

WV 

L-2.0 

18.4 

7.26 

11.99 

5.62 

3.85 

93% 

98% 

5.71 


TUDelft 




1. Probability output is effective 

2. Misunderstanding Rate is high 

14 

12 

10 

8 

6 

4 

MV 

HCV 

GV 

CGV 

WV 

2 

0 

Detected Screen Corner Deviation 


TUDelft 


Synthetic Sample Dataset 

• Generation 

Border 

P 

β 

α 

Content 


TUDelft 




Sample Size 





• Corner samples: 


TUDelft 





TUDelft 




RVM 


KF 

RVN 

HCV 

MV 

GV 

CGV 

MUR 

CD 

WV 

G-0.5 

4.4 

36.94 

5.38 

3.47 

2.57 

84.% 

100% 

6.65 

40 

35 

30 

25 

20 

15 

10 

5 

MV 

HCV 

GV 

CGV 

WV 

0 



TUDelft 


Adaptive Sample Dataset 


Sample Size 






TUDelft 



• Non-corner samples selection (three categories) 

Base: The samples that can be miss-classified 

Background: The samples far from corners 

Adaptive: The samples miss-classified before 


TUDelft 



• Samples: 



TUDelft 




RVM 


KF 

RVN 

HCV 

MV 

GV 

CGV 

MUR 

CD 

WV 

L-2.0 

15.6 

13.01 

7.56 

4.33 

2.84 

29% 

100% 

4.93 

14 

12 

10 

8 

6 

4 

MV 

HCV 

GV 

CGV 

WV 

2 

0 



TUDelft 


Sample datasets comparison 

40 

35 

30 

25 

20 

15 

10 

5 

0 

MS 

SYN 

ADAP 

100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

MS 

SYN 

ADAP 

MV HCV GV CGV WV 

MUR 

20 

18 

16 

14 

12 

10 

8 

6 

4 

2 

0 

MS 

SYN 

ADAP 

RVN 


TUDelft 


Sample datasets comparison 

1. Manually selected dataset is not convenient for users 

2. Adaptive dataset makes classification slow 

3. Synthetic dataset is suitable for current use 

Feature extraction speed 

Target Size 

Speed (s/sec) 

10x10 

718 

20x20 

40x40 

199 

49 

So we used synthetic dataset in final system. 


TUDelft 

Rectangle Filter 


TUDelft 


Candidates-Winners Result By far 


TUDelft 


Rectangle Filter with Corner Type Information 

• Select out the final screen corners 


TUDelft 


Rectangle Filter Before RVM 

• Reduce candidates number 

15 candidates decreases to 9 


TUDelft 


Results 

Less than 1 millionsecond per sample 


TUDelft 

ROI Tracking 


TUDelft 

ROI Tracking 

Purposes 

1. Accelerate detection speed 

2. Store the trace information 


TUDelft 

ROI Tracking 

ROI Tracking Filter 

Cannot work well if move fast 


TUDelft 

ROI Tracking 

ROI Prediction 

• Simple motion predication 

y 

MV1 

ROI 

mv = 

predicted 

mv 

1 

mv1 

/ 

* mv 

2 

Pn-2 

MV2 

Pn-1 

Pn 

MVpredicated 

α 

predicted 

= α w 

1 

+ ∗ ( α1 

− α 

2 

) 

o 

ROI 

Pn+1 

x 

⎪⎧ 

mv 

w = ⎨ 

⎪⎩ mv 

2 

1 

/ mv 

/ mv 

1 

2 

, 

, 

if 

if 

mv 

mv 

2 

1 

≤ 

≤ 

mv 

mv 

1 

2 


TUDelft 

ROI Tracking 

ROI Prediction Results 


TUDelft 

ROI Tracking 

Missing Corners Prediction and Alignment 

• Type I missing 

Without screen information 

• Type II missing 

With screen detected in previous frames 

Type I missing seldom happens; 

Need to do more work on Type II missing 


TUDelft 

ROI Tracking 

Type II Missing 

• Miss less than 4 corners 

• Miss 4 corners 


TUDelft 

ROI Tracking 

Improvement Results 

Prediction without alignment 

Prediction with alignment 


TUDelft 



TUDelft 


Gesture Recognition Procedure 

Gesture Pattern 

gesture 

! Observations capturing 

Observation 

capturing 

observations 

A 

E D 

B 

F 

C 

! Feature extraction 

G 

H 

I 

Feature extraction 

A: < a1,b1,c1,d1 > 

! Classification 

B: < a2,b2,c2,d2 > 

C: < a3,b3,c3,d3 > 

... 

H: < a8,b8,c8,d8 > 

Classification 

I: < a9,b9,c9,d9 > 

gesture 1 gesture 2 gesture 3 


TUDelft 


Gesture Recognition Technologies 

! HMM model for classification 

--- States transformation, probability 

! RVM model for classification 

--- Disjoint regions divided ,feature vectors 


TUDelft 


HMM for Gesture Recognition I 

S1 

S2 

S3 

S4 

S5 

C1 C2 C3 C4 C5 

! Training phase 

! Classification phase 


TUDelft 


HMM for Gesture Recognition II 

Phase 1: 

Step 1 Step 2 

Initial 

HMM 

model 

Extended Baum- 

Welch algorithm 

for parameters 

reestimation 

Updating HMM 

model 

Trained 

HMM 

model 

Phase 2: 

Using Viterbi algorithm for gesture recognition 


TUDelft 


HMM for Gesture Recognition III 

! One HMM for each gesture (initial work) 

! Need well-trained HMM (cost time) 

! Speed up technology 

! More fitful for continuous gesture recognition 


TUDelft 


New Idea for Gesture Recognition 

! We just apply simple gestures 

! Gesture is a pattern 

! We already have an existing classifier 

----RVM 


TUDelft 


RVM for Gesture Recognition 

! Training phase 

! Classification phase 


TUDelft 


RVM for Gesture Recognition 

2D screen 

corner 

points 

G e ttin g 

observations 

O b s e rv a tio n lis t 

analysis 

Observations 

re lo c a tio n 

Feature 

extraction 

Make training 

sam ples 

RVM training 

RVM m odel 

Classification 

Output label 


TUDelft 


RVM Training Phase 

! Getting observations 

! Selecting candidate gesture traces 

! Interpolating candidates traces (relocation observations) 

! Extracting feature vectors 

! Training RVM (training samples generation) 


TUDelft 


RVM Training Phase I 

--- Getting observations 

! ROI for tracking screen corners 

! Four screen corners positioning 

! Geometric center of the screen corners 

! A sequence of positions as observations 


TUDelft 


RVM Training Phase II 

---Selecting candidate gesture traces 

Real gesture 

MOVE 

HOLD 

HOLD 

HOLD 

MOVE_START 

MOVE 

MOVE 

MOVE_END 

HOLD 

HOLD 

HOLD 

 

 

 

Data flowing direction 

Position status distinguish 

! Observation list analysis 

--- List length (>=50 elements) 

--- Element states (MOVE, HOLD, MOVE_START, MOVE_END, INITIAL) 

-- Thresholds (MOVEMENT_SENSITIVITY, HOLDING_NUM_UNITTIME, 

MISSING_TOLERANCE) 


TUDelft 


RVM Training Phase III 

--- Interpolating candidate traces 

! Balanced observations distribution 

! Well-shaped gesture pattern 

! Keep the original trace shapes 


TUDelft 


RVM Training Phase IV 

--- Extracting feature vectors 

Gesture trace 

B 

( x 

j 

, y j 

) 

Motion Vector of A->B 

x 

j 

− x , y 

i 

j 

− 

y 

i 

0 0 

A 

( x 

i 

, y i 

) 

Original 2D space 

Motion space 

! Motion vectors (direction and distance) 

! Without normalizing (keep velocity information) 

! 2*N-dimensional vectors (each for one gesture) 


TUDelft 


RVM Training Phase V 

---Training RVM model 

! Gesture traces generation (follows I,II,III,IV) 

! Non-gesture traces generation (see examples) 

! Training RVM (changes kernel function) 

K ( x , x ) ( x x + 1) 

m 

n 

= 

m n 

r 


TUDelft 


Examples 

Short Middle Long 

Random straight lines 

Three different lengths 

Eight different main directions 

Random lines 

Selected main direction 

Random little reflection 

A random straight line non-gesture trace with observations 

a) b) 

Chosen length 

Chosen length 

Move direction 

Random main direction 

Random move direction 

Random reflection range 


TUDelft 


RVM Classification Phase 

! Getting observations 

! Selecting candidate gesture traces 

! Interpolating candidate gesture traces 

! Feature extraction 

! Putting into RVM for classification 

--- Gesture type can be specified. 


TUDelft 


RVM Gesture Recognition Tests I 

Left-Right Right-Left Cross 

Non-gesture 

Non-gesture 

Non-gesture 


TUDelft 


RVM Gesture Recognition Tests II 

---Sequence 1 

25% 

46% 

50% 

50% 50% 

50% 

54% 

75% 

Left-Right Right-Left Cross Non 

20% 

80% 

Right 

Recognition 

Wrong 

recognition 

Total 


TUDelft 


RVM Gesture Recognition Tests II 

---Sequence 2 

0% 

25% 

50% 

50% 

50% 

50% 

100% 

Left-Right Right-Left Cross Non 

75% 

27% 

Total 

73% 

Right 

Recognition 

Wrong 

recognition 


TUDelft 


RVM Gesture Recognition Conclusions 

Problems: 

! Testing results are not so satisfied as our imagination 

--- Training sample qualities and quantities 

Future works: 

! Thresholding every gesture for recognition 

! Continuous real-time processing 


TUDelft 

Part III: System Implementation 

Design 

! UI-Wand System Design 

! UI-Wand Utilities Design 


TUDelft 

UI-Wand System Design 


TUDelft 


Visipirin Framework 

• Application class and Algorithm Interface 


* 

Application 

PIXELTYPE 

1 

DetectionAlgorithm 

AnnotationAlgorithm 

FeatureExtractionAlgorithm 

OutputAlgorithm 

InputAlgorithm 


TUDelft 


Visipirin Framework 

• Application execution 

The command line format: 

(a) applicationname parameters ./theframesyouwanttorun/* 

(b) applicationname parameters.par ./theframesyouwanttorun/* 

The parameters have the following format: 

(a) Scopename=AlgorithmName 

(b) Scopename::ParameterName=ParameterValue 


TUDelft 


UI-Wand System Use Cases 

Synthetic Screen Corners 

Dataset Generation 

Application 

Screen Corners 

Training Application 

Adaptive Screen Corners 

Dataset Collection 

Application 

Gestures Training 

Application 

Users 

Gestures Dataset 

Collection Application 

UI-Wand Application 


TUDelft 


UI-Wand Application Class Diagram 


P IX E L T Y P E 

F ile In p u tP P M 

1 

DetectionAlgorithm 

SojkaCornerD etector 

1 

1 


SojkaC ornerDetection 

* 


AnnotationFeatures 

1 


* 


ClassificationAlgorithm 

R V M C la s s ifie r 

A p p lic a tio n 

1* 

R elevanceVectorM achine 

1 

1 1 

1 


UIW andApplication 

TraceAnalysisAlgorithm 

TraceInterpolation 

1 

O u tp u tA lg o rith m 


O u t p u t P P M 

1 

AnnotationAlgorithm 


D raw Annotation 

1 

1 


ScreenC ornersFilter 

UIW andApplicationUtil 

1 


TUDelft 


UI-Wand Application Sequence Diagram 

UIWandApplication FileInputPPM SojkaCornerDetection AnnotationFeatures RVMClassifier1 ScreenCornerFilter TraceInterpolation RVMClassifier2 

creatFrame 

init 

init 

cornerDetection 

rectangleFilterWithoutType 

cornersFeatureExtraction 

cornersClassification 

rectangleFilterWithType 

missingcornerpredication 

ROIpredication 

gestureFeatureExtraction 

gestureRecognition 

updateROIs 

updateAngle 

updateAngle 


TUDelft 

UI-Wand Utilities Design 


TUDelft 


UI-Wand Utilities Use Cases 

Reference Corners 

Utility 

Visualization 

Utility 

User 

Error Analysis 

Utility 


TUDelft 


Visualization Utility Class Diagram 

Application 

PIXELTYPE 

1 

PIXELTYPE 

ScanImageAppication 

1 

RelevanceVectorMachine 

1 

1 

1 

* 

1 

PIXELTYPE 

PIXELTYPE 

PIXELTYPE 

FileInputPPM 

WindowsFeatures 

1 

RVMClassifier 



ClassificationAlgorithm 



TUDelft 


Error Analysis GUI Utility 


TUDelft 


Error Analysis Results Table 

• Corner detection analysis results 


TUDelft 



• RVM classification analysis results 


TUDelft 



• Complete application detection analysis results 


TUDelft 

Part IV: System Tests and Future 

Works 

! System Tests 

! Conclusions and Future works 


TUDelft 

System Tests 


TUDelft 

System Tests 

System Test Conditions 

! Hardware conditions: 

--- Test machine (PIV-2.4G, 256M) 

--- Test monitor (PHILIPS brilliance 180P2 

black LCD computer screen) 

--- Devices (UI-Wand components) 

! Software conditions: 

--- Operation systems (Red Hat Linux, Windows XP) 

--- Error analysis tool (UI-Wand Error Analysis Utility) 


TUDelft 

System Tests 

System Tests I 

! Off-line tests I 


TUDelft 

System Tests 

System Tests II 

! Off-line tests II 


TUDelft 

System Tests 

System Tests III 

! On-line tests 


TUDelft 

System Tests 

System Tests Conclusions 

! High accuracy 

! Fast speed 


TUDelft 

Conclusions and Future 

Works 


TUDelft 

Conclusions and Future Works 

Conclusions 

• Detect screen corners by Candidates-Winners approach 

• Invent a model for gestures recognition. 

• The System is robust with some lighting changes. 

• It can run different applications. 

• It can work in a range of working space. 

• Its speed is faster than 10 frames/sec. 

• Developed evaluation utilities 

• The development is based on Vispirin, the existing framework 


TUDelft 

Conclusions and Future Works 

Future Works 

• Current system improvement 

1. Parameters optimization 

2. Try Kalman filter to do the motion predication 

3. Do some research on color images 

4. Improve gesture recognition model 

• Invent a new structure 

Accelerate RVM Classification speed by using 

adaptive dataset, which should be a very 

promising way to go 


TUDelft 

Questions? 


TUDelft 

Thank you! 

Soon….

Sojka Corner Detection Algorithm

Create successful ePaper yourself

Delete template?

Save as template?