Improvements of the Lucas-Kanade Optical-Flow Algorithm

Improvements of the Lucas-Kanade Optical-Flow Algorithm 

An Improvement of the Lucas-Kanade Optical-Flow 

Algorithm for every Circumstance 

Lorenz Gerstmayr 

Computer Engineering Group 

Faculty of Technology 

University of Bielefeld 

2008-06-11 

Corrected version: 2008-08-05


Today’s talk 

1 Basics about optical flow 

2 Standard Lucas-Kanade method 

3 Improvements 

4 Discussion


Basics about optical flow 

Definition 

Definition of optical flow 

Definiton 

P ′ (1) 

u 

Nodal point 

P(1) P(2) ∈ R 3 

P ′ (2) ∈ R 2 

Image 

Example 

http://www.ezls.fb12.uni-siegen.de/lehre/WebOne/index.htm



Definition 

Image Brightness Constancy Constraint (IBCC) 

B: 

A: 

Image brightness constancy constraint: 

A(p + u(p)) = B(p)



Common problems 

Common problems 

Aperture problem Correspondence problem 

After: B. Jähne, Dig. Img. Proc., 2000


Standard Lucas-Kanade method 

Basics 

Optic flow equation 


Optic flow equation: 

A(p + u(p)) = B(p) 

First order Taylor approximation (with u = [ux, uy] ⊤ ): 

A(p) + Ax(p)ux(p) + Ay(p)uy(p) = B(p) 

 

A + [Ax, Ay] 

 

ux 

[Ax, Ay] 

uy 

ux 

uy 

+ (A − B) 

 

At 

= B 

= 0 

B. Jähne, Dig. Img. Proc., 2000; B.D. Lucas & T. Kanade, Proc. Im. Underst. Workshop, 1981



Derivation 

Derivation 

Optic flow equation: 

 

ux 

[Ax, Ay] + At = 0 

uy 

Underdetermined problem (aperture problem) 

Approach: 

Assuming optical flow to be locally constant 

Weighting w.r.t. distance to center pixel p of image patch Ω: 

e(u) = 

w 2 

 

ux(x) 

(x) [Ax(x), Bx(x)] + At(x) 

uy(x) 

x∈Ω 

2 

! 

= min 




Derivation 

Derivation 

Weighted least squares approach 

e(u) = 

w 

x∈Ω 

2 

2 ux 

! 

[Ax, Ay] + At = min 

uy 

Partial derivatives: 

∂e 

= 2 

∂ux 

 

w 

x∈Ω 

2 

 

ux 

[Ax, Ay] + At Ax 

uy 

! = min 

∂e 

= 2 

∂uy 

 

w 

x∈Ω 

2 

 

ux 

[Ax, Ay] + At Ay 

uy 

! = min 




Derivation 

Derivation 

Weighted least squares approach: 

Sorting: 

∂e 

= 2 

∂ux 

 

w 

x∈Ω 

2 

 

ux 

[Ax, Ay] + At Ax 

uy 

! = min 

∂e 

= 2 

∂uy 

 

w 

x∈Ω 

2 

 

ux 

[Ax, Ay] + At Ay 

uy 

! = min 

 

 

〈AxAx〉 〈AxAy〉 ux 〈AxAt〉 

= − 

〈AxAy〉 〈AyAy〉 uy 〈AyAt〉 

 

G 

b 

〈a〉 = 

w 2 (x)a(x) 

x∈Ω



Derivation 

Solution 

Weighted least squares approach: 

Aperture problem 

Eigenvalues λ1, λ2 of G: 

Gu = b 

u = G −1 b 

λ1, λ2 > 0 λ1 > 0, λ2 ≈ 0 λ1, λ2 ≈ 0 




Limitations and restrictions 

Limitations and possible improvements 

Standard LK 

Iterated LK 

Multiscale LK 

baseline 

most efficient 

more 

accurate 

large 

displacements 

Illumination Tolerance 

illumination 

changes 

LK with flowfield constraints 

Global LK 

types of 

flowfields 

global flow 

computation


Improvements 

Iterated Lucas-Kanade algorithm 

Motivation 

Sketch Motivation 

Deviations due to first order 

approximation 

Large displacements within 

aperture 

Iteratively refine the flow vector 

Improvement: 

More accurate flow estimates 

J.Y. Bouguet, Tech. Report, Intel Cooperation, 2000




Motivation 



approximation 


aperture 


Improvement: 






Motivation 



approximation 


aperture 


Improvement: 






Motivation 



approximation 


aperture 


Improvement: 






Motivation 



approximation 


aperture 


Improvement: 






Motivation 



approximation 


aperture 


Improvement: 






Derivation 


Refined IBCC: 

A(p + u(p)) = B(p) 

A(p + u (κ−1) (p) + t (κ) (p)) = B(p) 

Iterations 1 ≤ κ ≤ k 

Recursion: u (κ) from u (κ−1) and t (κ) 

Initial estimate: u 0 = 0 

Sketch 





Derivation 


 

(κ) 

e t (κ) 

= 

w 

x∈Ω 

2 ⎛ ⎛ 

(x) ⎝A ⎝x + u (κ−1) 

+t 

 

η 

(κ) 

⎞ ⎞2 

⎠ − B(x) ⎠ 

e(t) = 

w 2 (A (η + t) − B) 2 

x∈Ω 

Taylor-approximation of A(η + t) (omitting x): 

= 

w 

x∈Ω 

2 

⎛ 

 

⎜ 

tx 

⎜ 

⎝A(η) − B + [Ax(η), Ay(η)] 

 

ty 

At(η) 

⎞2 

⎟ 

⎠ 

! 

= 0




Derivation 


e(t) = 

w 

x∈Ω 

2 

 

2 tx 

At(η) + [Ax(η), Ay(η)] 

ty 

Partial derivatives: 

 

∂e 

, 

∂tx 

∂e 

⊤ = 2 

∂ty 

 

w 

x∈Ω 

2 

 

 

tx Ax(η) 

At(η) + [Ax(η), Ay(η)] 

ty Ay(η) 

Solution (with η = x + u (κ−1) ): 

 

 

 

〈Ax(η)Ax(η)〉 〈Ax(η)Ay(η)〉 tx 〈Ax(η)At(η)〉 

= − 

〈Ax(η)Ay(η)〉 〈Ay(η)Ay(η)〉 ty 〈Ay(η)At(η)〉




Simplified solution 

Solution 

 

 

 


= − 

〈Ax(η)Ay(η)〉 〈Ay(η)Ay(η)〉 ty 〈Ay(η)At(η)〉 

 

=G(η) 

=b(η) 

Problem 

G depends on η = x + u (κ−1) 

⇒ Recomputation of G −1 needed for each iteration 

A and B contain identical gradient information 

Compute Bx and By instead of Ax and Ay 






Solution 

 

 

 


= − 

〈Ax(η)Ay(η)〉 〈Ay(η)Ay(η)〉 ty 〈Ay(η)At(η)〉 

 

=G(η) 

=b(η) 


 

 

〈BxBx〉 〈BxBy〉 tx 〈BxAt(η)〉 

= − 

〈BxBy〉 〈ByBy〉 ty 〈ByAt(η)〉 

 

=G 

=b(η) 





Building blocks 

Refinement vector 

Image mismatch vector 

Temporal derivative 

Recursion 

b (κ) = − 

t (κ) = G −1 b (κ) 

A (κ) 

t = B(p) − A 

 

〈BxA (κ) 

t 〉, 〈ByA (κ) 

⊤ t 〉 

 

p + u (κ−1) 

u (κ) = u (κ−1) + t (κ)



Multiscale Lucas-Kanade algorithm 

Motivation 

Sketch Motivation and approach 

Improvement 

Applicable for large image displacements 

IBCC and Taylor only valid for 

small movements 

Larger displacements are common 

Multi-scale approach 

Coarse-to-fine propagation 

Identical aperture sizes 

but shifted center position 





Motivation 


Improvement 













Motivation 


Improvement 













Motivation 


Improvement 













Motivation 


Improvement 













Motivation 


Improvement 













Motivation 


Improvement 













Gaussian image pyramid 

Theory 

Pyramidal images: I (κ) 

Pixel positions: p (κ) = 1 

2 κ p 

coarse 

fine 

λ = ℓ − 1 

λ = 2 

λ = 1 

λ = 0




Gaussian image pyramid 

Theory 

Pyramidal images: I (κ) 

Pixel positions: p (κ) = 1 

2 κ p 

coarse 

fine 

λ = ℓ − 1 

λ = 2 

λ = 1 

λ = 0 

Example 

fine ↔ coarse 

http://www.prip.tuwien.ac.at/research/research-areas/image-pyramids/




Algorithm 

Input: 

Two image pyramides A (λ) and B (λ) , 0 ≤ λ ≤ ℓ − 1 

Coarse-to-fine recursion (at level λ) 

Flow estimate u (λ) is given (initial estimate: u (ℓ−1) = 0) 

Compute refinement vector t (λ) : 

 

e t (λ) 

= 

x∈Ω 

Propagation to finer level λ − 1: 

 

2 

w(x) A x + u (λ) + t (λ) 

2 − B(x) 

u (λ−1) 

= 2 u (λ) + t (λ) 




Tolerance against illumination changes 

Motivation 

Sketch 

Improvement 

Tolerance against 

illumination changes 

IBCC 

Refined IBCC 

A(p + u(p)) = B(p) 

A(p + u(p)) = α(p)B(p) + β(p) 

Brigthness change model 

Linear gray value changes 

α and β locally constant 

Compute t = [ux, uy, α, β] ⊤ 

S. Negahdaripour, IEEE PAMI, 1998; H.W. Haussecker & D.J. Fleet, Proc. CVPR, 2000; 

Y. Altunbasak et.al., IEEE TIP, 2003; Y.H. Kim et.al., Img. Vis. Comp., 2005




Motivation 

Sketch 

Improvement 



IBCC 

Refined IBCC 

A(p + u(p)) = B(p) 

A(p + u(p)) = α(p)B(p) + β(p) 










Motivation 

Sketch 

Improvement 



IBCC 

Refined IBCC 

A(p + u(p)) = B(p) 

A(p + u(p)) = α(p)B(p) + β(p) 










Motivation 

Sketch 

Improvement 



IBCC 

Refined IBCC 

A(p + u(p)) = B(p) 

A(p + u(p)) = α(p)B(p) + β(p) 










Motivation 

Sketch 

Improvement 



IBCC 

Refined IBCC 

A(p + u(p)) = B(p) 

A(p + u(p)) = α(p)B(p) + β(p) 










Derivation 

Error function (omitting x) 

e(t) = 

w 2 

 

[Ax, Ay] ⊤ 

 

x∈Ω 

 

ux 

uy 

+ A − αB − β 

2 

S. Negahdaripour, IEEE PAMI, 1998




Derivation 

Partial derivatives (omitting x) 

∂e 

= 2 

∂ux 

 

w 

x∈Ω 

2 

 

[Ax, Ay] ⊤ 

 

 

ux 

+ A − αB − β Ax 

uy 

∂e 

= 2 

∂uy 

 

w 

x∈Ω 

2 

 

[Ax, Ay] ⊤ 

 

 

ux 

+ A − αB − β Ay 

uy 

∂e 

= −2 w 

∂α 

x∈Ω 

2 

 

[Ax, Ay] ⊤ 

 

 

ux 

+ A − αB − β B 

uy 

∂e 

= −2 w 

∂β 

x∈Ω 

2 

 

[Ax, Ay] ⊤ 

 

 

ux 

+ A − αB − β 

uy 





Solution 

Error function (omitting x) 

e(t) = 

w 2 

 

[Ax, Ay] ⊤ 

 

Solution 

x∈Ω 

 

ux 

uy 

+ A − αB − β 

t = G −1 b 

⎛ ⎞ 

ux 

⎜ ⎟ 

⎜uy 

⎟ 

⎜ ⎟ 

⎝ α ⎠ 

β 

= 

⎛ 

〈AxAx〉 

⎜ 〈AxAy〉 

⎜ 

⎝〈−AxB〉 

〈AxAy〉 

〈AyAy〉 

〈−AyB〉 

〈−AxB〉 

〈−AyB〉 

〈BB〉 

⎞−1 

〈−Ax〉 

〈−Ay〉 

⎟ 

〈B〉 ⎠ 

〈−Ax〉 〈−Ay〉 〈B〉 〈−1〉 

⎛ ⎞ 

〈−AxA〉 

⎜ 

⎜〈−AyA〉 

⎟ 

⎜ ⎟ 

⎝ 〈AB〉 ⎠ 

〈−A〉 

2 





Discussion 

Properties of G 

Full rank → invertible 

Ill-conditioned 

Areas of weak/regular texture 

Multiple interpretations of motion and illuminance 

Radiometric cues α and β 

Discontinuous at motion boundaries or occlusions 

⇒ errouneous estimates 

⇒ large residuals 

Physical interpretations possible: 

e.g. irradiance, surface normals, reflectance 

S. Negahdaripour, IEEE PAMI, 1998; H.W. Haussecker & D.J. Fleet, Proc. CVPR, 2000



Movement constraints 

Motivation 

Motivation 

Known camera motion 

Approach 

Known object motion 

Flowfield “patterns” 

Parameterize the flowfields: 

Solve for a 

u(x) = f(x, a) 


More accurate flow fields 

Uses available knowledge 

about flow fields 

Derive motion parameters 

Flowfield segmentation




Motivation 

Motivation 

Known camera motion 

Approach 

Known object motion 

Flowfield “patterns” 

Parameterize the flowfields: 

Solve for a 

u(x) = f(x, a) 

Example: motion segmentation 

http://www-bcs.mit.edu/people/jyawang/demos/garden-layer/




Motion models 

Translational model 

Affine model 

 

ux a1 

= 

uy a2 

 

 

ux a1x + a2y + a3 

= 

uy a4x + a5y + a6 

Y. Altunbasak et.al., IEEE TIP, 2003




Solution 

Refined image brightness constancy constraint 

A(p + f(p, a)) = B(p) 

Example: Taylor approximation for affine model 

Properties and discussion 

A(p) + Ax(p)(a1px + a2py + a3) 

No interesting properties mentioned 

+Ay(p)(a4px + a5py + a6) = B(p) 

Y. Altunbasak et.al., IEEE TIP, 2003



From local to global optical flow 

Local vs. global optical flow 

Local optical flow 

“Lucas-Kanade” 

Sparse flow fields 

Analytical solution 

Parsimonious 

Robust against noise 

Global optical flow 

“Horn-Schunck” 

Dense flow fields (fill-in) 

Iterative solution 

Computationally cheap 

Less robust 

B.D. Lucas & T. Kanade, Proc. Im. Underst. Workshop, 1981; B.K.P. Horn & B.G. Schunck, AI, 1981; 

A. Bruhn et.al., Proc. DAGM, 2002; A. Bruhn et.al., ĲCV, 2005




Horn-Schunck in a nutshell 

Global error functional (omitting x) 

⎛ 

e(u) = ⎜ 

⎝ 

x∈Γ 

(Axux + Ayuy + At) 2 

 

+ϕ |∇ux| 

 

IBCC 

2 + |∇uy| 2 

⎟ 

⎠ 

 

Divergence 

Smoothness weight ϕ > 0, Γ is the whole image 

Solution 

Compute optimal u which minimizes e(u) 

1 Euler-Lagrange mechanism ⇒ system of equations 

2 Numerical solution (Gauss-Seidel, SOR, . . .) 

A. Bruhn et.al., Proc. DAGM, 2002; A. Bruhn et.al., ĲCV, 2005 

⎞




Lucas-Kanade meets Horn-Schunck 

Horn-Schunck error functional 

e(u) = 

x∈Γ 

Combined error functional 

 

(Axux + Ayuy + At) 2 

+ ϕ |∇ux| 2 + |∇uy| 2 

e(u) = 

⎛ 

⎝ 

x∈Γ 

 

x ′ w 

∈Ω 

2 

2 ux 

[Ax, Ay] + At + . . . 

uy 

 

ϕ |∇ux| 2 + |∇uy| 2 





Lucas-Kanade meets Horn-Schunck 

Combined error functional 

Euler-Lagrange solutions 

e(u) = eLKHS(u) + ϕeR(u) 

0 = ∆ux − 1 

ϕ (〈AxAx〉ux + 〈AxAy〉uy + 〈AxAt〉) 

0 = ∆uy − 1 

ϕ (〈AxAy〉ux + 〈AyAy〉uy + 〈AyAt〉) 

Laplaceans ∆ux, ∆uy 

Discrete approximation 

Standard solvers for sparse sets of linear equations (SOR) 



Discussion 

Summary 

Limitations and possible improvements 

Standard LK 

Iterated LK 

Multiscale LK 

baseline 

most efficient 

more 

accurate 

large 

displacements 


illumination 

changes 


Global LK 

types of 

flowfields 

global flow 

computation


Discussion 

Summary 

Results?!? 

Unifying review 

Does not exist 

Reviewing myself 

Different data sets 

Unclear parameters 

Missing visualizations for parameters 

From some papers only a small subset was presented 

Quality of figures 

Benchmarking myself 

Availability of implementations 

Costs and benefits?


Discussion 

Outlook 

Relevance for my work 

Standard LK 

Iterated LK 

Multiscale LK 

tested 

12 ◦ to 15 ◦ 

tested 

8 ◦ to 11 ◦ 

tested 

5 ◦ to 8 ◦ 


TODO 


Global LK 

motion seg.? 

parameters? 

role of fill-in 

effects? 

A. Vardy & R. Möller, Conn. Sci., 2005; L. Gerstmayr, Tech. Report, Uni Bielefeld, 2007

Improvements of the Lucas-Kanade Optical-Flow Algorithm

Create successful ePaper yourself

Delete template?

Save as template?