Tutorial-ImageSegmentationGraph-cut1-Shi

Tutorial
Graph Based Image Segmentation
Jianbo Shi, David Martin, Charless Fowlkes, Eitan Sharon

Topics
• Computing segmentation with graph cuts
•
•
Segmentation benchmark, evaluation criteria
Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation

i
Wij
j
W ij
G = {V, E}
V: graph nodes
E: edges connection nodes
Image = { pixels }
Pixel similarity

Topics
• Computing segmentation with graph cuts
•
•
Segmentation benchmark, evaluation criteria
Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation

Topics
• Computing segmentation with graph cuts
•
•
Segmentation benchmark, evaluation criteria
Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation

Original Image
Texture
Brightnes
s
L*
Colo
r
a*
b*
Boundary Processing
Region
Processing
Textons
D
E
Proximity
C
B
A
χ 2
χ 2
W ij
C
A
B

Topics
• Computing segmentation with graph cuts
•
•
Segmentation benchmark, evaluation criteria
Image segmentation cues, and combination
• Muti-grid computation, and cue aggregation

Part I: Graph and Images
Jianbo Shi

Graph Based Image Segmentation
i
Wij
j
W ij
G = {V, E}
V: graph nodes
E: edges connection nodes
Image = { pixels }
Pixel similarity
Segmentation = Graph partition

Right partition cost function?
Efficient optimization algorithm?

Graph Terminology
adjacency matrix,
degree,
volume,
graph cuts

Graph Terminology
Similarity matrix S = [ Sij ]
i
Sij
j
is generalized adjacency matrix
1
20
10
0
20
40
60
80
0.9
0.8
0.7
0.6
0.5
0.4
!10
100
0.3
!20
!10 0 10 20 30 40
120
140 0
20 40 60 80 100 120 140
0.2
0.1

Graph Terminology
Degree of node:
d i = ∑ j
S ij
i
1
20
10
0
20
40
60
80
0.9
0.8
0.7
0.6
0.5
0.4
!10
100
0.3
!20
!10 0 10 20 30 40
120
140 0
20 40 60 80 100 120 140
0.2
0.1

Graph Terminology
Volume of set:
vol(A) = ∑ i∈A
d i , A ⊆ V
A
1
20
10
0
20
40
60
80
0.9
0.8
0.7
0.6
0.5
0.4
!10
100
0.3
!20
!10 0 10 20 30 40
120
140 0
20 40 60 80 100 120 140
0.2
0.1

Cuts in a graph
cut(A, Ā) =
∑
i∈A,j∈Ā
S i,j
1
20
10
0
20
40
60
80
0.9
0.8
0.7
0.6
0.5
0.4
!10
100
0.3
!20
!10 0 10 20 30 40
120
140 0
20 40 60 80 100 120 140
0.2
0.1

Graph Terminology
Similarity matrix S = [ Sij ]
i
Sij
j
Degree of node:
i
d i = ∑ j
S ij
Volume of set:
Graph Cuts
A

Useful Graph Algorithms
• Minimal Spanning Tree
• Shortest path
• s-t Max. graph flow, Min. cut

Minimal/Maximal Spanning Tree
Tree is a graph G without cycle
Graph
Maximal
Minimal

Kruskal’s algorithm
• sort the edges of G in increasing order by length
• for each edge e in sorted order
if the endpoints of e are disconnected in S
add e to S
Randonmalized version can compute Typical cuts

Leakage problem in MST
Leakage

Graph Cut and Flow
Source
Sink
1) Given a source (s) and a sink node (t)
2) Define Capacity on each edge, C_ij = W_ij
3) Find the maximum flow from s->t, satisfying the capacity constraints
Min. Cut = Max. Flow

Problem with min cuts
Min. cuts favors isolated clusters

Normalize cuts in a graph
• (edge) Ncut = balanced cut
Ncut(A, B) = cut(A, B)(
1
vol(A) + 1
vol(B) )
NP-Hard!

Representation
Partition matrix:
segments
X = [X 1 , ..., X K ]
pixels
Pair-wise similarity matrix W
Degree matrix D:
D(i, i) = ∑ j
W i,j
Laplacian matrix D-W

Graph weight matrix W
(a)
W
n=nr * nc
i1
(b)
nr
i2
nc
W(i1,j)
(c)
W(i2,j)
n=nr * nc
(d)

Laplacian matrix D-W
Let x = X(1,:) be the indicator of group 1
asso(A, A) =
x T W x
x T
W
1
x
20
0.9
0.8
40
0.7
60
0.6
0.5
80
0.4
100
120
0.3
0.2
0.1
140 0
20 40 60 80 100 120 140

Laplacian matrix D-W
x T Dx
Cut(A, V-A) = vol(A) -
x T W x
asso(A,A)
1
20
0.9
0.8
40
0.7
60
0.6
0.5
80
0.4
100
120
0.3
0.2
0.1
140 0
20 40 60 80 100 120 140
Cut(A, V − A) = x T (D − W )x

Ncut(X) = 1 K
K∑
l=1
cut(V l , V − V l )
vol(V l )
1
20
40
0.9
0.8
0.7
60
80
0.6
0.5
0.4
100
0.3
120
0.2
0.1
140 0
20 40 60 80 100 120 140
= 1 K
K∑
l=1
X T l (D − W )X l
X T l DX l
X ∈ {0, 1} N×K , X1 K = 1 N

Step I: Find Continuous Global Optima
Scaled partition matrix.
Z = X(X T DX) − 1 2
X =
⎡
⎢
⎣
1 0 0
1 0 0
0 1 0
0 1 0
0 0 1
⎤
⎥
⎦
Z =
⎡
⎢
⎣
√
1
0 0
vol(A)
√
1
0 0
vol(A)
1
0 √ 0
vol(B)
1
0 √ 0
vol(B)
1
0 0 √
vol(C)
⎤
⎥
⎦

Step I: Find Continuous Global Optima
Ncut
= 1 K
K∑
l=1
X T l (D − W )X l
X T l DX l
becomes
Ncut(Z) = 1 K tr(ZT W Z)
Eigensolutions
(D − W )z ∗ = λDz ∗
Z T DZ = I K
y 2 i
A
i
Z ∗ = [z ∗ 1, z ∗ 2, ..., z ∗ k]
y 2 i
A
i

Interpretation as a Dynamical System

Interpretation as a Dynamical System

Step I: Find Continuous Global Optima
X =
⎡
⎢
⎣
Partition
1 0 0
1 0 0
0 1 0
0 1 0
0 0 1
⎤
⎥
⎦
⎡
Z =
⎢
⎣
Scaled Partition
√
1
0 0
vol(A)
√
1
0 0
vol(A)
1
0 √ 0
vol(B)
1
0 √ 0
vol(B)
1
0 0 √
vol(C)
⎤
⎥
⎦
⎡
Z ∗ =
⎢
⎣
Eigenvector solution
√
1
0 0
vol(A)
√
1
0 0
vol(A)
1
0 √ 0
vol(B)
1
0 √ 0
vol(B)
1
0 0 √
vol(C)
⎤
⎥
⎦
[ cos(θ) -sin(θ)
×
sin(θ) cos(θ)
]
.
(D − W )Z ∗ = λDZ ∗
If Z* is an optimal, so is {ZR : R T R = I K }

Step II: Discretize Continuous Optima
Z2
Target
partition
X =
⎡
⎢
⎣
1 0
1 0
0 1
0 1
0 1
⎤
⎥
⎦
Z1
Rotation R
Eigenvector
solution
Z ∗ =
⎡
⎢
⎣
1 -1.4
1 -1.3
1 0.8
1 0.9
1 0.7
Rotation R can be found exactly in 2-way partition
⎤
⎥
⎦

Brightness Image
Segmentation

ightness image
segmentation

Part II:
Segmentation Measurement,
Benchmark