What Energy Functions Can be Minimized via Graph Cuts

What Energy Functions Can
Be Minimized via Graph Cuts
Yu Miao
2011.5.10

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

Image Denoising
Pairwise MRF
Noisy Image, I
Higher order MRF
A Simple Example
I 4
MAP-MRF
I 5
I 6
I 7
I 8
I 9
I 10
I 11
I 12
How to minimize
this energy
function ?
Likelihood(Data term) Prior(Smooth term)
Complex prior
structure may lead
to better results
Potts
model

Image Denoising
Noisy Image
The Applications of Graph Cuts
Stereo Correspondence
Pairwise MRF Higher order MRF
L R Disparity map
Stereo Pair

Image Synthesis
The Applications of Graph Cuts
Image Segmentation
…
+
Multi-View Reconstruction

• What?
• Find what kind of energy functions can be minimized via graph
cuts
• How?
Motivations
• Give a general-purpose graph construction for minimizing this
kind of energy functions

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

Pseudo-Boolean Function

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

v 1
5
2
Source
1
2
Sink
The st-Mincut Problem
9
v 2
4
Graph (V, E, C)
Vertices V = {v 1, v 2 ... v n}
Edges E = {(v 1, v 2) ....}
Costs C = {c (1, 2) ....}

v 1
5
2
Source
1
2
Sink
The st-Mincut Problem
9
v 2
4
What is a st-cut?

v 1
5
2
Source
1
2
Sink
The st-Mincut Problem
9
v 2
4
5 + 2 + 9 = 16
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
What is the cost of a st-cut?
Sum of cost of all edges
going from S to T

v 1
5
2
Source
1
2
Sink
The st-Mincut Problem
9
v 2
4
2 + 1 + 4 = 7
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
What is the cost of a st-cut?
Sum of cost of all edges
going from S to T
What is the st-mincut?
st-cut with the
minimum cost

The st-Mincut Problem
一个S,T割所对应的能量函数可表示为Pseudo-Boolean
function

St-mincut and Energy Minimization
S st-mincut
T
E: {0,1} n → R
Minimizing a Pseudo-
Boolean function E(x)
Functions of boolean variables Pseudo-Boolean?
E(x) = ∑ c i x i + ∑ c ij (1-x i)x j
i
i,j
c ij≥0
Polynomial time st-mincut algorithms
require non-negative edge weights

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

Energy
Move Making Algorithms
Solution Space

Energy
Computing the Optimal Move
Solution Space
Current Solution
Search
Neighbourhood
Optimal Move

Energy
Bigger move
space
Move Making Algorithms
x c
Current Solution
Search
Neighbourhood
Optimal Move
(t) Key Property
Solution Space
• Better solutions
Move Space
• Finding the optimal move hard

Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy
Move Space (t) : 2 N
Space of Solutions (x) : L N
Moves using Graph Cuts
Current Solution
Search Neighbourhood
N Number of
Variables
L Number of
Labels

Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy
Move to new
solution
Moves using Graph Cuts
Current Solution
Construct a move
function
Minimize move function
to get optimal move
How to
minimize move
functions?

Expansion Move
• Variables take label a or retain current label
[Boykov, Veksler, Zabih]

Status: Initialize Expand Ground House Sky with Tree
Expansion Move
• Variables take label a or retain current label
Tree
Ground
House
Sky
[Boykov, Veksler, Zabih]

Expansion Move
• Variables take label a or retain current label
这样就使问题转换为求解
Pseudo-Boolean Optimization的问题
这是我们选择
Pseudo-Boolean能量
函数的原因
• It is important to note that this sub-problem is an
energy minimization problem over binary variables.
[Boykov, Veksler, Zabih]

Expansion Move
这就是所谓的Graph-Representable问题

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

现在的问题是什么样
的能量函数是graphrepresentable,以
及如何构造graph.
Graph-Representable
Pseudo - Boolean
Energy Function E
E(X)
=
Cuts
const + minimum s-t cut(corresponds to
the configuration X’)
在一次α-expansion中(即
在求E(X)时),const不变。
在不同的α-expansion时,
const不相同
这样在用max-flow算法求st-Mincut时,所得到的X,就是使E(X)取最小值的X。且他们的值
之间有上述关系。这就使得最小化Pseudo-Boolean Optimizing E(X)的问题转化成了求最小
割的问题。从而可用高效的max-flow算法进行求解。

Graph-Representable
先考虑第一个问题,什么样的函数可以graph-representable?
Definition of regular
?
?

Graph-Representable
Theorem 3.7 其充要条件如何证明?对于其必要性,kolmogorov给出了对任意
Pseudo-Boolean Functions的证明。此处给出其结论(证明略)
该定理指出,对于任意的Pseudo-Boolean Functions,只有当它是Regular时,才有
可能是Graph-Representable
Regular和
Graph-Representable的Pseudo-Boolean Functions必定是Regular的
Submodular等价

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

在构造之前,先给出一个非常重要的定理:Additivity
?

Theorem 3.7 的充分性的证明采用的是构造性的证明方法。该方法也是一个
general-purpose graph construction for minimizing any function in this class.
是 的子集,所以只要给出 的构造方法就可。但是由于 的构造相对简单,
所以还是先讨论 的构造方法。
在构造之前,先给出一个非常重要的定理:Additivity
证明略
根据该定理,只要能证明函数的每一项都是graph-representable,则整个函数就
是graph-representable.
现在要证明的是,若 是regular,则其为graph-representable。
下面即针对 和 分别构造其Graph
前面所提到的第
二个问题:如何
构造Graph?

The graph will contain n+2 vertices: Each nonterminal vertex
will encode the binary variable
验证该图是否
满足graphrepresentable
的定义

x i
For E i,j (x i,x j)
A = E ij (0,0) B = E ij (0,1) C = E ij (1,0) D = E ij (1,1)
0
1
x j
0 1
A B
C D
= A +
E ij (x i,x j) = E ij (0,0)
0
1
0 1
0 0
C-A C-A
if x i=1
add C-A
1
0 1
0 0 D-C 0
+ +
0 D-C
if x j=1
add D-C
+ (E ij (1,0)-E ij (0,0)) x i + (E ij (1,1)-E ij (1,0)) x j
1
0 1
0
B+C
-A-D
0 0
+ (E ij (0,1) + E ij (1,0) - E ij (0,0) - E ij (1,1)) (1-x i) x j
B+C-A-D 0 is true from the regularity of E ij

For E i,j (x i,x j)
x i
A = E ij (0,0) B = E ij (0,1) C = E ij (1,0) D = E ij (1,1)
0
1
x j
0 1
A B
C D
= A +
E ij (x i,x j) = E ij (0,0)
0
1
0 1
0 0
C-A C-A
if x i=1
add C-A
1
0 1
0 0 D-C 0
+ +
0 D-C
if x j=1
add D-C
+ (E ij (1,0)-E ij (0,0)) x i + (E ij (1,1)-E ij (1,0)) x j
1
0 1
0
B+C
-A-D
0 0
+ (E ij (0,1) + E ij (1,0) - E ij (0,0) - E ij (1,1)) (1-x i) x j
B+C-A-D 0 is true from the regularity of E ij

For E i,j (x i,x j)
x i
A = E ij (0,0) B = E ij (0,1) C = E ij (1,0) D = E ij (1,1)
0
1
x j
0 1
A B
C D
= A +
E ij (x i,x j) = E ij (0,0)
0
1
0 1
0 0
C-A C-A
if x i=1
add C-A
1
0 1
0 0 D-C 0
+ +
0 D-C
if x j=1
add D-C
+ (E ij (1,0)-E ij (0,0)) x i + (E ij (1,1)-E ij (1,0)) x j
1
0 1
0
B+C
-A-D
0 0
+ (E ij (0,1) + E ij (1,0) - E ij (0,0) - E ij (1,1)) (1-x i) x j
B+C-A-D 0 is true from the regularity of E ij

For E i,j (x i,x j)
x i
A = E ij (0,0) B = E ij (0,1) C = E ij (1,0) D = E ij (1,1)
0
1
x j
0 1
A B
C D
= A +
E ij (x i,x j) = E ij (0,0)
0
1
0 1
0 0
C-A C-A
if x i=1
add C-A
1
0 1
0 0 D-C 0
+ +
0 D-C
if x j=1
add D-C
+ (E ij (1,0)-E ij (0,0)) x i + (E ij (1,1)-E ij (1,0)) x j
1
0 1
0
B+C
-A-D
0 0
+ (E ij (0,1) + E ij (1,0) - E ij (0,0) - E ij (1,1)) (1-x i) x j
B+C-A-D 0 is true from the regularity of E ij

For E i,j (x i,x j)
x i
A = E ij (0,0) B = E ij (0,1) C = E ij (1,0) D = E ij (1,1)
0
1
x j
0 1
A B
C D
= A +
E ij (x i,x j) = E ij (0,0)
0
1
0 1
0 0
C-A C-A
if x i=1
add C-A
1
0 1
0 0 D-C 0
+ +
0 D-C
if x j=1
add D-C
+ (E ij (1,0)-E ij (0,0)) x i + (E ij (1,1)-E ij (1,0)) x j
1
0 1
0
B+C
-A-D
0 0
+ (E ij (0,1) + E ij (1,0) - E ij (0,0) - E ij (1,1)) (1-x i) x j
B+C-A-D 0 is true from the regularity of E ij

对 而言,E是regular等价于E i,j 是regular
而对 而言,E是regular并不意味着E中的每一项(each E i,j E i,j,k )是regular
由Additivity theorem及前面 的结论,我们仅需对满足regular要求的E i,j,k 构造其
Graph即可证明Theorem 3.7的充分性

Const x i =1 x j =1 x k =1 (x j=0,x k=1) (x i=1,x k=0) (x i=0,x j=1) (x i=1, x j=1, x k=1)
Where P 1= F-B, P 2= G-E, P 3= D-C,
P 23= B+C-A-D, P 31= B+E-A-F, P 12= C+E-A-G
Only the last
term will be
considered
E i,j,k is regular, therefore by considering projections E i,j,k [x 1=0], E i,j,k [x 2=0], E i,j,k [x 3=0]
we conclude that P 23, P 31, P 12 are non-negative.

(x i=1, x j=1, x k=1)
If x i=x j=x k=1, the cost
of the minimum cut is
0;otherwise the
minimum cut is P

Summary of graph constructions
For a function E(x 1,…x n) we define
1.
2.

3.
Summary of graph constructions

Background
Outline of the Talk
Pseudo-Boolean Function
The st-Mincut Problem
Move Making Algorithms
Graph-Representable
A General-Purpose Graph Construction for F 3
Conclusions & Future Work

Conclusions

Future Work
Non-submodular Energy Functions
Mixed(Real-Integer) Problems
Higher Order Energy Functions

The “Graph” of Graph Cuts
Yuri Boykov,
The University of
Western Ontario
Ramin Zabih,
Cornell University
Vladimir Kolmogorov,
University College London
Endre Boros,
Rutgers University
Nikos Komodakis,
University of Crete,
INRIA-Saclay;
Ecole Centrale Paris
Philip Torr,
Oxford Brookes University
Carsten Rother,
Microsoft Research
Cambridge
Pushmeet Kohli,
Microsoft Research
Cambridge

Thanks. Questions?