Compute certainty map from correlations input depth map certainty ...

Compute certainty map from correlations
input depth map certainty map
1

scanline
Left Right
SSD

scanline
Left Right
Norm. corr

• If necessary, rectify the two stereo images to transform epipolar lines
into scanlines
• For each pixel x in the first image
Find corresponding epipolar scanline in the right image
Examine all pixels on the scanline and pick the best match x’
Compute disparity x-x’ and set depth(x) = B*f/(x-x’)

Textureless surfaces
Non-Lambertian surfaces, specularities
Occlusions, repetition

• The similarity constraint is local (each reference
window is matched independently)
• Need to enforce non-local correspondence
constraints

• Uniqueness
For any point in one image, there should be at
most one matching point in the other image

Usually, order of points in two images is
same.
Is this always true?
If we match pixel i in image 1 to pixel j in
image 2, no matches that follow will affect
which are the best preceding matches.
Example with pixels (a la Cox et al.).
8

• Uniqueness
For any point in one image, there should be at most
one matching point in the other image
• Ordering
Corresponding points should be in the same order in
both views
Ordering constraint doesn’t hold

Smoothness: disparity usually doesn’t change too
quickly.
Unfortunately, this makes the problem 2D again.
Solved with a host of graph algorithms, Markov Random
Fields, Belief Propagation, ….
Uniqueness constraint (each feature can at most
have one match)
Occlusion and disparity are connected.
10

• Try to coherently match pixels on the entire scanline
• Different scanlines are still optimized independently
Left image Right image

q
t
s
p
Can be implemented with dynamic programming
Ohta & Kanade ’85, Cox et al. ‘96
Left
occlusion
Right
occlusion
Left image
Right image
Slide credit: Y. Boykov

Find the minimum-cost path going monotonically
down and right from the top-left corner of the
graph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).
• Arcs = matched intervals along the epipolar lines.
• Arc cost = discrepancy between intervals.
13

Find the minimum-cost path going monotonically
down and right from the top-left corner of the
graph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).
• Arcs = matched intervals along the epipolar lines.
• Arc cost = discrepancy between intervals.
CS 223b
14

• Scanline stereo generates streaking artifacts
• Can’t use dynamic programming to find spatially
coherent disparities/ correspondences on a 2D grid

Regions without texture
Highly Specular surfaces
Translucent objects
17

What if the scanlines are not aligned ?
Given general displacement how to warp the views
Such that epipolar lines are parallel to each other
How to warp it back to canonical configuration
(more details later)
(Seitz)

• Rectified Image Pair
• Corresponding epipolar lines are aligned with the scan-lines
• Search for dense correspondence is a 1D search
19

Basic Equations
Epipolar Geometry
Image Rectification
Reconstruction
Correspondence
Active Range Imaging Technology
Dense and Layered Stereo
Smoothing With Markov Random Fields
21

Space-time stereo scanner
uses unstructured light to aid
in correspondence
Result: Dense 3D mesh (noisy)
22

ectified
23

• Project “structured” light patterns onto the object
Simplifies the correspondence problem
Allows us to use only one camera
projector
camera
L. Zhang, B. Curless, and S. M. Seitz.
Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic
Programming. 3DPVT 2002

L. Zhang, B. Curless, and S. M. Seitz.
Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic
Programming. 3DPVT 2002

http://bbzippo.wordpress.com/2010/11/28/kinect-in-infrared/

Many slides adapted from S. Seitz

• Generic problem formulation: given several images of
the same object or scene, compute a representation of
its 3D shape

The third view can be used for verification

Goal: Assign RGB values to voxels in V
photo-consistent with images

Space Carving
Voxel based methods
Silhoute based methods
31

http://www.cs.washington.edu/homes/furukawa/
gallery/
Yasutaka Furukawa and Jean Ponce,
Accurate, Dense, and Robust Multi-View Stereopsis, CVPR 2007.

YouTube video, high-quality video
Yasutaka Furukawa, Brian Curless, Steven M. Seitz and Richard Szeliski,
Towards Internet-scale Multi-view Stereo,CVPR 2010.