Stereoscopic 3D Video Depth Adjustment for Viewer ... - scien

scien.stanford.edu

Stereoscopic 3D Video Depth Adjustment for Viewer ... - scien

L A B O R A T O R I E S O F A M E R I C A

3D Imaging Workshop 2011

Stereoscopic 3D Video Depth Adjustment

for Viewer Preference and Prevention of

Visual Discomfort

Hao Pan, Chang Yuan, Scott Daly

Advanced Video and Display Technologies

Sharp Labs of America

Camas, Washington

http://www.sharplabs.com

1


L A B O R A T O R I E S O F A M E R I C A

Motivations

Motivation 1: Viewers’ preferences (Increased depth gives higher quality)

Stage Style

(depth behind the display)

Uncrossed (positive)

disparity

3D Imaging Workshop 2011

Hologram Style

(depth into the room)

Crossed (negative)

disparity

Remote control

with 3D knob!

2


L A B O R A T O R I E S O F A M E R I C A

Motivations

Motivation 2: Prevention of visual comfort

Excessive disparity will cause eye fatigue

The 3D content providers cannot master digital depth rendering for all

display sizes, viewing distances, etc.

On-screen disparity = image disparity * pixel size

3D Imaging Workshop 2011 3


L A B O R A T O R I E S O F A M E R I C A

Disparity scaling => scaled depth range

3D Depth Adjustment: Scaling and Shifting

d'

( x,

y)

= d(

x,

y)

+

Disparity shifting => slightly distorted depth range δ

Behind the screen

Screen Depth

In front of screen

z

Expanded

Depth

Compressed

Depth

Original Depth

Range

d' ( x,

y)

= αd(

x,

y)

Forward Shifted

Depth

Backward Shifted

Depth

Ref.: Woods et al., “Image Distortions in Stereoscopic Video Systems,” SD&A 1993 (assuming parallel camera)

3D Imaging Workshop 2011 4


L A B O R A T O R I E S O F A M E R I C A

The Overall Block-diagram

New image pairs:

“original L + synthesized R” or

“synthesized L + original R”

3D Imaging Workshop 2011 5


L A B O R A T O R I E S O F A M E R I C A

1. HVS 3D Model (1)

HVS: Human Visual System

Ref: Hoffman et al., “Vergence-accommodation conflicts hinder visual performance

and cause visual fatigue,” Journal of Vision, 8(3):33, 1-30 (2008).

3D Imaging Workshop 2011 6


L A B O R A T O R I E S O F A M E R I C A

Empirical human visual model:

Percival’s Zone of Comfort

The 3D area which human

eyes can see clearly and

comfortably

Image level prediction

Check if the disparities in the

whole image fall into the

comfort zone and decide

whether to shift or scale the

images

3D Imaging Workshop 2011

1. HVS 3D Model (2)

DEPTH POSITION (m)

CLOSE DISTANCE

FAR

OF DISPLAY

SURFACE

Applied in this approach

Pixel level prediction

Check if a certain area in the

image is comfortable and then

replace these areas with more

comfortable versions

More details in a paper under

review cyan (teal) => blue => red:

increasing discomfort

7

COMFORT RANGE

# OF PIXELS


L A B O R A T O R I E S O F A M E R I C A

1. HVS 3D Model (3)

Vergence: eyes’ fixation point; Accommodation: eyes’ “focal length”; Exotropia: normal vergence at infinity

3D Imaging Workshop 2011 8

8


L A B O R A T O R I E S O F A M E R I C A

1. HVS 3D Model (4)

The comfortable disparity range is affected by display size, viewing distance, …

“Soft” boundary

Temporal variations (“depth budget”) to be considered later

3D Imaging Workshop 2011 9


L A B O R A T O R I E S O F A M E R I C A

2. Disparity Estimation (1)

3D Imaging Workshop 2011 10


L A B O R A T O R I E S O F A M E R I C A

2. Disparity Estimation (2)

One of the oldest unsolved computer vision problems

State-of-the-art algorithms generate disparity maps with holes

Causes of ambiguities in disparity estimation:

1. Occlusions: matched regions do not exist

2. Feature-less regions: matched regions are difficult to determine.

3. Regions with repetitive features: matched regions are difficult to determine.

4. Small objects with large disparities

5. Variations on intensity and colors in two views

3D Imaging Workshop 2011

2 3

1

1

4 4

11


L A B O R A T O R I E S O F A M E R I C A

Various methods: block matching, dynamic

programming, graph cut, …

Our solution: regularized block matching

algorithm for minimizing the cost function

2. Disparity Estimation (3)

Σ {Color_difference} + Σ {Disparity_difference}

A weighting mechanism is used to select pixels

in local windows

Similar to the bilateral filter: the weight is

bigger when the color difference between a

pixel and the center is smaller.

Benefits:

3D Imaging Workshop 2011

Can use bigger window (resolving

ambiguity 2, 3, 4)

Faster converging speed

“Disparity_difference” enforces the smoothness

of disparity map

Left View Right View

12


L A B O R A T O R I E S O F A M E R I C A

Iterative multiresolution

algorithm:

3D Imaging Workshop 2011

2. Disparity Estimation (4)

225 Set

window size

227 Set

window size

229 Set

search range

231 Set

search step

L image

201 Lowpass

filtering with L n

203 Spatial downsampling

by m n

207 Cost function

optimization

209 Spatial upsampling

the DM by

m n

DM n

No

0

n=n-1

n=0?

Yes

R image

DM 0 (LtoR or RtoL

maps) & matching

errors

201 Lowpass

filtering with L n

203 Spatial downsampling

by m n

205 Prediction from

previous DM

221 Set lowpass

parameter L n

223 Set downsampling

factor

m n

Other improvement: adaptive search range that changes from layer to layer, and from pixel to pixel

13


L A B O R A T O R I E S O F A M E R I C A

3. Occlusion Detection (1)

3D Imaging Workshop 2011 14


L A B O R A T O R I E S O F A M E R I C A

Occlusion detection is important

Many artifacts in a synthesized

view can be traced back to

improper handling of occlusion

Separation of foreground and

background objects

Detection using one-side disparity

map and the original matching cost

value is not reliable

Higher matching cost =>

occlusion: Not always!

Occlusion => higher matching

error: Not always!

3D Imaging Workshop 2011

3. Occlusion Detection (2)

15


L A B O R A T O R I E S O F A M E R I C A

3. Occlusion Detection (3)

Improvement: Detection using horizontal scan lines in the two-side disparity map

Detect occlusion in L image using rising edges in R disparity map: position, height, …

Multiple cues are used

3D Imaging Workshop 2011 16


L A B O R A T O R I E S O F A M E R I C A

3D Imaging Workshop 2011

Disparity adjustment & New-view synthesis

Disparity adjustment generates the disparity map for the new view by scaling the L2R

and R2L disparity maps

New-view synthesis fetches the proper pixels from L and R image (more details in a

paper under review)

Any holes are filled in the disparity maps

17


L A B O R A T O R I E S O F A M E R I C A

“Monsters vs Aliens 3D”: Left View

3D Imaging Workshop 2011 18


L A B O R A T O R I E S O F A M E R I C A

“Monsters vs Aliens 3D”: Right View

3D Imaging Workshop 2011 19


L A B O R A T O R I E S O F A M E R I C A

Synthesized View without Occlusion Detection

3D Imaging Workshop 2011 20


L A B O R A T O R I E S O F A M E R I C A

Synthesized View with Occlusion Detection

3D Imaging Workshop 2011 21


L A B O R A T O R I E S O F A M E R I C A

Under-the-hood Analysis of “Monster” Image

3D Imaging Workshop 2011 22


L A B O R A T O R I E S O F A M E R I C A

Anaglyph 3D Images Demo

Now view the stereo 3D images in anaglyph format

The time-multiplex version (120Hz + active shutter glasses) of

these images and more 3D images/videos will be shown in the

interactive session this afternoon!

Warning: possibly excessive disparities!

3D Imaging Workshop 2011

23


L A B O R A T O R I E S O F A M E R I C A

Summary & discussions

Described a 3D depth adjustment system

a HVS 3D model to guide the depth adjustment

a multi-resolution disparity estimation algorithm

a novel occlusion detection algorithm based on bidirectional disparity maps

Discussion

3D Imaging Workshop 2011

The performance is content dependent

How to hide artifacts.

3D is easier to hide or more difficult to hide?

The dominant eye effect.

Human factors; visual comfort model

24


L A B O R A T O R I E S O F A M E R I C A

Thank you!

Q&A

We have several full-time and intern job openings. Visit our website

(http://www.sharplabs.com) for more information (click on “Careers”).

Display algorithm researcher, video coding algorithm researcher, summer interns

Our EI/SD&A 2011 paper is shared at: http://cyuan.org/ei2011.pdf

This presentation is shared at: http://cyuan.org/3dworkshop2011.pdf

3D Imaging Workshop 2011 25

More magazines by this user
Similar magazines