Colorization: History

Colorization

Colorization: History
• Hand tinting
– http://en.wikipedia.org/wiki/Hand-colouring

Colorization: History
• Film colorization
– http://en.wikipedia.org/wiki/Film_colorization
Colorization in 1986
Colorization in 2004

Overview
• Colorization by example
• Colorization using “scribbles”

Transferring Color to Greyscale Images
T. Welsh, M. Ashikhmin, and K. Mueller
SIGGRAPH 2002

The Basic Approach
• Convert source image to decorrelated lαβ color space
– l: luminance
– α, β: chromatic channels (yellow/blue and red/green)
• Perform luminance remapping (histogram matching)
• Take ~200 color samples from the source image
• For each pixel in the target image (in scanline order):
– Find best matching source pixel (compare luminance and std. dev.
of luminance values in neighborhood)
– Transfer color from source pixel to target pixel
source target result

Recall: Image Analogies

Problem
• Global procedure fails when corresponding
colors don’t have corresponding luminance
values
Source image Target image Colorized target
Grayscale source
image

Solution
• The user specifies corresponding swatches in
the source and target images

Colorization with swatches
Transfer between
swatches
Global transfer Extend to the
rest of image

Colorization with swatches: Details
• Transfer color from source to target swatches
– Perform luminance remapping between corresponding
swatches
– Take ~50 samples from each source swatch
• Extend colorized swatches to the rest of image
– For each grayscale pixel, find best matching pixel in a
colorized swatch in the target image
• Matching function is SSD of grayscale neighborhoods
– Transfer color from matching pixel to grayscale pixel

Example results

Example results

Example results: Scientific visualization

Video colorization
• First transfer color between swatches for a
single frame
• Use the colorized swatches in the single
frame to transfer color to the rest of the
sequence

Video colorization results

Video colorization results

Brain volume colorization

Discussion of implementation choices
• Effect of color space
• Sampling scheme for source pixels
• Matching function between source and
target pixels
• Additional constraints for search (i.e., spatial
coherence)
• Selection of sample image

Colorization Using Optimization
A. Levin, D. Lischinski, Y. Weiss
SIGGRAPH 2004
http://www.cs.huji.ac.il/~yweiss/Colorization/

Overview
Input: grayscale image with color
“scribbles”
Output: Colorized image

The Approach
• Two neighboring pixels r, s should have
similar colors if their intensities are similar
• The goal is to minimize the difference
between the color U(r) at pixel r and the
weighted average of the colors at
neighboring pixels

Objective function
sum over all pixels
color of pixel r
sum over pixels in
the neighborhood of r
affinity between
r and s
color of pixel s

Objective function
Possible affinity functions:
Neighborhood definition: for video, take optical flow into account
Constraints: color of user-specified pixels remains fixed
Optimization: sparse linear system

Results
• Colors from the original image used for the
“scribbles”
• Processing time: ~15sec/frame

Comparison to segmentation-based colorization
Segmented image Segmentation + flood fill
Colorization by optimization

Recoloring
Original image Mask and scribbles Final image

More recoloring
Original image Scribbles Final image

Progressive colorization

Video colorization
Grayscale video Input scribbles

Video colorization
Grayscale video Colorized video

Video colorization
Grayscale video Input scribbles

Video colorization
Grayscale video Colorized video

Video colorization
Grayscale video Input scribbles

Video colorization
Grayscale video Colorized video

Colorization by Example
R. Irony, D. Cohen-Or, and D. Lischinski
Eurographics Symposium on Rendering, 2005

Motivation
• Improve spatial consistency of examplebased
transfer methods such as Welsh et al.
(2002)
• Reduce the amount of manual supervision of
scribble-based methods such as Levin et al.
(2004)

The importance of spatial coherence
Source image Target image Image colorized by
method of Welsh et al.

The importance of spatial coherence
Source image Target image Proposed method

Overview of approach

Example result
Reference
(source) image
Automatic
segmentation
Target image Pixel classification
Smoothed pixel
classification Colorized target

Example result
Reference
(source) image
Target image
Manual
segmentation
Automatic
classification
Colorized target
image

Manual vs. automatic segmentation
Source image
Target image
Manual
segmentation
Classification based
on manual
segmentation
Automatic
segmentation
Classification based
on automatic
segmentation
Colorization

Colorizing multiple frames

Natural Image Colorization
L. Qing, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu,
H. Shum
Eurographics Symposium on Rendering, 2007

Motivation
• Reduce the amount of user interaction necessary
to produce complex, nuanced color images
• Handle highly textured images – non-adjacent
regions of similar texture
Colorization by optimization (Levin et al.)

Motivation
• Reduce the amount of user interaction necessary
to produce complex, nuanced color images
• Handle highly textured images – non-adjacent
regions of similar texture
Proposed method

Outline of method
1. The user draws strokes indicating regions that (roughly)
share the same color
2. Strokes are used for automatic texture segmentation
3. The user selects color for a few pixels in each region
4. Color is transferred automatically based on segmentation and
selected colors

Segmentation
• Iterative process: propagate labels to regions
similar in intensity and texture, but not
necessarily spatially contiguous

Color mapping
• Piecewise-linear interpolation of selected
colors inside each region
• Soft blending of colors around the region
boundaries

Comparison
Levin result 1 Levin result 2 Proposed method

Comparison
Proposed method
Levin et al.

More results

More results

More results

Difficult example

Closeup

Colorization: Summary
• Example-based methods
– Transferring color to grayscale images (Welsh et al. 2002)
• Shortcoming: spatial coherence
– Colorization by example (Irony et al. 2005)
• Spatially coherent texture segmentation
• Stroke-based methods
– Colorization using optimization (Levin et al. 2004)
• Shortcomings: color leaking, too many strokes required for
textured images
– Natural image colorization (Qing et al. 2007)
• Handle images with non-contiguous textures