Sampling, Filtering, Reconstruction

**Sampling**,**Filtering**,**Reconstruction**Image Manipulation andComputational PhotographyCS294-69 Fall 2011Maneesh Agrawala[Some slides from James Hays , Derek Hoiem,Alexei Efros and Fredo Durand]**Sampling** and **Reconstruction**1

UndersamplingWhat if we missed things between the samples?Simple example: undersampling a sine wave• unsurprising result: information is lost• surprising result: indistinguishable from lower frequency© 2006 Steve MarschnerUndersamplingWhat if we missed things between the samples?Simple example: undersampling a sine wave• unsurprising result: information is lost• surprising result: indistinguishable from lower frequency• also was always indistinguishable from higher frequencies• aliasing: signals traveling in disguise as other frequencies© 2006 Steve Marschner4

Aliasing in VideoSlide by Steve SeitzAliasing in images5

What’s Happening?Input signal:Plot as image:x = 0:.05:5; imagesc(sin((2.^x).*x))Alias!Not enough samplesAntialiasingWhat can we do about aliasing?Sample more often• Join the Mega-Pixel craze of the photo industry• But this cant go on foreverMake the signal less wiggly• Get rid of some high frequencies• Will loose information• But it looks better than aliasing6

Preventing AliasingIntroduce lowpass filters:• remove high frequencies leaving only safe, low frequencies• choose lowest frequency in reconstruction (disambiguate)© 2006 Steve MarschnerLinear **Filtering**: A Key IdeaTransformations on signals; e.g.:• bass/treble controls on stereo• blurring/sharpening operations in image editing• smoothing/noise reduction in trackingKey properties• linearity: filter(f + g) = filter(f) + filter(g)• shift invariance: behavior invariant to shifting the input– delaying an audio signal– sliding an image aroundCan be modeled mathematically by convolution© 2006 Steve Marschner7

Moving AverageIdea: define new function by averaging over sliding windowa simple example to start off: smoothing© 2006 Steve MarschnerWeighted Moving AverageCan add weights to our moving averageWeights […, 0, 1, 1, 1, 1, 1, 0, …] / 5© 2006 Steve Marschner8

Weighted Moving Averagebell curve (gaussian-like) weights […, 1, 4, 6, 4, 1, …]© 2006 Steve MarschnerMoving Average In 2DWhat are the weights H?0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 00 0 0 90 90 90 90 90 0 00 0 0 90 90 90 90 90 0 00 0 0 90 90 90 90 90 0 00 0 0 90 0 90 90 90 0 00 0 0 90 90 90 90 90 0 00 0 0 0 0 0 0 0 0 00 0 90 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0© 2006 Steve Marschner • 20Slide by Steve Seitz9

Cross-Correlation **Filtering**Equation with averaging window size (2k+1)x(2k+1):Allow different weights for different neighboring pixels:This is called a cross-correlation operation and written:H is called the filter,kernel, or mask.© 2006 Steve MarschnerSlide by Steve SeitzGaussian **Filtering**Gaussian kernel: less weight further from window center0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 00 0 0 90 90 90 90 90 0 00 0 0 90 90 90 90 90 0 00 0 0 90 90 90 90 90 0 00 0 0 90 0 90 90 90 0 00 0 0 90 90 90 90 90 0 00 0 0 0 0 0 0 0 0 00 0 90 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 01 2 12 4 21 2 1H[u,v] is an approximation of a continuousGaussian:Slide by Steve Seitz10

Mean vs. Gaussian filteringSlide by Steve SeitzConvolutionCross-correlation:A convolution operation is a cross-correlation where the filter is flippedboth horizontally and vertically before being applied to the image:It is written:Suppose H is a Gaussian or mean kernel. How does convolution differfrom cross-correlation?Slide by Steve Seitz11

Convolution is Nice!Notation:Convolution is a multiplication-like operation• commutative• associative• distributes over addition• scalars factor out• identity: unit impulse e = […, 0, 0, 1, 0, 0, …]Conceptually no distinction between filter and signalUsefulness of associativity• often apply several filters one after another: (((a * b 1 ) * b 2 ) * b 3 )• this is equivalent to applying one filter: a * (b 1 * b 2 * b 3 )© 2006 Steve Marschner12

Tricks with ConvolutionsUCB UCBUCB10203040=506010 20 30 40 50 6015

Yucky detailsWhat about near the image edge?• the filter window falls off the edge of the image• need to extrapolate• methods:– clip filter (black)– wrap around– copy edge– reflect across edge– vary filter near edge© 2006 Steve Marschner • 4316

Image Half-SizingThis image is too big tofit on the screen. Howcan we reduce it?How to generate a halfsizedversion?Sub-sampling1/81/4Throw away every other row andcolumn to create a 1/2 size image- called image sub-samplingSlide by Steve Seitz17

Sub-sampling1/21/4 (2x zoom) 1/8 (4x zoom)Aliasing! What do we do?Slide by Steve SeitzDownsamplingoriginalimagedirectlydownsampled18

Pre-filtering, **Reconstruction**, Resamplingoriginalimageconvolve withGaussianreconstructedsignaldownsampledreconstructionGaussian (lowpass) pre-filteringG 1/4G 1/8Gaussian 1/2Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?Slide by Steve Seitz19

Subsampling with Gaussian pre-filteringGaussian 1/2G 1/4 G 1/8Slide by Steve SeitzCompare with...1/21/4 (2x zoom) 1/8 (4x zoom)Slide by Steve Seitz20

Resampling to Enlargedownsampledreconstructionconvolve withbox filterreconstructedsignal??upsampledreconstruction??reconstructedimageGaussian (lowpass) pre-filteringG 1/8G 1/4Gaussian 1/2Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?Slide by Steve Seitz21

Jean Baptiste Joseph Fourier (1768-1830)Had crazy idea (1807):Any periodic function canbe rewritten as a weightedsum of sines and cosines ofdifferent frequencies.Don’t believe it?• Neither did Lagrange,Laplace, Poisson and otherbig wigs• Not translated into Englishuntil 1878!But its true!• called Fourier SeriesA Sum of SinesOur building block:Asin(ωx +φ)Add enough of them to getany signal f(x) you want!How many degrees offreedom?What does each control?Which one encodes thecoarse vs. fine structure ofthe signal?22

Fourier TransformTo understand frequency ω let’s reparametrize the signal by ω:f(x)FourierTransformF(ω)For every ω from 0 to inf, F(ω) holds the amplitude Aand phase φ of the corresponding sineHow can F hold both? Complex number trick!F ( ω)= R(ω)+ iI(ω)A = ± R ω + I2 2( ) ( ω)We can always go back:Asin(ωx +φ)− I(ω)φ = tan 1R(ω)F(ω)Inverse FourierTransformf(x)Time and Frequencyexampleg(t) = sin(2pf t) + (1/3)sin(2p(3f) t)23

Time and Frequencyexampleg(t) = sin(2pf t) + (1/3)sin(2p(3f) t)= +Frequency Spectraexampleg(t) = sin(2pf t) + (1/3)sin(2p(3f) t)= +24

Frequency SpectraUsually, frequency is more interesting than the phaseFrequency Spectra= +=25

Frequency Spectra= +=Frequency Spectra= +=26

Frequency Spectra= +=Frequency Spectra= +=27

Frequency Spectra=A∞∑k=11 sin(2 π kt )kFrequency Spectra28

Extension to 2Din Matlab, check out: imagesc(log(abs(fftshift(fft2(im)))));Man-made Scene29

Can Change Spectrum, then ReconstructLow and High Pass filtering30

The Convolution TheoremThe greatest thing since sliced (banana) bread!The Fourier transform of the convolution of two functions is theproduct of their Fourier transformsF[ g ∗h]= F[ g]F[h]The inverse Fourier transform of the product of two Fouriertransforms is the convolution of the two inverse Fourier transformsF−1[ gh]=F−1[ g]∗F−1[ h]Convolution in spatial domain is equivalent to multiplication infrequency domain!2D Convolution Theorem Examplef(x,y)|F(s x ,s y )|*h(x,y)|H(s x ,s y )|g(x,y)|G(s x ,s y )|31

Fourier Transform PairsLow-pass, Band-pass, High-pass filtersLow-pass:High-pass / band-pass:32

Edges in ImagesWhat Does Blurring Remove?original33

What Does Blurring Remove?smoothed (5x5 Gaussian)High-Pass Filtersmoothed – original34

Unsharp Masking100200300400500- =200 400 600 800+ α =Hybrid Images[Oliva, Torralba & Schyns 05] Assignment 0: Implement Hybrid Images 35

Hybrid Images[Oliva, Torralba & Schyns 05] Assignment 0: Implement Hybrid Images Image PyramidsKnown as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]• A precursor to wavelet transformSlide by Steve Seitz36

A bar in the bigimages is ahair on thezebras nose;in smallerimages, astripe; in thesmallest, theanimals nose Figure from David ForsythGaussian Pyramid Constructionfilter maskRepeat• Filter• SubsampleUntil minimum resolution reached• can specify desired number of levels (e.g., 3-level pyramid)The whole pyramid is only 4/3 the size of the original image!Slide by Steve Seitz37

Band-pass filteringGaussian Pyramid (low-pass images)Laplacian Pyramid (subband images)Created from Gaussian pyramid by subtractionLaplacian PyramidOriginalimageNeed this!How can we reconstruct (collapse) this pyramidinto the original image?38

Pyramid Based GraphicsLaplacian pyramid introduced in classic paper: [Burt & Adelson 83]Applications summarized in 84 paper• Image interpolation[Ogden, Adelson, Bergen & Burt 84] • Merging images into mosaics• Creating realistic shadows and shading• Generation of natural looking textures using fractals• Real-time animation of fractalsAssignment 0: Extra Credit Image GradientThe gradient of an image:The gradient points in the direction of most rapid change in intensityThe gradient direction is given by:• how does this relate to the direction of the edge?The edge strength is given by the gradient magnitude39

Effects of NoiseConsider a single row or column of the image• Plotting intensity as a function of position gives a signalThHow to compute a derivative?Where is the edge?Solution: Smooth FirstThWhere is the edge?Look for peaks in40

Derivative Theorem of ConvolutionThis saves us one operation:Laplacian of GaussianConsiderLaplacian of GaussianoperatorWhere is the edge?Zero-crossings of bottom graph41

2D Edge Detection FiltersLaplacian of GaussianGaussianderivative of Gaussianis the Laplacian operator:42