The Light Field Camera: Extended Depth of Field, Aliasing, and ...

More documents

Recommendations

Info

BISHOP AND FAVARO: THE LIGHT FIELD CAMERA: EXTENDED DEPTH OF FIELD, ALIASING, AND SUPERRESOLUTION 981h LIsði; uÞ ¼hMLkðiÞ ð qðiÞ;uÞh LkðiÞ ð qðiÞ;uÞ; ð24Þwhere kðiÞ and qðiÞ are given by (7). In a Lambertian scene,the image l captured by the light field camera is thenZlðiÞ ¼ h LIs ði; uÞrðuÞdu: ð25ÞWe define the microlens point spread function h LkðiÞconsidering the main lens diameter to be infinite. This gives8< 1h LkðiÞ ð qðiÞ;uÞ ¼ b 2 qðiÞ ðuÞðc kðiÞ uÞ ðuÞ2 z0 fz 0 fand negative otherwise.7.1.2 Main Lens VignettingAs seen in Fig. 4 a microlens may only be partially hit by themain lens blur disc, which results in clipped microlens PSF;this effect is modeled by the product of h LkðiÞand hMLkðiÞ . Alsonotice that depending on the camera settings and thedistance of the object from the camera, the PSF under eachmicrolens may be flipped.7.1.3 DiscretizationTo arrive at a computational model, we discretize thespatial coordinates as u n with n 2f1...Ng and use thepixel coordinates ½i m ;j m Š T with m 2f1...Mg. Then, (25)can be rewritten in matrix-vector form l ¼ H s r as in (1).rðnÞ ¼ : rðu n Þ and lðmÞ ¼ : lði m ;j m Þ are the discrete vectorizedversions of l and r. H s 2 IR MN is the sparse matrixH s ðm; nÞ ¼ : h LIsðu n Þ ði m ;u n Þ;ð28Þwhere each column is the PSF corresponding to theupsampled depth map at that point (i.e., it is a nonstationaryoperator). Also, we are free to choose the spacing ofthe samples u n as ; setting ¼ 1 recovers the sameresolution as the original sensor; however, depending onthe camera settings, too high a resolution will not revealadditional detail. Hence, we choose based on our analysisof the limits of the LF camera as described in the previoussections. Note that, in particular, if the microlens diameteris reduced, more image detail will be visible (although lesslight efficient). Typically, we use ¼ 2 to reduce computationalload. The upsampling factor compared to the originalviews is Q=.7.2 Bayesian SuperresolutionWe use the Bayesian framework to estimate r, where allunknowns are treated as stochastic quantities. Withadditive Gaussian observation noise w Nð0; 2 wIÞ, themodel becomes l ¼ H s r þ w, and the probability ofobserving a given light field l in (1) may be written aspðlr; H s ; 2 w ;sÞ¼Nðl H s ;r; 2 w I Þ.We then introduce priors on the unknowns (assuming sis already estimated). Many recent image restoration worksmake use of nonstationary edge preserving priors. Forexample, total variation or modeling heavy-tailed distributionsof image gradients or wavelet subbands are popular[38]. We apply a recently developed Markov random field(MRF) prior [31], [32] which extends such ideas tomodeling higher order neighborhoods. It uses a localautoregressive (AR) model, whose parameters are alsoinferred, and leads to a conditionally Gaussian priorpðrj a; v :Þ¼Nðrj 0; C 1 Q v C T :Þ, where matrix C applieslocally adaptive regularization using AR parameters a; Q vis a diagonal matrix of local variances v . We estimate theseparameters using conjugate priors, which lets us setconfidences on their likely values. The resulting Gaussian—inverse-gammacombination also represents inferencewith a heavy-tailed Student-t, considering the marginaldistribution, corresponding to sparsity in the texture model.The SR inference procedure therefore involves finding anestimate of the parameters r; a; v ; 2 wgiven the observations land an estimate of H s . Direct maximization of the posteriorpðr; a; v ; w j l; H s Þ/pðljH s Þpðrj a; v Þpða; v ; 2 wÞ is intractable;hence, we use variational Bayes estimation withthe mean field approximation to obtain an estimate of theparameters.7.3 Numerical ImplementationThe variational Bayesian procedure requires alternateupdating of approximate distributions of each of theunknown variables. The approximate distribution for r isa Gaussian withIE½rŠ ¼cov½rŠ 2w HT s l;ð29Þcov½rŠ 1 ¼ C T Q 1v C þ w 2 HT s H s : ð30ÞThis update of IE½rŠ can be found solving M T Mr ¼ M T y,with Q 1v¼ L T L, M ¼½w 1HTs jCT LŠ T , and y ¼½w 1lTj0Š T .However, due to the large size of this linear system, we solveefficiently using CG for each step. Each CG iteration requiresmultiplying by H s and its transpose once, which weimplement as a sparse matrix formed using a look-up tableof precomputed PSFs from each point in 3D space. Theimage restoration procedure is run in parallel tasks acrossrestored tiles of size up to around 400 400 pixels, whichare trimmed to the valid region and then seamlesslymosaiced (size is limited such that the nonstationary H scan be preloaded into memory). We run experiments inMATLAB on an 8-core Intel Xeon processor with 2 GB ofmemory per task. Precalculating the look-up table can takeup to 10 minutes per depth plane (depending on PSF size);however, restoration is faster: Each CG iteration takesaround 0.1-0.5 seconds (again depending on depth). Goodconvergence is achieved after typically 100 to 150 iterations.The AR model parameters are recomputed every few CGiterations, taking around 1-2 seconds. Notice that since themodel is linear in the unknown r, convergence is guaranteed
982 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 5, MAY 2012TABLE 1Average L 2 Norm Disparity Error (Pixels per View 10 3 ),against Noise Levels, Filtering Method: No (A), Ideal (B),and Iterative Filtering (C), 4 textures and at 2 scales:Fine (1), Coarse (2)Fig. 12. Synthetic data: antialiasing filtering. Top: LF image of a sum-oftwo-sinusoidstexture (at five depth planes). Bottom: (a) One extractedview, containing aliasing. (b) The view filtered with the estimated depth(left) and the corresponding high-frequency image (right). (c) As in (b),but using the true depth; there is little difference between the two.(d)-(f) Enlarged portions of the last depth step in (a)-(c).by the convexity of the cost functional. The total runtime pertile is typically 30-60 seconds, depending on depth.8 EXPERIMENTSThe antialiased 3D depth estimation method is tested on bothsynthetic and real data in Section 8.1. Then, the proposed SRmethod is tested similarly in Section 8.2. We also evaluaterestoration performance in Section 8.2.2, comparing to othercomputational and regular camera systems.8.1 Antialiased Depth Estimation8.1.1 Synthetic DataThe scene in Fig. 12 consists of five steps at different depths,with a texture that is the sum of sinusoids at 0.2 and 1.2 timesthe views’ Nyquist rate f 0 , therefore the higher frequency isaliased in the views (but not in the subimages). The scene hasdisparities in the range s ¼ 0:24 to 0.44 pixels per view. Wesimulate a camera with Q ¼ 15, ¼ 9:05 10 6 m, d ¼0:135 mm, v ¼ 0:5 mm, f ¼ 0:5 mm, v 0 ¼ 91:5 mm, andF ¼ 80 mm. Note that for these settings, the microlens bluris small but nonnegligible, varying between 1.2 and 2.1 pixelsradius. We use the 9 9 pixel central region of each subimagefor depth estimation. Fig. 12 shows one of the 81 extractedviews, and the result of filtering with the estimated and truedisparity maps, with the aliased component separated out. InFig. 13, we compare the resulting disparity maps recoveredwith no antialiasing filtering, the iterative method, with thecorrect antialiasing filter (the errors at depth transitions occurdue to lack of occlusion modeling in the synthesized data),and the ground truth.We also test the algorithm’s performance, repeating theexperiment at 16 depths (for s ¼ 0:1 to 0.6). Results in Table 1show the average L 2 norm per pixel of the error between theRow header is in the format (Texture/Scale).ground truth and the disparity maps obtained with differentfiltering. We use the sinusoidal pattern and three texturestaken from the Brodatz data set (http://sipi.usc.edu/database/database.cgi?volume=textures). Each texture isresized to 380 380 pixels, and tested again with the central50 percent enlarged to this size to give a coarser scale. Wetest both the noise-free case and with additive Gaussianobservation noise at two different levels. The texturescontain different proportions of high and low frequencies;when more high frequencies are removed due to aliasing,matching performance decreases as less texture contentremains to match.8.1.2 Real DataThe use of small f=10 microlens apertures and a largemicrolens spacing (27 pixels per subimage) results insignificant aliasing in the data from Georgiev andLumsdaine [22] (kindly made available by Georgiev athttp://www.tgeorgiev.net), hence it is a good test for theantialiasing algorithm. Other camera settings are asdescribed in [6], [22]. Subimages and view details of thisdata are shown in Fig. 10, for different filters, toappreciate the effect of an incorrect size. In Fig. 14, weshow the disparity maps obtained at different steps of theiterative algorithm (the first iteration is essentially astandard multiview stereo result), along with the finalregularised result. Notice that there is a progressivereduction in the number of artifacts and that the disparitymap becomes more and more accurate. The estimatedFig. 13. Depth estimates. From top to bottom: Results obtained withoutfiltering, with the iterative method, with the correct filtering, and theground truth. Each row shows the disparity map (a) without and (b) withregularization. Notice how the results obtained with the estimated depthare extremely similar to those obtained with the correct filter.Fig. 14. Depth estimates on real data. (a) Disparity map estimateobtained without regularization (left) and for increasing filtering iterations(middle and right). (b) L 1 regularized disparity map, from the energy ofthe third iteration. (c) Regularized depth map for the puppets data set(Fig. 18).
Page 3 and 4: 974 IEEE TRANSACTIONS ON PATTERN AN
Page 6 and 7: BISHOP AND FAVARO: THE LIGHT FIELD

The Light Field Camera: Extended Depth of Field, Aliasing, and ...

Create successful ePaper yourself

Delete template?

Save as template?