The Light Field Camera: Extended Depth of Field, Aliasing, and ...

More documents

Recommendations

Info

BISHOP AND FAVARO: THE LIGHT FIELD CAMERA: EXTENDED DEPTH OF FIELD, ALIASING, AND SUPERRESOLUTION 977Fig. 7. (a) Choice of aperture sizes for fully tiling subimages. (b) Imagingthe purple vector under different microlenses. A point u on the radiancehas a conjugate point indicated by the triangle at u 0 , which is imaged by amicrolens centered at an arbitrary center c, giving a projected point atangle . A second microlens positioned at u images the same point at thecentral view ¼ 0 on the line through O. The scaling of the purple vectorto the green vectors is .The coordinates ðq; kÞ will be used often in the next sectionsto parameterize the light field, which is originally capturedwith respect to the coordinates ½i m ;j m Š T ¼ q þ Qk. Theinverse mapping isiðq; kÞ ¼ mod mþ Q $ %!Qj m 2 ;Q 2 ; ½i m ;j m Š Tþ 1 : ð7ÞQ 24.1.5 Ray Space RepresentationIn Fig. 8, we show an alternate view of sampling, aliasing,and blurring effects in the plenoptic camera, via the (2D) rayspace representation of Levin et al. [7]. In our version, weuse the internal camera coordinates: u (spatial) and theprojection of onto the main lens (angular). A point in thisspace represents a ray through the corresponding positionson the main lens and microlens array, while a ray withconstant color corresponds to a particular conjugate point(e.g., the vertical red rays are points on the microlens array,conjugate to the main lens plane in focus in space; differentslopes represent other depths, with the same colors as theplanes in Fig. 4). Each gray parallelogram indicates the raysin the LF that a sensor pixel integrates; their shear anglecorresponds to the conjugate plane the microlenses arefocused on. A column of parallelograms is one subimage,and a row represents a view. Gaps between parallelogramsare due to pixel fill-factor vertically, and microlens apertureshorizontally. To achieve the highest sampling accuracyachievable, the gray parallelogram should be aligned alonga ray of constant intensity (so that pixel integration does notlead to a loss of information). rðuÞ is the slice of the raysalong the x-axis.5 SUPERRESOLUTION LIMITS AND ANALYSIS OFMODEL5.1 SuperresolutionSo far we have found relations between points in space andtheir projections on the sensor, assuming infinitesimalmicrolens apertures. However, in general the image of apoint p 0 on the sensor is a pattern called the system PSF,whose shape depends on the projection of the finite aperturesonto the sensor. We will see that these blur sizes, as well asthe number of times a point is imaged under differentmicrolenses, can affect the SR quality.Fig. 8. Ray space diagram [7], with internal camera coordinates.(a) Apertures on the microlenses correspond to parallelograms. Theyellow region corresponds to objects inside the camera, while the greenline marks the conjugate image at 1, i.e., the object at distance F fromthe camera. (b) Microlens blur with finite apertures (vertical scale here isvin rather than 0v ).We would like to superresolve at resolutions close tothat of the original sensor. While we may render theestimate of r at any resolution, the actual detail that will berecovered will depend on a combination of factors relatedto how ill posed the inversion is, and how good our systemcalibration and priors are. Previous SR studies [34], [35],[36] showed in general that the performance of SRalgorithms decreases as blur size increases; it alsodecreases when the ratio of upsampling factor to numberof observations remains constant, but the upsamplingfactor increases. While our imaging model is ratherdifferent from those mentioned for regular SR, the samegeneral principles apply. We also point out some importantdesign considerations based upon these limitations.5.1.1 Main Lens DefocusThe cone of rays passing through p 0 causes a blur on themicrolens array determined by the main lens aperture. Thisblur disc determines how many microlenses capture lightfrom p 0 . With the Lambertian assumption, p 0 casts the samelight on each microlens, and this results in multiple copiesof p 0 in the LF (see first and third image in Fig. 4).Approximating this blur by a Pillbox with radius B 0 (i.e.,the unit volume cylinder, hðuÞ ¼ 1Bfor kuk 2 2 2 B2 and 0otherwise), the main lens blur radius isB ¼ Dv01 12 z 0 v 0 :ð8ÞTo characterize the number of repetitions of the samepattern in the scene, we must count how many microlensesfall inside the main lens blur disc. Therefore, in eachdirection we have#repetitions ¼ 2B d ¼ Dv01 1d z 0 v 0 : ð9ÞThis ratio is also evident in the ray space (Fig. 8) byconsidering how many columns (subimages) the blue rayon the right covers. In our SR framework, this numberdetermines how many subimages can be used to superresolvethe LF.5.1.2 Microlens BlurA necessary condition to superresolve the LF is that theinput views are aliased so that they sample different
978 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 34, NO. 5, MAY 2012 ¼ : z 0system and by generalizing the microlenses to virtual onesvv 0 v 0 z : ð12Þ centered in the continuous coordinates c, rather than0discrete positions c k .Notice the relation to Lumsdaine and Georgiev’s magnificationfactor [21]. Here, however, we refer everything to acommon reference frame in r in order to compare multipledepths. Note that the number of repetitions may also berewritten in terms of as#reps ¼ D v 0 z 0d z 0 ¼ v0v 0 z 0v 0 þ vz 0 v v 0 ¼ 1 QFig. 9. (a) and (b) Microlens blur radius b versus scene depth z (in logjj d ;scale), for several settings of the microlens-to-CCD spacing v. Theð13Þmicrolens focal length f is 0.35 mm (note that Ng et al. [3] sets v ¼ f). where we used (6). Thus, since Q dis constant, as theThe main lens plane in focus is at 700 mm. Our model can work with anynumber of repetitions increases, the size of subimagesuitable settings. (c) Magnification factor between the radiance at themicrolens plane and each subimage, under similar settings.features decreases.The above equation formalizes the constraint that, forinformation, i.e., they are not just shifted and interpolatedversions of the same image. We discuss aliasing further inSection 6, but here it suffices to note that increasingsubimage blur reduces complementary information in theLF available to perform SR. The blur of each microlens thateach depth, the amount of information from r remainsconstant, but is split across a different number of subimages.What does change, however, is the effective blur, orthe integration region in r of a sensor pixel, as it scales withthe magnification factor .is fully covered by the main lens PSF (some are not, seeSection 7.1.2) is also a pillbox due to the microlens aperture, 5.1.4 Coincidence of Samples and Undersamplingwith radius (see Fig. 5):In this section, we show that samples from differentb ¼ dvmicrolenses coincide in space on some fronto-parallel1 1 12 fv 0 z 0 v planes. On these planes the aliasing requirements are notsatisfied and the SR restoration performance will decrease.where f is the microlens focal length. In Fig. 8, theWe shall see this experimentally in Section 8.2.2. One suchmicrolens blur is the horizontal projection of a ray onto plane is when the conjugate image lies on the microlensthe subimage pixels it intersects. Restoration performance isoptimal at depths where b ! 0 (rays like the purple one inFig. 8, with the same slope as the pixel shear), obtained forplane. Another is shown in Fig. 6, where r 0 is positionedsuch that the blue, red and black rays intersect inside thecamera at this depth. In Fig. 8, this degeneracy correspondspoints p 0 at a distance z 0 ¼ v 0 vfv f, and it will degrade to a ray passing through exactly the same point in aaway from these depths. When a microlens is not fully parallelogram in different subimages.inside the main lens PSF, then its blur radius is smaller (seeSection 7). For simplicity, consider the blur radius b 0 whenboth microlens and main lens share the same optical axis(see the general case in Section 7):To have an exact replica of a sample of r 0 under twomicrolenses, it must simultaneously project on two discretepixel coordinates. Considering two microlenses separatedby Nd, with N 2 Z, then the coordinate 0 under the firstb 0 ¼ min2Bb microlens must correspond to a coordinate NdðuÞ ¼T,d ;b : ð11Þ where T 2 Z. Also, for the corresponding pixel to be fullyinside the subimage, T< QNotice that as z 0 ! v 0 2. By substituting the expression, although b !1, the radius b 0 for T, we see that such microlenses must satisfy N
Page 3 and 4: 974 IEEE TRANSACTIONS ON PATTERN AN
Page 8 and 9: BISHOP AND FAVARO: THE LIGHT FIELD

The Light Field Camera: Extended Depth of Field, Aliasing, and ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?