Shadow Multiplexing for Real-Time Silhouette ... - EDM - UHasselt

Shadow Multiplexing for Real-Time Silhouette ExtractionTom Cuypers Yannick Francken Johannes Taelman Philippe BekaertHasselt University - tUL - IBBTExpertise Centre for Digital Media, Belgium{firstname.lastname}@uhasselt.beMethod overview. The setup consists of multiple colored point light sources, a diffuser and a digitalcamera. For each light source a silhouette is extracted from the captured shadows of the scene. Thesesilhouettes are used for visual hull reconstruction as well as image based collision detection.AbstractIn this work we propose a real-time implementation forefficient extraction of multi-viewpoint silhouettes using asingle camera. The method is based on our previously presentedproof-of-concept shadow multiplexing method. Wereplace the cameras of a typical multi-camera setup withcolored light sources and capture the multiplexed shadows.Because we only use a single camera, our setup ismuch cheaper than a classical setup, no camera synchronizationis required, and less data has to be captured andprocessed. In addition, silhouette extraction is simple as weare segmenting the shadows instead of the texture of objectsand background. Demultiplexing runs at 40 fps on currentgraphics hardware. Therefore this technique is suitable forreal-time applications such as collision detection. We evaluateour method on both a real and a virtual setup, andshow that our technique works for a large variety of objectsand materials.1. IntroductionMany virtual reality applications require a natural interactionbetween the real and the virtual world. An intuitiveway to achieve this is to create a 3D model of the humanactor, perform collision detection and provide an appropriateresponse in the virtual and/or real scene. As this processneeds to operate in real-time without being visuallyrendered, an approximate model often suffices. A traditionalapproximation for real-time modeling is the shapefrom-silhouettetechnique [2, 17]. In a multi-camera setup,silhouettes are obtained by performing background subtractionon every image. They are then combined to reconstructthe original volume. As an increasing number of silhouettesoften results in a better reconstruction, a large amountof calibrated and synchronized cameras is needed to produceaccurate results, making this an expensive and complexsetup. The vast amount of data generated by thesecameras makes it ill-suited for real-time full-resolution processingon a single computer.Therefore we proposed a technique that extracts multiplesilhouettes from different views in a single image, byreplacing the cameras with colored light sources and capturingthe cast shadows on a diffuser [6]. This methodis related to shadow carving [22] and coplanar shadowgrams[27], but instead of extracting a set of silhouettesfrom subsequent images, we obtain multiple silhouettesfrom a single image. In this way, our approach is very similarto the recent work of Lanman et al. [16], although theirmethod obtains silhouettes from a light field by using a specializeddense lens array, where we achieve our silhouettesfrom segmenting colors on a common diffuser. Our setup

LLL0 I 0 I 0(a) Red (b) Green (c) BlueFigure 3. Relationship between the luminance and current of a (a)red, (b) green and (c) blue LED. Notice the strong green responseof the CCD camera for the blue LED.I4.2. CalibrationIn order to provide our algorithm with the light propertiesof the setup and the relative positions of the hardwarecomponents, we will now cover radiometric and geometriccalibration.4.2.1 RadiometricEach LED is connected to a controllable power source. Byadjusting the current sent through the LED, more light willbe emitted. As can be seen in Figure 3, the luminance willnot respond linearly in function of the current. This plot canbe measured or found in the characteristics provided by theLED manufacturer. In practice this means that the currentrequired for getting a certain intensity value can simply befound in a lookup table containing the LED characteristics.4.2.2 GeometricThe geometric calibration of the setup consist of scanningthe 3D structure of the diffuser and defining the locationof each point light source. We assumed that our diffuseris planar, simplifying the calibration. We calibrate it by attachinga checkerboard pattern and using a standard calibrationmethod [3, 10]. More in general all kinds of 3D scanningtechniques, such as structured light [23] can be usedfor scanning the diffuser.Finding the location of each light is achieved using thecalibration proposed by Yamazaki et al. [27]. It analyzesthe ellipsoidal shadows cast from occluding spheres.5. ResultsIn this section we will show results we obtained and theproblems we encountered. We will also demonstrate twoapplications where the extracted silhouettes are used.In figure 7 we show a couple of extracted silhouettes.Herefore we use two red, two green and two blue LEDs withdifferent intensity. These results indicate that this techniqueis suitable for different types of materials like low albedo orspecular surfaces.(a) Demultiplexed shadow(b) PostprocessingFigure 4. Captured images contain soft shadows: (a) Straight forwardshadow demultiplexing method (b) Figure (a) after performingerosion and dilation5.1. ProblemsThe straightforward method for demultiplexing cansometimes introduce some errors and artefacts which weshow can be reduced.5.1.1 Soft BordersA point on the diffuser does not only receive light fromeach light source, but also from the neighbouring illuminatedpoints. The diffusion due to scattering within the diffuserresults in a smoothened transition at the borders ofthe shadows. These soft borders sometimes create a ghostsilhouette as shown on Figure 4(a). Most of the time theyhave a size of only a few pixels. Therefore they can easilybe removed by performing morphological operations suchas erosion and dilation as shown on Figure 4(b).Soft borders can also occur due to motion blur. We canminimize these by lowering the shutter time of our camera.Therefore we require brighter LEDs. LED strobing isa common method for obtaining the same intensity with alower shutter time reducing motion blur. However, the OS-RAM Diamond Dragon LEDs do not allow significant extralight output in pulse operation, so this technique will requirea different selection of LEDs.5.1.2 Camera noiseShadow demultiplexing is a local algorithm meaning thatcamera noise can result in faulty results at some pixels.Even though this noise is already minimal, it will even befurther reduced by the erosion operation.5.1.3 Object MaterialOur method works well for sufficiently opaque materialssuch as high and low albedo diffusers, glossy and specularsurfaces, and many translucent materials. However, as isthe case for standard background subtraction, this method

Figure 5. Collision detection overview (a) input, a virtual object we want to check collision with (b) the virtual object converted into apoint cloud (c) the point cloud is projected onto the demultiplexed shadows of the real object (d) red points are inside every silhouette afterprojection, while blue points are projected outside at least one silhouette.5.3. ApplicationsIn this section we will demonstrate the usefulness of ourmethod for interactive collision detection. The first applicationshows how we can reconstruct a visual hull from thesesilhouettes, which can then be used for collision detectionwith virtual objects. As a second application we show howthe silhouettes can be used for collision detection in imagespace.5.3.1 Visual Hull ReconstructionFigure 6. Artifacts due to (a) specular reflections and (b) refractionswill break down for mainly transparent materials. Occasionallysmall artefacts appear when highly specular materialsreflect their light directly toward the surface as shownin Figure 6.5.1.4 Blue-Green Response OverlapBecause we do not use a multispectral camera, the numberof multiplexed blue and green LED intensities was limitedby two per light source. This is due to the large responseoverlap between green and blue on CCD cameras. This isillustrated in Figure 3.5.2. TimingsThe method is implemented on an NVidia GForce 8800GTX. We store up to 4 silhouettes into a single texture, reducingrequired memory storage. It also takes advantage ofthe scalar processors. The number of render passes is alsoreduced by 4 and the erosion and dilation can be calculatedsimultaneously. For input images containing 6 silhouetteswith a resolution of 640 ×480 we achieve a framerate of 40fps.The visual hull of the object can be reconstructed usingclassical shape-from-silhouette techniques. Because thesemethods usually require silhouettes from the viewpoint ofmultiple cameras we have to perform a transformation onthe demultiplexed shadow. Since the setup is calibrated wecan use projective texture mapping for an efficient transformation.We model the diffuser on which we project ourmultiplexed shadows. Then we render it from the positionof the light source. Because the shadows are already demultiplexedon the GPU, this step can easily and efficiently beadded. An overview of this method is shown in the teaserimage.5.3.2 Image-Based Collision DetectionCollision detection can also be done directly in image space.Therefore we divide our virtual objects into small voxelsand project each voxel onto the input image. We can thendetermine if this voxel is projected inside or outside the realobject’s silhouette. When a projection falls inside every silhouette,we assume a collision has occured. This way collisionresponse can be handled directly in image space. Thisis similar to the work of De Decker et al. [8] . An overviewof this method is shown in Figure 5.6. ConclusionWe proposed a technique to efficiently obtain multiplefull resolution silhouettes of an object from multiplexedshadows using a single camera, allowing to construct a visualhull or provide image based collision detection in realtimeon a single pc. Because only one camera is required,

no synchronization is needed and less data has to be obtainedand processed, increasing the performance and accuracy.Furthermore, easy and robust background subtractionis achieved for a variety of object materials.7. Future WorkCurrently we manually control the intensities of ourLEDs. We are working on a computer-controlled LEDdriver to automate LED intensity calibration, providing amore accurately tuned configuration.In order to increase the number of silhouettes in our prototype,we are looking to replace the RGB camera with amultispectral camera since some of the method’s potentialis lost due to the considerable blue-green response overlapof RGB cameras.AcknowledgementsThe authors (Cuypers Tom and Philippe Bekaert) acknowledgefinancial support by the European Commission(FP7 IP ” 2020 3D media”). We gratefully expressour gratitude to the European Regional Development Fund(ERDF), the Flemish Government and the Flemish Interdisciplinaryinstitute for Broadband Technology (IBBT),which are funding part of the research at the Expertise Centrefor Digital Media. We also would like to thank our colleaguesfor their time and effort in this work.References[1] S. Balcisoy and D. Thalmann. Interaction between real andvirtual humans in augmented reality. In CA 1997, page 31,1997. 2[2] B. G. Baumgart. Geometric modeling for computer vision.PhD thesis, Stanford University, Stanford, CA, USA, 1974.1[3] J.-Y. Bouguet. Camera calibration toolbox for matlab, 2006.http://www.vision.caltech.edu/bouguetj/calib doc/. 5[4] K. Cheung, T. Kanade, J. Bouguet, and M. Holler. A realtime system for robust 3d voxel reconstruction of human motions.In CVPR 2000, pages 714 – 720, 2000. 2[5] S. S. Cheung and C. Kamath. Robust techniques for backgroundsubtraction in urban traffic video. Proceedings ofVideo Communications and Image Processing, pages 881–892, 2004. 2[6] T. Cuypers, Y. Francken, and P. Bekaert. Shadow multiplexingfor single camera visual hull reconstruction. In PRO-CAMS ’08: Proceedings of the 5th ACM/IEEE InternationalWorkshop on Projector camera systems, pages 1–2, 2008. 1,2[7] T. Cuypers, C. Vanacken, Y. Francken, F. V. Reeth, andP. Bekaert. A multi-camera framework for interactivevideogames. In GRAPP 2008, pages 443–449, 2008. 2[8] B. D. Decker, T. Mertens, and P. Bekaert. Interactive collisiondetection for free-viewpoint video. In GRAPP 2007,pages 114–120, 2007. 2, 6[9] Y. Francken, T. Cuypers, T. Mertens, J. Gielis, andP. Bekaert. High quality mesostructure acquisition usingspecularities. In CVPR 2008, pages 1–7, 2008. 2[10] R. I. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press, secondedition, 2004. 5[11] J.-M. Hasenfratz, M. Lapierre, and F. Sillion. A real-timesystem for full body interaction with virtual worlds. EurographicsSymposium on Virtual Environments, pages 147–156, 2004. 2[12] M. Hatzitheodorou. Shape from shadows. Journal of Complexity,pages 63–84, 1998. 2[13] B. Horn. Obtaining shape from shading information. In ThePsychology of Computer Vision, pages 115–155, 1975. 2[14] P.-H. Huang and S.-H. Lai. Contour-based structure fromreflection. CVPR 2006, pages 379–386, 2006. 2[15] H. Kawasaki, R. Furukawa, R. Sagawa, and Y. Yagi. Dynamicscene shape reconstruction using a single structuredlight pattern. In CVPR 2008, pages 1–8, 2008. 2[16] D. Lanman, R. Raskar, A. Agrawal, and G. Taubin. Shieldfields: Modeling and capturing 3d occluders. SIGGRAPHAsia 2008, 27(5), 2008. 1, 2[17] A. Laurentini. The visual hull concept for silhouette-basedimage understanding. IEEE PAMI 1994, pages 150–162,1994. 1[18] B. Lok, S. Naik, M. Whitton, and J. F.P. Brooks. Incorporatingdynamic real objects into immersive virtual environments.In I3D 2003, pages 31–40, 2003. 2[19] A. M. McIvor. Background subtraction techniques. Imageand Vision Computing, 2000. 2[20] Osram. Osram opto semiconductors - products, 2008.http://www.osram-os.com/osram os/EN/Products. 4[21] PointGreyResearch. Imaging products - grasshopper, 2008.http://www.ptgrey.com/products/grasshopper. 4[22] S. Savarese, H. Rushmeier, F. Bernardini, and P. Perona.Shadow carving. ICCV 2001, pages 190–197, 2001. 1, 2[23] D. Scharstein and R. Szeliski. High-accuracy stereo depthmaps using structured light. CVPR 2003, pages 195–202,2003. 2, 5[24] Y. Schechner, S. Nayar, and P. Belhumeur. A theory of multiplexedillumination. ICCV 2003, pages 808–815, 2003. 3[25] A. R. Smith and J. F. Blinn. Blue screen matting. In SIG-GRAPH 1996, pages 259–268, 1996. 2[26] R. Woodham. Photometric method for determining surfaceorientation from multiple images. Optical Engineering,pages 139–144, 1980. 2[27] S. Yamazaki, S. G. Narasimhan, S. Baker, and T. Kanade.Coplanar shadowgrams for acquiring visual hulls of intricateobjects. In ICCV 2007, pages 1–8, 2007. 1, 2, 5

LOW ALBEDOTRANSLUCENTSPECULARDIFFUSEFigure 7. Left: The object we want to obtain multi-viewpoint silhouettes from, middle: Input image consisting of multiplexed shadows,right: Six demultiplexed shadows