Covariant Derivatives and Vision - Todor Georgiev

Covariant Derivatives and VisionTodor GeorgievAdobe PhotoshopPresentation at ECCV 20061

Original2

Selection to clone3

Poisson cloning from dark area4

Selection to clone5

Poisson cloning from illuminated area6

PoissoncloningCovariantcloning(see next)7

Poisson cloning can be viewed as anapproximation to covariant cloning.Outline of our theory:8

Thanks to Jan Koenderink10

Retina / Cortex Adaptation• The image is just a record of pixel values.• We do not see pixel values directly.• What we see is an illusion generated fromthe above record through internal adaptation.We can not compare pixels.11

Models of Image Space:12

(1) Cartesian Product• A pair (location, intensity)• Multiple copies of the intensity line.• We can compare intensities. The image is a functionthat specifies an intensity at each point.13

(2) Fiber Bundle• Two spaces and a mapping (vertical projection)Total space EBase space B• Fiber is the set of points that map to a single point.We will use vector bundles, where fibers are vectorspaces.14

Section in a Fiber Bundle• Mapping from base B to total space ESectionsreplacefunctions• We can not compare intensities. Horizontalprojection is not defined. We have forgotten it.• Perceptually correct model of the imageImage = graph of a section15

Derivatives in a Fiber BundleDefinition:Derivative is a mapping from one section toanother that satisfies the Leibniz rule relativeto multiplication by functions:In the Cartesian product space this definition isequivalent to the conventional derivative.16

Derivatives in a Fiber BundleIf we express a section as a linear combination of somebasis sectionsthen the derivative will be:17

Derivatives in a Fiber BundleIf we represent the section in terms of the functionsthat define it in a given basis (not writing thebasis vectors), the last equation can be written as:The functions are called “color channels” inPhotoshop, and D is called “Covariant Derivative”.It corresponds to the derivative in the Cartesianproduct model.18

The covariant derivative is a rigorous mathematical toolfor perceptual pixel comparison in the fiber bundle modelof image space. It replaces the conventional derivative ofthe Cartesian product model as:19

- Reconstructing images with thecovariant Laplace equation,based on adaptation vector field, A.- Reconstructing surfaces based on gradient field.Recent work by R. Raskar et. al. Covariant Laplaceshould produce better results than Poisson.How can we know A?It can be extracted based on the idea of covariantlyconstant section, next:21

Assume perceived gradient of image g(x, y) is zero.This means complete adaptation:Substitute in covariant Laplace:Poisson equationCovariantcloning22

PoissoncloningCovariantcloning23

CovariantPoisson24

Detailed Example:25

Original Damaged Area26

Laplace27

Poisson28

Laplace29

InpaintingThanks to Guillermo Sapiro and Kedar Patwardhan30

Poisson31

Covariant32

InpaintingThanks to Guillermo Sapiro and Kedar Patwardhan33

Structure and Texture InpaintingBertalmio – Vese – Sapiro – Osher34

Covariant Inpainting35

Day36

Night37

Covariant cloning from day38

Poisson cloning from day39

Cloning from night to dayThanks to R. Raskar and J. Yu40

Gradient Domain HDR CompressionChanging the lighting conditions. The visual system isrobust. It compensates for the changes in illuminationby adaptation vector field A:Simplest energy invariant to those transforms is:41

Gradient Domain HDR CompressionEuler-Lagrange equation for the above energy:Exactly reproduces the result of the Fattal-Lischinski-Werman paper. They assume log; we derive log.Any good visual system needs to be logarithmic!42

Conclusion:The covariant (adapted) derivative providesa way to perform perceptual image processingaccording to how we see images as opposed to -how images are recorded by the camera.Useful for Poisson editing, inpainting or any PDE,HDR compression, surface reconstruction fromgradients, night/day cloning, graph cuts, bilateraland trilateral filters in terms of jet bundles, andpractically any perceptual image editing.43

Appendix:Bilateral interpreted in 3D image spaceThe image is a distribution in 3D:or -perceptual?Integrate the following 3D filter expression over zand evaluate it on the original surface. Result:(1)44

Bilateral(1)Same procedure on the logarithmic expressionproduces (using “delta function of function” formula):Now, bilateral filter is exactly expression (2) dividedby expression (1). Paris-Durand paper derives a similarresult (based on intuition) and a speed up algorithm.(2)45