Lecture 1: Introduction to Biological Vision - Vision Research ...

Lecture 1• Overview of primate visual system–The computational problem the visual system issolving• Jobs of vision–Form & function: retina to extra-striate cortexWednesday, August 29, 12

The computational problemWhat do you perceive?Visual perception relieson local retinalmeasurements of lightintensity...10 to 100million per second.How to extract features,and integrate them toproduce usefulbehavior?L(x,y)From: Mumford, 20026Wednesday, August 29, 12

The computational problemLyxYour brain’s solutionYour eye’s input as intensity orluminance “data”Shape, material, size, depth,identity, intentions, ...L(x,y,t)Wednesday, August 29, 12

• Object perception– Object properties: shape, material– Recognition:• Problem of variation: view, illumination,...• Viewer-object relations– Navigation• heading--motion and optic flow• Time-to-contact--motion and optic flow– Manipulation/grasp– Tracking and eye movements• Object-object relations– Object trajectories -- (depth from moving castshadows)– Scene interpretation, planningWednesday, August 29, 12Jobs of visionSame L(x,y), butdifferent apparent materialFrom AdelsonFrom Kersten et al.No change in L(t), butdifferent apparent motion indepth8

Visual neural architectureComplex system needed to do the jobs ofvision–Up to 50% of macaque monkey gray matter isvisually sensitive–Multiple pathways–Multiple interacting visual areas:• feedforward and feedback connections–Neural architecture reflects task-dependency• -- the jobs of visionWednesday, August 29, 12

Challenges in VisionTheoretical challenge– Understand, in principle, how vision makes reliableinferences and produces useful, flexible and diverseactions given local ambiguity, and the complexity ofnatural images• Theoretical and computational studiesEmpirical challenge– Testing quantitative theories of biological visual behavior• Psychophysics or behavioral studies– Understanding visual behavior in terms of neural systems• Neural electrophysiology, brain imaging, etc..Wednesday, August 29, 12

Measurements of vision• Image measurements support inferences,jobs of vision– Intensity change in space (edges)– Color change– Texture change– Binocular disparity– Intensity change in time and space (optic flow,motion)Measurements derived from transformations of physicallight energy flux (“radiance”) as a function of space, timeand wavelength:R(x, y, t, )Wednesday, August 29, 12

Overview of human/primate visualsystem• Outline of form and function–The eyeball–Optics–Retina–Pathways from eye to cortex–Primary visual cortex–Extra-striate areasWednesday, August 29, 12

Wednesday, August 29, 12Eyeball: receptor density

Eyeball• Eye musclesFrom: http://www.yorku.ca/eye/muscle.htmWednesday, August 29, 12

Eyeball & muscles• Form & function:– Get the interesting part of the image placed where acuity isbest (where receptor density highest)-- the fovea.– 6 of the fastest moving muscles in the body control eyemovements.–Fixations correlated with attention, action plans• Active work to compute salient visual directions fromnatural images by computing “saliency maps”, perhapsformed in V1 (primary visual cortex)Itti and Koch, 2001; Koch and Ullman, 1985; Wolfe, 1994,Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G.W. (2008); Zhang, X., Zhaoping, L., Zhou, T., & Fang, F.(2012) ...Wednesday, August 29, 12

Overview of human/primate visualsystem• Outline of form and function–The eyeball–Optics–Retina–Pathways from eye to cortex–Primary visual cortex–Extra-striate areasWednesday, August 29, 12

Wednesday, August 29, 12Optics

Optics• Functions– Image the world• simple & complex eyes indicate the importance of gathering patterninformation about nearby spatial relationships, i.e. visual direction• Refraction– cornea - bulk of refraction (fixed, ~43 diopters)– focus, lens (adaptive, “accommodation”)• focus range when young, up to 15 diopters• pupil - some control over light level (2 to 8mm diameter, or x16 areachange), depth-of-fieldWednesday, August 29, 12

• Why aren’t the optics better? Anti-aliasing?– Receptor packing near fovea is regular• (distribution of receptor types is not)Wednesday, August 29, 12

Retinal mosaic & regularsamplingFrom Roorda & Williams, 1999. Figure 3 Pseudocolour image of the trichromaticcone mosaic. Blue, green and red colours represent the S, M and L cones,respectively. a, b, Subject JW's temporal and nasal retina, respectively, at onedegree of eccentricity. c, Subject AN's nasal retina, at one degree of eccentricity.We performed a statistical test for randomness according to Diggle27. Wecompared the distribution of all intercone distances of the measured M-conearray with 100 simulations derived from the same mosaic in which the samenumber of M cones were randomly assigned. JW's array was no different fromrandom at either location. AN's array showed significant clumping of the data(€ , 0X01) but, because of optical blur, the possibility of a random assignmentof M cones cannot be ruled out. Scale bar represents 5 arcmin of visual angle.Wednesday, August 29, 12

Overview of human/primate visualsystem• Outline of form and function–The eyeball–Optics–Retina–Pathways from eye to cortex–Primary visual cortex–Extra-striate areasWednesday, August 29, 12

RetinayxReceptive field weights: W(x,y)Responsivity10.80.60.40.2-3 -2 -1-0.21 2 3-0.4xReceptive field weights: W(x)Wednesday, August 29, 12

• Functions of the retina– discrete image samplingRetina• Why aren't the receptors packed closer together? Physical wave guide limits?– automatic gain control• Retina codes contrast, not intensity– band-pass spatial filtering (“lateral inhibition”)• I.e. neither slow nor very rapid spatial changes in intensity are seen– economical coding (e.g. predictive coding)– front-end for edge detection– band-pass temporal filtering• I.e. neither slow nor very rapid temporal changes are seen– economical coding– front-end for motion measurement– color coding• (some animals have motion selective cells in retina, but not cats andprimates)• remove correlations from cone signals• front-end for color constancy mechanismsR L(x, y, t) L avgL avgW(x,y)R ~ W(x,y)*L(x,y)Wednesday, August 29, 12

...but even moreEvidence that the mammalian retina supportsdetecting object expansion and trajectoryextrapolation, and pattern adaptationGollisch, T., & Meister, M. (2010). Eye Smarter than Scientists Believed: Neural Computations in Circuitsof the Retina. Neuron, 65(2), 150–164. doi:10.1016/j.neuron.2009.12.009Meister, M., & Berry, M. J. (1999). The neural code of the retina. Neuron, 22, 435–450.Wednesday, August 29, 1228

Overview of human/primate visualsystem• Outline of form and function–The eyeball–Optics–Retina–Pathways from eye to cortex–Primary visual cortex–Extra-striate areasWednesday, August 29, 12

Pathways out of the eyeFrom: Milner and Goodale, 1995Wednesday, August 29, 12

Pathwaysout of the eye“Area 17”“Striate cortex”“Primary visual cortex”“V1”Wednesday, August 29, 12

Optic nerveOptic tractOptic radiationLots of feedbackconnections fromcortex to LGNWednesday, August 29, 12

Pathways from eye tocortex• Retina (about 10 7 to 10 8 cones) ->–optic nerve (about 10 6 axons) ->–optic chiasm ->–-optic tract• -> about 20% of the fibers to superior colliculus• -> lateral geniculate nucleus–optic radiation ->–layer IV of primary visual cortex• Lateral geniculate nucleus–magnocellular pathway (layers 1 & 2)–parvocellular pathway (layers 3,4,5,6)Wednesday, August 29, 12

Pathways from eye to• Functions:cortex–chiasm :• routes axons so that information from correspondingpoints on the left and right eyes can come together(eventually at cortex).–lateral geniculate nucleus:• more band-pass spatial filtering, even smaller response tochanges in mean level -- i.e. the excitatory (+) andinhibitory (-) components to the receptive field W(x,y) arebalanced. Again, absolute light level not important tovisual processing• feedback from cortex (10-fold) in optic radiation suggestspossible role in information gating/selective attention,binding, multi-modal interactions...?Wednesday, August 29, 12

Overview of human/primate visualsystem• Outline of form and function–The eyeball–Optics–Retina–Pathways from eye to cortex–Primary visual cortex–Extra-striate areasWednesday, August 29, 12

Feedforward flow of image informationDorsalPosteriorAnteriorVentralV1Wednesday, August 29, 12

Primary visual cortex (V1)Anteriorone of ~30 visual areasDorsalVentralPosteriorWednesday, August 29, 12

Primary visual cortex• Basic properties–~200,000,000 neurons in one hemisphere ofmacaque–Large-scale• non-linear topographic map (cm)• (the first of several retinotopic cortical areas)– Fine-scale• hypercolumn organization (1 to 2 mm)Wednesday, August 29, 12

V1 large scale organization:RetinotopyA topographic mapping transforms retinal coordinates to V1cortical coordinates--a retinotopic map. “Log polar” in V1.PosteriorAnteriorFixate here.The contracting wave ofcontrast on your retinaproduces a traveling wave ofactivity in your V1,represented by theanimations on the right.FixationNow fixate here.And play the animation again.Rotating waves on yourretina produce approximatelytranslating waves of activityin your V1.Visual fieldV1(upside-down)Wednesday, August 29, 12

Neuroimaging of the radialdirectionHuk, A. C. (2008) Visual Neuroscience: Retinotopy meets Percept-otopy, Current Biology, 18,21, R1005-1007.Wednesday, August 29, 12

V1 small scale• Basic properties–V1 architecture• hypercolumns organized into columns perpendicular tosurface in which cells prefer same orientation anddegree of binocularity• hypercolumns are on the scale of 1-2 mm, and spanrange of orientations and degree of left/right eyepreference. A hypercolumn is believed to be acomputational unit that analyzes a “point” of visualspace.Wednesday, August 29, 12

V1 small scaleLayer I near skullLayer VI deeperHypercolumnReceptive fieldmaps characterizethe spatial filteringproperties of V1cells. The forms ofW(x,y) are nolonger primarilycircular.Wednesday, August 29, 12

V1 small scale• Basic properties– 6 layers I-VI (“gray matter”)• 100,000 neurons under 1 sq mm of cortex– cortical thickness, ~2 to 4mm• Layer IV receives LGN outputs– Apart from those neurons that LGN neurons form synapses with, V1neurons have the following properties (macaque):• spatial frequency selective (i.e. scale or size)• orientation selective• binocularity• motion direction selective• non-oriented cells, wavelength selective– Receptive fields properties modifiable by early visual experience• many computational neural models of self-organizationWednesday, August 29, 12

Psychophysics of spatial frequencyadaptation using square-wave gratingsWednesday, August 29, 12Note that the two patterns on the right are identical. Now let your eyes scan back and forth overthe long red bar on the left for about a minute, and then look at the small red square on the right.Do the upper and lower right patterns still look the same? How are they different? It is believedthat populations of neurons with different spatial frequency tuning are responsible for thiseffect.44

Contrast sensitivity of some V1neuronsOne can characterizethe “tuning” orspatial filteringproperties of a celleither by mapping outthe response topoints of light, or asfor retinal ganglioncells, by mapping outthe response tosinusoidal gratings ofvarious spatialfrequencies.Wednesday, August 29, 12From Wandell Book: Spatial frequencyselectivity of 6 neurons in V1 macaque.DeValois, 198245

Orientation adaptation:“tilt aftereffect”Here’s an orientation version of the spatial frequencydemo. First fixate region of red bar for about 1 minute,allowing eyes to move. Then look at red dot on right. Dothe vertical lines still appear vertical?46Wednesday, August 29, 12

Functions of localspatial filtering?Simple cell representationthe family of receptive fieldweights can be modeled asa “basis set”OrientationFilter symmetries:orientationscale-- “self-similar”Relation to “wavelets”Spatial scale (frequency)Wednesday, August 29, 12

ontal and vertical spatial frequencies? (Answer: this is the spatial frequency equivalent to applying a Laplaciantor in the space domain).an play with changes to the amplitude or phase spectrum to see what happens. For example, what happens if youply the amplitude spectrum by an array whose values are proportional to the negative of the sum of the squares of the“Neural image” or“channel”, based on ae assorted functions and filtersconvolution model20 Fourier_neural_Out[117]=W =R i,j =In[118]:=filterft = Fourier@shift@filter, hsizeDD;ArrayPlot@fface = Chop@t = InverseFourier@filterft faceftDD,Mesh Ø False, ColorFunction Ø "GrayTones", DataReversed Ø TrueDx,yW i x,j y L x,y]=The DensityPlot above shows a view of the "receptive field" of an oriented band-passOut[118]=cell. Let's plot the filter along thediagonal:48Wednesday, August 29, 12DataReversed Ø TrueD

Functions of cortical local spatiotemporalfiltering ?• V1 neuron receptive field properties• How does a neuron respond to input patterns ofluminance L?–Assuming the spatial filters (receptive field weights W)are approximately identical across space, thepopulation of responses is often modeled linearly andrepresented by convolution operation:R i,j (t) = (W i x,j y L x,y (t))x,y49Wednesday, August 29, 12

But V1 cells are that “simple”• Receptive field types– Simple cells• First-order approximation:– linear spatio-temporal filter (i.e. cross-correlation ofimage with weight function) followed by half-waverectificationR i,j (t) = ( W i x,j y L x,y (t))– Complex cells• roughly modeled as (if!) a linear combination ofsimple cells spanning a small region of space,combining a range of phases in the filters (e.g.sine and cosine)x,y• involved in build-up of translation and scaleinvariance for object recognition?W x,ySine phase--”edge detector”Cosine phase--”bar detector”Wednesday, August 29, 12

ming := 0.25` HRandomReal@D - 0.5`L + 0.5`;maxg2 := 0.15` HRandomReal@D - 0.5`L + 0.5`;d1 = TableBIfB i - width22+ j - width8i, 1, width

• edge and bar detectorsvs.V1 cell functions• local quasi-Fourier analysis of image–neuron's receptive field represents basis function–firing rate represents the coefficient–Why?• economical coding of image– coefficients statistically uncorrelated?• analysis of image over a range of spatial scales useful for– edge detection, textures, build-up of invariances,...– integration of information globally?Wednesday, August 29, 12

Well-known problems with localedge detectorsSharp edges around objects are aneveryday visual “illusion”–Local information for an edge is almostalways ambiguous• Lots of “noise” in images• Edge identity is ambiguous– i.e. is the edge due to a change in illumination,shape, depth, material, noise? Some edges are lessimportant than others–Noise vs. smoothingWednesday, August 29, 1253

Efficient coding and higher orderEfficient (“predictive”) coding givenredundancies in natural images. E.g.L(x,y) can be predicted by L(x’,y’) whenx’ and y’ are near x and y.dependenciesGives rise to neuralnetwork models that areclosely related toprinciples of imagecompression developed insignal processing theory,as in “difference coding”R(x) = L(x) - L(x-1)which exploits theobservation that L(x) isoften ~ L(x-1)Kersten, D. (1987). Predictability and redundancy ofnatural images. J Opt Soc Am A, 4(12), 2395-2400.54Wednesday, August 29, 12

Population coding and decodingCurrent area of research: How is information aboutorientation stored and used in V1?Zemel, R. S., Dayan, P., & Pouget, A. (1998). Probabilistic interpretation ofpopulation codes Neural Computation, 10(2), 403–430.Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006). Bayesianinference with probabilistic population codes. Nature Neuroscience, 9(11),1432–1438.Wednesday, August 29, 12

Advantages?• Optimal information integration requiresrepresentation of uncertainty–Does biological vision integrate local featureinformation based on uncertainty?• Indirect psychophysical evidence from local motionintegration in humans, cf. Weiss, Y., Simoncelli, E. P., &Adelson, E. H. (2002). Motion illusions as optimalpercepts. Nature Neuroscience, 5(6), 598–604.Wednesday, August 29, 1260

Primary visual cortex (V1)• Summary of overall functions:–Local coding, economical, emphasizes changes(edges)–binocular processing• front-end for stereopsis, eye of origin–early motion detection -- but only local, and thusambiguous–front-end for linking information likely to belong tosingle scene cause, e.g. single surface region orobject boundary (jobs of vision)Wednesday, August 29, 12