Toward Understanding the Visual System: the emergence of simple ...

informatik.uni.hamburg.de

Toward Understanding the Visual System: the emergence of simple ...

Toward Understanding the Visual System:

the emergence of simple cells in V1

Cornelius Weber

in collaboration with

Jochen Triesch

FIAS

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.1/51


Talk Contents

Visual System Biology

Models of Edge Detectors in V1

Weak Non-Linearities

Strong Non-Linearities

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.2/51


Visual System Biology

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.3/51


Visual System

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.4/51


Areas in the Visual System

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.5/51


Hierarchy of Visual Areas

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.6/51


Response Latencies of Visual Areas

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.7/51


Cortical Layers

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.8/51


Orientation Tuning in V1

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.9/51


Cortical Maps (V1)

Orientation Columns

p31

p42

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.10/51


Cortical Maps (V1)

Orientation Columns

p31

p42

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.10/51


Lateral Connections (V1)

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.11/51


Hubel & Wiesel 1960’s

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.12/51


Tilt Aftereffect (TAE)

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.13/51


TAE

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.14/51


TAE

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.15/51


TAE

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.16/51


TAE

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.17/51


TAE

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.18/51


TAE Data

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.19/51


Dynamic Responses

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.20/51


Tilt Illusion

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.21/51


Tilt Illusion

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.22/51


Models of Edge Detectors in V1

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.23/51


Localized Edge Detector

input ⃗x output z(y) = ey

1+e y

1

z

0.5

0

−4 −2 0 2 4

y

"net input"

y i = ∑ w

j

}{{} ij

weights

x j −

b i }{{}

threshold

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.24/51


Localized Edge Detector

input ⃗x output z(y) = ey

1+e y

1

z

0.5

0

−4 −2 0 2 4

y

"net input"

y i = ∑ w

j

}{{} ij

weights

x j −

b i }{{}

threshold

... to be learnt

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.24/51


Weak Non-Linearities – Sparseness

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.25/51


Statistics of Edges in Images

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.26/51


A Sparse Forward Model

Data examples ⃗x

p = probability each line present

p 2 = probability 2 lines co-occurring

Weights W bu

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.27/51


A Sparse Forward Model

Data examples ⃗x

p = probability each line present

p 2 = probability 2 lines co-occurring

Weights W bu

z

W bu

W lat

x

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.27/51


A Sparse Forward Model

Activation update

∆z i = g( ⃗w bu

i

·⃗x+ ⃗w lat

i ·⃗z −b i )

z i −→ 0 , 1

Thresholds

∆b i ≈ z i − p

Lateral weights

≈ p 2 − z i z j

∆w lat

ij

BU weights

∆ ⃗w i ≈ z i (⃗x − ⃗w i )

z

W bu

x

W lat

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.27/51


A Sparse Forward Model

Data examples ⃗x

Weights W bu

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.28/51


A Sparse Forward Model

Data examples ⃗x

Weights W bu

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.28/51


A Sparse Forward Model

Data examples ⃗x

Weights W bu

... should be:

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.28/51


Entropy Max and Generative Model

H(⃗z(⃗y(W bu ))) = MAX!

z

z

W bu

W td

x

x

(⃗x − W td ⃗z) 2 = MIN!

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.29/51


Entropy Max

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

z(y)

1

y

1

y

x

W

1

p (y)

p (y)


+

p (z)

p (z)


+

1

z

p (x)

+

bu

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.30/51


Generative Model

+

p (y)

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡

¡ ¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

¢

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

£

z(y)

1

y

1

y

x

1

p (y)


p (z)


1

z

p (x)

+

p (x)


W td Toward Understanding the Visual System:the emergence of simple cells in V1 – p.31/51


W is Unlearnt

z

1 1

z

-2 -1 0 1 2

y 1

z

y 2

y 2

2

Wx

y 1

-2 -1 0 1 2

z

2

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.32/51


W is Learnt

z

1 1

z

-2 -1 0 1 2

y

1

z

2

y

2

Wx

y

1

-2 -1 0 1 2

z

2

y

2

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.33/51


Reconstruction Error for Learning

y

Reconstruction error

˜x = ⃗x − W td ⃗y

An energy function

E(W, ⃗y) = 1 2 ˜x2

W bu

W td

Activation update

∆⃗y ≈ − ∂E

∂⃗y = W bu˜x

x

Learning

∆W td ≈ − ∂E = ˜x ⃗y T

∂W td

W bu = (W td ) T

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.34/51


Reconstruction Error for Learning

y

Reconstruction error

˜x = ⃗x − W td ⃗y

An energy function

E(W, ⃗y) = 1 2 ˜x2

W bu

W td

Activation update

∆⃗y ≈ − ∂E

∂⃗y = W bu˜x

x

Learning

∆W td ≈ − ∂E = ˜x ⃗y T

∂W td

W bu = (W td ) T

Linear model does not make any sense!

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.34/51


Sparse Distribution Requested

z = g(y)

!

f (z) ~ e −z

z

y = W x

x

~ td

x = W z

g(y) =

eay−b

1 + e ay−b

intrinsic plasticity parameters a, b

medium time course of parameter adaptation

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.35/51


Adaptation of a and b


∂t a i

= −η a


∂a i

KL(f z (z)‖ 1 µ e−z/µ )

= ...

= η a ( 1 a i

+ y i − 2y i z i − 1 µ y iz i + 1 µ y iz 2 i )


∂t b i

= −η b


∂b i

KL(f z (z)‖ 1 µ e−z/µ )

= ...

= η b (1 − 2z i − 1 µ z i + 1 µ z2 i )

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.36/51


Resulting Edge Detectors

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.37/51


TAE adaptation

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.38/51


TAE – how a and b adapt

0.2

decrease of parameter

0

−0.2

−0.4

−0.6

−0.8

a adapt

b adapt

−1

−90 −70 −50 −30 −10 10 30 50 70 90

orientation difference

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.39/51


TAE – Model-Generated

a adapt

b adapt

a,b adapt

5

TAE

1

0

-1

-5

-90

-60

-30

0

30

60

90

orientation difference

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.40/51


TAE – Model-Generated

a adapt

b adapt

a,b adapt

5

TAE

1

0

-1

-5

-90

-60

-30

0

30

60

90

orientation difference

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.40/51


TAE – Non-Saturating in Model

10

5

10

90

1x180

2x180

5x180

10x180

20x180

40x180

80x180

TAE

1

0

-1

-5

-90

-60

-30

0

30

60

90

orientation difference

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.41/51


Strong Non-Linearities – Competition

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.42/51


Hierarchical Linear Model

y

⃗y

2

2 = W 2 ⃗y 1

W 2

W 1

y

1

= W 2 W 1 ⃗x

x

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.43/51


Hierarchical Linear Model

W 1

y

⃗y 2

W ~ y1

= ˜W⃗x

2

= W 2 ⃗y 1

W 2

= W 2 W 1 ⃗x

x

makes no sense

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.43/51


Non-Linearities

Retinal input

V1 activation

linear sparse competitive winner

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.44/51


Non-Linearities

Retinal input

V1 activation

linear sparse competitive winner

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.44/51


Non-Linearities

Retinal input

V1 activation

linear sparse competitive winner

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.44/51


Setting up Competition

0.6

w i

lat

Weight profile

0.4

0.2

activation

Evolving activities

0

-0.2

i

cells

1

0.8

0.6

0.4

0.2

Activation update

˜z i (t + 1) = g( ⃗w i

lat · ˜z)

0

cells

time

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.45/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Other evidence for competition

retrieve stimulus parameter with max likelihood

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Functions of Competitive Non-Linearity

noise removal

categorization/discretization of continuous data

computing invariances (complex cells)

segmentation/binding – via (de-) synchronization

Other evidence for competition

retrieve stimulus parameter with max likelihood

contrast-invariant orientation tuning curve widths

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.46/51


Learning the Lateral Weights

W lat

z~

Activation initialization

˜z i (t 0 ) = z i (t 0 )

z = g(y)

Activation update

˜z i (t + 1) = f( ⃗w i

lat · ˜z(t))

W bu

W td

x

Lateral weights

∆wij

lat ≈

(z i (t end ) − ˜z i (t end )) · ˜z j (t end−1 )

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.47/51


W lat

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.48/51


W lat and W bu

lateral RF

BU RF’s 0

BU RF’s 2

Toward Understanding BUthe Visual RF’s System:the emergence 1 of simple cells in V1 – p.49/51


Results

100

0

Lateral weights

200

spatial frequency

orientation

-100

0 0.5 PI PI

0.8

0.6

0.4

Orientation tuning

1

10.0

8.0

6.0

4.0

2.0

0.2 0.0

0

−PI 90

0 PI 90

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.50/51


Results

100

0

Lateral weights

200

spatial frequency

orientation

-100

0 0.5 PI PI

0.8

0.6

0.4

Orientation tuning

1

10.0

8.0

6.0

4.0

2.0

0.2 0.0

0

−PI 90

0 PI 90

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.50/51


Outlook ...

Toward Understanding the Visual System:the emergence of simple cells in V1 – p.51/51

More magazines by this user
Similar magazines