AD Tutorial at the TU Berlin

The Reverse Mode by Hand:
Recall: y = f (x) = sin(x 1 + cos(x 2 )x 1 )
independent v −1 = x 1 = 3
independent v 0 = x 2 = 7
v 1 = φ 1 (v 0 ) = cos(v 0 )
v 2 = φ 2 (v 1 , v −1 ) = v 1 v −1
v 3 = φ 3 (v −1 , v 2 ) = v −1 + v 2
v 4 = φ 4 (v 3 ) = sin(v 3 )
dependent y = v 4
Reverse Mode by Hand: Successive Pullbacks
dy = dφ 4 (v 3 ) = ∂φ 4(z)
∂z
˛ dv 3 = cos(v 3 )
˛z=v3
= ¯v 3 dφ 3 (v −1 , v 2 ) = ¯v 3 dv −1 + ¯v 3
|{z} |{z}
=¯v −1
= (¯v −1 + ¯v 2 v 1 ) dv −1 + ¯v 2 v −1 dv 1
| {z } | {z }
=¯v −1
=¯v 1
= ¯v −1 dv −1 + (−¯v 1 sin(v 0 )) dv 0
| {z }
=¯v 0
Interpretation: ¯v −1 ≡ df and ¯v dx 0 ≡ df
1 dx 2
Need to store v 0 , v 1 , v 3 , v 4 for the reverse mode!
dv 3
| {z }
=¯v 3
=¯v 2
dv 2
Sebastian F. Walter, HU Berlin () Not So Short Tutorial OnAlgorithmic Differentiation Wednesday, 04.06.2010 14 / 39