Algorithmic Differentiation in Python with Application Examples
Algorithmic Differentiation in Python with Application Examples
Algorithmic Differentiation in Python with Application Examples
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
Forward Mode vs Reverse Mode<br />
Task: compute Jacobian J = dF<br />
dx for F : RN → R M<br />
Forward Mode:<br />
J = dF<br />
dx · S ,<br />
where S = I ∈ R N×N .<br />
Reverse Mode:<br />
J = ¯S T · dF<br />
dx ,<br />
where ¯S ∈ R M×M .<br />
Gradient: The number of arithmetic operations (OPS) for the gradient<br />
evaluation ∇f (x) ∈ R N is only a small constant multiple of the OPS for<br />
the function f itself.<br />
Example: If f : R 2500 → R and runtime(f )=30 sec then SD/FD would<br />
require about 2500 ∗ 30 sec ≈ 21 hours but only a couple of m<strong>in</strong>utes<br />
us<strong>in</strong>g AD<br />
Mode Operations Memory<br />
Forward ∝ N OPS(F) MEM(J) N MEM(F)<br />
Reverse ∝ M OPS(F) MEM(J) ∝ OPS(F)<br />
Sebastian F. Walter, Humboldt-Universität zu Berl<strong>in</strong> <strong>Algorithmic</strong> () <strong>Differentiation</strong> <strong>in</strong> <strong>Python</strong> <strong>with</strong> <strong>Application</strong> <strong>Examples</strong> Wednesday, 10.07.2010 21 / 27
Change language