Algorithmic Differentiation in Python with Application Examples

Algorithmic Differentiation in Python with 

Application Examples 

Sebastian F. Walter, Humboldt-Universität zu Berlin 

Wednesday, 10.07.2010 

Sebastian F. Walter, Humboldt-Universität zu Berlin Algorithmic () Differentiation in Python with Application ExamplesWednesday, 10.07.2010 1 / 27

Part I: Intro to Algorithmic Diff. 

Comparison to Symbolic/Numerical Differentiation 

The Forward Mode by Taylor Arithmetic 

The Reverse Mode 

Part II: Advanced Application Examples 

Differentiation of Differential Equations 

Differentiation of Numerical Linear Algebra Functions 

Optimum Experimental Design 

Standard Reference: 

Griewank, Evaluating Derivatives 

x(t; p) 

4 

3 

2 

1 

0 

−1 

−2 

−3 

−4 

0 1 2 3 4 5 

t 

10 

control func. u(t) 

8 state x(t) 

dx/dp1(t) 

6 

dx/dp2(t) 

x1(t; p = 1.0) 

x2(t; p = 1.0) 

dx1/dp(t; p = 1.0) 

dx2/dp(t; p = 1.0) 

h(t, x, p) 

dh/dp(t, x, p) 

d 2 x1/dp 2 (t; p = 1.0) 

d 2 x2/dp 2 (t; p = 1.0) 

state x(t) 

4 

2 

Research Community Website: 

www.autodiff.org 

0 

−2 

−4 

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 

time t [sec] 


What is Algorithmic Differentiation (AD) 

Name confusion: Algorithmic Differentiation aka Automatic 

Differentiation aka Computational Differentiation aka AD 

Considered one of the most important algorithmic techniques “invented” 

in the 20’th century 1 

Can be used to differentiate large-scale problems, e.g. in PDE 

constrained optimization. 

Generally much more efficient than symbolic/numerical differentiation 

and also accurate close to machine precision 

1 Nick Trefethen, http://www.comlab.ox.ac.uk/nick.trefethen/inventorstalk.pdf 


Software Used in this Talk: 

Name Description Status LOC 

algopy forward/reverse UTPM in Python alpha 10388 

www.github.com/b45ch1/algopy 

pysolvind Python Bindings to SolvIND/DAESOL-II alpha 9743 

pyadolc Python Bindings to ADOL-C (C++) stable 6895 

www.github.com/b45ch1/pyadolc 

pycppad Python Bindings to CppAD (C++ ) stable 1334 

www.github.com/b45ch1/pycppad 

taylorpoly ANSI-C with Python bindings alpha 9276 

www.github.com/b45ch1/taylorpoly 

easyodoe Opt. Exp. prototype alpha 8345 


Why not Symbolic Differentiation 

A Raytracing Example 

1.0 

0.5 

0.0 8 

6 

−0.5 

1 

3 

5 

7 

9 

Cylindrical mirror described by 0 = g(x) = x 2 1 + x2 2 − 1, 

laser beam enters at x = (0, −1) and direction v. 

Recursive algorithm for next reflection point x + and 

direction v + : 

„ x + 

v + « 

0 

B 

= F(x, v) = @ x + 

r ! 

“ ” x T 2 

v ‖x‖ 

‖v‖ − 2 −1 

2 ‖v‖ 2 − xT v 

‖v‖ 2 

P(x + )v 

4 

2 

0 

−1.0 

−1.0 −0.5 0.0 0.5 1.0 

where P(x + ) = I − 2 wwT 

‖w‖ 2 , w = w(x + ) = ∇ xg(x + ) 

Goal: compute sensitivity of the 10’th reflection point x (10) w.r.t. initial 

direction v (0) , i.e. dx(10) 

dv (0) . 


Why not Symbolic Differentiation (cont) 

Compute recursively x (k) , v (k) = F(x (k−1) , v (k−1) ) as symbolic 

expression 

Use sum,product and chainrule to compute the wanted derivative 

Problem: Expression swell (show live example) 

import p y l a b ; import numpy ; from numpy import s q r t , dot , cos , s i n , pi , l i n 

import sympy ; from sympy import s q r t 

def F ( x , v ) : 

””” computes next r e f l e c t i o n p o i n t x and d i r e c t i o n v ””” 

c = d o t ( v , v ) 

x2 = [ x [ 0 ] + v [ 0 ] ∗ ( s q r t ( ( d o t ( x , v ) / c )∗∗2 −( d o t ( x , x ) − 1 . ) / c)− d o t ( x , v ) / c ) , 

x [ 1 ] + v [ 1 ] ∗ ( s q r t ( ( d o t ( x , v ) / c )∗∗2 −( d o t ( x , x ) − 1 . ) / c)− d o t ( x , v ) / c ) ] 

w = x2 

v2 = [ ( v [ 0 ] − 2∗ w[ 0 ] ∗ d o t (w, v ) / d o t (w,w) ) , 

( v [ 1 ] − 2∗ w[ 1 ] ∗ d o t (w, v ) / d o t (w,w ) ) ] 

return x2 , v2 

x1 , x2 , v1 , v2 = sympy . symbols ( ’ x1 ’ , ’ x2 ’ , ’ v1 ’ , ’ v2 ’ ) 

x = [ x1 , x2 ] ; v = [ v1 , v2 ] 

x , v = F ( x , v ) 

p r i n t ’x , v=\n ’ , x , v 

#x , v = F ( x , v ) 

Sebastian # p rF. i Walter, n t ’x Humboldt-Universität , v=\n ’ , x , v zu Berlin Algorithmic () Differentiation in Python with Application ExamplesWednesday, 10.07.2010 6 / 27

Why not Finite Differences 

Problem: Finite Precision Arithmetic 

f (x) true value, ˜f (x) numerically computed value (assume x = ˜x) 

d˜f (x; v) = ˜f (x + tv) − ˜f (x) 

= f (x + tv) + δ 1 − f (x) + δ 2 

t 

t 

= 

f (x + tv) − f (x) 

+ δ 1 + δ 2 

t 

t 

= 

r(x; tv) 

df (x; v) − + δ 1 + δ 2 

, 

t t 

where δ 1 and δ 2 random errors due to finite precision arithmetic 

Difference numerical and true derivative: 

d˜f (x; v) − df (x; v) = 

r(x; tv) 

− δ 1 + δ 2 

, 

} {{ 

t 

} } {{ 

t 

} 

Question: What is the best t ∈ R 

t→0 

→ 0 

t→0 

→ ∞ 

if f ∈ C 2 (R), then r(x; tv)/t = f ′′ (ξ)t and therefore t = 

√ 

δ1 +δ 2 

f ′′ (ξ) 


Why not Finite Differences (cont.) 

absolute FD error 

10 33 

10 30 

10 27 

10 24 

10 21 

10 18 

10 15 

10 12 

10 9 

10 6 

10 3 

test function: f(x) = 1 + sin(x), x = 1 

FD 1st order 

FD 2nd order 

FD 3rd order 

10 0 

10 −3 

10 −6 

10 −9 

10 −16 10 −14 10 −12 10 −10 10 −8 10 −6 10 −4 10 −2 10 0 

step width t 

machine EPS ≈ 10 −16 for 64bit IEEE-754 floats 

higher-order derivatives by FD quickly get to large 

best t is not known a priory and often has to be guessed by careful tests 


Part I: 

Intro Algorithmic Differentiation 


Computational Model and the Evaluation Trace 

All computer programs are a sequence of elementary functions 

φ l ∈ {+, −, ∗, /, sin, exp, . . . } 

Symbolic dependency is resolved at each elementary function: 

pushforward of numerical values v j≺l 

Example: Evaluate Function f (3, 7): 

f : R 2 → R 

x ↦→ y = f (x) = sin(x 1 +cos(x 2 )∗x 1 ) 

Computational Graph: 

Computational Trace: 

1 Id 

independent v −1 = x 1 = 3 

independent v 0 = x 2 = 7 

v 1 = φ 1 (v 0 ) = cos(v 0 ) 

2 cos 

v 2 = φ 2 (v 1 , v −1 ) = v 1 v −1 

v 3 = φ 3 (v −1 , v 2 ) = v −1 + v 2 

3 __mul__ 

v 4 = φ 4 (v 3 ) = sin(v 3 ) 

4 __add__ 

5 sin 

0 Id 

Sebastian F. Walter, Humboldt-Universität zu Berlin Algorithmic () Differentiation in Python with Application Examples Wednesday, 10.07.2010 10 / 27

Code Tracing with PYADOLC and ALGOPY 

import a d o l c ; import a l gopy ; import numpy ; from numpy import s i n , cos ; 

def f ( x ) : 

return s i n ( x [ 0 ] + cos ( x [ 1 ] ) ∗ x [ 0 ] ) 

a d o l c . t r a c e o n ( 1 ) 

x = a d o l c . a d o u b l e ( [ 3 , 7 ] ) ; a d o l c . i n d e p e n d e n t ( x ) 

y = f ( x ) 

a d o l c . d e p e n d e n t ( y ) ; a d o l c . t r a c e o f f ( ) 

a d o l c . t a p e t o l a t e x ( 1 , [ 3 , 7 ] , [ 0 . ] ) 

cg = a l g o p y . CGraph ( ) 

x = [ a l g o p y . F u n c t i o n ( 3 . ) , a l g o p y . F u n c t i o n ( 7 . ) ] 

y = f ( x ) 

cg . t r a c e o f f ( ) 

cg . i n d e p e n d e n t F u n c t i o n L i s t = [ x [ 0 ] , x [ 1 ] ] ; cg . d e p e n d e n t F u n c t i o n L i s t = [ y ] 

cg . p l o t ( ’ c g r a p h s i m p l e f u n c t i o n . svg ’ ) 

code op loc loc loc loc double double value value val 

33 start of tape 

39 take stock op 2 0 3.000000e + 00 n 

1 assign ind 0 3.000000e + 00 

1 assign ind 1 7.000000e + 00 

20 cos op 1 3 2 7.000000e + 00 6.569866e − 

15 mult a a 2 0 3 7.539023e − 01 3.000000e + 

11 plus a a 0 3 4 3.000000e + 00 2.261707e + 

21 sin op 4 6 5 5.261707e + 00 5.221055e − 

2 assign dep 5 

0 death not 0 6 −8.528809e − 

Sebastian32 F. Walter, endHumboldt-Universität of tape 

zu Berlin Algorithmic () Differentiation in Python with Application Examples Wednesday, 10.07.2010 11 / 27

PART I.1: 

The Forward Mode of AD by 

Univariate Taylor Polynomial (UTP) Arithmetic 


Univariate Taylor Polynomial Arithmetic (UTP) 

Basic Observation 1: Let f : R N → R, then 

d 

dt f (x + e it) 

∣ = (∇ x f (x)) T · e i = ∂f 

t=0 

∂x i 

Basic Observation 2: Hessian 

d 2 

f (x + e i t 1 + e j t 2 ) ∣ 

dt 1 dt 2 

∣ 

t1 =t 2 =0 

= e T i ∇ 2 xf (x)e j = ∂2 f 

∂x i ∂x j 

e i = (0, . . . , 1, . . . , 0) is the i’th cartesian basis vector. 


Univariate Taylor Polynomial Arithmetic (UTP) (cont.) 

Problem can be formulated as arithmetic on univariate Taylor 

polynomials (UTP) 

D−1 

∑ 

[x] D = [x 0 , . . . , x D−1 ] = x d T d ∈ R(T)/(T D ) , 

d=0 

T is an indeterminate, i.e. a formal parameter 

x d ∈ R is called Taylor coefficient 

Define extension of Functions f : R → R, y = f (x): 

E D (f ) : R[T]/(T D ) → R[T]/(T D ) 

[x] D ↦→ [y] D := ∑ 1 d d D−1 

d! dt d f ( ∑ 

x d t d ) 

T d , 

∣ 

d=0 

k=0 

} {{ t=0 

} 

≡y d 


Univariate Taylor Polynomial Arithmetic (UTP) (cont.) 

Let f (x) = (h ◦ g)(x) = h(g(x)) be a composite function, then 

E D (f ) = E D (h) ◦ E D (g) . 

I.e. E D is a homomorphism that preserves the function composition. 

Therefore: Need algorithms to compute 

[y 0 , . . . , y D−1 ] = E D (φ)([x 0 , . . . , x D−1 ]) 

only for the elementary functions φ ∈ {+, −, ∗, /, . . . } ! 

Suggests implementation by function and operator overloading, i.e. 

univariate Taylor polynomial (UTP) arithmetic. 


Algorithms for Univariate Taylor Polynomials over Scalars (UTPS) 

binary operations 

unary operations 

z = φ(x, y) d = 0, . . . , D OPS MOVES 

x + cy z d = x d + cy d 2D 3D 

x × y z d = P d 

k=0 h 

x ky d−k D 2 3D 

x/y z d = 1 x y d − P i 

d−1 

0 k=0 z ky d−k D 2 3D 

y = φ(x) d = 0, . . . , D OPS MOVES 

h 

ln(x) ỹ d = 1 ˜x x d − P i 

d−1 

0 k=1 x d−kỹ k D 2 2D 

exp(x) ỹ d = P d 

k=1 y d−k˜x k D 2 2D 

√ h 

x yd = 1 x 2y d − P i 

d−1 

0 k=1 y 1 

ky d−k 2 D2 3D 

h 

x r ỹ d = 1 r P d 

x 0 k=1 y d−k˜x k − P i 

d−1 

k=1 x d−kỹ k 2D 2 2D 

sin(v) ˜s d = P d 

j=1 ṽjc d−j 2D 2 3D 

cos(v) ˜c d = P d 

j=1 −ṽ js d−j 

tan(v) ˜φd = P d 

j=1 w d−jṽ j 

˜w d = 2 P d 

j=1 φ d−j “ 

˜φ j 

arcsin(v) ˜φd = w −1 

0 ṽ d − P d−1 

j=1 w d−j ˜φ 

” 

j 

˜w d = − P d 

j=1 v d−j “ 

˜φ j 

arctan(v) ˜φd = w −1 

0 ṽ d − P d−1 

j=1 w d−j ˜φ 

” 

j 

˜w d = 2 P d 

j=1 v d−jṽ j 


Live Example: Directional Derivatives using TAYLORPOLY 

Interpretation: extract derivatives from Taylor coefficients 

if [x] D = [x 0 , 1, 0, 0, . . . , 0], then 

y d = 1 d d D−1 

d! dt d f ( ∑ 

x d t d ) 

= dd f 

∣ dx d (x 0)1 , 

t=0 

Example: f : R 2 → R 

k=0 

x ↦→ y = f (x) = sin(x 1 + cos(x 2 )x 1 ) 

(( ) ( )∣ 

Compute df 

dx 1 

(3, 7) = d 3 1 ∣∣∣t=0 

dt f + t 

7 0) 

import numpy ; from numpy import s i n , cos ; from t a y l o r p o l y import UTPS 

def f ( x ) : 

return s i n ( x [ 0 ] + cos ( x [ 1 ] ) ∗ x [ 0 ] ) + x [ 1 ] ∗ x [ 0 ] 

x = [ UTPS ( [ 3 , 1 ] ) , UTPS ( [ 7 , 0 ] ) ] 

y = f ( x ) 

p r i n t ’ normal f u n c t i o n e v a l u a t i o n y 0 = f ( x 0 ) = ’ , y . d a t a [ 0 ] 

p r i n t ’ g r a d i e n t e v a l u a t i o n df / dx 1 = ’ , y . d a t a [ 1 ] 


PART I.2: 

The Reverse Mode of AD 


The Reverse Mode by Hand: 

Recall: y = f (x) = sin(x 1 + cos(x 2 )x 1 ) 

independent v −1 = x 1 = 3 

independent v 0 = x 2 = 7 

v 1 = φ 1 (v 0 ) = cos(v 0 ) 

v 2 = φ 2 (v 1 , v −1 ) = v 1 v −1 

v 3 = φ 3 (v −1 , v 2 ) = v −1 + v 2 

v 4 = φ 4 (v 3 ) = sin(v 3 ) 

dependent y = v 4 

Reverse Mode by Hand: Successive Pullbacks 

dy = dφ 4 (v 3 ) = ∂φ 4(z) 

∂z 

˛ dv 3 = cos(v 3 ) 

˛z=v3 

= ¯v 3 dφ 3 (v −1 , v 2 ) = ¯v 3 dv −1 + ¯v 3 

|{z} |{z} 

=¯v −1 

= (¯v −1 + ¯v 2 v 1 ) dv −1 + ¯v 2 v −1 dv 1 

| {z } | {z } 

=¯v −1 

=¯v 1 

= ¯v −1 dv −1 + (−¯v 1 sin(v 0 )) dv 0 

| {z } 

=¯v 0 

Interpretation: ¯v −1 ≡ df and ¯v dx 0 ≡ df 

1 dx 2 

Need to store v 0 , v 1 , v 3 , v 4 for the reverse mode! 

dv 3 

| {z } 

=¯v 3 

=¯v 2 

dv 2 


Semi-Automatic Forward/Reverse Mode by Manual Tracing 

import numpy ; from numpy import s i n , cos ; from t a y l o r p o l y import UTPS 

x1 = UTPS ( [ 3 , 1 , 0 ] , P = 2 ) ; x2 = UTPS ( [ 7 , 0 , 1 ] , P=2) 

# forward mode 

vm1 = x1 ; v0 = x2 

v1 = cos ( v0 ) 

v2 = v1 ∗ vm1 

v3 = vm1 + v2 

y = v4 = s i n ( v3 ) 

# r e v e r s e mode 

v4bar = UTPS ( [ 0 , 0 , 0 ] , P = 2 ) ; v3bar = UTPS ( [ 0 , 0 , 0 ] , P=2) 

v2bar = UTPS ( [ 0 , 0 , 0 ] , P = 2 ) ; v1bar = UTPS ( [ 0 , 0 , 0 ] , P=2) 

v0bar = UTPS ( [ 0 , 0 , 0 ] , P = 2 ) ; vm1bar = UTPS ( [ 0 , 0 , 0 ] , P=2) 

v4bar . d a t a [ 0 ] = 1 . 

v3bar += v4bar ∗ cos ( v3 ) 

vm1bar += v3bar ; v2bar += v3bar 

v1bar += v2bar ∗ vm1 ; vm1bar += v2bar ∗ v1 

v0bar −= v1bar ∗ s i n ( v0 ) 

g1 = y . d a t a [ 1 : ] ; g2 = numpy . a r r a y ( [ vm1bar . d a t a [ 0 ] , v0bar . d a t a [ 0 ] ] ) 

p r i n t ’ f o r w a r d g r a d i e n t g ( x 0 )=\ n ’ , g1 , ’\ n r e v e r s e g r a d i e n t g ( x 0 )=\ n ’ , g2 

p r i n t ’ H e s s i a n H( x 0 )=\ n ’ , numpy . v s t a c k ( [ vm1bar . d a t a [ 1 : ] , v0bar . d a t a [ 1 : ] ] ) 

can automatize this using a code tracer or source code transformation, 

e.g. with PYADOLC or ALGOPY 


Forward Mode vs Reverse Mode 

Task: compute Jacobian J = dF 

dx for F : RN → R M 

Forward Mode: 

J = dF 

dx · S , 

where S = I ∈ R N×N . 

Reverse Mode: 

J = ¯S T · dF 

dx , 

where ¯S ∈ R M×M . 

Gradient: The number of arithmetic operations (OPS) for the gradient 

evaluation ∇f (x) ∈ R N is only a small constant multiple of the OPS for 

the function f itself. 

Example: If f : R 2500 → R and runtime(f )=30 sec then SD/FD would 

require about 2500 ∗ 30 sec ≈ 21 hours but only a couple of minutes 

using AD 

Mode Operations Memory 

Forward ∝ N OPS(F) MEM(J) N MEM(F) 

Reverse ∝ M OPS(F) MEM(J) ∝ OPS(F) 


Minimal Surface problem with PYADOLC 

Example where the Reverse Mode Excels 

(discretized) objective function: 

u : [0, 1] 2 → R , u ∈ C 1 

s 

Z 1 Z 1 

u ↦→ 

1 + 

0 0 

≈ 

m−1 X m−1 X 

O ij (u) 

„ « ∂u 2 

+ 

∂x 

„ « ∂u 2 

∂y 

i=0 j=0 

" 

Õ ij (u) := h 2 1 + (u # 

i+1,j+1 − u i,j ) 2 + (u i,j+1 − u i+1,j ) 2 

4 

Nonlinear Program with box constraints: 

u ∗ ∈ R m×m = argmin u Õ(u) 

therefore ∇ u Õ(u) ∈ R m×m , e.g. m = 50 

yields a gradient with 2500 elements ⇒ use 

reverse mode 


A Minimal Surface Problem 

part of unit test of pyadolc: /pyadolc/tests/complicated tests.py 

import a d o l c ; import numpy ; 

def O t i l d e ( u ) : 

””” o b j e c t i v e f u n c t i o n of the minimal s u r f a c e problem ””” 

M = numpy . shape ( u ) [ 0 ] 

h = 1 . / (M−1) 

return M∗∗2∗h∗∗2+numpy . sum ( 0 . 2 5 ∗ ( ( u [ 1 : , 1 : ] − u [0: −1 ,0: −1])∗∗2+( u [ 1 : , 0 : − 1 

M = 5 0 ; h = 1 . /M; u = numpy . z e r o s ( (M,M) , d t y p e = f l o a t ) 

u [ 0 , : ] = [ numpy . s i n ( numpy . p i ∗ j ∗h / 2 . ) f o r j in r a n g e (M) ] 

u [ −1 ,:] = [ numpy . exp ( numpy . p i / 2 ) ∗ numpy . s i n ( numpy . p i ∗ j ∗ h / 2 . ) f o r j 

u [ : , 0 ] = 0 

u [: , −1]= [ numpy . exp ( i ∗h∗numpy . p i / 2 . ) f o r i in r a n g e (M) ] 

# t r a c e the o b j e c t i v e f u n c t i o n 

a d o l c . t r a c e o n ( 1 ) 

au = a d o l c . a d o u b l e ( u ) 

a d o l c . i n d e p e n d e n t ( au ) 

ay = O t i l d e ( au ) 

a d o l c . d e p e n d e n t ( ay ) 

a d o l c . t r a c e o f f ( ) 

# compute g r a d i e n t 

g AD = a d o l c . g r a d i e n t ( 1 , numpy . r a v e l ( u ) ) . r e s h a p e ( numpy . shape ( u ) ) 

g AD [ : , 0 ] = 0 ; g AD [ 0 , : ] = 0 ; g AD [: , −1] = 0 ; g AD [ −1 ,:] = 0 # on the ed 

# compute dot ( Hessian , v ) , v random v e c t o r 

Hv AD = a d o l c . h e s s v e c ( 1 , numpy . r a v e l ( u ) , numpy . random . r and ( u . s i z e ) ) 


PART II: 

Advanced Applications Examples 


Optimum Experimental Design in Chemical Engineering 

Tetramethyl 

Cyclohexadien 

+ 

k2 + Cat 

Pi−Complex + 

Cat 

λ 

Maleinacid 

Anhydrid 

Maleinacid 

Anhydrid 

Deactiv. Cat 

k1 

k3 

Diels−Alder 

Product 

− Cat 

non-catalyzed and catalyzed reaction path 

deactivation of the catalyst 

batch process 

measurements: product mass concentration 

control of educt molar numbers, catalyst 

concentration, temperature profile 

five unknown model parameters 

ṅ 1 = −k · n1 · n 2 

m tot 

, n 1 (0) = n a1 

ṅ 2 = −k · n1 · n 2 

m tot 

, n 2 (0) = n a2 

ṅ 3 = k · n1 · n 2 

m tot 

, n 3 (0) = 0 

k = k 1 · exp − E 1 

R · 

1 

T − 1 

!! 

T ref 

+ k kat · c kat · exp (−λ · t) · exp − E kat 

R · 

n 4 = n a4 T = ϑ + 273 

m tot = n 1 · M 1 + n 2 · M 2 + n 3 · M 3 + n 4 · M 4 

1 

T − 1 

!! 

T ref 


Objective Function of Opt. Exp. Design 

Part I: Computation of J 1 and J 2 

J 1 [n mts, :] = 

√ wmts 

d 

(h(tnmts, x(tnmts; s, u(tnmts; q), p))) 

σ nmts (x(t nmts ; s, u(t nmts ; q), q) d(p, s) 

J 2 = 

d 

r(q, p, s) 

d(p, s) 

Part II: Numerical Linear Algebra 

„ 

J T 

C(J 1 , J 2 ) = (I, 0) 1 J 1 J2 

T « −1 „ « I 

J 2 0 0 

” 

= 

“Q T 2 (Q 2J1 T J 1Q T 2 )−1 Q 2 

Φ = λ 1 (C) , max. eigenvalue 

where J2 T = (QT 1 , QT 2 )(L, 0)T 

Computational Graph 

[p] 

[h], [r] [J 1 ], [J 2 ] [C] [Φ] 

[q] 

[s] [x 0 ] [x 1 ] [x 2 ] [x 3 ] [x 4 ] . . . [x N mts−1] [x N mts ] 

statex(t) atmeasurementtimes (mts) 

independent/dependent variables 

N mts Number measurement times, w measurement weight, σ std of a measurement, q controls, p nature 

Sebastian givenF. Walter, parameter, Humboldt-Universität s pseudo-Parameter zu Berlin Algorithmic ()(e.g. initial Differentiation values), in uPython control with functions Application Examples Wednesday, 10.07.2010 26 / 27

Algorithm: Forward UTPM of the Rectangular QR Decomposition 

input : [A] D = [A 0 , . . . , A D− 1], where A d ∈ R M×N , d = 0, . . . , D − 1, M ≥ N. 

output: [Q] D = [Q 0 , . . . , Q D− 1] matrix with orthonormal column vectors, where Q d ∈ R M×N , 

d = 0, . . . , D − 1 

output: [R] D = [R 0 , . . . , R D− 1] upper triangular, where R d ∈ R N×N , d = 0, . . . , D − 1 

Q 0 , R 0 = qr (A 0 ) 

for d = 1 to D − 1 do 

∆F = A d − P d−1 

k=1 Q d−kR k 

S = − 1 P d−1 

2 k=1 QT d−k Q k 

P L ◦ X = P L ◦ (Q T 0 ∆FR−1 0 − S) 

X = P L ◦ X − (P L ◦ X) T 

R d = Q T 0 ∆F − (S + X)R 0 

Q d = (∆F − Q 0 R d )R −1 

0 

end 


Example for Differentiation of Numerical Linear Algebra 

Compute directional derivatives of the largest eigenvalue, 

„ 

J T 

∇ qλ max (I, 0) 1 J 1 J2 

T « −1 „ « ! I 

. 

J 2 0 0 

import numpy 

from a l g o p y import CGraph , F unction , UTPM, dot , inv , z e r o s , e i g h 

def P h i f c n (C ) : 

””” return max e i g e n v a l u e ””” 

return e i g h (C) [ 0 ] [ − 1 ] 

def Cfcn ( J1 , J2 ) : 

””” compute c o v a r i a n c e matrix ””” 

Np = J1 . shape [ 1 ] ; Nr = J2 . shape [ 0 ] 

tmp = z e r o s ( ( Np+Nr , Np+Nr ) , d t y p e =J1 ) 

tmp [ : Np , : Np ] = d o t ( J1 . T , J1 ) 

tmp [ Np : , : Np ] = J2 

tmp [ : Np , Np : ] = J2 . T 

return i n v ( tmp ) [ : Np , : Np ] 

D, P ,Nm, Np , Nr = 2 , 1 , 2 0 0 0 , 6 , 3 

cg = CGraph ( ) 

J1 = F u n c t i o n (UTPM( numpy . random . r and (D, P ,Nm, Np ) ) ) 

J2 = F u n c t i o n (UTPM( numpy . random . r and (D, P , Nr , Np ) ) ) 

Phi = P h i f c n ( Cfcn ( J1 , J2 ) ) 

cg . i n d e p e n d e n t F u n c t i o n L i s t = [ J1 , J2 ] ; cg . d e p e n d e n t F u n c t i o n L i s t = [ Phi ] 

p r i n t ’ o b j e c t i v e f u n c t i o n Phi =\n ’ , Phi 

Sebastian # cg F. . Walter, p l o t Humboldt-Universität ( ’ odoe cgraph zu Berlin . svg Algorithmic () ’ ) Differentiation in Python with Application Examples Wednesday, 10.07.2010 28 / 27


Summary: Software Used in this Talk: 

Name Description Status LOC 

algopy forward/reverse UTPM in Python alpha 10388 

www.github.com/b45ch1/algopy 

pysolvind Python Bindings to SolvIND/DAESOL-II alpha 9743 

pyadolc Python Bindings to ADOL-C (C++) stable 6895 

www.github.com/b45ch1/pyadolc 

pycppad Python Bindings to CppAD (C++ ) stable 1334 

www.github.com/b45ch1/pycppad 

taylorpoly ANSI-C with Python bindings alpha 9276 

www.github.com/b45ch1/taylorpoly 

easyodoe Opt. Exp. prototype alpha 8345 

API is fairly well documented, about 30% of the LOCs 

quite complete unit test and many examples (included in LOCs) 

ready to be used!

Algorithmic Differentiation in Python with Application Examples

Create successful ePaper yourself

Delete template?

Save as template?