Learning Motor Skills - Intelligent Autonomous Systems

2007], such a derivation results in a lower bound on the expected return using Jensen’s inequality and
the concavity of the logarithm. Thus, we obtain
log J(θ ′ ) = log
≥
¢
¢
p θ ′ (τ) R (τ) dτ = log
p θ (τ) R (τ) log p θ ′ (τ) dτ + const,
p θ (τ)
¢
p θ (τ)
p θ (τ) p θ ′ (τ) R (τ) dτ,
which is proportional to
where
−D p θ (τ) R (τ) ‖p θ ′ (τ) = L θ (θ ′ ),
D p (τ) ‖q (τ) =
¢
p (τ) log p (τ)
q (τ) dτ
denotes the Kullback-Leibler divergence, and the constant is needed for tightness of the bound. Note
that p θ (τ) R (τ) is an improper probability distribution as pointed out by Dayan and Hinton [1997]. The
policy improvement step is equivalent to maximizing the lower bound on the expected return L θ (θ ′ ), and
we will now show how it relates to previous policy learning methods.
Resulting Policy Updates
In this section, we will discuss three different policy updates, which are directly derived from the results
of Section 4.2.2. First, we show that policy gradients [Williams, 1992, Sutton et al., 1999, Lawrence et al.,
2003, Tedrake et al., 2004, Peters and Schaal, 2006] can be derived from the lower bound L θ (θ ′ ), which
is straightforward from a supervised learning perspective [Binder et al., 1997]. Subsequently, we show
that natural policy gradients [Bagnell and Schneider, 2003, Peters and Schaal, 2006] can be seen as an
additional constraint regularizing the change in the path distribution resulting from a policy update when
improving the policy incrementally. Finally, we will show how expectation-maximization (EM) algorithms
for policy learning can be generated.
Policy Gradients.
When differentiating the function L θ (θ ′ ) that defines the lower bound on the expected return, we
directly obtain
¡ ∑T
∂ θ ′ L θ (θ ′ ) =
p θ (τ)R(τ)∂ θ ′ log p θ ′(τ)dτ = E
t=1 ∂ θ ′ log π(a t|s t , t) R(τ) , (4.2)
where
∂ θ ′ log p θ ′ (τ) = ∑ T
t=1 ∂ θ ′ log π a t |s t , t
denotes the log-derivative of the path distribution. As this log-derivative depends only on the policy we
can estimate a gradient from rollouts, without having a model, by simply replacing the expectation by a
sum. When θ ′ is close to θ , we have the policy gradient estimator, which is widely known as episodic
REINFORCE [Williams, 1992]
lim θ ′ →θ ∂ θ ′ L θ (θ ′ ) = ∂ θ J(θ ).
See Algorithm 4.1 for an example implementation of this algorithm and Appendix 4.A.1 for the detailed
steps of the derivation. A MATLAB implementation of this algorithm is available at http://www.robotlearning.de/Member/JensKober.
44 4 Policy Search for Motor Primitives in Robotics