Learning Motor Skills - Intelligent Autonomous Systems

4 Policy Search for Motor Primitives in Robotics
Many motor skills in humanoid robotics can be learned using parametrized motor primitives. While
successful applications to date have been achieved with imitation learning, most of the interesting motor
learning problems are high-dimensional reinforcement learning problems. These problems are often
beyond the reach of current reinforcement learning methods. In this chapter, we study parametrized
policy search methods and apply these to benchmark problems of motor primitive learning in robotics. We
show that many well-known parametrized policy search methods can be derived from a general, common
framework. This framework yields both policy gradient methods and expectation-maximization (EM)
inspired algorithms. We introduce a novel EM-inspired algorithm for policy learning that is particularly
well-suited for dynamical system motor primitives. We compare this algorithm, both in simulation and on
a real robot, to several well-known parametrized policy search methods such as episodic REINFORCE,
‘Vanilla’ Policy Gradients with optimal baselines, episodic Natural Actor Critic, and episodic Reward-
Weighted Regression. We show that the proposed method out-performs them on an empirical benchmark
of learning dynamical system motor primitives both in simulation and on a real robot. We apply it in the
context of motor learning and show that it can learn a complex Ball-in-a-Cup task on a real Barrett WAM
robot arm.
4.1 Introduction
To date, most robots are still taught by a skilled human operator either via direct programming or
a teach-in. Learning approaches for automatic task acquisition and refinement would be a key step
for making robots progress towards autonomous behavior. Although imitation learning can make this
task more straightforward, it will always be limited by the observed demonstrations. For many motor
learning tasks, skill transfer by imitation learning is prohibitively hard given that the human teacher is
not capable of conveying sufficient task knowledge in the demonstration. In such cases, reinforcement
learning is often an alternative to a teacher’s presentation, or a means of improving upon it. In the highdimensional
domain of anthropomorphic robotics with its continuous states and actions, reinforcement
learning suffers particularly from the curse of dimensionality. However, by using a task-appropriate policy
representation and encoding prior knowledge into the system by imitation learning, local reinforcement
learning approaches are capable of dealing with the problems of this domain. Policy search (also known
as policy learning) is particularly well-suited in this context, as it allows the usage of domain-appropriate
pre-structured policies [Toussaint and Goerick, 2007], the straightforward integration of a teacher’s
presentation [Guenter et al., 2007, Peters and Schaal, 2006] as well as fast online learning [Bagnell
et al., 2004, Ng and Jordan, 2000, Hoffman et al., 2007]. Recently, policy search has become an accepted
alternative of value-function-based reinforcement learning [Bagnell et al., 2004, Strens and Moore,
2001, Kwee et al., 2001, Peshkin, 2001, El-Fakdi et al., 2006, Taylor et al., 2007] due to many of these
advantages.
In this chapter, we will introduce a policy search framework for episodic reinforcement learning and
show how it relates to policy gradient methods [Williams, 1992, Sutton et al., 1999, Lawrence et al.,
2003, Tedrake et al., 2004, Peters and Schaal, 2006] as well as expectation-maximization (EM) inspired
algorithms [Dayan and Hinton, 1997, Peters and Schaal, 2007]. This framework allows us to re-derive or
to generalize well-known approaches such as episodic REINFORCE [Williams, 1992], the policy gradient
theorem [Sutton et al., 1999, Peters and Schaal, 2006], the episodic Natural Actor Critic [Peters et al.,
2003, 2005], and an episodic generalization of the Reward-Weighted Regression [Peters and Schaal,
2007]. We derive a new algorithm called Policy Learning by Weighting Exploration with the Returns
(PoWER), which is particularly well-suited for the learning of trial-based tasks in motor control.
41