Machine Learning - DISCo

CHAF'TER 12 COMBINING INDUCTIVE AND ANALYTICAL LEARNiNG 347
can specify certain derivatives of the target function in order to express our prior
knowledge that "the identity of the character is independent of small translations
and rotations of the image."
Below we describe the TANGENTPROP algorithm, which trains a neural network
to fit both training values and training derivatives. Section 12.4.4 then describes
how these training derivatives can be obtained from a domain theory
similar to the one used in the Cup example of Section 12.3. In particular, it
discusses how the EBNN algorithm constructs explanations of individual training
examples in order to extract training derivatives for use by TANGENTPROP.
TANGENTPROP and EBNN have been demonstrated to outperform purely inductive
methods in a variety of domains, including character and object recognition, and
robot perception and control tasks.
12.4.1 The TANGENTPROP Algorithm
TANGENTPROP (Simard et al. 1992) accommodates domain knowledge expressed
as derivatives of the target function with respect to transformations of its inputs.
Consider a learning task involving an instance space X and target function f. Up
to now we have assumed that each training example consists of a pair (xi, f (xi))
that describes some instance xi and its training value f (xi). The TANGENTPROP
algorithm assumes various training derivatives of the target function are also
provided. For example, if each instance xi is described by a single real value,
then each training example may be of the form (xi, f (xi), q lx, ). Here lx,
denotes the derivative of the target function f with respect to x, evaluated at the
point x = xi.
To develop an intuition for the benefits of providing training derivatives as
well as training values during learning, consider the simple learning task depicted
in Figure 12.5. The task is to learn the target function f shown in the leftmost plot
of the figure, based on the three training examples shown: (xl, f (xl)),(x2, f (x2)),
and (xg, f (xg)). Given these three training examples, the BACKPROPAGATION algorithm
can be expected to hypothesize a smooth function, such as the function g
depicted in the middle plot of the figure. The rightmost plot shows the effect of
FIGURE 12.5
Fitting values and derivatives with TANGENTPROP. Let f be the target function for which three examples
(XI, f (xi)), (x2, f (x2)), and (x3, f (x3)) are known. Based on these points the learner might
generate the hypothesis g. If the derivatives are also known, the learner can generalize more accurately
h.