Machine Learning - DISCo

12.4.3 Remarks
To summarize, TANGENTPROP uses prior knowledge in the form of desired derivatives
of the target function with respect to transformations of its inputs. It combines
this prior knowledge with observed training data, by minimizing an objective function
that measures both the network's error with respect to the training example
values (fitting the data) and its error with respect to the desired derivatives (fitting
the prior knowledge). The value of p determines the degree to which the network
will fit one or the other of these two components in the total error. The behavior
of the algorithm is sensitive to p, which must be chosen by the designer.
Although TANGENTPROP succeeds in combining prior knowledge with training
data to guide learning of neural networks, it is not robust to errors in the prior
knowledge. Consider what will happen when prior knowledge is incorrect, that
is, when the training derivatives input to the learner do not correctly reflect the
derivatives of the true target function. In this case the algorithm will attempt to fit
incorrect derivatives. It may therefore generalize less accurately than if it ignored
this prior knowledge altogether and used the purely inductive BACKPROPAGATION
algorithm. If we knew in advance the degree of error in the training derivatives,
we might use this information to select the constant p that determines the relative
importance of fitting training values and fitting training derivatives. However, this
information is unlikely to be known in advance. In the next section we discuss
the EBNN algorithm, which automatically selects values for p on an example-byexample
basis in order to address the possibility of incorrect prior knowledge.
It is interesting to compare the search through hypothesis space (weight
space) performed by TANGENTPROP, KBANN, and BACKPROPAGATION. TANGENT-
PROP incorporates prior knowledge to influence the hypothesis search by altering
the objective function to be minimized by gradient descent. This corresponds to
altering the goal of the hypothesis space search, as illustrated in Figure 12.6. Like
BACKPROPAGATION (but unlike KBANN), TANGENTPROP begins the search with an
initial network of small random weights. However, the gradient descent training
rule produces different weight updates than BACKPROPAGATION, resulting in a different
final hypothesis. As shown in the figure, the set of hypotheses that minimizes
the TANGENTPROP objective may differ from the set that minimizes the BACKPROP-
AGATION objective. Importantly, if the training examples and prior knowledge are
both correct, and the target function can be accurately represented by the ANN,
then the set of weight vectors that satisfy the TANGENTPROP objective will be a
subset of those satisfying the weaker BACKPROPAGATION objective. The difference
between these two sets of final hypotheses is the set of incorrect hypotheses that
will be considered by BACKPROPAGATION, but ruled out by TANGENTPROP due to
its prior knowledge.
Note one alternative to fitting the training derivatives of the target function
is to simply synthesize additional training examples near the observed training
examples, using the known training derivatives to estimate training values for
these nearby instances. For example, one could take a training image in the above
character recognition task, translate it a small amount, and assert that the trans-