Machine Learning - DISCo

12.7 SUMMARY AND FURTHER READING
The main points of this chapter include:
0 Approximate prior knowledge, or domain theories, are available in many
practical learning problems. Purely inductive methods such as decision tree
induction and neural network BACKPROPAGATION fail to utilize such domain
theories, and therefore perform poorly when data is scarce. Purely analytical
learning methods such as PROLOG-EBG utilize such domain theories, but
produce incorrect hypotheses when given imperfect prior knowledge. Methods
that blend inductive and analytical learning can gain the benefits of both
approaches: reduced sample complexity and the ability to overrule incorrect
prior knowledge.
One way to view algorithms for combining inductive and analytical learning
is to consider how the domain theory affects the hypothesis space search.
In this chapter we examined methods that use imperfect domain theories to
(1) create the initial hypothesis in the search, (2) expand the set of search
operators that generate revisions to the current hypothesis, and (3) alter the
objective of the search.
A system that uses the domain theory to initialize the hypothesis is KBANN.
This algorithm uses a domain theory encoded as propositional rules to analytically
construct an artificial neural network that is equivalent to the domain
theory. This network is then inductively refined using the BACKPROPAGATION
algorithm, to improve its performance over the training data. The result is
a network biased by the original domain theory, whose weights are refined
inductively based on the training data.
TANGENTPROP uses prior knowledge represented by desired derivatives of
the target function. In some domains, such as image processing, this is
a natural way to express prior knowledge. TANGENTPROP incorporates this
knowledge by altering the objective function minimized by gradient descent
search through the space of possible hypotheses.
EBNN uses the domain theory to alter the objective in searching the hypothesis
space of possible weights for an artificial neural network. It uses
a domain theory consisting of previously learned neural networks to perform
a neural network analog to symbolic explanation-based learning. As in
symbolic explanation-based learning, the domain theory is used to explain
individual examples, yielding information about the relevance of different
example features. With this neural network representation, however, information
about relevance is expressed in the form of derivatives of the target
function value with respect to instance features. The network hypothesis is
trained using a variant of the TANGENTPROP algorithm, in which the error to
be minimized includes both the error in network output values and the error
in network derivatives obtained from explanations.
FOCL uses the domain theory to expand the set of candidates considered at
each step in the search. It uses an approximate domain theory represented