Machine Learning - DISCo

additional detail that must be considered: There are many alternative sets of Horn
clauses entailed by the domain theory. The remaining component of the inductive
bias is therefore the basis by which PROLOG-EBG chooses among these alternative
sets of Horn clauses. As we saw above, PROLOG-EBG employs a sequential covering
algorithm that continues to formulate additional Horn clauses until all positive
training examples have been covered. Furthermore, each individual Horn clause
is the most general clause (weakest preimage) licensed by the explanation of the
current training example. Therefore, among the sets of Horn clauses entailed by
the domain theory, we can characterize the bias of PROLOG-EBG as a preference
for small sets of maximally general Horn clauses. In fact, the greedy algorithm of
PROLOG-EBG is only a heuristic approximation to the exhaustive search algorithm
that would be required to find the truly shortest set of maximally general Horn
clauses. Nevertheless, the inductive bias of PROLOG-EBG can be approximately
characterized in this fashion.
Approximate inductive bias of PROLOG-EBG: The domain theory B, plus a preference
for small sets of maximally general Horn clauses.
The most important point here is that the inductive bias of PROLOG-EBGthe
policy by which it generalizes beyond the training data-is largely determined
by the input domain theory. This lies in stark contrast to most of the other learning
algorithms we have discussed (e.g., neural networks, decision tree learning), in
which the inductive bias is a fixed property of the learning algorithm, typically
determined by the syntax of its hypothesis representation. Why is it important
that the inductive bias be an input parameter rather than a fixed property of the
learner? Because, as we have discussed in Chapter 2 and elsewhere, there is
no universally effective inductive bias and because bias-free learning is futile.
Therefore, any attempt to develop a general-purpose learning method must at
minimum allow the inductive bias to vary with the learning problem at hand.
On a more practical level, in many tasks it is quite natural to input domainspecific
knowledge (e.g., the knowledge about Weight in the SafeToStack example)
to influence how the learner will generalize beyond the training data.
In contrast, it is less natural to "implement" an appropriate bias by restricting
the syntactic form of the hypotheses (e.g., prefer short decision trees). Finally,
if we consider the larger issue of how an autonomous agent may improve its
learning capabilities over time, then it is attractive to have a learning algorithm
whose generalization capabilities improve as it acquires more knowledge of its
domain.
11.3.4 Knowledge Level Learning
As pointed out in Equation (11.2), the hypothesis h output by PROLOG-EBG follows
deductively from the domain theory B and training data D. In fact, by examining
the PROLOG-EBG algorithm it is easy to see that h follows directly from B alone,
independent of D. One way to see this is to imagine an algorithm that we might