Machine Learning - DISCo

each hypothesis h(x) is of the form
(a) Derive a gradient descent rule that minimizes the same criterion as BACKPROP-
AGATION; that is, the sum of squared errors between the hypothesis and target
values of the training data.
(b) Derive a second gradient descent rule that minimizes the same criterion as
TANGENTPROP. Consider only the single transformation s(a, x) = x + a.
12.5. EBNN extracts training derivatives from explanations by examining the weights
and activations of the neural networks that make up the explanation. Consider the
simple example in which the explanation is formed by a single sigmoid unit with
n inputs. Derive a procedure for extracting the derivative 91,=,~
where xi is a
particular training instance input to the unit, f(x) is the sigmoid unit output, and
xi denotes the jth input to the sigmoid unit. You may wish to use the notation x!
to refer to the jth component of xi. Hint: The derivation is similar to the derivation
of the BACKPROPAGATION training rule.
12.6. Consider again the search trace of FOCL shown in Figure 12.8. Suppose that the
hypothesis selected at the first level in the search is changed to
Cup t -.HasHandle
Describe the second-level candidate hypotheses that will be generated by FOCL as
successors to this hypothesis. You need only include those hypotheses generated
by FOCL's second search operator, which uses its domain theory. Don't forget to
post-prune the sufficient conditions. Use the training data from Table 12.3.
12.7. This chapter discussed three approaches to using prior knowledge to impact the
search through the space of possible hypotheses. Discuss your ideas for how these
three approaches could be integrated. Can you propose a specific algorithm that
integrates at least two of these three for some specific hypothesis representation?
What advantages and disadvantages would you anticipate from this integration?
12.8. Consider again the question from Section 12.2.1, regarding what criterion to use
for choosing among hypotheses when both data and prior knowledge are available.
Give your own viewpoint on this issue.
REFERENCES
Abu-Mostafa, Y. S. (1989). Learning from hints in neural networks. Journal of Complexity, 6(2),
192-198.
Bergadano, F., & Giordana, A. (1990). Guiding induction with domain theories. In R. Michalski et
al. (Eds.), Machine learning: An art$cial intelligence approach 3 (pp. 474-492). San Mateo,
CA: Morgan Kaufmann.
Bradshaw, G., Fozzard, R., & Cice, L. (1989). A connectionist expert system that really works. In
Advances in neural information processing. San Mateo, CA: Morgan Kaufmam.
Caruana, R. (1996). Algorithms and applications for multitask learning. Proceedings of the 13th
International Conference on Machine Learning. San Francisco: Morgan Kaufmann.
Cooper, G. C., & Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks
from data. Machine Learning, 9, 309-347.
Craven, M. W. (1996). Extracting comprehensible modelsfrom trained neural networks (PhD thesis)
(UW Technical Report CS-TR-96-1326). Department of Computer Sciences, University of
Wisconsin-Madison.