Machine Learning - DISCo

(b) You now draw a new set of 100 instances, drawn independently according to the
same distribution D. You find that h misclassifies 30 of these 100 new examples.
Can you give an interval into which this true error will fall with approximately
95% probability? (Ignore the performance over the earlier training data for this
part.) If so, state it and justify it briefly. If not, explain the difficulty.
(c) It may seem a bit odd that h misclassifies 30% of the new examples even though
it perfectly classified the training examples. Is this event more likely for large
n or small n? Justify your answer in a sentence.
REFERENCES
Angluin, D. (1992). Computational learning theory: Survey and selected bibliography. Proceedings
of the Twenty-Fourth Annual ACM Symposium on Theory of Computing (pp. 351-369). ACM
Press.
Angluin, D., Frazier, M., & Pitt, L. (1992). Learning conjunctions of horn clauses. Machine Learning,
9, 147-164.
Anthony, M., & Biggs, N. (1992). Computational learning theory: An introduction. Cambridge,
England: Cambridge University Press.
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. (1989). Learnability and the Vapnik-
Chemonenkis dimension. Journal of the ACM, 36(4) (October), 929-965.
Ehrenfeucht, A., Haussler, D., Kearns, M., & Valiant, L. (1989). A general lower bound on the
number of examples needed for learning. Informution and computation, 82, 247-261.
Gold, E. M. (1967). Language identification in the limit. Information and Control, 10, 447-474.
Goldman, S. (Ed.). (1995). Special issue on computational learning theory. Machine Learning,
18(2/3), February.
Haussler, D. (1988). Quantifying inductive bias: A1 learning algorithms and Valiant's learning framework.
ArtGcial Intelligence, 36, 177-221.
Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory. Cambridge,
MA: MIT Press.
Laird, P. (1988). Learning from good and bad data. Dordrecht: Kluwer Academic Publishers.
Li, M., & Valiant, L. G. (Eds.). (1994). Special issue on computational learning theory. Machine
Learning, 14(1).
Littlestone, N. (1987). Learning quickly when irrelevant attributes abound: A new linear-threshold
algorithm. Machine Learning, 2, 285-318.
Littlestone, N., & Warmuth, M. (1991). The weighted majority algorithm (Technical report UCSC-
CRL-91-28). Univ. of California Santa Cruz, Computer Engineering and Information Sciences
Dept., Santa Cruz, CA.
Littlestone, N., & Warmuth, M. (1994). The weighted majority algorithm. Information and Computation
(log), 212-261.
Pltt, L. (Ed.). (1990). Special issue on computational learning theory. Machine Learning, 5(2).
Natarajan, B. K. (1991). Machine learning: A theoretical approach. San Mateo, CA: Morgan Kaufmann.
Valiant, L. (1984). A theory of the learnable. Communications of the ACM, 27(1 I), 1134-1 142.
Vapnik, V. N. (1982). Estimation of dependences based on empirical data. New York: Springer-
Verlag.
Vapnik, V. N., & Chervonenkis, A. (1971). On the uniform convergence of relative frequencies of
events to their probabilities. Theory of Probability and Its Applications, 16, 264-280.