Learning binary relations using weighted majority voting

More documents

Recommendations

Info

268 GOLDMAN AND WARMUTH 3mn21nk +4c~p(mn-c~p)+4c~(mn-c~) ( ln~ ) -- 2 1 < in I+Z 2 2 In 1+/3 : - \ - - + V/ 3mn 2 In k where z = 2~(~-~)" So by applying Lemma 1 with and /3 = g(z) we obtain the worst-case mistake bound s given in the theorem. • In addition to presenting their algorithm to make at most km+ nv/(k - 1)m mistakes, Goldman, Rivest, and Schapire (1989) present an information-theoretic lower bound for a class of algorithms that they call row-filter algorithms. They say that an algorithm A is a row-filter algorithm if A makes its prediction for Mrj strictly as a function of j and all entries in the set of rows consistent with row r and defined in column j. For this class of algorithms they show a lower bound to f~(nvZm ) for m > n on the number of mistakes that any algorithm must make. Recently, William Chen (1991) has extended their proof to obtain a lower bound of f~(n~) for m > n lg k. Observe that Learn- Relation is not a row-filter algorithm since the weights stored on the edges between the rows allows it to use the outcome of previous predictions to aid in its prediction for the current trial. Nevertheless, a simple modification of the projective geometry lower bound of Goldman, Rivest, and Schapire (1989) can be used to show an f~(nv/--m) lower bound for m > n on the number of prediction mistakes by our algorithm. Chen's extension of the projective geometry argument to incorporate k does not extend in such a straightforward manner; however, we conjecture that his lower bound can be generalized to prove that the mistake-bound we obtained for Learn-Relation is asymptotically tight. Thus to obtain a better algorithm, more than pairwise information between rows may be needed in making predictions. 6. Concluding Remarks We have demonstrated that a weighted majority voting algorithm can be used to learn a binary relation even when there is noise present. Our first algorithm uses exponentially many weights. In the noise-free case this algorithm is essentially optimal. We believe that by proving lower bounds for the noisy case (possibly using the techniques developed by Cesa-Bianchi, Freund, Helmbold, Haussler, Schapire, and Warmuth (1993)) one can show that the tuned version of the first algorithm (Theorem 1) is close to optimal in the more general case as well. The focus of our paper is the analysis of our second algorithm that uses a polynomial number of weights and thus can make predictions in polynomial time. In this algorithm a number of copies of our algorithm divide the problem among themselves and learn the relation cooperatively. It is surprising that the parallel application of on-line algorithms using multiplicative weight updates can be used to do some non-trivial clustering with provable perfor- -
LEARNING BINARY RELATIONS 269 mance (Theorem 3). Are there other applications where the clustering capability can be exploited? For the problem of learning binary relations the mistake bound of the polynomial algorithm (second algorithm) which uses (2) weights is still far away from the mistake bound of the exponential algorithm (first algorithm) which uses U'/k! weights. There seems to be a tradeoff between efficiency (number of weights) and the quality of the mistake bound. One of the most fascinating open problem regarding this research is the following: Is it possible to significantly improve our mistake bound (for either learning pure or non-pure relations) by using say O(n 3) weights? Or can one prove, based on some reasonable complexity theoretic or cryptographic assumptions, that no polynomial-time algorithm can perform significantly better than our second algorithm? Aeknowledgments We thank William Chen and David Helmbold for pointing out flaws in earlier versions of this paper. We also thank the anonymous referees for their comments. Appendix { 67 n? n(n--1) We now demonstrate that the function f(6i, n~) = 6ini - -~ + -~ß lg ~~(n~-]) is con- cave for ni _> 2 and 6~
Page 1 and 2: Machine Leaming, 20, 245-271 (1995)
Page 3 and 4: LEARNING BINARY RELATIONS 247 When
Page 5 and 6: LEARNING BINARY RELATIONS 249 ~,~,.
Page 7 and 8: LEARNING BINARY RELATIONS 251 l J i
Page 9 and 10: LEARNING BINARY RELATIONS 253 Learn
Page 11 and 12: LEARNING BINARY RELATIONS 255 We no
Page 13 and 14: LEARNING BINARY RELATIONS 257 Solvi
Page 15 and 16: LEARNING BINARY RELATIONS 259 In th
Page 17 and 18: LEARNING BINARY RELATIONS 261 depen
Page 19 and 20: LEARNING BINARY RELATIONS 263 m V S
Page 21 and 22: LEARNING BINARY RELATIONS 265 Final
Page 23: LEARNING BINARY RELATIONS 267 32500
Page 27: LEARNING BINARY RELATIONS 271 Kivin

Learning binary relations using weighted majority voting

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?