Data Mining: Practical Machine Learning Tools and ... - LIDeCC

6.5 NUMERIC PREDICTION 253nique. It also compares favorably with far more sophisticated ways of enhancingNaïve Bayes by relaxing its intrinsic independence assumption. Locallyweighted learning only assumes independence within a neighborhood, notglobally in the whole instance space as standard Naïve Bayes does.In principle, locally weighted learning can also be applied to decision treesand other models that are more complex than linear regression and Naïve Bayes.However, it is beneficial here because it is primarily a way of allowing simplemodels to become more flexible by allowing them to approximate arbitrarytargets. If the underlying learning algorithm can already do that, there is littlepoint in applying locally weighted learning. Nevertheless it may improve othersimple models—for example, linear support vector machines and logisticregression.DiscussionRegression trees were introduced in the CART system of Breiman et al. (1984).CART, for “classification and regression trees,” incorporated a decision treeinducer for discrete classes much like that of C4.5, which was developed independently,and a scheme for inducing regression trees. Many of the techniquesdescribed in the preceding section, such as the method of handling nominalattributes and the surrogate device for dealing with missing values, wereincluded in CART. However, model trees did not appear until much morerecently, being first described by Quinlan (1992). Using model trees for generatingrule sets (although not partial trees) has been explored by Hall et al.(1999).Model tree induction is not so commonly used as decision tree induction,partly because comprehensive descriptions (and implementations) of the techniquehave become available only recently (Wang and Witten 1997). Neural netsare more commonly used for predicting numeric quantities, although theysuffer from the disadvantage that the structures they produce are opaque andcannot be used to help us understand the nature of the solution. Although thereare techniques for producing understandable insights from the structure ofneural networks, the arbitrary nature of the internal representation means thatthere may be dramatic variations between networks of identical architecturetrained on the same data. By dividing the function being induced into linearpatches, model trees provide a representation that is reproducible and at leastsomewhat comprehensible.There are many variations of locally weighted learning. For example, statisticianshave considered using locally quadratic models instead of linear ones andhave applied locally weighted logistic regression to classification problems. Also,many different potential weighting and distance functions can be found in theliterature. Atkeson et al. (1997) have written an excellent survey on locally