Data Mining: Practical Machine Learning Tools and ... - LIDeCC

342 CHAPTER 7 | TRANSFORMATIONS: ENGINEERING THE INPUT AND OUTPUTDougherty et al. (1995) give a brief account of supervised and unsuperviseddiscretization, along with experimental results comparing the entropy-basedmethod with equal-width binning and the 1R method. Frank and Witten (1999)describe the effect of using the ordering information in discretized attributes.Proportional k-interval discretization for Naïve Bayes was proposed by Yang andWebb (2001). The entropy-based method for discretization, including the useof the MDL stopping criterion, was developed by Fayyad and Irani (1993). Thebottom-up statistical method using the c 2 test is due to Kerber (1992), and itsextension to an automatically determined significance level is described by Liuand Setiono (1997). Fulton et al. (1995) investigate the use of dynamic programmingfor discretization and derive the quadratic time bound for a generalimpurity function (e.g., entropy) and the linear one for error-based discretization.The example used for showing the weakness of error-based discretizationis adapted from Kohavi and Sahami (1996), who were the first to clearly identifythis phenomenon.Principal components analysis is a standard technique that can be found inmost statistics textbooks. Fradkin and Madigan (2003) analyze the performanceof random projections. The TF ¥ IDF metric is described by Witten et al.(1999b).The experiments on using C4.5 to filter its own training data were reportedby John (1995). The more conservative approach of a consensus filter involvingseveral learning algorithms has been investigated by Brodley and Friedl(1996). Rousseeuw and Leroy (1987) describe the detection of outliers in statisticalregression, including the least median of squares method; they alsopresent the telephone data of Figure 7.6. It was Quinlan (1986) who noticedthat removing noise from the training instance’s attributes can decrease aclassifier’s performance on similarly noisy test instances, particularly at highernoise levels.Combining multiple models is a popular research topic in machine learningresearch, with many related publications. The term bagging (for “bootstrapaggregating”) was coined by Breiman (1996b), who investigated the propertiesof bagging theoretically and empirically for both classification and numeric prediction.Domingos (1999) introduced the MetaCost algorithm. Randomizationwas evaluated by Dietterich (2000) and compared with bagging and boosting.Bay (1999) suggests using randomization for ensemble learning with nearestneighborclassifiers. Random forests were introduced by Breiman (2001).Freund and Schapire (1996) developed the AdaBoost.M1 boosting algorithmand derived theoretical bounds for its performance. Later, they improved thesebounds using the concept of margins (Freund and Schapire 1999). Drucker(1997) adapted AdaBoost.M1 for numeric prediction. The LogitBoost algorithmwas developed by Friedman et al. (2000). Friedman (2001) describes how tomake boosting more resilient in the presence of noisy data.