Data Mining: Practical Machine Learning Tools and ... - LIDeCC

7.7 FURTHER READING 341deal with weighted instances in Section 6.5 under Locally weighted linear regression(page 252). One way of obtaining probability estimates from support vectormachines is to fit a one-dimensional logistic model to the output, effectivelyperforming logistic regression as described in Section 4.6 on the output. Excellentresults have been reported for text classification using co-EM with thesupport vector machine (SVM) classifier. It outperforms other variants of SVMand seems quite robust to varying proportions of labeled and unlabeled data.The ideas of co-training and EM—and particularly their combination in theco-EM algorithm—are interesting, thought provoking, and have striking potential.But just what makes them work is still controversial and poorly understood.These techniques are the subject of current research: they have not yet enteredthe mainstream of machine learning and been harnessed for practical datamining.7.7 Further readingAttribute selection, under the term feature selection, has been investigated in thefield of pattern recognition for decades. Backward elimination, for example, wasintroduced in the early 1960s (Marill and Green 1963). Kittler (1978) surveysthe feature selection algorithms that have been developed for pattern recognition.Best-first search and genetic algorithms are standard artificial intelligencetechniques (Winston 1992, Goldberg 1989).The experiments that show the performance of decision tree learners deterioratingwhen new attributes are added are reported by John (1997), who givesa nice explanation of attribute selection. The idea of finding the smallest attributeset that carves up the instances uniquely is from Almuallin and Dietterich(1991, 1992) and was further developed by Liu and Setiono (1996). Kibler andAha (1987) and Cardie (1993) both investigated the use of decision tree algorithmsto identify features for nearest-neighbor learning; Holmes and Nevill-Manning (1995) used 1R to order features for selection. Kira and Rendell (1992)used instance-based methods to select features, leading to a scheme calledRELIEF for Recursive Elimination of Features. Gilad-Bachrach et al. (2004) showhow this scheme can be modified to work better with redundant attributes. Thecorrelation-based feature selection method was developed by Hall (2000).The use of wrapper methods for feature selection is due to John et al. (1994)and Kohavi and John (1997), and genetic algorithms have been applied withina wrapper framework by Vafaie and DeJong (1992) and Cherkauer and Shavlik(1996). The selective Naïve Bayes learning method is due to Langley and Sage(1994). Guyon et al. (2002) present and evaluate the recursive feature eliminationscheme in conjunction with support vector machines. The method of racedsearch was developed by Moore and Lee (1994).