Data Mining: Practical Machine Learning Tools and ... - LIDeCC

5.7 COUNTING THE COST 171should choose method A, which gives a false positive rate of around 5%, ratherthan method B, which gives more than 20% false positives. But method B excelsif you are planning a large sample: if you are covering 80% of the true positives,method B will give a false positive rate of 60% as compared with method A’s80%. The shaded area is called the convex hull of the two curves, and you shouldalways operate at a point that lies on the upper boundary of the convex hull.What about the region in the middle where neither method A nor methodB lies on the convex hull? It is a remarkable fact that you can get anywhere inthe shaded region by combining methods A and B and using them at randomwith appropriate probabilities. To see this, choose a particular probability cutofffor method A that gives true and false positive rates of t A and f A , respectively,and another cutoff for method B that gives t B and f B . If you use these twoschemes at random with probability p and q, where p + q = 1, then you will gettrue and false positive rates of p.t A + q.t B and p.f A + q.f B . This represents a pointlying on the straight line joining the points (t A ,f A ) and (t B ,f B ), and by varying pand q you can trace out the entire line between these two points. Using thisdevice, the entire shaded region can be reached. Only if a particular scheme generatesa point that lies on the convex hull should it be used alone: otherwise, itwould always be better to use a combination of classifiers corresponding to apoint that lies on the convex hull.Recall–precision curvesPeople have grappled with the fundamental tradeoff illustrated by lift charts andROC curves in a wide variety of domains. Information retrieval is a goodexample. Given a query, a Web search engine produces a list of hits that representdocuments supposedly relevant to the query. Compare one system thatlocates 100 documents, 40 of which are relevant, with another that locates 400documents, 80 of which are relevant. Which is better? The answer should nowbe obvious: it depends on the relative cost of false positives, documents that arereturned that aren’t relevant, and false negatives, documents that are relevantthat aren’t returned. Information retrieval researchers define parameters calledrecall and precision:recall =number of documents retrieved that are relevanttotal number of documents that are relevantnumber of documents retrieved that are relevantprecision = .total number of documents that are retrievedFor example, if the list of yes’s and no’s in Table 5.6 represented a ranked list ofretrieved documents and whether they were relevant or not, and the entirecollection contained a total of 40 relevant documents, then “recall at 10” would