Scalable approaches for analysis of human genome-wide ...

B. Supplementary Results for Sparse Linear Modelsall cutoff values. The ROC curves can be summarised using the Area Under the receiver operatingcharacteristic Curve (AUC) (Hanley and McNeil, 1982), computed through numericalintegration or alternatively estimated asÂUC =N1+N + N − ∑N −∑ {I(ŷ i > ŷ j ) + 1 2 I(ŷ i = ŷ j )} ,i=1 j=1(B.1)where N + and N − are the number of cases and controls respectively, ŷ i is ith the predictionfor the ith case, ŷ j is the prediction for the jth control, and I(⋅) is the indicator functionevaluating to 1 if its argument is true and to zero otherwise. Eq. B.1 shows that the sampleAUC is the maximum likelihood estimate of the probability of correctly ranking a randomlyselectedcausal SNP more highly than a randomly-selected non-causal SNP (with correctionfor ties). The expected AUC for a classifier producing random predictions is 0.5, perfectpredictions have AUC = 1.0 and perfectly-wrong predictions have an AUC = 0.0.Another useful statistic is the Area under the Precision-Recall Curve (APRC, also knownas Average Precision), which can be integrated numerically, but is usually approximated asÂPRC = 1 MM∑ Prec m ,m=1(B.2)where Prec m is the precision for the mth level of recall, out of M levels. The expectedAPRC for a classifier producing random predictions is the proportion of positive samples. Forestimating APRC, we used the program perf (http://kodiak.cs.cornell.edu/kddcup/software.html).Unlike the APRC, the AUC does not depend on the relative proportions of the classes(the class balance). However, the AUC, as commonly used, can be misleading when usedfor comparing classifiers when the proportion of causal SNPs is very small, as is the case inGWA. Consider our HAPGEN simulations, where only 148 of the 73, 832 SNPs are true causalSNPs. To see why AUC is not informative in these settings, consider a thought experimentsimilar to that used by Sonnenburg et al. (2006), where we have a classifier that at some cutoffcorrectly classifies 100% of the true causal SNPs (TPR=1), but also wrongly classifies 1%of the non-causal SNPs (FPR=0.01). The AUC is the area under the curve induced by theTPR and the FPR, and is monotonically increasing. Therefore, the AUC in this case mustbe ≥ 0.99, which seems like very good discrimination. However, when there are 73 684 noncausalSNPs, even the low false positive rate of 1% implies 0.01 × 73 684 ≈ 737 false positiveson average. In comparison, even assuming a fixed recall (=TPR) of 1, so that the APRCis equal to the precision, then the number of false positives needs to be as low as 148 (thenumber of causal SNPs) for the precision and APRC to be 0.5, and conversely, with a falsepositive rate of just 0.5% leading to ∼ 368 false positives on average, both the precision andthe APRC drop to 148/(148 + 368) = 0.287. In many real world settings, such extreme results218