SEKE 2012 Proceedings - Knowledge Systems Institute

TABLE I
INTRINSIC EVALUATION OF SERIOUS VERSUS LOW-GRADE EVENT CATEGORIZATION. THE RESULTS FOR C4.5 DECISION TREE, K-MEANS CLUSTERING
AND PCLS ARE COMPARED WITH THE LABELS FROM HAND-CRAFTED RULES.
year
2002
2003
2004
2005
2006
class
C4.5 Decision Tree (DT) K-Means Clustering (KM) PCLS
Pre Rec F Avg. F Pre Rec F Avg. F Pre Rec F Avg. F
serious event .701 .603 .648
.024 .162 .042
.610 .716 .659
.746
.366
low-grade event .835 .852 .843 .757 .634 .690 .879 .698 .778
.718
serious event .675 .491 .569
.256 .536 .347
.328 .428 .371
.668
.319
low-grade event .813 .725 .767 .908 .174 .292 .792 .514 .623
.497
serious event .633 .376 .472
.251 .577 .350
.688 .414 .517
.599
.429
low-grade event .706 .745 .725 .631 .425 .508 .690 .701 .695
.606
serious event .536 .635 .581
.624 .895 .735
.560 .932 .700
.657
.688
low-grade event .780 .692 .733 .731 .571 .641 .691 .694 .693
.696
serious event .603 .554 .577
.008 .128 .016
.692 .791 .738
.646
.320
low-grade event .708 .722 .715 .547 .726 .624 .710 .664 .686
.712
We developed a semi-supervised learning method, Progressive
Clustering with Learned Seeds, that suited the problem.
Intrinsic evaluation displays the stability and consistency of
knowledge preservation in accordance with a set of very
limited labeled data. Extrinsic evaluation shows the superior
actual performance on a blind test. Our problem engineering
method can also be implemented and adapted to other domains
providing an exemplary approach that uses computational
intelligence technology for industrial applications.
Fig. 3. Visualized results of extrinsic evaluation. It shows how the structures
that actually have events are captured at the top 5,000 rank list. The vertical
axis is the ranking, and the horizontal axis is a count of the number of
structures. A lower curve is preferred, and the one more stretched to the
right is preferred.
Table II summarizes the results of blind evaluation for
Manhattan; larger scores for AUC and DCG, and lower for
PNorm, correspond to superior performance. PCLS excels
the other methods in all three measures. Figure 3 plots the
vulnerable structures from the blind evaluation year (horizontal
axis) against the top 5000 structures in the ranked lists from
the four methods (vertical axis). As shown, PCLS outperforms
the other methods in terms of all three measures (AUC, DCG,
PNorm). Since the actual performance and scores are not linear
correlated, even though PCLS is literally only 3.7% higher in
AUC and 1.2% higher in DCG, the improvement is actually
quite substantial. From Figure 3, in the top 5000 of the rank
list, PCLS retrieves 51 vulnerable structures and C4.5 DT is
31, which is 64.5% improvement. When compared to KM,
PCLS captures 3 more structures and the positions of these
structures in the list are significantly ranked higher. Moreover,
KM labels many more tickets as serious, as indicated by the
very low precision and relatively high recall for this class
(Table I), leading to much higher computational complexity
of the structure ranking task for KM in contrast to PCLS.
VIII. CONCLUSION
We have presented an electrical event categorization task
on a power grid application system. The characteristics of
the categorization problem require an approach that can adapt
existing knowledge about the data model over time.
REFERENCES
[1] J. C. Schlimmer and R. H. Granger, “Incremental learning from noisy
data,” Machine Learning, vol. 1, no. 3, pp. 317–354, 1986.
[2] P. E. Utgoff, “Incremental induction of decision trees,” Machine Learning,
vol. 4, no. 2, pp. 161–186, 1989.
[3] L. Kaufman and P. J. Rousseeuw, Finding Groups in Data: An Introduction
to Cluster Analysis. Wiley, Canada, 1990.
[4] J. M. Pena, J. A. Lozano, and P. Larranaga, “An empirical comparison
of four initialization methods for the K-means algorithm,” Pattern
Recognition Letters, vol. 20, pp. 1027 – 1040, 1999.
[5] D. Arthur and S. Vassilvitskii, “K-means++: The advantages of careful
seeding,” in Proceedings of the 18th Annual ACM-SIAM Symposium on
Discrete Algorithms, June 2007, pp. 1027 – 1035.
[6] P. S. Bradley and U. M. Fayyad, “Refining initial points for K-means
clustering,” in Proceedings of the 15th International Conference on
Machine Learning (ICML), 1998, pp. 91 – 99.
[7] B. Liu, Y. Xia, and P. S. Yu, “Clustering through decision tree construction,”
in Proceedings of the 9th International Conference on Information
and Knowledge Management (CIKM), McLean, VA, 2000.
[8] C. Rudin, R. J. Passonneau, A. Radeva, H. Dutta, S. Ierome, and
D. Isaac, “A process for predicting manhole events in manhattan,” Mach.
Learn., vol. 80, no. 1, pp. 1–31, Jul. 2010.
[9] C. Rudin, “The P-Norm Push: A simple convex ranking algorithm
that concentrates at the top of the list,” Journal of Machine Learning
Research, vol. 10, pp. 2233–2271, Oct 2009.
[10] R. J. Passonneau, C. Rudin, A. Radeva, and Z. A. Liu, “Reducing noise
in labels and features for a real world dataset: Application of nlp corpus
annotation methods,” in Proceedings of the 10th International Conference
on Computational Linguistics and Intelligent Text Processing, 2009.
[11] J. Cohen, “A coefficient of agreement for nominal scales,” Educational
and Psychological Measurement, vol. 20, pp. 37–46, 1960.
[12] R. Kittredge, “Sublanguages,” American Journal of Computational Linguistics,
pp. 79–84, 1982.
[13] G. Forman, “An extensive empirical study of feature selection metrics for
text classification,” Machine Learning Research, pp. 1289–1305, 2003.
[14] I. H. Witten and E. Frank, Data Mining: Practical Machine Learning
Tools and Techniques. San Francisco, CA: Morgan Kaufmann, 2005.
[15] J. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo, CA:
Morgan Kaufmann Publishers, 1993.
[16] J. R. Galliers and K. Sparck Jones, “Evaluating natural language
processing systems,” Computer Laboratory, University of Cambridge,
Tech. Rep. Technical Report 291, 1993.
105