A Data Streaming Algorithm for Estimating Entropies of OD Flows

Fraction of experiments10.90.80.70.60.50.40.30.20.100 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5Relative Error35.5%14.5%5.3%3.3%2.5%Figure 8: Varying fraction of traffic from ingressFraction of experiments10.90.80.70.60.50.40.30.20.10Entropy error0 0.05 0.1 0.15 0.2 0.25Relative ErrorFigure 9: Error distribution for actual entropysource and destination. In Figure 9 we observe that the errorplot for the entropy is comparable to that for the entropynorm. Hence, this confirms the fact that our algorithm is arobust estimator of the entropy of OD flows.9. RELATED WORKThere has been considerable previous work in computingthe traffic matrix in a network [18, 21, 22, 23, 27]. The trafficmatrix is simply the matrix defined by the packet (byte)counts between each pair of ingress and egress nodes in thenetwork over some measurement interval. For fine-grainednetwork measurement we are sometimes interested in theflow matrix [28], which quantifies the volume of the trafficbetween individual OD flows. In this paper we propose thecomputation of the entropy of every OD flows, which givesus more information than a simple traffic matrix and hasconsiderably less overhead than maintaining the entire flowmatrix.In the last few years, entropy has been suggested as a usefulmeasure for different network-monitoring applications.Lakhina et al. [15] use the entropy measure to perform anomalydetection and network diagnosis. Information measures suchas entropy have been suggested for tracking malicious networkactivity [11, 24]. Entropy has been used by Xu etal. [25] to infer patterns of interesting activity by using itto cluster traffic. For detecting specific types of attacks,researchers have suggested the use of entropy of differenttraffic features for worm [24] and DDoS detection [11]. Recently,it has been shown that entropy-based techniques foranomaly detection are much more resistant to the effects ofsampling [3] than other, volume-based methods.The use of stable distributions to produce a sketch wasfirst proposed in [12] to measure the L 1 distance betweentwo streams. This result was generalized to the L p distancefor all 0 < p ≤ 2 in [8, 13]. This sketch data structure isthe starting point for the one that we propose in this paper.The main difference is that we have to make several keymodifications to make it work in practice. In particular, wehave to ensure that the number of updates per packet issmall enough to feasibly perform in realtime.Other than for p = 1, 2, there is no known closed formfor the p-stable distribution. To independently draw valuesfrom an arbitrary p-stable distribution, we make use of theformula proposed by [6]. Since this formula is expensiveto compute, we create a lookup table to hold several precomputedvalues.In [7] it is suggested that the stable distribution sketchcan be used as a building block to compute empirical entropy,but no methods for doing this are suggested. Moreimportantly, there are already known to be algorithms thatwork well for the single stream case [16] and we believe thatit is for this distributed (i.e., traffic matrix) setting that thestable sketch really shines.10. CONCLUSIONIn this paper we motivate the problem of estimating theentropy between origin destination pairs in a network andpresent an algorithm for solving this problem. Along theway, we present a completely novel scheme for estimatingentropy, introduce applications for non-standard L p norms,present an extension of Indyk’s algorithm, and show howit can be used in our distributed setting. Via simulationon real-world data, collected at a tier-1 ISP, we are able todemonstrate that our algorithm is practically viable.11. REFERENCES[1] A. Chakrabarti, K. Do Ba, and S. Muthukrishnan.Estimating entropy and entropy norm on data streams. InSTACS, 2006.[2] L. Bhuvanagiri and S. Ganguly. Estimating entropy overdata streams. In ESA, 2006.[3] D. Brauckhoff, B. Tellenbach, A. Wagner, M. May, andA. Lakhina. Impact of packet sampling on anomalydetection metrics. In IMC, 2006.[4] G. Casella and R. L. Berger. Statistical Inference. Duxbury,2nd edition, 2002.[5] A. Chakrabarti and G. Cormode. A near-optimal algorithmfor computing the entropy of a stream. In SODA, 2007.[6] J. M. Chambers, C. L. Mallows, and B. W. Stuck. Amethod for simulating stable random variables. Journal ofthe American Statistical Association, 71(354), 1976.[7] G. Cormode. Stable distributions for stream computations:It’s as easy as 0,1,2. In Workshop on Management andProcessing of Data Streams, 2003.[8] G. Cormode, P. Indyk, N. Koudas, and S. Muthukrishnan.Fast mining of massive tabular data via approximatedistance computations. In ICDE, 2002.[9] M. Durand and P. Flajolet. Loglog counting of largecardinalities. In ESA, 2003.[10] C. Estan and G. Varghese. New Directions in TrafficMeasurement and Accounting. In SIGCOMM, Aug. 2002.[11] L. Feinstein, D. Schnackenberg, R. Balupari, andD. Kindred. Statistical approaches to DDoS attackdetection and response. In Proceedings of the DARPAInformation Survivability Conference and Exposition, 2003.[12] P. Indyk. Stable distributions, pseudorandom generators,embeddings and data stream computation. In FOCS, 2000.[13] P. Indyk. Stable distributions, pseudorandom generators,embeddings, and data stream computation. J. ACM,53(3):307–323, 2006.