Anomaly Detection for Monitoring

More documents

Recommendations

Info

Dealing with Trend Trends break models because the value of a time series with a trend isn’t stable, or stationary, over time. Using a basic, fixed control chart on a time series with an increasing trend is a bad idea because it is guaranteed to eventually exceed the upper control limit. A trend violates a lot of simple assumptions. What’s the mean of a metric that has a trend? There is no single value <strong>for</strong> the mean. Instead, the mean is actually a function with time as a parameter. What about the distribution of values? You can visualize it using a histogram, but this is misleading. Because the values increase or decrease over time due to trend, the histogram will get wider and wider over time. What about a simple moving average or a EWMA? A moving average should change along with the trend itself, and indeed it does. Un<strong>for</strong>tunately, this doesn’t work very well, because a moving average lags in the presence of a trend and will be consistently above or below the typical values. Figure 4-1. A time series with a linear trend and two exponentially weighted moving averages with different decay factors, demonstrating that they lag the data when it has a trend. How do you deal with trend? First, it’s important to understand that metrics with trends can be considered as compositions of other metrics. One of the components is the trend, and so the solution to deal‐ 34 | Chapter 4: Dealing with Trends and Seasonality
ing with trend is simple: find a model that describes the trend, and subtract the trend from the metric’s values! After the trend is removed, you can use the models that we’ve previously mentioned on the remainder. There can be many different kinds of trend, but linear is pretty common. This means a time series increases or decreases at a constant rate. To remove a linear trend, you can simply use a first difference. This means you consider the differences between consecutive values of a time series rather than the raw values of the time series itself. If you remember your calculus, this is related to a derivative, and in time series it’s pretty common to hear people talk about first differences as derivatives (or deltas). Dealing with Seasonality Seasonal time series data has cycles. These are usually obvious on observation, as shown in Figure 4-2. Figure 4-2. A server’s load average, showing repeated cycles over time. Seasonality has very similar effects as trend. In fact, if you “zoom into” a time series with seasonality, it really looks like trend. That’s because seasonality is variable trend. Instead of increasing or decreasing at a fixed rate, a metric with seasonality increases or decreases with rates that vary with time. As you can imagine, things like EWMAs have the same issues as with linear trend. They lag behind, and in some cases it can get so bad that the EWMA is com‐ Dealing with Seasonality | 35
Page 3 and 4: Anomaly Detection for Monitoring A
Page 5 and 6: Table of Contents Foreword. . . . .
Page 7: Foreword Monitoring is currently un
Page 10 and 11: tion” to anomaly detection is imp
Page 12 and 13: Why do we assume these things? Are
Page 14 and 15: Conclusions If you are like most of
Page 17 and 18: CHAPTER 2 A Crash Course in Anomaly
Page 19 and 20: How can you achieve similar results
Page 21 and 22: cause most anomaly detection techni
Page 23 and 24: CHAPTER 3 Modeling and Predicting A
Page 25 and 26: (say, the size of the drill bit), a
Page 27 and 28: lies. To fix this problem, the cont
Page 29 and 30: ally decaying window. This is made
Page 31 and 32: ack again, meaning that they smooth
Page 33 and 34: emains consistent across all of the
Page 35 and 36: Evaluating Predictions One of the m
Page 37 and 38: No. You’ve stumbled into statisti
Page 39: As an aside, there’s a rumor goin
Page 44 and 45: pletely out of phase with the seaso
Page 46 and 47: mon situations. You can probably gu
Page 49 and 50: CHAPTER 5 Practical Anomaly Detecti
Page 51 and 52: If you can get close to that, you m
Page 53 and 54: We previously discussed how frequen
Page 55 and 56: First, observe how odd this metric
Page 57 and 58: After differencing, it looks like w
Page 59 and 60: Perhaps, instead of differencing, w
Page 61 and 62: CHAPTER 6 The Broader Landscape As
Page 63 and 64: We could apply a EWMA control chart
Page 65 and 66: figure out how unlikely it was for
Page 67 and 68: predict or find structure in data w
Page 69 and 70: Graphite and RRDTool Graphite and R
Page 71 and 72: APPENDIX A Appendix Code Control Ch
Page 73 and 74: var ma = new movingAverage(0.5); ma
Page 75: Acknowledgments We’d like to than

Anomaly Detection for Monitoring

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?