Anomaly Detection for Monitoring

Evaluating Predictions
One of the most important and subtle parts of anomaly detection
happens at the intersection between predicting how a metric should
behave, and comparing observed values to those expectations.
In anomaly detection, you’re usually using many standard deviations
from the mean as a replacement for very unlikely, and when you get
far from the mean, you’re in the tails of the distribution. The fit
tends to be much worse here than you’d expect, so even small deviations
from Gaussian can result in many more outliers than you theoretically
should get.
Similarly, a lot of statistical tests such as hypothesis tests are deemed
to be “significant” or “good” based on what turns out to be statistician
rules of thumb. Just because some p-value looks really good
doesn’t mean there’s truly a lot of certainty. “Significant” might not
signify much. Hey, it’s statistics, after all!
As a result, there’s a good chance your anomaly detection techniques
will sometimes give you more false positives than you think they
will. These problems will always happen; this is just par for the
course. We’ll discuss some ways to mitigate this in later chapters.
Common Myths About Statistical Anomaly
Detection
We commonly hear claims that some technique, such as SPC, won’t
work because system metrics are not Gaussian. The assertion is that
the only workable approaches are complicated non-parametric
methods. This is an oversimplification that comes from confusion
about statistics.
Here’s an example. Suppose you capture a few observations of a
“mystery time series.” We’ve plotted this in Figure 3-6.
Evaluating Predictions | 27