Anomaly Detection for Monitoring

lem in your system, that’s great. Go ahead and alert on it. But otherwise,
we suggest that you don’t alert on things that may have no
impact or consequence.
Instead, we suggest that you record these anomalous observations,
but don’t alert on them. Now you have essentially created an index
into the most unusual data points in your metrics, for later use in
case it is interesting. For example, during diagnosis of a problem
that you have detected.
One of the assumptions embedded in this recommendation is that
anomaly detection is cheap enough to do online in one pass as data
arrives into your monitoring system, but that ad hoc, after-the-fact
anomaly detection is too costly to do interactively. With the monitoring
data sizes that we are seeing in the industry today, and the
attitude that you should “measure everything that moves,” this is
generally the case. Multi-terabyte anomaly detection analysis is usually
unacceptably slow and requires more resources than you have
available. Again, we are placing this in the context of what most of
us are doing for monitoring, using typical open-source tools and
methodologies.
Conclusions
Although it’s easy to get excited about success stories in anomaly
detection, most of the time someone else’s techniques will not translate
directly to your systems and your data. That’s why you have to
learn for yourself what works, what’s appropriate to use in some situations
and not in others, and the like.
Our suggestion, which will frame the discussion in the rest of this
book, is that, generally speaking, you probably should use anomaly
detection “online” as your data arrives. Store the results, but don’t
alert on them in most cases. And keep in mind that the map is not
the territory: the metric isn’t the system, an anomaly isn’t a crisis,
three sigmas isn’t unlikely, and so on.
14 | Chapter 2: A Crash Course in Anomaly Detection