Anomaly Detection for Monitoring

work. When a system or process is highly unstable, it becomes
extremely difficult for models to work well. We highly recommend
implementing filters to reduce the number of false positives. Some
of the filters we’ve used include:
• Instead of sending an alert when an anomaly is detected, send
an alert when N anomalies are detected within an interval of
time.
• Suppress anomalies when systems appear to be too unstable to
determine any kind of normal behavior. For example, the
variance-to-mean ratio (index of dispersion), or another dimensionless
metric, can be used to indicate whether a system’s
behavior is stable.
• If a system violates a threshold and you trigger an anomaly or
send an alert, don’t allow another one to be sent unless the system
resets back to normal first. This can be implemented by
having a reset threshold, below which the metrics of interest
must dip before they can trigger above the upper threshold
again.
Filters don’t have to be complicated. Sometimes it’s much simpler
and more efficient to just simply ignore metrics that are likely to
cause alerting nuisances. Ruxit recently published a blog post titled
“Parameterized anomaly detection settings” 5 in which they describe
their anomaly detection settings. Although they don’t call it a “filter,”
one of their settings disables anomaly detection for low traffic applications
and services to avoid unnecessary alerts.
Tools
You generally don’t have to implement an entire anomaly detection
framework yourself. As a significant component of monitoring,
anomaly detection has been the focus of many monitoring projects
and companies which have implemented many of the things we’ve
discussed in this book.
5 http://bit.ly/ruxitblog
60 | Chapter 6: The Broader Landscape