Lecture Notes in Computer Science 5185

The Problem of False Alarms 141
produced an unacceptably high rate of false positives, which rose above the
50% ROC’s guess line rate. Unfortunately, no further explanation was given to
describe the nature of false alarms.
Interestingly, a paper by Kayacik and Zincir-Heywood [11] discussed the benefit
of implementing intrusion detection systems working together with a firewall.
The paper had demonstrated a benchmark evaluation of three security management
tools (Snort, Pakemon and Cisco IOS firewall). Significantly, the result
showed that none of the tools could detect all the attacks. In fact, Snort IDS
was found to have produced 99% of false alarm rate, the highest rate compared
to the other IDS (Pakemon). The result had also revealed that Cisco IOS had
performed well and raised only 68% of false alarm rate. This has suggested the
implementation of a firewall-based detection,whichinturndecreasestheattack
traffic being passed to the IDSs.
Apart from the two studies above, which focused upon Snort performance,
there are a large number of studies that have used the 1998 and 1999 DARPA
dataset to evaluate the performance of IDSs. One of those studies is that of
Lippmann et al [13], which managed to demonstrate the need for developing
techniques to find new attacks instead of extending existing rule-based approach.
The result of the evaluation demonstrated that current research systems can reliably
detect many existing attacks with low false alarm rate as long as examples
of these attacks are available for training. In actual fact, the research systems
missed many dangerous new attacks when the attack mechanisms differ from the
old attacks. Interestingly, a similar paper had also been written by Lippmann et
al [14], focusing upon the performance of different IDS types, such as host-based,
anomaly-based and forensic-based in detecting novel and stealthy attacks. The
result of this analysis had proposed a number of practical approaches applied to
improve the performance of the existing systems.
Alharby and Imai [2] had also utilised 1999 DARPA dataset to evaluate the
performance of their proposed alarm reduction system. In order to obtain the
normal alarm model, alarm sequence is collected by processing the alerts generated
by Snort from the first and third weeks (free-attacks traffic) of DARPA
1999 dataset. From these alarm sequences, the sequential patterns are then extracted
to filter and reduce the false alarms. The same dataset (using the first
and third weeks of the 1999 DARPA dataset) had also been applied by Bolzoni
and Etalle [7] to train and evaluate the performance of the proposed false positive
reduction system. Similarly, Alshammari et al [3] had also used such data
to experiment their neural network based alarm reduction system with the different
background knowledge set. The final result has proved that the proposed
technique has significantly reduced the number of false alarms without requiring
much background knowledge sets.
Unlike other papers discussed above, our experiment focuses specifically upon
the issue of false alarms, rather than the performance of IDS (true alarms) in
general. In this study, we propose to investigate in a more detailed manner some
of the shortcomings that caused the generation of false alarms.