6th European Conference - Academic Conferences

Large-Scale Analysis of Continuous Data in Cyber-Warfare
Threat Detection
William Acosta
University of Toledo, USA
william.acosta@utoledo.edu
Abstract: Combating cyber/information warfare threats requires analyzing vast quantities of diverse data. The
data required to detect attacks as they occur (on-line analysis of live data) and predict future threats (forensic
analysis/data mining) is not only large, but is growing at a staggering rate. Data such as network traffic logs,
emails, and social networking posts, SMS message, and cell phone call logs are, by nature, continuous and
growing. The problem addressed in this research is that current systems are not designed to handle either the
scope or nature of the analysis or the data itself. For example, distributed data processing systems like Google’s
Map-Reduce provide the ability to process large data sets, but they are not designed to easily support processing
of changing data sets or data-mining algorithms. In light of this, Google has itself recently stopped using
MapReduce for building its web-index, opting instead for a custom mechanism that can more quickly respond to
and process new content. Non-traditional databases, like vertically-partitioned/column-store databases, can
efficiently support analysis algorithms on large quantities of data, but they are not designed to support
continuously changing data sets. The goal of this research is to explore and design new data management
system that can handle large quantities of incrementally growing data as well as direct support for data mining
and analysis algorithms. Specifically, this research proposes a new distributed data processing system that
exploits the parallel and distributed resources/computation of cloud computing infrastructures. It makes use of
summary data structures that can be updated incrementally and continuous queries to support analysis and data
mining algorithms natively. This approach allows for larger-scale and more robust analysis on continuously
growing data that can help detect, predict and respond to cyber-warfare threats.
Keywords: data-mining, databases, text-search, cloud computing, data integration
1. Introduction
Protection against cyber/information warfare threats requires understanding the nature, methods, and
patterns of those attacks. Such understanding can allow for early detection and, possibly prediction,
of attacks. Gaining an understanding of the patterns and mechanisms used in cyber/information
warfare attacks requires analyzing large amounts of diverse data such as server logs (Myers et al.
2010), emails, SMS messages, and social-networking data. Not only is the data diverse, but it is also
continuous; new data gets generated every day. Furthermore, analysis of this data can require
equally diverse approaches: graph-theoretic algorithms (detecting patterns in social-networking), data
mining algorithms (associations between events), statistical models, clustering algorithms, etc. The
diverse nature of the data and analysis algorithms as well as the large quantity of data to be analyzed
poses problems to both traditional databases and storage systems. In order to provide the analysis of
diverse and continuous data required for cyber-warfare threat detection, a new system is needed for
managing large quantities of diverse data that can support equally diverse analysis algorithms.
The need to incrementally process large quantities of data is applicable to wide range of applications.
For example, Google replaced MapReduce (Dean & Ghemawat 2004), its current web-indexing
system, in order to enable faster updates of its index (Metz 2010, Peng & Dabek 2010). Similarly,
detecting and responding to information security threats requires a mechanism that cannot only
manage large quantities of data, but also provide for fast response time of complex, continuous
analysis. This paper proposes a new distributed data-analysis framework that is designed to meet the
needs of applications that require analysis of continuous data. Next, Section 2 presents the design of
the proposed system in the context of related work. Section 3 then provides concluding remarks.
2. Design and requirements of a continuous data analysis system
Cyber-warfare threat detection requires analyzing large quantities of diverse data that is continuously
generated. The properties of the raw data in this type of application impose some constraints on the
analysis and data storage systems. These applications require analyzing not only current data, but
also prior/historical from many heterogeneous sources. Because the raw data is continuously
generated, old data must be kept for analysis while new data is integrated into the storage and
analysis framework. Because old data must be kept and not changed, the system need not support
updates of raw data. Effectively, raw data is append-only. This can be leveraged to improve storage
efficiency and performance; it is easier to implement and support distributed storage as no write-
317