6th European Conference - Academic Conferences

William Acosta
locking of existing data is necessary. It also allows for the analysis framework to make use novel
summary data structures and algorithms that can incorporate the changes made to the data without
requiring analysis of the full dataset.
2.1 Storage and data management
The large quantity of data makes a centralized storage solution unfeasible; instead, a distributed
storage solution Is favored. The parallel nature of many of the algorithms makes a distributed solution
not only more feasible, but also desirable. Distributed storage systems such as Google’s BigTable
(Chang et al. 2006), Yahoo’s PNUTS (Cooper et al. 2008), and Amazon’s Dynamo (DeCandia et al.
2007) provide the low-level mechanisms for storing, and managing large quantities of data. These
systems were designed to support coordinated reads and updates of data in a distributed
environment. To support the needs of applications like cyber-warfare threat detection, a distributed
storage system should provide efficient, low-level support for append-only writes of raw data, as well
as efficient tracking of incremental additions and updates of the dataset.
2.2 Distributed processing of data
Recently, there has been a great deal of research in Google’s MapReduce (Dean & Ghemawat 2004)
distributed computing software framework for processing large datasets. However, its batch-oriented
nature was not designed to deal with incremental or continuous data updates. This makes it
unsuitable for a variety of applications including cyber-warfare threat analysis and detection. Systems
like Haloop (Bu et al. 2010) and MapReduce Online (Condie et al. 2010) have sought to add
continuous query support to MapReduce. To achieve this, these systems had to make fundamental
changes to the API and underlying architecture of MapReduce. This paper argues that what is
needed instead is a system designed from the ground-up to support the demands of analysis and
mining algorithms on large sets of continuously generated data.
2.3 Data management and analysis
The problem of analyzing continuous data has been explored by stream databases (Abadi et al. 2005,
Shah et al. 2004). Similarly, continuous queries in databases have been proposed with systems such
as TelegraphCQ (Chandrasekaran et al. 2003) and CQL (Arasu et al. 2006). These systems can
handle processing queries on streams of data with long-running/continuous queries. However, they
lack the ability to support analytic algorithms over a large and diverse dataset. In contrast, verticallypartitioned
databases such as C-Store (Stonebraker et al. 2005) excel at fast and efficient support of
complex analytics. Unfortunately, vertically-partitioned databases suffer from poor performance on
writes. In essence, insertions and updates require that the index be rebuilt. Although performance of
reads is very fast once the index is built, building the index is very expensive. What is needed is a
system that can perform complex analytics on continuous data without requiring a complex index to
be completely rebuilt as a result of data updates. This paper proposes a new, incremental indexing
system that keeps track of summarized historical data while allowing for many small [incremental]
updates to be incorporated. The key difference is that, unlike traditional database indexes, the new
incremental index would not be build off-line (batch-process). Instead, the index would incorporate the
many incremental updates on-line so that the index of past data is always active and valid.
In addition to the storage and distributed computing framework, it is also important to consider the
needs of the algorithms that will be used in the system. Applications with such diverse data require
equally diverse analysis. For example, detecting hidden correlations and associations between events
seen in server logs requires mining association rules (Agrawal & Srikant 1994) whereas detecting
interaction of attackers in a network may involve graph theoretic algorithms.
3. Conclusion
This paper presents a case for a new distributed computing system that is explicitly designed to meet
the unique needs of applications such as cyber-warfare threat detection. The system should support
large quantities of diverse data such as server logs, emails, social-network data, etc. It should allow
for a variety of mining and analysis algorithms and support for those algorithms to be processed in a
parallel and distributed manner. The system must not only meet these needs, but also do so in a way
that can efficiently support continuous analysis of data that is continuously generated.
318