The Data Lake Survival Guide

The Data Lake Survival Guide
Real implementations are much more complex, usually involving a data staging area
where data is placed prior to ingestion into the data warehouse. This may be necessary
for operational reasons such as the data warehouse needing to limit data ingest to
particular times. Alternatively data may need to be cleaned or restructured before
ingest. In some instances, because data took too long to flow through to a data mart,
yet another database - called an Operational Data Store (ODS) - would be created to
provide a more timely service to BI dashboards.
The need for such awkward manoeuvres might have been eliminated in time by the
increasing power of hardware. However, this approach was constrained by other factors
all of which we will identify and discuss in detail later.
The Value of Data
Businesses do not remain static. Their processes change and evolve, their business
models change and the markets they serve are gradually reshaped. Precisely how this
happens varies, but generally we can think of there being a simple feedback loop, which
governs the process. We illustrate this in Figure 2.
The feedback loop has three steps:
• Plan (business process design and implementation)
• Run operational business processes
• Review operational business processes
There may be many manual elements in this; it is rarely
driven by IT, although IT normally contributes. BI and
analytics have the clear role of providing information either
to assist operational business processes or to assist planning
and change management activities.
Planning &
Change
Management
Operational
Activity
Conceptually, there is nothing new at all about this view of
company behavior and the role of BI and Analytics. What
has drawn attention to Big Data and the BI & Analytics
applications it supports, is that the technology parameters
have shifted dramatically, as we shall discuss later in
this report. If the Big Data opportunity is pursued, the
efficacy of this corporate feedback loop will be improved.
Organizations will be more “data driven” than before and
their success will be more dependent on making effective
Business
Intelligence
& Analytics
Figure 2. Change
use of technology for this purpose. This is why some businesses are now chanting the
mantra, “data driven, data driven, data driven.”
The triumph in 1997 IBM’s Deep Blue computer in a chess match with world chess
champion, Gary Kasparov, the later victory in 2011 by IBM’s Watson computer system
against three Jeopardy champions and the recent (2016) victory by Google AI against the
world Go champion have demonstrated beyond argument that computer intelligence
can now outstrip the most intelligent humans in well-defined contexts.
2