The Data Lake Survival Guide

The Data Lake Survival Guide
If we now look at
Figure 10, which
illustrates the data
lake complete, the
only two processes,
we have not yet
discussed are data
extracts and data lifecycle
management.
While data lake
processing can be
fast, if particularly
high performance
is required for some
applications, there
will be a need to
export data to a
fast data engine or
database. It will
probably be many
years before data
lake data access
speed gets close
to a purpose built
database.
The truth is that
the focus of data
lake architecture is
data ingest and data
governance. There
are many processes,
all of them important,
competing for the
same resources.
Servers, Desktops, Mobile, Network Devices, Embedded
Chips, RFID, IoT, The Cloud, Oses, VMs, Log Files, Sys
Mgt Apps, ESBs, Web Services, SaaS, Business Apps,
Office Apps, BI Apps, Workflow, Data Streams, Social...
Data
Governance
Data Lake
Mgt
Ingest
Transform &
Aggregate
Archive
Data
Security
Life Cycle
Mgt
DATA LAKE
Real-Time
Apps
Metadata
Mgt
Data
Cleansing
Extracts
Search &
Query
BI, Visual'n
& Analytics
Other
Apps
To
Databases
Data Marts
Other Apps
Figure 10. The Data Lake Complete
There will always be a limit to the capacity of the data lake, and governance processes
naturally take priority, so it will prove necessary to replicate data to other data lakes or
data marts to properly serve some applications or users.
As regards data archive, data life-cycle management can be regarded as an aspect of
data governance. It can best be thought of as a background process. The exact rules
of if and when data needs to be deleted may be influenced by business imperatives
(regulation), but may also be determined by storage costs. Ideally, archive will be an
automatic process.
32