The Data Lake Survival Guide

The Data Lake Survival Guide
We represent this reality, in Figure 7, by showing real-time
applications running directly against the real-time data
stream within the Data Bus and also accessing a disk-based
data source. In practice this is likely to be achieved by a
Lambda or Kappa architecture, which is far ore involved
than teh diagram suggests.
In practice it is far more likely, for latency reason, that realtime
apps will not be located anywhere near the data lake,
but will instead be as close as possible to the data stream(s)
that feed them. Nevertheless they will likely pass the data
stream they process directly to the data lake. Since such
applications run prior to any data governance it may be
necessary to pass data to these applications if anything important is discovered during
the governance processes.
Governance Applications
We need to distinguish between the governance processing that occurs on ingest and
the governance processing that may occur later. One of the rules of governance itself
may be to specify what processes
have to take place before data is
made available to data lake users.
It makes sense to do as much
processing as possible while data is
on the Data Bus (i.e. held in memory)
since it can be accessed very
quickly. The ideal would be to do
all governance processing on ingest,
but this may be impossible. Some
data cleansing activity and some
metadata discovery activity requires
human intervention and it may not
be practical to do it on ingest. For
some data, the chosen policy may
be to implement schema-on-read so
Data
Security
Data
Transforms
Data
Aggregat'n
Metadata
Mgt
Data
Cleansing
Figure 8. Ingest Governance Apps
metadata gathering will occur after ingest. There can be other competing dynamics. The
desire may be to encrypt all data (or at least all data that is destined to be encrypted)
on ingest. If data is to be stored in the write-only HDFS, this is doubly important, as it
is a write-only file system. However data cleansing will require data to be unencrypted.
The data transform and aggregation activities shown in the diagram are not governance
activities per se. From an efficiency perspective it will be better to perform data transformations,
aggregations and other data calculations that are known to be required before data is written
to disk. Of course, some will need to be done later simply because not all the data they
required is in the data stream.
D
A
T
A
B
U
S
Real-Time
Apps
Data
Figure 7. Real-Time
30