The Data Lake Survival Guide

The Data Lake Survival Guide
The main point is that there can be multiple physical data lakes, each with ingest
capabilities and governance processes, that constitute a logical data lake, as illustrated
in Figure 11. While the diagram implies that all the physical data lakes are running
roughly the same processes, this may not be the case. There are different reasons for
having multiple physical data lakes. Some might exist entirely for disaster recovery
or as a reserve resource for unexpected processing demand or as a dedicated analyst
sand box for a specific group of users. Some may simply be established for time zone
or geographical reasons.
Having multiple physical data lakes will complicate the global approach to governance
and establishing a system of record. However, with an intelligent deployment of Kafka
(and possibly also NiFi) to manage the replication and export of data, ensuring that the
physical data lakes correspond to a logical data lake is achievable.
The system of record is likely to be logical (i.e. spread physically across multiple
systems and data lakes) for several reasons. One particular cause for this that we
believe is worth discussing is Internet of Things (IoT) applications, where the source
data is created and is likely to remain physically remote.
The Internet of Things
The IoT is currently in its infancy, although some IoT applications have existed for many
years, particularly those involving mobile phones. The Uber and Lyft applications, for
example, are complex internet of things applications.
However, such applications are not
what normally spring to mind when the
IoT is mentioned. The general idea is that
there will be some physical domain – a
building or many buildings or a transport
network or a pipeline of a chemical
plant or a factory – and this domain will
be peppered with sensors, controllers
or even embedded CPUs in various
locations that are, at the very minimum,
recording information but may also be
running local applications.
Sensors, Controllers, CPUs
Data
Depot
Depot
Source
Proc.
Depot
Proc.
Figure 12 illustrates a typical scenario.
Consider an example, let us say a car or a
truck or an airplane engine, loaded with
sensors. The data gathered locally needs
to be marshaled in a local data depot,
which may contain considerable amounts
of data, maybe terabytes. Some of that
data will probably be processed and used
locally and there may be no need to send
it to a central data hub. However, there
Data
Central
Central
Hub
Proc.
Figure 12. IoT in Overview
36