The Data Lake Survival Guide

Data Lake Architecture
The Data Lake Survival Guide
Let’s begin with the idea that the data lake is the system of record. And let’s be clear
what we mean by this. The system of record is the system that records all the data that
is used by the business. It also holds the golden copy of each data record.
For simplification the system of record should be thought of as a logical system. It may
be possible to implement it on a single cluster of servers, but this is not a requirement
and should not be a goal. In practice the whole configuration will probably involve
multiple clusters if, for no other reason, than to provide disaster recovery.
The system of record should also be the system where governance processes are
applied to data. Clearly, data needs to be subject to governance wherever it is used
within the organization. Some governance processes, particularly data security, need
to be applied to data as soon as possible after its creation or capture. For that reason,
the data lake will inevitably be the landing zone for external data brought into the
business, so governance processes can quickly and easily be applied it. Data created
within the organization should be passed to the data lake immediately after creation so
that governance processes can be applied.
It is best to think of the data lake as a store of event records. We can think of it in
this way: events are atoms of data and transactions (the traditional paradigm with its
traditional data structures) are molecules of information. The analogy works reasonably
well.
Consider for example a web site visit. A user lands on site, clicks through a few pages,
decides to buy something, enters credit card details, and clicks on the “confirm” button.
In transactional terms we may think of this as a purchase: a molecule of data, that’s
quickly followed by a delivery transaction, another molecule of data to record.
But in reality it’s a stream of events, a series of atoms of data. Every user mouse
action creates an event. The computer records these events and responds to each one
immediately; it displays new web pages or expands text descriptions or whatever.
The purchase confirmation is just another event, distinguished only by the fact that it
generates a cascade of other related events in the application or in other applications.
When you look at it like that, business systems consist of applications generating events
and sending event information to other applications. There is nothing particularly
special about web applications in this respect; all applications are like that. They
respond to events and send messages or data to other applications. That’s been the
nature of computing for decades.
When you scan all the servers and network devices in a data center you find them
awash with log files that store data about events of every kind: network logs, message
logs, system logs, application logs, API logs, security logs and so on. Collectively the
logs provide an extensive audit trail of the activity of the data center, organized by
time stamp. It happens at the application level, at the data level and lower down at
the hardware level. Hiding within this disparate set of data can be found details of
anomalies, error conditions, hacker attacks, business transactions and so on.
28