The Data Lake Survival Guide

The Data Lake Survival Guide
We will discuss these processes one by one:
1. Assigning data provenance and lineage. Accurate data analytics needs to be certain
of the provenance and lineage of the data precisely. Prior to the dawn of the “big data
age” this was rarely a problem as the data either originated within the business or came
from a traditional and reputable external source. As the number of sources of data
increases the difficulties of provenance/lineage will increase.
We expect the ability of data to self-identify for the sake of provenance to increase,
although it may be many years before a general standard is agreed. For the sake of
lineage and provenance each event record would need to record the time of creation,
geolocation of creation, ID of creating device, ID of the process/app which created the
data, ownership of the data, the metadata, and the identity of the data set or grouping
it belongs to. To this we can add the details of derivation if the data was derived in
some way, which would allow lineage to be deduced.
Where such precise details are absent on ingest to the data lake, it should be possible at
least, to know and record where the data came from and how. As very few data records
self-identify as described some compromise is inevitable until standards emerge. With
the advent of the Internet of Things, the need to self-defining data will increase.
2. Data security. The goal of data security is to prevent data theft or illicit data usage.
Encryption is one primary dimension of this and access control is the other. Let us
consider encryption first.
Encryption needs to be planned and, ideally, applied as data enters the lake. In a
world of data movement, security rules that are applied need to be distributable to
wherever encrypted data is used. Ideally with encryption, you will encrypt data as
soon as possible and decrypt it at the last moment, when it needs to be seen “in the
clear.”
The reason for this approach is twofold. First, it makes obvious sense to minimize the
time that data is in the clear. Secondly, encryption and decryption make heavy use of
CPU resources and hence minimizing such activity reduces cost.
To implement this approach, format-preserving encryption (FPE) is necessary. The
point about FPE is that it does not change the characteristics of the data such as format
and sort order, it simply disguises the data values. There are FPE standards and vendors
that specialize in their application.
Data access controls require the existence of a reasonably comprehensive identity
management system and access rights associated with all data. Access rights may
distinguish between the right to view and the right to process the data using a particular
program or process.
3. Data Compliance. Data compliance regulations are now common, and likely to
become increasingly complicated with the passage of time. The EU is hoping to establish
a General Data Protection Regulation (GDPR) for personal data that is implemented
across the world and has devoted considerable effort to formulating rules for that. GDPR
22