The Data Lake Survival Guide

The Data Lake Survival Guide
is rare because of communications error checking procedures (in flight corruption is
most likely hacking). It can be corrupted by any software that rewrites the data. It can
be corrupted by database error (DBA error). To deal with such possibilities, some form
of checksum integrity can be applied.
7. Disambiguation. Disambiguation is an aspect of data cleansing and data reliability.
It refers to the removal of ambiguities in the data. It is a process that would be unlikely
to be applied when data enters the data lake, since it can be a time consuming process.
There is a particular problem with data ambiguity when it comes to people. There is no
bullet-proof global identity system, so identity theft is a fact of life.
There is no standard for people’s names. Sometimes just the first name and surname is
asked for. Sometimes a middle initial or a middle name is asked for. Sometimes the full
name. The name is a poor identifier. People can change their names legally and women
change their names by marriage. People sometimes disguise their names deliberately
for fraudulent reasons. But they may do so legitimately (in an effort to anonymize
their data). Some attributes change (address, telephone number, etc.) and usually do so
without the information being easily gathered. To complicate the picture, the structure
of the customer entity changes over time. For example, the social media identities (on
twitter, etc.) were born only recently.
The consequence of this is that for many businesses cleansing customer data also
means disambiguating the data. There are few software tools that are effective at
disambiguation. Novetta has this capability and IBM also, but they are the only two
software providers we know of with this capability.
8. Audit trail of data usage. A record of who used what data and when needs to be
maintained both for security purposes and for usage analytics. Usage analytics play a
part in query optimization as well as in data life-cycle management.
9. Data life-cycle management. Few companies have formally implemented a general
data life cycle strategy. More likely is that they have a strategy for some data, such as
data covered by compliance regulations, but not all data. Others may have no strategy
at all.
With analytical applications in particular, the need to manage data life cycles is
important, because much of the data used in data exploration may eventually be
discarded as worthless. There is no point in retaining the data, beyond recording that
it was once explored. As data lakes are also used for archive, the use of a data lake
creates an opportunity to implement or tighten up the procedures around data life cycle
management. Life-cycle management can be thought of as the strategy for moving data
to least cost locations as its usage diminishes. Deletion is a possible destination in this.
Metadata and Schema-on-read
The only aspect of governance that we have not yet discussed is metadata management.
The metadata situation is complex and thus we are devoting more time to it than
other aspect of data governance. In overview, the situation is simple: since metadata
determines the meaning of data, the natural preference is for metadata be as complete
24