The Data Lake Survival Guide

The Data Lake Survival Guide
Data Lake Products
The alternative to acquiring technical staff to understand the nuts and bolts of the
Apache Hadoop Stack to buy technology from one of the Hadoop integration vendors
or as they are also called “data lake” vendors: Cask, Unifi, Cambridge Semantics et al.
Such vendors provide a data lake capability “out of the box.” Many companies have
found this a more productive way to build a data lake than building it from scratch.
Consider the position of a company that wishes to build a data lake that uses, say, 20
components of the Apache Stack. The issues they will inevitably face are as follows:
• Technical skills: New staff with appropriate technical skills may need to be
hired.
• Operational Management: The server cluster will need to be monitored and
managed, altering configurations, provisioning new hardware and tuning for
performance as needed. The software environment for this may need to be built.
• Upgrades: Upgrade management of the Apache Stack is a potential headache.
The Apache Stack upgrades are not going to be as smooth as, for example, a
Windows Server upgrade - simply because the Apache Stack is not under the
close control that a vendor like Microsoft or IBM provides.
• Standards and Integration Issues: The question here is how to align the Apache
Stack with common data center standards (e.g. security) for security, data life
cycle management, etc.
In reality what the data lake vendors do is provide an abstraction layer between the
Apache Stack and the applications built on top of it. If this is done well then the user
could, for example, migrate from Cloudera’s stack to MapR’s stack or even move the
data lake applications into the cloud without concern for whether the application will
function as expected.
The Data Layer OS
As we introduced the concept of a data layer OS it is worth explaining here why
we do not currently believe that the Apache Stack has earned that description. If you
ignore all the infrastructure software that is there to make applications possible, then
we can think in terms of there being three types of applications:
• OLTP applications: These are the applications that process the events and
transactions of the business
• Office applications: This category embraces communications activity from
email to multimedia collaboration and includes personal applications from the
word processor to graphics software. (We would even include development
software here).
• BI and Analytics applications: These are the applications that analyze the
business and provide feedback.
16