The Data Lake Survival Guide

The Data Lake Survival Guide
This collaborative approach to software development established a trend. Facebook,
with its exabyte volumes of data was built on open source from the ground up. Imitating
Yahoo, it also had gifts to give Apache: Hive, the quasi SQL capability and the graph
processing system Giraph. Its most recent donation was Presto, the distributed SQL
query engine, for which Teradata now offers commercial support.
Linked In’s open sourcing of Kafka (a persistent distributed message queue) and
Voldemort (a distributed key-value store) can be added to the list of Apache gifts - as
can Parquet (a column-store capability developed in a collaboration between Twitter
and Cloudera).
There is a huge difference between a new Hadoop component, whose development
only recently started, and one that drops magically from the sky into the ecosystem,
fully tested and implemented over years by a highly scaled web business. The first kind
of component may take a long time to mature, while the other is born fully formed, like
Hercules of Greek myth.
The addition of Kafka to the Hadoop ecosystem was particularly important. Kafka
provides a publish-subscribe capability that enables data to be streamed in a managed
fashion from one application to another or from one Hadoop or Spark cluster to another.
In mid-2015, Kafka was joined by another important communications capability that
goes by the name of NiFi, short for Niagra Files. The software was developed by the
NSA over a period of 8 years. It had the goal of automating the flow of data between
systems at scale, while dealing with the issues of failures, bottlenecks, security and
compliance, and allowing changes to the data flow. Taken together, Kafka and NiFi
provide a complete data flow solution that scales as far as the eye can see, and beyond.
If you consider the whole Apache Hadoop stack, the collection of open source software
that emerged in the wake of Hadoop, it obviously constitutes a new environment for
building and deploying parallel software applications. It is remarkably inexpensive,
both because of its open source nature and the commodity hardware on which it runs.
We sometimes think of this stack as constituting a data layer OS, a distributed operating
system for data.
It doesn’t quite quality, but it is close and it is moving in that direction.
14