The Data Lake Survival Guide

The Data Lake Survival Guide
Parallelism Playing Havoc
By 2010 we, at The Bloor Group, began to observe a significant increase in the speed of
server software occurring as a consequence of parallelism. That was in the early days
of the Hadoop project, which had been inspired by Google and Yahoo’s use of scaledout
server networks.
Let’s think about this. Google and Yahoo were the first two Web 2.0 businesses. They
had been forced to find their own solutions to big data problems. Searching the web
was a completely new application. No developers had ever built software to solve a
problem like that. First you send out spiders; you run software that accessed every
web site, including new web sites, gathering information about new web pages and
changes to old pages. After you harvest the data, you compress it to the digital limit
and add it to your big data heap. That’s the relatively easy part of the problem. The
hard part is updating the indexes on the huge data heap and letting the world pick at
it. It was the first of these two problems that gave MapReduce its start in life.
The idea of “mapping” and “reducing” was not new. It is a relatively old technique that
emerged from functional programming, a 1970s programming paradigm. MapReduce,
as invented by Google, was a development framework that was scalable over grids of
servers and ran in a parallel manner. It provided a solution to the indexing problem.
It was research activity by Doug Cutting and Mike Cafarella at Yahoo which spawned
Hadoop, with its Hadoop Distributed File System (HDFS) and MapReduce framework.
Quite likely the project would not have taken flight if Yahoo had not decided to throw
it to Apache as an open source project with Doug Cutting in the pilot’s seat. Soon
Hadoop distribution companies sprouted up: Cloudera in 2008, MapR in 2009 and
Hortonworks in 2011.
Under the auspices of The Apache Software Foundation (ASF), Hadoop acquired a
coterie of complementary software components: Avro and Chuckwa in 2009, HBase
and Hive in 2010, then Pig and ZooKeeper in 2011. Soon ASF - a nonprofit corporation
devoted to open source software - acquired a destiny. It provided a well-honed process
for incubating the development and assisting the delivery of open source products. It
wasn’t long before it was supervising over 100 such projects, about one fifth of which
were Hadoop related.
The commercial early adopters of Hadoop began to trickle in around 2012. The trickle
soon became a stream, and then the stream became a river, and the river flowed into a
lake. Hadoop and its growing band of open source jig saw pieces were truly inexpensive.
It was free software until you needed support, and if you cared to assemble a few old
servers, you could prototype applications and experiment with parallel computing for
almost nothing.
Before Hadoop danced into the data center with its scale-out file system, no such
scale-out file system existed. The assumption had always been that if you had data that
needed to scale out in a big way – hundreds of terabytes or petabytes or beyond – you
needed to put it in a database or a data warehouse. Hadoop changed all that.
12