The Data Lake Survival Guide

The Data Lake Survival Guide
The Emergence of Data Governance
Data is not what it used to be. Let’s take this on board. The data we knew and loved
was pretty much static, sitting in databases or files fairly close to the applications that
used it. It wasn’t until the blooming of BI in the late 1990s that the industry felt obliged
to let data flow around. That desire gave birth to the data warehouse into whence data
flowed and from whence data trickled into data marts.
Things began to change. Every year that passed witnessed an increase in the need
for data movement. With the advent of CEP technology to process data streams of
stock and commodity prices for trading banks, we got the first hints of real time data
processing and as time marched forward, such activity grew larger.
It was obvious to web businesses like Yahoo and Amazon, that we lived in a real-time
world. The web logs that drove their businesses were the digital footprints of users
visiting their web sites. A transaction-based world was giving way to event-based
world.
Had we thought about it at the time, we might have realized that it wasn’t just web
sites that lived and died by events. Network devices and operating systems and
databases and middleware and applications were all happily recording their events in
log files that squandered disk space until they were eventually deleted. The computer
networks of the world were already event oriented. They were born that way, but the
applications we built were not.
The first software vendor to notice this was Splunk. It detected and mined a thick
vein of gold in the log files that litter the corporate networks. The advent of Splunk
was a boon to IT departments that often needed to consult collections of log files to
identify the causes of application error. Security teams were also appreciative of the
technology as it helped them to hunt network intruders and vagrant viruses. That we
were entering an event based world was transparently obvious to users of Splunk since
all the data they gathered was event data.
But it was still not obvious to others, and even now with the advent of streaming
analytics, it is still not obvious to everyone.
The Dawn
You would be hard put to find any references to the term “data governance” before
the year 2000. In fact you wont find many references to it prior to 2005. And let’s be
clear about this, it was not that businesses did not care about the governance of data in
earlier years. It’s just that pretty much all corporate data was internal data generated
by the business and most of it stayed put, where it was born, or if it moved anywhere
it made its way into a data warehouse.
Under those circumstances governing the data was not such a pressing need. But, as
we have described, data found the need to move much more often and the volume of
data exploded.
18