The Data Lake Survival Guide

More documents

Recommendations

Info

The Data Lake Survival Guide Analytics Development Analytics Implementation Passive Decision Support Data Set User Analytic Exploration New Knowledge Interactive Decision Support Data Scientist Data Set Data Set User Automation Data Set Figure 4. Analytics and BI, Development and Implementation Alternatively, the knowledge may be automatically included in an operational system improving it in some way. The illustration does not try to elaborate on how an analytic discovery is made operational, since this varies according to context. The Data Lake Dynamic The fundamental assumption of the data warehouse architecture was that there needed to be a very powerful query engine (database) at the center of the data flow. It thus suggested a centralized architecture where, first of all data flowed to the data warehouse. It was then used in place or it was distributed from there for use elsewhere. The fatal flaw of this architecture was that it did not scale out well. However this limitations did not become apparent until a whole series of forces came into play. They were: • The need to analyze unstructured data, both external and internal. The need for this continues to grow. • External data sources began to multiply. Particularly prominent in this was social media data, but it was by no means the only source. Until recently, selling or renting data was a niche activity but this has ceased to be the case. An expanding amount of valuable data is now bought and sold publicly. • Traditionally analytics applications lived in “walled gardens” served by their own data mart. However a data lake could do service as an analytics sandbox 6
The Data Lake Survival Guide - a useful idea given the incursion of external data that data analysts wished to explore. • Hadoop (and later Spark) with their Open Source software ecosystems gained traction and those ecosystems began growing apace. Such software is very low cost to adopt. There were significant economies available just for off-loading tired data from data warehouses, to a data lake. • The parallelism of these Open Source environments made it possible to run some analytic applications much faster than had previously been possible. • The Hadoop/Spark ecosystems were further strengthened. New metadata capture and data cleansing products emerged. Security products appeared that strengthened the initially poor security capabilities of these environments. • The concept of a data lake (or data hub) quickly gained acceptance as an architectural idea for data management. • Cloud vendors, particularly Amazon with EMR and Microsoft with Azure quickly identified the opportunity and built easily deployed data lake environments. • The release of Open Source Kafka in combination with Spark’s micro-batch capability caused some companies to experiment with near real-time applications significantly expanding the data lake’s areas of application towards real-time analytics. This move will no doubt continue as other open source streaming capabilities, such as Flink, mature. • Kafka can now be regarded as a foundational data lake component (or if you use MapR then MapR Streams fulfils the same role). It was donated to Open Source by Linked In in a fully proven form. As a fast publish-subscribe capability it enables replication and disaster recovery configurations as well as making it possible to treat multiple physical data lakes as a single logical data lake. • The traditional data warehouse was never an ingest point of itself. Data was prepared in various ways, in a “staging area” before going to the data warehouse. In contrast, a data lake is capable of being a staging area, not just for corporate data but for all data, including unstructured data. • The data lake provided a means of acquiring data prior to modeling the data (for inclusion in a database or data warehouse). Since a good deal of data could be processed immediately without such modeling, considerable time could be saved. For applications where “time to market” matters, the data lake delivers. • A whole series of data governance issues began to raise their heads. Data governance had never been an important issue, in fact it hadn’t even been well defined until the possibility of gathering large collections of data into data lakes became a possibility. Then a whole series of issues from data security through data lineage to data archive stumbled into the spotlight. The question of how should data be governed has now become a pressing question. 7
Page 1 and 2: The Bloor Group The Data Lake Survi
Page 3 and 4: The Genesis of the Data Lake The Da
Page 5 and 6: The Data Lake Survival Guide No dou
Page 7: The Data Lake Survival Guide Unders
Page 11 and 12: The Data Lake Survival Guide their
Page 13 and 14: The Data Lake Survival Guide The Ty
Page 15 and 16: The Data Lake Survival Guide YARNin
Page 17 and 18: The Data Lake Survival Guide The Ne
Page 19 and 20: The Data Lake Survival Guide Right
Page 21 and 22: The Data Lake Survival Guide Let us
Page 23 and 24: A General Definition of The Data La
Page 25 and 26: The Data Lake Survival Guide will b
Page 27 and 28: The Data Lake Survival Guide as pos
Page 29 and 30: The Data Lake Survival Guide • Th
Page 31 and 32: The Data Lake Survival Guide Clearl
Page 33 and 34: The Data Lake Survival Guide Data L
Page 35 and 36: The Data Lake Survival Guide Next G
Page 37 and 38: The Data Lake Survival Guide Ingest
Page 39 and 40: The Data Lake Survival Guide are ma

The Data Lake Survival Guide

Create successful ePaper yourself

Delete template?

Save as template?