The Data Lake Survival Guide

The Data Lake Survival Guide
By employing schema-on-read you get value from the data as soon as possible. It does
not impose any structure on the data because it does not change the structure that the
data had when loaded. It can be particularly useful when dealing with semi-structured,
poly-structured, and unstructured data. It was difficult and often impossible to model
some of this data and ingesting it into a data warehouse, but there is no problem getting
it into the data lake.
In general, schema-on-read allows for all types of data and encourages a less rigid
organization of data. It is easier to create two different views of the same data with
schema-on-read. Also, it does not prevent data modelling, it just makes it necessary to
defer it.
The Data Catalog and Master Data Management (MDM)
There needs to be a catalog for all the data in the lake. It is important and will become
increasingly important the data catalog to be as rich as possible, embodying as much
“meaning” as possible. It may help here if we explain what we mean by “data catalog,”
as the term is open to interpretation.
The data catalog for an operating system (such as Windows or Linux) is the file system.
The metadata it stores for each file is incomplete, as its primary purpose is to enable
programs to find and access files. The programs themselves will know the layout of
the data and thus how to interpret the values in any given file. This arrangement is not
ideal because the data cannot be shared with programs that do not understand the file
structure. It is adequate only for data that never needs to be shared.
The data catalog for a database is the schema. The schema defines the structure of all
the physical data held in the data sets or tables of the database and seeks to provide
a logical meaning to each data item by attaching a label to it (Customer_ID, Amount,
Date_of_Birth, etc). The amount of meaning that can be derived from the data catalog
depends upon the specific database product. For relational databases the catalog
roughly reflects the meaning that is captured in an ER Diagram, where the relationships
between specific data entities are indicated. With NoSQL and document databases the
situation is similar, although how the schema is used varies from product to product
With an RDF database (sometimes called a Semantic Database) the catalog can record
a greater level of meaning. This is an area where Cambridge Semantics, with its Smart
Data Lake solutions currently excels. The simple fact is that semantic technology
allows you to capture much more meaning for the data catalog than you can using ER
modelling.
A traditional data warehouse metadata catalog can be complex, but nothing like as
complex as a data lake catalog, which may include many sources of unstructured
data (document data, graph data) that may require ontologies (semantic structures) to
accurately define the metadata.
The point of the data lake’s data catalog is to provide users (and programs) with a
data self-service capability. On a simple level, if you compare the pre-data-lake world
of data warehouses and data marts to the data lake world, two obvious facts emerge:
26