The Data Lake Survival Guide

The Data Lake Survival Guide
provided by orders, contact records and so on. If you look at a set of customer data
it may yield information that is not part of any single record. As soon as you have
multiple records you can calculate categories (gender, age, location, etc.). From a
“business intelligence” perspective you are creating information with such activities.
When you link customer data to all the orders placed by a customer, you can generate
useful information about that customer’s buying patterns.
We define information to be: collections of data linked together for human consumption.
In general BI products are software products that present information, possibly as
reports or visually on dashboards. Some BI products are interactive, enabling the user
to slice and dice the information. BI tools such as spreadsheets or OLAP tools can be
thought of as user workbenches for the further analysis of information. The databases
and data warehouses that feed such tools store information in a semi-refined form.
Knowledge
We define knowledge to be information that has been refined to the point where it is
actionable.
Consider the BI tools that simply present information for decision support. The user
knows their own context and consumes the information to take an action, such as
resolving an insurance claim or approving a loan. The user has the knowledge of what
to do and the BI tools assists by providing information.
In the case of the BI tools that enable data exploration, the user has some idea of what
he or she needs to know, explores the information to create that knowledge and then
applies it to their business context. As such the knowledge of how to explore the data
lives in the user. Such a user can accurately be described as a knowledge worker.
Knowledge can also be stored in computer systems. This is where we encounter rules
based systems and all the technology that is normally classified as AI. But knowledge
manifests in computer systems in many other ways. The people who run a business
create business processes to carry out particular activities. These are normally
improved over time on the basis of acquired experience (feedback). They may even be
fully automated - converted into software and implemented without the need for any
human intervention. This is implemented knowledge.
Indeed, all software, no matter what it does, can be classified as implemented
knowledge. Nevertheless within any business there will also be other knowledge:
rules, procedures, guidelines and policies that are not automated and are implemented
by staff.
Data analytics, or Data Science as it has now been named, is the activity of trying to
discover new knowledge from data by applying mathematical techniques to reveal
previously unknown patterns. It is science of a kind, in the sense that the data scientist
may formulate and then test hypotheses, although there are some brute force techniques
that can discover patterns without the need to hypothesize. It is not the only way to
discover new knowledge, but it can be a very powerful and rewarding route.
4