DM Nov-Dec 2023

Dm MANAGEMENT: DATA ARCHITECTURE
Embracing open data architecture
Matt Peachey, Vice President, International at Dremio, argues that open is the smart way
forward for data management
For the past decades, data has been
propelling business operations.
Whether an organisation offers
tangible goods or intangible services,
crucial information about partners,
workforce, processes, and clients forms
the backbone of a company's wellbeing.
At the heart of any computing system is
data storage, therefore the selection of
the appropriate solution to store it will
significantly impact how efficiently an
organisation's network and
accompanying infrastructure cater to the
business requirements.
The primary expectation from a data
storage system is to safely keep valuable
data while allowing users and applications
to retrieve it seamlessly and swiftly when
required. However, with the volume of
data growing exponentially and never
deleted, businesses started to add more
storage capacity.
The issue deepens where data warehouse
vendors store data in a proprietary format.
Data gets locked into the platform,
making it difficult and costly to extract if
and when a business wants to. Further,
maintaining and troubleshooting issues
often require teams with deep subject
matter expertise in the ecosystem - an
expensive outlay.
Given the multitude of data storage
alternatives and system setups,
organisations can get dragged into the
rabbit hole whilst adding more data to
their systems - a very inefficient approach.
Organisations must embrace open-source
standards, technologies and formats to
ensure fast and cost-effective analytics
with the best engine for each workload.
This provides the agility to innovate with
the next wave of technology without
draining resources or time.
EVOLUTION OF DATA ARCHITECTURE
Previously, companies depended on
conventional databases or warehouses for
their Business Intelligence (BI) demands.
However, these systems presented certain
difficulties. The typical data warehouse
setup requires investing in expensive onpremises
hardware, maintaining structured
data in proprietary formats, and
dependence on a centralised IT and data
department for analysis. Other obstacles
included technical interoperability, system
orchestration, and, more critically, scalability.
However, things changed in 2006 with the
launch of Hadoop, built on the Map-Reduce
paradigm capable of parallel processing and
producing enormous data sets over large
clusters of commoditised hardware. This
framework facilitated handling vast datasets
distributed over computer clusters, making
it immensely appealing for businesses
accumulating more data with each passing
day. Still, databases like Teradata and Oracle
encapsulated storage, computation, and
data within a single, interconnected system,
offering no separation of compute and
storage components.
Between 2015 and 2020, however, the
widespread usage of the public cloud
altered this approach, enabling the
separation of compute and storage. Cloud
data vendors like AWS and Snowflake
facilitated this separation in cloud
warehouses, enhancing scalability and
efficiency. Nevertheless, data still had to be
ingested, loaded, and duplicated into a
single proprietary system, which was
attached to a solitary query engine.
Employing multiple databases or data
warehouses necessitated the storage of
multiple data copies. Moreover, companies
were still charged for transferring their data
into and out of the proprietary system,
which resulted in excessive costs.
Enter more contemporary and open data
architecture, where data exists as an
independent layer. This includes highlighting
a clear division between data and compute.
Data is stored in open-source file formats
and table formats and accessed by
decoupled and elastic compute engines.
Consequently, different engines can access
the same data in a loosely tied architecture.
In these architectures, data is stored as its
own independent tier source in open
formats within the company's cloud account
and made accessible to downstream
consumers through various services.
This transformation parallels the shift in
applications from monolithic architectures
to microservices. A comparable transition
is presently occurring in data analytics,
with companies migrating from
proprietary data warehouses and ceaseless
ETL (Extract, Transform, Load) processes to
open data architectures like cloud data
lakes and lakehouses.
SEPARATING COMPUTE AND STORAGE
FOR EFFICIENCY
Over the years, there have been many
discussions around the detachment of
compute from storage within the industry,
primarily due to its contribution to
enhancing efficiency, which resulted in
several advantages.
Firstly, the reduction in raw storage costs
was so significant that they practically
disappeared from IT budget spreadsheets.
Secondly, compute costs became
segregated, leading to customers paying
only for what they utilised during data
processing, which lowered overall expenses.
30 November/December 2023 www.document-manager.com
@DMMagAndAwards