ST Nov-Dec 2023

Dremio 16.qxd 01-Dec-23 10:58 AM Page 2
MANAGEMENT: DATA ARCHITECTURE
af
EMBRACING OPEN DATA ARCHITECTURE
MATT PEACHEY, VICE PRESIDENT, INTERNATIONAL AT DREMIO,
ARGUES THAT OPEN IS THE SMART WAY FORWARD FOR DATA
MANAGEMENT
For the past decades, data has been
propelling business operations. Whether
an organisation offers tangible goods or
intangible services, crucial information about
partners, workforce, processes, and clients
forms the backbone of a company's wellbeing.
At the heart of any computing system is data
storage, therefore the selection of the
appropriate solution to store it will significantly
impact how efficiently an organisation's network
and accompanying infrastructure cater to the
business requirements.
The primary expectation from a data storage
system is to safely keep valuable data while
allowing users and applications to retrieve it
seamlessly and swiftly when required. However,
with the volume of data growing exponentially
and never deleted, businesses started to add
more storage capacity.
The issue deepens where data warehouse
vendors store data in a proprietary format.
Data gets locked into the platform, making it
difficult and costly to extract if and when a
business wants to. Further, maintaining and
troubleshooting issues often require teams with
deep subject matter expertise in the ecosystem -
an expensive outlay.
Given the multitude of data storage
alternatives and system setups, organisations
can get dragged into the rabbit hole whilst
adding more data to their systems - a very
inefficient approach. Organisations must
embrace open-source standards, technologies
and formats to ensure fast and cost-effective
analytics with the best engine for each
workload. This provides the agility to innovate
with the next wave of technology without
draining resources or time.
EVOLUTION OF DATA ARCHITECTURE
Previously, companies depended on
conventional databases or warehouses for their
Business Intelligence (BI) demands. However,
these systems presented certain difficulties. The
typical data warehouse setup requires investing
in expensive on-premises hardware,
maintaining structured data in proprietary
formats, and dependence on a centralised IT
and data department for analysis. Other
obstacles included technical interoperability,
system orchestration, and, more critically,
scalability.
However, things changed in 2006 with the
launch of Hadoop, built on the Map-Reduce
paradigm capable of parallel processing and
producing enormous data sets over large
clusters of commoditised hardware. This
framework facilitated handling vast datasets
distributed over computer clusters, making it
immensely appealing for businesses
accumulating more data with each passing
day. Still, databases like Teradata and Oracle
encapsulated storage, computation, and data
within a single, interconnected system, offering
no separation of compute and storage
components.
Between 2015 and 2020, however, the
widespread usage of the public cloud altered
this approach, enabling the separation of
compute and storage. Cloud data vendors like
AWS and Snowflake facilitated this separation
in cloud warehouses, enhancing scalability and
efficiency. Nevertheless, data still had to be
ingested, loaded, and duplicated into a single
proprietary system, which was attached to a
solitary query engine. Employing multiple
databases or data warehouses necessitated the
storage of multiple data copies. Moreover,
companies were still charged for transferring
their data into and out of the proprietary
system, which resulted in excessive costs.
Enter more contemporary and open data
architecture, where data exists as an
independent layer. This includes highlighting a
clear division between data and compute. Data
is stored in open-source file formats and table
formats and accessed by decoupled and elastic
compute engines. Consequently, different
engines can access the same data in a loosely
tied architecture. In these architectures, data is
stored as its own independent tier source in
open formats within the company's cloud
account and made accessible to downstream
consumers through various services.
This transformation parallels the shift in
applications from monolithic architectures to
microservices. A comparable transition is
presently occurring in data analytics, with
companies migrating from proprietary data
warehouses and ceaseless ETL (Extract,
Transform, Load) processes to open data
architectures like cloud data lakes and
lakehouses.
SEPARATING COMPUTE AND STORAGE
FOR EFFICIENCY
Over the years, there have been many
discussions around the detachment of compute
from storage within the industry, primarily due
to its contribution to enhancing efficiency, which
resulted in several advantages.
Firstly, the reduction in raw storage costs was
so significant that they practically disappeared
from IT budget spreadsheets. Secondly,
compute costs became segregated, leading to
customers paying only for what they utilised
during data processing, which lowered overall
expenses. Lastly, the independent scalability of
both storage and compute facilitated ondemand,
elastic resource precision, adding
flexibility to architecture designs.
However, these changes took time to
materialise. Expensive Storage Area Networks
(SANs) and less costly but often complex
Network Attached Storage (NAS) systems have
existed for quite a while. Both storage models
were limited due to administrative and
16 STORAGE Nov/Dec 2023
@STMagAndAwards
www.storagemagazine.co.uk
MAGAZINE