DM Nov-Dec 2023

MANAGEMENT: DATA ARCHITECTURE Dm
"DECOUPLING COMPUTE AND STORAGE IN PUBLIC CLOUDS IS MORE STRAIGHTFORWARD TO
ADMINISTER AND RELATIVELY INEXPENSIVE. BESIDES, THESE COMPUTE AND STORAGE CLOUD SERVICES
ARE VIRTUALLY UNLIMITED IN SCALABILITY, ELIMINATING LEGACY HARDWARE PROCUREMENT ISSUES.
THEY ALSO OFFER SUPREME LEVELS OF AVAILABILITY AND PERFORMANCE."
Lastly, the independent scalability of both
storage and compute facilitated ondemand,
elastic resource precision, adding
flexibility to architecture designs.
However, these changes took time to
materialise. Expensive Storage Area
Networks (SANs) and less costly but often
complex Network Attached Storage (NAS)
systems have existed for quite a while. Both
storage models were limited due to
administrative and procurement
overheads. Mass adoption of separating
compute and storage only became feasible
with public cloud computing.
Decoupling compute and storage in
public clouds is more straightforward to
administer and relatively inexpensive.
Besides, these compute and storage cloud
services are virtually unlimited in scalability,
eliminating legacy hardware procurement
issues. They also offer supreme levels of
availability and performance. Therefore, the
separation of compute from data brings
forth three immediate benefits:
A significant reduction in complicated
and expensive data copies and
movements as the data warehouse as
the sole source of truth gets replaced
by accessing data in open formats in
the data lake, eliminating data silos.
Open data standards and formats
provide universal data access from
infinite services and applications,
creating the freedom to pick the best
solutions.
An open architecture ensures that
future cloud services can directly access
the data, avoiding going through a
data warehouse vendor's proprietary
format or moving/copying data from
the data warehouse.
THE OPPORTUNITIES OF OPEN
ARCHITECTURE
Cloud data warehouse providers enticed
firms with the allure of scalability and costefficiency
that was unsustainable with onpremises
solutions. However, after
uploading their data into the warehouse,
organisations were restricted entirely to the
vendor's ecosystem or denied access to
other promising technologies that could
extract more value from their data.
Open architecture is a significant
advantage of cloud data lake/lakehouse
over the data warehouse. As a result,
organisations are reassessing their
strategies to use an open architecture that
promotes flexibility and re-establishes
ownership of their data. This shift signifies
three things:
The flexibility to utilise various superior
services and engines on the company's
data. This allows the use of diverse
technologies like superior SQL,
Databricks or any other data-processing
tool. Given that companies have
numerous use cases and requirements,
utilising the best-suited tool yields
higher productivity - especially for data
teams - and lower cloud costs. It's also
important to remember that no single
vendor can offer all the processing
capabilities a company requires.
Not being confined to one vendor.
Platform changes become profoundly
challenging when dealing with a data
warehouse holding up to a million
tables and hundreds of complex
ingestion pipelines. Comparatively, if an
organisation uses a superior SQL on its
cloud data lake today and a new tool
emerges tomorrow, it's possible to
query the existing data with the new
system without migrating it.
The ability to benefit from future
technological advancements. Avoiding
becoming locked-in is crucial, as it
keeps vendors from exploiting a
company financially. But more
significant is the capacity to adopt and
benefit from emerging technology,
even if the current vendor remains
favourable. If a superior machine
learning service or a better batch
processing engine is invented,
organisations can have peace of mind
that they can use the tool freely.
Application architectures have
demonstrated that a service-oriented
approach allows maximum scale,
flexibility, and agility. While separating
compute and storage marked an essential
first step in reducing analytic costs, it
doesn't offer the kind of benefits visible in
modern application architectures.
However, by disengaging compute from
data, the benefits of application design
can now be used for data analytics,
especially given the critical importance of
data for all businesses.
As a result, open data architecture brings
forth many benefits, from flexibility,
independence, and future-proofing to
creating new avenues for gaining valuable
business insights. In the rapidly evolving
digital era, embracing open data
architectures is more than a strategic
choice; it's a decisive move towards a more
flexible, scalable, and insightful future.
More info: www.dremio.com
www.document-manager.com
November/December 2023
@DMMagAndAwards
31