Data engineering is undergoing significant, rapid growth, where new technologies redefine how enterprises deliver secure, trusted, and accessible data. The proliferation of tools to meet data engineering demands has given many organizations pause as to which solutions are right for them—and which will give way to the most innovative opportunities.
Data engineering experts joined DBTA’s webinar, The Future of Data Engineering: AI, Automation, and the Cloud, to examine key technologies and best practices that are playing significant roles in how data engineering can best manifest in the enterprise.
David Jayatillake, VP of AI at Cube, kicked off the discussion by explaining that navigating data complexity and availability remains a main challenge for organizations in a variety of industries.
Cube aims to remediate this challenge with Cube Cloud’s Universal Semantic Layer, an independent yet interoperable solution that allows every data endpoint—including BI tools, embedded analytics, or AI agents and chatbots—-to work with the same semantics and underlying data.
The Universal Semantic Layer creates a single source of truth with consistent metrics that centralizes and enforces fine-grained data access controls to achieve faster, more cost-efficient outcomes from enterprise data, according to Jayatillake.
“You can’t scale if only certain people can use your interface,” as legacy ETL and cloud data platform solutions lack either performance or usability, noted Mei Long, founding member, product at Prophecy. This challenge is what prompted Prophecy to “bring all data players together into the same arena,” eradicating the disconnect between data engineers, data analysts, and data scientists.
Prophecy offers a Data Transformation Copilot that enables every user to build, deploy, and observe data pipelines that accelerate AI and analytics, without locking them into a proprietary space. Prophecy’s Copilot delivers an AI-powered visual designer for developing data pipelines that turn into native Spark or SQL code, democratizing data engineering for the entire enterprise.
According to Seth Weisman, director of field engineering at Materialize, businesses that act on fast-changing data outperform their competition. “If you can pre-empt your customers' needs, if you can solve a problem before they even know it occurs, they’re going to be happy [and] they’re going to want to work with you over the competition,” explained Weisman.
Yet, current approaches for real-time workloads are challenging, often resulting in increased costs, strains on core systems and operations, high latencies, time consuming processes, specialized engineering talent, and more.
To remediate these pains, Materialize offers the Operational Data Warehouse, a data warehouse that combines real-time data with SQL support to help teams deliver fresh data with the ease and familiarity of a data warehouse. Focusing on data that is most valuable in the moment for immediate actionability, the Operational Data Warehouse:
- Computes on write incrementally and efficiently
- Reflects the current state of the world by joining data without losing consistency
- Drives accessibility with a familiar, declarative interface
- Fits into your existing stack
Matthew Groves, DevRel engineer at Couchbase, argued that yesterday’s data architectures are fundamentally not ready to support AI initiatives. Architectures relying on purpose-built databases lead to unnecessary complexities and data sprawl, driving costs, inefficiencies, and overall time and effort.
Couchbase’s differentiated architecture delivers on performance, flexibility, and efficiency, spanning use cases such as high-speed caching, user experience and personalization, product catalogs, customer 360, mobile and IoT applications, and operational analytics.
Couchbase Capella is a distributed, SQL++ document DBaaS designed for global applications with in-memory read, write, replication, and synchronization. Supporting environments from the cloud to the edge, including mobile and IoT devices, Couchbase Capella helps to eliminate ongoing data management challenges and make way for GenAI implementation, according to the company.
For the full, in-depth discussion of the latest technologies shaping data engineering, you can view an archived version of the webinar here.