With the majority of DBTA subscribers reporting the existence of budgets for modernizing their data platforms, the widespread demand for greater scalability and agility becomes a difficult task to balance. While grappling with this challenge, discussions around data architecture—especially regarding how it handles integration, governance, security, self-service, and real-time analytics—become a key focal point.
Experts joined DBTA’s webinar, Top Trends in Modern Data Architecture for 2024, to examine how the evolving world of data architecture can be harnessed to successfully meet growing demands amid a flood of challenges.
Jessica Laughlin, head of product at Materialize, positioned the operational data warehouse as a forerunner in 2024 data architecture trends. Its adoption is motivated by the difficulties associated with building with streaming data—which lags in the technological advancements otherwise benefiting batch data.
Laughlin posed a question: What if working with the freshest data was as easy as working with batch data?
With Materialize’s Operational Data Warehouse, enterprises can act on operational data—or data that enables businesses to build on the present to determine the future—the moment it changes.
“Materialize is an operational data warehouse purpose-built to keep the results of your SQL queries up to date as your data changes,” explained Laughlin. “Just like with your analytical data warehouse, you can send all different types of data to Materialize—your relational data, your SaaS data—but with an operational data warehouse…you can act on that changing data in milliseconds.”
Operational data can be used to query the data live, power operational dashboards or real-time applications, or push that data back to SaaS tools via a real-time, reverse ETL integration. This data can still be sent back to an enterprise’s data warehouse, as well as leveraged to execute big analytical queries on top of it—after the operational window passes.
Saket Saurabh, co-founder and CEO at Nexla, focused on how a metadata-approach can revolutionize data engineering and data management in 2024. With the popularity of generative AI (GenAI) and a continuously growing need to ingest, prepare, and deliver data at scale with reliability and quality, data engineers must contend with massive, unstable workloads.
Saurabh pointed to two factors that can alleviate the pressure of demand on data workers: automation (through a data fabric architecture) and collaboration (through a data product framework).
To successfully implement these concepts, Saurabh noted that it “starts with capturing metadata…[and] taking that metadata into our architectural approach is an essential part in becoming key people supporting those increasing demands that come to the data teams.”
Metadata is the metaphorical glue that binds the data fabric and data mesh frameworks, according to Saurabh. Recognizing that metadata can be generated and captured at every step of the data processing cycle—from the data connector to the database—enables organizations to configure how they work with their data.
With a metadata-driven architecture, enterprises can benefit from:
- Auto-generated connectors
- Logical data products
- Automated data monitoring
- Auto-scaling pipelines
Transitioning to the role of security and access governance in modernizing the data stack, Dhruv Jain, co-founder and CPO at Acante, explained that the mission of Acante is “to help these data teams to unlock the tremendous innovation power of their data without compromising on the security and governance requirements that they have to invariably solve for.”
Jain pointed to two significant challenges: 80% of data attack surfaces are still unaddressed today while security continues to be a top barrier halting the adoption of modern data platforms. Additionally, data platforms are becoming security blackholes that make secure configuration, sensitive data tracking, data privilege management, and accidental or malicious data leakage quite difficult.
Ultimately, as modernization demands grow exponentially, addressing security and access governance is a critical component in actually achieving that digital transformation, according to Jain.
Acknowledging these current modernization challenges, Acante offers a single platform for data security, observability, and access governance, entirely designed for modern data architectures. Acante’s Data Security Intelligence Platform secures lakehouse environments with data sensitivity mapping, data privilege management, data leak protection, and data security guardrails that enables organizations to:
- Setup a secure data infrastructure
- Speed-up provisioning of access to data
- Ensure continuous access governance and compliance
- Proactively prevent sensitive data misuse or leakage
It is nearly impossible to discuss 2024 data architecture modernization trends without paying attention to the impacts of GenAI. Sean Martin, CTO and co-founder at Cambridge Semantics, emphasized that knowledge graph technology will be a significant player in empowering enterprise modernization through GenAI.
Grounding data in actual facts, especially regarding GenAI, is critical for enabling successful AI. According to Martin, “context is everything,” and implementing the mega trends of 2024 starts with unlocking knowledge.
To combat the traditional approaches of siloed data sets, Cambridge Semantics delivers a cohesive data layer that integrates and elevates siloed data across an enterprise. By knitting together a data fabric that is unique to the company’s data, enterprises are empowered to answer ad-hoc, multi-source questions at speed and scale with AI.
This is achieved via knowledge graph technology which simplifies access to complex data to address unanticipated questions. Knowledge graphs can:
- Quickly profile, connect, and harmonize data from multiple sources, including unstructured data
- Present tailored views, services, and experiences to different personas with semantic conceptual models
- Flexibly accommodate new data sources and use cases on the fly, with minimal impact
- Become scalable with CSI
For an in-depth discussion of modern data architecture trends, you can view an archived version of the webinar here.