As modern applications are expected to maintain fast, scalable, flexible, and user-friendly qualities, data management technologies and strategies are expected to meet those same conditions to support demanding apps.
DBTA held a webinar, “Rethinking Data Management for Microservices and Cloud-Native Apps,” to highlight a variety of modern data management approaches that will accommodate the world of cloud-native applications that form today’s data-driven businesses.
Mary Grygleski, streaming developer advocate at DataStax, began the discussion by listing the different aspects of data management that include:
- Data Storage
- Data Catalogs
- Data Modeling
- Data Integration
- Data Architecture
- Data Processing
- Data Pipelines/ETL
- Data Security
- Data Governance
- Data Lifecycle Management
Clearly, data management is no simple task, as its subcategories cover a wide range of data handling that can impact the overall business. Traditional ways of handling data, like archaic, monolith systems and data pipelines/ETL, impose significant drawbacks that drain time, finances, and productivity.
The demands of the modern, digital world are extensive; modern systems must be cloud-native, responsive, scalable, resilient, and most importantly, real-time.
Grygleski explained that several shifts in the data industry are driving this change. The growth of real-time, digital experiences that give enterprises a competitive edge, data science, and building AI/ML to enhance business, as well as edge computing to meet the demands of large volumes of data create a demand for a different approach to data management.
According to Grygleski, these conditions require a paradigm shift—perhaps an event-driven approach will remediate the tensions between enterprises and modern application needs.
As microservice architectures, messaging middleware, and the demands of consumers become today’s norm for data management, Grygleski pointed to Astra Streaming with Apache Pulsar from DataStax as the effective solution.
Apache Pulsar, a unified, distributed messaging and streaming platform that is both open source and cloud-native, guarantees message delivery with a lightweight, serverless functions framework and tiered storage offloads.
DataStax’s Astra Streaming and Luna Streaming provide management for Apache Pulsar, as well as enterprise support backed by an open source community.
Luna Streaming is a support offering for Apache Pulsar or DataStax Luna Streaming distribution of Pulsar in a customer-managed/on-premises environment, while Astra Streaming distributes Apache Pulsar as a service, in a fully-cloud offering that is part of Astra.
Astra Streaming is an event streaming platform that offers pre-built and sink connections, managed Apache Pulsar with low latency and scalability, as well as flexible consumption that allows customers to pay as you consume.
Grygleski explained that event streaming can transform your business to meet modern application needs. By applying machine learning models, enterprises can make data-driven decisions with quick responses and cost optimization for changing events. Real-time dashboards and data enrichment further enhance the data quality an organization can possess for modern application needs.
Steve Wilcockson, product marketing for data science technologies at KX, emphasized Grygleski’s point that the rise in microservice popularity changed everything for enterprise data management.
According to Snyk’s JVM Ecosystem Survey, 49% of businesses reported that microservices was the architecture of the main application they develop, while a significant 42% reported they still build on a monolith architecture. Regarding how many microservices enterprises have in their primary applications, the survey found that 34% have 1-5 microservices, 36% have 5-10, and 16% have 20-plus microservices.
The vast amount of microservices a single enterprise can have results in a minefield of complexity that absolutely requires efficient configurations to perform effectively.
Wilcockson then presented kdb Insights by KX, where “not a single byte is wasted in the pursuit of optimal efficiency.”
kdb Insights is a cloud-ready time series database and analytics engine that aids enterprises in modernizing their cloud-native analytics stacks and developing MLOps pipelines. With the smallest memory footprint (800KB), kdb Insights avoids data movement and natively supports array operations and parallel computations with both Python and SQL language support.
Leveraging kdb Insights for data analytics, paired with other services such as Azure as a cloud provider, Azure Data factory for pipeline orchestration, and Azure Machine Learning, MLflow, KX, and Python for model machine learning lifecycle, can help “squish the stack” in simplifying microservice architectures.
Wilcockson concluded by stressing that data and configuration guide modern software design. Furthermore, while microservices can help scale and simplify, they can increase complexity.
By centralizing the cloud data pipeline in a primary platform, or providing specialist services in a monoglot or polyglot homebrew, enterprises can aid in meeting the ramping demands of modern applications.
For in-depth examples and discussions of data management technologies and strategies to accommodate modern application needs, you can view an archived version of the webinar here.