To meet the needs of the digital economy of the 2020s, data architecture has evolved into a different animal than it was 10, or even 5, years ago. Most notably, there are three trends that have changed the way enterprises look at and design their data architectures. These are “the rapid rise in cloud services adoption, the increasingly agile and decentralized nature of corporate cultures, and the growing complexity of business problems,” according to Asanka Abeysinghe, chief technology evangelist with WSO2.
Ten years ago, SQL Server instances were cautiously being moved to on-prem virtual machines, observed Kathi Kellenberger, editor of the Simple Talk blog at Redgate and a Microsoft Data Platform MVP. “Today, there are many more options, especially with cloud offerings. For example, in Azure, you have the choice of running SQL Server in a VM but also in a managed instance or even as a database as a service with many options for scaling and performance. It’s also possible to use containers like Docker, which is especially useful for CI/CD pipelines.”
The shift to real-time requirements is also reshaping data architectures in dramatic ways. The desire for scalable, real-time data with proactive or prescriptive analytics has required previous high-latency, disk-based, on-prem batch processes to transition to low-latency, in-memory, cloud-based, real-time streaming data architectures, said Steve Wilkes, CTO and co-founder of Striim.
The catch is that much of the data architecture that has been designed up to this point is not intended to support real-time environments. “Data is generated and moved into a large persistence and analysis layer for ex-post analysis, after the original data has been generated—data at rest,” said Christoph Strnadl, vice president of innovation and architecture at Software AG. “This neglects the power of real-time data and streaming analytics, where data is captured and analyzed during its journey from source to destination—data in motion. Organizations need to foresee a real-time streaming data path for analyzing in flight—the data and event stream.”
The Internet of Things (IoT) is another element that needs to be considered. Previous data architectures “disregarded the vast IoT realm, including Industrial IoT,” said Strnadl. “With [forecasts for] 50 billion IoT endpoints in 2025, devices will be generating roughly 80 zettabytes of data. To capture and monetize this, an IoT platform must be part of the overall data architecture.”
The amount of unstructured data created on-premise, at the edge, and in the cloud on a daily basis “is becoming untenable, far outpacing legacy architectures,” said Ben Gitenstein, vice president of product at Qumulo. “IT and application teams have spent the past decade scrambling to keep up. It’s not enough to just keep adding servers to scale. A modern data architecture needs the flexibility to ingest data from the edge and leverage the scale of data lakes in the cloud, and must also take advantage of cloud-native services like machine learning.”