With the abundance of technologies and strategies for modernizing the data infrastructure circulating the industry, IT leaders and data architects are left with an overwhelming heap of architecture patterns and enabling technologies to choose from. Between cloud data warehouses, data lakehouses, data fabrics, data mesh, real-time analytics, streaming IoT, and more, making the choice among these options—and making that choice a successful reality—is a weighty endeavor.
Unisphere and DBTA, in partnership with Radiant Advisors, gathered real-world experts to discuss and evaluate popular data architecture plans in DBTA’s recent webinar, 2023 Modern Data Architecture Trends: New Market Research and Guidance, using its latest market study on these innovative strategies and technologies.
John O'Brien, principal advisor and industry analyst at Radiant Advisors, provided a brief overview of the 2023 survey, explaining that its design was intended to capture respondents’ perceptions of the top five architecture trends.
The first figure posed the following question: What are your company’s drivers for considering and adopting a new data architecture? The results, split into three categories, demonstrated that:
- 49.5% reported its driver as increased operational real-time analytics
- 48.6% reported its driver as enabling AI and ML analytics use case adoption
- 47.1% reported its driver as increased analytics performance, scalability, and agility
- 46.7% reported its driver as enabling broader analytics and self-service
- 41.4% reported its driver as having an integrated data management platform
- 40.5% reported its driver as reduced risk by improving compliance, security, and transparency
Jonathan Thornbury, chief architect at Informatica, concurred that these drivers certainly reflect the industry at this point. Furthermore, he argued that putting analytics ahead of data management and risk, from an architecture standpoint, doesn’t make sense. On the other hand, from the business perspective, it does.
“The point of analytics is to drive insights and make data-driven decisions, so it makes sense that that’s at the forefront,” said Thornbury. “Ultimately, data architects are going to have to play catch up to deal with data management and address the risk.”
Steve Sarsfield, director of analytics and AI at OpenText, agreed with this sentiment, adding that going real time adds a particular complexity despite its allure. Many companies, when asked what real time means, don’t really know—whether it means an hour, sub-second, or how to achieve it.
It also changes depending on the data you’re dealing with, according to Sarsfield. If it’s customer data, one hour latency is real-time enough, yet machine data—where downtime and outages are critical data points—necessitates sub-second latency. Once again, though, these categories are compounded by their quantity; machine data tends to be larger than customer data, increasing overall complexity in achieving real time.
Chad Meley, CMO at Kinetica, explained that “it makes a lot of sense that the two most transformational types of initiatives—real time and AI/ML—are on the top.” He added that, intuitively, people know that business is moving faster, and they must make faster decisions.
“There’s an analytic revolution,” said Meley. “If they’re not investing in AI, they’re going to fall behind.”
Practically, Meley said that to even begin traversing the AI and ML space, enterprises must seriously consider modernizing their data architectures.
Later in the webinar, the panel explored data mesh technology, diving into both the challenges to overcome and business value outcomes.
The top challenge reported in the realm of data mesh was IT organizational or process changes (44%) followed shortly by data governance and security (43%). Regarding business value outcomes, the top three included improved agility and scalability (46%), improved data ownership and data sharing among business groups (44%), and improved data management and quality (44%).
Sarsfield remarked that “the good news about data mesh is that it gives everyone access to all the data that exists in your organization…the bad news is you’re giving access to everyone in your organization. You can see why security rates pretty high here.” Things like access control and security encryption, he further explained, will be key points to focus on.
The panel then moved toward discussing streaming IoT data, divided into the same categories for data mesh. The top two challenges to overcome for this strategy came in at data security and privacy with IoT devices (49%) and data volumes and performance of analytics (48%). For business value outcomes, 42% reported predictive analytics for faster intervention, 41% reported the ability to create new real-time data products, and 40% reported improved real-time business analytics and insights.
Meley, who defined IoT streaming data as a subset of real-time analytics marked by time series and spatial attributes, said that the critical insight surfaced here is that a lot of companies struggle with the volume and performance of analytics. When attempting to analyze this data, most databases are not equipped to handle its specific needs—whether it's a temporal or spatial join. Fortunately, Meley believes that most of the patterns and technology for IoT streaming data is beginning to catch up with the area’s aspirations.
Thornbury added that the ability to have personalized interfaces with customers in real time, as well as the ability to scale, will be crucial factors to consider when moving toward IoT streaming data.
“[IoT streaming data] will give us that 1950s, sci-fi, Jetsons view that we’ve all been waiting for,” he said.
For an in-depth discussion of DBTA’s 2023 market study, featuring more findings, insights, and more, you can view an archived version of the webinar here.