BUSINESS BEFORE PLATFORMS
What kind of platform or technology—be it data warehouse, lake, fabric, or mesh—is proving to be most effective at maintaining data and delivering insights? For starters, it’s important to think in terms of business requirements, versus platforms, Werner pointed out. "Don’t put technology before people and processes,” he advised. “Given the skills of the people you have now, and how you want to upgrade those skills over time, what operating model for your data capability will deliver immediate value and then allow change over time? Depending on wider organizational strategy and current maturity, the right way forward could be wildly different. Only then can the question about the best flavor of data platform be meaningfully answered. The most mature organizations will likely need a blend of several."
Well-designed business analytics shouldn’t be dependent on technology choices. “The question for organizations shouldn’t be about determining the fit for one technology or paradigm, but rather it should be ‘what’s the use case?’” Gnau said. “It’s most important to understand the principles of each and know your business’s goals to decide which approach, or combination of approaches/solutions, is best suited for the use cases at hand. Ultimately, a data lake or warehouse can help store data, but without a well-orchestrated architecture, the data remains inaccessible and wasted—or not efficiently addressable—regardless of where it lives.”
Curran, on the other hand, sees continuing value in data warehouses. “In terms of insights in the purest form, the data warehouse still reigns supreme,” she said. “Insights are the result of the analysis of clean, consistent, and timely data.” However, technology choices still should be superseded by business choices, she added. “There is a misnomer that data insights will magically result from a technology. In reality, organizations that want to keep a competitive edge will have multiple data solutions and embrace a data-first culture.”
Still, older technologies may likely impede efforts to meet increasingly stringent data privacy regulations and localization laws. “The complexity of these regulations combined with the limitations of traditional centralized stores of raw data, such as data warehouses or data lakes, can slow down innovation,” Stalla-Bourdillon said. Demands around compliance will help drive movement to modern data architectures such as data mesh and data lakehouses, she predicted. “In these new paradigms, we think the winners will be those that can also achieve strong data security and governance. There are no shortcuts.”
DATA WAREHOUSING IN THE 2020s
Data warehouses are still effective, but their users and administrators need to proceed with extreme caution. “During the big data era, the so-called enterprise data warehouse got a bad reputation for being slow, expensive, limited, and centralized,” said Werner. “The reaction against it probably swung too far in the other direction. We still meet customers dealing with the fallout of dumping 10-plus years of data into a lake that can still only be accessed by a narrow subset of expert individuals.”
These lessons need to be applied to finding ways to repurpose data warehouses for broader enterprise use, Werner continued. “For example, most enterprises still need a way of reconciling and joining customer data across all their different business systems, to enable financial reporting at subsidiary and group level. Organizations must also be realistic that an incredible amount of value in data can be realized without it first being centralized and perfected."
Data warehouse technology has not stood still, however. “Companies are expecting more capabilities from data warehouse vendors than simply storing data,” said Curran. “Performance expectations have increased with the introduction of cloud and customers are looking for more functionality. This is giving rise to new data platforms that will not store and query data but will be able to simplify an increasingly complex data management stack.”
The typical data warehouse today “is quite different than what ran large enterprises 10 or 15 years ago,” McGuigan agreed. For example, many data warehouse platforms now support both analytical and transactional capabilities in the same data set—a trend “that has been happening for years now but is finally becoming mainstream,” said Stalla-Bourdillon. “The cloud has helped break down that traditional wall between OLTP and OLAP workloads. Organizations will still need core data warehouse functionalities, but they will push this paradigm to the limits in terms of its ability to handle more workloads within the same platform, or better opt for a unified data architecture capable of merging the best of those two worlds—data lakes and data warehouses, or data lakehouses—thereby avoiding the need to have a new data set deployed for every new use case.”
The data warehouse paradigm promised a “single source of truth for all of an organization’s data,” said Gnau. “However, exploding data volumes and distributed data challenges are making moving all of this data into a centralized location, whether in the cloud or on-premises, a tall task on its own. This is forcing businesses to rethink the movement of data altogether. Even with cloud data warehouses, the challenges of centralization remain.”