The progression of modernization throughout enterprises in a variety of industries is a catalyst for data architecture complexity; while organizations look to cloud adoption and implementing the latest technologies to stay competitive, remnants of legacy structures remain within their infrastructure.
Catering to both modern and legacy components result in data architecture intricacy that impedes business success. This transformation period for many enterprises calls for new approaches to designing, implementing, integrating, and managing systems that handle data.
DBTA held a webinar, “Building a Next-Generation Data Architecture: Key Capabilities and Strategies,” to uncover the ways in which IT decision-makers and data processionals pilot their data architectures to best maintain a market edge in the thick of infrastructure complexity.
Anthony Deighton, chief product officer at Tamr, explained that 2023 is the year of managing data as a product.
Deighton said that data products are an organizing framework for building a data-driven organization. The data product owner will have particular significance, as Deighton predicts that the CDO will view data products as the primary artifact they deliver to their organization.
Responsible for guiding the vision of particular data, engaging the business layer in understanding data needs, and testing and evaluating each iteration of the data, the data product owner must become the translation layer between data scientists/managers and business workers to maximize the value of an enterprise’s data architecture.
Data downtime is the metaphorical house fire that data teams contend with, as Jon So, head of product marketing at Monte Carlo, put it.
Thirty to 50% of data engineering time is spent on data quality issues; 80% of data science and analytics teams spend time on collecting, cleaning, and preparing data; and 12-27% of negative revenue impacts are due to poor quality data, according to research from Monte Carlo, Crowdflower, and Experian. There is a proportional relationship between business outcomes and data quality; the better the data, the better the business.
Ultimately, So noted that detection, resolution, and schema management workflows are the challenge impacting data quality. Data downtime results from a myriad of inabilities, including seeing downstream from data sources, predicting data breakage in data warehouses and lakes, and knowing when data is bad or who to ask for help at the data product stage.
Data downtime looks similar at all companies, which can be remedied by data observability, upheld by five pillars: freshness, volume, quality, schema, and lineage.
Fortunately, data observability platforms are here to detect data issues, triage BI, users, and tables, and then troubleshoot the existing issue. With the Monte Carlo data observability platform, data can be accompanied by metrics, logs, and metadata throughout its journey (from data lakes, orchestration tools, data warehouses, to BI tools) to feed quality data more effectively to an enterprise’s infrastructure.
Paul Lewis, CTO at Pythian, asserted that by aligning your data strategy objectives with your data governance programs, you can ensure the organization’s planning and execution is managed, thoughtful, and measured—enabling a future-focused data architecture.
Data strategy permits for frameworks to do the following, according to Lewis:
- Increase buy-in across stakeholders with data literacy and data trust.
- Establish an early plan that builds upon your future needs.
- Quickly assess potential use cases based on ROI and risk to determine where to begin your organization's data journey.
- Identify the organizational, technological, and people investments needed to migrate to a data-driven organization.
- Map journeys for employees and customers to defined, friction-free engagements driven by data.
Data strategy comes with a particular lifecycle, emphasizing education of corporate objectives, alignment with context, and prioritizing data products; identifying ROI and designing around that figure; and executing business processes with measurable impact. Lewis emphasized that your organizational platform should be designed for your data strategy, and not the other way around.
Data governance also plays a critical role in ensuring that your data strategy can effectively net positive for organizational data architecture; it is the enabler when implemented early and aligned with data strategy initiatives.
Data must be easy to find, trustworthy, timely, and protected to ensure a beneficial relationship between data strategy and governance. Said governance can provide both a defensive and offensive layer of innovation and strength that promotes operational excellence and business transformation, ultimately decreasing OPEX and increasing revenue.
Dan DeMers, CEO and co-founder of Cinchy, highlighted dataware as the answer to solving data architecture complexity.
Quoting the Eckerson Group regarding dataware, “Dataware is a bold reimaging of the relationship between data and applications. It promises to reduce the burden of integration by providing a shared data layer for all of an organization’s applications while simultaneously serving as the base for analytics.”
According to DeMers, dataware reduces the intricacy of data architectures by making API/ETL integration obsolete with bidirectional synchronization and real-time collaboration.
Cinchy’s dataware platform pushes beyond data warehouses and data lakes, enabling it to transition siloed data into enterprise data networks that are both autonomous and fully-collaborative, blending the advantages of data meshes and fabrics within a single platform.
The benefits of this revolutionary approach are numerous, according to DeMers. These include:
- Domain-driven federated governance.
- Cross-application controls consistency.
- Data decoupling from applications.
- Elimination of app-to-app integration.
- Schema plasticity/data contract avoidance.
- Metadata-driven adaptive UI’s.
- Elimination of EUCs for core business processes.
To learn more about strategies for simplifying and powering data architectures, you can view an archived version of the webinar here.