Rahman cautioned against assuming a single source of truth or 100% trust; after all, Rahman stated, the world is not perfect. Having highly interoperable, visible, and transparent data is key for businesses that are dependent on that data to trust it as much as possible. SLA’s must continue to evolve, emphasized Rahman, keeping in mind the maturity of data and how it impacts your business. Ultimately, data observability is becoming more of a necessity than a luxury—and will continue to prove its importance in 2023.
Data contracts—what they are, how to use them, and how to make them successful—is on the top of many data teams’ minds as it becomes an industry buzzword.
Rahman explained that data contracts are seeing a resurgence from their prior popularity in the 2010s. Data contract testing has been circling for some time; from an implementation perspective, it exists in a similar socio-technological intersection as data mesh.
Data contracts rely on both running the contract tests as well as informing relevant parties—the upstream data provider as well as the consumer. Every time a release is made, running a contract test is critical for informing and detecting the impact analysis, which then prompts each party to meet and make the appropriate corrections to the schema.
Murray added that data contracts can be most useful for commerce data and its platforms, as an example. The need for SLO’s and agreed upon schema is growing, and data contracts can be significant in navigating that space.
To learn more about the predictions for data engineering teams in 2023, you can view an archived version of the webinar here.