There is no doubt that data mesh principles resonate with so many data professionals, particularly those looking to move beyond brittle, monolithic architecture.
However, adopting data mesh can seem daunting, due to both a scarce but improving ecosystem of tools, as well as organizational change management.
At Data Summit 2024, Elliott Cordo, CEO/founder/builder, Data Futures, LLC, presented “From Daunting to Doable: The Evolution of Data Mesh.”
The annual Data Summit conference returned to Boston, May 8-9, 2024, with pre-conference workshops on May 7.
Data mesh lends itself to evolutionary adoption, helping organizations to leverage existing platform investment and gain incremental value, according to Cordo.
“What I’m really focused on is the maturation and modernization of data,” Cordo said. “I’m a software engineer first but my superpower is data engineering.”
Adopting data mesh principles can actually bring data engineering to similar Maturity as other well engineered software products.
A data mesh is a composable data architecture of data products with decentralized ownership and development, he explained.
The 4 data mesh principles include:
- Domain driven ownership
- Data as a Product
- Self-service data platform
- Computational governance
“Data is not an afterthought, it’s not at the end of the pipeline of the application,” Cordo said.
According to Cordo, there are several questions to ask when in the Data as a Product phase:
- What are my analytic requirements? How will I measure the success of this feature/product?
- What are the requirements of my data, both within my domain/team, and that of my customers?
- How can I robustly share my data model as fully as possible?
“This is a big cultural [process],” Cordo said. “Maybe have an award system around some metrics.”
When it comes to the technology part of data mesh, data should be easy to discover and understand; easy to publish; easy to consume; and …address multiple modalities, usage patterns—tables, streaming, files, etc. This enables the federated model.
“The industry is understanding that this is how they want to work, and they are developing tools,” Cordo said. “I do believe that AI helps us move into these proactive capabilities.”
Because creating a data mesh involves a culture change, Cordo recommends investing in product manager data literacy skills.
“Don’t take your data engineers and split them up from their teams,” Cordo said. “I highly recommend starting semi-embedded.”
The best place to start is to first focus on making agile teams data producers. You can adopt data mesh principles by publishing and consuming within a single data warehouse or data lake environment or via data sharing.
“Interoperability is key,” Cordo said.
Many Data Summit 2024 presentations are available for review at https://www.dbta.com/DataSummit/2024/Presentations.aspx.