A new methodology is on the rise at insights-hungry enterprises looking to bring improved quality and reduced cycle times to data analytics.
Borrowing from Agile Development, DevOps and statistical process control, DataOps is poised to revolutionize data analytics with its eye on the entire data lifecycle, from data preparation, to reporting.
However, improving the flow of data between managers and consumers within an organization through greater communication, integration and automation is no simple task, and it requires cultural changes as well as enabling technologies.
DBTA recently held a roundtable webinar featuring Dan Potter, VP of product marketing, Qlik; Douglas McDowell, chief strategy officer, SentryOne; and Chris Bergh, CEO and head chef, DataKitchen, who discussed key success factors and emerging best practices in the DataOps space.
Data analytics users now demand a “subscription experience,” said Bergh. Yet, analytic teams are failing to deliver this subscription experience.
A Gartner survey titled, “Data Management Struggles to Balance Innovation and Control,” says that only 22% of the time spent by data analytics teams delivers innovation and new insight.
What you do is much less important than how you do it, according to Bergh. There has to be a change in focus and mindset to include:
- Decreasing cycle time of change, continuous deployment
- Lowering error rates in production, increasing customer data trust
- Improved collaboration inter and intra team; less meetings and bureaucracy
- Measure your process show increased productivity, lower cost and more insight
DataOps are the technical practices, cultural norms, and architecture that enable: rapid experimentation and innovation for the fastest delivery of new insights to our customers; low error rates; collaboration across complex sets of people, technology, and environments; and clear measurement and monitoring of results.
Intelligent DataOps is analytics for DataOps, according to McDowell. The process allows for observability across the entire data pipeline and observability for process. Data testing is not optional and it allows users to map their data estate. Intelligent DataOps also relies on SQL because DataOps needs observability and performance and RDBMS deliver.
According to Potter, Qlik Data Integration can efficiently capture large volumes of changed data from heterogeneous sources and deliver analytics-ready data in real-time to Cloud Platforms where it can be catalogued for discovery and provisioning to analysts and data scientists.
DevOps mixed with DataOps can accelerate time to insight, Potter said. There are 5 key DataOps technology requirements, including:
- Continuous integration
- Universal integration
- Automated data pipeline
- Agility
- Trust
An archived on-demand replay of this webinar is available here.