DataOps is emerging as a methodology for data scientists, developers, and other data-focused professionals to enable an agile workflow while also adhering to data governance requirements.
Kellyn Pot'Vin-Gorman, Technical Intelligence Manager, Delphix, presented her session, “Making Big Data Bite-Size with DataOps” on Wednesday, May 23, 2018 during Data Summit 2018. Pot'Vin-Gorman discussed how to move from simple DevOps with tools and automation, while making the data central to the solution.
Data Summit 2018 is taking place at the Hyatt Regency Boston, May 22-23, with pre-conference workshops on Monday, May 21. Cognitive Computing Summit will also be co-located at the event.
“DevOps is a big deal,” Pot’Vin-Gorman said. “Everything is getting in demand. We need to marry data and DevOps.”
DataOps takes DevOps to the next level, recognizing that many DevOps projects have data integrated into them and requires that data to move at the same speed the rest of development and testing, Pot’Vin-Gorman explained.
If those managing the data sources aren’t included and working well with those needing that data to produce features and products, friction is the result. DBAs are working with robust, but archaic utilities that guarantee outcome, rarely speed of results. This means developers and testers are spending 80% of their time not carrying out their primary duties.
Virtualization is the answer, according to Pot’Vin-Gorman. A DBA is “only as good as their last backup” and many database tools take considerable time to recover. DevOps is often about automation- automating an “undo” for development and testing that includes data rewind.
Virtualization includes a self-service tool that will allow for rewind without DBA intervention and allows for data version control and DataOps, the next step in DevOps.
With Virtualization, DBAs can create containers or data pods. She explained that creating containers or data pods offers benefits including:
- Development is done by project or feature vs. tier or product.
- With the introduction to the cloud, the business requires those that can do more at a global scale.
- Ease of management and maintenance
- Provide more value to the business
Now the Developers and Testers can work at the speed the business needs them to, she said.
When it comes to upcoming GDPR standards masking the data can assist companies working with DataOps looking to adhere to new privacy guidelines.
DevOps can become DataOps by automating repeatable and complex processes, locating heterogeneous tools/scripting languages that can support multiple tiers of technology, using tools to speed up the development cycle, securing all the data before GDPR, and understanding that automation does not lessen anyone’s value.
Data Summit 2019, presented by DBTA and Big Data Quarterly, is tentatively scheduled for May 21-22, 2019, at the Hyatt Regency Boston with pre-conference workshops on May 20.
Many presentations from Data Summit 2018 have been made available for review at https://www.dbta.com/DataSummit/2018/Presentations.aspx.