However, “it also presents unique data management challenges,” Scott-Murphy continued. “Specifically, edge computing—a key component in processing and storing sensor data—adds another layer in moving sensor data to the cloud for AI, machine learning, and analytics,” he said, with much of this data remaining trapped at the edge or on-prem.
While available data streaming solutions “can move limited datasets to the cloud in piecemeal fashion, they cannot manage massive data transfers, offer limited flexibility in how data pipelines are constructed, and require constant orchestration—all serious impediments in dealing with petabyte and even exabytes of IoT data,” he said. “Additionally, the distributed nature of edge data centers makes coordinating IoT data transfers all the more challenging. Overcoming these challenges will be key in enabling enterprises to realize their IoT ambitions, and the cloud service industry is beginning to make progress in these areas.”
IMAGE ANALYTICS
One of the more compelling use cases for IoT and device connectivity is a new class of applications for imaging, such as video surveillance, video analytics, and sensors. “When it comes to video data from a network camera, there is a lot of actionable information contained in just one image,” said Robert Muehlbauer, senior manager of business development at Axis Communications. “Today’s network cameras have the capability of edge processing where only metadata is transferred, saving bandwidth, storage, and making it much more efficient and faster to analyze and act upon the results. There are analytics that can recognize objects such as vehicles and people. They can also provide descriptive information such as colors and direction an object is traveling. In addition, advanced network cameras with machine learning or deep learning processors can provide more granular and descriptive information such as classification of the vehicle.”
The challenge with this kind of data “is how to manage millions or billions of images to extract the value, and the answer is—metadata, which provides rich, descriptive information in more actionable sizes,” said Muehlbauer. “Education and understanding of the complete system needed to enable a successful implementation is the biggest roadblock.”
The benefits of computer imaging “are only limited by your imagination,” he added. “Traditionally, security cameras have been used to provide safety and security. Now with analytics, there is a tremendous amount of business intelligence that can be obtained from the rich metadata the cameras produce. Currently, there are solutions on the market that can count people, read license plates, monitor traffic flows, recognize loitering, and more. In addition, cameras with open development platforms enable independent software developers to create customized analytics to solve specific use cases.”
ACTIVE AND AUGMENTED METADATA
There is an increasingly important role for metadata, or data about data, across the data-driven enterprise. Metadata traces data lineage and enables the monitoring and observability of data resources. While metadata was previously stored away, it is now seen as a way to deliver value to business operations, said Anu Mohan, director of product management for Teradata. “Active metadata tells the story about the underlying data by providing deeper context, breadth of data sources, and orchestration to create data maps. This, augmented with machine learning for generating insights on the metadata, truly unlocks multiple data dimensions.”
Now, AI and machine learning on metadata “with underlying semantic maps informs data quality, placement of data, optimization of workloads across hybrid, multi cloud deployments,” said Mohan. “With active metadata, data platforms become more autonomous in architecting for optimal costs and data access.”
This includes “laying the foundation with capabilities like augmented data catalogs, data lineage, and intelligent data classification,” said Mohan. “Catalog technologies today automate cataloging and enable users to discover, understand, and access data in a standardized way. Semantic maps-based searches with rich business glossaries enable easier data discovery and take organizations closer to the democratization of data.”
The challenge, Mohan continued, is bringing together “diverse data sources and disparity in formats and various types for metadata. In addition, hybrid multi-cloud deployment models mean the data and metadata landscape is highly distributed.”
DATA GOVERNANCE
With the variety of data management approaches that are becoming prominent, the challenge is to ensure the data is ready for the business. As a result, “data governance is emerging as a crucial technology or set of technologies that help organizations harness data while at the same time keeping them in compliance with various regulatory or internal mandates,” said Rajiv Dholakia, senior vice president of products at Privacera.
Data governance platforms ensure that the data flowing through businesses is trusted data. “The starting point is to match democratization and free access to data with the requirements to use that data in a responsible and trusted manner,” Dholakia stated. Data governance platforms also help ensure “the accelerated delivery of data for new analytical demands.”