Accordingly, “it’s important to leverage all the information and knowledge collectively about integrations,” McNabb agreed. “Metadata management is an area that has never taken off. The real value is in the metadata around integrations. The more you understand about the metadata element of integration—around processes, usage, mapping—the more insight you will gain, dramatically improving speed and quality. You can get more value out of integration tools, if you can leverage the metadata.”
The lesson enterprises and their data managers have beenlearning as they move to big data environments—such as Hadoop—is that new types of tools are required.
Decision makers are looking for a 360-degree view of their business, and metadata provides that capability. “They need to be able to accommodate new types of data, ask new questions of that data, and have the answers delivered in real time,” said Gorbet. “This requires an agile data integration platform that can accommodate complex structured and unstructured data with heterogeneous and constantly evolving schema. For that you need schema flexibility so that you can preserve all your data and metadata in one place,” he said. “You also need powerful indexing of that data so you can perform flexible and complex queries in real time without having to design your data model around the queries, as well as semantics to draw the linkages between data of different types, and security and lifecycle management to ensure that the data is governed appropriately.”
Be Open to Introducing New Tools and Platforms
In the emerging new data integration world,“traditional relational database tools are not sufficient,” said Benavides. “It goes beyond data warehouse and intelligence systems. Applying complex algorithms such as machine learning or predictive analytics will give businesses a competitive edge over others. Granted, data managers will have more challenges in identifying and operating these tools in this complex data growth scenario but there are new choices popping up every day—from both technology startups and traditional product vendors.”
Of course, moving to new tools and platforms is not an overnight task. Planning and preparation are key. “To maximize the value of their data integration investments, companies need to take an approach that allows them to keep up with the latest innovations but insulates them from the disruption and cost of rewriting all their jobs and acquiring new skills every time a new compute framework comes out,” said Yogurtçu.
The lesson enterprises and their data managers have been learning as they move to big data environments—such as Hadoop—is that new types of tools are required. However, once organizations had moved the data to those big data environments, they found that the level of complexity and time it took to process the data made it inaccessible to the business user, said Ajay Anand, vice president of products for Kyvos Insights. “Connecting to this data from the tools they were familiar with was a frustrating process because the tools were not designed to deal with data at this scale.” It’s important to pay attention to emerging solutions that can help address this bottleneck. “Technologies such as OLAP on Hadoop, sophisticated indexing solutions, parallelized query mechanisms, and others are now becoming more prevalent,” Nijhawan said. “These solutions are leading the way to truly democratize big data and unlock the potential for business use.”
Ultimately, while the job and processes associated with data integration can be complex, simplicity is the best way to go. Transforming to a data-driven organization can be overwhelming at times, said Chatelain, who advised that companies take an outcome-driven approach. “View the transformation as a number of small steps instead of a giant leap.” Another key rule is that if you always keep the end customer in mind, you won’t go wrong, said Wes Higbee, president of Full City Tech Co. “Design your investments to have a marked impact on your customers and you’ll be much more successful, and much less wasteful.”