Streamlining
Data integration continues to introduce complexity but also delivers numerous technical benefits as well. Storage is one area that benefits from best practices in this area. “Well-integrated data is easier to store efficiently,” says Akred. “It can also unlock self-service BI, via tools like Tableau or Spotfire.”
Management is also smoother when data is well-integrated. “Simply put, achieving comprehensive data integration provides much faster project implementation times,” says Goodson. “Customers have told us that 40%–60% of their typical IT project budgets are spent on integration. Lowering that figure speeds up project completion but also improves time to value.”
Data integration enables a more streamlined data management operation as well. An emerging approach such as data virtualization, for example, is capable of bringing the concept of single sign-on or a data-access layer which significantly reduces the cumbersome coding requirements of heterogeneous environments, Anne Buff, evangelist in the SAS Best Practices organization, tells DBTA. “Cloud models bring multiple advantages such as improved fiscal planning, reduced maintenance costs, scalability, and flexibility.” Plus, now smaller to medium-sized businesses can take advantage of data services, self-service, and automation, formerly only available to larger enterprises.
“Whether all data is physically moved, accessed virtually through virtualization methods or some combination of approaches, having a single end-to-end view of business data allows for efficiencies in information management, accessibility, usability, and user training,” says Lucker. “You don’t want multiple generations of data appearing in a variety of sources throughout the enterprise. Data redundancy is also eliminated, external data and software licensing can be consolidated with resulting savings, and data currency, timeliness, and accuracy can be better managed and assured.”
With improved data accuracy and quality, Lucker adds, “a business can also have more confidence that a number, metric, or KPI has a consistent meaning whenever used.”
Ultimately, a well-designed data integration effort pays dividends. “The ability to have your data well-organized, harmonized, accessible, and usable for analytics by people without advanced degrees in computer science or statistics is what can help make a company more nimble, innovative, and customer-centric,” says Lucker. “Without quality data integration, it's very hard to speedily access, assess and analyze the data necessary for understanding business issues and achieving strategic objectives.”
According to the IOUG-Oracle research examining big data challenges and opportunities, while 46% of respondents are not yet sure how big data will be incorporated into their BI analysis, 32% indicate that they pre-process big data, then load it into their data warehouse for integrated analysis. This suggests that many data managers who are using big data find the greatest value in integrating the emerging unstructured data world with existing relational data environments. Another 14% say they conduct big data analysis separately from traditional enterprise analysis.
Some analysts do, in fact, suggest that organizations need to give up on the holy grail of integration—the idea that everything can be consolidated and viewed within one system of record. “We’ve all seen, for decades, that it just doesn’t happen,” says Berry. “MDM programs take years and millions of dollars to complete. Integrating social media into everything isn’t feasible. Integrating all the systems that should provide date into, for example, a CRM system, is costly, difficult to maintain and, let’s admit it, provides a less than optimal user experience.”
No Silver Bullet
Goodson cautions “there is no silver bullet for comprehensive data integration. No one solution is going to get business there. This means that organizations need to invest in a variety of products as well as some new skillsets. Just acquiring the expertise to manage these new types of data is expensive. For example, a company might have to now employ someone or buy new software to manage a Hadoop system.”
Thus, the true costs of data integration may be difficult to measure. “Data can be integrated via a multitude of methods and techniques using a variety of tools,” says Lucker. “The costs of these approaches vary widely and the tools vary from highly proprietary to open source. In short, when asking if data integration is expensive, the answer is ‘it depends’—on the objective, time frame, robustness, architecture, and design, support mechanism, and business strategy.”
Still, open source solutions may help in driving down some of the costs associated with data integration. “Utilizing open source Hadoop tools and NoSQL databases is a cost-effective data integration strategy,” says Diana. “Many vendors have integrated Hadoop with their offerings; of course, there will be associated cost. With the plethora of open source and ‘community’ editions of data filtering and integration platforms, it makes sense to exploit these tools to feed the data models for one’s BI and other analytics tools. Don’t forget that these vendors will be two steps behind the leading edge, but they are rapidly catching up—look at SAP’s HANA in-memory big data solution.”
Are there hidden costs such as labor and skills in data integration efforts? “There is no question that human capital is scarce and expensive in the data integration domain,” says Lucker. “Seeding a team with experienced talent and building organically around that is the only approach that promises success,” Akred agrees. He adds, though, that deploying cloud-based approaches will help address data integration with a minimum of new skills.
Ultimately, the best path to success in data integration is effective planning, SAS’s Buff emphasizes. “The technology is often the smallest portion of the data integration solution,” she says. “When implementing a data integration solution, it is important to note what business problems you intend to solve and ensure the rules, standards, and processes implemented as part of the solution actually solve the problem. Without appropriate planning, the technology will just automate your existing problems.”