Is it possible to move data any faster than it now travels? There’s no question that the pace of data movement has quickened dramatically in recent years. This calls for new strategies for integrating data at the speed of business. That is the challenge as companies increasingly rely on data analytics in their decision making. In a new survey, a majority of managers and professionals (57%) state their business leaders now rely heavily on analytics in their day-to-day decision making. Conversely, about the same number complain about a lack of complete information. However, most organizations are not where they want to be in terms of data delivery.
The survey, covering 303 data managers and professionals and conducted by Unisphere Research, a division of Information Today, Inc., finds that organizations are employing a range of new strategies and approaches to improve the speed of data delivery and integration. The survey, among members of the Independent Oracle Users Group (IOUG), and sponsored by Oracle, included respondents from organizations of all sizes and across various industries.
Appliances and cloud have become a large part of the data warehouse scene, according to the survey. About one-third use data warehouse appliances, and close to a third of enterprises now use cloud-based data warehouses to fulfill their data integration needs. (See Figure 1.) While conventional data warehouses, built in-house, still dominate, there are a substantial number of companies now doing their data integration in the cloud. In addition, when looking at workloads, 19% say that a significant portion of their analytical workloads are stored or managed in the cloud.
Types of Data Warehouse Platforms in Use
Data warehouse platform configured/built in-house: 55%
Data warehouse appliance/machine: 32%
Software as a service or cloud data warehouse: 30%
Federated data warehouse spread across multiple data environments: 22%
None: 13%
Don’t know/unsure: 8%
Other: 1%
While the ETL (extract, transform, and load) model is still popular, there are a number of additional data integration technologies that are emerging. Data integration is also being accomplished through data replication or the use of networked databases. Common data storage—in which data and documents are available to all applications, independent of formats or architecture—is employed at two in five organizations.
Data Integration Approaches Currently In Use
Data replication: 53%
Extract, transform, and load: 53%
Networked databases: 43%
Common data storage: 40%
Extract, load, and transform: 38%
Manual scripting: 37%
Application service integration: 36%
Cloud-based integration solutions/platforms: 36%
Data services layer/data virtualization: 35%
Middleware/broker: 30%
Common user interface: 23%
Componentized web services: 20%
None: 2%
Don’t know/unsure: 7%
Other: 1%
Are these approaches enough to integrate and deliver data within minutes or seconds, as required in many of today’s businesses? Fifty-seven percent of respondents state that there is now strong demand for delivery of real-time information within their organizations. Only one-third, however, say these approaches are currently capable of delivering most of their data in real-time mode. The key inhibitors to moving data faster are database performance issues and network performance, cited by 48% and 45%, respectively. Data quality, cited by 45%, also stands out as a leading inhibitor to the faster delivery of data.
Overall, 29% indicate that up to 100GB of data is moved to their analytical systems on a daily basis, while 26% indicate that anywhere between 100GB and one terabyte of data is moved. Another 10% move volumes of data that scale into the hundreds of terabytes.
In-memory databases and platforms offer an option to rapidly accelerate data analytics, making it a key step toward real-time integration and delivery. About 28% of organizations now employ in-memory technologies, and another 23% are piloting or evaluating the approach. Lack of understanding may be inhibiting in-memory deployments, however.
In-Memory Database Technology Use
Yes, extensive use: 9%
Yes, limited use: 19%
Yes, pilot project: 6%
No, but evaluating: 17%
No, but considering adoption:12%
No, and no plans to: 24%
Don’t know/unsure: 14%
Greater industry education is called for. Close to half (46%) of data managers and professionals admitted they had only a basic, limited, or minimal understanding of the technology, and 8% said they had no knowledge at all of in-memory.
The most often-used applications support transactional databases and serve as engines for predictive analytics. The top benefits seen so far are improvements in query response times, accelerated access to detailed data, and the ability to eliminate performance tuning—such as aggregates, indices, and duplicate data/systems.
While new approaches promise faster and more comprehensive data delivery, the reach of data warehouses is still limited. A majority (62%) indicate that the primary users of their data warehouses are analysts and researchers. Another 52% count IT professionals among their users, while 54% say their top executives comprise the userbase. Interestingly, only 36% have marketing department staff among their users.