Cloud to Cloud and Beyond: Enterprises Tackle New Levels of Data Integration

Image courtesy of Shutterstock.

For years, data integration was a difficult, yet clear-cut task. As new applications were added to an enterprise’s roster, data managers and professionals were tasked with introducing ways to either capture key data elements for analysis within data warehouses, writing scripts, or setting up connectors or interfaces that would enable data from one application to be piped to another. Typically, the data was shifted from one end of the data center to another.

Nowadays, data moves well beyond the data center—as well as beyond corporate walls. The challenge is to support data as it moves from data center to cloud, from cloud to cloud, and from cloud to data center. That was the challenge explored in a recent survey conducted by Unisphere Research among the members of the Independent Oracle Users Group (IOUG). The survey sought to examine the current state of data integration in the cloud era, including the key issues, priorities, and solutions being adopted by organizations. A total of 342 qualified responses were collected and analyzed. Respondents came from organizations of all sizes and across various industries. The survey was underwritten by Oracle (“2015 IOUG Data Integration Cloud Survey,” May 2015).

In today’s diverse computing environments, data ends up in many different places, addressing many demands. Data may need to be moved between on-premises systems and public clouds, between private and public clouds, between different hybrid clouds, or between different public clouds. The rise of big data—in varying formats and file sizes often required at real-time speeds—is proving to be an overwhelming data integration challenge. In the past, data professionals could simply stitch together manual scripts, or plug in connectors or adapters to enable two different applications to pass datasets to each other. Or, administrators and developers could point data sources at their extract, transform, and load environments to bring information into data warehouse environments or analytic applications.

Now, there is high demand for real-time views and insights into data coming from both within and outside organizations. This data needs to be rapidly synced, stored, and managed—while maintaining peak, always-on performance. The traditional methods of data integration are proving too slow, costly, and unreliable for these emerging requirements.

Cloud services—particularly private and hybrid clouds—are now part of mainstream IT, and with it, comes greater comfort in not only storing and fetching data, but also doing the complex, behind-the-scenes integration work organizations now need. The opportunities presented by cloud data integration occur on two levels. First, storing data in the cloud itself affords a degree of standardization and access that is often difficult to replicate across on-premises enterprise systems. Second, as organizations move to hybrid cloud approaches, the need to enable the movement of data between cloud and on-premises environments requires new thinking about the data integration challenge.

The survey found that a majority of enterprises are now implementing or considering cloud-based applications and infrastructure to manage their mission-critical systems. Much of the movement has been to private and hybrid cloud architectures, but public cloud adoption is not too far behind. Close to 30% of respondents have public software as a service; around 16% use public platform as a service; and 16% use infrastructure as a service. Among cloud adopters, close to one-third deploy database as a service or application platform as a service, and one in five deploys data integration as a service.

Data-driven requirements—such as big data analytics and backup—are the motivating factors behind many cloud initiatives, yet may also be holding back cloud deployments. About one in four respondents also cites application and data interoperability as one of the top challenges in moving to cloud

There are now an abundance of choices for achieving enterprise data integration. Most enterprises still rely on the established, tried-and-true strategies for data integration that were perfected in the 1980s and 1990s. Extract, transform, and load and data replication approaches dominate within enterprises used by a majority of organizations in this survey. Many even still integrate with manual scripting. Close to one-third now also employ cloud-based services for application or data integration.

For a majority of enterprises, data integration is a key requirement of their cloud plans. Those leveraging data integration within cloud environments report they are seeing faster data movement to target applications, as well as reduced costs, and increased agility. Eighty-seven percent say the amount of data maintained within cloud systems will increase during the next 3 years, and 42% predict this growth will be “substantial.” 

Two-fifths of data managers and professionals say that they currently integrate cloud and on-premises data.  For both implementers of private and public cloud systems, at least nine out of 10 recognize data integration as important to their efforts going forward. Half of enterprises also now require real-time data synchronization between cloud and on-premises systems—reflecting the challenges and opportunities that lie ahead.

The executive summary is available now and IOUG members may access the full report here.