As companies have evolved toward digital business models and undertaken digital transformation initiatives, they have increasingly faced two challenges. First, the data they need to drive their real-time business processes is typically spread across multiple, siloed datastores. Second, their existing applications often cannot scale to address the increase in end-user demands for real-time engagement.
Thanks to the relatively low cost of RAM today and the availability of open source solutions, in-memory computing technologies have progressed dramatically over the last few years, becoming a foundation for accelerating and scaling real-time business processes in support of the range of digital transformation and big data/fast data initiatives. As we move through 2020, in-memory computing will be particularly important in enabling data centers to accelerate the use of the following new strategies for supporting real-time business processes and analytics:
- Highly performant digital integration hub (DIH) architectures for accessing data in real-time from multiple, siloed datastores and data streams. The datastores may span on-premise and cloud databases as well as SaaS datastores.
- Hybrid transactional/analytical processing (HTAP), also called hybrid operational/analytical processing (HOAP), to enable transaction processing and analytics on the same system.
- Creating end-to-end business processes based on a combination of mainframe data and other datastores that leverage DIH architectures.
- Combining in-memory computing platforms with non-volatile RAM to fine tune the balance between optimal performance, data protection and overall system cost.
DIH Architectures
Consider a financial services institution that offers a variety of products, including mortgages, credit cards, core banking services. As customers interact with the company about one of its products, it may want to take advantage of cross-sell and upsell opportunities. To offer those additional products, the institution needs to be able to access a customer’s data from all the disparate business units to understand their current situation and which additional products might be a fit for them. To ensure a positive experience, the required data must be processed and a decision on which new products to offer must be made in real time.
Today, most enterprises will struggle to implement such real-time business processes because the required data may reside in multiple internal and external systems as well as in separate data warehouses and data lakes.
To aggregate and process data in support of end-to-end business processes, developers must typically make API calls to multiple data sources for every action. However, waiting for data to be retrieved from disparate data silos can be slow compared to the real-time experience demanded by the web visitor. The company may also face limitations on the number or types of API calls it can make on the datastores. Further, once the data is aggregated, it must be processed in real time. The complexity of overcoming these challenges makes it extremely difficult or even impossible for many organizations to create the real-time business processes they need.
An approach to solving this challenge is what Gartner calls a DIH architecture. A DIH architecture creates a common data access layer for aggregating and processing data from multiple on-premise and cloud-based sources and streaming data feeds. The DIH architecture’s API services layer automatically synchronizes changes made by the applications consuming the data to the backend datastores.
A DIH architecture can enable multiple customer-?facing business applications to access a single view of the aggregated data and process the data at in-memory speeds without movement of the data over the network. By caching the required data, calls to the siloed datastores can be reduced using change data capture to periodically update the datastore, and delays caused by waiting for data to be retrieved from some datastores can be eliminated. This architecture can also eliminate the need for business applications to make direct API calls to each datastore every time a new customer interacts with the company.
At a recent industry conference, 24 Hour Fitness described its use of a DIH architecture. The system caches data from the company’s SaaS billing system, updating the cache every 15 minutes. This cached data can then be accessed simultaneously and in real time by multiple business applications, eliminating the need for an API call to the SaaS billing system for each operation. Application logic can also be deployed on the server nodes of the in-memory computing platform cluster, enabling parallel processing of the cached data with no data movement over the network. This architecture provides customers and business analysts with fast access to the information they need, and the company can easily implement additional functionality.
Other examples of the use of DIH architectures include:
- Combining current purchase data with inventory levels and web page visit data to enable retailers to power real-time recommendation engines.
- Creating a common data layer for IoT platforms, so companies can process and query current and archived sensor data to drive real-time system awareness.
- Combining and processing travel reservation data from many travel providers in real time to instantly present visitors to a travel reservation website with multiple travel options.
To create their DIH architectures, many companies are turning to in-memory data grids, which can cache data from a variety of underlying datastores, including databases, SaaS applications, and incoming data streams. Application code can then be deployed on the servers in the in-memory data grid cluster to perform parallel processing with no data movement across the network. With the data cached in memory for instant access and with collocated computing on the cluster nodes, line-of-business applications, including consumer-facing websites and back-office applications, can implement real-time business processes that would be impractical or impossible to achieve without a DIH.
Hybrid Transactional/Analytical Processing
For decades, companies have relied on bifurcated data infrastructure to meet their performance SLAs. Separating OLAP systems from OLTP systems ensures analytics won’t impact operational systems. The downside of this approach is that it requires a time-consuming ETL process to periodically copy data from the OLTP system to the OLAP system. Today, the delays inherent in periodic ETL are an obstacle to driving real-time business processes based on OLAP real-time analytics.