SAP AG has introduced a new version of its Sybase IQ disk-based column store analytics server. The overriding theme of this new release, which will be generally available later in the first quarter, “is positioning IQ 16 to go from terabytes to petabytes,” Dan Lahl, senior director of product marketing at SAP, tells 5 Minute Briefing. To accomplish this, IQ 16 provides enhancements in three critical areas – exploiting the value of big data, gaining deeper insights into data, and guaranteeing enterprise class performance.
Enhancing the Value of Big Data
With this release, the column store has been totally rewritten to do better compression for better I/O and traveling metadata management for faster answers, notes Lahl. “We are much better able to compress the data on disk and so we are reducing the input/output even further, which is the slowest part of analytics, even further. In addition, rather than the metadata being stored on separate dictionary pages, we are actually storing the metadata on the pages themselves in order to reduce the I/O even further when the system does dictionary lookups - and we reorganized how we do the dictionary as well.”
SAP has also enhanced the multiplex capability of IQ in two ways to improve batch window data loading. “We have fully parallelized our bulk loading,” says Lahl. Now SAP Sybase IQ can fully break up large tables and split that over multiple nodes so data gets into the system much more quickly. And second, post processing has been improved, he notes. “We have totally rewritten how we are ingesting data into the system.” Now data is ingested into an in-memory row system, so as soon as that data becomes available as a row, it becomes immediately available to any query that wants to query that brand new data, and in the background, IQ converts that row to a column. This process is called “row level versioning” and makes it faster to get data into the system, explains Lahl.
Deeper Insights into Big Data
To support better insights into big data, SAP has scaled out the query engine in IQ 16 by enhancing the parallel capability of the query system across multiplex nodes, says Lahl. “We have continued to push the boundaries on how parallelized queries can become. You can never get to 100% parallelized but we moved up by a great degree.” In addition, he notes, SAP has added the ability to index in the background so that every time data is loaded, it is very consistent. A third key capability in the area of deeper insights is the ability to now have temporary storage, whether it is memory or SSD or hard drive on the nodes themselves and then IQ can pin the most used pages and tables into the caching area. This is called “soft data affinity” and serves to move IQ toward a hybrid shared nothing / shared everything architecture.
Enterprise-Ready Features
The new release also adds total web call administrations so DBAs don’t have to be inside the data center to administer the system and can deal with emergencies wherever they are. In addition, there is added resiliency to the multiplex system so that if, for example, a node goes down, IQ will automatically push a query to another node. And, finally, there is advanced security with LDAP support and granular role-based access support for users.
For more information, go here.