Qubole, a big data-as-a-service company, is open sourcing its Quark platform, a cost-based SQL optimizer. The Quark project is also available in a SaaS implementation via the Qubole Data Service (QDS).
Traditionally, the datasets generated by data teams are aggregated and copied to a variety of analytics systems to balance performance and cost, but making it difficult for analysts to query data for quick business decisions. Quark, the company says, improves access to data by managing relationships between tables across an organization's databases
With the open sourcing of Quark, Qubole is offering a simple solution to allow developers to route SQL queries across data warehouses, big data SQL engines, and data marts, according to Ashish Thusoo, co-founder and CEO.
Quark models relationships between datasets using database concepts such as materialized views and OLAP cubes, enabling data analysts to take advantage of the fastest and most efficient dataset for their queries. “Quark provides a single endpoint of where the analysts can put their SQL,” Thusoo said. “Think of it as a SQL virtualization layer which makes a decision by looking at the transformations that the analysts have given.”
Quark is distributed as a JDBC jar and works with the majority of tools that integrate with JBDC. Qubole’s global shared Hive metastore provides the central view into data along with connectivity to data warehouse engines such as Redshift and relational databases.
“This is something we saw as being needed by a lot of accounts in our customer base,” Thusoo said. “We eventually decided to give this back to the community with the hope that it will help in simplifying the analysts' experience.”
Future updates to the platform will include extension to the cloud and the ability to stream analytics, according to Thusoo.
“I think with Quark we are hoping that this project can significantly help in simplification of the user experience, especially around the analysts,” Thusoo said.
For more information about Quark, visit the Qubole website and Github.