Cloudera is collaborating with Microsoft to build a new open source platform that will reduce the burden on application developers leveraging Spark.
The two entities, together with other open source contributors, have built a new open source Apache licensed REST-based Spark Service, called Livy, which is still in early alpha development.
Livy provides a way for applications to interface with Spark, submit jobs, and programmatically retrieve results.
At its core, Livy is a REST server for submitting, running, and managing Spark jobs and contexts.
Its Client API enables fine grained Spark job submission and retrieval of results synchronously or asynchronously.
The platform’s key benefits include reduced friction in spark consumption, the ability to enable third-party applications to use Spark, and allows the use of new architectures.
“Microsoft is focused on simplifying big data and advanced analytics to make technologies like Apache Hadoop and Spark available for everybody,” said Tiffany Wissner, senior director of Data Platform Marketing at Microsoft. “The collaboration on Project Livy was able to make interacting with Spark easier for developers through a REST web service and able to make Spark enterprise-ready as a robust back-end for running interactive notebooks.”
For more information about Livy, visit www.cloudera.com.