As the Hadoop ecosystem matures, the consensus in the industry has been that adoption of Hadoop technology is steady amid continuing disruptive innovation within the open-source framework.
To get a better understanding of exactly what’s going on in the market today, Decisionworx, in collaboration with The Bloor Group, has published findings from its Hadoop in Transition, From Proof-of-Concept to Production’ survey. The final analysis shows some interesting findings and might surprise some market observers who have been following with Hadoop adoption.
This graph shows results on the promoted benefits of Hadoop, for all respondents answering the question.
At a high level, respondents reported many benefits of Hadoop. However, despite the growing pool of reported success stories, the researchers found that we are still in the early stages of Hadoop adoption.
There seems to be a gap in organizations’ abilities to engage experienced resources with Hadoop skills early in the adoption and "productionalization" stages. Clearly, acquiring or developing skills is one of the most widely cited challenges of integrating Hadoop into the enterprise, as is solution complexity and system integration.
Of those who indicated they have no plans to use Hadoop, 37% noted information technology as their job function and 18% indicated that they were consultants. This large percentage may be indicative of a growing interest in what the Hadoop ecosystem has to offer and a desire for awareness of the environment as opportunities for adoption emerge over time.
The survey results also indicate that the most frequently cited business drivers for Hadoop adoption include predictive analytics, the ability to use a data lake, data warehouse augmentation and real-time analytics. The researchers suspect that the more mundane use cases such as a data lake or data warehouse augmentation provide practical value while positioning the organization to embrace more sophisticated application capabilities like analytics. However, data ingestion, enterprise integration, application development and integration within the data architecture are considered challenging.
As a result, researchers conclude that organizations desiring to stay competitive as Hadoop becomes more mainstream should invest in proactive education and training programs for their staff members. In addition, organizations should expect from the outset that they will most likely need to partner with Hadoop support providers and “scaffolding” technology vendors to supplement and hasten the design, development, deployment and continuous operations of their Hadoop solutions.
Although Hadoop’s adoption has continued to increase steadily worldwide during this early stage, the data integration process is often underestimated and cobbled together, requiring significant investment in IT resources. Thus, leading companies are looking to leverage effective data integration solutions to help automate the process, which can significantly reduce complexity and the need for additional hires and training. Companies can now install and ramp up a solution quickly to move big data from many different sources into and out of Hadoop and data lakes with ease.
The technology to make big data valuable, from dashboards and storage architectures like Hadoop, to the replication tools needed to connect everything, are available and becoming more affordable. The historical divide between database professionals and analytics leaders has become pronounced but must be bridged to unlock value. Businesses must erect a robust infrastructure to deal with the new status quo, complete with specialized solutions that enable automation, timely data ingest and data usage analytics, which help manage and monitor Big Data infrastructures. Taking the abstract ideas behind information use and turning them into a real plan is admittedly a tall order - but it is this process that eases us out of transition and makes for a successful big data deployment.