LS: People’s view on big data depends on their needs. For an on-premise company, big data could be dozens of terabytes or even petabytes and more. But if you look at a customer who is doing work in the cloud - and we have a customers there using our CloudBeam product - big data could be just a few terabytes. What big data means is different for different companies.
And, it is not just about volume and variety; it is about what you can do with it and how quickly you can get there. A lot of organizations buy into the value of big data. They understand the value of Hadoop and they understand they can process a lot more for a lot less and get their hands around a much larger data set. Or, they may look at an Amazon Redshift solution and understand the flexibility and the cost, and the value of it. However, there is often a big gap between that understanding, and being able to capitalize on that value. Getting a new project up and running with Hadoop often takes a lot longer than people expect, and getting things up and running in the cloud can take a long time - and that is where we come in and simplify the process.
DBTA: If a company has unstructured data and traditional data it has got to merge the two at some point to derive benefit.
LS: Absolutely.
DBTA: What are some of the use cases?
LS: Let’s say you want to do more of the near real-time analytics on that, or you might have a lot of social or external data and you want to figure out what is important and wade through the whole data lake and find out what is important to put in the data warehouse that you will use over and over. That is one example of a nice way to marry the two - do all the preprocessing in Hadoop and figure out what you want to do - and then put it in a higher performant, structured environment.
DBTA: The traditional data warehouse still has a lot of value.
LS: At the end of the day, if you are ever going to do a query more than once, it makes sense to put it in a structured environment. Hadoop does not obviate the need for a structured environment, but the two can be combined for additional value.
DBTA: Do most customers of your have a big data game plan at this point?
LS: I would say a good number do. And if they have not implemented something like Hadoop, it is certainly part of most conversations with our sales staff. Customers are trying to understand how they can do it and how all of the pieces fit together, and they come to us for advice and counsel on it. It is hard to find people that are not doing it or thinking about how to implement it, and that is a pretty significant change from a year ago, when it was more of a conversation but not a targeted project focus.From what I have seen and heard in the market, about one-third are in production with a big data deployment, maybe 25% or 30% are doing some sort of pilot, and maybe one-third are in some kind of evaluation phase.
DBTA: What is the problem that you see on the horizon for customers trying to leverage big data?
LS: The biggest hurdle I see is that overall there is a paucity of big data scientists out there. Finding the talent is difficult. I was at an event in Boston and I ran into a professor from MIT Sloan School of Management, which is starting a data analytics master’s program. More universities are getting into this, but there is going to be that gap for a while between the need for data scientists and the supply, and I think that will create issues for businesses. The industry will get there, but it will be challenging until that gap gets filled.