A View from the Summit – How best to climb the Big Data Mountain

Big Data is a hot topic at the moment; however, I notice as I speak with those at the coal face of generating business results from these new technologies, there is a great deal of uncertainty about how best to proceed. However, there is no doubt that companies need to grasp the nettle and decide on a big data strategy to stay competitive. I attended the Hadoop Summit in Silicon Valley earlier this month where, Geoffrey A. Moore (of Crossing the Chasm fame) put it this way in his keynote address: “Big Data are becoming the eyes and ears of business. Without it, companies have little idea who their customers are and, quite frankly, they’re flying blind and will fail unless they harness the power of big data.” My observations from the summit confirm that Big Data / Hadoop is very much in the early adopters stage of development. The vast majority of the presentations were by technologists for technologist and focused on detailed descriptions of how Hadoop can be implemented to solve very specific, large scale problems. By looking at the vendors at the summit we can also see that there is a huge push to make Map Reduce more accessible to business users – whether through visualisations or through GUI interfaces. This need for a readily accessible interface to Map Reduce was also behind the development of Aster’s SQ-MR and SQL-H technology. So there is clearly an appetite for analyses on large data, subtly structured data sets, but exactly how this analysis is done in a business environment is still an open question. 

Ross Farrelly