I’m sure you’ve seen myriad stories about how Big Data is the next big thing and if you aren’t on board you’ll be left behind. I’m sure you’ve also read how Big Data is just data combined hype, and that the bubble is about to burst. I think, as is often the case, the truth is somewhere in the middle.
Approaching how you might exploit Big Data is no different to the process of investigating any other technology opportunity or new source of information. In this case you are trying to find ‘nuggets’ of valuable information in a large river of data. The Big Data tools might be new and unfamiliar, the skills may be different, but the process is not rocket science.
The first task is to develop a plan (sounds different so far, doesn’t it). I’m assuming you will need to get approval for spending time and money on this activity and so some management or governance process applies. Often the first steps into the world of Big Data are taken in an Advanced Analytics, Research or Innovation group – who are tasked with finding new analytical techniques to increase sales, identify fraud or reduce costs.
These groups generally have some leeway to invest in new ideas compared to operational business groups.
The plan should include the new sources of data that you’re planning to exploit – perhaps logs from web sites, location data from phones, RFID tags on items in your warehouses, geophysical or network sensors. Quantifying the exact amount of cost reduction or increase in income or customer satisfaction can be tricky before undertaking the work, but remember we are proposing a discovery process, not a fully-fledged implementation.
The budget will also depend on the cost of acquiring and storing the data you are interested in. The number of staff and duration of the project will depend on how ‘close to the surface’ the valuable data is, and how much work you thing there is to expose the nuggets. Of course, it can help to talk to other organisations or experts who have worked on this type or project before. People with the right expertise and a track record are in high demand, but Teradata has skilled people who can provide assistance.
That help could be simply providing details of case studies demonstrating the benefits to similar organisations. Or it could be helping to develop a business plan, then executing the project to show the return on investment possible when you exploit Big Data.
One last suggestion is to make a space for experimentation in the project, but give it a time-box so that it is not open-ended. There is a fair chance that you’ll make a serendipitous discovery that was not expected. Leave room for this possibility.
By now you should be thinking that getting into Big Data is not too different to investigating any new technology. Have an idea of what you’re trying to achieve, know how long and how much you can afford to invest in this discovery process – and keep an eye out for the unexpected. There may be diamonds in the haystack, as well as needles.
Justin Knol is a Senior Solution Architect for Teradata who is responsible for technology & architecture across Australia and New Zealand. He has extensive experience in a number of ICT disciplines including analytics, application development and delivery as well as project management.