Part 3 – If Hadoop is not Big Data, what is?
In Part 1 we let history show us that the world has changed with Big Data. We no longer need to carefully decide on a small portion of the data which will be kept and analysed; now we can keep all the data we have, add to it data we don’t have, store it for ever and analyse it to gain business advantage.
In Part 2 we saw that you need three ingredients for success in Big Data:
- A Hadoop-like data storage – allowing storage of large amounts of data, doing it cheaply, with no schema, no tuning and a guaranteed retrieval method.
- An Exploration Platform – where ideas can be tried quickly.
- A Data Warehouse – where the definitive enterprise data is stored and accessed.
More than that, you want all three to be integrated. You will need to write queries that combine all three sources, otherwise you need complex load-unload mechanisms between them, combined with data translations and mappings.
And this is why the title of this blog is “Hadoop is not Big Data”. Because Hadoop gives you just one piece of the puzzle: the data storage & retrieval. Your Big Data architecture must have the other two pieces.
So Hadoop can be a component of your Big Data enterprise architecture, but it can’t be your Big Data enterprise architecture.
So, how do you combine all three components into a coherent architecture? The answer can come from one of two places: either the Open Source community will deliver Hadoop integration to every possible database environment (Hadoop has some initial signs of SQL integration), or (and this is more likely) the main vendors will provide seamless Hadoop integration. This is already happening.
Here’s a link to a coherent approach to Big Data Enterprise Architecture that includes all three components: A Unified Data Architecture.
Ben Bor is a Senior Solutions Architect at Teradata ANZ, specialist in maximising the value of enterprise data. He gained international experience on projects in Europe, America, Asia and Australia. Ben has over 30 years’ experience in the IT industry. Prior to joining Teradata, Ben worked for international consultancies for about 15 years and for international banks before that. Connect with Ben Bor via Linkedin.