Big Brother and Big Data

I read recently that US chain Wal-Mart bought the startup Social Calendar, a popular Facebook calendar app which allows users to record special events and birthdays etc. Who would have thought that listing to friends your birthday or holiday details would end up being cross-referenced by Wal-Mart with its own data to generate a more complete profile of their customers?

The data currently collected on individuals and groups and the interesting (and commercial) ways in which this data can be used is still largely unknown. Given enough data, computing power and intelligence, companies can connect the dots in ways we currently could not imagine.

In one recent high-profile example I read about, a US man discovered his teenage daughter was pregnant because coupons for baby food and clothing were arriving at his address from the US superstore Target. The girl, who had not registered her pregnancy with the chain, had been identified by a system that looked for pregnancy patterns in her purchase behaviour.

There are no doubt a myriad of ways to exploit your own company's data to gain a competitive advantage given the right data, tools and motivation. With companies quickly attempting to get to grips with Hadoop and realise some of this potential in the data they already have, it will only be a matter of time before Big Data is at the forefront of an organizations data architecture.

Teradata have recently announced additional functionality to better analyse the data in your Hadoop cluster with the release of the Teradata Aster SQL-HTM software. This is the first to offer business analysts a seamless bridge between standard business intelligence (BI) applications and big data sets stored in Apache Hadoop.

Steven Lawton