Democratizing “big data”

I shouldn't pick favourite locations for our EMEA CTO Road Show, because everywhere we have gone, Teradata CTO Stephen Brobst and I have had the pleasure of meeting lots of smart, engaged people. But I will admit that I always especially look forward to trips to Egypt; a country with a history and culture so bold and rich - and a population so young and energetic. I send my heartfelt best wishes to our Egyptian friends and hope that they can work through the current challenges that stand between them and a prosperous, peaceful and democratic future. And I look forward to my next visit; although if I never again meet the particularly aggressive Red Sea mosquitos that ate me alive in Ain El Sokhna this year, it will be too soon!

I'm also a big fan of witty quotes and adages. The Hindu proverb, "Call on God, but row away from the rocks", ranks, I think as my favourite aphorism – but Egyptian Nobel Laureate Naguib Mahfouz's saying, "You can tell whether a man is clever by his answers; you can tell whether a man is wise by his questions" comes a close second.

Many of my peers in the technology marketing game have clearly never come across Mahfouz's maxim – or at any rate have never stopped to consider its implications – because one of the industry's favourite pastimes is to position existing solutions as the answer to whatever new questions its customers can come up with, regardless of their suitability, apparently heedless of the fact that this behaviour reveals that too many of us are not even very clever, still less wise. Bow your head in shame the vendor currently claiming that your in-memory technology is a "big data" solution - despite the fact that traditional, record-oriented data warehouses are typically getting bigger faster than the unit price of memory is getting cheaper, never mind that multi-structured, "big data" volumes are, in general, growing faster still. And join them in ignominy the vendor currently suggesting that because one customer, somewhere has achieved a 204x compression ratio on an unnamed and unspecified data-set using its technology, that you, too, will see similar compression ratios on all of your data, regardless of their demographics. This latter stunt, is, of course, a neat twist on the established technology vendor dirty trick of Meaningless Comparison And Extrapolation; follow this link for another recent egregious example.

Fortunately for those of us in Marketing roles at Teradata, talented engineers like CTO Stephen Brobst – my travelling companion on tour and the star of our travelling show – have the whip hand at Teradata and keep us from the temptation of emulating our more excitable peers in the industry.

We've been keeping those same Engineers pretty busy recently. After the Prague event, a colleague of mine asked me to help him list the "industry firsts" that we have been responsible for delivering to the market in the last few years, to help him construct an RFP response. I wasn't sure how long the list would be; after all, the trick isn't necessarily to be first, but to implement new functionality so that the resulting solutions perform, scale and can be simply and easily deployed and managed.

Actually, the list from just the last few years is pretty strong. And as I compiled it in a hurry in the departure lounge at Frankfurt airport, I am almost certain that it is incomplete. I reproduce my list at the foot of this post and offer a small prize for any readers who can help me complete it! 

And this week we have another genuine industry first to brag about: SQL-H.

SQL-H gives business analysts "self-service" access to data stored in Hadoop, by extending the Teradata-Aster SQL-MR interface so that objects stored in Hadoop can be accessed as if they were regular objects stored on the Teradata-Aster platform. That means that those same business analysts can now directly analyze vast amounts of multi-structured data without the requirement to master complex MapReduce programming skills, or develop an understanding of how data are stored within the Hadoop Distributed File System (HDFS). Instead, the HDFS objects can now be accessed using ANSI standard SQL and/or the Teradata-Aster SQL-MR extension to ANSI standard SQL that enables end-users to invoke complex MapReduce functions as if they were table functions. The data can be accessed directly and in parallel in HDFS, and/or moved to Teradata-Aster for more flexible and more performant exploratory analysis, as required. And of course, once the data are in the Aster environment, they can then be moved into the Teradata environment so that they can be combined with the data stored in the Enterprise Data Warehouse.

This is a huge step forward for "big data" analytics. Hadoop has enormous potential to enable organizations to collect large volumes of multi-structured data to which we may need to apply multiple different schemas – or none at all. But as Stephen points out in his "Big Data" presentation, it was designed by computer scientists, for computer scientists. The Teradata-Aster SQL-MR interface has already brought MapReduce processing to business analysts without a PhD in parallel computing; now the SQL-H extension brings native access to data stored in Hadoop to the masses. It's the ultimate expression of "big data" democratization.

Martin Willcox
Director of Platform & Solutions Marketing, Teradata (EMEA)

Teradata industry firsts from the last 4/5 years
 

• First analytic appliance vendor to introduce a "platform family" (and still the vendor with the most complete range of models, all of which run the same industry-leading RDBMS);
• First analytic appliance vendor to introduce an all SSD appliance;
• First analytic appliance vendor to support "hybrid storage" (and still the only vendor to extend the RDBMS to include fine-grained and automated data temperate measurement / movement (TVS));
• First analytic DBMS to provide full support for bi-temporal semantics;
• First analytic DBMS to implement true hybrid row / column storage that passes the six "Stonebraker" tests;
• First analytic DBMS to implement automatic, temperature-based data management (automatically compress the "cool" data, whilst leaving the frequently accessed data uncompressed);
• First data synchronization / intelligent query routing middleware framework ("Unity") to support High Availability deployments.

"