I often talk about the Data Artist and the key role this highly-skilled professional plays in enterprise-level analytics (see my recent blog post on the subject and my chat with CNBC's Chris Morris on the topic for more). The data artist blends engineering and statistical know-how with intuition and novel problem-solving abilities to uncover insights and create value from data. But you can’t get very far without an analytic environment that has the power and flexibility to match the artist’s own imagination. That’s where a discovery platform comes into play.
The discovery platform is a critical component within the big data architecture. Neither the first stop for data loading nor the final word in enterprise reporting, the discovery platform is the data-driven laboratory where trends, connections and insights can emerge under the watchful guidance of the data artist. Here is where many of today’s “aha” moments happen, often in striking ways not unlike how traditional film photography buffs must feel when – between the early steps of loading negatives and the later stages of selecting prints for distribution – there is that superb moment when darkroom chemicals coax new, often beautiful patterns into existence for the very first time.
That’s where the analogy ends, because rather than just seeing what develops, the data artist is actively managing the discovery platform to extract both patterns and solutions from data. As my colleague Randy Lea has written, the key is to go beyond simple transaction analysis and instead study the interaction between diverse forms of structured, unstructured, and multi-structured data by combining multiple analytic techniques – Map-Reduce, SQL and Statistical functions – to truly uncover unique insights. This discovery approach has revolutionized use cases ranging from fraud prevention in finance and yield optimization in manufacturing, to risk assessment in insurance and customer behavior in retail, healthcare and other service fields. A quick look at our own market-leading Teradata Aster Discovery Platform – used by Gilt Groupe, Barnes & Noble and other major companies – helps show how this is done.
Take a key goal like reducing customer complaints, where we employ end path analysis to reverse engineer the many ways complaints can arise and then use statistics to isolate the strongest paths to target as complaint conduits. Comprised of Teradata Aster Database and Teradata Aster Discovery Portfolio, our Teradata Aster Discovery Platform achieves this by examining interactions between web, text, machine, sensor and other forms of data to reap crucial insights that, in turn, can lead to new lines of inquiry and iterative drill-downs. This process can isolate specific behaviors (not just human, but also product, machine and supply chain behaviors) to target as crucial factors that govern whether a complaint is made. With minimal time, cost and effort, Teradata Aster Discovery Platform is the fastest path to business value from raw big data and the easiest and most productive platform for the data artist to develop Big Data apps.
A chief advantage lies in the requirements: Teradata Aster Discovery Platform does not demand the same data completeness needed for data warehousing, and it can analyze transactional, non-relational and variously structured data at multiple volumes and speeds of acquisition without requiring strict service level agreements, or extensive data modeling and pre-preparation. The data artist working a discovery platform takes a “good enough” approach to data, focusing on discovery and exploration instead of balancing books down to the penny. Our method at Teradata is to use the discovery platform in tandem with an integrated data warehouse to help enterprises get both flexibility and precision from data, because you need both to reap the best insights and value.
I have to stress that we’re not talking here about the discovery platform as a simple sandbox for data experiments. Teradata Aster Discovery Platform is enterprise-ready and SQL compliant for production scale analytics. And the same powerful capabilities that afford flexibility for the data artist can also be leveraged for accessibility by enterprise users. Teradata Aster Discovery Platform provides a single SQL interface to access, join and analyze multi-structured data with a patented SQL-MapReduce framework that is easy to use by any SQL-savvy analyst or business user. We offer the industry’s first Visual SQL-MapReduce® Functions and out-of-the box big data functionality for integrated data acquisition, data preparation, analysis and visualization in a single SQL statement.
In some ways, the data artist is a close cousin to the police detective, journalist, research scientist or any other good investigator who must know how to identify new leads and follow them to new insights. But like any other professional, the data artist needs the right tools. Chief among these is a discovery platform for dynamic, flexible exploration of data. The result is more innovation, value and competitive advantage from analytics.