#bars12

Big Insights from Big Analytics Roadshow

Posted on: January 25th, 2013 by Teradata Aster No Comments

 

Last month in New York we completed the 4th and final event in the Big Analytics 2012 roadshow. This series of events shared ideas on practical ways to address the big data challenge in organizations and change the conversation from “technology” to “business value”. In New York alone, 500 people attended from across both business and IT and we closed out the event with two speaker panels. The data science panel was, in my opinion, one of the most engaging and interesting panels I’ve ever seen at an event like this. The topic was on whether organizations really need a data scientist (and what’s different about the skill set from other analytic professionals). Mike Gualtieri from Forrester Research did a great job leading & prodding the discussion.

Overall, these events were a great way to learn and network. The events had great speakers from cutting-edge companies, universities, and industry thought-leaders including LinkedIn, DJ Patil, Barnes & Noble, Razorfish, Gilt Groupe, eBay, Mike Gualtieri from Forrester Research, Wayne Eckerson, and Mohan Sawhney from Kellogg School of Management.

As an aside, I’ve long observed that there has been a historic disconnect between marketing groups and the IT organizations and data warehouses that they support. I noticed this first when I worked at Business Objects where very few reporting applications ever included Web clickstream data. The marketing department always used a separate tool or application like Web Side Story (now part of Adobe) to handle this. There is a bridge being built to connect these worlds – both in terms of technology which can handle web clickstream and other customer interactional data, but also new analytic techniques which make it easier for marketing/business analysts to understand their customers more intimately and better serve them a relevant experience.

We ran a survey at the events, and I wanted to share some top takeaways. The events were split into business and technical tracks with themes of “data science” and “digital marketing”. Thus, the survey data compares the responses from those who were more interested in technology than the business content, so we can compare their responses. The survey data includes responses from 507 people in San Francisco, 322 in Boston, 441 in Chicago, and 894 in New York City for a total of 2164 respondents.

You can get the full set of graphs here, but here are a couple of my own observations / conclusions in looking at the data:

1)      “Who is talking about big data analytics in your organization?” - IT and Marketing were by far the largest responses with nearly 60% of IT organizations and 43% of marketing departments talking about it. New York had slightly higher # of CIO’s and CEO’s talking about big data at 23 and 21%, respectively

 Survey Data: Figure 1

 

 

 


 

 

 

 

 

 

 

2)      “Where is big data analytics in your company” - Across all cities, “customer interactions in Web/social/mobile” was 62% - the biggest area of big data analytics. With all the hype around machine/sensor data, it was surprisingly only being discussed in 20% of organizations. Since web servers and mobile devices are machines, it would have been interesting to see how the “machine generated data” responses would have been if we had taken the more specific example of customer interactions away

 Survey Data: Figure 2

 

 

 

 


 

 

 

 

 

 

3)      This chart is a more detailed breakdown of the areas where big data analytics is found, broken down by city. NYC has a few more “other.” Some of the “other” answers in NYC included:

  1. Claims
  2. Client Data Cloud
  3. Development, and Data Center Systems
  4. Customer Solutions
  5. Data Protection
  6. Education
  7. Financial Transaction
  8. Healthcare data
  9. Investment Research
  10. Market Data
  11.  Predictive Analytics (sales and servicing)
  12. Research
  13. Risk management /analytics
  14. Security

 Survey Data: Figure 3

 

 

 

 

 

 


 

 

 

 

4)      “What are the Greatest Big Analytics Application Opportunities for Businesses Today? – on average, general “data discovery or data science” was highest at 72%, with “digital marketing optimization” as second with just under 60% of respondents. In New York, “fraud detection and prevention” at 39% was slightly higher than in other cities, perhaps tied to the # of financial institutions in attendance

 Survey Data: Figure 4

 


 

 

 

 

 

 

 

 

 

In summary, there are lots of applications for big data analytics, but having a discovery platform which supports iterative exploration of ALL types of data and can support both business/marketing analysts as well as savvy data scientists is important. The divide between business groups like marketing and IT are closing. Marketers are more technically savvy and the most demanding for analytic solutions which can harness the deluge of customer interaction data. They need to partner closely with IT to architect the right solutions which tackle “big analytics” and provide the right toolsets to give the self-service access to this information without always requiring developer or IT support.

We are planning to sponsor the Big Analytics roadshow again in 2013 and take it international, as well. If you attended the event and have feedback or requests for topics, please let us know. I hear that there will be a “call for papers” going out soon. You can view the speaker bios & presentations from the Big Analytics 2012 events for ideas.

 

We live in interesting times!

In the past 30 years, data was used to record business events and report on business events. Over the last 5 years, data has gotten closer to business. Now data is being used to record business events, report on business events as well as influence business events. We now realize that the more data we record, the more comprehensively data can influence business events.

Hence the excitement of "big data" - it is a business opportunity for each line of business - to influence business events to have favorable outcomes.

The responsibility for technologists is to provide the right platforms and tools to make influencing business easy and simple.

There are TWO relentless forces that are playing out in the big data space to which technology has to respond.

The first force is the diversity of data. As we record more data, we end up having different formats of data to manage. About 20% is relational, but we also have text, emails, PDF, Twitter feeds, Facebook profiles, social graphs, CDRs, Apache logs, JSON formats, …

The second force is the richness of analytics. As we influence more business, we end up having richer analytics to perform. About 20% is SQL, but we also have time series analysis, statistical analysis, geo-spatial analysis, graph analysis, sentiment analysis, entity extraction, …

Note that I am not saying MapReduce doesn't have a diverse set of analytics to do: MapReduce is a way of programming to do analysis - time series, statistical, geo-spatial - each require different MapReduce programs to be written.

Today, the platforms and tools for big data are very complex. They expect lines of business owners to write programs to manage different forms of big data, to write sophisticated programs to analyze big data, to master the management and administration of big clusters and be self-sustaining in managing data quality. This last point is very important - data values change over time. We have to keep values consistent, otherwise our analysis will be wrong and our influence on business will be negative - garbage in, garbage out rule of computing.

As a result, big data is in danger of entering the DIY (do it yourself) space. A line of business is now expected to support big clusters = big administration = big programs = big friction = low influence.

We have to acknowledge these challenges as technologists. If we let big data solutions be a DIY solution, only pockets of enterprise will embrace big data - the rest of the non-technology savvy business leaders will be left out of the opportunity.

We have to simplify this equation. We need to enable line of business owners to benefit from big data a lot more easily. We have to make it simpler for business leaders to get from big data to big analytics.

Our goal, big data = small clusters = easy administration = big analytics = big influence.

This entails solving the following problems:

[1] Make platform and tools to be easier to use to manage and curate data. Otherwise, garbage in = garbage out, and you will get garbage analytics.

[2] Provide rich analytics functions out of the box. Each line of programming cuts your reachable audience by 50%.

[3] Provide tools to update or delete data. Otherwise, data consistency will drift away from truth as history accumulates.

[4] Provide applications to leverage data and find answers relevant to business. Otherwise the cost of DIY applications is too high to influence business - and won't be done.

At Teradata Aster, we are continuing to lead the big data revolution. We have led the revolution for the past 5 years, and helped shape the market and technologies. We are convinced that the path to big data success is to connect it with Big Analytics in the coming 5 years.