Author Archive

How Smartphones Turn Sport into Art

Posted on: April 11th, 2014 by Hermann Wimmer No Comments


Apart from all the interesting speeches and discussions, there is one other thing I really like about the Universe: every year the conference takes us to another amazing city which I’m always keen to explore. What impressed me most in Prague is the city’s lively atmosphere and active character. Wherever I looked, I saw athletic people jogging and running along the banks of the Moldova with colorful clothes, headbands, earplugs – and of course smartphones. They seem to have become the inevitable accessory for any serious runner. Smartphones not only support sportsmen and women by playing their favorite music or audiobooks, they also allow them to take snapshots during a beautiful nature run. And – increasingly important – they let them quantify their sports activities. As I recently mentioned, my colleague Stephen Brobst gave an interesting talk about the trend towards self-quantification on Tuesday’s Universe keynote and predicted that it might lead to personal data warehousing with cloud technologies.

Many people also like to share their results on Facebook, Twitter or specific sports platforms like Runtastic. Why? Because it motivates them: Just like your real-world running partner will make fun of you if you cancel your weekly running date for the third time in a row, your athletic social media friends are able to virtually survey your fitness activities and can put a little more pressure on your inner couch potato.

Sharing the results of workouts and training sessions is not limited to hobby runners. Professional athletes like the German triathlete and double winner of the Ironman Triathlon World Championship in Hawaii, Normann Stadler also let the world participate in their activities. But while normal people’s postings usually just briefly pop up and then disappear again, celebrities’ updates are often shared and commented by hundreds and thousands of other users. A single post might travel a long journey through the virtual space. Therefore, our Tuesday keynote speaker Jer Thorp, self-proclaimed “artist, educator, thinker” invented his “Cascade” project: it illustrates the way a single tweet evolves in social media by turning texts and pictures, hashtags and links, comments, tweets, and retweets that we see on our one-dimensional timeline into informative yet artistic videos. When passengers on a plane at Kennedy International Airport observed a flight-attendant having a fight and then quitting his job via the emergency-evacuation chute, one of them spread the word on Twitter. You can imagine that this tweet has come a long way of sharing and retweeting. That’s why Cascade created this beautiful impression of the tweet’s journey.

And Jer is not the only person to create art with social media. In the “Phototrails” project, for instance, scientists analyzed millions of Instagram photos taken in major cities around the world and combined them to infographics. They show the urban lifestyle and rhythm of metropolis in a fascinating way, for example a normal day in San Francisco or Bangkok. Even times of crisis can be displayed in an impressive manner, for instance with this graphic of pictures taken in Brooklyn, NY during Hurricane Sandy.

Taking our smartphone with us wherever we go and whatever we do (even for running a marathon), allows us to virtually share our thoughts, activities and emotions 24/7. Be it with status updates on our physical activities or with pictures we take of (more or less) exciting objects. In a sum, that data is much more than just information about an individual person. It allows us to recognize and illustrate cultural contexts and movements, thus displaying our generation’s zeitgeist. And really, isn’t that just what art is all about?

Data Discovery, Paradigm Shift for the Business

Posted on: April 8th, 2014 by Yousaf Mirza No Comments


Interpreting, analysing and understanding data has been the holy grail of informed decision making. Be it for tracking the performance of the business, competitive analysis, research or as simple as managing your expenses. Humans apart from gut feeling or previous knowledge now more than ever rely on getting key facts and figures before making a decision.

"Data itself is of no value if it is just being stored and not converted into useful information or actionable insight." 

From the overall business perspective the main reasons that are making it harder and harder for businesses to stay alive are:

  1. Market saturation
  2. Cut throat competition
  3. Proliferation of profits

The business analysts are now being challenged and pushed to get additional information and insights that is not the same as the regular and traditional reporting which is used to monitor the business. Yet, everybody these days accepts and works on the principal of collecting more and more data about their business. After working with many companies, in my experience most of the departments in a company selectively pick and collect as much data that they can handle and analyse efficiently.

Interestingly if you talk to decision makers about their problems, the key focus would be that they are not getting all or most of the data they need to understand their business better and stay competitive.

So the fear of the unknown lies in these basic questions which rattle the minds of business analysts to get to the business insight:

  1. Is there value in the data that is discarded or not collected?
  2. If we get the data how will we explore and understand the data?
  3. What if we don’t know the format of the data?
  4. How to apply new/complex/advanced techniques to analyse new types of data?
  5. Do we have any system or technology to play with the new data?

All these questions and problems point to a Data discovery solution and environment where the business analyst could have the luxury of:

  • Exploring and trying new data sources and formats
  • Possibility of late binding the structure to the data if required
  • Try new functions that are suited to different data formats and multiple sources
  • Last but not the least a platform to explore new  ideas with ease of use

Teradata’s Aster discovery platform has these key benefits which help the business analyst to apply pre built functions to handle, manipulate and explore multi structured data that was previously out of their reach. Getting a single environment to discover data is a crucial part of keeping the thought process alive and answering new questions that are critical to the growth of the business.

Yousaf Mirza is the Senior Analytics Consultant at Teradata Australia/New Zealand. Yousaf has experience in the field of Advanced Analytics and Big Data. Recently he has been involved in Big Data projects using Aster Data.


Experts – a Threatened Species?

Posted on: April 8th, 2014 by Hermann Wimmer No Comments


Against the background of our latest technology announcements, which I described this morning, some of these questions seem to be inevitable: While technology is making business intelligence faster and easier, is it fair to say that experts are a threatened species? Will the future be an era of agile generalists who can play every role, supported by data-driven decisions? The panelists of Sunday’s media roundtable at Teradata Universe including data journalist Nate Silver, Forrester analyst Martha Bennett and Teradata CTO Stephen Brobst, together with moderator Duncan Ross discussed the answers in detail.

Nate surprised his listeners with an opening statement that was quite humble for a data journalist, saying that data is not magic and there is a considerable risk of using them inappropriately because, as he put it, "we're all dumber than we think we are". Predictions often fail, simply because fitting a data set is not the same as predicting future developments. Just think of Google’s analysis of flu trends. The tool that tracks web data such as keyword searches led experts to drastically overestimate swine flu cases in 2012/13. People did not google swine flu because they were actually ill but because of all the media reporting. Martha also warned that during the current hype, business specialists might commission data scientists to do technically sophisticated analyses and then come back with hardly surprising results like "people are more likely to buy warm boots in winter". This gave Stephen the idea to make an ironic attempt at defining experts: he would differentiate between self-proclaimed experts who mainly use data to justify their existing opinions, and experts who truly have expertise and use data to form their opinion and improve their decisions. So does the problem lie in the human race? Should we just step aside and let the data take all decisions?

No, then we would definitely have got the panelists wrong. They all came up with lots of examples from various industries where predictions deliver highly accurate results. The question is whether data scientists did their homework properly in the first place: providing excellent data quality and the right analytic and predictive models. Because the old term 'garbage in - garbage out' remains valid and big data increases its relevance by orders of magnitude. Stephen profiled a good data scientist as someone who masters the art of designing experiments accurately and sticks to measurement rather than assumptions. He sees the dotcoms at the frontline of this new experimentation era with a daring attitude of simply trying everything.

Up to this moment, the panelists achieved such consent on the issues that moderator Duncan Ross switched to a provocative question: can traditional journalism survive given the rise of data journalism and a growing preference for facts rather than opinion. Nate clearly favored a writing approach, backed by more evidence and less speculation. Journalism should go back to its roots and ask good questions. Stephen put it in a nutshell: once you have the right question, finding the right answer is easy. Well, it of course still depends on the technology you use, as the CTO of Teradata is well aware of. A good data scientist and a good journalist have one thing in common: their profession is mainly about finding the right question. Martha added that data scientists should adopt the journalist’s skills of storytelling. Throwing a bunch of numbers at a decision-maker might not be enough to convince him. You need to be able to bring the data to life!

Ancient Town, New Technologies

Posted on: April 8th, 2014 by Hermann Wimmer No Comments


The Teradata Universe is running in high gear and just like the other participants I always find it difficult to choose from the huge amount of first-class sessions I would like to attend. Although there never is much time available, we took the opportunity to explore the so-called "golden city" of Prague on Sunday. Since our business is all about gaining insights, information and intelligence, the Czech capital as one of the first and most renowned university cities in the world offers an ideal environment for this year’s Universe conference. Just think of the term university, which by definition describes a community of learners and teachers. Now isn't that exactly the kind of community that takes shape at every Universe conference?

Prague has a rich history as a center and hub of information, research and higher education dating back to the Middle Ages. It is home to the Charles University, the oldest one in Central Europe founded back in 1348 and the Czech Technical University, one of the oldest institutes of technology in Central Europe, already founded in 1707. Most interesting from our business intelligence point of view may be its Department of Cybernetics which conducts research in fields like pattern recognition, biomedical engineering, data mining, machine learning and artificial intelligence – all of which is highly relevant for the solutions we develop at Teradata. Speaking of which, here is a short summary of all the exciting news we presented at the media briefing on Sunday afternoon:

For starters, we announced the new Teradata Active Enterprise Data Warehouse 6750. With its capacity of 61 petabytes of data and the ability to manage the workload of virtually unlimited concurrent users, it sets new efficiency standards for data warehousing. And we introduced our new flagship, the Teradata Database 15. Being a new database version, it comes with several enhancements and industry-leading innovations. The most important one perhaps is the Teradata QueryGrid™ capability, which allows users to orchestrate queries across several databases and other data sources throughout the enterprise. This way, users can analyze data that is not currently stored in the data warehouse but in other repositories such as open-source Hadoop.

With QueryGrid™, Teradata has launched another major innovation and continues to drive the shift from a single data warehouse to the Logical Data Warehouse – a term introduced by Gartner analyst Mark Beyer in 2011 to describe an analytic architecture which combines the strengths of a traditional RDBMS with alternative data management and access strategies. Teradata introduced such a Unified Data Architecture in October 2012.

Besides the Query Grid™ functionality, the Teradata Database 15 also allows for new programming languages such as Perl, Ruby, Python or R to give application developers more flexibility. We extended its temporal and geospatial capabilities which now include the “z coordinate” to represent height or altitude. This will be advantageous, for example, when analyzing the performance of products in a supermarket by their height in the shelf or when it comes to city planning.

Last but not least, customers can now add data in JSON (Java Script Object Notation) to their data warehouse and analyze it with all the features and functionalities of the Teradata Database. That innovation is particularly significant for the Internet of Things and Industry 4.0, as many sensors and microprocessors generate data in JSON format. I am convinced that this innovation will soon be put to good use by our growing customer base – especially when it comes to enhancing manufacturing processes, a field in which we have a major customer announcement coming up on Tuesday. So stay tuned!


It’s difficult to make predictions – especially about the future. Yes, you heard that quote attributed to Yogi Berra probably more than once. But as I am writing this preview for our upcoming Teradata Universe Conference in Prague, I have to think about it over and over again. The conference will begin tonight and I am in a “Universe State of Mind”. The big agenda for the coming week already pushes me to consider many different issues. And part of it is about prediction. We at Teradata provide various industries with technologies that enable them to precisely predict customer churn and take action before the customer turns away. You can read one of my posts on the power of predictive analytics here. The secret here is that it is simply not enough to rely on historical data. You also need to deploy sound statistical data modelling to guide your forecast.

One major reason I am so excited about attending Universe is Nate Silver – “the Lord and God of the Algorithm” (Jon Stewart). Nate Silver, who will be one of the keynote speakers, is most certainly best qualified to teach us how to make better predictions than others, because that is exactly what defines his astonishing career. Although he most recently revealed that he hates data-driven as a term, he probably invented data-driven journalism. His award-winning website FiveThirtyEight has gone through quite a development since its early days. Back in 2008, the surprising outcomes of the Clinton-Obama primaries trashed traditional pollsters’ prediction models. But Nate Silver stood out because he was able to do the right math. Early on election day in 2012, FiveThirtyEight gave Obama a 92 percent probability of winning the national election, while pundits saw the race as “too close to call”. In the end, Nate Silver had correctly forecasted the winners of all 50 states.

Silver has a lot more to talk about than political polls for sure. After working as a consultant for KPMG, he used sabermetrics for analyzing baseball (I blogged about it from last year’s Universe) and even earned a living as a professional poker player before he started his own blog. Meanwhile, FiveThirtyEight moved from the New York Times to ESPN, covering subjects such as travel and lifestyle. In his book “The Signal and the Noise: Why So Many Predictions Fail - but Some Don’t”, Nate Silver introduces us to the art of prediction. The piece is a must-read for every data-savvy professional out there. In his view, foxes are considerably better at forecasting than hedgehogs because they choose from a set of different methods and don’t just look at new information in order to come up with the groundbreaking idea. They are less likely to mistake noise for a sign and therefore less likely to fail.

Failure is also one of the main topics of the popular British behavioral economist Tim Harford – “our chief Economist storyteller” (The Independent), who will address the conference on Tuesday. In his 2011 release “ADAPT: Why Success Always starts with failure”, Harford reminds us that we do not allow enough complexity to enter our thinking. Solving complex problems requires running trial and error – and must include the readiness to fail. I guess this is hard to take in a world where we are all under pressure to be efficient and want to make it right the first time. In politics we prefer decisive leaders with big plans that are supposed to get rid of all problems at once. And we particularly dislike politicians who retreat from their announcements – although this is often the very right thing to do in a fast-changing environment. As Mr. Harford stated in an interview with Ezra Klein, eventually failure does pay off: “The theory of relativity and Google and penicillin more than make up for all the failed experiments, theories and businesses.” It’s the strategy of muddling through that leads to success. Small incremental moves add up to big gains.

Understanding reality requires more than data and technology. I believe that next week’s meeting will lead us to a broad discussion on how we extract the highest value from a complex and data-rich world – a prediction that, for once, I find easy to make.

Making the Job Easier for the Chief Financial Officer

Posted on: April 3rd, 2014 by Guest Blog No Comments


The role of today’s CFO office is not what it used to be, it is no longer enough to just be delivering financial results and efficiently funding the business, over the years the role has evolved into also advising on strategic planning, analysing and delivering operational efficiencies and driving value creation within the organisation.

There is a need to make faster decisions, adapt quickly to market changes and have early insights into leading indicators, a need to predict future income and costs with a higher degree of reliability and a need for all management reporting, analytics and business intelligence to utilise the same data for enterprise wide consistency.

To successfully deliver on this finance we have to look outside of the world of the ERP and siloed Finance Data Marts. The need to integrate ERP data with the detailed operational data from systems across the organisation has never been so great . This was reflected in the results of the latest IBM CFO Survey (2014) in which 82 percent of the 570 global CFOs interviewed see the value of integrating enterprise-wide information however a big concern is that only 24 percent of the same CFOs feel that they, their teams and organisation are currently up to the task.

Out of the same IBM CFO survey there are 3 significant calls to action for the CFO office:

Hit the Speed Dial – Align your data platform with your business priorities, embed analytics in every process and automate recurring analytical processes. The faster you can analyze the information you collect, the faster you can make decisions

Merge to Surge – Integrate financial and operational data to get deeper understanding of complex questions such as how much it really costs to serve individual customers, which ones are the most profitable and what else you can offer them to generate sustainable increases in profit.

Read the Signs – Use advanced analytic techniques to predict future trends and prescribe the best course of action. It is impossible to be sure what tomorrow will bring, but analysing the variables provides a much clearer picture of the range of future possibilities – and your resulting options.

Teradata’s Financial Reference Architecture provides a picture of what, based on the above calls to action, the CFOs end state vision needs to look like and Teradata continue to make this vision easier and faster for the CFOs office to deliver on.

In January this year Teradata announced the release of Teradata Analytics for SAP (TAS) an out-of-the-box solution to simplify the integration of SAP ERP data on to the Teradata Data Warehouse. This rapid deployment solution enables the delivery of results and business value in days and weeks rather than months, provides business users access to business data in a business friendly format which opens new opportunities for all your users to make differentiated business decisions based on rich cross functional analysis, rapidly. For additional information on Teradata Analytics for SAP please click here.

David Hudson is a Senior Solutions Consultant at Teradata ANZ. He has 10 Years Data Warehousing Experience, primarily focussed on Enterprise Data Model solutions. This includes data integration, ETL design and Logical Data Modelling. Connect with David Hudson on Linkedin.

How to Innovate in the Age of Big Data

Posted on: April 2nd, 2014 by Monica Woolmer 1 Comment


This week Teradata CTO  Stephen Brobst,  spoke at Teradata Summit 2014 in Canberra to an audience with shared interests in two key areas, data and government.

Brobst took us on a journey talking about how companies and organisations around the world are innovating today, in the era of big data.   Brobst shared that during US President Barrack Obama’s first term he was a member of the President’s Innovation and Technology Advisory Committee (PITAC), a working group of the President's Council of Advisors on Science and Technology (PCAST).    Brobst noted that the number one finding was that all federal agencies need a big data strategy in order to use data to improve quality of life for citizens.   How best to do this?   First is to focus less on the organisation and more on the citizen or what Brobst called Consumer Intelligence, providing information direct to the consumer to help them make better decisions whether it is regarding healthcare, banking, insurance or even government services.

The second topic Brobst addressed was the Open Data Revolution.  Brobst shared that the US government policy went from encouraging publication of data to mandating data sharing (with exceptions to protect privacy and security).    What has been learned is that the more open the data is the more value that can be created from the data.  One of the earliest examples of open data in the US is GPS information.  There are now more than 3 million jobs that are directly dependent on the availability of open GPS data and the estimated value is $90 billion a year.

A related topic to open data is the concept of crowd sourcing – using others outside of your organisation to identify insights.  A Canadian company published data and invited PHD students to help identify the best locations to drill for oil.   Todd Park the previous CTO of the US Department of Health and Human Services created a “Datapalooza“ festival where de-identified health care data (including procedures, hospitals, costs, outcomes) was provided to participants in order to create software products with the data.  There were 1600 participants and the Department then provided R&D money to subsidise good ideas generated.  One such idea was the iTriage app to help people locate the most effective hospital services for their particular situation.  This app has been downloaded over 9 million times.

Another example of crowd sourcing is in using GPS-enabled phones to improve data capture.  In the Punjab area (border between India and Pakistan), Dengue fever is quite prevalent and leads to many deaths every year.  Dengue fever is spread by mosquitos and the key problem was identifying breeding locations in and near cities.  A smart phone application was built that health care professionals could use to enter the details on new cases and government employees could use to report potential breeding locations so that preventative measures could be taken.   Pictures of the locations were taken, geo-coded and uploaded for further analysis.  

The picture below shows Brobst pointing out a visualisation of the resulting analysis, the red dots are outbreaks and the blue circles show identified breeding areas.  The result was that incidents of Dengue fever went down significantly and that last year there were only 20 deaths reported.

Brobst's last topic was to discuss innovation and governance and I like many thought that never the two shall meet!    As Brobst pointed out, it is the perception that needs to change.    Brobst referenced a study conducted by Capgemini and MIT Sloan School of Management and proceeded to walk through examples within each of four quadrants shown in the picture below.   Brobst did say that in the report the lower left hand quadrant was called Beginners and not Losers.   He chose that descriptor to make a point, stay in that quadrant too long and your company won’t be around into the future

The Cash Cows are those existing on older products, fixed telephone lines, for example.   There is a very limited future for this market, so organisations must innovate to evolve.   The Fashonistas have high innovation and no governance and often chase the “shiny new object”.   IT can easily fall into this category; think “cloud”, “big data”, “Hadoop”.   It is in focusing on the technology, not on the business value delivered by the technology.   The quadrant to be in is the Digeratti, characterised by high innovation and governance, organisation that are sophisticated in their use of digital assets.

Brobst went on to provide an example of the differences between Fashonistas and Digeratti – social media analytics.   A company was looking for ways to improve customer service within their call centre and were investigating the potential to review feedback received via social media.  Using a collaborative process between business and IT a decision was made not to chase the “shiny new object” (social media) rather to analyse the content of call centre logs, where agents captured a record of their interaction with the customer via a text field. 

Turns out the tools and techniques would have been the same; however the result would have been different.  By focusing on the “boring” call centre data that was never before used for analytics, the company was able to identify opportunities for increasing the quality of service.

So what was Brobst's advice in terms of strategies for those working in government agencies to identify the motivation to invest in innovation?  

Look for someone who cares about improving the quality of service and provide them a way to benchmark.  This can be done by exploring best practices across other agencies in Australia or other countries.   In some cases, commercial organisations can serve as a benchmark.   If you are in an analytics type of role for your organisation or you lead others in the space, challenge what you think of as the best way; use data and experimentation to test and measure effectiveness.  Make no investment unless you know how you will measure it.  If you are not measuring success then you will not succeed - unless you are very lucky.

Monica Woolmer has over 25 years of IT experience and has been leading data management and data analysis implementations for over 15 years. As an Industry Consultant Monica’s role is to utilise her diverse experience across multiple industries to understand client's business, articulate industry vision and trends and to identify opportunities to leverage analytics platforms to support, enable and facilitate the client's strategic business improvement and change agendas. Monica’s focus is assisting Public Sector clients across Australia and New Zealand. Connect with Monica via LinkedIn.

Animals 4.0

Posted on: March 31st, 2014 by Hermann Wimmer 1 Comment


Self-quantification – this term describes a phenomenon that is increasingly affecting our daily lives. More and more people make use of step counters, fitness apps or sleep diaries to learn more about their body and health condition, and ultimately improve their lifestyle. Day by day, developers surprise us with new tools for self-quantification that you can read about, amongst others, on the Quantified Self blog. Even Teradata CTO Stephen Brobst is going to dedicate his keynote speech next week at the Teradata Universe Conference in Prague to “the quantified self”.

What amazes me about this trend is that it’s by no means limited to us human beings! Modern technologies easily allow us to interconnect with and track other things and species, too. Just think of self-driving cars, smart homes or – animals. Yes, you got that last one right! At the US east coast, scientists have started to equip white sharks with tags that track their routes through the sea and even record their individual moves and behaviour.

With the help of three different devices that are implanted or attached to the shark, Chris Fischer and his crew at non-profit organization Ocearch are able to get unprecedented insights into the lives of these mysterious animals. An acoustic tag sends radio frequency signals to underwater buoys, while a “Smart Position or Temperature Transmitting” (SPOT) device connects with a satellite whenever the shark’s dorsal fin breaks the surface of the sea. These two devices allow a pretty concrete location of the shark’s route through the water. But what is most interesting, and most similar to our human self-quantification, is an accelerometer package. It tracks fine-scale data of the shark’s movement – similar to a Wii Motion Plus remote controller – along with external factors such as water depth and temperature. This device detaches itself after just a couple of days and floats back to the surface, together with tons of information stored in its memory that provide the scientists with valuable data.

"On average, we're collecting 100 data points every second – 8.5 million data points per day. It's just phenomenal," Nick Whitney, a marine biologist with the Mote Marine Laboratories in Sarasota, Florida, said in an interview with "Second by second, we can pick up every tail beat and change in posture." With the help of this information, Whitney, Fischer and their colleagues want to get to know white sharks better; not solely for scientific reasons, but also to learn how to protect and preserve them. Best of all, the data is not only accessible to the scientists. Anyone interested can simply choose a shark and track its route through the deep blue sea on

And sharks are not the only animals 4.0! Over at techcrunch blog, I recently came across this hilarious tool: iCPooch, “an internet enabled device that lets you video chat and deliver your dog a treat from anywhere in the world”. iCPooch basically is a simple plastic tower that can be filled with dog goodies. The trick is an appliance that allows you to attach an old smartphone onto the tower. So people can call their dog from wherever they are and have a “conversation” via Skype. The corresponding iCPooch app also includes a “drop cookie” button which you can push during the call and thus treat your dog with a little snack whenever you feel like it. iCPooch just reached its goal of 20,000 $ at Kickstarter two weeks ago and will now go from prototype to production.

These two cases clearly show that modern technologies and data analytics not only help us humans to improve our lifestyle and health condition, but also to support our friends with fur, feathers and fins. We can simply use them to calm our pets down when home alone or we can produce scientific studies to do greater good, for instance gather enough relevant data to maybe save an entire species. And who knows, perhaps they will even help to save our very own species, too?!

The 6 Skills Required to be a Good Data Scientist

Posted on: March 28th, 2014 by David Stewardson No Comments


Presenting at Teradata Summit 2014 across Australia and New Zealand this week, Teradata CTO Stephen Brobst discussed the skills that are required to be a good data scientist :

- Curiosity: Dives into the data head first.

- Intuition: Has good business sense and will explore in directions that yield results but is not afraid to fail.

- Data Gathering:  Knows how to find data and knows how to design experiments to obtain data when it is not available.

- Statistics:  Understands causality versus correlation, expected value theory, statistical significance etc.  This is different from the base math, but covers the understanding of creating a viable sample size, understanding the basis of valid and sound experiments etc.

- Analytic Modelling: Uses historical data to predict the future without over-fitting the data.

- Communication: Ability to explain the results of data exploration without using math terms.

"So, where do you find people like this?  Applied physicists or applied science people are typically good candidates for these roles.  Social science people involved in field surveys also understand data and statistics, so hire them if the stats and maths geeks are not available." said Brobst.

What can you do to make the Data Scientist successful ?

- Give them self-provisioning to data. Don't put castle guards (DBAs) in place between the data and the data scientist.  Don't put a ROI test in front of the test, as the test will determine whether there is a point in having an ROI discussion.  Provide data in un-modelled form for raw analysis. If you try to put these blocks in place, a black market in data will develop.

- Ensure data visualisation is available, as data scientists look for patterns in data.

- Data Scientists should be able to have a dedicated space ("data lab") where they can load new data that is not yet integrated into the data warehouse.  This avoids people downloading data from the data warehouse to their local PC or server for their analysis.  This download approach has security and performance implications that are far worse than allowing controlled processing and data loading on the data warehouse.

David Stewardson is a Senior Consultant in the Teradata Solutions Group. He has a very strong technical background and business acumen with over 23 years’ experience in the Data Warehouse business, specialising in Business Intelligence. During his extensive career, he worked in 6 countries, across 8 different industries (including Mining, Finance and Insurance, Utilities and Telecoms) and has been responsible for managing teams of varying sizes from five up to 150 in previous Business Analysis, Project Manager, Program Manager and Program Director roles. Connect with David Stewardson on Linkedin.

In God We Trust Whilst All Others Bring Data

Posted on: March 28th, 2014 by David Stewardson No Comments


Moving to a culture for data-driven decision making has been proven to be a key differentiator between high performing organisations and the average performing organisation.  The movement from using art to make decisions to using science as the basis for decisions is a key aspect in this cultural change.

Presenting at Teradata Summit 2014 in Australia and New Zealand this week, Teradata CTO Stephen Brobst illustrated the difference between these two approaches with a range of examples.  One example was an airline with a frequent flyer who was on a delayed flight that had to stay overnight in the airport.  The decision facing the airline was how to react to this situation, with options ranging from doing nothing through to financial recompense.  Using test and control groups, the airline was able to determine the best outcome to achieve the best lifetime value outcome from this poor situation.

Another example was the problem facing a market research company to select the right incentive to encourage executives to participate in a market survey.  "Engineers will do anything for a free t-shirt" said Brobst.  Healthcare workers and females will choose the $25 donation to a charity of your choice, but letting people choose one of a range of different incentives is actually the worst choice of all.  Where people don't care enough about the choice, they will avoid the issue and not respond at all.  Too many choices is a bad thing as it causes cognitive dissonance.

Last example was how to price a new product that offers substantial benefits over competitor products.  The recommended approach is to build a price elasticity curve based on test marketing to small regional centers to gain market feedback.  A related example was a pricing experiment in a winery sales where two alternative prices were placed on the same bottle of wine over two consecutive weekends. Brobst talked about the Veblen effect where in some goods a higher price provides a higher quality signal to the market and generates more demand for the products.

Having presented these examples, Brobst proposed that the right answer to each of these questions is to design scientific tests, model the results and produce the answer, but that most organisations expect the more senior employees to know the answer and the managers try to be the "oracle on the hill" and give answers from their previous experience but this can often be the wrong answer.    It's also a challenge because senior managers typically have risen to their positions by being a good intuitive decision maker.  Some of these people find it very confronting to be using math and science to make decisions.   Brobst referred to this approach as the HiPPO principle by using the Highest Paid Person's Opinion.

Moving to competing on analytics and using data or analytics to make the decisions has been proven to be 3-6% advantage in profit for those organisations that have achieved it.  When the data is not available, run an experiment to produce the required data.  Structuring your organisation and IT infrastructure to support testing and experiments will lower the bar to experiments and encourage more decisions to be based on data.

However, all science and no art can risk only incremental improvement rather than the dramatic improvements that a whole new approach can some bring.  Creative new ideas are needed but should be tested against current process to show the benefits.

 It's not Art VERSUS Science, but more a matter of Art AND Science.

David Stewardson is a Senior Consultant in the Teradata Solutions Group. He has a very strong technical background and business acumen with over 23 years’ experience in the Data Warehouse business, specialising in Business Intelligence. During his extensive career, he worked in 6 countries, across 8 different industries (including Mining, Finance and Insurance, Utilities and Telecoms) and has been responsible for managing teams of varying sizes from five up to 150 in previous Business Analysis, Project Manager, Program Manager and Program Director roles. Connect with David Stewardson on Linkedin.