Lateral Thinking in Data Science

Monday May 25th, 2015

Can market basket analysis be used to improve the performance of a website? How about path analysis on text extracted from CRM notes to identify cross sell opportunities? Many data science techniques are developed to address a specific problem within a certain industry. But often there is great benefit in re-purposing these analyses to solve problems in other contexts.

Data Science is a broadly applicable approach to solving many different types of business problems. However, one of the challenges is to knowing which technique is best suited to solving a given problem. When an analyst only draws on standard techniques from his or her industry, solutions may not be optimised.

To address this issue it is useful to firstly be aware of techniques and approaches from many different fields, and then to think laterally about the problem under consideration. This allows the analyst to combine and apply a plurality of techniques in novel and interesting ways.

This has a number of benefits. It expands the range of tools and techniques which are brought to bear on specific problems and also allows the business to benefit from techniques which have been developed in different but related contexts.

I propose a three step solution adaption cycle (Figure 1) to implement this approach.

Step1 – Current Situation: Understand the problem you are looking to address and the context in which it is being framed. Understand the data you have available or ideally the data which needs to be collected in order to address the problem.

Step 2 – The meta-problem: Think about the class of problem being solved and the paradigm into which it naturally falls. Is there a related technique from a another field which would apply to this situation? Would a combination of techniques be applicable?

Step 3 – Adapt and Apply: Select a relevant technique or techniques (if they exist) from your problem-solving toolkit and adapt them to the new situation. Ensure all assumptions are met before applying. Also ensure you understand how to interpret the results in the new paradigm in which your original problem exists.

One example of this approach is as follows. Market basket analysis was developed by retailers to understand items commonly purchased together. Items are grouped together by basket and the associations between commonly purchased items are analysed and used for store planning and recommendations.

Thinking about this paradigm abstractly, to use this technique we need items and groupings. So, for example when analysing website usage, we could use each session as a basket and each web page visited in that session as an item. Or, if we were looking to make recommendations, we could use forms downloaded as items and then make “forms you may be looking for” type recommendations in order to reduce the customer effort score.

This simple paradigm of items and baskets is applicable to many different situations including customer complaints, part replacement and even share trading.

Thinking laterally about your problem and looking beyond standard analyses can vastly enrich your data science solution toolkit.

Ross Farrelly is the Chief Data Scientist for Teradata ANZ, Ross is responsible for data mining, analytics and advanced modeling projects using the Teradata Aster platform. Previously Ross ran Datamilk, an independent bespoke data mining consultancy specialising in data mining and advanced predictive analytics. Ross is a six sigma black belt and has had many years of experience in a variety of statistical roles including Business Development Management at Minitab and as a SAS Analyst at New Frontier Publishing. Connect with Ross Farrelly on Linkedin.

The following two tabs change content below.
avatar

Ross Farrelly

Chief Data Scientist at Teradata
Ross Farrelly is the Chief Data Scientist for Teradata ANZ, Ross is responsible for data mining, analytics and advanced modeling projects using the Teradata Aster platform. Previously Ross ran Datamilk, an independent bespoke data mining consultancy specialising in data mining and advanced predictive analytics. Ross is a six sigma black belt and has had many years of experience in a variety of statistical roles including Business Development Management at Minitab and as a SAS Analyst at New Frontier Publishing.
Category: Ross Farrelly Tags: ,
avatar

About Ross Farrelly

Ross Farrelly is the Chief Data Scientist for Teradata ANZ, Ross is responsible for data mining, analytics and advanced modeling projects using the Teradata Aster platform. Previously Ross ran Datamilk, an independent bespoke data mining consultancy specialising in data mining and advanced predictive analytics. Ross is a six sigma black belt and has had many years of experience in a variety of statistical roles including Business Development Management at Minitab and as a SAS Analyst at New Frontier Publishing.

Leave a Reply

Your email address will not be published. Required fields are marked *


*