There is an ever increasing need for businesses to engage in analytic innovation – exploring new, disparate and more data to gain insights to new products, services or other opportunities for an organization and its customers. Analytic innovation is really about seeing where the data takes you – determining in a scientific manner what actions can be predicted based on past performance. Much of the demand for analytic innovation is being driven by the era of big data – the availability of new data sources provides new and previously unimagined insights.
So, what does it really mean to be innovative? What constitutes analytic R&D and how does this discovery capability relate to the other components of the data strategy? Innovation is synonymous with risk taking as an idea or hypothesis must, by definition, be replicable at an economical cost and must satisfy a specific need – stated or unstated. To create an environment where innovation can flourish, an organization must create a culture that encourages exploration and risks, and accepts and actually welcomes failures.
“We made too many wrong mistakes.”
As I hike the hills of Southern California, I’m accustomed to following the trail maps, but sometimes on a leisurely hike it’s worthwhile to venture off the beaten path. Frequently, that detour leads to a dead end – perhaps a good place to have a snack, but ultimately I’ve got to get back on the trail. Sometime it leads to the discovery of a beautiful canyon or cave I didn’t know was there, or it could yield a short cut that makes my journey to the top a little faster or more interesting. This exploration is not an instance of deciding to take a hike without a trail map or to venture off the trail or out of the park (remember, there are lots of snakes in those hills)! Rather, it’s a desire to explore the unknown to see what I can discover beyond the obvious or familiar. If I discover a really interesting trail, I might add it to my hiking options and revisit it on a regular basis. If I don’t discover anything new or discover that a deviation leads to something undesirable, I simply won’t do that again and my discovery is at least that valuable to me.
Effective Analytic R&D
This is a very exciting time for analytics – the landscape is changing and evolving every day. Dynamic changes and big challenges exist for all organizations. We should strive not just for innovation, but business agility – the ability to make data-driven decisions faster and with more confidence. Ultimately, the goal is to increase revenue and decrease costs, while meeting the customer’s ever-increasing demands for personalized products and services.
In business, an analytic R&D environment is a key component of innovation. An analytic R&D environment supports rapid experimentation and evaluation of data with less formality of data management rules than applied to the production analytics. While this is a discovery zone, and by its nature meant to be less restricted by rules, the key to success is to apply the right amount of governance and structure. No matter what spontaneous choices I make on the hiking trail, I don’t disregard the basic rules of safety, environmental responsibility or common sense.
Effective R&D data management and governance practices allow for exploration but strive to create order from the chaos that can ensue and drive a culture of innovation. These practices consider the iterative and explorative nature of research and development, understanding that new discoveries are sometimes born from previously “failed” endeavors. In fact, the failures are required to develop the new insights and hypotheses – how many attempts did Thomas Edison make before he developed the light bulb?
An effective data strategy has a path for both production and R&D analytics. And, when the R&D effort yields gold, there must be a path back to the production environments and a way to incorporate the innovation into the pipeline of projects that make up the production portfolio. For instance, identify the business processes that need to be modified and the individuals that should be trained to make appropriate use of the data driven insight.
How do you strike a balance between too much control and too little? The goal should always be to preserve the value of the data and ensure that the customers (internal and external) have confidence in the data irrespective of the source or data type. Some of the questions that must be answered include:
- Infrastructure and Platform – can the data be accessed irrespective of where it is stored? Can data from different platforms be analyzed without arduous and time-consuming data integration efforts?
- Data Architecture – is the data (including unstructured and semi-structured data) understood within the context of the broader enterprise and the supported business initiatives? Has the data been modeled? Is it loosely coupled or tightly coupled data?
- Data Quality – is the data fit for purpose? Can you measure and report on that data quality? For certain data types, is there a lower acceptable quality standard; e.g., social media records.
- Master Data – does the data need to be mastered; i.e., a single “golden record” created? Does the data need to be combined with mastered data; e.g., is it necessary to integrate social media data with a customer record to analyze customer satisfaction?
- Metadata – is it possible to report on data lineage, describing where the data originated and how it was transformed in its journey to analytics? How much definition needs to be applied to the data to facilitate effective self-service?
- Data Integration – does the data used for R&D need to be integrated? Is the requirement for batch or real-time, or something in between? Can it be integrated at the time of the analytics; e.g., dynamically modeled? Is self-provisioning an option?
- Data Security and Privacy – how much data security is required? Do the same privacy rules apply as those in the production analytics environments? How damaging is a data breach to the organization?
- Program and Project Management – are there ways to fund and measure the R&D projects that are consistent with the goals for the program and the business initiatives supported? Are there appropriate gating processes in place; i.e., when do you know that the hypotheses is not providing the business value anticipated? How can you build on the previous “failures” when appropriate – does that include sharing the hypotheses, the data, or the techniques applied?
There are new data sources, new technologies, and new skills being developed to exploit these opportunities. But, as with most changes that we have seen over the last 30 years, the answer to addressing the opportunity leads back to traditional concepts and topics. We don’t simply throw out everything we have learned over the years and start again with each new technological advance. That would be a little like discovering a new potential path to the top of the hill and deciding that going forward we didn’t need the same things we used previously to climb the hill – throw out the shoes, the trail map, the water! Throw out the preparation and planning and production quality processes – just start moving! That’s not innovation or business agility, and it’s certainly not progress.
Innovation can flourish when we understand our data strategy – our vision for the organization required to meet the business initiatives – and apply the appropriate management and governance controls, building on what we know works, and leveraging new techniques and technologies.
Vinnie Dessecker is a senior consultant for big data within Teradata’s Strategy and Governance Center of Excellence. Her work aligns business and technical goals for data and information initiatives, including master data management, metadata management, data quality, content/document management, and the analytic roadmap and business intelligence ecosystems.