Monthly Archives: November 2017

Clint Eastwood And Data Scientists. Can Cowboy Processes Drown Your Business?

November 30, 2017


Cowboys have a reputation of getting stuff done, often regardless of the consequences. One of my favorite tough-guy western actors is Clint Eastwood. You never got in his way, or crossed him.

Cowboys are stereotypically resourceful, determined, and ultimately productive, but may lack a longer term vision resulting in undesirable consequences. If not recognized, embraced and managed, you end up with the Wild West. Sometimes, today’s data scientists are not much different than yesterday’s cowboys. They also can be resourceful, determined, and ultimately productive, but also may lack a longer term vision. If their talents and personalities are not recognized, embraced and managed, your analytic organization could also turn into the wild-wild west, drowning in cowboy processes.

Let me explain. Many years ago, when website personalization was being used by a handful of ecommerce companies, my analytic team developed a substantial improvement to the way products were displayed on a website. Instead of immediately implementing this function in a production environment (which would have required massive IT investment), we decided to prototype it, test it and estimate the actual upside first. The results were astounding, many millions of dollars straight to the top and bottom-line. In a meeting with senior executives, there was palpable excitement, until…

… IT said, “We estimate this will take about 6 months to operationalize. Which of these critical initiatives do you want taken off the roadmap to accommodate that timeline?”

The company president responded “We can’t delay any of these.” Looking straight at me he asked, “Scott, can you continue to execute this until we find space on the roadmap?”

With millions of dollars at stake, what else could I say other than: “Of course!”

Fast-forward a few years, and my team was still manually running the analytic process daily. Everything was working fine. We had implemented checks, versioning and documentation of the process to ensure it would continue to be executed daily. We were doing what IT is great at, but definitely weren’t doing it as well. And there was no plan to properly operationalize this process.

Business teams are often frustrated that the prototypes built by their analytic teams cannot be operationalized quickly. To a non-IT resource the long development time does not make sense. After all, the data exists, they have working code… all they need is someone to implement this logic in a production environment with proper monitoring and governance. How hard can that be? And, the frustration is magnified when adjustments (even minor) have to be made to the logic, IT still requires many weeks or even months to make those changes.

So, what causes these types of bottlenecks? Well, there are a few specific causes, including:

  • Antiquated Development Processes:  IT implementing every analytic process with traditional IT standards, which often is a detailed, methodical, and time-consuming approach. Analytic processes require much more agility.
  • Misguided SLAs:  Business requiring strict SLA’s around downtime or performance, which requires the process be built with ironclad reliability, eliminating the flexibility and agility truly required by the business.
  • Isolated Analytic Teams:  The analytic technologies are so specialized, that there are no IT resources available that can understand and implement the analytical processes effectively.
  • Poor Requirements:  To an analytic professional, writing requirements can be a tedious task.   And, often these are done without a long-term view on how the process will evolve.
  • No Productionalized Analytic Technology:  Analytic technology does not exist in the production environment, requiring analytic process prototype code to be completely rewritten.

This lack of agility forces the business to look for alternatives. Analytic teams have many of these skills, and can work in a very agile way. So why not have the analytic team create and execute this business process?

I call these “cowboy processes,” which are regular, business critical, well-defined analytical tasks or jobs (often based on a Proof of Concept (POC) prototype) running on an analytical (i.e. “non-production”) system that ether is…

  • … waiting for IT resources and prioritization
  • …not on the IT queue as the business expects it would take too long

Analytic teams, specifically data scientists, are very efficient at creating these cowboy processes. While not ideal, they are allow the business additional agility, but if not intentionally monitored they could become a large organizational risk.

What are the benefits of Cowboy Processes?

Analytic teams are uniquely trained to plow through data quickly and efficiently. Their tools are designed to extract data from many sources, shuffle it together and spit out an answer. And since they have first-hand knowledge of the complex math used in the algorithms and formulas, it is relatively simple for them to execute these in a no production way.

This results in two substantial benefits of cowboy processes:

  1. Immediate implementation –  Once the prototype has been completed, impact has been tested and the executive team desires a rollout, the analytic team simply needs to execute the prototype at scale. Unless the volumes are beyond the capabilities of the analytic tool, this is usually simple to do.
  2. Agility: abilty to adjust logic quickly- since this code is not “production,” it is usually very simple for the analytic team to adjust and “test” and implement the process.
  3. Implementation of smaller analytic initiatives – since the investment required to develop cowboy processes is so much shorter than traditional development, it allows for analytics to be applied to much more targeted and specialized applications.

Are Cowboy Processes a Problem?

When I talk with business executives, they ask the obvious question: “Are cowboy processes a problem?”

They can be if you can’t adequately answer the following questions:

  • Do you know how many/what business critical processes are dependent on the analytic team for execution?  If not, you need to take an inventory. You could have massive exposure here.
  • What percentage of your analytic team’s time is being spent on these processes? Are there other things they could be doing that would provide more value to the organization? What would happen if you lost 10 percent, 20 percent or 50 percent of your team – would the cowboy processes dominate their time, keeping them from innovating?
  • What would happen if these processes were not executed? Would anyone notice? Who? Would there be a business impact? The processes that create the most exposure should be prioritized high to operationalize.
  • Do you know if these processes can only be executed by a single individual?  If so, there needs to be a deliberate attempt to cross-train and document the process to reduce the risk of that person leaving.
  • Is the organization aware (business, IT, executives, where the jobs are executed?

Maybe the organization is willing to accept the above risks with cowboy processes, but there needs to be awareness of those risks.

Ideally Cowboy Processes shouldn’t exist.  But this is not practical without an infrastructure that allows analytic solutions to be very rapidly implemented (hours or days, not weeks or months) in production with IT providing oversight and the control of the logic and implementation being driven and owned by the business and analytic teams.

How Can I Prevent Cowboy Processes?

Once a cowboy process is running, it is very difficult to justify having IT operationalize it – after all, there is no measurable financial “upside” in passing it over to IT once these processes are running, only a difficult-to-quantify reduction in risk.

You need to recognize that not every cowboy process should be operationalized. If it is not critical to the business (e.g. not much revenue exposure if it periodically fails) or it is simple to execute (and would require a substantial IT investment to operationalize), then it might not justify the resources to needed.

At the core of it, operationalizing conversations with IT and senior executives need to happen before the prototype is developed. There needs to be an agreement on the lift that will result in which potential actions. Three factors should determine the action:

  1. Financial Upside Thresholds – Based on “what if” results from the prototype testing
  2. Cost to Operationalize – Back of the envelope IT development estimates based on analytic team input of high-level requirements. (Note these requirements must include the ability to adjust the rules in a matter of hours or days, not weeks or months.) At the same time, IT has to communicate what would and would not be costly to implement, resulting in a deprioritization of very difficult/costly process.
  3. Effort and risk to manually execute – How much of analytic team resources will be required to maintain and execute this process, and how critical is it to ensure this is executed reliably?

The efficiency of your analytic team is critical to your business success.  Their cowboy skills should be embraced and managed; otherwise you could find your organization living out of control in the Wild West.

Scott LangfeldtScott Langfeldt is a Senior Analytic Consultant for Teradata. In this role, he assists organizations become analytic innovators. This includes identifying obstacles, such as process and organizational challenges, and help organizations overcome them. He also helps organizations align and prioritize analytical opportunities. Previously he lead analytical teams at focused nearly every area of the business including website merchandizing and personalization, pricing, competitive analysis, email strategy and testing, and fraud detection. Prior to that he worked at Blockbuster Entertainment, developing analytically-driven, automated product allocation and supply chain solutions. His 20 years of data science experience designing and implementing analytical business process solutions has given him an first-hand appreciation of the how analytics can impact a P&L, and how to effectively operationalize analytic solutions into production processes.

The culture of value measurement

November 29, 2017

The culture of value measurement - Sundara Raman

“You cannot manage what you don’t measure.” You may have heard that before. Certainly this truism holds today. However, John Hauser, professor of marketing at MIT Sloan School of Management, takes it further: metrics influence the behaviour of the entire organisation, and affect both actions and decisions related to future strategies. According to Hauser, “you are what you measure”! I’m inclined to agree.

Let’s say a firm measures a and b, but not y or z. Pretty soon, managers begin to pay more attention to a and b. Managers who perform well against a and b are promoted or given more responsibilities, increased pay and bonuses. Recognising these rewards, managers start asking their employees to make decisions and take actions that improve these metrics. Often, management won’t even need to ask – employees will start to focus on delivery against a and b of their own volition. These metrics become embedded in the culture of the organisation, and the entire organisation focuses on ways to improve a and b. The firm even gains core strengths in producing a and b. In essence, the firm becomes what it measures.

Value of data and analytics

This is also true of metrics associated with investments in data and analytics. Some organisations consider such investments an overhead. As an overhead, data and analytics investment is viewed as a necessary cost, but one that does not add value. An alternative approach is to view such investments as part of the process of creating value for the organisation. Differences in measurement approach determine the culture of the organisation: either cost driven or growth driven.

What is the right approach to measurement?

In Australia there’s currently a cost versus benefits public debate raging, with a focus red-light speed cameras. A quick calculation shows the differing views of cost versus value, with each having implications on investment decisions.

Cost-based approach:

  • Total annual costs are $5.6million for 227 fixed camera sites and 5 mobile vehicles
  • To reach break-even point the police must catch 93,000 offenders per annum, each with a $60 average infringement fee

Value-based approach:

  • The associated speed warning and safety awareness campaign has reduced deaths from 120 per annum to 81 per annum
  • ‘Financial benefit is $64.5million from avoidance of cost per death of $1.65m (ambulance, pathologists, hospital, police, etc.)
  • ‘Real benefit is 39 lives saved

The best of both worlds!

Of course, it is prudent to focus on efficiencies of cost management while at the same time taking measures to improve revenue growth. However, many organisations face difficulties on both accounts as a result of not leveraging their data assets appropriately.

Sometimes the overall cost of data is not visible, and therefore escapes scrutiny because data is duplicated and spread across several data marts in the organisation. The consequence of not having all information in one place is that the business executives will have great difficulty in seeing the entire situation of the enterprise, resulting in suboptimal decision making. As a result of disconnected information, businesses experience some common pain points:

  • Inability to identify customers affected by mobile network coverage and service quality due to lack of correlated telecommunication network and customer data
  • Inaccurate forecasts resulting from disparate data
  • Imprecise view of profitability caused by inability to charge for overuse and lack of detailed usage data
  • Inability to identify unprofitable products or most valuable customers due to insufficient information
  • Inability to meet customer commitments due to inaccurate service metrics

What can the CFO do to encourage the ideal value-measurement culture?

Focus on cost economics where appropriate, be it establishing a data lake or integrating the data in an ecosystem. Establish a framework for governance that takes into consideration regular measurement of return on capital. The framework should include the regular audit of data and analytics assets that can provide a strong incentive for managers and executives to use them efficiently and effectively.

SundaraRamanSundara Raman, Senior Telecom Industry Consultant, Teradata Australia

Sundara has been a Telecom professional for the over 30 years with a wide range of interests and multi-national experience in product management, solution marketing, presales for new generation networks and services, information management strategy, business intelligence, analytics and enterprise architecture development.

At Teradata, Sundara focuses on Business Value Consulting, Business Intelligence, Discovery Analytics and Customer Experience Management solutions.

Sundara has a Master’s Degree in Business and Administration with research on economic value of information from Massey University, New Zealand.

For the last 20 years, Sundara has been living in Sydney, Australia. In his spare time, Sundara enjoys walking and maintaining an active life style. Sundara is an inventor and joint holder of an Australian patent with his clinical psychologist wife. The invention is an expert system in cognitive mental health that applies machine learning algorithms.

Q&A with Sri Raghavan: Applying Advanced Analytics to Health Care

November 28, 2017


Sri Raghavan, senior global product marketing manager at Teradata, answered a few questions on how analytics are impacting the field of health care, and approaches to dealing with privacy and false positives in that industry.

It seems like there is a lot of excitement around health care and AI. Where do you see that field evolving, and is there any other area that you think holds a lot of immediate promise?

In general I think health care is one of our biggest frontiers. Part of the reason health care is so big is because the outcomes really make a difference in our lives. If I went to an entertainment company where I could use artificial intelligence, the impact is something which affects the quality of life in that you’ll get better entertainment, which is a terrific outcome. But it’s not quite as appealing and effective as looking in someone’s eyes and telling them that heart disease is something you they shouldn’t be concerned about. From that standpoint, while every industry is important, there are certain industries that have much more immediacy to our public good and from that standpoint I fully believe it’s health care.

With health care, how do you ensure the data is everywhere it needs to be to inform everyone in the chain, from the patient to the provider to an insurance company?

To me the question is not so much to make the data available everywhere. It’s not the data that needs to be available — it’s the insights. The data availability part is important in the sense that the guys that are doing the analytics, they can can pick that data from any source or location. But insights, the ability to be able to provide insights in many different form factors and visualizations and tables and what have you in a manner that every idiosyncratic audience group understands — that makes a big difference, and that’s a key thing we are ensuring with the availability of ways insights can be communicated. That’s something we try to do a lot of.

What about privacy? That seems like a much bigger factor in this industry.

You know, it seems like one of those cliched issues like, “Oh, we need to do security and privacy.” But the question of health care really makes a big difference.

I’ll give you an example. I was talking to a customer a couple days ago, and they said, “Look, I have a lot of information about diseases and afflictions and prescriptions and treatment plans and demographics and other contextual data, which really allows me to be able to, on the one hand, provide enough information to medical care providers to tell them what kinds of treatments work with which kinds of people.” Great, but they said, “I also happen to know about your lifestyle. I also know that you’re are eating six gallons of ice cream.”

Yes, it’s bad to be eating six gallons of ice cream, but that is not your business. However, health care companies, because there’s this risk of using your lifestyle information with your affliction information, you could potentially easily envision a scenario where I can go to you and say, “Hey I know you are doing some things like smoking three times a day. Maybe you shouldn’t do that.” The problem with that is it impinges upon your privacy, so that’s why we’re making governance processes to make sure those kinds of instances not happen, that certain kinds of data aren’t collected by law and you are limited by using that information.

What is the baseline standard of what level of what false-positive rate is good enough to roll out these technologies?

Some thresholding has to happen. And the thresholding happens at two two levels. It’s very contextual. Every patient is different and every disease is different. There are certain thresholds which may work in certain cases that don’t work in other cases. It’s very specific to the kinds of treatment areas that are being worked on.

Then, of course, there is this huge legal implication, because now certain decisions are going to be made by data that is generated by machines. Which means that now the onus seemingly is on an inanimate object, which cannot be sued and nothing can be done to it. So it seems like the patients are left out on a limb, because no one is to blame when certain errors are being made. So I think certain legal costs and protections can come into place to ensure there is.

People should not be worried that artificial intelligence is going to take over doctors. No, it won’t, because it’s practically impossible for us to do that. We need someone to go back to as a recourse for certain things, and that someone has to be responsible within reason.

Are you seeing overlap in your models with medical false positives and, say, the models that are working for fraud detection and banking that focus on champion/challengers?

Yes, except in certain industries, the false positives are more palatable than others. Predicting the demand for certain eyeliner products and I make a mistake: No big deal. Someone bought wrong eyeliner, who cares? But when you are looking at heart disease and concluding, “There is no heart disease, and the arteries are really not blocked. Don’t worry!” you have a different set of problems.

But, to your point, it does get better over time, absolutely the models do. But an enormous amount of work needs to go into that data to deliver those types of numbers just to establish a threshold of success. If 90 percent of the time you are able to say this case acted fraudulently, that’s damn good in banking. But you still need precautions in place for the 10 percent.

Click here for more on Teradata’s work in the area of healthcare, from helping find solutions to America’s opioid crisis to advancing breakthrough medical innovations at Sonafi using analytics.

sri-raghavan-hadoop16Sri Raghavan is a Senior Global Product Marketing Manager at Teradata and is in the big data area with responsibility for the AsterAnalytics solution and all ecosystem partner integrations with Aster. Sri has more than 20 years of experience in advanced analytics and has had various senior data science and analytics roles in Investment Banking, Finance, Healthcare and Pharmaceutical, Government, and Application Performance Management (APM) practices. He has two Master’s degrees in Quantitative Economics and International Relations respectively from Temple University, PA and completed his Doctoral coursework in Business from the University of Wisconsin-Madison. Sri is passionate about reading fiction and playing music and often likes to infuse his professional work with references to classic rock lyrics, AC/DC excluded.

The 13 rules for measuring the Turing time of your AI

November 27, 2017

The 13 rules for measuring the Turing time of your AI - Danko Nikolic

Last time we discussed how the Turing test, flawed as it is, still holds some relevance for AI development – especially when it comes to intelligent assistant applications. For more on that, read more here.

However, consistently measuring the test is a methodological problem, as it focuses on psychological measurements. Because a human makes the judgement on whether the test subject is a machine or not, these judgements must be established as accurately as possible, objectively and without bias.

What follows are the proposed rules that every measurement of Turing time should satisfy.

Rule 1: No naïve tests

The judge must know they are taking part in a test. For example, someone running a chat bot might not tell visitors, and only later ask if they noticed anything strange or whether they thought who they were chatting to was a machine. Such a test would be biased towards giving bots incorrectly long Turing times.

Rule 2: Be yourself

This holds for every human taking part in the tests – be it the assessor or the test subject. Every participant should be instructed not to imitate a machine in any way. They simply need to be themselves – to be human. At the end of the test, they should be rewarded for not being mistaken for a machine.

Rule 3: 50-50

The probability that a human judge is talking to a machine in any single test should remain at 50%. That is, there should always be a 50% chance that a test subject is human, and 50% that it is a machine. Also, the judge should know this likelihood.

Rule 4: Average Joe

The judges shall represent the education and culture of the general population. The judges should by no means be experts in AI, or experts in human behaviour. Therefore, judges should be sampled from the general population.

Rule 5: Free choice of topics

There should be no limits to the topics discussed. This means also that no theme of conversation can be assigned – implicitly or explicitly. Similarly, there should be no limits to topic changes. For example, it should not be considered a proper Turing test if one is limited to conversations about medical issues.

Of course, it is legitimate to determine how long it takes for users of a medical AI to notice that they are not talking to a human physician, but that result cannot be called Turing time. The result may be called something like limited domain detection time. What Alan Turing meant by imitation game was whether a machine can replicate human intelligence, and this is what we are measuring under Turing time.

Rule 6: No time or word limit

We don’t ask whether AI can be uncovered within 30 minutes, or after 1000 words have been exchanged. We are also not interested in how often the presence of machine is revealed in a 30-minute test. The imitation game is by its nature asymmetric; a machine imitates a human, not the other way around. This is why the test ends the moment the assessor can make a confident judgement on which of the two test subjects is a machine.

Rule 7: Human faults

The machine is expected to exhibit human faults, and thus can be detected through super-human performance. For example, if the machine has an encyclopaedic knowledge or can quickly performs complex calculation, the judge could legitimately conclude that this is a machine. Of course, there is nothing wrong with endowing AI with super-human abilities outside the imitation game, but here we are determining whether and how long the machine can hold the illusion that it is human. A large part humanity is fallibility.

Rule 8: Multiple tests, minimal time

In order to derive a robust Turing time measurement, multiple tests must be taken using multiple judges and human test subjects, with the same AI. The Turing time is defined as the minimum of all those times. We do not consider the average or median.

The Turing time is not about how long it takes on average for the machine to be detected, but how long the machine is guaranteed to hold the illusion.

The reasoning behind this rule is the same as that of two-year product warranties. If you purchase a car – or any other product – you don’t want it to work with some likelihood, you want it to work, period.

Similarly, if you get a new state-of-the-art AI that imitates human intelligence, you don’t want that imitation to maybe work for a bit – it must be guaranteed to work for a period of time. Good user experience – i.e. good, human-like interaction with AI – is about how well the technology performs for everyone, across all types of applications. This is why Turing time is defined as the minimal time for the test to end. This also means that once Turing time is measured, the result is not fixed – it can be reduced later to a lower value.

Rule 9: Turing word count

An additional way of indicating the amount of interaction needed to detect that one is interacting with machine is in the number of words used in the communication. This number may be published along with Turing time, and can be referred to as the Turing word count.

Rule 10: Valid justification

Each detection of machine-like error must be documented with an explanation as to why it has been decided to consider it an error. One needs to document how the machine responded, and explain why this was not accepted as a human-like response. Also, it would be good to state how a human could have communicated in that situation.

For example, if the assessor asked, “I need to fly to London next week. Can you find a flight for me?”, and the machine responded with, “I cannot find a flight to destination: London Next Week.”, then one would document that the machine had misunderstood the destination and time frame. A human may ask for more clarity: narrowing the timeframe with a follow-up question such as, “Which day next week would work for you?”. Through proper documentation, anyone should be able to see how the Turing time had been established.

Rule 11: Intermittent conversation

The judge should be permitted to break the interaction, returning to continue the test at a later point in time. The Turing time should count from the very beginning of the entire interaction with that AI. As AI advances significantly in the future, such measurements of Turing time will become a necessity.

Rule 12: Under measurement

If a machine has not revealed itself for longer than 50 hours, its Turing time can be reported as “under measurement” until the machine finally makes a mistake. Imagine the year 2117; a new state-of-the-art AI is already five years old. The Turing tests began the moment the AI went live, but to date no single machine-like error has been detected. The machine’s Turing time is reported as “under measurement” and with an indicator of how long has it been e.g., “1540 hours without error”.

Rule 13: Don’t alter your AI

During the period of testing, the AI should not be altered by a third party. The machine is allowed to learn, but only in the same way a human would learn. For example, the AI could obtain information about recent world events, as it could enable interaction with the assessor. However, the underlying technology of AI should not be improved whilst the AI itself is being assessed.

If the AI is altered significantly outside of a period of testing, that AI should be considered as new, and a separate set of tests should begin.


Of course, on top of these thirteen rules, all the other known methodological factors of scientific measurements and experimentation must be considered and applied. This includes proper randomization, sampling, double blind designs, replicability of results, and so on.

What do you make of these rules? What did I miss? Tweet me to discuss your AI.

Danko NikolicDanko Nikolic is a brain and mind scientist, as well as an AI practitioner and visionary. His work as a senior data scientist at Teradata focuses on helping customers with AI and data science problems. In his free time, he continues working on closing the mind-body explanatory gap, and using that knowledge to improve machine learning and artificial intelligence.

It all Started with ‘CARE’ – Reasons to Pay it Forward

November 23, 2017

Male and female hikers climbing up mountain cliff and one of them giving helping hand. People helping and, team work concept. ; Shutterstock ID 547233985; PO: redownload; Job: redownload; Client: redownload; Other: redownload

Tho H. Nguyen works for Teradata Global Alliances. Tho has many reasons to pay it forward because of his own personal story — and it all started with the CARE organization (humanitarian aid company). His family fled Vietnam in 1979 when he was just seven years old. “I was one of the boat people,” says Tho. “My parents sacrificed all their savings to book passage for themselves and their seven children on a journey to freedom.”

Bound for Hong Kong, the fragile boat Tho’s family was on quickly became too heavy and all passengers had to throw their luggage overboard. On day four at sea without food and water, the boat ran out of gas and was stranded. Fortunately, a European ship came by and towed the boat to a refugee camp in Indonesia. The refugees chopped wood, built barracks that were shared by multiple families, and learned how to fish and hunt for food to survive. Tho recalls his experience, “We were the lucky ones – other families who fled were pirated at sea and families were split up, kidnapped or even killed.”

When the CARE organization approached his parents at the refugee camp six months later about where they wanted to live, they said, unequivocally, the United States. As luck would have it, St. Francis Episcopal Church in Greensboro, North Carolina was looking to sponsor a family. Tho’s family came to the United States, where the church provided a home, taught the family English and helped them acclimate to their new environment. “We are very fortunate and we would not be in the United States if it wasn’t for the church.”

As a Teradata employee today, Tho enjoys giving back and paying it forward for others through Teradata Cares, a great way to for our employees to have fun, meet passionate people and feel good volunteering – all at the same time. Tho has supported DataKind organization and Summit Rotary International. “DataKind is a great charity whose mission is to utilize data for good.” Tho participated in a DataDive and appreciated the opportunity to meet the organizers and volunteers who bring so much passion and knowledge for good causes.

DataKind – Tho and Jake at a DataDive

DataKind – Tho and Jake at a DataDive

The Summit Rotary International gave Tho scholarships – the McKnight Scholarship aided in his undergraduate degree and the prestigious Ambassadorial Scholarship gave him a chance to earn his MBA in International Business from England. Tho says, “The Summit Rotary Club gave me a chance to see and experience the world, which I will never forget.”

Summit Rotary – Giving back to fund scholarships

Summit Rotary – Giving back to fund scholarships

Finally, Tho serves as the Vice President for the local Vietnamese-American Association of Raleigh (VAAR). VAAR has given Tho an opportunity to reconnect with his roots, maintain the Vietnamese language and preserve the rich culture and traditional values. Being a part of this community helps Tho appreciate the past, honor the present and anticipate for the future. “I count my blessings every day,” Tho said.

Teradata Cares at Partners 2017 – stuffing backpacks and bears for charity

Teradata Cares at Partners 2017 – stuffing backpacks and bears for charity

Speaking of blessings, his two-year-old daughter Ana gives Tho another reason to give back. “I want to pay it forward,” Tho said. “And help Ana’s future and future generations.” This is one more reason Tho is donating his proceeds from his book Leaders and Innovators: How Data-Driven Organizations are Winning with Analytics to charity.


teradata-caresJust as Teradata is dedicated to helping our business customers drive results through analytics, we also focus our efforts on ways non-profit organizations around the globe can use data to tackle the problems they face everyday. This data philanthropy strategy aligns our corporate strengths, consulting services and technology with the demands of those organizations which all too often lack the money or manpower needed to exploit and analyze the volumes of data they have accumulated.

Teradata Cares is a program designed to build strong and vibrant communities, improve quality of life and make a positive difference where we live and work. By working together, we can build a better world.


Can We Trust Hadoop Benchmarks?

November 22, 2017

Trust broken shutterstock_579284215

Over the last 7 years, there have been dozens of Hadoop and Spark benchmarks. A vast majority of them are a scam.  Sometimes the perpetrators are ignorant of how to do a benchmark correctly (i.e. Hanlon’s Razor.)  But too often, some zealot just wants to declare their product the winner.   They ignore or adjust the facts to fit their objective. The novice data management buyer then makes decisions based on misleading performance results. This leads to slow performing implementations if not outright purchasing disasters.  Business users are then told to abandon slow running queries, they don’t need them. (Yeah, right.)  That company then falls behind their competitors who made better purchasing decisions.  Never forget: query performance is vital for employee productivity and results accuracy. When hundreds of employees make 1000s of informed decisions every day, it adds up.

Today’s misleading big data benchmarks mirror historical misbehavior.  In the 1990s, poorly defined vendor benchmarks claimed superiority over competitors. Customers got bombarded with claims they couldn’t understand nor use for comparisons.   The Transaction Processing Council was formed to define standard OLTP and decision support benchmark specifications.  TPC benchmarks require auditors, full process disclosure, and pricing to the public.  TPC-DS and TPC-H are the current decision support benchmark specifications.  Ironically, TPC-DS and the older TPC-H queries are used in nearly all tainted benchmark boasts.  IBM Research complained loudly about Hadoop TPC-DS benchmarks back in 2014. But the fake news continues, sometimes called bench-marketing.  But most of today’s misleading benchmarks come from academics and vendor laboratories, not marketing people.

Here are some of the ways Hadoop benchmark cheating occurs.

Grandpa’s hardware. Comparing old hardware and software to new is the oldest trick in the book.  Comparing a 2012 server with four cores to 2017 servers with 22 cores its pure cheating.  Furthermore, each CPU generation brings faster DRAM memory and PCI busses. Is it any surprise the new hardware wins all benchmark tests?  But then some startup vendor publishes “NoSQL beats Relational Database X”, leaving out inconvenient details.   There’s a variation cheat where hard disk servers are compared to SSD only servers.  Solution: publish full configuration details.  And always use the same hardware and OS for every comparison

SQL we don’t like. None of the SQL-on-Hadoop offerings support all the ANSII SQL-99 functions.  The easy solution is to discard SQL the product cannot run.   While TPC-DS has roughly 100 SQL queries, no Hadoop vendors report more than 25 query results.   Worse, they modify the SQL in ways that invalidates the purpose of the test.  Solution: Disclose all SQL modifications.  

Cherry pickers. By reporting only the query tests that they win, the vendor claims they beat the competitor 100% of the time.  These are easy to spot since every bar chart shows the competitor losing by 2X to 100X slower.  Solution: publish all TPC test results.  But this will never happen which is why customers must run their own benchmarks

Tweaks and no tweaks. First, install the competitor out-of-the-box product, preferably a back level version.  Run the tests.  Next load the cheater products on the system. Now tweak, tune, and cajole the cheater software to its’ highest performance.  Too bad the competitor couldn’t do any tweaks.  Solution: test out of the box products only

Amnesia cache. Run the benchmark a few times so that all the data ends up in memory cache.  Run it again and “Wow! That was fast.”  Next, reboot and run the competitor software with empty memory caches forcing it to read all the data from disk.  Surprise! The competitor loses. Solution: clear the in-memory cache before every test.

Manual optimizer (an oxymoron). Lacking an optimizer, Hadoop and Spark systems process tables in the order encountered in the SQL statement.  So programmers reorder the SQL clauses for the best performance.  That seems fair to me.  But if a Tableau or Qlik user issues a query, those tweaks are not possible.  Oops –not fair to the users.  Solution: run the TPC test SQL ‘as-is’ without reordering.

Ignorance is easyThis is common with university students who lack experience testing complex systems.  Benchmarking is a career path, not a semester’s experiment.  Controlling all the variables is much harder than most people know.  There are many common mistakes that students don’t grasp and repeatedly apply.  Solution: be skeptical of university benchmarks. Sorry about that Berzerkly.

Scalability that is not. Many benchmarks double the data volume and call it a scalability test.  Wrong.  That’s a scale-up stress test.  Scale-out is when more servers are added to an existing cluster.  Solution: Double the data and the cluster hardware.  If response time is within 5% of the prior tests, it’s a good scalability test.  Note to self: Teradata systems consistently scale-out between 97-100%.

Teradata abandoned TPC testing in 2003.  TPC workloads represent the easiest queries our customer’s run.  We find business users evolve from simple queries to increasingly more complex analytics.  Which is how the phrase “query from hell” was born.  Missing from the simplistic TPC tests are:

  • Continuous ingest of data into tables being queried
  • Testing 100-1000 concurrent users. This is exceptionally difficult to do correctly
  • Severely complex joins with non-colocated fact tables (matching keys on different nodes)
  • Complex joins — 20-50 table joins with 4-10 huge fact tables


TPC-DS testing falls into the gray pie slices shown. The gray slices are data mart workloads while the entire pie chart is a data warehouse workload.  Table scans represent 7-10% of a data warehouse workload but they are 60-80% of Hadoop and Spark workloads.  Simple joins combine many small dimension tables with a couple fact tables.  Tactical lookups in Hadoop must run in HBase to achieve sub-second performance.  Which is like saying “Let’s install two RDBMS systems: one for decision support, and one for tactical queries and keep them in sync.”

Most Teradata customers run 100 to 1000 concurrent users all day long. We call these Active Data Warehouses.  ADWs deliver real time access, continuous data loading, and complex analytics simultaneously.  It’s not unusual for a Teradata customer to run ten million queries a day while loading data every five minutes in eight concurrent batch jobs. One customer has a 50 page SQL statement they run sub-second.  Others have 100-200 page SQL statements generated by modern BI tools.  All this runs with machine learning grinding along in the ‘severe joins’ category.

One way to tell if a benchmark is honest is that the publisher loses some of the test cases.  Teradata has published two such results:

  1. WinterCorp compares Hadoop to the Teradata 1700 system (2015)
  2. Radiant Advisors comparing SQL-on-Hadoop software (2016)

We encouraged these independent analysts to report all test results, even where our products did not win. Why?  We want customers to use the right tool for the workload.  These independent analysts pushed back occasionally.  We learned a lot from each other.

Custom benchmarks tests are the only way to make an informed platform decision. It’s time consuming and expensive for both the vendor and the customer.  But it’s vital information.  Product performance has a huge influence on business user productivity and accuracy.  Yet, benchmark performance is a small part of a platform selection decision.  Other product selection criteria of equal importance includes:

  • Total cost of ownership
  • BI and ETL software interoperability
  • Customer references you can talk to
  • Governance capabilities
  • Security features
  • High availability vetting
  • Public and hybrid cloud support
  • System administration tools
  • Ease of use and flexibility for business users
  • Analyst reports and inquiries

Platform selection is a complex task.  If you need help, Teradata has an unbiased professional services engagement to crawl through all the issues.  The results of those comparisons show some workloads go on Hadoop, some on the data warehouse, some on other platforms.  We practice best fit engineering and you should too.

I hope you now will realize the real lies with your real eyes.

Or trust in Dilbert’s answer  “A misleading benchmark test can accomplish in ten minutes what years of good engineering can never do.”

dan graham

With over 30 years in IT, Dan Graham has been a DBA, Product Manager for the IBM RS/6000 SP2, Strategy Director in IBM’s Global BI Solutions division, and General Manager of Teradata’s high end 6700 servers.  At IBM he managed four benchmark centers including large scale mainframes and parallel PSeries systems.  He is currently Technical Marketing Director for the Internet of Things at Teradata.

Building Trust in AI: How to Get Buy in for the Black Box

November 21, 2017


Maybe it’s something innate to human nature, or maybe we’ve all seen one too many sci-fi movies (I’m looking at you Hal and 2001: A Space Odyssey), but people tend to view new technology skeptically. This is especially true when it comes to technology that makes recommendations or tells us how to do something. A prime example of this is GPS we all depend on GPS to get us to and from our destinations now, but when it first came on the market, many people preferred hard-copy maps to digital ones.

We’re now entering a world where AI will play an increasingly large and important role in our lives. Yet when it comes to the enterprise, many people at all levels of organizations do not trust AI-generated insights. If you’re someone tasked with getting buy-in across a company to motivate adoption of AI-based tools, the fear of the unknown and mistrust of AI can be a huge impediment.

Based on my experience, I see overcoming skepticism of AI as a two-step process that first requires users to understand how AI works so that they can then learn to trust it. Mistrust arises when the trip between the data and the recommendation from AI is more complex than people can immediately understand on their own. Assisting people to comprehend how AI made this trip is how you establish trust with the tool.

Cultivate Understanding …

Helping users understand how AI works is the crucial first step in the process of adoption. The best way to increase understanding is to provide people insight into important variables and trends around the outputs your AI tool is targeting.

We all remember our elementary school teachers telling us to show our work when working on algebra problems. Well, the same is true when trying to develop understanding of AI. Showing is more powerful than telling to aid understanding.

There are four main ways you can do this:

  • Change variables affected by the algorithm: No human is going to be able to understand an AI algorithm completely. But to improve understanding of AI, you can show how the outputs of the algorithm are sensitive to changes with certain variables. For instance, if you remove income of the customer as a factor in detecting fraudulent account activity, the difference in results will help users to see what variables the AI tool is using to make its recommendations.
  • Change the algorithm itself. Within an algorithm, you have a complex network of many nodes. If you remove a layer of nodes and then assess the impact this has on output, you can gain understanding into how it works. You’re essentially performing a sensitivity analysis. For instance, if you change the threshold for a certain variable from 4.5 to 7.5, and the output changes significantly, you know that variable played a big role in the outcome. Or to put it more metaphorically: Think of an algorithm as a machine with a bunch of knobs. If each knob has a range of intensity, you can alter those parts of the algorithm to show users how this changes outcomes. If you turn one knob up to 11 and another to 1, you can tell from the results what variables were most important to the AI tool in making its determinations.
  • Build global surrogate models: Surrogate models are built in parallel to an AI algorithm, but are simpler and easier to understand. These could be a tree, linear model, or linear regression that mimic the more complex AI network. Now, the results will never 100% align, but if the results from the surrogate model strongly echo those from the AI tool, users will understand some of the steps involved in the AI process.
  • Build LIME models: LIME models, or local interpretable model-agnostic explanations, are surrogate models that are localized. Instead of trying to replicate the entire model in a linear way, with LIME models you generate synthetic samples around a particular event and then linearly create models just for that event in a local way. From this, you get an understanding of which features are important in doing a linear classification around the event.  

…and then Establish Trust

After using one, or a combination of these four methods to help users better understand how AI machine learning works, you then have the foundation to establish the trust that’s crucial to getting people to actually use the tool. There are three ways I’ve found that enable this trust to be built most effectively:

  • Detect events and trends that conform to human expectations: The old adage of “I don’t believe it until I see it with my own eyes” applies to AI. If you can run AI on events and trends that are part of users’ domain knowledge and context, and that then produces results that confirms their expectations, you show that the AI model can be trusted. For example, if you have a transaction that the AI model has flagged as fraud and when asked, a fraud detection expert confirms that it’s a plausible fraud event, you help to promote buy-in.
  • Event and non-event cases should use different criteria. When a human is trying to detect fraud, their brain goes through different processes when examining a case that looks like fraud and one that doesn’t. Taking this same intuitive approach with AI, in which the AI tool can show why one event triggers a fraud alert, and another doesn’t, helps to show users that the AI tool operates in a way that is familiar and trustworthy.
  • Detected outcomes remain consistent. All scientific inquiry is predicated on the idea that for results to be statistically significant, they must remain consistent over time and be replicable. The same is true for AI. When a possible fraud event is run through an AI model, it should be flagged consistently each time. Stability is key to establishing trust. Companies can build user interfaces that help to the backend of the AI tool into the daylight to illustrate to users what is occurring.

At the end of the day, if you want to make AI less of a black box, you must first make users understand how the AI tool works so that they can then have trust the results. It’s natural to view any new technology with a skeptical eye, but because the insights generated by AI can so dramatically help to reshape how a company operates, companies need to do everything they can to ensure buy-in.

Chancal ChatterjeeChanchal Chatterjee, Ph.D., has had several leadership roles focusing on machine learning, deep learning and real-time analytics. As Senior Director of AI Solutions at Teradata, he is leading AI solutions for several market verticals. His AI focus is on financial fraud, preventive maintenance, recommender systems, smart manufacturing, and financial credits. Previously, he was the Chief Architect of Dell EMC at the Office of the CTO where he helped design end-to-end deep learning and machine learning solutions for smart buildings and smart manufacturing for leading customers. He has also worked on service assurance solutions for NFV data centers, and spearheaded the deployments with large service providers in Europe and Middle East. He has also been instrumental in Industrial Internet Consortium, where he published a smart manufacturing analytics framework for large enterprise customers.

Put your AI to the test – it’s Turing time!

November 20, 2017

Put your AI to the test – it’s Turing time - Danko Nikolic

Whilst imitating a human is only a small part of what an AI can potentially do, the pursuit of human imitation has implications in a number of areas. For that reason, I’m proposing a contest.

Everyone is invited. Bring your AI and see who can take on the Turing test. The goal: achieving the longest Turing time.

In the future, I can see software giants competing directly with one another, their market shares rising and falling in accordance with the performance of their AI contenders.

The open source community may collaborate; contributing to the overall quality of the technology and reducing the importance of proprietary solutions.

Importantly, the competition will create excitement around AI, and as a result start-ups could explore risky but potentially rewarding ways to drive AI forward, and even find novel applications for the technology.

Even academia could get involved: performing meta research, monitoring the strength of AI development and how much positive impact the technology has on the society.

Each contender in this contest may have their moment of glory, and contribute to the advancement of technology.

Independent body

Any competition will need an independent body to oversee proceedings and ensure a fair result. The task of that body would be to keep track of the Turing time assessment results, and to define (and perhaps refine) the official rules of the game. This organization should help establish the legitimacy of the test, and could maybe even perform measurements of its own.

The organization will need stewards and administrators on a volunteer basis, as well as funding. Suggestions and contributions are welcome. I’ve even reserved a domain name for us to get started:

If done right, the contest will produce reliable measurements and we will accurately monitor the progress of our chatty AIs.

Turing time give us the numbers to beat, and the rules by which to play. Contenders for the competition are anything but lacking.

So, let the imitation games begin.

Danko NikolicDanko Nikolic is a brain and mind scientist, as well as an AI practitioner and visionary. His work as a senior data scientist at Teradata focuses on helping customers with AI and data science problems. In his free time, he continues working on closing the mind-body explanatory gap, and using that knowledge to improve machine learning and artificial intelligence.


ETL is changing: How to transform a TLA*

November 16, 2017

ETL is changing - How to transform a TLA - Andrew Johnston_Preview

*TLA = Three Letter Acronym.

TLAs. The world is full of them. But as processes shift, so too does the language with which we describe them. The process of preparing data for analytical usage is transforming. Gone are the days of writing business requirements, mapping new data feeds from written specifications and handing off to a select bunch of extract, transform, load (ETL) developers.

IT departments are finally cottoning on to the fact that it is far more efficient to place tools in the hands of the subject-matter experts that understand the data domains. The role of IT changes from doers, to that of the enablers: making the right tools available in a secure and robust data ecosystem.

This federated approach reinforces the concept of business data ownership. It brings to the fore the role of data stewardship – prioritising developments that directly support the development of new business use cases. It also aligns better to the agile working methods that many companies are now adopting.

This isn’t to say that we encourage data federation or duplication; we still want to integrate once and reuse many times through the use of data layers across an integrated data fabric.

So, why does the ETL TLA need to change? Let’s examine the rationale.

From Extract, to Ingest

Data no longer comes from a relatively small number of on-premises product processing systems – where data was typically obtained in batches according to a regular schedule.

From transactions and interactions to IoT sensors, there are now many more sources of data. As we move to service-oriented and event-based architectures, data is much more likely to be continuous and streamed through infrastructure like an enterprise service bus (ESB). We need to be able to tap into these streams and rapidly ingest, standardise and integrate the source data.

From Transform, to Wrangle

The explosion in the use of data across diverse information products means that data is repurposed many times for each downstream consuming application. The adoption of the ELT paradigm reinforces this point, where data must be integrated with other data before complex transformations can occur.

Data may need to be derived, aggregated and summarised, to be optimised for consumption by use case or application. Creating complex new transformations requires a greater understanding of the data domain. New data-wrangling tools can also help business users accomplish these types of tasks.

From Load, to Project(ion)

We no longer simply load data to tables in a traditional enterprise data warehouse. Most organisations use an analytical ecosystem using open source technologies to supplement traditional data warehouses. The term ‘data projection’ extends the way data may be consumed – as a logical view spanning multiple platforms in the analytical ecosystem, or delivered via an API to a consuming application. Think of a projection as a set of flexible layers to access the data.

ETL is changing - How to transform a TLA - Andrew Johnston_Inline

Change behaviours, not just vocabulary

The ultimate goal is to accelerate the creation of new information products and improve efficiencies. These new tools are intended to identify and automate commonly used data transformation workflows, based on how business analysts and others typically use the data. When done right, it will result in less work on the IT side, and create more self-service capabilities for business end users.

Historically, an organisation selected a single vendor for data integration tasks. Where software from multiple vendors were necessary, they often didn’t play very well together. Building a data integration workbench, running on open source technologies, allows mix-and-match use of best-of-breed tools for different tasks such as ingestion, wrangling, cleansing or managing the data catalogue.

Democratising the creation of information products can act as a real catalyst to drive innovation and deliver superior business outcomes; faster and at a lower cost than the legacy approach.

At Teradata, we constantly evaluate the latest data integration tools and can help you create the right workbench to transform your data management capabilities. For example, Kylo™ – our open source data pipeline ingest and design tool – won Best Emerging Technology at Computing Magazine UK’s annual Big Data and IoT Excellence Awards in June 2017.

If you’re eager to unleash the value of your data, our Velocity services portfolio could be just the thing you’re looking for. Find out more here:

Andy JohnstonAndy Johnston, Senior Industry Consultant, Finance Centre of Excellence, Teradata International

Andy has more than 20 years’ UK Financial Services marketing and customer management experience, including work transforming the customer management capability at Lloyds Banking Group. He’s managed analytical teams and run operational marketing systems. And as an expert direct-marketing practitioner, he has helped set-up numerous multi-channel and multi-stage contact programmes, to drive customer advocacy and to meet customer needs.

Taking customer journey from mapping to guiding

November 15, 2017


I’ve recently gotten a lot of questions from colleagues about the need for software to create customer journey maps and this recent analyst report by Richard Snow, “Why All the Fuss about Customer Journey Maps?” confirms that it’s moving from interesting professional discussion to business trend.

Yes, as Mr. Snow discusses, it’s progress to move from inside-out looking, Post-it note-based journeys to tools that use data to map actual journeys that real customers take as they try to meet their needs across the myriad of channels available today. It seems akin to me to going from the Babylonian Map of the World to satellite images. The Babylonian Map of the World is the earliest surviving map of the globe from circa 600 B.C. It is a figurative representation of the world, tied closely to the Babylonians’ religious beliefs. And while the endeavor to create this map was likely a Herculean effort, the result is a map that doesn’t accurately represent the world and is skewed to favor the Babylonian’s vantage point.

When put that way, it’s striking how similar this description is to how well-intentioned corporate teams — locked in conference rooms, buzzing with caffeine and great ideas — create customer journey maps.

There was a long, tech-fueled road of progress between this ancient map and satellite images. But now the stakes are even higher — satellite imagery and real-time insights are both at play when people navigate from point A to point B. Shouldn’t the journeys we provide our customers do the same?

The Teradata clients that move from mapping to real-time navigation are generating significant financial returns. A top 10 global bank that implemented real-time customer journey mapping realized $30 million in incremental profit based on improvements acquisition and cross-sell — and its technology investment cost was recouped in the first quarter.

Teradata has a proven architecture and the implementation experience to create an integrated analytics, marketing and customer experience ecosystem leveraging the best of open-source and Teradata technology, and builds on existing client investments in data, analytics and interaction capabilities. Teradata was the only vendor to be listed as a leader in three Forrester Wave reports published this year, spanning Customer Journey Analytics Orchestration Platforms, Customer Journey Analytics Visioning Platforms and Real-Time Interaction Management.

Customer journeys can be mapped, significant events can be predicted, and interactions can be poised and ready to react to use the breadth of customer knowledge to guide each customer to the optimal destination in an automated, “always on” system. When done right, customer satisfaction and revenue goes up, while marketing and customer care costs go down.

Customer journey mapping software will help you move from the Babylonian map to the satellite image. But the next step is integrating real-time navigation — something your customers are used to. Isn’t it time you provided it? Let me show you how!

Kathy Koontz