We were recently faced with a challenge which we rose to in true Ferret style. One of our automotive clients wanted to find out what their customers really wanted from their brand. Using Impact Analysis, we told them.
Our client wanted to know at which points of their customers’ journey issues arose and what those issues were.
To get to the heart of the matter, we took customer feedback comments from survey data over a two year period (over 50,000 comments). This cohort was broken down into two areas: sales and service. After all, these are two distinct business functions and as such have different processes and pinch points that we wanted to identify.
To make sense of this large data set, we used Impact Analysis Predictive Modelling to identify the key themes, both positive and negative, that were the key drivers of satisfaction and dissatisfaction for both departments.
Once these key topics were identified, they were overlaid against the client’s own customer journey maps. The impact analysis highlighted many very positive elements in the “initial greeting to purchase” and several negative issues during the “purchase to finish” phases which led to dissatisfaction amongst clients.
As a result of the full analysis, we drafted a list of things the customers expect during the sales and service process. We identified a number of high level priorities for the business to focus on improving that most impacted satisfaction and loyalty. These included communication improvements with the client post-purchase, better internal processes required for busy times at dealerships, and changes to the standards with which sales and service customers are treated – every customer is equally important.
The response from our client to the analysis was overwhelmingly positive:
“I would like to thank you for this work and acknowledge that this report is very clear, concise and valuable to us. The content has been socialised within the business and as such we are using the report findings to streamline our approach in our key customer service and core process strategies.”
Purpose of this Document
This paper sets out the Impact Analysis modelling that Feedback Ferret carries out after the text analysis processing is completed. It covers the modelling process itself, together with ideas of how to use the findings. The purpose of the Impact Analysis modelling is to understand the influence that each topic has on customer loyalty. To understand Impact Analysis, we need to start with a summary of the Reporting Topics that we create as the main output from the text analysis process.
Background to Feedback Ferret Reporting Topics
From the text analytics engine processing outputs, Feedback Ferret typically codes 300-600 topics which are customised to each client. The purpose of creating this extensive topic set is two-fold:- Enable ‘Discovery’ – ie, what is it in the data that is causing some kind of outcome? For example, what’s causing dissatisfaction with particular outlets or with particular products or services, etc? Enable ‘Validation’ – ie, you may have an idea that a particular issue is causing a problem, or someone has asked about a particular topic, and you want to understand more about it – how frequently does it occur, trending over time, penetration across outlets, influence on products or rating scores, etc. Both these avenues of interrogation require a comprehensive set of topics that describe the full customer experience, at a level of detail that the company finds useful and can act on when required. We work closely with clients at the start of a project to establish a workable and useful set of reporting topics. Inevitably, these topics need to evolve over time, and we continue to work closely with clients to maintain the topic set so that it continues to meet the two primary objectives outline above, whilst also enabling access to new issues.
Typically over time, we find that the reporting topics evolve:-
The Purpose of Impact Analysis Modelling
We use Pearson Correlation Coefficient statistical modelling to provide a coefficient to indicate the predictive power of each topic in terms of its influence on your customer loyalty.
The Individual Customer Score that we create for this modelling creates a value that closely mimics actual behavioural loyalty. This is the dependent variable for the modelling process.
Where a client is able to provide transaction data from which we can derive actual customer loyalty and defection behaviour, this can replace the Individual Customer Score as the dependent variable for the modelling process.
The Individual Customer Score takes into account the three attributes of:
These individual elements can be weighted where clients have reservations about rating scores. For example, there may be survey manipulation by retail outlets and so Sentiment score may need to be up-weighted in importance.
The Impact Analysis (IA) correlation modelling generates a coefficient score between -1-to +1 for each topic, together with a p-score to indicate whether the findings are statistically significant.
The Pearson Correlation calculation incorporates both the volume of records and the extent to which each topic predicts the Individual Customer Score, so the ranking of topics in the model does not then need to multiply out by the volume of survey hits – that is already factored in to the coefficient score and therefore the topic ranking.
The IA score is added to the Feedback Ferret database against each qualifying topic. The highest positive and negative coefficient scores can be represented in a simple chart, as in the example above. The score itself is not vital; what is more important is the ranking and comparative ratios that the coefficient gives to the Topics.
Where a client has surveys/feedback from different transactions or interactions with the business, we usually run separate IA models for each of those. For example, an automotive company would have different scores for sales vs. service transactions
Customer Defection Risk Score
The verbatim comment(s) of each customer will result in different combinations of IA topics that have been coded in the text analysis process. Some customers may get no IA topic hits, others may have a large number of positive or negative topics. We sum the coefficient scores of IA topic hits for each customer, giving a net positive or negative score for each customer, which we call the Customer Defection Risk score (CDR).
To make it easy to work with this new metric, we assign each customer to a percentile, therefore giving a range of 0-100 to the defection risk score. Those customers with higher scores have a greater predicted likelihood to defect, having mentioned more of the negative rated IA topics in their verbatims.
This simple defection likelihood score can further help you to identify and ring fence those customers for whom remedial action needs to be taken to retain their business.
What to do with the Impact Analysis Results
So – what can we do with the IA results data? How can this additional level of analysis help clients to improve their customer experience and loyalty, and to drive improvements in business performance?
Scale of Improvements
Overall, it is worth considering what scope the topics have across the business, and what actions you may want to take, based on:-
A) Just the surveys that show these topic hits; ie, if you can only act on the specific issues for these specific customers.
For example, a sales person being rude to a customer or their spouse can only be assumed to affect that one customer, and can only be taken up with that individual customer. You can’t issue a broadcast apology to all customers for the sales person’s rudeness! (but you could expand the task from an individual customer apology to tackling the rude sales person – especially if there are multiple instances of such rude behaviour).
b) Across the entire organisation – ie, do the IA findings enable you to take widespread action that will potentially impact a much larger group of customers, rather than just the specific survey responders who have mentioned these topics?
For example, if the opening hours of an outlet are being criticised, and you change the time the store is open, it is entirely reasonable to assume that everyone will benefit from the improved accessibility. Also, it is a great idea to contact all customers and say that as a result of feedback received, the store hours are being changed.
Acting on Positive or Negative issues?
At Feedback Ferret we continually look for evidence from our clients and from published best practice about whether it is best to try to eradicate the negative issues, or to amplify the positives. It seems that the jury is still out on this, with no conclusive evidence on either side, so it really means each organisation needs to work on what it deems to be the best solution under the circumstances. In most cases, it is likely to lead to a mix of tackling negative issues whilst also trying to maximise the positive.
Some practioners claim that you get greatest results from moving NPS ‘Neutrals’ to Promoters, whereas others advise trying to eradicate the things that cause Detractors to be so dissatisfied (Forrester & others).
Back in the archives, the originator of the NPS metric, Fred Reichheld, was of the opinion that to maximise loyalty, you needed customers to rate you extremely highly – simply being ‘satisfied’ was not sufficient to stop you switching your loyalty at your next purchase.
You will find multiple examples where companies have tackled the issues that are driving very low ratings – often referred to as the ‘brand assassins’ – customers scoring 0 or 1 out of 10.
The Dirichlet model of loyalty points to the fact that most customers maintain a portfolio of brands – the ‘consideration set’ – from which they buy or would consider buying. Most often, this consideration set is heavily dictated by market share of those brands. The main effort for brands is to ensure that they don’t do anything that causes customers to knock the brand out of this consideration set, and reduce potential purchases or repurchases from existing buyers. Text Analytics can be very illuminating on what causes customers to remove brands from that consideration set.
Using Impact Analysis to prioritise improvements
1 Initial Exploration
Prioritise topics for action
2 Root Cause Analysis
IA results act as a fast-track into Root Cause Analysis to identify why customers are dissatisfied and why they are most likely to defect. Drill down from IA priority topics into the underlying verbatims to read and get a feel for why customers raise those issues. Explore the data more deeply with:-
Cross Tab views
3 Hot Alerts
Working through the first two steps of deep dives and Root Cause Analysis helps to flush out the requirements for Hot Alert triggers.
Hot Alerts are typically triggered in 3 areas:-
1. Customers needing fast remedial action:-
2. Staff needing actions taking:-
3. Inanimate objects or processes that needing fixing / changing
Hot Alerts can be triggered from any combination of:-
Feedback Ferret can either configure automated Hot Alerts to be triggered in near real-time (depending on survey upload frequencies) or clients can action these themselves directly from results data uploads from the Feedback Ferret platform.
4 Feedback Ferret 8-Point Improvement Plan
Feedback Ferret provides a pro-forma template for taking action based on feedback response data and the insight we unearth. This framework identifies 8 key areas of the business that can be mapped to topics to address positive or negative issues, ideas, suggestions, and other improvements. The 8 key areas are:
The 8 key areas can be further sub-divided by factors such as Customer Journey stage or touch-point, and of course, any initiatives taken can then be monitored over time to assess the impact on customer experience from future feedback.
Although cost reduction opportunities (for example, refunds, goodwill payments, compensation claims, ombudsman charges, warranty claims, etc.) may not be the primary goal of the feedback analysis project, using IA and Root Cause Analysis we can unearth where changes are necessary to drive down these specific costs to the business.
5 Return on Investment modelling
For many clients, we are able to calculate a financial value correlation against their primary rating metric. For example, each gain in NPS rating equates to £x,000 increase in net income per year.
From that point, you can use the IA weighting scores against each topic to run predictive modelling to calculate the payback you can expect by increasing the presence of positive topics, or reducing the occurrence of negative topics.