Impact Analysis Predictive Modelling - Feedback Ferret
Download PDF

Impact Analysis Predictive Modelling

best kind of feedback

CASE STUDY: How Impact Analysis can help a business

We were recently faced with a challenge which we rose to in true Ferret style. One of our automotive clients wanted to find out what their customers really wanted from their brand. Using Impact Analysis, we told them.

Our client wanted to know at which points of their customers’ journey issues arose and what those issues were.

To get to the heart of the matter, we took customer feedback comments from survey data over a two year period (over 50,000 comments). This cohort was broken down into two areas: sales and service. After all, these are two distinct business functions and as such have different processes and pinch points that we wanted to identify.

Impact analysis with automotive client

To make sense of this large data set, we used Impact Analysis Predictive Modelling to identify the key themes, both positive and negative, that were the key drivers of satisfaction and dissatisfaction for both departments.

Once these key topics were identified, they were overlaid against the client’s own customer journey maps. The impact analysis highlighted many very positive elements in the “initial greeting to purchase” and several negative issues during the “purchase to finish” phases which led to dissatisfaction amongst clients.

As a result of the full analysis, we drafted a list of things the customers expect during the sales and service process. We identified a number of high level priorities for the business to focus on improving that most impacted satisfaction and loyalty. These included communication improvements with the client post-purchase, better internal processes required for busy times at dealerships, and changes to the standards with which sales and service customers are treated – every customer is equally important.

The response from our client to the analysis was overwhelmingly positive:

“I would like to thank you for this work and acknowledge that this report is very clear, concise and valuable to us. The content has been socialised within the business and as such we are using the report findings to streamline our approach in our key customer service and core process strategies.”

Purpose of this Document

This paper sets out the Impact Analysis modelling that Feedback Ferret carries out after the text analysis processing is completed. It covers the modelling process itself, together with ideas of how to use the findings. The purpose of the Impact Analysis modelling is to understand the influence that each topic has on customer loyalty. To understand Impact Analysis, we need to start with a summary of the Reporting Topics that we create as the main output from the text analysis process. 

Background to Feedback Ferret Reporting Topics

From the text analytics engine processing outputs, Feedback Ferret typically codes 300-600 topics which are customised to each client. The purpose of creating this extensive topic set is two-fold:- Enable ‘Discovery’ – ie, what is it in the data that is causing some kind of outcome? For example, what’s causing dissatisfaction with particular outlets or with particular products or services, etc? Enable ‘Validation’ – ie, you may have an idea that a particular issue is causing a problem, or someone has asked about a particular topic, and you want to understand more about it – how frequently does it occur, trending over time, penetration across outlets, influence on products or rating scores, etc. Both these avenues of interrogation require a comprehensive set of topics that describe the full customer experience, at a level of detail that the company finds useful and can act on when required. We work closely with clients at the start of a project to establish a workable and useful set of reporting topics. Inevitably, these topics need to evolve over time, and we continue to work closely with clients to maintain the topic set so that it continues to meet the two primary objectives outline above, whilst also enabling access to new issues.

Typically over time, we find that the reporting topics evolve:-

  • New topics – new issues arising from client requirements – new products, marketing campaigns, etc
  • Split existing topics – where more granularity is required about a particular issue
  • Merge existing topics – where less detail and granularity is required
  • Delete topics – where there is no interest or action that can be taken
  • Renaming topics – to better engage with people across the business so they get a better understanding of what is being talked about
  • Re-organisation/restructuring topic sets – this tends to happen when topics need to align with other corporate requirements, or where topics need to better reflect a third party naming and structure convention, such as Nielsen or Kantar in retail markets, or JD Power in automotive.

continuous improvement

The Purpose of Impact Analysis Modelling

We use Pearson Correlation Coefficient statistical modelling to provide a coefficient to indicate the predictive power of each topic in terms of its influence on your customer loyalty.

The Individual Customer Score that we create for this modelling creates a value that closely mimics actual behavioural loyalty. This is the dependent variable for the modelling process.

Where a client is able to provide transaction data from which we can derive actual customer loyalty and defection behaviour, this can replace the Individual Customer Score as the dependent variable for the modelling process.

IA diagram

The Individual Customer Score takes into account the three attributes of:

  • The main rating score (typically NPS or 5 star satisfaction rating)
  • Sentiment Score
  • The strong verbal statements about loyalty, defection and recommendation.

These individual elements can be weighted where clients have reservations about rating scores. For example, there may be survey manipulation by retail outlets and so Sentiment score may need to be up-weighted in importance.

The Impact Analysis (IA) correlation modelling generates a coefficient score between -1-to +1 for each topic, together with a p-score to indicate whether the findings are statistically significant.

The Pearson Correlation calculation incorporates both the volume of records and the extent to which each topic predicts the Individual Customer Score, so the ranking of topics in the model does not then need to multiply out by the volume of survey hits – that is already factored in to the coefficient score and therefore the topic ranking.

The IA score is added to the Feedback Ferret database against each qualifying topic. The highest positive and negative coefficient scores can be represented in a simple chart, as in the example above. The score itself is not vital; what is more important is the ranking and comparative ratios that the coefficient gives to the Topics.

Where a client has surveys/feedback from different transactions or interactions with the business, we usually run separate IA models for each of those. For example, an automotive company would have different scores for sales vs. service transactions


Customer Defection Risk Score

The verbatim comment(s) of each customer will result in different combinations of IA topics that have been coded in the text analysis process. Some customers may get no IA topic hits, others may have a large number of positive or negative topics. We sum the coefficient scores of IA topic hits for each customer, giving a net positive or negative score for each customer, which we call the Customer Defection Risk score (CDR).

To make it easy to work with this new metric, we assign each customer to a percentile, therefore giving a range of 0-100 to the defection risk score. Those customers with higher scores have a greater predicted likelihood to defect, having mentioned more of the negative rated IA topics in their verbatims.

This simple defection likelihood score can further help you to identify and ring fence those customers for whom remedial action needs to be taken to retain their business.


What to do with the Impact Analysis Results

So – what can we do with the IA results data? How can this additional level of analysis help clients to improve their customer experience and loyalty, and to drive improvements in business performance?

take action

Scale of Improvements

Overall, it is worth considering what scope the topics have across the business, and what actions you may want to take, based on:-

A) Just the surveys that show these topic hits; ie, if you can only act on the specific issues for these specific customers.

For example, a sales person being rude to a customer or their spouse can only be assumed to affect that one customer, and can only be taken up with that individual customer. You can’t issue a broadcast apology to all customers for the sales person’s rudeness! (but you could expand the task from an individual customer apology to tackling the rude sales person – especially if there are multiple instances of such rude behaviour).

b) Across the entire organisation – ie, do the IA findings enable you to take widespread action that will potentially impact a much larger group of customers, rather than just the specific survey responders who have mentioned these topics?

For example, if the opening hours of an outlet are being criticised, and you change the time the store is open, it is entirely reasonable to assume that everyone will benefit from the improved accessibility. Also, it is a great idea to contact all customers and say that as a result of feedback received, the store hours are being changed.

Acting on Positive or Negative issues?

At Feedback Ferret we continually look for evidence from our clients and from published best practice about whether it is best to try to eradicate the negative issues, or to amplify the positives. It seems that the jury is still out on this, with no conclusive evidence on either side, so it really means each organisation needs to work on what it deems to be the best solution under the circumstances. In most cases, it is likely to lead to a mix of tackling negative issues whilst also trying to maximise the positive.

Some practioners claim that you get greatest results from moving NPS ‘Neutrals’ to Promoters, whereas others advise trying to eradicate the things that cause Detractors to be so dissatisfied (Forrester & others).

Back in the archives, the originator of the NPS metric, Fred Reichheld, was of the opinion that to maximise loyalty, you needed customers to rate you extremely highly – simply being ‘satisfied’ was not sufficient to stop you switching your loyalty at your next purchase.

You will find multiple examples where companies have tackled the issues that are driving very low ratings – often referred to as the ‘brand assassins’ – customers scoring 0 or 1 out of 10.

The Dirichlet model of loyalty points to the fact that most customers maintain a portfolio of brands – the ‘consideration set’ – from which they buy or would consider buying. Most often, this consideration set is heavily dictated by market share of those brands. The main effort for brands is to ensure that they don’t do anything that causes customers to knock the brand out of this consideration set, and reduce potential purchases or repurchases from existing buyers. Text Analytics can be very illuminating on what causes customers to remove brands from that consideration set.


Using Impact Analysis to prioritise improvements

1 Initial Exploration

Prioritise topics for action

  • Use the IA ranking report to prioritise issues for remedial action
  • Drill down in Ferret Explore dashboards to the verbatims linked to the topics to get a quick understanding of the issues
  • Address negative staff issues with changes to recruitment or training
  • Explore and compare these topics across outlets or call centre teams, trends/spikes over time, trends by products, services, etc.

Negative Topics

  • Identify the priority issues that cause dissatisfaction with customers
  • Explore how you can reduce or stop the occurrences of these issues
  • Dig further into the root causes of these issues – see below Positive Topics
  • Find outlets or call centre teams where the positive issues are underrepresented, and work on increasing the presence of these issues
  • Learn from, and spread good practice from high performing locations
  • Monitor uplift in positive topic feedback over time
  • Maximise the use of positive aspects in sales and marketing communications

2 Root Cause Analysis

IA results act as a fast-track into Root Cause Analysis to identify why customers are dissatisfied and why they are most likely to defect. Drill down from IA priority topics into the underlying verbatims to read and get a feel for why customers raise those issues. Explore the data more deeply with:-

Cross Tab views 

  • How do the IA topics map out across outlets, touch-points, products?
  • Where are the spikes in the data? Or are these issues ‘flat’ across the business, with equal levels of occurrences everywhere?
  • A quick and simple approach to finding the ‘spikes’ in data is to compare the percentage of associated ‘entities’ in which 50% of the topics occur. For example, if 50% of a topic (eg ‘Poor Booking Experience’) occurs in just 10% of outlets, then you know where to focus the efforts to stop this happening. If 50% of the problems occur in 50% of outlets, the issue is likely to be generic across the business and not dependent on individual outlet actions.
  • Mapping IA topics to time periods also enables trending by time of day, day of week, as well as time trends by day, week, month, etc.

Intersect Topics 

  • Using Ferret Explore dashboards, explore the high ranking topic ‘intersects’ that are also present in i) the same sentences or ii) whole verbatim answers where customers also mention the high ranking IA topics.
  • Often, several high ranking IA topics will intersect with each other, resulting in a compounding of negative customer experience, eg:- – Food – Poor Quality AND – Staff – Poor Availability AND – Staff – Poor Customer Service AND – Cost – Poor Value for Money
  • In this example, it becomes clear that there is a ‘domino’ effect of staff making themselves scarce where the customer wants to complain about poor quality food, with a knock-on result of the customer complaining about the costs as well
  • In other outlets, you may find that customers complain about poor quality food, but staff members are available and are helpful in resolving the issue – leading to higher satisfaction and loyalty rates
  • The comparative rating scores for these two cohorts of customers can be used to calculate a RoI on making improvements (see the RoI section below)
  • Understanding at which outlets the negative issues occur and the time of day can start to pinpoint where the company needs to focus efforts on staff training or resource levels to meet customer expectations.

3 Hot Alerts

Working through the first two steps of deep dives and Root Cause Analysis helps to flush out the requirements for Hot Alert triggers.

Hot Alerts are typically triggered in 3 areas:-

1. Customers needing fast remedial action:-

  • One-to-one customer interactions
  • Small groups of customers with common issues – who may not need individual contact

2. Staff needing actions taking:-

  • Management actions
  • Changing staff behaviours / actions 3. Inanimate objects or processes that needing fixing / changing ● Improve toilet cleanliness ● Improve online booking process

3. Inanimate objects or processes that needing fixing / changing

  • Improve toilet cleanliness
  • Improve online booking process

Hot Alerts can be triggered from any combination of:-

  • Topics
  • Emotion topics and scores
  • Survey rating scores
  • Sentiment scores
  • Customer Defection Risk score (CDR)
  • Structured data (outlet, product, service, survey type, etc)

Feedback Ferret can either configure automated Hot Alerts to be triggered in near real-time (depending on survey upload frequencies) or clients can action these themselves directly from results data uploads from the Feedback Ferret platform.

4 Feedback Ferret 8-Point Improvement Plan

Feedback Ferret provides a pro-forma template for taking action based on feedback response data and the insight we unearth. This framework identifies 8 key areas of the business that can be mapped to topics to address positive or negative issues, ideas, suggestions, and other improvements. The 8 key areas are:


8 point improvement plan

The 8 key areas can be further sub-divided by factors such as Customer Journey stage or touch-point, and of course, any initiatives taken can then be monitored over time to assess the impact on customer experience from future feedback.

Although cost reduction opportunities (for example, refunds, goodwill payments, compensation claims, ombudsman charges, warranty claims, etc.) may not be the primary goal of the feedback analysis project, using IA and Root Cause Analysis we can unearth where changes are necessary to drive down these specific costs to the business.

5 Return on Investment modelling

For many clients, we are able to calculate a financial value correlation against their primary rating metric. For example, each gain in NPS rating equates to £x,000 increase in net income per year.

From that point, you can use the IA weighting scores against each topic to run predictive modelling to calculate the payback you can expect by increasing the presence of positive topics, or reducing the occurrence of negative topics.