NPS (‘Net Promoter Score’) - a
popular index
An
investigation of its features
The index is used when evaulating the result of an audit. This document invesigates some of the main features and limits of the index. A more theoretical treatment is included at the end of the document.
The outcome from an audit (integers 0 - 10) is separated into three proportions – ‘Ambassadors’ (p1), ‘Passive’ (p2) samt ‘Critical’ (p3). NPS is limited (-100 < NPS < 100) and is defined in the following way:
\[ \begin{align} \Large NPS=\frac{\text{Number of Ambassadors}-\text{Number of Criticals} }{\text{Number of Respondents}}\cdot100 \end{align} \]
A hypothetical table contains two extreme and illuminating cases:
| Case | Number of Ambassadors | Number of Passive | Number of Critical |
|---|---|---|---|
| A | 3 | 94 | 3l |
| B | 48 | 4 | 48 |
A hypothetical outcome. Using the expression and the hypothetical data we get the following two NPS-values:
\[ \begin{align} \Large NPS=\frac{3-3}{100}\cdot100=0 & \phantom{litet tomt uterymme} \Large NPS=\frac{48 -48}{100}\cdot100=0 \end{align} \]
It is somewhat strange that two completely different cases give the same
NPS-value. Case A can hardly cause the same amount of concern and
activities as case B where approximately half the group is
‘Critical’.
If we instead place weights or costs to the three
categories we can achive at least some discrepancy. Suppose that we have
the following weights: ’Ambassador 1’, ’Passive 0’, ’Critical -10’ where
a single ‘critical’ must be upweighted by 10 ’Ambassadors’:
\[ \begin{align*} \Large NPS_{B}=\frac{48\cdot1+4\cdot0+48\cdot(-10)}{100}\cdot100=-432 \end{align*} \]
This suggestion gives a clear difference between A and B. However, note that this is only a suggestion, one of many possibilities.
Further analysis and some simulations.The statistical litterature contains a number of requirements of an estimator such as NPS. These requirements are important but they are of mathematical nature and are therefore omitted here.
NPS is thus a ratio between two integers and the definition can, after some rewriting, also be expressed as the difference between two proportions and multiplied ba a factor 100:
\[
\begin{align}
\Large NPS=\left(p_{1}-p_{3} \right)\cdot100
\end{align}
\]
The result is rounded to interger values and can be within the interval
+/-100. (Note that it is not obvious to calculate the difference
between the ratios, another option is the ratio
i.e. p1/p3 but that would give
results from 0 to infinity. If all members answered correctly and
according their own opinion we would get exact values of the different
proportions and act accordingly.
However, in general the response is
ofte much less than 100% and thus the result must be considered as a
sample from the population and thus contain a random component.
Simulation. The simulation below is based on the proportion Ambassadors and Criticals is 0.50 and 0.40. The following parameter values and assumptions are used:
• Size of region/section: 50 persons
• Proportion replies: 35 %
• Proportion Ambassadors: 50 %
• Proportion Criticals: 40 %
Using these parameter values 10 000 surveys are performed, twice. The
result is shown in two graphs below. One histogram with all results and
one histogram with the differences between the two surveys.
Note
that all assumptions are kept constant, the variation reflects the
randomness, no changes are induced.
#klippy::klippy(position = c('top', 'right'))
k2 <- 0.35 # Proportion responding.
k3 <- 50 # Population size.
k4 <- round(k2*k3, 0) # Number responding.
k5 <- 10000 # Number of populations.
k6 <- k4*k5 # Total number of answers (1 or 2 or 3).
propAmb <- 0.50
propNeutral <- 0.10
propCrit <- 0.40
proportions <- c(propAmb, propNeutral, propCrit)
c3 <- sample(1:3, size = k6, replace = TRUE, prob = proportions) # Random answers of the 3 alternatives.
c6 <- ifelse(c3 == 1, 1, 0) # 1 = "Ambassador"
c9 <- ifelse(c3 == 3, 1, 0) # 3 = "Critical"
c4 <- rep(1:k5, each = k4) # Creates group number.
c7 <- aggregate(c6, list(c4), FUN = mean) # Calculates group average
c10 <- aggregate(c9, list(c4), FUN = mean)
c12 <- (c7$x - c10$x)*100 # NPS-value, i.e. difference 'Ambassador' and 'Critical'
c13 <- round(c12, 0) # Round, no decimal
m1 <- mean(c13)
mydata1 <- data.frame(c13) # Creates a data frame
library(ggplot2)
hist1 <- ggplot(mydata1, aes(c13)) + geom_histogram(binwidth = 10, color='black', fill='red', alpha = 0.3,
position = 'identity')
hist1 <- hist1 + annotate("text", x = -55, y = Inf, label="Simulated distribution of NPS-values", hjust=0,
vjust = 2, size = 5, colour = "blue" )
hist1 <- hist1 + geom_point(x = m1, y = 0, shape = 17, colour = "red", size = 5.5)
hist1 <- hist1 + xlab("NPS-value")
hist1
Comments figure 1. The histogram shows 10 000 simulated values and the expected NPS-average is 10 as the proportion Ambassadorsis 0.50 and the poportion Criticals is 0.40, this seems to be reasonable according to the histogram. The random variation seems to be between -80 to +90. See the summary below.
library(kableExtra)
library(vtable)
st(mydata1, vars=c('c13'), title='', summ=c('notNA(x)', 'mean(x)', 'sd(x)', 'min(x)', 'max(x)'),
col.width=c(14, 15, 18, 18, 20, 20), summ.names=c('n', 'average', 'std', 'min', 'max'), digits=3)| Variable | n | average | std | min | max |
|---|---|---|---|---|---|
| c13 | 10000 | 9.78 | 22.4 | -67 | 83 |
Difference between two surveys
It is of course natural to compare two different surveys and pose the question ‘is there an improvement or not?’
Suppose that we have NPS-results from two different quarters described as ‘Investigation quarter Q1 2016’ and ‘Investigation quarter Q1 2017’ and that we calculated the differences. Following histogram shows 10 000 such comparisons.
c3 <- sample(1:3, size = k6, replace = TRUE, prob = proportions)
c6 <- ifelse(c3 == 1, 1, 0)
c9 <- ifelse(c3 == 3, 1, 0)
c4 <- rep(1:k5, each = k4)
c7 <- aggregate(c6, list(c4), FUN = mean)
c10 <- aggregate(c9, list(c4), FUN = mean)
c12 <- (c7$x - c10$x)*100
c14 <- round(c12, 0)
c17 <- c13 - c14
m2 <- mean(c17)
mydata2 <- data.frame(c17)
hist2 <- ggplot(mydata2, aes(c17)) + geom_histogram(bins=24, color='black', fill='red',
alpha = 0.3, position = 'identity')
hist2 <- hist2 + annotate("text", x = -100, y = Inf, label="Simulated difference between Q1 and Q2",
hjust=0, vjust = 2, size = 5, colour = "blue" )
hist2 <- hist2 + geom_point(x = m2, y = 0, shape = 17, colour = "red", size = 5.5)
hist2 <- hist2 + xlab("NPS-value")
hist2
Comments figure 2. The histogram shows a large difference between the two measurements even if there is no difference! This means that there is a risk for completely incorrect conclusions regarding the development. Because of comparing two random values the standard deviation is now much larger.
library(kableExtra)
library(vtable)
st(mydata2, vars=c('c17'), title='', summ=c('notNA(x)', 'mean(x)', 'sd(x)', 'min(x)', 'max(x)'), col.width=c(14, 15, 18, 18, 20, 20),
summ.names=c('n', 'average', 'std', 'min', 'max'), digits=3)| Variable | n | average | std | min | max |
|---|---|---|---|---|---|
| c17 | 10000 | -0.252 | 31.8 | -106 | 123 |
Final comments. The ‘non-response’ usually gives some extra problem in an investigation and often is regarded as a quality problem. How should these ‘non-respondes’ be handled? Call all Ambassadors or Criticals? Should the observed proportion of Criticals be applied to the result and thus artificially increase the sample size?
Appendix
Suppose that we have the following proportions:
p1 = Proportion Ambassadors
p2 = Proportion Passive
p3 = Proportion Criticals
The variance V(pi) of such a proportion is:
\[ \begin{align} \Large V(p_{i}) =\frac{p_{i}\cdot(1 - p_{i})}{n} \end{align} \]Above it is stated that NPS can be expressed as:
\[ \begin{align} \Large NPS=\left(p_{1}-p_{3} \right)\cdot100 \end{align} \]
NPS is thus a so-called linear combination (i.e. the sum or
difference of variables) and the variation of NPS is effected by
this. Sometimes there is also a covariation (so-called covariance)
between the values and this covariance is added to the total
variance.
The covariance is either negative, zero or positive,
depending om the variables. (There is a negative covariance if one value
in a pair is higher while the other is lower. In a diagram this is
visible as a negative trend across the graph.)
The covariance
between p1 and p3 is expected to be
negative - if one proportion tends to be high, the other tends to be
low, the sum of all proportions is of course 1.
This covariance
Cov(p1, p3) has the following
mathematical expression:
The general expression for the linear combination NPS becomes (see the literature for the details):
\[ \begin{align} \Large V(NPS) =[V(p_{1})+V(p_{3})+2\cdot Cov\left(p_{1}, p_{3}\right)]\cdot100^{2} \end{align} \]
With substitrution the following expression is obtained:
\[ \begin{align} \Large V(NPS) =\left[\frac{p_{1}\cdot(1 - p_{1})}{n} + \frac{p_{3}\cdot(1 - p_{3})}{n} + 2\cdot \frac{p_{1}\cdot p_{3}}{n}\right]\cdot100^{2} \end{align} \]With some rearranging following expression is obtained:
\[ \begin{align} \Large V(NPS) =\left[\frac{p_{1}+p_{3} - (p_{1} - p_{3}) ^2}{n} \right]\cdot100^{2} \end{align} \]For the interpretation the standard deviation is perhaps simpler. The standard deviation is the square root of the variance and with indata from above we get the following:
\[ \begin{align} \Large \sigma_{NPS} = \sqrt{ \left[\frac{0.5+0.4 - (0.5 - 0.4) ^2}{n} \right]\cdot100^{2}}=22.2 \phantom{invisible}\text{(NPS-units)} \end{align} \]
The denominator ’18’ comes from a population of 50 but only 35% responses. A common rule-of-thumb is that +/- 3 standard deviations encompass of the random variation, here approximately +/- 66. We expected a NPS-value of (0.5 0.1) 100 I.e. 10 but it is obvious that the random variation can give a result far from the expected and thus causing a risk for incorrect decision.
The standard deviation of these differences becomes 31.4 and corresponds well with the simulated result from histogram 2 above:
\[ \begin{align} \Large \sigma_{Diff} = \sqrt{ 22.2^{2} + 22.2^{2} }=31.4 \phantom{invisible}\text{(NPS-units)} \end{align} \]