About the poll: Telephone interviews were conducted by live, professional interviewers with 405 voters selected at random from registered voters in Washington state June 9-13. Margin of sampling error is ±5% at the 95% level of confidence.

That’s a solid statement. But what struck me was that the commentary, based on the chart I’m reproducing here, might seem inconsistent with the reliability statement above.

The accompanying text reads “More Washingtonians claim allegiance to Democrats than to Republicans, but independents are tilting more towards the GOP.” How can this be, when the difference is only 4% (6% more Democrats, 10% more Republicans)? The answer lies in how statistical testing works and the fact that statistical tests take into account the differences arising from different event probabilities.

First, let’s dissect the reliability statement. It means that results from this survey will be within ±5% of the true population, registered voters in this case, 19 out of 20 times if samples of this size were drawn from the registered voter list and surveyed. (One time in 20 the results could be outside of that ±5% range; that’s the result of sampling.) This ±5% range is actually the worst case and is only this high at for 50% event probabilities – meaning the situation where responses are likely to be equally split. Researchers use the worst case figure to ensure that they sample enough people for the desired reliability whatever the results are. In this case, the range for Independents leaning towards Democrats is ±2.3% (i.e. 3.7% to 8.3%) while the range for Independents leaning towards the GOP is ±2.9% (i.e. 7.9% to 12.9%). But these ranges overlap so how can the statement about tilting more to the Republicans be made with confidence?

We need to run statistical tests to apply more rigor to the reporting. In this case t-tests or z-tests will show the answer we need. The t-test is perhaps more commonly used because if works with smaller sample sizes, although we have a large enough sample here for either. Applying a t-test to the 6% and 10% results we find that the t-score is 2.02 which is greater than the 1.96 needed for 95% confidence. The differences in proportions are NOT likely due to random chance, and the statement is correct.

To illustrate the impact of event probability on statistical testing, this diagram shows how smaller differences in proportions are more able to discriminate differences as the event probability gets further away from the midpoint. Note that even at 6% difference results between about 20% and 70% (for the lower proportion) won’t generate a statistically significant difference, while at 8% difference the event probability doesn’t matter. Actually, 7% is sufficient – just.

Without using statistical testing, you won’t be sure that the survey results you see for small differences really mean that the groups in the surveyed population differ. How can you prioritize your efforts for feature A versus feature B if you don’t know what’s really important? Do your prospects differ in how they find information or make decisions to buy? You can create more solid insights and recommendations if you test.

Tools for statistical testing

The diagram above shows how things work, and is a rule of thumb for one type of testing. But it is generally best to use one or more tools to do significance testing.
Online survey tools don’t generally offer significance testing. The vendors tell me that users can get into trouble, and they don’t want to provide support. So you are need to find your own solutions. If you are doing analysis in Excel you can use t-tests and z-tests that are included in the Data Analysis Toolpak. But these only work on the individual results so if you are trying to look at aggregate proportions (as might be needed when using secondary research as I did above) you need a different tool. Online calculators are available from a number of websites, or you might want to download a spreadsheet tool (or build your own from the formulae). These tools are great for a quick check for a few data points without having to enter a full data set.

SPSS has plenty of tests available, so if you are planning on doing more sophisticated analysis yourself, or if you have a resource you use for advanced analysis then you’ll have the capability available. But SPSS, besides being expensive, isn’t all that efficient for large numbers of tests. I use SPSS for regressions, cluster analysis and the like, but I prefer having a set of crosstabs to be able to quickly spot differences between groups in the target population. We still outsource some of this work to specialists, but have found that most of full-service engagements include so we recently added WinCross to our toolbag. We are also making the capability available for our clients who subcontract to 5 Circles Research.

WinCross is a desktop package from The Analytical Group (http://www.analyticalgroup.com/index.html) offering easy import from SPSS or other data formats. Output is available in Excel format, or as an RTF file for those who like a printed document (like me). With the printed output you can get up to about 25 columns in a single set (usually enough, but sometimes two sets are needed), with statistical testing across multiple combinations of columns. Excel output can handle up to 255 columns. There are all sorts of features for changing the analysis base, subtotals and more, all accessible from the GUI or by editing the job file to speed things up.

Conclusion

I hope I’ve convinced you of the power of statistical testing, and given you a glimpse of some of the tools available. Contact us if you are interested in having us produce crosstabs for your data.

Idiosyncratically,
Mike Pritchard

• 1 – Not at all Satisfied
• 2 – Not at all Satisfied
• 3 – Not at all Satisfied
• 4 – Not at all Satisfied
• 5 – Not at all Satisfied
• 6 – Not at all Satisfied
• 7 – Somewhat Satisfied
• 8 – Somewhat Satisfied
• 9 – Highly Satisfied
• 10 – Highly Satisfied

What’s this all about?  As a survey taker I’m confused.  The question has a 10 point scale, but why does every numeric point have text (anchors). What’s the difference between 1, 2, 3, 4, 5 and 6 that all have the same anchoring text?   Don’t they care about the difference between 3 and 5?  Oh, I get it, this is really a 3 point scale disguised as a 10 point scale.

With these and other variations on the theme of “what were the survey authors thinking?”  on my mind I talked to a representative from the sponsoring company, AOTMP.  I was told that the question design was well-thought out and appropriate, being modeled on the well-known Net Promoter Score.   Well of course it is  – like an apple is based on an orange (both grow on trees).  But not really:

• The Net Promoter question is for Recommendation, not Satisfaction.  There were a couple of other similar questions in the short survey, but nothing about Recommendation. Frederick Reichheld’s contention is that recommendation is the important measure and also incorporates satisfaction; you won’t recommend unless you are satisfied.
• The NPS question uses descriptive text only at the end points (Extremely Unlikely to Recommend and Extremely Likely to Recommend).  It is part of the methodology to avoid text anywhere in the middle in order to give the survey taker the maximum flexibility.  That’s consistent with survey best practices.
• The original NPS scale is from 0 to 10, not 1 to 10.  Maybe that’s a small point, although the 0 to 10 scale does allow for a midpoint which was part of the the NPS philosophy.

Other than the fact that this survey question isn’t NPS, what’s the big deal?  Well, this pseudo 10 point scale really doesn’t work.  The survey taker is likely to be confused about whether there is any difference between “3, Not at all Satisfied” and “4, Not at all Satisfied”. Perhaps the intention was to make it easier for survey takers, but either they’ll take more time worrying about the meaning, or just give an unthinking answer, and the survey administrator has no way of knowing.  Why not just use the 3 point scale instead?  I suppose you could, but then it would be even less like NPS. Personally, I like the longer scale for NPS.  I don’t use NPS on its own very much, but the ability to combine with other satisfaction measures with longer scales (Overall Satisfaction and Likelihood to Reuse) means that I’ve got the option of doing more powerful analysis as well as the simple NPS.  More importantly, I don’t have to try to persuade a client to stop using NPS as long as I include other questions using the same scale.  Ideally, I’d prefer to use a 7 or 5 point scale instead, but 10 or 11 points works fine – as long as only the end-points are anchored. For more on combining Net Promoter with other questions for more powerful analysis, check out “Profiting from customer satisfaction and loyalty research”

There’s no justification for this type of scale in my opinion.  If you disagree, please make a comment or send me a note.   If you want to use a scale with every point textually anchored, use the Likert scale with every point identified (but no numbers). Including both numbers and too many anchors will make the survey takers scratch their heads – not the goal for a good survey.

Perhaps the people who created this survey had read economist J.K. Galbraith’s comment without realizing it was sarcastic.- “It is a far, far better thing to have a firm anchor in nonsense than to put out on the troubled seas of thought.”

Idiosyncratically,
Mike Pritchard

Many thanks to Greg Weber of Priorities Research for clarifying the practice and the philosophy of the Net Promoter Score.

If you are going to fudge the numbers, you’d better be very clever.

Last December’s Annual Year in Ideas issue of the New York Times magazine included an idea titled “Forensic Polling Analysis” describing how Nate Silver analyzed results published by a polling firm called Strategic Vision. Silver decided to take a look at the results because Strategic Vision had been censured by the American Association for Public Opinion Research for not revealing details about how polls were conducted. After looking at 100 sets of poll results for the previous 4 years Nate concluded that the distribution of the last digit wasn’t random as it should have been. In addition to examining Strategic Vision’s numbers, he analyzed results from Quinnipiac (a well-respected pollster according to the New York Times), and found the last digit distribution conformed to what might be expected from chance. Silver’s conclusions were confirmed by a retired University of Illinois physics professor, Michael Weissman, who used Fourier analysis to come up with the chance of 1 in 5,000 of Strategic Vision’s results being produce by a legitimate poll.

Silver’s article [http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html] describes in more detail his use of Benford’s law to perform the analysis, and how it is used for forensic accounting (i.e. fraud detection). Oddly, although the Wikipedia article [http://en.wikipedia.org/wiki/Benford%27s_law] leads to discussions of forensic accounting and predictive analytics for fraud detection, Benford’s law is concerned with first digit, not the last. Still, the real point is that it is very difficult to manually generate random numbers. In fact, people often don’t recognize randomness (read Leonard Mlodinow’s entertaining “The Drunkard’s Walk” for more on the subject).

Apparently, Strategic Vision still hasn’t revealed how their polls are conducted, but they did threaten to sue Nate Silver. Hopefully I won’t be a target as a mere reporter of others work.

Mystery shopping should be a mystery.

For accurate results when testing service quality, it is important that the transaction is normal, receiving no special treatment. We’ve probably all been in situations where we wonder if that’s really the case. If our experience at a restaurant was so bad, why does it get good reviews? Or why does the car dealer have a five-star rating when everyone we know hates them? Recently in Britain, the postal watchdog Postcomm is considering action against the Royal Mail following allegations that lists of customers involved in a test were circulated so that the deliveries to these customers could be ensured of being on time. Staff were also able to recognize and prioritize the test letters. Apparently the cheating has been going on for nearly 4 years, with thousands of people involved. Ironically, the published results haven’t been improved as a result. More details are at http://www.telegraph.co.uk/news/uknews/royal-mail/7431654/Cheats-at-Royal-Mail-fix-delivery-times.html

Encouraging the customer to lie is bad for everyone

Over the past months I’ve had a couple of deliveries from Sears. In both cases, one of the delivery team told me that I’d be getting a phone call to ask how the delivery went. Fair enough. But then they held up a card showing me the ‘5’s that they wanted me to give them. Obviously, I was offended. The request was presumptuous, whether or not they told me that the scores were important to their performance reviews (both did). Involving customers in creating inaccurate results doesn’t improve performance or customer satisfaction. In fact, the sour taste keeps the bad parts of both experiences in my mind much longer. The refrigerator was supposed to be leveled but the delivery team didn’t bring a level. And the people delivering the lawn tractor didn’t check that their oversized truck could be driven up the driveway and didn’t think to ask if I had gas available so the tractor could be driven to our house. By contrast, another appliance delivery from a small local company was handled completely and competently, with no reference to a follow up survey.

Don’t get me wrong – I’m a fan of Sears. I’ve had good experience with their brands and generally find the sales people helpful. But this approach to customer satisfaction doesn’t help them improve. Apparently Sears thinks managing by fear is appropriate, or they aren’t directing employees properly. That’s too bad.

Idiosyncratically,

Mike Pritchard

But I’m getting ahead of myself.  Let’s start with a couple of definitions:

A correlation is simply the test of the relationship between two variables.  Pearson’s coefficient, commonly used to test linear relationships between scale variables, will be 1 (or -1) for perfect correlation.  Other coefficients are used for different types of variables. Tools such as SPSS that calculate correlation coefficients generally provide some guidance as to whether the relationship is significant – the strength of the correlation.

What correlation tells you is given the value of the one variable, what to expect for the value of another variable.

Causality, on the other hand, is a statement that if the value of one variable is changed then the value of the second variable will change accordingly.  Correlation is necessary, but not sufficient, for a cause-and-effect relationship.

It is easy to find good examples of correlations where assuming a causal relationship would be absurd. The Wikipedia article on the topic shows a chart of Mexican lemons imported from Mexico to the US plotted against total US highway fatalities. This is an example of a coincidental correlation.

Another type of misinterpretation occurs when the order of the cause and effect is reversed. Daniel Huff’s excellent “How To Lie With Statistics” discusses the relationship between smoking and college grades.  Apparently the results were used to promote the idea that giving up smoking would lead to improved grades.  But it is equally feasible that lower grades caused students to take up smoking.

We can get into trouble by using more sophisticated statistical techniques without paying enough attention to the meaning of the data and the variables being used to express results. Regression analysis is a powerful tool, but look at the correlations first.  Even the jargon can encourage misinterpretation and misstatements; when you are performing analysis for the ‘dependent’ variable it is easy to conclude causality where none exists.

More subtle problems can occur when some other factor is the cause for both the correlated variables.   This article describes a study where eating breakfast was correlated with elementary school success.  This could have resulted in the conclusion that breakfast eating caused them to be better learners. The article continues, “It turns out, however, that those who don’t eat breakfast are also more likely to be absent or tardy — and it is absenteeism that is playing a significant role in their poor performance. When researchers retested the breakfast theory, they found that, independent of other factors, breakfast only helps undernourished children perform better.” The article is from the Statistical Assessment Service – STATS – which is a non-partisan resource whose mission is to provide education on the use and abuse of science and statistics in the media.

I can’t be sure which of the fallacies were behind the ill-considered statements that were the inspiration for this article without access to the raw data.  The Kauffman Foundation does some excellent work studying entrepreneurship.  But their report on “The Use of Credit Card Debt by New Firms” draws some conclusions that are not justified by the data shown. The report states that “credit card debt reduces a firm’s probability of survival” (emphasis mine).  It appears that the authors want to warn entrepreneurs to avoid using credit cards. All the more surprising then that two positive examples for credit card funding (Spike Lee and the Blair Witch Project movie) are named in the report. I don’t want to be hypercritical of Kaffman or the report, as there are some interesting and useful results presented.  But from the data shown it seems equally likely that the businesses that failed were going to fail anyway, regardless of taking on credit debt.  In fact, businesses that failed during the three years of the study actually had lower credit card debt at the end of the first year.  Perhaps they did not borrow aggressively enough!

How then do you avoid drawing the wrong conclusions about cause-and-effect?  And how can you deliver results from research that provide useful guidance for actions that forward the organizational goals?

First, avoid making statements that imply the correlations imply causality.  Consider the other possibilities such as reverse causality or another variable that wasn’t measured.  However, don’t be too pedantic or academic either.  It is often fair to say that there may be a cause-and-effect relationship.  And frequently the changes that will positively impact one variable will be beneficial to the organization as long as they make sense on the face of it.

If you really need to confirm causality, you’ll generally need to do some sort of study that is repeated over time.  By including the same people in the sample, you’ll have good assurance that changes you see in Overall Satisfaction can be connected with the changes you make from one wave to the next – such as for Speed of Connecting to a Customer Service Representative.  If you don’t use the same people, you’ll have to take more care to make sure the samples are the same as far as possible.

For more examples that will help you critically review your own and others’ work, check out this great list of correlation/causality fallacies.

And finally, I couldn’t resist this cartoon on the topic from XKCD:

Idiosyncratically,

Mike Pritchard

http://www.asa.org.uk/asa/adjudications/Public/TF_ADJ_46477.htm

Generally, the best approach is to build up the intensity, starting from less important questions and then moving to the critical questions as quickly as possible, building up the survey taker’s engagement as you go.  Then reduce the intensity with clarifying questions and demographics.  That way, if someone bails out early, you’ll still have the most important information (assuming that your survey tool and/or your sample company allow you to look at partial surveys).

There are exceptions of course, and one comes from the use of online panels, particularly when you set up quotas and pay for completed surveys.  In this case, one or more demographic questions, used for screening, will be placed very early. 

Or sometimes the topic of the survey dictates the order, as with awareness studies where unaided awareness is usually one of the first questions.  You might also order the questions based on the survey logic. 

If you need to include a response from an earlier question in a later question (piping), or if the answer to one question will determine which other questions are asked (skip logic), this may impose a question order. 

For complex surveys, there are likely to be tradeoffs that are best decided by careful review of the questionnaire (as a document) before starting programming.  This is why questionnaire writing is a combination of experience and science with a little bit of guesswork thrown in for good measure.

One example of how a softer start helped was a survey for an organization considering new services.  The original questionnaire launched straight into the questions for the new services after a brief introduction.  Responses trickled in slowly.  When a question about membership in the organization was moved up to the beginning, the response rates jumped and we were able to complete the survey on time.

If you show respect for your survey takers, they’ll appreciate it and they’ll reward you by completing the entire survey.  Good luck!
Mike

First, a refresher. Van Westendorp’s Price Sensitivity Meter is one of a number of direct techniques to research pricing. Direct techniques assume that people have some understanding of what a product or service is worth, and therefore that it makes sense to ask explicitly about price. By contrast, indirect techniques, typically using conjoint or discrete choice analysis, combine the price with other attributes, ask questions about the total package, and then extract feelings about price from the results.

I prefer direct pricing techniques in most situations for several reasons:

• I believe people can usually give realistic answers about price.
• Indirect techniques are generally more expensive because of setup and analysis.
• It is harder to explain the results of conjoint or discrete choice to managers or other stakeholders.
• Direct techniques can be incorporated into qualitative studies in addition to their usual use in a survey.

Remember that all pricing research makes the assumption that people understand enough about the landscape to make valid comments. If someone doesn’t really have any idea about what they might be buying, the response won’t mean much regardless of whether the question is direct or the price is buried. Lack of knowledge presents challenges for radically new products. This aspect is one reason why pricing research should be treated as providing an input into pricing decisions, not a complete or absolute answer.

Other than Van Westendorp, the main direct pricing research methods are these:

• Direct open-ended questioning (“How much would you pay for this”).  This is generally a bad way to ask, but you might get away with it at the end of a in-depth (qualitative) interview.
• Monadic (“Would you be willing to buy at $10”). This method has some merits, including being able to create a demand curve with a large enough sample and multiple price points. But there are some problems, chief being the difficulty of choosing price points, particularly when the prospective purchaser’s view of value is wildly different from the vendor’s. Running a pilot might help, but you run the risk of having to throw away results from the pilot. But if you include open-ended questions for comments, and people tell you the suggested price is ridiculous, at least you’ll know why nobody wants to buy at the price you set in the pilot. Monadic questioning is pretty simple, but it is generally easy to do better without much extra work.
• Laddering (“would you buy at $10”, then “would you buy at $8” or “$12″). Don’t even think about using this approach, as the results won’t tell you anything. The respondent will treat the series of questions as a negotiation rather than research. If you wanted to ask
  about different configurations the problem is even worse.
• Van Westendorp’s Price Sensitivity Meter uses open-ended questions combining price and quality. Since there is an inherent assumption that price is a reflection of value or quality, the technique is not useful for a true luxury good (that is, when sales volume increases at higher prices). Peter Van Westendorp introduced the Price Sensitivity Meter in 1976 and it has been widely used since then throughout the market research industry.

How to set up and analyze using Van Westendorp questions

The actual text typically varies with the product or service being tested, but usually the questions are worded like this:

• At what price would you begin to think product is too expensive to consider?
• At what price would you begin to think product is so inexpensive that you would question the quality and not consider it?
• At what price would you begin to think product is getting expensive, but you still might consider it?
• At what price would you think product is a bargain – a great buy for the money

There is debate over the order of questions, so you should probably just choose the order that feels right to you.

The questions can be asked in-person, by telephone, on paper or (most frequently these days) online questionnaire. In the absence of a human administrator who can assure comprehension and valid results, online or paper surveys require well-written instructions. You may want to emphasize that the questions are different and highlight the differences. Some researchers use validation to force the respondent to create the expected relationships between the various values, but if done incorrectly this can backfire (see my earlier post). If you can’t validate in real-time (some survey tools won’t support the necessary programming), then you’ll need to clean the data (eliminate inconsistent responses) before analyzing. Whether you validate or not, remember that the questions use open-ended numeric responses. Don’t make the mistake of imposing your view of the world by offering ranges.

Excel formulae make it easy to do the checking, but to simplify things for an eyeball check, make sure the questions are ordered in your spreadsheet as you would expect prices to be ranked, that is Too Cheap, Bargain, Getting Expensive, Too Expensive.

Ensure that the values are numeric (you did set up your survey tool to store values rather than text didn’t you? – if not another Excel manipulation is needed), and then create your formula like this:

IF(AND(TooCheap<=Bargain,Bargain<=GettingExpensive, GettingExpensive<=TooExpensive), OK, FAIL)

You should end up with something like this extract:

Perhaps respondent 3 didn’t understand the wording of the questions, or perhaps (s)he didn’t want to give a useful response.  Either way, the results can’t be used.  If the survey had used validation, the problem would have been avoided, but we would also have run the risk of annoying someone and causing them to terminate, potentially losing other useful data.  Not an easy call.

Now you need to analyze the valid data.  Van Westendorp results are displayed graphically for analysis, using plots of cumulative percentages. I use Excel’s Histogram tool to generate the values for the plots. You’ll need to set up the buckets,so it might be worth rank ordering the responses to get a good idea of the right approach.  Or you might have an idea of price increments that make sense.

Create your own buckets, otherwise the Excel Histogram tool will make its own from the data, but they won’t be helpful.

Just to make the process even more complicated, you will need to plot inverse cumulative distributions (1 minus the number from the Histogram tool) for two of the questions – Too Cheap and Bargain.  Warning: if you search online you may find that plots vary, particularly in which questions are flipped. What I’m telling you here is my approach which seems to be the most common, and is also consistent with the Wikipedia article, but the final cross check is the vocalizing test, which we’ll get to shortly.

Van Westendorp Example

Before we get to interpretation, let’s apply the vocalization test.  Read some of the results from the plots to see if everything makes sense intuitively.

“At $200, about 25% think the product is a bargain, and at $50, about 55% think it is a bargain.”

“40% think it is too cheap at about $35, but at $300 only 5% think it is too cheap.”

“At $400, about 70% think it is too expensive, while 75% think it is getting expensive.”  (Remember these are cumulative – the 75% includes the 70%).”

Interpretation

Much has been written on interpreting the different intersections and the relationships between intersections of Van Westendorp plots. Personally, I think the most useful result is the Range of Acceptable Prices.   The lower bound is the intersection of Too Cheap and Expensive (sometimes called the point of marginal cheapness).  The upper bound is the intersection of Too Expensive and Cheap (the point of marginal expensiveness).  In the chart above, this range is from $50 to $100.  As you can see, there is a very significant perception shift either side of the $50 and $100 price points.

Some people think there is so-called optimal price (the intersection of Too Expensive and Too Cheap) is useful, but I think there is a danger of trying to create static perfection in a dynamic world, especially since pricing research is generally only one input to a pricing decision. For more on the overall discipline of pricing, Thomas Nagle’s book is a great source.

Going beyond Van Westendorp’s original questions

As originally proposed, the Van Westendorp questions provide no information about willingness to purchase, and thus nothing about expected revenue or margin.

To provide more insight into demand and profit, we can add one or two more questions.

The simple approach is to add a single question along the following lines:

At a price between the price you identified as ‘a bargain’ and the price you said was ‘getting expensive’, how likely would you be to purchase?

With a single question, we’d generally use a Likert scale response (Very unlikely, Unlikely, Unsure, Likely, Very Likely) and apply a model to generate an expected purchase likelihood at each point. The model will probably vary by product and situation, but let’s say 60% of Very Likely + 25% of Likely as a starting point. It is generally better to be conservative and assume that fewer will actually buy than tell you they will, but there is no harm in using what-ifs to plan in case of a runaway success, especially if there is a manufacturing impact.

A more comprehensive approach is to ask separate questions for the ‘bargain’ and ‘getting expensive’ prices, in this case using percentage responses.  The resulting data can be turned into demand/revenue curves, again based on modeled assumptions or what-ifs for the specific situation.

Conclusion

Van Westendorp pricing questions offer a simple, yet powerful way to incorporate price perceptions into pricing decisions.  In addition to their use in large scale surveys described here, I’ve used these questions for in-depth interviews and focus groups (individual responses followed by group discussion).

Idiosyncratically,
Mike Pritchard

References

Wikipedia article: http://en.wikipedia.org/wiki/Van_Westendorp’s_Price_Sensitivity_Meter

The Strategy and Tactics of Pricing, Thomas Nagle,

Van-Westendorp PH,(1976), NSS Price Sensitivity Meter – a new approach to the study of consumer perception of price. Proceedings of the 29th Congress, Venice ESOMAR

Digging in further, I was struck by a number of points:

• The article I first noticed, by Karen Kaplan of the Los Angeles Times, was very clearly written and didn’t mangle the facts or interpretations. Such clarity is unusual and deserves a commendation. Read the article for the details – I don’t need to repeat.
• The scale of the study was unusually large compared with many medical studies, including some that draw dubious conclusions from a very small data set. The research team (from England, France, Sweden and the U.S.) matched 300 cancer patients with 571 healthy controls who had similar demographics. These groups are only a small fraction of the entire database of nearly 50,000 people in Golestan province whose tea drinking habits have been studied, so we can expect future refinement and expansion of results.
• The original article in the BMJ (formerly the British Medical Journal), BMJ 2009;338:b929, is a well-written source document, complete with properly explained tables and a video.

This is a good example of a well researched and reported project. The results are made available under an open access Collective Commons License, that doubtless encourages completeness.

After evaluating the details, I decided to review my own tea and coffee rituals. The study concluded that the most likely causal mechanism is the temperature, so regardless of what hot liquid you drink it might be a good to be cautious about temperature. I don’t drink anywhere near the quantity of hot liquids that the study participants imbibe daily (nearly 1.2 liters on average – that’s over 2.1 British pints or 2.5 U.S. pints), but the damage may be cumulative and I want to be a tea drinker for many more years. It seems that my latte drinks are cool enough, but I should probably wait for a few minutes after brewing to drink my tea at around 140 degrees Fahrenheit. Perhaps I’ll start to put the tea cosy on after the first cup, but I don’t think I can bring myself to stop warming the teapot. My wife is the smart one – she’s always preferred to cool down her tea with water from the tap.

Idiosyncratically,
Mike Pritchard

But even here there are shining examples; Southwest Airlines leads other airlines in satisfaction and, not coincidentally, has been profitable for 36 straight years – despite the turmoil of the economy and fuel prices.

In case you need more convincing, there are plenty of published studies showing the link between customer satisfaction and financial performance. One such paper from the Burke Institute (http://www.burke.com/Library/WhitePapers/B.WhitePaperVol5Iss3.pdf) uses the Secure Customer Index methodology that is described in the rest of this post.

Benefits from satisfied customers

Each industry is slightly different, but there are some consistent principles:

1. Satisfied customers tend to continue to buy from the same company. They are easier to market and sell to (for repeat purchases, increased usage or cross selling).
2. It costs much less to retain existing customers than to acquire new ones.
3. Satisfied customers tell others about their positive experiences, while dissatisfied customers tell even more people about their negative experience.

Why conduct customer satisfaction research

The current economic conditions make customer satisfaction even more important.  But don’t make the mistake of thinking that research can only tell you what’s happened in the past.  Sure, the report-card aspect has some value, but the real power comes from insights that help provide guidance for the future.

Research can tell you which customers are really satisfied, and why.  Remember, most of your customers are silent.  The outspoken customer is generally not typical, and satisfying the squeaky wheel may not be helpful overall, in fact, it may be counterproductive. What if you enhance your offerings to support a customer whose hot buttons aren’t similar to your good customers? What happens if layoffs force you to concentrate on fewer customers?  Learning what you should do to better support good customers is generally the best approach.

What should you measure?

The three high-level measures you should use are Overall Satisfaction, Likelihood of Future Purchase, and Likelihood to Recommend.  The wording may vary with your situation, but the concepts remain constant. These three measures allow you to measure the current situation, to predict retention (loyalty), and to balance marketing and other costs against the value of customer groups.

Can’t I just use the Net Promoter Score?

I don’t want to get into a debate about Net Promoter. You’ll find plenty of discussion on the methodology online if you are interested in the controversy.  Suffice it to say that I don’t accept the premise that the NPS is the one number you need (as Frederick Reichheld stated) for powerful customer satisfaction research. If you want to calculate the Net Promoter Score you can do that from data you already collect; the Likelihood to Recommend question is the same and the results can be used in different ways.

How do you analyze?

Combining the results from the three high-level satisfaction measures allows you to understand how each customer is classified.  You’ll need to decide how to code the responses.  A common approach for 5 point scales is to only count a score of 5 for the most satisfied etc.

Classifying customers based on satisfaction

The percentage classified as Secure is known as the Secure Customer Index.

Secure customers are those who are most satisfied overall, most likely to repurchase, and most likely to recommend (scoring top on all three questions).  These are the most valuable customers overall – because they buy the most, are the best advocates, and generally cost less to service.  They probably won’t need expensive changes to remain classified as secure, but it is important that the company continues to provide appropriate support to keep them in the category.  For the example study in the article, this group was 88% likely to remain a customer after 1 year, and 33% were likely to increase purchases.  These are only example results, and the effects of the current crisis will depress these numbers, but the difference between secure and other customers is likely to remain.

Satisfied customers are generally well satisfied, scoring top or second for all the three questions.  Improvements are often directed at this category because they are the easiest to move to the secure category where they become even more valuable.  In the example study, this group was 57% likely to remain a customer, and 20% likely to increase purchasing.

The Indifferent group is those who have middle of the road scores on all measures.  This group is not usually as important to target as others, in part because the impact of changes is not as assured.  Over time, the percentage of customers in this group should be minimized.

The Vulnerable group is comprised of those customers who score low on any of the three satisfaction measures.  It is often tempting to focus energy on making changes that improve perceptions by this group, but this may not pay off.  Rather, learning the causes of the dissatisfaction will help the company to avoid seeking more customers who may also be dissatisfied for the same reasons when there are no immediate fixes.  For example a customer who is driven by low prices is probably not a desirable customer for a company seeking to differentiate through added services.  Better targeting should minimize the size of this group.

Taking action

Once the customers are classified, you can profile them to understand what makes them different and take appropriate action.

• Are the secure customers less likely to be using a product that has some problems?  Perhaps it has some bugs, or perhaps competitors have a better solution.  It would be a good idea to address those issues before marketing the flawed product heavily, or you might risk losing the goodwill of your best customers.
• Are some customers less satisfied because they have run into customer service or support issues?  Maybe those lower satisfaction levels can be traced to specific customer-facing personnel who need training.
• Can you identify combinations of products or services that are used by more satisfied customers? Cross-selling these combinations will likely yield good results, not only for immediate revenue, but also to increase loyalty.

Extending the research

The above analysis can be done with the three satisfaction questions combined with demographics and other profile questions.

To take the research to another level you can include detailed questions about customers’ perceptions of importance and performance of specific attributes and features. Analysis of these importance/performance questions is useful standalone, but can also be combined with the higher level satisfaction questions and the classification to provide deeper insights.

• Which groups of customers value specific features, and which desire improvements?  If your dissatisfied customers are the only ones who are complaining about a particular issue, perhaps fixing it will cost too much. It is generally better to focus on enhancements that are appreciated by customers who are already favorable, although you should pay attention to competition too.
• What do customers really think is important versus what they tell you?  Customers may tell you that they want a lower price, but is that really going to pay off?  Even today, most people aren’t buying purely on price.  Think of your own purchasing behavior and motivations.  Would you switch to a different chiropractor or car mechanic just because someone else is offering lower prices?  Conversely, will you buy more ice-cream because the price is lower (maybe, in my case).

The point is that you shouldn’t just lower prices reactively. Sophisticated analysis may be needed to tease out all the information in your data, but you can learn quite a lot from a simple importance-performance matrix.

Importance and Performance

Now is a great time to get started with customer satisfaction research.

Idiosyncratically,

Mike Pritchard

Secure Customer Index classification