About the poll: Telephone interviews were conducted by live, professional interviewers with 405 voters selected at random from registered voters in Washington state June 9-13. Margin of sampling error is ±5% at the 95% level of confidence.

That’s a solid statement. But what struck me was that the commentary, based on the chart I’m reproducing here, might seem inconsistent with the reliability statement above.

The accompanying text reads “More Washingtonians claim allegiance to Democrats than to Republicans, but independents are tilting more towards the GOP.” How can this be, when the difference is only 4% (6% more Democrats, 10% more Republicans)? The answer lies in how statistical testing works and the fact that statistical tests take into account the differences arising from different event probabilities.

First, let’s dissect the reliability statement. It means that results from this survey will be within ±5% of the true population, registered voters in this case, 19 out of 20 times if samples of this size were drawn from the registered voter list and surveyed. (One time in 20 the results could be outside of that ±5% range; that’s the result of sampling.) This ±5% range is actually the worst case and is only this high at for 50% event probabilities – meaning the situation where responses are likely to be equally split. Researchers use the worst case figure to ensure that they sample enough people for the desired reliability whatever the results are. In this case, the range for Independents leaning towards Democrats is ±2.3% (i.e. 3.7% to 8.3%) while the range for Independents leaning towards the GOP is ±2.9% (i.e. 7.9% to 12.9%). But these ranges overlap so how can the statement about tilting more to the Republicans be made with confidence?

We need to run statistical tests to apply more rigor to the reporting. In this case t-tests or z-tests will show the answer we need. The t-test is perhaps more commonly used because if works with smaller sample sizes, although we have a large enough sample here for either. Applying a t-test to the 6% and 10% results we find that the t-score is 2.02 which is greater than the 1.96 needed for 95% confidence. The differences in proportions are NOT likely due to random chance, and the statement is correct.

To illustrate the impact of event probability on statistical testing, this diagram shows how smaller differences in proportions are more able to discriminate differences as the event probability gets further away from the midpoint. Note that even at 6% difference results between about 20% and 70% (for the lower proportion) won’t generate a statistically significant difference, while at 8% difference the event probability doesn’t matter. Actually, 7% is sufficient – just.

Without using statistical testing, you won’t be sure that the survey results you see for small differences really mean that the groups in the surveyed population differ. How can you prioritize your efforts for feature A versus feature B if you don’t know what’s really important? Do your prospects differ in how they find information or make decisions to buy? You can create more solid insights and recommendations if you test.

Tools for statistical testing

The diagram above shows how things work, and is a rule of thumb for one type of testing. But it is generally best to use one or more tools to do significance testing.
Online survey tools don’t generally offer significance testing. The vendors tell me that users can get into trouble, and they don’t want to provide support. So you are need to find your own solutions. If you are doing analysis in Excel you can use t-tests and z-tests that are included in the Data Analysis Toolpak. But these only work on the individual results so if you are trying to look at aggregate proportions (as might be needed when using secondary research as I did above) you need a different tool. Online calculators are available from a number of websites, or you might want to download a spreadsheet tool (or build your own from the formulae). These tools are great for a quick check for a few data points without having to enter a full data set.

SPSS has plenty of tests available, so if you are planning on doing more sophisticated analysis yourself, or if you have a resource you use for advanced analysis then you’ll have the capability available. But SPSS, besides being expensive, isn’t all that efficient for large numbers of tests. I use SPSS for regressions, cluster analysis and the like, but I prefer having a set of crosstabs to be able to quickly spot differences between groups in the target population. We still outsource some of this work to specialists, but have found that most of full-service engagements include so we recently added WinCross to our toolbag. We are also making the capability available for our clients who subcontract to 5 Circles Research.

WinCross is a desktop package from The Analytical Group (http://www.analyticalgroup.com/index.html) offering easy import from SPSS or other data formats. Output is available in Excel format, or as an RTF file for those who like a printed document (like me). With the printed output you can get up to about 25 columns in a single set (usually enough, but sometimes two sets are needed), with statistical testing across multiple combinations of columns. Excel output can handle up to 255 columns. There are all sorts of features for changing the analysis base, subtotals and more, all accessible from the GUI or by editing the job file to speed things up.

Conclusion

I hope I’ve convinced you of the power of statistical testing, and given you a glimpse of some of the tools available. Contact us if you are interested in having us produce crosstabs for your data.

Idiosyncratically,
Mike Pritchard

• 1 – Not at all Satisfied
• 2 – Not at all Satisfied
• 3 – Not at all Satisfied
• 4 – Not at all Satisfied
• 5 – Not at all Satisfied
• 6 – Not at all Satisfied
• 7 – Somewhat Satisfied
• 8 – Somewhat Satisfied
• 9 – Highly Satisfied
• 10 – Highly Satisfied

What’s this all about?  As a survey taker I’m confused.  The question has a 10 point scale, but why does every numeric point have text (anchors). What’s the difference between 1, 2, 3, 4, 5 and 6 that all have the same anchoring text?   Don’t they care about the difference between 3 and 5?  Oh, I get it, this is really a 3 point scale disguised as a 10 point scale.

With these and other variations on the theme of “what were the survey authors thinking?”  on my mind I talked to a representative from the sponsoring company, AOTMP.  I was told that the question design was well-thought out and appropriate, being modeled on the well-known Net Promoter Score.   Well of course it is  – like an apple is based on an orange (both grow on trees).  But not really:

• The Net Promoter question is for Recommendation, not Satisfaction.  There were a couple of other similar questions in the short survey, but nothing about Recommendation. Frederick Reichheld’s contention is that recommendation is the important measure and also incorporates satisfaction; you won’t recommend unless you are satisfied.
• The NPS question uses descriptive text only at the end points (Extremely Unlikely to Recommend and Extremely Likely to Recommend).  It is part of the methodology to avoid text anywhere in the middle in order to give the survey taker the maximum flexibility.  That’s consistent with survey best practices.
• The original NPS scale is from 0 to 10, not 1 to 10.  Maybe that’s a small point, although the 0 to 10 scale does allow for a midpoint which was part of the the NPS philosophy.

Other than the fact that this survey question isn’t NPS, what’s the big deal?  Well, this pseudo 10 point scale really doesn’t work.  The survey taker is likely to be confused about whether there is any difference between “3, Not at all Satisfied” and “4, Not at all Satisfied”. Perhaps the intention was to make it easier for survey takers, but either they’ll take more time worrying about the meaning, or just give an unthinking answer, and the survey administrator has no way of knowing.  Why not just use the 3 point scale instead?  I suppose you could, but then it would be even less like NPS. Personally, I like the longer scale for NPS.  I don’t use NPS on its own very much, but the ability to combine with other satisfaction measures with longer scales (Overall Satisfaction and Likelihood to Reuse) means that I’ve got the option of doing more powerful analysis as well as the simple NPS.  More importantly, I don’t have to try to persuade a client to stop using NPS as long as I include other questions using the same scale.  Ideally, I’d prefer to use a 7 or 5 point scale instead, but 10 or 11 points works fine – as long as only the end-points are anchored. For more on combining Net Promoter with other questions for more powerful analysis, check out “Profiting from customer satisfaction and loyalty research”

There’s no justification for this type of scale in my opinion.  If you disagree, please make a comment or send me a note.   If you want to use a scale with every point textually anchored, use the Likert scale with every point identified (but no numbers). Including both numbers and too many anchors will make the survey takers scratch their heads – not the goal for a good survey.

Perhaps the people who created this survey had read economist J.K. Galbraith’s comment without realizing it was sarcastic.- “It is a far, far better thing to have a firm anchor in nonsense than to put out on the troubled seas of thought.”

Idiosyncratically,
Mike Pritchard

Many thanks to Greg Weber of Priorities Research for clarifying the practice and the philosophy of the Net Promoter Score.

If you are going to fudge the numbers, you’d better be very clever.

Last December’s Annual Year in Ideas issue of the New York Times magazine included an idea titled “Forensic Polling Analysis” describing how Nate Silver analyzed results published by a polling firm called Strategic Vision. Silver decided to take a look at the results because Strategic Vision had been censured by the American Association for Public Opinion Research for not revealing details about how polls were conducted. After looking at 100 sets of poll results for the previous 4 years Nate concluded that the distribution of the last digit wasn’t random as it should have been. In addition to examining Strategic Vision’s numbers, he analyzed results from Quinnipiac (a well-respected pollster according to the New York Times), and found the last digit distribution conformed to what might be expected from chance. Silver’s conclusions were confirmed by a retired University of Illinois physics professor, Michael Weissman, who used Fourier analysis to come up with the chance of 1 in 5,000 of Strategic Vision’s results being produce by a legitimate poll.

Silver’s article [http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html] describes in more detail his use of Benford’s law to perform the analysis, and how it is used for forensic accounting (i.e. fraud detection). Oddly, although the Wikipedia article [http://en.wikipedia.org/wiki/Benford%27s_law] leads to discussions of forensic accounting and predictive analytics for fraud detection, Benford’s law is concerned with first digit, not the last. Still, the real point is that it is very difficult to manually generate random numbers. In fact, people often don’t recognize randomness (read Leonard Mlodinow’s entertaining “The Drunkard’s Walk” for more on the subject).

Apparently, Strategic Vision still hasn’t revealed how their polls are conducted, but they did threaten to sue Nate Silver. Hopefully I won’t be a target as a mere reporter of others work.

Mystery shopping should be a mystery.

For accurate results when testing service quality, it is important that the transaction is normal, receiving no special treatment. We’ve probably all been in situations where we wonder if that’s really the case. If our experience at a restaurant was so bad, why does it get good reviews? Or why does the car dealer have a five-star rating when everyone we know hates them? Recently in Britain, the postal watchdog Postcomm is considering action against the Royal Mail following allegations that lists of customers involved in a test were circulated so that the deliveries to these customers could be ensured of being on time. Staff were also able to recognize and prioritize the test letters. Apparently the cheating has been going on for nearly 4 years, with thousands of people involved. Ironically, the published results haven’t been improved as a result. More details are at http://www.telegraph.co.uk/news/uknews/royal-mail/7431654/Cheats-at-Royal-Mail-fix-delivery-times.html

Encouraging the customer to lie is bad for everyone

Over the past months I’ve had a couple of deliveries from Sears. In both cases, one of the delivery team told me that I’d be getting a phone call to ask how the delivery went. Fair enough. But then they held up a card showing me the ‘5’s that they wanted me to give them. Obviously, I was offended. The request was presumptuous, whether or not they told me that the scores were important to their performance reviews (both did). Involving customers in creating inaccurate results doesn’t improve performance or customer satisfaction. In fact, the sour taste keeps the bad parts of both experiences in my mind much longer. The refrigerator was supposed to be leveled but the delivery team didn’t bring a level. And the people delivering the lawn tractor didn’t check that their oversized truck could be driven up the driveway and didn’t think to ask if I had gas available so the tractor could be driven to our house. By contrast, another appliance delivery from a small local company was handled completely and competently, with no reference to a follow up survey.

Don’t get me wrong – I’m a fan of Sears. I’ve had good experience with their brands and generally find the sales people helpful. But this approach to customer satisfaction doesn’t help them improve. Apparently Sears thinks managing by fear is appropriate, or they aren’t directing employees properly. That’s too bad.

Idiosyncratically,

Mike Pritchard

But I’m getting ahead of myself.  Let’s start with a couple of definitions:

A correlation is simply the test of the relationship between two variables.  Pearson’s coefficient, commonly used to test linear relationships between scale variables, will be 1 (or -1) for perfect correlation.  Other coefficients are used for different types of variables. Tools such as SPSS that calculate correlation coefficients generally provide some guidance as to whether the relationship is significant – the strength of the correlation.

What correlation tells you is given the value of the one variable, what to expect for the value of another variable.

Causality, on the other hand, is a statement that if the value of one variable is changed then the value of the second variable will change accordingly.  Correlation is necessary, but not sufficient, for a cause-and-effect relationship.

It is easy to find good examples of correlations where assuming a causal relationship would be absurd. The Wikipedia article on the topic shows a chart of Mexican lemons imported from Mexico to the US plotted against total US highway fatalities. This is an example of a coincidental correlation.

Another type of misinterpretation occurs when the order of the cause and effect is reversed. Daniel Huff’s excellent “How To Lie With Statistics” discusses the relationship between smoking and college grades.  Apparently the results were used to promote the idea that giving up smoking would lead to improved grades.  But it is equally feasible that lower grades caused students to take up smoking.

We can get into trouble by using more sophisticated statistical techniques without paying enough attention to the meaning of the data and the variables being used to express results. Regression analysis is a powerful tool, but look at the correlations first.  Even the jargon can encourage misinterpretation and misstatements; when you are performing analysis for the ‘dependent’ variable it is easy to conclude causality where none exists.

More subtle problems can occur when some other factor is the cause for both the correlated variables.   This article describes a study where eating breakfast was correlated with elementary school success.  This could have resulted in the conclusion that breakfast eating caused them to be better learners. The article continues, “It turns out, however, that those who don’t eat breakfast are also more likely to be absent or tardy — and it is absenteeism that is playing a significant role in their poor performance. When researchers retested the breakfast theory, they found that, independent of other factors, breakfast only helps undernourished children perform better.” The article is from the Statistical Assessment Service – STATS – which is a non-partisan resource whose mission is to provide education on the use and abuse of science and statistics in the media.

I can’t be sure which of the fallacies were behind the ill-considered statements that were the inspiration for this article without access to the raw data.  The Kauffman Foundation does some excellent work studying entrepreneurship.  But their report on “The Use of Credit Card Debt by New Firms” draws some conclusions that are not justified by the data shown. The report states that “credit card debt reduces a firm’s probability of survival” (emphasis mine).  It appears that the authors want to warn entrepreneurs to avoid using credit cards. All the more surprising then that two positive examples for credit card funding (Spike Lee and the Blair Witch Project movie) are named in the report. I don’t want to be hypercritical of Kaffman or the report, as there are some interesting and useful results presented.  But from the data shown it seems equally likely that the businesses that failed were going to fail anyway, regardless of taking on credit debt.  In fact, businesses that failed during the three years of the study actually had lower credit card debt at the end of the first year.  Perhaps they did not borrow aggressively enough!

How then do you avoid drawing the wrong conclusions about cause-and-effect?  And how can you deliver results from research that provide useful guidance for actions that forward the organizational goals?

First, avoid making statements that imply the correlations imply causality.  Consider the other possibilities such as reverse causality or another variable that wasn’t measured.  However, don’t be too pedantic or academic either.  It is often fair to say that there may be a cause-and-effect relationship.  And frequently the changes that will positively impact one variable will be beneficial to the organization as long as they make sense on the face of it.

If you really need to confirm causality, you’ll generally need to do some sort of study that is repeated over time.  By including the same people in the sample, you’ll have good assurance that changes you see in Overall Satisfaction can be connected with the changes you make from one wave to the next – such as for Speed of Connecting to a Customer Service Representative.  If you don’t use the same people, you’ll have to take more care to make sure the samples are the same as far as possible.

For more examples that will help you critically review your own and others’ work, check out this great list of correlation/causality fallacies.

And finally, I couldn’t resist this cartoon on the topic from XKCD:

Idiosyncratically,

Mike Pritchard

I don’t have any problems with the basic findings of the study.  But equating real-world with real-life leads to an incorrect conclusion that recommendations online are inherently less effective than those made offline. 

What’s important is the strength of the connection, regardless of how the connection is made.  I have friends and associates online that I have never met, but whose opinions I trust.  Maybe I don’t know them well enough to invite them to dinner, or to ask them to dogsit, but I know enough about them to value their suggestions about a piece of software, an LCD projector, or a training class.   The same is true of consumer purchases.  I’ll treat an anonymous review of a refrigerator with caution; perhaps it will highlight some feature that I didn’t think of, or give me a reason to double check some aspect.  But if I know more about the person and their use of the product I have more confidence in the review’s relevance for me.  And if is from someone that I’ve become acquainted with, I will be even more likely to pay attention to the recommendation.

Don’t get me wrong.  I’m not suggesting that online connections are a replacement for real-world friendships.  But there is room for both, and either one may be valuable for recommendations.  So companies needn’t despair that their efforts in social media won’t pay off.  Perhaps the net effect of suggestions online is lower, but I suspect that for with strong connections the results will be a lot closer than the 5:1 ratio you might assume from the Mintel report.

Idiosyncratically.
Mike Pritchard

http://www.mintel.com/press-release/Mintel-finds-people-still-prefer-reallife-recommendations-to-online?id=358

In this half-day interactive training class, attendees will learn:

• How to increase the number of people who open and respond to your survey
• How to reduce the number of people abandoning your survey
• How to ask the right questions for your business issues and get useful answers

If you are already running your own online surveys you know that there are lots of reasons to survey, from exploring new markets to gaining better understanding of customers.  But you’ve probably also experienced frustrations that can be avoided with professional insight.

• Are you considering your own surveys and don’t want to waste time?
• Do you want to create effective surveys (and avoid looking like those companies that annoy you as a survey taker)?

Online survey tools aren’t just for marketing; they can also be used to qualify prospects, to gather feedback from employees, or to evaluate classes. Online survey software is inexpensive and easy to use.  BUT it takes more than a good tool to create a good survey that will deliver useful and accurate results. 

This class is dedicated to the needs of do-it-yourself surveyors. If you are already running your own surveys you’ll find plenty of information to help you improve your results. Or, if you are a beginner, this seminar will get you started on the right path.  Don’t wade through masses of online information (frequently contradictory), read textbooks with heavy statistical jargon, or attend an expensive class for full-time research professionals.  Instead, learn much faster in an interactive class along with others who are interested in learning the best practices for do-it-yourself surveys.  The seminar is filled with practical tips, with just enough theory so you can understand why the recommendations are made.

Mike Pritchard, your trainer, has been involved in online research since it began. With experience spanning marketing, engineering, and research, for organizations large and small, Mike can help attendees improve all kinds of surveys. Great workshop! You know this field cold and it’s refreshing to see someone focused on research for entrepreneurs. ( Maria Ross, Owner, Red Slice)

You’ll learn how to run your own survey from start to finish, if that’s your preference. Or if you’d rather get help in some areas (for example questionnaire design or complex analysis), you will learn how to work with an outside consultant cost effectively.

Topics covered

Who should attend?

The seminar is designed to meet the needs of those whose responsibilities include surveys and research, but who aren’t part of a dedicated research group. This includes marketing professionals, human resources, sales managers, and consultants. Small business owners can also learn how to do online surveys for all aspects of the business.

Sign me up!

The next seminars will be held in Redmond on July 28 and Seattle on July 30.

For just $129, you’ll receive a half day of training that will save many hours and frustrations.
Sign up by July 17 for early bird pricing of $110.

Hope to see you there!

Idiosyncratically,
Mike Pritchard

Some question types and situations are not appropriate for random ordering.  For example:

• Where the option order is inherent, such as education level or a word based rating question (Likert scale)
• Where there is an ‘Other’ or ‘Other – please specify’ option.  It is often a good idea to offer an ‘Other’ option for a list of responses such as performance measures in case the survey taker believes that the list provided isn’t complete, but the ‘Other’ should be the last entry.
• A very long list, such as a list of stores, where a random order is likely to confuse or annoy the survey taker.

As with other aspects of questionnaire development, think about whether randomization will be best for the questions you include.

Idiosyncratically,
Mike Pritchard

It isn’t totally clear how closely the new program is associated with the Hyatt’s Gold Passport loyalty program.  The Times article states that recipients don’t have to be members, but Mark Hoplamazian (Hyatt C.E.O) writes in a guest blog post for USA Today that the “random acts of generosity” program is being run by the Gold Passport team.

It is certainly clear that current loyalty programs are generally poor performers in terms of creating grateful customers whose relationship extends much beyond treating the loyalty card as a discount program. And I buy into the notion of gratitude as a powerful motivator.  But I’m not so sure that Hyatt’s plan will be able to walk the tightrope necessary to achieve their objectives.

• The idea of randomness is troubling to me, in part because I wonder how well it will be applied in practice.  Will a customer receiving a free massage see the gift in a positive light, or be suspicious?  Will someone else who doesn’t receive a “random act of generosity” perceive unfairness?  In a planned paper on gratitude, the  importance of elements of randomness or discretion is mentioned.  Perhaps the giveaways will become merely discretionary, used as ways to appease an unhappy customer, or be perceived as such. 
• I’m also thinking of the random aspects of B.F. Skinner’s operant conditioning.  Is this what’s intended – to generate a feeling among customers that they should return because they might be the recipient of benefit next time (much like the dog who doesn’t know when they’ll receive a treat for good behavior).  If that’s the case, perhaps it would be better to be upfront with a truly randomized system.  That approach worked well for a funky burger joint in Portland, Oregon, where the possibility of a free meal was part of the schtick, but it could backfire for the Hyatt if customers simply see it as a different way to apply discounts (and perhaps would prefer lower prices). 
• Hyatt is in a bind on how to publicize the program.  On the one hand, if they promote the new program actively, they might be seen as doing this for very self-serving purposes.  Of course, that’s their intent, but they don’t want it to be obvious.  On the other hand, will word-of-mouth pay off quickly enough, or be accurate?
• Perhaps a simpler approach would be instead to emphasize the aspects of service that don’t have as direct an impact on the consumer’s wallet.  The Times article mentions Zappos’ ability to generate gratitude by helping shoppers find a product that Zappos doesn’t have in stock.  Some of my most positive experiences of hotels, and the ones I’ll use for recommendations, are for places that go above and beyond to provide suggestions for local services, or advice for a future stay.  Perhaps Hyatt thinks that tactic has run its course?

For more information on research into the role of gratitude in relationship marketing, look for  “The Role of Customer Gratitude in Relationship Marketing“, by Robert W. Palmatier, Cheryl Burke Jarvis, Jennifer R. Bechkoff, & Frank R. Kardes. This will be published in September 2009 issue of the Journal of Marketing, but the draft is available now.

Hyatt’s goal should be to be seen as a chain that offers a better experience for all customers, not just the lucky few. Will the “random acts of generosity” program hit the mark?  It remains to be seen.

Idiosyncratically,
Mike Pritchard