About the poll: Telephone interviews were conducted by live, professional interviewers with 405 voters selected at random from registered voters in Washington state June 9-13. Margin of sampling error is ±5% at the 95% level of confidence.

That’s a solid statement. But what struck me was that the commentary, based on the chart I’m reproducing here, might seem inconsistent with the reliability statement above.

The accompanying text reads “More Washingtonians claim allegiance to Democrats than to Republicans, but independents are tilting more towards the GOP.” How can this be, when the difference is only 4% (6% more Democrats, 10% more Republicans)? The answer lies in how statistical testing works and the fact that statistical tests take into account the differences arising from different event probabilities.

First, let’s dissect the reliability statement. It means that results from this survey will be within ±5% of the true population, registered voters in this case, 19 out of 20 times if samples of this size were drawn from the registered voter list and surveyed. (One time in 20 the results could be outside of that ±5% range; that’s the result of sampling.) This ±5% range is actually the worst case and is only this high at for 50% event probabilities – meaning the situation where responses are likely to be equally split. Researchers use the worst case figure to ensure that they sample enough people for the desired reliability whatever the results are. In this case, the range for Independents leaning towards Democrats is ±2.3% (i.e. 3.7% to 8.3%) while the range for Independents leaning towards the GOP is ±2.9% (i.e. 7.9% to 12.9%). But these ranges overlap so how can the statement about tilting more to the Republicans be made with confidence?

We need to run statistical tests to apply more rigor to the reporting. In this case t-tests or z-tests will show the answer we need. The t-test is perhaps more commonly used because if works with smaller sample sizes, although we have a large enough sample here for either. Applying a t-test to the 6% and 10% results we find that the t-score is 2.02 which is greater than the 1.96 needed for 95% confidence. The differences in proportions are NOT likely due to random chance, and the statement is correct.

To illustrate the impact of event probability on statistical testing, this diagram shows how smaller differences in proportions are more able to discriminate differences as the event probability gets further away from the midpoint. Note that even at 6% difference results between about 20% and 70% (for the lower proportion) won’t generate a statistically significant difference, while at 8% difference the event probability doesn’t matter. Actually, 7% is sufficient – just.

Without using statistical testing, you won’t be sure that the survey results you see for small differences really mean that the groups in the surveyed population differ. How can you prioritize your efforts for feature A versus feature B if you don’t know what’s really important? Do your prospects differ in how they find information or make decisions to buy? You can create more solid insights and recommendations if you test.

Tools for statistical testing

The diagram above shows how things work, and is a rule of thumb for one type of testing. But it is generally best to use one or more tools to do significance testing.
Online survey tools don’t generally offer significance testing. The vendors tell me that users can get into trouble, and they don’t want to provide support. So you are need to find your own solutions. If you are doing analysis in Excel you can use t-tests and z-tests that are included in the Data Analysis Toolpak. But these only work on the individual results so if you are trying to look at aggregate proportions (as might be needed when using secondary research as I did above) you need a different tool. Online calculators are available from a number of websites, or you might want to download a spreadsheet tool (or build your own from the formulae). These tools are great for a quick check for a few data points without having to enter a full data set.

SPSS has plenty of tests available, so if you are planning on doing more sophisticated analysis yourself, or if you have a resource you use for advanced analysis then you’ll have the capability available. But SPSS, besides being expensive, isn’t all that efficient for large numbers of tests. I use SPSS for regressions, cluster analysis and the like, but I prefer having a set of crosstabs to be able to quickly spot differences between groups in the target population. We still outsource some of this work to specialists, but have found that most of full-service engagements include so we recently added WinCross to our toolbag. We are also making the capability available for our clients who subcontract to 5 Circles Research.

WinCross is a desktop package from The Analytical Group (http://www.analyticalgroup.com/index.html) offering easy import from SPSS or other data formats. Output is available in Excel format, or as an RTF file for those who like a printed document (like me). With the printed output you can get up to about 25 columns in a single set (usually enough, but sometimes two sets are needed), with statistical testing across multiple combinations of columns. Excel output can handle up to 255 columns. There are all sorts of features for changing the analysis base, subtotals and more, all accessible from the GUI or by editing the job file to speed things up.

Conclusion

I hope I’ve convinced you of the power of statistical testing, and given you a glimpse of some of the tools available. Contact us if you are interested in having us produce crosstabs for your data.

Idiosyncratically,
Mike Pritchard

If you are going to fudge the numbers, you’d better be very clever.

Last December’s Annual Year in Ideas issue of the New York Times magazine included an idea titled “Forensic Polling Analysis” describing how Nate Silver analyzed results published by a polling firm called Strategic Vision. Silver decided to take a look at the results because Strategic Vision had been censured by the American Association for Public Opinion Research for not revealing details about how polls were conducted. After looking at 100 sets of poll results for the previous 4 years Nate concluded that the distribution of the last digit wasn’t random as it should have been. In addition to examining Strategic Vision’s numbers, he analyzed results from Quinnipiac (a well-respected pollster according to the New York Times), and found the last digit distribution conformed to what might be expected from chance. Silver’s conclusions were confirmed by a retired University of Illinois physics professor, Michael Weissman, who used Fourier analysis to come up with the chance of 1 in 5,000 of Strategic Vision’s results being produce by a legitimate poll.

Silver’s article [http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html] describes in more detail his use of Benford’s law to perform the analysis, and how it is used for forensic accounting (i.e. fraud detection). Oddly, although the Wikipedia article [http://en.wikipedia.org/wiki/Benford%27s_law] leads to discussions of forensic accounting and predictive analytics for fraud detection, Benford’s law is concerned with first digit, not the last. Still, the real point is that it is very difficult to manually generate random numbers. In fact, people often don’t recognize randomness (read Leonard Mlodinow’s entertaining “The Drunkard’s Walk” for more on the subject).

Apparently, Strategic Vision still hasn’t revealed how their polls are conducted, but they did threaten to sue Nate Silver. Hopefully I won’t be a target as a mere reporter of others work.

Mystery shopping should be a mystery.

For accurate results when testing service quality, it is important that the transaction is normal, receiving no special treatment. We’ve probably all been in situations where we wonder if that’s really the case. If our experience at a restaurant was so bad, why does it get good reviews? Or why does the car dealer have a five-star rating when everyone we know hates them? Recently in Britain, the postal watchdog Postcomm is considering action against the Royal Mail following allegations that lists of customers involved in a test were circulated so that the deliveries to these customers could be ensured of being on time. Staff were also able to recognize and prioritize the test letters. Apparently the cheating has been going on for nearly 4 years, with thousands of people involved. Ironically, the published results haven’t been improved as a result. More details are at http://www.telegraph.co.uk/news/uknews/royal-mail/7431654/Cheats-at-Royal-Mail-fix-delivery-times.html

Encouraging the customer to lie is bad for everyone

Over the past months I’ve had a couple of deliveries from Sears. In both cases, one of the delivery team told me that I’d be getting a phone call to ask how the delivery went. Fair enough. But then they held up a card showing me the ‘5’s that they wanted me to give them. Obviously, I was offended. The request was presumptuous, whether or not they told me that the scores were important to their performance reviews (both did). Involving customers in creating inaccurate results doesn’t improve performance or customer satisfaction. In fact, the sour taste keeps the bad parts of both experiences in my mind much longer. The refrigerator was supposed to be leveled but the delivery team didn’t bring a level. And the people delivering the lawn tractor didn’t check that their oversized truck could be driven up the driveway and didn’t think to ask if I had gas available so the tractor could be driven to our house. By contrast, another appliance delivery from a small local company was handled completely and competently, with no reference to a follow up survey.

Don’t get me wrong – I’m a fan of Sears. I’ve had good experience with their brands and generally find the sales people helpful. But this approach to customer satisfaction doesn’t help them improve. Apparently Sears thinks managing by fear is appropriate, or they aren’t directing employees properly. That’s too bad.

Idiosyncratically,

Mike Pritchard

I don’t have any problems with the basic findings of the study.  But equating real-world with real-life leads to an incorrect conclusion that recommendations online are inherently less effective than those made offline. 

What’s important is the strength of the connection, regardless of how the connection is made.  I have friends and associates online that I have never met, but whose opinions I trust.  Maybe I don’t know them well enough to invite them to dinner, or to ask them to dogsit, but I know enough about them to value their suggestions about a piece of software, an LCD projector, or a training class.   The same is true of consumer purchases.  I’ll treat an anonymous review of a refrigerator with caution; perhaps it will highlight some feature that I didn’t think of, or give me a reason to double check some aspect.  But if I know more about the person and their use of the product I have more confidence in the review’s relevance for me.  And if is from someone that I’ve become acquainted with, I will be even more likely to pay attention to the recommendation.

Don’t get me wrong.  I’m not suggesting that online connections are a replacement for real-world friendships.  But there is room for both, and either one may be valuable for recommendations.  So companies needn’t despair that their efforts in social media won’t pay off.  Perhaps the net effect of suggestions online is lower, but I suspect that for with strong connections the results will be a lot closer than the 5:1 ratio you might assume from the Mintel report.

Idiosyncratically.
Mike Pritchard

http://www.mintel.com/press-release/Mintel-finds-people-still-prefer-reallife-recommendations-to-online?id=358

In this half-day interactive training class, attendees will learn:

• How to increase the number of people who open and respond to your survey
• How to reduce the number of people abandoning your survey
• How to ask the right questions for your business issues and get useful answers

If you are already running your own online surveys you know that there are lots of reasons to survey, from exploring new markets to gaining better understanding of customers.  But you’ve probably also experienced frustrations that can be avoided with professional insight.

• Are you considering your own surveys and don’t want to waste time?
• Do you want to create effective surveys (and avoid looking like those companies that annoy you as a survey taker)?

Online survey tools aren’t just for marketing; they can also be used to qualify prospects, to gather feedback from employees, or to evaluate classes. Online survey software is inexpensive and easy to use.  BUT it takes more than a good tool to create a good survey that will deliver useful and accurate results. 

This class is dedicated to the needs of do-it-yourself surveyors. If you are already running your own surveys you’ll find plenty of information to help you improve your results. Or, if you are a beginner, this seminar will get you started on the right path.  Don’t wade through masses of online information (frequently contradictory), read textbooks with heavy statistical jargon, or attend an expensive class for full-time research professionals.  Instead, learn much faster in an interactive class along with others who are interested in learning the best practices for do-it-yourself surveys.  The seminar is filled with practical tips, with just enough theory so you can understand why the recommendations are made.

Mike Pritchard, your trainer, has been involved in online research since it began. With experience spanning marketing, engineering, and research, for organizations large and small, Mike can help attendees improve all kinds of surveys. Great workshop! You know this field cold and it’s refreshing to see someone focused on research for entrepreneurs. ( Maria Ross, Owner, Red Slice)

You’ll learn how to run your own survey from start to finish, if that’s your preference. Or if you’d rather get help in some areas (for example questionnaire design or complex analysis), you will learn how to work with an outside consultant cost effectively.

Topics covered

Who should attend?

The seminar is designed to meet the needs of those whose responsibilities include surveys and research, but who aren’t part of a dedicated research group. This includes marketing professionals, human resources, sales managers, and consultants. Small business owners can also learn how to do online surveys for all aspects of the business.

Sign me up!

The next seminars will be held in Redmond on July 28 and Seattle on July 30.

For just $129, you’ll receive a half day of training that will save many hours and frustrations.
Sign up by July 17 for early bird pricing of $110.

Hope to see you there!

Idiosyncratically,
Mike Pritchard

http://www.5circles.com/wordpress/blog/2009/04/surveymonkey-sold-what-does-this-mean-2/mike-pritchard/ http://www.5circles.com/wordpress/blog/2009/04/surveymonkey-sold-what-does-this-mean-2/mike-pritchard/#comments Thu, 23 Apr 2009 01:05:20 +0000 Mike Pritchard http://www.5circles.com/wordpress/blog/2009/04/surveymonkey-sold-what-does-this-mean-2/mike-pritchard/ SurveyMonkey is being acquired by an investor group.  Dave Goldberg, who previously led Yahoo’s music business, will take over as CEO, but founders Ryan (current CEO) and Chris Finley will remain with the company according to the news. 

The company will be opening an office in Menlo Park, CA, where Goldberg is based.  From the current job openings, it looks like SurveyMonkey will be moving marketing and administrative functions to Menlo Park, while development and operations will remain in Portland, OR.

This acquisition is a tribute to the great work done by Ryan and Chris over the past 10 years in building the SurveyMonkey capabilities and brand. It’s also evidence of the growing importance of self-service tools for online surveys.  That’s why 5 Circles Research created SurveysAlaCarte^TM modular services and training.

It remains to be seen if the acquisition will mean increased attention on enhancing SurveyMonkey features and capabilities. Kara Swisher’s BoomTown column about the news shows that she thinks SurveyMonkey offers analytics.  I hope the comment just shows lack of understanding.  Low-end self-service tools across the board have limited is analytics and reporting.  The best that any of them offer is filtering and “cross-tabs”, but their cross tabulation capabilities are very limited.  In particular, if I want statistical significance testing I need to export the data and use other tools to do individual tests (SPSS or Excel), or run a full report using Wincross or some other product.  Either way, this process is time-consuming and expensive.  Looks like a good opportunity for SurveyMonkey or the competitors to differentiate, or for a third party to provide a decent, inexpensive, add-on.

In any case, congratulations to the SurveyMonkey founders!

Idiosyncratically,
Mike Pritchard

]]> http://www.5circles.com/wordpress/blog/2009/04/surveymonkey-sold-what-does-this-mean-2/mike-pritchard/feed/ 0