About the poll: Telephone interviews were conducted by live, professional interviewers with 405 voters selected at random from registered voters in Washington state June 9-13. Margin of sampling error is ±5% at the 95% level of confidence.

That’s a solid statement. But what struck me was that the commentary, based on the chart I’m reproducing here, might seem inconsistent with the reliability statement above.

The accompanying text reads “More Washingtonians claim allegiance to Democrats than to Republicans, but independents are tilting more towards the GOP.” How can this be, when the difference is only 4% (6% more Democrats, 10% more Republicans)? The answer lies in how statistical testing works and the fact that statistical tests take into account the differences arising from different event probabilities.

First, let’s dissect the reliability statement. It means that results from this survey will be within ±5% of the true population, registered voters in this case, 19 out of 20 times if samples of this size were drawn from the registered voter list and surveyed. (One time in 20 the results could be outside of that ±5% range; that’s the result of sampling.) This ±5% range is actually the worst case and is only this high at for 50% event probabilities – meaning the situation where responses are likely to be equally split. Researchers use the worst case figure to ensure that they sample enough people for the desired reliability whatever the results are. In this case, the range for Independents leaning towards Democrats is ±2.3% (i.e. 3.7% to 8.3%) while the range for Independents leaning towards the GOP is ±2.9% (i.e. 7.9% to 12.9%). But these ranges overlap so how can the statement about tilting more to the Republicans be made with confidence?

We need to run statistical tests to apply more rigor to the reporting. In this case t-tests or z-tests will show the answer we need. The t-test is perhaps more commonly used because if works with smaller sample sizes, although we have a large enough sample here for either. Applying a t-test to the 6% and 10% results we find that the t-score is 2.02 which is greater than the 1.96 needed for 95% confidence. The differences in proportions are NOT likely due to random chance, and the statement is correct.

To illustrate the impact of event probability on statistical testing, this diagram shows how smaller differences in proportions are more able to discriminate differences as the event probability gets further away from the midpoint. Note that even at 6% difference results between about 20% and 70% (for the lower proportion) won’t generate a statistically significant difference, while at 8% difference the event probability doesn’t matter. Actually, 7% is sufficient – just.

Without using statistical testing, you won’t be sure that the survey results you see for small differences really mean that the groups in the surveyed population differ. How can you prioritize your efforts for feature A versus feature B if you don’t know what’s really important? Do your prospects differ in how they find information or make decisions to buy? You can create more solid insights and recommendations if you test.

Tools for statistical testing

The diagram above shows how things work, and is a rule of thumb for one type of testing. But it is generally best to use one or more tools to do significance testing.
Online survey tools don’t generally offer significance testing. The vendors tell me that users can get into trouble, and they don’t want to provide support. So you are need to find your own solutions. If you are doing analysis in Excel you can use t-tests and z-tests that are included in the Data Analysis Toolpak. But these only work on the individual results so if you are trying to look at aggregate proportions (as might be needed when using secondary research as I did above) you need a different tool. Online calculators are available from a number of websites, or you might want to download a spreadsheet tool (or build your own from the formulae). These tools are great for a quick check for a few data points without having to enter a full data set.

SPSS has plenty of tests available, so if you are planning on doing more sophisticated analysis yourself, or if you have a resource you use for advanced analysis then you’ll have the capability available. But SPSS, besides being expensive, isn’t all that efficient for large numbers of tests. I use SPSS for regressions, cluster analysis and the like, but I prefer having a set of crosstabs to be able to quickly spot differences between groups in the target population. We still outsource some of this work to specialists, but have found that most of full-service engagements include so we recently added WinCross to our toolbag. We are also making the capability available for our clients who subcontract to 5 Circles Research.

WinCross is a desktop package from The Analytical Group (http://www.analyticalgroup.com/index.html) offering easy import from SPSS or other data formats. Output is available in Excel format, or as an RTF file for those who like a printed document (like me). With the printed output you can get up to about 25 columns in a single set (usually enough, but sometimes two sets are needed), with statistical testing across multiple combinations of columns. Excel output can handle up to 255 columns. There are all sorts of features for changing the analysis base, subtotals and more, all accessible from the GUI or by editing the job file to speed things up.

Conclusion

I hope I’ve convinced you of the power of statistical testing, and given you a glimpse of some of the tools available. Contact us if you are interested in having us produce crosstabs for your data.

Idiosyncratically,
Mike Pritchard

I don’t have any problems with the basic findings of the study.  But equating real-world with real-life leads to an incorrect conclusion that recommendations online are inherently less effective than those made offline. 

What’s important is the strength of the connection, regardless of how the connection is made.  I have friends and associates online that I have never met, but whose opinions I trust.  Maybe I don’t know them well enough to invite them to dinner, or to ask them to dogsit, but I know enough about them to value their suggestions about a piece of software, an LCD projector, or a training class.   The same is true of consumer purchases.  I’ll treat an anonymous review of a refrigerator with caution; perhaps it will highlight some feature that I didn’t think of, or give me a reason to double check some aspect.  But if I know more about the person and their use of the product I have more confidence in the review’s relevance for me.  And if is from someone that I’ve become acquainted with, I will be even more likely to pay attention to the recommendation.

Don’t get me wrong.  I’m not suggesting that online connections are a replacement for real-world friendships.  But there is room for both, and either one may be valuable for recommendations.  So companies needn’t despair that their efforts in social media won’t pay off.  Perhaps the net effect of suggestions online is lower, but I suspect that for with strong connections the results will be a lot closer than the 5:1 ratio you might assume from the Mintel report.

Idiosyncratically.
Mike Pritchard

http://www.mintel.com/press-release/Mintel-finds-people-still-prefer-reallife-recommendations-to-online?id=358

It isn’t totally clear how closely the new program is associated with the Hyatt’s Gold Passport loyalty program.  The Times article states that recipients don’t have to be members, but Mark Hoplamazian (Hyatt C.E.O) writes in a guest blog post for USA Today that the “random acts of generosity” program is being run by the Gold Passport team.

It is certainly clear that current loyalty programs are generally poor performers in terms of creating grateful customers whose relationship extends much beyond treating the loyalty card as a discount program. And I buy into the notion of gratitude as a powerful motivator.  But I’m not so sure that Hyatt’s plan will be able to walk the tightrope necessary to achieve their objectives.

• The idea of randomness is troubling to me, in part because I wonder how well it will be applied in practice.  Will a customer receiving a free massage see the gift in a positive light, or be suspicious?  Will someone else who doesn’t receive a “random act of generosity” perceive unfairness?  In a planned paper on gratitude, the  importance of elements of randomness or discretion is mentioned.  Perhaps the giveaways will become merely discretionary, used as ways to appease an unhappy customer, or be perceived as such. 
• I’m also thinking of the random aspects of B.F. Skinner’s operant conditioning.  Is this what’s intended – to generate a feeling among customers that they should return because they might be the recipient of benefit next time (much like the dog who doesn’t know when they’ll receive a treat for good behavior).  If that’s the case, perhaps it would be better to be upfront with a truly randomized system.  That approach worked well for a funky burger joint in Portland, Oregon, where the possibility of a free meal was part of the schtick, but it could backfire for the Hyatt if customers simply see it as a different way to apply discounts (and perhaps would prefer lower prices). 
• Hyatt is in a bind on how to publicize the program.  On the one hand, if they promote the new program actively, they might be seen as doing this for very self-serving purposes.  Of course, that’s their intent, but they don’t want it to be obvious.  On the other hand, will word-of-mouth pay off quickly enough, or be accurate?
• Perhaps a simpler approach would be instead to emphasize the aspects of service that don’t have as direct an impact on the consumer’s wallet.  The Times article mentions Zappos’ ability to generate gratitude by helping shoppers find a product that Zappos doesn’t have in stock.  Some of my most positive experiences of hotels, and the ones I’ll use for recommendations, are for places that go above and beyond to provide suggestions for local services, or advice for a future stay.  Perhaps Hyatt thinks that tactic has run its course?

For more information on research into the role of gratitude in relationship marketing, look for  “The Role of Customer Gratitude in Relationship Marketing“, by Robert W. Palmatier, Cheryl Burke Jarvis, Jennifer R. Bechkoff, & Frank R. Kardes. This will be published in September 2009 issue of the Journal of Marketing, but the draft is available now.

Hyatt’s goal should be to be seen as a chain that offers a better experience for all customers, not just the lucky few. Will the “random acts of generosity” program hit the mark?  It remains to be seen.

Idiosyncratically,
Mike Pritchard

The Pew Internet and American Life study on online classifieds contains no big surprises.  Use of online classified ads sites has risen dramatically since 2005 while the revenue from newspaper classified ad plummeted as Craigslist became a force.

Craigslist has over 75% of the unique visitors for online classifieds in the US, and that was before the extra promotion (notoriety) that came from the change of policy around their adult services section. 

A more comprehensive look at the newspaper industry can be found in the PriceWaterhouseCoopers report, “Outlook for Newspaper Publishing in the Digital Age“.  I’m not going to attempt to summarize the study here, but just to comment on a couple of points.  You can read the press release here, and the full 50 page report is here.  PWC surveyed 4,900 people worldwide, identifying a number of different perspectives (not just the stereotype of younger news consumers preferring online).  Yes, preference for paper increases with age, but news consumers still prefer paper over online in every age bracket.  The study includes findings for why reading news on mobile devices or electronic paper (Amazon’s Kindle and the like) is not attractive to many consumers.  In the case of electronic paper, the major reason is lack of familiarity, whereas dislike of mobile devices also stems from difficulty of reading, and a desire for more in-depth information. PWC put together a nice summary of different consumer perspectives around the world in this video.

I hope that PWC’s generally upbeat perspective on the future for the industry is driven by more than the fact that the report was co-sponsored by the World Association of Newspapers.  The report acknowledges the expectation of casualties among well-known papers, but offers hope for newspapers to take advantage of the opportunities afforded by a digital world.  Perhaps PWC is right – I hope so.  But the newspaper industry doesn’t have a very good history of responding well to technology.

What of the New York Times?  I can’t blame the Daily Show for taking potshots at my favorite paper.  But I hope that the announcement in today’s magazine that it was being trimmed in size isn’t the forerunner of what’s been going on with the Seattle Times that feels like death by a thousand (OK, less than 10) cuts. Newsprint costs have increased, for sure, in part driven by transportation costs. But newsprint suppliers have also been cutting production, at a time when worldwide demand seems to be increasing (democracy anyone?). How ironic if the suppliers are cutting their own throats.

Bloggers, Online Ads, the Great Recession.  Big threats bring big opportunities.  Will the existing newspaper industry evolve to meet the challenges?

Idiosyncratically,
Mike Pritchard

Read the full report to draw your own conclusions (you can sign up here) but here are a few impressions:

Predictably, Q1 2009 saw the end of strong growth seen over the past several years.   I think the results are positive enough to be heartening for continuing success of online retail, although some of the growth probably comes at the expense of offline, as people are increasing the use of online to seek lower prices.

Online retail spending may have bottomed out, but it is unclear when it will start to grow again.  The current overall flatness is a result of a combination of factors for different groups. Lower income households (under $50K) show reduced spending over the same period last year, while higher incomes show some growth.  There is also distinction between age groups, with those under 44 increasing online spending and older consumers holding off.  Looks like younger people are less concerned because of longer time horizons or generally don’t want to defer spending any longer, while the older brackets are saving to rebuild their retirement assets instead of purchasing.  Depending on your perspective on the role of consumer purchasing in the U.S. economy and levels of saving, this is either a good thing or scary for the speed of the recovery.

Online prices, lower at the turn of the year through February, have now increased as inventories have been worked off, and promotional activity reduced to match.

Presumably reflecting the increased significance of comparison shopping and other money saving tools, the Internet has become more important to buying decisions than a year ago.  Three-quarters of consumers do online research before buying offline (I don’t know if this is an increase). And more people are using coupons than ever before, including from online sources.  No surprise, the role of the Internet as an integral part of shopping – both online and offline – is confirmed during tough economic times.  Regardless of whether the sale is completed offline, retailers must pay attention to providing useful information (not just discounts and sales, but also product information).   I was reminded of this recently buying a refrigerator from Sears. Maybe not the best use of time, but it was more efficient to do some preliminary research online, then discuss benefits with a sales person in the store.  In this case, we made the purchase in the store, then changed our minds after looking more thoroughly at home (and had to run the gamut of the Sears phone system to make the change – but that’s another story).  Next time, it will probably be better to do the final check after talking to the sales person with a laptop or smartphone.

comScore’s figures for incremental offline sales from search or display advertising (16% display only, 82% search only) might have you agreeing with the idea that search advertising is much more effective, but comScore points out that the reach is typically much higher for display, therefore the dollar lift may be higher for display.  In addition, the synergy for combined search and display (119% increase) is clear.  Cost effectiveness will vary with situation.

Enjoy the full report!

Idiosyncratically,
Mike Pritchard