Michael Phelps, 4x100m relay, Beijing 2008 Olympics		John Nunn winning his place on the US 2012 Olympic team in the 50K racewalk

The last time I commented on a Nate Silver article, he was predicting winners at the Academy Awards. Nate’s performance that time wasn’t good. He was out of his element in an event that often has upsets.

Besides returning to his roots, Nate is playing it safe by not predicting the outcome of specific events. He took the same tack in an article a couple of weeks ago, “Let’s Play MedalBall!” which gave advice to nations aspiring to achieve Olympic medals.

But let’s return to the topic of Olympic records. I liked the analysis in the article, as well as some aspects of the presentation of the results. Silver calculated percentage improvements in performances between the 1968 Olympics (Mexico City) and 2012 (London). To avoid effects of outliers, the statistical approach incorporated all Olympic performances, not just records. I don’t know if there was any correction made for the 7,300 feet altitude in Mexico City, but any effect would have been eliminated over the 40 years of the data. The calculations were based on time for the most part, but distance was used for field events like javelin, discus, and long jump.

The main conclusion of the analysis is that some types of events have exhibited overall greater performance improvements than others; these are the events where records are more likely be broken. In particular, swimming events improved by an average of 10.3% from 1968 to 2008, while track and field events improved by an average of 4.1%. In fact, in track and field performance has actually declined in a couple of events (javelin and shot-put), but as can be seen easily from the chart these are anomalies. Also notable is that the greatest improvements in track and field are for the longer events including racewalking (who knew?)

Silver offers some reasons for the differences, but I don’t know if any formal correlation analyses were done for his independent variables. He suggests that technology has benefited swimming in particular through better costumes and better pools, whereas runners haven’t had any significant tools to help them over the same period. Also, Nate writes, those from poorer nations have less access to swimming pools which means that the group of potential stars was limited as compared to athletics where little equipment is needed. It seems possible to me that these new stars are added to the pool (pun intended) through economic improvements in their own countries as well as some migration; I haven’t analyzed this – it’s just a theory.

Reporting

The article uses a long horizontal bar chart that works well in the broadsheet format of the New York Times. Silver combines male and female (distinguished by bold), and uses color to identify different types of events, arrayed in order of performance improvements.  Nice job!

But how could you convey something similar in a normal style of research report – landscape format PowerPoint, with limited room on the vertical axis?

• Turning the whole thing on its side isn’t going to work well. The length of the text for the events wouldn’t look good along the X axis, even when the text is angled. And using vertical bars might not convey the differences as well, but in any case there are still too many events for the effects to be properly communicated.
• I’d use a version of the chart as an inset, as large as possible, and then pull out subsets to show specific points. This would perhaps work even better. Events could be grouped by type and gender, perhaps separating gender within sports. The current chart makes it fairly clear that female swimming has improved more than male, but with the inclusion of some field events in the mix the point is less clear.  Three or four additional smaller charts supporting the main chart should do the trick. And you could hover over the PowerPoint to confirm anything that’s unclear to the people in the back of the room.

Actually the online version of the article uses only a clipped version of the chart as a teaser. The full chart is accessible in a separate browser window.

I hope this post has given you a few ideas about reporting a complex topic.  As for records at the 2012 Olympics, it’s too soon to know if the trends seen in the article will continue, as many of the events with the most improvement haven’t yet been held. There have already been some new records in swimming. Other records include weightlifting and archery, which weren’t covered in the article. Personally, I’d like to see a gold medal or two for my homeland, never mind a record. After the disappointment with synchronized diving, even a win in a lower profile sport might boost Britons’ morale. No predictions from me, but I’ll be keeping an eye out for trampoline and rowing (where Katherine Grainger and Anna Watkins have already broken the Olympic record).

Update August 3rd: Grainger and Watkins succeeded, and Britain is now in 3rd place for medals, behind China and the U.S. (showing the home country boost). There have been quite a few Olympic records broken in swimming, consistent with Nate Silver’s analysis. Most of the other events he analyzed are still under way.

Idiosyncratically,

Mike Pritchard

Image sources:

John Nunn Racewalking: By U.S. Army (Flickr: John Nunn wins 50K) [CC-BY-2.0], via Wikimedia Commons
Michael Phelps: By Jmex60 (Own work) [GFDL or CC-BY-SA-3.0-2.5-2.0-1.0], via Wikimedia Commons

The post Predicting Olympic Records appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

Many marketing people have been promoting the value of QR codes for quite a while. After all, the promise seems obvious – post a targeted code somewhere, make it easy for someone to reach the website, and track the results of different campaigns.

Studies such as this February 2011 survey from Baltimore based agency MGH seem to confirm the positives. 415 smartphone users from a panel were surveyed. 65% had seen a QR code, with a fairly even split between male and female. Of those who’d seen a code, 49% had used one, and 70% say they would be interested in using a QR code (including for the first time). Reasons for the interest include:

• 87% to get a coupon, discount, or a deal
• 64% to enter a sweepstake
• 63% to get additional information
• 60% to make a purchase

31% say they’d be “Very Likely” to remember an ad with a QR code, and a further 41% say they’d be “Somewhat Likely” to remember.

The published survey results don’t cover whether people actually made purchases, or did anything else once they’d visited the site (32%). But let’s look at what gets in the way of using the QR code in the first place.

The February 2012 of Quirk’s Magazine has a brief article, titled “QR Codes lost on even the savviest“, referencing work done by Archival (a youth marketing agency). The thrust is that if QR codes are to succeed, they should be adopted by college students who are smartphone users. However, although 80% had seen a QR code, and 81% owned a smartphone, only 21% successfully scanned the QR code used as part of the survey, and 75% say they are “Not Likely” to scan a QR code in future. A few more details from the study and a discussion are at http://www.archrival.com/ideas/13/qr-codes-go-to-college. I suspect the Archrival results reflect market reality more than MGH, but in any case QR codes are not living up to expectations. When was the last time you saw someone use a QR code?

Some may place the blame with marketers who don’t do as good as job as they should of communicating the benefits, and indeed having something worthwhile in the landing page. But technology is probably the most important factor. Reasons noted by the students include:

• Needing to install an app. Why isn’t something pre-installed with more phones?
• Expecting just to be able to take a picture to activate the QR code. Why shouldn’t this work?
• Takes too long. Of course, they are right.

To these reasons, I’d add that there is currently some additional confusion caused by the introduction of new types of codes. Does the world need Microsoft Tag and yet another app?

Maybe QR codes will suffer the same fate as some previous technology driven attempts to do something similar. Does anyone remember Digimarc’s MediaBridge from 2000? Did it ever seem like a good idea to scan or photograph an advertisement in a printed page to access a website? What about that RadioShack CueCat? Perhaps Digimarc has a better shot with their new Discover™ service that includes a smartphone app as well as embedded links in content. If you are already a Digimarc customer, or don’t want to sully the beauty of your images with codes – maybe it’s the answer. But that seems like a limited market compared with the potential that’s available for QR codes done right.

Come on technologists and marketers – reduce the friction in the system!

Idiosyncratically,

Mike Pritchard

The post QR codes not hitting the spot appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

[Warning: if you are a technical type interested in surveys, you might find this interesting. But if you aren't in that category, I won't be offended if you stop reading.]

Scalability

The obvious interpretation of scalability for IT applies to online surveys too. Make sure the survey tool you use is capable of handling the current and predicted usage for your online surveys.

• If you use an SaaS service, such as SurveyGizmo or QuestionPro, does your subscription level allow you to collect enough completed surveys? This isn’t likely to be an issue if you host your own surveys (perhaps with an open-source tool like Lime Survey) as you’ll have your own database.
• Do you have enough bandwidth to deliver the survey pages, including any images, audio or video that you need? Bandwidth may be more of an issue with self-hosted surveys. Bandwidth might fit more into availability, but in any case think about how your needs may change and whether that would impact your choice of tools.
• How many invitations can you send out? This applies when you use a list (perhaps a customer list or CRM database), but isn’t going to matter when you use an online panel or other invitation method. There are benefits to sending invitations through a survey service (including easy tracking for reminders), but there may be a limit on the number of invitations you can send out per month, depending on your subscription level. You can use a separate mailing service (iContact for example), and some are closely integrated with the survey tool. Perhaps the owner of the customer list wants to send out the invitations, in which case the volume is their concern but you’ll have to worry about integration. Most market researchers should be concentrating on the survey, so setting up their own mail server isn’t the right approach; leave it to the specialists to worry about blacklisting and SPF records.
• Do you have enough staff (in your company or your vendors) to build and support your surveys? That’s one reason why 5 Circles Research uses survey services for most of our work. Dedicated (in both senses) support teams make sure we can deliver on time, and we know that they’ll increase staff as needed.

Perhaps it’s a stretch, but I’d also like to mention scales for research. Should you use a 5-point, 7-point, 10-point or 11-point scale? Are the scales fully anchored (definitely disagree, somewhat disagree, neutral, somewhat agree, definitely agree)? Or do you just anchor the end points? IT professionals are numbers oriented, so this is just a reminder to consider your scales. There is plenty of literature on the topic, but few definitive answers.

Usability

Usability is a hot topic for online surveys right now. Researchers agree that making surveys clear and engaging is beneficial to gathering good data that supports high quality insights. However, there isn’t all that much agreement on some of the newer approaches. This is a huge area, so here are just a few points for consideration:

• Shorter surveys are (almost) always better. The longer a survey takes, the less likely it is to yield good results. People drop out before the end or give less thoughtful responses (lie?) just to get through the survey. The only reason for the “almost” qualifier is that sometimes survey administrators send out multiple surveys because they didn’t include some key questions originally. But the reverse is the problem in most cases. Often the survey is overloaded with extra questions that aren’t relevant to the study.
• Be respectful of the survey taker. Explain what the survey is all about, and why they are helping you. Tell them how long it will take – really! Give them context for where they are, both in the form of textual cues, and also if possible with progress bars (but watch out for confusing progress bars that don’t really reflect reality). Use survey logic and piping to simplify and shorten the survey; if someone says they aren’t using Windows, they probably shouldn’t see questions about System Restore.
• Take enough time to develop and test questions that are appropriate for the audience and the topic. This isn’t just a matter of using survey logic, but writing the questionnaire correctly in the first place. Although online survey data collection is faster than telephone, it takes longer to develop the questionnaire and test.
• Gamification of surveys is much talked about, but not usually done well. For a practical, business-oriented survey taker, questions that aren’t as straightforward may be a deterrent. On the other hand, a gaming audience may greatly appreciate a survey that appears more attuned to them. Beyond the scope of this article, some research is being conducted within games themselves.

Reliability

One aspect of reliability is uptime of the server hosting the survey tool. Perhaps more relevant to survey research are matters related to survey and questionnaire design:

• Representativeness of the sample within the target population is important for quality results, but the target depends on the purpose of the research. If you want to find out if a new version of your product will appeal to a new set of prospects, you can’t just survey customers. An online panel sample is generally regarded as representative of the market.
• How you invite people to take the survey also affects how representative the sample is. Self selection bias is a common issue; an invitation posted on the website is unlikely to work well for a general survey, but may have some value if you just need to hear from those with problems. Survey invitations via email are generally more representative, but poor writing can destroy the benefit.
• As well as who you include and how you invite them, the number of participants is important. Assuming other requirements are met, a sample of 400 yields results that are within ±5% at 95% reliability. The confidence interval (±5%) means that the results from the sample will be within that range of the true population’s results. For the numerically oriented, that’s a worst case number, true for a midpoint response; statistical testing takes this into account. The reliability number (95%) means that the results will conform 19 out of 20 times. You can play with the sample size, or accept different levels of confidence and reliability. For example, a business survey may use a sample of 200 (for cost reasons) that yields results that are within ±7% at 95% reliability.
• Another aspect of reliability comes from the questionnaire design. This is a deep and complex subject, but for now let’s just keep it high-level. Make sure that the question text reflects the object of the question, that the options are exclusive, single thoughts, exhaustive (with don’t know, none of the above, or other/specify as appropriate).

Security

Considerations for survey security are similar to those for general IT security, with a couple of extra twists.

• Is your data secure on the server? Does your provider (or do you if you are hosting your own surveys) take appropriate precautions to make sure that the data is backed up properly and is guarded against being hacked into?
• Does the connection between the survey taker and the survey tool need to be protected? Most surveys use HTTP, but SSL capabilities are available for most survey tools.
• Are you taking the appropriate measures to minimize survey fraud (ballot stuffing?) What’s needed varies with the type of survey and invitation process, but can include cookies, personalized invitations, and password protection.
• Are you handling the data properly once exported from the survey tool? You need to be concerned with overall data in the same way that the survey tool vendor does. But you also need to look after personally identifiable information (PII) if you are capturing any. You may have PII from the customer list you used for invitations, or you may be asking for this information for a sweepstake. If the survey is for research purposes, ethical standards require that this private information is not misused. ESOMAR’s policy is simple – Market researchers shall never allow personal data they collect in a market research project to be used for any purpose other than market research. This typically means eliminating these fields from the file supplied to the client. If the project has a dual purpose, and the survey taker is offered the opportunity for follow up, this fact must be made clear.

Availability

No longer being involved in engineering, I’d have to scratch my head for the distinction between availability and reliability. But as this is about IT terms as they apply to surveys, let’s just consider making surveys available to the people you want to survey.

• Be careful about question types that may work well on one platform and not another, or may not be consistently understood by the audience. For example, drag and drop ranking questions look good and have a little extra zing, but are problematic on smart phones. Do you tell the survey taker to try again from a different platform (assuming your tool detects properly), or use a simpler question type? This issue also relates to accessibility (section 508 of the Rehabilitation Act, or the British Disability Discrimination Act). Can a screen reader deal with the question types?
• Regardless of question types, it is probably important to make sure that your survey is going to look reasonable on different devices and browsers. More and more surveys are being filled out on smartphones and iPads. Take care with fancier look and feel elements that aren’t interoperable across browsers. These days you probably don’t have to worry too much about people who don’t have JavaScript available or turned on, but Flash could still be an issue. For most of the surveys we run, Flash video isn’t needed, and in any case isn’t widely supported on mobile devices. HTML5 or other alternatives are becoming more commonly used.
• Instead of accessing web surveys from any compatible mobile devices, consider other approaches to surveying. I’m not a proponent of SMS surveys; they are too limited, need multiple transactions, and may cost the survey taker money. But downloaded surveys on iPad or smartphone have their place for situations where the survey taker isn’t connected to the internet.

I hope that these pointers are meaningful for the IT professional, even with the liberties I’ve taken. There is plenty of information As you can tell, just like in the IT world there are reasons to get help from a research professional. Let me know what you think!

Idiosyncratically,

Mike Pritchard

The post IT terminology applied to surveys appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

The survey was for follow up satisfaction after some time with a new version of ACT! Call me a dinosaur, but after experiments with various online services, I still prefer a standalone CRM. Still, this post isn’t really about ACT! – I’m just giving a little background to set the stage.

• The survey title is ACT! Pro 2012 Customer Satisfaction Survey. Yet one of the questions asks the survey taker to compare ACT 2011 with previous versions. How dumb does this look?
  
  
• This same question has a text box for additional comments. The box is too small to be of much use, but also the box can’t be filled with text. All the text boxes in the survey have the the same problem.
  
  
• If you have a question that should be multiple choice, set it up correctly.
  
  
  
  Some survey tools may use radio buttons for multiple choice (not a good idea), but this isn’t one of them. This question should either be reworded along the lines of “Which of these is the most important social networking site you use“, or – probably better – use a multiple choice question type.
• Keep up to date.
  
  
  
  What happened to Quickbooks 2008, or more recent versions? It would have been better to simply have Quickbooks as an option (none of the other products had versions). If the version of Quickbooks was important (I know that integration with Quickbooks is a focus for Sage) then a follow up with the date/version would work, and would make the main question shorter.
• There were a couple of questions about importance and performance for various features. I could nitpick the importance question (more explanation about the features or an option something like “I don’t know what this is” would have been nice), but my real issue is with the performance question. 20 different features were included in both importance and performance. That’s a lot to keep in mind, so it’s good to try to make the survey taker’s life easier by keeping the order consistent between importance and performance. The problem was that the order of the performance list didn’t match the first. I thought at first that the lists were both randomized separately, instead of randomizing the first list and using the same order for the second. This is a common mistake, and sometimes the survey software doesn’t support doing it the right way. But after trying the survey again, I discovered the problem was that both lists were fixed orders, different between importance and performance. Be consistent. Note, if your scales are short enough, and if you don’t have a problem with the survey taker adjusting their responses as they think about performance and importance together (that’s a topic of debate among researchers) you might consider showing importance and performance together for each option.
• Keep up to date – really! The survey asked whether I used a mobile computing device such as a smartphone. But the next question asked about the operating system for the smartphone without including Android. Unbelievable!
  
  

There were a few other problems that I noted, but they are more related to my knowledge of the product and Sage’s stated directions. But similar issues to those above occur on a wide variety of surveys. Overall, I score this survey 5 out of 10.

These issues make me as a customer wonder about the competence of the people at Sage. A satisfaction survey is designed to learn about customers, but should also create the opportunity to make the customers feel better about the product and the company. However, if you don’t pay attention to the details you may do more harm than good.

Idiosyncratically,

Mike Pritchard

The post Survey Tip: Pay Attention to the Details appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

The question asked about the TOP activity, so people needed to prioritize. The most popular answers were “planning next year“, and “delivering to customers“, recognizing both looking forward and (presumably) the need to complete tasks. It would be interesting to see if planning is as popular at a time of the year when New Year isn’t a factor. Reviewing last year wasn’t as common as response. Perhaps people are doing continual reviews (I doubt it), or more likely they have recognized the need and the opportunity for bigger shifts and looking back isn’t as relevant.

A fellow member of BizEnrich and expert in collaborative strategy planning, Robert Nitschke of Arago Partners, tells me that many companies take until the end of Q1 to complete their strategic plan for the year. When will yours be done?

Idiosyncratically,
Mike Pritchard

The post Top holiday business activities appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

In years gone by, cell phone users were rarely included in surveys. There was uncertainty about likely reaction of potential respondents (“why are you calling me on my mobile when I have to pay for incoming calls?”, “is this legal?”). Although even early on surveyors were nervous about introducing bias through not including younger age groups, studies showed that there were only insignificant differences beyond those associated with technology. When cell phone only households were only 7% researchers tended to ignore them. Besides, surveying via cell phone cost more, due to requirements that auto-dialing techniques couldn’t be used, increased rejection rates, compensating survey takers to compensate for their costs, and also a need for additional screening to reduce the likelihood of someone taking the survey from an unsafe place. Pew Research Center’s landmark 2006 study http://www.pewinternet.org/Reports/2006/Americans-and-their-cell-phones.aspx focused on cell phone usage and related attitudes, but also showed that the Hispanic population was more likely to be cell phone only.

Over the course of the next couple of years, Pew conducted several studies (e.g. http://people-press.org/report/391/the-impact-of-cell-onlys-on-public-opinion-polling ) showing that there was little difference in political attitudes between samples using landline only and those using cell phones. At the same time, Pew pointed out that other non-political attitudes and behaviors (such as health risk behaviors) differed between the two groups. They also noted that cell phone only households had reached 14% in December 2007. Furthermore, while acknowledging the impact of cost, Pew studies also commented on the value of including cell phone sampling in order to reach certain segments of the population (low income, younger).

Time marches on. Not surprisingly give the chart above, for more and more studies, cell phone sample is now being included. With cell phone only households now estimated at upwards of 25% this increasingly makes sense. But not apparently for most political polls, despite criticism. The Economist, in an article October 7, 2010, http://www.economist.com/node/17202427 summarizes the issues well. Cost of course is one factor, but this impacts different polling firms and types differently. Pollsters relying on robocalling (O.K. IVR or Interactive Voice Response if you don’t want to associate these types of polls with assuredly partisan phone calls), are particularly affected by cost considerations. Jay Leve of SurveyUSA estimates costs would double for firms to change from automated calling to human interviewers as would be needed to call cell phones. And as the percentage of cell phone only households varies across states, predictability is even less likely. I suspect that much of this is factored into Nate Silver’s assessments on his FiveThirtyEight blog,  but he is also critical of the pollsters for introducing bias (http://fivethirtyeight.blogs.nytimes.com/2010/10/28/robopolls-significantly-more-favorable-to-republicans-than-traditional-surveys/ ). Silver holds Rasmussen up as having a Republican bias due to their methodology, and recently contrasted Rasmussen results here in Washington State with Elway (a local pollster using human interviewers) who has a Democratic bias according to FiveThirtyEight.

I’ve only scratched the surface of the discussion. We are finally seeing some pollsters incorporating cell phones into previously completely automated polls and this trend will inevitably increase as respondents are increasingly difficult to reach via landlines. Perhaps the laws will change to allow automated connections to cell phones, but I don’t see this in the near future given the recent spate of laws to deter use while driving.

But enough of politics. I’m fed up with all the calls (mostly push, only a few surveys); unfortunately my VOIP phone still counts as a landline. Still, I look forward to dissecting the impact of cell phones after the dust has settled from November 2^nd.

What’s the impact for researchers beyond the political arena?

• If your survey needs a telephone data collection sample for general population, you’d better consider including cell phone users despite the increased cost. Perhaps you can use a small sample to assess bias or representativeness, but weighting alone will leave unanswered questions without some current or recent data for comparison.
• Perhaps it’s time to use online data collection for all or part of your sample. Online (whether invitations are conducted through panels, river sampling, or social media) may be a better way to reach most of the cell phone only people. Yes, it’s true that the online population doesn’t completely mirror the overall population, but differences are decreasing and it may not matter much for your specific topic. Recent studies I’ve conducted confirm that online panelists aren’t all higher income, broadband connected, younger people. To be sure, certain groups are less likely to be online, but specialist panels can help with, for example, Hispanic people.

The one thing you can’t do is to ignore the cell phone only households.

By the way, if you are in the Seattle area, you might be interested in joining me at the next Puget Sound Research Forum luncheon on November 18, when REI will present the results of research comparing results from landline, cell phone and online panel sample for projectability.  http://pugetsoundresearchforum.org/

Good luck with your cell phone issues!

Idiosyncratically,

Mike Pritchard

The post Impact of cell phones on 2010 Midterms and beyond politics appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

About the poll: Telephone interviews were conducted by live, professional interviewers with 405 voters selected at random from registered voters in Washington state June 9-13. Margin of sampling error is ±5% at the 95% level of confidence.

That’s a solid statement. But what struck me was that the commentary, based on the chart I’m reproducing here, might seem inconsistent with the reliability statement above.

The accompanying text reads “More Washingtonians claim allegiance to Democrats than to Republicans, but independents are tilting more towards the GOP.” How can this be, when the difference is only 4% (6% more Democrats, 10% more Republicans)? The answer lies in how statistical testing works and the fact that statistical tests take into account the differences arising from different event probabilities.

First, let’s dissect the reliability statement. It means that results from this survey will be within ±5% of the true population, registered voters in this case, 19 out of 20 times if samples of this size were drawn from the registered voter list and surveyed. (One time in 20 the results could be outside of that ±5% range; that’s the result of sampling.) This ±5% range is actually the worst case and is only this high at for 50% event probabilities – meaning the situation where responses are likely to be equally split. Researchers use the worst case figure to ensure that they sample enough people for the desired reliability whatever the results are. In this case, the range for Independents leaning towards Democrats is ±2.3% (i.e. 3.7% to 8.3%) while the range for Independents leaning towards the GOP is ±2.9% (i.e. 7.9% to 12.9%). But these ranges overlap so how can the statement about tilting more to the Republicans be made with confidence?

We need to run statistical tests to apply more rigor to the reporting. In this case t-tests or z-tests will show the answer we need. The t-test is perhaps more commonly used because if works with smaller sample sizes, although we have a large enough sample here for either. Applying a t-test to the 6% and 10% results we find that the t-score is 2.02 which is greater than the 1.96 needed for 95% confidence. The differences in proportions are NOT likely due to random chance, and the statement is correct.

To illustrate the impact of event probability on statistical testing, this diagram shows how smaller differences in proportions are more able to discriminate differences as the event probability gets further away from the midpoint. Note that even at 6% difference results between about 20% and 70% (for the lower proportion) won’t generate a statistically significant difference, while at 8% difference the event probability doesn’t matter. Actually, 7% is sufficient – just.

Without using statistical testing, you won’t be sure that the survey results you see for small differences really mean that the groups in the surveyed population differ. How can you prioritize your efforts for feature A versus feature B if you don’t know what’s really important? Do your prospects differ in how they find information or make decisions to buy? You can create more solid insights and recommendations if you test.

Tools for statistical testing

The diagram above shows how things work, and is a rule of thumb for one type of testing. But it is generally best to use one or more tools to do significance testing.
Online survey tools don’t generally offer significance testing. The vendors tell me that users can get into trouble, and they don’t want to provide support. So you are need to find your own solutions. If you are doing analysis in Excel you can use t-tests and z-tests that are included in the Data Analysis Toolpak. But these only work on the individual results so if you are trying to look at aggregate proportions (as might be needed when using secondary research as I did above) you need a different tool. Online calculators are available from a number of websites, or you might want to download a spreadsheet tool (or build your own from the formulae). These tools are great for a quick check for a few data points without having to enter a full data set.

SPSS has plenty of tests available, so if you are planning on doing more sophisticated analysis yourself, or if you have a resource you use for advanced analysis then you’ll have the capability available. But SPSS, besides being expensive, isn’t all that efficient for large numbers of tests. I use SPSS for regressions, cluster analysis and the like, but I prefer having a set of crosstabs to be able to quickly spot differences between groups in the target population. We still outsource some of this work to specialists, but have found that most of full-service engagements include so we recently added WinCross to our toolbag. We are also making the capability available for our clients who subcontract to 5 Circles Research.

WinCross is a desktop package from The Analytical Group offering easy import from SPSS or other data formats. Output is available in Excel format, or as an RTF file for those who like a printed document (like me). With the printed output you can get up to about 25 columns in a single set (usually enough, but sometimes two sets are needed), with statistical testing across multiple combinations of columns. Excel output can handle up to 255 columns. There are all sorts of features for changing the analysis base, subtotals and more, all accessible from the GUI or by editing the job file to speed things up. It’s not the only package out there, but we like it, and the great support.

Conclusion

I hope I’ve convinced you of the power of statistical testing, and given you a glimpse of some of the tools available. Contact us if you are interested in having us produce crosstabs for your data.

Idiosyncratically,
Mike Pritchard

The post Why you should run statistical tests appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

• 1 – Not at all Satisfied
• 2 – Not at all Satisfied
• 3 – Not at all Satisfied
• 4 – Not at all Satisfied
• 5 – Not at all Satisfied
• 6 – Not at all Satisfied
• 7 – Somewhat Satisfied
• 8 – Somewhat Satisfied
• 9 – Highly Satisfied
• 10 – Highly Satisfied

What’s this all about?  As a survey taker I’m confused.  The question has a 10 point scale, but why does every numeric point have text (anchors). What’s the difference between 1, 2, 3, 4, 5 and 6 that all have the same anchoring text?   Don’t they care about the difference between 3 and 5?  Oh, I get it, this is really a 3 point scale disguised as a 10 point scale.

With these and other variations on the theme of “what were the survey authors thinking?”  on my mind I talked to a representative from the sponsoring company, AOTMP.  I was told that the question design was well-thought out and appropriate, being modeled on the well-known Net Promoter Score.   Well of course it is  – like an apple is based on an orange (both grow on trees).  But not really:

• The Net Promoter question is for Recommendation, not Satisfaction.  There were a couple of other similar questions in the short survey, but nothing about Recommendation. Frederick Reichheld’s contention is that recommendation is the important measure and also incorporates satisfaction; you won’t recommend unless you are satisfied.
• The NPS question uses descriptive text only at the end points (Extremely Unlikely to Recommend and Extremely Likely to Recommend).  It is part of the methodology to avoid text anywhere in the middle in order to give the survey taker the maximum flexibility.  That’s consistent with survey best practices.
• The original NPS scale is from 0 to 10, not 1 to 10.  Maybe that’s a small point, although the 0 to 10 scale does allow for a midpoint which was part of the the NPS philosophy.

Other than the fact that this survey question isn’t NPS, what’s the big deal?  Well, this pseudo 10 point scale really doesn’t work.  The survey taker is likely to be confused about whether there is any difference between “3, Not at all Satisfied” and “4, Not at all Satisfied”. Perhaps the intention was to make it easier for survey takers, but either they’ll take more time worrying about the meaning, or just give an unthinking answer, and the survey administrator has no way of knowing.  Why not just use the 3 point scale instead?  I suppose you could, but then it would be even less like NPS. Personally, I like the longer scale for NPS.  I don’t use NPS on its own very much, but the ability to combine with other satisfaction measures with longer scales (Overall Satisfaction and Likelihood to Reuse) means that I’ve got the option of doing more powerful analysis as well as the simple NPS.  More importantly, I don’t have to try to persuade a client to stop using NPS as long as I include other questions using the same scale.  Ideally, I’d prefer to use a 7 or 5 point scale instead, but 10 or 11 points works fine – as long as only the end-points are anchored. For more on combining Net Promoter with other questions for more powerful analysis, check out “Profiting from customer satisfaction and loyalty research”

There’s no justification for this type of scale in my opinion.  If you disagree, please make a comment or send me a note.   If you want to use a scale with every point textually anchored, use the Likert scale with every point identified (but no numbers). Including both numbers and too many anchors will make the survey takers scratch their heads – not the goal for a good survey.

Perhaps the people who created this survey had read economist J.K. Galbraith’s comment without realizing it was sarcastic.- “It is a far, far better thing to have a firm anchor in nonsense than to put out on the troubled seas of thought.”

Idiosyncratically,
Mike Pritchard

Many thanks to Greg Weber of Priorities Research for clarifying the practice and the philosophy of the Net Promoter Score.

The post Poor question design means questionable results: A tale of a confusing scale appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

If you are going to fudge the numbers, you’d better be very clever.

Last December’s Annual Year in Ideas issue of the New York Times magazine included an idea titled “Forensic Polling Analysis” describing how Nate Silver analyzed results published by a polling firm called Strategic Vision. Silver decided to take a look at the results because Strategic Vision had been censured by the American Association for Public Opinion Research for not revealing details about how polls were conducted. After looking at 100 sets of poll results for the previous 4 years Nate concluded that the distribution of the last digit wasn’t random as it should have been. In addition to examining Strategic Vision’s numbers, he analyzed results from Quinnipiac (a well-respected pollster according to the New York Times), and found the last digit distribution conformed to what might be expected from chance. Silver’s conclusions were confirmed by a retired University of Illinois physics professor, Michael Weissman, who used Fourier analysis to come up with the chance of 1 in 5,000 of Strategic Vision’s results being produce by a legitimate poll.

Silver’s article [http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html] describes in more detail his use of Benford’s law to perform the analysis, and how it is used for forensic accounting (i.e. fraud detection). Oddly, although the Wikipedia article [http://en.wikipedia.org/wiki/Benford%27s_law] leads to discussions of forensic accounting and predictive analytics for fraud detection, Benford’s law is concerned with first digit, not the last. Still, the real point is that it is very difficult to manually generate random numbers. In fact, people often don’t recognize randomness (read Leonard Mlodinow’s entertaining “The Drunkard’s Walk” for more on the subject).

Apparently, Strategic Vision still hasn’t revealed how their polls are conducted, but they did threaten to sue Nate Silver. Hopefully I won’t be a target as a mere reporter of others work.

Mystery shopping should be a mystery.

For accurate results when testing service quality, it is important that the transaction is normal, receiving no special treatment. We’ve probably all been in situations where we wonder if that’s really the case. If our experience at a restaurant was so bad, why does it get good reviews? Or why does the car dealer have a five-star rating when everyone we know hates them? Recently in Britain, the postal watchdog Postcomm is considering action against the Royal Mail following allegations that lists of customers involved in a test were circulated so that the deliveries to these customers could be ensured of being on time. Staff were also able to recognize and prioritize the test letters. Apparently the cheating has been going on for nearly 4 years, with thousands of people involved. Ironically, the published results haven’t been improved as a result. More details are at http://www.telegraph.co.uk/news/uknews/royal-mail/7431654/Cheats-at-Royal-Mail-fix-delivery-times.html

Encouraging the customer to lie is bad for everyone

Over the past months I’ve had a couple of deliveries from Sears. In both cases, one of the delivery team told me that I’d be getting a phone call to ask how the delivery went. Fair enough. But then they held up a card showing me the ‘5’s that they wanted me to give them. Obviously, I was offended. The request was presumptuous, whether or not they told me that the scores were important to their performance reviews (both did). Involving customers in creating inaccurate results doesn’t improve performance or customer satisfaction. In fact, the sour taste keeps the bad parts of both experiences in my mind much longer. The refrigerator was supposed to be leveled but the delivery team didn’t bring a level. And the people delivering the lawn tractor didn’t check that their oversized truck could be driven up the driveway and didn’t think to ask if I had gas available so the tractor could be driven to our house. By contrast, another appliance delivery from a small local company was handled completely and competently, with no reference to a follow up survey.

Don’t get me wrong – I’m a fan of Sears. I’ve had good experience with their brands and generally find the sales people helpful. But this approach to customer satisfaction doesn’t help them improve. Apparently Sears thinks managing by fear is appropriate, or they aren’t directing employees properly. That’s too bad.

Idiosyncratically,

Mike Pritchard

The post Researchers: remember, honesty is the best policy appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.

But I’m getting ahead of myself.  Let’s start with a couple of definitions:

A correlation is simply the test of the relationship between two variables.  Pearson’s coefficient, commonly used to test linear relationships between scale variables, will be 1 (or -1) for perfect correlation.  Other coefficients are used for different types of variables. Tools such as SPSS that calculate correlation coefficients generally provide some guidance as to whether the relationship is significant – the strength of the correlation.

What correlation tells you is given the value of the one variable, what to expect for the value of another variable.

Causality, on the other hand, is a statement that if the value of one variable is changed then the value of the second variable will change accordingly.  Correlation is necessary, but not sufficient, for a cause-and-effect relationship.

It is easy to find good examples of correlations where assuming a causal relationship would be absurd. The Wikipedia article on the topic shows a chart of Mexican lemons imported from Mexico to the US plotted against total US highway fatalities. This is an example of a coincidental correlation.

Another type of misinterpretation occurs when the order of the cause and effect is reversed. Daniel Huff’s excellent “How To Lie With Statistics” discusses the relationship between smoking and college grades.  Apparently the results were used to promote the idea that giving up smoking would lead to improved grades.  But it is equally feasible that lower grades caused students to take up smoking.

We can get into trouble by using more sophisticated statistical techniques without paying enough attention to the meaning of the data and the variables being used to express results. Regression analysis is a powerful tool, but look at the correlations first.  Even the jargon can encourage misinterpretation and misstatements; when you are performing analysis for the ‘dependent’ variable it is easy to conclude causality where none exists.

More subtle problems can occur when some other factor is the cause for both the correlated variables.   This article describes a study where eating breakfast was correlated with elementary school success.  This could have resulted in the conclusion that breakfast eating caused them to be better learners. The article continues, “It turns out, however, that those who don’t eat breakfast are also more likely to be absent or tardy — and it is absenteeism that is playing a significant role in their poor performance. When researchers retested the breakfast theory, they found that, independent of other factors, breakfast only helps undernourished children perform better.” The article is from the Statistical Assessment Service – STATS – which is a non-partisan resource whose mission is to provide education on the use and abuse of science and statistics in the media.

I can’t be sure which of the fallacies were behind the ill-considered statements that were the inspiration for this article without access to the raw data.  The Kauffman Foundation does some excellent work studying entrepreneurship.  But their report on “The Use of Credit Card Debt by New Firms” draws some conclusions that are not justified by the data shown. The report states that “credit card debt reduces a firm’s probability of survival” (emphasis mine).  It appears that the authors want to warn entrepreneurs to avoid using credit cards. All the more surprising then that two positive examples for credit card funding (Spike Lee and the Blair Witch Project movie) are named in the report. I don’t want to be hypercritical of Kaffman or the report, as there are some interesting and useful results presented.  But from the data shown it seems equally likely that the businesses that failed were going to fail anyway, regardless of taking on credit debt.  In fact, businesses that failed during the three years of the study actually had lower credit card debt at the end of the first year.  Perhaps they did not borrow aggressively enough!

How then do you avoid drawing the wrong conclusions about cause-and-effect?  And how can you deliver results from research that provide useful guidance for actions that forward the organizational goals?

First, avoid making statements that imply the correlations imply causality.  Consider the other possibilities such as reverse causality or another variable that wasn’t measured.  However, don’t be too pedantic or academic either.  It is often fair to say that there may be a cause-and-effect relationship.  And frequently the changes that will positively impact one variable will be beneficial to the organization as long as they make sense on the face of it.

If you really need to confirm causality, you’ll generally need to do some sort of study that is repeated over time.  By including the same people in the sample, you’ll have good assurance that changes you see in Overall Satisfaction can be connected with the changes you make from one wave to the next – such as for Speed of Connecting to a Customer Service Representative.  If you don’t use the same people, you’ll have to take more care to make sure the samples are the same as far as possible.

For more examples that will help you critically review your own and others’ work, check out this great list of correlation/causality fallacies.

And finally, I couldn’t resist this cartoon on the topic from XKCD:

Idiosyncratically,

Mike Pritchard

The post Correlation isn’t Causality appeared first on 5 Circles Research: full service market research, consulting and training for do-it-yourself surveys.