• 1 – Not at all Satisfied
• 2 – Not at all Satisfied
• 3 – Not at all Satisfied
• 4 – Not at all Satisfied
• 5 – Not at all Satisfied
• 6 – Not at all Satisfied
• 7 – Somewhat Satisfied
• 8 – Somewhat Satisfied
• 9 – Highly Satisfied
• 10 – Highly Satisfied

What’s this all about?  As a survey taker I’m confused.  The question has a 10 point scale, but why does every numeric point have text (anchors). What’s the difference between 1, 2, 3, 4, 5 and 6 that all have the same anchoring text?   Don’t they care about the difference between 3 and 5?  Oh, I get it, this is really a 3 point scale disguised as a 10 point scale.

With these and other variations on the theme of “what were the survey authors thinking?”  on my mind I talked to a representative from the sponsoring company, AOTMP.  I was told that the question design was well-thought out and appropriate, being modeled on the well-known Net Promoter Score.   Well of course it is  – like an apple is based on an orange (both grow on trees).  But not really:

• The Net Promoter question is for Recommendation, not Satisfaction.  There were a couple of other similar questions in the short survey, but nothing about Recommendation. Frederick Reichheld’s contention is that recommendation is the important measure and also incorporates satisfaction; you won’t recommend unless you are satisfied.
• The NPS question uses descriptive text only at the end points (Extremely Unlikely to Recommend and Extremely Likely to Recommend).  It is part of the methodology to avoid text anywhere in the middle in order to give the survey taker the maximum flexibility.  That’s consistent with survey best practices.
• The original NPS scale is from 0 to 10, not 1 to 10.  Maybe that’s a small point, although the 0 to 10 scale does allow for a midpoint which was part of the the NPS philosophy.

Other than the fact that this survey question isn’t NPS, what’s the big deal?  Well, this pseudo 10 point scale really doesn’t work.  The survey taker is likely to be confused about whether there is any difference between “3, Not at all Satisfied” and “4, Not at all Satisfied”. Perhaps the intention was to make it easier for survey takers, but either they’ll take more time worrying about the meaning, or just give an unthinking answer, and the survey administrator has no way of knowing.  Why not just use the 3 point scale instead?  I suppose you could, but then it would be even less like NPS. Personally, I like the longer scale for NPS.  I don’t use NPS on its own very much, but the ability to combine with other satisfaction measures with longer scales (Overall Satisfaction and Likelihood to Reuse) means that I’ve got the option of doing more powerful analysis as well as the simple NPS.  More importantly, I don’t have to try to persuade a client to stop using NPS as long as I include other questions using the same scale.  Ideally, I’d prefer to use a 7 or 5 point scale instead, but 10 or 11 points works fine – as long as only the end-points are anchored. For more on combining Net Promoter with other questions for more powerful analysis, check out “Profiting from customer satisfaction and loyalty research”

There’s no justification for this type of scale in my opinion.  If you disagree, please make a comment or send me a note.   If you want to use a scale with every point textually anchored, use the Likert scale with every point identified (but no numbers). Including both numbers and too many anchors will make the survey takers scratch their heads – not the goal for a good survey.

Perhaps the people who created this survey had read economist J.K. Galbraith’s comment without realizing it was sarcastic.- “It is a far, far better thing to have a firm anchor in nonsense than to put out on the troubled seas of thought.”

Idiosyncratically,
Mike Pritchard

Many thanks to Greg Weber of Priorities Research for clarifying the practice and the philosophy of the Net Promoter Score.

Neither approach is optimal.  They don’t look great to your primary customer — the survey taker — perhaps reducing your response rate. What’s more, you may be limiting your options for effective survey logic.

Every question on a new page

The survey taker has to check the “Next” button too many times, with each click giving an opportunity to think about quitting.  Each new page requires additional information to be downloaded from the survey host, causing extra time delay.  If the survey taker is using dialup, or your survey uses lots of unique graphics, the additional delay is likely to be noticeable, but in any case you create an unnecessary risk of looking stupid.

One reason for surveys being created like this is is a hangover from early days of online surveying when limitations were common, and as a result surveyors may think it is a best practice.  Another possibility is leaving a default set in the online survey design tool for placing each question on a new page.  But, rather than just programming without thinking, try to put yourself in the mind of the survey taker, and consider how they might react to the page breaks.

Most surveys have enough short questions that can be easily combined to reduce the page count by 20% or more.

It is generally easy to save clicks at the end of the survey, by combining demographic questions, and this is a great way of reducing fatigue and early termination.  However, try hard to make improvements at the beginning also, to minimize annoyances before the survey taker is fully engaged.  If you have several screening questions there should be opportunities to combine questions early on.

Be careful that combining pages doesn’t cause problems with survey logic.  Inexpensive survey tools often require a new page to use skip patterns.  Even if you are using a tool with the flexibility of showing or hiding questions based on responses earlier in a page this usually requires more complex programming.

Everything on one long page

People who create surveys on a single long page seem to be under the impression that they are doing the survey taker a favor, as their invitations generally promote a single page as if that means the survey is short.  Surveys programmed like this tend to look daunting, with little thought given to engaging with the survey taker.  There might be issues for low bandwidth users (although generally these surveys are text heavy with few graphics, so the page loading time shouldn’t be much of an issue).

Single page surveys rarely use any logic, even when it would be helpful.  As described above it may more difficult to use logic on a single page.  I often recommend that survey creators build a document on paper for review before starting programming, but single page surveys often look like they started with a questionnaire that could have been administered on paper (even down to “if you answered X to question 2, please answer question 3“), but that misses the benefits of surveying online.  One benefit of surveying online that isn’t always well understood is being able to pause in the middle of a survey and return to it later.  This feature is helpful when you are sending complex surveys to busy people who might be interrupted, but it only works for pages that have been previously submitted.

One of the most extreme examples of overloading questions on pages I’ve seen recently printed out as 9 sheets of paper!  It also included numerous other errors of questionnaire design, but I’ll save them for other posts.

In the case of long pages, consider splitting up the questions to keep just a few logical questions together.  For some reason, these long page surveys are usually (overly) verbose so it may be best to just use one question per page, or, more productively, reviews by other people to distill the questionnaire to the most important elements with clear and concise wording.

To finish on a positive note, one of the best online surveys I’ve seen recently was a long page survey from the Dutch Gardens company.  There were two pages of questions, one with 9 questions and the second with 6, plus a half-page of demographics.  The survey looked similar to a paper questionnaire in being quite dense, but it didn’t look overwhelming because it made effective use of layout and varied question types to keep the interest level high.  None of the questions were mandatory, refreshing in itself.  And the survey was created with SurveyMonkey — it just goes to show what a low-end tool is capable of.  This structure was possible because the survey was designed without needing logic.

I hope that you’ll get some useful ideas from this post to build surveys with page structure that helps increase the rapport with your survey takers.

Idiosyncratically,
Mike Pritchard

To understand this, think of what happens when you are asked to choose a question by a telephone interviewer.  When the list of choices are presented for a single choice question, you might be think of the first option as more of a fit, or perhaps the last option is top-of-mind.   The problem is even more acute when the person answering the survey has to comment on each of several attributes, for example when rating how well a company is doing for time taken to answer the phone, courtesy, quality of the answer, etc.   As survey creators, we don’t know exactly how the survey taker will react to the order, so the easiest way is to eliminate the potential for problems by presenting the options in a random order.  Telephone surveys with reasonable sample sizes are almost always administered with question options randomized for this reason, using CATI systems (computer assisted telephone interviewing).

When we create a survey for online delivery, a similar problem exists.  It could be argued that the survey taker can generally see all of the options so why is a random order needed?  But the fact is that we can’t predict how survey takers will react to the order of the options.  Perhaps they give more weight to the option nearest the question, or perhaps to the one at the bottom.  If they are filling out a long matrix or battery of ratings, perhaps they will change their scheme as they move down the screen.  They might be thinking something like “too many highly rated, that doesn’t seem to fit how I feel overall, so I’ll change, but I don’t want to redo the ones I already did”.    Often there could be an effect from one option being next to another that might be minimized by separating them, which randomizing will do (randomly).   The results from these options being next to each other would likely be very different:

Some question types and situations are not appropriate for random ordering.  For example:

• Where the option order is inherent, such as education level or a word based rating question (Likert scale)
• Where there is an ‘Other’ or ‘Other – please specify’ option.  It is often a good idea to offer an ‘Other’ option for a list of responses such as performance measures in case the survey taker believes that the list provided isn’t complete, but the ‘Other’ should be the last entry.
• A very long list, such as a list of stores, where a random order is likely to confuse or annoy the survey taker.

As with other aspects of questionnaire development, think about whether randomization will be best for the questions you include.

Idiosyncratically,
Mike Pritchard

Not a provider that I would think of at first, but I probably would not consider it

OK, let me give some context. This is from a survey on business internet services. The researcher wants to know what would be my likely consideration for each of several providers if I’m choosing a new one. The choices are as follows:

• The only provider I’d ever consider
• One of the providers I’d consider above others
• One of the providers I’d consider above others
• Not a provider that I would think of at first, but I might consider it
• Not a provider that I would think of at first, but I probably would not consider it
• A provider I would never consider

If I think about it, especially with the ordering they’ve offered, I guess the research company wants to know if I would be unlikely to consider it (somewhere between “might consider” and “would never consider”).  But was there an actual phrase that they were trying to come up with?  Beats me.

It’s hard to tell whether they are losing any useful data from this poor question wording, other than running the risk of respondents terminating from confusing.

I saw this issue 11% of the way through the survey, so I wondered how bad the rest would be.  Fortunately there were no other major problems.

Idiosyncratically,
Mike Pritchard

• At what price would you consider the product starting to get expensive, so that it is not out of the question, but you would still consider buying it? (Expensive/High Side)

• At what price would you consider the product to be so expensive that you would not consider buying it? (Too expensive)

• At what price would you consider the product to be priced so low that you would feel the quality couldn’t be very good? (Too cheap)

• At what price would you consider the product to be a bargain—a great buy for the money? (Cheap/Good Value)

There is some debate about the order of questions, but in this example the questions were asked in the order shown.  The wording was slightly different.  Researchers are sometimes concerned about whether the respondents understand the questions correctly, especially since the wording is so similar (the Expensive, Cheap etc. designations are usually not inclined in the question as seen by a survey taker).   One way to address this concern is to highlight the differences.  Or you might point out that the questions are slightly different and encourage the respondent to read carefully.

The other approach is to apply validation that tests the numerical relationship.   Correctly entered numbers should be Too Cheap < Good Value  < Expensive < Too Expensive. (We usually ask these questions on separate pages so as to get independent thoughts from the respondents as far as possible, rather than letting them see the group of questions as one and making them consistent or nicely spaced).

In this case, the research vendor chose to validate, but messed up big-time.  When I entered a value for ‘Too Expensive’ that was higher than the value for ‘Expensive’, I was told to make sure your answer is smaller or equal to the previous answer.  Yes, they forced me to provide an invalid response!  I hope they caught the problem before the survey had gathered all the completes, but maybe they didn’t – given how fast online surveys often fill.  They probably had to field the survey again because the pricing questions were integral to the research objectives.

Why did this happen, and how can you prevent a similar problem in your surveys?

My guess is that the underlying cause was that debate about question order that I mentioned earlier.  The vendor probably had the questions switched when the validation was tested, and then changed the order before the survey was launched.

But the real message is that proper testing could have identified the issue in time to correct a very expensive error.  There is no excuse for what happened.  This doesn’t even fall into the class of problems that the pilot or soft-launch would be needed to catch.

So, test, test, and test again.   In particular, test using people who aren’t research professionals or experienced survey takers.

If you are creating your own surveys, don’t let this kind of problem stop you.  You can do just as good a job of testing as the big companies, and big companies aren’t immune.  This survey was delivered by one of the top 10 U.S. market research firms.  I won’t publish the company name here, but I’ll probably tell you if you catch me at one of my workshops (coming soon).

Idiosyncratically,

Mike Pritchard

Please note, this survey contains media that is not compatible with Firefox Internet Browser

The invitation continued with instructions to copy and paste the link into an Internet Explorer window if Firefox is my default browser.

Let’s look at this in more detail. To dispose of the title question first, the only obvious logical reason why someone fielding online surveys wouldn’t provide support for Firefox users would be if they were surveying people who don’t use it. Perhaps even that isn’t exactly logical, but at least it’s a reasonable excuse. If you are creating something that requires significant development effort, and you are screening for Internet Explorer users, why bother with Firefox?

Unfortunately, that theory doesn’t fit the situation. I’ve seen invitations with this warning for over a year, covering Consumer Package Goods and Retail Stores. I have yet to come up with a good reason, and the research company hasn’t offered me one.

But why is it such a bad idea?

First, Sample Bias. Systematically excluding a segment of the overall population you want to survey is generally a bad practice. It is easy to gather results that are biased, for reasons that may be obvious or less so.

Remember the days of telephone surveys? (I know, we are still collecting data via the telephone, but many people are only familiar with online surveys.) Best practices include calling at random times of the day and night, and also letting the phone ring for quite a while. Why? To increase the chances of the respondent being a person who works, and also to increase the coverage of people who might be elderly or infirm – and who might take longer to reach the phone. Without these measures, you might end up with a disproportionately large number of fit stay-at-home respondents. Some corrections could be done with weighting, but this adds unnecessary complexity versus just improving the representivity of the sample in the first place.

In the case of eliminating surveys from Firefox users, it would probably be a good idea to understand the potential impact through browser share numbers. Unfortunately, this isn’t quite as easy as it might seem, which is probably why we see percentages ranging from 14% to several times higher for Firefox usage in the US. These differences are caused by data collection methodologies and also browser behavior, but this article isn’t about browser share so let’s just settle on an approximation of 20% user share for Firefox. So these surveys are systematically excluding about one fifth of the US population. I could easily come up with some imagined differences  between Firefox users and users of other browsers, but fortunately there is some real research out there. comScore reported in 2007 on a study that looked into the differences between Firefox and Internet Explorer users. The results showed that Firefox users were more likely to be younger, higher income, and male than the average Internet user. Would this impact a project covering food items in the grocery store? You bet. comScore’s study also showed that Firefox users are more likely to have a broadband connection and that their site visitation profile varied from the average – which could impact advertising placement and content.

The other impact for concern, although probably a lesser concern in this case than sample bias, is that of Lower Response Rates. Without hard evidence we can only speculate on the impact, but it seems likely that some people who receive an invitation excluding Firefox might decide not to participate even though they could do so fairly easily by starting Internet Explorer and pasting the link. The additional steps involved are a deterrent. Unfortunately, these particular surveys don’t even work by changing to Internet Explorer rendering within Firefox (something that is common practice to allow usage of sites that are not standards compliant). Longer term, continued invitations that are less easy to use may result in more people leaving the panel.

In conclusion, make your surveys sample as representative as possible, and don’t do anything in the invitation or survey to turn people off.

One last note on this subject. The problem invitations specifically state that the surveys don’t work with Firefox. Even if Firefox is the only excluded browser, it represents over 20% of the overall market as of Dec 2008 according to Net Applications.  It probably doesn’t make sense to invest in development for older browsers, but as Safari (7.9%) and Chrome (1%) usage grows the challenges for survey developers are going to increase.  Overall, browsers other than Internet Explorer are currently about one-third of total usage.

Idiosyncratically,
Mike Pritchard
5 Circles Research