http://www.asa.org.uk/asa/adjudications/Public/TF_ADJ_46477.htm

To understand this, think of what happens when you are asked to choose a question by a telephone interviewer.  When the list of choices are presented for a single choice question, you might be think of the first option as more of a fit, or perhaps the last option is top-of-mind.   The problem is even more acute when the person answering the survey has to comment on each of several attributes, for example when rating how well a company is doing for time taken to answer the phone, courtesy, quality of the answer, etc.   As survey creators, we don’t know exactly how the survey taker will react to the order, so the easiest way is to eliminate the potential for problems by presenting the options in a random order.  Telephone surveys with reasonable sample sizes are almost always administered with question options randomized for this reason, using CATI systems (computer assisted telephone interviewing).

When we create a survey for online delivery, a similar problem exists.  It could be argued that the survey taker can generally see all of the options so why is a random order needed?  But the fact is that we can’t predict how survey takers will react to the order of the options.  Perhaps they give more weight to the option nearest the question, or perhaps to the one at the bottom.  If they are filling out a long matrix or battery of ratings, perhaps they will change their scheme as they move down the screen.  They might be thinking something like “too many highly rated, that doesn’t seem to fit how I feel overall, so I’ll change, but I don’t want to redo the ones I already did”.    Often there could be an effect from one option being next to another that might be minimized by separating them, which randomizing will do (randomly).   The results from these options being next to each other would likely be very different:

Some question types and situations are not appropriate for random ordering.  For example:

• Where the option order is inherent, such as education level or a word based rating question (Likert scale)
• Where there is an ‘Other’ or ‘Other – please specify’ option.  It is often a good idea to offer an ‘Other’ option for a list of responses such as performance measures in case the survey taker believes that the list provided isn’t complete, but the ‘Other’ should be the last entry.
• A very long list, such as a list of stores, where a random order is likely to confuse or annoy the survey taker.

As with other aspects of questionnaire development, think about whether randomization will be best for the questions you include.

Idiosyncratically,
Mike Pritchard

Key advantages for sample from LinkedIn as I see it:

• Profiling information is entered by the LinkedIn user for reasons unconnected with survey taking. Regardless of of how much of a problem you think lying on sample panel profiling or screening questionnaires might be,  a LinkedIn user’s description of themselves is likely to be fairly accurate – and useful to a survey researcher.  LinkedIn claims that their users inflate career history less than resumes on job seeking sites such as Monster because the information is visible to colleagues.
• This isn’t a panel. The primary reason for LinkedIn membership isn’t to take surveys.  While response rates may be lower from LinkedIn than from panels, I really care about quality.  Response rate figures are meaningless if you are talking to the wrong people, as long as there isn’t a non-response bias.  Surveys using  LinkedIn sample still have the potential for response bias, of course, but the reasons are less to do with sample than with questionnaire and invitation design.
• LinkedIn says that they will minimize the number of invitations sent to users, perhaps with a limit of no more than 1 or 2 per month.  Although I’m skeptical about the actual numbers, I accept the point that LinkedIn’s focus isn’t sample and that frequent invitations would annoy members so I am optimistic that the LinkedIn sample will continue to be lightly surveyed.

Results shared with the audience seem to bear out the truth of the LinkedIn sample promise.  A small telephone study validated the accuracy of status, title and start dates for LinkedIn members.  Results from LinkedIn sample and a B2B panel for online study of  U.S. IT decision makers (a notoriously over-surveyed group) showed some interesting differences.  In particular, the panel delivered a high percentage of completes between the hours of 3am and 7am.  Other data supported the suspicion that many of the responses were from India and China, not from the U.S.  Additionally, the panel respondents were more likely than the LinkedIn sample to complete the survey very quickly, meaning that these were probably not the target audience.  Of course, LinkedIn presented information that showed them in the best light, but it was convincing.

I’ll be looking at LinkedIn sample for B2B projects in future, both for my self-service(SurveysAlaCarte) and full-service clients.

Idiosyncratically,

Mike Pritchard

But even here there are shining examples; Southwest Airlines leads other airlines in satisfaction and, not coincidentally, has been profitable for 36 straight years – despite the turmoil of the economy and fuel prices.

In case you need more convincing, there are plenty of published studies showing the link between customer satisfaction and financial performance. One such paper from the Burke Institute (http://www.burke.com/Library/WhitePapers/B.WhitePaperVol5Iss3.pdf) uses the Secure Customer Index methodology that is described in the rest of this post.

Benefits from satisfied customers

Each industry is slightly different, but there are some consistent principles:

1. Satisfied customers tend to continue to buy from the same company. They are easier to market and sell to (for repeat purchases, increased usage or cross selling).
2. It costs much less to retain existing customers than to acquire new ones.
3. Satisfied customers tell others about their positive experiences, while dissatisfied customers tell even more people about their negative experience.

Why conduct customer satisfaction research

The current economic conditions make customer satisfaction even more important.  But don’t make the mistake of thinking that research can only tell you what’s happened in the past.  Sure, the report-card aspect has some value, but the real power comes from insights that help provide guidance for the future.

Research can tell you which customers are really satisfied, and why.  Remember, most of your customers are silent.  The outspoken customer is generally not typical, and satisfying the squeaky wheel may not be helpful overall, in fact, it may be counterproductive. What if you enhance your offerings to support a customer whose hot buttons aren’t similar to your good customers? What happens if layoffs force you to concentrate on fewer customers?  Learning what you should do to better support good customers is generally the best approach.

What should you measure?

The three high-level measures you should use are Overall Satisfaction, Likelihood of Future Purchase, and Likelihood to Recommend.  The wording may vary with your situation, but the concepts remain constant. These three measures allow you to measure the current situation, to predict retention (loyalty), and to balance marketing and other costs against the value of customer groups.

Can’t I just use the Net Promoter Score?

I don’t want to get into a debate about Net Promoter. You’ll find plenty of discussion on the methodology online if you are interested in the controversy.  Suffice it to say that I don’t accept the premise that the NPS is the one number you need (as Frederick Reichheld stated) for powerful customer satisfaction research. If you want to calculate the Net Promoter Score you can do that from data you already collect; the Likelihood to Recommend question is the same and the results can be used in different ways.

How do you analyze?

Combining the results from the three high-level satisfaction measures allows you to understand how each customer is classified.  You’ll need to decide how to code the responses.  A common approach for 5 point scales is to only count a score of 5 for the most satisfied etc.

Classifying customers based on satisfaction

The percentage classified as Secure is known as the Secure Customer Index.

Secure customers are those who are most satisfied overall, most likely to repurchase, and most likely to recommend (scoring top on all three questions).  These are the most valuable customers overall – because they buy the most, are the best advocates, and generally cost less to service.  They probably won’t need expensive changes to remain classified as secure, but it is important that the company continues to provide appropriate support to keep them in the category.  For the example study in the article, this group was 88% likely to remain a customer after 1 year, and 33% were likely to increase purchases.  These are only example results, and the effects of the current crisis will depress these numbers, but the difference between secure and other customers is likely to remain.

Satisfied customers are generally well satisfied, scoring top or second for all the three questions.  Improvements are often directed at this category because they are the easiest to move to the secure category where they become even more valuable.  In the example study, this group was 57% likely to remain a customer, and 20% likely to increase purchasing.

The Indifferent group is those who have middle of the road scores on all measures.  This group is not usually as important to target as others, in part because the impact of changes is not as assured.  Over time, the percentage of customers in this group should be minimized.

The Vulnerable group is comprised of those customers who score low on any of the three satisfaction measures.  It is often tempting to focus energy on making changes that improve perceptions by this group, but this may not pay off.  Rather, learning the causes of the dissatisfaction will help the company to avoid seeking more customers who may also be dissatisfied for the same reasons when there are no immediate fixes.  For example a customer who is driven by low prices is probably not a desirable customer for a company seeking to differentiate through added services.  Better targeting should minimize the size of this group.

Taking action

Once the customers are classified, you can profile them to understand what makes them different and take appropriate action.

• Are the secure customers less likely to be using a product that has some problems?  Perhaps it has some bugs, or perhaps competitors have a better solution.  It would be a good idea to address those issues before marketing the flawed product heavily, or you might risk losing the goodwill of your best customers.
• Are some customers less satisfied because they have run into customer service or support issues?  Maybe those lower satisfaction levels can be traced to specific customer-facing personnel who need training.
• Can you identify combinations of products or services that are used by more satisfied customers? Cross-selling these combinations will likely yield good results, not only for immediate revenue, but also to increase loyalty.

Extending the research

The above analysis can be done with the three satisfaction questions combined with demographics and other profile questions.

To take the research to another level you can include detailed questions about customers’ perceptions of importance and performance of specific attributes and features. Analysis of these importance/performance questions is useful standalone, but can also be combined with the higher level satisfaction questions and the classification to provide deeper insights.

• Which groups of customers value specific features, and which desire improvements?  If your dissatisfied customers are the only ones who are complaining about a particular issue, perhaps fixing it will cost too much. It is generally better to focus on enhancements that are appreciated by customers who are already favorable, although you should pay attention to competition too.
• What do customers really think is important versus what they tell you?  Customers may tell you that they want a lower price, but is that really going to pay off?  Even today, most people aren’t buying purely on price.  Think of your own purchasing behavior and motivations.  Would you switch to a different chiropractor or car mechanic just because someone else is offering lower prices?  Conversely, will you buy more ice-cream because the price is lower (maybe, in my case).

The point is that you shouldn’t just lower prices reactively. Sophisticated analysis may be needed to tease out all the information in your data, but you can learn quite a lot from a simple importance-performance matrix.

Importance and Performance

Now is a great time to get started with customer satisfaction research.

Idiosyncratically,

Mike Pritchard

Secure Customer Index classification

• At what price would you consider the product starting to get expensive, so that it is not out of the question, but you would still consider buying it? (Expensive/High Side)

• At what price would you consider the product to be so expensive that you would not consider buying it? (Too expensive)

• At what price would you consider the product to be priced so low that you would feel the quality couldn’t be very good? (Too cheap)

• At what price would you consider the product to be a bargain—a great buy for the money? (Cheap/Good Value)

There is some debate about the order of questions, but in this example the questions were asked in the order shown.  The wording was slightly different.  Researchers are sometimes concerned about whether the respondents understand the questions correctly, especially since the wording is so similar (the Expensive, Cheap etc. designations are usually not inclined in the question as seen by a survey taker).   One way to address this concern is to highlight the differences.  Or you might point out that the questions are slightly different and encourage the respondent to read carefully.

The other approach is to apply validation that tests the numerical relationship.   Correctly entered numbers should be Too Cheap < Good Value  < Expensive < Too Expensive. (We usually ask these questions on separate pages so as to get independent thoughts from the respondents as far as possible, rather than letting them see the group of questions as one and making them consistent or nicely spaced).

In this case, the research vendor chose to validate, but messed up big-time.  When I entered a value for ‘Too Expensive’ that was higher than the value for ‘Expensive’, I was told to make sure your answer is smaller or equal to the previous answer.  Yes, they forced me to provide an invalid response!  I hope they caught the problem before the survey had gathered all the completes, but maybe they didn’t – given how fast online surveys often fill.  They probably had to field the survey again because the pricing questions were integral to the research objectives.

Why did this happen, and how can you prevent a similar problem in your surveys?

My guess is that the underlying cause was that debate about question order that I mentioned earlier.  The vendor probably had the questions switched when the validation was tested, and then changed the order before the survey was launched.

But the real message is that proper testing could have identified the issue in time to correct a very expensive error.  There is no excuse for what happened.  This doesn’t even fall into the class of problems that the pilot or soft-launch would be needed to catch.

So, test, test, and test again.   In particular, test using people who aren’t research professionals or experienced survey takers.

If you are creating your own surveys, don’t let this kind of problem stop you.  You can do just as good a job of testing as the big companies, and big companies aren’t immune.  This survey was delivered by one of the top 10 U.S. market research firms.  I won’t publish the company name here, but I’ll probably tell you if you catch me at one of my workshops (coming soon).

Idiosyncratically,

Mike Pritchard

As you might imagine, broadband usage is unevenly distributed. People living in rural areas are less likely to have a high speed connection, as are lower income and African Americans (Hispanic broadband access is similar to the overall population). Notably, broadband adoption now increases with age, with the highest rate among those 65 and above.

All very interesting you might say, but what’s the point for me as a researcher or marketer? When I dug deeper into the report, I found some nuggets about why broadband isn’t being used that lead to implications about research and product. Here are a couple of points to ponder:

• Some people say they don’t want broadband. Of course, availability and cost are issues for some, but 19% say that nothing would convince them to get broadband. I’m sure that some of the naysayers would actually become subscribers if they were to try broadband (most marketing still focuses on speed, ignoring the benefits of an always-on connection), but there will still be some who won’t make the move of their own volition. Slowing adoption rates confirm that these people aren’t just late adopters, they are laggards, and they will probably only convert when forced by suppliers. As we know from the Technology Adoption LifeCycle model, the stages correlate to different psychographic profiles. These people are different!

• Many of those who want broadband do not have access (particularly in rural areas), or cannot afford it. This has implications for the design and implementation of market research.

• Beyond the broadband versus dial-up split, 35% of adult Americans do not have any form of Internet access at home. The most significant demographic differences shown in the Pew report summarizes are age, income, and education – truly a digital divide.

Lessons for marketers and researchers

1. You still need to consider bandwidth capabilities for online surveys. Perhaps your research topic is such that you don’t care if you deter dial-up users, but often you should be concerned about non-response bias, and in any case the things that improve surveys for lower bandwidth are good practice for all. In particular,
   • Combine pages when it makes sense. We’ve all seen surveys that have every question on a new page with no good reason. Every page load takes more time for a dial-up user. Sometimes your logic requires a new page (but be careful when choosing a tool that you aren’t forced into poor practices just because of the tool), but it is usually possible to put a few questions together with the result looking better for the responder. Demographics and related satisfaction questions are good candidates, but try especially hard to make the front of the survey look good when viewed over a slower connection. Note that the advice to combine pages doesn’t just mean put everything on a single page. Remember, you are trying to engage the respondent. Think of the survey like a conversation. A long single page online survey can be very daunting, and almost as frustrating as endless clicks.
   • Make your graphics small files. There’s nothing wrong with some graphical elements for branding or just to make the pages more interesting, but be sure to keep the files small. That great picture of the product was probably taken with a multi-megapixel camera, meaning that the file is hundreds of kilobytes. But it doesn’t need to be very high resolution, regardless of the speed of the connection; 72dpi is probably plenty.
   • Avoid gratuitous or physically large graphics, for the same reason as the previous recommendation. Your respondents are doing you a favor, so don’t show them images just because you think they make the survey more interesting.
2. Is an online survey really the best approach? Usually online surveys work very well, but don’t be blind to other techniques. Is your target market online? If your product is aimed at one of the demographics that are underrepresented online you may be increasing the potential for bias. Weighting and oversampling might be helpful, but could increase costs and you may miss out on insights from some of the key targets. Even if you are surveying existing customers (rather than using a panel) be aware of the potential for problems, especially if your email coverage isn’t representative. Perhaps telephone or mail data collection would allow you to better reach the full range of psychographic profiles.
3. Match your marketing to your targets. No surprise here, but remember a couple of things. Your customers may be looking at your messages online, just not at home, by using the Internet in the library or at work. [Side note - speedy access is one reason why online shopping at work is popular at lunchtime.] Don’t alienate the less technically attuned consumers. Differentiated advertising through different media is probably a good idea.

As with many other aspects of research and marketing in general, the real message here is to think, not assume. Try to put yourself in the mind of your respondents, prospects, and customers.

Idiosyncratically,
Mike

Please note, this survey contains media that is not compatible with Firefox Internet Browser

The invitation continued with instructions to copy and paste the link into an Internet Explorer window if Firefox is my default browser.

Let’s look at this in more detail. To dispose of the title question first, the only obvious logical reason why someone fielding online surveys wouldn’t provide support for Firefox users would be if they were surveying people who don’t use it. Perhaps even that isn’t exactly logical, but at least it’s a reasonable excuse. If you are creating something that requires significant development effort, and you are screening for Internet Explorer users, why bother with Firefox?

Unfortunately, that theory doesn’t fit the situation. I’ve seen invitations with this warning for over a year, covering Consumer Package Goods and Retail Stores. I have yet to come up with a good reason, and the research company hasn’t offered me one.

But why is it such a bad idea?

First, Sample Bias. Systematically excluding a segment of the overall population you want to survey is generally a bad practice. It is easy to gather results that are biased, for reasons that may be obvious or less so.

Remember the days of telephone surveys? (I know, we are still collecting data via the telephone, but many people are only familiar with online surveys.) Best practices include calling at random times of the day and night, and also letting the phone ring for quite a while. Why? To increase the chances of the respondent being a person who works, and also to increase the coverage of people who might be elderly or infirm – and who might take longer to reach the phone. Without these measures, you might end up with a disproportionately large number of fit stay-at-home respondents. Some corrections could be done with weighting, but this adds unnecessary complexity versus just improving the representivity of the sample in the first place.

In the case of eliminating surveys from Firefox users, it would probably be a good idea to understand the potential impact through browser share numbers. Unfortunately, this isn’t quite as easy as it might seem, which is probably why we see percentages ranging from 14% to several times higher for Firefox usage in the US. These differences are caused by data collection methodologies and also browser behavior, but this article isn’t about browser share so let’s just settle on an approximation of 20% user share for Firefox. So these surveys are systematically excluding about one fifth of the US population. I could easily come up with some imagined differences  between Firefox users and users of other browsers, but fortunately there is some real research out there. comScore reported in 2007 on a study that looked into the differences between Firefox and Internet Explorer users. The results showed that Firefox users were more likely to be younger, higher income, and male than the average Internet user. Would this impact a project covering food items in the grocery store? You bet. comScore’s study also showed that Firefox users are more likely to have a broadband connection and that their site visitation profile varied from the average – which could impact advertising placement and content.

The other impact for concern, although probably a lesser concern in this case than sample bias, is that of Lower Response Rates. Without hard evidence we can only speculate on the impact, but it seems likely that some people who receive an invitation excluding Firefox might decide not to participate even though they could do so fairly easily by starting Internet Explorer and pasting the link. The additional steps involved are a deterrent. Unfortunately, these particular surveys don’t even work by changing to Internet Explorer rendering within Firefox (something that is common practice to allow usage of sites that are not standards compliant). Longer term, continued invitations that are less easy to use may result in more people leaving the panel.

In conclusion, make your surveys sample as representative as possible, and don’t do anything in the invitation or survey to turn people off.

One last note on this subject. The problem invitations specifically state that the surveys don’t work with Firefox. Even if Firefox is the only excluded browser, it represents over 20% of the overall market as of Dec 2008 according to Net Applications.  It probably doesn’t make sense to invest in development for older browsers, but as Safari (7.9%) and Chrome (1%) usage grows the challenges for survey developers are going to increase.  Overall, browsers other than Internet Explorer are currently about one-third of total usage.

Idiosyncratically,
Mike Pritchard
5 Circles Research