• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

5 Circles Research

  • Overview
  • SurveysAlaCarte
  • Full Service
  • About
  • Pricing Gurus
  • Events
  • Blog
  • Contact

Leave a Comment

When Validation Backfires

I just came across an interesting issue with validation in an online survey using a Van Westendorp pricing model.  Van Westendorp is one of the common ways to test pricing by directly questioning prospective purchasers.  This post isn’t about Van Westendorp, also known as the Price Sensitivity Meter (you can find plenty of references online, including  a starting point on Wikipedia) but you need to know a little to understand the issue.  Survey respondents are asked a series of questions about price perceptions, as follows:

  • At what price would you consider the product starting to get expensive, so that it is not out of the question, but you would still consider buying it? (Expensive/High Side)
  • At what price would you consider the product to be so expensive that you would not consider buying it? (Too expensive)
  • At what price would you consider the product to be priced so low that you would feel the quality couldn’t be very good? (Too cheap)
  • At what price would you consider the product to be a bargain—a great buy for the money? (Cheap/Good Value)

There is some debate about the order of questions, but in this example the questions were asked in the order shown.  The wording was slightly different.  Researchers are sometimes concerned about whether the respondents understand the questions correctly, especially since the wording is so similar (the Expensive, Cheap etc. designations are usually not inclined in the question as seen by a survey taker).   One way to address this concern is to highlight the differences.  Or you might point out that the questions are slightly different and encourage the respondent to read carefully.

The other approach is to apply validation that tests the numerical relationship.   Correctly entered numbers should be Too Cheap < Good Value  < Expensive < Too Expensive. (We usually ask these questions on separate pages so as to get independent thoughts from the respondents as far as possible, rather than letting them see the group of questions as one and making them consistent or nicely spaced).

In this case, the research vendor chose to validate, but messed up big-time.  When I entered a value for ‘Too Expensive’ that was higher than the value for ‘Expensive’, I was told to make sure your answer is smaller or equal to the previous answer.  Yes, they forced me to provide an invalid response!  I hope they caught the problem before the survey had gathered all the completes, but maybe they didn’t – given how fast online surveys often fill.  They probably had to field the survey again because the pricing questions were integral to the research objectives.

Why did this happen, and how can you prevent a similar problem in your surveys?

My guess is that the underlying cause was that debate about question order that I mentioned earlier.  The vendor probably had the questions switched when the validation was tested, and then changed the order before the survey was launched.

But the real message is that proper testing could have identified the issue in time to correct a very expensive error.  There is no excuse for what happened.  This doesn’t even fall into the class of problems that the pilot or soft-launch would be needed to catch.

So, test, test, and test again.   In particular, test using people who aren’t research professionals or experienced survey takers.

If you are creating your own surveys, don’t let this kind of problem stop you.  You can do just as good a job of testing as the big companies, and big companies aren’t immune.  This survey was delivered by one of the top 10 U.S. market research firms.  I won’t publish the company name here, but I’ll probably tell you if you catch me at one of my workshops (coming soon).

Idiosyncratically,

Mike Pritchard

Filed Under: Methodology, Questionnaire, Surveys, Testing

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Follow us on LinkedIn Subscribe to Blog via RSS Subscribe to Blog via Email
Every conversation with Mike gave me new insight and useful marketing ideas. 5 Circles’s report was invaluable in deciding on the viability of our new product idea.
Greg HowePresidentCD ROM Library, Inc.
You know how your mechanic knows what’s wrong with your car when you just tell them what it sounds like over the phone? Well, my first conversation with Mike was like that — in like 10 seconds, he gave me an insight into my market research that was something I’d been struggling trying to figure out. A class like this will help you learn what you can do on your own. And, you’ll have a better idea of what a research vendor can do for you.
Roy LebanFounder and CTOPuzzazz
Since becoming our contracted consultant for market research services in 2010, 5 Circles Research has revolutionized our annual survey of consumer opinion in Washington. Through the restructuring of survey methodology and the application of new analytical tools, they have provided insights that are both wider in their scope and deeper in their relevance for understanding consumer values and behavior. As a result, the survey has increased its significance as a planning and evaluation tool for our entire state agency. 5 Circles does great work!
Blair ThompsonDirector of Consumer CommunicationsWashington Dairy Products Commission
Many thanks to you for the very helpful presentation on pricing last night. I found it extremely useful and insightful. Well worth the drive down from Bellingham!
G.FarkasCEOTsuga Engineering
I have come to know both Mike and Stefan as creative, thoughtful, and very diligent research consultants. They were always willing to go further to make sure respondents remained engaged and any research results were applicable and of immediate use to us here at Bellevue CE. They were partners and thought leaders on the project. I am happy to recommend them to any public sector client.
Radhika Seshan, Ph.DRadhika Seshan, Ph.D, Executive Director of Programs Continuing Education Bellevue College
First, I thought it was near impossible to obtain good market information without a large scale, complex market study. Working with 5 Circle Research changed that. We were able to put together a comprehensive survey that provided essential information the company was looking for. It started with general questions gradually evolving to specifics in a fast pace, fun to take questionnaire. Introducing “a new way of doing things” like Revollex’ induction heating-susceptor technology can be challenging. The results provided critical data to help understand the market demand. High quality work, regard for schedule, thorough understanding of the issues are just a few aspects of an overall exceptional experience.
Robert PoltCEORevollex.com
I hired Mike to develop, execute and report on a market research project involving a potential business opportunity. I was impressed with his ability to learn the industry and subsequently develop a framework for the market research project. He was able to execute the research and collect data efficiently and effectively. Throughout the project, he kept me abreast of the progress to allow for any adjustments as needed. The quality and quantitative output of the results exceeded my expectations and provided me with more confidence in the direction of the business opportunity.
Mike ClaudioVice President Marketing and Business DevelopmentWizard InternationalSeattle
5 Circles Research has been a terrific research partner for our company. Mike combines a wealth of experience in research methodology and analytics with a truly strategic perspective – it’s a unique combination that has helped our company uncover important insights to drive business decisions.
Daniel WiserBrand ManagerAttune Foods Inc.
Great workshop! You know this field cold, and it’s refreshing to see someone focused on research for entrepreneurs.
Maria RossOwnerRed Slice
What we were doing was offering not just a new product, but a new market niche. We needed to understand traditional markets well to characterize the new one. Most valuable was 5 Circles ability to gather research data and synthesize it.
Will NeuhauserPresident Chorus Systems Inc.

Featured Posts

Dutch ovens: paying a lot more means better value

An article on Dutch ovens in the September/October 2018 of Cook’s Illustrated gives food for thought (pun intended) about the relationship of between price and value. Sometimes higher value for a buyer means paying a lot more money – good news for the seller too. Dutch ovens (also known as casseroles or cocottes) are multipurpose, [Read More]

Profiting from customer satisfaction and loyalty research

Business people generally believe that satisfying customers is a good thing, but they don’t necessarily understand the link between satisfaction and profits. [Read More]

Customer satisfaction: little things can make a big difference

Unfulfilled promises by the dealer and Toyota of America deepen customer satisfaction pothole. Toyota of America and my local dealer could learn a few simple lessons about vehicle and customer service. [Read More]

Are you pricing based on cost rather than value? Why?

At Pricing Gurus, we believe that value-based pricing allows companies to achieve higher profitability and a better competitive position. Some companies disagree with that perspective, or feel they are stuck with cost-based pricing. Let’s explore a few reasons why value-based pricing is generally superior. [Read More]

Recent Comments

  • Mike Pritchard on Van Westendorp pricing (the Price Sensitivity Meter)
  • Marshall on Van Westendorp pricing (the Price Sensitivity Meter)
  • 📕 E mail remains to be the most effective SaaS advertising channel; Chilly emails that work for B2B; Figuring out how it is best to worth… - hapidzfadli on Van Westendorp pricing (the Price Sensitivity Meter)
  • Isabelle Spohn on Methow Valley Ski Trails gets pricing right
  • Microsoft Excel Case Study: Van Westendorp-un "Price Sensitivity Meter" modeli | Zen of Analytics on Van Westendorp pricing (the Price Sensitivity Meter)

Categories

  • Overview
  • Contact
  • Website problems or comments
Copyright © 1995 - 2023, 5 Circles Research, All Rights Reserved