A tale of three types of cheating.
If you are going to fudge the numbers, you’d better be very clever.
Last December’s Annual Year in Ideas issue of the New York Times magazine included an idea titled “Forensic Polling Analysis” describing how Nate Silver analyzed results published by a polling firm called Strategic Vision. Silver decided to take a look at the results because Strategic Vision had been censured by the American Association for Public Opinion Research for not revealing details about how polls were conducted. After looking at 100 sets of poll results for the previous 4 years Nate concluded that the distribution of the last digit wasn’t random as it should have been. In addition to examining Strategic Vision’s numbers, he analyzed results from Quinnipiac (a well-respected pollster according to the New York Times), and found the last digit distribution conformed to what might be expected from chance. Silver’s conclusions were confirmed by a retired University of Illinois physics professor, Michael Weissman, who used Fourier analysis to come up with the chance of 1 in 5,000 of Strategic Vision’s results being produce by a legitimate poll.
Silver’s article [http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html] describes in more detail his use of Benford’s law to perform the analysis, and how it is used for forensic accounting (i.e. fraud detection). Oddly, although the Wikipedia article [http://en.wikipedia.org/wiki/Benford%27s_law] leads to discussions of forensic accounting and predictive analytics for fraud detection, Benford’s law is concerned with first digit, not the last. Still, the real point is that it is very difficult to manually generate random numbers. In fact, people often don’t recognize randomness (read Leonard Mlodinow’s entertaining “The Drunkard’s Walk” for more on the subject).
Apparently, Strategic Vision still hasn’t revealed how their polls are conducted, but they did threaten to sue Nate Silver. Hopefully I won’t be a target as a mere reporter of others work.
Mystery shopping should be a mystery.
For accurate results when testing service quality, it is important that the transaction is normal, receiving no special treatment. We’ve probably all been in situations where we wonder if that’s really the case. If our experience at a restaurant was so bad, why does it get good reviews? Or why does the car dealer have a five-star rating when everyone we know hates them? Recently in Britain, the postal watchdog Postcomm is considering action against the Royal Mail following allegations that lists of customers involved in a test were circulated so that the deliveries to these customers could be ensured of being on time. Staff were also able to recognize and prioritize the test letters. Apparently the cheating has been going on for nearly 4 years, with thousands of people involved. Ironically, the published results haven’t been improved as a result. More details are at http://www.telegraph.co.uk/news/uknews/royal-mail/7431654/Cheats-at-Royal-Mail-fix-delivery-times.html
Encouraging the customer to lie is bad for everyone
Over the past months I’ve had a couple of deliveries from Sears. In both cases, one of the delivery team told me that I’d be getting a phone call to ask how the delivery went. Fair enough. But then they held up a card showing me the ‘5’s that they wanted me to give them. Obviously, I was offended. The request was presumptuous, whether or not they told me that the scores were important to their performance reviews (both did). Involving customers in creating inaccurate results doesn’t improve performance or customer satisfaction. In fact, the sour taste keeps the bad parts of both experiences in my mind much longer. The refrigerator was supposed to be leveled but the delivery team didn’t bring a level. And the people delivering the lawn tractor didn’t check that their oversized truck could be driven up the driveway and didn’t think to ask if I had gas available so the tractor could be driven to our house. By contrast, another appliance delivery from a small local company was handled completely and competently, with no reference to a follow up survey.
Don’t get me wrong – I’m a fan of Sears. I’ve had good experience with their brands and generally find the sales people helpful. But this approach to customer satisfaction doesn’t help them improve. Apparently Sears thinks managing by fear is appropriate, or they aren’t directing employees properly. That’s too bad.