"The Ultimate Question": Science or Snake Oil?

Being in the customer research business, our group was amazed a few years ago by the impact of Fred Reichheld's book The Ultimate Question. In a nutshell, Reichheld wrote that the willingness of your customers to recommend your business to others was a sure-fire forecaster of your companies growth. It seemingly provided companies with a magic bullet. Busy CEOs suddenly had one simple question that would tell them what they needed to know. For a time, it seemed that all of our clients wanted to rearrange their surveys to ask the 'ultimate question' and get their 'net promoter score'.

The Ultimate Question has recently come under tremendous fire in places like The Journal of Marketing. A recent article in Colloquy went so far as to liken The Ultimate Question debate to a matter of 'faith versus science'.

So what's the answer?! I've always appreciated the wisdom of Dr. Bob, one of my professors, who taught me that "truth is often found at the point of tension between the two extremes". It certainly applies the current Ultimate Question debate. The truth is that there is no magic bullet and asking one question is not going to give you a perfect crystal ball into your company's future. By the same token, I certainly don't believe that Reichheld deserves the total vilification he's receiving from detractors.

Having customers who are willing to promote your business to others is a positive sign that you're doing many things right. It's also a good indicator that you can expect to reap the profitable results of that customer loyalty. But, how do you get customers that satisfied? For that, you need to strategically ask a lot more questions.

Creative Commons photo courtesy of Flickr and hfabulous

Comments

[Amy Madsen]

Thanks for an interesting post. Indeed, the "Ultimate Question" is one question, but there's lots to do after tabulating the Net Promoter Score from the answers.

We have a number of Net Promoter blogs that may be of interest to your readers. Net Promoter blogger Jeanne Bliss, author of "Chief Customer Officer" has some hands-on advice such as "We Know Our NPS. Now What?" and "Mining for Gold: Listening Hard to 'Detractors.'"

Her blogs can be found at this link: http://netpromoter.typepad.com/jeanne_bliss/

[Tim Keiningham]

I believe that there are two additional points that need to be made regarding the Net Promoter debate.

The first involves your defense of Mr. Reichheld from his detractors despite the failure of his claims regarding Net Promoter and its linkage to growth and customers' future loyalty behaviors. It is important to note that Mr. Reichheld has made outrageous claims regarding Net Promoter while denigrating other metrics as useless in linking to growth; it is interesting that he was not called to task for these irresponsible statements at the time they were being made. Now that they are proven false, it is ridiculous to portray him as a victim for having to face the music for his conduct.

Second, and most important, our research presented in the Journal of Marketing presented VERY strong evidence of research bias. In particular, we were able to replicate a subset of Reichheld's reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Our findings clearly show that when using Reichheld’s own data, Net Promoter wasn’t superior to the ACSI. It is difficult to imagine a scenario other than research bias as the cause of this finding. This is a serious problem. We expect published research to be free of bias in management science, just as we do in all other fields of study. Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no HBR paper introducing Net Promoter without the research.

Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.

Net Promoter “Believers” typically focus on “testimonials” from users or “qualifications” regarding the usefulness of the metric in response to evidence contradicting their earlier assertions and beliefs. Unfortunately, as “researchers” this is the equivalent of ignoring the 900-pound gorilla in the room; the research upon which Net Promoter is based is the sole reason for its existence. If it is biased, then it is disqualified—period.

Sincerely,

-- Tim Keiningham

[Tom Vander Well]

Thank you, to both camps, for your comments. I'm glad that IowaBiz can help continue the conversation.

I was disappointed, Amy, that your mention of Jeanne's post "Listening Hard to Detractors" might lead readers to an official defense of NPS against the accusations of bias.

Tim, please understand that what Reichheld calls "the ultimate question" has been a question our group has asked as part of statistically valid, comprehensive customer satisfaction research projects for years. It can provide a piece of data which, when placed in context with other data, can yield value for any company trying to understand where they stand in their customer's mind. Calling it "the ultimate question", however, and making it the sole focus of your customer sat research, as Reichheld promotes it, is a faulty notion.