Does Marketing Research Live in an Echo Chamber? – Part 1

I’ve been working in the MR industry since 1978, and the one unchanging theme of that long period has been the constant complaint by the senior corporate executives – who fund the industry by their demand for research-based decision making – that MR just isn’t very good at identifying opportunities or pointing to the new products, services, and ventures with the best chances of successfully capitalizing on them.

I remember being surprised when I first encountered a CEO denouncing the entire MR enterprise at a major conference – which probably happened sometime within my first or second year. Was my new, desperately needed first job after grad school, and maybe the whole industry, going to go down in flames before I even got started? It hasn’t but the drumbeat of C-suite dissatisfaction has never lessened.

This all came to me as I was reading one of the many articles from the “nerdiest election” in which Romney had not prepared a concession speech because – by all the accounts I’ve seen – he didn’t think he’d need one. It sounds like Romney really got blindsided. Not only did he not have a concession speech held in reserve, but he also planned to celebrate the election with a fireworks display over Boston Harbor and his campaign even had a “Romney Wins” website up and ready. (Someone posted screen grabs, of course.)

As Slate’s John Dickerson said, “He got the numbers wrong … in the end Romney and Ryan had to watch CNN to find out how their campaign was doing.”

The blog posts and news stories lay blame on what David Frum labeled the “conservative entertainment complex.” And it looks like the Romney’s “marketing research” group followed the media’s lead in asking the questions they wanted asked, hearing the answers they wanted to hear, and reinforcing an internal viewpoint that, in the end, failed as MR and left the CEO “gobsmacked” and angry.

I read all of this with the shock of recognition. My mind snapped back to 1985 and the introduction of New Coke. All the research that had been done! And, the magnitude of the disaster that followed! The astonishment of everyone inside Coca-Cola that their carefully constructed edifice had been built on sand. I had seen all of it and had enjoyed having the inside scoop thanks to two colleagues who had come from Coke’s MR department and understood the backstory, which had included a huge research effort.

The post-election stories seem to be revealing a distinctive, internal worldview shared by the campaign and its media supporters. Republican pollster, Whit Ayres, described the research being drive by “rosy assumptions on a likely electorate…at…substantial variance with recent history.”

An “echo chamber” bubble developed when the campaign research staff and its clients elaborated a shared narrative about polling and sampling methods and how to interpret results. Outside that bubble, the academic social scientists and media stat nerds – with Nate Silver as their symbolic leader – were using different methods, asking different questions, and interpreting their findings differently.

They were right in the end, and the people in the bubble were wrong.

A case can be made that MR and our clients have created a similar bubble, in which we talk to each other in an echo chamber. And, a case can be made that the academic social scientists and techie stat nerds are “threatening” traditional MR with everything – including Big Data, social media and web based behavioral analytics, location data, remote facial analysis, and eye tracking – living in an entirely different world.

The GRIT survey as well as reports coming out of consultancies like Cambiar (site registration required), say clearly that corporate research departments are eagerly welcoming all the new “non-MR” vendors knocking on their doors. What’s worrisome is what would happen if the CEOs start sending more work to the guys outside the MR world, and they started having some successes calling the game. That’s what the voices that expect the leading “MR” vendors of the end of the decade to be companies like Google, Facebook, and Twitter.

Why didn’t all of these new approaches arise out of marketing research? At the very least, why weren’t traditional marketing research companies their earliest and most eager adopters? Why didn’t clients hear about all of these developments from their MR vendors? Is it because the conversation was “closed” and things like that simply had no place?

I’m not arguing that MR is operating in bad faith – only that they and their client audiences may have constructed a particular shared view of commissioned and conducted research that has become closed, limited, and overly rigid.

Marketing researchers are almost universally serious, sober people who see themselves as technical experts in a field that demands hard work, clear thinking, and ingenuity to provide vital information to decision makers while often suffering little respect, diminished budgets, and constricting timelines. But make no mistake about it, marketing research has been working with the same basic approaches for at least a couple of generations. I made a stab at some of the qualitative issues in a previous post, so in part 2 to this post, I will examine a few things about quantitative research.