The Future of Sample Panels

What, exactly, is a “sample panel,” and what will one look like tomorrow?

The modern MR sample panel is a descendant of the Paleolithic MR “mail panel,” a list of people or households, usually selected to be representative of some larger population, like grocery shoppers or readers of magazines, from which survey samples can be drawn. When MR survey methods migrated from the phone room – where “sample” meant chosen phone numbers derived from the probabilistic arcana of number blocks and the telephone company rules for their assignment – to online – where there were no analogues of assigned phone numbers or listings of email addresses – sample providers assembled new-age opt-in panels following the model of the mail panels to fill the gap.

Potential respondents were recruited from an array of sources, profiled, and usually at least somewhat balanced to reflect the larger online population and, eventually, as internet penetration and usage allowed, the general population. Sample lists, like mailing lists, were pulled to match a target profile and were leased to researchers, usually for a single use, rather than sold.

The individual panel members were seen as assets by the sample providers, bought and paid for by their recruitment efforts, and as with mail lists, unauthorized re-use of a sample was seen as a form of theft. A panelist, once recruited was seen more-or-less as a permanent member of the panel, and although panelists did inevitably drop out, the panel providers did their best to retain them for as long as possible, leasing them out as often as they could.

A panel was like a village, and the villagers, once moved in and settled, were expected to stay put.

It turns out that real sample panels became less and less like villages over time (if, in fact, they ever really were much like them). True to form, the Internet rather quickly generated a mob of information-sharing Google Searchers swapping leads on ways to maximize income from taking surveys, generally monetizing their villager status. Tips were traded, methods (and software) for enrolling multiple times on the same panel were discovered, and at least some sub-set of “villagers” developed Doppelgangers who moved into every “village” under the sun. Even the less entrepreneurial “villagers” eventually joined a few panels and came and went as they pleased rather than settling down and tilling the village soil. Turnover rates grew, response rates dropped, and “data quality” became an increasingly important issue as the villagers refused to behave like honest hardworking peasants, speeding and straight lining their way to incentive payoffs.

Easily gathering or utilizing a lot of the mobile data that smartphones can generate means utilizing apps. Although the browser-based MR model of the recent past can be stretched onto mobile, to really make the process simple for the respondent and maximize the approach means using MR-specific apps. Community interactions, location, in-the-moment interviewing, and many other mobile-oriented approaches can hardly be done gracefully, if at all, through the generic browser. So eventually, data gatherers will move to develop their own apps, which they will integrate with their various research platforms, analysis/reporting tools, and collaboration systems. Eventually, as more respondents and clients become mobile-centric, this evolution will become inevitable.

Step back a minute and ask this question: “What do you call the group of people who have downloaded a particular researcher’s MR app on their mobile devices?”

You may wonder if they should be called anything much at all. “The people who took part in our recent projects,” perhaps. But think about it a bit and you will realize that the very act of installing an app created a sort of panel: a group of profiled respondents who can be recontacted and invited to participate in other projects. The more projects you conduct using a particular app the larger your “panel” grows and, if your work is concentrated by the interests of a group of recurring clients, for whom you recruit from a repeating selection of segments, the more representative of that population your panel is likely to become. If your project portfolio is broad enough and your sampling wide enough, the more representative of the general mobile population it will become. And, as happened with panels that were first representative of the online population, over time your panel will eventually mirror the general population.

Of course, the ties between your panel and your panelists may be more tenuous than the ties of the traditional panel village. People may install your app and never use it again, or choose to de-install it. Or they may keep it on their devices but refuse to give you permission to push notifications to them. They may never again open the app to see if you’ve tried to contact them. Or, you may get someone now and again seeking a diversion or the chance to win an incentive, who refuses most invitations they receive. But still, these cases are analogous to cases in traditional sample panels, and are really in no way odd. Your app-based “panel” is recognizably a fairly traditional “sample panel.”

But it’s not a village; it’s more like the crowd of people who pass through a train station or drive along a highway during rush hour. They come together for a brief period of time, and then disperse, re-forming periodically with a constantly shifting membership. You can’t really count on the crowd coming together again in exactly the same way, and you certainly can’t count on having consistent profiling data on its various members.

But it is still a panel – a mobile-first, app-based panel – and you built it simply by doing research using your app. Moreover, like a traditional panel, it’s a corporate asset that you can monetize, reuse, repurpose, and build on.

This only slightly odd sort of panel is also a disruptive development in the relationship between researchers and sample vendors.

Some sample vendors will certainly resist allowing researchers to install their apps to conduct projects, insisting that they use the vendors’ apps instead, preserving their villagers from being stolen. Others will realize that their so-called villagers aren’t really theirs for long and will move to quickly give access to them as often as possible while they’re staying, briefly in their village. A single individual may be a member of many different panels simultaneously, and this will not be seen as in any way unusal.

Clients who are building customer communities will create their own sample panels by using apps for community communication, and some sample vendors will allow them to raid their villages rather than turn down this growing revenue source. Researchers will also find other avenues for placing their apps on respondents’ devices, distributing them through mobile ad networks and other channels, building panels from outside the sample providers’ villages among the larger proportion of people who never joined a sample panel but who do see mobile ads and may download various mobile apps — as many as 90% of the population by some estimates.

All of this portends a period of extreme change in MR that will be felt by and will have to be negotiated among sample providers, researcher suppliers, and clients as the internet does what it seems to do best, bringing about the disruption of existing models and the re-invention of existing methods.