A Comprehensive Guide to
product testing in market research

How do you choose the right design for your product testing study?

As with any research, you must start with an objective. What do you hope to accomplish? Knowing why you’re testing the product and how you plan to use the findings from the research helps to ensure the correct design and analysis for the project. Beyond that, there are many other questions to answer about the product itself, the user, and the usage.

1. Understand constraints and product considerations

First, we like to think about the possible constraints and product considerations, as these influence the options on which method to choose. You can start by asking yourself these questions:

• What is your budget?
• How much product can you obtain?
• What timing must you meet?
• Is there confidentiality around the product? Do you need to get it back after respondents use it?
• Are there constraints with the product itself? Can it be sent in the mail? Does it require complex instructions? Must it be kept frozen or refrigerated? Is it fragile?

2. think about the product itself

• How do you expect the product to be used? How often, where, by whom, in what situations, etc.?
• Is there a need for natural, in-the-home interaction? Is it a product that needs to be evaluated over time?
• What level of involvement is required with the product? Is it ready for use, or does it require preparation, building, or installation to experience it? Does one’s experience with it require time (such as apparel that should be worn and washed over time)?
• Who is the user? Is it an individual, or is the product for the entire household?
• Is the product fully developed, or is it still being optimized?

3. think about what you want to learn

And in particular, how you’ll use the research findings or data:

• Is this exploratory, requiring qualitative?
• Is the product in need of validation, requiring quantitative data?
• Is there a benchmark (either a current in-market product, a competitive product, an alternative prototype, etc.)?
• Based on the objectives, who do you want to talk to? Should you be talking with new users or current users? Must they be brand acceptors? If testing food, they should be accepting of the flavor.
• What are the key questions/unknowns surrounding the product? Is it more about how it’s used in the home or simply a disaster check on taste/flavor, fit, or usability? Will the unboxing of the product be a moment that you
  want to capture?
• Should the product be branded or blind? Will knowing the product’s brand name skew the results?

Asking yourself these questions about your objectives, any constraints, and the product experience itself will help lead you to the right methodology. We detail the various methods below, from qualitative to quantitative and from In Home to Central Location.

when to choose a Qualitative approach

We typically recommend qualitative methods when our clients are looking for a deep understanding of consumers’ attitudes and experiences with the product. This is typically needed for products that are not yet ready for market and need some direction on how to optimize the product experience. Qualitative product testing can stand alone or be paired with quantitative to add deeper understanding of the product experience. Specifically, qualitative product
testing can:

• Bring consumers’ experience with the product to life
• Provide an understanding of customers’ needs and expectations to identify relevant product changes and optimizations
• Answer the “whys” behind consumers’ attitudes towards a product and brand (e.g., appeal, frustrations, usage, relevance, and fit)
• Help position the product vs. its competition

If it’s decided that a qualitative approach will best fit our client’s needs and objectives, the next step is to determine if an In-Home Usage Test (or IHUT) or a Central Location Test (or CLT) is more appropriate, whether the research should be conducted online or in person—or if a combination of these approaches is needed.

the qualitative in-home usage test
(or ihut)

Both online and in-person qualitative IHUTs offer unique ways to truly capture and understand consumers’ experiences with a product in a natural setting to garner more realistic responses.

Product experience over time

For example, we had a consumer electronics client who wanted to understand how consumers adapt to using a new cell phone. We created a six-month online community of consumers, starting with tasking consumers to order and customize their phone online. Then, once their new phone was delivered, consumers captured their unboxing process and initial setup and usage on video and shared that experience with us through the community. In addition to having access to these consumers for ad hoc projects while the community was live, we had open threads on the research platform to capture delights and frustrations in the moment. We created a feedback loop for the client’s engineering team to be able to address these issues head-on.

product comparison

In another online qualitative IHUT, we helped a CPG client position new products, which were recipe shortcuts. We conducted a two-week community with consumers. First, we had them document (via video) their experiences making one of the two key recipes from scratch to understand their process, goals, and pain points. Then, they reviewed the product concepts and detailed their reactions. Next, they used the test products to make the recipe—again on video. They then reflected on the product concept to see where it delivered versus where their expectations fell short.

A key learning was that, while the two test products seemed similar in that they were both canned recipe shortcuts, their frame of reference was very different, impacting how each recipe was received. One product was seen as fresher and better than the typical powdered spice mix most consumers use. However, for the second recipe, the product was seen as a downgrade from the fresh ingredients consumers preferred. Learnings helped the client appropriately position each product.

the quantitative + qualitative
in-home usage test

Sometimes (but not too often), there is a need to combine qualitative and quantitative methodologies into one IHUT. When a study’s objective calls for combined methods, with C+R Research’s established best practices and qualitative and quantitative research divisions under one roof, we are easily able to accomplish this.

inspiration for new product development

For example, one client wanted to measure consumers’ behaviors around a household appliance and explore their level of frustration to inform their innovation product plans. We developed a new methodology for observing and measuring consumers’ natural behaviors in their home. We set up motion-sensor cameras, created an inventory measurement system using RFID scanners to capture what was being used by the appliance, and administered an app-based survey that was triggered by the consumers’ activities.

Because of the level of involvement required, we first piloted the program with a small group of consumers to optimize their interactions with each piece of the system. Then, we launched the full-scale study where we collected not only the survey data, but also analyzed the scanner data to understand the sequencing, timing, and frequency for each event, and coded thousands of hours of video. We also conducted in-home interviews with consumers who had outlier behavior and high levels of frustration with the appliance to deepen our understanding.

the qualitative central location test
(or clt)

In other situations, a central location for qualitative product research can be more appropriate. For instance, we often recommend focus groups to our clients as a flexible method when consumer feedback is needed to further develop product concepts or prototypes and/or when a product taste test is just one part of the overall research objectives.

new product launch

One example comes from a European-based client who was interested in launching a new product concept to consumers in the U.S.—a chilled bread proposition. We were challenged to ground our client in the bread market landscape in the U.S., while also uncovering opportunities and considerations for their chilled bread product offering.

As an introduction to the U.S. bread category, we leveraged our partnership with Numerator’s InfoScout Omnipanel™ to provide our client with receipt-based shopper behavior insights in the category. Next, we conducted focus groups in three major markets among bread purchasers to understand consumers’ receptiveness to the client’s chilled bread product concept. While the product’s taste was favorable, the product’s perceived usage, packaging, and intended in-store location negatively impacted the product’s overall appeal and acceptance.

Our client went to work to revise their product concept based on the consumer feedback. Through two more rounds of revisions and focus groups (and a half- day, C+R-led product-naming workshop), we were able to identify three product formats with multiple perceived usage occasions and three flavors that “wowed” consumers. Additionally, the optimized product branding, product name, and product packaging design communicated the product as artisan and high quality, which the client desired. Upon completion of the project, we also produced a video sizzle reel, which included highlights from the consumer focus groups, to help the product appeal to grocery retailers. The first product phase was launched in select grocery retailers in October 2018.

Product usability

Focus groups are also a great way to explore usability, allowing consumers to get hands-on with products and enabling us to observe how easily they can open, use, and carry a test product. By seeing consumers explore and demo the product, we can identify what is and isn’t intuitive, and where instructions need to be clearer. In addition, we often recommend focus groups when a new product’s confidentiality is paramount.

For example, we had a technology-driven client come to us to gain consumer feedback on two hair tool products. Of the brand’s two products, the first was already available in the market and needed refinement. The second product was a highly confidential working prototype that was still being engineered (it couldn’t leave the engineers’ sight).

The company’s stakeholders wanted to evaluate consumer experiences with each product’s system, as well as the relevance of introducing this brand into an established category. Everything from the product’s relevance, functional, emotional, and social benefits, to product usability, aesthetics, packaging, and pricing needed to be tested qualitatively.

While online video journals were most appropriate for the client’s current product, we needed a different approach for their new prototype, where confidentiality was a primary concern. To address this, we conducted in-person usability testing in a bathroom environment (e.g., sink, counter space, two-way mirror, spacious back room, etc.). Each woman walked in with a bag of her personal hair styling products and tools; by the end, she had also used the working prototype on her hair, sharing her contrasting perceptions, behavior, and needs from the experience. This hybrid qualitative methodology, topped off with compelling video reporting, was the solution our client needed to quickly substantiate user errors, expose what was working, and optimize its products.

when to choose a quantitative approach

Quantitative research will validate if a product is ready for launch, and what small modifications, if any, need to be made to make it successful. Again, quantitative product testing can stand alone, or be paired with qualitative to add deeper understanding of the product experience.

Depending on the level of control necessary and the length of the usage period needed to explore the product experience, one could product test in home through an IHUT or in a Central Location Test (CLT).

the quantitative in-home usage test
(or ihut)

The in-home-usage test is our recommended approach when the true product experience is more important than a completely controlled setting. Respondents have the freedom to use the products in their everyday lives, while we set some parameters to measure and compare experiences accurately. Consumers interact with products differently depending on the occasion, so IHUTs are a valuable method when we want to capture the breadth of usage occasions over a longer usage period than a single setting in a CLT (Central Location Test) would allow. Given the longer time frame, we also recommend this method when attempting to diagnose sensory delighters and pain points that surface after continual use.

manufacturing change

In one case, we were approached by an apparel client whose R&D team modernized the production line for one of its top-selling bras. Knowing the move to a new production line could create undesirable changes in a tried-and-true product, we designed an IHUT for women to wear both the originally produced bra and the newly produced bra. The longer usage period allowed women to more accurately compare their sensory experiences between the two bras and provide our client with peace of mind—the manufacturing changes did not impact women’s sensory experiences with the new bra, and they could move forward with the new production line.

logistics

Product logistics can make or break an IHUT. It’s imperative that the correct product is shipped to the correct respondent without damage or delay. C+R Research excels in this area. We have a dedicated logistics team with a wealth of experience in procuring, labeling, packing, and shipping all types of products across the country—and we’ve formulated best practices. We take a cross functional approach to product logistics, designing and programming surveys that capture contact information in a safe and secure manner, and “assigning” products to respondents in a way that’s easily trackable by our analysts and field team. Our analysts, operations, and logistics teams work together—with many checks and balances along the way—to ensure respondents receive and evaluate the products as designed.

respondent freedom

Before moving forward with an IHUT, timing and tolerance for respondent freedom should be considered. We often test multiple products in a sequential order and rotate product use to eliminate position bias. We can control which products are evaluated when by sending products in separate shipments and following up with a dedicated survey after each phase. Timing constraints may prevent us from shipping products out in a sequential manner, requiring all products be shipped to the respondent from the start. While we can carefully label and instruct respondents on which product to use when, we ultimately sacrifice some of that control. The respondents’ freedom inherent to IHUTs also makes their experience a greater unknown and harder to control and account for.

the quantitative central location test (clt)

The Quantitative Central Location Test (CLT) is our recommended approach when a completely controlled setting is necessary (and a long-term, in-home usage experience is not).

a step-up story

For example, we had a client who needed to better understand their food waste disposers. While we could have installed these in homes, the point of the research was not the installation process or constant use. The objective of the research was to understand consumers’ ability to discriminate between levels of benefits and identify features that drive satisfaction in the food waste disposal experience. This required a controlled environment where everyone was grinding the same measured batch of food. Through Triangle Testing¹ at a central location, with support from the product’s engineers who were on hand to install the products, we were able to obtain actionable results for our client that helped them to communicate differences in each model, with a strategy of “step-up story.”

¹A Triangle Test is used to see if differences in products are noticeable. Respondents evaluate three products, two of which are the same and one that is different. They are asked which one of the three is different from the other two. Through statistical methods, we can determine if the difference is noticeable.

product reformulation

On the other end of the spectrum, we have also successfully leveraged CLT in testing a reformulation of a premium skincare product. Specifically, the team wanted to evaluate aesthetics such as texture, smell, and feel on skin to see if current users liked the new product. Again, the focus was not on the long-term-usage experience, but more focused on elements/features of the product. Moreover, the CLT format allowed for a more controlled environment for the use of this product than what could be accomplished through an IHUT. After conducting a CLT, it was clear that some aspects of the new skincare line were well-received, while other aspects were disliked and could alienate a fair portion of current users. Additionally, advanced analysis was able to reveal which disliked aspects were alienating current users the most and, therefore, needed to be corrected before launching.

Just like with IHUTs, product logistics can make or break a CLT. As with an IHUT, it’s key to get the right products to the right places. But, it’s also critical with a CLT to pick the right facility based on the needs of the study. It’s also essential to be sure the instructions for product storage, preparation, usage, rotation, etc. are being executed exactly as instructed and the same from facility to facility if more than one facility is used.

the quantitative + qualitative
central location test

Many times, when the objectives call for more exploratory feedback on a product, we may add on qualitative to a quantitative CLT, or vice versa.

new product launch

Recently, an appliance manufacturer approached C+R for research focused on a new-to-market large kitchen appliance. They wanted high-level quantitative feedback on the product:

• Do consumers like its look?
• Do they like the interior features?
• Do they expect it to fit into their kitchen décor, etc.?

They also desired more in-depth qualitative feedback to help craft the communication of the product’s benefits and direct any optimization before going to market. Over a couple of days, hundreds of consumers evaluated the client’s product and five competitive models.

Respondents were pre-recruited to a specific session and evaluated the models in a random order designed by C+R. Data was captured via tablets in real-time, evaluated in the back room by C+R analysts and the client team, and used to determine who to ask to stay for the focus group that followed each quantitative session. A series of focus groups took place where consumers discussed the appliances, guided by the quantitative results. Not only does C+R Research have expertise in qualitative and quantitative research, we also have teams that work closely to execute hybrid studies—making the analysis of the quantitative and qualitative data come together in one cohesive story. Therefore, the integration of the methodologies is flawless from logistics to the final deliverable.

advanced analytics in quantitative research: a critical component for successful product testing

We’ve covered a lot of ground on approaches, methods, and logistics; but, to ensure you have a successful product test, a critical component is making sure you analyze the data correctly. That’s why our quantitative research analysts work together with our Advanced Analytics team to make sure our recommendations are solid. Here are the types of analyses we typically conduct for product testing:

1. Penalty Analysis: This tells us how the client can “fix” the product to create higher overall appeal. This analysis connects respondents’ product attribute ratings to the overall appeal of the product rating. By doing this, we can identify where changes to the product would have the greatest impact and/or risk.
2. Opportunity-Risk Analysis: This looks at a model that is built around likers vs. non-likers of the product. This helps us understand which variables are risky to manipulate, and which provide opportunity for improvement without alienating those that like the product. Think of this as a “balancing act.” What do you need to do to the product to convert non-likers to likers—but, at the same time, not lose any of the current likers.
3. Drivers’ Analysis/Regression: This is very familiar to many of us in the research world—this is not specific to product testing, but a good analysis to include. This
   analysis measures the joint effects of general and specific attribute ratings on either purchase interest or appeal (a dependent variable). This allocates the importance of
   separate categories of attributes, such as taste, texture, and appearance.
4. Bootstrap Resampling: This is used in product value engineering to assess the risk of attribute replacement and is an effective way to comprehensively compare
   products when the client’s goal is no difference between products. This analysis uses advanced statistical methods to simulate replications of the product test and measure the risk of replacement. Specifically, this process keeps replicating the sample and then conducts thousands of significance tests, often revealing systematic risk that is not easy to detect in standard statistical tests. Learn more about how we used bootstrapping resampling in a frozen pie IHUT study we conducted.

Product Testing best practices

In the end, the output of a product test will only be as good as the inputs. Here are a few best practices for product testing that we have collected over the years:

• Design the study based on the objectives.
• Include a control or benchmark for comparison.
• Choose the right location when conducting a CLT, based on the needs of the study.
• Quantitatively, choose a sample size large enough to make the differences significant.
• Screen for the right people.
• Utilize a non-disclosure agreement and include all product specs/ingredients to limit liability.
• Don’t recruit people for the study until products are in hand or delivery is confirmed (delays compound issues if a study gets started before the product is in hand and ready to ship).
• Over-recruit to ensure sufficient base sizes (we have standards around quantitative response rates based on experience). We also double-screen, set clear expectations with respondents, confirm addresses through the USPS if shipping, have a help desk set up for questions, and more.
• Limit the number of products you are asking a respondent to provide feedback on (three is usually the
  maximum).
• Product logistics are key. Be sure products are correctly labeled and shipped to the right people—and, if possible, keep them on hand while analyzing the data to ensure the results make sense.
• If testing multiple products, rotate the order to ensure there is not order bias (and, if doing it through an IHUT, mail the products one at a time).
• When conducting product research remotely (and when video is a key data point), consider sending consumers phone stands so they can easily record themselves using the products.
• Always offer an incentive (above and beyond the product—if the respondent can keep it).
• Video deliverables are a great way to help communicate the products’ usability to stakeholders and can help to sell-in the products to retailers. Video can be captured qualitatively or quantitatively.

Product testing offers a wealth of insights but can be complex given the logistical factors and analysis required. Next time you are contemplating how product testing could benefit your business, reach out and let’s chat about the best way to design the research to ensure you get actionable results that drive your business forward.