Product Testing: Handle with Care

Product testing plays an important role in both the innovation process for new products and the brand management of existing products. A great new idea or brand repositioning is doomed to failure unless the product delivers on the concept. Product testing can provide the answers to important marketing questions, including:

• Which product formulation is the best among my alternatives?
• How well does my best formula stack up against competition?
• Does my product fit with my concept?
• How much sales revenue will this product generate?
• Will a lower-cost formula perform as well as the product currently on the market?
• Will a product improvement attract new customers without alienating current customers?
• How can my product formulation be optimized?

Answers to these critical questions will steer the course of product development and determine the success or failure of the product in the marketplace.

The best approach to obtain maximum learning from your product testing is to design a product testing system. Three benefits of a system are: (1) consistency of approach in terms of sample size, confidence range, methodology options, questionnaire design, analysis, and report format; (2) the ability to generate a normative database on key measures repeated from study to study; and (3) the opportunity to conduct meta analysis across product tests to understand the product elements that drive ratings or preference.

The design of a product test must be handled with care. The methodology employed depends on the objectives of the study (i.e., the marketing questions listed above). A key consideration is whether the test will be monadic, paired comparison, proto-monadic, sequential monadic, or multi-product (three or more products). Another important factor is whether respondents will be exposed to blinded test product or identified test product.

After decades of conducting product tests for Fortune 500 consumer packaged goods companies, the Consumer Products Division of Ipsos Insight has established guidelines to ensure that product tests are designed correctly.

Monadic Versus Paired Product Tests

There are two general types of product tests: monadic and paired, which includes paired comparison, proto-monadic, and sequential monadic tests. Another type of product test is multi-product (three or more products). These tests are very useful when the goal is to optimize a product by determining the product elements that drive performance. Multi-product tests require product formulations based on an experimental design of product elements. As such, the design of multi-product tests is quite different from monadic or paired tests. Our subsequent discussion focuses only on monadic and paired tests, which are described below.

Monadic Test

Each respondent tries one product and answers one survey relevant to that product.

Paired Comparison

Each respondent tries two products and then answers one survey, which includes questions asking the respondent to compare the two products.

Proto-Monadic

Each respondent tries two products, and answers two surveys. Specifically, each respondent tests the first product and answers a monadic survey and then tests the second product and answers a comparative survey. The order of products tested is rotated among the respondents. By comparing monadic results between cells, you will learn whether the differences between the products are meaningful when tested separately; by pooling comparative results across cells, you will learn which product is preferred.

Sequential Monadic

Each respondent tries two products and answers two surveys, which are consecutive monadic surveys. Specifically, each respondent tests one product and then answers a monadic survey. Next, each respondent tests a second product and then answers a second monadic survey. Preference questions are asked at the end of the second survey. The first product is evaluated without knowledge that a second product will follow. Sequential monadic tests are used to obtain both monadic and comparative ratings. However, once the first product is tested, the ratings of the second product are no longer monadic and can be difficult to interpret. For this reason, we recommend proto-monadic tests over sequential monadic tests whenever possible.

The monadic and paired methods differ from one another largely based on two issues: validity and sensitivity. Monadic testing offers greater validity as consumers use only one product at a time as they would in the real world. Paired testing offers greater sensitivity, as consumers are exposed to two separate stimuli, and using products one after the other magnifies differences. So, how does one decide which design to apply?

Generally, monadic product tests should be used when there are readily discernible product differences. This scenario typically occurs during product development when the issue is whether consumers like the product or hate it. Monadic tests are also very suitable for testing innovative products (for which no benchmark or competition exists) and for testing products having long purchase cycles (which require a long usage period making it hard to compare products).

When the variations between alternative formulas are minor, monadic tests are unlikely to pick up differences without large base sizes and, consequently, high costs. In these cases, paired comparison, proto-monadic, or sequential monadic tests are recommended because they provide better discrimination.

This scenario occurs when testing product reformulations, in which there would be only subtle differences among the test alternatives or versus competition. Paired tests are more feasible when the products tested have a short usage cycle, or when they would normally be used almost simultaneously.

Many clients prefer proto-monadic/sequential monadic designs, in order to provide both sensitivity and validity in the same test. Although these paired tests are usually less expensive than monadic tests, they usually take longer to execute.

Blinded Versus Identified Product Tests

Respondents in a product test are exposed to either blinded or identified test product. In blind tests, the test product is disguised so that the tester--often a brand user--cannot readily identify it. The choice of blind versus identified depends on the stage of product development and how the results will be used.

When to Use Blinded Product Tests

Test product should be blinded when the purpose is to compare product formulations. When blinded product is used, no brand expectations exist; hence, the performance qualities of each formulation being tested are magnified. Blind testing should be conducted during early stages of product development or during restages.

Product tests should be designed to use blinded test product when the manufacturer has the following objectives:

• What formulation leads to optimal perceptions?
• How do the product components interact? Is there synergy?
• Which attributes drive overall product performance/consumer liking?
• How does my test product compare to my current product based on formulation alone?
• How does my test product compare to my competitors' products based on formulation alone?
• If a more expensive formulation is used, will consumers notice a difference?
• If a less expensive formulation is used, will consumers notice a difference?

Before respondents are exposed to blinded product, the product should be labeled with neutral letters and/or numbers. It is important not to use labels like A or B, or 10 or 100, as there is an implied ranking to these. Labels like J38 and K23 should be used.

When analyzing the results of a blind product test, it is important to recognize that differences between products are typically overstated. This overstatement is acceptable since the goal is usually to maximize the opportunity to see differences. However, if differences are not observed in a blind test, then it is not likely that differences will be observed during an identified test - or in the real world. Even if differences are observed in a blind test, they still may not be recognized in the real world. For that reason, we recommend following up a blind test with an identified test.

When to Use Identified Product Test

When respondents are exposed to an identified product, they bring to the test the same expectations they would have when using the brand in the real world. Thus, identified tests reflect real-world evaluation of product performance.

A product test should be designed as an identified test when the goal is to measure product performance taking into account the brand name. For example, when manufacturers implement a formula change, they often simultaneously reposition the brand. A concept/product test is then appropriate for determining whether the reformulated product delivers based on the new positioning. Identified product is used in this type of test, which is typically done monadically.

In keeping with the attempt to reflect real-world conditions, identified testing should be employed in the natural settings of respondents' own homes, with a usage period long enough to allow repeated opportunities to use the product. For new-to-the world consumer packaged goods, if repeat usage is uncertain, the usage period may be extended for a long period. For example, some products may be used several times before they are discarded, wear out, or are used up.

It is also important to monitor use over a long enough time to evaluate when performance declines. Other times it may be of interest to see exactly how respondents use the product, specifically, how often, for what occasions, and how much is used on each occasion. In-home testing allows for these scenarios.

When analyzing the attribute ratings of an identified product test, it is important to recognize the halo effect of recognizable brands. A popular brand will score higher than a less popular or less-known brand among a general population sample. These differences can be reduced by analyzing results within brand user groups. It is especially important to evaluate identified product tests among brand users and competitive users to see if new formulations appeal to new users or alienate current users. When you have these halo effects, it is more difficult to disentangle which attributes are driving product performance. In these cases, we recommend applying more advanced statistical techniques, such as ridge regression or Shapley Value analysis. Still, identified tests provide an evaluation of the total product offering.

Product tests must be designed with care. It is critical to correctly identify the business objectives of a study so that a test can be designed which correctly focuses on validity versus sensitivity - or a combination of both. It is also important to determine whether the brand should influence the results, and to analyze the data carefully and deeply when the brand does have an influence. In general, blind product tests are used to understand product performance, and, consequently, they are often used in paired tests (e.g., tests comparing the test product to the current product or competition). Identified product tests are used more for marketing purposes and are typically used in monadic tests (e.g., concept/product tests to understand the concept/product fit).