Towards more agile and efficient product testing

The historic assumption that larger sample sizes are needed for product tests derives from the hypothesis that there is a risk of unreliable and varying consumer responses, i.e., variance. Today, large consumer panels and databasing of consumer responses allow us to revisit historic variance assumptions for product development.

In this paper we investigate the variance of products tested across regions and categories using different scales from 36,500 consumers in our database. We assess how reliable a smaller sample size of n=50 is versus larger samples of n=150 or more when testing products in the early stage and later stage of the product development life cycle..

Our findings suggest small sample sizes can be considered when the objective is to create differentiating products such as for pre-screening of prototypes. Such pre-screening allows us to save cost and time by reducing the number of products required for further testing. However, for other types of testing such as cost rationalization studies, or when subgroups need to be analyzed, larger sample sizes are recommended.