The power of research panels
Accessing the right respondents at the right time is the foundation of effective market research. A technology-driven evolution of data collection methods has resulted in the emergence of different channels to access participants – particularly when it means costs can be reduced.
Not all research companies choose to own and operate proprietary research panels, but Ipsos chooses to continually invest in panels because they deliver a consistent quality advantage over other sources.
The success of high-quality panel research is built on five main pillars:
- Sample Stability
- Rigorous Sampling Process
- Quality Respondents
- Respondent Engagement
- Respondent Trust
In this paper, we explore these five aspects in more detail comparing panel and non-panel sources. Also find out about new developments, accelerated by the COVID-19 pandemic, which include carrying out high-engagement qualitative research online.
Ipsos believes that the future of market research insights will continue to depend on access to properly managed research panels.
Accessing the right respondents at the right time is the foundation of effective market research – it has been so since the inception of the discipline. Data collection for market research activities has experienced a cumulative evolution, driven by technological transformations. Technological advancements not only change people’s habits and expectations, but they have also resulted in the emergence of different channels to access research participants – particularly when it means costs can be reduced. Market research began with face-to-face (FTF) interviews using paper and pencil. Telephone interviewing preceded the ability to conduct computer assisted telephone interviews (CATI), which became more prevalent, as did computer assisted personal interviewing (CAPI). Today, in developed markets (and increasingly in developing markets) more and more surveys are conducted online, both through web and mobile access routes. And as internet penetration increases globally, the online and mobile channel is becoming increasingly important. Although FTF and CATI will always exist as the best-in-class data collection method for specific studies and programmes, online access and engagement are a significant focus for the future of market research.1 The online sample market as we know it today is around 20 years old and is estimated to be worth $1.5 billion. In the market research industry, there is an ongoing proliferation of “insights providers”, including traditional research firms, technology companies, and small boutique agencies. But not all of these have the resources or the need to manage their own panels of respondents.
WHAT ARE ACCESS PANELS?
Online Access Panels are groups of internet users who have agreed to take part in online market research surveys. Upon registration, the panel members provide contact details and demographic information about themselves which allows for a better sample selection. Furthermore, the rapid evolution of market research fieldwork has required researchers to either choose a preferred methodology or to leave that decision to someone else – typically the supplier of that fieldwork. But the way in which market research fieldwork is conducted has a significant impact on the ultimate quality and reliability of that research. Understanding the choices available and the implications they have is therefore paramount. Online access research panels are hard to build and maintain, but easy to use. Ipsos has invested in a global network of online access research panels (hereto referred to as research panels) for the past 20 years to deliver the highest standard of research insights together with the highest quality. There are five main pillars to achieving this standard, which this paper will address:
- Sample Stability
- Rigorous Sampling Process
- Quality Respondents
- Respondent Engagement
- Respondent Trust
SAMPLE STABILITY
Simple, single-origin survey designs are most desirable for ensuring research quality. Ipsos has tried and tested several different methods for sourcing respondents, both internally and externally. Experience demonstrates that many factors have an impact on the characteristics and quality of the sample, from how respondents are recruited to how they are incentivized and allocated for a study. By understanding the dynamics of these potential sources of bias, we can know how to control them, ensuring sample stability. Stability is vital in fieldwork as the goal of many research projects is to identify changes in consumer behaviour. The best way to isolate behavioural changes is to minimize the “noise” that can be caused by changes in sample. This is one of the reasons that Ipsos operates its own research panels. These panels recruit respondents from a wide variety of sources, and panelists are offered a points-based incentive program designed to keep them engaged long-term. Technological advancements have enabled a shift from panels to other sources of people research fieldwork. Market research technology is derived from Advertising Technology as a means for companies to quickly monetize all types of audiences or databases. Audiences are created when people purchase things or sign up for services, resulting in clear biases. This means that certain audiences do better than others when presented with the opportunity to take a survey.
DEFINITION: SAMPLE
When we refer to the ‘sample’ of a given study, we mean the subset of the population or universe of interest which is interviewed. Typically, the sample is pulled using random methods so that everyone has an equal chance of inclusion.
DEFINITION: SAMPLE EXCHANGES & AGGREGATORS
Sample exchanges and/or aggregators bring together buyers and sellers of sample/respondents and automate the interaction between them, respondents and surveys. They are generally considered to be uncontrolled sample sources but controls can be applied if needed. Exchange sample is simply a compilation of audiences that are rolled together to form a respondent source for surveys. Exchanges and sample aggregators use their technology to mix traffic from these various non-panel sources. This results in a constant shifting of respondent traffic sources with different incentives offered for survey completes, resulting in limited sample stability. A recent Research-on-Research programme by Ipsos analysed multiple sample sources to test the sample stability that panel sources and non-panel sources provided in comparison to one another. Key findings include:
1 THERE WERE SIGNIFICANT EFFECTS BASED ON SOURCE
MANOVA analyses showed significant differences on most measures, including key KPIs.
2 THERE WERE DIFFERENCES BY SOURCE.
Multiple comparisons tests demonstrated that some non-panel sources contributed more significant differences than panel sources.
3 KPIS WERE SIGNIFICANTLY HIGHER AMONGST NON-PANEL SOURCES
The elevated KPIs means that the results from non-panel sources were above what is expected on average and would therefore result in false positives Another reason that research panels provide stability is that they are not normally used by “professional survey takers”. Through proper panel management, it is easy to control how many surveys a panelist can complete in a given time period. Given non-panel sources do not provide such controls, many non-panel audiences complete far more surveys than panelists do. To highlight that non-panel sources are often the source of professional respondents, we found that over a one-week period in the US, i-Say panelists (Ipsos’ proprietary access panel) arrived at an Ipsos study via 0.4 additional sample sources, compared to up to 1.5 additional sources for non-panel sources.
DEFINTION: PROFESSIONAL RESPONDENT
A person who is familiar with interviewing practices may use this knowledge to secure qualification in a particular study, whentheir real profile mayotherwise disqualify them. Their views, opinions, and ultimately surveyanswer choices may therefore not represent those of the demographic or profile the study wishes to access. These “professional” respondents can compromise the research quality.
RIGOROUS SAMPLING PROCESS
Sampling is the process of how individuals are selected or allocated from a sample source to a survey. To improve sample quality, panel providers employ systematic approaches based on market research fundamentals. Sampling is a long-established way to ensure quality research fieldwork. When operating a research panel, a rigorous sampling process is followed, which involves identifying a sample universe and making exclusions to eliminate those who are deemed ineligible. A sample is then selected from that universe according to the balancing and quota requirements of a study. Ipsos uses a proprietary sampling application to construct complex samples based on target and screening requirements to choose respondents that balance according to the targets.
Exchange sample or other non-panel sources don’t have the same rigor as research panel sources. They direct web traffic to the study based upon what is known about the definition of the type of people required and open quotas. While non-panel approaches can be quicker and more efficient, these shortcuts can lead to data quality concerns within a study. That said, there is a place for respondents from non-panel sources; they can be useful for a quick directional read on an issue, but it is important to note that the data may not hold up under scrutiny as well as research panel data.
QUALITY RESPONDENTS
Fraud is a real threat in market research, and it targets panel and non-panel sources alike. To mitigate the risk of fraud, a well-managed panel deploys numerous preventative measures including technology such as AI algorithms and quality experts. Panelist performance is assessed over time and new recruits are constantly evaluated – similar to the way a new joiner would be reviewed during their probation period with a company. It is also possible to complete a more in-depth profile of in-survey and cross-survey activity. Only if new panelists perform well are they invited to participate in higher value, more complex research tasks that would typically attract fraudulent respondents. When sourcing from across the sample market, it is important to make choices based on expertise and experience as to which sources are highest quality and reliable. Continuously evaluating vendors on a variety of in-survey quality measures is essential to ensuring continued quality answers. There are no such controls with non-panel sources, resulting in frequent and/or professional respondents. Ultimately, panels focus on ensuring that fraud via bots or click farms cannot occur.
DEFINITIONS: BOTS AND CLICK FARMS BOTS
Autonomous internet or network-based programs that interact with systems to automatically complete surveys.
CLICK “FARMS”
Organized/coordinated efforts to complete surveys on a large scale for rewards, creating meaningless data. These click farms have been involved in recent incidents with Facebook “likes farming” (generating false clicks or likes for a fee).
Irrespective of source, most of the time survey respondents do not qualify for the surveys they are attempting to complete. With panels, there is roughly a 50% higher chance that a panelist will qualify for a survey as more is known about them than non-panel companies know about their respondents. Over time, panelists are encouraged to complete a variety of profiles to enable the pre-identification of surveys for which there is a greater chance they can qualify. This also minimises the risk of fraudulent respondents or bots from accessing surveys. For example, if a client is seeking owners of convertible sports cars, a research panel will have that information readily available as panelists will have completed an automotive profile as part of their registration. Non-panel sample would have to be screened for convertible sports car ownership, and while best practice is to disguise the topic of the survey, many non-panel respondents may overstate qualifications to attempt to be accepted into a study (see figure 3). This is fraudulent behaviour and means that respondents will not meaningfully engage with the survey.
RESPONDENT ENGAGEMENT
The shift in traditionally offline research being conducted via online channels has been underway for years, but only recently has high-engagement research (requiring significant time commitment or completion of multiple activities) started to move online too. The COVID-19 pandemic has only accelerated the desire to complete high-engagement research online. One example of the accelerating shift is the move to online qualitative research. This is an interesting growth area as it was originally thought that online respondents would not be as eager or as expressive as offline respondents (see figure 4). We find that research panel members often find online qualitative discussions engaging and are eager to participate, whereas respondents from non-panel sources are often less expressive and less welling to participate. Panelists are more likely to engage in the research programme with full interest and engagement, ensuring clients achieve the insights they require. A significant proportion of Product Usage Testing is still completed offline, but online in-home use tests (IHUTS) are becoming increasingly important in some markets. Irrespective of the data collection mode, both clients and respondents must significantly invest in Product Usage tests, either through the provision of product and its shipment or through ongoing respondent engagement at recruitment, usage and the product review stage. For online in-home use tests, experience demonstrates that only well-managed research panels can meet the necessary benchmark for acceptable participation throughout the process (see figure 5). We have also seen a shift from surveys being completed only on personal computers to being predominantly completed on mobile devices4. This is important as mobile devices are now truly the centre of a person’s digital life.Mobile devices also offer additional rich data sources including GPS, eCommerce, health and exercise, media, and internet search information, which add value when linked to survey data. Capturing this data requires engaged panelists who are willing to share such personal information. Ipsos’ panel application strategy allows all data points to be collected in one convenient place, with the appropriate explicit permissions and the compensation channels set up to reward panel members for access.
RESPONDENT TRUST
In online research, as is the case everywhere, trust is earned. Ipsos is one of the largest and best-known research companies in the world. This is advantageous to operating research panels as people, and ultimately panelists, trust the Ipsos brand. But trust is also gained through the contract panels have with panelists. Panelists do not need to be concerned that their data will be sold to third parties or about any breach of privacy as they agree to terms and conditions reassuring them that their information and answers will be only be used for research purposes, will be fully anonymized (unless explicit consent is received), and will be deleted after a period of time. Trust is key when considering the future of market research. There is increasing interest in bringing together attitudinal and behavioural datasets, and while behavioural data of all types is widely available for digital advertising, in most cases it is anonymized and can only be tied to attitudinal data with permission. Receiving permission to enable these connections requires trust, and this is more likely to be gained through panel sources.
USING PANELS FOR REAL-TIME QUANT+QUAL RESEARCH
Research that is traditionally conducted offline is gradually transitioning online - and the COVID-19 pandemic has accelerated this transition. Moving real-time research
(including qualitative research) online has been possible for years, but reservations about participation rates and overall quality of participants has held it back. The push towards resilient methods has forced these concerns to be put aside. Regardless of market, Ipsos panelists have embraced participating in real-time research, surpassing participation rates and ensuring exceptional quality to deliver the insights required. Below we set out an example schedule of an international panel-based study.
REAL-TIME INTEGRATED QUANT+QUAL EVENTS
Our real-time integrated quantitative and qualitative events combine recruitment of 100+ participants 5-7 days before with 30-minute instant questions among 50-100 people and 60-minute instant chat sessions via group video discussions with 4-6 participants. Live observation and interaction provides instant answers and real-time reporting with instant AI processing.
CONCLUSION
Not all research companies choose to own and operate proprietary research panels. However, Ipsos chooses to continually invest in and develop new panels because they deliver a consistent quality advantage over other sources. While the availability of research panels has diminished over the last few years in favour of lower-priced and sometimes more immediate traffic sources, Ipsos still prioritizes its own research panels. This is not the least expensive way to source research participants, but we believe it is the best way because it ensures quality answers, which produce quality data, in turn meaning that Ipsos can provide quality information and insights to clients. It is clear that the future of quality insights will continue to depend on access to properly managed research panels.
