Embracing Mixed Mode Research

Mixed Mode Research is not a new topic in the world of market research, but it is one that has not yet been readily embraced. The disruptive forces that shaped 2020 have made Mixed Mode research increasingly relevant, especially for social and audience measurement research.

To allow for continuity of research programmes during Covid-19, Ipsos has worked to redesign studies using multiple methods in recruitment and data collection. But Mixed Mode is not only a plan B. It brings a variety of benefits such as improving representativity, response rates, and cost-savings – when managed well.

There is reason to believe that Mixed Mode research will become increasingly popular and considered good practice. In the spirit of future-proofing research studies, this paper outlines;

• Why now is the time to embrace Mixed Mode
• Benefits and best practices
• Ipsos’ “Contactless” Mixed Mode approach
• Examples of studies that have successfully made this methodological transition.

For an introduction to Mixed Mode research, you may find it useful to refer to our previous paper, published in June 2020.

THE ATTRACTION OF MIXED MODE

Inertia, both in physics and in business and policy, is an incredibly strong force. In terms of the latter, decisionmakers may choose to accelerate towards a new approach or remain steadfast in their commitment to an established practice. Many market forces (and not only Covid-19) emerged in 2020 and disrupted the pull of inertia working to either slow or hasten the direction of travel. This moment of disruption provides an opportunity for change that may not have previously been present, and can often produce better outcomes for those involved. The disruptive forces that shaped 2020 have made Mixed Mode research increasingly relevant, especially for social and audience measurement research (where the forces of inertia are particularly strong).

This paper will outline:

• why now is the time to embrace Mixed Mode research;
• how to leverage the benefits of Mixed Mode;
• Mixed Mode best practices;
• why Mixed Mode research helps to future-proof studies, and
• “Contactless” Mixed Mode: Ipsos’ unique approach to safe and effective interviewing during the Covid-19 pandemic.

You will also find cases of Mixed Mode studies, some of which were designed before 2020, and some of which transitioned quickly and effectively to a ‘Plan B’ Mixed Mode approach as a result of Covid-19 – often bringing additional benefits.

WHY IS NOW THE TIME TO MIX MODES?

Mixed Mode is the use of two or more different data recruitment or collection modes (e.g. CATI, CAPI, online and postal) within a single survey, within a country. And it can take various forms, including:

• sequential use of different data collection modes within a survey (commonly seen in longitudinal or academic surveys to maximise response rate), or
• parallel use of different data collection modes, extending to potentially real-time ‘switch mode’.

Mixed Mode Research is not a new topic in the world of market research. In social and audience measurement research, Mixed Mode research has not been readily embraced, with decision-makers instead focusing on the need for the continuity of method, stability, consistency and reliability of data, together with a representative sample. The very public and high-profile nature of decisions taken as a result of the research findings proved persuasive in continuing what was done before, even if it used the highest cost data collection method. This meant that many markets and projects continued to use the ‘gold standard’ (often face-to-face) as it provided the broadest population coverage and was often key to accessing niche or difficult to reach target groups. The decision to continue with singlemode research, even when it combined highest cost and a declining ability to access people on their doorstep, was often made to prevent the disruption of trends, commercial activities and ultimately decision-making. But trends and commercial activities have been disrupted. Before the impact of Covid-19, pragmatists within social and audience measurement research embraced Mixed Mode methodologies as an opportunity to improve representativeness and to capitalise on the cost and convenience of web-based methods. While face-to-face will need to be a part of the mix in many markets, especially for long and complex questionnaires, the representativeness of final samples can be improved by using multiple methods in recruitment and data collection. At an absolute minimum, given the types of disruption of offline research forced by Covid-19, studies should be planned with a Mixed Mode ‘Plan B’ to allow research continuity.

MANAGING MIXED MODE RESEARCH: SOME CHALLENGES

In social and audience measurement research, many efforts to test new approaches to counteract the growing cost of offline methods have been hampered by the very different results generated and the concern that the data produced may be inaccurate or misleading. These differences are the result of mode and/or selection effects. The two can be difficult to untangle, which makes design decisions both challenging and critical. Mode effects need not be a barrier to progress. They can be successfully managed if considered at the survey design stage, and if ‘unimode’ questionnaire design is applied. Selection effects are best managed in large continuous or repeat studies where projects can be run in parallel and differences can be accounted for, and therefore calibrated and controlled. Proper management of mode and selection effects allows the benefits of Mixed Mode to be maximised.

MODE EFFECTS

Different data collection methods can produce data differences. This is the result of survey questions being adapted to suit the mode of research being conducted, changing how they are presented to and interpreted by respondents. For example, questions presented visually to participants in a self-completion setting can be interpreted differently by participants to questions asked in an interviewer-administered environment. As the trend is towards self-administration of questionnaires, either online or even face-to-face, mode effects are increasingly abated.

SELECTION EFFECTS

Selection effects occur when different modes are used for recruitment, as each mode tends to appeal to different groups of people. For example, online recruitment generally appeals to younger, highereducated respondents than do offline methods. A Mixed Mode survey may therefore have different sample compositions via selection effects.

UNIMODE QUESTIONNAIRE

An unimode questionnaire ensures a questionnaire is consistent across data collection modes meaning that all respondents are presented with the same question and response categories for the meaning and intent of the question and response options to stay consistent. Questions or instructions sometimes need to be modified to be understood by respondents the same way across different modes, but any such variation is kept to a minimum.

CASE STUDY: AUDIENCE MEASUREMENT

To refresh and improve the capability of an audience measurement programme, Ipsos supported the implementation of a Mixed Mode design across sampling, staffing/resourcing and study launch. This was adapted to the realities of today and future-proofed for tomorrow. The Mixed Mode design is inclusive and relevant, providing better coverage of target populations and taking into account characteristics of specific demographics in the market. It is proven to work in readership studies across the world (Canada, UK, Hong Kong, Germany, Mexico and Australia). The same approach is used for our national representative studies and regional representative work such as election polling. 12 months in, overall field progress for readership is sound and meeting targets for both online and CATI across key demographics.

BEFORE:

• Different modes (telephone, IVR, access panel, etc.) were used to target different audiences. Demographic differences corrected post-field.
• Cash incentives on ‘per complete’ basis.
• Gaps in coverage across age, gender and race/ ethnicity.
• Declining landline usage: Nearly half of homes across the nation use a cell phone as their primary phone line and don’t own a landline.

THE NEW MIXED MODE RECIPE:

• Nationally representative sample for 75% of the total recruits, holding back 25% for “rotating sample boost” in major markets.
• Weekly targeted completes of a readership survey.
• Balancing age, gender, household income and market.
• “Dual Frame Random Digit Dialling Design” using 65% cell phones and 35% landline.

MAXIMISING THE BENEFITS OF MIXED MODE

Mixed Mode brings a variety of benefits that span access to people, measurement and response rates. Implementing a Mixed Mode research design can either be introduced into a study as part of participant recruitment or as part of data collection. With recruitment, the increased population reach from incorporating multiple modes can improve research quality and/ or representativeness by accessing participants who would otherwise be excluded from a survey, especially in markets where online penetration is limited. When it comes to data collection, Mixed Mode has the potential to reduce fieldwork costs by conducting a significant proportion of the study through low-cost modes (e.g. online), and only high cost modes (e.g. face-to-face and telephone) where absolutely necessary. Furthermore, when a sequential mixed mode approach to data collection is taken, response rates can be significantly improved (and non-response error mitigated). This works by following the initial data collection mode with another mode. For example, push-to-web designs send a letter to participants inviting them to complete an online survey, followed by a reminder mailing with a paper questionnaire to help improve response rates. When establishing a new continuous or longitudinal survey, a Mixed Mode approach is a forward-looking approach. The key benefit is that studies are not limited to high-cost data collection approaches that are subject to falling response rates (see figure 1). Furthermore, as the world’s economy is now mobile and digital, including a digital component in a Mixed Mode study not only reduces costs and supports access, but also makes the study future-proof.

ACHIEVING THE SYNERGY OF A MIXED MODE DESIGN

Create a unified questionnaire structure to minimise mode effects. Ensure the least expensive data collection modes are used first to minimise costs. Incorporate a sequential Mixed Mode approach to improve response rates. If an online data collection mode is incorporated, ensure that the questionnaire is designed to be mobile first. Design your study in consultation with Ipsos’ Mixed Mode research and operations experts.

That said, for ongoing continuous and longitudinal studies it is important to assume a pragmatic approach to unlock the benefits of Mixed Mode. One opportunity is to integrate complementary modes into an existing study in a way that adds additional benefits or insights, keeping data integrity intact. Examples of how Mixed Mode could add to an existing study include:

Increasing sample sizes at a more cost-effective rate.
Adding a type of participant or target for whom face-to-face may not be the optimum method due to availability (presence), or convenience (face-to-face interviews are rarely requested at optimal times).

A benefit that may not be instantly evident is that Mixed Mode is one of the best overall designs to cope with the disjuncture between what researchers would like participants to do and what participants themselves are willing to do. As there is a wide gap between technological advancement/ change and human adaptation to this change, there is a need to understand what stage participants are at and provide the appropriate modes to support their response to studies. In other words, while there may be the desire to leverage newer technologies quickly in research, it is important to understand respondent willingness and comfort levels as they participate in newer approaches.

PUSH-TO-WEB

Push-to-web surveys, also known as “web-first” or “web-push” surveys are increasingly being used in social research where random probability samples are still required, making online research through opt-in panels inappropriate. A push-to-web design leverages a sequential Mixed Mode approach whereby an offline data collection mode is used to encourage participants to complete an online survey. After one or two reminders, this may be followed with an offer to complete a paper questionnaire. The potential cost savings, (and recognition that lower response rates don’t necessarily mean lower quality data), has resulted in governments adopting the approach for census collection (e.g. Australia, Canada and Japan) and other policy research (e.g. the United States). Irrespective of the research topic, “push-to-web” research will be more successful if the participants trust the research sponsor. Ipsos’ global breadth and local depth, in terms of ability to reach people, often in partnership with a government agency, provides the trust and reputation necessary to properly achieve the necessary data and insights.

CASE STUDY: EUROPEAN WORKPLACE SURVEY

A flagship pan-EU representative sample survey of business establishments with at least 10 employees collects data on workplace practices with regard to work organisation, human resource management, skills use, skills strategies, digitalisation, direct employee participation and social dialogue. All previous surveys had been conducted via CATI (computer assisted telephone interviewing), which is the standard approach for business surveys, given the need to identify suitable interviewees with company gatekeepers. Ipsos came on board to optimise data quality and update and future-proof the survey through online data collection, specifically using push-to-web recruitment. This involved telephone contact to identify the survey participants who were then asked to complete the interview online. The method ultimately proved successful, despite concerns about conversion rate. The response rate was lower than a CATI-only approach (as expected), but analysis of non-response bias suggests this was largely controlled by the weighting, and Ipsos was able to deliver an online survey in all countries.

LESSONS FROM THE COVID-19 PANDEMIC

As we have said, significant disruption always marks an opportunity for significant change. To ensure the safety of participants, clients and staff during the pandemic, Ipsos introduced Contactless Mixed Mode to support data collection that was traditionally captured face-to-face. While many ongoing programmes were halted, many continued, requiring study designs to be adapted fast and at global scale. There was no single recipe for successful transition from single-mode research to Mixed Mode research. Some studies and programmes experienced a move towards push-toweb, some to CATI@home interviews (using the original face-to-face interviewing teams), and some a combination of mail preparation with CATI interviewing or CATI@home and WhatsApp interviewing. Given the significant disruption to trends, some programmes fully embraced online interviewing, recognizing that post-pandemic there is an opportunity to rebuild in the most robust yet cost efficient Mixed Mode approach (see forthcoming case studies). For those programmes that were paused and not restarted, the trends are broken, and a new reality has emerged. Invariably, as most will start over, Mixed Mode will now become an increasingly large part of the conversation for social research and audience measurement, especially given the increased costs and decreased access to face-to-face.

COVID-RESILIENT FACE-TO-FACE AND CHANGING MODES

1. A GLOBAL WOMEN’S EDUCATION AND TRAINING PROGRAMME

There was time pressure for a baseline survey to be completed in one location to enable full implementation across the multi-country project. The adapted design allowed insights to be delivered across target groups for a comprehensive baseline study. The team responded quickly, remaining agile and adaptable to meet the study requirements. Conducting the study remotely had an added benefit of increasing the profitability of the project.

THE INITIAL RESEARCH PLAN

In-home methods including:

• 1.700 CAPI in four states with target group in intervention and control areas.
• 185 F2F stakeholder interviews.
• 48 Focus Group discussions with girls and women.

THE COVID-19 ADJUSTED RESEARCH PLAN

• Database generated for interviews in control locations.
• Questionnaire development adapted to telephone interviews.
• 1.100 telephone iField interviews pre-survey for feasibility.
• 155 stakeholder in-depth interviews and discussion through WhatsApp.
• In-depth telephone interviews.

2. LABOUR FORCE SURVEY

The challenge was to ensure continuity in a highly important public interest survey measuring employment and economic conditions of the population, used at national and regional level to define social and economic policies. The fieldwork was carried out 100% during lockdown, leading to the current Mixed Mode technique. The transition to telephone interview worked well, with good quality and productivity that allowed the planned full sample to be completed.

THE INITIAL RESEARCH PLAN In-home, one visit CAPI study:

• 45-minutes interview.
• 8.500 interviews per month.
• 1.100 sample points.
• +400 F2F interviewers in field.

THE COVID-19 ADJUSTED RESEARCH PLAN

• Limited stop to data collection while reengineering the process.
• Switch to telephone interview performed by the same in-person interviewer’s network.
• Interviewer re-training.
• Same questionnaire as F2F.

3. LONGITUDINAL HEALTHCARE STUDY

Currently in its tenth year, Ipsos was tasked with ensuring continuity of a large-scale prospective longitudinal study of children and young people in out-of-home care throughout the Coronavirus crisis.

THE INITIAL RESEARCH PLAN

• 900 in-home interviews with a cohort of children/young people and their caregivers.
• 30-minute Audio-Computer Assisted Self- Interview with children/young people plus 1hr 45-minute caregiver interview administered via CAPI.

THE COVID-19 ADJUSTED RESEARCH PLAN

• Data collection stopped while the methodology was re-designed.
• Questionnaire redesigned for telephone interviewing.
• Additional questions on Covid-19 integrated.
• Approval of revised approach from ethics committee.
• Pre-mailing process set-up to administer incentives and provide documentation the children/young people and caregivers would otherwise have received during the interview.
• Re-training of F2F interviewers.
• Same interviewing team used for participants to ensure consistency across waves.

The value of the study relies on its 24-month interval between the interviews with children and young people. Had interviewing stopped mid-fieldwork, the study cohort interviewed this wave would have been vastly reduced. The team was quick to respond and adapt the design to facilitate the continuity of data collection and maintain the integrity of the longitudinal data.

THE FUTURE OF MIXED MODE

Moving beyond the pandemic, there is reason to believe that Mixed Mode research will become increasingly popular and considered good practice - both for recruiting participants and collecting data. It is possible that post-pandemic more people may be reluctant to complete studies face-to-face, especially if it involves inviting people into their home, and in many markets, telephone access is neither representative nor feasible. However, many of these people would be happy to participate in ways which are less disruptive than completing a survey live with an interviewer – such as completing questionnaires in their own time, online, or by telephone appointment. Ipsos is uniquely placed to support clients to unlock the benefits Mixed Mode through its strength across all data collection modes and strong geographical coverage. A leading mode-agnostic position in data collection, coupled with social research and audience measurement expertise, ensures the most appropriate Mixed Mode design can be incorporated into each study. Furthermore, Ipsos continues to innovate in and explore new data collection modes that intersect with peoples’ behaviours and needs. Irrespective of whether data collection is conducted via ‘Chat’ or ‘Voice’, our approach to Mixed Mode will continue to integrate multiple sources of data in a coherent way that allows for actionable decision-making.