Mobile Methodology: Where Do We Stand?

The world is witnessing a dramatic shift in terms of how people access the internet. Let's look at some industry statistics on mobile adoption and use:

• Over half (58%) of American adults now have a smartphone. Over 40% now have a tablet and 12% of all media consumption takes place on mobile devices.
• Almost three quarters of shoppers access a mobile phone while shopping and 85% of those accessed the store site to search for information.
• The younger you are, the more likely you are to use a mobile device for search. Of those 18-29 years of age, almost half use their mobile device daily to search. That makes sense given that over 80% of the people in that age group have a smartphone compared to 49% of those 49 to 64 years of age.
• The conversion rates on mobile searches are high. About two thirds of searches done for a product on a mobile device result in a purchase, with men being more likely to make a purchase after a mobile search than women.
• The restaurant industry reports a conversion rate of about 90% when people search for a restaurant on a mobile device.

Smartphone penetration is climbing at an extraordinary rate. Just two years ago, smartphone penetration was 20% lower than it is now. At this rate, in a few short years, an overwhelming majority of Americans will be accessing smartphones on a daily basis and tablets will not be far behind. Mobile will be the standard for internet access. Consumers have already come to expect content delivery to be mindful of their context and customized to integrate seamlessly into the device of their choice. Widespread mobile adoption and use is not just a phenomenon limited to North America. In the second half of 2013, it was estimated that almost a fifth of global internet traffic at any given time was going through mobile devices.

While the smartphone adoption curve has been following closely behind desktop internet access over the past few years in markets like the United States, Japan, and Russia, consumers in China and India have been early adopters of mobile technology. Developing markets are experiencing a "leapfrog" effect where the vast majority of citizens appear to be skipping an entire technological stage of desktop internet access by increasingly going online using mobile devices. Roughly three quarters of the population in China and India already access the internet on mobile devices, leaving just under one-third and less than 10% respectively using desktop access only. Similar trends are also present in specific demographic groups among Americans, the most relevant being the US Hispanic and African American populations, as well as younger generations that have grown up online. Device costs, convenience, context relevance, and the personal nature of the devices are among the top drivers of trends defining mobile adoption.

Since people are changing how they interact with, and access the internet, the market research industry has to adapt as well or we will lose respondents the same way telephone methodologies have taken a back seat to online research over the last 15 years. Mobile devices are already becoming an integral part of mystery shopping studies. Mystery shoppers can use a smartphone to take pictures of the product or service they are evaluating. The GPS features on mobile device can be used to ensure that the agent is where they say they are. They can complete an evaluation using an app, and that information can be ready for analysis in real time, instead of days or weeks later.

Mobile applications are being used for collecting customer satisfaction information. We are starting to see servers bring a mobile device to the table along with the bill. These systems operate in real time and the store manager can be notified of unpleasant experiences before the customer approaches the cash register to pay. Public transportation systems are using smartphone apps to collect rider information in real time as well. These apps improve the quality of information as the experience is still extremely fresh in a person's mind as they provide an evaluation of the service. Do we expect this information to be different from evaluations of experience done in the past? Yes we do. We have always known that recall of an event is sharper (and more accurate) when the time between the event we are measuring and the actual evaluation is minimized.

The impact of mobile can also be seen on our standard general population surveys. About a third of the people who respond to an invitation to complete a survey, come to our servers using a mobile device. It is one of the reasons why we have optimized many of our online surveys for mobile internet access. In our experience, tablets users can access and complete our surveys with ease. Tablets have enough screen space to display questions and answers in the same way it would appear on a regular computer. The ability to shrink and enlarge what you see on a tablet allows the respondent to adjust the size of what they see so that they can complete the questionnaire easily. There are methodological concerns when they adjust the size of images that are being evaluated. On these types of studies we have to evaluate the tradeoffs of standardizing the stimulus across all respondents versus the ease of evaluating the stimulus, or design specific execution frameworks enabled by native mobile apps that exert a very tight control over content scaling by the respondents.

Our research to date has identified two significant issues in optimizing studies for mobile research. The first is that smartphones are used differently than tablets. Smartphones are personal devices and we use them almost everywhere. In fact, it is estimated that almost three quarters of smartphone users take their device into the bathroom with them! Since people access smartphones as they are going about their daily lives, they are willing to complete surveys when they have pockets of time available to them. However, this makes them less tolerant of longer surveys. As a result, questionnaires over ten minutes have significant drop off rates. We recommend the optimal time of about seven minutes for a mobile app that is geared towards smartphones.

The second issue when optimizing studies for mobile research focuses on what you see. The sizes of mobile screen displays limit what we can do with long scales. For example, if you try to fit a ten point scale in an attribute battery going across the screen, the answer bullets run into each other and are barely discernible, or they run off the page, forcing the respondent to scroll back and forth. Additionally, if you have long list of answer categories and run the ten point scale up and down, the answer categories that appear first are significantly more likely to be answered. For example, we recently found on a patient satisfaction survey that the scores shifted by about 5% when we presented a respondent with a ten point scale that rolled off the bottom of the screen. Even though people can scroll up and down, it doesn't mean they actually will! This is a modern variation of something we have dealt with for decades in survey research: the top box phenomenon. In order to mitigate device type bias, we have developed proprietary survey templates that adapt to various devices' screen types and sizes. Respondents have come to expect content to adapt to their device of choice, so research design needs to acknowledge and manage the issues that arise from this. When we control for demographics of respondents and optimize the questionnaire for the internet, the information we gather on a mobile device is very similar to what we gather using any self-administered methodology such as computers or even traditional paper and pencil surveys.

Can we conduct surveys that are reflective of the population using only mobile devices? This is a similar question we faced 15 years ago from clients who wanted to know if we could reliably conduct online research for business decisions as we crossed the threshold between telephone and internet surveys! Polling in the last presidential election showed that online surveys had the most accurate predictions of final voting results and Ipsos' online poll was among the best of the forecasts. We could not do that using only mobile devices today, at least not yet anyway. Our research shows those people who access our surveys using mobile devices are likely to be significantly younger. As many as 60% of new Ipsos panel members in the United States are 18 to 34 years old, and are visiting our websites on mobile devices. Mobile research is helping us fill in the holes in our sampling, especially gaps related to traditionally underrepresented demographics such as youth in general, and older technology adapters.

While mobile research does not yet have the accuracy to predict final vote numbers within a percent of the actual results, it does offer reliable results for a slew of other questions that do not require similar precision. The ability of mobile to give us quick reads that point us in the right direction for doing more critical research is invaluable. In a few short years, we will probably be making predictions on election outcomes with surveys that rely heavily on respondents using their mobile devices.