The latest news from the meaning blog

Smartphone surveys – the rule of thumb

Smartphone surveys – the rule of thumb

It’s two years since I presented a paper at ASC’s international conference in Bristol about mobile survey technology. According to Twitter, it came in for a few namechecks at last night’s MRS Live event on smartphone surveys, in which Google’s Mario Callegaro was presenting. My 2011 paper seems to have earned the name “the paper on mobile apps”, which is due, as Mario no doubt said last night too, because very little actual research has been done on survey techniques and methodology. But the paper covers much more than that.

The paper was based on a two-step survey looking at mobile self-completion methods and tools. First, I spoke with mobile survey practitioners and did a lit search to see what kinds of issues people had mentioned with respect to mobile research, and from this came up with a list of make-or-break capabilities needed to deal with these issues. I  then I contacted all the main software vendors to see if they offered that kind of support. Because there are, in essence, two ways you can do research on a smartphone – using a dedicated app, or pushing the survey to the phone’s built in browser – I covered both.

“Smartphone surveys” or mobile research still seems to polarize the research community into advocates on the one hand, and skeptics and agnostics on the other. But our annual technology survey among research companies shows that respondents are voting with their thumbs, judging by those now attempting to take online surveys on mobile devices. Across participants reporting this data, it averaged 6.7% in 2011. This had jumped to 13.1% in 2012. We’re just about to repeat the question in the 2013 survey. Anyone want to hazard a guess?

Like it or not, mobile is here to stay – it time to start looking after our new little brother because he’s growing up fast.



Mobile goes plural

“There’s a fault line running through mobile research technology that makes it not just a disruptive technology but one that is fragmenting research approaches. Forget mobile technology – we need to think ‘mobile technologies’”, says Tim Macer in an article that appears in the May 2013 issue of ESOMAR’s Research World.

In the article, he interviewed four leading developers working in very distinct areas of mobile research technology. He discusses their different approaches, and summarises the pros and cons of using a dedicated app or surveys modified to work within the browser of a mobile device.



Mobile research growing up fast

Hands holding out a collection of mobile phonesGlobalpark, organisers of the 2011 Mobile Research Conference asked me to chair day two of the event. I decided, rather ambitiously, to close the conference with a round-up of all the presentations that day. Here, in prose, is what I verbalised at the end of long day of very interesting presentations.

Don’t be surprised if you don’t recognise many of the names here. It is true to say that the early adopters of this method are not necessarily the usual suspects – there were some familiar firms present – but as the industry as a whole continues to see only problems with mobile research, it was illuminating to hear from those who are not only convinced of the value of mobile research, but are developing expertise, best practice and clients hungry for more.

Though organised by Globalpark, the event was software vendor agnostic, with examples from rival software providers presented too. In a few sentences for each session, here is what came up:

Bruce Hoang (Orange Advertising Network) – presented a multi-country study of mobile media consumption by mobile data users in the UK, France, Spain and Poland – countries with marked differences is adoption and usage. He has concluded that Web-optimised sites are more popular with consumers in mature markets than using apps to access content. “The web browser in the mobile device is the killer app” according to Bruce. He advocates sticking to web browser-based surveys on the mobile as it most closely aligns with respondents’ preferences and experiences.

Guy Rolfe (Kantar Operations) – asserted that Mobile apps for surveys definitely have their place. Kantar find participants are willing to download survey apps, which can enrich their survey experience. In parallel with this, many consumer product manufacturers and retailers are now creating lifestyle apps that capture a lot of useful data which are proving to be very popular with consumers – they don’t have a research purpose at their heart but the data they collect they could be very useful to researchers, if they can get their hands on it.

Jeroen de Rooij (ValueWait) – presented a lifestyle case study that proved it possible to use mobile research to ask 60-question-long surveys, with modest incentives, if you do it with care. The survey also asked respondents to email in pictures after completing the survey, and a very high proportion were willing to go to this effort.

Peter Lynn (University of Essex) – explained that from a social science and statistical perspective, the focus of scientific survey literature has tended to emphasise the negative – seeing mobile samples as a problem. This needs to be questioned. If you take a Total Quality perspective, there are many areas in which mobile samples are no better or worse than others – coverage may be better. There is also a one to one relationship between respondent and phone unlike landlines. Other sources of error are reduced, e.g. people are more willing to answer some kinds of questions, it avoids recall error, being in the moment, and overall, the responses are not otherwise fundamentally different from other modes. It’s strength surely lies in complementing other modes.

Michael Bosnjak (Free University of Bozen-Bolzano) and Sven Scherrer (Globalpark) – took us through some early results of study on how useful voice capture and voice recognition might be in overcoming the Achilles’ Heel of mobile research – capturing lengthy verbatim responses. Low response and high drop-off is often observed in mobile surveys when these questions get asked. The study pitched standard touch-screen entry with voice capture and voice recognition. From the preliminary results presented, voice did not come out well from a respondent perception point of view. Touch screen entry was preferred over voice entry – voice recognition was the least preferred and the spread of responses indicated a divergence of opinions here. Interestingly, respondents seemed to warm to those methods, particularly voice capture, when asked about it 5 days later. The actual effect on the data has not been measured yet – those results are due out soon.

Justin Bailey (The Nielsen Company) and Sean Conry (Techneos) — presented a case study on using BlackBerry Curve devices with a recruited panel of South Africans during the period of the world cup showed the extent to which low response really does not need to be a feature of survey based research. The study was monitoring media and advertising consumption and some brand recognition over the period of the World Cup. Very high response rates were sustained throughout an extended survey. Pictures were collected too, and Nielsen ended with a library of 60,000 submitted pictures. The case study offered a real feel-good moment for mobile research.

Thaddeus Fulford Jones (Locately) – has created a panel of mobile phone users in the US who are willing to allow the firm to capture location data and use this to model actual behavior. You can learn about the extent to which consumers do go to some outlets and often will drive past rivals in order to reach them. Raw location data is used to identify locations such as retailers, leisure destinations and other important consumer touchpoints. It tends to be most powerful when combined with other data to provide context. Location data also reveal useful temporal data – e.g. how long people really have spent in a store or even waiting at the checkout.

Hannu Verkasalo (Zokem) – spoke of  “on-device measurement” or using the mobile phone for passive data gathering. What came out was just how much you can measure passively, free from the response bias of a survey, when using a mobile device, from sites accessed, search terms entered and time spent on different activities to location data – what was accessed at home, at work or on the road. He also revealed  the very different ways that people consume mobile content on mobile devices compared to the web, and again the different profile of apps versus browsers in the content that people access. Hannu’s prognosis is that the mobile app is in the ascendant – which contradicted Bruce Hoang’s earlier analysis.

AJ. Johnson (IPSOS Mori) – chaired a panel session entitled “Debunking the myths of mobile research” and asserted that research needs to treat mobile very differently. People will be engaged, if you approach them via mobile research, but as researchers we have to be very transparent, open and honest with respondents.

Paul Berney (Mobile Marketing Association) – challenged research to take greater interest in mobile research. Mobile is the natural home of the digital native – the under 30’s who have grown up knowing nothing other than the internet and the mobile phone. It’s already changing the way that retailers are working and it fundamentally changes the engagement model for brands. It is a mistake to think that mobile is about the technology – it’s about people. Mobile is a two-way channel and if we don’t go there with our research, then others will.

To round up, a few common themes to emerge across the event that struck me as fresh:

  1. Mobile surveys can be a bit longer than we may have first thought. 8-12 questions is a common-sense length, but examples were presented of 30 and 60 questions, and much longer, when over an extended period. But is trying to push up the limit the start of the same slippery slope that has led to the downfall of online research?
  2. The experience in emerging markets and less mature markets is very different. The penetration of mobile is so high in emerging markets that it far exceeds every other channel except face-to-face – it is the natural equivalent of online research.
  3. In developed economies, there is an assumption that mobile research is a replacement for online. In reality, it seems to supplement it, and it is more of a replacement for telephone, face to face.
  4. Mobile research is not one thing – it’s a multimodal channel in its own right, embracing self completion, interview-administered, quantitative or qualitative, visual, textual, voice and image, or passive, observational, which can be augmented with location or temporal data.
  5. The sphere of mobile research is changing fast and it is continuing to evolve. It is not something that research can afford to ignore.