The latest news from the meaning blog

 

Read the results: Globalpark Annual MR Software Survey 2009

We’re excited to be able to share with you the results of the Globalpark Annual Market Research Software Survey, which is published today. We have been conducting this international survey – which looks at software and technology used in the market research industry – every year since 2004, the results of which we make freely available to all.

Pie chart showing the perceived viability of moble self-completion research

In 2009 we introduced some new questions about mobile research and communities.  These reveal the MR industry is split over the merits of self-completion mobile-based research, with 45% seeing it as viable 48% only seeing it as close to becoming viable (and not really viable at present) with a further 7% never expecting it to be viable (chart to left). It is also very striking, that large companies have a far more optimistic opinion about the viability of mobile research.

Our questions on communities also proved to be interesting. For example, we learnt that only one in six research firms was operating a community in 2009, and those that do are operating very few communities.

Four- and five-year trends

Chart showing the growth of the mixed-mode integrated platform

Over the years, our tracker questions have revealed some interesting trends. For example, we ask respondents whether they use an integrated platform for their multimode research or whether they switch between modes (chart to right). We first asked this question in 2006 and since then we have detected a gradual but consistent trend to wards integrated platforms. In another question we found that nearly all respondents (84%) thought that multimode data collection was either essential, very important or moderately important when choosing a new data collection tool. Software developers, please take note!

Member Login

Mobile fallout – to be ignored at your peril

Hands holding out a collection of mobile phonesFrom the Mobile Research Conference 2010, Globalpark, London, 8-9 March 2010.

Mobile research, as a method, may still be in its infancy, but researchers already need to be aware of the fallout from the growing phenomenon of mobile communications, both in telephony and in data communications and the mobile web. The effects cannot be avoided and need to be understood. It was clear from last week’s Mobile Research Conference, organised by software firm Globalpark, that respondents are already taking online surveys designed for conventional PCs and laptops on their web-enabled smartphones in small but significant numbers. The response to simply exclude them by closing the survey when an iPhone is detected is not a neutral decision from the sampling perspective.

But neither is it smart to exclude them simply because the survey then behaves in a way that does not let them continue, or which makes it difficult for them to select some responses. It is no longer safe to assume survey participants are using a conventional browser as their preferred means of accessing the Internet, and that trend will accelerate as other portable devices, such as Apple’s iPad and the imitators it will spawn, start to emerge.

The same is the case with mobiles replacing landlines – the figures I found were that 20% of households in the USA were mobile only last year, and that is likely to be 25% now – so a quarter of the population will now fall outside any RDD sampling frame in the USA. Marek Fuchs from the Technical University of Darmstadt, in one of the sessions at the event that I was chairing, presented some astonishing figures on the extent to which people were giving up their landlines in many other European countries at an even faster rate. He presented a Europe-wide average from Eurobarometer data in late 2009 of 30%. It is even higher in some counties, notably 66% of people in Finland and 75% in the Czech Republic who have only a cellphone to answer.

Mobile web may not quite be mothholing sampling frames to the extent mobile voice is to CATI, but the greater cause for concern here is just what these respondents do who do participate online. Mick Couper, who knows more about interview effects than anyone, warned that the effect on completion is barely understood yet, but one thing is clear – making assumptions from web surveys would be very risky. Even if survey tools are set up to convert full screen surveys for gracefully to the smaller format, as  Google’s Mario Callegaro said, a concern for him would be to know on what basis this conversion was being done and what lay behind the decision-making process adopted by web survey developers or software providers.

The uncomfortable truth is that we just don’t know the answers to many of these questions yet.

Snapshot of mobile interviewing

As the opening keynote speaker at today’s Mobile Research Conference organised by Globalpark in London, Tim gave a definitive snapshot of the state of mobile interviewing in research.

As source material, he used new research-on-research he conducted before the event. His summary of the benefits and the challenges faced in mobile research was echoed throughout the conference, by early adopters and specialist developers alike.

Download this presentation and other presentations by meaning

External link to Mobile Research 2009 Conference

Mopinion reviewed

In Brief

Mopinion

The 3rd Degree, UK
Date of review: October 2008

What it does

SMS and WAP interviewing software, provided as a hosted web-based service which allows you to design and deliver short surveys via SMS using a free shortcode so there is no cost of reply to the respondent (in the UK) or via WAP to mobile web-enabled handsets.

Our ratings

Score 4 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 5 out of 5Value for money

Cost

From £1500 per month which give access to a zero-rated UK shortcode and international long code, capacity for 50,000 messages per month (more than enough for four 500-respondent surveys), authoring, admin tools, training and support. Message costs charged in addition: typically 6p outbound and 5p inbound on the UK free shortcode. Price breaks for higher volumes are offered.

Pros

  • Analyse and cross-tab verbatim responses as you would any standard question
  • Platform-independent set-up: can use any modern browser on Windows, Mac, Linux
  • Clean, easy-to-use interface requires no technical skills
  • Good back-end support for integrating with other systems or data collection tools

Cons

  • Sampling options a bit fiddly
  • Limited reporting capabilities
  • No online help
  • Hosted solution only: no enterprise version available (yet)

In Depth

Mopinion has come a long way since we first looked at it in May 2005, when the first version had just been released. Back then, it was very much limited to asking a few simple closed questions in a straight line, for delivery as an SMS interview. Now the software has broadened to include a range of supporting tools and options and, to an extent, support for WAP surveys and even web surveys.

There are many other useful changes, such as support for uploading samples or panels and some rudimentary top-line reports to give you a snapshot of response and preliminary results. There is also a complete administrator interface, in which you can define the permissions you wish to give to different back office users – so you can define different roles for survey authors, survey testers, data editors and so on.

Surveys can be more ambitious now, with support for scale questions, open-ended questions, much better support for multiple- response questions. Unlike before, you can route questions on prior responses and you can you can pipe text in from other answers or from the sample, such as the respondent’s name or the name of the brand they selected. You can also create ‘cyclic’ question groups, where the software will randomly select one or more questions from a pool of questions – a handy way to pose ten questions but keep the interview to only five for any one individual. The conventional wisdom is half a dozen questions is just about the limit in an SMS interview. The trade-off can be fieldwork times of an hour from start to finish.

Last time, we marked the software down for being unrealistically tolerant of errors in the data. This time, it is due for some well-deserved praise, as a range of strategies for weeding out and correcting errors has been added. The tightest control will put the interview into a hold state until someone given editing rights reviews the response and makes a data correction. There is also an auto-correct feature which will learn by example, so that common mistypes or systematic errors (e.g. people who type the response ‘yes’ to a question instead of using the numeric code indicated) can be substituted without manual intervention, so you should only get clean data out of the other end.

The interface is clean and easy to use – the software does not try to do too much, so it is relatively quick to learn and navigate around, and works even on relatively slow internet connections. Mopinion does benefit from being a very standard HTML implementation – there is no Flash or Java involved, so it will run on just about any web browser and does not mind what operating system you are using. With patience, you could even write it on an iPhone. It offers a range of data outputs, including a hassle-free Triple-S export.

This almost ruthless simplicity means it is not always quite as friendly as it could be though. Defining sample selections is rather clunky, as is editing existing questions – some changes are only achieved by deleting and adding again. The system would also benefit from having some online help or documentation, which was not in evidence – and as more features are added, this will become imperative.

We learned there are some very useful additions in the pipeline, such as support for diary surveys, where a survey segment can be repeated – the survey will be kept open for the respondent to submit each diary entry over an extended period. We also understand the rather rudimentary reports are about to get a makeover too.

A real strength of mOpinion’s SMS implementation, though, is the SMS gateway that The 3rd Degree (T3D) provides as an integral part of the service. Without this, SMS interviewing is a nightmare of multiple SIM cards and modems. In the UK you can use T3D’s own shortcode number, which is free to the respondent – you will be billed 5p per incoming text, which we consider to be a bargain. T3D say this is the wholesale rate and they add no margin to it. If you try to rent your own shortcode from a mobile network it will cost you tens of thousands of pounds annually, even before any call charges. The firm is very experienced now in the vagaries of the mobile networks, both in the UK and internationally, and can provide low-cost solutions in other markets too. You would need to be doing very high volumes indeed for it to be worth your while trying to run your own gateway.

Mopinion now supports three interviewing modes: SMS, WAP and web, though a survey can only be one of these, and you are committed to that mode once you have started on it. WAP surveys are rather more sophisticated in what you can do than SMS, and the software helpfully provides templates for defining each screen.

Don’t expect the web survey capabilities to match up to those of the other specialist web survey tools, as options are effectively limited by the scope of WAP and SMS surveys. However, it does mean you can use the same tool to manage a simple web recruit as a screener to the SMS interview, without needing to move data between one tool and another. Indeed, you can recruit directly off the web, sign a respondent up and initiate the SMS interview without any manual intervention, as an alternative to mass-invites by SMS message, from samples or panel data. All in all, mOpinion now provides a safe and research-savvy way to get into SMS interviewing with little fuss and for a very reasonable cost.

Client perspective: Ipsos Mori, London

Ipsos Mori launched the Orange Business Jury last year, which is a panel of over 1,000 small business owners and SME decision-makers. Where a very fast reaction to breaking news is sought, the panel is contacted by SMS. The SMS component is driven by Mopinion, as AJ Johnson at Ipsos Mori Online explains:

“What I like about The 3rd Degree is the way they are willing to integrate their technology with other systems, so that we can combine their niche area of data collection with more mainstream activities. We have been able to integrate this seamlessly with our main interviewing platform, and link it to a subsection of our panel, so we are not having to move data around between systems.

“It provides excellent PR-type feedback and research. With the Orange Business Jury, it has been very successful and the results are coming in as fast as everyone says they will with text messaging. It is very good for PR-type research and instant feedback.

“The software side is important, but with SMS, it is of even greater importance to have an understanding of the mobile networks, having relationships with the networks and having a gateway that is as stable as it can be. In our experience, The 3rd Degree is strong in all of this.

“Because the surveys are simple, with just three or four questions, we have been able to move the survey into the business areas, to people working on the research side, who are able to run their own surveys and call on the technical people only if they need to. They like it because it is easy to use and they can go right through all the steps of a project without it becoming too technical for them and get results back very quickly, often in just a few hours.

“It is also an advantage that it works with WAP too. I do think WAP is in a better place for research than it was a year ago, and with people switching to BlackBerries and iPhones, the market is swinging in that direction. But I am still a big fan of SMS, because representivity-wise, it is better even than online research.

“The 3rd Degree have also fully covered any worries about compensating our respondents or ensuring that they are not out of pocket, because in the UK they provide completely free text messaging.

Looking to the future, Johnson is excited by the potential to add location data to each survey response, either from the mobile network or using GPS to pinpoint the position. He remarks: “SMS and WAP have huge potential for point-of-experience surveys and could answer some of our sampling issues for location-based interviewing. This would provide the big benefit of being able to interview people who are on the move, if you can know exactly where they are each time they respond.”

A version of this review first appeared in Research, the magazine of the Market Research Society, October 2008, Issue 50

AccessPoint reviewed

In Brief

What it does

Mobile CAPI solution for complex or simple surveys which works on a range of standard consumer PDAs under Windows with integrated web-based real-time fieldwork management and monitoring tools.

Supplier

Global Bay

Our ratings

Score 5 out of 5 Ease of use

Score 4 out of 5Compatibility with other software

Score 4.5 out of 5Value for money

Cost

Starts at £20 per interviewer per month with volume discounts available; includes technical support. Devices and airtime contracts extra.

Pros

  • Good for both simple linear surveys and complex non-linear data gathering activities
  • Handles multi-level hierarchies and repeating sections well
  • Web application available for survey/interviewer assignment and administration
  • Can integrate with other applications through open database and web services interfaces

Cons

  • No built-in support for design templates
  • Complex set-up for anything but the simplest of surveys
  • Only works with handhelds using one of the mobile Microsoft Windows variants.

In Depth

Perceptively, the people at Global Bay have realised that demand for handheld interviewing tends not to come from people wanting to do simple twenty question surveys in the street, but from altogether more complex data collection activities such as diaries, mystery shopping and retail audits. In re-writing their Access Point handheld data capture system, they had added a new layer of advanced capabilities for surveys of fiendish complexity, while retaining the ability to be able to knock out a simple survey with a few questions, and get it into the field in a few hours, if need be.

The big change in version 4 is a new data model and scripting language that provide support for hierarchical surveys with a variable number of repeating sections, such as diaries or surveys of households and individuals. These also provide support for what could be described as ‘non-linear’ surveys, where it is the observations being made that determine the flow of the data gathering exercise. Surveys such as this defy programming in most data collection systems.

There are essentially five components to AccessPoint: the ‘Form Builder’ to design the survey instrument; the interviewing client, which can sit on a variety of standalone PDAs or wireless devices, provided they use one of the Microsoft Windows variants; the central database and data synchronisation engine – you can either use GlobalBay’s hosted service, or invest in your own web server; plus a web application for fieldwork administration and another web application for managers or even clients to log in and review topline results or even build some dashboard reporting. All in all, AccessPoint is now a very comprehensive offering, though the results reporting capabilities are fairly rudimentary – great to check up on the data, but not really for in depth data analysis or any data manipulation.

Mobile communications always have been a strength of AccessPoint. GlobalBay, though based in the US, works closely with several of the UK mobile networks through its UK office, and can offer some very competitive bundled deals on hardware and airtime contracts to sit alongside their software. These also bring the advantage of only one supplier to confront if the system goes uncommunicative on you.

Aware that configuring these kinds of devices can cause all sorts of headaches to fieldwork managers and interviewers alike, the software can be self-configuring. For example, if a device is lost in transit to an important assignment, an interviewer could buy a replacement from a High Street retailer, and by logging onto the AccessPoint server, all the special drivers and applications will be downloaded, along with the day’s interviewing assignments.

Administrators can set up assignments in advance, according to interviewers’ availability, or allocate work in real-time, controlling what the interviewer will do next, according to availability.

Another change is that surveys can be distributed already populated with key items of data relevant to the interviewee or the location being audited. This can be lifted directly out of an external database, such as a CRM system, because also new is a choice of three open interface methods to share data with other systems. There is a database mapping tool that will either allow data to be downloaded periodically, or for actual synchronisation to take place – such as where name and address information is being updated in the host system, or where a customer care team needs a real-time alert about an urgent performance issue. And there is a web services interface, for any of the burgeoning number of web-enabled tools that offer this kind of low-level over-the-web data linkage.

On the non-techie side, there are several noticeable improvements for interviewers. Rather than needing to use a stylus to record data, finger-activated questions means that some kinds of surveys can be made entirely ‘finger driven’ now. Interview screens can look a lot nicer too, as it is possible to fine-tune their appearance and to put multiple data capture fields on the same screen, such as drop-down boxes. There is a degree of template control within the design but unfortunately this stops short of being able to apply overall design templates, or use cascading style sheets – a missed opportunity in streamlining the set-up process.

Interviewers can also capture other data now, including photos, if their device has an integral camera. There is a special photo question, which will present a prompt “Do you want to take a photo?” and the next picture the interviewer takes will be captured as the answer to the question and get transmitted back to the server the next time the data are synchronised. There is even an option now to deal with barcodes by taking a picture of them – the image is then translated into numeric data.

At the back-end, the big innovation is the Form Builder tool. Previously, anything other than the very simplest of surveys required custom programming by Global Bay’s technical gurus. Now, there is a wizard to set up very basic questionnaires, which is quick and efficient to use, or a complete programming environment for more complex jobs. Unfortunately, this is a tortuous process that involves separate definitions for questions, screen layouts and a java-like scripting language for the logic that glues it all together. Though not for the faint hearted, this is actually a very sophisticated questionnaire development environment that makes just about any survey possible. The only lack I could see was for multi-language surveys, which are not supported.

The result of this is that surveys seem to take either two hours or two weeks to set up: there is no middle ground. Because this is a programming tool, it means you must plan time for testing and debugging, as all sorts of errors of commission are possible. However, if you are programming a multi-hierarchical diary survey driven by complicated changing schedules, overlaid with a myriad of local exceptions, and a three year contract for the job, not only would this be a fortnight well spent, but the downstream benefits could amount to whole year’s of effort saved.

Customer viewpoint: Andreas Stübi, DemoSCOPE, Switzerland

Andreas Stübi is Head of MR Information Technologies at DemoSCOPE, a full service research agency in Switzerland. DemoSCOPE have now completed three different surveys using Access Point. Andreas judges the experience to date to have been positive: “We chose Global Bay because it was really the only company that could offer us a complete solution. They support a nice range of PDAs on the market, and the software they provide can create hierarchical models which was important for the project we were starting with – a media project that involved the use of diaries.

Our clients really like these kinds of surveys using PDAs, because the Interviewers can walk into a store with one of these things and they don’t stand out – which is especially important for mystery shopping. These are now everyday devices which people are familiar with.

One of the nice things with this software is that you do not need to have an online connection the whole time. So long that you do not have to provide a new questionnaire, it does not matter if an individual interviewer is unable to sync their data until one or two days later.

The online reporter tool is very handy – it is quite powerful. The idea is you can easily control and follow the fieldwork while collecting data in the field. We really made use of this on the media survey, as there are a lot of day quotas to manage. It also means you can react very quickly if you see errors from interviewers or in the actual interview. You can contact the interviewer and explain what needs to happen.”

He also singles out the support he has received from Global Bay. “They are always very helpful if you have any questions, or need some tips with the programming.”

A version of this review first appeared in Research, the magazine of the Market Research Society, December 2007, Issue 499