The latest news from the meaning blog

 

AccessPoint reviewed

In Brief

What it does

Mobile CAPI solution for complex or simple surveys which works on a range of standard consumer PDAs under Windows with integrated web-based real-time fieldwork management and monitoring tools.

Supplier

Global Bay

Our ratings

Score 5 out of 5 Ease of use

Score 4 out of 5Compatibility with other software

Score 4.5 out of 5Value for money

Cost

Starts at £20 per interviewer per month with volume discounts available; includes technical support. Devices and airtime contracts extra.

Pros

  • Good for both simple linear surveys and complex non-linear data gathering activities
  • Handles multi-level hierarchies and repeating sections well
  • Web application available for survey/interviewer assignment and administration
  • Can integrate with other applications through open database and web services interfaces

Cons

  • No built-in support for design templates
  • Complex set-up for anything but the simplest of surveys
  • Only works with handhelds using one of the mobile Microsoft Windows variants.

In Depth

Perceptively, the people at Global Bay have realised that demand for handheld interviewing tends not to come from people wanting to do simple twenty question surveys in the street, but from altogether more complex data collection activities such as diaries, mystery shopping and retail audits. In re-writing their Access Point handheld data capture system, they had added a new layer of advanced capabilities for surveys of fiendish complexity, while retaining the ability to be able to knock out a simple survey with a few questions, and get it into the field in a few hours, if need be.

The big change in version 4 is a new data model and scripting language that provide support for hierarchical surveys with a variable number of repeating sections, such as diaries or surveys of households and individuals. These also provide support for what could be described as ‘non-linear’ surveys, where it is the observations being made that determine the flow of the data gathering exercise. Surveys such as this defy programming in most data collection systems.

There are essentially five components to AccessPoint: the ‘Form Builder’ to design the survey instrument; the interviewing client, which can sit on a variety of standalone PDAs or wireless devices, provided they use one of the Microsoft Windows variants; the central database and data synchronisation engine – you can either use GlobalBay’s hosted service, or invest in your own web server; plus a web application for fieldwork administration and another web application for managers or even clients to log in and review topline results or even build some dashboard reporting. All in all, AccessPoint is now a very comprehensive offering, though the results reporting capabilities are fairly rudimentary – great to check up on the data, but not really for in depth data analysis or any data manipulation.

Mobile communications always have been a strength of AccessPoint. GlobalBay, though based in the US, works closely with several of the UK mobile networks through its UK office, and can offer some very competitive bundled deals on hardware and airtime contracts to sit alongside their software. These also bring the advantage of only one supplier to confront if the system goes uncommunicative on you.

Aware that configuring these kinds of devices can cause all sorts of headaches to fieldwork managers and interviewers alike, the software can be self-configuring. For example, if a device is lost in transit to an important assignment, an interviewer could buy a replacement from a High Street retailer, and by logging onto the AccessPoint server, all the special drivers and applications will be downloaded, along with the day’s interviewing assignments.

Administrators can set up assignments in advance, according to interviewers’ availability, or allocate work in real-time, controlling what the interviewer will do next, according to availability.

Another change is that surveys can be distributed already populated with key items of data relevant to the interviewee or the location being audited. This can be lifted directly out of an external database, such as a CRM system, because also new is a choice of three open interface methods to share data with other systems. There is a database mapping tool that will either allow data to be downloaded periodically, or for actual synchronisation to take place – such as where name and address information is being updated in the host system, or where a customer care team needs a real-time alert about an urgent performance issue. And there is a web services interface, for any of the burgeoning number of web-enabled tools that offer this kind of low-level over-the-web data linkage.

On the non-techie side, there are several noticeable improvements for interviewers. Rather than needing to use a stylus to record data, finger-activated questions means that some kinds of surveys can be made entirely ‘finger driven’ now. Interview screens can look a lot nicer too, as it is possible to fine-tune their appearance and to put multiple data capture fields on the same screen, such as drop-down boxes. There is a degree of template control within the design but unfortunately this stops short of being able to apply overall design templates, or use cascading style sheets – a missed opportunity in streamlining the set-up process.

Interviewers can also capture other data now, including photos, if their device has an integral camera. There is a special photo question, which will present a prompt “Do you want to take a photo?” and the next picture the interviewer takes will be captured as the answer to the question and get transmitted back to the server the next time the data are synchronised. There is even an option now to deal with barcodes by taking a picture of them – the image is then translated into numeric data.

At the back-end, the big innovation is the Form Builder tool. Previously, anything other than the very simplest of surveys required custom programming by Global Bay’s technical gurus. Now, there is a wizard to set up very basic questionnaires, which is quick and efficient to use, or a complete programming environment for more complex jobs. Unfortunately, this is a tortuous process that involves separate definitions for questions, screen layouts and a java-like scripting language for the logic that glues it all together. Though not for the faint hearted, this is actually a very sophisticated questionnaire development environment that makes just about any survey possible. The only lack I could see was for multi-language surveys, which are not supported.

The result of this is that surveys seem to take either two hours or two weeks to set up: there is no middle ground. Because this is a programming tool, it means you must plan time for testing and debugging, as all sorts of errors of commission are possible. However, if you are programming a multi-hierarchical diary survey driven by complicated changing schedules, overlaid with a myriad of local exceptions, and a three year contract for the job, not only would this be a fortnight well spent, but the downstream benefits could amount to whole year’s of effort saved.

Customer viewpoint: Andreas Stübi, DemoSCOPE, Switzerland

Andreas Stübi is Head of MR Information Technologies at DemoSCOPE, a full service research agency in Switzerland. DemoSCOPE have now completed three different surveys using Access Point. Andreas judges the experience to date to have been positive: “We chose Global Bay because it was really the only company that could offer us a complete solution. They support a nice range of PDAs on the market, and the software they provide can create hierarchical models which was important for the project we were starting with – a media project that involved the use of diaries.

Our clients really like these kinds of surveys using PDAs, because the Interviewers can walk into a store with one of these things and they don’t stand out – which is especially important for mystery shopping. These are now everyday devices which people are familiar with.

One of the nice things with this software is that you do not need to have an online connection the whole time. So long that you do not have to provide a new questionnaire, it does not matter if an individual interviewer is unable to sync their data until one or two days later.

The online reporter tool is very handy – it is quite powerful. The idea is you can easily control and follow the fieldwork while collecting data in the field. We really made use of this on the media survey, as there are a lot of day quotas to manage. It also means you can react very quickly if you see errors from interviewers or in the actual interview. You can contact the interviewer and explain what needs to happen.”

He also singles out the support he has received from Global Bay. “They are always very helpful if you have any questions, or need some tips with the programming.”

A version of this review first appeared in Research, the magazine of the Market Research Society, December 2007, Issue 499

Cluetec mQuest reviewed

In Brief

What it does

Handheld interviewing for Windows mobile devices, with capabilities for mystery shopping, public transport measurement and self-completion diary surveys

Supplier

Cluetec, Germany

Our ratings

Score 4 out of 5Ease of use

Score 3 out of 5Compatibility with other software

Score 5 out of 5Value for money

Cost

Pay-per-use model from €0.30 per interview, with support contracts from €100 (inc 1 hr phone support). Other pricing plans available for longer-term users.

Pros

  • QuestEditor authoring tool very simple and easy to use
  • Special ‘traffic’ version is ideal for transport and travel surveys
  • Robust and reliable in the field
  • Cluetec offers PDA rental and per-survey cost

Cons

  • Cannot pre-populate interviews with case data
  • No support for quotas
  • Lacks tools to manage the allocation of fieldwork to individuals and devices
  • Windows Mobile/Pocket PC devices only, not Palm

In Depth

“Great idea, but not really practical.” This is a common reaction from experienced fieldwork managers to handheld interviewing for long and complex surveys – and it is fair to say that it has also been the experience of some users when trying to do demanding surveys such as mystery shopping or travel audits. Long surveys with complex routing are often no problem, but surveys where the interviewer needs to follow a routing determined by what they are observing rather than a pre-determined flow of questions can be very tricky to present on a PDA.

Cluetec, a German software company, has developed a palmtop interviewing system which aims to provide support for these problem children of mobile interviewing, as well as the more standard fare of face-to-face interviews.

Surveys are created in a Windows-based tool called QuestEditor. It is fairly standard fare, with a tree-view on the left and tabbed forms where you define your question texts, answers, conditions and validation. It is virtually syntax-free, though some of the routing and validation logic can be a bit cryptic, and branches are achieved using a ‘goto’ type construct, which has the potential to become very confusing for complex routings. A block-structured approach would be safer all round.

Cluetec offers a special ‘Traffic’ version of mQuest which is aimed at public transport service measurement, though it could be equally useful in a variety of other mystery shopping situations. Traffic lets the auditor toggle between two surveys, which is ideal for on-board measurement on a train, tram or bus, as a full audit of the service and the vehicle can be carried out when the vehicle is moving. However, when the vehicle reaches a stop, the auditor can toggle to a second survey which lets them count those boarding and alighting, and in a number of categories such as by age, disability, with bikes, pieces of luggage and so on.  The boarding survey is already populated with information about the route, which is downloaded to all the devices. All the auditor needs to do is enter the route code and the survey will then anticipate each stop in sequence. There appears to be no practical limit on how many routes can be loaded on the palmtop device for any transportation region.

An auto-completion feature, available in mQuest, makes travel and mystery shopping very effective, and overcomes the constraint of typing letters with a stylus from a pop-up keyboard on screen. As each letter is entered, the list will narrow down to the most likely candidates that begin with or contain those letters. It will make light work of several thousand destination or transport interchange points.

The software is strong on validation – both rigorous error checking and lighter-touch plausibility checking, and contains several other features to make completion both fast and reliable in the field. Single click mode moves to the next question as soon as an answer is given – and this can be turned on and off as required. Global variables allow some questions, such as location, to be filled in automatically, until the interviewer changes it. Also ideal for mystery shopping is its ability to integrate any pictures or short movies taken using the PDA’s built-in camera with the interview, or audio, so that these images or recordings are passed back along with the rest of the data. There are special question types for each of these, making programming a breeze.

There are also some surprising lacks in the product. There is no support for quota control, which is perhaps forgivable if the emphasis is mystery shopping. Less easy to understand is the absence of the means to pre-populate surveys with case-specific data, such as data pertaining to the places being mystery shopped. Neither is there a decent solution for allocating work to fieldworkers or respondents yet. Improvements are promised here for a version due out in June. Though multiple languages are supported, the authoring tool only shows one at a time. As you enter the translation, the original disappears from view, which is an accident waiting to happen, in my view.

Underneath all of this, the software is developed in Java, and Cluetec have written their own emulator and delivery platform, which means the software has the potential to run on any platform that supports Java, not just Windows. Market demand, however, means that mQuest is currently only supported on Windows mobile devices and support for PalmOS has recently been withdrawn. But the emulator, which you can use to test surveys, works in any Java environment device. It worked faultlessly on my Mac, for example.

The server, which the handhelds connect to for up- and downloading, works under Linux or UNIX as well as Windows. But the QuestEditor authoring tool insists that you are a Windows user. Yet the experience of users seems to point to the product being rock solid for reliability when used in the field. Data transfer can be achieved wirelessly, using cellular telephony for distributed fieldwork, or Bluetooth for short distances such as at a conference centre, or simply by docking the devices’ flash memory cards.

Conveniently. Cluetec also maintains a large stock of loan devices, and allows renters to use its servers for data transfer and fieldwork management. With rental charged on a per-interview basis, it can offer both a low-cost and a low-risk was to dip a toe into the fast-moving waters of mobile interviewing.

Media company MindShare has made extensive use of mQuest for its MindSet media survey in Germany. Here, Christian Franzen, Director Advanced Techniques Group and Christian Maerten, Project Manager MindSet, speak of their experiences.

Customer Perspective: MindShare, Germany

Media company MindShare has made extensive use of mQuest for its MindSet media survey in Germany. Here, Christian Franzen, Director Advanced Techniques Group and Christian Maerten, Project Manager MindSet, speak of their experiences.

CF: “We belong to a media agency, so we do research on media usage and ad effectiveness. The aim of our MindSet study is to evaluate total media usage of people, because all we have are highly separated studies in this market. We do not have something that does not cover all media – not just TV, radio and newspapers but all the other media such as posters, beer mats, internet, email and so on.

CM: “Our aim was to do a study where people filled in a questionnaire every hour for three days. The idea was to have something that was something between a questionnaire and measurement – very near to measurement. We wanted to do this via PDA so [respondents] could fill in the questionnaire on their own. We wanted a solution that was simple for them to use. The software did not do all of this, but Cluetec were very open to change, and created a special version for us.

CF: “With our system we could have a very detailed look at what the person is watching or has seen, and also ask them about their attention and general feeling. This provides a very good combination and overcomes the problem of people either forgetting things or changing things. A lot of things get lost [with traditional methods]. We chose mQuest on the basis that it would be very good for our research.

CM: “The technology for respondents is very easy to use. We did a pilot and in that asked people if they would participate again and 70 per cent said ‘yes’. Only two per cent said they did not like it. The technique is amazingly stable and Cluetech have made it very robust. For example, the PDA crashes for some reason – and that usually does not happen – immediately the software starts again. This is very important if you are handing out a PDA to your respondent.

Working with this software is really easy. It’s no problem to create a questionnaire and it does not take much time to learn – you can learn what you need to make the questionnaire after one or two hours.

CF: “We have been amazed by the technology – when we started, our concerns were about whether people could work with it and if it would was stable enough to do a quick study. It was. We eventually had 200 PDAs out in the field.”

CM: “Our questionnaire is huge and contains a lot of complex filtering, but this means is not time-consuming for the respondent. It takes about a minute and a half to complete each time. But if you print it out, the questionnaire is 120 pages.”

CF: “We are definitely going to do more studies this way.

A version of this review first appeared in Research, the magazine of the Market Research Society, May 2007, Issue 492