The latest news from the meaning blog

 

AccessPoint reviewed

In Brief

What it does

Mobile CAPI solution for complex or simple surveys which works on a range of standard consumer PDAs under Windows with integrated web-based real-time fieldwork management and monitoring tools.

Supplier

Global Bay

Our ratings

Score 5 out of 5 Ease of use

Score 4 out of 5Compatibility with other software

Score 4.5 out of 5Value for money

Cost

Starts at £20 per interviewer per month with volume discounts available; includes technical support. Devices and airtime contracts extra.

Pros

  • Good for both simple linear surveys and complex non-linear data gathering activities
  • Handles multi-level hierarchies and repeating sections well
  • Web application available for survey/interviewer assignment and administration
  • Can integrate with other applications through open database and web services interfaces

Cons

  • No built-in support for design templates
  • Complex set-up for anything but the simplest of surveys
  • Only works with handhelds using one of the mobile Microsoft Windows variants.

In Depth

Perceptively, the people at Global Bay have realised that demand for handheld interviewing tends not to come from people wanting to do simple twenty question surveys in the street, but from altogether more complex data collection activities such as diaries, mystery shopping and retail audits. In re-writing their Access Point handheld data capture system, they had added a new layer of advanced capabilities for surveys of fiendish complexity, while retaining the ability to be able to knock out a simple survey with a few questions, and get it into the field in a few hours, if need be.

The big change in version 4 is a new data model and scripting language that provide support for hierarchical surveys with a variable number of repeating sections, such as diaries or surveys of households and individuals. These also provide support for what could be described as ‘non-linear’ surveys, where it is the observations being made that determine the flow of the data gathering exercise. Surveys such as this defy programming in most data collection systems.

There are essentially five components to AccessPoint: the ‘Form Builder’ to design the survey instrument; the interviewing client, which can sit on a variety of standalone PDAs or wireless devices, provided they use one of the Microsoft Windows variants; the central database and data synchronisation engine – you can either use GlobalBay’s hosted service, or invest in your own web server; plus a web application for fieldwork administration and another web application for managers or even clients to log in and review topline results or even build some dashboard reporting. All in all, AccessPoint is now a very comprehensive offering, though the results reporting capabilities are fairly rudimentary – great to check up on the data, but not really for in depth data analysis or any data manipulation.

Mobile communications always have been a strength of AccessPoint. GlobalBay, though based in the US, works closely with several of the UK mobile networks through its UK office, and can offer some very competitive bundled deals on hardware and airtime contracts to sit alongside their software. These also bring the advantage of only one supplier to confront if the system goes uncommunicative on you.

Aware that configuring these kinds of devices can cause all sorts of headaches to fieldwork managers and interviewers alike, the software can be self-configuring. For example, if a device is lost in transit to an important assignment, an interviewer could buy a replacement from a High Street retailer, and by logging onto the AccessPoint server, all the special drivers and applications will be downloaded, along with the day’s interviewing assignments.

Administrators can set up assignments in advance, according to interviewers’ availability, or allocate work in real-time, controlling what the interviewer will do next, according to availability.

Another change is that surveys can be distributed already populated with key items of data relevant to the interviewee or the location being audited. This can be lifted directly out of an external database, such as a CRM system, because also new is a choice of three open interface methods to share data with other systems. There is a database mapping tool that will either allow data to be downloaded periodically, or for actual synchronisation to take place – such as where name and address information is being updated in the host system, or where a customer care team needs a real-time alert about an urgent performance issue. And there is a web services interface, for any of the burgeoning number of web-enabled tools that offer this kind of low-level over-the-web data linkage.

On the non-techie side, there are several noticeable improvements for interviewers. Rather than needing to use a stylus to record data, finger-activated questions means that some kinds of surveys can be made entirely ‘finger driven’ now. Interview screens can look a lot nicer too, as it is possible to fine-tune their appearance and to put multiple data capture fields on the same screen, such as drop-down boxes. There is a degree of template control within the design but unfortunately this stops short of being able to apply overall design templates, or use cascading style sheets – a missed opportunity in streamlining the set-up process.

Interviewers can also capture other data now, including photos, if their device has an integral camera. There is a special photo question, which will present a prompt “Do you want to take a photo?” and the next picture the interviewer takes will be captured as the answer to the question and get transmitted back to the server the next time the data are synchronised. There is even an option now to deal with barcodes by taking a picture of them – the image is then translated into numeric data.

At the back-end, the big innovation is the Form Builder tool. Previously, anything other than the very simplest of surveys required custom programming by Global Bay’s technical gurus. Now, there is a wizard to set up very basic questionnaires, which is quick and efficient to use, or a complete programming environment for more complex jobs. Unfortunately, this is a tortuous process that involves separate definitions for questions, screen layouts and a java-like scripting language for the logic that glues it all together. Though not for the faint hearted, this is actually a very sophisticated questionnaire development environment that makes just about any survey possible. The only lack I could see was for multi-language surveys, which are not supported.

The result of this is that surveys seem to take either two hours or two weeks to set up: there is no middle ground. Because this is a programming tool, it means you must plan time for testing and debugging, as all sorts of errors of commission are possible. However, if you are programming a multi-hierarchical diary survey driven by complicated changing schedules, overlaid with a myriad of local exceptions, and a three year contract for the job, not only would this be a fortnight well spent, but the downstream benefits could amount to whole year’s of effort saved.

Customer viewpoint: Andreas Stübi, DemoSCOPE, Switzerland

Andreas Stübi is Head of MR Information Technologies at DemoSCOPE, a full service research agency in Switzerland. DemoSCOPE have now completed three different surveys using Access Point. Andreas judges the experience to date to have been positive: “We chose Global Bay because it was really the only company that could offer us a complete solution. They support a nice range of PDAs on the market, and the software they provide can create hierarchical models which was important for the project we were starting with – a media project that involved the use of diaries.

Our clients really like these kinds of surveys using PDAs, because the Interviewers can walk into a store with one of these things and they don’t stand out – which is especially important for mystery shopping. These are now everyday devices which people are familiar with.

One of the nice things with this software is that you do not need to have an online connection the whole time. So long that you do not have to provide a new questionnaire, it does not matter if an individual interviewer is unable to sync their data until one or two days later.

The online reporter tool is very handy – it is quite powerful. The idea is you can easily control and follow the fieldwork while collecting data in the field. We really made use of this on the media survey, as there are a lot of day quotas to manage. It also means you can react very quickly if you see errors from interviewers or in the actual interview. You can contact the interviewer and explain what needs to happen.”

He also singles out the support he has received from Global Bay. “They are always very helpful if you have any questions, or need some tips with the programming.”

A version of this review first appeared in Research, the magazine of the Market Research Society, December 2007, Issue 499

Converso Enterprise reviewed

In Brief

What it does

Platform-independent Java-based multi-modal interviewing and analysis platform with an integrated portal-style front-end

Supplier

Conversoft, France

Our ratings

Score 3.5 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 3 out of 5Value for money

Cost

In euro (€): Most modules €3,000 per user, plus €20,000 for Enterprise platform and €6,000 for web server module: all one-off costs. Maintenance: 18% of licence cost annually, or 25% for ‘gold’ support.

Pros

  • Extremely easy to use for moderators and participants
  • Can present a wide range of stimulus material
  • Offers several novel research techniques
  • Provides a complete transcript for analysis at the end

Cons

  • Only supports Windows for both moderator and participants
  • Not completely DIY yet: management module to be developed
  • Real time groups only – no support for asynchronous participation

In Depth

Converso Enterprise is an ambitious redevelopment project which deserves much praise for embracing what Web 2.0 technology has to offer head on. The portal-building and alert capabilities are excellent, and the main data collection platform is robust and sophisticated. But as an end-to-end solution it is still very much a work in progress. Substantial chunks are ready for production work now, but the gaps within and between these modules are just a bit too wide for comfort at this point. Given Conversoft’s recent rate of development, It is likely to look much more complete in as little as six month’s time, so for anyone planning to upscale their software platform next year, this is definitely one for the short list.

Converso Enterprise follows an entirely different architectural principal to most of the other new-generation research platforms on the market. Conversoft rejected developing in Microsoft’s .NET framework in favour of using Java, both J2EE, for desktops, laptops and servers, and J2ME (the flavour of Java for mobile devices). This approach does not on its own give the product a Web 2.0 pedigree, but it is a good start. It means that the software is totally platform-independent, so all users – researchers, respondents, technicians or end-clients – can use the browser or the operating system they want – Apple, Linux or any of the Windows varieties. This technical agnosticism extends to the relational database at the heart of the product, for survey data and panels, if used, which could be any of the modern database platforms – Microsoft, Oracle, or open source databases like MySQL or Postgres.

Conversoft also intends to create an open-source development platform to allow customers to extend the capabilities of Converso for themselves, but this does not exist yet.

What does exist is a wonderful portal-building tool that lets you snap into place any of the components of the Enterprise toolkit. You can create your own portal just for you or for entire groups of users – and then you can selectively switch on controls that will allow them to tailor the portal you gave them, to add in their own favourite things.

It could be the survey editing tool, a summary report showing the latest set of KPIs, an RSS news feed from the BBC or a link to Google Maps. This is where it gets exciting, because, once the missing developer tools have been developed, the techie people would be able to build whatever components you wanted to create so called ‘mash-ups’ of data from different sources on the internet, alongside your survey data – for example, to present geodemographic data in map form. What is more, Converso Enterprise components can be used as applets in other portals – so you could broadcast your poll results to other sites, or even Facebook.

Already, there is a rich library of components to choose from, particularly in the reporting area – which was never a strength for Converso in the past. It is relatively straightforward to create client data portals and dashboards that will present data graphically or as cross-tabs, or use intelligent reporting methods to highlight exceptions and provide alerts. Alerts are defined as triggers – really rather like dynamic filters that operate against the data and present a message. It all works fine with published data, but at the moment, you would struggle to show any real-time data from live surveys – such as to track response or get a live snapshot in a topline report. For these you need to resort to some of the legacy modules still.

Similarly, you can deploy new surveys through the portal, define your sample, and even use the very comprehensive access rights management tool from the portal – all of these are java programs. But the survey authoring tool is still a Windows program, and uses the old and rather complicated Converso scripting interface. That is promised for later this year, although nothing was available for Interface to obtain a preview.

These are not the only gaps waiting to be plugged. These are being addressed – and they need to be – though given Conversoft’s recent track record, the current feeling of being on a new highway where the cones are still in place, should have gone by the middle of 2008.

On the plus side, there is true multimodal interviewing with CATI. Web CATI is an integrated and very versatile handheld interviewing capability that will work on a very broad range of smartphones and BlackBerrys. The mobile interviewing is a new development and is impressive. There seems to be complete backwards compatibility with the old Windows-based CATI too.

Panel management exists but is not fully developed yet – the main panel management and respondent selection capabilities are there, but the panellist recruitment and community part is still missing. When it comes, it will offer integration with CRM systems, to use customers as a sample source, or to create customer panels.

The analytical tools are starting to look impressive too. A range of tables and charts can be created and presented directly in Word, Excel or Powerpoint, and it will populate native Excel or PowerPoint objects with data, to permit dynamic linkage. But if you wish to move data out to other MR analysis tools, then you are stuck until the planned Dimensions and Triple-S links are ready.

Converso Enterprise is an ambitious redevelopment project which deserves much praise for embracing what Web 2.0 technology has to offer. The portal-building and alert capabilities are excellent, and the main data collection platform is robust and sophisticated. But as an end-to-end solution it is still very much a work in progress. Substantial chunks are ready for production work now, but the gaps within and between these modules are just a bit too wide for comfort at this point. Given Conversoft’s recent rate of development, it is likely to look much more complete in as little as six months’ time, so for anyone planning to upscale their software platform next year, this is definitely one for the shortlist.

A version of this review first appeared in Research, the magazine of the Market Research Society, November 2007, Issue 498

nQual Rich Focus reviewed

In Brief

What it does

Hosted software solution for online focus groups in real time. Features replicate many of the capabilities of conventional groups.

Supplier

?

Our ratings

Score 5 out of 5Ease of use

Score 2 out of 5Compatibility with other software

Score 4 out of 5Value for money

Cost

Single group is £450. 10 groups £350 each, with volume discounts available. Includes technical support and moderator training.

Pros

  • Extremely easy to use for moderators and participants
  • Can present a wide range of stimulus material
  • Offers several novel research techniques
  • Provides a complete transcript for analysis at the end

Cons

  • Only supports Windows for both moderator and participants
  • Not completely DIY yet: management module to be developed
  • Real time groups only – no support for asynchronous participation

In Depth

To some, the notion of conducting a focus group online is as unsatisfactory as one of those ‘restricted view’ theatre tickets where all you see is a triangle of mostly bare stage. Whether behind the pillar or in an online focus group, you may need to try harder to discern what is going on without eye contact, body language or other visual clues – and all can be deceptive at times. Yet a restricted seat is infinitely better than no seat.

To the generation that thinks nothing unusual of making friends and sharing comments with strangers through such social network sites as Second Life, MySpace and Facebook, nqual’s Rich Focus research platform actually makes much sense as a coherent research method than much of the current research hubris around Web 2.0. What is lost is made up through access to targets that would be impossible to reach by other routes.

Rich Focus is a web-based for online research in real time but in virtual space. In many ways it resembles web-conference tools for business meetings like Webex or GoToMeeting – yet without the accompanying, and often excruciating inefficient 10-way telephone conversation. Rich Focus assumes all the conversation is carried out in instant messaging chat, but in a very research-literate way.

The process starts conventionally enough as you write your topic guide, which you upload in advance. You can recruit offline, or from your own panel, and once you have agreement, you send them an initial email with a software download, which is a one-off activity. One plug in will work with Internet Explorer, but a better solution is to use the dedicated nqual plug-in and browser, which locks down the capabilities and avoids all problems with participants’ browser settings.

Initiations are then issued with an embedded link and at the appointed hour, everyone logs on and can ‘see’ who else is online – identified by a name and a colour. For respondents, the screen is carved up into three panels: one where they can see their name and other member group, including the moderator; another where the questions, answers and discussion takes place, and a third area where stimulus material can be presented.

Moderators have a few more controls, but not so many to make it complicated. As you click on the topic in the guide, a timer starts, if you have added timings to your guide – you can see it counting down, but tactfully, neither participants nor clients can. All contributions from participants are colour coded to match their names – a simple aid to differentiate people and their comments. When a respondent types, a visual light bulb by their name tells you that a response is imminent. It is a simple but useful device that calms down the tangle of overlapping comments that is familiar to most Second Life and other online chat users.

As you work your way through the deceptively simple but intelligently thought out capabilities in this software, you soon see other ways that the more obvious lacks of the online focus groups are compensated for by the new tricks you get in return.

You can introduce stimulus material – images, sound or video – and you can give your respondents control to add pointers, positive ticks or negative crosses, or to circle things and add annotations – a versatile set of tools which opens up endless possibilities for concept testing, ideation, co-creation and more. At any point, you can also turn off the ability for participants to see what others are saying, to get reactions uninfluenced by others, then it can be switched back on once everyone has said their bit, for general discussion. A built in polling tool lets participants vote in secret, then the results can be viewed as a group, for more discussion.

The ability to move unobtrusively between public and private comment also allows a moderator to make a one-to-one comment to a contributor, or even for a co-moderator to step in and do some offline how-did-you-know-that type of questioning. Client observers can also talk one-to-one to group moderators, and moderators can even appeal for help if the something they have not briefed you on should rear its head. In a ‘real’ group, such behaviour would be highly destructive.

When the group is over, the greatest joy is to get the full transcript right away, as a Word or Word or Excel file. The Excel output is organised with comments in one column and contributor name in another, which makes for very simple sorting and filtering. It is easy to import this into CAQDAS tools such as MAXQDA or XSight, though unfortunately the structure is lost – a structured exchange (like triple-S for quant data) would be very nice here.

At present, the software lacks the management tools to let you start new surveys, upload the guide, issue invitations and download transcripts for yourself, though these features are planned. For the time being, nqual performs these tasks for you. It is also a pity the respondent download works on Windows PCs, effectively cutting out any Mac users you’re your research, which could be an issue in some sectors. These lacks at the periphery bound to be remedied as demand increases – there is much good research to be done with what Rich Focus already provides.

Customer viewpoint: Paul Dixon, ICD Research, London

ICD Research is a full-service agency in London providing a typical mix of qual and quant research – around 70 per cent of which is carried out online. Group MD Paul Dixon has used Rich Focus extensively, and is impressed:

“I have been interested in online groups for a long time. From a research perspective you can tell that nqual has been designed with researchers in mind, as it captures a lot of what you want to replicate in an offline focus group. It is very well laid out in terms of your topic guide but still allows you the flexibility to follow an impromptu direction. It also works extraordinarily well in terms of being be able to have a second moderator liaising and talking privately to the client and for the client to be able to feed into the group indirectly when they need to. If a client has to interrupt a conventional group, it really disrupts the group and can undermine your authority.

“Clients really like it – they see it as very distinct in its application. There will always be a need for offline groups for particular topics or where the analysis focuses more on the interactions and non verbal elements, but this works extraordinarily well for proposition testing, website evaluation, advertising testing and it offers clients a way for them to engage more with their customers.”

One of Paul’s favourite features is image tool, where respondents can mark up pictures, illustrations or even websites being displayed directly. “They can circle what they see, or put a tick or cross. It almost gives you quantitative information in what they react to. Nqual have just managed to get all the little components and the small barriers to online research right.  For example, the ability to see who is typing in real time dispels the barrier that online can be disjointed. You might ask another question before people have answered the first one. With this, you can see when people are typing and wait for their response.

“The transcript is probably one of the biggest advantages. With offline you send off your videotape and get the transcript a week later and after a hefty fee – and you get no names next to the transcript. Here, within a matter of hours, you get transcript with the persons name, what they have said, and the ability to filter it. It is making life so much easier for the team here. And because we select our respondents from our online panel, we have all the background information too. I like the simplicity of it – from uploading the topic guide to downloading the transcript, it is a seamless process.

“It is never going to replace offline, but it is a viable methodology in it own right. It works and we have proven it over the 30 or so online groups we have conducted online over the last two years. I think I am actually getting richer data, and from people or audiences that sometimes you would not be able to attract to attend a group in Central London at say, 6pm. I think nQual are onto a winner with this.”

A version of this review first appeared in Research, the magazine of the Market Research Society, October 2007, Issue 497

Remark Office OMR and Remark Office Web 6 Reviewed

In Brief

What it does

A pair of related survey tools designed for DIY surveys on paper, with automated data entry using robust OMR scanning, or online surveys, which share a common data format and set of analytical tools at the back end.

Supplier

Gravic

Our ratings

Score 4 out of 5Ease of use

Score 2 out of 5Compatibility with other software

Score 5 out of 5Value for money

Cost

Remark office OMR $895. Remark Office Web $950. One off costs which includes telephone or email support and software updates within current version plus 50%+ discount on new versions.

Pros

  • Simple straightforward survey tool at a bargain price.
  • Does not require special printing for scannin
  • Can tie together data from paper and web
  • Everything runs on your own PC and web server

Cons

  • No character recognition capability in scan module
  • Lacks email invitations and reminders web module
  • Limited reporting options, especially for cross-tabs
  • Fiddly to create really nice-looking web surveys

In Depth

here are almost too many online survey tools to choose from these days – yet the choice narrows down considerably when you seek a single solution for fielding surveys on paper and online. If you need both and are on a tight budget, then Remark Office could give your productivity and independence a welcome boost.

Remark Office actually comprises two different products, which you purchase separately: Remark Office OMR, for automatic data capture of survey forms, and Remark Office Web, for online surveys. Though they work independently, the underlying data format is the same, and a survey set up on paper can be imported into the web module, so that the data can share the same layout, and they can be brought together easily for analysis, using the integrated cross-tab and statistical reporting module.

To use the OMR module you design your questionnaire in Word, with tickboxes and write-in fields to capture the answers. There are a few simple conventions to follow when laying these out, which are explained in tutorials provided by Gravic. Basically, keep things lined up, avoid gratuitous rules and boxes round boxes, and don’t put things too close together.

The software actually recognises three different types of data: tickbox fields for pre-coded questions, image fields, either for open-ended or numeric questions, or barcodes.

Barcodes are useful for customer surveys, so that you can tie the questionnaire into demographics or other data already known, and pull in data from an external file for analysis. This is simply achieved by having a unique identifier for each record, which could be a customer ID or your own made up number. You then use Word’s mailmerge to print this identifier on each separate survey form and send out the appropriate forms to the appropriate people. Gravic even provide you with a special barcode font to use in Word. Simply applying this font to your identifier converts it into a barcode readable by the scanner.

Tickboxes can also take on a variety of shapes beyond just squares. Where there is less flexibility is over write-in fields for text or numbers. Remark is strictly OMR only, and there is no support for handprint number or letter recognition, unlike many other data capture solutions. It makes the task of collecting a price or a postcode into a lengthy manual process. Gravic claim this is due to the ‘unreliability’ of intelligent character recognition as a technology. It is a specious argument: handprint recognition reached a level of viability fifteen years ago, and has only got better since. The case for leaving it out is probably more down to cost, and it may not be something everyone needs. For me, this is a serious omission.

Remark Office OMR is essentially a legacy forms tool, so to get the form ready for data capture you scan in a blank form and then use their software to define the page in terms of the questions, answers and expected values. When you define a new project there is a wizard that steps you through the task in a relatively painless way, and the software has the intelligence to be able to recognise what are likely to be answer fields when you draw a box around the relevant area of the scanned image using your mouse. All in all, you are likely to spend little more than 15–30 minutes setting up each A4 sheet.

The web survey tool has the advantage that it is fairly quick to use, through it does not have the instant ease and web feel that some of the rival online tools have and feels a bit dated. At least survey instruments are page-oriented, not just one long scrolling form. You can add several questions to one page very easily, and incorporate routing too. But the look of the online survey is uninspiring and appears hard to change.

It can be spiced up with some graphics, which are easy enough to add, but the product lacks the concept of CSS-driven independent style templates.

There is support for password-controlled access to surveys, and the ability for respondents to break off and resume a survey at a later time. Oddly, there is no invitation or reminder facility. However, Gravic are currently re-working the web module, so there is hope that some of these lacks will be overcome. It does have the big advantage that surveys can be deployed on your own web server, once again offering a very low-cost alternative to even the cheapest hosted solutions.

Reporting capabilities are reasonably good. As you might expect, this is not a power-reporting tool, but it goes further than many DIY products with the range of statistics that it offers. It will create statistical reports, summary reports and charts for all of your questions automatically, which will get you off to a flying start with your analysis. Cross tabs are different, and you have to create these one by one – and one question by another is the limit for each table. For in-depth analysis and segmentation, you can use the SPSS .SAV file export route to take it into another program.

It’s a program that does what it says it will, does it accurately, reliably and doesn’t cost the earth. Look on its simplicity as a virtue, and you could be in on a real bargain.

Country Report: United States. Remark Office in action at The City College of New York

Ed Silverman is the Director of Institutional Research at City College, part of the City University of New York, and is responsible for compiling data and carrying out research on the college and its courses. This he does single handedly for the most part, with the aid of Remark Office. Ed is passionate about what can be learned from surveys among students, employers and other ‘customers’ of the college.

Ec explains: “An example of one our marketing-type surveys was to find out ‘what do adult learners want to do?’ for our continuing education programme here. With Remark on paper, you can create a form which is simple to fill in, and which will capture what they say. We took it to events like street fairs, handed them out and we asked people to fill them in. We gave them a bunch of options and asked them what else they would like to see. Remark will capture what they wrote. That’s very useful when can you bring all the comments together and sort them in Word, because you start to notice things you had never thought of – and that means other people haven’t either. It’s great if we can offer courses that other people aren’t.”

“With employers, it is hard to get them to fill out a survey. But if we send them a URL and the survey is short, we get good results. But we also find that some of the managers, especially those in the older age groups, are uncomfortable with the web, so we send them a paper survey. By taking this mixed approach, we have had a number of responses from people who have never responded before.

“For me it is a time game. I have a huge amount of work, and the less time I can take, the happier I am. I can put together a survey in Remark in minutes – especially if you use templates. You can simply copy and paste questions from one survey into another, and that way, when someone asks you for something urgent, you can create something literally in minutes.”

Asked to sum up the benefits, Ed reels off a lengthy list: “It is inexpensive. For scanning, it does not require any special paper or printing and it’s really quick and easy to learn. It is reliable – there just aren’t any glitches. It is incredibly easy to upload to the web, and then to download the data when you are done. It makes savings all round.

Besides the surveys, I have a lot of other work to do. It saves an enormous amount of time. I could not survive without this.

Working with this software is really easy. It’s no problem to create a questionnaire and it does not take much time to learn – you can learn what you need to make the questionnaire after one or two hours.

A version of this review first appeared in Research, the magazine of the Market Research Society, September 2007, Issue 496

Warp reviewed

In Brief

What it does

Web-based cross-tabular and charting suite for researchers and clients, which extends the capabilities of Quanvert and works with Quantum/Quanvert databases.

Supplier

Warp Online

Our ratings

Score 5 out of 5Ease of use

Score 2 out of 5Compatibility with other software

Score 3 out of 5Value for money

Cost

Per month: hosted solutions from $1500 (£750) per user annually; purchase from $90,000 (£45,000) for a 10-concurrent user licence, plus 20% annual maintenance.

Pros

  • Lets you distribute existing reports, define your own or both
  • Real-time access to one central database ensures that users work with the latest data
  • By using Flash, provides a very Windows-like drag-and-drop user experience
  • Good intuitive support for hierarchical data

Cons

  • Only works with data prepared in Quantum and Quanvert
  • Charting in Flash only, which is not always compatible across platforms or media
  • Cannot work offline

In Depth

While travelling at warp speeds beyond that of light was the core fantasy of Star Trek, travelling forward at any speed remains the fantasy of users and companies with a major investment in data processed through Quantum and Quanvert. Still seen by many as the gold standard of MR data analysis, both programs have been on a trajectory to nowhere since acquisition by SPSS in 1998, thereby missing out completely on the internet revolution.

The newer, web-friendly products on the market rarely approach the depth of functionality to handle tricky analyses such as hierarchical data, re-weighting, rolling averages, indexing or creating composites such as Key Performance Indicators. Warp, a totally web-based analysis, reporting and presentation tool, does all this, and will do so directly from Quantum or Quanvert outputs in the form of a Quanvert .PKD file. The deliberate aim of San Francisco-based Warp Online is to create a worm-hole through which Quanvert users can travel instantly from an era when browsers were something only to be found in a public library, to today’s world where distance is no longer a barrier to information.

Indeed, to anyone familiar with Quanvert, Warp is an easy transition to make, as it still contains the same concepts of variables, tables and table axes. There is a separate web-based system administrators’ interface which allows for users and groups to be set up and for the interface to be customised for different clients, both in look-and-feel and in the options they may select – according to the extent to which you trust their skills. It is here you load in the Quanvert datafiles and then set permissions as to who is allowed to see them in their lists of available projects. This process can also be automated to a large extent, for continuous projects. It will also make a lot of helpful auto-adjustments when new waves of data are added, and there are changes to resolve.

For the end user, the interface is deceptively simple. The program is highly context-sensitive, so it tends only to display those capabilities or options which are relevant at that time. As you dig down, some very sophisticated capabilities emerge. Variables are selected simply by dragging and dropping – even though it is all web-based – as it makes extensive use of Flash to provide a very comfortable and productive working environment that really feels like desktop software. Being completely web-based, there is no software to install, no files to distribute, and it means everyone is always working on the latest set of data. Warp will only work with Microsoft IE6 (or IE7 with a special patch applied), which may disappoint some users, but IE6 is the preferred browser for most corporate users, and Warp promise to support IE7 once there is demand for it. However, this is not a program that will suit you if you need to work for long periods with no Internet connection.

With every table you produce you can choose to save, or not. By letting you choose, you can keep your reports folder uncluttered, which can be a problem with many reporting tools. This is also how you create a new variable or filter – any combination of rows or columns on a saved table can then be used to create a filter or a variable, and the top of the table can be saved as a breakdown or banner. These and most other options are available from a context sensitive right-click menu, which also reinforces the desktop feel of the software. Any variable or filter you create goes into a ‘My Variables’ folder. You can also apply filters or breakdowns globally from the right-click menu to existing tables, or make this the default for any subsequent reports you create.

Reports can also be graphs, which are produced using Warp’s own charting facility using Flash. This does not limit you to the rather pedestrian charts that tend to come out of Excel or PowerPoint – you can even create animated charts to show actual growth or change. However, some version of MS Office struggle to accept the Shockwave files produced by Warp, which could cause your presentation to unravel before a client, if you are not using your own laptop.

Also within very easy reach of the non-technical user is the ability to create new weighting schemes, to work with hierarchical data, and to perform significance testing.

All the way through, the program demonstrates a profound understanding of what the user needs to do. The program will also work out what is the appropriate weighting scheme to add when working on different levels of data, and restrict your choice of breakdown variables to those which are appropriate, given the questions already placed in the table.

Warp actually holds all project data in one highly optimised database. This means that end users can also share items that they have created with other users, such as filters or new variables. The optimised database also means that tables are crunched very fast.

Ultimately, the success of Warp will depend on how it lets users break free of the Quantum paradigm, and for that, it needs to be able to handle other types of metadata than Quanvert. For the moment, to coin a phrase, that remains the final frontier.

Customer viewpoint – Dave Bostock, MarketTools, USA

MarketTools, a US-based research and technology provider specialising in online research, uses Warp to deliver online analysis to its clients. Dave Bostock Director and Business Architect at Market Tools, explains: “We conduct large trackers for clients, where we have to go back repeatedly to the same respondents to see how opinions shift over time and how that impacts the perception of brand and market share. We’ve looked at a number of third-party reporting tools to deliver this type of data set and we have not found any that can compete with the speed, usability and power of Warp.

This application actually gives the MarketTools researchers and our clients the ability to generate their own analysis and conduct deeper dives into the data without having to go through the traditional table specification and creation stage. This has always been the big drawback in providing additional insightful analysis. This type of solution gives an analyst the flexibility to view data, query results and run reports via their desktop. Our clients are extremely happy with the solution.

Other programs require a lot of specialised or stylised work at the set up stage. Warp allows us to take our normal production process using Quantum and, with a quick conversion into Quanvert clients can access the project data remotely. Clients access Warp via secure login. One key advantage that Warp provides us with is the ability to publish a project to a client user group from one dataset, with different nuances for each user – filtering which reports, charts and variables are seen by each user for example. The drag and drop interface is really important in making it an easy, engaging and enjoyable experience for the user”.

“Our mission is to provide our clients with on-demand research solutions, and Warp helps us accomplish this. The research business, like the internet overall, is becoming more and more driven by consumers – Warp is a tool that empowers our clients. Reactions [from them] have been very positive.”

Dave is also impressed with Warp’s advanced analytical and table manipulation capabilities, which is delivered in simple steps allowing users to take full advantage of them. “I like the ease of use of the hierarchical data functionality, and the ability to apply pre-determined weights on the fly. Profiling respondents very quickly is also a huge benefit. There are several things I have not yet seen in any other tool, such as Warp’s very handy column differentiator test, or how you can create share tables by easily adding a numeric variable to the increment area on the screen. Our clients need to feel empowered, and Warp certainly gives them that. It is a huge win for us.”

A version of this review first appeared in Research, the magazine of the Market Research Society, August 2007, Issue 495

Vision Critical reviewed

In Brief

What it does

Web-based, hosted solution for high quality web surveys with an integrated, fully featured panel management solution and some support for online qualitative research.

Supplier

Vision Critical, Canada

Our ratings

Score 3.5 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 4.5 out of 5Value for money

Cost

Per month: approx £1,250 for first 5000 panelists, £500 for each additional 10,000, with no cap on number of surveys, interviews or users.

Add-on modules for analysis, communities etc between £150-450 per month.

Allow £7-10,000 on-off for set-up, design and customisation.

Pros

  • Complete range of panel management and community building toolsSophisticated Web 2.0-like survey capabilities
  • Links together all a panellist’s responses to any survey across the database
  • Lets in-house researchers or smaller research firms run their own panels

Cons

  • Authoring and management is Windows and IE only

  • Limited animation support in survey tool

  • No enterprise version offered

In Depth

Given that many researchers wishing to do online research are likely to want to develop and run their own panels, it is odd that the Achilles heel of most online survey tools is still in providing decent panel management capabilities. Some get no further than giving you a database, a recruitment survey and some invitation management capabilities, others not even that far. Not so with Vision Critical, a Vancouver-based company that has spent the last three years building a very different online survey tool from the normal fare. Panel management lies at the heart of its Vision Critical web-hosted online survey tool, with not only facilities to fine-tune the selection of respondents from the panel database, but a complete range of tools to aid building the panel, nurturing a panel community and communicating with panel members and keeping it in peak interviewing condition.

Vision Critical is the first foray into technology for veteran Canadian researcher Angus Reid, and former owner of Angus Reid Group, which was acquired by IPSOS in 2000. The benefit of having a researcher shaping the technology is apparent throughout this software. More than any other online research tool I have reviewed to date, this is a product that acknowledges almost equally the needs of researchers, research clients and, with impeccable grace and style, the often overlooked needs of respondents. If you are seeking a more respondent-friendly way of conducting research, then this is certainly one to review.

The software is provided exclusively as a hosted ASP solution – Vision Critical have no plans to produce a version to install in-house. Therefore all of its capabilities are invoked from a web-browser interface. It is clear that Vision Critical is pitching for the corporate user rather than the agency, though the product should also appeal greatly to smaller research companies with no desire to run their own IT. The software comprises five key functional components: panel member administration, sampling, survey authoring tool, survey deployment and two reporting modules: a simple real-time tool, and a more advanced tool, about to emerge from beta testing.

Though it is not released yet, the new Dynamic Reports module module is intriguing – allowing you to use the same tool to run one-off reports for instant viewing, building complex reports as PDF, Word or Excel files, or publishing them to the web in the manner of a data portal for controlled viewing by clients and stakeholders – which will then change dynamically when you release more data into them. The approach here, and throughout the suite, is to not try to be too ‘clever’ in terms of functionality – avoiding options for options’ sake and going for sensible defaults so that it is easy to get started and become more sophisticated over time.

The Panel Plus module is fully released and in widespread use. It contains everything you need to run your own panel without having to recruit an army of admin staff to run it. It works with the sensible assumption that panel members should be able to self-administer their profiles, participation and reward redemption for themselves through the capabilities you can build into the panel member site. Though you can do all of this yourself, if you wish, designing the site and customising it is part of the set-up service that Vision Critical offers.

It also takes pays more than lip-service to the notion of a panel being an online community. For example, it provides the means to edit and publish your own online newsletter for panel members, which could be daily if you wished. It offers a content management system to let you publish findings and reports to your panel members, which is critical when creating ‘professional’ panels. And most interestingly in the context of Web 2.0, it offers the opportunity to blur the edge between researcher-led and participant-led qualitative discussion by integrating online forums or even blogs. It will also interact directly with commonly used Web 2.0 social networking portals.

There is a nice survey authoring tool too, which is well laid out and easy to use, with a very full range of functionality. It will also let you integrate with Java or Flash components, though it is disappointing that this otherwise futuristic tool does not have more in the way of animation and interactivity at present – though these are supposedly in the pipeline.

It is in the connection between panel and surveys that another astonishing difference emerges. The classic organisation for survey tools is to make each survey a discrete set of records, maybe linked to sample or panel indirectly, but with no actual interconnection. Some provide the mean to post back key variables into a respondent’s profile, but only to a limited extent. With Vision Critical, everything is interlinked, with every survey question answered being attached directly to the respondent’s record, and every prior answer being available for re-use. In case this sounds like chaos in the making, with a minestrone soup of a million questions to choose from, the coup de grace is that the variables are still presented as organised by survey, even though it is effectively just a slice through the database. But it means that at any time, all prior responses that a respondent has given are available to you to use for routing, sample selection or quota control – and also at the analysis stage, such as for profiling or comparison. So long as the respondent was asked the question and provided an answer, anything can be cross-referenced by anything.

But isn’t that what panel-based interviewing systems should by like?

Customer viewpoint: Dru-Ann Love, Business Week, New York

Dru Ann Love is a Primary Research Analyst at the US Publication BusinessWeek Research Services, based in New York. In addition to carrying out surveys and polls that often appear in the pages of BusinessWeek, and surveys among their print and online subscribers, the research group operates as an independent full-service custom research provider. One of its prized assets is its BusinessWeek Market Advisory Board — a controlled access panel of many thousands of business leaders and opinion formers.

This year, BusinessWeek Research Services switched to Vision Critical in order to bring control of its panel in house – although the panel is actually hosted on Vision Critical’s servers, Dru Ann is able to carry out all of the management aspects concerning the panel’s operation for herself.

Dru Ann recalls: “When we had the demo, our reaction was ‘wow!’ you can do all of this. And once we started using Vision Critical, it was really good. I enjoyed it – it was really fun.”

One of the first tasks for Vision Critical was to migrate BusinessWeek’s high-profile panel into their panel management suite.

“Their role was instrumental as we migrated our existing panel,” reports Dru Ann. “And they did a better profiling questionnaire than we had previously.” This has enabled the Research Services team to obtain much more information about their panellists.

Dru Ann also points out the sample selection capabilities as being a particular strength of this software: “I have control over who I selected, and I can verify the source of my target markets. For example, if I need to select a target of ‘all Directors or above’, I can see exactly who I’ve got, and I know to my satisfaction that I’ve got the group that I am targeting.”

Her experience with the questionnaire design module is also that it is “attractive and easy to use”. Again, Vision Critical, as a part of the migration process, defined templates so that the panel members’ portal, and also the surveys have a consistent look and feel, reflecting the BusinessWeek design ethos.

Dru Ann uses the built-in reporting features while the surveys are active. She comments: “I really like their online status report which has a feature which shows completes, so it gives me an idea of how many people are taking my survey at any particular time. If I start noticing that I am not reaching my target, then I can send another email blast to get some more respondents.” More detailed analysis is carried out in SPSS, using the capability provided in Vision Critical to create a complete SPSS file of the data.

She concludes: “I have no complaints. The software is really nice, and the people have been nice too, whenever I have needed to contact them.”

A version of this review first appeared in Research, the magazine of the Market Research Society, July 2007, Issue 494.