The latest news from the meaning blog

 

Revelation reviewed

In Brief

What it does

Online qualitative research environment for asynchronous or bulletin-board style depth interviewing, discussions and auto-ethnography, allowing research to take place over several days or even weeks. No special software or plug-in is required to participate.

Supplier

Revelation Inc., Portland, OR

Our ratings

Score 5 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 4 out of 5Value for money

Cost

Single project licenses start at $1,500 US.  Discounts available for larger volumes, annual licenses and longer projects (3 months+).  Helping Hands project support and translations are costed on a per-project basis.

Pros

  • Daily activities can be set up in advance and launched automatically
  • Participants can blog, view stimulus material and upload content using any browser
  • Integrated translation service for discussion guides and transcripts
  • Full transcripts easily exported at any time

Cons

  • Discussion groups lack versatility
  • Can be difficult to analyse quail-quant type pre-coded questions
  • Can be overwhelmed by emails from a busy board
  • Currently cannot personalise the welcome email text

In Depth

Online qualitative research does not need to be a pale imitation of conventional face-to-face groups and depths, and it is better not to try. Revelation is a piece of web-based software that provides a rich environment for qualitative researchers to design Internet-age research projects that play to the strengths of the medium. Respondents tend to welcome the convenience of being able to participate whenever they choose from the comfort of their own home or office. Researchers and client may enjoy the same, but more importantly, by moving beyond the temporal and spatial constraint of the single-point-in-time group, they may find they get richer and more considered insights.

A Revelation group can take place over several days, or even weeks, with new questions or exercises being presented on a daily basis. Participants can be encouraged to contribute much more in the way of content, taking the focus group into the realms of auto-ethnography and co-creation. Revelation allows you to decide, for each question you ask, whether the responses are to be visible to others – before they respond (“influenced”), only after they respond (“uninfluenced”) or even withheld from all but the moderator and any client observers (“private”).

The way the software works as a researcher, is that you log into your account on the Revelation server and create a series of activities for your respondents to take, building different tasks from a toolbox of stimuli. You can cue them exercises to start on different days, and if you are planning some kind of diary activity, an exercise can be made to repeat. You can build an exercise very simply from a toolbox of components.

There’s everything you would expect there, from open questions and closed, pre-coded questions, places to provide descriptions, welcome texts and explanations, cues to present any multi-media stimulus material you may wish to display and questions where you request an upload of a photo, document or even a video. You could simply present a series of open questions each day or you could lead your participant through creating an daily blog illustrated with photos they have taken of their actual experiences. Either way, you can also probe away to your heart’s content, and even change the direction of the research part way through.

The software also lets you manage your participants, send out initial invites and get them to fill out a short profile survey, which you can customise. You can also import participant lists from Excel. Projects are divided into segments, which you can use in multiple ways: to divide your participants into smaller subgroups, assign them to different moderators or to assign different tasks to different subgroups.

The respondent interface presents some very simple tabbed areas to view – things to do, things already done and direct messages to or from the moderator. This and all the interfaces have a very pleasing “Facebook era” design which make them pretty much self-explanatory.

As a moderator, is very easy to track participation, view all the new content, add probes and send email reminders or messages to participants that don’t seem to be logging in.  You too will get emails whenever anyone completes one of your tasks – if you are running several large groups, their can be a tidal wave of emails coming your way.

Revelation also allows you to conduct discussion groups online. Here, I found the software to be a little less flexible. It forces you to divide each discussion topic into a series of different tasks. You follow up any point with a probe to that individual, but the tool currently lacks the ability to ask follow-up questions of the group as a whole, or simply open out a probe to everyone without setting up an entirely new discussion task.

In the current version, there are some other minor niggles, such as not being able to personalise the welcome email, and it not being very easy to output and analyse or present the answers to closed questions. However, in the piece of research I used this for (a group among six IT professionals), I was astonished with the quality and clarity of the responses I got. Going online clearly cuts out the waffle, as respondents draft their responses carefully, and consider what they are saying. The result is data that is relatively easy to analyse with very little padding to cut away.

It would be wrong to consider this method a replacement for all groups or depths, but it does provide a credible alternative, and this software certainly encourages creativity in the actual research design.

Client perspective: Claire Dally, GfK Automotive, London

Claire Dally is a Research Manager at GfK Automotive in London, and has recently completed a multi-country study of over 190 people across 15 participant groups using Revelation.

She describes her experiences: “Revelation is very intuitive, easy to use and has a visually appealing interface.  We took advantage of the Helping Hands package, where we were given a dedicated member of the Revelation team to guide us through this multi-market project and set up some of the scripts. They were extremely supportive throughout the whole process.  We also used their translation service and found the quality of the translation was excellent.

“A transcript created during a Revelation session will often be much longer than that of a focus group, because respondents generally have more time to consider their answers and to write down their opinions in detail.  With online qual I find you need to recruit double the number of respondents you actually need.  There are a number of reasons why you lose people: sometimes they are not available during the whole fieldwork session; others may lose interest along the way.

“You do lack some of the rapport you gain face-to-face.  Participants log on at different times, so you don’t necessarily get the chance to probe on things straight away, and therefore you can lose momentum.  However, there are ways of building rapport; you need to invest some time in the start of the fieldwork warming up respondents, and making sure you keep on top of each person’s response, so that they feel someone is reading their comments.  Participants can also upload photographs or videos to illustrate their ideas.

“Clients find the software easy to use too.  They are able to log on as observers, watch comments being made in real time and suggest probes via the moderator.   This means that clients feel really involved in the moderation process and that all of their questions are being addressed.  However, Revelation is not suitable for all types of projects.  If your client wants participants to view confidential stimuli, you have no guarantee that any material tested is not screen grabbed, copied down or viewed by others.

“Participants also find the experience very positive too; they are able to log on at a time suitable for them, and many have commented about how much they have enjoyed taking part.  You can be slightly less structured in your approach, allowing the topic guide to evolve over the fieldwork session, rather than relying on a pre-determined list of questions.  New questions can be loaded up on a daily basis if required.

“It is a cost-effective way of running online qual.  You can bring together participants from different locations without needing them to be in one place at the same time.  For less than the price of a UK focus group of eight respondents you could run a Revelation session of around 20 participants.  Our clients are becoming increasingly interested in online qual and we believe that this interest will only become stronger in the future.  We’ve been very happy with this software and what you can get out of it.”

A version of this review first appeared in Research, the magazine of the Market Research Society, April 2010, Issue 527



Has the Insight Show overheated?

Technology was an aspect of this week’s Insight Show that the exhibition’s promoters were majoring on, yet on the ground the number of technology providers exhibiting at the show was thinner than ever – I found just 13. Who was there? End-to end  mixed mode providers were represented by Askia, Confirmit, Merlinco, Nebu and Snap, plus online specialists Itracks and the newcomers on the block, ebox software. The niche providers were represented by E-Tabs (a niche maker in their own right for report automation), Centurion and Cint, for panel management, Intellex Dynamic Reporting for interactive analyis,  OnePoint for mobile data collection, Think Eyetracking, for, well, eye tracking, Visions Live, a new qualitative research platform, plus, rather strangely, a presence from Panasonic, featuring their Toughbooks as a rugged CAPI device.

Part of the reason for the shift of the Insight Show from the back end of the year to the middle (last year’s show was barely seven months ago, in November), was to merge four of Centuar’s marketing-related shows together under one roof, where they were colour coded and branded as MarketingWeekLive! Insight was in the orange corner. But lo and behold! Over in the blue corner, was SPSS, a big fish in the diminutive Data Marketing Show. They weren’t the only MR-relevant supplier to show up in the other quadrants – there were some research and fieldwork firms that had taken up positions elsewhere too. To the visitor, it was a bit of a muddle.

The Insight Show does have the feel of being on the wane since its heyday, if you listen to the crowd. But then I hear exhibitors moan each year that traffic is very slow, and most time is spent standing around in an excruciatingly expensive way: but identifying its heyday is elusive and illusory. This year, it seems day one was busier than the day two, when I was there. Yet I can remember being told there wasn’t a busy day at all in past years. Still, the day I was there  seemed to be the one when competing sales teams converged on the orange carpet between their stands to chat about who was up to what and complain about the heat.

I had assumed much of the reason for the merged format was because the Insight Show (which used to be big and standalone) was in danger of disappearing altogether, and alongside the other shows, it would find itself in the naughty corner. Not so. The Insight show was only second in size to the big and bold In-Store show. If the Point-of-Sale people can’t put on a good show, what hope is there for us research boffins? But it did make me wonder how many people, out shopping for illuminated fascias and storefront signage might find some online focus groups coming in handy, or those looking for a decent panel provider  being wowed by the ‘innovative trolley and basket systems’ on display next door.

Apart from the exhibitors, what was hot in the Orange corner? 2009 seems to be the year of online qual. Not only does Visions Live have a very interesting new multilingual realtime and asynchronous (or bulletin board) product which has come out of New Zealand, and already has a significant footprint in Asia Pacific, but then the other newcomers, ebox, seem to have put as much effort into developing qual tools as they have the quant online data collection.  It’s all very Research 2.0, although Itracks, who were also there, would make the point that they’ve been doing online qual from the time when people were still discovering their @ signs. And today I’ve just been given a private preview of yet another virtual qualie tool  (a very nice one in the making too) that locates the group experience in a virtual worlds paradigm.

Beyond that, software providers are talking seriously about automation – as they have for a long time – but they were also showing me things that were starting to make sense in simplifying tasks and saving time. Centurion have a new web-based interface out for their panel and sampling platform, called Marsc.net, which looked very nice – and they have built in lots of heuristic models for drawing samples for trackers. Intellex Dynamic Reporting had a number of smart new reporting goodies on display to make life easier, and can now go straight out to PowerPoint for report automation. The bright people at Nebu, on the other hand, have simplified the panel set-up process so that someone using their panel solution, could create and start populating a new online panel or custom community in just an hour or so – or as long as it takes to create the branding and imagery, in fact – their ‘panel in a box’.

But as I left, I was wondering if someone in Centuar had misheard what I certainly heard last year, that ‘the show would make more sense as a biennial event’ and optimistically decided to make it a biannual event. Hardly more than six months later was really too soon for this event, and the show definitely suffered as a result from the visitor’s point of view.

Converso Enterprise reviewed

In Brief

What it does

Platform-independent Java-based multi-modal interviewing and analysis platform with an integrated portal-style front-end

Supplier

Conversoft, France

Our ratings

Score 3.5 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 3 out of 5Value for money

Cost

In euro (€): Most modules €3,000 per user, plus €20,000 for Enterprise platform and €6,000 for web server module: all one-off costs. Maintenance: 18% of licence cost annually, or 25% for ‘gold’ support.

Pros

  • Extremely easy to use for moderators and participants
  • Can present a wide range of stimulus material
  • Offers several novel research techniques
  • Provides a complete transcript for analysis at the end

Cons

  • Only supports Windows for both moderator and participants
  • Not completely DIY yet: management module to be developed
  • Real time groups only – no support for asynchronous participation

In Depth

Converso Enterprise is an ambitious redevelopment project which deserves much praise for embracing what Web 2.0 technology has to offer head on. The portal-building and alert capabilities are excellent, and the main data collection platform is robust and sophisticated. But as an end-to-end solution it is still very much a work in progress. Substantial chunks are ready for production work now, but the gaps within and between these modules are just a bit too wide for comfort at this point. Given Conversoft’s recent rate of development, It is likely to look much more complete in as little as six month’s time, so for anyone planning to upscale their software platform next year, this is definitely one for the short list.

Converso Enterprise follows an entirely different architectural principal to most of the other new-generation research platforms on the market. Conversoft rejected developing in Microsoft’s .NET framework in favour of using Java, both J2EE, for desktops, laptops and servers, and J2ME (the flavour of Java for mobile devices). This approach does not on its own give the product a Web 2.0 pedigree, but it is a good start. It means that the software is totally platform-independent, so all users – researchers, respondents, technicians or end-clients – can use the browser or the operating system they want – Apple, Linux or any of the Windows varieties. This technical agnosticism extends to the relational database at the heart of the product, for survey data and panels, if used, which could be any of the modern database platforms – Microsoft, Oracle, or open source databases like MySQL or Postgres.

Conversoft also intends to create an open-source development platform to allow customers to extend the capabilities of Converso for themselves, but this does not exist yet.

What does exist is a wonderful portal-building tool that lets you snap into place any of the components of the Enterprise toolkit. You can create your own portal just for you or for entire groups of users – and then you can selectively switch on controls that will allow them to tailor the portal you gave them, to add in their own favourite things.

It could be the survey editing tool, a summary report showing the latest set of KPIs, an RSS news feed from the BBC or a link to Google Maps. This is where it gets exciting, because, once the missing developer tools have been developed, the techie people would be able to build whatever components you wanted to create so called ‘mash-ups’ of data from different sources on the internet, alongside your survey data – for example, to present geodemographic data in map form. What is more, Converso Enterprise components can be used as applets in other portals – so you could broadcast your poll results to other sites, or even Facebook.

Already, there is a rich library of components to choose from, particularly in the reporting area – which was never a strength for Converso in the past. It is relatively straightforward to create client data portals and dashboards that will present data graphically or as cross-tabs, or use intelligent reporting methods to highlight exceptions and provide alerts. Alerts are defined as triggers – really rather like dynamic filters that operate against the data and present a message. It all works fine with published data, but at the moment, you would struggle to show any real-time data from live surveys – such as to track response or get a live snapshot in a topline report. For these you need to resort to some of the legacy modules still.

Similarly, you can deploy new surveys through the portal, define your sample, and even use the very comprehensive access rights management tool from the portal – all of these are java programs. But the survey authoring tool is still a Windows program, and uses the old and rather complicated Converso scripting interface. That is promised for later this year, although nothing was available for Interface to obtain a preview.

These are not the only gaps waiting to be plugged. These are being addressed – and they need to be – though given Conversoft’s recent track record, the current feeling of being on a new highway where the cones are still in place, should have gone by the middle of 2008.

On the plus side, there is true multimodal interviewing with CATI. Web CATI is an integrated and very versatile handheld interviewing capability that will work on a very broad range of smartphones and BlackBerrys. The mobile interviewing is a new development and is impressive. There seems to be complete backwards compatibility with the old Windows-based CATI too.

Panel management exists but is not fully developed yet – the main panel management and respondent selection capabilities are there, but the panellist recruitment and community part is still missing. When it comes, it will offer integration with CRM systems, to use customers as a sample source, or to create customer panels.

The analytical tools are starting to look impressive too. A range of tables and charts can be created and presented directly in Word, Excel or Powerpoint, and it will populate native Excel or PowerPoint objects with data, to permit dynamic linkage. But if you wish to move data out to other MR analysis tools, then you are stuck until the planned Dimensions and Triple-S links are ready.

Converso Enterprise is an ambitious redevelopment project which deserves much praise for embracing what Web 2.0 technology has to offer. The portal-building and alert capabilities are excellent, and the main data collection platform is robust and sophisticated. But as an end-to-end solution it is still very much a work in progress. Substantial chunks are ready for production work now, but the gaps within and between these modules are just a bit too wide for comfort at this point. Given Conversoft’s recent rate of development, it is likely to look much more complete in as little as six months’ time, so for anyone planning to upscale their software platform next year, this is definitely one for the shortlist.

A version of this review first appeared in Research, the magazine of the Market Research Society, November 2007, Issue 498

nQual Rich Focus reviewed

In Brief

What it does

Hosted software solution for online focus groups in real time. Features replicate many of the capabilities of conventional groups.

Supplier

?

Our ratings

Score 5 out of 5Ease of use

Score 2 out of 5Compatibility with other software

Score 4 out of 5Value for money

Cost

Single group is £450. 10 groups £350 each, with volume discounts available. Includes technical support and moderator training.

Pros

  • Extremely easy to use for moderators and participants
  • Can present a wide range of stimulus material
  • Offers several novel research techniques
  • Provides a complete transcript for analysis at the end

Cons

  • Only supports Windows for both moderator and participants
  • Not completely DIY yet: management module to be developed
  • Real time groups only – no support for asynchronous participation

In Depth

To some, the notion of conducting a focus group online is as unsatisfactory as one of those ‘restricted view’ theatre tickets where all you see is a triangle of mostly bare stage. Whether behind the pillar or in an online focus group, you may need to try harder to discern what is going on without eye contact, body language or other visual clues – and all can be deceptive at times. Yet a restricted seat is infinitely better than no seat.

To the generation that thinks nothing unusual of making friends and sharing comments with strangers through such social network sites as Second Life, MySpace and Facebook, nqual’s Rich Focus research platform actually makes much sense as a coherent research method than much of the current research hubris around Web 2.0. What is lost is made up through access to targets that would be impossible to reach by other routes.

Rich Focus is a web-based for online research in real time but in virtual space. In many ways it resembles web-conference tools for business meetings like Webex or GoToMeeting – yet without the accompanying, and often excruciating inefficient 10-way telephone conversation. Rich Focus assumes all the conversation is carried out in instant messaging chat, but in a very research-literate way.

The process starts conventionally enough as you write your topic guide, which you upload in advance. You can recruit offline, or from your own panel, and once you have agreement, you send them an initial email with a software download, which is a one-off activity. One plug in will work with Internet Explorer, but a better solution is to use the dedicated nqual plug-in and browser, which locks down the capabilities and avoids all problems with participants’ browser settings.

Initiations are then issued with an embedded link and at the appointed hour, everyone logs on and can ‘see’ who else is online – identified by a name and a colour. For respondents, the screen is carved up into three panels: one where they can see their name and other member group, including the moderator; another where the questions, answers and discussion takes place, and a third area where stimulus material can be presented.

Moderators have a few more controls, but not so many to make it complicated. As you click on the topic in the guide, a timer starts, if you have added timings to your guide – you can see it counting down, but tactfully, neither participants nor clients can. All contributions from participants are colour coded to match their names – a simple aid to differentiate people and their comments. When a respondent types, a visual light bulb by their name tells you that a response is imminent. It is a simple but useful device that calms down the tangle of overlapping comments that is familiar to most Second Life and other online chat users.

As you work your way through the deceptively simple but intelligently thought out capabilities in this software, you soon see other ways that the more obvious lacks of the online focus groups are compensated for by the new tricks you get in return.

You can introduce stimulus material – images, sound or video – and you can give your respondents control to add pointers, positive ticks or negative crosses, or to circle things and add annotations – a versatile set of tools which opens up endless possibilities for concept testing, ideation, co-creation and more. At any point, you can also turn off the ability for participants to see what others are saying, to get reactions uninfluenced by others, then it can be switched back on once everyone has said their bit, for general discussion. A built in polling tool lets participants vote in secret, then the results can be viewed as a group, for more discussion.

The ability to move unobtrusively between public and private comment also allows a moderator to make a one-to-one comment to a contributor, or even for a co-moderator to step in and do some offline how-did-you-know-that type of questioning. Client observers can also talk one-to-one to group moderators, and moderators can even appeal for help if the something they have not briefed you on should rear its head. In a ‘real’ group, such behaviour would be highly destructive.

When the group is over, the greatest joy is to get the full transcript right away, as a Word or Word or Excel file. The Excel output is organised with comments in one column and contributor name in another, which makes for very simple sorting and filtering. It is easy to import this into CAQDAS tools such as MAXQDA or XSight, though unfortunately the structure is lost – a structured exchange (like triple-S for quant data) would be very nice here.

At present, the software lacks the management tools to let you start new surveys, upload the guide, issue invitations and download transcripts for yourself, though these features are planned. For the time being, nqual performs these tasks for you. It is also a pity the respondent download works on Windows PCs, effectively cutting out any Mac users you’re your research, which could be an issue in some sectors. These lacks at the periphery bound to be remedied as demand increases – there is much good research to be done with what Rich Focus already provides.

Customer viewpoint: Paul Dixon, ICD Research, London

ICD Research is a full-service agency in London providing a typical mix of qual and quant research – around 70 per cent of which is carried out online. Group MD Paul Dixon has used Rich Focus extensively, and is impressed:

“I have been interested in online groups for a long time. From a research perspective you can tell that nqual has been designed with researchers in mind, as it captures a lot of what you want to replicate in an offline focus group. It is very well laid out in terms of your topic guide but still allows you the flexibility to follow an impromptu direction. It also works extraordinarily well in terms of being be able to have a second moderator liaising and talking privately to the client and for the client to be able to feed into the group indirectly when they need to. If a client has to interrupt a conventional group, it really disrupts the group and can undermine your authority.

“Clients really like it – they see it as very distinct in its application. There will always be a need for offline groups for particular topics or where the analysis focuses more on the interactions and non verbal elements, but this works extraordinarily well for proposition testing, website evaluation, advertising testing and it offers clients a way for them to engage more with their customers.”

One of Paul’s favourite features is image tool, where respondents can mark up pictures, illustrations or even websites being displayed directly. “They can circle what they see, or put a tick or cross. It almost gives you quantitative information in what they react to. Nqual have just managed to get all the little components and the small barriers to online research right.  For example, the ability to see who is typing in real time dispels the barrier that online can be disjointed. You might ask another question before people have answered the first one. With this, you can see when people are typing and wait for their response.

“The transcript is probably one of the biggest advantages. With offline you send off your videotape and get the transcript a week later and after a hefty fee – and you get no names next to the transcript. Here, within a matter of hours, you get transcript with the persons name, what they have said, and the ability to filter it. It is making life so much easier for the team here. And because we select our respondents from our online panel, we have all the background information too. I like the simplicity of it – from uploading the topic guide to downloading the transcript, it is a seamless process.

“It is never going to replace offline, but it is a viable methodology in it own right. It works and we have proven it over the 30 or so online groups we have conducted online over the last two years. I think I am actually getting richer data, and from people or audiences that sometimes you would not be able to attract to attend a group in Central London at say, 6pm. I think nQual are onto a winner with this.”

A version of this review first appeared in Research, the magazine of the Market Research Society, October 2007, Issue 497

Vision Critical reviewed

In Brief

What it does

Web-based, hosted solution for high quality web surveys with an integrated, fully featured panel management solution and some support for online qualitative research.

Supplier

Vision Critical, Canada

Our ratings

Score 3.5 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 4.5 out of 5Value for money

Cost

Per month: approx £1,250 for first 5000 panelists, £500 for each additional 10,000, with no cap on number of surveys, interviews or users.

Add-on modules for analysis, communities etc between £150-450 per month.

Allow £7-10,000 on-off for set-up, design and customisation.

Pros

  • Complete range of panel management and community building toolsSophisticated Web 2.0-like survey capabilities
  • Links together all a panellist’s responses to any survey across the database
  • Lets in-house researchers or smaller research firms run their own panels

Cons

  • Authoring and management is Windows and IE only

  • Limited animation support in survey tool

  • No enterprise version offered

In Depth

Given that many researchers wishing to do online research are likely to want to develop and run their own panels, it is odd that the Achilles heel of most online survey tools is still in providing decent panel management capabilities. Some get no further than giving you a database, a recruitment survey and some invitation management capabilities, others not even that far. Not so with Vision Critical, a Vancouver-based company that has spent the last three years building a very different online survey tool from the normal fare. Panel management lies at the heart of its Vision Critical web-hosted online survey tool, with not only facilities to fine-tune the selection of respondents from the panel database, but a complete range of tools to aid building the panel, nurturing a panel community and communicating with panel members and keeping it in peak interviewing condition.

Vision Critical is the first foray into technology for veteran Canadian researcher Angus Reid, and former owner of Angus Reid Group, which was acquired by IPSOS in 2000. The benefit of having a researcher shaping the technology is apparent throughout this software. More than any other online research tool I have reviewed to date, this is a product that acknowledges almost equally the needs of researchers, research clients and, with impeccable grace and style, the often overlooked needs of respondents. If you are seeking a more respondent-friendly way of conducting research, then this is certainly one to review.

The software is provided exclusively as a hosted ASP solution – Vision Critical have no plans to produce a version to install in-house. Therefore all of its capabilities are invoked from a web-browser interface. It is clear that Vision Critical is pitching for the corporate user rather than the agency, though the product should also appeal greatly to smaller research companies with no desire to run their own IT. The software comprises five key functional components: panel member administration, sampling, survey authoring tool, survey deployment and two reporting modules: a simple real-time tool, and a more advanced tool, about to emerge from beta testing.

Though it is not released yet, the new Dynamic Reports module module is intriguing – allowing you to use the same tool to run one-off reports for instant viewing, building complex reports as PDF, Word or Excel files, or publishing them to the web in the manner of a data portal for controlled viewing by clients and stakeholders – which will then change dynamically when you release more data into them. The approach here, and throughout the suite, is to not try to be too ‘clever’ in terms of functionality – avoiding options for options’ sake and going for sensible defaults so that it is easy to get started and become more sophisticated over time.

The Panel Plus module is fully released and in widespread use. It contains everything you need to run your own panel without having to recruit an army of admin staff to run it. It works with the sensible assumption that panel members should be able to self-administer their profiles, participation and reward redemption for themselves through the capabilities you can build into the panel member site. Though you can do all of this yourself, if you wish, designing the site and customising it is part of the set-up service that Vision Critical offers.

It also takes pays more than lip-service to the notion of a panel being an online community. For example, it provides the means to edit and publish your own online newsletter for panel members, which could be daily if you wished. It offers a content management system to let you publish findings and reports to your panel members, which is critical when creating ‘professional’ panels. And most interestingly in the context of Web 2.0, it offers the opportunity to blur the edge between researcher-led and participant-led qualitative discussion by integrating online forums or even blogs. It will also interact directly with commonly used Web 2.0 social networking portals.

There is a nice survey authoring tool too, which is well laid out and easy to use, with a very full range of functionality. It will also let you integrate with Java or Flash components, though it is disappointing that this otherwise futuristic tool does not have more in the way of animation and interactivity at present – though these are supposedly in the pipeline.

It is in the connection between panel and surveys that another astonishing difference emerges. The classic organisation for survey tools is to make each survey a discrete set of records, maybe linked to sample or panel indirectly, but with no actual interconnection. Some provide the mean to post back key variables into a respondent’s profile, but only to a limited extent. With Vision Critical, everything is interlinked, with every survey question answered being attached directly to the respondent’s record, and every prior answer being available for re-use. In case this sounds like chaos in the making, with a minestrone soup of a million questions to choose from, the coup de grace is that the variables are still presented as organised by survey, even though it is effectively just a slice through the database. But it means that at any time, all prior responses that a respondent has given are available to you to use for routing, sample selection or quota control – and also at the analysis stage, such as for profiling or comparison. So long as the respondent was asked the question and provided an answer, anything can be cross-referenced by anything.

But isn’t that what panel-based interviewing systems should by like?

Customer viewpoint: Dru-Ann Love, Business Week, New York

Dru Ann Love is a Primary Research Analyst at the US Publication BusinessWeek Research Services, based in New York. In addition to carrying out surveys and polls that often appear in the pages of BusinessWeek, and surveys among their print and online subscribers, the research group operates as an independent full-service custom research provider. One of its prized assets is its BusinessWeek Market Advisory Board — a controlled access panel of many thousands of business leaders and opinion formers.

This year, BusinessWeek Research Services switched to Vision Critical in order to bring control of its panel in house – although the panel is actually hosted on Vision Critical’s servers, Dru Ann is able to carry out all of the management aspects concerning the panel’s operation for herself.

Dru Ann recalls: “When we had the demo, our reaction was ‘wow!’ you can do all of this. And once we started using Vision Critical, it was really good. I enjoyed it – it was really fun.”

One of the first tasks for Vision Critical was to migrate BusinessWeek’s high-profile panel into their panel management suite.

“Their role was instrumental as we migrated our existing panel,” reports Dru Ann. “And they did a better profiling questionnaire than we had previously.” This has enabled the Research Services team to obtain much more information about their panellists.

Dru Ann also points out the sample selection capabilities as being a particular strength of this software: “I have control over who I selected, and I can verify the source of my target markets. For example, if I need to select a target of ‘all Directors or above’, I can see exactly who I’ve got, and I know to my satisfaction that I’ve got the group that I am targeting.”

Her experience with the questionnaire design module is also that it is “attractive and easy to use”. Again, Vision Critical, as a part of the migration process, defined templates so that the panel members’ portal, and also the surveys have a consistent look and feel, reflecting the BusinessWeek design ethos.

Dru Ann uses the built-in reporting features while the surveys are active. She comments: “I really like their online status report which has a feature which shows completes, so it gives me an idea of how many people are taking my survey at any particular time. If I start noticing that I am not reaching my target, then I can send another email blast to get some more respondents.” More detailed analysis is carried out in SPSS, using the capability provided in Vision Critical to create a complete SPSS file of the data.

She concludes: “I have no complaints. The software is really nice, and the people have been nice too, whenever I have needed to contact them.”

A version of this review first appeared in Research, the magazine of the Market Research Society, July 2007, Issue 494.