The latest news from the meaning blog

 

Confirmit Horizons Reviewed

In Brief

What it does

A complete mixed mode interviewing system with high-quality support for CATI, CAPI and web interviewing, integrated with panel and community management, and telephony integration for CATI, operating in a web-based or cloud computing environment.

Supplier

Confirmit

Our ratings

Score 5 out of 5Ease of use

Score 4.5 out of 5Compatibility with other software

Score 3.5 out of 5Value for money

Cost

Confirmit Horizons CATI hosted solution. Entry-level system from £8,000 per year. Web surveys and other pricing on application.

Pros

  • Captive application for web CATI interviewers and supervisors (rather than a web browser interface)
  • Switch easily from Web to CAPI or CATI and back again
  • Clean, modern and customisable look throughout
  • Open system capabilities through a range of API extensibility kits

Cons

  • Sample loading capabilities are rudimentary
  • No offline scripting capabilities

In Depth

There is a new professional telephone interviewing system on the block with a surprising name to those familiar with it – Confirmit. Confirmit Horizons, released earlier this year, extends this granddaddy of a Web survey platform into the phone room and also the street and shopping mall, with full online/offline CAPI too.

Presumptions that CATI would dwindle away as an interviewing channel, in the face of Internet research have so-far been wide of the mark, rather like those predictions about long-awaited paperless office. Industry stats show decline, but CATI is far from collapsing. For many years Confirmit (in its earlier FIRM days) appeared to procrastinate over whether it would or would not introduce CATI into its Web survey offering. Then two years ago, it bought up Pulse Train, which appeared to have two jewels in its crown – a superior reporting platform, Pulsar, and a real workhorse of a CATI system, Bellview. The combined firm set about merging together the two product lines – an initiative that is not for the feint-hearted.

In a surprisingly short period of time, a merged set of tools for data collection has emerged, and in doing so, the Confirmit product line has matured into a very comprehensive offering for the whole spectrum of quant research. The manufacturer has clearly learned from the mistakes of others. First, there is an upgrade route for Bellview customers to convert their legacy QSL scripts into Confirmit, and secondly, the revised platform is not a hotch-potch of legacy modules and new modules – the capabilities of the old Bellview system have almost all been reproduced within Horizons without compromising the Confirmit environment or way of working.

Bellview was a very flexible system for CATI, giving a lot of control to supervisors over the management of interviewers, sample and callbacks through its admin interface and offered limitless possibilities to the script writer. Scripts were written either using a proprietary script language called QSL or a graphical (GUI) authoring tool called Visual QSL — though in reality, most people wrote QSL syntax. Confirmit, on the other hand, has always been a GUI system, and the developer took the brave decision to go with GUI alone. There are those who will defend the syntax approach to the bitter end on the grounds that it is ‘more efficient’ and quicker to do certain tasks. But QSL was quirky and took time to learn – and it was always easier to make mistakes than it was to notice them and correct them.

The Confirmit authoring GUI is, in practice, highly efficient and the developer has worked on optimizing it. There is often a fear that a Web-based editing interface will be sluggish in operation. Not so with Horizons – moving from screen to screen is instant and effortless. There will be a fairly steep learning curve for anyone experienced in QSL that has not worked with Confirmit, as the design interface is completely different.

The legacy code bridge will not convert everything – it seems to get you around 80 to 90 per cent of the way to a working CATI script in Confirmit, and it deals with all of the more tedious aspects of converting texts, even in multiple languages, variables and logic, but you will probably have to patch up any complex execution logic or clever scripting by hand. Another bonus is that a lot of the concepts are very similar between the two systems, such as the block-structured skip logic, and those that were missing from Confirmit (such as QSL’s ‘pblocks’ for handling beginnings, endings and handover from one mode to another) all now appear in Confirmit as ‘call blocks’. QSL users will, however, miss some of the syntax tricks that were possible – the Confirmit interface is not as slick as QSL in writing logic and performing operations on variables.

Better-looking CATI

There is no separate CATI-specific module – all of the interviewing and supervision capabilities are also implemented now through the main Confirmit platform, which has been skilfully enhanced to provide a true CATI experience to users, within a Web-based environment. However, to avoid the problem of Interviewer and Supervisors having to work in a Web browser, both of these interfaces are provided as dedicated applications, or ‘consoles’. This is easily set up on Interviewer machines, or if interviewers and supervisors are working remotely, they can be emailed a link to download and install the package.

CATI interviewer screens now take on a very web-like appearance – you can control their exact look and feel through the same template gallery as a Web interview, and there are some templates optimised for CATI too. However, the console environment also allows interviewers to control all aspects of the interview from the keyboard, rather than the mouse, which is typical in a Web interview – though they can also use the mouse if they wish.

Clever reporting

The web environment certainly improves the look of the CATI screen – they can be branded, for instance, with the clients logo, or made distinctive for different projects. These are not merely cosmetic matters – theming the screens can help with project recognition, and better screen design can aid concentration and legibility. They can also contain many more answer options.

The supervisor console similarly packs a punch, with added style, for supervisors. Again, how it does things is a little different for those familiar with Bellview, but the concepts are similar, and a lot of effort has gone into making the features intuitive to use. A real innovation here is the alerting report – triggered by events such as an interviewer not entering any data, or an appointment falling overdue. This is potentially a big money-saver for CATI units, as it makes it feasible for a single supervisor to supervise 15 or more interviewers as effectively as 6-10 with a more conventional system. This could even free up a few monitoring stations for interviewer use instead.

The next day CATI centre

Telephony integration is also possible through Magnetic North, a telephony specialist, with interfaces for both conventional telephone lines or VOIP. It is a web-based dialler, which means it will support either conventional call centres or satellite offices and homeworkers. When an interviwer logs in, they are asked to enter the number of the phone on their desk, and a call is placed to that number. The line is then kept open, and the dialler then starts placing calls. The dialler will support every level of automation from click-to-dial through to full predictive dialling, with the supervisors able to monitor and control the nuisance call rate through the alerting reports.

The telephony support also allows for recording of chosen segments of the interview, such as an open-ended question, or the entire interview. The web telephony also means that you could set up a real or a virtual call centre using the cloud computing model pretty much overnight.

Where does it end?

The sheer depth of functionality becomes apparent when you take a look at the other interfaces offered, to allow you to integrate Confirmit with other software. The software is developed using Microsoft SQL Server and using the Microsoft .NET framework, which already makes integration relatively straightforward for other Microsoft users. Confirmit have also embraced Web Services with a passion – this allows applications to exchange data and drive one-another across the Internet, effectively going in through the back-door but in a highly secure and also efficient way. There are currently nine different APIs for such touch-points as reporting, data transfer, quotas and panel integration. There is also an ‘extensibility framework’ to make it easy to develop specialist applications within Confirmit. One has already been designed by a third party for fraud detection and another is being developed to support interviewing on iPhones.

It is going to be hard in future for the humble reviewer to find anything that this software cannot do, since there are likely to be many more of these third party plug-ins. Indeed, the few niggles I do have are relatively minor in scope – sample management is still a bit too Web interview-oriented, and you have to be a Windows user to exploit the software. Confirmit are cagey about the pricing too, but it appears to be towards the top end.

Confirmit have pulled off quite a feat with Horizons – it’s a serious product with a friendly face that makes for a hard act to follow.

A version of this review first appeared in Research, the magazine of the Market Research Society, September 2009, Issue 520.

Key Survey reviewed

In Brief

Key Survey version 7.1

Worldapp
Date of review: April 2009

What it does

Online survey data collection and reporting system provided on a Software-as-a-Service basis by a USA provider represented in the UK and elsewhere. Sophisticated mid-range web survey tool with a lot of flexibility and sophistication, at a highly competitive price.

Our ratings

Score 4 out of 5Ease of use

Score 3.5 out of 5Compatibility with other software

Score 5 out of 5Value for money

Cost

Annual fee for an unrestricted licence in US dollars: single user $3,950, additional users at around $3,500 (cost reduces by number of users). Annual usage-based licence for up to 10,000 completes at $1,950 per user and 25¢ for extra pre-paid completes.

Pros

  • Browser based and system independent – works on PC or Mac browsers
  • Simple GUI for most operations with a powerful scripting language in the background
  • Highly customisable and extensible through a series of plug-ins and a software interface (API)
  • Built-in error checking capabilities

Cons

  • No panel management
  • Analysis capabilities somewhat restrictive
  • Only exports to SPSS and Excel. No Triple-S.
  • Cannot create your own plug-ins.

In Depth

Researchers looking for a flexible web survey tool will be familiar with the conundrum that the more affordable entry-level tools lack the flexibility you need, while the grown-up ones not only cost a fortune but can be bewildering to use. Key Survey is a relatively new entrant to the UK market which focuses on the middle-ground and is aimed at researchers who want to build and deploy sophisticated surveys for themselves. Compared to other products on the market, Key Survey sits towards the high end of the middle ground. For a relatively low fixed cost, based on the number of users, you get a hosted solution with no restrictions on the number of interviews you do each year – something that is almost unheard of in the software-as-a-service market.

Key Survey not only covers all the basics in style – like all the standard question types, all kinds of survey routing and logic, text piping, logic on answer lists, question libraries, look-and-feel templates, multiple languages, survey invitations and reminders – it also leads the non-technical researcher into some very sophisticated territory. It offers analysis and online reports, but does not provide any panel management support.

It has a question library, which comes populated with hundreds of well-worded questions organised by subject, and very useful search capabilities that other tools often lack. You can add to the library, or treat any prior survey as part of the library. Using this could help you to standardise survey design and harmonise demographics across surveys.

There are over a hundred different design templates to choose from, and it is easy to take any of these as starting point and design your own templates too. These are all based on CSS (cascading style sheets) so they are sound at a technical level, but you do not have to get your hands dirty with any actual CSS coding, unless you actually want to.

One of the most versatile capabilities is the collection of plug-ins. These allow you extend the basic functionality into untold areas of sophistication – and there are no extra costs for using them. Each essentially goes off and performs a task and then returns any resulting data back into the questions you have defined in your survey. There are plug-ins for various kinds of flash animated questions, such as sliders, calendars for selecting dates, or using Google Maps to choose a location. There are plug-ins to validate and clean data on the fly, reach out to an external database, or to perform geo-IP checking on survey respondents, to detect those pretending to be in countries they are not. There are already over 20 plug-ins, and whenever a customer requests a new one, WorldAPP make it available to all customers too.

Logic-related errors are a common and expensive problem with surveys as they become more complex. I particularly like the Key Survey approach, which has a level of logical scrutiny built into it so that it will advise you of ill-formed logic, questions that may never get asked, and so on. Furthermore, wherever you make logic selections for routing, filters or text piping, a simple on-screen assistant anticipates the choices you are about to make, and presents you with context-sensitive overlay of the questions and answer options available – it is ingenious and very intuitive, and will actively help you stamp out most common scripting errors at source.

My only concern over the design interface is that the survey questions and answers are presented as a single scrolling page (though they can be displayed in the survey on different screens) and on long and complex surveys, this could become difficult to navigate through – especially the buttons and actions you need at the top of the page quickly scroll out of sight. It feels more web page than web app, in this respect.

The software does offer support for other self-completion streams of data too – there are additional cost modules offered for print/mail/scan surveys on paper, and for IVR on the phone, which share the same design and reporting environment.

The analysis and reporting side is a bit of a disappointment after all the capabilities in the upstream areas. It does make it relatively easy to create simple reports showing frequencies and charts, and these can also include a wide range of statistics. For some users, this may be sufficient, but the reporting formats are inflexible and most researchers will struggle to use the tool to probe their data fully and then build reports to communicate what is salient. What it does seem to be good at is letting you create quick snapshot reports that can be updated regularly as results arrive and which you can publish and share with your clients.

Users wanting to do deeper analysis are likely to want to move the data into other cross-tab or statistical tools, and here your luck runs out if it is not SPSS that you use, as that is currently the only export route that provides you with labels and definitions ready to run. WorldAPP is working on further exports, though the next one is likely to be SAS.

A particularly nice feature of the software is the live help button – this connects you immediately to a support representative who will answer any question you have using instant messaging chat. WorldAPP provides support from its three locations in Massachusetts, London and Kiev, in the Ukraine, so someone is usually available to answer questions around the clock. Support is also included in the fixed annual fee.

Customer viewpoint: Monica Coetzee, Research Manager, The College of Law

The College of Law is the UK’s largest dedicated postgraduate law school, operating from six campuses across England. It’s been using Key Survey for the last nine months in its research division, which carries out a broad range of surveys including student surveys on course quality, HR surveys, a continuous customer satisfaction survey on IT services and even ad testing, all done online with samples that are often in the thousands.

Monica Coetzee, Research Manager, explains: “We were previously using a desktop survey tool, but because there was only one person trained to use this software, it was causing a bottleneck. We wanted to restructure the department anyway and I wanted to change to working online.”

“As a non-profit organisation, price was critical for us when choosing a replacement, but we also needed something with a lot of advanced features. We reviewed several tools and Key Survey came out on top.

Monica’s team now have around 30 surveys done with Key Survey under their belts. “What I like about it is that it allows all three of us to have access to the software online from anywhere, giving us greater flexibility, for example when working in different College centres. Each person has responsibility for an entire survey project and not just for specific tasks.” This, in many cases, includes exporting report-ready data into SPSS. “It exports it directly with all the data and value labels correctly filled in”, she reports.

“We now know there is nothing that we cannot do – and we use all sorts of advanced features like show/hide options and complicated skip routing. We very often bring data we already have into the survey and use the autofill function to use it for verification or routing. And it all works very well.”

A particular requirement for Monica is the visual appearance of surveys, and she was pleased to be able to produce a set of templates matching the corporate visual identity of the College that could be used consistently across all surveys. “Although they have a good range of templates, there is the option to go into the HTML programming, so I can play around and see what happens to make my own. With very little HTML knowledge we have produced some really nice-looking surveys.”

Monica also appreciates the built-in warnings and safety features of the system. She remarks: “If you try to do some things that would result in an error, it will give you a warning or prevent you from doing it. But once you have gone live with a survey, if you need to make a change, you can react so quickly with this – it will make that change instantly. It must be said, my favourite function is the live help button. They almost always answer within a few seconds, and they are always very friendly and helpful. I actually prefer using a typed interface to a phone calls – I find I am generally more comfortable communicating online in this way.”

Published in Research, the magazine of the Market Research Society, April 2009, Issue 515

Catglobe reviewed

In Brief

Catglobe version 5.5

Catinét a/s, Denmark Date of review: February 2009

What it does

Web-based software-as-a-service (SaaS) product for mixed mode data collection and analysis, including CATI, CAPI, CAWI and integrated panel management.

Our ratings

Score 3 out of 5

Ease of use Score 4.5 out of 5

Compatibility with other software Score 4 out of 5Value for money

Cost

Variable cost based upon usage. Start-up costs typically €3,500 for configuration and training then €0.015 per panel member and €0.03 per interview minute, with some additional charges applicable

Pros

  • Completely web browser-based – supports Internet Explorer or Firefox on PC or Mac
  • Simple GUI for most operations with a powerful scripting language in the background
  • Strong on panel management and sampling capabilities
  • Good range of imports and exports including Triple-S, SPSS and Excel

Cons

  • Data analysis is inflexible and limited in scope
  • GUI questionnaire editor is cumbersome to use
  • Some performance issues – complex sample queries can be slow to run

In Depth

It may seem as if we are spoilt for choice with data collection software packages, but if you are looking for a multimodal interviewing solution that is also web-based, the choice is relatively narrow – especially if web-based CATI is part of the mix. So it’s good to welcome a new entrant on the scene in the guise of Catglobe, a mixed mode interviewing system offered as a SaaS (software as a service) product by the software division of the Danish fieldwork company Catinét. The SaaS model makes it very easy to get started with little infrastructure in-house – all that is needed is a reasonable internet connection and a web browser. It is not fussy about which one: Firefox on Windows or Mac, or Internet Explorer on Windows works equally well. Catglobe is a surprisingly vast system, and the fact that it has been extensively road-tested by Catinét’s in-house fieldwork team is evident in the range of capabilities and options provided. There are different modules for sampling, questionnaire authoring, fieldwork management, reporting and some report automation. The interviewing module supports CATI, laptop CAPI and CAWI and even has a special Hall Test mode for a temporary local network of interviewing stations. All of these modules are accessed from a central home page through a pop-up menu similar to the Windows start button. Behind all of this is a single relational database which holds all of the assets or resources relating to your surveys – questionnaires, survey responses, respondents or panellists, interviewers, reports and so on. This is one of those advanced systems that moves you away from rigid boundaries of the survey to define how data are organised. The concept of the survey still exists – however more as a workflow concept. The system presents the surveys you have available to work on as a folder structure, which you can model as you wish. However, in the background, all the survey does is provide a convenient, organising view of the central data repository. Questions and response data from one survey are easily accessible from others, if you can make a connection through questions or respondents in common. This opens up endless possibilities for using your data more intelligently both in sampling and in analysis, and it makes the logistics of running one or more panels really simple. Panel management is an area Catglobe handles particularly well. At the sample selection stage, there is a wonderful tool for building ‘groups’ – which are effectively a database query. You use a group to pull a sample from the respondent database. However, this is a query tool that understands concepts such as key demographics, sample frames, frequency of previous response and interviewer resting rules. It then ties in seamlessly with the ‘communications’ module that serves invitations and reminders for web surveys. These work directly from a library of templates, so it is very quick to set up an invitation from an existing project and adapt it slightly for the survey. The system is fully multilingual, so invitations can be templated in several languages then dispatched in the appropriate one for each respondent. As it is also truly multimodal, samples can be drawn for CATI or CAWI in parallel. The workflow is well-designed, so it is not only quick to run through the process from end to end, but also flexible when changes are needed, or if the sample requires a boost part-way through the fieldwork. Panel recruitment works equally well, and there is considerable scope to automate this, including ongoing top-up recruitment. Recruitment can be by web or by telephone, and a phone recruit can be used to trigger an immediate web survey invitation for new panellists to complete their profile data. There is also an elaborate points allocation and redemption capability too, if you wish to incentivise your panel. Access to surveys and functionality is managed from the HR module, which allows you to define roles and allocate individuals to roles. Respondents and panellists are treated in the same way: everyone from the system administrator to the panel member is registered as a user and has usage rights associated with them. The majority of the system has a very cohesive appearance, which is simple to follow – it passes the test of deceiving you that you are using a desktop program, when in fact it is a browser-based web app. At the bottom of the screen are two buttons – one labelled Tools, which is the ‘start’ button that gets you to all the different modules; the other is the Folders button, which takes you to a tree view of folders containing surveys, questionnaires, templates – essentially all your data. The questionnaire editor has a somewhat different feel, and is not as well-crafted as the other modules. It does provide pretty much all of what you need, but it feels clumsy to use. You can view a list of the questions, but important details, such as the answers, can’t be seen without going into the question itself. There is no overview of the logic or routing, which makes like difficult for the scripter. There is a powerful scripting language available, and parts of the questionnaire created in the GUI can be exported out into this too, which makes writing it (and learning it) much easier. This is an excellent capability. Unfortunately, you are likely to need it more than you should if writing web or CATI interviews of even a medium level of complexity, such as creating a constrained sum set of questions. Really, more options should be built into the questionnaire editor. Also on the downside, actual users have reported sluggish performance with some of the database operations such as drawing sample or exporting results once the number of records is in the hundreds of thousands – though Catinét report that they have worked to improve this. The reporting capabilities are also unlikely to meet most users’ needs at present – there are some nice features there, but Catinét have ambitious development plans for the reporting side, so this is likely to be improved over the current year.

Customer viewpoint: Ólafur Thor Gylfason, Market and Media Research, Reykjavik

MMR moved to Catglobe a year ago, in order to move to a single interviewing platform for CATI, CAPI and web interviewing replacing a range of different packages in use to that point. As Ólafur explains: “The good thing is we have been able to use this platform for everything we do from CATI recruiting of panels to CATI phone interviews, CAPI and CAWI. “There is a powerful programming language within the software so when we do complicated surveys such as international surveys where you have to produce an exact data map afterwards, we can write the data handling programs in advance, so that when the survey is finished, we can export the data in the exact format the client requires straight away. With this programming language there is nothing you can’t do with the software, provided you have a little bit of programming experience. “Another positive thing about the software is that we use it for open-ended coding and this capability is very powerful – we can do this on the fly so that the turnaround time on projects can be reduced considerably. “We use it for CATI recruiting, and once the phone phase is completed, the automated CAWI questionnaire is sent out immediately, and everything is always interlinked, so it is very good for us. “With panel management, there are two key points. Firstly, because it is completely multimodal, and all recruitment is done by phone but recruits are immediately served a web survey to complete their profile. Everything happens at the same time. Secondly, their sampling is very easy to work with. It makes sure there is the right load across the sample and making sure that panellists get the right number of invitations, keeping track of invitations and reminders . Their ‘group builder’ is very powerful and very easy to use and the communicator, which is the email part of the system, links in with the group builder. “We have run into some problems, but the main thing for us has been that the support has been excellent – they are almost acting as a division within our company when it comes to support, so you forget about the bad things very quickly.” Published in Research, the magazine of the Market Research Society, February 2009, Issue 512

DatStat Illume 4.5 reviewed

In Brief

What it does

Complete end-to-end web-based interviewing solution with a database and object-based approach to allow greater freedom in working across surveys and incorporating other applications or sources of data into the research process.

Supplier

DatStat

Our ratings

Score 4 out of 5Ease of use

Score 4 out of 5Compatibility with other software

Score 4 out of 5Value for money

Cost

One-off licence fee starts at $25,000 for entry-level user. All licensees pay a variable transaction charge per interview, e.g. 85 cents when purchased in volume. One-off hosted surveys from $5,000.

Pros

  • Intelligent, intuitive user interface
  • Database approach makes it easy to link responses across surveys
  • Repository provides an managed library of reusable questions and objects
  • Comprehensive SDK (programmer interface) for easy linkage to other applications or to create your own objects

Cons

  • User interface, though web-based, only operates under Windows
  • More advanced features require some direct HTML coding
  • Online reporting not as well developed
  • Support and training from West Coast USA only

In Depth

Stand back from research, and you can see that the vehicle of the survey is as arbitrary in defining the boundaries of most research endeavours as is the nation state in defining language, culture and markets. Most of today’s research technology reinforces the silo effect of the survey, with the data collected and isolated from other surveys or other processes that may be taking place in parallel.

DatStat Illume, on the other hand, offers a way to break through these barriers by making it easy to build bridges between surveys, borrow questions or data from one to use in another, or link surveys to other activities taking place simultaneously online.

Not that Illume aggressively confronts you with this radical difference. The software presents very much like other online data collection and analysis products. It still has the concept of questionnaires or projects and these you create online by building up questions in a pleasantly attractive point-and-click authoring environment. There is the typical tree structure on the left, and space on the right to write your questions and answers, select options and tweak their appearance. Furthermore, you can work online, or offline, as you prefer. If you wish to work offline, you simply book your project out to you, so you receive a local copy to work on, which also prevents others from updating it online in the meantime.

You could happily switch to Illume and remain unaware of just how subversively permeable the software is underneath, with its rigorous and imaginative application of true database technology and object oriented architecture to the business of market research.

In Illume, each question is considered a self-contained object, as are folders or groups of questions called collections. Any object in the database is accessible, subject to permissions, from any other object. This brings tremendous flexibility. You can add your own objects too, such as a different kind of question, or a resource, such as a data feed from a CRM system or a gateway to panel provider to request sample top-ups on demand and in real time.

An entire survey is also considered an object, and any survey can refer to data from any other survey during interview or analysis, either in aggregate or though respondent linkage, where this can be achieved. This can bring all sorts of benefits to longitudinal research or cohort studies, to panels or as a means of generating new samples.

Creating objects is a task for a programmer, but it does not require a software change from the software manufacturer. Once created, objects can be re-used in all your future work too.

The object-driven architecture means the developers have been able to incorporate much better logic diagnostics into the survey authoring process than is typical. For example, if you update a survey and move a question on which branch logic is dependent, you will get a warning if the logic check now occurs before the question has been asked. It will also check logic to ensure you have not accidentally created any orphaned questions in your survey that will never get asked.

The object approach also helps explain how the software has one of the best library capabilities I have seen in a research package. This can contain questions, answer lists, questionnaire sections or entire model surveys. There is a permissions-based workflow covering who is allowed to submit new repository content, and who may approve it, and even diagnostics on the effects of changes. When you use something from the repository, it remains under the control of the repository. So, for example, a list of car makes can be updated once, and all surveys using that repository item will be refreshed automatically – unless you choose to sever the link and convert the instance in your survey into a local copy.

This is a highly accomplished interviewing platform, with all the customary built-in options to support everything from the simplest to the most complex survey. If there is a criticism in this area it is that some of the more advanced stunts you might wish to play require some tricky HTML coding within your survey texts – though some might just view this as additional flexibility.

Survey deployment is a doddle, with a highly intuitive survey administration tool that handles samples, invitations and reminders, and allows you to work with other survey modes such as offline mobile interviewing and an intelligent set of real-time response reports.

On the back end is what looks as if it is going to develop into a very promising researcher-driven online reporting and data portal module with the ability to create multi-user enterprise dashboards. However, to build such dashboards or portals at present requires expert help from DatStat. As a result, the tabulation and reporting elements in the package appear not to be as well advanced as the data collection tools.

The software’s underlying dependence on Microsoft technologies also means that, though largely web-based software, survey authors and administrators need to be running Windows. And though all support comes from Seattle, there are vast online support resources, including an efficient and responsive support issue logging and tracking, and excellent documentation.

Customer Perspective: Mindwave Research, Austin, Texas, USA

Mike Skrapits is vice president of research at Mindwave Research, a full-service research company based in Austin, Texas. Mindwave uses DatStat Illume for the large volumes of online research it carries out, which includes several large and complex tracking studies and numerous international projects. The company switched over to Illume seven months ago, having found it had reached the capacity of another web-browser based survey package.

Mike Skrapits explains: “When we did our evaluation, DatStat offered a little something different and this really had to do with the architecture of the software. Scalability had become an issue for us. The architecture of DatStat Illume means it is highly efficient – and the performance gain we have experienced from that is enormous. We had needed five quad core servers with the previous system, and we still had performance issues. We are now running everything on a single server and enjoy better performance too. It is also very stable: we have experienced no unplanned downtime over the seven months we have been running it.

“We are very pleased with the flexibility of the software when designing questionnaires. I like the look and feel of the software and many of our clients have commented on this and how much they like it. From the perspective of the end user, it is visually appealing and from the perspective of the programmer, it is very flexible.

The system’s modular, object-oriented architecture has also encouraged Mindwave to take advantage of the savings that can be achieved by re-using components and also to develop some novel research methods of its own, as these in turn become components that can used on any project whenever required. “The database architecture allows us to effectively create libraries that we can borrow from,” says Mike, “and that has absolutely been beneficial. We have also created some code of our own to create our own specialised question types. The flexibility is there in the architecture to let you do this.”

A version of this review first appeared in Research, the magazine of the Market Research Society, April 2008, Issue 502

Remark Office OMR and Remark Office Web 6 Reviewed

In Brief

What it does

A pair of related survey tools designed for DIY surveys on paper, with automated data entry using robust OMR scanning, or online surveys, which share a common data format and set of analytical tools at the back end.

Supplier

Gravic

Our ratings

Score 4 out of 5Ease of use

Score 2 out of 5Compatibility with other software

Score 5 out of 5Value for money

Cost

Remark office OMR $895. Remark Office Web $950. One off costs which includes telephone or email support and software updates within current version plus 50%+ discount on new versions.

Pros

  • Simple straightforward survey tool at a bargain price.
  • Does not require special printing for scannin
  • Can tie together data from paper and web
  • Everything runs on your own PC and web server

Cons

  • No character recognition capability in scan module
  • Lacks email invitations and reminders web module
  • Limited reporting options, especially for cross-tabs
  • Fiddly to create really nice-looking web surveys

In Depth

here are almost too many online survey tools to choose from these days – yet the choice narrows down considerably when you seek a single solution for fielding surveys on paper and online. If you need both and are on a tight budget, then Remark Office could give your productivity and independence a welcome boost.

Remark Office actually comprises two different products, which you purchase separately: Remark Office OMR, for automatic data capture of survey forms, and Remark Office Web, for online surveys. Though they work independently, the underlying data format is the same, and a survey set up on paper can be imported into the web module, so that the data can share the same layout, and they can be brought together easily for analysis, using the integrated cross-tab and statistical reporting module.

To use the OMR module you design your questionnaire in Word, with tickboxes and write-in fields to capture the answers. There are a few simple conventions to follow when laying these out, which are explained in tutorials provided by Gravic. Basically, keep things lined up, avoid gratuitous rules and boxes round boxes, and don’t put things too close together.

The software actually recognises three different types of data: tickbox fields for pre-coded questions, image fields, either for open-ended or numeric questions, or barcodes.

Barcodes are useful for customer surveys, so that you can tie the questionnaire into demographics or other data already known, and pull in data from an external file for analysis. This is simply achieved by having a unique identifier for each record, which could be a customer ID or your own made up number. You then use Word’s mailmerge to print this identifier on each separate survey form and send out the appropriate forms to the appropriate people. Gravic even provide you with a special barcode font to use in Word. Simply applying this font to your identifier converts it into a barcode readable by the scanner.

Tickboxes can also take on a variety of shapes beyond just squares. Where there is less flexibility is over write-in fields for text or numbers. Remark is strictly OMR only, and there is no support for handprint number or letter recognition, unlike many other data capture solutions. It makes the task of collecting a price or a postcode into a lengthy manual process. Gravic claim this is due to the ‘unreliability’ of intelligent character recognition as a technology. It is a specious argument: handprint recognition reached a level of viability fifteen years ago, and has only got better since. The case for leaving it out is probably more down to cost, and it may not be something everyone needs. For me, this is a serious omission.

Remark Office OMR is essentially a legacy forms tool, so to get the form ready for data capture you scan in a blank form and then use their software to define the page in terms of the questions, answers and expected values. When you define a new project there is a wizard that steps you through the task in a relatively painless way, and the software has the intelligence to be able to recognise what are likely to be answer fields when you draw a box around the relevant area of the scanned image using your mouse. All in all, you are likely to spend little more than 15–30 minutes setting up each A4 sheet.

The web survey tool has the advantage that it is fairly quick to use, through it does not have the instant ease and web feel that some of the rival online tools have and feels a bit dated. At least survey instruments are page-oriented, not just one long scrolling form. You can add several questions to one page very easily, and incorporate routing too. But the look of the online survey is uninspiring and appears hard to change.

It can be spiced up with some graphics, which are easy enough to add, but the product lacks the concept of CSS-driven independent style templates.

There is support for password-controlled access to surveys, and the ability for respondents to break off and resume a survey at a later time. Oddly, there is no invitation or reminder facility. However, Gravic are currently re-working the web module, so there is hope that some of these lacks will be overcome. It does have the big advantage that surveys can be deployed on your own web server, once again offering a very low-cost alternative to even the cheapest hosted solutions.

Reporting capabilities are reasonably good. As you might expect, this is not a power-reporting tool, but it goes further than many DIY products with the range of statistics that it offers. It will create statistical reports, summary reports and charts for all of your questions automatically, which will get you off to a flying start with your analysis. Cross tabs are different, and you have to create these one by one – and one question by another is the limit for each table. For in-depth analysis and segmentation, you can use the SPSS .SAV file export route to take it into another program.

It’s a program that does what it says it will, does it accurately, reliably and doesn’t cost the earth. Look on its simplicity as a virtue, and you could be in on a real bargain.

Country Report: United States. Remark Office in action at The City College of New York

Ed Silverman is the Director of Institutional Research at City College, part of the City University of New York, and is responsible for compiling data and carrying out research on the college and its courses. This he does single handedly for the most part, with the aid of Remark Office. Ed is passionate about what can be learned from surveys among students, employers and other ‘customers’ of the college.

Ec explains: “An example of one our marketing-type surveys was to find out ‘what do adult learners want to do?’ for our continuing education programme here. With Remark on paper, you can create a form which is simple to fill in, and which will capture what they say. We took it to events like street fairs, handed them out and we asked people to fill them in. We gave them a bunch of options and asked them what else they would like to see. Remark will capture what they wrote. That’s very useful when can you bring all the comments together and sort them in Word, because you start to notice things you had never thought of – and that means other people haven’t either. It’s great if we can offer courses that other people aren’t.”

“With employers, it is hard to get them to fill out a survey. But if we send them a URL and the survey is short, we get good results. But we also find that some of the managers, especially those in the older age groups, are uncomfortable with the web, so we send them a paper survey. By taking this mixed approach, we have had a number of responses from people who have never responded before.

“For me it is a time game. I have a huge amount of work, and the less time I can take, the happier I am. I can put together a survey in Remark in minutes – especially if you use templates. You can simply copy and paste questions from one survey into another, and that way, when someone asks you for something urgent, you can create something literally in minutes.”

Asked to sum up the benefits, Ed reels off a lengthy list: “It is inexpensive. For scanning, it does not require any special paper or printing and it’s really quick and easy to learn. It is reliable – there just aren’t any glitches. It is incredibly easy to upload to the web, and then to download the data when you are done. It makes savings all round.

Besides the surveys, I have a lot of other work to do. It saves an enormous amount of time. I could not survive without this.

Working with this software is really easy. It’s no problem to create a questionnaire and it does not take much time to learn – you can learn what you need to make the questionnaire after one or two hours.

A version of this review first appeared in Research, the magazine of the Market Research Society, September 2007, Issue 496