The latest news from the meaning blog

 

Ruby Reviewed

In Brief

What it does

Modern, GUI-driven cross-tabulation, analysis and charting suite for market research data aimed at the tabulation specialist. Capable of handling large and complex data sets, trackers and other ‘difficult’ kinds of research project.

Supplier

Red Centre Software, Australia

Our ratings

Score 3 out of 5Ease of use

Score 5 out of 5Compatibility with other software

Score 4.5 out of 5Value for money

Cost

Full version $4,800 (allows set-up); additional analyst versions $2,400. Annual costs; volume discounts available.

Pros

  • Cross-tabs and charts of every kind from large or complex datasets, and so much more
  • Quick and efficient to use for DP specialist, using a choice of GUI access and scripting
  • Push-pull integration with Excel and PowerPoint for report preparation and automation
  • Superb proprietary charting to visualize MR data more effectively than in Excel or PowerPoint
  • Excellent support for managing trackers

Cons

  • Interface is bewildering to beginners: a steep learning curve
  • No simple web-browser interface for end users or to provide clients with portal access to studies

In Depth

We always try to present something new in these software reviews, but this time, we think we are onto something that could break the mold: a new tabulation software package from an Australian producer, Red Centre Software, that leaves most of the existing choices looking decidedly dated. It’s refreshing, because for a while, most efforts in market research software seem to have gone into improving data collection and making it work across an ever-broadening spectrum of research channels. Innovation at the back-end seems to have focused on presentation, and has often left research companies and data processing operations with a mish-mash of technology and a few lash-ups along the way to transform survey data into the range of deliverables that research clients expect today.

Ruby could easily be mistaken for yet another end-user tabulation tool like Confirmit’s Pulsar Web or SPSS’s Desktop Reporter, with its GUI interface and drag-and-drop menus. The reality is that it is a fully-fledged tabulation and reporting system aimed squarely at the data processing professional. If you are looking for a Quantum replacement, this program deserves a test-drive.

As far as I could see, there were no limits on the data you could use. It will import data from most MR data formats, including Quantum, Triple S and SPSS. Internally, it works with flat ASCII files, but it is blisteringly fast, even when handling massive files. It will handle hierarchical data of any complexity, and offers the tools to analyse multi-level data throughout, which is something modern analysis tools often ignore.

It is equally at home dealing with textual data. The producers provided me with a series of charts and tables they had produced from analyzing Emily Brontë’s Wuthering Heights by treating the text as a data file. The same could be done for blogs, RSS feeds and the mass of other Web 2.0 content that many researchers feel is still beyond their grasp.

More conventionally, Ruby contains a broad range of tools specifically for handing trackers, so that you are not left having to automate the reconciliation of differences between waves due to variations in the question set and answer lists.

Ruby is a very intelligent tool to use when it comes to processing the data. The data in the tables reported or charted in MR have often gone through a long chain of transformations, and in the old tools, there could be yards of ‘spaghetti code’ supporting these transformations. Trying to work out why a particular row on a table is showing zeroes when it shouldn’t do can take an age in the old tools, as you trace back through this tangle of code, but Ruby will help you track back through the chain of definitions in seconds, and even let you see the values as you go. It is the kind of diagnostic tool that DP professionals deserve but rarely get.

In Ruby, you will probably make most of these data combinations and transformations visually, though it does also allow you to write your own syntax, or export the syntax, fiddle with it, and import it again (the combination that DP experts often find gives them the best of both worlds). However, Ruby keeps track of the provenance of every variable, and at any point, you can click on a variable and see exactly where the data came from, and even see the values at each stage.

The range of options for tabulation and data processing is immense, with a broad range of expressions that can be used to manipulate your data or columns and rows in tables. There is complete flexibility over percentaging and indexing values off other values, or basing one table on another, so it is great for producing all of those really difficult tables where every line seems to have a different definition

With charting, Ruby gives you the choice of using its own proprietary charting engine, or pushing the data out to PowerPoint or Excel charts. The native Ruby charts are a treat to work with, as the developers seem to have gone out of their way to redress the inadequacies of Excel and PowerPoint charts. For time-series charts, concepts such as smoothing and rolling periods are built-in. You can add trend lines and arbitrary annotations very easily. Charts can be astonishingly complex and can contain thousands of data points or periods, if you have the data. Yet it will always present the data clearly and without labels or points clashing, as so often happens in Excel.

Excel and PowerPoint charts are also dynamic, and the Ruby data source will be embedded in the chart, so that the charts can be refreshed and updated, if the underlying data changes.

Amy Lee is DP Manager at Inside Story, a market research and business insights consultancy based in Sydney, Australia, where she has been using Ruby for two years, alongside five other researchers and analysts. Ruby is used to analyze custom quantitative projects and a number of large-scale trackers.

Asked if the program really did allow a DP analyst to do everything they needed to, Amy responds: “We were able to move to Ruby a couple of years ago, and it is now the main program we use, because it can do everything we need to do. I find it is an extremely powerful and flexible tool. Whenever I need to do anything, I always feel I can do it with Ruby. Other tools can be quite restrictive, but Ruby is very powerful and completely flexible.”

Amy considered the program went beyond what more traditional DP cross-tab tools allowed her. She observes: “Compared with other programs I have used, Ruby allows me to filter and drill down into the data much more than I could with them. It’s especially good at exporting live charts and tables into documents.

“Once they are in PowerPoint or Word, trend charts can be opened up and adjusted as necessary.  When it is a live chart, it means you can update the data, and instead of having to go back to Ruby, open it up and try, find the chart and then read the data, you can just double click it inside PowerPoint, and you can see all the figures change.  And there is even an undo feature, which is good for any unintentional errors.”

Amy freely admits that this is not a program you can feel your way into using, without having some training, and allowing some time to get to understand it.  “It is really designed for a technical DP person,” she explains. “If you have someone with several years’ experience of another program they will have no problem picking this up as everything will be very familiar to them. But we also had a client who wanted to use it, someone with a research rather than a DP background, and they found it a bit overwhelming, because it can do so much, and it is not that simple. It looks complex, but once you get the hang of it, you can do what you need very quickly.”

Among the other distinguishing features Amy points to are the speed of  the software, which is very fast to process large amounts of data and produce large numbers of tables and charts; its in-built handling of time-series, allowing you to combine or suppress periods very easily,  and the range of charts offered, in particular the perceptual maps.

Some of the research companies I speak with are becoming uneasy that the legacy data processing tools they depend on have fallen so far behind, and are in some cases, dead products. They have endured because the GUI-based ‘replacements’ at the back of the more modern data collection tools just don’t cover the breadth of functionality that is needed. You get breadth and depth with Ruby – even in the sheer range of functionality it offers is bewildering to the newcomer.

A version of this review first appeared in Quirk’s Marketing Research Review, August 2009.

Cognicient Link

In Brief

Cognicient Link version 1.0

Cognicient, UK
Date of review: May 2009

What it does

Software utility that allows you to create a consolidated database of all numerous survey datasets for meta analysis and aggregation of survey questions and their responses across different surveys by creating flexible taxonomies that work independently of the original data structures to resolve variations at the survey level. Data may be queried directly within Link or extracted to statistical packages for analysis or modelling.

Our ratings

Score 3 out of 5Ease of use

Score 4 out of 5Openness

Score 3.5 out of 5Value for money

Cost

Annual fee £20,000 for the core system and a single data upload user and £5,000 for each additional user.

Pros

  • All your survey data in one place – analyse anything by anything
  • Works with just about any format of survey data, and will import directly from Triple-S and SPSS and SPSS legacy formats (Quancept, Surveycraft etc)
  • Flexible in how you treat and resolve similarities and differences
  • Robust and highly scalable

Cons

  • Steep learning curve – requires some technical expertise to use
  • Some complex manual intervention required during set-up stages
  • Exports limited to SPSS or raw data files

In Depth

It is a pipe dream for many big research buyers that one day they will have a single database that contains absolutely all of their surveys, so they can play with all their data as if it was just one big survey. Talk to the people at Cognicient, and that dream can become reality. Cognicient, a small UK/US consulting company that specialises in survey data management, decided to make available the tools it had developed for its own internal use that allow it to create vast data warehouses of survey data, offer them as a package for others to install and use for themselves, or as a managed service through Cognicient.

It’s a common problem that, as research agencies or research buyers accumulate surveys on different products, different markets and at different times, it becomes increasingly difficult to pull these together in any meaningful way in order to make comparisons. The individual dataset becomes a straightjacket. Minor differences in the formats of the questions, the answer lists or rating scales used, or even discrepancies in the underlying format of the data make it impossible to put the data together for analysis, unless you are able to grab a few figures from previously reported data that just happen to meet all your criteria for comparison.

Cognicient Link is a database application based on a standard Microsoft SQL Server database, that lets you create a data warehouse of all your past surveys. In doing so, it breaks down the artificial barriers that normally exist between surveys. The importer lets you load data from each survey without needing to reformat it, and alongside that, to load in metadata on the survey too, such as how the sample was constructed, the fieldwork dates and method used or any other relevant information. This metadata can also be used, alongside the actual survey data, for queries and comparisons, adding another dimension to the data.

At the other end of the process is a range of tools that let you query the data and extract sets of variables selected from the database, whether they were original survey questions or from metadata added the import stage, to provide a working set of variables for analysis or modelling in stats programs like SPSS or SAS. Link does not attempt to offer any analytical tools other than providing simple counts when querying the data, and currently only outputs data as SPSS or raw data.

At the heart of system is what Cognicient call the “taxonomy”. This is the truly clever bit – it’s a sophisticated master list of all of the fields you have in the database which categorises them for comparison. At the point where you import new data into the database, you supplement the taxonomy to point it to the specific variables in the incoming survey, and provide enough information for Link to be able to bring in the data and, if necessary, transform the data into a standardised format. It therefore maintains an indirect link between the source data and the consolidated data in the database, so you can add more data later if you need to. You can create your taxonomies to be very specific – to define a particular rating used on one product in a tracker, or make it generic, such as a ‘value for money rating’ which might be found in any survey. And the taxonomy could even accommodate variations in how that question was asked, essentially adding another dimension to that question which can be used in analysis or filtering. The concept is simple, but taking decisions about the best taxonomy to use is a complex process, and one that the people at Cognicient prefer to be involved with directly, when introducing Link to new customers.

Taxonomies are created as Excel worksheets, to a model specified by Cognicient. You also create a Survey Information Sheet, in which you specify the survey-specfic metadata and information on file names. It will cope with multi-level or hierarchical data, and you specify this here. With the definition complete, you move into loading the data, and here some rather ugly edits are required to the SQL Server database tables – something that Cognicient are planning to automate in a future version. You are then ready to open the Link Manager tool, from where all the other operations are performed, including importing the data, resolving differences, associating the questions in the current survey with the master list in the taxonomy.

The interface is functional, rather than elegant – but this is not software you spend time looking at – it is the means to an end. The tools it contains to query the data and make up extracts are basic but easy to use and exports are performed surprisingly quickly, usually in seconds or minutes. It is a pity that you cannot see more within the database itself though, and my plea would be for Cognicient to provide some tools that allow some real-time analysis in the future. Link is a fairly expensive product, and the process of adopting it requires a considerable commitment, but what it will let you do, through pitching all of your surveys into one big melting pot, is very exciting indeed.

Customer viewpoint: Brett Matheson, Vice President of Synovate MarketQuest

Brett Matheson is Vice President of Synovate MarketQuest, the firm’s global product design and development practice, where he has been co-ordinating an initiative to build a global database of concept, product, and packaging ratings for international consumer product clients. This database allows their analysts to perform a wide range of meta-analyses across the entire range of consumer studies, which could be from understanding the performance of a format of product packaging across different international markets, or identifying seasonality effects on concept and product test results, to observing the relative effectiveness of different kinds of questions in obtaining consistent results.

Brett explains: “Cognicient Link fundamentally does two things: it provides the platform for combining data from a wide variety of sources and a wide variety of formats into a single database, and it provides the tools to efficiently get the data out in a meaningful way. The Link software allows us to do things that really set our database apart. First, the database is at the respondent level, not aggregate. That allows us to use much more powerful analytics. Second, instead of requiring strict adherence to common data collection protocols, Link allows us to embrace the variability of different approaches used by different clients. This makes construction of the database much more difficult, but it also gives us the flexibility that our clients demand.

“We already have hundreds of studies in the database and this is just the beginning – we add to it every day. We have many thousands of product, concept, and package tests, and we need a lot of data because when you start to drill down into individual categories or regions, you need to have data there to support your analysis.”

Analysts in Synovate MarketQuest use the query tools provided within Link to interrogate the database and extract relevant datasets to the question or hypothesis they are working with. Usually, the data are then exported in SPSS format for analysis in SPSS, or for some advanced modelling in SAS, which can also read the SPSS formatted data delivered from Link.

Brett continues: “We are very excited about what we’ve been able to do with it — it has already been an enormous benefit to us and our clients. There is a certainly a lot of client interest in it. We now have all this data in one place and we are discovering new uses for it all the time.

“Some of the research on research we have done with this is focused on how we can help our clients make more efficient use of the funds they have for research. This includes things like examining the impact of sample size and composition on research conclusions.

“There is a very steep learning curve, because what you are trying to do is complex. But in the end you have an extremely effective process that allows you to do a lot of things that you might have always wished but you couldn’t very easily because they were just too expensive and took too long to do.”

A version of this review first appeared in Research, the magazine of the Market Research Society, May 2009, Issue 516.