The latest news from the meaning blog


Double standards, triple costs

According to a recent white paper produced by Oracle’s BI Consulting Group (“The Great Debate: buy versus build”) it costs between two-and-a-half and three-and-a-half times as much to build your own business intelligence and analytics system than buy and adapt an off-the-shelf solution. It must be admitted, the white paper concerned is a marketing piece to support this firm’s services in – you guessed it – adapting standard Oracle systems to provide BI solutions. But their figures are based on experiences with some 250 large-scale implementations, which is a decent number to conjure with.

They also found that such projects when built from scratch by system developers averaged 34 weeks from start to finish and consumed 925 person days, whereas implementing then adapting a generic platform took half the time – 17 weeks – and less than a third of the effort, with an average of 290 days required. When you consider the amount of effort involved in managing a project twice as long, and the drain that this has internally on the stakeholders involved with prototyping and early adoption, there are also likely to be sizable hidden internal costs that can be avoided. MR applications may differ in some respects, and the development projects around them may be smaller in scale, but the experience is unlikely to be radically different from this scenario.

There is a surprisingly large amount of custom-built MR software around, developed in-house from scratch. It’s not all legacy stuff either. I continue to hear of firms developing their own tools across the range of research applications, from panel management through to dashboards and portals. As the article from Oracle points out, their experience is that the out-of-the-box solution tends to meet between 70% and 80% of needs, so development effort is concentrated instead on the 20% to 30% that needs to be accommodated.

Of course, it is always possible that, lurking in that 20-30% are some very nasty problems that require almost infinite resource to resolve, but that remains true whatever route you take. Perhaps decision-makers are not comparing like-with-like when opting for own-grown. An off-the-shelf solution has known limitations – these limits are usually cited as reason for rejection. Despite this, they are limitations that may often be overcome through customisation. Open systems now make it readily feasible to customise rather than bribe your software vendor into changing their core software product or live with the limitations. Yet it is a choice that is often neglected.

The build-your-own route is seductive – theoretically, it appears to have no limits in what is achievable. The danger lurks in those obscure user requirements where effort starts to spiral towards infinity in achieving them, and where hard decisions are needed – always assuming you’ve noticed in time.

If you can find something where 70% of the work was already done without having to think about it, surely that’s a good thing? Yet, “it only met 70% of our requirements” is often the siren call that lures project teams into the unfathomed depths of what astute system developers call “the vanity project”. It’s flattering to be told your requirements are so special that a system must be built from scratch to meet them. It’s also very very unlikely. But to counter vanity with vanity, who likes finding out they paid more than three times over the odds?

CASRO Tech 09 and a field of poppies

It was great to be at CASRO’s annual Technology conference again. They have changed the format since I was last there in 2005 and instead of a roster of invited speakers, they now follow the call-for-presentations model, which has upped the standard considerably as the organising committee has been able to pick from the best. The CASRO Tech presentations have always tended to be pretty grounded, focusing on methodology as well as technology, and with speakers unafraid of saying what does not work along with what does. You are often hearing from people who are speaking from a well of practical experience, which I find a tonic from the customary highly choreographed statements of the obvious that seem to dominate the showpiece research conferences. (more…)

Accessible access panels

I have just had a visit from Stephen Hughes, who was until recently the UK representative of CfMC, but is the UK rep for Cint. I am intrigued as Cint is a kind of panel solution provider, but with a twist. It’s a Swedish firm and is strong in the Nordic countries, having been around for about ten years now, and it provides a web-based solution for panel management. That is unremarkable in itself, as there are others out there doing just that. What caught my ear is that they are working on providing panel exchange – essentially a marketplace for those with panels and those looking for panel samples to do business – and also on automation.

It also appeals to me because I’m getting a presentation ready for the Research conference on Online Research at the beginning of June, and it ticks a few boxes on the kinds of things I am going to be talking about – their approach looks as if it might hold the keys to resolving a couple of common problems with panels at present.

The thinking goes like this. We know now that there is a pool of respondents who are willing to participate in surveys, and this is a diminishing subset of the population as a whole. Some of these people join panels or communities and they have done this because they enjoy doing surveys. An increasing number of firms are creating their own specialist panels and communities. Often this is because they have a specialist need, or are researching something where the incidence in the general population is fairly low, and the general consumer panels don’t offer a good fit. But these panel owners don’t necessarily always have the work to keep the panel ‘ticking over’ with enough survey invitations to make the panel member feel wanted. Therefore, if such panels could be openend up – safely that is – to other firms doing legitimate and relevant research, then there should be benefit all round: the panel member gets more invitations, the panel owner keeps the panel moving, and keeps each panellist’s status and other stats up to date, and the client gets their survey done. Furthermore, the panel owner can defray some of their operational costs by charging an access fee.

Well this is what Cint has the potential to do. The catch is that you must be using their own panel software solution – so they are creating a marketplace, but only among their own customers. At present, Cint customers have a strong bias towards the Nordic countries, though Stephen assures me that there are some ‘big announcements’ about to come regarding panels in other parts of Europe – at present, the UK is particularly limited. No doubt this is Stephen’s immediate task in his new role. But I speculated that the sharing concept could be so much greater if Cint offered an interface, ideally as a web service, so that they could also provide access to panels in other systems too. It is an area where some cross-industry standards are required.

Unfortunately ours is an industry virtually untouched by technical standards. Such standards that we do have almost always relate to quality evaluation. We have the triple-s standard for the exchange of survey data and metadata between one application and another which is much used, and not very widely adopted tabs-ml standard for tables, which has already fractured into two incompatible dialects of it.

A standard for exchanging panel metadata would be very handy indeed. It needs to be a bit different though, as it must incorporate metadata at at least two different levels. It needs to allow users to discover the panel’s overall characteristics at the high level, and in some detail, and then it needs to have an agreed protocol for revealing and transmitting the actual panel member records. But some of this data is confidential, such as the crucial email address, and needs to be concealed from the end user. This is where a panel exchange, such as that proposed by Cint, could be so very useful, as it provides a trusted intermediary that will allow the would-be purchaser to dip in and see what is there, but can then handle the extraction of the data and invitation management without letting the panel buyer harvest the emails as well. It would mean that the panels do not all have to reside in the one database., and will therefore increase choice, and critical mass and viablilty of the solution.

Perhaps Cint can be encouraged to work with other providers, like Centurion, Globalpark, Confirmit or Nebu, as some of the panel software providers, to agree standard interfaces so that panel can be exchanged in this way. However, many mass panel companies, like SSI, Research Now, Toluna and GMI, have built their own proprietary software for panels, and they are less likely to be interested in an exchange, since they are aiming to provide a one-stop-shop service. However, the reality is that the panel buyer goes to where the panel is, and tends to shop around among these big providers too.

Cint is already taking a very responsible attitude towards safety, permission and control, with respect to additional groups of researchers making use of panels built and owned by someone else – which would be even more necessary in a more open context. If nothing else, it would be good if there could be a more standardised and automated way of accessing panels from any of the interviewing solutions in use out there, like Confirmit, MR Interview or Globalpark. It is an area where many could benefit – the problem is, it requires quite a diverse crowd to agree to something and this could mean it is something that gets talked about for a very long time.