Just before the end of last year, I decided to join one of the online panels to get a better understanding of their workings. I have to say that the whole experience has left me open-mouthed in shock. In the short period since I have been a member, I have encountered error after error as well as, frankly, poor questionnaire design and basic scripting errors.
Typical errors I have experienced are:
- Emails in various foreign languages inviting me to participate in surveys
- Numerous broken links in emails
- Requests to enter passwords that I don’t have
- Another time, I clicked the link and got a message saying ‘this survey has not started yet’. I went back later and clicked the link and got a message saying that I had completed.
For those few surveys where I have not been screened out, I have encountered numerous impossible lines of questioning. There never seems to be a way of skipping a question, so your only options are to just click anything (to make sure you get your payment for completion) or to give up. Some questionnaires allow you to make a comment at the end, so you can at least explain that you were forced to give an opinion on the personalities of different washing up liquid brands, even though you have no opinion on the matter because you regard washing up liquid as a commodity and just buy the cheapest, irrespective of its name, color, perfume etc.
In another survey, which was about radio stations, I was presented with a list of radio stations and asked which one I had most recently listened to. I picked a talk and news station. From then on, I was asked endless, detailed, non-applicable questions such as whether the presenters interrupted the music or how fun the competitions were.
And perhaps, even worse, I have recently seen a question which asked me which of a list of brands I had bought. The one I had bought was not on the list and there was no ‘other’ option! Again, I was forced to select an incorrect answer or give up.
I did think that perhaps I was just biting into the one bruised apple of the basket, but coincidentally, I spent New Year’s Eve with a friend who is managing a new panel. She has worked with numerous panel suppliers and was not in the least bit surprised by my tale of woe.
Having had this discussion with my friend, I was wondering whether market researchers are accepting unrealistic deadlines. Clients know that with the technology we now have, it is possible to script a project and complete the fieldwork online in a matter of hours. But perhaps they don’t know about the amount of thought, effort and creativity that is required for decent questionnaire design and in particular thorough quality control. There’s no point in conducting a survey if the questions are impossible to answer honestly and accurately. Presumably, some important business decisions are based on some rather dubious survey results.