The second day of the IWIS09 Internet Workshop in South Korea focused on practical measures and finding in improving response in online surveys (in addtion to those already reported here and here).
Jan Zajac (University of Warsaw) overviewed factors which can drive participation rates in online surveys, both to boost them and, in some cases, diminish them too. His own experiments, carried out in Poland, optimising email survey invitations to boost response found that including a picture of ‘the researcher’ made a surprisingly large improvement to response. Less surprisingly, pretty, young and female researchers seem best in pulling in the respondents – though not only from males but females too.
Pat Converse (Florida Institute of Technology) revisited Dillman’s Tailored Design Method to see the differences in response rates to in mixed-mode paper and web surveys, and the extent combining both best improves response. It seems paper is far from dead. His analysis across a wide range of published survey results results seem to show that a 34% response rate is about middling for Internet only surveys whereas mail surveys still typically achieve a 45% response. In his experiment, he looked at how effective using a second mode to follow up non-response at the first mode can be – and clearly it will improve response. Surprisingly, the greatest improvement was in following up a web survey invitation that had got nowhere, with an approach by mail: almost 50% of those approached responded, taking overall response to 72%. The best response came from mail first with web as the fall-back, though this is likely to be the most costly, per interview. Web first, with a switch to mail could hit the sweet spot in terms of cost, when a high response really matters – such as for a low incidence sample.
Presenters from National Statistics services in New Zealand, Singapore, Estonia and Colombia all provided insights into how web-based research had been helping them, and how they had been ensuring both high quality and acceptably high response in order to reach the entire population. This too was typically achieved by using the web as one channel in a multimodal approach. Web was generally favoured because of its cost and convenience, and empirically, speakers had observed little significant variation between the responses between modes. Even where internet pentration is still low, as it is in Colombia, with only around 12% of the population enjoying an Internet connection, online is used to supplement fieldwork carried out using 10,000 PDAs that use Geo-location.
As an event, these two days have effectively provided a cross-section of the state of current knowledge and inquiry into Internet research. There was talk of making the papers and presentations available, and if so, I’ll provide a link here.
Statistical Center, Daejeon, home of Statistic Korea
More insights into market and social research in Korea emerged in day two of the Internet Survey International Workshop, hosted by Statistics Korea.
South Korea is one of the most technically advanced nations in the world, with a young and growing population. Virtually 100% of those aged under 40 are Internet users and across the board, South Korea ranks eighth globally for Internet penetration: higher than both the USA and the UK. Using Internet panels is therefore very appealing for national statisticians and social researchers – if only ways could be found to overcome coverage and non-response bias.
Sunghill Lee (UCLA) proposed an advance on Harris Interactive style of propensity weighting, to nudge panels towards national representativeness by supplementing propensity weights with a stage of calibration against a reference dataset which nationally representative, or from a true random probabilty sample. Her model was capable of halving the observed discrepency, but at a cost, as the sample variability tended to increase.
Prof. Cho Sung Kyum (Chungnam National University, Korea) had noticed others’ attempts to weight their panels in the direction national representivity tended to use demographic data, including some measures that were hard to calibrate, such as occupation. There is often frustration in being able to get hold of robust reference data. Prof. Cho had noticed that many national statistics offices around the world conduct a Time Use study among the general population. These meet most criteria for good reference data – large, robust, random probability samples that are representative of the population. They also cover Internet-specific information, as one use of time which is tracked in these studies, in some detail.
In his test online surveys, he asked respondents some time characteristics that could be cross-matched, such as the typical time home from work, typical bedtime and time spent online. Matching by six measures, his model provided near perfect adjustments for questions relating to leisure, education or media consumption; but it offered no improvement for income or work-related questions. However, his work is ongoing, and he hopes to identify other variables that could narrow the gap in future.
Online on a slow burn
In MR, online research has only a ten per cent share in Korea, an astonishingly low figure given the very high Internet penetration in Korea, stated Choi In Su, CEO of Embrain, an Asian panel provider. Face-to-face still tends to dominate, as telephone is not particularly useful either with less than 70% of Koreans having a fixed phone line. However, he predicted quite rapid change, expecting the share to reach 20% or more.
The reluctance among MR firms also stems from the same concerns that the statisticians had been airing – coverage and non-response error, and low quality in particiation. Mr Choi outlined a number of unusual characteristics of the Embrain panels designed to combat these limitations – which include a combination of online and offline recruitment, rigorous verification of new recruits against registers or other trusted sources, a range of fraud detection measures, and good conduct training for panel members. A key measure of success is the consistent 60% response rate from survey invitations.
It felt as if the social statisticians were ahead of the game. Kim Ui Young from the Social Surveys division of Statistics Korea spoke of two successful online introductions of large-scale regular surveys. A key driver had been to reduce measurement error and respondent burden, and one diary study of household economic activity provided a good example of this. In fact, Kostat had gone as far as to work with online banking portals to allow respondents to access their bank statements securely, and then import specific transactions directly into the online survey, which a lot of respondents found much easier to do.
In my concluding blog entry, tomorrow, I will cover the highlights from international case studies and new research on research, which were also presented today.
I’m at the First International Workshop on Internet Survey at Daejeon, Korea. It is hosted by Statistics Korea (or Kostat) which has put together an impressive roster of presentations on leading edge thinking in using online research for public policy research and other nationally representative surveys: eighteen speakers, fourteen from around the world, and a nice fat 320-page book of scholarly papers to accompany the event.
My own talk was on software and technology (what else?) and how appropriate technology can help control measurement and non-response error: but unlike many of these events, I did not find myself the pariah for speaking technology. There has been explicit acknowledgment throughout the first day of this two-day event for the need for researchers to be more discriminating and more demanding of the technology being used, in order to improve response, reduce respondent burden and control error more effectively — as well as reducing cost.
The event started with Yi Insill, the Commissioner of Statistics Korea, who predicted “a significant increase in demand for Internet Surveys” in National Statistics work in Korea. “We are expecting them to reduce non-participation and make them engaging for participants,” she stated. She also acknowledged that national statisticians had been reluctant to use online surveys because they were not based on random probability samples and “have been criticised for poor quality”, but that was now changing as the methodology was being understood and tested. Preparations were well advanced for the 2010 e-Survey in Korea, and we heard more of this later on.
One good paper followed another – but I will pull out a few highlights. Frederik Funke (Tübingen University) showed how Visual Analog Scales (VAS), when applied to online surveys, can dramatically reduce measurement error, while showing that conventional 5-point scales, applied to online surveys by convention (and possibly for no better reason) can enforce measurement error on participants by restricting their options – to the extent that different results will arise from a VAS which appear to be more accurate.
Surveys that leak cash
Lars Kaczmirek (GESIS, Mannheim) followed through with three practical changes to survey design that would improve response and reduce error. He showed the results of some experiments that showed how, compared to the effect of providing an incentive on a survey or not, some simple changes to survey design were actually more effective. In other words, you could chop the incentive, improve the design, and still be slightly better off in terms of response.
Kaczmirek was also critical of the way in which new technology was sometimes applied to surveys uncritically, even though it would increase non-response. Another example was the automatic progress bar – inaccurate or misleading progress bars, particularly those that jump due to routing, are such a turn-off to respondents that actually removing them altogether will often improve response. Accurate bars, or bars where jumps are smoothed and averaged out, do better than no bar, though.
Boxes for Goldilocks
Marek Fuchs (University of Kassel) gave us the latest thinking on verbatim response box size and design in online surveys: getting the size right can mean more characters and potentially, more concepts – like Goldilocks and the porridge, they should not be too small or too large. Adding in a Twitter-style count of how many characters remain can also boost response length, provided the starting number is realistic (a couple of hundred, not a thousand characters). However, too much trickery, such as dynamically appearing or extending boxes will also send any gains into reverse. As with the wonky progress bars, the point is that any feedback must be realistic and honest for it to act as a positive motivator.
Questionnaires with added AJAX
Peter Clark (Australian Census Bureau) talked us through the 10 per cent uptake of an online census option in Australia for the 2006 Census, and the plans being made to increase this now to 25% for the 2011 Census. ACB had appointed IBM as its technology partner for 2006 and again for 2011. IBM had pioneered adding browser-based processing in AJAX (a Web 2.0 technology) to the 2011 e-Census form, to cut down server load. It has saved them a fortune in hardware requirements, as the server load is now a third of what it was. For the many participants on slower dial-up connections, the form took longer to load, but once loaded, was actually faster, as all further traffic to the server was minimal and therefore very fast to the user.
Australia, along with other speakers describing their e-census strategies in Singapore and Estonia, had added an online option to the national census as a means of reducing cost. For the obvious coverage reasons, e-census is offered as an option to back up self-completion by mail, and as a last resort, face-to-face for non-responders.
Pritt Potter (Webmedia, Estonia) spoke of the work he had done in providing novel respondent validation methods to the forthcoming e-census in Estonia, which included using trusted third parties such as banks to offer verification through their normal online banking security, and then pass on to the census bureau key identification data. Another method offered to respondents is mobile phone verification (provided that the phone is registered). Both methods have the advantage that the public can respond to ads in the media, visit a website and self-verify, instead of the census bureau having to send out numerous unique passcodes.
And there is more in store tomorrow…
Internet surveys may now be the favoured method for a majority of commercial market research surveys, yet uptake has been slower in areas of social policy research and particularly official statistics. When internet penetration is high – and globally, South Korea is among the highest – Internet surveys start to become useful for those compiling official statistics too. It’s a subject the Korea National Statistical Office thinks is now worth exploring and it has convened its first International Workshop on Internet Surveys to bring together experts and practitioners from around the world.
Among the panel of international speakers KNSO has invited to the event, Tim Macer will be presenting a paper on IT applications and their role in supporting online research. The workshop will be taking place in the Metropolitan City of Daejeon in central South Korea in September 2009. The city is South Korea’s science and technology hub, and the home of the national statistics service.