Environmental Economics

Mode Effects and Other Potential Biases in Panel-based Internet Surveys: Final Report

During the spring and summer of 2008 the Wyoming Survey & Analysis Center (WYSAC) at the University of Wyoming directed national surveys by telephone (1273 respondents) and mail (904), and also a web survey of KN panelists (1162). Each sample was administered a questionnaire about air quality in national parks. By design, the questionnaire was nearly identical for all three modes, as was the sampling frame.
The response rate was much lower for the web survey than by phone or by mail. Response was best in the mail survey, which also showed the greatest yield from additional survey efforts aimed at encouraging response. Exploratory analyses gave indications of differential non-response bias by mode, apparently due to mode-related variation in the mechanisms of self-selection as a survey participant. Phone and mail surveys may involve more self-selection of respondents interested in the topic of a particular survey, whereas a web panel may self-select for those with sedentary lifestyles.
Weighting and matching the respondents did not eliminate significant demographic, behavioral, and attitudinal differences across modes. For example, web respondents were less likely than either phone or mail respondents to have children in the home, to be members of any environmental organizations, to participate in various kinds of outdoor recreation, and to express high satisfaction with the National Park Service. Their econometric models therefore incorporated statistical controls for variables likely to be correlated with both mode of survey administration and willingness-to-pay (WTP).
Results showed that using either a panel-based Internet survey or a mail survey produces a more conservative dollar value for WTP than using a phone survey. Communication with a live interviewer over the phone seems to yield over-statement of true WTP. Though face-to-face interviewing was not part of their research design, the apparent upward bias on WTP due to the effects of social desirability in a phone survey would also be expected in a face-to-face survey.
They found, further, that the variance in WTP left unexplained by their model was higher for the web panel than for either of the other two survey modes. There was a slight negative effect on WTP from panel conditioning (as measured by duration of panel membership or number of web surveys completed), whereas survey fatigue had no consistent effect in the other two modes. Statistical interactions between mode of survey administration and other explanatory variables were of little substantive importance. In all three modes, the factors affecting WTP were similar and the signs of their effects were consistent with plausible theoretical expectations. Hence, with appropriate controls, a WTP estimate derived from a KN web survey should be no less accurate than that obtained from a well-designed and well-executed mail or phone survey.
The cost of data collection proved to be highest by mail and lowest by phone. The web survey was in the middle on cost, but closer to the high end than the low. Any of these three modes would be much cheaper than face-to-face interviewing if the goal is to obtain a large representative sample.
Strictly speaking, a KN web panel survey does not solve the methodological problems associated with changes in telephone use, because KN‟s recruitment process is itself reliant on a fairly traditional telephone survey approach. For example, the low cumulative response rate in their web survey, with the attendant risk of non-response bias, results mainly from KN‟s low response rate during telephone recruitment of panelists. Hence, to keep pace with cultural and technological changes in telephony, KN will need to adopt the same kinds of tactics for recruitment that all telephone surveyors are now using to improve the representativeness of their samples. These tactics include dual-frame sampling (cell and landline), multi-mode initial contacts (mail, phone, and email), non-contingent incentives, and multi-mode follow-up with non-respondents. Such tactics can be expensive, and their use could well close the fairly narrow gap between the cost of a KN web panel survey and that of a thoroughly designed and implemented mail survey.
Meanwhile, advances in address-based sampling are improving the potential for representativeness of mail surveys. The techniques for maximizing response and minimizing non-response bias in a mail survey are also being refined through systematic research (e.g., Dillman 2007). Like a web survey, a mail survey is well-suited to the use of photos or other visual aids that may be especially helpful when asking about environmental quality. And unlike phone-recruited web panels, mail surveys are not much affected by cultural and technological changes in telephone use.
In sum, their findings demonstrate that mail surveys and probability-based web surveys both merit consideration as alternatives to phone or face-to-face interviewing in studies of willingness to pay for environmental quality.