The effects of incentives on data quality in online panels
Final Report Abstract
In view of the wide distribution of web-based data collection the project develops evidence-based guidelines for the short-term as well as long-term employment of incentives in online panels to collect high-quality data in a cost-conscious manner. One cross-sectional experiment and one longitudinal experiment were conducted. In the cross-sectional experiment 5,857 panellists were offered one out of four rewards: no reward, money via bank transfer, money via PayPal, or a donation to a charity. Bank transfer, PayPal and donation were offered in one of two amounts: 1 EUR or 3 EUR. Half of the panellists were offered a result summary. Two dependent measures reflected response quantity, namely response and retention, and 23 dependent measures reflected response quality. Offering panellists a payment into their bank account promoted data quality best, followed by a payment via PayPal. The donation to charity was counterproductive. In the longitudinal experiment 3,201 newly recruited members of a non-commercial online panel were invited to seven study waves. Half of the panellists were offered a result summary in each wave. One out of six rewards was offered: no reward, money via bank transfer, money via PayPal, redeemable loyalty points, a donation to a particular charity, or a donation to a selectable charity. Three dependent measures reflected response quantity, that is, attrition, response and retention, and three dependent measures reflected response quality, that is, number of written words in open-ended answers, item nonresponse and midpoint response style. Monetary rewards (i.e., bank transfer, PayPal, and loyalty points) failed to promote response quantity except for a few sporadic effects, but they improved response quality. Nonmonetary incentives (i.e., donations and summary) reduced response quantity but improved response quality. This project shows that by offering incentives researchers cannot maximize response quantity and response quality at the same time. Instead, incentives promote one facet of data quality at the expense of the other. Incentives hardly improve and often reduce response quantity, whereas they augment response quality. These trade-offs render it impossible to derive all-purpose recommendations on the employment of incentives in online panels. Researchers need to weigh the pros and cons of using particular incentives according to the aims and budget of their studies.
