zurück zur Homepage

zurück zur Publikationsliste

Material incentives, i.e. small gifts, have proven to be an effective means to increase the response rate of mail surveys (cf. Church 1993; Fox et al. 1988; Hare et al. 1998; Heberlein and Baumgartner 1978; James and Bolstein 1990, 1992; Kanuk and Berelson 1975; Linsky 1975; Nederhof 1983; Warriner et al. 1996; Yu and Cooper 1983). Although much research has been conducted on the use of material incentives in cross-sectional mail surveys, little is known about their effectiveness in mail panel surveys. It is the aim of this research note to fill this gap. By means of an experimental design in a two-wave mail panel survey we examine the effects of material incentives (phone cards) on return rate, panel attrition and sample composition.

Why do incentives work, and which type of motivation do they create? In the Total Design Method (TDM), their use is embedded in a comprehensive theoretical framework. Taking part in a survey is seen as a process which only takes place if a) the respondent’s reward for participating is higher than his or her (perceived) costs and b) if the prospective respondent trusts the researcher (Dillman 1978: 12; see also Brehm 1994: 48f).

In a survey, the basic reward for the respondent is intrinsic, i. e. of the type predominantly found in processes of social rather than economic exchange (Frey 1997). The reward is the satisfaction of being "consulted" in an important matter (Dillman 1978: 13). Incentives are usually included in the first letter sent to the respondents (ex ante-incentives) but can also be provided upon return of the questionnaire (ex post-incentives). Church (1993: 73) has demonstrated that incentives of the ex post-type, which represent an extrinsic reward (payment or compensation), have no significant effect on the response rate, while incentives given ex ante usually work very well.

Even experiences in everyday life give an idea why this holds true: If someone presents you with a gift or does you a favor, you usually feel obliged to return the favor. The ex ante-incentive is a means to establish a relation of social exchange between the researcher and the respondent, a "symbol of trust" (Dillman 1978: 16) which creates an obligation to comply with the request for an interview. By sending back the questionnaire, respondents can obey the norm of reciprocity (Gouldner 1960), which may in fact be the greater intrinsic reward. This is why the incentive should definitely be given in advance rather than after the interview as a kind of payment.

If given ex ante, incentives work remarkably well in cross-sectional studies at which TDM is directed. But using them in a panel study is a different story, because (1) virtually nothing is known about effects on panel attrition, (2) because, if given only in the second wave to reduce panel attrition (Martinez-Ebers 1997: 79), some respondents may see it as an ex-post payment for their participation in the first wave and will shift from social to economic exchange (cf. Brehm 1994: 48f), which may damage intrinsic motivation to participate further; and (3) because, if given in both waves (at additional cost), the interview may increasingly be seen as economic exchange by some, while for other respondents the relationship of social exchange may deepen.

Because there is little empirical evidence on the pressing question whether and when incentives should be used in mail panel surveys, we decided to test their impact in a two-wave short-time panel with a true experimental design. By means of this design the following questions can be answered:

1. Does a material ex-ante incentive promote a higher return rate in both waves?
2. Does an incentive granted in the first wave influence the return rate in the second wave, i.e. is there a long-term effect of hysteresis after the respondent "returned the favor"?
3. Is there an interaction effect between the two incentives, i.e. will the effect of the second incentive be intensified (positive reinforcement) or weakened (shift to economic sphere) if the target persons received an identical incentive in the first wave?
4. Do incentives have an influence on the composition of the sample? That is: Does the effect of the incentive vary between certain groups in the population? Such variation would obviously lead to a different composition and could either increase or decrease sampling bias and bias due to panel attrition.

The vehicle we used for our experiment was a mail survey of citizens of the state of Hamburg. That survey was part of a research project on voting behavior. Data collection started immediately after the state election of September 21, 1997 and was administered adhering to the TDM. Hamburg is a city-state which has only one registration office. This office was exceedingly cooperative and agreed to draw a single-stage random sample (n=4,000) from the registration files for us. This was done six weeks before the election, so the sampling frame matched the population almost perfectly and information on addresses was complete and up-to-date.

In addition to the address we were also informed about the target person’s year of birth, gender and the district of residence, which allowed us to add context variables to the data set for later multilevel analyses. The questionnaire was in the legal format A 5 (approximately the size of this journal) and contained 12 pages. At the core of the questionnaire the respondents were asked to rank a set of 10 hypothetical party manifestos according to their preferences (see Arzheimer an Klein 1998 for details). The pretests showed that this task required quite a cognitive effort. Our expectations regarding the return rate were not very optimistic, but we employed every measure recommend by Dillman and conducted the survey literally "by the book". Five months later, the second wave of the panel was administered adhering to TDM, too, but we did not send the last of Dillman’s three reminders because it had displayed only a very modest effect on the return rate of the first wave.

In both panel waves we used a phonecard with a value of 6 DM (approximately $ 3.60) as an incentive. Phonecards are lightweight, small in size and – in contrast to lighters, ballpoints or similar "presents" – they have a genuine monetary value. On the other hand, they are not as close to the sphere of economic exchange as the equivalent amount of cash. Therefore, they can cross the threshold between nonmonetary and monetary incentives, which have proven to yield larger overall effects on response rates (Church 1993: 71). As an additional benefit, phonecards underscore the researchers willingness to answer the questions respondents might have regarding the study.

To estimate the effect of the incentive we employed a 2´ 2 design: In the first wave 75 percent of the target persons received an incentive along with the first questionnaire mailed, 25 percent did not. In the second wave, 50 percent of the persons in each of these groups received an incentive while the other 50 percent did not. The respondents were randomly assigned to the treatment groups immediately after the sample was drawn, so we have a true experimental control for the internal validity of our findings.

Of the total of 4,000 addresses 96.9 percent (3,874) were valid. 58.5 percent of these 3,874 persons participated in the first panel wave; 1,606 people either explicitly refused to participate or did not react to any of our reminders. About 95 percent of the 2,268 respondents in the first wave were also willing to take part in the second. Meanwhile 57 persons had moved to an unknown address or did not live in Hamburg any longer, and 51 persons had rendered illegible the ID number on the envelope of the first wave. 87.1 percent of the remaining 2,160 respondents answered a second time.

Main Effect

In the first wave, the return rate of the experimental group was 61.7 percent and that of the control group was 49.4 percent. The probability for a target person of the experimental group to take part in the study was therefore almost 25 percent higher than that of the control group. The difference of 12.3 percentage points is statistically and substantially significant (Cramer’s V= 0.11, p=0.000).

In the second wave, the incentive had an effect as well. The return rates were 84.0 without and 90.3 percent with incentive. This effect is statistically significant, too, but somewhat weaker (Cramer’s V=0.09, p=0.000). This lower magnitude can be explained by some sort of self-selection: The sample for the second wave consists of respondents who participated in the first wave and thereby showed some interest in the topic of the survey. It seems plausible that the level of willingness to participate in a second survey in this group is higher than in the general public, so the relative gain from using an incentive must be smaller than in the first wave.

Hysteresis Effect

Is there a positive effect of the first incentive on the reinterview rate? For those who received a phonecard in the first wave the reinterview rate is 87.1 percent, whereas it is 87.3 percent for the control group. This small difference is not statistically significant, so there is no evidence of any lasting effect of the first incentive after the respondent has fulfilled his/her obligation to return the questionnaire.

Interaction Effect

Table 1 shows the reinterview rates for all groups in the 2´ 2 design. If there had been an interaction effect because either a shift from social to economic exchange or an effect of positive reinforcement occurred, there should be a marked difference between the ratios of the reinterview rates in the rows of table 1, but the actual difference (1.11:1 vs. 1.07:1) is very small.

Table 1 Reinterview rate in the second wave dependent on giving incentives (percent).

The conjecture that there is in fact no interaction effect can be formally tested with a log-linear model for the 2,160 target persons of the second wave. Participation in the second wave is the dependent variable while the two incentives form the independent variables. If the difference between the two ratios is within the range of random variation, a model containing only the main effects of the two incentives should fit the empirical distributions well. This is the case (Likelihood-Ratio c ² = 1.17, df=1, p=0.279), so there is indeed no reason to believe that an interaction effect between the incentives exists. Both incentives work completely independent of one another.

Composition of the Sample

Do incentives modify sample composition? If existing sampling bias could be reduced by the use of incentives, such an effect would be very welcome. However, if incentives should induce additional sampling bias, researchers would risk to trade quality for quantity.

We were able to examine possible effects on the composition of the sample by means of the information on the target persons obtained from the Hamburg registration office: the person’s gender, age and the district of residence. The latter gives us the opportunity to employ additional context variables, e. g. the average income in this area. We suspected that the incentive might have a stronger effect in poorer districts.

The incentive influences the sample composition if its effect is modified by one of the aformentioned variables. We tested this with a second log-linear model. In order to examine as many cases as possible we restricted the analysis to the first wave. A model with main effects only (participation in the first wave as the dependent and age, average income in the district and incentive as independent variables) and without interactions between any of the main effects fits the empirical data sufficiently well (Likelihood-Ratio c ² = 122.03, df=108, p=0.088), so there is no indication of group-specific effects of the incentive that could increase or decrease sampling bias. Especially, the effect of the incentive is independent of the average income in the district of residence (see also Warriner et al. 1996: 557f.).

The analyses reported so far referred to demographic and economic variables solely. However, among the factors that determine participation in a survey, interest in its topic is very important (Martinez-Ebers 1997: 80). A certain amount of bias regarding this variable seems almost inevitable. Can the undersampling of the less interested be reduced by means of an incentive? We cannot investigate these question directly for obvious reasons, but as "interest in politics" was included in the first wave’s questionnaire, we can measure the association between political interest and the willingness to take part in the second wave. There is indeed a modest positive correlation (g =0.13, p<0.05) between political interest and participating in the second wave, and incentives seem to have a greater effect on people with low political interest than on persons who are very interested in politics (see Table 2).

Table 2 Return rate in second wave by political interest and incentive given (percent)

The statistical significance of this difference can again be tested with a log-linear model. A model which only includes main effects fits the data very well (Likelihood-Ratio c ² = 0.73, df=2, p=0.695). Hence the incentive does not significantly reduce the bias which arises from the correlation between political interest and survey participation.

Total Response

Finally, let us have a look at the total response rate of the survey over both panel waves. As expected, the group of persons who received incentives in both waves has the highest return rate (55.3 percent), while the group of persons who had not received any incentives in either of the two waves featured the lowest return rate of only 40.9 percent. If the incentive was granted in only one wave, the return rate lies in-between these two values: If the incentive was employed in the first wave only, total return was 51.9 percent as compared to 45.3 percent if the incentive was granted only in the second wave. Although there is no hysteresis effect, it is more efficient to use the incentive in the first wave, in cases where only one incentive can be given.

According to our results, the Total Design Method with an appropriate ex ante-incentive is a successful alternative to the more common face-to-face or telephone interviews for conducting panel studies. Even in the group of persons who did not receive any incentives at all, the return rate was almost 50 percent in the first and more than 80 percent in the second wave. Taking into account that this study was done in a big city, we feel that this result is quite convincing.

Our results underscore that phonecards are very convenient and efficient in mail surveys and cross the threshold between nonmonetary and monetary incentives. In the first wave, the phonecard increases the return rate by 12 percentage points, i. e. 25 percent. This effect is comparable to the impact of a 5-dollar-bill. Furthermore, our experiment indicates that an appropriate incentive implemented in both panel waves increases not only the return rate of the first wave (which has been demonstrated before) but also the reinterview rate in the second wave.

These two effects are completely independent of one another, i. e. there is no interaction effect. Moreover, our data allow us to demonstrate that both effects are independent of variables such as age, gender or socio-economic context of the target person. Sampling bias regarding these variables is neither increased nor decreased by the use of a material incentive. A modest decrease in bias regarding political interest as measured in the first wave is not statistically significant.

Our data therefore do not support the conjecture that the extensive use of incentives may destroy the intrinsic motivation of respondents. However, the employing of incentives may actually make respondents expect an incentive whenever they are asked to participate in a survey as shown by Singer et al. (1998), and little is known about the long-term effects of such expectations.

Because of financial restrictions, most researchers will be able to use only one incentive. Our results show that it is more efficient to use the incentive in the first wave of the panel.

However, it is not clear if our findings apply to long-term multi-wave panels as well. Further research on mail panels in general and the appropriate use of incentives in particular is definitely needed.

Kai Arzheimer is an assistant at the department of Political Science at the University of Mainz. His research focus is on the voters and supporters of left-wing, right-wing, and green parties.

Markus Klein is a research assistent at the Central Archive for Empirical Social Research at the University of Cologne. His research interests are voting behavior and value change.

Address correspondence to Kai Arzheimer, Institute for Political Science, Johannes Gutenberg University Mainz, Colonel-Kleinmann-Weg 2, 55099 Mainz, Germany,

zurück zur Homepage

zurück zur Publikationsliste