1/21/99
I. Introduction
In an attempt to slow the rate of attrition for the Survey of Program Dynamics (SPD), we are requesting permission to offer a prepaid incentive of $40 to all eligible sample cases in the upcoming 1999 data collection. This procedure would be part of a campaign to slow the rate of attrition for this important longitudinal survey. Justification for our request is given in the following section. Section III. offers more details about the operational plan for implementing the incentive procedure. Section IV. offers the most current results from both the Survey of Income and Program Participation (SIPP) and SPD incentive experiments as well as information about other surveys that have used incentives. We would need OMB approval by March 1, 1999 to be able to implement the procedure successfully.
II. Justification
The SPD is a unique survey in the federal statistical system. It is a longitudinal survey with a one-time window of opportunity for success.
The SPD is the vehicle for assessing true changes in behavior from the 1996 welfare reforms, because it is the only survey that provides a source of both baseline data and longitudinal data on individual and family outcomes. The data gathered for the 10-year period (1992-2002) will aid in assessing short- to medium-term consequences of outcomes of the welfare legislation. This is the only time that this data can be collected and will be the only source of data of this type available.
Congress specifically directs the Bureau of the Census to continue to collect data on the 1992 and 1993 panels of the Survey of Income and Program Participation (SIPP) in Title 42, United States Code, Section 614 (Public Law 104-193, Section 414, signed August 22, 1996). The use of the SIPP panels provides a good baseline for pre-welfare reform data. However, it does provide some major obstacles for collection of quality statistical data.
Nonresponse to the SPD is a major concern of project staff. The SIPP respondents provided 9 or 10 waves of detailed data over a three-year period. The SIPP data collection has a burden of 30 minutes per adult respondent per wave. So the average SIPP household (2.1 adults per household) has provided more than 10 hours of their time in burden. At the end of the last wave of SIPP interviews, respondents were thanked for their time and told that there would be no more interviews. Then one to two years later, the respondents were contacted and told they were still in a panel survey. Therefore, it was not surprising that SPD would have nonresponse problems.
The SPD inherited a 26.6 percent sample loss rate from the SIPP sample. After two waves of SPD, the sample loss rate is 50 percent (See Table 1). Previous studies on SIPP sample loss has shown that the sample loss is not uniform. Households in and near poverty attrit at a higher rate than other households. Since poverty households are a key target population in the study of welfare reform, there is some concern about nonresponse bias.
Table 1. Sample Loss- An Average of the 1992 and 1993 Panels and SPD
| Interview | Eligible HHs | Interviewed HHs | Average Sample Loss Rate (%) |
| 1 | 43,394 | 39,446 | 8.8 |
| 2 | 44,225 | 37,936 | 14.4 |
| 3 | 45,043 | 37,882 | 16.3 |
| 4 | 45,468 | 37,477 | 18.1 |
| 5 | 45,985 | 36,985 | 20.3 |
| 6 | 46,437 | 36,676 | 21.9 |
| 7 | 46,704 | 36,133 | 23.6 |
| 8 | 47,030 | 35,761 | 25.1 |
| 9 | 47,273 | 35,291 | 26.6 |
| *10 | 17,804 | 13,337 | 26.6 |
| **1997 SPD Bridge | 48,633 | 30,125 | 41.3 |
| ***1998 SPD | 32,800 | 16,400 | 50.0 |
** Only those Hhs interviewed in the last wave of the 1992 or 1993 panels were sent to the Field for the SPD Bridge.
*** Only those Hhs interviewed in the Bridge and selected during the subsampling were eligible for the 1998 SPD.
The use of incentives is standard among long-term panel studies similar
to the SPD. While there has been little experimental research on the effects
of these incentives, many the panel studies provide their respondents
some enumeration. A summary of several prominent panel studies' purpose,
sample, sample loss and incentives can be found in Attachment A. Table
2. shows a preliminary response rates for SPD 1998, Panel Survey of Income
Dynamics (PSID), and National Longitudinal Survey of Youth (NLSY). All
the response rates in Table 2. are calculated in the same manner so that
they are directly compariable.
Table 2. Response Rates for SPD, PSID, and NLSY: Period-Specific and Total
| Period-Specific | SPD1 | PSID2 | NLSY3 | ||
| SRC | SEO | Total | |||
| Sample-Selection to Interview 1 | 90.9 | 76 | 76 | 73 | 87.0 |
| Interview 1 to Most Recent Interview (Deceased Removed from Base) | 60.1 | 53.0 | 53.0 | 53.0 | 71.5 |
| Sample-Selection to Most Recent Interview (Deceased Removed from Base) | 50.0 | 40.3 | 36.3 | 38.7 | 62.2 |
2- PSID is based on a combined sample from the Survey of Economic Opportunity (SEO) (1966/1967) and fresh sample selected by the Survey Research Center (SRC) for the survey in 1968. Response rates are based on the 26th interview collected in 1993. More current information has been requested, but not received.
3-Response rates for NLSY are based on the 17th interview collected in 1996.
We have been trying to keep SPD response rates up using other enhancements as well. At the last interview, we provided respondents with a portfolio filled with Census Stat Briefs based on the results of the SIPP panels that the respondents participated. This showed the usefulness of the information they had previously provided and was used to encourage continued participation.
In February, 1999, respondents are being sent an "interim mailing" which has information culled from the 1997 SPD Bridge. This mailing serves two purposes, it gives respondents a reminder of the importance of their continued participation and it gives our Field staff a two month head start on tracking down people who have moved since the last interview. Reducing the number of people lost due to moving will also reduce the nonresponse.
We are also studying the feasibility of bringing low income people who left the SIPP sample back into SPD. This is expected to be a costly method of reducing the attrition rate. The Panel Study of Income Dynamics (PSID) Attrition Study brought back in approximately 35% of the noninterview cases they attempted to interview. Our two questions from the SPD attrition study are 1) can we locate low income people that we lost in the 1992/1993 SIPP panels; and 2) once we find them, can we convince them to participate in a lengthy questionnaire?
The Census Bureau is trying various methods to deal with the SPD attrition problem. We now believe that incentives are next method that we need to include to maintain a sample without major nonresponse bias. Incentives are the standard for long-term longitudinal surveys. (Again see Attachment A.)
We know that OMB is reluctant to set precedents regarding incentive use, but we feel SPD would not set precedents because it is a unique survey in the federal statistical system:
- It is a longitudinal survey studying a phenomena at a place in time;
- The respondents are from a previous study (SIPP) and believed that they had completed their obligation to the government; and
- The selection of the sample was Congressionally mandated.
A $40 incentive per household will be given to every SPD sample household eligible for interview in the upcoming interviewing cycle scheduled for 1999. The incentive will be prepaid by enclosing, in the advance letter prior to the interviewer's visit, a $40 debit card along with a PIN for redeeming the amount at an ATM. Each eligible sample household will be allowed to cash in the incentive regardless of the interview outcome (response or nonresponse). To ensure that every sample household gets the incentive, each interviewer will be given additional debit cards to offer to the households who had not received the debit cards through the mail prior to the interview.
IV. Results of Incentive Research.
A. Results of the SIPP Incentive Tests.
1. SIPP Waves 1 Incentive Experiment Results
In Wave 1 of the 1996 Panel, households were given either a $0, $10, or $20 dollar incentive to test whether it would reduce nonresponse rates at the initial interview and reduce item nonresponse rates for those who answered the questionnaire.
The results summarized below are extracted from Mack, S., Huggins, V., Keathley, D, and Sundukchi, M. (1998), "Do Monetary Incentives Improve Response Rates in the Survey of Income and Program Participation?", to be published in 1998 Proceedings of the Survey Research Section of the American Statistical Association. (For details see Attachment B.)
- The $20 incentives reduced (with statistical significance)
household, person, and item
(gross wages) nonresponse rates in the initial interview
(Wave 1).
- The $20 incentives reduced (with statistical significance)
household nonresponse rates in
subsequent interviews as well (Wave 2 through Wave
6).
- The $20 incentives were particularly effective for
reducing the household nonresponse rates
of poverty and black households.
- The $10 incentives did not substantially reduce nonresponse rates.
2. SIPP Wave 7 Incentive Experiment Results
In Wave 7, a $20 booster incentive was given to households who received the incentive in Wave 1 and were also low income in Wave 1. This incentive has had a positive effect on reducing attrition in Wave 7. (See Attachment C for details.) The results are given below with actual nonresponse rates in Table 3.
- The $20 incentives reduced (with statistical significance)
household nonresponse rates in
Wave 7.
- The $20 incentives reduced (with statistical significance)
household nonresponse rates in
Wave 7 for households below 150% of their poverty threshold
in Wave 1.
-The $20 incentive did not significantly reduce the
nonresponse rates for households above
150% of their poverty threshold in Wave 1.
Table 3. Weighted Type A Nonresponse Rates for Wave 7 by Incentive Groups
| Incentive Groups | Overall | Below 150% | Above 150% |
| $0 | 7.89% | 6.92% | 8.00% |
| $20 | 5.97% | 5.69% | 6.78% |
3. SIPP Wave 8 and Wave 9 Incentive Experiment Results.
In Wave 8, households that were Type A nonrespondents for the first time in Wave 7 were given either $0, $20, and $40 incentive during nonresponse conversion. A similar procedure is being done in Wave 9 for Wave 8 Type A's. The incentives have resulted in a significant increase in the Type A conversion rate in Wave 8. Results are given below with actual rates in Table 4. (See Attachment D for more detailed results.)
- The $40 incentive significantly increased the overall
conversion rate of Wave 7 Type A
noninterviews.
- The $40 incentives significantly increased the Wave
8 conversion rate for those households
that said they refused to participate in Wave 7 (i.e.,
hard refusals).
- Incentives did not significantly increase the conversion
rates for households where no one
was home after several attempts or were temporarily
absent in Wave 7 (i.e., soft refusals).
- Priority mail alone has increased the Wave 8 conversion
rate for refusals compared with
Wave 7 conversion rates (32.1% to 37.6%).
Table 4. Conversion Rates for Wave 7 Noninterviews in Wave 8 Incentive Experiment
| Wave 8 Treatments | All Type As | Refusals | No one home and
Temporarily absent |
| $0 Group | 47.0% | 37.6% | 60.8% |
| $20 Group | 49.9% | 38.7% | 65.5% |
| $40 Group | 56.0% | 47.5% | 68.2% |
| *Wave 7 Conversion Rates | 44.0% | 32.1% | 62.4% |
B. Results of the SPD Bridge Incentives Test.
A $20 voucher was given to a test group of low income cases and their neighbors in the first interview of the SPD. Based on the results of this incentive test, providing a $20 incentive to households has had a positive, but not significant, effect on response rates overall, as well as by demographic characteristics. However, within the experimental group, the households having received and cashed vouchers had significantly higher response rate than the households having received but not cashed or having not received vouchers. (See Attachment E for detailed results.)
C. Results of Other Survey Incentive Usage
The literature regarding incentives overwhelmingly supports the benefits of incentive use in general. Recently, Mosher, Pratt, and Duffer (1994) proposed the use of incentives for cycle 5 of the National Survey of Family Growth (NSFG). They suggested that previous research (e.g., Groves, Cialdini and Couper, 1992) implies that incentives are effective because they: create a reciprocation norm; create an informal contract between the interviewer and respondent, resulting in an exchange of goods for services; or are viewed as compensation for the respondent's time. Mosher et al. further argued that several federal social and health surveys used incentives because their surveys "...are long, sensitive, involve repeated interviews, and sometimes ask the respondent to leave their home or keep detailed records." (pp. 61). Accordingly they conducted a pretest using three conditions: a $20 incentive for an in-home interview; a $40 incentive for an outside-home interview; and a no incentive control condition. Results indicated that the $20 incentive significantly increased response rates, mostly because of fewer refusals. Incentives also increased data quality (e.g., incentive groups were more likely to report accurate levels of abortion than the non-incentive group) and decreased the amount of time spent locating and converting respondents (by over two hours), resulting in cost savings nearly equal to the incentive amount. The $40 incentives significantly improved response rates compared to the no-incentive group, but there was no significant difference between the $20 and $40 incentive groups. The authors concluded that $20 is an effective incentive for an in-home interview. Similar results were reported by Duffer, Lessler, Weeks, and Mosher (1994).
Other social scientists have also found incentives to be an effective tool for increasing both survey response and retention, and for reducing costs. Berlin, Mohadjer, Waksberg, Lolstad, Kirsch, Rock, and Yamamoto (1992) paid respondents for the National Adult Literacy Survey (NALS) incentives of $0, $20 or $35 at the completion of a face-to-face interview. Results showed that incentives significantly increased response rates but there was no significant difference between the $20 and $35 incentive groups. Results also showed that survey costs per interview were lower for the $20 incentive group than for the $0 and $35 incentive groups. Kerachsky and Mallar (1981) provided $5 per interview to a portion of respondents during face-to-face interviews in a longitudinal study of economically disadvantaged youths. The results showed that incentive payments were increasingly effective in each successive wave of the study. By the third wave of the study, the payment group was significantly more likely than the nonpayment group to: return update postcards (27% vs. 17%); be located (87% vs. 83%); and complete interviews (85% vs. 80%).
There is also considerable evidence that prepaid incentives achieve more substantial gains in response rates than do promised incentives, especially in mail surveys. Armstrong (1975) reviewed 18 empirical studies of monetary incentives used in mail surveys and concluded that only prepaid incentives show substantial reductions in nonresponse rates. Similarly, Church (1993) conducted a meta-analysis of 38 experimental and quasi-experimental studies in order to determine the effects of incentives on mail survey response rates. Significant increases in response rates were found only for surveys in which incentives were given in the initial mailing, with no evidence that incentives contingent on the return of the survey increased response rates. Compared to control conditions, the response rates increased an average of 19.1 percent for monetary incentives and an average of 7.9 percent for non-monetary incentives. These findings are consistent with those of Peck and Dresch (1981), and Berry and Kanouse (1987), which found better response rates for prepaid incentives than for promised or no incentives for mail surveys.
Berk, Mathiowetz, Ward, and White (1987) investigated the effects of both pre- vs. post-paid and monetary vs. nonmonetary incentives ($5) for face-to-face and telephone interviews for the longitudinal National Medical Expenditure Survey (NMES). Results showed that prepaid incentives, but not promised incentives, increased survey response rates and lowered the item nonresponse rates compared to no incentives, with only a moderate increase in cost. Berk et al. (1987) suggest that the benefit of prepaid incentives may result by decreasing the respondents' perceived burden, increasing their satisfaction for participating, and indicating the importance of the survey.
Recently, in a meta-analysis of 39 experiments, Singer, Gebler, Raghunathan, Van Hoewyk, and McGonagle (in press) found that payment of incentives significantly increases response rates for telephone and face-to-face interviews, especially for surveys with low initial response rates. Results also showed that prepaid incentives were more effective than promised incentives, although not at a statistically significant level. However, the authors also found that when comparing prepaid versus promised incentives within the same study, prepaid incentives are significantly more effective than promised payments. In addition, the results suggest that incentives help to increase response rates in interview situations with high respondent burden (e.g., a survey is over an hour in length, contains diary, tests, or sensitive questions, or is a panel study).
Finally, there has been some concern that incentives given early in longitudinal studies may create expectations of further incentives that may cause reductions in the rate or quality of response if these expectations are not met in later waves. Lengacher, Sullivan, Couper, and Groves (1995) investigated differences in cooperation rates in face-to-face interviews for the Health and Retirement Study (HRS) when a large incentive was given in the first wave followed by a smaller incentive in the second wave, compared to consistent incentives across waves. The authors concluded that "commitment to a longitudinal survey is not marginally harmed by large incentives in the first wave as a method to induce entry into a panel" (p.1034). They suggest that large incentives in the first wave do not necessarily create respondent expectations for large incentives in the second wave, but rather that respondents may feel "a surplus of reward" (p.1034), creating a positive feeling about the survey or a feeling of owed reciprocation to the survey organization. Similarly, Singer, Van Hoewyk, and Maher (1998) reported that respondents who received a monetary incentive in the past were more likely to participate in subsequent survey waves than those who had not received an incentive. The authors suggest that respondents may feel that their current participation was covered by the initial incentive.
V. Timing
We propose giving the incentive to all eligible SPD cases starting with the 1999 SPD which would be in the field at the end of April 1999. We would need OMB approval by March 1, 1999 to be able to implement the procedure successfully.
References
Attachments A - E
References:
Armstrong, J. S. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39, 111-116.
Berk, M. L., Mathiowetz, N. A., Ward, E. P., and White, A. A. (1987). The effect of prepaid and promised incentives: Results of a controlled experiment. Journal of Official Statistics, 3, 449-457.
Berlin, M., Mohadjer, J., Waksberg, J., Lolstad, A, Kirsch, I., Rock, D., and Yamamoto, K. (1992). An experiment in monetary incentives. Proceedings of the Survey Research Methods Section of the American Statistical Association, 393-398.
Berry, S. H., and Kanouse, D. E. (1987). Physician response to a mailed survey. Public Opinion Quarterly, 51, 102-114.
Church, A. H. (1993). "Estimating the effect of incentives on mail survey response rates" A meta-analysis. Public Opinion Quarterly, 57, 62-79.
Duffer, A., Lessler, J., Weeks, M., and Mosher, W. (1994). "Effects of incentive payments on response rates and field costs in a pretest of a national CAPI survey." Proceedings of the Survey Research Methods Section of the American Statistical Association, 2, 1386-1391.
Groves, R. M., Cialdini, R. B., and Couper, M. P. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56, 475-495.
Kerachsky, S. H., and Mallar, C. D. (1981). The effects of monetary payments of survey responses: Experimental evidence from a longitudinal study of economically disadvantaged youths. Proceedings of the Survey Research Methods Section of the American Statistical Association, 258-263.
Lengacher, J. E., Sullivan, C. M., Couper, M. P., and Groves, R. M. (1995). Once reluctant, always reluctant? Effects of differential incentives on later survey participation in a longitudinal study. Proceedings of the Survey Research Methods Section of the American Statistical Association, 1029-1034.
Mosher, W. D., Pratt, W. F., and Duffer, A. P. (1994). CAPI, event histories, and incentives in the NSFG cycle 5 pretest. Proceedings of the Survey Research Methods Section of the American Statistical Association, 1, 59-63.
Peck, J. K., and Dresch, S. P. (1981). Financial incentives, survey response, and sample representativeness: Does money matter? Review of Public Data Use, 9, 245-266.
Singer, E., Gebler, N., Raghunathan, T., Van Hoewyk, J., and McGonagle, K. (in press) "The effect of incentives on response rates in face-to-face and telephone surveys." Journal of Official Statistics.
Singer, E., Van Hoewyk, J., and Maher, M. P. (1998). Does the payment of incentives create expectation effects? Public Opinion Quarterly, 62, 152-164.
Attachment A
Incentive Use in Panel Surveys
NELS:88
Amount
Dropouts paid $25-$75 dependent on time commitment
Teachers paid dependent on number students
School coordinator paid $25,$50,$75 ($50 average)
Time Promised Before interview
Time Delivered By check, upon completion of interview
Interview Mode/Length Personal.
Differentials As above, dependent on time commitment
Incentive Effects
No.
Tested?
After first follow-up, school coordinator fee was varied, but no reports
released.
General Survey
The NELS is a school-based survey. The sample is based on a
Information
stratified, national probability sample of 1052 public and private
eight-grade schools (yielding over 25000 student respondents).
Students were administered questionnaires in class,
administrators completed surveys about the school, teachers
completed surveys about the students, themselves and the school,
and a sub-sample of parents completed surveys about
their children's home life and activities.
Response Rate(s)
Contact Person Steven Ingels 773-256-6275
High School and Beyond (HSB)
Amount
Dropouts paid $30-$50 dependent on time commitment
School coordinators paid $30; 1980, 1982 waves
(Note: Web page indicates incentives were given to all respondents,
yet administrator says no. Ask him again)
Time Promised Before interview
Time Delivered By check, upon completion of interview
Interview Mode/Length Personal.
Differentials As above, dependent on time commitment
Incentive Effects
No.
Tested
General Survey
The HSB is a school-based survey. The sample is a stratified,
Information
national sample of over 1100 secondary schools. The total number of
respondents totaled over 58,000 students at 1015 public and private
schools. The survey frame yielded student questionnaires, cognitive
tests, a school questionnaire (completed by school administrators), a
teacher comment list, and a subsample of parents completed a parent
questionnaire.
Response Rate(s)
Contact Person Steven Ingels 773-256-6275
National Survey of Families and Households Wave I & II (NSFH1 and NSFH2)
Amount $20 per respondent. Up to two respondents per household.
Time Promised
Time Delivered By check, upon completion of interview
Interview Mode/Length Personal. Wave I: 100 min. Wave II: avg 90 min.
Differentials No.
Incentive Effects
No.
Tested?
General Survey
National probability sample of adults (n=13,007 in Wave I).
Information
Minorities, single parent and step-families, cohabiters and
newlyweds were oversampled). One adult per household was
designated the respondent; spouses were also asked to complete
related questionnaires.
Response Rate(s) 82%
Contact Person Jim Sweet 608-262-2182
Panel Study of Income Dynamics (PSID)
Amount
$20 per respondent (one per household)
Formerly, $15 per respondent, plus $5 for completing change of
address postcard
Time Promised Before first interview (letter)
Time Delivered By check, upon completion of interview
Interview Mode/Length
Differentials
Finder's fee of $5 or $10 for helping to locate respondents. Towards
end of field season, interviewers have discretion to offer additional
money as a refusal conversion ($10)
Incentive Effects
In general, no.
Tested?
In one wave, $20 versus $50 was tested as a one-time only test. $50
was more effective, but created expectations for next wave
(respondents' hopes were dashed, however, as PSID went
back to $20) (am waiting to hear if there is a citation)
General Survey
A panel study begun in 1968. Beginning with 5000 households,
Information
the PSID is a nationally representative sample of US individuals. Over
time, grown children of initial respondents have been followed as
they
form their own households.
Response Rate(s) 53%-60% cumulative (depends on base; see Don Hernandez report)
Contact Person Tom Gonzalez 734-936-0307
National Longitudinal Surveys - Youth (NLS-Y)
Amount $20 per household head, plus $5 per child in household
Time Promised Before interview (letter)
Time Delivered
Upon completion of interview
Interview Mode/Length Personal (20% via phone). Avg 75 minutes
for main interview, plus
additional time for child assessment and young adult interview.
Differentials
Varied by household composition (number children). Interviewers had
discretion to offer additional refusal conversion fees. Interviewers
could also offer non-monetary conversion fees (e.g., pizza, movies for
the kids)
Incentive Effects
In general, no. Refusal conversion fee amounts were tested. $100
tested? fees statistically significantly increased responses.
A sliding
scale was just as effective, however. There is no report yet, as the test
was conducted September, 1998.
General Survey
Begun in 1979 as a nationally representative sample 12,686 men
Information
women aged 14-22 years. Minorities and poor whites were
oversampled; some of these subsamples were dropped in 1991.
Response Rate(s) 84% (see Dan Weinberg or Steve McClaskie for more info.)
Contact Person Randy Olsen 614-442-7348
National Longitudinal Study of Adolescent Health (AddHealth)
Amount
$20 per respondent (cash)
$1000 per school
Time Promised
Before interview (in consent form)
Time Delivered Upon completion of interview
Interview Mode/Length Personal. 90 minutes.
Differentials No.
Incentive Effects
No.
Tested?
General Survey
The AddHealth is a school-based survey. Eligible high schools
Information
had to have at least 30 students; students were selected from the school
rosters.
Response Rate(s)
Wave I: 80% Wave II: 90% of Wave I (which would be 72%
cumulative)
Contact Person Jo Jones 919-962-8412
National Evaluation of Welfare to Work Strategies (JOBS)
Amount
Waves I & II: $10 to $20 plus $5 gift for child, dependent on length
of
survey and whether child also participated.
Time Promised Before interview (letter)
Time Delivered By check, upon completion of interview
Interview Mode/Length Personal. Wave I: 30-90
min., dependent on additional modules.
Wave II: 35-70 min., dependent on additional modules.
Differentials Yes. Dependent on time commitment and survey wave.
Incentive Effects
No. However, researchers believe incentives increased responses.
Tested?
General Survey
The JOBS survey evaluates seven state welfare employment
Information
programs begun under the 1988 Family Support Act and continued under
TANF. The sites are: Atlanta, GA; Grand Rapids, MI; Riverside, CA;
Columbus, OH; Detroit, MI; Oklahoma City, OK; and Portland, OR.
Response Rate(s) Wave I: 83% Wave II: Approx 82% (still in field)
Contact Person Greg Hertz (sp?) 212-340-8670
Detroit Area Study, 1996 (DAS)
Amount
$5 to respondents
$25 refusal conversion fee
Time Promised Before interview
Time Delivered
$5 - in letter before interview
$25 - in cash upon completion of interview (B: verify)
Interview Mode/Length Personal. 60 minutes.
Differentials
$5 sent to only 2/3 of sample households
$25 randomly assigned to half the refusal households
Incentive Effects
Yes. $5 increases responses at p<.05.
Tested?
$25 - increases responses of refusers, but not statistically
significant (base is refusers)
General Survey
The DAS is an area-probability sample of the Detroit
Information
metropolitan area. The purpose of the survey was two-fold: 1)
the study of social/geographic stratification; and 2) the testing
of incentive effects and study of attitudes toward incentive use.
Response Rate(s) 68%
Contact Person Barbara Downs 301-457-2465
Health and Retirement Survey (HRS)
Amount $20 per respondent. Up to two respondents per household
Time Promised Before interview (letter)
Time Delivered By check, before interview
Interview Mode/Length
Personal. Avg 65 minutes. Wave I was a few minutes longer, as
baseline information was collected then.
Differentials
$100 refusal conversion at end of Waves I and II. Respondents
received FedEx letters promising the $100 in exchange for an
interview ($200 in couple households).
Incentive Effects
No. Conversion test indicates those who received money and
Tested?
letter were more likely to complete the interview than those who
only received a letter (correspondence with Dan Hill).
General Survey
The HRS is an area-probability sample of households. The target
Information
population is all adults in the contiguous US, born 1931-1941, who
reside in households. The HRS follows adults who move into
institutions after the initial interview. Blacks, Hispanics, and
Floridians were oversampled.
Response Rate(s)
Wave I: 80.2%-82.1% (depends on base; I have documentation
available)
Wave II: 92.06% Wave III: 87.65% (Waves II and III base is all
respondents from Wave I who are still alive at the respective
subsequent interviews.
Contact Person
Dan Hill
dhhill@umich.edu
DO MONETARY INCENTIVES IMPROVE RESPONSE RATES IN THE SURVEY OF INCOME AND PROGRAM PARTICIPATION?
Stephen Mack, Vicki Huggins, Donald Keathley, and Mahdi
Sundukchi, U.S. Bureau of the Census
Stephen Mack, U.S. Bureau of the Census, Demographic Statistical
Methods Division, Washington D.C. 20233
Key Words: Incentives, Nonresponse
Abstract
The Survey of Income and Program Participation (SIPP) used a monetary incentive in the initial interview of the 1996 panel to lower nonresponse rates. As in other longitudinal surveys, nonresponse rates increase in SIPP panels over time. We plan to interview sample households in the 1996 SIPP panel over a longer period than previous panels, 48 months versus 32 months. Consequently, we expect nonresponse levels to reach record levels, 30% or more by the end of the panel. We conducted an experiment to study the effect of $10 and $20 incentives on nonresponse and interviewing costs. James [1997] analyzed data from the first year of the panel. She found that the $20 incentive was effective in lowering nonresponse rates and that any incentive lowered the number of interviewer visits needed per case. This paper extends the analysis to cover interviews over two years, studies additional population subgroups, and looks at item completion rates.
I. Introduction
The SIPP is a longitudinal survey conducted by the U.S. Census Bureau which provides national estimates of sources, amounts, and determinants of income for households, families, and persons. The principle goal of the SIPP is to provide information to federal policy makers to assist in evaluation and reform of welfare programs, taxes, and entitlement programs. In order to achieve these goals, the SIPP provides both cross-sectional and longitudinal estimates (such as transition probabilities and spell durations).
Interviewing of SIPP panel members usually starts in February of the panel year (the 1984 and 1996 panels are exceptions). Subsequent interviews take place at four month intervals until the panel ends. One round of interviewing of the entire panel is called a wave. SIPP panels are divided into four rotation groups of approximately equal size. One rotation group is interviewed each month. This arrangement smooths out interviewing workloads and reduces bias in transition estimates.
In the initial interview, all persons living at sample addresses are listed as household members. Persons who are 15 years of age and older are interviewed and become original sample persons. Original sample persons are the units of observation for SIPP and are followed for the life of the panel. Exceptions include those who die, move abroad, or move into an institution or military barracks. Persons who move into households with original sample persons after wave 1 are also interviewed as long as they continue to reside with an original sample person.
Details of SIPP panels, such as sample size and panel length, vary among panels. More substantial changes are made after each Decennial Census when we update the sample frame and select new sample. The 1990 redesign of the SIPP took effect with the 1996 panel. We reduced cluster sizes, oversampled for poverty, introduced computer assisted interviewing, and made other changes.
In the first interview of the 1996 panel, wave 1, we obtained interviews from 92% of eligible households; about 36,700 interviews. Like other longitudinal surveys, SIPP noninterview rates increase as panels get older. The household noninterview rate of the 1996 panel stood at 26.4% as of the end of wave 6.
The SIPP conducted an incentive experiment in the initial interview of the 1996 panel to study the effect of incentives on nonresponse rates. SIPP primary sample units (psu's) were divided into three groups to receive no incentive, a $10 incentive, or a $20 incentive. Sample addresses in rotations 2,3, and 4 in the $10 and $20 groups were given vouchers (redeemable by mail) by interviewers immediately before the interview. James [1997] reported on the effectiveness of the incentive up through wave 3. She looked at nonresponse rates and interview cost data among households that were sent out for interviewing; we do not attempt further interviews with households that do not respond in wave 1 or have two consecutive noninterviews. James found that $20 incentives were effective in lowering nonresponse rates in waves 1-3 and that any incentive lowered the number of interviewer visits needed per case in wave 1.
In this paper, we will cover incentive results through wave 6. We compare household nonresponse between population subgroups defined by within-psu stratum (high poverty/low poverty), March poverty status, race, and education. Cumulative household nonresponse rates are used throughout the paper rather than wave nonresponse; i.e., households we no longer attempt to interview due to prior nonresponse are counted as nonrespondents.
Another issue we consider is whether incentives are effective at a person level. Some researchers have suggested that incentives can influence the quality and amount of information obtained from persons. To study this issue, we look at a few person-level rates: noninterview rates of persons within interviewed households (Type Z's), proxy interview rates, and nonresponse rates for gross wages.
II. Literature Review
There are many reports of positive results from using incentives. Ferber and Sudman [1974] reviewed a number of incentive studies. They found that the effect of incentives depends on respondent burden (i.e., the effort needed to cooperate), the amount of the incentive, and the economic level of the respondent. Berlin, et al. [1992] reported that a $20 incentive increased response rates for subgroups with low levels of literacy and lowered interviewer costs. Incentives may increase the willingness of respondents to provide information. A variable incentive was used in an education assessment study (Chromy and Horvitz [1978]). Young adults, age 26 to 35, were asked to take one or more assessment packages. Most respondents decided to take the maximum number of assessments to receive the highest incentive. The literature is mixed, but the following results were found in many studies:
* Large incentives increase response rates more than
small incentives.
* Incentives are effective for underrepresented populations,
such as low income and low
education.
* Incentives are effective in surveys with high respondent
burden such as panel or diary
studies.
* Incentives can reduce interviewer time and costs.
* Incentives may increase respondent cooperation; i.e.,
respondents may provide more
information when given incentives
(Chromy and Horvitz [1978]).
Gbur [1988] reported on an incentive experiment in the SIPP 1987 panel. A small gift was given to households scheduled for April 1987 interviews, about 25% of the total sample. The remainder of the panel was interviewed in February, March, and May. Interview rates were 1% higher for gift-recipient households than for nonrecipient households.
III. Design of the SIPP Incentives Experiment
SIPP sample psu's were sorted by size and divided into incentive groups
using systematic sampling. Incentives were distributed to sample addresses
in $10 and $20 incentive groups during rotations 2,3, and 4 of wave 1.
Incentives were not distributed in rotation 1. Table 1 gives counts of
eligible households by incentive group and incentive versus nonincentive
rotations.
| Table 1. Wave 1 households eligible for interviewing. | ||
|
group |
rotation 1
(no incentive) |
rotations 2-4
(incentive) |
|
|
3529
|
10328
|
|
|
3219
|
9686
|
|
|
3388
|
10038
|
Vouchers for $10 and $20 were distributed by SIPP interviewers at the door immediately after verifying the address. Interviewers gave vouchers to noninterviewed as well as interviewed households. Recipients were instructed to fill in their name, check the address, and return the voucher to the Census Bureau in the postage paid preaddressed envelope. After receiving the voucher, the Census Bureau mailed a check to the recipient within 2 to 3 weeks.
In this paper, we compare response rates and imputation rates. All estimates
are weighted. We use base weights; i.e., the inverse of the probability
of selection, or final weights as noted.
Differences are examined using two-tailed tests based on the normal distribution.
Significance is reported at the 10% level. Two types of comparisons are
made:
* differences of rates. The nonresponse rates of households
in rotations 2,3, and 4 are
compared between incentive groups. Significantly
lower nonresponse rates in the $20
incentive group are expected if a $20 incentive
is effective in lowering nonresponse.
* differences of differences. The differences of nonresponse
rates from rotation 1 to rotations
2,3, and 4 are compared between incentive
groups. If the $20 incentive is effective in
reducing nonresponse, then the change in
nonresponse rates should be greatest in the $20
incentive group.
IV. Nonresponse Rates
Within PSU Stratum
We oversampled for low-income households using a stratification approach proposed by Waksburg [1973]. Two within-psu strata were formed, one with a high concentration of poverty and one with a low concentration. In wave 1, we found a poverty rate of 27% in the high poverty stratum and 11% in the low poverty stratum.
Table 2 gives nonresponse rates in rotations 2-4 by poverty stratum. Nonresponse rates are significantly lower in every wave for the $20 incentive group when compared to the $0 and $10 incentive groups: for the high poverty stratum; for the low poverty stratum; and overall.
Differences in nonresponse rates in rotation 1 and rotations 2-4 are
shown in Table 3. Positive differences indicate lower nonresponse rates
in rotations 2-4 than in rotation 1. Significant overall decreases in
rates occur in waves 2 through 6 within the $20 incentive group. The $20
incentive was particularly effective in the high poverty stratum where
relatively large differences occurred in all waves.
| Table 2. Household nonresponse by poverty stratum. Rotations 2-4 only, weighted by base weights. | |||||||
|
|
group |
|
Poverty Stratum |
|
|||
|
1 |
|
.30%
|
9.14%
|
9.18%
|
|||
|
|
8.12%
|
9.51%
|
9.26%
|
||||
|
|
5.91%
|
*+ |
8.16%
|
*+ |
7.72%
|
*+ | |
|
2 |
|
16.06%
|
14.88%
|
15.13%
|
|||
|
|
13.77%
|
* |
14.44%
|
14.32%
|
|||
|
|
11.40%
|
*+ |
13.05%
|
*+ |
12.72%
|
*+ | |
|
3 |
|
19.18%
|
18.10%
|
18.33%
|
|||
|
|
17.65%
|
18.17%
|
18.08%
|
||||
|
|
14.39%
|
*+ |
16.12%
|
*+ |
15.77%
|
*+ | |
|
4 |
|
22.36%
|
21.22%
|
21.46%
|
|||
|
|
20.74%
|
21.27%
|
21.18%
|
||||
|
|
16.91%
|
*+ |
19.33%
|
*+ |
18.85%
|
*+ | |
|
5 |
|
25.53%
|
24.48%
|
24.70%
|
|||
|
|
24.26%
|
24.24%
|
24.24%
|
||||
|
|
21.06%
|
*+ |
22.78%
|
*+ |
22.44%
|
*+ | |
|
6 |
|
28.98%
|
27.27%
|
27.64%
|
|||
|
|
27.10%
|
26.70%
|
26.77%
|
||||
|
|
23.00%
|
*+ |
25.22%
|
*+ |
24.78%
|
*+ | |
| * significantly different from $0 incentive
group + significantly different from $10 incentive group |
|||||||
| Table 3. Household nonresponse by poverty stratum. Difference of rotation 1 and rotations 2,3, and 4 weighted by base weights. | |||||||
|
|
group |
|
|
|
|||
|
|
|
-0.05%
|
-0.33%
|
-0.27%
|
|||
|
|
|
0.34%
|
-0.33%
|
-0.21%
|
|||
|
|
2.78%
|
-0.31%
|
0.31%
|
||||
|
|
|
0.67%
|
0.51%
|
0.55%
|
|||
|
|
|
2.64%
|
1.09%
|
1.37%
|
|||
|
|
5.18%
|
* |
1.81%
|
2.49%
|
* | ||
|
|
|
-0.58%
|
0.45%
|
0.23%
|
|||
|
|
|
1.07%
|
-0.12%
|
0.09%
|
|||
|
|
6.11%
|
*+ |
2.30%
|
*+ |
3.07%
|
*+ | |
|
|
|
-1.35%
|
-0.76%
|
-0.88%
|
|||
|
|
|
0.27%
|
-0.53%
|
-0.39%
|
|||
|
|
5.42%
|
*+ |
1.45%
|
* |
2.25%
|
*+ | |
|
|
|
0.49%
|
-0.38%
|
-0.17%
|
|||
|
|
|
0.79%
|
0.35%
|
0.43%
|
|||
|
|
3.94%
|
2.38%
|
* |
2.69%
|
*+ | ||
|
|
|
-0.73%
|
-1.47%
|
-1.30%
|
|||
|
|
|
1.71%
|
0.49%
|
0.70%
|
|||
|
|
4.64%
|
* |
1.78%
|
* |
2.34%
|
* | |
| * significantly different from $0 incentive
group + significantly different from $10 incentive group shaded differences are significantly different from 0 |
|||||||
The change in nonresponse rates from rotation 1 to rotations 2-4 is often larger in the $20 incentive group than in other incentive groups. Overall nonresponse rate differences are largest within the $20 incentive group for waves 3,4, and 5.
The $10 incentive does not appear to significantly influence nonresponse rates overall or within poverty strata. The only significant result, i.e., positive result, for the $10 incentive group occurs in wave 2.
Wave 2+ Rates by Poverty Status
Analysis of wave 1+ nonresponse rates is limited to the few variables whose values are known for wave 1 nonrespondents. Geographic and sampling variables are known. Interviewers are asked to provide their best guess of the householder=s race and sex as well as household size and tenure. For other characteristics, we can study the effect of incentives on wave 2+ nonresponse rates; i.e., nonresponse of wave 1 respondents.
Incentives are thought by many researchers to be most effective in low-income
areas. Wave 2+ noninterview rates are shown in Tables 4 and 5 by the March
poverty status of the original household. Nonresponse is lower in rotations
2-4 for both poverty and nonpoverty households in the $20 incentive group,
except for wave 4 poverty. The $20 incentive appears, at first glance,
to be more effective for poverty households than for nonpoverty households;
however, the differences are not statistically significant except for
wave 2.
| Table 4. Wave 2+ nonresponse rates for households in poverty as of March, weighted by base weights. | |||||||
|
wave |
group |
1 |
2-4 |
r[1]-r[2-4] |
|||
|
2 |
|
7.10%
|
7.87%
|
-0.77%
|
|||
|
|
10.06%
|
5.97%
|
* |
4.09%
|
* | ||
|
|
13.53%
|
* |
7.73%
|
+ |
5.80%
|
* | |
|
3 |
|
10.28%
|
10.81%
|
-0.53%
|
|||
|
|
13.59%
|
11.12%
|
2.47%
|
||||
|
|
16.70%
|
* |
10.69%
|
6.01%
|
* | ||
|
4 |
|
13.24%
|
14.83%
|
-1.59%
|
|||
|
|
15.66%
|
13.56%
|
2.10%
|
||||
|
|
16.46%
|
14.70%
|
1.76%
|
||||
|
5 |
|
15.55%
|
19.03%
|
-3.48%
|
|||
|
|
20.80%
|
* |
17.11%
|
3.69%
|
* | ||
|
|
21.91%
|
* |
17.49%
|
4.42%
|
* | ||
|
6 |
|
16.39%
|
23.50%
|
-7.11%
|
|||
|
|
24.88%
|
* |
19.64%
|
* |
5.24%
|
* | |
|
|
25.27%
|
* |
20.63%
|
* |
4.64%
|
* | |
| * significantly different from $0 incentive
group + significantly different from $10 incentive group shaded differences are significantly different from 0 |
|||||||
| Table 5. Wave 2+ nonresponse rates for households not in poverty as of March, weighted by base weights. | |||||||
|
wave |
group |
1 |
2-4 | ||||