4. DATA COLLECTION PROCEDURES

The next three chapters are about SIPP data collection procedures. This chapter describes the current data collection procedures, highlighting those features most likely to influence the quality of the data, e.g., interview mode, respondent rules, the structure of the interview and the questionnaire, interviewer characteristics and training, and quality assurance procedures. Chapters 5 and 6 present available information on how these features of the data collection process affect quality. The results of experiments in which alternative procedures were tested are also covered in Chapters 5 and 6. Chapter 5 covers nonresponse error and Chapter 6 covers measurement or response error.

References that provide useful information about current data collection strategies and procedures, as well as changes since the start of interviewing in 1983, include Kalton et al. (1986) and SIPP Working Papers 8401 (Nelson et al., 1985) and 8830 (Kasprzyk, 1988). Persons interested in more detail may also wish to consult the SIPP-5010 Interviewers’ Manual (U. S. Bureau of Census, 1988b), which is periodically updated, and may be obtained by writing or calling Customer Services, Data User Services Division, U. S. Bureau of the Census, Washington, DC 20233, (301) 457-4100.

4.1 Basic data collection features

Interviewing every 4 months

Interviews of sample persons and households take place at 4-month intervals. Information on core items, such as labor force participation and income recipiency and amounts, is collected for each of the 4 months included in the reference period. Interviews are conducted in the month immediately following the 4-month reference period and the majority of them are completed in the first 2 weeks of that month.

Mode of interviewing

For SIPP panels between 1984 and 1991, the preferred mode of interview and the one used in the great majority of interviews was face-to-face. Telephone interviewing was permitted to follow-up for information not obtained at the initial face-to-face interview, to interview persons who will not or cannot participate otherwise, and to interview sample persons who have moved to locations more than 100 miles from a SIPP sample PSU. For the 1984 panel, about 5.3 percent of the interviews were conducted by telephone because the households were inaccessible for face-to-face interviews. The corresponding rate for the 1985 panel was 6.3 percent. For the most part, the proportion of interviews conducted by telephone increased from the second through the final wave of each panel.

The mode of data collection was switched to maximum telephone interviewing in 1992 commencing with wave 7 of the 1990 panel and wave 4 of the 1991 panel. For the 1992 and 1993 panels, waves 1, 2, and 6 were conducted primarily using face-to-face interviewing as before. The remaining waves were conducted through telephone interviewing, to the extent feasible. Section 4.5 discusses the studies that monitored effects of maximum telephone interviewing on data quality.

Interviewing time

The length of a household interview depends in part on the number of adults (persons 15 and over) in the household. Table 4.1 shows the median length of interview (in minutes) by wave and number of household members age 15 and over for the 1985 and 1993 panels. The median for all households ranged from a low of 27 minutes to a high of 40 minutes for the eight waves. For the 1985 panel, the high occurred in the first wave and the low in the second wave, which was the only wave after the first which had no topical modules. For the 1993 panel, the high again occurred in the first wave and there is no significant evidence that telephone interviewing shortens interview times. The interview time for the subsequent waves vary, depending on the topical modules included in each wave.

Table 4.1 Median household interview times (in minutes)
by wave and number of persons aged 15 and over in household: 1985 and 1993 panels

Source: Table prepared by A. Feldman-Harkins, Division of Housing and Household Economic Statistics, U.S. Bureau of Census.

¹Per = face-to-face interview, Tele = telephone interviews.

Self and proxy respondents

Response to the survey is voluntary. The respondent rule for SIPP is that all persons aged 15 and over who are present at the time of the interview should report for themselves unless not physically or mentally able to do so. Proxy informants are accepted for persons absent or incapable of responding. Exhibit 4.1 shows the hierarchy of preferred proxy respondents established for SIPP. The procedure for identifying a proxy with telephone interviews is the same as personal visits.

Exhibit 4.1 SIPP Respondent Rules

During the first wave of the 1985 panel, 67 percent of the sample persons for whom data were obtained reported for themselves. In subsequent waves this figure was quite stable at the level of 63-64 percent (Kasprzyk and McMillen, 1987). Similar results have been observed for other panels. However, for persons completing all interviews, Table 4.2 shows that the percentage reporting for themselves in every wave was about 40 percent for the 1984 panel and about 36 percent for the 1992 panel. The distributions of sample persons by patterns of self and proxy response for the first three waves of the 984 and 1992 SIPP panels are shown in Table 4.3.

Table 4.2 Persons aged 15 and over¹ by number of proxy interviews: 1984 and 1992 panels

¹Age determined as of first interview.
Souce: Data for the 1984 panel was from Kasprzyk and McMillen (1987).

Table 4.3 Self (S) and proxy (P) response patterns for waves 1 to 3: 1992 panels

Source: Data for the 1984 panel was from Kalton et al. (1986).

Following sample persons

The SIPP "following rules," which determine which persons should be interviewed after the initial wave, are considered to be a part of the sample selection procedure and were described in Section 3.4. The preferred procedure is for the same interviewer to interview sample persons and households at each wave, but because of interviewer turnover and moves of sample persons to other locations, this is not always possible.

4.2 Data collection instruments

This section describes the SIPP data collection instruments and how they are used in the interviews for successive waves of a panel. Copies of the data collection instruments for each completed panel are included in a SIPP Informational Booklet, which may be obtained by writing or calling Customer Services, Data User Services Division, Bureau of the Census, Washington, DC 20233, (301) 457-4100. The primary data collection instruments for SIPP are the control card and the questionnaire. The Control Card is the basic field record for the sample unit and remains in the Regional Office during the life of the panel. Questionnaires for each wave are sent to a central point for processing.

SIPP control card

For each sample address, a control card is completed at the first (wave 1) interview and updated at each subsequent wave. At the start of the wave 1 interview, the interviewer uses the control card to record basic demographic characteristics for each person residing at the sample address, plus a few characteristics of the household and the housing unit. The telephone number of the household is recorded both for use in telephone interviewing in subsequent waves (introduced in February 1992) and for callbacks needed to obtain information not available at the time of the interview. To assist in following movers, each household is asked for the name and telephone number of a nonhousehold member who would be likely to know the new location of households or persons who have moved.

At the close of the first-wave interview, the interviewer transcribes a few key data items for each adult to the control card, such as names of employers, sources of income received, and assets, to aid in collecting data for these items in subsequent interviews. Social Security numbers are recorded for household members who have them, for possible use in connection with estimation and research projects requiring linkage of data from SIPP interviews with data from administrative records. A separate section of the control card is used to record information about household noninterviews when they occur.

In interviews for the second and later waves, the interviewer begins by updating the control card to identify changes in the household roster and in the demographic characteristics of persons, e.g., a change in marital status. Selected data items from the control card are transcribed to the current-wave questionnaire to help the interviewer determine which specific questions need to be asked for each person. At the close of each interview, except in the final wave, changes in the data items used for tracking employment, income, and assets are recorded on the control card.

SIPP questionnaire

A different version of the questionnaire is used for each wave and at each wave a separate questionnaire is completed for each adult. Core items, mainly about labor force participation and sources and amounts of income, are included on the questionnaire for each wave. Except in wave 1 for all panels and wave 2 for the 1984 and 1985 panels, each questionnaire has included one or more fixed and/or variable topical modules.

The interview proceeds on a person-by-person basis until a questionnaire has been completed for each adult in the household. The core items for a person are divided into four sections:

• Section 1, Labor Force and Recipiency, has two main goals—to obtain labor force status on a weekly basis for the 4-month reference period and to identify all sources of income during the 4-month period, including earnings from wages and self-employment, program benefits, and income-producing assets, such as stocks and savings accounts.
  
• Section 2, Earnings and Employment, obtains details, including monthly earnings, for jobs and separate types of self-employment reported by or for the person in Section 1. Data are obtained for employment with up to two employers and for self-employment in up to two businesses.
  
• Section 3, Amounts, obtains the amounts of income received from all sources other than earnings. In Part A, General Amounts, monthly income amounts are obtained for program benefits and pensions. A standard set of questions is completed for each income source in this category. Parts B to F cover all other income sources, such as interest, dividends, rents, and royalties. For each of these sources, only the total amount for the 4-month reference period is obtained.
  
• Section 4, Program Questions, covers participation in subsidized housing, school lunch, and school breakfast programs.

Section 4 is followed by the topical modules. After they are completed, the interview proceeds to the next adult in the household. Exhibit 4.2 shows the main steps in completing or updating the control card and completing the questionnaires for a single interview at a sample address.

Other SIPP instruments

Other instruments used in the data collection process are advance letters and respondent aids. Before each wave of interviewing, an advance letter signed by the Director of the Census Bureau is sent to each sample address. The letter informs the residents that an interviewer will be calling in a few days, explains the purpose of the survey, gives the authority for conducting the survey, and strongly encourages the recipient(s) to participate. Exhibit 4.3 shows the version of the letter used for the 1993 panel.

Respondent aids, sometimes called flash cards, play an important role in the interview. A key one is the Calendar of Reference Months, which is frequently kept in sight during the entire interview to make clear the specific months and weeks for which respondents are being asked to provide information. Other aids are lists of the categories used to respond to questions on topics such as race, origin or descent, types of income, and types of assets. These cards are shown to respondents at the appropriate stages in the interview. There are also aids for use primarily by interviewers, such as an Age Verification Chart, which the interviewer can use to determine whether reported ages and dates of birth are consistent. Checklists for interviewers remind them what steps are needed in order to prepare for an interview and what needs to be done after the interview has been completed. Respondent aids are left with the respondents after the initial face-to-face interviews. If the respondents cannot find them at the time of telephone interviews, the interviewers would read the categories to the respondents.

Exhibit 4.3 Advance Letter to Respondents: 1993 Panel

Exhibit 4.3 (continued) Reverse Side of Advance Letter

4.3 Data collection strategies

The quality of SIPP data is affected by certain broad strategies that were adopted in designing the data collection instruments and the interview process. These strategies are identified in this section. Some of them could be applied in any survey; others are applicable only in longitudinal surveys. Available information on how these strategies affect quality and some results of experiments to test alternative strategies are presented in Chapters 5 and 6.

One strategy is redundancy, i.e., collecting more than the minimum information needed to meet the survey’s data requirements and using the redundant information either to check for internal consistency or to impute missing information. The most obvious example of this in the SIPP is the Annual Roundup, a topical module used in waves 5 and 8 (waves 6 and 9 in the 1984 panel) to collect calendar year information on certain kinds of income that have already been covered on a month-by-month basis in prior interviews. The timing of the Annual Roundup is such that most respondents will have already completed their tax returns for the prior year. For some types of income, the availability of such records might lead to more accurate reporting.

Another aspect of strategy is sequencing of the items. As described in the preceding section, all types of income received are identified for each person before the person is asked for amounts and other details about each type. An alternative would be to collect detailed information on each type of income recipiency identified before proceeding to other types. Likewise, the decision to collect all data for each person before proceeding to the next person is an aspect of sequencing. An alternative would be to cover each topic for the entire household before proceeding to the next topic.

Use of records, such as pay vouchers, tax returns, and bank statements, during the interview may help respondents to provide more accurate information. One would expect the use of copies of tax returns and wage and tax statements (Form W-2) to be especially helpful in completing the Annual Roundup modules. Interviewers are instructed to encourage respondents to use records. The introduction to Section 5, Amounts Section, in the SIPP Interviewers’ Manual states: "Wherever possible the use of available records showing amounts received should be encouraged in order to ensure the best possible responses" (U.S. Bureau of the Census, 1988b).

An important strategy question that applies only to longitudinal surveys is whether each interview should be independent or dependent with respect to preceding interviews. A mixed strategy is followed in SIPP. Questions on sources of income are asked in a dependent mode. Respondents are asked about sources of income reported in the previous interview, and then are asked about new sources of income during the 4 months covered by the current interview. Starting with the 1986 panel, questions on occupation and industry of employed persons have also been asked in a dependent mode. Respondents are asked whether their activities or duties have changed during the past 8 months. A negative response would eliminate the detailed occupation and industry questions. On the other hand, questions on assets in wave 7 have been asked without reference to information on assets reported a year earlier, in wave 4. An exception was made for half of the 1984 panel. This Asset Feedback Experiment is discussed in Section 6.3.1.

4.4 Field representatives: Characteristics, training, and supervision

Characteristics

The formal titles of Census Bureau interviewers and supervisors are field representatives (FRs) and supervisory field representatives (SFRs), respectively. The FRs locate sample addresses and collect most of the data. The SFRs do follow-up work to convert potential refusals, handle assignments that are sometimes partially completed by FRs, or act as team leaders for a group of FRs.

As of September 1995, there were 335 FRs and about 50 SFRs working on the 1992 and 1993 panels. The turnover rate for SIPP FRs in Fiscal Year 1995 was 19 percent. This rate was calculated by dividing the number of FRs trained in Fiscal Year 1995 (64) by the total number of FRs (335). The average monthly assignment size in Fiscal Year 1995 was about 14 households. FRs have the entire month to complete the work, although they are encouraged to finish about 90 percent during the first 2 weeks; the last 2 weeks are spent following up movers and potential refusal cases. Both FRs and SFRs are paid on an hourly basis. As of September 1995, the starting salary ranged from $7.55 to $8.48, excluding any increases reflecting high cost-of-living geographic areas.

Training

Training for SIPP FRs includes home study, classroom training, on-the-job training, and refresher training. Training is continuous and includes:

• Initial training. FRs new to SIPP receive intensive training, including 1 day of advance self-study and 3 days of classroom training. For the first wave of a panel, there is an additional half day of training on listing operations. Training sessions include lectures, audiovisual presentations, a mock interview exercise and discussions. Trainees receive detailed information on their jobs, the concepts and definitions used in the survey, and specific interviewing techniques, such as probing. As part of the initial training, a supervisor or senior field representative (SFR) observes each new FR during his or her first 2 or 3 days of interviewing.
  
  
• Refresher training. About twice a year, FRs receive training on new topical modules and on special aspects of the survey, such as conversion of noninterviews, movers’ rules, editing, and transcription.
  
  
• Supplemental training. FRs found to be weak in certain aspects of the survey, such as completion rates and accuracy, are given supplemental training to help them meet the Census Bureau’s standards in these areas of their work.
  

Supervision and quality assurance

The work of FRs is monitored and feedback provided to them in several ways.

• Questionnaire checks: Completed questionnaires are sent to the Census Bureau’s regional offices, where they are subjected to clerical edits and simple computer edits that are incorporated into the data-entry programs. More complex edits are performed on the computerized records when they are received at Census headquarters. In some instances, FRs may be contacted to resolve problems identified in these edits.
  
  
• Performance observation. The work of each experienced FR is observed once a year by a SFR who checks the FR’s performance in establishing rapport with respondents, asking questions in an appropriate manner, probing, and recording answers accurately. The results of the observations are discussed with the FRs. FRs whose performance is below standard in some respect will have their work observed more often, as needed.
  
  
• Reinterviews. A systematic reinterview program serves the dual purposes of checking a sample of the work of individual FR’s and identifying aspects of the field procedures which may need improvement. The reinterview sample covers five to eight of the sample addresses for one-sixth of the SIPP interviewers each month. The reinterviews, which are completed as soon as possible after the original interview, are conducted on the telephone by SFRs or other members of the supervisory staff. The reinterviews are used to determine whether the FRs visited the correct units, classified noninterviews correctly, and determined household composition correctly. Several questionnaire items are checked to verify that the FR asked these items during the original interview. The results of the reinterviews are used to take corrective action, such as supplemental training and observation for FRs whose work is below standard.
  
  
• Performance standards. There are specific performance standard guidelines established for the SIPP. These are conveyed to the regional offices as examples of how they might measure performance. However, the standards are only guidelines and the regional offices are empowered to change the standards as they deem appropriate. If they choose to change the standards, they must apply the standards equally across all SIPP FRs for the same period of time. Performance standard guidelines have been established for response rates (proportion of eligible households interviewed) and productivity (average time per household, including travel), as follows (U.S. Bureau of the Census, 1988b):
  

  FR Response Rates:
  Adjective rating	Response rate
  Outstanding	100.0 - 97.5
  Commendable	97.4 - 95.5
  Full successful	95.4 - 91.5
  Marginal	91.4 - 88.0
  Unsatisfactory	87.9 and under
  Productivity:
  Adjective rating	Hours per household
  Outstanding	0.0 - 3.0
  Commendable	3.1 - 3.5
  Full successful	3.6 - 4.0
  Marginal	4.1 - 4.5
  Unsatisfactory	4.6 and over

FRs receive annual ratings based on these standards. SFR may take account of extenuating circumstances when they assign adjective ratings.

There is a concern that the current method for calculating response rates is not sensitive to increases in the number of noninterviews for unlocated movers (Type D noninterview). A Type D noninterview reduces the eligible workload by one and the response rate is computed based on the reduced workload (in the denominator). With this method, increases in the number of Type D noninterviews have little impact on the response rate, and therefore may offer little incentive for FRs to find unlocated movers. The Census Bureau is reviewing alternative methods to calculate response rates and to provide incentives for locating movers (see Section 4.5)

4.5 Evaluations of data collection procedures

This section is about response errors that are associated with the data collection phase to SIPP. The level of measurement error is affected by virtually all design features including: length of reference periods, interview mode, respondent rules, errors associated with interviewers, questionnaire length and structure, and use of records by respondents. This section discusses evaluation studies in each of theses areas and improvements to the data collection procedures to enhance data quality.

4.5.1 Length of reference period

SIPP interviews are conducted at four monthly intervals. In the core portion of the SIPP interview, respondents are asked to report on labor force activity, income recipiency, and amounts of income for the 4 months immediately preceding the month of the survey interview. For most items, they are asked to report separate data for each of the months included in the 4-month reference period. Although respondents sometimes refer to records when answering these questions, they often rely only on memory.

Memory errors in surveys have been the subject of many evaluation studies and experiments (e.g., Neter and Waksberg, 1965). Experience suggests that memory errors can be reduced by the use of relatively short reference periods and, in longitudinal surveys, by using some form of dependent interviewing such as bounded recall, where interviewers at each round of a survey have access to information reported by respondents on previous rounds.

3-month versus 6-month

The frequency of interviews and length of reference periods was a key issue in the design of SIPP and was addressed in two studies conducted as part of the ISDP. In the first of these studies (Olson, 1980; Kasprzyk, 1988), a single interview using a 6-month recall period was compared with two consecutive interviews, both using 3-month reference periods. The same 6 calendar months were covered in both sets of interviews. Not surprisingly, the proportion of respondents who reported some income from sources such as wages, Aid to Families with Dependent Children (AFDC), and unemployment compensation for the first 3 months of this period was greater for the interviews that used a 3-month reference period.

Hill (1986) used an additive error model to analyze the joint effects of length of recall and time in sample (number of waves) on earnings data for the first 9 months of the 1984 panel. The analysis was restricted to men aged 25 to 55 who reported for themselves in all three interviews, had at least 2 months with some employment, and had no self-employment and no imputed data - a total of 1,378 sample persons. He found evidence of significant differential reporting bias and response variance associated with both length of recall and time in sample.

4-month versus 6-month

The choice between 4-month and 6-month reference periods was considered for the 1996 SIPP redesign. Since a 6-month recall period involves only two interviews per year, SIPP could afford to increase sample size and reduce sampling error. However, a longer recall may reduce reports for small sources of income and incomes received for short durations, and exacerbate the seam effect observed at the transition between waves of data collection (see Section 6.1 for a discussion of the seam problem). Huang (1993) presented the results of a Recall Length Study to examine this choice. This report presents a literature review and the results of simulation studies on the effects of length of recall period on sampling error and estimates of poverty and program participation. The conclusion is that the increase in the precision of estimates cannot compensate for the increase in biases. The recommendation is therefore to maintain the 4-month reference period. More recent findings on recall bias from a record-check study are discussed in Chapter 6.

4.5.2 Interview mode

Until February 1992, face-to-face interview was the preferred mode of interviewing for SIPP. Telephone interviews were allowed when a sample person had moved more than 100 miles from a SIPP primary sampling unit or when the interview cannot otherwise be completed. Under this maximum personal visit interviewing mode, about 95 percent of all interviews were face-to-face.

Pretesting telephone interview

In a continuing effort to improve the efficiency of the SIPP design, tests of a maximum telephone mode, i.e., conducting interviews by telephone except when circumstances make it impossible, have been built into the ongoing survey operations, in order to determine how the quality of results and the cost of data collection might be affected by shifting to greater use of telephone interviews.

An off-line feasibility test of maximum telephone interviewing in a single wave was conducted in June 1985 with 279 households, in two regional offices, that had been dropped from the 1984 panel because of budget cuts. Subsequently, an on-line experiment was carried out in two phases. From August to November 1986, half of the national sample for the 1986 panel was designated for maximum telephone interviewing in either the second or third wave of the panel. The allocation to the two modes was by segment and most of the telephone interviews were done by regular SIPP field representatives, calling from their homes. In the second phase, from February to April 1987, some of the households assigned to the telephone mode in the first phase were assigned to that mode again for the wave 4 interviews. As a result, there were three experimental groups and a control group, as follows:

The experimental procedures and some results have been discussed by Durant and Gbur (1988); Carmody et al. (1988); Kasprzyk (1988); Petroni et al. (1989); Gbur and Petroni (1989); and Waite (1990). Household and person noninterview rates did not differ significantly for the experimental and control groups; item nonresponse rates were slightly but not significantly higher in the telephone interviews. The proportion of persons whose data were supplied by proxy respondents was higher for the maximum telephone group, 37.8 percent versus 35.3 percent. In interviewer debriefings, about three-fourths of the interviewers felt that telephone interviewing could be used successfully in SIPP, but others believed it would not work because of the sensitive nature of the questions and because it would be more difficult to establish and maintain respondent rapport and trust.

Implementing maximum telephone interviewing

The encouraging results from this experiment, and the cost saving benefits, led to a decision to make greater use of telephone interviewing in SIPP. Maximum telephone interviewing began in February 1992 with wave 7 of panel 1990 and wave 4 of panel 1991. For the 1992 and 1993 panels, wave 1, 2, and 6 were conducted primarily using face-to-face interview as before, but the remaining waves were conducted using telephone interviewing to the extent feasible. For waves in which telephone interviewing is the preferred mode, FRs were instructed to record the reason for obtaining a personal versus telephone interview. The reasons are mostly the following: respondent requests, no telephone or unlisted number, reluctant households (as determined by the FR), elderly persons, people with hearing problems, and language barriers.

Monitoring the effects

Since implementation, the Census Bureau has monitored the effects of telephone interviewing on cost savings and on data quality. Analyses of cost showed savings in terms of cost per household unit, interviewing hours per unit, and miles per unit (Allen, 1993). The effects of telephone interviewing on data quality showed no adverse effects on response rate (King, 1995) and on record usage by respondents during interviews (Huggins, 1994).

King (1995) found that although both Type A (refusal, no one home, etc.) and Type D (unlocated movers) nonresponse rates have risen during the 1990s, there is no consistent evidence that telephone interviewing is the cause of the increases. She compared the nonresponse rates by wave for the 1986-1993 panels and concluded that the increase in Type A rate cannot be attributed to telephone interviewing because most of the Type A nonresponse occurred in the first two waves which continued to rely on face-to-face interviews. The Type D nonresponse rate showed an increase of approximately 12 percent in the unlocated mover rate per wave in panels where telephone interviewing is used heavily, but there is no evidence of the telephone interviewing as the cause of the increase.

James (1994) reviewed the studies by Ikeda (1993) and Sutterlin (1993a) which evaluated SIPP estimates from the Education Financing module. Estimates of educational cost are derived based on data from the "telephone interview" wave (wave 5 of Panel 1991 and wave 8 of panel 1990) and from a "personal visit" wave (wave 5 of panel 1990). These estimates were compared with estimates from administrative data from the College Board and Department of Education. The results showed no evidence of an effect from telephone interviewing.

4.5.3 Respondent rules: Self versus proxy respondents

As explained in Section 4.1, all sample persons aged 15 and over who are present at the time of an interview are asked to respond for themselves unless not physically or mentally competent to do so. Proxy responses are accepted for those who are absent or incapable of responding (see Exhibit 4.1 above). Under this rule, about 65 percent of sample persons report for themselves at each interview wave. It has been generally believed that the level of accuracy for self-response is higher than for proxy response (but see Moore, 1988, for a contrary view); however, achieving a higher proportion of self-response would increase data collection costs and might lead to some increase in person nonresponse rates.

Evaluation studies

In the ISDP, the current SIPP respondent rule was experimentally compared with a maximum self-response rule, which required callbacks to interview sample persons not present at the first interview for a household (Coder, 1980; Kasprzyk, 1988). Household and person noninterview rates were slightly higher for the maximum self-response group and the estimated costs of data collection were about 4 to 6 percent higher. Under the self-response rule, records were consulted more often by persons answering wage and salary questions and response rates for hourly wage rates were higher. There was not conclusive evidence favoring either set of respondent rules, so the less stringent requirement was adopted, with the addition of a callback procedure to obtain certain critical data items if they were not obtained at the initial interview of a household.

Under the current respondent rules, most unmarried college students living away from home while attending college are considered to be members of their parents’ households and information for them is usually obtained from proxy respondents. Roman and O’Brien (1984) describe the results of an ISDP study in which data for students obtained by proxy from members of their parents’ households were compared with data collected from them by self-response at their school residences. The analysis was limited by flaws in the implementation of the test; however, it suggested that proxy response led to greater underreporting of recipiency and more nonresponse on amounts of income and expenses, especially for smaller amounts.

Hill (1987) used data from a single month of the 1984 panel in a model-based analysis of the effects of proxy versus self-response on reported earnings for a sample of men aged 25 to 55. He estimated that self-reports of earnings were about 20 percent below proxy reports. The validity of Hill’s results depends on the correctness of his assumptions concerning a common behavioral structure for proxy and self-respondents. Furthermore, without an external measure of validity, it cannot be determined which type of report is of higher quality.

The above discussion has been in terms of the effects of self and proxy responses on SIPP cross-sectional estimates. There is an additional concern about the effects of changing the informant between waves (e.g., from self respondent to proxy respondent, or from one proxy respondent to another) or longitudinal estimates, particularly on changes between waves (Kalton et al., 1989). No research has been conducted on this issue.

The evidence currently available on the effects of accepting proxy reporting does not clearly indicate the advisability of changing SIPP respondent rules regarding proxy or self–respondents. There are, therefore, no plans at present to change these rules.

4.5.4 Errors associated with interviewers

Interviewer variance

Formal interviewer variance studies carried out in connection with U.S. population censuses have shown that when the data are collected by interviewers (as opposed to the use of self-administered questionnaires), interviewer variance can contribute a substantial proportion of the total mean square error for small area counts and sample estimates (see, for example, U.S. Bureau of the Census, 1968). There have been fewer studies of the contribution of interviewer variance to error in household surveys, but a study carried out in connection with the National Crime Survey in eight cities demonstrated that it can be an important source of error for some variables (Bailey et al., 1978). An analysis by Tepping and Boland (1972) of interviewer variance in the Current Population Survey gave estimates of 0.50 or greater for the ratio of interviewer (correlated response) variance to sampling variance for several items included in the survey.

There have been no formal interviewer variance studies in connection with SIPP; however, the findings cited from other surveys and from censuses suggest that interviewer variance could be a significant source of error for some items. The contribution of interviewer variance to total error depends on the size of interviewer assignments. The larger the assignments, the greater the effect on total error. In SIPP the average interviewer assignment is about 14 per month, or 56 households per wave (each interviewed three times during a 12-month period), as compared to approximately 21 households per month or 125 households per 6-month interviewing cycle in the Central Cities Sample for the National Crime and Victimization Survey. Monthly interviewer assignments in the Current Population Survey presently average between 25 and 30 households.

Reinterview checks

The continuing SIPP reinterview program (see Section 4.4) provides some information on measurement errors attributable to interviewers. Selected questions from five content groups: job, income, benefits, health insurance, and assets are asked again in the reinterviews. The original and reinterview responses are compared and differences reconciled. When differences are confirmed, the reinterviewer judges whether or not the original interviewer was responsible for the difference in responses. Pennie (1988), summarized results of this process for 1984 through 1987. The rates presented are based on the number of content groups in each reinterview with one or more confirmed differences. Overall difference rates for the five content groups ranged from a high of 3.1 percent in 1984 to 2.4 percent in 1987. The proportion of confirmed differences attributed to interviewers stayed within a fairly narrow range and averaged 43.3 percent for the 4-year period. Although the manner in which the rates were calculated and presented does not permit precise comparisons between content groups, it appears that the health insurance and asset groups have relatively high difference rates as compared to the other three groups. The health insurance questions were modified after the 1984 panel, but the difference rates for that content group remained relatively high.

Reinterviewers also check the original interviewers’ control card listing of persons present in the household and usual residents of the household who were temporarily absent at the time of the interview. The gross errors detected in this household composition check have been consistently small, averaging 0.23 percent of persons listed in the initial interview over the 4-year period (Pennie, 1988). The average net error rate was -0.05 percent, representing a very small undercount in the initial interviews. However, results of demographic analyses and other types of evidence suggest that the proportion of persons missed in interviewed households may be considerably larger than is indicated by the reinterview results (see, for example, Hainer et al., 1988; Shapiro et al., 1993). The low figure just cited may simply mean that reinterviewers are not much more successful than initial interviewers in obtaining complete listings of household members. The Census Bureau is examining methods to modify the roster questions to improve coverage (see Section 3.5).

Standards have been established for each section of the interview that is checked in the reinterviews. Failure rates have been low; generally, less than 2 percent of the interviewers for each section checked. The usual response to failures is to discuss the errors found with the interviewers. Occasionally interviewers are reassigned or are dismissed or suspended.

Research with reinterview data

Hill (1989) undertook a detailed analysis of reinterview data for waves 2 and 3 of the 1984 panel (1,559 cases), using a data file containing reinterview reports merged with data for the same respondents from the public use file. He found the highest overall discrepancy rates, 5 to 7 percent, for items relating to Medicare and health insurance coverage. For many items included in the reinterview, he found that a high proportion of the discrepancies were procedural, i.e., the question was asked by the interviewer and not by the reinterviewer, or vice versa. He attributed this finding to unnecessarily complex skip patterns in the relevant sections of the questionnaire.

Hill also examined the relationship between response inconsistencies and respondent characteristics, using a Poisson regression procedure. He concluded that "Data quality appears to be significantly lower for low income, black, and either very young or very old respondents." His general conclusion was that the SIPP reinterview results can be valuable in understanding nonsampling error issues, and he recommended some changes to make the reinterview program even more useful.

4.5.5 Questionnaire length and structure

The ISDP included two methodological experiments which had a direct influence on the length and structure of the SIPP core modules (Olson, 1980; Kasprzyk, 1988). The first of these experiments compared the use of short and long forms to collect income data for each household member. With the short form, respondents were asked a series of direct questions about receipt of selected types of income. For all "yes" answers, questions on amounts were asked before proceeding to the next income type. The long form made extensive use of cues and probes in an attempt to help respondents recall types of income received by putting them in a broader context. Questions on amounts were not asked until all sources of income had been determined, and a longer set of questions was used to determine the amounts.

Several kinds of data were used in the analysis. Except for Social Security income, which was reported at a higher level on the long forms, the comparison of income reporting levels was inconclusive (Olson, 1980, pp. 6-46). In debriefing sessions, the consensus of interviewers was that the short form was easier for respondents, but that respondents thought more carefully about their answers when the long form was used (Olson, 1980, pp. 2-22). Not surprisingly, the long forms took longer to complete. The form ultimately adopted for SIPP was a variation of the long form used in the ISDP experiment.

The second experiment compared a household screening format with a person-by-person format for obtaining information on income recipiency and amounts. The household format was intended to reduce burden by asking a single household respondent, for each income type, whether any household members had received that type of income. For affirmative responses, the household members who had received that type of income were identified. Data on amounts were obtained only after completion of screening for all household members. The person-by-person format used a separate form for each adult household member. No significant differences in income recipiency rates and item nonresponse were observed for the two formats. Use of the person-by-person format added about 5 minutes per household to the length of the interview. The latter format, with moderate changes, was adopted for SIPP (Kasprzyk, 1988).

4.5.6 Use of records by respondents

One of the SIPP data collection strategies is to encourage respondents to use financial records to aid response. In a debriefing of a sample of 1985 panel respondents in connection with reinterviews following wave 8 interviews, 56 percent of the respondents said they had referred to their W-2 forms and completed tax forms in the final (wave 8) interview. In response to a question about the use of records such as pay stubs and bank statements in all interviews, 61 percent said they routinely referred to these kinds of records (Meier, 1988). Anecdotal information from interviewer debriefings suggests that the frequency of record use is significantly affected by the extent to which individual interviewers encourage it (Chapman, 1988).

The usage of records, as reported by interviewers, is somewhat lower. For persons responding to questions in the tax and annual roundup module, interviewers indicate in check boxes on the questionnaire which ones refer to W-2 forms and tax forms during the interview. Table 4.4 shows the results for the 1984 and 1985 panels (Altman, 1989) which indicate that records were used by around 35-45 percent of the respondents.

Table 4.4 Rate of record usage reported by interviewers

Source: Altman (1989).

Huggins (1994) summarized the studies which compared the rate of record usage between personal visit waves and telephone visit waves, and concluded that there is no evidence to suggest that the rate of record usage is affected by telephone interviewing. Jones (1993) compared the percentage of records used for all income sources reported on the Income Source Summary sheet for personal visit waves (wave 2 of 1991 panel and wave 5 of 1990 panel) and for telephone interview waves (wave 7 of 1990 panel and wave 3 of 1992 panel). She found that the average record usage rate is about 20 percent and there is no significant difference by interviewing mode. The percentages reported for the 1990-1993 panels, however, are lower, at about 20 percent. James (1992) found similar results comparing the rates for wave 4 of 1990 (personal visit wave) and wave 4 of the 1991 panel (telephone interview wave).

Guarino (1996) studied the rate of record usage by respondents in the 1990-1993 panel. She found that the rate of respondents using at least one type of record is about 20 percent for each panel. Within each panel, she observed that record usage appears to increase as the waves progress. She attributed this to a possible correlation between record usage and attrition. Those respondents who refuse to use records in the initial waves are more likely to drop out of the survey.

4.5.7 Other improvements of data collection procedures

The Census Bureau has examined different methods to improve the data collection procedure and data quality. This section summarizes some of the efforts in this area.

• In response to suggestions from the SIPP Current Operations and Research and Evaluation Committee, Singh (1989a) summarizes the Bureau’s decisions on procedural changes designed to improve data quality. Some of the changes are: better training for FRs to emphasize data quality, encouraging FRs to probe for more information, providing FRs with more information about SIPP, improving efforts to reassure respondents about confidentiality, and exploring potential improvement of the questionnaire. The Committee has also suggested several experiments, most of which were not conducted.
  .
• Singh (1989b) proposes several changes to the SIPP questionnaire for panels in the 1990s. The first change involved having FRs explain to respondents that certain months of the year contain 5 paydays for workers paid weekly and 3 paydays for workers paid every 2 weeks before the questions about monthly earnings. The second change involves having FRs explain to respondents that some persons receive more than one payment per month for certain sources of income such as unemployment compensation and Aid to Families with Dependent Children (AFDC). This explanation is to be given before the questions about monthly amounts from income sources. Including these explanations affects two questions in the Earnings and Employment section, and one question in each of the Amounts section. The goal is to improve the quality of data from these items.
  .
• Singh (1991) presents the results of an opinion survey of FRs on the usefulness of information about the government programs. A questionnaire was sent to 468 FRs and they were asked whether the availability of reference material about the government programs would help them with data collection. About 70 percent of the FRs responded to the survey. Among the respondents, about half indicated that they were using some reference aids on their own, and about 80 percent would use the aid if provided.
  .
• Allen (1991) discusses the method of probing in data collection and provides examples of how probes can be used to obtain a better response regarding specific amounts, specific lengths of time, and to clarify answers that are too general or vague. This technique is recommended for obtaining better quality data for items in the Labor Force and Recipiency Section.

SIPP Quality Profile (NEW !) Page

SIPP Main Page

Census Home Page