Georgia welfare leavers study technical appendices

Georgia Welfare Leavers Study
Technical Appendices
Prepared for: Georgia Department of Human Resources Division of Family and Children Services
January 2000
Prepared by:
E. Michael Foster, Ph.D. Principal Investigator Dana K. Rickman, Project Manager
Applied Research Center Andrew Young School of Policy Studies
Georgia State University

Table of Contents Technical Appendix I -- Design of the Georgia Study Technical Appendix II -- Design of Other Leavers Studies Technical Appendix III -- Progress to Date Technical Appendix IV -- Analysis of Non-Response Technical Appendix V -- Results from Other Leavers Studies References

1 8 13
22 28
33

Technical Appendix I -- Design of the Georgia Study The following provides an overview of the Georgia State study and compares it to that of leavers studies in other states. While there are similarities, this study differs in several crucial ways from these other studies. Definition of Leavers Like the other funded ASPE studies, the Georgia leavers study defines leavers as cases not having received cash assistance for two months. However, unlike the other studies, Georgia is the only study including child-only cases in their population of leavers. It is also important to note that we do not exclude cases that return to the rolls from our analyses. We believe this offers a fuller picture of how leavers are faring as a whole. (Of course, we can compare cases that have and have not returned to the rolls.) Data Sources Like the other studies, our study relies on interview data as well as data from various administrative databases. Telephone Interviews. As designed, the study involved ongoing telephone interviews with a sample (n=4,800) of families having left TANF. We are interviewing approximately 200 women per month over a 24-month period, extending from July 1999, through June 2001. Our sample size is quite large compared to the other ASPE studies. Only four other studies have a sample size more than 1,000. Use of Administrative Data. We are linking survey respondents to various state databases. The initial sample of cases is drawn from DFCS' database of closed clients. The TANF database also

Leavers StudyInitial Report Technical Appendices (2/4/00)

1

provides basic demographic information, including race, age, gender and relationship to other persons in the household. Additionally, the database provides information on case status, issued payments, case and client id numbers, address, telephone number, and county, office, and supervisor number. The initial closed-case file is matched with the TANF Emergency file that provides supplemental information at the time the case closed, such as food stamp and Medicaid receipt, work eligibility status and work experience, reported earnings, and family structure. Three to four months later, these files are again matched with DFCS' database of active cases to determine recidivism rates and possible location information.
This study also uses the Child Support Enforcement (CSE) Database. The CSE database provides two important pieces of information. First, administrative matches are used to verify or update address and telephone information. CSE clients still receive their checks through the mail; therefore, CSE database is more likely to have accurate information than TANF records. Second, the CSE database provides benefit history and amount information.
Taken together, the combined administrative databases provide information on employment, food stamp use, past use of TANF (both cash payments and other involvement, such as sanctions), current uses of TANF, as well as receipt of child support.
For a complete comparison survey and administrative data across studies, see Table 11. Research Topics
When combined, the administrative and interview data cover a wide range of topics consistent with those covered in similar leaver studies.

Leavers StudyInitial Report Technical Appendices (2/4/00)

2

C

Employment and Earnings

The telephone survey asks questions pertaining to the individual's present employment situation, hours

worked, earnings, and type of employment. Questions also probe job search strategies, and length of

employment or unemployment. The TANF Emergency file also provides information on earned and

unearned income and on work activities before exit.

C

Other Income Supports

In addition to earnings through employment, the telephone survey provides detailed information about

income support, including CSE payments, SSI payments, and earned income credits.

C

Health Insurance

The telephone survey includes questions about Medicaid and employer-provided health insurance on

their surveys for adults and children.

C

Child Care

We are relying solely on survey data for this information. Detailed questions are asked concerning

types of child care, frequency of use, and satisfaction of child care options.

C

Child Well-being

The combination of questions in the telephone survey is aimed at assessing child-well being. Basic

information about health insurance, school attendance, participation in child care, and disability is

collected. In addition, we also ask information about the involvement of absent fathers, home

environment, and domestic violence issues.

C

Barriers to Self-Sufficiency

Leavers StudyInitial Report Technical Appendices (2/4/00)

3

The telephone survey address such subjects as disability, maternal depression, illiteracy, domestic

violence, and the lack of education/skills. The Mother's Mental Health/Domestic Violence module

includes the Rosenberg self-efficacy scale and the CDI depression scale.

C

Deprivation/Insecurity

The survey asks whether former recipients encounter severe problems in meeting basic needs, including

hunger, access to health care, and housing problems. We address food insecurity, living arrangements,

and health care.

C

Attitudes Toward TANF and Work

We are surveying former recipients about their attitudes toward TANF or welfare reform. We pose

questions concerning their satisfaction with TANF, how they feel now that they have left, their

confidence they will stay off, and their general knowledge of the welfare laws.

In sum, the Georgia State study has features in common with the other ASPE studies, but key

differences remain. Like the ASPE studies, the definition of a leaver is based on a two-month period

during which a case receives no cash assistance. The Georgia State and other state studies utilize

similar sources of administrative data and provide information on a similar range of research topics.

Georgia is also similar to the other states in terms of the content of the survey interview.1

However, key differences remain between the Georgia State study and the other leavers

studies. As noted, this study includes child-only cases and does not exclude cases that have returned to

1Because it is the only study to employ matrix sampling, Georgia's survey interview is among the most comprehensive. There are some topics that we have emphasized less than other states, such as recidivism or detailed neighborhood or community characteristics.

Leavers StudyInitial Report Technical Appendices (2/4/00)

4

the rolls. In addition, the number of interviews being conducted in this study is generally much larger than in other states. This larger size provides the depth necessary to examine multiple comparisons according to (1) rural-urban location, (2) race, (3) education of leaver; (4) poverty status of the neighborhood, (5) time spent by leaver on welfare and, (6) reason leaver moved off roles (e.g., whether sanctioned). (For an overview of planned comparisons and those made in other studies, see table 12). We believe this information is critical. It is likely that the effects of welfare reform differ across families of different types. For example, a black never-married woman with young children living in an urban area may be affected quite differently than a white divorced woman living in a rural area. Describing such diversity requires a large sample.
There are other differences. The Georgia State study focuses on cases that have closed since the summer of 1999. Therefore, we have no information about those who left under AFDC or immediately after TANF was implemented. Furthermore, our ability to examine recidivism is limited by data availability. Currently, we have only received information on current receipt only for the respondent and only for a single month.

Leavers StudyInitial Report Technical Appendices (2/4/00)

5

Table 11: Study Overviews

Population

Administrative Data (All sites include TANF, Medicaid enrollment,
and Food Stamps

Sample Size

Sample Mode

Adult TANF Leavers Child Only Cases Sanctioned
Eligible not enrolled Diverted
Employment CSE
Child Welfare Foster Care WIC
Housing Asst. SSI
Health Services Mixed Mode Telephone

Georgia Arizona Cuyahoga County District of Columbia
Florida Illinois Los Angeles Massachusetts Missouri New York San Mateo Washington Wisconsin South Carolina Total

///

/

/

7800 completes

/

/

/

/

/

800 completes

/

/

/

/

/

300 completes

/

500 sampled

/

375 completes

/

/

/

15,000 completes

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

//

/

800 completes

/

330 completes

/

600 completes

/

4800 completes

/

1200 sampled

/

/

950 completes

/

/

// /

/

/

//

/

/

1000 completes

/

/

/

2000 completes

/

/

/// /

/

/

/

1000 sampled

/

14 1 3 2 5 11 9

6

2

1

2

1

3

2649 (average) 13 1

6

Table 12: Research Topics

Subgroups of Analysis

Urban/Rural Sanctioned Cases Time limit leavers
Race/ethnicity Age of youngest child
Age of recipient Length of Spell Single v. married
Type of exit Recidivism
Number of Children Education and training levels
Housing Assistance AFDC v. TANF leavers

Georgia Arizona Cuyahoga County District of Columbia Florida Illinois Los Angeles Massachusetts Missouri New York San Mateo Washington Wisconsin South Carolina Total

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

/

10

9

4

12

6

4

3

4

6

3

2

3

2

7

7

Technical Appendix II -- Design of Other Leavers Studies This section addresses the research design for a sample of other leaver studies. It is inevitable that our findings will be compared with those in other states, and so it is important to describe the research designs of these studies. Since the Georgia State project received supplemental funding from the Assistant Secretary of Planning and Evaluation (ASPE) of the US Department of Health and Human Services, our review focuses on the other states who received funding at the same time: Arizona, Cuyahoga County (Ohio) , District of Columbia, Florida, Illinois, Los Angeles County, Massachusetts, Missouri, New York, San Mateo County (California), Washington, Wisconsin, and South Carolina. These studies, most notably South Carolina and Wisconsin, include some of the most publicized leaver studies to date. We present an outline of these studies and review how leavers are defined, sources of data, and research topics. Definition of Leavers The ASPE studies share a common definition of a "leaver"--all cases that leave cash assistance for at least two months. This definition excludes cases that reopen within one or two months; the rationale is that these cases involve 'churning' of families on and off the roles rather than a true 'exit' from welfare. States differ along a variety of other dimensions, however. South Carolina, for example, defines leavers as individuals who have left cash assistance for two months and who do not return during the course of the study. Clearly, economic hardship is associated with a return to the rolls, and ignoring

Leavers StudyInitial Report Technical Appendices (2/4/00)

8

these individuals produces a rather unrepresentative picture of all leavers. Other states include individuals who return to the rolls as part of the leavers population.
Another restriction involves child-only cases. All of the ASPE grantees (except Georgia) exclude child-only cases. This is particularly unfortunate. Those cases comprise a substantial portion of the welfare rolls; in Georgia, they represent 20 percent of the leavers. Furthermore, TANF may alter the movement of children into and out of child-only cases, and as a result, ignoring those cases may produce a rather incomplete picture of the impact of welfare reform. Furthermore, if this process differs across states, then it becomes virtually impossible to make cross-state comparisons of study findings.
Finally, the states vary in the time period with which they define leavers. About half the grantees2 are drawing their first cohort of leavers from the fourth quarter of calendar year 1996. As a result, their analyses include women and children who have left welfare before welfare reform was enacted. Data Sources
All studies are employing a combination of linked administrative data and survey data to study outcomes for families leaving TANF. For virtually all the studies, the administrative data being used include TANF, Food Stamps, Medicaid, and some type of wage record database. Most of the wage records come from the Unemployment Insurance system; however, two of the states are using the state's Department of Revenue records and the New Hire database. Child welfare, child support, and

2Grantees looking at populations from 1996 are Arizona, Cuyahoga County, Los Angeles County, Missouri, San Mateo County, Washington, and Wisconsin.

Leavers StudyInitial Report Technical Appendices (2/4/00)

9

JOBS or JOBS successor data are also being analyzed. A few of the studies are analyzing information

about child care subsidies, general assistance, or SSI data. Other examples of administrative data

being used include substance abuse, publically funded mental health services, WIC, housing assistance,

vocational education, school attendance, emergency, and JTPA data.

In addition to administrative data, each study is interviewing leavers. The majority of the surveys

take 20-30 minutes. Sample sizes for the first year are generally between 600-1200 completed

interviews, although three studies are using sample sizes of less than 400.

With the exception of Florida, the studies are all engaging in a mixed-mode survey, with first

contact made by telephone. In-person interviews are planned for hard-to-locate individuals and those

without telephones. Follow-up methods to locate leavers who are initially unreachable by telephone are

as follows:



using other administrative databases to locate a current phone number,



using commercial locating services,



sending a letter with an 800 number and an offer of an incentive payment to the last

known address,



sending an in-person interviewer to the last know address to question current residents

and neighbors, also offering an incentive payment, and



obtaining tracking information at exit including contact information for several relatives

or friends.

Research Topics

Leavers StudyInitial Report Technical Appendices (2/4/00)

10

Each of the studies combines administrative and survey data to address a number of important

research questions. These can be grouped into general areas: employment and earnings, other income

supports, health insurance, child care, child well-being, barriers to self-sufficiency,

insecurity/deprivation, and attitudes toward TANF and work. Key topics include

C

Employment and Earnings

All of the studies are using a combination of administrative data and survey data to examine such issues

as employment status, quarterly earnings, wage levels, hours worked, and types of jobs or occupations.

C

Other Income Supports

Each of the studies also investigates the receipt of food stamps and child support through administrative

and survey data. Other types of income supports specifically addressed in the surveys include SSI,

general assistance, housing assistance, energy assistance, and EITC.

C

Health Insurance

Most of the studies are using administrative data to examine Medicaid and employer-provided health

insurance on their surveys.

C

Child Care

All of the studies are analyzing child care use by families leaving welfare. Most of the states are relying

solely on survey data for this information.

C

Child Well-being

Many of the studies include some indicators of child well-being. The most common source of

Leavers StudyInitial Report Technical Appendices (2/4/00)

11

administrative data is the child welfare system (i.e., data on child abuse and neglect, foster care, or

both). Furthermore, a majority of the studies are including types of survey questions concerning

children's living arrangement and/or interactions with the child welfare system. A few surveys also

address questions about child health status, children's school attendance, and child behavior.

C

Barriers to Self-Sufficiency

All of the studies examine barriers to self-sufficiency primarily through survey questions. The survey

questions address such subjects as disability, maternal depression, substance abuse, illiteracy, domestic

violence, and the lack of education/skills.

C

Deprivation/Insecurity

For the most part, the studies are using survey questions to determine whether former recipients

encounter severe problems in meeting basic needs, that is, issues associated with hunger, access to

health care, use of emergency services, and housing problems.

C

Attitudes Toward TANF and Work

Some studies are surveying former recipients about their attitudes toward TANF or welfare reform.

Additionally, leavers are being asked about their attitudes toward work.

Leavers StudyInitial Report Technical Appendices (2/4/00)

12

Technical Appendix III -- Progress to Date Missing contact information in the administrative data represents a significant challenge to conducting the interviews required for this project. From the data provided by DFCS, approximately 15 percent of the cases have no telephone number (this number does not include those numbers deemed invalid or disconnected), and another 40 percent have incomplete address information. In order to locate individuals, we have implemented a step-by-step process that utilizes all of the resources available to us. This process represents a significant improvement over what was included in our original proposal. Much of this expanded work has been paid for using the ASPE funding. For this reason, we review our current efforts to locate respondents and then describe the fruits of those efforts to date. Definitions for Selecting Respondents To identify which individual in the case to interview, we combine two different codes in the TANF database: the relationship code and the financial responsibility code. For single-parent cases, we select the individual whose relationship code is "SE" (self, head of household) and their financial responsibility code is "RE" (recipient). Teen mothers are identified the same manner, but the search is limited to those under 18 years of age. We identify child-only cases based on the same two codes. Those cases with a relationship code of "SE" (self, head of household) and a financial responsibility code that is not "RE" (recipient) are identified as as respondents in child-only cases. This coding was confirmed by the work participation code in the TANF Emergency file.

Leavers StudyInitial Report Technical Appendices (2/4/00)

13

Efforts to Locate Respondents The process of locating respondents begins with contact letters that are mailed (at the start of
the month) to all individuals eligible to be interviewed. The letter announces that we will be contacting them over the next month, explains the study, and describes how their name was selected. The letter also provides a toll-free telephone number for the individual to call the survey lab if they feel we will have difficulty locating them over the telephone. The letter also offers $25 to any individual who calls the Applied Research Center to complete the interview.3
After the letters are mailed, we being our efforts to contact respondents by phone. The Applied Research Center's telephone survey lab utilizes a variety of techniques to increase our response rate. First, numbers are called up to seven times until they can be verified as disconnected or invalid. Calls are made to hard-to-reach households at various times throughout the day and week. For numbers with answering machines, messages (with the toll-free number) are left, and incentives are offered for participation. Disconnected and invalid numbers are recalled after two weeks to allow for the reconnection of telephone service. As we locate additional phone numbers through our search efforts (described below), contact information is updated, and the process continues for up to two months or more.4
For individuals for whom DFCS has no phone number or for whom the DFCS number is

3The incentive is also offered to those we locate through our in-person tracking efforts. For one cohort, June interviewees, we lowered the incentive payment to $15.00 and experienced a dramatic decrease in the number of call-ins. We raised the incentive payment back to $25.00 in August and the call-in rates have increased.
4We originally attempted to contact respondents for one month, but we soon learned that this period was too short.

Leavers StudyInitial Report Technical Appendices (2/4/00)

14

disconnected or otherwise invalid, we begin a series of steps to locate valid contact information. First, all individuals are matched to the food stamp database to update address and telephone information. We then use directory assistance and reverse directory look-ups to identify new numbers. (This is done using a software program called Pro-CD. It contains nationwide addresses and telephone numbers. It can be utilized as a source for directory assistance or reverse directory look-ups.)
Numbers are then matched against the Child Support Enforcement (CSE) database. CSE mails checks directly to their clients. Therefore, the addresses in their database tend to be more accurate than the food stamp information. Finally, when no valid phone number can be located, inperson trackers go to the last know address of the respondent and start their field tracking procedures. Currently, in-person trackers are limited to the Atlanta metropolitan area, which is defined as inside the I-285 perimeter. Once individuals are located, the trackers provide them with a cellular phone to complete the interview over the phone. Multiple visits are made until the respondent's address is verified. If the respondent is not home, a card notifying them of the tracker's visit and the lab's phone number is left at the address. Scheduled appointments are also made if the respondent is busy. Response Rates
The Georgia leavers study has been running smoothly since June 1999. A change in data management systems (from PARIS to SUCCESS) delayed the receipt of reliable data for approximately six months. Once we began to receive data again, approximately 10 percent of the counties were missing from our sample. As a result, the surveys from September 1998-April 1999 (completed during the spring and early summer of 1999) were only partial state samples of those who

Leavers StudyInitial Report Technical Appendices (2/4/00)

15

left during 1998. The June survey cohort was the first complete sample we received using the new system. Therefore, the results presented below begin with the June cohort.
June Survey: The June survey ran from June 10 thru August 12 with Success data from cases that closed during January and February 1999. One hundred eight (18%) of the June sample did not have a telephone number when the sample was drawn. Directory assistance and field visits were attempted for these cases as well as for those whose phone numbers from the sample file proved incorrect. A total of 156 surveys were completed from the original sample of 600 (26.0 % response rate). However, excluding cases in which the respondent could not be located resulted in a sample of 293 cases and an adjusted response rate of 53.2%. The more thorough step-by-step procedures outlined above were not implemented until the August sample. Furthermore, the incentive payment for June was only $15.00, further lowering the response rate. (See table 13 for a breakdown of the results).
August Survey: The August survey was in the field from August 12 thru November 11, and involved cases that closed during March 1999. The detailed, step-by-step locating procedure was implemented with this sample, and the incentive payment was raised back to $25.00. A total of 204 completed interviews were obtained from the original sample of 600 (34.1% response rate). This number represents 55.1% of cases for whom a respondent could be located. Two refusals were received for a cooperation rate of 99%. Table 15 summarizes the sample dispositions.
As indicated by the field visits (table 16), efforts to locate respondents were unsuccessful about two-thirds of the time. Most unsuccessful attempts were due to empty households, incomplete

Leavers StudyInitial Report Technical Appendices (2/4/00)

16

addresses, or unsafe environments where no attempt was made to contact the respondent. September Survey: The September survey began on September 16, and involved cases that
closed during April 1999. These results are based on completed interviews as of November 11. At that time, interviews were still being conducted. A total of 204 completes had been obtained from the original sample of 600 (34% response rate). Excluding cases in which the respondent could not be located after exhausting all available resources results in a sample of 413 potential cases and an adjusted response rate of 49.4%. Four refusals were received for a cooperation rate of 95.8%. Since the survey is still in the field, we anticipate all response rates will increase. Table 17 clearly shows that the September sample contained more valid phone numbers compared to the August sample, resulting in fewer calls and fewer respondents that could not be located. The success rate for field visits in September increased over the rate for August. For this cohort, fewer addresses were incomplete or were in an unsafe environment. Results are presented in Table 18.
October Survey: The results from the October survey are not included in this report as the data were not available in time. However, preliminary results from the October survey are encouraging. As of November 17, a total of 226 completed surveys had been obtained from the original sample of 600 (37.7% response rate). This response rate is after only four weeks in the field. By comparison, four weeks into the August cohort, we only had a response rate of 18.1%. Based on our current progress, the final response rate for the October cohort should reach well above 50%. One refusal has been received for a cooperation rate of 99.6 percent. These encouraging results coincide with the implementation of a more rigorous and thorough system of locating respondents through our

Leavers StudyInitial Report Technical Appendices (2/4/00)

17

step-by-step tracking process initially implemented in the August survey. It also coincides with the availability of more accurate and current data being provided by DFCS.

Leavers StudyInitial Report Technical Appendices (2/4/00)

18

Table 13

June Survey

Monthly Response Rates

Percent

N

Average # of Calls

Total sample size

600

6.7

Valid sample size*

48.8

293

9.1

Response rate (based on N=600)

26.0

156

3.8

Adjusted response rate (based on N=293)

53.2

156

3.8

Refusal rate (based on N=293)

1.7

5

11.4

Noninterview rate** (based on N=293)

45.1

132

15.4

Unable to locate

51.2

307

4.3

* Excludes those where no working telephone number could be established ** Includes households where no one was ever reached, respondent unavailable, or no adult in the household.

Table 14 June Field Visits

Percent

N

Visits made

41

No one at home

29.3

12

Bad address/unsafe neighborhood

17.1

7

Person at address, but never heard of respondent

9.8

4

Person knows respondent, but respondent does not live there

7.3

3

Respondent at address, scheduled call back

22.0

9

Completed survey

14.6

6

Table 15

August Survey

Total sample size

Percent

N

Average # of Calls

600

5.6

Leavers StudyInitial Report Technical Appendices (2/4/00)

19

Valid sample size*

61.7

370

5.8

Response rate (based on N=600)

34.1

204

5.2

Adjusted response rate (based on N=370)

55.1

204

5.2

Refusal rate (based on N=370)

.5

2

3

Noninterview rate** (based on N=370)

27.8

103

6.6

Unable to locate (based on N=600)

28.2

229

5.4

* Excludes those where no working telephone number could be established ** Includes households where no one was ever reached, respondent unavailable, or no adult in the household.

Table 16 August Field Visits

Percent

N

Visits made

72

No one at home

33.3

24

Bad address/unsafe neighborhood

30.6

22

Person at address, but never heard of respondent

13.9

10

Person knows respondent, but respondent does not live there

9.7

7

Respondent at address, scheduled call back

5.6

4

Completed survey

6.9

5

Table 17 September Survey
Total sample size Valid sample size* Response rate (based on N=600)**

Percent
68.8 34.0

N

Average # of Calls

600

5.2

413

6.1

204

4.2

Leavers StudyInitial Report Technical Appendices (2/4/00)

20

Adjusted response rate (based on N=413)

49.9

204

4.2

Refusal rate (based on N=413)

2.2

4

7.6

Noninterview rate** *(based on N=413)

30.5

126

8.4

Unable to locate (based on N=600)

31.2

187

3.6

* Excludes those where no working telephone number could be established ** As stated, these complete are as of Nov. 11. The survey is still in the field. *** Includes households where no one was ever reached, respondent unavailable, or no adult in the household.

Table 18 September Field Visits

Percent

N

Visits made

141

No one at home

41.1

58

Bad address/unsafe neighborhood

9.9

14

Person at address, but never heard of respondent

5.0

7

Person knows respondent, but respondent does not live there

10.6

15

Respondent at address, scheduled call back

14.2

20

Completed survey

12.8

18

Leavers StudyInitial Report Technical Appendices (2/4/00)

21

Technical Appendix IV -- Analysis of Non-Response The response rate for this project is less than ideal but is in the range of response rates among comparable projects. The quality of a study, however, depends not only on the response rate but on the extent to which respondents and non-respondents differ. The response rate could be rather high (80%), but the study might be very misleading if the 20% who do not respond differ substantially from those who complete interviews. At the same time, a study with a much lower response rate might describe the population of leavers accurately if respondents and non-respondents are similar.5 Generally, any bias induced by non-response is difficult to diagnose. In many instances, one knows very little about these individuals. In the case of leavers studies, however, we know a great deal about these individualsbasic demographics, history of welfare use at the time they left the rolls and whether they have returned to the rolls. This information is extremely valuable and provides important insights into the nature of any biases induced by non-response. In Table 19, we consider differences among individuals for whom the project was and was not able to complete telephone interviews. The table presents the results of a logit model, a form of regression well-suited to dichotomous outcomes. In that model, we consider whether one could have predicted who would complete interviews based on various characteristics (taken from administrative data). We estimated statistical models using the roughly 1800 individuals we tried to contact for the study. (To make sure we obtained 200 complete interviews per month, we attempted to locate 600

5Of course, all else equal, a higher response rate is desirable because (1) the number of observations is greater, increasing statistical power; and (2) the potential bias caused by differences between respondent and non-respondents is greater at higher levels of non-response. (If the response rate is 98%, then the potential bias is still rather small even if nonrespondents are quite different.)

Leavers StudyInitial Report Technical Appendices (2/4/00)

22

individuals per month.) The table has five columns. The first is the estimated beta coefficient from the regression model; and the second is the corresponding standard error. The third and fourth columns present t-statistics and the corresponding p-value. These gauge the statistical significance of the relationship between a given characteristic and the likelihood of response. Finally, because the beta coefficient has no straightforward interpretation, we translate that coefficient into a 'marginal effect'the impact of the characteristic of interest on the likelihood that an individual was interviewed. For example, individuals with no phone number in the DFCS administrative data ("nophone" cases) were 23 percentage points less likely to complete an interview.
Table 19 indicates thatin generalvery little distinguishes survey respondents from nonrespondents. The two groups do not differ in terms of age, race, household size, the presence of young children, household type, number of months on the rolls, or whether they had returned to the rolls by September of 1999. Overall, the model has very little predictive power. The pseudo-R square is .02. (Given that this is a logit model, the pseudo-R square is not a measure of explained variance. However, it is bounded by 0 and 1 and does measure the explanatory power of the model. In this case, that power is very low.6)
Two differences, however, separate individuals we interviewed from those we did not. First, respondents were more likely to have a phone number in the original DFCS data. (By this, we mean that the data contained any phone number. The number still may have been disconnected.) As noted, the results of the logit model tell us that we were 23 percentage points less likely to locate individuals

6 See Davidson, Russell, and James G. MacKinnon. 1993.

Leavers StudyInitial Report Technical Appendices (2/4/00)

23

for which we originally had no phone number. Second, we were more likely to interview individuals who were receiving food stamps when they left TANF. Presumably, this is because they were more likely to maintain correct contact information with the state after leaving welfare. This effect is relatively small (5 percentage points).
What are the consequences of these results? In general, they are very reassuring. Our survey data do not over-represent the experiences of individuals who have been on the rolls for a long (or short) period of time. Nor are the data unrepresentative in terms of whether an individual returned to the welfare rolls. The data appear to be representative along racial and demographic lines. However, as noted, there are two differences that require some interpretation andpotentiallysome statistical adjustment to the data.
How do these differences affect our results? One key factor is whether the characteristics that differuse of food stamps and "nophone" statusare related to outcomes of interest. If they are not, then the fact that our data are not representative with regard to these characteristics is of no consequence. There is some evidence that no such relationships exist. After all, we have actual data on some key outcomes (past and future welfare use), and the data do not indicate that respondents and non-respondents differ.
However, as a precaution, we considered the impact of correcting for non-response on our findings. The most straightforward adjustment one might make to the data would be to assign probability weights. These weights would be the inverse of the predicted probability that a respondent completed an interview. The practical effect of using probability weights is to increase the emphasis

Leavers StudyInitial Report Technical Appendices (2/4/00)

24

given to those respondents who are in under-represented groups. In the case at hand, this means that we will give more emphasis to the "nophone" cases that we interview, inflating their impact on tabulations to better represent their presence in the population of welfare leavers.
What happens when we calculate sample weights and use them in our calculations? The answer is very little. Consider table 20: it documents the distribution of monthly income by category with and without weights. One can see that the two columns of numbers are virtually identical. Weighting has no impact on the estimated distribution of income. This is because the factors that influence response"nophone" status and receipt of food stamps when leaving TANF are unrelated to family income.
What the table does not show is that weighting reduces the level of statistical precision (Thompson 1992). This effect can be rather substantial. Balancing the limited gains from weighting against the certain costs, we have chosen not to weight our tabulations.

Leavers StudyInitial Report Technical Appendices (2/4/00)

25

Table 19: Analysis of Non-Response

Demographics

Beta

SE T-value

Age of respondent

0.0074 0.0061 1.21

Household size

-0.0068 0.0573 -0.12

Presence of child in household ages 5 or under-0.0545

0.0779 -0.70

Race (white=1 ; 0 = other)

-0.0608 0.1369 -0.44

p-value 0.23 0.91 0.48 0.66

Marg. Effect 0.00 0.00 -0.01 -0.01

Characteristics of case when leaving TANF

Household type

Two-parent household1

-0.1558 0.4999 -0.31

0.76

Child-only case1 0.0509 0.1736

0.29

0.77

Amount of benefits in last month

0.0000 0.0007 -0.04 0.97

# Months on rolls when left

0.0032 0.0063 0.51 0.61

# Months squared

0.0000 0.0010 -0.02 0.98

Received Food Stamps when left rolls

0.2465 0.1161 2.12 0.03

-0.03 0.01 0.00 0.00 0.00 0.05

Other Characteristics Returned to the rolls by September No phone number in DFCS data Intercept

.0.0412 0.1673 0.25 0.81 -1.0549 0.1757 -6.01 0.00 -1.0147 0.2906 -3.49 0.00

0.01 -0.23
NA

Number of observations=1738 Log Likelihood=-1058.541 Pseudo-R square=.02

NOTES 1Relative to omitted category, one-parent households. Statistically significant relationships shaded.

26

Table 20: Impact of Weighting on

Distribution of Monthly Income

Unweighted Weighted

<$500

13%

14%

$500-$799

27%

27%

$800-$999

28%

27%

$1,000-$1,200

18%

18%

$1,201-$1,499

5%

5%

$1,500-$2,500

7%

7%

>$2,500

3%

2%

Leavers StudyInitial Report Technical Appendices (2/4/00)

27

Technical Appendix V -- Results from Other Leavers Studies In this final appendix, we briefly review findings from other leavers studies. We used these studies to identify key outcomes and characteristics on which to focus our analyses. As noted, these studies are not directly comparable to this work because of differences in design. However, other studies can suggest key outcomes and trends which we can examine. We also can look for broad consistencies. We begin our review with results from the other 13 ASPE studies. To date, published results from those studies have focused on administrative data. As a result, we fill out our picture of prior research using findings from a broader array of studies that include results from interviews. Preliminary Analysis of Administrative Data Seven of the FY 1998 ASPE-funded grantees -- Arizona, Georgia, Missouri, New York, Washington, San Mateo County in California, and Cuyahoga County in Ohio -- have released interim reports. These reports are based on the use of linked administrative data for families who left welfare in late 1996/early 1997. These interim reports provide interesting preliminary findings about former AFDC/TANF recipients in the areas of employment, earnings, returns to cash assistance, and participation in other programs. Preliminary findings from the eight studies were consistent, particularly in the areas of employment, earnings, and recidivism. Employment. According to the eight reports, between 50 and 60 percent of former TANF recipients found work immediately after leaving TANF. Employment rates fell slightly throughout the first year after exit. Over the 12-month period, some former recipients lost their jobs, while others

Leavers StudyInitial Report Technical Appendices (2/4/00)

28

found new employment. This trend resulted in cumulative employment rates of 65 to 75 percent, measured as those who were ever employed within the first 12 months of exit.
Earnings. The studies also used administrative data from the UI system to determine the earnings of welfare leavers. In the quarter immediately following exit from TANF, mean quarterly earnings ranged from $2,185 to $3,868. In every location, earning steadily rose over the course of the year following exit.
Recidivism. Data from six states suggest that between 5 and 20 percent of leavers were receiving welfare again one quarter after exit. Many of these leavers re-entered in the third month itself since cases that reopen after one or two months were excluded from the study population. The proportion of former recipients receiving TANF reached between 13 and 28 percent at two quarters after exit, and then rose more slowly reaching 13 to 29 percent one year after exit. The proportion that ever returned for at least one month over the first 12 months after exit was somewhat higher, ranging from 24 percent in San Mateo County to 35 percent in Cuyahoga County.
Participation in other programs. A few states provided information about food stamp receipt, Medicaid, and other work support programs with administrative data bases. Medicaid enrollment varied over time and unit of analysis. Among those studies that reported Medicaid enrollment among both adult leavers and their children (Missouri, New York, San Mateo County), enrollment among children was slightly higher. Medicaid enrollment also tended to decline over time in the year after exit from AFDC/TANF. For example, in Missouri, 35 percent of adults and 41 percent of children were enrolled in Medicaid 3 months after exit; at 12 months after exit, the percentages

Leavers StudyInitial Report Technical Appendices (2/4/00)

29

dropped to 15 percent of adults and 37 percent of children. A substantially higher percentage of leavers reported ever being enrolled in Medicaid than claimed to have been enrolled in each of the four quarters.
In the fourth quarter following exit from TANF, between 14 and 40 percent of leavers were receiving food stamps. Among the studies reporting Food Stamp data, the percentage of leavers participating in the Food Stamp program appeared to be lower than the percentage who received Medicaid. Survey Results from Other Studies
In April 1999, the General Accounting Office (GAO) published a report examining seventeen studies of families who left AFDC or TANF during or after 1995. Each of these states reported the economic status of leavers, and most reported family composition and child well-being. Because states generally initiated tracking studies to meet their own information needs, the studies differ in many important ways, including the categories of families tracked, geographic coverage, time-periods covered, and the timing and frequency of follow-up. The studies also differed in the sources of data used (i.e., mail surveys, telephone surveys, in-person interviews, or administrative data). Because of these differences, the findings were not completely comparable across states. However, these studies provide an indication of the status of families who have left welfare.
The studies had consistent findings on employment and earnings. Employment rates ranged from 61 to 87 percent for adults in families that had left welfare. Average quarterly earnings for former recipients ranged from $2,378 to $3,786. If these earnings are the only source of income for the

Leavers StudyInitial Report Technical Appendices (2/4/00)

30

families after they left welfare, many of them remain below the federal poverty level. While receiving AFDC or TANF, families generally received Medicaid. However, the GAO
found that whether Medicaid benefits are retained after leaving welfare depends on many factors. For the children who left welfare, approximately 9 percent in South Carolina, 20 percent in Oklahoma, and 35 percent in Indiana did not have health insurance at the time of follow-up. For adults who left welfare in these states, 24 percent in Oklahoma, 48 percent in South Carolina, and 54 percent in Indiana did not have health insurance.
In general, the studies provided little information on family and child well-being. Although a major goal of welfare reform was the promotion of two-parent families and the reduction of out-ofwedlock pregnancies, the tracking studies report only minimal information on family composition at the time of data collection, and no information on changes that may have occurred just before of after leaving welfare. With regard to measures on child well-being, six states included data on homelessness or separation of children from their parents. They reported no indication of increased incidences of these outcomes at the time of follow-up.
Two studies, South Carolina and Wisconsin, asked former recipients to compare several aspects of their general well-being after leaving welfare with their situation when they were on welfare. Former welfare recipients in both states were more likely to experience some deprivation after leaving welfare than while on welfare. At the same time, in South Carolina and Wisconsin, 76 and 68 percent, respectively, disagreed or strongly disagreed with the statement that "life was better when you were getting welfare."

Leavers StudyInitial Report Technical Appendices (2/4/00)

31

In its conclusion, the GAO states that the studies were able to provide limited information about the status of former welfare recipients. In general, the information on economic status of the families being tracked indicate that many are finding only low-paying jobs. These low wages highlight the importance of income supports, such as subsidized medical and child care, and the earned income credit.

Leavers StudyInitial Report Technical Appendices (2/4/00)

32

References Davidson, Russell, and James G. MacKinnon. 1993. Estimation and Inference in Econometrics.
Oxford University Press: New York. General Accounting Office. 1999. Status of Former Welfare Recipients. Review of studies
conducted by Idaho, Indiana, Iowa, Kentucky, Louisiana, Maryland, Michigan, Montana, New Jersey, New Mexico, Oklahoma, Pennsylvania, South Carolina, Tennessee, Washington, Wisconsin, and Wyoming. GAO/HEHS-99-48. Washington DC: US Government Printing Office. Loprest, Pamela. 1999. Families Who Left Welfare: Who Are They And How Are They Doing? The Urban Institute Report 99-02. Risler, Ed and Larry Nackerud, 1999. The Georgia Welfare Reform Research Project: The Remaining TANF Recipients. Athens: University of Georgia, School of Social Work. National Survey of American Families [On-line]. Available http://newfederalism.urban.org/nsaf/. Thompson, Steven K. 1992. Sampling. John Wiley & Sons, Inc.: New York.

Leavers StudyInitial Report Technical Appendices (2/4/00)

33