The Learnfare Program

AN EVALUATION OF

THE LEARNFARE PROGRAM

FINAL REPORT

DEPARTMENT OF

WORKFORCE DEVELOPMENT

APRIL 1997

State of Wisconsin

Legislative Audit Bureau



LEGISLATIVE AUDIT BUREAU

The Bureau is a nonpartisan legislative service agency responsible for conducting financial and program evaluation audits of state agencies. The Bureau’s purpose is to provide assurance to the Legislature that financial transactions and management decisions are made effectively, efficiently, and in compliance with state law, and that state agencies carry out the policies of the Legislature and the Governor. Audit Bureau reports typically contain reviews of financial transactions, analyses of agency performance or public policy issues, conclusions regarding the causes of problems found, and recommendations for improvement.

Reports are submitted to the Joint Legislative Audit Committee and made available to other committees of the Legislature and to the public. The Audit Committee may arrange public hearings on the issues identified in the report, and may introduce legislation in response to the audit recommendations. However, the findings, conclusions, and recommendations in the report are those of the Legislative Audit Bureau. For more information, contact the Bureau at 131 W. Wilson Street, Suite 402, Madison, WI 53703, (608) 266 2818.



State Auditor - Dale Cattanach

Editor of Publications - Jeanne Thieme

Audit Prepared by

Judith Frye, Director - Contact Person

Emma Caspar

TABLE OF CONTENTS

LETTER OF TRANSMITTAL1

SUMMARY3

INTRODUCTION5

The Evaluation Design5

Measures of School Participation10

Implementation Lessons13

THE TEENAGERS’ EXPERIENCE WITH LEARNFARE15

The Families’ Introduction to Learnfare15

Learnfare Sanctions and Services17

EFFECTS OF LEARNFARE ON SCHOOL PARTICIPATION21

School Participation Among All Research Sample Teenagers22

School Participation for Weighted Sample23

School Participation Among Dropouts24

School Participation Among Enrolled Students26

School Participation Among Teenage Parents27

School Participation Among Teenagers in Different Age Groups29

School Participation Among Teenagers Within and Outside Milwaukee County36

School Completion41

EFFECTS OF LEARNFARE ON PUBLIC ASSISTANCE RECEIPT AND DURATION OF LEARNFARE PARTICIPATION 43

Receipt of Public Assistance43

Effects of the Duration of Learnfare Participation43

INTERPRETATION OF RESULTS47

Sample Attrition47

Decreasing Exposure to Learnfare51

Programs Without Attendance Requirements51

Conversion of the Statewide AFDC Case Records System51

LEARNFARE IMPLEMENTATION: LESSONS FOR OTHER STATES53

School Data53

Program Management55

Support Services55

Case Management Service Linkage57

Due Process and Appeals Procedures58

Data Management System59

APPENDIX I - METHODOLOGY

APPENDIX II - RESULTS FOR ALL SUBGROUPS AND STATISTICAL SIGNIFICANCE

DETAIL

APPENDIX III - EXAMPLE OF RESPONSE TO ATTENDANCE PROBLEM UNDER

LEARNFARE FOR MILWAUKEE PUBLIC SCHOOL STUDENTS

****

April 28, 1997

Ms. Linda Stewart, Secretary

Department of Workforce Development

201 East Washington Avenue

Madison, Wisconsin 53703

Dear Secretary Stewart:

This report contains the results of the fourth analysis of outcome data for the evaluation of the Learnfare program. It describes the study teenagers’ enrollment, attendance, unexcused absences, and school completion, as well as the amount of benefits received from Aid to Families with Dependent Children during families’ first four semesters following entry into the study, as described in the December 1992 research design. Six semesters of data are reported for those who entered the study earliest. This is the final report for attendance and enrollment outcomes, and it includes a summary of implementation lessons from the various Learnfare studies completed by the Legislative Audit Bureau.

In previous interim reports, we found that the early positive effects of Learnfare on the school participation of several groups of teenagers had virtually disappeared after three semesters. The fourth semester results largely corroborate these findings.

Fourth semester analyses include school data for only about 75 percent of our original research sample. Subjects not in the analysis include those who had graduated from high school or its equivalent, left Wisconsin, or reached 20 years of age. We were also unable to locate some sample members, and some who were enrolled in programs that do not collect attendance could not be included in the attendance analyses. Individuals for whom we were missing data were evenly divided between the Learnfare and control groups, and we do not believe that this attrition caused bias in the results.

We appreciate the courtesy and cooperation extended to us by staff of the Department of Workforce Development and the school districts, technical colleges, and private schools that assisted in the collection of data.

Sincerely,

Dale Cattanach

State Auditor

DC/JF/aw

SUMMARY

Wisconsin’s Learnfare program is intended to encourage enrollment, regular attendance, and high school graduation or the completion of high school equivalency programs among 13- to 19-year-old recipients of Aid to Families with Dependent Children (AFDC). These teenagers, who can be either dependent children or parents, risk losing part or all of their families’ monthly AFDC grants if they do not maintain enrollment and acceptable school attendance. In addition, the program offers participating families assistance in identifying and correcting the causes of attendance problems, and services such as day care for the teenagers’ children, transportation, and referral to alternative education programs. The program was administered by the Department of Health and Social Services until July 1996, when that responsibility was transferred to the new Department of Workforce Development.

This evaluation report describes the effects of Learnfare on the school participation and school completion of teenagers and on their families’ public assistance payments. Because individuals entered the sample at different times, some were in the study for longer than others. All sample members were tracked for at least four semesters after introduction to Learnfare. Six semesters of data are reported for those who entered the sample earliest.

The study uses an experimental design. Between March 1993 and April 1994, 3,205 teenagers in ten counties were selected for this study. Before any were introduced to the Learnfare program, one-half were randomly chosen to experience Learnfare as usual. The remaining teenagers were assigned to a control group and made exempt from the program. Learnfare was not explained to these teenagers or their families, so that their behavior will indicate how the Learnfare teenagers might have behaved in the absence of the program. To observe the long-term effects of the program and to preserve the comparability of the Learnfare and non-Learnfare groups, teenagers remain in the study and in the group to which they were assigned for the duration of the study, regardless of their continuing participation in AFDC or Learnfare.

As subjects entered the study, we began to collect information regarding their monthly school participation and AFDC case histories. This information was used to analyze the effects of Learnfare on the group as a whole and on different groups of subjects: teenagers who were and were not enrolled in school when they entered the study; teenage parents and non-parents; teenagers by age group; and teenagers within and outside Milwaukee County.

In the first and second semesters, when the school participation of specific groups of teenagers was examined separately, some beneficial effects of the program were evident for dropouts, teenage parents, older teenagers, and teenagers who lived outside Milwaukee County. However, these effects had virtually disappeared by the third semester. In the fourth semester, the lack of effects continued. The only detectable fourth-semester effect was a negative effect on enrollment for the group of teenagers who were enrolled in school when they entered the study. Within this group, the Learnfare teenagers had a slightly lower average number of months enrolled in school than the non-Learnfare teenagers.

Data describing the levels of school participation among the groups of teenagers who responded to Learnfare in any of the first four semesters, and for the full sample in any of the six semesters, are included in the body of this report; additional detail and results for those groups of teenagers for whom we found no detectable response are described in Appendix II.

The consequences that additional school participation might have for the teenagers’ later success in school, or any other longer-term consequences, can be examined to some extent after four semesters of data have been collected. Although the majority of the sample is too young to have graduated from high school by the end of the fourth semester, we did examine the results for those who were old enough. The analysis found that Learnfare had no significant effect on the rate of school completion in this group or in any subset of it.

The effects of Learnfare on public assistance receipt were evaluated by examining AFDC eligibility and receipt data. There was no significant difference in the amount of AFDC received by Learnfare families compared to non-Learnfare for the full sample, or for any subgroup except Milwaukee County in the first semester. There was no difference between the Learnfare and control groups in the number of periods of AFDC receipt. The Learnfare group did have, on average, slightly shorter periods of receipt than the control group.

This report includes observations about program design and administration that might be useful to other states considering implementing a program similar to Learnfare. These implementation lessons are based on previous Audit Bureau reviews of Learnfare program administration, sanction delays in Milwaukee County, and Learnfare case management.

****

1

INTRODUCTION

Wisconsin’s Learnfare program is intended to promote enrollment, improve school attendance, and increase high school graduation or the completion of high school equivalency programs among 13- to 19-year old recipients of Aid to Families with Dependent Children (AFDC). Teenagers subject to Learnfare may be either dependent children or teenage parents receiving AFDC for their own children.

If these teenagers do not maintain enrollment and acceptable school attendance, they risk losing all or part of their families’ monthly AFDC grants. In addition, the program offers participants opportunities to work with case managers to identify and correct the causes of the attendance problems and offers services such as day care, transportation, and referral to alternative education programs.

This is the fourth and final Audit Bureau report in a long-term evaluation of the effects of Learnfare. It updates earlier interim reports that described the teenagers’ school participation during the first three semesters after introduction to Learnfare. An earlier report released in February 1994, “An Evaluation of Learnfare Case Management Services,” reviewed the delivery of supportive services associated with the Learnfare program. In Fall 1994, the Learnfare program was extended on an experimental basis in four counties to children younger than 13; the effect of Learnfare on these younger children is the subject of a separate study.

The Evaluation Design

The research design of this study was described in “Research Design: Evaluation of the Learnfare Program” (December 1992) and was approved by the State of Wisconsin Department of Health and Social Services and the U.S. Department of Health and Human Services, Administration for Children and Families. The study is designed to examine the effects of the Learnfare program upon four outcomes:

• enrollment;
  
• attendance;
  
• graduation from high school or completion of high school equivalency programs; and
  
• receipt of public assistance.

In addition to determining the effects of Learnfare upon the AFDC teenage population as a whole, analyses are included to determine the program’s effectiveness among different groups of teenagers:

• teenage parents and teenagers without children;
  
• teenagers who were and were not enrolled in school when introduced to Learnfare;
  
• teenagers of different ages; and
  
• teenagers within and outside Milwaukee County.

Teenagers were selected for the study at the time they were scheduled to be introduced to Learnfare, usually when a new AFDC case was initiated or as part of a routine eligibility review after the oldest child in a family had turned 13. Before Learnfare was introduced to the families, they were randomly assigned either to the Learnfare group or to a comparison, or control, group.

Members of the Learnfare group experienced Learnfare as usual. Control-group members were not introduced to Learnfare and are neither subject to Learnfare requirements nor eligible for Learnfare services for the duration of the study. Control-group teenagers remain eligible for similar services provided to all AFDC families through other programs, subject to availability.

Random assignment of research subjects to treatment and control groups is the key to the experimental design. It ensures that any unmeasurable factors that might affect school participation, such as individual motivation, parental involvement, and local school policies, will occur approximately equally among treatment and control subjects. Once assigned to either the Learnfare or non-Learnfare group, teenagers remain in that group for the duration of the study, even if their families leave AFDC or if they in some other way become exempt from Learnfare. In this way, we are able to take into account long-term effects of Learnfare participation and preserve the comparability of the two groups.

Selection of the research subjects and their random assignment to the experimental groups took place between March 1993 and April 1994. Ten counties participated: Brown, Douglas, Eau Claire, Kenosha, La Crosse, Marathon, Marinette, Milwaukee, Portage, and Racine. These counties were selected as representative of Wisconsin’s AFDC population and include both urban and rural areas. Results of the sample-selection process are shown in Table 1.

1

Table 1

Baseline Characteristics of Learnfare Evaluation Research Sample

Learnfare / Control / Percentage
Group / Group / Total / of Total
Total Sample / 1,620 / 1,585 / 3,205 / 100.0%
Parent or Pregnant / 275 / 307 / 582 / 18.2
Non-parent / 1,345 / 1,278 / 2,623 / 81.8
Milwaukee County / 1,006 / 1,016 / 2,022 / 63.1
Non-Milwaukee County / 614 / 569 / 1,183 / 36.9
Enrolled in School / 1,376 / 1,323 / 2,699 / 84.2
Not Enrolled in School / 244 / 262 / 506 / 15.8
Ethnicity
Black / 697 / 780 / 1,477 / 46.1
White / 557 / 483 / 1,040 / 32.4
Hispanic / 224 / 184 / 408 / 12.7
Asian / 115 / 113 / 228 / 7.1
Native American / 27 / 25 / 52 / 1.6
Male / 667 / 648 / 1,315 / 41.0
Female / 953 / 937 / 1,890 / 59.0
Age at entry
into the study
13 / 881 / 866 / 1,747 / 54.5
14 / 160 / 147 / 307 / 9.6
15 / 149 / 131 / 280 / 8.7
16 / 119 / 111 / 230 / 7.2
17 / 106 / 129 / 235 / 7.3
18 / 139 / 141 / 280 / 8.8
19 / 66 / 60 / 126 / 3.9

1

Teenagers selected for the study were those who met program requirements for participation in Learnfare. They were:

• between the ages of 13 and 19, inclusive;
  
• not yet graduates of high school or high school equivalency programs; and
  
• either parents themselves or living with a parent.

In addition, each teenager was to be introduced to the program for the first time in one of three situations:

• as a member of a family group applying to open a new AFDC case, either as a dependent teenager or as a teenage head-of-household;
  
• as a member of an ongoing AFDC case who had recently turned 13; or
  
• as a teenager not previously participating in Learnfare who was being added to an ongoing AFDC grant, such as a teenager who moved from the home of a non-AFDC parent to the home of a parent receiving AFDC.

As each research subject entered the study, we began to collect information on the teenager’s month-by-month school enrollment and attendance, and on high school completion and the family’s receipt of public assistance. Details of the methodology used to complete the study are described in Appendix I.

Results were analyzed by semester. Because individuals entered the sample at different points in time, some were in the study for longer than others. Those who entered the sample earliest have been in for six semesters, while those who entered last have been in for four semesters. Most of the data presented here are for the four semesters during which the entire sample was in the study. Some results include data from six semesters for those individuals who entered the sample earliest. In every case, semesters are counted relative to when an individual entered the sample. So, for example, “first semester” refers to the semester in which an individual entered the study, or the first semester following entry for individuals assigned during a school recess.

The effects of Learnfare can be observed by comparing the average school participation of the Learnfare group in each semester to that of the control group in the same semester. No conclusions about Learnfare can be drawn by comparing any group’s school participation rate from one semester to the next: both the Learnfare and non-Learnfare groups are likely to enroll and attend less as they get older. Instead, any change in the effects of Learnfare over time is shown in the changing magnitude of the differences between the Learnfare and non-Learnfare teenagers.

Comparing length of enrollment between the first and later semesters, for any group, is misleading for another reason. All subjects were in the study for the entire duration of their second and following semesters (4.5 months), but many subjects had a shorter possible first-semester enrollment because they entered the study at some point after the start of the school year. Therefore, the apparent increase in most groups’ average length of enrollment from the first to the second semester is not directly indicative of increasing levels of enrollment.

To produce the findings described in this report, statistical analyses were performed to separate the effects related to one of the subjects’ characteristics, such as age, while controlling for the effects related to others, such as whether teenagers were parents. For example, the effects of parental status were held constant in the analysis of results by age group, while the effects of age were held constant in the analysis of effects by parental status. In this way, we can observe Learnfare effects related to the quality of being 18 or 19 years old separately from those related to being a teenage parent, although many subjects are in both groups. The analyses also controlled for the effects of other observable characteristics, including sex, ethnicity, family size, and length of participation in Learnfare.

Because some variation between any two groups of people will occur simply by chance, the differences that remained after controlling for observable characteristics were then tested for statistical significance. These tests determined the probability that each difference might have occurred by chance even if there was, in fact, no difference between the two groups.