Discussion, Consideration and Possible Action Regarding Program Year 2017

Adult Education & Literacy Grantee Performance Measures & Targets

Introduction –TWC contracted two sets of targets to AEL Grantees in PY16: AEL Participants Served and Measurable Skills Gains (which were contracted with separate targets for each of the 11 Educational Functioning Levels). PY17/SFY18 represents the beginning of a new biennium and TWC is now accountable for also achieving post-exit outcomes using new WIOA-based measures, which we submitted as part of or Legislative Appropriations Request last summer. Staff have developed recommendations for PY17 AEL Grantee targets as outlined below.

PY17 Measures and Targets – Staff have developed the following recommendations for PY17 AEL Grantee targets as outlined below:

  • Participants Served – TWC’s target of 82,036 Participants in SFY18 assumed a mix of three tiers of service:
  • Tier I Basic AEL
  • Tier II Intensive AEL (Work-based, International Professional, and Transition to Reentry & Post Release Services)
  • Tier III Integrated Education & Training (IET) AEL

While we originally assumed that we would serve 3,000 in Intensive and 4,000 in IET, those numbers are not part of our formal measure. This gives us flexibility in both the program/casemix and the distribution of targets. Staff proposed casemixes/targets based on recent grantee experience and then let Grantees either agree to the proposals or submit counterproposals. Negotiations involved these cost assumptions applied to their PY17 Allocations[1]:

  • Tier I Basic AEL (including EL Civics without IET) at $679.25 per;
  • Tier II Intensive AEL at $1,179.25 (Tier I + $500) per; and
  • Tier III IET AEL (including Integrated El Civics) at $2,479.25 (Tier I + $1,800).

Our final recommendations continue to promote utilization of advanced AEL models consistent with the agency’s AEL Strategic Plan and the recommendations of the TriAgency Taskforce. The negotiated targets are on page 3.

  • Measurable Skills Gains – Last year, TWC negotiated both PY16 and PY17 targets with the Office of Career, Technical, and Adult Education (OCTAE) by setting separate targets for each of the 11 different Educational Functioning Levels that AEL uses for reporting. When we contracted performance with the grantees, we figured out a way to do it using a blended target approach to try to remove the disincentive for late year enrollments by giving lower targets for those enrolled in the 4th quarter. That approach, along with significant technical assistance by AEL staff to communicate the importance of offering year-round services to those who need them, may have contributed to an increase in 4th quarter enrollments with 18.76% of individuals becoming Participants in the 4th Quarter of PY16 vs 15.82% in PY15.[2] Staff developed PY17 target recommendations based on the same principal, which is discussed in further detail on Page 4. The following table shows the PY17 targets we negotiated for each EFL and how we propose to split it out into separate subtargets for those who were initially Participants in Q1-Q3 vs those in Q4 Targets:

Educational Functioning Level / PY17
ED Target / PY17
Q1-3 SubTarget / PY17
Q4 SubTarget
ABE Beginning Literacy / 63.0% / 65.0% / 55.4%
ABE Beginning Basic Education / 56.0% / 57.6% / 49.9%
ABE Intermediate Low / 55.0% / 56.7% / 48.8%
ABE Intermediate High / 47.0% / 49.6% / 37.7%
ASE Low / 53.0% / 56.5% / 41.0%
ESL Beginning Literacy / 56.0% / 56.3% / 54.7%
ESL Beginning Low / 62.0% / 63.0% / 57.0%
ESL Beginning High / 65.0% / 65.8% / 61.0%
ESL Intermediate Low / 63.0% / 64.5% / 54.5%
ESL Intermediate High / 53.0% / 53.4% / 50.8%
Advanced ESL / 53.0% / 58.4% / 20.6%

One change that we are recommending for PY17 is that we collapse the 11 different EFL measures into one overall Measurable Skills Gain measure, which is how OCTAE now does it. OCTAE had us propose 11 separate targets and then blended them together based on expected casemix between the 11 EFL groups. We recommend using a similar blended approach for the Grantees, whereby we use the state negotiated targets for each of the 11 EFLs and we blend them together based on the Grantee’s actual casemix during the year. One of the benefits of this approach is that if a Grantee underperforms slightly in one area, their overperformance in another area can make up for it. In the interest of transparency, to reduce potential confusion, and to ensure that we can monitor performance for each EFL level, staff will still produce data at the EFL level.

  • WIOA-based Outcome Measures – As noted, PY17/SFY18 represents the beginning of a new biennium and TWC is now accountable for also achieving post-exit outcomes using new WIOA-based measures we submitted as part of or Legislative Appropriations Request last summer. Therefore, staff are recommending that the Commission set the 3 new WIOA-based post-exit outcome measures using the state targets:
  • Employed/Enrolled in Q2 Post-Exit 34%
  • Employed/Enrolled in Q2-4 Post-Exit at 83%
  • Credential Rate at 32%.

Request for Commission Actions – Staff request the Commission approve staff recommendations for PY17 AEL Grantee Performance Measures and the outlined target methodologies which staff will apply AEL Grantee performance evaluation through PY17.

PY17 Participants Served Targets as Negotiated between Staff and Grantees

Based on PY17 Allocations and Average Cost Assumptions on Page 1

AEL Grantee / Grantee # / EL Civics / EL Civics IET / Intensive / IET / Total
Amarillo College / 502 / 39 / 40 / 12 / 50 / 1,430
Region 17 ESC South Plains / 526 / 26 / 27 / 121 / 50 / 1,298
Region 9 ESC / 528 / 21 / 22 / 20 / 30 / 700
Denton ISD / 536 / 118 / 119 / 34 / 144 / 4,165
Navarro College / 521 / 11 / 15 / 19 / 75 / 1,254
Weatherford ISD / 533 / 3 / 10 / 7 / 51 / 463
Tarrant County (formerly FWISD) / 537 / 89 / 151 / 280 / 108 / 5,283
Dallas County LWDB / 509 / 208 / 209 / 300 / 225 / 8,608
Paris Jr College / 523 / 16 / 17 / 5 / 19 / 571
Texarkana ISD / 531 / 7 / 8 / 4 / 17 / 387
Literacy Council of Tyler / 519 / 34 / 35 / 46 / 112 / 2,519
Abilene ISD / 501 / 0 / 40 / 45 / 27 / 1,069
Socorro ISD / 529 / 90 / 104 / 90 / 90 / 3,148
Howard College Permian Basin / 517 / 0 / 3 / 3 / 11 / 197
Odessa College / 522 / 28 / 28 / 114 / 8 / 1,398
Region 17 ESC Permian Basin / 525 / 3 / 4 / 4 / 0 / 88
Howard College Concho Valley / 516 / 21 / 22 / 10 / 16 / 588
McLennan Community College / 520 / 40 / 22 / 55 / 30 / 1,167
Austin Community College / 504 / 78 / 78 / 107 / 63 / 2,908
Community Action Inc / 508 / 85 / 86 / 75 / 175 / 1,795
Brazos Valley COG / 505 / 30 / 40 / 35 / 10 / 985
Angelina College / 503 / 25 / 25 / 57 / 31 / 1,266
Region 5 ESC / 527 / 27 / 28 / 30 / 41 / 1,243
Victoria Co Jr College / 532 / 22 / 23 / 25 / 25 / 651
Region 20 ESC / 512 / 94 / 95 / 269 / 210 / 6,575
Laredo Community College / 518 / 38 / 38 / 54 / 20 / 1,272
Del Mar College / 510 / 15 / 35 / 15 / 106 / 2,192
Region 1 ESC / 524 / 70 / 100 / 245 / 40 / 3,926
Brownsville ISD / 506 / 0 / 56 / 17 / 73 / 1,833
Grayson College / 514 / 21 / 22 / 20 / 28 / 612
Central Texas College / 534 / 4 / 5 / 4 / 16 / 336
Temple College / 535 / 24 / 25 / 42 / 25 / 1,049
Southwest Texas Jr College / 530 / 29 / 30 / 54 / 3 / 895
Houston-Galveston Area Council / 515 / 484 / 485 / 169 / 721 / 19,204
Total / NA / 1,800 / 2,047 / 2,387 / 2,650 / 81,075

Measurable Skills Gain Target Methodology Details

Distinct Populations – Measurable Skills Gains (MSG) performance is distinctly different for those who become participants late in the year than those early in the year. The fourth quarter (Q4) cohort is a distinct population compared to those who become participants in the other three quarters (Q1-3). If we break out PY16 performance for the first 3 quarters versus the 4th quarter, the difference in performance is striking:

Educational Functioning Level / Overall Performance / Overall Participants / Q1-3 Performance / Q1-3 Participants / Q4 Performance / Q4 Participants
ABE Beginning Literacy / 44.11% / 2,786 / 45.49% / 2,216 / 38.77% / 570
ABE Beginning Basic Education / 38.44% / 11,531 / 39.56% / 9,097 / 34.27% / 2,434
ABE Intermediate Low / 39.82% / 13,633 / 41.02% / 10,767 / 35.31% / 2,866
ABE Intermediate High / 34.27% / 9,736 / 36.16% / 7,614 / 27.47% / 2,122
ASE Low / 37.73% / 2,412 / 40.21% / 1,870 / 29.15% / 542
ESL Beginning Literacy / 44.17% / 16,471 / 44.39% / 13,497 / 43.17% / 2,974
ESL Beginning Low / 48.70% / 15,784 / 49.46% / 13,227 / 44.74% / 2,557
ESL Beginning High / 52.13% / 7,959 / 52.75% / 6,667 / 48.92% / 1,292
ESL Intermediate Low / 54.72% / 3,867 / 56.02% / 3,292 / 47.30% / 575
ESL Intermediate High / 41.91% / 4,176 / 42.24% / 3,499 / 40.18% / 677
Advanced ESL / 43.90% / 287 / 48.37% / 246 / 17.07% / 41

Blended Proportionate Targets – To address the issue of distinct populations in target setting, TWC used a system referred to as the “Proportionate Target Setting Method,” in which TWC sets separate targets for each of the two populations and then combines them together using a weighted average. There are three key benefits to this approach:

1) It recognizes the inherent differences in results likely to be achieved for each population and accounts for it in the target;

2) If the Grantee has a significant shift in the case mix (such as by increasing enrollment in the 4th quarter), the overall target automatically adjusts to account for the change, thus removing a potential disincentive to improving year-round enrollments; and

3) Although the target is made up of 2 sub-targets, it doesn’t double the number of measures the Grantees are expected to meet. If the Grantee meets each sub-target, then they will automatically meet the overall target. However, if a Grantee is a little low on one sub-target, they might still meet the measure because their performance with the other population can help make up the slack.

Staff took the 11 individual EFL targets we submitted to OCTAE for PY17, split them into a slightly higher target for those persons who were initially Participants in the first 3 quarters of the year (Q1-3), and applied a much lower target for those from Q4 using the ratio between Q1-3 and Q4 performance from PY16. The results were shown on page 2.

1

[1] In addition, Grantees will be responsible for serving additional Participants, where Grantees did not fully meet their PY16 Participant Targets between their original allocations and the supplemental funding approved by the Commission in December 2016. These additional numbers will be added to the PY17 Participant Served Targets.

[2] 16,650 of the PY16 88,642 Participants served became Participants in the 4th Quarter vs. 14,807 of the 93,615 PY15 Participants.