Discussion, Consideration and Possible Action Regarding Local Workforce Development Area Performance Measures and Targets for Board Contract Years 2016 and 2017

BCY17 WIOA-based Measure Targets

Introduction

WIOA §116(c) identifies siloed, Title I performance measures to be used for Local Workforce Development Board (Board) performance accountability and requires use of a federally developed statistical model to help negotiation and agreement on Board targets to account for economic conditions and characteristics of participants. U.S. Department of Labor (DOL) published the model to be used for 11 of the Title I performance measures on June 29, 2016. However, WIOA Rule 677.210 further provides that the “negotiations will include adiscussion of circumstances not accounted forin themodel.”

In addition, TWC previously approved several ‘Integrated’ WOIA-based measures to be used for BCY17 to replace the old integrated Common Measures. While these measures are WIOA-based, we’re not required to follow WIOA §116 in setting their targets.

Staff have completed negotiations with the Boards on both the WIOA Title I siloed measures and the TWC Integrated measures and developed target recommendations for Commission approval. Most of the remainder of this briefing document focuses on the process for negotiating WIOA Title I measure targets, as it was the most involved.

Evaluation of the Statistical Model for WIOA Title I Measures

Staff evaluated DOL’s model and found a number of issues which created concerns about how well the model would work in the negotiation process. These concerns only grew as staff tested the model against actual Board-level data from BCY16 and found that only 20-25% of the model’s predictions were within 10% of the actual performance.

Although DOL’s actual statistical work was excellent, there were several key issues that led to this outcome:

1)The source data was based on the old “Pre-WIOA” standards for Exit (which was often later WIOA);

2)There was a change in the Credential Rate definition after the model was built;

3)The DOL methodology used for state-specific “correction factors” to address fundamental differences from state to state since the impact of each factor may not be universal but we didn’t have time to implement such factors for each Board[1]; and

4)DOL’s data requires each Participant/Period of Participation record to be assigned one and only one Board, even if the Participant was served by multiple Boards (which is very common in Texas).

Negotiation Process for WIOA Title I Measures

Staff realized that while the model itself might not be as accurate as we want for setting the actual target, we could still use it in a manner which would allow negotiations in accordance with WIOA’s requirements to use the statistical model to account for economic and demographic characteristics. Staff developed a negotiation tool, which used the BCY16 casemix and economic conditions and the anticipated BCY17 casemix and economic conditions (since we already know the exiters), to determine how things would change for each Board on 50 factors. We then used the model weighting to estimate how much performance should go up or down from BCY16 levels and applied a floor and ceiling (as we commonly do on targets for other measures) and sent those to the Boards as proposals for their responses.[2] Staff reviewed those responses and provided feedback to the Boards, which led to dialogs with several Boards.

Seventeen Boards agreed to the target proposals without question or comment. Several others had questions or misunderstandings about one or more of the proposals, which we were able to easily resolve. We also had significant dialog with 7 of the Boards reviewing their questions and input, doing additional analysis and providing new views of the data for their consideration. In some cases, we ended up accepting the Boards alternate proposals. In others, we found that the Board was raising important points that were not applicable to this particularly set of targets. For example, one Board was concerned about the impact of discontinuing a special program and we were able to show that the program was still operational for the period being reported in BCY17, so that their concern might be relevant in BCY18 but not this year.

One key item is that all this is very new to the Boards and TWC staff and there will be more to learn. Boards did not have the benefit of access to participant-level performance data, to better understand what went into their BCY16 performance and how it might change for BCY17. Further, TWC is still months away from fully implementing a permanent WIOA performance reporting solution and WIOA provides for target adjustments at the end of the year,which means there is a lot left to learn and there will be an opportunity to apply that knowledge at the end of the year.

Negotiation Process for Integrated Title I Measures

While DOL requires states to negotiate targets for 11 siloed, Title I WIOA measures for BCY17, the Commission also approved a staff recommendation that we replace our old integrated Pre-WIOA Common Measures with a set of four new WIOA-based measures that are integrated across programs.

However, staff ran into difficulty developing a methodology and targets for one of the measures: the All Job Seeker Credential Rate. We were trying to develop the measure by taking the federal Title I Credential Rate measure and applying it to all programs but as it turns out, that’s a bit more complicated than anticipated, partially due to our inability to get answers to several questions from DOL. Therefore, staff did not develop targets for this measure and will continue to work to develop the measure for BCY18.

On the remaining measures, staff developed target proposals based on the average level of performance over the last several years, slightly weighted towards more recent performance and no Board requested an alternate target.

Commission Request

Staff request the Commission approve the BCY17 targets listed on pages 3 and 4 with the understanding that staff will continue to work with Boards to review the measures, performance, and the statistical model to support yearend adjustments.

1

Board / # / Adult Employed Q2 / Adult Median Wages Q2 / Adult Employed Q4 / Adult Credential / DW Employed Q2 / DW Median Wages Q2 / DW Employed Q4 / DW Credential / Youth Employed / Enrolled Q2 / Youth Employed / Enrolled Q4 / Youth Credential
Panhandle / 1 / 81.3% / $6,530 / 84.5% / 69.9% / 84.5% / $7,550 / 72.7% / 68.5% / 76.6% / 82.0% / 71.1%
South Plains / 2 / 80.2% / $7,590 / 84.5% / 67.7% / 85.2% / $8,070 / 83.5% / 90.0% / 72.4% / 81.1% / 76.8%
North Texas / 3 / 85.0% / $4,930 / 76.7% / 53.3% / 87.3% / $6,940 / 72.2% / 90.0% / 67.2% / 80.6% / 61.8%
North Central / 4 / 77.7% / $4,490 / 75.4% / 67.8% / 80.0% / $7,130 / 76.9% / 64.4% / 68.8% / 82.0% / 66.6%
Tarrant County / 5 / 80.9% / $5,000 / 77.5% / 77.3% / 87.3% / $8,070 / 81.8% / 85.6% / 71.9% / 79.4% / 72.3%
Dallas / 6 / 75.2% / $6,500 / 73.0% / 74.0% / 86.0% / $8,070 / 83.5% / 73.5% / 59.8% / 71.5% / 47.6%
North East / 7 / 79.4% / $6,460 / 74.1% / 60.8% / 86.5% / $6,660 / 76.1% / 55.6% / 76.6% / 82.0% / 76.8%
East Texas / 8 / 72.3% / $4,340 / 70.0% / 61.6% / 76.9% / $5,470 / 74.1% / 71.7% / 71.9% / 78.1% / 65.7%
West Central / 9 / 77.6% / $4,090 / 69.3% / 78.1% / 71.7% / $7,140 / 60.2% / 54.6% / 63.8% / 60.5% / 47.6%
Upper Rio Grande / 10 / 85.0% / $7,590 / 79.2% / 72.3% / 76.7% / $5,840 / 68.7% / 77.6% / 70.4% / 72.5% / 60.7%
Permian Basin / 11 / 84.6% / $7,590 / 84.5% / 78.1% / 87.3% / $8,070 / 83.5% / 89.7% / 76.6% / 82.0% / 70.7%
Concho Valley / 12 / 85.0% / $6,650 / 84.5% / 78.1% / 87.3% / $8,070 / 81.1% / 90.0% / 65.3% / 61.6% / 76.8%
Heart of Texas / 13 / 85.0% / $7,380 / 84.5% / 71.7% / 79.5% / $6,220 / 75.2% / 74.8% / 76.6% / 82.0% / 62.2%
Capital Area / 14 / 77.5% / $7,470 / 84.5% / 68.6% / 82.2% / $8,070 / 78.4% / 78.4% / 76.6% / 75.8% / 76.8%
Rural Capital / 15 / 75.2% / $6,010 / 78.5% / 75.4% / 81.5% / $7,850 / 83.4% / 90.0% / 71.3% / 69.3% / 68.3%
Brazos Valley / 16 / 82.8% / $4,490 / 77.8% / 76.9% / 84.8% / $5,990 / 67.4% / 79.6% / 73.3% / 71.6% / 61.1%
Deep East Texas / 17 / 83.9% / $5,240 / 83.6% / 69.7% / 80.4% / $6,210 / 67.4% / 90.0% / 63.8% / 67.2% / 50.2%
Southeast Texas / 18 / 79.9% / $5,350 / 74.8% / 65.8% / 83.1% / $6,660 / 83.5% / 79.3% / 76.6% / 82.0% / 66.7%
Golden Crescent / 19 / 81.9% / $7,590 / 82.8% / 65.0% / 82.1% / $6,410 / 83.5% / 90.0% / 73.3% / 82.0% / 76.8%
Alamo / 20 / 80.9% / $5,990 / 77.4% / 73.0% / 84.3% / $6,870 / 83.2% / 86.4% / 66.5% / 67.6% / 56.0%
South Texas / 21 / 83.0% / $7,590 / 71.5% / 78.1% / 64.1% / $4,190 / 50.7% / 90.0% / 76.2% / 73.4% / 68.0%
Coastal Bend / 22 / 85.0% / $5,940 / 83.9% / 60.2% / 82.3% / $4,980 / 77.9% / 64.7% / 70.8% / 74.4% / 73.7%
Lower Rio Grande / 23 / 79.0% / $7,590 / 82.2% / 74.4% / 84.2% / $4,780 / 79.2% / 88.8% / 68.4% / 74.3% / 76.8%
Cameron County / 24 / 85.0% / $5,500 / 80.9% / 78.1% / 87.3% / $8,070 / 83.5% / 90.0% / 76.6% / 82.0% / 76.8%
Texoma / 25 / 85.0% / $7,590 / 84.5% / 74.6% / 76.8% / $7,780 / 83.5% / 67.7% / 76.6% / 73.1% / 44.2%
Central Texas / 26 / 85.0% / $7,590 / 84.5% / 78.1% / 87.3% / $8,070 / 78.1% / 85.0% / 73.7% / 77.8% / 76.8%
Middle Rio Grande / 27 / 81.5% / $5,040 / 81.9% / 78.1% / 87.3% / $6,330 / 70.4% / 90.0% / 65.6% / 76.6% / 55.3%
Gulf Coast / 28 / 75.0% / $3,910 / 71.5% / 47.7% / 81.3% / $6,980 / 76.1% / 66.6% / 63.8% / 67.7% / 41.7%
Proposed Target Ceiling and Floor / Adult Employed Q2 / Adult Median Wages Q2 / Adult Employed Q4 / Adult Credential / DW Employed Q2 / DW Median Wages Q2 / DW Employed Q4 / DW Credential / Youth Employed / Enrolled Q2 / Youth Employed / Enrolled Q4 / Youth Credential
Ceiling / 85.0% / $7,590 / 84.5% / 78.1% / 87.3% / $8,070 / 83.5% / 90.0% / 76.6% / 82.0% / 76.8%
Floor / 75.2% / $4,490 / 71.5% / 60.8% / 76.8% / $5,470 / 67.4% / 64.7% / 63.8% / 67.6% / 47.6%
Board / # / All Job Seeker Employed/ Enrolled Q2 / All Job Seeker Median Earnings Q2 / All Job Seeker Employed/ Enrolled Q2-4
Panhandle / 1 / 67.0% / $4,790 / 79.9%
South Plains / 2 / 64.3% / $3,930 / 77.9%
North Texas / 3 / 64.6% / $4,310 / 79.4%
North Central / 4 / 61.0% / $5,010 / 81.4%
Tarrant County / 5 / 63.1% / $4,520 / 80.9%
Dallas / 6 / 62.3% / $4,340 / 80.4%
North East / 7 / 63.4% / $3,970 / 78.8%
East Texas / 8 / 64.0% / $4,330 / 79.9%
West Central / 9 / 65.9% / $4,150 / 78.9%
Upper Rio Grande / 10 / 61.8% / $3,820 / 79.4%
Permian Basin / 11 / 68.1% / $5,970 / 79.7%
Concho Valley / 12 / 66.5% / $4,650 / 80.4%
Heart of Texas / 13 / 64.1% / $3,780 / 79.6%
Capital Area / 14 / 64.2% / $5,120 / 81.9%
Rural Capital / 15 / 65.1% / $5,700 / 83.0%
Brazos Valley / 16 / 63.9% / $4,000 / 78.3%
Deep East Texas / 17 / 62.2% / $3,920 / 77.3%
Southeast Texas / 18 / 65.4% / $5,040 / 79.5%
Golden Crescent / 19 / 69.1% / $5,310 / 81.5%
Alamo / 20 / 63.2% / $4,760 / 81.0%
South Texas / 21 / 62.7% / $4,130 / 79.6%
Coastal Bend / 22 / 65.3% / $4,920 / 79.9%
Lower Rio Grande / 23 / 67.2% / $4,470 / 78.0%
Cameron County / 24 / 65.1% / $4,190 / 79.3%
Texoma / 25 / 64.6% / $4,590 / 79.6%
Central Texas / 26 / 60.2% / $4,340 / 79.8%
Middle Rio Grande / 27 / 66.8% / $4,120 / 76.7%
Gulf Coast / 28 / 61.4% / $4,080 / 78.3%

1

[1] For example, the impact of a change in natural resource sector employment in Texas is a much bigger issue than in Vermont. The same is true for between Board areas like Permian Basin and Cameron County.

[2] The floors were applied in a slightly more lenient manner for this first year.