NRS Webinar Questions/Answers from June 8, 2012

  1. So is there a difference between performance standards, as measured by ACLS and the NRS reports? Would we get credit for meeting goalsdefined by ESEwhich do not correspond to criteria related to outcomes followed within any of the NRS cohorts?

A:The National Reporting System (NRS) and the Massachusetts Performance Standards are both accountability systems. The NRS is the federal reporting system which each state that receives federal funds for ABE must use to report information about the number of students served, demographic information about those students and the certain outcomes achieved by those students. For the NRS, we report aggregate information and as a state, we are held accountable for achieving outcomes based on performance targets that are negotiated with the USDOE. Beginning in FY13, the NRS will be capturing outcomes for some of its measures differently. Specifically, students will be assigned to cohorts for the outcomes related to high school completion, entered employment and entered post-secondary education.

For more information about the NRS:

The Massachusetts Performance Standards are used to measure performance at the program level. There are six performance standards that have been informed by statistical analysis of multiple years of data, a goals pilot, and input from the field of ABE practitioners in multiple task groups and workshops at statewide conferences. Each of the standards is set at or near the Massachusetts state average for performance in the following areas: attendance, average attended hours, pre and post testing percentage, learner gains, setting and meeting student goals, and Educational Functioning Level completion. The performance standards, as one part of a larger system of accountability, encourage continuous improvement, effective program administration, and positive student outcomes. For more information about the performance standards:

  1. What are the reasons for these changes? Does this have to do with improving ways that goals and outcomes are captured?

A:The USDOE is implementing these changes in order to capture the maximum number of student outcomes. Additionally, it ensures that all states are using the same criteria for reporting purposes.

  1. So if students are automatically being placed into cohorts, will programs still need to set the "A" goals?

A:The change to automatic cohorts does not impact the importance of authentic goal setting. Please review the goal setting section in the Massachusetts Guidelines for Effective Adult Basic Education (FY 2013), pp.55-61

  1. Will students with no attendance for 90 days be automatically exited by the SMARTT system or will we still have to go to the SMARTT system in order to exit them?

A:Students are not automatically exited; rather this is a reporting feature to determine inclusion in a cohort. For NRS reporting purposes, student information will be treated in the same manner as that of an exited student if they have no attendance for 90 days or the program has exited the students. The date of exit or the last date of participation combined with other characteristics where they apply such as current employment status, date of GED test completion, etc. are determining factors for cohort designation.

These new reporting procedures will not affect or change in any way the data that is actually stored within the SMARTT system. Programs will still be responsible for maintaining the accuracy of data they store within the SMARTT system.

  1. Is it true that both of the dates concerning entry and exit from programsdetermine entry into NRS follow-up cohorts? In what ways will information within SMARTT be used for determining cohort entry?

A:At the state level, students will be considered exited if they have no attendance for 90 days or the program has exited the students. The date of exit or the last date of participation combined with other characteristics where they apply such as current employment status, date of GED test completion, etc.are determining factors for cohort designation.

  1. We don't usually exit students from the program until the end of the fiscal year because they sometimes return. How does this impact their cohort designation?

A:For the purposes of NRS reporting, criteria used for determining cohort entry concerning those whose last date of attendance was at least 90 days ago would be the same as that used if the student had instead been exited at such time.

  1. How would we handle students who have left for the summer but are returning in the fall. They would have 90 days of inactive attendance. Any suggestions? Some programs have 90 day summer vacations.

A:Keep mind that these reports are based in fiscal years.

  1. Why do we only care about becoming employed following exit? What about students who gained employmentbefore exiting?

A:For NRS reporting purposes, only students becoming employed after they exit the program are included in the performance measure. ACLS will continue to care about the number of students who enter employment and remain in the program.

  1. Will incarcerated ABE students who obtain employment through work release before exiting be considered members of the entered employment cohort?

A:For NRS reporting purposes, only students including incarcerated students becoming employed after they exit the program are included in the performance measure.

  1. Currently, we record an employment goal as met if a student has a job one quarter after setting the goal. Therefore,sometimes students will set and meet goals during the time they are enrolled. Will this possibility still exist? That is, if a student is unemployed at entry and obtainsemployment before exit, which cohort (if any) will they be in?

A:If a student is unemployed at entry and gets a job before exiting the program, that outcome will not be counted in the NRS report.

  1. How will the state count data on students who become employed while remaining in the program?

A:The NRS report requires that the student be exited. The state standards do not require a student to be exited.

  1. Do you make any distinction between full and part time and temporary work?

A:The NRS does not make a distinction between types of employment.

  1. What if a student is retired - over the age of 65? Will this matter when determining their entry into a cohort?

A:Age is not a factor used for designation into any cohort. However, retired persons are among those considered to not be in the labor force. For that reason, retired persons of any age are ineligible for classification into the entered employment and retained employment cohorts. Since age is not used as a basis for determining entry into programs designed for the purposes of obtaining a secondary or post-secondary credential, it cannot be used as a basis for determining entry into either of the education cohorts.

  1. What about ESOL learners who have a post secondary credential but have no way to prove it? Wehave folks in our highest level college-prep ESOL class. Are they all considered part of the same cohort?

A:Learners are not required to provide documentation.All students enrolled in a class specifically designed for transitioning to college are included in the post secondary education cohort.

  1. Should Example #3 from the webinar power point be considered a realistic example? To be realistic, wouldn’t this need to account for the 2014 GED changes?

Example 3: A student who enrolls in December 2012, exits in March 2013, takes 3 tests then, takes 2 more in January 2014, would not be included in any cohort.

A:Regardless of the new changes to the GED test series, the student must complete all five tests within the same calendar year.

  1. Will entry into or completion of occupational training certificate program count toward criteria concerned withdetermining entry into or outcomes related to the post-secondary cohort? What if a student passes all GED tests but remains in the program to build skills?

A:Post-secondary education and occupational training are included in the post-secondary education cohort. Inclusion of students into the post-secondary cohort requires their exit froma program regardless of when they may have passed all of the GED tests.

  1. What if a student enters in September 2012, exits December 1, 2012, and takes all five GED tests during 2013?

A:This student will be included in the obtaining a secondary credential cohort if s/he attends at least 12 hours during the fiscal year. In this case, the requirements for cohort entry would be met since the student would be enrolled during the 2012-13 program year and will have completed the five GED tests before the end of the reporting period for that year (December 31, 2013).

  1. Does the post-secondary cohort include students enrolled in ESOL courses specifically designed as preparatory classes for post-secondary programs?

A:To be included in the post-secondary education cohort, a student must have earned a secondary credential while enrolled or hold a secondary credential or they are in a class specifically designed for transitioning to community college.

  1. Will ESE fund transition to college classes not at the community colleges?

A:For information about the types of classes funded at Community Adult Learning Centers, please see the FY 13 Guidelines for Effective ABE Program, pp. 22, and 29-31.

  1. Will all students be included in a cohort? Or will there be some students who are not included in a cohort?

A:No, not all students will be included in a cohort.

  1. How long will students be tracked following exit? If someone enters college 3 years following exit will the program get credit?

A:For NRS reporting purposes, programs are responsible for follow-up regarding the post-secondary education cohort through the end of the fiscal year following the calendar year in which the student exited from the program. Current performance standards capture outcomes achieved within a fiscal year.

  1. What are the implications for multi-site programs that have students who transfer between sites throughout the year and from one year to the next? Consider: a.) When students may have exited from a site but are considered current due to a more recent entry intoanother program. b.) In terms of determining follow-up cohorts, would re-intake into a new site within the same program have any affect?

A:At the state level, a student will be counted only once regardless of the number of programs or sites in which she/he is enrolled.

  1. Will ESE be modifying the regular goal setting system? If so, what might such changes look like?

A:The FY13 Student Goal Form has been modified. We combined the current occupation goals with monthly timeframes (<6 months, 6-12 months, >12 months) into one goal – “Enter Occupational Training.” We deleted “Retain” and “Complete” occupational goals listed below that had specific timeframes:

  • Retain Occupational Training > 12months
  • Complete Occupational Training < 6 months
  • Complete OccupationalTraining 6-12 months.
  • Complete Occupational Training > 12 months
  1. Will there be a newly modified goal form for adults in correctional institutions posted on the website?

A: It is anticipated that revisions to the goal form for correctional institutions will be available soon.

  1. Is there a minimum number ofyearly contact hours a teacher needs in order for that time to be counted as a year of teaching experience?

A:The NRS has not provided specific guidelines. However, only ABE experience should be included.

  1. Under the new NRS system, will programs be allowed to use the same data-capturing modes (data match or follow-up survey) when determining the cohort outcomes used for measuring program performance?

A:The achievement of goals can be captured through data match or survey follow-up which includes phone calls, emails or letters to students. Regardless of how the goal achievement is captured, it is included in the measure of the program’s performance.

  1. Due to the differing procedures related to when follow-up occurs, which system – ESE or NRS – has an earlier release date related to their annual reports?

A:Statewide release of the annual report by ESE occurs earlier. The NRS reports are usually released a year later.

  1. Is thePowerPointfrom the webinar posted?

A:Yes, it may be accessed by clicking on the file with title ‘Webinar: Understanding the NRS Changes’ within the “SMARTT News” section of the following webpage:

1