U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Plan: New York PYs 2016-2017
Combined Plan C

Section: Adult Education and Family Literacy Act Program

Narrative: f.

Published
Located in:

f. Assessing Quality

Describe how the eligible agency will assess the quality of providers of adult education and literacy activities under title II and take actions to improve such quality, including providing the activities described in section 223(a)(1)(B) of WIOA.

Current Narrative:

NYSED has a very comprehensive system of providing high quality professional development programs to improve the instruction provided pursuant to local activities required under section 231(b), including instruction incorporating the essential components of reading instruction as such components relate to adults, instruction related to the specific needs of adult learners, instruction provided by volunteers or by personnel of a State, and dissemination of information about models and promising practices related to such programs.

The Adult Education and Literacy Programs Adult Student Information System and Technical Support (ASISTS) is a NYSED–funded, custom built management information system. This system has been used by funded programs in NYS since PY2007–08. In addition to serving as the database for all program data related to adult literacy, it also provides reports designed to check for data quality and compliance with NRS. To ensure consistent and continual availability of training support, ASISTS produced a number of training videos, focusing on daily required action with program data. Round table discussions are scheduled quarterly to promote data quality.

Section 223 State Leadership funding will continue to support a contract for technical assistance, training, and troubleshooting to improve data quality and monitor program performance in every program. The NYSED NRS accountability specialist funded by this NYSED contract works directly with NYSED staff. Duties include focusing on the largest providers with significant enrollments and programs who need intensive technical assistance to achieve improvement.

Continuing in 2016–17, under the Combined State Plan, three programs from each region (for a total of 21) will be selected for inclusion in the Individual Technical Assistance Program (ITAP). The Big5 city school district programs (NYC, Yonkers, Syracuse, Rochester, and Buffalo) consistently receive support from ITAP, given their large size. All ITAP programs will be desk monitored, accompanied by monthly conference calls and an annual scheduled site visit. All communication and quarterly data analysis will be collected electronically and stored on the NYS accountability website in a secure portal for each program. The portal is accessible by the NRS specialist, the NYSED Regional Liaison, the RAEN Director, and the program manager. Each fiscal year, report card data is reviewed by NYSED, and any program’s performance that places that program in Quartile 1 and missed one or more of the state benchmarks, is placed under Corrective Action. The program will be required to correspond electronically on the accountability website.

Quarterly data reviews will be conducted by the NRS Accountability Specialist and subsequent conference calls will be scheduled with those programs exhibiting poor performance or inconsistent data patterns. The NRS Accountability Specialist will assist in identifying problems and trends on a statewide and regional level. Professional development in the areas of data collection, recording, and reporting continues to be a priority. Added to that is the constant need for NRS training at all levels. The Accountability Specialist will provide a minimum of two training events in each of the seven geographic regions annually. Topics include NRS Foundations, NRS for Teachers, and NYS Report Card Training.

Beginning in 2007, all funded agencies continue to receive NYSED issued annual NRS report cards. This will continue in 2016–17 and each year of the Combined State Plan. The report cards evaluate programs under two distinct methodologies:

1. The first is a standard benchmark measurement. NYS benchmarks for performance in Educational Gain, Post Test Rate, and Follow up Outcomes are set each year by NYSED based on the averages derived from the prior year’s state data. When programs meet all three benchmarks, they are scored as “proficient.”

2. The second method of evaluation is based on a quartile ranking among all programs in the state. In the case of Educational Gain, the program’s scores are weighted based on the educational functioning levels indicating the strongest proponent for gain. The weighting of each EFL is determined by the aggregate data analysis of the prior year’s data. The quartiles are determined using the following rubric:

• 50% of the score is from the weighted educational gain;

• 20% of the score is from the post–test rate;

• 20% of the score is based on the four follow–up measures, i.e., total students with a follow–up goal who achieved the goal upon exiting divided by the total exiters with one of the identified goals; and

• 10% of the score will be based on achieving both a 50% and 70% response rate on the survey of all students who exited the program with a follow–up goal.

Local program data will be entered quarterly into ASISTS according to the following dates:

• Quarter I (7/01–9/30) data entered and reviewed by 11/30;

• Quarter II (10/01–12/31) data entered and reviewed by 2/28;

• Quarter III (1/01–3/31) data entered and reviewed by 4/30; and

• Quarter IV (4/01–6/30) data entered and reviewed by 8/31.

Note that the program has until 10/31 each year to finalize the annual data submitted over the previous four quarters. It is the responsibility of the program to have data entry up to date for each quarter. This timeline was shortened in 2015–16 in anticipation of the final WIOA NRS rule and NRS policy guidelines that may require a significant shortening of the State NRS reporting timeline.

Intensive Technical Assistance Plan (ITAP) continues as a tool and process for asssessing the quality of funded providers in 2016–17. ITAPs identify agencies in need of intense technical assistance. The contracted NRS Accountability Specialist, NYSED ACCES–AEPP regional staff, and RAEN directors work closely with these programs to identify gaps in quality and performance, and implement an effective program improvement plan. Effective use of webinars and conference calls provide a consistent line of communication with these programs as they implement change to improve their performance. As programs improve their performance, they are removed from ITAP status and others are then identified.

Beginning in November 2014 and continuing through 2016–2017, NYSED annually issues a Program Evaluation Report for each funded program. The annual Program Evaluation Report will identify whether the agency met the NYS benchmarks for Educational Gain, Post Test Rate, and Follow up Outcomes. These are aggregate benchmarks set annually by NYSED based upon NRS targets negotiated with the U.S. Education Department and previous year performance data from programs funded by NYSED. Programs that do not meet these NYS annual benchmarks will be required to undertake a formal Corrective Action Planning (CAP) process. The program must submit and receive NYSED approval for a comprehensive CAP plan that identifies specific steps for meeting NYS annual benchmarks by June 30, 2017. For programs identified for corrective action in November of 2015 or 2016, they must meet their annual benchmarks by June 30, 2017 and June 30, 2018, respectively. After the Corrective Action Plan (CAP) process is initiated, NYSED staff will conduct virtual calls/webinars to include the local program staff identified by the adult education director, the NRS Accountability Specialist under contract with NYSED and the program’s RAEN director to track progress toward meeting the NYS annual benchmarks.

For a program placed in CAP in 2015–2016, 2016–17 and each year for the duration of the combined state plan, the program will have two years to raise performance to meet or exceed the New York State annual NRS benchmarks. All communication is documented on the accountability website. Program data and related data reports are stored in secure, user–protected, individual program accounts accessible only to NYSED staff, the RAEN director, and the NRS Accountability Specialist. The website and all secure documents are maintained through the office of the NRS Accountability Specialist, funded under contract with NYSED. As part of the comprehensive CAP, there is a Continuous Evaluation Process (CEP) under which programs and NYSED must identify quarterly progress milestones on key indicators such as educational gain that lead to meeting NYS annual benchmarks within two years. NYSED staff, the RAEN director, and the NRS Accountability Specialist will conduct a quarterly review to determine whether approved quarterly milestones were met for each agency on corrective action. Failure to meet the quarterly progress milestones for any three consecutive quarters could result in the termination of the grant at the end of that program year. This means that a grant could be terminated as early as June 30, 2016.

Both the CAP and CEP processes will be reported and tracked through the accountability website at: http://www.adult–education–accountability.org for each program funded through this RFP. Under the federal NRS for adult literacy programs, NYS, and ultimately local service providers, will be evaluated based on performance indicators. Successful applicants must submit data based on the federal requirements of the NRS. Information on the NRS may be found on the web page www.nrsweb.org.

Tracking data in ASISTS follows a four category process. These categories are sequential and allow for the diversity among New York’s adult literacy programming. The categories are as follows:

Category I – Setup. Programs enter the following categories of data to set up their data account in ASISTS: • Funding sources – Programs can track services funded by multiple sources. • Sites – Providers enter data pertaining to each of their sites and connect it to services so they can run reports separately by site. • Outcomes – In addition to tracking required outcomes, programs may also track outcomes that are specific to their own specialized programming, such as training or career pathways. • Personnel – Providers add teacher information including name, experience, certification information, and longevity. • Classes/Services – Providers create their roster of classes adding the schedule, teacher, and type/level of classes.

Category II – Registration. All state funded programs must use a standard registration form, the Individual Student Record Form (ISRF). • Student information – This includes name; DOB; contact information; and demographic information such as gender, ethnicity, employment status, educational background, and public assistance status. ASISTS also allows providers to note whether a person is a parent or disabled. • Initial Assessment – All providers use one of two standardized assessments (the Test of Adult Basic Education (TABE) for Adult Basic Education (ABE) students or BEST Plus for English Language Learner (ELL) students) with registrants at intake or within 12 hours of instruction. This assessment is used to place students in literacy classes. • Enrollment – Once registration is complete, registrants are assigned to classes based on their assessments and availability.

Category III – Services. • Hours – Adult Education providers use ASISTS to track attendance hours on a daily, weekly, or monthly basis. • Test scores – Post–tests are given by providers at regular intervals to demonstrate learning gains. ASISTS calculates the educational gain based on their standardized assessment scores.

Category IV – Outcomes. Adult Education providers track student outcomes regularly, both during and after their participation in a program. • Educational Gain – NRS guidelines define a student as demonstrating educational gain when his or her test score places him or her in an NRS level (one of six predefined educational functioning levels) that is higher than the one determined by that student’s initial or pre–test. • Primary follow–up outcomes – Providers focus on the following three key NRS outcomes: o Employment and Retention of Employment; o Attaining the NYS High School Equivalency Diploma; and o Entering postsecondary education or training. • Other Outcomes – ASISTS also facilitates the tracking of a wide variety of outcomes that are customized per program. Review – Reports and Downloads.

ASISTS includes the following reports to review the data and use it for reporting and management purposes;

• Funding reports – Reports that present data critical to reporting following state funding guidelines. The most critical report is the Program Evaluation report indicating on one page the most critical performance statistics NYSED uses to evaluate programs. Employment Preparation Education (EPE) claim forms are also generated from ASISTS.

• Data management reports primarily meant for tracking and internal program management purposes.

• Data checks that are specifically designed to help providers detect and correct issues relating to data quality and completeness. All of these resources will serve as the basis of core program alignment to support WIOA success. Dissemination of high quality programs and practices are comprehensively shared throughout the state through seven Regional Adult Education Network (RAEN) centers through monthly program managers meetings by each RAEN, workshop presentations at state conference (NYACCE, statewide incarcerated conference, NYC Adult Basic Education conference).