Located in:
- Program-specific Requirements for Adult Education and Family Literacy Act Programs
The Unified or Combined State Plan must include a description of the following as it pertains to adult education and literacy programs and activities under title II of WIOA, the Adult Education and Family Literacy Act (AEFLA).
f. Assessing Quality
Describe how the eligible agency will assess the quality of providers of adult education and literacy activities under title II. Describe how the eligible agency will take actions to improve such quality, including providing the activities described in section 223(a)(1)(B) of WIOA. Describe how the eligible agency will use evaluations, data analysis, program monitoring, or other methods to improve the quality of adult education activities in the State.
Current Narrative:
The Adult Education and Family Literacy Program’s Adult Student Information System and Technical Support (ASISTS) is an ACCES-AEPP-funded, custom built, online-management information system. This system has been used by funded programs in NYS since PY 2007. In addition to serving as the database for all program data related to adult literacy, it also provides reports designed to check for data quality, performance measure, and compliance with NRS. To ensure consistent and continual availability of training support, ASISTS provides a number of training videos, focusing on daily required activities for program data recording and reporting.
Leadership will continue to support a contract for technical assistance, training, and troubleshooting to improve data quality and monitor program performance statewide. The NRS accountability specialist funded by this NYSED contract works directly with ACCES-AEPP staff. Duties include focusing on the largest providers with significant enrollments and programs needing intensive technical assistance to achieve improvement. Annually, under the Combined State Plan programs from each RAEN region are selected for inclusion in the Individual Technical Assistance Program (ITAP). The Big 5 city school district programs (New York City, Yonkers, Syracuse, Rochester, and Buffalo) are consistently receiving support from ITAP, given their large size and possible impact on statewide performance measures. All ITAP programs are desk monitored, accompanied by monthly conference calls and an annual scheduled site visit. All communication and quarterly data analysis is collected electronically and stored on the NYS accountability website in a secure, password protected portal for each program, website: https://adult-education-accountability.org. The portal is accessible by the NRS Office, the AEPP NYSED Regional staff, the respective RAEN Director, and the program manager.
Each fiscal year, report card data is reviewed by NYSED. In addition to measuring progress toward performance benchmarks, the NYS report cards quartile rank all funded programs. When a program’s performance places that program in Quartile 1 and the program has missed one or more of the state benchmarks, the program is placed under Corrective Action. The program is required to submit a program improvement plan and must correspond electronically on the accountability website. Quarterly data reviews are conducted by the NRS Accountability Office and subsequent conference calls and/or web meetings are scheduled with those programs exhibiting poor performance or inconsistent data patterns. The NRS Accountability Office assists in identifying problems and trends on a statewide and regional level.
Professional development in the areas of data collection, recording, and reporting continues to be a priority. Added to that is the continual need for NRS training at all levels. The Accountability Specialist will provide a minimum of two training events in each of the seven geographic regions annually. Topics include NRS Foundations, NRS for Teachers, and NYS Report Card Training.
With the exception of the three fiscal years impacted by the pandemic (FY2020, FY2021, FY2022), all funded agencies receive NYS-issued report cards. The report cards evaluate programs under two distinct methodologies. The first is a standard benchmark measurement. NYS benchmarks for performance in Measurable Skill Gain (MSG), Post Test Rate, and Follow up Outcomes are set each year by AEPP Office based on the averages derived from the prior year’s state data. When programs meet all three benchmarks, they are scored as “proficient.”
The second method of evaluation is based on a quartile ranking among all programs in the state. In the case of MSG, the program’s scores are weighted based on the NRS Educational Functioning Levels (EFLs) indicating the strongest proponent for gain as evidence in the data. The weighting of each EFL is determined by the aggregate data analysis of the prior year’s data.
The quartiles are determined using the following rubric:
• 40% of the score is from the weighted MSG;
• 20% of the score is from the post-test rate;
• 20% of the score is based on the four follow-up measures; and
• 10% of the score will be based on HSE achievement.
Local program data is entered monthly into ASISTS. Data is reviewed on a quarterly basis:
• Quarter I (7/01-9/30) reviewed by 10/31;
• Quarter II (10/01-12/31) reviewed by 1/31;
Note that the program has until October 31st each year to finalize the annual data submitted over the previous four quarters. It is the responsibility of the program to have data entry up to date for each quarter. ITAP will continue to support those programs presenting with the need for intense technical assistance. ITAPs identify agencies in need of intense technical assistance. The contracted Accountability Office, AEPP Regional staff, and RAEN directors work closely with these programs to identify gaps in quality and performance, and implement an effective program improvement plan. Effective use of webinars and conference calls provide a consistent line of communication with these programs as they implement change to improve their performance. As programs improve their performance, they are removed from ITAP status and others are then identified.
NYSED issues a Program Evaluation Report for each funded program accessible in the management information system on a daily basis as data is updated each evening. The annual Program Evaluation Report will identify whether the agency met the NYS benchmarks for MSG, Post Test Rate, Follow up Outcomes, and HSE achievement. Program staff are able to monitor their progress and attend to any deficiencies or alternate trends. These are aggregate benchmarks set annually by NYSED based upon NRS targets negotiated with the U.S. Education Department and previous year performance data from programs funded by ACCES-AEPP. Programs that do not meet these NYS annual benchmarks will be required to undertake a formal Corrective Action Planning process. The program must submit and receive NYSED approval for a comprehensive Corrective Action Plan (CAP) that identifies specific steps for meeting NYS annual benchmarks by June 30 of the current year.
- Both the CAP and Continuous Evaluation Process processes will be reported and tracked through the accountability website at http://www.adult-education-accountability.org for each program funded through this RFP.
- Under the federal NRS for adult literacy programs, NYS, and ultimately local service providers, will be evaluated based on performance indicators. Successful applicants must submit data based on the federal requirements of the NRS. Information on the NRS may be found on the web page www.nrsweb.org.
- Tracking data in ASISTS follows a four-category process. These categories are sequential and allow for the diversity among NYS’s adult literacy programming.
The categories are as follows:
Category I – Setup
Programs enter the following categories of data to set up their data account in ASISTS:
• Funding sources - Programs can track services funded by multiple sources.
• Sites - Providers enter data pertaining to each of their sites and connect it to services so they can run reports separately by site.
• Outcomes - In addition to tracking required outcomes, programs may also track outcomes that are specific to their own specialized programming, such as training or career pathways.
• Personnel - Providers add teacher information including name, experience, certification information, and longevity.
• Classes/Services - Providers create their roster of classes adding the schedule, teacher, and type/level of classes.
Category II – Registration
- All state funded programs must use a standard registration form, the Individual Student Record Form (ISRF).
- Student information - This includes name; Date of Birth (DOB); contact information; and demographic information such as gender, ethnicity, employment status, educational background, and public assistance status. ASISTS also allows providers to note whether a person is a parent or disabled. Social Security Numbers (SSNs) are also recorded when made available by the student.
- Initial Assessment - All providers use one of two standardized assessments (the (TABE) for Adult Basic Education (ABE) students or BEST Plus for English Language Learner (ELL) students with registrants at intake or within 12 hours of contact with the student). This assessment is used to place students in literacy classes and diagnose learning needs.
- Enrollment - Once registration is complete, registrants are assigned to classes based on their assessments and availability.
Category III – Services
- Contact Hours - Adult Education providers use ASISTS to track attendance hours on a daily, weekly, or monthly basis.
- Test scores - Post-tests are given by providers at regular intervals based on NYS Assessment Policy submitted to OCTAE to demonstrate learning gains. ASISTS calculates the MSG based on their standardized assessment scores.
Category IV – Outcomes
- Adult Education providers track student outcomes regularly, both during and after their participation in a program.
- MSG - NRS guidelines define a student as demonstrating MSG when their test score places him or her in an NRS level (one of six predefined educational functioning levels) that is higher than the one determined by that student's initial or pre-test.
- Primary follow-up outcomes - Providers focus on the following key NRS outcomes: o Employment;
- Median wage earning;
- Attaining the NYS HSE diploma;
- Attaining an industry-recognized credential; and
- Entering postsecondary education or training.
- Other Outcomes - ASISTS also facilitates the tracking of a wide variety of outcomes that are customized per program.
Review – Reports and Downloads
ASISTS includes the following reports to review the data and use it for reporting and management purposes.
- Funding reports - Reports that present data critical to reporting following state funding guidelines. The most critical report is the Program Evaluation report indicating on one page the most critical performance statistics NYSED uses to evaluate programs. The state-funded Employment Preparation Education (EPE) state aid claim forms are also generated from ASISTS.
- Data management reports primarily meant for tracking and internal program management purposes.
- Data checks that are specifically designed to help providers detect and correct issues relating to data quality and completeness.
All of these resources will serve as the basis of core program alignment to support WIOA success. NYSED‘s Adult Career and Continuing Education Services – Adult Education Policies and Programs (NYSED/ACCES-AEPP) will continue to refine measurable skill gain targets annually, including definitions of measures and approved assessments for individuals with barriers to employment “who are English language learners, individuals who have low levels of literacy, and individuals facing substantial cultural barriers.”
Seven Regional Adult Education Network (RAEN) centers and the National Reporting System (NRS) accountability specialist, funded with WIOA leadership funds, are preparing to mount professional development and training for agencies that are WIOA-funded. The federal office set the reporting deadline annually for NRS data to August 15. ACCES- AEPP has taken steps to comply with required deadlines. All training and performance reporting policies will be closely coordinated with partner programs to ensure a smooth transition.