Located in:
- III. Operational Planning ElementsThe Unified or Combined State Plan must include an Operational Planning Elements section that support the State’s strategy and the system-wide vision described in Section II.(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and PoliciesThe Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in Section II Strategic Elements . This includes—
- b. State Operating Systems and Policies
III. b. 4. A. Assessment of Core Programs
Current Narrative:
Core Program and One-Stop Partner Assessments
Under WIA, NYSDOL tracked several Customer Service Indicators (CSIs). CSIs are designed to assist LWDAs in meeting their WIOA Common Measure goals and to move the Career Center System in NYS toward specific goals that NYSDOL deems important to the good of the state. First instituted during PY07 under WIA, NYSDOL has used many different CSIs that have evolved over time. They have included measures to reinforce data entry policy for staff, categorize participants based on their job readiness and prescribe additional services, leverage new technology, and effectively serve business customers. Over the past year, we have focused our efforts on ensuring we comply with the mandated WIOA Common Measure reporting requirements of WIOA. NYSDOL now expects to return to developing additional CSIs to assist local areas in achieving further performance goals.
As discussed previously, NYSDOL, in partnership with NYS OMH, developed and implemented NYESS. NYESS is made up of various computer applications and data sets. Employment related data collection is accomplished by all NYESS partners using the existing NYSDOL case management system, OSOS. Legacy data sets from the partners and current data sets, which include but are not limited to, OSOS; vocational rehabilitation agencies; DOH; Social Security Administration (SSA); and others are pulled together in a data warehouse. A web-based reporting portal designed and maintained by OMH will provide cross-agency report card like information to the general public (aggregate data) and to the individual agencies and their contracted partner staff.
The Adult Education and Literacy Programs Adult Student Information System and Technical Support (ASISTS) is an ACCES-AEPP-funded, custom built management information system. This system has been used by funded programs in NYS since PY2007-08. In addition to serving as the database for all program data related to adult literacy, it also provides reports designed to check for data quality and compliance with NRS. To ensure consistent and continual availability of training support, ASISTS produced a number of training videos, focusing on daily required action with program data. Round table discussions are scheduled quarterly to promote data quality.
Leadership will continue to support a contract for technical assistance, training, and troubleshooting to improve data quality and monitor program performance in every program. The NRS accountability specialist funded by this NYSED contract works directly with ACCES-AEPP staff. Duties include focusing on the largest providers with significant enrollments and programs who need intensive technical assistance to achieve improvement. In 2016-17, under the Combined State Plan, three programs from each region (for a total of 21) were selected for inclusion in the Individual Technical Assistance Program (ITAP). The Big 5 city school district programs (New York City, Yonkers, Syracuse, Rochester, and Buffalo) consistently receive support from ITAP, given their large size. All ITAP programs are desk monitored, accompanied by monthly conference calls and an annual scheduled site visit. All communication and quarterly data analysis is collected electronically and stored on the NYS accountability website in a secure portal for each program. The portal is accessible by the NRS specialist, the NYSED Regional Liaison, the RAEN Director, and the program manager.
Each fiscal year, report card data is reviewed by NYSED. The NYS report cards quartile rank all funded programs. When a program’s performance places that program in Quartile 1 and the program has missed one or more of the state benchmarks, the program is placed under Corrective Action. The program is required to submit a program improvement plan and must correspond electronically on the accountability website. Quarterly data reviews are conducted by the NRS Accountability Specialist and subsequent conference calls are scheduled with those programs exhibiting poor performance or inconsistent data patterns. The NRS Accountability Specialist assists in identifying problems and trends on a statewide and regional level.
Professional development in the areas of data collection, recording, and reporting continues to be a priority. Added to that is the constant need for NRS training at all levels. The Accountability Specialist will provide a minimum of two training events in each of the seven geographic regions annually. Topics include NRS Foundations, NRS for Teachers, and NYS Report Card Training.
Beginning in 2007, all funded agencies have continued to receive NYS-issued report cards. This will continue in 2016-19. The report cards evaluate programs under two distinct methodologies. The first is a standard benchmark measurement. NYS benchmarks for performance in Measurable Skill Gain (MSG), Post Test Rate, and Follow up Outcomes are set each year by NYSED based on the averages derived from the prior year’s state data. When programs meet all three benchmarks, they are scored as “proficient.”
The second method of evaluation is based on a quartile ranking among all programs in the state. In the case of Measurable Skill Gain, the program’s scores are weighted based on the NRS Educational Functioning Levels (EFLs) indicating the strongest proponent for gain. The weighting of each EFL is determined by the aggregate data analysis of the prior year’s data. The quartiles are determined using the following rubric:
- 50% of the score is from the weighted Measurable Skill Gain;
- 20% of the score is from the post-test rate;
- 20% of the score is based on the four follow-up measures; and
- 10% of the score will be based on High School Equivalency (HSE) achievement.
Local program data will be entered quarterly into ASISTS, according to the following dates:
- Quarter I (7/01-9/30) data entered and reviewed by 10/31;
- Quarter II (10/01-12/31) data entered and reviewed by 1/31;
- Quarter III (1/01-3/31) data entered and reviewed by 4/30; and
- Quarter IV (4/01-6/30) data entered and reviewed by 7/31. Note that the program has until 10/31 each year to finalize the annual data submitted over the previous four quarters. It is the responsibility of the program to have data entry up to date for each quarter. ITAP continues in PY 2016-19. ITAPs identify agencies in need of intense technical assistance. The contracted Accountability Specialist, NYS regional staff, and RAEN directors work closely with these programs to identify gaps in quality and performance, and implement an effective program improvement plan. Effective use of webinars and conference calls provide a consistent line of communication with these programs as they implement change to improve their performance. As programs improve their performance, they are removed from ITAP status and others are then identified.
Beginning in November 2014 and continuing through PY 2016-2019, NYSED issues a Program Evaluation Report for each funded program. The annual Program Evaluation Report will identify whether the agency met the NYS benchmarks for Measurable Skill Gain, Post Test Rate, Follow up Outcomes, and HSE achievement. These are aggregate benchmarks set annually by NYSED based upon NRS targets negotiated with the U.S. Education Department and previous year performance data from programs funded by ACCES-AEPP. Programs that do not meet these NYS annual benchmarks will be required to undertake a formal Corrective Action Planning process. The program must submit and receive NYSED approval for a comprehensive Corrective Action Plan (CAP) that identifies specific steps for meeting NYS annual benchmarks by June 30 of the current year. For programs identified for corrective action in November of 2017 or 2018, they must meet their annual benchmarks by June 30, 2018 and June 30, 2019, respectively.
Both the CAP and Continuous Evaluation Process processes will be reported and tracked through the accountability website at: http://www.adult-education-accountability.org for each program funded through this RFP.
Under the federal NRS for adult literacy programs, NYS, and ultimately local service providers, will be evaluated based on performance indicators. Successful applicants must submit data based on the federal requirements of the NRS. Information on the NRS may be found on the web page www.nrsweb.org.
Tracking data in ASISTS follows a four-category process. These categories are sequential and allow for the diversity among NYS’s adult literacy programming. The categories are as follows:
Category I - Setup.
Programs enter the following categories of data to set up their data account in ASISTS:
- Funding sources - Programs can track services funded by multiple sources.
- Sites - Providers enter data pertaining to each of their sites and connect it to services so they can run reports separately by site.
- Outcomes - In addition to tracking required outcomes, programs may also track outcomes that are specific to their own specialized programming, such as training or career pathways.
- Personnel - Providers add teacher information including name, experience, certification information, and longevity.
- Classes/Services - Providers create their roster of classes adding the schedule, teacher, and type/level of classes.
Category II - Registration.
All state funded programs must use a standard registration form, the Individual Student Record Form (ISRF).
- Student information - This includes name; DOB; contact information; and demographic information such as gender, ethnicity, employment status, educational background, and public assistance status. ASISTS also allows providers to note whether a person is a parent or disabled. Social Security Numbers are also recorded when made available by the student.
- Initial Assessment - All providers use one of two standardized assessments (the Test of Adult Basic Education (TABE) for Adult Basic Education (ABE) students or BEST Plus for English Language Learner (ELL) students with registrants at intake or within 12 hours of contact with the student. This assessment is used to place students in literacy classes and diagnose learning needs.
- Enrollment - Once registration is complete, registrants are assigned to classes based on their assessments and availability.
Category III - Services.
- Contact Hours - Adult Education providers use ASISTS to track attendance hours on a daily, weekly, or monthly basis.
- Test scores - Post-tests are given by providers at regular intervals based on NYS Assessment Policy submitted to the U.S Education Department’s Office of Career, Technical and Adult Education to demonstrate learning gains. ASISTS calculates the Measurable Skill Gain based on their standardized assessment scores.
Category IV - Outcomes.
Adult Education providers track student outcomes regularly, both during and after their participation in a program.
- Measurable Skill Gain - NRS guidelines define a student as demonstrating Measurable Skill Gain when his or her test score places him or her in an NRS level (one of six predefined educational functioning levels) that is higher than the one determined by that student's initial or pre-test.
- Primary follow-up outcomes - Providers focus on the following key NRS outcomes:
o Employment;
o Median wage earning;
o Attaining the NYS HSE diploma;
o Attaining an industry recognized credential; and
o Entering postsecondary education or training.
- Other Outcomes - ASISTS also facilitates the tracking of a wide variety of outcomes that are customized per program.
Review - Reports and Downloads.
ASISTS includes the following reports to review the data and use it for reporting and management purposes.
- Funding reports - Reports that present data critical to reporting following state funding guidelines. The most critical report is the Program Evaluation report indicating on one page the most critical performance statistics NYSED uses to evaluate programs. The state-funded Employment Preparation Education (EPE) state aid claim forms are also generated from ASISTS.
- Data management reports primarily meant for tracking and internal program management purposes.
- Data checks that are specifically designed to help providers detect and correct issues relating to data quality and completeness.
All of these resources will serve as the basis of core program alignment to support WIOA success.