Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. A. Assessment of Core and One-Stop Program Partner Programs.
Describe how the core programs will be assessed each year based on State performance accountability measures described in section 116(b) of WIOA. Describe how other one-stop delivery system partner program services and Combined State Plan partner programs included in the plan will be assessed each year. This State assessment must include the quality, effectiveness, and improvement of programs broken down by local area or provider. Such state assessments should take into account local and regional planning goals.
Current Narrative:
Assessment of Core Programs
Over the past few years, NYSDOL has focused its efforts on ensuring compliance with the mandated WIOA Primary Indicators of Performance reporting requirements. These efforts have included measures to reinforce data entry policy for staff, categorize participants based on their job readiness and prescribed additional services, leverage new technology, and effectively serve business customers. NYSDOL has provided targeted training and technical assistance with regards to the WIOA Primary Indicators of Performance through regular communication, sharing of best practices, individualized assistance for areas that are struggling, and required performance improvement plans (PIPs). With the release of the quarterly performance report cards, NYSDOL also provides lists of customers who were included in the Credential and Measurable Skill Gains (MSG) indicators for LWDAs to review, giving them the opportunity to correct any data entry errors and improve overall performance. Beginning in Quarter 4 of PY 2021, NYSDOL also began providing the local areas with lists of customers in the Adult Priority of Service metric to ensure data entry is being captured appropriately.
Performance goals for LWDAs are set through a negotiation process and assessment scores are used to measure performance outcomes. Proposed goals are determined prior to the negotiation process and act as a starting point for the negotiations. Negotiated goals are agreed upon by the parties involved in the negotiation process. Adjusted goals are calculated after the end of the PY by recalculating the negotiated goals using the statistical adjustment model to account for circumstances that take place during the PY that could not be anticipated during the negotiation process.
NYSDOL provides quarterly performance reports. These reports include negotiated goals as well as cumulative information for the PY on participants, exiters, Q2 and Q4 employment rates, median earnings, credential attainment, and measurable skills gains for each LWDA and NYS. NYSDOL also provides ad-hoc reports for self-evaluation at the request of LWDAs. These reports allow the LWDA to keep track of its progress and highlight any areas that may need attention before they become a problem. These reports also allow NYSDOL to see which areas are successful and which areas may need assistance. If an area looks like it may fail to reach its goals, NYSDOL will reach out to determine if technical assistance or training could help. If an area is successful, NYSDOL will reach out to document best practices. These best practices are shared through training and regular meetings with LWDB Directors.
Program performance at the State level is assessed by using individual indicator scores, program scores, and State indicator scores. State performance is considered unsatisfactory when any of the individual indicator scores fall below 50% for the PY, any program score falls below 90% for the PY, or any State indicator score falls below 90% for the PY.
Program performance at the local level is assessed using only the individual indicator scores and is limited to performance of the Title I programs. Actual performance outcomes will be compared to the LWDB’s adjusted goals after the close of the PY to determine if local performance goals were achieved. NYSDOL has set an unsatisfactory performance threshold at an outcome-to-goal ratio of less than 80% per indicator for a PY. The LWDB must develop a PIP for unsatisfactory performance of an indicator in a PY. NYSDOL will also provide technical assistance, which may include assistance in the development of the PIP and/or other actions designed to assist the local area in improving performance. If a local area receives unsatisfactory performance for the same indicator for three (3) consecutive PYs, the Governor will take corrective action, which includes development of a reorganization plan.
As discussed previously, NYSDOL, in partnership with OMH, developed and implemented NYS OMH/NYESS. NYS OMH/NYESS is made up of various computer applications and data sets. Employment related data collection is accomplished by all NYS OMH/NYESS partners using the existing NYSDOL case management system, OSOS. Legacy data sets from the partners and current data sets, which include but are not limited to, OSOS, VR agencies, DOH, SSA, and others are pulled together in a data warehouse. NYSDOL, in collaboration with NYS OMH/NYESS, continues to improve this partnership and employment related data collection through continuous communication and system enhancements.
The Adult Education and Family Literacy Program’s ASISTS is an ACCES-AEPP-funded, custom built, online-management information system. This system has been used by funded programs in NYS since PY 2007. In addition to serving as the database for all program data related to adult literacy, it also provides reports designed to check for data quality, performance measure, and compliance with NRS. To ensure consistent and continual availability of training support, ASISTS provides a number of trainings, focusing on daily required activities for program data recording and reporting.
Leadership will continue to support a contract for technical assistance, training, and troubleshooting to improve data quality and monitor program performance Statewide. The NRS accountability specialist funded by this NYSED contract works directly with ACCES-AEPP staff. Duties include focusing on the largest providers with significant enrollments and programs needing intensive technical assistance to achieve improvement. Annually, under the Combined State Plan, programs from each RAEN region are selected for inclusion in the Individual Technical Assistance Program (ITAP). The Big Five city school district programs (NYC, Yonkers, Syracuse, Rochester, and Buffalo) are consistently receiving support from ITAP, given their large size and possible impact on Statewide performance measures. All ITAP programs are desk monitored, accompanied by monthly conference calls and an annual scheduled site visit. All communication and quarterly data analysis is collected electronically and stored on the NYS accountability website in a secure, password protected portal for each program. The portal is accessible by the NRS Office, the NYSED/ACCES-AEPP Regional staff, the respective RAEN Director, and the program manager.
At the conclusion of each fiscal year, report cards for each funded project are generated from the data in the MIS system and are reviewed and published by NYSED. In addition to measuring progress toward performance benchmarks, the NYS report cards quartile rank all funded programs. Through this effort, programs are able to see their performance measures against federally negotiated benchmarks but are also able to scale their performance against all other similarly funded projects throughout the State. When a program’s performance places that program in Quartile 1 and the program has missed one (1) or more of the State benchmarks, the program is placed under Corrective Action. The program is required to submit a program improvement plan and must correspond electronically on the accountability website. Quarterly data reviews are conducted by the NRS Accountability Office and subsequent conference calls and/or web meetings are scheduled with those programs exhibiting poor performance or inconsistent data patterns. The NRS Accountability Office assists in identifying problems and trends on a Statewide and regional level.
Professional development in the areas of data collection, recording, and reporting continues to be a priority. Added to that is the continual need for NRS training at all levels. The Accountability Specialist will provide a minimum of two (2) training events in each of the seven (7) geographic regions annually. Topics include NRS Foundations, NRS for Teachers, and NYS Report Card Training.
The report cards evaluate programs under two (2) distinct methodologies. The first is a standard benchmark measurement. NYS benchmarks for performance in Measurable Skill Gain (MSG), Post Test Rate, Enrollment and Follow up Outcomes are set each year by the NYSED/ACCES-AEPP Office based on the averages derived from the prior year’s State data. When programs meet all three (3) benchmarks, they are scored as “proficient.”
The second method of evaluation is based on a quartile ranking among all programs in the State. In the case of MSG, the program’s scores are weighted based on the NRS Educational Functioning Levels (EFLs) indicating the strongest proponent for gain as evidence in the data. The weighting of each EFL is determined by the aggregate data analysis of the prior year’s data. This methodology assures that programs serving those students populations that are the hardest to serve are not penalized for lower performance levels.
The quartiles are determined using the following rubric:
- 50% of the score is from the weighted MSG;
- 20% of the score is from the post-test rate;
- 20% of the score is based on the four (4) follow-up measures; and
- 10% of the score will be based on High School Equivalency (HSE) achievement.
Local program data is entered monthly into ASISTS. Data is reviewed on a quarterly basis:
- Quarter I (7/01-9/30) reviewed by 10/31;
- Quarter II (10/01-12/31) reviewed by 1/31;
- Quarter III (1/01-3/31) reviewed by 4/30; and
- Quarter IV (4/01-6/30) reviewed by 7/31.
Note that the NYSED/ACCES-AEPP program office has until October 1st each year to finalize the annual data submitted over the previous four (4) quarters. It is the responsibility of the program to have data entry up to date for each quarter. ITAP will continue to support those programs presenting with the need for intense technical assistance. The contracted Accountability Office, NYSED/ACCES-AEPP Regional staff, and RAEN directors work closely with these programs to identify gaps in quality and performance and implement an effective program improvement plan. Effective use of webinars and conference calls provide a consistent line of communication with these programs as they implement change to improve their performance. As programs improve their performance, they are removed from ITAP status.
NYSED issues a Program Evaluation Report for each funded program accessible in the management information system on a daily basis as data is updated each evening. The Program Evaluation Report will identify whether the agency is progressing toward and/or meeting the NYS benchmarks for MSG, Post Test Rate, Follow up Outcomes, and survey success. Program staff, NYSED/ACCES-AEPP Regional Staff, RAEN Directors, and the Accountability office are able to monitor program progress and attend to any deficiencies or alternate trends. The benchmarks set annually by NYSED based upon NRS targets negotiated with USED and previous year performance data from programs funded by NYSED/ACCES-AEPP. Programs that do not meet these NYS annual benchmarks will be required to undertake a formal Corrective Action Planning process. The program must submit and receive NYSED approval for a comprehensive Corrective Action Plan (CAP) that identifies specific steps for meeting NYS annual benchmarks by June 30th of the current year.
Both the CAP and Continuous Evaluation Process processes will be reported and tracked through the accountability website at http://www.adult-education-accountability.org for each program funded through this RFP.
Under the federal NRS for adult literacy programs, NYS, and ultimately local service providers, will be evaluated based on performance indicators. Successful applicants must submit data based on the federal requirements of the NRS that demonstrate their effectiveness. Additional information on the NRS may be found on the web page www.nrsweb.org.
Tracking data in ASISTS follows a four-category process. These categories are sequential and allow for the diversity among NYS’s adult literacy programming. The categories are as follows:
Category I – Setup
Programs enter the following categories of data to set up their data account in ASISTS:
- Funding sources – Programs can track services funded by multiple sources.
- Sites – Providers enter data pertaining to each of their sites and connect it to services so they can run reports separately by site.
- Outcomes – In addition to tracking required outcomes, programs may also track outcomes that are specific to their own specialized programming, such as credentials achieved, training or career pathways.
- Personnel – Providers add teacher information including name, experience, certification information, and longevity.
- Classes/Services – Providers create their roster of classes adding the schedule, teacher, and type/level of classes.
Category II – Registration
All State-funded programs must use a standard registration form, the Individual Student Record Form (ISRF). Through meaningful conversations, the completion of the ISRF form aims to help programs capture the student’s Barriers to Employment. This information is significant in referring students for supportive services with other core partners, providing students with the necessary academic supports to assure their success in the program and, ultimately, impact the program’s academic retention and employment efforts.
- Student information – This includes name; Date of Birth (DOB); contact information; and demographic information such as gender, ethnicity, employment status, educational background, barriers to employment, and public assistance status. ASISTS also allows providers to note whether a person is a parent or disabled. Social Security Numbers (SSNs) are also recorded when made available by the student.
- Co-enrollment – Programs, during the intake process, are required to collect information from students regarding their co-enrollment in other WIOA programs to include Titles I, III, and IV. These data are reported in ASISTS and updated annually.
- Initial Assessment – All providers use one of two standardized assessments (the TABE 11/21 for Adult Basic Education (ABE) students or BEST Plus for English Language Learner (ELL) students with registrants at intake or within 12 hours of contact with the student). This assessment is used to place students in literacy classes and diagnose learning needs.
- Enrollment – Once registration is complete, registrants are assigned to classes based on their assessments and availability.
Category III – Services
- Contact Hours – Adult Education providers use ASISTS to track attendance hours on a daily, weekly, or monthly basis. Minimum requirement is monthly.
- Test scores – Post-tests are given by providers at regular intervals based on NYS Assessment Policy submitted to OCTAE to demonstrate learning gains. ASISTS calculates the MSG based on their standardized assessment scores.
Category IV – Outcomes
Adult Education providers track student outcomes regularly, both during and after their participation in a program.
- MSG – NRS guidelines define a student as demonstrating MSG when their test score places him or her in an NRS level (one of six predefined educational functioning levels) that is higher than the one determined by that student's initial or pre-test. MSG may also be earned when a student achieves the HSE diploma, or when a student achieves a credential pre-approved by NYSED. MSG is realized when a student completes any one of these areas, multiple achievements are not factored.
- Primary follow-up outcomes – Providers focus on the following key NRS outcomes:
- Employment, second and fourth quarter after exit;
- Median wage earning;
- Attaining the NYS HSE diploma;
- Attaining an industry-recognized credential; and
- Entering postsecondary education or training.
- Other Outcomes – ASISTS also facilitates the tracking of a wide variety of outcomes that are customized per program.
Review – Reports and Downloads
ASISTS includes the following reports to review the data and use it for reporting and management purposes.
- Funding reports – Reports that present data critical to reporting following State funding guidelines. The most critical report is the Program Evaluation report indicating on one page the most critical performance statistics NYSED uses to evaluate programs. The State-funded Employment Preparation Education (EPE) state aid claim forms are also generated from ASISTS.
- Data management reports primarily meant for tracking and internal program management purposes.
- Data checks that are specifically designed to help providers detect and correct issues relating to data quality and completeness.
All of these resources will serve as the basis of core program alignment to support WIOA success.
NYSED/ACCES-AEPP will continue to refine MSG targets annually, including definitions of measures and approved assessments for individuals with barriers to employment “who are English language learners, individuals who have low levels of literacy, and individuals facing substantial cultural barriers.”
Seven (7) RAEN centers and the National Reporting System (NRS) accountability specialist, funded with WIOA leadership funds, are preparing to mount professional development and training for agencies that will be WIOA-funded. The federal office set the reporting deadline annually for NRS data to August 15. NYSED/ACCES-AEPP has taken steps to comply with required deadlines. All training and performance reporting policies will be closely coordinated with partner programs to ensure a smooth transition.
NYSED/ACCES-VR and OCFS/NYSCB are both assessed annually by the six (6) common performance measures under WIOA: Employment Rate 2nd Quarter after Exit, Employment Rate 4th Quarter after Exit, Median Earnings in the 2nd Quarter after Exit, Credential Attainment Rate, MSG, and Effectiveness in Serving Employers. The first five (5) performance measures are reported jointly by OCFS/NYSCB and NYSED/ACCES-VR. The sixth performance measure, Effectiveness in Serving Employers, is reported at a Statewide level for all partners.
Assessment of One-Stop Partner Programs
Over the past few years, NYSDOL has focused its efforts on ensuring compliance with the mandated WIOA Primary Indicators of Performance reporting requirements. These efforts have included measures to reinforce data entry policy for staff, categorize participants based on their job readiness and prescribed additional services, leverage new technology, and effectively serve business customers. NYSDOL has provided targeted training and technical assistance with regards to the WIOA Primary Indicators of Performance through regular communication, sharing of best practices, individualized assistance for areas that are struggling, and required PIPs. With the release of the quarterly performance report cards, NYSDOL also provides lists of customers who were included in the Credential and MSG indicators for LWDAs to review, giving them the opportunity to correct any data entry errors and improve overall performance. Beginning in Quarter 4 of PY 2021, NYSDOL also began providing the local areas with lists of customers in the Adult Priority of Service metric to ensure data entry is being captured appropriately.
Program performance at the local level is assessed using only the individual indicator scores and is limited to performance of the Title I programs. Actual performance outcomes will be compared to the LWDB’s adjusted goals after the close of the PY to determine if local performance goals were achieved. NYSDOL has set an unsatisfactory performance threshold at an outcome-to-goal ratio of less than 80% per indicator for a PY. The LWDB must develop a PIP for unsatisfactory performance of an indicator in a PY. NYSDOL will also provide technical assistance, which may include assistance in the development of the PIP and/or other actions designed to assist the local area in improving performance. If a local area receives unsatisfactory performance for the same indicator for three (3) consecutive PYs, the Governor will take corrective action, which includes development of a reorganization plan.
Similarly, the TAA program uses data provided through Participant Individual Record Layout (PIRL) submission and analyzed through the TAA Data Integrity (TAADI) report card process to measure performance. The TAADI framework identifies a series of measures that target both potentially inaccurate data, as well as accurately reported data that reflect substandard program administration. TAADI requires states to review performance against 18 measures quarterly and to specify by when issues will be resolved, categorize the types of issues, and provide a plan for resolution. The TAA program, working with NYSDOL’s R&S Division, analyzes each customer record which contributed to underperformance in each quarter. This assessment informs training for case managers, as well as clarification to system guidance. In addition to TAADI reviews, TAA case records are subject to Program Monitoring and Financial Oversight and Technical Assistance (FOTA) reviews.
NYSED/ACCES-VR and OCFS/NYSCB are both assessed annually by the six (6) common performance measures under WIOA including: Employment Rate 2nd Quarter after Exit, Employment Rate 4th Quarter after Exit, Median Earnings in the 2nd Quarter after Exit, Credential Attainment Rate, Measurable Skill Gains, and Effectiveness in Serving Employers. The first five (5) performance measures are reported jointly by OCFS/NYSCB and NYSED/ACCES-VR. The sixth performance measure, Effectiveness in Serving Employers, is reported at a Statewide level for all partners.
The annual process of data validation and on-site monitoring for the SCSEP program provides NYSOFA staff the opportunity to ensure regional SCSEP providers are in compliance with SCSEP regulations, and that communicated information has been implemented.
The NYSOFA monitoring tool reviews all required SCSEP policies and regulations and is an effective indicator in determining if the providers are following USDOL statutes, regulations, and policies. Follow-up letters are sent to the providers indicating the results, strengths, and weaknesses (if any) found during the monitoring visit. If any issues related to SCSEP policies and regulations are found, a corrective action plan is required, and the Program Director works with the provider until the issue is resolved. Data validation coupled with the follow-up of those results provide further assurance that understanding and compliance with SCSEP regulations are implemented throughout the program area.
NYSOFA fiscal staff will monitor each regional provider through onsite monitoring. Fiscal staff previously monitored SCSEP when conducting onsite monitoring of each AAA since the program was previously administered by 29 AAAs. Now that the program is operated through a regional model, fiscal staff created a SCSEP-specific monitoring tool and will conduct onsite visits with each regional provider after the final claims are submitted.
Some local social service districts contract out their TANF and SNAP employment operations to local employment and training services, who co-locate with the district at the local Career Center. Each contract is developed directly between the one-stop partner and the district through a process that might involve regular meetings, review of fiscal documentation and developing outcome measures. The contracts involve specific policy requirements that are based in State regulations, with OTDA as the contact agency to provide guidance and interpretation of said regulations.
System-wide assessment and implementation of the WIOA Primary Indicators of Performance measures is burdensome. When the measures were initially phased in, performance measures used baseline data and expected rates were then negotiated with the use of a statistical adjustment model at the federal level. This statistical adjustment model did not account for events and circumstances experienced throughout the review period, for example, a global pandemic that reduced employment opportunities, shut down education systems and paused direct service provision. Expected and negotiated rates for the common performance measures are detailed for each partner in the Plan but are negotiated with the specific core partner and their federal oversight agency’s designated liaison. The challenges presented with needing to modify and adjust case management systems, technical assistance on required documentation and the calculation of performance measures as well as the impact of the COVID-19 pandemic made implementation, training, and monitoring of the progress on these performance measures burdensome for all core partners in NYS. Partners continue to evaluate and mitigate variables impacting performance measures as well as review, update and implement training and guidance at the staff level for effective reporting and collection of the performance measures.