U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Plan: New York PYs 2016-2017
Combined Plan C

Section: WIOA State Plan Common Elements

Narrative: III. b. 4. A.

Published
Located in:
  • III. Operational Planning Elements
    The Unified or Combined State Plan must include an Operational Planning Elements section that support the State’s strategy and the system-wide vision described in Section II.(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—

III. b. 4. A. Assessment of Core Programs

Describe how the core programs will be assessed each year based on State performance accountability measures described in section 116(b) of WIOA. This State assessment must include the quality, effectiveness, and improvement of programs broken down by local area or provider. Such state assessments should take into account local and regional planning goals.

Current Narrative:

Core Program and One–Stop Partner Assessments

In addition to the WIOA performance standards, NYSDOL also tracks several Customer Service Indicators (CSIs). CSIs are designed to move the Career Center System in NYS toward specific goals that NYSDOL deems important to the good of the state, and/or to assist Local Areas in meeting their WIA/WIOA Common Measure goals. First instituted during PY07, NYSDOL used a number of different CSIs that have evolved over time. For PY15, CSIs cover the following topics:

• Ensuring the provision of Initial Assessment as the first staff–assisted service;

• Ensuring customers have a complete customer record (termed “Employability Profile”) in OSOS;

• Ensuring customers identified as needing Job Search Ready Services (JSRS) receive a referral within 5 days of the Initial Assessment;

• Increasing the number of JSRS customers performing resume based searches in JobZone, which populates the NYTalent Bank;

• Ensuring Youth participants complete a CareerZone Portfolio;

• Ensuring Individual Training Accounts (ITAs) are done in occupations identified as regional priorities by LWDBs in a region; and

• Ensuring follow–up with business customers who have referrals on their job orders.

As discussed previously, NYSDOL, in partnership with NYS OMH, developed and implemented NYESS. NYESS is made up of various computer applications and data sets. Employment related data collection is accomplished by all NYESS partners using the existing NYSDOL case management system, OSOS. Legacy data sets from the partners and current data sets, which include but are not limited to, OSOS; vocational rehabilitation agencies; DOH; Social Security Administration (SSA); and others are pulled together in a data warehouse. A web–based reporting portal designed and maintained by OMH will provide cross–agency report card like information to the general public (aggregate data) and to the individual agencies and their contracted partner staff.

Anticipated 1st year metrics include, but may not be limited to:

• Number of shared customers across various agencies.

• Standardized “Report Card” like public reports (aggregate level) available on new web portal (in development): Statewide/Regional/Local; Agency/Provider; and Cross Agency.

Over the next three years, NYESS and NYS DDPC will develop an integrated web–based platform called DB101. This system will be integrated with NYESS, CareerZone, and JobZone to provide accurate, up–to–date information and benefits calculators so participants can better assess how going to work will impact their access to publicly funded healthcare and income support like Supplemental Security Income, Social Security Disability Insurance, the Supplemental Nutrition Assistance Program, housing assistance, and other public benefits.

The Adult Education and Literacy Programs Adult Student Information System and Technical Support (ASISTS) is a NYSED–funded, custom built management information system. This system has been used by funded programs in NYS since PY2007–08. In addition to serving as the database for all program data related to adult literacy, it also provides reports designed to check for data quality and compliance with NRS. To ensure consistent and continual availability of training support, ASISTS produced a number of training videos, focusing on daily required action with program data. Round table discussions are scheduled quarterly to promote data quality.

Leadership will continue to support a contract for technical assistance, training, and troubleshooting to improve data quality and monitor program performance in every program. The NRS accountability specialist funded by this NYSED contract works directly with NYSED staff. Duties include focusing on the largest providers with significant enrollments and programs who need intensive technical assistance to achieve improvement. In 2016–17, under the Combined State Plan, three programs from each region (for a total of 21) will be selected for inclusion in the Individual Technical Assistance Program (ITAP). The Big5 city school district programs (NYC, Yonkers, Syracuse, Rochester, and Buffalo) consistently receive support from ITAP, given their large size. All ITAP programs will be desk monitored, accompanied by monthly conference calls and an annual scheduled site visit. All communication and quarterly data analysis will be collected electronically and stored on the NYS accountability website in a secure portal for each program. The portal is accessible by the NRS specialist, the NYSED Regional Liaison, the RAEN Director, and the program manager. Each fiscal year, report card data is reviewed by NYSED, and any program’s performance that places that program in Quartile 1 and missed one or more of the state benchmarks, is placed under Corrective Action. The program will be required to correspond electronically on the accountability website. Quarterly data reviews will be conducted by the NRS Accountability Specialist and subsequent conference calls will be scheduled with those programs exhibiting poor performance or inconsistent data patterns. The NRS Accountability Specialist will assist in identifying problems and trends on a statewide and regional level.

Professional development in the areas of data collection, recording, and reporting continues to be a priority. Added to that is the constant need for NRS training at all levels. The Accountability Specialist will provide a minimum of two training events in each of the seven geographic regions annually. Topics include NRS Foundations, NRS for Teachers, and NYS Report Card Training.

Beginning in 2007, all funded agencies continued to receive NYS issued report cards. This will continue in 2016–17. The report cards evaluate programs under two distinct methodologies. The first is a standard benchmark measurement. NYS benchmarks for performance in Educational Gain, Post Test Rate, and Follow up Outcomes are set each year by NYSED based on the averages derived from the prior year’s state data. When programs meet all three benchmarks, they are scored as “proficient.”

The second method of evaluation is based on a quartile ranking among all programs in the state. In the case of Educational Gain, the program’s scores are weighted based on the educational functioning levels indicating the strongest proponent for gain. The weighting of each EFL is determined by the aggregate data analysis of the prior year’s data. The quartiles are determined using the following rubric:

• 50% of the score is from the weighted educational gain;

• 20% of the score is from the post–test rate;

• 20% of the score is based on the four follow–up measures, i.e., total students with a follow–up goal who achieved the goal upon exiting divided by the total exiters with one of the identified goals; and

• 10% of the score will be based on achieving both a 50% and 70% response rate on the survey of all students who exited the program with a follow–up goal.

Local program data will be entered quarterly into ASISTS according to the following dates:

• Quarter I (7/01–9/30) data entered and reviewed by 11/30;

• Quarter II (10/01–12/31) data entered and reviewed by 2/28;

• Quarter III (1/01–3/31) data entered and reviewed by 4/30; and

• Quarter IV (4/01–6/30) data entered and reviewed by 8/31. Note that the program has until 10/31 each year to finalize the annual data submitted over the previous four quarters. It is the responsibility of the program to have data entry up to date for each quarter. This timeline is being reconsidered for FY15/16 and may result in an abbreviated version.

ITAP continues in 2015–16. ITAPs identify agencies in need of intense technical assistance. The contracted Accountability Specialist, New York State regional staff, and RAEN directors work closely with these programs to identify gaps in quality and performance, and implement an effective program improvement plan. Effective use of webinars and conference calls provide a consistent line of communication with these programs as they implement change to improve their performance. As programs improve their performance, they are removed from ITAP status and others are then identified.

Beginning in November 2014 and continuing through 2015–2016, NYSED issues a Program Evaluation Report for each funded program. The annual Program Evaluation Report will identify whether the agency met the NYS benchmarks for Educational Gain, Post Test Rate, and Follow up Outcomes. These are aggregate benchmarks set annually by NYSED based upon NRS targets negotiated with the U.S. Education Department and previous year performance data from programs funded by NYSED. Programs that do not meet these NYS annual benchmarks will be required to undertake a formal Corrective Action Planning process. The program must submit and receive NYSED approval for a comprehensive CAP plan that identifies specific steps for meeting NYS annual benchmarks by June 30, 2017. For programs identified for corrective action in November of 2015 or 2016, they must meet their annual benchmarks by June 30, 2017 and June 30, 2018, respectively.

After the Corrective Action Plan (CAP) process is initiated, NYSED staff will conduct virtual calls/webinars to include the local program staff identified by the adult education director, the NRS Accountability Specialist under contract with NYSED and the program’s RAEN director to track progress toward meeting the NYS annual benchmarks. For a program placed in CAP in 2015–2016, the program will have two years to raise performance to meet or exceed the New York State annual benchmarks. All communication is documented on the accountability website. Program data and related data reports are stored in secure, user–protected, individual program accounts accessible only to NYSED staff, the RAEN director, and the NRS Accountability Specialist. The website and all secure documents are maintained through the office of the NRS Accountability Specialist, funded under contract with NYSED.

As part of the comprehensive CAP, there is a Continuous Evaluation Process (CEP) under which programs and NYSED must identify quarterly progress milestones on key indicators such as educational gain that lead to meeting NYS annual benchmarks within two years. NYSED staff, the RAEN director, and the NRS Accountability Specialist will conduct a quarterly review to determine whether approved quarterly milestones were met for each agency on corrective action. Failure to meet the quarterly progress milestones for any three consecutive quarters could result in the termination of the grant at the end of that program year. This means that a grant could be terminated as early as June 30, 2016.

Both the CAP and CEP processes will be reported and tracked through the accountability website at: http://www.adult–education–accountability.org for each program funded through this RFP.

Under the federal NRS for adult literacy programs, NYS, and ultimately local service providers, will be evaluated based on performance indicators. Successful applicants must submit data based on the federal requirements of the NRS. Information on the NRS may be found on the web page www.nrsweb.org.

Tracking data in ASISTS follows a four category process. These categories are sequential and allow for the diversity among New York’s adult literacy programming. The categories are as follows:

Category I – Setup.

Programs enter the following categories of data to set up their data account in ASISTS:

• Funding sources – Programs can track services funded by multiple sources.

• Sites – Providers enter data pertaining to each of their sites and connect it to services so they can run reports separately by site.

• Outcomes – In addition to tracking required outcomes, programs may also track outcomes that are specific to their own specialized programming, such as training or career pathways.

• Personnel – Providers add teacher information including name, experience, certification information, and longevity.

• Classes/Services – Providers create their roster of classes adding the schedule, teacher, and type/level of classes.

Category II – Registration.

All state funded programs must use a standard registration form, the Individual Student Record Form (ISRF).

• Student information – This includes name; DOB; contact information; and demographic information such as gender, ethnicity, employment status, educational background, and public assistance status. ASISTS also allows providers to note whether a person is a parent or disabled.

• Initial Assessment – All providers use one of two standardized assessments (the Test of Adult Basic Education (TABE) for Adult Basic Education (ABE) students or BEST Plus for English Language Learner (ELL) students) with registrants at intake or within 12 hours of instruction. This assessment is used to place students in literacy classes.

• Enrollment – Once registration is complete, registrants are assigned to classes based on their assessments and availability.

Category III – Services.

• Hours – Adult Education providers use ASISTS to track attendance hours on a daily, weekly, or monthly basis.

• Test scores – Post–tests are given by providers at regular intervals to demonstrate learning gains. ASISTS calculates the educational gain based on their standardized assessment scores.

Category IV – Outcomes.

Adult Education providers track student outcomes regularly, both during and after their participation in a program.

• Educational Gain – NRS guidelines define a student as demonstrating educational gain when his or her test score places him or her in an NRS level (one of six predefined educational functioning levels) that is higher than the one determined by that student’s initial or pre–test.

• Primary follow–up outcomes – Providers focus on the following three key NRS outcomes:

o Employment and Retention of Employment;

o Attaining the NYS High School Equivalency Diploma; and

o Entering postsecondary education or training.

• Other Outcomes – ASISTS also facilitates the tracking of a wide variety of outcomes that are customized per program.

Review – Reports and Downloads.

ASISTS includes the following reports to review the data and use it for reporting and management purposes.

• Funding reports – Reports that present data critical to reporting following state funding guidelines. The most critical report is the Program Evaluation report indicating on one page the most critical performance statistics NYSED uses to evaluate programs. Employment Preparation Education (EPE) claim forms are also generated from ASISTS.

• Data management reports primarily meant for tracking and internal program management purposes.

• Data checks that are specifically designed to help providers detect and correct issues relating to data quality and completeness.

All of these resources will serve as the basis of core program alignment to support WIOA success.