Narrative: III. b. 4. C.

Beginning with the state plan modification in 2018 and for subsequent state plans and state plan modifications, provide the results of assessments of the effectiveness of the core programs and other one-stop partner programs and Combined State Plan partner programs included in the Unified or Combined State plan during the preceding 2-year period (i.e. the 2-year period of the plan modification cycle).  Describe how the State is adapting its strategies based on these assessments.

Representatives from the Workforce Innovation and Opportunity Act (WIOA) Steering Committee report results of Section 116 performance data to the State Workforce Development Board as do the local workforce development boards (WDB). These results are used to assess program effectiveness of core, one-stop partners. The WIOA Steering Committee uses this data to review trends, look for improvement opportunities, and set program and policy direction. The primary responsibilities of the WIOA Steering Committee are to oversee the execution of the combined state plan, oversee and direct subcommittees focused on plan strategies and to evaluate partner performance on a quarterly basis to ensure the quality and effectiveness of service delivery to workforce system participants. The committee also utilizes performance outcomes and quality assessment information to implement strategies and best practices to improve any deficiencies.

Missouri Vocational Rehabilitation (MVR) and Rehabilitation Services for the Blind (RSB) previously used federal standards and indicators to evaluate their performance. With the passage of WIOA, MVR and RSB are now utilizing the required performance indicators found in Section 116 of the Act for program evaluation. The review and analysis of this quarterly data is a valuable assessment tool and aids MVR and RSB in determining goal attainment as well as providing the opportunity for comparison across core partner agencies. MVR and RSB have adjusted their case management systems to support the common performance measures.

Until this plan period, MVR and RSB were only required to report baseline data for the first five performance indicators as performance data collection was not mandatory until PY 2017. For the PY 2020 – 2023 state plan, MVR and RSB were required to only report an expected level of performance for measurable skill gains (MSG).

MVR and RSB use MSG to evaluate how well clients and employers are being served as MSG is a good measure of program effectiveness and success. MVR and RSB analyzed the following MSG data:

RSB

PY 2017: 74.3%

PY 2018: 49.5%

PY 2017 and PY 2018 Average: 61.9%

MVR

PY 2017: 15.9%

PY 2018: 18.0%

PY 2017 & PY 2018 Average: 16.9%

PY 2019: 43.7%

PY 2020: 78%

PY 2019 & PY 2020 Average: 60.9%

Based on PY 2017 and PY 2018 data, Missouri estimated its expected MSG level of performance for PY 2020 would be 18.0% and 18.5% for PY 2021. The agencies negotiated 20.0% for PY 2020 and 20.5% for PY 2021 with Rehabilitation Services Administration (RSA).

After reviewing its performance on indicator 5, MSG, MVR determined an increase of this measure was necessary. MVR developed recommendations and strategies to improve clients’ attainment of MSG. Upon implementation of these, MVR exceeded 20.0% with an MSG rate of 43.7% for PY 2019 and 78% for PY 2020. Performance negotiations for indicators 1 through 5 are expected to be completed spring 2022.

In Fiscal Year (FY) 2021, 7906 students enrolled in Adult Education, meaning they were pre-tested and persisted 12 hours or more out of a total of 9761 reportable individuals, as reported through the Missouri Literacy Adult and Community Education System (LACES). This is an 81% persistence rate. The drastic drop in enrollment from previous years is attributed to the global pandemic’s impact on AEL’s ability to serve students when classrooms were suddenly closed for extended timeframes. In response, Missouri AEL developed a new Distance Education policy and provided opportunities for all local service providers to reach students virtually leading to an immediate increase in enrollment.

Also in FY 2021, 68% of the participants were administered post-tests, which is slightly less than the pre-pandemic years and higher than 2020, which was the first year of the pandemic.

In FY 2021, 67% of post-tested students completed an educational functioning level or advanced to higher educational functioning levels, which is a decrease of 4% from the previous year. This decrease is also attributed to the pandemic. With the development and expansion of distance education, DESE AEL expects this to return to normal levels.

Development, expansion, and improvement of adult education and literacy service delivery in Missouri are driven by enrollment and performance data. Monthly, AEL staff review program enrollment, financial, and performance data and provide immediate technical assistance based on those examinations. Every other month, AEL staff share statewide performance dashboards with local programs and discuss the strategic path forward to address gaps shown by the statistics. At the conclusion of each program year, AEL staff evaluate the statewide data and work with local programs to adjust service delivery. Analysis of enrollment and performance data ensures equitable access to adult education programming for all individuals and allows local programs to address the success of students with certain demographic classifications. AEL determines funding allocations and bonus distributions using assessed past effectiveness as a factor.

The Office of Workforce Development (OWD) utilized the MoPerforms reporting system to track performance outcomes. OWD was not able to meet 90% of Credential Performance in Youth in PY2019 but has significantly increased performance for PY 2020 in Youth and have maintained high performance levels in the other programs even with COVID restrictions and closures. OWD was quick to respond and adapt when COVID restrictions were initially into place to allow for participants to still receive services or begin new services helping to keep performance moving in the right direction.

During PY 20, the methods of service delivery for Missouri’s citizen and employer customers changed as a result of the pandemic. These shifts in service delivery resulted in strong customer satisfaction ratings that OWD will build upon for future years.

OWD leveraged technology to serve employers. Missouri served employers virtually throughout most of the year to deliver one-on-one services. In addition, virtual hiring events allowed employers the opportunity to connect with workers while distancing. During the program year, 23 virtual job fairs were held, with 738 employers receiving log-in credentials. Missouri received 339 surveys from employers during the program year. As a direct correlation to the return of in-person hiring events, 55%of surveys were received during the fourth quarter.  Overall, employers rated services at 95.5%during PY 20, an increase over the past program year.

Citizens were also served differently during the program year.  In addition to providing reemployment services to citizens, OWD assisted with Unemployment Insurance (UI) claims for weekly benefits. This allowed staff to provide information on the employment and training initiatives available to those affected by layoffs as they planned for returning to the workforce. Satisfaction with these services was rated very high. Nearly 5,850 citizens returned satisfaction surveys with an overall rating of 97.2%, matching the previous program year.

Efforts going forward will be on increased quantity of surveys collected across all regions and Missouri Job Centers as well as heightening interest and skill in the delivery of services from a satisfaction standpoint. 

The Jobs For Veterans State Grant Program (JVSG)

The JVSG program undergoes an audit of our case management system annually by the DOL-DVET to ensure programmatic compliance with regulations to include services to non-veterans, identification of barriers, and a plan to overcome those barriers.  They also perform site visits to the AJCs to interview staff in order to confirm compliance with Priority of Service.  DVOPs and LVERs are interviewed during the site visits to ensure role adherence.  Quarterly, Missouri sends a Managers Report to the DVET that outlines the responsibilities of the individual roles and what has been done to meet performance metrics.

Quarterly we submit a Technical Performance Narrative to DOL with our performance metrics in comparison to the negotiated rate.  

Missouri has also established an internal auditing process to occur monthly by the Intensive Services Coordinator as part of the Compliance and Monitoring Team.  We are using the same metrics established by DOL-VETS to ensure compliance with regulations.

In early 2020, the WIOA core partners met to design a "WIOA Partner Satisfaction Survey.” The survey was targeted to both frontline staff and leadership to determine cross-agency satisfaction with not only delivery of services to participants, but the level of engagement and collaboration between partners. These survey results are also being evaluated by the WIOA Steering Committee.

Core partners have completed performance negotiations with their respective federal agencies; moving forward, progress toward these outcomes will be evaluated by the partners through the WIOA Steering committee.