Narrative: III. b. 4. C.

The Arkansas Workforce Development Board commissioned the University of Arkansas' Center for Business and Economic Research at the Walton College of Business to complete a comprehensive WIOA Systems Evaluation, Skills Gap Analysis, and a series of Customer Surveys.

The entire evaluation report can be found at: https://www.dws.arkansas.gov/programs/workforce-innovation/

The results of this evaluation, along with the review of the WIOA Titles I - IV program performance, annual monitoring, and ADWS' risk management protocol for our WIOA formula and discretionary grants under the purview of ADWS will continue to serve as the foundation for our ability to determine the effectiveness of our core programs.

The overall recommendations from this evaluation concluded:

1. Co-enrollment - While co-enrollments exists, data documenting rates of co-enrollment is not readily available. It was recommended that a system of data collection to effectively track and monitor co-enrollments is developed.
2. Partnering Across Programs - Many partner organizations reported that they do not have information available to adequately partner with the Workforce System. It was recommended that the State and Local Workforce System Partners work more closely together and provide the necessary training and information to assist all potential partners.
3. Business Services - The results of the Business Customer Surveys found that less than half of the businesses surveyed believed that they received all of the needed services. It was recommended that additional training be developed and provided to the Local WIOA Business Services Teams to improve their interactions with business customers. Additionally, it was recommended that training and promotional materials specific to the workforce system be developed and distributed.
4. Customer Services - Many customers felt that customer services training was needed for staff. Some respondents found that staff was either uninformed or inattentive to their needs. It was recommended that additional customer service training for all workforce center staff is provided. Another issue was that the current phone system does not appear suited to surge demand of ADWS or WIOA services. While it is possible that the increased demand on the phone system due to the COVID pandemic is at least partially responsible for the problems reported by customers, the evaluators highly recommend that the phone system be professionally evaluated immediately.
5. High Performing Centers - There existed a significant difference in the quality and success of the services provided across workforce centers. It was highly recommended that methods to identify high performing centers and workforce areas is developed, along with additional training for all workforce centers.

In addition to addressing those recommendations from this evaluation, the State Board's Program and Performance Evaluation committee will continue to focus on partners’ performance and eligible training providers. Not only do they establish and monitor partner performance, they will also look at continuous improvement strategies related to credential, skills gains, and business performance.

Adult Education Assessments

The Arkansas Division of Workforce Services, Adult Education Section conducts a several assessments of subgrantees including quarterly desk audits, annual site visits, and cyclic intensive onsite program reviews. The result of this information is used to provided technical assistance and professional development training.

One of the assessments the Arkansas Division of Workforce Services, Adult Education Section monitors programs through quarterly reports and data analysis. Through this process, programs enter statistical and performance data that has been collected and reported in the data management information system (LACES) in a spreadsheet to the state office. Some of the information collected include a program’s enrollment numbers, measurable skill gains performance, number of high school equivalency diplomas earned, and the demographic data associated with student progress, and much more. For some data, program’s performance in the current year is compared to the previous year in hopes of identifying trends. This process ensures that the responsibility is placed on providers to stay current on the program’s progress towards performance measures. In addition, this also gives providers an opportunity to make necessary adjustments to program operations as the fiscal year goes on versus waiting until the final quarter, which is oftentimes too late to make necessary changes. At the state level, the information is used to provide targeted technical assistance and professional development training to providers.

Another assessment the Arkansas Division of Workforce Services, Adult Education Section uses to monitor and assesses programs is continuous desk audits, which includes items such as average attendance hours, educational functioning level (EFL) gains across demographic groups, number of students post-tested, etc. This allows state staff to review day to day performance of programs to ensure compliance with state and federal policies and guidelines. The desk audits also allow programs to review operational aspects of the program such as the percentage of funds expended, completion of required professional development, and completed personnel activity reports. In addition, like the quarterly reports, desk audits give state staff an opportunity to provide technical assistance and provide targeted professional development training.

Arkansas Division of Workforce Services, Adult Education Section also performs an annual site visit for each adult education and literacy program to ensure that state and federal laws and policies are followed, to review program performance, and to connect with Local Education Agency Administrators. Items included on the site visits may vary year to year and is, oftentimes, based on findings from the more extensive three-day Program Review Monitoring visit. During site visits, the following items may be reviewed by state staff: 

1. Accuracy of student records
2. Data collection measures and analysis (performance, student information, SSNs, and manual outcomes)
3. Professional Development trainings of staff
4. Student and staff handbook policies
5. Complete and accurate financial data
6. Program’s recruitment and retention plan
7. Development of workplace, digital, and financial literacy skills
8. How distance learning is being integrated in the program
9. WIOA partner engagements

Based upon the documentation and discussion during the site visit, regional program advisors provide technical assistance to program directors and staff to ensure continuous improvement. Within 10 days following the site visit, the program receives a summary report of the visit that indicates areas of compliance, recommendations, or commendations. If a program has a repeat site visit finding on an item two years in a row, a Performance Improvement Plan (PIP) must be completed. A plan of action must be submitted. In addition, scheduled follow-up meetings are set with their Program Advisor until the PIP has been satisfied. The program director and program advisor work together to ensure that each compliance issue is corrected, at which time the program and Local Education Agency are sent a compliance letter from the State Director.

Additionally, approximately every four years, nine of the 37 providers are selected for an intense three-day Program Review Monitoring visit. When a new Request for Proposal for AEFLA funds is competed and providers are approved for funding, programs are randomly selected for the next four years on which year they will receive a Program Review. Some of the criteria that may be used in the determination of Program Reviews include the region of the state where the program is located, the type of Local Education Administrator including community-based organizations, community colleges, school district, number of findings (for currently funded programs), leadership turnover, amount of grant funds, etc. A team of program advisors conducts an intensive program review through which Division staff assesses each program in the following areas:

1.             Program Management and Reporting: Staff manual/continuous program improvement
2.             Financials: Timely report submissions and completed audits
3.             Advisory Committees: Local Adult Education Advisory Committees
4.             Personnel and Professional Development: Ensuring staff meets requirements
5.             Student Intake, Documentation, and Orientation: Maintenance of student information and supporting documentation
6.             Program Policies: Inclusion of required policies such as anti-bullying and student handbook
7.             Instruction: Course offerings and lesson planning based on the College Career Ready Standards
8.             Teaching/Classroom Observation: Observation of classroom activities
9.             Technology: Maintenance and utilization of adequate technology
10.            Recruitment/Retention (Reviewing recruitment and retention plans
11.           Collaboration: Partnerships with WIOA partners and other stakeholders to meet the needs of participants
12.           Facilities: Safety and adequacy of facilities conducive to learning
13.           Americans with Disabilities Act of 1990:  Adherence to ADA requirements

The program review process allows program advisors and the State Disabilities Project Manager to thoroughly assess program operations, observe classroom instruction, and review state and federal laws and policies. Each program review is accompanied by a full financial review by the state agency’s finance department.

The state office requires adult education directors to attend a State Administrators’ meeting twice a year. During this time, state staff review policies and procedures, state performance, answer questions, and give providers an opportunity to share best practices.

Additionally, the Adult Education Section annually conducts an Effective and Efficient (E&E) evaluation of each program’s performance based on the federal benchmarks. Programs that do not meet the overall benchmark requirement are provided additional technical assistance and monitoring. In the 2019-2020 program year, 18 of the state’s 37 programs (48.6%) did not meet the Effective and Efficient requirement.  In 2020-2021, that number was reduced to 6 of 37 programs (16.2%). Based on the findings, we conclude that the intense technical assistance that includes the review of data, program processes and procedures, instructional content and methods and the professional development of staff, through which a performance improvement plan is developed, is valuable in improving the effectiveness of programs. Fortunately, in spite of some providers not meeting the federal and state benchmarks, Arkansas exceeded its Measurable Skill Gains performance for the past two years.

One of the most common difficulties identified within the past two years is understanding and developing integrated education and training (IET) programs and timely submission of monthly and annual financial documents. To increase the understanding and the number of IET offerings, professional development has been offered during semiannual Administrator Meetings, and regional Program Advisors have provided one-on-one technical assistance to program administrator as well. Additionally, partnerships with the TANF and SNAP E&T Programs has been helpful in providing options for funding the workforce training portion of the program so that participants can earn industry-specific credentials that along with the workforce preparation and the basic skills prepare them for a new career pathway.

In summary, the Division of Workforce Services, Adult Education Section uses a variety of assessment tools to determine program effectiveness. Desk audits, site visits, and program reviews provide a variety of options to ensure compliance. In addition, this gives state staff an opportunity to provide intense technical assistance and build positive relationships with providers. Lastly, assessments are ongoing and have been very beneficial in the continuous improvement of programs and services to clients in Arkansas.

Rehabilitative Services Assessments

To establish baselines and meet benchmarks for state with respect to the performance accountability measures under section 116 of WIOA. 

In PY19 the number of participants increased to 13,182, with 2,236 successfully employed participants in the 2^nd quarter after exit (57.3%) and 1,554 successfully employed participants in the 4^th quarter after exit (38.6%). The credential rate was at 17.5% and the measurable skills gains rate was at 45.5%. During PY20 VR participant numbers reduced slightly to 11,830. The 2^nd quarter after exit rate rose to 60.4 % with 2,283 exiters. The 4^th quarter after exit rate climbed to 57.2% with 2,080 participants maintaining steady employment. The credential rate rose to 26.2% and the measurable skills gain rate was at 54.6%.

These steady and increasing baselines paved the way for benchmarks to be set in April 2022.

Increase the availability of Supported Employment (SE) services to youth and adults with significant disabilities. 

ARS increased services to youth and adults with significant disabilities by placing a high priority on Supported Employment outcomes and by partnering with other State of Arkansas agencies, as indicated above. During PY19 the program was implemented, and staff were trained on levels of disability significance for supported employment services. 5,061 or 46.2% were determined significantly disabled Individuals in PY2020 and 642 were most significantly disabled, all qualifying for supported employment services.

Improve public relations, marketing, information, outreach, and referrals.

ARS has launched a marketing campaign to promote the services of the agency through digital platforms, video clips, publications, flyers, and by updating our website materials. ARS has contracted with communications providers to offer materials in alternative languages and in alternative formats. ARS has also begun to host outreach events for special populations, like the Deaf and Hard of Hearing Community to meet their unique needs. An electronic referral was developed that can be sent via a hyperlink that takes less than 30 seconds to complete to begin the referral process. This year ARS also embarked on the Comprehensive Statewide Needs Assessment to gather data on the needs of clients to improve services, marketing, and outreach efforts. The data from that survey will be used to develop the goals for PY22, PY23 and beyond.

Increase collaboration with business that leads to increased employment outcomes. 

ARS has increased support to businesses by improving the process for On-the-Job Training and Work-Based Learning. The programs support paid work experiences for clients that can result in employment offers upon successful completion of the training. ARS has also successfully partnered with the National Apprenticeship Training Foundation and the Career Technical Education program of local high schools to bring apprenticeship and pre-apprenticeship programs to clients in a supported setting where they are succeeding and becoming gaining employed in the medical field and in transportation/logistics upon completion of the programs.

Expand and improve transition services program. 

The transition services programs have expanded to include work-based learning programs throughout the school districts across the State where students are engaged in small business efforts on campus or in work opportunities both on and off campus. Over 6,000 students have participated in transition services since PY17, with the largest increase being in PY19, with 4,290 students participating. Transition includes the Youth Leadership Program, summer Film Camp for careers in the film industry, College Bound Arkansas, the Transition Employment Program, Project Search, and special projects for work-based learning for students in the custody of the State of Arkansas living in juvenile detention facilities.

Develop and improve Community Rehabilitation Programs.

The Arkansas Career Technical Institute (ACTI) has transformed into the Arkansas Career Development Center (ACDC). ACTI was the Community Rehabilitation Program managed by Arkansas Rehabilitation Services. ACDC has moved into a model that provides virtual services, as well as services designed to bring training to students in the areas where they live and work. With pre-apprenticeship and apprenticeship programs, as well as support services that build essential communications and needed resume’ writing, interviewing, and technical communication skills. ACDC also trains other Community Rehabilitation Providers on service delivery of hard skills certifications that can be offered to participants in Pre-Employment and Transition programs throughout the State.

Improve training, resources, and continuing education.

ARS is currently partnering with the Rehabilitation Services Administration, the Council for State Administrators of Vocational Education, the Technical Assistance Center on Quality Employment and San Diego State University to offer direct training on vocational rehabilitation service provisions, management of the vocational rehabilitation program, and continuing education units. ARS also offers direct training supports that are approved through the Commission on Rehabilitation Counselor Certification for CEU credential training for counselors throughout the year.

Improve ARS’ ability to act as a resource when assisting individuals with disabilities and employers in addressing accommodation needs. 

ARS has improved it’s reach to State agencies and to businesses in offering information and assistance on ADA accommodations and technical assistance on ergonomics, technological advances, and how to use ARS assistance to meet accommodation needs for existing employees. Through the Stay-at-Work/Return-to-Work program employers are educated and assisted to make modifications and accommodations in the workplace.

Increase the efficiency and effectiveness of service delivery for agency clients. 

ARS is currently implementing a new case management system with enhanced features to improve the efficiency of data entry and to reduce the amount of entry required. ARS has also implemented new counselor training for all staff, created an online database for training, complete with assessments, training guides, and examples for counselor use. Duties that are primarily administrative have been shifted to other staff to increase counselor time with clients. ARS has also increased the number of vendors participating in job placement and work-based learning activities for clients. The Access and Accommodations section was given additional staff to increase psychological assessment output to improve timeframes for counselors to engage clients in plan development. The ACDC staff added a new program that supports counselors that have students in college, to ensure they have a direct contact while in school for retention purposes and job placement assistance upon completion.

Division of Service for the Blind Assessments

To establish baselines and meet benchmarks for state with respect to the performance accountability measures under section 116 of WIOA.  During FFY2020, DSB had 147 successful employment outcomes, and during FFY 2021, DSB had 128 successful employment outcomes. While the overall outcomes are down FFY 2018, the Covid-19 pandemic greatly impacted the availability of employment options and the safety of accepting work for consumers with a disability. DSB’s median income and measurable skill gains performance rates increased from PY2018 to PY2020.

Increase the availability of Supported Employment (SE) services to youth and adults with significant disabilities.  DSB is working with the VR Technical Assistance Center for Quality Employment to enhance the Supported Employment program. This work includes updated policy, procedures, and training for staff.

Improve public relations, marketing, information, outreach, and referrals. DSB has hired an Outreach Manager and Business Engagement Coordinator to assist with conducting outreach across the state. DSB has updated our brochures to provide information about Business Engagement, Pre-Employment Transition Services, Older Individuals Who Are Blind program, and the Vocational Rehabilitation program. DSB has increased membership in local area chamber of commerce organizations.

Expand and improve transition services program.  DSB has promulgated new Pre-Employment Transition Services policy. Staff continue to receive training on providing these services in the most effective manner possible. DSB is continuing to work with the National Technical Assistance Center on Transition: the Collaborative (NTACT:C) to improve forms, processes, and training for staff.

 Improve training, resources, and continuing education. DSB continued to provide training for staff throughout the year and at the two annual trainings each year. While trainings have been virtual during the Covid-19 pandemic, this also allowed staff to participate in more trainings than in the past. DSB continues to work on finalizing new drafts of policy and procedures with feedback from the Technical Assistance Centers and RSA.

Increase the efficiency and effectiveness of service delivery for agency clients.  DSB provides training to field staff on common errors and concerns in service delivery. New policy was developed and prepared for approval by the DSB Board with the support of WINTAC to ensure compliance with WIOA.  New policy is an ongoing process with chapters being reviewed by all stake holders and then sent for promulgation to improve service delivery.