Narrative: III. b. 4. C.

The Arkansas Workforce Development Board commissioned the University of Arkansas' Center for Business and Economic Research at the Walton College of Business to complete a comprehensive WIOA Systems Evaluation, Skills Gap Analysis, and a series of Customer Surveys.

The entire evaluation report can be found at: https://www.dws.arkansas.gov/programs/workforce-innovation/

The results of this evaluation, along with the review of the WIOA Titles I - IV program performance, annual monitoring, and ADWS' risk management protocol for our WIOA formula and discretionary grants under the purview of ADWS will continue to serve as the foundation for our ability to determine the effectiveness of our core programs.

Evaluation & Research Projects

A new evaluation is underway and being conducted by the University of Arkansas’ Counselor Education program.

The Contractor will work closely with ADWS, the Arkansas Workforce Development Board, and workforce partners throughout the evaluation process, and will implement processes and procedures necessary to ensure that accurate and relevant data are used for evaluation purposes.

For each assigned local area, contractors will interview leadership to identify partner programs and non-profits that collaborate with the Workforce Centers. For these partners, the contractor will document and report on the existence of memoranda of understanding between the ADWS and partners, written referral procedures, documentation of training for partners regarding the services provided or alternative print materials to assist in collaboration OR, an alternative method to refer clients between the partners effectively.

The contractor will research and identify entities in the assigned local areas that provide services to the specific targeted populations required in the Bid Solicitation Document (See Appendix A).

For all programs identified as Workforce Innovation and Opportunity Act (WIOA), Wagner-Peyser Act Employment Service programs, and Adult Education and Family Literacy Act (AEFLA) programs, the Contractor will identify, from a sampled participant list, whether clients were referred or co-enrolled in other programs and will develop case studies on effectiveness of identifying and closing Skills-Gaps. The Contractor will interview staff for examples of skills-gap identification on either the occupation level, the demand level, or the skill-level based on interaction with an employer and will determine the services that were provided to address the skills-gap. The Contractor will also assess and report on the effectiveness of the services in achieving the objective using surveys, and interview techniques.

The Contractors will review a sample of Arkansas Workforce Centers against the State’s certification policy and applicable laws and regulations for six centers (four comprehensive and two affiliate centers). Each of the six centers will be located in different Local Workforce Development Areas.

Contractors will conduct a customer service survey for exiting participants of WIOA Title I and Title III sampled over the most recent 12 months available.

Objective 1: In the Comprehensive Centers for the 3 Local Areas selected for sampling, the Contractor will report case studies on effectiveness of identifying and serving individuals from targeted populations.

Objective 2: In the Workforce Centers assigned for the study, evaluate the extent to which innovative tools as well as emerging and current technology are used to enhance service delivery.

Objective 3: Review a sample of Arkansas Workforce Centers against the State’s certification policy and applicable laws and regulations for four comprehensive centers and two affiliate centers. Each of the six centers selected will be located in different Local Workforce Development Areas.

Objective 4: Conduct a statewide customer service survey for business customers of WIOA Title I and Title III receiving services over the most recent 12 months available.

Objective 5: Review existing policies and procedures in Arkansas and in other states and make recommendations for changes that will lead to optimal efficiency in the provision of services in the state in the following areas:

Objective 6 – a: Geographic alignment; do workforce center locations provide optimum access for customers, both businesses and individuals, related to public transportation, urban vs. rural communities and other considerations?

Objective 6 – b: Demographic alignment; do workforce center locations provide optimum outreach and access to priority and underrepresented populations as identified in the state’s data measures?

Objective 6 – c: Fiscal considerations; given rising costs associated with brick-and-mortar operations, what cost effective models can be implemented virtually to maintain the quality of services to our customers? How do we strike a balance between virtual and in-person services that ensures continued quality of services while lessening the financial impact of maintaining physical spaces?

Objective 7: Conduct additional research in Jefferson, Crittenden, Cross, Phillips, and St. Francis Counties in Arkansas. According to the American Community Survey (ACS), these counties have unemployment rates in excess of 4.5%, compared to the current statewide rate of 2.8%. The Contractors will attempt to document the factors that are responsible for this excessive unemployment rate and will make recommendations for change.

DELIVERABLES/PRESENTATION OF MATERIAL:

Written reports detailing project activities, methods of data collection, and data analysis will be provided to ADWS at designated times. These reports include: (1 ) Progress reports provided monthly and upon completion of major parts of the project as stated in the scope of work; (2) An Executive Summary which summarizes the major findings and recommendations; and (3) A Final Report which includes: background information, methodology used in the project, study findings, a discussion of the implications, and recommendations.

The Executive Summary and Final Report will be provided in print, disk, and electronic formats. Both the Executive Summary and Final Report will be appropriately bound to enhance access to the material. The data and the analysis of the data will be presented in charts, graphs, and tables, and in other configurations that promote access to and understanding of the information. Sample survey questions

will be included in the Final Report to assist in providing an understanding of the information presented. The Contractor will provide a sufficient number of reports to meet the needs of ADWS.

The contractor will provide evaluation status reports on a monthly basis on overall project goals. A preliminary report is due NLT March 30, 2024 and a final report NLT June 20, 2024.

Governor's Workforce Cabinet

In response to the evolving workforce challenges, Executive Order 23-16 was issued in early 2023 to establish the Governor's Workforce Cabinet.  The cabinet is composed of key state department leaders, including the Secretaries of Commerce, Education, Corrections, Human Services, Labor and Licensing, Transformation and Shared Services, and Veterans Affairs. The group is chaired by the Governor's Chief Workforce Officer, who oversees the coordination and execution of state workforce development policies and career education strategies for the State of Arkansas.

Data-Driven Workforce Strategy

The Executive Order requires the Governor’s Workforce Cabinet to prepare and implement a data-driven strategic plan for workforce development in Arkansas.  The development of the workforce strategy involved a collaborative, user-centered design process, engaging a wide array of stakeholders.  This included state, regional, and national partners, ensuring that the strategy benefits from a diverse range of insights and best practices. Input was also solicited from local businesses, non-profits, educational institutions, and community groups, ensuring that the strategy is grounded in the real-world needs of Arkansas's workforce and employers.

LAUNCH Skills-Based Workforce Platform

A key outcome of this collaborative process is the design of LAUNCH, a user-centered platform tailored to Arkansas's specific workforce needs.  LAUNCH is envisioned as a comprehensive tool that will facilitate easier access to employment and educational resources for residents and provide a streamlined talent acquisition and management system for employers.  The platform, being collaboratively developed with Research Improving People’s Lives (RIPL), is designed to integrate data-driven insights, leveraging the state's robust data-sharing infrastructure and advanced technologies like AI, to offer personalized and intuitive support for learning and employment.

LINKED OPEN DATA ON CREDENTIALS

Credential Registry

Arkansas has made significant progress towards compliance with DOL requirements and best practices for ensuring that individuals, employers, educators, and training providers have access to the most complete, current and beneficial information about providers, programs credentials, and competencies supported with these public, federal funds.  In support of this goal, we have established a statewide credential registry that is publicly accessible and supports full transparency and interoperability using Credential Transparency Description Language (CTDL) specifications.  Arkansas is making rapid progress, as featured in a recent report from the American Enterprise Institute (AEI), on publication of all ETPL, higher education, adult education, career and technical education, and registered apprenticeship programs and credentials, which will establish the interoperable foundation for many other state efforts, such as interoperable Learning and Employment Records (LERs).

Skills Data Coordinator

In order to support credential transparency, Learning and Employment Records, skills-based hiring, and data-driven alignment of workforce and education pipelines with evolving employer demand, Arkansas has established the position of Skills Data Coordinator.  This role, established out of Arkansas’s participation in the NGA Skills-Driven State Community of Practice, works to raise awareness of skills-based practices, cultivate communities of practice, provide educational resources, and establish statewide shared service data infrastructure to support the interoperability and use of data related to skills and credentials.

INTEGRATED SERVICE DELIVERY

In alignment with the WIOA Integrated Service Delivery Vision, Strategic Goal 2 of the Arkansas WIOA State Plan seeks to enhance service delivery to employers and jobseekers through:

• Development of a common intake process for jobseekers and businesses that will efficiently connect them with services available from all workforce development partner programs and identify any barriers to employment that need to be address.
• Development of an integrated data system that will enable the sharing of information between partner agencies to more efficiently serve both employers and jobseekers.
• Exploration of data sharing opportunities with non-governmental organizations that are committed partners to the state’s workforce center system that will lead to improved intake, referral, and case management for customers served by multiple agencies (both public and private).

Significant progress was made towards this goal in program year 2022 with the modification of the open source CiviForm service discovery and eligibility screening platform to meet WIOA common intake needs with the support of Google.Org fellows.  An integrated data system was also developed for facilitating electronic referrals between case management systems.  Data sharing opportunities were explored and are progression with governmental programs, community-based organizations, and healthcare systems to support a “No Wrong Door” service delivery strategy.

TALENT PIPELINE MANAGEMENT

Arkansas workforce staff participated in the Multistate Career Readiness cohort of the Talent Pipeline Management (TPM) Academy hosted by the U.S. Chamber of Commerce Foundation alongside Arkansas eligible training providers.  Talent Pipeline Management (TPM) is a cutting-edge, data-driven approach to meeting talent demand that enables employers to play an active role in creating and managing their talent supply chains.

Throughout the six-month program, teams from Arkansas, Florida, and Tennessee learned and applied proven strategies to create career pathways for students and workers with talent pipelines aligned to dynamic business needs.  Data on demand, supply, skills, competencies, and credentials plays a critical role in TPM, and the longitudinal data system developed under WDQI was a key support, that will be incredibly valuable during implementation of these strategies in PY 2023 and beyond in coordination with Next-Gen Sector Partnerships.

JOBS AND EMPLOYMENT DATA EXCHANGE (JEDX)

Arkansas was one of seven states that actively contributed to working groups, standards, and protocols for the Jobs and Employment Data Exchange (JEDx), an initiative of the U.S. Chamber of Commerce Foundation and the T3 Innovation Network to develop a public-private approach for collecting and using standards-based jobs and employment data.

Working groups, surveys, and employer engagement during program year 2022 identified multiple value propositions for JEDx to improve not only the timeliness, granularity, breadth, and utility of labor market information (LMI), but also to reduce employer reporting burden, support interoperable Learning and Employment Records, support integrated service delivery, and provide valuable talent analytics to employers, providers, researchers, and government.

Arkansas participated in “bridge phase” planning in PY 2022 and submitted a letter of intent for participation in the pilot phase of the JEDx partnership in PY 2023.

ELIGIBLE TRAINING PROVIDER REPORTING

Heading into our second year of “all participants” eligible training provider reporting (ETA 9171), we have started to simplify the process for providers by starting earlier in the process and aligning data collected in WIOA case management system (AJL) with the administrative data that is contained in the Arkansas Higher Education Information System (AHEIS).  By ensuring that CIP Codes entered in AJL match those in AHEIS and capturing the unique degree codes of the programs, we were able to improve the efficiency and effectiveness of the data match for the for-credit programs on the ETPL.

For non-credit programs and providers, we have provided a roster to fill out for each of their programs with in-depth instructions, illustrations, and recorded presentations on how to provide the needed data for all participants in a secure manner.  We are making use of an easy-to-use encrypted transfer system that does not require the providers to have any specialized knowledge or skills related to secure file transfer because that can be a limiting factor in getting them to engage in the data collection.

Over the past year we have worked with providers to improve the quality and utility of ETPL data such as reporting on each individual credential as opposed to one program that can result in multiple credentials.  For example, providers have previously reported a single welding program that can result in three different for-credit credentials that ranged from a one semester certificate of proficiency to a two-year associate degree.  By increasing the grain and interoperability of ETPL reporting, we are able to more clearly differentiate participation, completion, and outcomes at the credential level. 

This is beginning to help improve consumer use of the ETPL by providing more transparent information about program costs, length of time needed to complete a program, and expected employment and earnings outcomes for different credential levels.

Chief Research Officer, Dr. Jake Walker, presented more information on this topic in a WDQI Web Convening presentation on "A World Without Waivers: Arkansas’ Journey Reporting ETPL All Participant Measures.”

EVALUATION AND RESEARCH – ENROLLMENT TO EMPLOYMENT

In order to support evaluation and research efforts relevant to state strategy, policy, and practice needs, Arkansas worked with the Coleridge Initiative to develop a new Applied Data Analytics training program curriculum, titled “Evaluating Enrollment to Employment Pathways”.  This program leverages the Arkansas workforce longitudinal data developed under the Workforce Data Quality Initiative (WDQI) to train participants on applied use of data analytics on integrated WIOA, SNAP, TANF, and work-based learning data to design and develop scalable research and products for supporting data-driven policy and practice, with a particular focus on co-enrollment and integrated service delivery.  Two cohorts of up to 30 students each will be supported under WDQI, and DOL ETA has funded two additional cohorts.

The curriculum leverages a scalable data model based on interoperable layouts to support collaborative development and use of continuously improving data products by interagency, multistate, and cross-sector teams.

ADULT EDUCATION JOINT PIRL DEVELOPMENT

To support an interoperable data foundation based on standard file layouts for the Evaluating Enrollment to Employment Pathways training, Arkansas worked with the case management software provider for Title II Adult Education programs to generate a WIOA Joint Participant Individual Record Layout (PIRL) file for Title II.  Representing Title II Adult Education programs in the same format as WIOA Titles I & III allows for integration and analysis across WIOA Titles I & III, Title II, and Title IV (via RSA-911 crosswalk) to better understand patterns of referral, co-enrollment, common exit, and outcomes of joint service delivery episodes.

ADMINISTRATIVE DATA RECORD LINKAGE ROADMAP

During PY 2022, Arkansas conducted research under the America’s DataHub Consortium to assess record linkage approaches for the foreign-born population and to recommend a roadmap for improving the fidelity of administrative data linkage on the foreign-born science and engineering workforce.

America’s Data Hub facilitates coordinated research to drive both infrastructure recommendations and response to relevant evaluation and research needs for current policy priorities through a series of research efforts for the development of a national secure data service.  The Foreign-Born Scientists and Engineers and the U.S. Workforce (FBSE) project is focused on improving the evidence base for measuring the economic impact of foreign-born scientists and engineers, which can be particularly affected by record linkage approached based upon Social Security Number (SSN), introducing coverage gaps and biases into data, evidence, and performance reporting.

Arkansas contributed to this work by researching record linkage approaches, assessing the quality of administrative data and the impact on record linkage performance for the foreign-born population, assessing and comparing the performance of a wide variety of record linkage approaches, and identifying methodologies for assessing, mitigating, and communicating record linkage bias.

Based on findings and consultation with a panel of experts, the Arkansas research team developed a roadmap of actionable recommendations for improving record linkage fidelity for the foreign-born population by:

• Raising awareness of record linkage bias
• Improving mitigation and communication of record linkage bias through education
• Increasing transparency of record linkage approaches, performance, and data quality
• Enhancing the collection and sharing of individual identifiers to improve record linkage performance

This research has broader implications for improving record linkage at the state and federal levels and will continue into PY 2023 with implementation or recommendations. 

Some recommendations have already been implemented through changes in Applied Data Analytics training program curriculum.  Some recommendations have relevance to discussions by the Workforce Information Advisory Council on enhanced collection and sharing of wage data.  Record linkage improvement will also be assessed during Jobs and Employment Data Exchange (JEDx) pilots.

WORKFORCE DATA QUALITY INITIATIVE

On June 23, 2021, the U.S. Department of Labor awarded Arkansas a Workforce Data Quality Initiative (WDQI) round 8 grant in the amount of $2,510,575.  This grant is designed to improve the delivery of training and employment services and to measure their effectiveness by supporting the development and expansion of workforce longitudinal databases and enhance their ability to share performance data with stakeholders.

Arkansas’ objective is to expand and enhance the longitudinal data systems around the needs of two complementary strategies for promoting continuous learning and improvement in workforce and education.  The first strategy is to help state leaders and service providers continuously improve programs and services and align resources to the evolving needs of citizens to deliver better workforce and education outcomes.

The second strategy is to facilitate continuous learning and collaboration through shared standards and governance, analytic capacity building, coordinated research and data products, and interagency, interstate, and cross-sector collaboration to support improved policy outcomes and an engaged community of practice.

WDQI efforts in PY 2021 were largely focused on employment, unemployment, postsecondary program participation and completion, and work-based learning.  This included development and delivery of an Applied Data Analytics training program on work-based learning to a multistate group of participants.  The final report for that training program is available here.

WDQI activities in PY 2022 were largely focused on inclusion of WIOA Titles I-IV, SNAP, and TANF in the longitudinal data system and development of the aforementioned “Evaluating Enrollment to Employment Pathways” training program.

TANF data was included using the Chapin Hall TANF data model from the TANF Data Collaborative.  SNAP data was incorporated using a model based on the USDA Gold Standard record layout used in the SNAP Longitudinal Data Project.  This use of standard record layouts contributes to multistate analyses and the ability to share and codevelop research and data products.

PY 2023 activities will largely focus on a number of identified evaluation and research projects and products and enhanced data sharing for federal and military employees.

STATEWIDE LONGITUDINAL DATA SYSTEMS GRANT PROGRAM

Arkansas applied for and was awarded a grant under the IES Statewide Longitudinal Data Systems Grant Program to modernize and expand the statewide longitudinal data system for early childhood through postsecondary education with modern data hub capabilities similar to those being implemented under WDQI.

Key work products under this grant include: 

• Development of a centralized, standardized, and interoperable non-traditional student management and information system to collect non-traditional provider, program, student, and outcomes data. This system will be accessible to all non-traditional education and training providers public and private and will be used to facilitate state and federal reporting related to these programs, inform state leaders and policy makers concerning program funding and need, administer/manage non-traditional sources of student financial aid.
• Modernization of early childhood to postsecondary longitudinal data to use Common Education Data Standards (CEDS) for multistate interoperability and collaborative development and sharing of research, data, and products.
• Incorporation of data on foster care and juvenile justice participants.
• Incorporation of more robust early childhood education data.
• Development and delivery of Applied Data Analytics training programs addressing Arkansas education and workforce priorities.
• Development and delivery of tiered-access tools and data products to facilitate access, analysis, and use of linked administrative education data by school and government staff, researchers, and the public.

This project will leverage, complement, and expand upon longitudinal data developed under the Workforce Data Quality Initiative (WDQI), Justice Reinvestment Initiative (JRI), Democratizing Our Data Challenge (DDC), and Arkansas Health and Opportunity for Me (Medicaid expansion) programs to support state needs with a comprehensive integrated data system.

Future longitudinal data system work across all of these efforts is planned to include robust integration with the credential registry, Learning and Employment Records (LERs), and enhanced labor market information such as that envisioned under the Jobs and Employment Data Exchange (JEDx).  Implementation of Talent Pipeline Management (TPM) is also expected to provide timely, actionable data on evolving employer demand which can be used in conjunction with state administrative data to support informed alignment of workforce and education talent pipelines and provide timely measures of credential quality and demand.  Consumer information products and user-centered platforms like LAUNCH can leverage this complete, timely, interoperable data foundation to provide highly actionable and personalized supports to residents for career awareness, exploration, pathway planning and navigation, employment, and connection with supportive services.

CUSTOMER SATISFACTION

The State assesses its core programs through on-going continuous improvement strategies including:

• State and local level fiscal and programmatic monitoring (on-site and/or desk reviews) of all programs within our combined WIOA state plan
• Customer Satisfaction Surveys (for both the jobseekers and employers) are conducted annually
• State and local level data validation of all programs within our combined plan
• The Program and Performance Evaluation Committee, a subcommittee of Arkansas Workforce Development Board, has the functions of making recommendations on performance standards for the forthcoming year and to review actual outcomes across programs, make recommendation for improvement and/or corrective action to ensure attainment of performance standards for all partners
• As a state agency, there are annual audits of programs conducted and specific, as needed, audits of local workforce areas are also conducted by third party vendors

The state also ensures that local workforce development areas meet performance accountability measures  and provide technical assistance as needed.

The Arkansas Workforce Development Board commissioned the University of Arkansas' Center for Business and Economic Research to complete a comprehensive WIOA Systems Evaluation, Skills Gap Analysis, and Customer Service Surveys. This evaluation is currently underway. 

This evaluation will identify any areas for improvement as well as recommendations designed to improve the effectiveness of the WIOA Programs' service delivery.

It is expected that the Board's Program and Performance Evaluation Committee will take the lead in developing a framework that provides a consistent snapshot of individual program results, despite the distinct variations in services, activities, and populations served.

The results of this evaluation, along with the review of the WIOA Titles I - IV program performance, annual monitoring, and ADWS' risk management protocol for our WIOA formula and discretionary grants under the purview of ADWS will continue to serve as the foundation for our ability to determine the effectiveness of our core programs.