Narrative: III. b. 4. C.

State evaluation efforts for workforce programs have been coordinated through the State Workforce Development Board (Workforce Board) under WIOA and prior to that under WIA.  

The net impact and cost-benefit evaluation study of Washington state’s workforce development programs: While most evaluations of workforce system programs identify average participant outcomes like earnings and employment, this net impact study takes it a step further by using a quasi-experimental design to estimate the causal relationship between programs and participant outcomes. The findings produce employment, earnings, and social assistance outcomes that are attributed to program participation itself by isolating other influencing factors, such as prior employment history, education, gender, or race. This makes it possible to calculate the average net benefit of these programs for individual participants and a return on investment to society and taxpayers. 

The most recent net impact studies have been conducted in-house by the Workforce Board. Bringing the study in-house was made possible by increased staff expertise and technical know-how in econometric research design, statistical programming, and project management. An independent review was conducted on the 2021 net impact study to validate its evaluation methodology. This new approach yields several advantages: cost-savings, standardization of research design and evaluation methodology, study repeatability, and increased research unit capabilities. 

Next, the Workforce Board plans to conduct the study more frequently, possibly as often as annually. The net impact study has two parts: 1) the effect of program participation on employment and earnings and 2) the costs of program participation. The first part is derived from administrative data sets and can be done more often. The second part on program cost is difficult and onerous to assess and will remain a once every 4- or 5-year effort. Therefore, there will likely be a comprehensive study done every 4 or 5 years with more frequent updates using updated employment and earnings data. 

Additional research and evaluations: In 2024-2028, the state of Washington will explore the possibility of conducting additional research projects and evaluations on activities under WIOA-related programs and services. Additional studies may include implementation, outcome, or net impact evaluations. These are contingent on availability of funding and the initial research exploration phase.  

The initial exploration phase will aim to determine which projects the State will pursue. The State will go through the project selection process to identify and prioritize research and evaluation projects based on their feasibility, impact, and potential value. The process of project prioritization will be done collaboratively across state agencies and state and local boards. This process will determine which studies the State pursues. 

Depending on the project prioritization and selection process, in 2024-2026, the State will conduct implementation evaluation(s) to explore how WIOA-related programs and services are delivered and implemented in practice. Implementation evaluations can be accompanied by customer experience studies to examine the experience of customers and program staff. Examples of potential implementation studies include: 

• An implementation study of co-enrollment to examine how co-enrollment practices have been implemented across different locations. 

• An implementation study of training services to document operations of classroom- and work-based training service opportunities used by WIOA Title I-B participants and describe how these training services are implemented in practice. 

• An implementation study of different service delivery approaches to document the operational aspects of different WIOA-related service delivery approaches (in-person, virtual, or hybrid). This study can also explore the experiences of program participants and providers with different service delivery approaches. 

• An implementation study of different outreach strategies to describe how different WIOA-related programs and services reach out to and engage with their potential and existing customers. This study can also explore the experiences of program participants and providers with different outreach strategies. 

• An implementation study to explore the process of implementing WIOA-related programs serving youth and determine whether these programs are implemented as intended.   

• A study to examine a job quality framework. This study will aim to develop a job-quality framework in the context of Washington state. The study will examine existing definitions and frameworks to measure job quality; conduct an exhaustive literature review of economics, including the research that quantifies job quality; outline the benefits and drawbacks of each of these definitions and frameworks; and conduct quantitative or qualitative research to improve understanding or definitions and measurement of job quality in Washington state. 

Outcome and net impact evaluations: Depending on the project prioritization and selection process, in 2025-2028, the State will be able to implement outcome evaluations to examine whether WIOA-related programs and services achieve their objectives or success measures and net impact evaluations to examine whether programs and services have impact on the main outcomes. Each of the evaluations assumes to have at least a six-month feasibility study prior to the actual evaluation to determine whether a study in question is achievable. Study feasibility depends on several factors, including timeframe, data reliability and accessibility, sample size, and the ability to satisfy econometric assumptions for a given research design. When feasible, the research staff will prioritize net impact evaluations to outcome evaluations and employ rigorous econometric methods. Examples of potential studies include: 

• An impact evaluation of co-enrollment practices to answer the following question: What is the impact of automatic co-enrollment of eligible individuals into the WIOA Title I-B adult and/or dislocated worker programs on employment, and earnings outcomes? Do the impacts differ by demographic characteristics, such as gender, race, ethnicity, age, disability status, or location? 

• An impact study of different service delivery approaches to answer the following question: What is the effect of different service delivery approaches (e.g., in-person, virtual, or hybrid) on program participation, service receipt, as well as other career-related outcomes? How are these effects different by demographic characteristics, such as gender, race, ethnicity, age, disability status, or location?  

• An impact study of different outreach methods to answer the following question: What outreach efforts are most effective in increasing customer engagement at the one-stop centers or via virtual services? How do these effects differ by demographic characteristics, such as gender, race, ethnicity, age, disability status, or location? 

• An outcome evaluation of co-enrollment practices to answer the following question: What are the outcomes of individuals who are automatically co-enrolled into the WIOA Title I-B adult and/or dislocated worker programs? What are the outcomes of individuals who are co-enrolled into the programs in a discretionary way or are not co-enrolled at all? Does this relationship differ by demographic characteristics, such as gender, race, ethnicity, age, disability status, or location? 

• An outcome evaluation of training services to answer the following question: What are the outcomes of WIOA Title I-B participants who receive work-based training (e.g., on-the-job training, registered apprenticeship)? What are the outcomes of those who receive classroom-based training (e.g., occupational skills training, customized training, class-size contract training)? Are there differences by demographic characteristics, such as gender, race, ethnicity, age, disability status, or location?  

• An outcome evaluation of different service delivery approaches to answer the following question: How do different service delivery approaches (e.g., in-person, virtual, or hybrid) are associated with program participation, service receipt, and other career-related outcomes? Are there differences by demographic characteristics, such as gender, race, ethnicity, age, disability status, or location?  

Coordination across partners and stakeholders. The preliminary ideas for additional research and evaluation were developed through a process that included multiple stakeholders. This process included:  

• Development of ideas through the Evaluation Peer Learning Cohort (Eval PLC);  

• Prioritization of topics through the Workforce Board;  

• Review by program partners and state agencies that administer the core and required programs under WIOA; 

• Review by several local workforce development councils; and 

• Feedback from the state Workforce Board. 

First, the Eval PLC is a technical assistance initiative sponsored by the Employment and Training Administration within the U.S. Department of Labor. The initiative aims to build research and evaluation capacity across the WIOA programs, with an emphasis on adult and youth activities. Washington took part in Eval PLC in 2022-2023. The group of participants from Washington included representatives from the Workforce Board, WIOA Titles I-III (Employment Security Department and State Board for Community and Technical Colleges) and local workforce development boards (Workforce Snohomish and Olympic Workforce Development Council). Eval PLC participants developed some of the initial questions. Second, the Workforce Board has prioritized several topics to be considered for evaluation initiatives. Lastly, multiple state agencies that administer WIOA programs, several local workforce development boards and the Workforce Board reviewed the proposed plan. 

Additional research and evaluations are contingent on availability of funding, as well as the initial research exploration process. As noted, the initial exploration phase will aim to identify and prioritize research and evaluation projects based on their feasibility, impact, and potential value. The process of prioritization and selection will be done collaboratively across state agencies and state and local boards. This process will determine which studies the State decides to undertake. 

Evaluation plan details and feasibility will be explored and determined during the period of performance in coordination with the state and local workforce boards and with state agencies responsible for the administration of all respective programs.