Narrative: III. b. 4. D.

CDLE contracted with Northern Illinois University (NIU), Center for Government Studies (CGS), to conduct a statewide two-phase evaluation in September 2020. Phase I, the quantitative analysis, NIU used data to verify employment outcomes through Colorado’s MIS and case note review.

Phase II, the qualitative analysis, NIU gathered and synthesized data from focus group interviews in the ten local areas. The teams interviewed primarily consisted of staff who collected, entered, and processed data. NIU submitted a final report focused on the return on investment in training-related placement and provided an overview of the mixed methodology evaluation approach. In addition, the report identifies promising practices in each local area and incorporates recommendations for both CDLE and the ten local areas for future reference.

Workforce Development Programs (WDP) contracted with Northern Illinois University (NIU), October 2021, to provide a deeper understanding of the effectiveness of workforce programs and build the evaluation capacity of state and local personnel.

NIU created The Evaluation Tool intended to independently evaluate outcomes of WIOA programs with an increased emphasis on serving disadvantaged populations and service delivery to improve programs. The Evaluation Tool is an evaluation approach with the objectives of: Identify promising practices that result in a better outcome for program participants and improve the capacity of WDP and Local Workforce Area personnel to conduct independent evaluations.

NIU will train evaluator leads in the ten local areas and WDP staff on using the Evaluation Tool. The local evaluators will work with NIU to determine and analyze recent, higher-quality data and identify outcomes leading to promising practices.

The data will provide demographics of WIOA participants and identify groups of people who underperform compared to the highest performing group, showing a disproportionate impact. In addition, NIU will train local evaluator staff on the application of the disproportionate impact analysis.

With the assistance of NIU, the evaluation leads will learn to craft qualitative analysis questions informed by quantitative data analysis. In addition, NIU will conduct qualitative research using the focus group approach and request local areas to identify staff to recognize promising practices.

In June 2022, NIU will release a final report and present the findings and recommendations to CDLE leadership and the local workforce directors.

USDOL Evaluation Peer Learning Cohort 

On November 1, 2019, Colorado submitted its application to become part of the The Evaluation Peer Learning Cohort (EvalPLC) which is an interactive technical assistance forum and comprised of cross-agency representatives from four to six states. State teams who represent core WIOA programs will collaborate and develop evaluation action plans designed to conduct and implement state and/or local research and evaluations.   

Participants will: 

• Learn from experienced evaluators, state and national subject matter experts (SMEs), and each other as they develop an evaluation plan.  
• Use the Evaluation Toolkit: Key Elements for State Workforce Agencies and two evaluation assessment tools to build evaluation capacity, the  
  • Evaluation Readiness Assessment - designed to help assess the degree to which your state is ready to implement research evaluation activities and projects; and 
  • Evaluation Design Assessment - developed to support states in designing a plan for evaluation. 
• The EvalPLC will also exchange ideas to support the creation of other evaluation tools and resources to enhance evaluation capacity for the workforce system.” 

Should Colorado be selected for this opportunity, we will integrate the tools and expertise gained with the implementation plan developed by Northern Illinois University.  

CDLE/WDP will also: 

1. Review existing program evaluation tools and processes 
2. Work with its partner agencies to determine whether additional resources are required, and develop additional methods to evaluate existing data 
3. Develop policies and guidance on how to identify and evaluate potential promising practices, and use evaluation as part of process improvement 
4. Require the use of logic models and outcome measures as part of the development of new projects and proposals 
5. Share information regarding promising practices identified through these activities with the appropriate state or federal entity for further evaluation to determine whether it might be an evidence-based practice 
6. Work with the CWDC, local workforce development boards and state partner agencies to develop and use quasi-research methods to evaluate the effectiveness of core programs and new initiatives, to the extent practicable and within the constraints of existing resources. 

CDLE/WDP will continue to utilize existing methods for program evaluation, such as annual compliance monitoring and data validation, routine tracking of state and local performance and expenditures, and look for ways to incorporate additional outcome measures. CDLE/WDP will work with its state partners to develop training in evaluation techniques for staff and foster a culture of informed inquiry. CDLE/WDP will also work with the local workforce areas to build capacity, and encourage the development and use of additional outcome measures and program evaluation as part of the selection and oversight processes for contracted workforce center operations.