Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. C. Evaluation
Describe how the State will conduct evaluations and research projects of activities carried out in the State under WIOA core programs; how such projects will be coordinated with, and designed in conjunction with, State and local boards and with State agencies responsible for the administration of all respective core programs; and, further, how the projects will be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA (WIOA Section 116(e )).
Current Narrative:
CDLE contracted with Northern Illinois University (NIU), Center for Government Studies (CGS), to conduct a statewide two-phase evaluation in September 2020. Phase I, the quantitative analysis, NIU used data to verify employment outcomes through Colorado’s MIS and case note review.
Phase II, the qualitative analysis, NIU gathered and synthesized data from focus group interviews in the ten local areas. The teams interviewed primarily consisted of staff who collected, entered, and processed data. NIU submitted a final report focused on the return on investment in training-related placement and provided an overview of the mixed methodology evaluation approach. In addition, the report identifies promising practices in each local area and incorporates recommendations for both CDLE and the ten local areas for future reference.
Workforce Development Programs (WDP) contracted with Northern Illinois University (NIU), October 2021, to provide a deeper understanding of the effectiveness of workforce programs and build the evaluation capacity of state and local personnel.
NIU created The Evaluation Tool intended to independently evaluate outcomes of WIOA programs with an increased emphasis on serving disadvantaged populations and service delivery to improve programs. The Evaluation Tool is an evaluation approach with the objectives of: Identify promising practices that result in a better outcome for program participants and improve the capacity of WDP and Local Workforce Area personnel to conduct independent evaluations.
NIU will train evaluator leads in the ten local areas and WDP staff on using the Evaluation Tool. The local evaluators will work with NIU to determine and analyze recent, higher-quality data and identify outcomes leading to promising practices.
The data will provide demographics of WIOA participants and identify groups of people who underperform compared to the highest performing group, showing a disproportionate impact. In addition, NIU will train local evaluator staff on the application of the disproportionate impact analysis.
With the assistance of NIU, the evaluation leads will learn to craft qualitative analysis questions informed by quantitative data analysis. In addition, NIU will conduct qualitative research using the focus group approach and request local areas to identify staff to recognize promising practices.
In June 2022, NIU released a final report, Performance and Equity in Colorado’s WIOA Programs, A Sequential Mixed-Methods Evaluation. The sequential mixed-method model evaluated program performance, measured equitable outcomes for WIOA Title I participants, and identified best practices for continuous program improvement. The finding and recommendation in the report was presented to CDLE leadership and the local workforce directors. NIU will prepare a final report each year after completion of the annual program evaluation process.
CDLE/WDP will also:
- Review existing program evaluation tools and processes
- Work with its partner agencies to determine whether additional resources are required, and develop additional methods to evaluate existing data
- Develop policies and guidance on how to identify and evaluate potential promising practices, and use evaluation as part of process improvement
- Require the use of logic models and outcome measures as part of the development of new projects and proposals
- Share information regarding promising practices identified through these activities with the appropriate state or federal entity for further evaluation to determine whether it might be an evidence-based practice
- Work with the CWDC, local workforce development boards and state partner agencies to develop and use quasi-research methods to evaluate the effectiveness of core programs and new initiatives, to the extent practicable and within the constraints of existing resources.
CDLE/WDP will continue to utilize existing methods for program evaluation, such as annual compliance monitoring and data validation, routine tracking of state and local performance and expenditures, and look for ways to incorporate additional outcome measures. CDLE/WDP will work with its state partners to develop training in evaluation techniques for staff and foster a culture of informed inquiry. CDLE/WDP will also work with the local workforce areas to build capacity, and encourage the development and use of additional outcome measures and program evaluation as part of the selection and oversight processes for contracted workforce center operations.