Narrative: III. b. 4. B.

Colorado has a commitment to continuous improvement within the talent development network and a focus on the use of data to inform practices and program design. Performance measures and indicators of effectiveness are regularly reviewed at the state and local level and assessment results are utilized to both develop statewide policy that embraces effective work and in the short- and long-term planning of local areas. On an annual basis, the CWDC leads the development with state partners of the Colorado Talent Pipeline Report. This report publishes overall statistics about the system and highlights the practices that have demonstrated results. Policy recommendations are made to the state legislature about opportunities for scaling effective strategies or for reducing barriers that have been identified. 

Since the 2018 State Plan modification the biggest adaptation of strategy has been the expansion of work-based learning. This move came from the examination of evidence-based practices in other states and countries and the approach has been further modified based on early results from activities under federal grants. The implementation practices now in use are based on what has been most successful so far in achieving program outcomes. 

At the local level, opportunities for planning across programs are provided regularly through conferences and the talentFOUND Quarterly Conversations implemented in 2019. Through these events the results of programming and special initiatives in one local area are shared with others across the state, and time is dedicated for staff and board members to create action plans with partners that will incorporate the results into their local work. Each core program engages in specific actions as well to assess effectiveness and adapt their strategies. Those results are discussed below, and the adaptations for 2020 will be incorporated into the local and regional planning processes that are currently underway in Colorado as of May 2020. 

Cost Effectiveness: Colorado has a continuing policy of examining the outcomes and cost effectiveness of all of our programs in order to guide us in this effort. In this analysis we consider both benefits for our clients and cost savings/increased revenue for government entities.  In many cases, there will be a reduction in governmental costs due to elimination or reduction of dependence on TANF, Food Stamps, Unemployment Insurance benefits, and other forms of public assistance.  

Latest Data - PY22: 

Adult Program 

• Expenditures  - $12,629,390 
• Cost per client served - $3,716 (served 3,399)  

Youth Program 

• Expenditures - $10,542,266 
• Cost per client served - $5,379 (served 1,960) 

Dislocated Worker Program 

• Expenditures - $6,924,351 
• Cost per client served - $4,863 (served 1,424) 

Workforce Intelligence Data Expert Workgroup (WIDE) 

During the implementation of WIOA in 2015, local workforce directors designated their local staff to participate in a statewide data expert group to ensure consistency of workforce-related data. As a result of the partnership, which is a collaboration of Colorado’s Workforce Center System and the Colorado Workforce Development Council, the WIDE group has created systematic changes in how all federally funded workforce programs utilize data as a decision-making tool for program evaluation and design.

WIDE worked closely with directors, State MIS, and other stakeholders to create a data system that provides actual wage data for WIOA Title I and Title III workforce programs to evaluate the effectiveness of programming through earnings outcomes for any program year back to 2010. In this report, the WIDE group is able to evaluate outcomes utilizing actual median earnings data for any exit cohort based on the following data points: 2 Quarters Prior to Program Registration; Entry Quarter Registration; 2 Quarters Post Exit; 4 Quarters Post Exit; Year 2 Post Exit; Year 3 Post Exit; Year 4 Post Exit.

Based on the methodology, the following measures were found for the Colorado Workforce System after WIOA implementation:

• Colorado statewide cost per participant for individuals receiving career services through any Title I or III programs.
• Colorado statewide cost per participant for individuals receiving training services through any Title I or III programs.
• In addition, WIDE has compiled the Return on Investment (ROI) for all programs in the previous two bullet points that are outlined in the charts below.

Please note a data lag in PY data due to the time required to report longitudinal wage data and Unemployment insurance wage reporting periods. This lag is generally 1 program year.  

All Workforce Programs (Including WP & WIOA)

WIOA Adult

WIOA Dislocated Worker

WIOA Youth

In addition to state and local area ROI reporting, the WIDE group has accomplished many other items during PY 21 and PY 22.  There is significant overlap and coordination between the WIDE team and the statewide WIOA evaluation workgroup facilitating the completion of  needed evaluation activities.  WIDE supported the CWDC and CDLE on a number of reports including the WIOA Annual Report, and data contained in some parts of the Talent Pipeline reports. Additionally, WIDE supports the CWDC on grant and funding requests including providing data for the Tech Hub application requests that were awarded in fall of 2023 data for Colorado’s behavioral health industry, and data for OEDIT and their regional roadmap planning efforts for the EDA.  WIDE is currently looking into a collaboration to create  of a white paper on electric fleet vehicles  to support CDOT, and is in the process of developing f a tool to quantify  the value of and determine the return on investment of business services efforts.  WIDE continues to collaborate and  share best practices among local regions and continues to provide  the CWDC with reliable, detailed  labor market and economic data. 

Program: Adult Education and Family Literacy 

Title II adult education conducts an internal quarterly review of overall program effectiveness. Colorado Department of Education Office of Adult Education Initiatives (AEI) staff review individual grantee as well as statewide progress toward federal performance targets. AEI staff also informally look at the costs per participant, but since those costs vary so widely based on local context and grantee organization type, AEI does not publish those amounts. Professional learning for both grantee and AEI staff is based off of these quarterly reviews to provide just-in-time support. 

Annually AEI utilizes the Data Quality Checklist in the National Reporting System to measure the effectiveness of both grantee and state agency staff data practices. As a result of this review, AEI worked with the MIS vendor, LiteracyPro, to develop a data dashboard within the LACES data system that provided more visibility into assessments, learner demographic data and enrollment data. AEI staff also attend the annual summer National Reporting System training, which provides an opportunity for Colorado to compare itself to other states and share best practices. These trainings focus on continuous improvement, and AEI staff have developed logic models and strategic plans to address targeted performance concerns. 

Program: Vocational Rehabilitation 

Every three years, the Colorado Division of Vocational Rehabilitation (DVR) and the State Rehabilitation Council (SRC), jointly develop and conduct a comprehensive statewide assessment of the rehabilitation needs of individuals with disabilities in Colorado. The triennial Comprehensive State Needs Assessment (CSNA) was developed and conducted in the fall of 2021, with additional planning and assessment activities completed throughout 2022. The results of DVR’s CSNA are described in the Vocational Rehabilitation Services Portion of the Combined State Plan. DVR also engages in ongoing and comprehensive review of data to monitor and evaluate program performance, including several key performance indicators in addition to the WIOA common performance measures. DVR monitors progress on all common performance measures throughout the year to ensure the program is on track to meet performance targets. 

DVR surpassed all negotiated targets, with the exception of Measurable Skills Gains. DVR continues to explore the challenges with meeting this accountability measure. Initially, monitoring overall progress throughout the year was a challenge due to changes to technical specifications in how this measure is calculated. DVR sought and received technical assistance from RSA to ensure internal tracking of this measure is consistent with RSA’s calculations and recent adjustments to DVR’s internal report specifications have supported more accurate monitoring of this measurement. Additionally, collecting timely and accurate data to document measurable skills gains is an ongoing challenge. While DVR partners closely with local education areas and post-secondary education providers, obtaining needed records timely from the training entities or participants is a challenge, as is ensuring consistent data entry. Both the technical requirements and business processes within DVR’s case management system, as well as complicated specifications regarding the calculation of this measure create barriers to accurate reporting. This is an area where DVR continues to put effort.