Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. C. Previous Assessment Results
Beginning with the state plan modification in 2018 and for subsequent state plans and state plan modifications, provide the results of assessments of the effectiveness of the core programs and other one-stop partner programs and Combined State Plan partner programs included in the Unified or Combined State plan during the preceding 2-year period (i.e. the 2-year period of the plan modification cycle). Describe how the State is adapting its strategies based on these assessments.
Current Narrative:
Colorado has a commitment to continuous improvement within the talent development network and a focus on the use of data to inform practices and program design. Performance measures and indicators of effectiveness are regularly reviewed at the state and local level and assessment results are utilized to both develop statewide policy that embraces effective work and in the short- and long-term planning of local areas. On an annual basis, the CWDC leads the development with state partners of the Colorado Talent Pipeline Report. This report publishes overall statistics about the system and highlights the practices that have demonstrated results. Policy recommendations are made to the state legislature about opportunities for scaling effective strategies or for reducing barriers that have been identified.
Since the 2018 State Plan modification the biggest adaptation of strategy has been the expansion of work-based learning. This move came from the examination of evidence-based practices in other states and countries and the approach has been further modified based on early results from activities under federal grants. The implementation practices now in use are based on what has been most successful so far in achieving program outcomes.
At the local level, opportunities for planning across programs are provided regularly through conferences and the talentFOUND Quarterly Conversations implemented in 2019. Through these events the results of programming and special initiatives in one local area are shared with others across the state, and time is dedicated for staff and board members to create action plans with partners that will incorporate the results into their local work. Each core program engages in specific actions as well to assess effectiveness and adapt their strategies. Those results are discussed below, and the adaptations for 2020 will be incorporated into the local and regional planning processes that are currently underway in Colorado as of May 2020.
Cost Effectiveness: Colorado has a continuing policy of examining the outcomes and cost effectiveness of all of our programs in order to guide us in this effort. In this analysis we consider both benefits for our clients and cost savings/increased revenue for government entities. In many cases, there will be a reduction in governmental costs due to elimination or reduction of dependence on TANF, Food Stamps, Unemployment Insurance benefits, and other forms of public assistance.
Latest Data - PY19:
Adult Program
- Expenditures - $9,321,861
- Cost per client served - $4,013 (served 2,323)
Youth Program
- Expenditures - $7,907,015
- Cost per client served - $4,135 (served 1,912)
Dislocated Worker Program
- Expenditures - $4,764,265
- Cost per client served - $4,347 (served 1,096)
Workforce Intelligence Data Expert Workgroup (WIDE)
During the implementation of WIOA in 2015, local workforce directors designated their local staff to participate in a state-wide data expert group to ensure consistency of workforce related data. As a result of the partnership, which is a collaboration of Colorado’s Workforce Center System and the Colorado Workforce Development Council, the WIDE group has created systematic changes in how all federally funded workforce programs utilize data as a decision-making tool for program evaluation and design. During PY18, WIDE worked closely with Directors, State MIS and other stakeholders to create a data system that provides actual wage data for WIOA Title I and Title III workforce programs to evaluate the effectiveness of programming through earnings outcomes for any program year back to 2010. In this report, the WIDE group is able to evaluate outcomes utilizing actual median earnings data for any exit cohort based on the following data points:
- 2 Quarters Prior to Program Registration
- Entry Quarter Registration
- 2 Quarters Post Exit
- 4 Quarters Post Exit
- Year 2 Post Exit
- Year 3 Post Exit
- Year 4 Post Exit
Based on the methodology, the following measures were found for the Colorado Workforce System after WIOA implementation:
- Colorado Statewide Cost per Participant for individuals receiving career services through any Title I or III programs.
- Colorado Statewide Cost per Participants for individuals receiving training services through any Title I or III programs.
- In addition, WIDE has compiled the Return on Investment (ROI) for all programs in the previous two bullet points that are outlined in the charts below. The aggregate Return on Investment for all Colorado workforce programs since the implementation of WIOA is $9,623,806,894. This includes program exiters from WIOA Title I and Wagner-Peyser.
Program: Wagner-Peyser
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 280,988 | $10,257,662 | N/A | N/A | N/A |
| PY15 | 258,045 | $10,201,840 | $7,428,262 | $28.79 | $ 776,338,170 |
| PY16 | 228,164 | $10,358,334 | $7,001,327 | $30.69 | $ 887,164,693 |
| PY17 | 203,882 | $10,049,862 | $6,651,025 | $32.62 | $ 915,017,351 |
| PY18 | 191,777 | $10,021,124 | $9,638,516 | $50.26 | $362,447,085 |
| PY19 | 313,338 | $9,822,955 | $10,577,286 | $33.76 | $280,542,153 |
PY20 data available June 2022.
Program: WIOA Adult
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 4,406 | $9,803,977 | $13,732,304 | $3,116.73 | N/A |
| PY15 | 4,152 | $9,328,713 | $14,537,998 | $3,501.44 | $ 34,889,622 |
| PY16 | 3,801 | $8,814,684 | $7,069,618 | $1,859.94 | $ 31,637,846 |
| PY17 | 3,180 | $7,893,417 | $7,268,095 | $2,285.56 | $ 23,315,345 |
| PY18 | 3,099 | $7,397,094 | $10,649,881 | $3,436.55 | $19,562,326 |
| PY19 | 2,327 | $6,664,946 | $9,321,861 | $4,005.96 | $8,587,206 |
PY20 data available June 2022.
Program: WIOA Dislocated Worker
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 2,096 | $12,658,118 | $6,322,090 | $3,016.26 | N/A |
| PY15 | 2,375 | $10,897,869 | $9,238,973 | $3,890.09 | $ 24,890,243 |
| PY16 | 2,065 | $9,858,705 | $7,315,295 | $3,542.52 | $ 28,639,129 |
| PY17 | 1,439 | $8,828,318 | $7,500,999 | $5,212.65 | $ 13,509,673 |
| PY18 | 1,241 | $6,102,421 | $4,455,415 | $3,590.18 | $16,655,302 |
| PY19 | 1,095 | $6,026,154 | $4,764,265 | $4,350.93 | $10,848,278 |
PY20 data available June 2022.
Program: WIOA Youth
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 3,123 | $10,552,245 | $11,106,180 | $3,556.25 | N/A |
| PY15 | 3,404 | $10,059,776 | $11,025,605 | $3,239.01 | $ 702,779 |
| PY16 | 3,086 | $9,505,469 | $7,928,173 | $2,569.08 | $ 5,633,107 |
| PY17 | 2,602 | $8,511,996 | $7,639,064 | $2,935.84 | $ 3,348,200 |
| PY18 | 2,624 | $7,952,674 | $8,828,866 | $3,364.66 | $2,608,266 |
| PY19 | 1,910 | $7,167,065 | $7,907,015 | $4,139.80 | -$646,127* |
*In PY19, more money was spent in the youth area and this population was one of the first to be impacted by COVID-19 layoffs and furloughs when data began to be collected (6 to 12 months after program exit). Therefore, we are beginning to see the effects, but will not see the full impact until June 2022 when all data is reported. Additionally, there were a number of PY19 youth program exiters in March through June of 2020 that either didn’t find employment or return to education during the timeframe.
PY20 data available June 2022.
Latest Data - PY19:
The WIDE group had several other accomplishments during PY19, including work on career pathways to help address the skill gap exacerbated by the COVID-19 pandemic. Research was conducted using data on skill transferability to allow sector partnerships to define training on-ramps and career pathways that take someone with low educational attainment and limited skills who was laid off from the restaurant sector, and upskill them so they can fill the current needs of business, which includes demand for IT professionals and consumer infrastructure occupations.
Program: Adult Education and Family Literacy
In the prior two years, Title II adult education has reviewed individual grantee performance and overall state performance on each of the shared performance outcomes annually once all data matches and federal report submission has been completed. In collaboration with the US Department of Education as part of the Corrective Action Plans, decisions around Request for Application Demonstrated Effectiveness and individual grantee targets were made. For example, Measurable Skill Gains outcomes were lower than the national average and the federal target, so the demonstrated effectiveness requirement was based off of the existing federal target and technical assistance was provided to grantees to help them reach the target. Grantee risk assessments were used to determine targeted technical assistance and which grantees would be on-site monitored.
Title II adult education conducts an internal quarterly review of overall program effectiveness. Colorado Department of Education Office of Adult Education Initiatives (AEI) staff review individual grantee as well as statewide progress toward federal performance targets. AEI staff also informally look at the costs per participant, but since those costs vary so widely based on local context and grantee organization type, AEI does not publish those amounts. Professional learning for both grantee and AEI staff is based off of these quarterly reviews to provide just-in-time support.
Annually AEI utilizes the Data Quality Checklist in the National Reporting System to measure the effectiveness of both grantee and state agency staff data practices. As a result of this review, AEI worked with the MIS vendor, LiteracyPro, to develop a data dashboard within the LACES data system that provided more visibility into assessments, learner demographic data and enrollment data. AEI staff also attend the annual summer National Reporting System training, which provides an opportunity for Colorado to compare itself to other states and share best practices. These trainings focus on continuous improvement, and AEI staff have developed logic models and strategic plans to address targeted performance concerns.
Program: Vocational Rehabilitation
Every three years, the Colorado Division of Vocational Rehabilitation (DVR) and the State Rehabilitation Council (SRC), jointly develop and conduct a comprehensive statewide
assessment of the rehabilitation needs of individuals with disabilities in Colorado. The triennial Comprehensive State Needs Assessment (CSNA) was developed and conducted in the fall of 2018. The results of DVR’s CSNA are described in the Vocational Rehabilitation Services Portion of the Combined State Plan. DVR also takes a comprehensive approach to using data to drive the management of the VR program. DVR has increased its capacity to identify, gather, and analyze critical data to improve services and outcomes for clients, while ensuring the accountability and credibility of VR to all stakeholders. DVR intends to continue this approach of metrics driven leadership to further enhance performance management, quality assurance, and outcomes. This information is used to help shape DVR’s goals, priorities, and strategies to improve vocational rehabilitation services in Colorado.