Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. C. Previous Assessment Results
Beginning with the state plan modification in 2018 and for subsequent state plans and state plan modifications, provide the results of assessments of the effectiveness of the core programs and other one-stop partner programs and Combined State Plan partner programs included in the Unified or Combined State plan during the preceding 2-year period (i.e. the 2-year period of the plan modification cycle). Describe how the State is adapting its strategies based on these assessments.
Current Narrative:
Colorado has a commitment to continuous improvement within the talent development network and a focus on the use of data to inform practices and program design. Performance measures and indicators of effectiveness are regularly reviewed at the state and local level and assessment results are utilized to both develop statewide policy that embraces effective work and in the short- and long-term planning of local areas. On an annual basis, the CWDC leads the development with state partners of the Colorado Talent Pipeline Report. This report publishes overall statistics about the system and highlights the practices that have demonstrated results. Policy reommendations are made to the state legislature about opportunities for scaling effective strategies or for reducing barriers that have been identified.
Since the 2018 State Plan modification the biggest adaptation of strategy has been the expansion of work-based learning. This move came from the examination of evidence-based pratices in other states and countries and the approach has been further modified based on early results from activities under federal grants. The implementation practices now in use are based on what has been most successful so far in acheiving program outcomes.
At the local level, opportunities for planning across programs are provided regularly through conferences and the talentFOUND Quarterly Conversations implemented in 2019. Through these events the results of programming and special initiatives in one local area are shared with others across the state, and time is dedicated for staff and board members to create action plans with partners that will incorporated the results into their local work. Each core program engages in specific actions as well to assess effectiveness amd adapt their strategies. Those results are discussed below, and the adaptations for 2020 will be incorporated into the local and regional planning processes that are currently underway in Colorado as of May 2020.
Cost Effectiveness: Colorado has a continuing policy of examining the outcomes and cost effectiveness of all of our programs in order to guide us in this effort. In this analysis we consider both benefits for our clients and cost savings/increased revenue for government entities. In many cases, there will be a reduction in governmental costs due to elimination or reduction of dependence on TANF, Food Stamps, Unemployment Insurance benefits, and other forms of public assistance.
Latest Data - PY18:
Adult Program
- Expenditures - $9,404,552
- Cost per client served - $3,028 (served 3,106)
Youth Program
- Expenditures - $6,897,070
- Cost per client served - $2,622 (served 2,630)
Dislocated Worker Program
- Expenditures - $4,797,651
- Cost per client served - $3,860 (served 1,243)
Workforce Intelligence Data Expert Workgroup (WIDE)
During the implementation of WIOA in 2015, local workforce directors designated their local staff to participate in a state-wide data expert group to ensure consistency of workforce related data. As a result of the partnership, which is a collaboration of Colorado’s Workforce Center System and the Colorado Workforce Development Council, the WIDE group has created systematic changes in how all federally funded workforce programs utilize data as a decision-making tool for program evaluation and design. During PY18, WIDE worked closely with Directors, State MIS and other stakeholders to create a data system that provides actual wage data for WIOA Title I and Title III workforce programs to evaluate the effectiveness of programming through earnings outcomes for any program year back to 2010. In this report, the WIDE group is able to evaluate outcomes utilizing actual median earnings data for any exit cohort based on the following data points:
- 2 Quarters Prior to Program Registration
- Entry Quarter Registration
- 2 Quarters Post Exit
- 4 Quarters Post Exit
- Year 2 Post Exit
- Year 3 Post Exit
- Year 4 Post Exit
Based on the methodology, the following measures were found for the Colorado Workforce System after WIOA implementation:
- Colorado Statewide Cost per Participant for individuals receiving career services through any Title I or III programs.
- Colorado Statewide Cost per Participants for individuals receiving training services through any Title I or III programs.
- In addition, WIDE has compiled the Return on Investment (ROI) for all programs in previous two bullet points that are outlined in the charts below. The aggregate Return on Investment for all Colorado workforce programs since the implementation of WIOA is $9,623,806,894. This includes program exiters from WIOA Title I and Wagner-Peyser.
Program: Wagner-Peyser
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 280,988 | $10,257,662 | N/A | N/A | N/A |
| PY15 | 258,045 | $10,201,840 | $7,428,262 | $28.79 | $ 776,338,170 |
| PY16 | 228,164 | $10,358,334 | $7,001,327 | $30.69 | $ 887,164,693 |
| PY17 | 203,882 | $10,049,862 | $6,651,025 | $32.62 | $ 915,017,351 |
PY18 data available April 2020.
Program: WIOA Adult
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 4,406 | $9,803,977 | $13,732,304 | $3,116.73 | N/A |
| PY15 | 4,152 | $9,328,713 | $14,537,998 | $3,501.44 | $ 34,889,622 |
| PY16 | 3,801 | $8,814,684 | $7,069,618 | $1,859.94 | $ 31,637,846 |
| PY17 | 3,180 | $7,893,417 | $7,268,095 | $2,285.56 | $ 23,315,345 |
PY18 data available April 2020.
Program: WIOA Dislocated Worker
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 2,096 | $12,658,118 | $6,322,090 | $3,016.26 | N/A |
| PY15 | 2,375 | $10,897,869 | $9,238,973 | $3,890.09 | $ 24,890,243 |
| PY16 | 2,065 | $9,858,705 | $7,315,295 | $3,542.52 | $ 28,639,129 |
| PY17 | 1,439 | $8,828,318 | $7,500,999 | $5,212.65 | $ 13,509,673 |
PY18 data available April 2020.
Program: WIOA Youth
| Year | Number of Participants | Allocation | Expense | Cost Per Participant | Total ROI |
|---|---|---|---|---|---|
| PY14 | 3,123 | $10,552,245 | $11,106,180 | $3,556.25 | N/A |
| PY15 | 3,404 | $10,059,776 | $11,025,605 | $3,239.01 | $ 702,779 |
| PY16 | 3,086 | $9,505,469 | $7,928,173 | $2,569.08 | $ 5,633,107 |
| PY17 | 2,602 | $8,511,996 | $7,639,064 | $2,935.84 | $ 3,348,200 |
PY18 data available April 2020.
The WIDE group had several other accomplishments during PY18:
- Completed several reports at the request of various stakeholders:
- Report on labor force availability to inform discussion of Colorado’s early childhood education and childcare systemic goals.
- Report on the economic impact of current aggregate student debt in Colorado estimating foregone demand for local goods and services caused by outflow of money for student loan remittance from the state’s economy.
- Report on the four generations currently in the labor force.
- Sundry regional reports designed for local workforce area stakeholders.
- Completed a review of official statewide designations of key industry clusters using the North American Industry Classification System (NAICS) and created profile templates for each of the key industry clusters as defined in Colorado.
- Arranged training for all EMSI users in the state’s workforce development system and provided training to its own members on the new methodology in performing program evaluation using actual earnings data as shown above.
- Is presently working with the Colorado Workforce Development Council to contribute to the Colorado Talent Pipeline Report, which is a legislated requirement for stakeholders in the state’s workforce development ecosystem.
Program: Adult Education and Family Literacy
Title II adult education conducts an internal quarterly review of overall program effectiveness. Colorado Department of Education Office of Adult Education Initiatives (AEI) staff review individual grantee as well as statewide progress toward federal performance targets. AEI staff also informally look at the costs per participant, but since those costs vary so widely based on local context and grantee organization type, AEI does not publish those amounts. Professional learning for both grantee and AEI staff is based off of these quarterly reviews to provide just-in-time support.
Annually AEI utilizes the Data Quality Checklist in the National Reporting System to measure the effectiveness of both grantee and state agency staff data practices. As a result of this review, AEI worked with the MIS vendor, LiteracyPro, to develop a data dashboard within the LACES data system that provided more visibility into assessments, learner demographic data and enrollment data. AEI staff also attend the annual summer National Reporting System training, which provides an opportunity for Colorado to compare itself to other states and share best practices. These trainings focus on continuous improvement, and AEI staff have developed logic models and strategic plans to address targeted performance concerns.
Program: Vocational Rehabilitation
Every three years, the Colorado Division of Vocational Rehabilitation (DVR) and the State Rehabilitation Council (SRC), jointly develop and conduct a comprehensive statewide assessment of the rehabilitation needs of individuals with disabilities in Colorado. The triennial Comprehensive State Needs Assessment (CSNA) was developed and conducted in the fall of 2018. The results of DVR’s CSNA are described in the Vocational Rehabilitation Services Portion of the Combined State Plan. DVR also takes a comprehensive approach to using data to drive the management of the VR program. DVR has increased its capacity to identify, gather, and analyze critical data to improve services and outcomes for clients, while ensuring the accountability and credibility of VR to all stakeholders. DVR intends to continue this approach of metrics driven leadership to further enhance performance management, quality assurance, and outcomes. This information is used to help shape DVR’s goals, priorities, and strategies to improve vocational rehabilitation services in Colorado.