Narrative: III. b. 4. A.

Adult, Dislocated Worker, Youth, and Wagner-Peyser 

In accordance with Colorado’s philosophy of local control, each local area may design and conduct its own evaluation. Additionally, the Colorado Department of Labor and Employment (CDLE) evaluates each local area’s outcomes by utilizing the performance accountability measures described in 116(b) of WIOA by updating its Connecting Colorado on-line reporting system. CDLE provides tools to each local area that enable them to track their performance on these measures as the year progresses and to track the impact on performance of each of their participants. In addition to local performance evaluation, CDLE uses a variety of assessment tools to evaluate and track program effectiveness and to promote a culture of continuous improvement, including: 

Connecting Colorado Data/Reporting System: Connecting Colorado enables local workforce areas and their partners to track WIOA Title I-B, TAA, Veteran, Wagner-Peyser, state and local programs, and discretionary grants in a single database. Reports generated from this system are “up to the minute,” which allows better management of daily activities and performance. In addition to federally-mandated reports, CDLE has developed a variety of locally requested reports that allow staff to track caseloads, correct data entry errors, and analyze performance at the state, local and desk level. The reports measure levels of service and outcomes by program or targeted groups, provide the ability to measure the impact of outreach programs and specialized initiatives on performance, and can be easily adapted to add new reports and coding to accommodate new programs and initiatives. Core programs and required partners may access and use the system through data sharing agreements with the state or the local workforce areas. Connecting Colorado has modules to track and report on employer-focused activities. These modules link employer services with system-generated surveys to obtain information on customer satisfaction, effectiveness and service outcomes. The reporting capability has been recognized in a White House Paper, “Ready to Work: Job-Driven Training and American Opportunity,” as a best practice. 

Program Monitoring: CDLE’s comprehensive monitoring guide, recognized as a model by the US Department of Labor (USDOL), is the result of an ongoing monitoring and technical assistance process that has built a strong working relationship between the state and local workforce areas. Quarterly, the State Monitoring team from CDLE meets with the local areas to review client service levels, program expenditure rates, and performance outcome results. Through these reviews, staff identifies areas that may require additional technical assistance and training to identify risk areas and rectify any deficiencies prior to the start of the annual state compliance monitoring. CDLE implemented a new tiered risk assessment policy specifically for discretionary grants that will determine the criteria for compliance issues and draw on promising practices in grants management to prevent performance issues. 

Return on Investment (ROI): The CWDC, CDLE Labor Market Information, and Workforce Development Programs sections, and the local workforce areas have used economic impact evaluations for several years to determine the economic benefits and fiscal contribution of individuals who receive services through the workforce system, obtain employment, and are working six months later. Colorado will explore strategies to enhance the model to improve its effectiveness, expand it to include partner programs, and explore other ROI measurements such as the cost and benefits derived from reduced reliance on various forms of public assistance (e.g. Unemployment insurance, Colorado Works). 

Continuous Improvement Management System (CIMS): The CWDC has worked in partnership with the local boards and USDOL to develop a performance incentive system for workforce areas that meets the needs of the local area and ensures a high level of quality and continuous improvement across the state. The Continuous Improvement Management System (CIMS) is aligned with CWDC strategic initiatives, and rewards local areas for pursuing: 

• A dual customer approach that emphasizes a focus on business needs 
• Strategies/partnerships to utilize industry sectors as the platform for increasing the number and level of training and educational opportunities, as well as for increasing the number of credentials achieved 
• Partnership initiatives with economic development, chambers of commerce, and small business organizations 
•  
• Participation in the Statewide Business Services Alignment Initiative 

CWDC and CDLE’s Workforce Development Programs develop the local planning guidelines for workforce areas and review the local plans submitted each year. The alignment of all strategic planning efforts combined with implementation of continuous improvement efforts helps maintain an effective and efficient statewide workforce system that embodies innovation and excellent customer service. 

Evidence Based Practice and Program Evaluation: CDLE is developing new policies and procedures to facilitate ongoing evaluation and research on current and new programs, including state initiatives, and to identify best practices that have the potential to become evidence based practice. This includes the development and use of logic models to establish benchmarks and outcome measures. CDLE will scale pilots that show promise to a statewide basis, or replicate the pilot through partner agencies who serve similar participants; similar evaluation methods will be used to determine whether the program warrants consideration as an Evidence Based Practice. To support this work, Colorado has worked with national experts at Results For America to implement an evidence continuum that defines the distinctions in concepts and practices that are: data informed, data driven, promising practice, proven practice, and evidence based practice. Evaluation will also help identify and address areas such as gaps in services, program inefficiencies, duplication of effort, etc. 

WIOA Evaluation Lead Training

Workforce Development Programs (WDP) contracted with Northern Illinois University (NIU), October 2021, to provide a deeper understanding of the effectiveness of workforce programs and build the evaluation capacity of state and local personnel.

NIU created The Evaluation Tool intended to independently evaluate outcomes of WIOA programs with an increased emphasis on serving disadvantaged populations and service delivery to improve programs. The Evaluation Tool is an evaluation approach with the objectives of: Identify promising practices that result in a better outcome for program participants, and improve the capacity of WDP and Local Workforce Area personnel to conduct independent evaluations.

NIU is training evaluator leads in the ten local areas and WDP staff on using the Evaluation Tool. The data will provide demographics of WIOA participants and identify groups of people who underperform compared to the highest performing group, showing a disproportionate impact. In addition, NIU will continue to train local evaluator staff on the application of the disproportionate impact analysis.

With the assistance of NIU, the evaluation leads have learned to craft qualitative analysis questions informed by quantitative data analysis. In addition, NIU will continue to conduct qualitative research using the focus group approach and request local areas to identify staff to recognize promising practices.

In June 2022, NIU released a final report, Performance and Equity in Colorado’s WIOA Programs, A Sequential Mixed-Methods Evaluation. The sequential mixed-method model evaluated program performance, measured equitable outcomes for WIOA Title I participants, and identified best practices for continuous program improvement.  The findings and recommendations in the report were presented to the CDLE leadership and the local workforce directors. NIU will prepare a final report each year after completion of the annual program evaluation effort. 

To build on the momentum of the Performance and Equity in Colorado’s WIOA Programs, WDP contracted with NIU, program year (PY)2023, to continue training the evaluation leads in the ten local areas and WDP staff on using the Evaluation Tool and conducting local evaluations using the sequential mixed-method evaluation. This approach incorporates quantitative and qualitative methods in analyzing the most current Participant Individual Report Layout (PIRL) data. 

The local evaluators are building their evaluation skills by analyzing their findings. They will submit yearly reports using quantitative analysis to interpret patterns of potentially inequitable outcomes based on participant characteristics, services, and outcomes. Following the quantitative analysis, the local evaluators will begin the qualitative analysis using a customer survey, and staff interviews to identify trends and promising practices in the area. These reports will help contribute to the more extensive statewide report, providing valuable insight into the WIOA Title I program on both a macro and micro level.

USDOL Evaluation Peer Learning Cohort 

WDP was awarded the EvalPLC to create an evaluation to develop an Evaluation Design and Implementation Assessment (EDA) tool to improve services for fostered and disconnected youth. With technical assistance from the DOL, the team completed the EDA tool, which identified areas of improvement, developing the capacity to share data across systems and creating an evaluation methodology and timeline to ensure we can provide the best possible services to this population. 

The Evaluation Peer Learning Cohort (EvalPLC) was an interactive technical assistance forum of cross-agency representatives from four to six states. State teams who represent core WIOA programs collaborated and developed evaluation action plans designed to conduct and implement state and/or local research and evaluations.   

Participants: 

• Learned from experienced evaluators, state and national subject matter experts (SMEs), and each other as they developed an evaluation plan.  
• Used the Evaluation Toolkit: Key Elements for State Workforce Agencies and two evaluation assessment tools to build evaluation capacity, the  
  • Evaluation Readiness Assessment - designed to help assess the degree to which your state is ready to implement research evaluation activities and projects; and 
  • Evaluation Design Assessment - developed to support states in designing a plan for evaluation. 
• The EvalPLC  also exchanged ideas to support the creation of other evaluation tools and resources to enhance evaluation capacity for the workforce system.” 

Colorado presented its Performance and Equity in Colorado’s WIOA Programs report at one of the EvalPLC sessions.

National Recognition for Colorado’s Evaluation Initiative: 

• The Performance and Equity in Colorado's WIOA Programs evaluation report noted above was presented at the WIOA Performance Reporting and Accountability Convening in Washington, D.C., on September 12 and 13 by Elise Lowe-Vaughn, Director of Workforce Development Programs, Barbara L. Larsen, Business Support Analyst, Workforce Boulder County, and Brian Richard, Ph.D.Assistant Director, Workforce Development Center for Governmental Studies, Northern Illinois University, recent convening in Washington, D.C.
• CDLE/WDP staff, local area representatives, and Northern Illinois University (NIU) WIOA Evaluation Team were invited to present at the Workforce GPS live event, Preparing for Equity-Focused Evaluation in Workforce Development, on May 22, 2023. The team presented  The Performance and Equity in Colorado’s WIOA Programs report. Over 800 people from across the country were in attendance.

Adult Education and Family Literacy 

As the entity responsible for management and oversight of Title II funds, the Colorado Department of Education (CDE) Office of Adult Education Initiatives (AEI) will develop and implement grantee monitoring and oversight consistent with the goals of the Colorado State Plan to emphasize quality, effectiveness, and  program improvement. The monitoring and oversight of adult education and family literacy programs in Colorado will help to ensure that programs will continue to demonstrate their effectiveness. This will include review of performance against the federally negotiated performance targets and program implementation and success metrics outlined in the grant request for proposal and subsequent continuation applications.  

Adult education and family literacy providers funded through Title II will be required to file comprehensive annual performance reports detailing both quantitative and qualitative results. Annual statistical reports from the information management system will track numbers of adult learners, demographic data such as sex and race/ethnicity, level of education, educational functioning level, and most importantly, progress at the local provider level. 

AEI will provide technical assistance both on a universal level and customized for individual programmatic needs, as deemed necessary based on quarterly reviews. 

Vocational Rehabilitation 

DVR reviews current performance on a monthly basis, analyzing performance on common performance measures, in addition to other relevant data points such as employment outcomes, wages at closure, timeliness of eligible and plan development, etc. The performance reports are provided on a statewide, regional, or local basis to provide a robust picture of performance throughout the state. To help inform the overall quality of the services being delivered, customer satisfaction surveys are available and results are reviewed to guide work moving forward. Routine quality assurance reviews are conducted, exploring the quality of case work and compliance with key policies and procedures. Results are shared quarterly with DVR field leadership to help support continuous improvement. The State Rehabilitation Council also completes customer satisfaction surveys, in addition to the data collected by DVR,  and reviews key performance metrics quarterly, using this information to inform their recommendations to DVR.  Finally, a Comprehensive Statewide Needs Assessment is conducted every three years, which includes a survey of staff members, clients, and stakeholders to provide ongoing information and data points about service delivery. 

ASSESSMENT OF ONE-STOP PARTNER PROGRAMS 

At a minimum, workforce delivery system partner program services, and the partner programs included in this Combined State Plan, will be assessed using performance measures they have in common, in addition to other performance accountability measures required by each program’s funding source(s) and grants. Performance will be reviewed at both the state and the local workforce area level. 

SCSEP 

GPMS (Grantee Performance Management System) is used for SCSEP data collection and client case management.  It's the primary source for entering, tracking and reporting data captured for participant cases.  Participant data is then fed into the quarterly and annual Quarterly Progress Report (QPR) through the WIOA Integrated Performance System (WIPS).  The main functional areas of the system are:  user management, case management, eligibility verification, services management, outcome management, exit management, follow-up management and reporting.

Colorado Works

Through the Federal TANF program, the State is held accountable to the Federal Work Participation Rate (FWPR). This rate is calculated by the Federal Administration of Children and Families within US Health and Human Services, and states are expected to meet or exceed performance targets set on an annual basis. The performance rate is established through the analysis of CBMS data for countable participants, and their ability to meet the federally required hours of participation in countable activities. This information is reviewed and managed by county and State human service staff, and may also be a performance indicator for workforce development sub-contractors who have a formal agreement for service provision through the local human service department in their area. CDHS is held accountable to this rate.

Colorado has further promoted employment entry as a meaningful outcome for program participants, and implemented system changes to enable the tracking of employment. A target rate for employment entry has been established in the Colorado Department of Human Services’ 5-Year Strategic Plan, and is expected to be 35% or better for the current performance year. The rate is calculated through the attainment of certain outcomes identified in CBMS. This goal is tracked monthly through the Department’s performance management system, known as C-Stat. 

TAA 

On July 1, 2022, the TAA program entered a phased termination while awaiting reauthorization.  During the termination phase, the TAA program will report performance under the WIOA measures used by core and partner programs. The TAA program currently reports all performance measures via the USDOL mandated quarterly PIRL. 

In addition to the PIRL report, USDOL has implemented the TAA Data Integrity (TAADI) report card that compares the financial and performance measures reported on the PIRL and the ETA 9130. If a state has any discrepancies between the two reports or any objectives that are not met, then the state must respond in writing on why the objective was not met and how it will be improved or fixed. The TAADI report card has provided areas of opportunity for improvement with the TAA program and will guide continued efforts in the future. All case files at the local workforce center are monitored on an annual basis by CDLE’s Workforce Development Programs Regional Services Team. The TAA program utilizes the monitoring reviews to discover trends within the program and provide additional training to the regional case managers. In addition to annual monitoring, TAA files are verified by USDOL’s annual Data Integrity review. 

Priority of Service for Veterans 

The demonstration of Priority of Service is the responsibility of American Job Center (AJC) staff. It is also the responsibility of the State and County leadership to ensure that Priority of Service is being successfully utilized by providing training, technical assistance, and monitoring accomplished by CDLE. CDLE staff will assist with these items are the Regional Services Unit (RSU), Local Veteran Employment Representative (LVER), Consolidated Veteran Services Representatives (CVSR: consolidated DVOP/LVER), CDLE Monitors, State Veterans Program Coordinator, CDLE Regional Directors, and Veteran Program Managers. 

JVSG funded staff are not responsible for applying priority of service but will have a foundational understanding of how the measure is applied and calculated and LVERs/CVSRs will meet with local AJC staff regularly during the quarter to assist with assessing POS outcomes and assist with modifying processes, as needed. 

Methods and means used to verify whether or not Priority of Service is being provided include the following but are not limited to: 

• Vet Priority of Service Report
• Interviewing of AJC staff 
• State monitoring tool 
• Review of program files and documentation 
• Customer surveys 
• Site visits 
• Accompanying DVET during federal audits 
• Quarterly Managers Report 

CDLE and USDOL/VETS monitor and review the provision of services to veterans at the state and local levels tri-annually or as directed, and provide the Executive Director, local directors, and the local workforce center manager with the findings, recommendations, and/or corrective actions. LVERs and CVSRs will conduct Priority of Service overview and training annually or as needed for additional training in their local area. A Priority of Service PowerPoint is available to the AJC regions as a training tool. This PowerPoint, in the absence of a LVER or CVSR, can be used for training purposes of AJC new hires. As a condition of receiving any USDOL funds for programs, local areas will provide Priority of Service to veterans and covered persons. This annual agreement is a contract between the State of Colorado and the local areas. The contract must be signed by both the state and local areas as a condition to receive all US Department of Labor funds. 

JVSG

JVSG reporting and evaluation are reported to USDOL VETS through the submission of Quarterly Reports. These reports include the following :

• VETS 402 Expenditure Detail Report
• VETS 403 Technical Performance Narrative
• ETA 9173
• Quarterly Managers Report
• WIOA Quarterly Performance Reports
• Standard Form (SF) 425 Federal Financial Report

These reports include information from LVERs/CVSRs, CDLE accounting, and Connecting Colorado data, and are complied by the JVSG State Veterans Coordinator. The data from the reports are evaluated by JVSG Regional Directors, the JVSG State Veterans Coordinator, and CDLE leadership.

MSFW 

The MSFW programs, as required by US Department of Labor (DOL) regulations, work closely with partner agencies to provide information and referrals to our shared customer base. Each significant office hands out MSFW packets to farm workers during outreach activities. These informational packets include resources for English as a Second Language (ESL), Migrant Education Program (MEP) and training opportunities available through the WIOA 167 provider (Rocky Mountain SER). The agency Memorandums of Understanding (MOUs) have provided a way to track some of our referrals between agencies. MOUs have been in place with two of our close partners. The Connecting Colorado (CC) database is used in part to collect data from referrals that have been made by Workforce agencies and outreach workers. The data entered in CC is used for monitoring purposes to ensure qualitative and quantitative proportionate referrals provided to the MSFW population 

The overall program performance measures include: Individuals Place in a Job; Median Earnings, Individuals placed in Long-Term non-Ag jobs; and Employment Rate. Receipt of the following referrals: Basic Career Services; Staff assisted Job Search; Staff Assisted Career Guidance; Unemployment Insurance Claim Assistance; Referred to Employment; Referred to Federal Training, Federal or State Assistance; and Individual Career Services. These performance measures are reviewed quarterly and targets are adjusted based on the economic conditions of Colorado. MOU referrals are also reviewed on a quarterly basis and discussed in meetings with our partners for reporting and evaluation purposes.