Narrative: III. b. 4. A.

Adult, Dislocated Worker, Youth, and Wagner-Peyser

In accordance with Colorado’s philosophy of local control, each local area designs and conducts its own evaluation. Additionally, the Colorado Department of Labor and Employment (CDLE) evaluates each local area’s outcomes by utilizing the performance accountability measures described in 116(b) of WIOA by updating its Connecting Colorado on-line reporting system. CDLE provides tools to each local area that enable them to track their performance on these measures as the year progresses and to track the impact on performance of each of their participants. In addition to local performance evaluation, CDLE uses a variety of assessment tools to evaluate and track program effectiveness and to promote a culture of continuous improvement, including:

Connecting Colorado Data/Reporting System: Connecting Colorado enables local workforce areas and their partners to track WIOA Title I-B, TAA, Veteran, Wagner-Peyser, state and local programs, and discretionary grants in a single database. Reports generated from this system are “up to the minute,” which allows better management of daily activities and performance. In addition to federally-mandated reports, CDLE has developed a variety of locally requested reports that allow staff to track caseloads, correct data entry errors, and analyze performance at the state, local and desk level. The reports measure levels of service and outcomes by program or targeted groups, provide the ability to measure the impact of outreach programs and specialized initiatives on performance, and can be easily adapted to add new reports and coding to accommodate new programs and initiatives. Core programs and required partners may access and use the system through data sharing agreements with the state or the local workforce areas. In PY 14, Connecting Colorado added new modules to track and report on employer-focused activities. These modules link employer services with system-generated surveys to obtain information on customer satisfaction, effectiveness and service outcomes. The new reporting capability was recognized in a White House Paper, “Ready to Work: Job-Driven Training and American Opportunity,” as a best practice.

Program Monitoring: CDLE’s comprehensive monitoring guide, recognized as a model by the US Department of Labor (USDOL), is the result of an ongoing monitoring and technical assistance process that has built a strong working relationship between the state and local workforce areas. Quarterly, the State Monitoring team from CDLE meets with the local areas to review client service levels, program expenditure rates, and performance outcome results. Through these reviews, staff identifies areas that may require additional technical assistance and training to identify risk areas and rectify any deficiencies prior to the start of the annual state compliance monitoring. CDLE implemented a new tiered risk assessment policy (PY16) specifically for discretionary grants that will determine the criteria for compliance issues and draw on promising practices in grants management to prevent performance issues.

Return on Investment (ROI): The CWDC, CDLE Labor Market Information and Workforce Development Programs sections, and the local workforce areas have used economic impact evaluations for several years to determine the economic benefits and fiscal contribution of individuals who receive services through the workforce system, obtain employment, and are working six months later. Colorado will explore strategies to enhance the model to improve its effectiveness, expand it to include partner programs, and explore other ROI measurements such as the cost and benefits derived from reduced reliance on various forms of public assistance (e.g. Unemployment insurance, Colorado Works).

Continuous Improvement Management System (CIMS): The CWDC has worked in partnership with the local boards and USDOL to develop a performance incentive system for workforce areas that meets the needs of the local area and ensures a high level of quality and continuous improvement across the state. The Continuous Improvement Management System (CIMS) is aligned with the Colorado Blueprint and CWDC strategic initiatives, and rewards local areas for pursuing:

• A dual customer approach that emphasizes a focus on business needs
• Strategies/partnerships to utilize industry sectors as the platform for increasing the number and level of training and educational opportunities, as well as for increasing the number of credentials achieved
• Partnership initiatives with economic development, chambers of commerce, and small business organizations
• Participation in the Key Industry Taskforce with the Governor’s Office of Economic Development and International Trade
• Participation in the Statewide Business Services Alignment Initiative

CWDC and CDLE’s Workforce Development Programs develop the local planning guidelines for workforce areas and review the local plans submitted each year. The alignment of all strategic planning efforts combined with implementation of continuous improvement efforts helps maintain an effective and efficient statewide workforce system that embodies innovation and excellent customer service.

Evidence Based Practice and Program Evaluation: CDLE is developing new policies and procedures to facilitate ongoing evaluation and research on current and new programs, including state initiatives, and to identify best practices that have the potential to become evidence based practice. This includes the development and use of logic models to establish benchmarks and outcome measures. CDLE will scale pilots that show promise to a statewide basis, or replicate the pilot through partner agencies who serve similar participants; similar evaluation methods will be used to determine whether the program warrants consideration as an Evidence Based Practice. To support this work, Colorado is working with national experts at Results For America to implement an evidence continuum that defines the distinctions in concepts and practices that are: data informed, data driven, promising practice, proven practice, and evidence based practice. In addition to national expertise, the CWDC will work to coordinate efforts with the Colorado Evidence-Based Policy Collaborative. Evaluation will also help identify and address areas such as gaps in services, program inefficiencies, duplication of effort, etc.

Evidence Based Practice Academy

To promote and accelerate the state evaluation process and activities, CDLE/Workforce Development Programs has contracted with Northern Illinois University to achieve the following outcomes:

“Northern Illinois University's Center for Governmental Studies (CGS) is proposing to work with the Colorado Department of Labor and Employment (CDLE) to design an Evidence-Based Practice (EBP) Academy for local and state-level workforce practitioners. Over a seven-week period, CGS will confer with a broad cross-section of Colorado workforce professionals and stakeholders regarding the feasibility, benefits, and challenges of creating an EBP Academy or other focused training opportunity. At the conclusion of this project, CDLE will be provided with a recommended structure, learning objectives, curriculum and suggested implementation plan for a Colorado EBP Academy based on information collected from prospective Academy participants and other Colorado workforce system stakeholders.” Contract work was completed during PY19 and the recommendation plan was implemented through a contract with the University of Denver.

USDOL Evaluation Peer Learning Cohort

On November 1, 2019, Colorado submitted its application to become part of the The Evaluation Peer Learning Cohort (EvalPLC) which is an interactive technical assistance forum and comprised of cross-agency representatives from four to six states. State teams who represent core WIOA programs will collaborate and develop evaluation action plans designed to conduct and implement state and/or local research and evaluations.  

Participants will:

• Learn from experienced evaluators, state and national subject matter experts (SMEs), and each other as they develop an evaluation plan. 
• Use the Evaluation Toolkit: Key Elements for State Workforce Agencies and two evaluation assessment tools to build evaluation capacity, the
  • Evaluation Readiness Assessment - designed to help assess the degree to which your state is ready to implement research evaluation activities and projects; and
  • Evaluation Design Assessment - developed to support states in designing a plan for evaluation.
• The EvalPLC will also exchange ideas to support the creation of other evaluation tools and resources to enhance evaluation capacity for the workforce system.”

Should Colorado be selected for this opportunity, we will integrate the tools and expertise gained with the implementation plan developed by Northern Illinois University. 

Data Coordination and Information Sharing: CWDC, CDLE and program partners will develop ways to share information across programs that take into account the federal and state rules that govern data collection and sharing and the release of information for each program. Colorado will then develop and implement the most effective and efficient solution for data coordination and information sharing, and use the resulting data to facilitate core program evaluations.

Adult Education and Family Literacy

As the entity responsible for management and oversight of Title II funds, the Colorado Department of Education (CDE) Office of Adult Education Initiatives (AEI) will develop and implement grantee monitoring and oversight consistent with the goals of the Colorado State Plan to emphasize quality, effectiveness, and  program improvement. The monitoring and oversight of adult education and family literacy programs in Colorado will help to ensure that programs will continue to demonstrate their effectiveness. This will include review of performance against the federally negotiated performance targets and program implementation and success metrics outline in teh grant request for proposal and subsequent continuation applications. 

Adult education and family literacy providers funded through Title II will be required to file comprehensive annual performance reports detailing both quantitative and qualitative results. Annual statistical reports from the information management system will track numbers of adult learners, demographic data such as sex and race/ethnicity, level of education, educational functioning level, and most importantly, progress at the local provider level.

AEI will provide technical assistance both on a universal level and customized for individual programmatic needs, as deemed necessary based on quarterly reviews.

Vocational Rehabilitation

DVR remains in a baseline period for all performance measures outlined in WIOA section 116 and anticipates beginning to negotiate targets for PY20. However, DVR reviews current performance on a quarterly basis, analyzing performance on common performance measures, in addition to other relevant data points such as employment outcomes, wages at closure, timeliness of eligible and plan development, etc, The performance reports can be run on a statewide, regional, or local basis to provide a robust picture of performance throughout the state. To help inform the overall quality of the services being delivered, a customer satisfaction surveys are available and results are reviewed quarterly to guide work moving forward. Finally, a Comprehensive Statewide Needs Assessment is conducted every three years, which includes a survey of staff members, clients, and stakeholders to provide ongoing information and data points about service delivery.