Narrative: III. b. 4. A.

Adult, Dislocated Worker, Youth, and Wagner–Peyser

In accordance with Colorado’s philosophy of local control, each local area designs and conducts its own evaluation. Additionally, the Colorado Department of Labor and Employment (CDLE) evaluates each local area’s outcomes by utilizing the Common Measures of performance for adults, dislocated workers, and youth programs, as outlined in TEGL 17–05, and is preparing to utilize the performance accountability measures described in 116(b) of WIOA by updating its Connecting Colorado on–line reporting system. CDLE provides tools to each local area that enable them to track their performance on these measures as the year progresses and to track the impact on performance of each of their participants.

In addition to local performance evaluation, CDLE uses a variety of assessment tools to evaluate and track program effectiveness and to promote a culture of continuous improvement, including:

Connecting Colorado Data/Reporting System – Connecting Colorado enables local workforce areas and their partners to track WIOA Title I–B, TAA, Veteran, Wagner–Peyser, state and local programs, and discretionary grants in a single database. Reports generated from this system are “up to the minute,” which allows better management of daily activities and performance. In addition to federally–mandated reports, CDLE has developed a variety of locally requested reports that allow staff to track caseloads, correct data entry errors, and analyze performance at the state, local and desk level. The reports measure levels of service and outcomes by program or targeted groups, provide the ability to measure the impact of outreach programs and specialized initiatives on performance, and can be easily adapted to add new reports and coding to accommodate new programs and initiatives. Core programs and required partners may access and use the system through data sharing agreements with the state or the local workforce areas.

In PY 14, Connecting Colorado added new modules to track and report on employer–focused activities. These modules link employer services with system–generated surveys to obtain information on customer satisfaction, effectiveness and service outcomes. The new reporting capability was recognized in a White House Paper, “Ready to Work: Job–Driven Training and American Opportunity,” as a best practice.

Program Monitoring – CDLE’s comprehensive monitoring guide, recognized as a model by the US Department of Labor (USDOL), is the result of an ongoing monitoring and technical assistance process that has built a strong working relationship between the state and local workforce areas. Quarterly, the State Monitoring team from CDLE meets with the local areas to review client service levels, program expenditure rates, and performance outcome results. Through these reviews, staff identifies areas that may require additional technical assistance and training to identify risk areas and rectify any deficiencies prior to the start of the annual state compliance monitoring. In PY16, CDLE will implement a new tiered risk assessment policy specifically for discretionary grants that will determine the criteria for compliance issues and draw on promising practices in grants management to prevent performance issues.

Return on Investment (ROI) – The CWDC, CDLE Labor Market Information and Workforce Development Programs sections, and the local workforce areas have used economic impact evaluations for several years to determine the economic benefits and fiscal contribution of individuals who receive services through the workforce system, obtain employment, and are working six months later. Colorado will explore strategies to enhance the model to improve its effectiveness, expand it to include partner programs, and explore other ROI measurements such as the cost and benefits derived from reduced reliance on various forms of public assistance (e.g. Unemployment insurance, Colorado Works).

Continuous Improvement Management System (CIMS) – The CWDC has worked in partnership with the local boards and USDOL to develop a performance incentive system for workforce areas that meets the needs of the local area and ensures a high level of quality and continuous improvement across the state. The Continuous Improvement Management System (CIMS) is aligned with the Colorado Blueprint and CWDC strategic initiatives, and rewards local areas for pursuing:

• A dual customer approach that emphasizes a focus on business needs
• Strategies/partnerships to utilize industry sectors as the platform for increasing the number and level of training and educational opportunities, as well as for increasing the number of credentials achieved
• Partnership initiatives with economic development, chambers of commerce, and small business organizations
• Participation in the Key Industry Taskforce with the Governor’s Office of Economic Development and International Trade
• Participation in the Statewide Business Services Alignment Initiative

CWDC and CDLE’s Workforce Development Programs develop the local planning guidelines for workforce areas and review the local plans submitted each year. The alignment of all strategic planning efforts combined with implementation of continuous improvement efforts helps maintain an effective and efficient statewide workforce system that embodies innovation and excellent customer service.

Evidence Based Practice and Program Evaluation – CDLE is developing new policies and procedures to facilitate ongoing evaluation and research on current and new programs, including state initiatives, and to identify best practices that have the potential to become evidence based practice. This includes the development and use of logic models to establish benchmarks and outcome measures. CDLE will scale pilots that show promise to a statewide basis, or replicate the pilot through partner agencies who serve similar participants; similar evaluation methods will be used to determine whether the program warrants consideration as an Evidence Based Practice. Evaluation will also help identify and address areas such as gaps in services, program inefficiencies, duplication of effort, etc.

Data Coordination and Information Sharing – CWDC, CDLE and program partners will develop ways to share information across programs that take into account the federal and state rules that govern data collection and sharing and the release of information for each program. Colorado will then develop and implement the most effective and efficient solution for data coordination and information sharing, and use the resulting data to facilitate core program evaluations.

Adult Education and Family Literacy

As the entity responsible for management and oversight of Title II funds, the Colorado Department of Education (CDE) Office of Adult Education Initiatives (AEI) will develop and implement assessment consistent with the goals of the Colorado State Plan to emphasize quality, effectiveness, and improvement of programs at the provider level. The assessment of adult education and family literacy programs in Colorado will help to ensure that programs will continue to demonstrate their effectiveness.

Assessment of adult education and family literacy providers funded through Title II will include performance consistent with the negotiated performance measures for adult education and family literacy. Additionally, programming will be assessed based on effective utilization of evidence–based practices associated with teaching learning, as documented in local provider proposals or applications for grant funding, for career pathways and family literacy purposes.AEI will design a rubric to be used for programmatic assessment that will account for quality of instructional design and delivery, effectiveness of practices, and continuous improvement. In the competitive grant process, applicants will be required to submit plans for implementing evidence–based instructional design and continuous improvement, which will be used for consideration of funding under Title II.

Adult education and family literacy providers funded through Title II will be required to file comprehensive annual performance reports detailing both quantitative and qualitative results. Annual statistical reports from the information management system will track numbers of adult learners, demographic data such as sex and race/ethnicity, level of education, educational functioning level, and most importantly, progress at the local provider level.

The instructional performance measures will focus on continuous improvement of adult education and family literacy activities and are evaluated annually by core indicators including:

• Measurable Skill Gains – demonstrated improvements in literacy skill levels in reading, writing, and speaking the English language; numeracy, problem–solving; and English language acquisition.
• Credential Attainment – receipt of a high school equivalency diploma and/or placement in, retention in, or completion of postsecondary education and training, unsubsidized employment, or career advancement

Throughout the program year, AEI state leadership staff will conduct mid–year reviews to determine progress to date on program goals and objectives related to the Colorado State Plan and referenced in the proposals or application for grant funds. AEI will provide technical assistance both on a universal level and customized for individual programmatic needs, as deemed necessary based on mid–year reviews.

Further, AEI will continue to require adult and literacy providers funded through Title II to submit annual performance reports, addressing Colorado State Plan goals and content from the grant proposal or application. Based on those programmatic reports, AEI will prepare a state report, which will assess programmatic performance and state leadership in meeting quality, effectiveness, and improvement initiatives to be submitted to the Department of Education Office of Career Technical and Adult Education.

Vocational Rehabiliation

DVR reviews the program’s performance each federal fiscal year based on the Standards and Indicators measures. Beginning in PY 2016 these criteria will be replaced with the performance measures outlined in WIOA section 116. The performance reports can be run on a statewide, regional, or local basis to provide a robust picture of performance throughout the state. To help inform the overall quality of the services being delivered, a quarterly client satisfaction survey is implemented and reviewed to guide work moving forward. Finally, a Comprehensive Statewide Needs Assessment is conducted every three years, which includes a survey of all staff members, to provide ongoing information and data points about service delivery.