U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Plan: Rhode Island PYs 2016-2017
Combined Plan C

Section: WIOA State Plan Common Elements

Narrative: II. b. 4.

Published
Located in:
  • II. Strategic Elements

    The Unified or Combined State Plan must include a Strategic Planning Elements section that analyzes the State’s current economic environment and identifies the State’s overall vision for its workforce development system. The required elements in this section allow the State to develop data-driven goals for preparing an educated and skilled workforce and to identify successful strategies for aligning workforce development programs. Unless otherwise noted, all Strategic Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs.

    • b. State Strategic Vision and Goals

      The Unified or Combined State Plan must include the State’s strategic vision and goals for developing its workforce and meeting employer needs in order to support economic growth and economic self-sufficiency. This must include—

II. b. 4. Assessment

Describe how the State will assess the overall effectiveness of the workforce development system in the State in relation to the strategic vision and goals stated above in sections (b)(1), (2), and (3) and how it will use the results of this assessment and other feedback to make continuous or quality improvements.

Current Narrative:

As Rhode Island responds to the legislative intent of WIOA to revamp and revitalize the methods and parameters state governments use to create effective workforce development intervention, developing the capacity to effectively measure the success of such interventions becomes imperative. Such measurements around the mechanics and effectiveness of an intervention provides policy makers the information necessary to determine the success of the program and effects future investment decisions. Although WIOA has been enacted to modernize our workforce development systems, it lacks sufficient performance metrics and uses measures which outcomes do not provide adequate documentation of progress. Rhode Island, through its innovative Real Jobs Rhode Island workforce development sector strategy program has taken upon itself to move beyond WIOA’s general output measurements and will use more precise outcome based measurements in order to better understand the included program’s efficacy on the state’s economy. These measures and others like them will be managed at the state level access programs, providing decision makers a clearer understanding of the effectiveness of the interventions provided. These core measures are required for all six core programs.

The primary indicators of WIOA for core programs miss their intended mark and do not provide for the reporting of the programs in a comprehensive way. The primary indicators include recordings of how many participants enroll and finish the program, their employment status six months and a year after exiting the program, their annual salaries after six months and a year after exiting the program, and whether or not they have obtained post-secondary diplomas or certificates after leaving a WIOA funded program.

What these primary indicators fail to produce is an idea of whether or not the programs or services are responsible for these measured successes, and create assumptions about participants and the overall effectiveness of programs. By only using output metrics via recording the number of clients entering and exiting a program and whether or not they gain and retain employment; WIOA assumes the participant completed the program, entered a job related to the training, and retained that employment due to the skills obtained within the program. Yet the metrics mandated in WIOA do not allow these assumptions to be verified. When analyzing data, it is indeed accurate to say “x” number of participants “exited” the program and are now employed as an output of a program, but this information is not as useful for future programs as reporting that “x” number of participants reached a specifically defined outcome based on career pathway planning activities.

The state of Rhode Island is emphasizing continuous service and not rushing premature exits to simply meet these output measures. The state recognizes that clients in the greatest need of service from multiple programs to both gain and retain employment will likely need services beyond the date of employment. Prematurely having a client exit may jeopardize the long term success of that client and limit their ability to connect to future services. The state will track the outcome measures that will be developed in addition to the complementary outputs to these required by WIOA. Such outputs will include measures such as “entered employment before exit of the program”, employment retention (6 month and a year after exit). Outcome measures will assist the state in tracking individuals between programs, and the change experienced by a client as a result of both the individual and collective effect of all interventions provided. This will provide the state information on the existence of gaps between programs and will allow decision makers to make informed decision regarding strengthening referrals, increasing co-enrollment, and identifying processes that may hinder the delivery of services concurrently.

For example, a program participant could be employed six months or a year after exiting a program, but their employment may be unrelated to the subsidized training they received, or their salaries may also not have been determined by the skills gained in the program. Furthermore, the language of “exiting the program” within WIOA does not indicate whether the participant completed the training programs or not. These metrics fall short of indicating program effects on employment rates, labor force skill acquirement, or wage rates increases. Although the Governors Workforce Board (GWB) under WIOA mandate is to be representative of businesses located in different geographical regions throughout each state, WIOA’s metrics do not attempt to measure whether or not these subsidized services and programs are being utilized by participants throughout these represented regions.

Such measures do not provide enough information to decision makers about the effectiveness of a program. Therefore, Rhode Island will develop outcome measures to determine the effectiveness of both individual programs and the strategies outlined in this plan to supplement the output measures required by WIOA. Outcome measures will be used to analyze the comprehensive effect of a program, and more broadly the workforce network as a whole. While individual outputs can track the efficiency of individual program elements, outcome measures will assess the overall quality of change experienced by program participants and employer partners.

Outcome measures focus on quality and the improvement or change experienced by a participant. Such measures allow for multiple aspects of a participant’s experience to be tracked at the same time as the participant progresses through a program. Outcome measures can be connected across programs to provide better data around the number of participants in multiple programs or who complete one program and enter another. By linking outcome measures between programs and analyzing the quality of change experience by participants as a result of a single program and the combined results of multiple programs, the acuity of serving individuals with different barriers to employment will be expressed and can inform future negotiated performance levels for individual programs. By having the necessary data to analyze who is being served and the level for service necessary to provide participants the skills they need, despite any barriers they may experience, programs will be able to set more realistic, evidence based performance levels for the WIOA common measures.

The mechanism for developing such comprehensive measures already exists; Rhode Island’s GWB under state law is responsible for gathering and distributing information from, and to, all agencies, departments, and councils within the coordinated-programs system. The board’s duties also include maintaining a comprehensive inventory and analysis of workforce development activities in the state to support the biennial statewide employment and training plan. This collection of data provided to the GWB through programs like RJRI allow the board to make comprehensive decisions about broader economic growth. Recording general output measurements alone may provide an unclear picture of the effectiveness of workforce service interventions.

Rhode Island’s Department of Labor and Training has begun this work in collaboration with the Harvard Kennedy School Government Performance Lab by developing the Real Jobs RI project that will utilize robust outcome based metrics that paint a complete picture of the program’s effectiveness in creating success for program sector partners and training participants alike. The work underway for this program provides a detailed example of the type of outcome based performance measurement Rhode Island seeks to create to more accurately measure the effectiveness of the strategies of this plan. These measurements aim to keep track of every level of the training process and all parties involved; the RIDLT will create reports for the following;

  • Individual Participant Background Measurements
  • Training Module Measurements
  • Participating Employer Measurements
  • Industry and Sector level Measurements
  • System Level Measurements

On an individual participant level, the measurements that the RI DLT will use in The Real Jobs Rhode Island program will allow the agency to accurately gauge the programs effect on creating upward mobility for participants. Understanding a participant’s background is imperative to knowing the possible barriers to employment and how to address them. These metrics also allow the department to follow individuals more closely on their advancement into the field they train for. Recording a participants pre-training wage gives the DLT and the GWB a standpoint in which to measure programs efficacy on these factors and allows the agency to compare them to post-training wages. This also gives a broader understanding of how programs affect the Rhode Island workforce as a whole.

The training module measurements the RIDLT will use to gauge the success of individual training programs go far beyond the reporting mandates in WIOA. By paying close attention to the relevancy of a participant’s employment after completion (not just exodus) of the program, we can ensure that it is in fact the program’s training that affected the participant’s employment status. It is critical to know whether a client staying in the industry in which they were trained resulted from these interventions provided by the programs. Without these measurements the

RIDLT could not know if interventions are assisting sector partners filling vacant positions, which is the main goal of a demand based strategy. These metrics will also pay close attention to the advancement of incumbent workers and their wages within their industries.

In order for RJRI to truly be the “demand driven” sector strategy that it is expected to be, the RIDLT will closely monitor how satisfied employers are with the number of participants they actually employ while taking into account the acuity of those positions filled. These metrics ensure that businesses are fully involved in every step of this process and the agency retains them as partners in future employment acquisitions. Having a close relationship with business partners will give the department the information to determine efficient protocols and processes and where to make changes. Through this metric system the RIDLT will be able to gather more accurate information on the effect of program intervention on productivity and employment quality and communicate this information to other workforce partners. By recording the number of positions retained as a result from programs we can follow which programs create the most stable employment and reproduce these results to later projects.

On a larger scale, beyond individual businesses, within the state’s industries and sectors the RIDLT will collect industry specific information to determine the program’s effect on sectors as a whole. Although Real Jobs Rhode Island is designed to give jobs to unemployed citizens across the state, the state seeks to also ensure that the industries that fuel our economy are benefitting from these programs across the board and not just support isolated employer needs that cannot be related to a larger industry based strategy. By recording the sum of positions filled across multiple businesses in a sector, a better understand of whether meaningful changes to the state’s economy have taken place will emerge. This also makes industries more competitive with our neighboring states and helps attract new businesses.

The System level measures the RIDLT will collect will give the state as a whole the information necessary to make statutory and legal changes in order for programs to operate more smoothly. These measures will be, in effect, an aggregated sum of the individual and sector level measures. This will shed light on where the state can cut red tape where necessary. These metrics will also allow the state government to know whether the system is becoming more responsive to the needs of local businesses and where they can create more efficient policies.

Although WIOA does not mandate such rigorous measurement requirements, Rhode Island believes proper and precise outcome based metrics lead to a greater understanding of how programs affect our state’s economy. Real Jobs Rhode Island is based on aligning agency and program goals with these new performance metrics. Rhode Island will take the comprehensive performance measurement structure developed for RJRI to scale, by adding additional outcome measures for all programs included in this plan.

As evident by the combined planning process, Rhode Island recognizes that it is the combined efforts of programs and services that truly affect change for clients and employers in the state. No agency or program can complete the work alone. When measuring progress towards the goal of providing the correct combination of investment and service to ensure individual workers are connected to opportunities in the labor market while ensuring employer demands for skilled workers are met, outcome measures for the network must be adapted. These reporting mechanisms will allow the agency to use recorded data to improve program operations, evaluate programs impacts on workers and employers while creating a funding feedback loop.

By using evidence based outcomes the state can gather real-time information on what aspects of services and programs are effective in employment interventions which, in turn, will affect funding allocation. By taking the initiative through RJRI Rhode Island will be the example of a laboratory of democracy that other states can use as an example in creating new workforce development programs. When developing comprehensive outcome measures for WIOA partner programs the state of Rhode Island will convene subject matter experts both government and non-government to develop the measures with support from Harvard’s Kennedy School of Government Performance Lab. Outcome measures for the six core programs will be developed over the next year, followed by the combined partner programs. Outcome measures for all programs should be complete by the midpoint of this plan.