Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. C. Evaluation
Describe how the State will conduct evaluations and research projects of activities carried out in the State under WIOA core programs; how such projects will be coordinated with, and designed in conjunction with, State and local boards and with State agencies responsible for the administration of all respective core programs; and, further, how the projects will be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA (WIOA Section 116(e )).
Current Narrative:
The Texas workforce system’s success depends on effectively serving its customer groups, engaging its employees (including TWC, Boards, and other workforce partners), and understanding its data. TWC’s Information Innovation & Insight (I|3) Division leverages relevant data and transforms it into actionable information to advance the workforce system’s overall efficiency and value.
TWC relies on a variety of analytical and statistical techniques to conduct evaluations, including quasi-experimental techniques and randomized controlled trials (RCTs). Although not as conclusive as RCTs, TWC frequently uses quasi-experimental study designs because they balance rigor with the ability to implement process changes and new initiatives quickly. Additionally, TWC is exploring how machine learning can identify strengths and weaknesses in its programs and suggest enhancements.
To conduct program evaluations, TWC collaborates with Boards, partners, and academic researchers on evaluations, and participates in federal studies such as the Workforce Investment Act Gold Standard Study. TWC proactively engages with the Secretaries of Labor and Education on studies that further the research priorities of TWC, Boards, and partners. Additionally, TWC believes that the webinars and regional calls held by DOL and the US Department of Education (ED) provide a valuable avenue to coordinate work and share results between the states and the departments. TWC also plays an active role in the Clearinghouse for Labor Evaluation and Research (CLEAR) community and shares results and best practices with other community members.
Business Transformation and Process Improvement
Historically, to translate evaluation results into systemic improvements and to measure success, TWC has used a blend of Six Sigma, Lean, and Theory of Constraints methodologies for a continuous improvement model called Rapid Process Improvement (RPI). RPI has been used to evaluate many system processes, identify opportunities for improvement, and test the results of implemented changes. Over the last four years, TWC has further expanded its continuous improvement model by designing a new methodology called Business Transformation (BT). The primary distinction between RPI and BT is the nature of the problem that is being targeted for improvement. RPI seeks to improve quality and efficiency in business processes, while BT is designed to uncover and address the root causes of system-level issues and limitations. BT tends to be used to address broader problems that consist of multiple processes. BT’s goal is not just efficiency and quality improvements, but to provide a strategic rethinking of entire business approaches, priorities, and value systems associated with core business functions.
Vocational Rehabilitation
One recent and ongoing BT effort was established as a result of TWC’s triennial Comprehensive Statewide Needs Assessment (CSNA) for Title IV Vocational Rehabilitation (VR). CSNA is a critical resource to understanding how the Texas VR program is viewed by Title IV program participants, VR service providers, employers, families, and disability support communities. Additionally, CSNA findings are only valuable insofar as they are analyzed and used to guide action. For example, VR service providers voiced their concerns about low reimbursement rates, the inconsistency and declining number of referrals from TWC, and the burdensome paperwork and requirements to do business with the Texas VR program, and, after analyzing the data, TWC validated the cumulative impact of their concerns. Over the past five years, the number of service provider locations that were doing business with the Texas VR program decreased by 51.5 percent (from 11,723 to 5,688). As of FY’20, only 24 percent of available provider locations were doing business with the VR program, which coincided with a significant reduction in referrals to many of those providers.
The decline in VR service provider locations and the use of service providers led to a reduction of options for VR participants and made it increasingly difficult for staff to find provider referrals. The existing RPI methodology was incapable of solving the issue because there were no existing processes that could be improved to reverse the declining provider and referral trends. Therefore, the issue became the focus of TWC’s initial BT project. TWC took the following actions:
- After examining reimbursement rates and history, TWC concluded that rates were too low and recommended rate adjustments, which TWC’s three-member Commission approved.
- As a result of several conversations with VR counselors, TWC determined that the reduction in referrals to service providers was driven in part by staff turnover. Newer staff members did not have relationships with local providers and were not able to easily search for available providers in their area. In response, TWC built a new Tableau-based provider locator that staff members were able to use to quickly find providers by location and types of services offered. In the first six months of use, the Provider Locator Tool received over 7,000 hits.
- To reduce provider burdens, TWC began reassessing established service provider requirements by comparing them to less burdensome policies in other states. This ongoing work is being performed in a targeted way to maximize the program’s impact.
- TWC began building solutions to quickly answer providers’ questions and offer them necessary guidance to ensure the provision of quality services to customers.
- TWC began building a data-driven approach to provider recruitment that would target the providers that were most in need based on customer and provider density in geographical areas.
- Due to provider recruitment efforts, the quarterly number of active contracted providers (accounting for seasonality) has increased steadily from PY’20 to PY’22, reflecting progress. As of Q4 PY’22, VR was actively using 597 contracted providers, a 10% increase compared to the baseline of 543 in Q4 PY‘20.
- There is a more accurate count of available providers because of VR’s systematic removal of inactive provider records from ReHabWorks. Since PY 20, VR has removed over 9,000 inactive providers (both contracted and non-contracted).
- The provider activity rate for contracted providers increased from 51% in Q4 PY’20 (meaning VR used around half of available contracted providers during the quarter) to nearly 65% in Q4 PY’22 (meaning VR used roughly two-thirds of available contracted providers that quarter).
- The Trend Analyzer dashboard developed to track payments to providers shows that 64% of onboarded providers in PY’22 were issued a service authorization within 28 days, indicating most providers are experiencing timely service authorizations. In PY’22, the average time from completion of a service authorization to issuing a warrant for the provider is six days. This represents a significant improvement from the 18 days in PY’21, demonstrating a more efficient service delivery and payment process.
TWC is engaged in several additional ongoing evaluations of its VR program. One project that has been submitted to DOL’s CLEAR involves a quasi-experimental evaluation of the impact of TWC’s Project SEARCH on individuals with developmental disabilities. Project SEARCH provides participants with on-the-job experience via three 10- to 12-week paid internships in competitive integrated employment settings. The goal is to obtain and retain permanent employment. Common local employers involved in Project SEARCH include hospitals, law firms, and supply chain industries, which offer diverse internships in areas such as office administration, data entry, assembly/packaging, and housekeeping and food preparation.
The results showed that participation in Project SEARCH leads to better vocational rehabilitation outcomes compared to traditional methods. TWC reviewed a sample of 904 individuals divided evenly into Project SEARCH and a VR program comparison group and determined that Project SEARCH participants have an 87.39 percent successful employment rate compared to 60.40 percent for other participants of the same demographic in the VR program. Based on a chi-square test, the result is significant, with greater than 95 percent confidence (p < 0.001). TWC also found that Project SEARCH participants enjoyed a greater employment retention rate during Q2–Q4 post-exit than their comparison group peers, with 95 percent confidence (70.35% versus 63.27%, p = 0.0238). TWC did not discover any significant impact on Q2 post-exit earnings. However, it is worth noting that over 16 quarters, including eight quarters before and after exit, the median wage for the Project SEARCH participants grew by about $2,600, versus around $2,100 for the comparison group.
This study is expected to meet CLEAR’s standard for moderate evidence of causality. This study will help determine the feasibility of increasing the number of SEARCH sites and host employers across the state. In 2023, three Project SEARCH sites were added (San Antonio, Midland, and El Paso), and an additional three sites have been approved to start in August 2024.
Another project still in development involves building a machine-learning model to identify VR participants who, six months into their cases, appear to be headed for an unsuccessful closure. The goal is to find commonalities that can be targeted with an intervention to change an unsuccessful case trajectory into a successful one. The VR program is developing the intervention, which will be tested using an RCT experimental design to identify cases at risk of unsuccessful outcomes and randomly assign them to receive the intervention. This approach will enable staff to determine whether the intervention has a causal impact on program outcomes. Although this work is being conducted within the VR program, the results likely will be applicable to all WIOA core programs and other partner programs.
Adult Education and Literacy
In 2022, TWC participated in a new national reporting system for the adult education evaluation learning community. This 15-month project brought together research and evaluation experts from many states to develop and conduct high-value evaluation projects for adult education and literacy (AEL). TWC evaluated different intake and onboarding mechanisms used by AEL grant recipients and their impact on variables such as length of participation. The final report was issued during PY’20 and was used to develop curriculum for a session at TWC’s AEL summer institute for AEL providers, which was held in August 2021. More recently, in August 2022, TWC proposed developing a machine-learning model to predict the likelihood that an AEL student receives a measurable skill gain (MSG) within a particular program year. TWC anticipates that the model will provide deeper insight into the interaction between demographic characteristics and TWC services, thereby helping to identify factors that the agency can influence to increase the ratio of MSGs to total AEL participants as a measure of program effectiveness and engagement.
TWC also contracts with several entities, including Texas A&M University’s Texas Center for the Advancement of Literacy and Learning (TCALL), that operate two of the state’s professional development centers and are required to conduct an annual statewide needs assessment for improvement in professional development activities that more closely align with the state’s goals. For PY’23, TCALL conducted a statewide needs assessment that resulted in several recommendations across four key partners. TWC will be working to ensure that state leadership projects support a responsive approach to recommendations, as further outlined in program specific plan elements.
Workforce Development
TWC and Texas A&M’s Public Policy Research Institution are collaborating on several evaluations for DOL-funded programs. The first study evaluates a new service that involves specialized job-coaching and related resources. The planned evaluation will involve the random selection of Reemployment Services and Eligibility Assessment (RESEA) claimants for access to such services and will serve as a WIOA DW evaluation. TWC and an external partner also are conducting a study that encompasses a quasi-experimental design to address the impact of the RESEA program on benefits usage and return to work. This study is expected to meet CLEAR’s standard for high evidence of causality. Another study, utilizing a broader quasi-experimental design, aims to investigate the impact of RESEA services on reducing UI duration and increasing reemployment outcomes, and is anticipated to meet CLEAR’s standard for moderate evidence of causality. Moreover, TWC regularly participates in multistate meetings to exchange ideas about RESEA interventions and evaluations.
TWC recently examined post‐program employment outcomes for Texans who participated in training as part of the Workforce Innovation and Opportunity Act (WIOA) Title I Adults program. TWC used a quasi‐experimental design to analyze three years of post‐program employment data for adults who completed the program between July 1, 2017, and June 30, 2018. Adults who completed training were employed in significantly more post‐program quarters than adults who either enrolled in training but did not complete the course or who were not enrolled in training. Analysis between adults with varying levels of employment barriers and who completed training did not result in a significant difference in post‐program employment outcomes. TWC determined that whether the study participants received additional TWC‐administered program services after completing the training service had no effect on the number of employment quarters post‐ program. However, an increase in the number of employment quarters five years before beginning a Title I Adults training service led to a statistically significant increase in post‐program employment quarters. This study is expected to meet CLEAR’s standard for moderate evidence of causality.
TWC dedicates substantial analytical and evaluation resources to prevent waste, fraud, and abuse within Texas’ UI system by developing predictive models used by UI’s benefit payment control and investigative staff. TWC is active within the national UI integrity sphere and supports other states’ efforts by sharing best practices and contributing to the National Association of State Workforce Agencies (NASWA), NASWA’s Integrity Data Hub (IDH), and associated workgroups.
Performance Accountability System
Texas completed its move to a WIOA-based performance accountability system in fall 2017 when TWC first reported WIOA-based program outcomes for state and local purposes.
TWC supplements WIOA statutory measures to address inconsistencies between the statutory measures and what TWC considers the statutory vision. For example, although WIOA emphasizes the importance of career pathways, its measures only recognize steps along a career pathway as success for former youth participants. From TWC’s perspective, helping a veteran who recently transitioned out of the military or a worker looking for a mid-career change constitutes a significant success, even if the workforce system is not funding the individual’s education or training. Therefore, TWC’s supplemental WIOA-based measures also identify either employment or educational enrollment in post-exit quarters.
Texas also supplemented the statutory employment/enrollment Q4 measures with a Q2–Q4 employment or enrollment retention measure. The measures build as follows:
- Employed or Enrolled Q2 Post-Exit: What percent of exiters were employed or enrolled in Q2?
- Median Earnings Q2 Post-Exit: How much were those employed in Q2 earning?
- Employed or Enrolled Q2–Q4 Post-Exit: Of those employed or enrolled in Q2, what percent were also employed or enrolled in both Q3 and Q4?
Target Setting
WIOA provides for a two-stage performance target process. DOL develops a statistical model based on the presumed case mix and economic conditions for the performance periods and cohorts. Each state then negotiates a set of initial targets based on these assumptions and other factors. At the end of the year, DOL reapplies the statistical model. For the current reporting year, TWC exceeded 90 percent of the originally negotiated targets across all programs, although the departments have yet to make their year-end adjustments.
Effectiveness in Serving Employers
Together with job seekers and employees, Texas employers are primary TWC customers in the Texas workforce system. Measuring the success of services provided to employers is important for the state’s economic success, and TWC continues to be concerned about the three measures that DOL and ED chose to pilot in response to the information collection requests that the departments published to obtain input on the proposals. Nevertheless, the agency has continued to report all three measures to ensure that DOL and ED have complete data for evaluating the measures and, ideally, developing improvements.
The Rosetta Initiative
Foundational to TWC’s ambitions to fully leverage its data is TWC’s EDW system, which supports advanced reporting, analytics, and high-quality evaluations that meet moderate and high causal evidence ratings. The system aims to streamline standard reporting to improve efficiency and provide more resources for analysis and evaluation work. The system also allows TWC to better connect data relating to participants, services, and outcomes across programs, which facilitates monitoring at the highest level (systemwide) as well as the program level (WIOA Titles I–IV), the characteristic level (that is, subpopulations across all programs), or a combination thereof (individuals in a program who meet specific conditions).
A key part of the work is the development of a common data model for customers in all programs. The system, called Rosetta, is designed to allow customer data to be transformed, standardized, and more easily combined. Rosetta also enables the development of Tableau dashboards that can work for any program.
The EDW buildout was supported by a Workforce Data Quality Initiative grant application that accelerated system development and brought additional capabilities online much sooner than would have been possible without the funding.
TWC has recently modernized several enterprise-level information technology (IT) systems that support core programs and services and is in the process of updating others. The systems are designed to provide high-quality data that can support the levels of evaluation that WIOA requires. TWC will ensure that its effort to modernize its IT systems pays off both in the short run, by serving customers, and in the long run, by yielding information about which efforts work under which conditions.
Once the Rosetta data system is fully implemented, TWC will apply market basket analysis to identify common customer groups like the way business marketing does when analyzing consumer purchasing patterns. However, this approach will reach further than the identification of common customer groups, as TWC intends to evaluate each group to determine whether service-based differences exist between high- and low-achieving customers in each group and then evaluate those services for effectiveness. It is assumed that, although most services have value, the value varies by individual and the degree in which services are provided in conjunction with other services or through other means. TWC has begun to do this using preliminary Rosetta infrastructure, for example, for SNAP E&T participants to evaluate longitudinal differences in outcomes between them and other career and training services participants, as well as investigate differences in outcomes for WorkInTexas.com users based on aspects of their created job profiles, such as the inclusion of links to external websites.
Through this approach, TWC hopes to implement a system that better uses data to determine appropriate service delivery strategies that are unique to each individual customer. This approach is not intended to remove local flexibility or to disregard staff experience, but rather to support staff by providing better evidence-based options that are based on precise data that is obtained through market basket analysis, to help plan an approach likely to lead to customer success. This approach will also help bring new staff up to speed more quickly.
TWC’s evidence-based data analysis and evaluation provide a more efficient way to serve employers. TWC works to identify employers that might benefit from outreach and education regarding short-term UI compensation to help employers retain their skilled labor force during a period of temporary financial difficulty.