Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. C. Evaluation
Describe how the State will conduct evaluations and research projects of activities carried out in the State under WIOA core programs; how such projects will be coordinated with, and designed in conjunction with, State and local boards and with State agencies responsible for the administration of all respective core programs; and, further, how the projects will be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA (WIOA Section 116(e )).
Current Narrative:
TWC relies on a variety of analytical and statistical techniques to conduct evaluations, including quasi-experimental techniques and randomized controlled trials (RCTs). Although not as conclusive as RCTs, TWC frequently uses quasi-experimental study designs because they balance rigor with the ability to implement process changes and new initiatives quickly. Additionally, TWC is exploring how machine learning can identify strengths and weaknesses in its programs and suggest improvements to customer service.
Collaborative Approach to Actionable Operations Research
To translate evaluation results into systemic improvements and measure success, TWC has used a blend of Six Sigma, Lean, and Theory of Constraints methodologies for a continuous improvement model called Rapid Process Improvement (RPI). RPI has been used to evaluate several system processes, identify opportunities for improvement, and test the results of implemented changes. TWC has expanded its continuous improvement model beyond RPI with the introduction of Business Transformation (BT). While RPI focuses on process efficiency and quality, BT addresses systemic issues and limitations by strategically rethinking entire business approaches and value systems to achieve broader, transformational improvements. Beginning in PY 2023, and continuing into PY 2024, TWC began applying BT methodologies to automate, coordinate, and simplify internal processes. Agency evaluations are dependent on analytics infrastructure such as databases, customer-facing visualizations to help spark research questions, and the development of self-service reports to reduce time spent on nonevaluative ad hoc requests.
To conduct program evaluations, TWC collaborates with Boards, partners, and academic researchers on evaluations and participates in federal studies, such as the Workforce Investment Act Gold Standard Study. TWC proactively engages with the Secretaries of Labor and Education on studies that further the research priorities of TWC, Boards, and partners. Additionally, TWC believes that the webinars and regional calls held by DOL and ED provide a valuable avenue to coordinate work and share results between the states and the departments. TWC also plays an active role in the Clearinghouse for Labor Evaluation and Research (CLEAR) community and shares results and best practices with other community members.
Performance Accountability System
TWC supplements WIOA statutory measures to address inconsistencies between the statutory measures and what it considers the statutory vision. For example, although WIOA emphasizes the importance of career pathways, its measures only recognize steps along a career pathway as success for former youth participants. From TWC’s perspective, helping a veteran who recently transitioned out of the military or a worker looking for a mid-career change constitutes a significant success, even if the workforce system is not funding the individual’s education or training. Therefore, TWC’s supplemental WIOA-based measures also identify employment or educational enrollment in post-exit quarters.
Texas also supplemented the statutory employment/enrollment Q4 measures with a Q2–Q4 employment or enrollment retention measure. The measures build as follows:
- Employed or Enrolled Q2 Post Exit: What percent of exiters were employed or enrolled in Q2?
- Median Earnings Q2 Post Exit: How much were those who were employed in Q2 earning?
- Employed or Enrolled Q2–Q4 Post Exit: Of those employed or enrolled in Q2, what percent were also employed or enrolled in both Q3 and Q4?
Data Validation
TWC takes the following four-pronged approach to validating data:
- TWC’s case management system uses data validation codes to ensure consistent data collection. In this system, field values must meet the required Participant Individual Record Layout (PIRL).
- TWC’s Subrecipient Monitoring (SRM) department conducts data validation testing for all Boards and AEL grant recipients. Beginning in September 2022, in coordination with the Business Support Section of FDCM, SRM developed a dedicated database to capture testing results while allowing TWC to generate testing outcomes by workforce area.
- TWC performs error-checking audits designed to conform with DOL reporting requirements.
- TWC’s case management system uses internal data integrity (DINT) functions that allow local system partners to make corrections to data entries while enabling TWC to validate and approve the corrections or invalidate them based on standardized procedures. The DINT process is engaged when edits or corrections are submitted for system entries that were made in the previous quarter plus 28 days, effectively giving Workforce Solutions Office staff 28 days after the previous quarter to make edits or corrections without the need to submit a DINT request. After that time, any changes to previous quarter entries require validation and approval by designated Board staff. TWC’s Board Service Strategies (BSS) department reviews DINT requests to ensure that they are being approved correctly.
Internally, TWC Analytics & Evaluation follows strict quality control procedures for reporting. The validation process is referred to as quality assurance (QA) and varies depending on the project and business needs, but it always includes:
- an independent run and verification by at least one other staff member of final code and/or results;
- comparison with other relevant published data to check the reasonableness of the numerators, denominators, rates, and other pertinent numbers, especially for high profile reports; and
- saving and reviewing the run log and other related documentation to rule out errors and for the purposes of future forensics and auditing.
Beyond these universal requirements, the manager or director determines appropriate additional QA processes. It is acceptable for a statistic to be reported (call it A) that deviates by up to 1 percent from the same statistic that was calculated from a transactional or base data source (call it B) due to lag between TWC’s information systems. To calculate percent deviation, we take |A – B| / B. For confirming statistics with other divisions, up to 1 percent deviation is acceptable. However, for federal, state, or external reporting, only 0.5 percent deviation is acceptable. When calculating deviation, B always denotes the data system that sets the standard and will change depending on the context. When deviation is too great, staff investigate and rectify.
Improving Service Delivery
In PY 2023, TWC launched an evaluation project to refine service delivery; however, the project is currently on hold due to computational constraints. The project's core hypothesis is that service effectiveness varies across different customer groups. Analyzing all customers together masks these variations. To address this, the project will develop statistically similar customer groups based on factors such as education, demographics, and work experience, thus allowing for a more precise analysis of each group's response to individual services. A customer analysis model will then classify customers into these groups, enabling targeted service recommendations. These recommendations will ideally include a prediction of the likelihood for success. Importantly, this project will enhance, not replace, the expertise of caseworkers and maintain participant autonomy, with recommendations entirely optional. Research indicates that such recommendation engines significantly improve staff efficiency, especially for newer staff, and could potentially offset the impact of staff turnover and associated training costs. TWC plans to resume exploring this project in PY 2025.
Another project, which has been in place since PY 2023, investigates the impact of the use of private networking in job searches. This quasi-experimental study uses the presence of a private networking profile in the Texas labor exchange system, WorkInTexas.com, as a proxy for private networking usage. Positive results could lead to a joint research project with private networking companies to identify effective private networking strategies.
Vocational Rehabilitation
Comprehensive Statewide Needs Assessment
TWC initiated a BT project in response to findings from the triennial Comprehensive Statewide Needs Assessment (CSNA) for Title IV Vocational Rehabilitation (VR). The CSNA provides insights into the perception of the Texas VR program from participants, service providers, employers, families, and disability support communities. The 2020 CSNA highlighted critical challenges to the provision of purchased services, including VR vendor shortages, low reimbursement rates, service inefficiencies, inconsistent referrals from TWC, and burdensome paperwork. In response, TWC initiated a BT project.
The 2023 CSNA indicates that the VR program made improvements, including reducing paperwork, streamlining administrative processes, and increasing the number of active contracted service providers (vendors who received payment for a service authorization [SA]). During the fourth quarter of calendar year 2024 (CY 2024), VR actively used 540 contracted providers to provide services, a 14 percent increase from the baseline of 469 that was set in the fourth quarter of CY 2020. Additionally, the percent of active contracted providers compared to all available contracted providers (provider activity rate) increased from 46 to 55 percent during the period. Improvements produced a 25 percent increase in the number of VR customers served with services purchased from the providers (both contracted and non-contracted) to over 41,000 in State Fiscal Year 2024 (SFY 2024) from approximately 33,000 in SFY 2019.
Strengthening the VR Program
TWC is implementing several key strategies to strengthen the VR program. In September 2022, TWC Vocational Rehabilitation (TWC-VR) Division contracted with Public Consulting Group (PCG) to analyze the program’s ability to be attractive to and its retention of vendors. The review provided an analysis of the activities and services that are currently provided, industry standards, provider costs, comparable rates and processes of peer states, and interviews with selected TWC-VR staff. Based on the data, PCG recommended rate increases, which TWC-VR implemented using a phased approach.
To reduce provider burdens, TWC is reassessing established service provider requirements by comparing them to less-burdensome policies in other states. This ongoing work is being performed in a targeted way to maximize the program’s impact. TWC also developed solutions to quickly answer providers’ questions and provide guidance to ensure the provision of quality services to customers.
TWC continues to run a Tableau-based provider locator that enables staff to quickly find providers by location and types of services offered. In the first six months of use, the Provider Locator Tool received over 7,000 hits. TWC also developed the Trend Analyzer dashboard to track provider payments, showing strong performance in Program Year 2024 (PY 2024) for timely SAs. While some providers will not get an SA in the first month after onboarding due to seasonality, such as summer programs providers that are onboarded in the spring, an increase is indicated from PY 2023 to PY 2024. In PY 2024, sixty-four percent of onboarded providers were issued a SA within 28 days, with over one-third receiving a SA within seven days. Moreover, the average time from completion of a SA to issuing a warrant for the provider decreased from six days in PY 2023 to five days in PY 2024. This continued improvement demonstrates increased efficiency in service delivery and payment processing.
Project SEARCH
TWC is engaged in several ongoing evaluations of its VR program. One project nearing completion involves a quasi-experimental evaluation of the impact of TWC’s Project SEARCH on individuals with developmental disabilities. Project SEARCH provides participants with on-the-job experience via three 10- to 12-week paid internships in competitive integrated employment (CIE) settings, with the goal of obtaining and retaining permanent employment. Local employers involved in Project SEARCH include hospitals, law firms, and supply chain industries, which offer diverse internships in areas such as office administration, data entry, assembly/packaging, housekeeping, and food preparation.
The ongoing study, covering PY 2020–2022, compares 500 Project SEARCH participants with 500 VR participants. Despite the challenges of the COVID-19 pandemic, Project SEARCH participants achieved a success rate 17.6 percentage points higher than the comparison group. Overall, Project SEARCH participants were 1.4 times more likely to achieve successful closure (defined as maintaining CIE for at least 90 days before exiting the VR program) than nonparticipants. While the analysis of retention rates (employment retention during the second through fourth quarters following VR program exit) using unemployment insurance wage records is still ongoing, preliminary findings reinforce the program’s positive impact. A previous study covering Federal Fiscal Years 2015–2019 also showed Project SEARCH success and retention rates of 87.4 percent and 70.35 percent, respectively, compared to 60.4 percent and 63.27 percent for a comparison group.
Both studies are expected to meet CLEAR’s standard for moderate evidence of causality and will help determine the feasibility of increasing the number of SEARCH sites and host employers across the state. In 2025, 223 VR customers completed the program within the 34 sites currently operating in Texas.
VR Services Guide
A new project is underway to create a Vocational Rehabilitation Services Guide (VRSG), a VR reference for counselors based on de-identified case data. The VRSG will help counselors, both new and experienced, to more efficiently identify potentially beneficial services for customers with similar circumstances and vocational objectives, based on past successful cases. By inputting limited customer data, the guide will generate a list of goods and services that were previously provided to similar customers that contributed to positive outcomes. The aim is to provide counselors with valuable insights, facilitate informed discussions with customers, and improve collaborative plan development. The VRSG leverages a cloud-based machine learning solution to generate personalized service recommendations. The system analyzes customer demographics, functional limitations, employment goals, and service history to generate tailored suggestions. Data are stored and processed in the cloud, training a recommendation model within a dedicated engine. Real-time recommendations are retrieved via an API. Key technical requirements include cloud service access, a robust data pipeline, and ongoing model monitoring and retraining.
Five-Year Return on Investment
A five-year return on investment (ROI) analysis demonstrated significant returns for VR program participants. The study examined the calendar year 2018 cohort of 21,023 program exiters by comparing their five-year post-exit earnings growth to total case expenditures. Findings showed that the average total amount expended over the life of the case was $5,645, compared to an additional return of about $31,951 in individual earnings over five years. Overall, for every dollar spent on a case, exiting participants made an additional $5.66 in income over the following five years. When broken down by age group, adults (aged 25 and over at time of exit) made an additional $3.57, while youth (aged 16–24 at time of exit) earned an additional $11.94. The following study results highlight the strong ROI and lasting positive impact of VR services on participant earnings:
- Total earnings for the 20 quarters (five years) before and after program exit were analyzed. “Additional five-year earnings after VR” represents the difference between post- and pre-exit earnings, calculated as [Total earnings five years after exit] – [Total earnings five years prior to exit]. This figure is then compared to total case expenditures to calculate ROI, calculated as [Additional five-year earnings after VR] / [Total case expenditures].
- Earnings in this study are not adjusted for inflation and are reported as nominal (actual) dollars.
Earnings Outcomes for VR Participants completing Postsecondary Education and Training
Additionally, TWC staff compared wage outcomes for participants who completed Postsecondary Education and Training during their VR case. Postsecondary education and training include community college, four-year college or university, graduate school, and occupational/vocational training programs. In PY 2023, VR participants who were employed in the second quarter following successful employment had a median wage gain of $2,007, comparing wages at the time an individualized plan for employment (IPE) was developed to earnings at Q2 post-exit. Participants who completed postsecondary education and training during participation had a significantly higher median wage gain of $6,861. Furthermore, postsecondary credential attainment significantly impacted employment outcomes. While the overall Texas VR employment rate was 51.4 percent, participants who earned a postsecondary credential achieved a notably higher employment rate of 67.3 percent, a 16-percentage point difference.
Reemployment Services and Eligibility Assessment Program
RESEA program evaluations started in 2022 with the launch of the IPE conducted by the Texas A&M University (TAMU) Public Policy Research Institute. Over the last two years, TAMU has visited several Workforce Solutions Offices in each Board area to conduct surveys and host focus groups to develop a baseline of qualitative data on program services. TAMU will use the data to support several impact studies as follows:
- Quasi Experimental Design I is currently underway to evaluate the impact of the initial RESEA appointment on employment outcomes and UI duration.
- Quasi Experimental Design II is currently underway to evaluate the impact of subsequent meetings on employment outcomes and UI duration.
- A random control trial is currently underway to evaluate the impact of different levels of service on program outcomes.
Texas is also participating in a DOL multi-state evaluation on the RESEA selection model.
Unemployment Insurance
TWC dedicates substantial analytical and evaluation resources to prevent waste, fraud, and abuse within Texas’ UI benefits and tax systems. Eligibility fraud, identification (ID) fraud, and fictitious employer detection is conducted through the development of predictive models used by investigative staff. TWC is active within the national UI integrity sphere and supports other states’ efforts by sharing best practices and contributing to the National Association of State Workforce Agencies (NASWA), NASWA’s Integrity Data Hub (IDH), and associated workgroups.
Adult Education and Literacy
TWC originally proposed developing a machine-learning model to predict the likelihood that an AEL student would receive a measurable skill gain (MSG) within a particular program year. However, House Bill (HB) 1602, 88th Texas Legislature, Regular Session (2023), requires TWC to establish performance criteria and prioritize funding to the providers that consistently meet them. As a result, this legislation changed TWC’s focus from predicting MSGs to building an education model on high school equivalency (HSE) completion and credential attainment.
The concept of a fast-track HSE program was reviewed by analyzing reported AEL program data from PY 2021 (July 1, 2021, to June 30, 2022) and comparing it to data from PY 2018. An interactive report was developed to give providers detailed data. The findings and interactive report were presented to providers at the annual TWC AEL Fall Institute in 2023.
TWC contracts with two entities, including Texas A&M University’s Texas Center for the Advancement of Literacy and Learning (TCALL) that operates two of the three professional development (PD) centers for the state (the Distance Education PD Center and the Statewide PD Center, which is the umbrella for the other PD centers for AEL). TCALL also conducts an annual statewide needs assessment to help ensure PD activities more closely align with the state’s goals. In PY 2023, TCALL conducted a statewide PD needs assessment that resulted in several recommendations across four key partners. TWC will work to ensure that state leadership projects (dependent on continued federal funding) support a responsive approach to recommendations, which will be further outlined in program specific plan elements. Educational Service Center in Region 6 operated the Career Pathways Professional Development Center (CPPDC), which is focused on PD activities for creating and implementing career pathways, in FY 2024, and a new grantee was procured for FY 2026. The contract started on October 1, 2025.
As recommended by the Texas AEL’s Advisory Committee in March of 2025, TWC has partnered with the Texas A&M Public Policy & Research Institute (PPRI) to conduct a year-long evaluation of AEL programming and data that will use focus groups and surveys to generate specific ideas for program improvement.
The Enterprise Data Warehouse and Rosetta Initiative
Foundational to TWC’s ambitions to fully leverage its data is TWC’s EDW system, which supports advanced reporting, analytics, and high-quality evaluations that meet moderate and high causal evidence ratings. The system aims to streamline standard reporting to improve efficiency and provide more resources for analysis and evaluation work. The system also allows TWC to better connect data relating to participants, services, and outcomes across programs. This facilitates monitoring at the highest level (systemwide), the program level (WIOA Titles I–IV), the characteristic level (that is, subpopulations across all programs), or a combination thereof (individuals in a program who meet specific conditions).
A key part of the work is the development of a common data model for customers in all programs. The system, called Rosetta, is designed to allow customer data to be transformed, standardized, and more easily combined. Rosetta also enables development of Tableau dashboards that can work for any program.
The EDW buildout was supported by a Workforce Data Quality Initiative grant application that accelerated system development and brought additional capabilities online much sooner than would have been possible without the funding.
Over the next few years, TWC plans to replace or significantly modify several key enterprise-level IT systems that support core programs and services. The systems are designed to provide high-quality data that can support the levels of evaluation WIOA requires. TWC will ensure its efforts to modernize its IT systems pay off both in the short term, by serving customers, and in the long term, by yielding information about which efforts work under which conditions.
Once the Rosetta data system is fully implemented, TWC will apply market basket analysis to identify common customer groups, similar to the way business marketing does when analyzing consumer purchasing patterns. However, this approach will reach further than the identification of common customer groups. TWC intends to evaluate each group to determine whether service-based differences exist between high- and low-achieving customers in each group and then evaluate those services for effectiveness. Although most services have value, that value varies by individual and the degree to which services are provided (in conjunction with other services or through other means).
Through this approach, TWC hopes to implement a system that better uses data to determine appropriate service-delivery strategies that are unique to each individual customer. This approach is not intended to remove local flexibility or disregard staff experience but rather to support staff by providing better evidence-based options from precise data, obtained through market basket analysis, to help plan an approach likely to lead to customer success. This approach will also help bring new staff up to speed more quickly.
TWC’s evidence-based data analysis and evaluation provide a more efficient way to serve employers. TWC works to identify employers that might benefit from outreach and education regarding short-term UI compensation to ultimately help them retain their skilled labor force during a period of temporary financial difficulty.