Located in:
- III. Operational Planning Elements
The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above. Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs. This section must include—
- b. State Operating Systems and Policies
The Unified or Combined State Plan must include a description of the State operating systems and policies that will support the implementation of the State strategy described in section II Strategic Elements. This includes—
- b. State Operating Systems and Policies
III. b. 4. D. Evaluation
Describe how the State will conduct evaluations and research projects on activities under WIOA core programs; how such projects will be coordinated with, and designed in conjunction with, State and local boards and with State agencies responsible for the administration of all respective core programs; and, further, how the projects will be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA.
Current Narrative:
TWC has historically used a blend of Six Sigma, Lean, and Theory of Constraints methodologies for a continuous improvement model called Rapid Process Improvement (RPI). RPI has been used to evaluate dozens of system processes, identify opportunities for improvement, and test the results of implemented changes. Over the last two years, TWC has further expanded its continuous improvement model to move beyond RPI by designing a new methodology called Business Transformation (BT). The primary distinction between RPI and BT is the nature of the problem that is being targeted for improvement. RPI seeks to improve quality and efficiency in business processes, while BT is designed to uncover and address root causes of system-level issues and limitations. BT tends to be used on broader problems that have multiple processes associated with them. BT’s goal is not just efficiency and quality improvements but a strategic rethinking of entire business approaches, priorities, and value systems associated with core business functions.
One recent and ongoing BT effort came out of TWC’s triennial Comprehensive Statewide Needs Assessment (CSNA) for Title IV vocational rehabilitation (VR). The CSNA is a critical resource to understanding how the Texas VR program is viewed by individuals who participate in Title IV programs and their VR service providers, employers, families, and disability support communities. CSNA findings are only valuable insofar as they are analyzed and used to guide action. Analysis of economic and related VR data substantially corroborated CSNA findings.
VR service providers voiced their concerns about low reimbursement rates, the inconsistency and declining number of referrals from TWC, and the burdensome paperwork and requirements to do business with the Texas VR program. I|3 examined the data and was able to validate and see the cumulative impact of these provider concerns. For example, over the past five years TWC saw a 51.5 percent reduction in the number of service provider locations (from 11,723 to 5,688) doing business with the Texas VR program. As of FFY’20, only 24 percent of available provider locations were doing business with the VR program, which coincided with a significant reduction in referrals to many of those providers.
The issue of declining availability and use of VR service providers resulted in secondary impacts, such as further reducing options for VR participants and making it more difficult for staff to find providers to refer them to. There was not a single process that needed to be improved to reverse the declining provider and referral trends. The existing RPI methodology was not going to solve the issue, which became TWC’s initial BT project. TWC took the following actions:
- After examining reimbursement rates and history, TWC concluded that rates were too low and recommended rate adjustments, which TWC’s Commissioners approved.
- As a result of several conversations with VR counselors, TWC found that the reduction in referrals to service providers was driven in part by staff turnover. Newer staff members did not have relationships with local providers and were not able to easily search for available providers in their area. In response, I|3 built a new Tableau-based provider locator that staff may use to quickly find providers by location and types of services offered.
- In an attempt to reduce provider burdens, TWC began reassessing established service provider requirements by comparing them to less burdensome policies in other states. This ongoing work is being performed in a targeted way to maximize the program’s early impact.
- TWC began building solutions to quickly answer providers’ questions and offer them the guidance necessary to assure the provision of quality services to our customers.
- TWC began building a data-driven approach to provider recruitment that targets the providers that are most in need based on customer and provider density in geographical areas.
TWC works with local Boards, other partners, and academic researchers on system evaluations and looks for opportunities to cooperate and partner with the Secretaries of Labor and Education on studies that complement and enhance the research priorities of TWC and its Boards and partners. In addition, TWC believes that the webinars and regional calls that DOL and ED hold provide an excellent avenue for coordinating such work and sharing results between the states and the departments.
Recently, TWC participated in a new adult education evaluation learning community project. This 15-month project brought together research and evaluation experts from many states to develop and conduct high-value evaluations for Adult Education and Literacy (AEL). TWC evaluated the impact that different forms of intake and onboarding mechanisms used by AEL grantees had on certain measures, such as length of participation. The final report was issued during PY’20 and used to develop the curriculum for a session at TWC’s AEL summer institute in August 2021.
TWC has also been performing evaluation work related to the VR program. One project involves a quasi-experimental evaluation of the impact of TWC’s Project SEARCH for individuals with developmental disabilities. Project SEARCH allows participants to compare three 10- to 12-week paid internships in competitive integrated employment settings, thus giving them the opportunity to focus on gaining proficiency in their preferred occupation with the goal of obtaining and retaining permanent employment. Common local employers involved in Project SEARCH include hospitals, law firms, and supply chain industries, which offer diverse internships in areas such as office administration, data entry, assembly/packaging, housekeeping, and food preparation.
Another project in the early stages of development involves a machine-learning model developed to identify VR participants who are six months into their cases and appear to be headed for an unsuccessful closure. The key to this project is not to identify the cases but to develop an intervention that may transform an unsuccessful trajectory into a successful one. The VR division is developing the intervention, which will be tested through a randomized controlled trial (RCT) to identify cases on the path to failure and randomly assign them to receive the new intervention. This RCT will enable staff to determine whether the intervention has a causal impact on program outcomes. Although this work is being conducted within the VR program, it should be applicable to many other programs, including WIOA core partner programs.
I|3 is working with TWC’s Workforce Development Division on several evaluations for DOL-funded programs. The first is an RCT for a new service that would be funded by WIOA dislocated worker (DW) statewide funding and involves special job-coaching and related resources. The evaluation will involve the random selection of Reemployment Services and Eligibility Assessment (RESEA) claimants for access to such services and will serve as both a WIOA DW and RESEA evaluation, as it is expected to meet CLEAR’s “high causality” standard. Another study encompasses a broader quasi-experimental design that addresses the impact of the RESEA program on benefits usage and employees returning to work. It is anticipated that this second study will meet CLEAR’s “moderate causality” standard.
Finally, I|3 is using its analytical resources to prevent waste, fraud, and abuse within the UI system by developing predictive analytics models used by UI’s benefit payment control and investigative staff. TWC supports other states’ efforts to promote integrity in their systems by sharing best practices and contributing to the UI Integrity Center’s Suspicious Actor Repository.
Foundational to TWC’s ambitions to fully leverage its data is TWC’s EDW system, which supports advanced reporting, analytics, and high-quality evaluations that will meet moderate and high causal evidence as described on the CLEAR website. The system will streamline standard reporting to improve efficiency and will provide more resources for analysis and evaluation work. The system also will allow TWC to better connect data relating to participants, services, and outcomes across programs, which will facilitate monitoring at the highest level (systemwide) as well as the program level (WIOA Titles I–IV and beyond), the characteristic level (that is, subpopulations across all programs), or a combination thereof (individuals in a program who meet specific conditions).
A key portion of this work is the development of a common data model for customers in all programs. This system, which will be called Rosetta, will allow customer data to be transformed, standardized, and more easily combined. This will allow us to build Tableau dashboards that can work for any program.
The EDW buildout is being supported by a Workforce Data Quality Initiative grant application that will accelerate system development and bring additional capabilities online much sooner than would have been possible without the funding.
Over the next two to five years, TWC plans to replace or significantly modify nearly all enterprise-level information technology (IT) systems supporting our core programs and services. These systems must provide high-quality data that can support the levels of evaluation that WIOA requires. TWC will expend significant effort on ensuring that its IT modernization pays off both in the short run by serving customers and in the long run by yielding information about which efforts work under which conditions.
Once the Rosetta data system is fully implemented, TWC will apply market basket analysis to identify common customer groups much the way business marketing does when analyzing consumer purchasing patterns. However, our approach will reach farther than the identification of common customer groups—we intend to evaluate each group to determine whether service- based differences exist between high- and low-achieving customers in each group and then evaluate those services for effectiveness. Our assumption is that although most services have value, the value varies by individual and the degree to which services are provided in conjunction with other services or through other means.
Through this approach, TWC hopes to implement a system that better uses data to determine appropriate service delivery strategies unique to each individual customer. This approach is not intended to remove local flexibility or to disregard staff experience, but rather to support staff by providing better evidence-based options, based on the more precise data obtained through market basket analysis, to help plan an approach likely to lead to customer success. This approach will also help bring new staff up to speed more quickly.
TWC’s evidence-based data analysis and evaluation provide a more efficient way to serve employers. We are working to identify employers that might benefit from outreach and education regarding short-term unemployment insurance (UI) compensation as a way to help the employer retain its skilled labor force during a period of temporary financial difficulty.
With the help of our Tri-Agency Workforce Initiative partners, TEA and THECB, TWC invested heavily in providing professional development for I|3 staff through the Coleridge Initiative’s data science class. The class was conducted with real workforce and education data that were deidentified in Coleridge’s data environment. Preparation for the training allowed for greater coordination and partnership between the Tri-Agencies, which resulted in the development of a modern data-sharing agreement and the design of the research questions and agenda for the class.
TWC recognizes the value of collaborating with evaluators outside the agency and even outside the state’s workforce system. The agency works with academic researchers, such as the Ray Marshall Center at the University of Texas and the Texas Policy Lab at Rice University, to evaluate the intersection between workforce preparation through education and labor market outcomes through Texas’ Education Research Centers (ERCs), which combine and make available K-12, postsecondary education (at public colleges and the university), and workforce wage and employment data. The Texas Legislature established the ERCs through legislation, which required development of the ERC system, and the system to focus its research to assist Texas’ elected officials, policymakers, and other stakeholders improve education in Texas. Additionally, recent legislation called for the creation of a more open and current workforce and education data repository in order to support identification of effective programs, measure the impact of changes on outcomes, and provide more robust data to stakeholders and partners.