U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Plan: Texas PYs 2020-2023
Combined Plan C

Section: WIOA State Plan Common Elements

Narrative: III. b. 4. D.

Published
Located in:
  • III. Operational Planning Elements

    The Unified or Combined State Plan must include an Operational Planning Elements section that supports the State’s strategy and the system-wide vision described in Section II(c) above.  Unless otherwise noted, all Operational Planning Elements apply to Combined State Plan partner programs included in the plan as well as to core programs.  This section must include—

III. b. 4. D. Evaluation

Describe how the State will conduct evaluations and research projects on activities under WIOA core programs; how such projects will be coordinated with, and designed in conjunction with, State and local boards and with State agencies responsible for the administration of all respective core programs; and, further, how the projects will be coordinated with the evaluations provided for by the Secretary of Labor and the Secretary of Education under WIOA. 

Current Narrative:

TWC’s Division of Operational Insight (DOI) was created to improve and leverage that data to advance the overall efficiency and effectiveness of the system. DOI has primary responsibility for conducting and coordinating evaluations of the Texas workforce system.

Although TWC uses a variety of statistical techniques to conduct evaluations, it uses primarily quasi-experimental evaluation techniques rather than random assignment trials. While less academic in approach than random assignment trials, quasi-experimental evaluation balances evaluation rigor with the ability to implement process changes or new initiatives quickly.

Additionally, TWC uses a blend of Six Sigma, Lean, and Theory of Constraints to develop “Rapid Process Improvement” (RPI) to evaluate system processes, identify opportunities for improvement, and test the results of implemented changes. RPI is part of the core of TWC’s approach for continuous improvement.

TWC works with local Boards, other partners, and academic researchers on system evaluations and has participated in numerous federal studies such as the Workforce Investment Act Gold Standard Study. TWC will look for opportunities to cooperate with the Secretaries of Labor and Education on studies that complement and enhance the research priorities of TWC and its Boards and partners. In addition, TWC believes that the webinars and regional calls that DOL and ED hold provide an excellent avenue for coordinating such work and sharing results between the states and the departments. TWC anticipates being particularly active within the Clearinghouse for Labor Evaluation and Research (CLEAR) community and sharing results and best practices with other community members.

Most recently, TWC applied and was accepted to participate in a new National Reporting System for Adult Education (NRS) Evaluation Learning Community. This 15-month project will bring together research and evaluation experts from many states to develop and conduct high-value evaluation projects for adult education and literacy (AEL). Because TWC has integrated all operational evaluation resources inside DOI, TWC will be able to leverage what it learns through the project to areas beyond AEL.

TWC’s BIG data warehousing system will support advanced reporting, analytics, and high-quality evaluations that will meet moderate and high causal evidence as described on the CLEAR website. The system will streamline standard reporting to improve efficiency and will provide more resources for analysis and evaluation work. The system also will allow TWC to better connect data relating to participants, services, and outcomes across programs, which will facilitate monitoring at the highest level (systemwide) as well as the program level (WIOA Titles I-IV and beyond), the characteristic level (that is, subpopulations across all programs), or a combination thereof (individuals in a program who meet specific conditions). TWC’s recently approved Workforce Data Quality Initiative grant application will accelerate system development and bring the system’s capabilities online much sooner than would have been possible without the funding.

Over the past several years, many of TWC’s analytic resources have been heavily focused on WIOA implementation activities, such as conducting an updated and developing Participant Individual Record Layout (PIRL) and RSA-911 logic and testing as well as supporting the implementation and rollout of TWC’s new WorkInTexas.com labor exchange system. Over the next four to five years, TWC plans to replace or significantly modify nearly all enterprise-level information technology (IT) systems. These systems must provide high-quality data that can support the levels of evaluation that WIOA requires. TWC will expend significant effort on ensuring that its IT modernization pays off both in the short run by serving customers and in the long run by yielding information about which efforts work under which conditions.

Continuous improvement and “doing more with less” are constant refrains in government, and now, statistical technique, computing power, and high-quality data can deliver on these goals. TWC is already using machine learning and artificial intelligence to detect and prevent fraud in its systems and looks forward to applying these techniques to service delivery.

Once the BIG data system is fully implemented, TWC will apply market basket analysis to identify common customer groups much the way business marketing does when analyzing consumer purchasing patterns. However, our approach will reach farther than the identification of common customer groups—we intend to evaluate each group to determine whether service-based differences exist between high- and low-achieving customers in each group and then evaluate those services for effectiveness. Our assumption is that although most services have value, the value varies by individual and to the degree that services are provided in conjunction with other services or are provided through other means.

Through this approach, TWC hopes to implement a system that better uses data to determine appropriate service delivery strategies unique to each individual customer. This approach is not intended to remove local flexibility or to disregard staff experience, but rather to support staff by providing better evidence-based options, based on the more precise data obtained through market basket analysis, to help plan an approach likely to lead to customer success. This approach will also help bring new staff up to speed more quickly.

TWC’s evidence-based data analysis and evaluation provide a more efficient way to serve employers. We are working to identify employers that might benefit from outreach and education regarding short-term unemployment insurance (UI) compensation as a way to help the employer retain its skilled labor force during a period of temporary financial difficulty.

TWC also has invested heavily in providing professional development to teachers in the AEL program. DOI will be working with TWC’s AEL program staff to determine whether TWC can measure the effectiveness of professional development by looking at student engagement and outcomes before and after teachers participate in professional development.

TWC will conduct several evaluation projects for the VR program. The VR program operates several specialized programs that serve specific groups of customers. DOI is working with the VR Division to evaluate some of the specialized programs, including the following:

  • Project Search - a program that uses a rotation of internships to improve competitive employment in an integrated setting for youth who have intellectual disabilities
  • Project Hire - a program that helps individuals with severe developmental disabilities to complete postsecondary education
  • Criss Cole Rehabilitation Center - a residential VR training facility that serves adults who are legally blind by helping them learn alternative techniques to prepare for, find, and retain employment and/or attend postsecondary education, and/or live independently in the community

TWC recognizes the value of collaborating with evaluators outside the agency and even outside the state’s workforce system. The agency works with academic researchers, such as the Ray Marshall Center at the University of Texas and the Texas Policy Lab at Rice University, to evaluate the intersection between workforce preparation through education and labor market outcomes through Texas’ Education Research Centers (ERCs), which combine and make available K-12, postsecondary education (at public colleges and the university), and workforce wage and employment data. The Texas Legislature established the ERCs through legislation, which required development of the ERC system, and the system to focus its research to assist Texas’ elected officials, policymakers, and other stakeholders improve education in Texas.

Most recently, a Texas delegation, which included TWC staff, engaged with Results for America to identify and incorporate strategies for improving outcomes for job seekers and employers by using evidence-based, results driven solutions. This initiative explored strategies for incorporating evidence-based approaches into how the state awards grant funds.

DOI is exploring ways to leverage the national WIOA regression models to help with program evaluation. The regression models can show the effect of numerous factors on program performance. TWC envisions building on the models to test the effect of various services.

Finally, DOI uses its analytic resources to help prevent waste, fraud, and abuse in the UI system by developing the predictive analytics models used by UI’s benefit payment control and investigative staff. TWC is active within the national UI integrity sphere and supports other states’ efforts to promote integrity in their systems by sharing best practices and contributing to the UI Integrity Center’s Suspicious Actor Repository.