Located in:
- Program-specific Requirements for Adult Education and Family Literacy Act Programs
The Unified or Combined State Plan must include a description of the following as it pertains to adult education and literacy programs and activities under title II of WIOA, the Adult Education and Family Literacy Act (AEFLA).
f. Assessing Quality
Current Narrative:
Ongoing Program Monitoring and Oversight
Evaluation of local provider services is a priority for the AEL program. TWC staff has worked to develop a strategic evaluation and monitoring process that draws from best practices developed across TWC’s divisions. Texas is implementing a program quality model to support student success that links evaluation of monthly program performance and information and findings from the agency’s monitoring department to just-in-time technical assistance and the deployment of targeted professional development to support continuous program improvement.
The AEL Department provides a Monthly Performance Report (MPR) to the director of the Workforce Development Division. The MPR meeting is facilitated by AEL program and contract management staff and led by the AEL director and AEL program supervisor. It includes a monthly snapshot of both program and financial performance across all AEFLA programs, including section 223 projects.
The MPR is the result of a multiphase process that uses timely data gathered from the Texas Educating Adults Management System (TEAMS) to measure program performance against contract deliverables as well as state and federal performance targets. TEAMS is Texas’ state-of-the-art, web-enabled system that maintains student-level data, including demographic, assessment, and outcome data, as well as class, site, program, and provider information. TEAMS users can analyze class, site, or program information using standard reports generated at the user level. TEAMS provides a wide range of information about adult education, including snapshots of student and program performance, personnel qualifications, and staff development activities.
AEL program providers are required to validate their data monthly and submit quarterly Data Sign-off Reports through TEAMS. The Data Sign-off Report requires program providers to describe areas of strength as well as concern. Program providers also are required to submit monthly expenditure reports in TWC’s Cash Draw and Expenditure Reporting (CDER) system and quarterly narrative reports using a predetermined template that requests information on specific TWC objectives and overall program objectives.
AEL contract deliverables reflect TWC’s commitment to strong performance and accountability. These deliverables include a combination of program and financial accountability and encourage program providers to exhibit strong planning through the development of project plans, continuous improvement plans, and standard operating procedures.
To prepare for the MPR, AEL program staff meet, weekly, at a minimum, to evaluate the overall health of each program, which is measured against contract deliverables; progress toward measures; state performance measures; and expenditures. Program staff members discuss trends in data and issues of concern and develop support plans to address those issues. Because services are aligned to workforce areas, staff can see where specific issues are isolated throughout the state. This provides a structure under which staff can engage TWC’s Statewide Professional Development Center and other Section 223-funded professional development and technical assistance support projects to address concerns.
If a program does not show improvement after being placed on a support plan, TWC places it on a Technical Assistance Plan (TAP). TAPs use a timeline of deliverables to improve and support program function. TAP actions may include additional targeted professional development support, one-on-one training in specific areas of weakness or concern, and technical assistance visits by TWC staff to meet and address issues. Members must meet specific benchmarks for a period of four consecutive months to have a TAP lifted. Programs that consistently fail to meet contract deliverables once placed on a TAP are placed on a Corrective Action Plan. Programs that fail to meet objectives beyond this point are subject to sanctions and deobligation of program funds.
Responsive professional development efforts funded under section 223(a)(1)(B) of WIOA is a central component of the quality model. TWC AEL assesses the quality and responsiveness of professional development through multiple approaches, including the review of local program improvement and remediation of deficiencies that may result after professional development and technical assistance efforts.
For example, through the quality model, a provider that month to month shows poor performance in reading or oral language scores will be advised by their TWC AEL program support specialist or assigned professional development specialist to schedule professional development in these areas. Once the program receives professional development services, the AEL department monitors program improvement to report in the MPR. Professional development is only one remedy to address program improvement and is deployed with other forms of technical assistance.
While changes in local program improvement is perhaps the most direct means of assessing the quality and effectiveness of professional development, it is not the only method of quality review in place. Professional development trainers are required to meet established quality and professional criteria developed and assessed by the state PD Center and AEL staff; training is reviewed by AEL staff to assure alignment with agency objectives and best practices based on the most rigorous or scientifically valid research available Specific professional development efforts funded under section 223 focus on developing curricula and training, including training-of-trainer models on ensuring support for instruction in the essential components of reading instruction and instruction related to the specific needs of adult learners. The Literacy Institute within the state PD Center is focused on professional development in the essential components of reading instruction and instruction related to the specific needs of adult learners. TWC AEL reviews both training curricula deliverables and training for these projects against best practices based on the most rigorous or scientifically valid research available. Section 223-funded projects, Literacy Texas and Career Pathways Expansion, focus on specifically training volunteers in the content areas, including reading and career pathways implementation and quality. These projects all have deliverables supporting the dissemination of information about relevant models and promising practices.
Additionally, all professional development events include session evaluations and these evaluations are provided to the AEL office, which reviews them monthly.
Subrecipient Monitoring
TWC’s Subrecipient Monitoring (SRM) department works with TWC AEL staff to ensure that all 36 grantees are compliant and are serving as good stewards of federal funds. Grantees are selected annually for on-site reviews based on a robust risk analysis that includes both objective criteria and input from AEL. All AEL grantees are monitored either in person or through a desk review, with all programs receiving on-site monitoring once every three or four years.
While on-site, SRM reviews expenditures that include, but are not limited to disbursements, cash management, financial reporting, and records management (including eligibility and assessment documentation). To ensure that the grantee has adequate internal controls, staff will conduct interviews with key staff to confirm that internal processes for participant intake, screening, and placement comply with TWC’s AEL policy. Testing samples are identified using the Texas Educating Adult Management System (TEAMS) database and compared with on-site records, testing will include the review of student test scores, credentials, and Measurable Skill Gains.
SRM hosts a preplanning meeting with TWC AEL staff to discuss any potential issues before the on-site visit. This gives SRM the opportunity to ask questions about specific items of interest identified during the desk review. It also provides SRM with any technical assistance items that AEL staff has reviewed and discussed with the grantee, including performance issues. The process ensures that SRM obtains an accurate review of program and financial activity and any ongoing issues
Before a review, SRM will request that the grantee submit documents for SRM to conduct a review and analysis to determine each area that will be tested on-site. At the end of the on-site review, SRM conducts an exit conference with the grantee, in which staff identifies any findings or areas of concern that will be identified later in an official report.
The results of SRM reviews improve the technical assistance targeted to local projects.
Deobligation of Funds
TWC is keenly focused on performance to ensure that funds are used effectively to serve customers and produce positive results. Contracted performance benchmarks and performance standards highlight TWC’s expectation that grantees will maintain or exceed performance standards through effective service delivery and innovation. TWC rule §800.78, which was amended December 2018, allows funds to be deobligated, including voluntarily, if they are not being used effectively and at expected levels at specified time frames to serve individuals who need AEL services. However, TWC staff works diligently with grantees to provide technical support and assistance in developing strategies to ensure that funds are used to serve customers efficiently and effectively, thereby mitigating risks.
Under current budgetary conditions—as in the past—the need for AEL services exceeds Texas’ available resources, and no workforce area is funded to fully meet its needs. TWC expects that AEL grant recipients will identify, and coordinate with, all available organizations and programs in their workforce areas to expand and leverage deliverable services beyond those provided solely through AEL funding.
Self-Evaluation and Monitoring of Local Programs
As part of their deliverables, all TWC grant recipients (sole providers and lead grant recipients for consortia of providers) are required to:
- submit a detailed project plan and expenditure projections for AEL-funded activities;
- submit a detailed local professional development plan;
- develop and maintain a procedure that outlines a process for continuous monitoring of subrecipients; and
- if applicable, use TEAMS to regularly monitor program performance across all providers.
Programs use the detailed project plan as an opportunity to analyze their performance, determine the cause of areas of weakness or concern, and develop strategies for improvement in the coming year. Programs are required to tie their professional development activities to this plan to ensure adequate training and support for program improvements.
TWC AEL staff also holds regular conference calls and biyearly meetings for program members to share challenges as well as best practices. These forums build on what is working in the field and provide peer-mentoring opportunities.
Additionally, programs are required to regularly solicit student evaluations to assess satisfaction with curricula, instructors, and institutions.