Located in:
- Program-specific Requirements for Adult Education and Family Literacy Act ProgramsThe Unified or Combined State Plan must include a description of the following as it pertains to Adult Education and Literacy programs under title II, the Adult Education and Family Literacy Act (AEFLA).
f. Assessing Quality
Current Narrative:
Massachusetts first implemented an ABE performance accountability system in 2006. CALCs are held accountable for six performance standards, assigned performance points based on their performance in each standard, and sorted into four tiers. Programs that fall into tier 4, and occasionally programs in tier 3 that are at risk of falling into tier 4, are deemed to be in need of targeted intervention (TI) and at risk of losing funding. TI is provided by the state PD system, SABES, a process overseen by ACLS staff. Additionally, performance points are used as one of the criteria during open and competitive funding years. Since instituting the performance accountability system, programs overall have earned higher levels of performance points per year and fewer programs have fallen into the lowest tier.
WIOA performance measures provide one key strategy for assessing the quality of adult literacy services. In FY16, ACLS convened a WIOA Performance Measures Task Force comprised of five Massachusetts adult education directors and ACLS staff to:
- Review the WIOA performance measures and program performance data;
- Decide how the federal measures should be part of a new state performance accountability system for awarding past performance points to local programs;
- Decide whether additional measures should be added;
- Weight and rank the measures;
- Address implications; and
- Create a plan for rolling out the new system including a performance measures pilot in FY17.
In addition to using revised performance measures, Massachusetts will monitor and evaluate the quality of, and improvement in, adult education and literacy activities and disseminate information about models and promising practices using:
- annual on-site monitoring visits of a subset of programs. See Section (e) State Leadership for more details regarding monitoring.
- annual site visits conducted by program specialists. See Section (e) State Leadership for more details regarding site visits.
- desk reviews and ongoing TA from state-assigned program specialists. See Section (e) State Leadership for more details regarding desk reviews and TA.
- SMARTT and Cognos reports to evaluate program performance and guide improvement planning.
- annual refunding applications that allow ACLS program specialists an opportunity to assess the quality of services, and how programs are addressing challenges encountered during the year and working to overcome those challenges the following year.
- required annual policy meetings for adult education directors.
- research-based policies and guidelines. See Massachusetts Policies for Effective Adult Basic Education in Community Adult Learning Centers and Correctional Institutions and Massachusetts Guidelines for Effective Adult Career Pathways Programs for examples of policy and guidance manuals.
- continuous improvement plans provide an opportunity to monitor the quality of adult education and advance emerging quality initiatives. In FY16, for example, programs were required to address at least one WIOA transition-related activity from among the activities highlighted in their continuation applications and encouraged to also include one or more WIOA-related curriculum activities.
- regular communications and guidance to the field (e.g., monthly mailings, project-specific meetings, SABES PD website).
In anticipation of WIOA data collection demands, ACLS is investing in a cutting-edge data collection and management system which will be fully deployed in FY17. This user-friendly, web-based application will be a significant improvement over the current system. The new system will:
- provide ACLS and local programs with intuitive tools to analyze and assess critical information.
- support local management of program operation (enrollment, testing, attendance, and staff assignments and credentials), thereby making it easier for directors to focus on ACLS Indicators of Program Quality and WIOA outcomes.
- create program and state performance reports on learner progress.
- match student records with WIOA partner data to report key employment and postsecondary education outcomes of shared customers.
ACLS will also collaborate with the ESE Audit and Compliance Unit to audit the data and finances of randomly selected programs each year to ensure quality.
VI.f. Assessing Quality:
The Commonwealth’s response did not include a description of how it will assess the quality of its professional development (PD) programs designed to improve: instruction in the essential components of reading instruction, instruction related to the specific needs of adult learners, instruction provided by volunteers or paid personnel, and dissemination of information about models and promising practices.
Response:
Evaluating PD begins with a strong planning process. In FY16, ACLS and the Massachusetts PD system, SABES, adopted 11 High Quality Professional Development (HQPD) Standards. HQPD is defined as a set of “coherent learning experiences that is systematic, purposeful, and structured over a sustained period of time with the goal of improving teacher practice and student outcomes.” PD development and planning are driven by information about where learners need to improve, and the practitioner change that is needed to support the desired learner change.
To align with HQPD Standards, SABES PD descriptions and learning objectives (LOs) focus on how teachers’ practice will change, and the potential impact on their students as a result. PD descriptions describe the knowledge, skills, and attitudes that teachers need to address the identified student needs.
In FY16, grounded in content and quality standards, ACLS and SABES also developed a common evaluation protocol for PD that involves various phases and levels of data collection and in-depth inquiry:
- All PD offerings (e.g., face-to-face, online, conferences, webinars) are evaluated immediately following the PD based on LOs, the connection to the CCRS and ESOL Curriculum Frameworks, the connection to student and program outcomes, and the effectiveness of the presenters. Online PD is additionally evaluated for the ease of navigation of the tools being used and the level of support provided.
- Follow-up evaluations are sent to all practitioners between six weeks and six months following the PD activity to determine if and how they are using what they learned, and what additional supports they need to implement the desired change.
- Evaluation summaries and analyses are provided to ACLS by SABES PD Centers through annual performance reports which include:
- Compiled quantitative and qualitative feedback from evaluations provided at the end of each PD activity.
- Summaries and analyses of follow-up evaluations.
- HQPD Rubrics that demonstrate the alignment of PD with HQPD Standards.
- Analysis of if and how PD has had an impact on classroom and program practice.
- Reflection on how feedback is used to inform revisions to PD activities as well as continuous improvement and planning.
- Evidence that PD is informed by current and rigorous research.
- Examples of local and national promising and best practices that are shared with programs, disseminated via the SABES website, and integrated into PD.
The effectiveness of PD is also evaluated by evidence and direct observation of practice. Examples include:
a. STAR-trained teachers are observed in the classroom for alignment with evidence-based reading instruction.
b. PD centers review and provide feedback on ABE program Curriculum Scope and Sequence and other materials aligned with the CCRS.
c. Participants in the online Integrating Math and ESOL course must submit a lesson plan that demonstrates how the teacher will make math accessible to ESOL learners.
d. New staff who complete the mandatory Foundations for New Staff must pass an online course including quizzes and a final test, and submit sample lesson plans.
e. Training for mandated, standardized NRS assessments requires participants to demonstrate competence by achieving a minimum score on practice tests.
f. New User Training for the data collection system (SMARTT) requires practitioners to demonstrate competence by completing sample data input.
g. ABE license-seekers are observed in the classroom or submit videos of classroom observations to determine if instructional practice aligns with license standards.
ACLS Program Specialists cull models and promising practices information from site visits, multi-day on-site monitoring, and annual continuation funding applications. These are disseminated by SABES on its website and in PD offerings for practitioners.