Located in:
- Program-specific Requirements for Adult Education and Family Literacy Act ProgramsThe Unified or Combined State Plan must include a description of the following as it pertains to Adult Education and Literacy programs under title II, the Adult Education and Family Literacy Act (AEFLA).
f. Assessing Quality
Current Narrative:
Massachusetts first implemented an AE performance accountability system in 2006. CALCs were held accountable for six performance standards, assigned performance points based on their performance in each standard, and sorted into four tiers. Programs that fell into tier 4, and occasionally programs in tier 3 that were at risk of falling into tier 4, were deemed to be in need of targeted intervention (TI) and at risk of losing funding. TI was provided by the state PD system, SABES, a process overseen by ACLS staff. Additionally, in the past, performance points were used as one of the criteria during open and competitive funding years. Since instituting the performance accountability system, programs overall have earned higher levels of performance points per year and fewer programs have fallen into the lowest tier.
WIOA performance measures provide one key strategy for assessing the quality of adult literacy services. In FY16, ACLS convened a WIOA Performance Measures Task Force comprised of five Massachusetts adult education directors and ACLS staff that:
- Reviewed the WIOA performance measures and program performance data;
- Decided how the federal measures should be part of a new state performance accountability system for awarding past performance points to local programs;
- Decided whether additional measures should be added;
- Weighted and ranked the measures;
- Addressed implications; and
- Created a plan for rolling out the new system including a performance measures pilot in FY17.
Based on the task force recommendations, the MA accountability system is aligned to WIOA measures. Currently ACLS has created one state performance standard: Measurable Skills Gain (MSG). The state MSG performance standard includes the following outcomes: Educational Functioning Level (EFL) completion, High School Equivalency (HSE) credential attainment, and Postsecondary Education or Training (PSE/T) enrollment after a student's exit from a program. In FY18 and annually thereafter, programs will be assigned an MSG target using a formula that incorporates (i.e., is weighted on):
a. The number of students who pretest into each of the National Reporting System (NRS) levels that a program serves, and
b. (Based on historical data) the degree of difficulty in achieving an MSG outcome.
Programs would be ranked based on how close they come to meeting or exceeding their MSG target, and performance will be the actual completion percent relative to the target. Additional information on MA Performance Accountability for adult education programs can be found on the ACLS website.
In addition to using revised performance measures, Massachusetts will monitor and evaluate the quality of, and improvement in, adult education and literacy activities and disseminate information about models and promising practices using:
- annual on-site program quality review visits of a subset of programs. See Section (e) State Leadership for more details regarding the PQR visits.
- annual site visits conducted by program specialists. See Section (e) State Leadership for more details regarding site visits.
- desk reviews and ongoing TA from state-assigned program specialists. See Section (e) State Leadership for more details regarding desk reviews and TA.
- data reports available to programs to evaluate program performance and guide improvement planning.
- annual refunding applications that allow ACLS program specialists an opportunity to assess the quality of services, and how programs are addressing challenges encountered during the year and working to overcome those challenges the following year.
- annual Directors’ Meeting for adult education directors.
- research-based policies and guidelines. See FY19 Massachusetts Policies for Effective Adult Education in Community Adult Learning Centers and Correctional Institutions and FY 2019 Massachusetts Guidelines for Distance Learning in Adult Education for examples of policy and guidance manuals.
- continuous improvement plans provide an opportunity to monitor the quality of adult education and advance emerging quality initiatives. In FY16, for example, programs were required to address at least one WIOA transition-related activity from among the activities highlighted in their continuation applications and encouraged to also include one or more WIOA-related curriculum activities.
- regular communications and guidance to the field (e.g., monthly mailings, project-specific meetings, PD website).
ACLS is currently procuring a vendor to provide a statewide data collection and management system which will replace the current SMARTT system in FY19. This user-friendly, web-based application will be a significant improvement over the current system. The new system will:
- provide ACLS and local programs with intuitive tools to analyze and assess critical information.
- support local management of program operations (e.g. enrollment, testing, attendance, and staff assignments and credentials), thereby making it easier for directors to focus on ACLS Indicators of Program Quality and WIOA outcomes.
- create program and state performance reports on learner progress.
- match student records with WIOA partner data to report key employment and postsecondary education outcomes of shared customers.
ACLS is also developing a risk analysis tool that will be in place for FY19 implementation. The tool will provide information on programs’ fiscal and data trends and will inform additional follow-up which may include a fiscal and/or data audit conducted by the ESE Audit and Compliance Unit.
Evaluating PD begins with a strong planning process. In FY16, ACLS and the Massachusetts PD system, SABES, adopted eleven High Quality Professional Development (HQPD) Standards. HQPD is defined as a set of “coherent learning experiences that is systematic, purposeful, and structured over a sustained period of time with the goal of improving teacher practice and student outcomes.” PD development and planning are driven by information about where learners need to improve, and the practitioner change that is needed to support the desired learner change.
To align with HQPD Standards, SABES PD descriptions and learning objectives (LOs) focus on how teachers’ practice will change, and the potential impact on their students as a result. The descriptions of PD offerings identify the knowledge, skills, and attitudes that teachers need to address specific student needs.
In FY16, grounded in content and quality standards, ACLS and SABES also developed a common evaluation protocol for PD that involves various phases and levels of data collection and in-depth inquiry:
- All PD offerings (e.g., face-to-face, online, conferences, webinars) are evaluated immediately following the PD based on LOs, the connection to the CCRSAE and ESOL Curriculum Frameworks, the connection to student and program outcomes, and the effectiveness of the presenters. Online PD is additionally evaluated for the ease of navigation of the tools being used and the level of support provided.
- Follow-up evaluations are sent to all practitioners between six weeks and six months following the PD activity to determine if and how they are using what they learned, and what additional supports they need to implement the desired change.
- Evaluation summaries and analyses are provided to ACLS by SABES PD Centers through annual performance reports which include:
- Compiled quantitative and qualitative feedback from evaluations provided at the end of each PD activity.
- Summaries and analyses of follow-up evaluations.
- HQPD Rubrics that demonstrate the alignment of PD with HQPD Standards.
- Analysis of if and how PD has had an impact on classroom and program practice.
- Reflection on how feedback is used to inform revisions to PD activities as well as continuous improvement and planning.
- Evidence that PD is informed by current and rigorous research.
- Examples of local and national promising and best practices that are shared with programs, disseminated via the SABES website, and integrated into PD.
The effectiveness of PD is also evaluated by evidence and direct observation of practice. Examples include:
a. Observations of STAR-trained teachers in the classroom for alignment with evidence-based reading instruction.
b. Review and feedback on ABE program Curriculum Scope and Sequence and other materials aligned with the CCRSAE.
c. Lesson plans by participants in the online Integrating Math and ESOL that demonstrates how the teacher will make math accessible to ESOL learners.
e. Demonstrations of competence (i.e., achievement of a minimum score on practice tests) by teachers participating in training for mandated, standardized NRS assessments.
f. Demonstrations of competence (i.e. completing sample data input) by teachers participating in New User Training for the data collection system (SMARTT).
g. Demonstrations of competence (i.e. classroom observations or videos of classroom performance) from ABE license-seekers to determine if instructional practice aligns with license standards.
Starting in FY19, the PQR team will create an annual report that will include promising practices as observed during the PQR visits and will be shared and disseminated to all programs. The PD Centers also are a vehicle for sharing best practices thought PD trainings and website postings.
ACLS also plans to hire an outside evaluator in FY19 to provide a multi-year evaluation of the PD system to continue to strengthen the system and inform its continued improvement.