Narrative: III. b. 4. A.

Maine Bureau of Employment Services has established a schedule of formal monitoring, which includes a program and fiscal review of each of the Title IB programs annually. Maine’s guidance on oversight and monitoring responsibilities spells out the roles and responsibilities of local area monitoring and identifies the criteria to be reviewed annually. ASSESSMENT OF TITLE II— ADULT EDUCATION TABLE 11: ADULT EDUCATION FEDERAL PARTICIPANT PERFORMANCE OUTCOMES Maine Statewide Adult Education Federal Participant Performance and Outcome Rates* Adult Basic and Secondary Educational Functioning Levels FY15 Negotiated FY15 Actual FY15 Outcome FY16 Negotiated FY16 Actual FY16 Outcome Beginning ABE Literacy 37% 28% Did Not Meet 37% 27% Did Not Meet Beginning Basic Education 42% 30% Did Not Meet 38% 35% Did Not Meet Low Intermediate Basic Education 40% 32% Did Not Meet 40% 34% Did Not Meet High Intermediate Basic Education 32% 23% Did Not Meet 31% 25% Did Not Meet Low Adult Secondary Education 33% 23% Did Not Meet 28% 19% Did Not Meet

Adult Basic and Secondary Educational Functioning Levels FY15 Negotiated FY15 Actual FY15 Outcome FY16Negotiated FY16 Actual FY16 Outcome Beginning ESL Literacy 48% 31% Did Not Meet 40% 43% Exceeded Low Beginning ESL 63% 43% Did Not Meet 52% 56% Exceeded High Beginning ESL 47% 45% Did Not Meet 41% 41% Met Low Intermediate ESL 39% 33% Did Not Meet 39% 34% Did Not Meet High Intermediate ESL 41% 29% Did Not Meet 34% 29% Did Not Meet Advanced ESL 24% 11% Did Not Meet 19% 14% Did Not Meet

Adult Basic and Secondary Educational Functioning Levels FY15 Negotiated FY15 Actual FY15 Outcome FY16 negotiated FY16 Actual FY16 Outcome Entered Employment 35% 41% Exceeded 41% 46% Exceeded Retained Employment 70% 72% Exceeded 69% 67% Did Not Meet Obtained HSE Credential or Its Equivalent 83% 97% Exceeded 94% 92% Did Not Meet Entered Postsecondary Education 20% 20% Met 22% 20% Did Not Meet *PERFORMANCE MEASURES AND OUTCOMES ARE FOR PARTICIPANTS IN FEDERAL PROGRAMS AND DO NOT INCLUDE MAINE COLLEGE TRANSITIONS DATA OR OTHER STATE FUNDED PROGRAMMING OUTCOMES. In FY15, 100% of the employment, postsecondary entry, and credential attainment outcomes exceeded or met state targets. In FY16, statewide performance measures exceeded or met state targets in 27% of the educational functioning levels (3 out of 11 categories). The overall completion rate for all 11 levels in FY16 was 30% which did not meet the overall state target of 34%, but did increase from the previous year by 2 points (from 28% in FY15 to 30% in FY16). To improve performance measures and outcomes for all learners, the state continues to address strategies to improve posttest percentage rates, enhance Career Pathway services, and increase academic rigor through evidence-based instructional practices aligned with the college and career readiness standards. ASSESSMENT OF TITLE III WAGNER—PEYSER PROGRAMS —ACCESSIBILITY — EEO PRACTICES Wagner—Peyser programs are assessed at the same time as Title IB program reviews are being conducted. Staff members are interviewed regarding knowledge and practice of explaining job order procedures and job seeker registration services and are asked to explain the ways in which they provide employer assistance and help in creation and resolution of jobs orders. A review of orders and assessment of staff regarding EEO and affirmative action requirements is also conducted and random review of staff knowledge of these requirements and Wagner Peyser regulations are also melded into the question review process. Processes to provide initial assessment and appropriate referrals to Info Center customers and front end procedures are also reviewed. In some instances participants may also be interviewed either directly on site or via telephone. Monitors use the Checklist provided under Section 188 to conduct the accessibility review. Center accessibility requirements are also assessed and staff members are asked to explain how customers can access the assistive technology in the centers; all required posted information is examined to ensure that it reflects the most up—to—date version of the regulations and whether or not centers are able to provide the information in Braille and other languages besides printed English. At least once annually a separate Equal Employment Opportunity review is conducted by the State EEO officer. The EEO officer reviews sub—recipient compliance with universal access and non—discrimination requirements through examination of participant applications and enrollments against demographic data. Likewise participant files are reviewed again to ensure that all staff assisted participants have been provided with the required EEO statement and their rights to file a complaint. Upon completion of the review, providers are supplied with a formal report of review outcomes and a corrective action plan if there are findings. ASSESSMENT OF TITLE IV PROGRAMS—REHABILITATION SERVICES Within the Department of Labor’s Bureau of Rehabilitation Services (BRS), Maine has two federally Designated State Units, the Division for the Blind and Visually Impaired (DBVI) and the Division of Vocational Rehabilitation (DVR), which deliver Vocational Rehabilitation (VR) services. Since the performance accountability measures required in Section 116(b) of WIOA were not included in the performance standards and indicators historically required of all VR agencies, BRS has been developing data sources to assess the performance of DBVI and DVR by these measures as they are defined by regulation. Initially, BRS will be using Maine Unemployment Insurance wage data to establish baseline and initial WIOA performance targets for employment retention and earnings. Given that this data set does not provide information on individuals who are placed in self— employment or other jobs, such as federal and out—of—state positions, BRS will be working with its core partners and the Rehabilitation Services Administration to develop alternate data sources that will provide a more complete report of the employment outcomes achieved by people with disabilities served by DVR and DBVI. Additionally, given the federal definition of the measures of credential attainment rate, skill gains, and effectiveness in serving BRS is working on ways to capture, report and assess DVR and DBVI performance on these indicators, including through the Maine Department of education and National Student Clearinghouse. In the interim, BRS will be reporting the performance results of FY 2017 State Plan goals in the DBVI and DVR portions of this Unified State Plan. [See (p) Evaluation and Reports of Progress: VR and Supported Employment Goals.] LOCAL BOARD AND PROGRAM ASSESSMENT The Bureau of Employment Services (BES) program monitoring tool is fashioned after the Core Monitoring Tool published by the USDOL—ETA. Each local area and its Title IB sub—recipients are monitored annually. Monitoring is conducted to ensure that the one—stop system is in compliance with the intent and substance of the rules governing funding streams, and to identify whether the systems are operating to achieve the State and local area or regional strategic workforce system goals. The monitoring tool used is designed to explore the working relationships between required partners within the system and in particular the core partners housed in the one—stops. Monitoring provides an opportunity to identify and share best practices across the State and to identify performance and compliance issues that need attention and address. Each year Bureau of Employment Services conducts on—site monitoring of the local board during which local board governance is evaluated and local area sub—recipient monitoring activity is assessed. The local area governance review includes examination of board policies, board membership and formal Memoranda of Understanding with required partners. It also includes a review of board minutes, requests for proposal, service and sub—recipient contracts and an overview of quarterly reports and most recent progress in achieving planned service levels and performance goals. Inadequate policies and/or lack of required board membership are identified as findings and technical assistance is provided upon request. The local area plan is reviewed against plan guidelines issued by the USDOL and the State Board. Plans that do not contain all of the required elements are not approved by the State Board until they comply with all of the requirements. Local areas are also evaluated on the process used to develop and create the local plan, to ensure that there has been adequate involvement of system partners in identifying and implementing strategies outlined in the plan and to ensure that the plan contains steps for implementation of identified strategies. Local board certification is approved only if the local board has an approved plan in place, meets negotiated performance measures, maintains required local board membership, addresses any outstanding findings or policy requirements and sustains fiscal integrity. Methods the local board uses to communicate, educate and inform sub—recipients and system partners is also evaluated to understand local area effectiveness in meeting plan goals and promoting continuous improvements. Local area sub—recipients are also monitored annually to ensure that the local board oversight is adequate and results in sub—recipient compliance. Sub— recipient monitoring includes intensive file reviews to ensure that adequate documentation of eligibility and participant services are in place. Program staff interviews are utilized to gauge whether program design and delivery is being conducted according to requirements and local area plans and participant interviews are conducted to gain insight on the participant’s perspective and satisfaction with the service being provided. BES has made a practice of reviewing participant files on the MIS system prior to conducting the on—site paper file reviews to identify prospective issues with data element validation requirements. Sometimes specific training programs are the focus of a review, such as On—the—Job Training, Customized Training and Work Experience contract reviews and interviews with employers providing the work—based training. Such contracts are reviewed for compliance with labor standards and non—displacement requirements and also to ensure that the level and quality of the training is commensurate with the needs of the participants being trained. Sub—recipient interviews allow the monitors to assess whether the service providers understand and are implementing service delivery according to the local area plan and whether they are familiar with and applying federal, state and local board policies correctly and meeting the requirements of specific programs. Tools that service providers use to conduct initial, comprehensive and academic assessments are also appraised as are the methods for providing and documenting provision of required services, including information about non—discrimination and customer complaint process, labor market and career information, and job search assistance processes. Monitors examine whether or not all required aspects of each of the programs are being implemented and spending thresholds attained and/or spending limits adhered to. Youth services staff are asked to explain how they assess each youth participant’s need for any of the required youth service elements and the number of enrolled youth who access each of the elements. Youth program case managers are interviewed to showcase how they develop an individual service strategy for a youth and whether they partner with outside entities to provide any of the service elements, such as Adult Mentoring. Work experience files and contracts are reviewed and staff members are asked to explain how Work Experience sites are monitored to determine that they are safe and to explain the process for addressing Work Experience site issues. Service providers must also provide an overview of how Work Experience site employers are educated about child labor law and safety requirements that may pertain to the particular site or participants. Youth policies are reviewed to identify whether and how they differ from Adult policies, such as supportive service policies or assessment and plan development methods. In January 2018, the State Workforce Board approved a motion by the Governor to implement a new policy that expands the definition of training services and requires 70% of the funds allocated to the Local Workforce Boards be expended on training. See Appendix XXV Minimum Training Expenditure Requirement for Workforce Innovation and Opportunity Act (WIOA) Adult and Dislocated Worker Funding policy ASSESSMENT OF EMPLOYER SERVICES One of the fundamental components of assessing quality of employer services is to identify if the information and services being provided by the formula program services providers and/or local board staff are being coordinated with similar outreach conducted by other one—stop system partners. Providers are asked to explain how they promote the services of the Maine Job Link and whether they collaborate with other Maine business assistance entities, such as the Department of Economic and Community Development’s account executives or Maine’s Small Business Development Centers or the Maine Community Colleges’ Business and Professional services leads. Also assessed is the number of public training resources that are packaged or bundled on behalf of employers and whether or how pools of participants are assessed and screened for new hire OJT training to meet the needs of employers. QUARTERLY PERFORMANCE ASSESSMENT Local areas are required to submit quarterly reports that identify actual to planned service levels and levels of attainment of negotiated performance goals. Local areas submit both quantitative and qualitative components and are asked to identify at least one service success story from each program every quarter. Planned spending is also reviewed against actual using quarterly fiscal reports; if there are spending thresholds to be met or spending caps which cannot be exceeded, these reports are used to identify and address risk. Quarterly reporting allows us to address and understand issues regarding service or spending levels that are below or above planned levels. Bureau of Employment Services responses to quarterly reports may be in written or oral form; whenever there are common issues across multiple local areas at once, a group conference call is initiated to share data and brainstorm action steps to address the issues directly with local board staff. DATA VALIDATION Each year the BES conducts data validation, an activity that involves careful scrutiny of both paper and MIS files to ensure that data reported is valid and properly documented. After each round of data element validation is completed, the BES provides reports to each local area office that identifies any of validation errors (failures); shortly thereafter the BES also provides training to each local area to go over the errors, how to address them going forward and to refresh staff about all of the required tracking and documentation. CUSTOMER SATISFACTION Job Seeker and employer customer satisfaction surveys are conducted at least once annually in addition to any local area customer satisfaction activities. These are conducted using survey monkey and may include short 10 minute phone interviews as well. In general the results provide us with information about the customer perceived value of required services and whether or not the services are meeting the needs of the customer. The quality of staff assistance is also ascertained through the questionnaire. The results of the service are explained in the Annual Report. PERFORMANCE RESULTS Each year new State performance goals for the Adult, Dislocated Worker, Youth and Wagner Peyser programs are negotiated with USDOL. In turn the Maine BES negotiates performance goals with each of the local areas who are also provided with data shared by USDOL on use of the regression model to project and propose appropriate levels of performance to be achieved. The ability to achieve proposed performance levels can be affected by a number of factors, including unanticipated series of mass layoffs, layoffs of low—skill workers in a high—skill job market, and rurality and lack of job openings in specific geographic areas. Over the last three program years Maine has achieved or exceeded all of the required levels with the exception of PY14 in which it failed to achieve the negotiated Literacy Numeracy measure for youth. The following steps are being taken to improve the literacy numeracy rate of Maine’s youth participants: • Best practice sharing takes place on a bi—monthly basis at the Younger Workers Committee meetings. The members of this committee include leaders from Maine’s WIOA youth service providers, Job Corps, Secondary Career and Technical Education programs, Adult Education, DHHS foster youth programs and more. • In addition to this meeting, BES has initiated a regular WIOA youth lead meeting that also takes place on a bi—monthly basis. This team is made up of WIOA Core service provider leads serving youth and includes WIOA youth service staff, Adult Education and Vocational Rehabilitation. Activities for this team include: identifying and sharing best practices, understanding new requirements under WIOA, understanding how core partners work with youth, and brainstorming solutions to challenges. • In identifying a solution to the Literacy/Numeracy gains issue, it was found that youth that are not fully engaged in a group cohort are less likely to come in for post—testing requirements as are homeless youth and youth involved with corrections. One strategy discussed was to incorporate more academics into activities, like work experience (as is required under WIOA), and to do so at a level that is appropriate for the particular youth participant; another is consideration of using incentives that reward youth who stick to and achieve the goals in the service strategy, which must include goals and methods for attaining skills gains and for post—test requirements. • Finally, formal professional development for youth staff has been identified as a critical need for Maine’s system, mainly because staff turnover is higher than ever (many long—term staff are retiring, and the wage levels for entry level youth case managers are not high enough to retain the special talent required of a youth counselor). These teams have also discussed strategies and partnerships to engage more out—of— school youth. A list of partners who refer youth has been identified and shared, the WIOA youth team has also shared a list of formal agreements they have with outside partners who serve the same youth participants. Some team members also sit on the Maine Youth Transition Council, which coordinates services for youth transitioning out of foster care.

Requested Revision;

III.b.4. A: Assessment of Core Program Activities Using Annual Performance Accountability Assessment Results.

Now that all four core partners are required to report on the same performance outcomes, the State will be better able to compare outcomes for quality improvement purposes - based not only on the barriers of the individuals served, but on the type and level of services received through each core program.

Using specific reporting elements, such as co-enrollment, level and type of Career Services provided, level and type of Training Services provided (or not), amounts spent on each participant for each service type, participant demographics and local economic factors, the State will be able to develop a quality assessment tool that more deeply explores all of the data that may affect performance achievement.

The additional data will allow the State to compare the programs, service packages, co-enrollments and specific approaches of local areas (and individual service providers) that meet or exceed planned and negotiated measures with those of local areas that are having difficulty doing so. In doing this, the State will be able to promote and/or require local areas that are struggling to achieve negotiated measures to replicate the practices of local areas that are exceeding goals. Such strategies may include, expanding the number of participants taking part in multiple core programs services simultaneously, such as Adult Education and Title I services for education and training. It may also identify that professional staff development around participant and/or employer outreach needs to be addressed. Finally, it may be that State, local area or service provider policies are actually restricting service blending possibilities and that by adjusting these the providers will be better able to address participant needs and provide supports for improved employment opportunities and longer term employment success.

Once the State becomes adept at utilizing the Statistical Adjustment Model it will have the ability to identify factors that are beyond the local area’s control, such as major downsizings, extreme rurality, higher levels of non-English speaking residents etc. and will be able to recommend proactive responses by core and other service partners in those areas that can begin to address those issues, by directing collaborative investments to improved English Literacy Programs that combine ESL and Occupational Training, or that establish partnerships that link individuals without access to service points, like One-Stops or affiliate sites because of extreme distances, through technology or new partnerships with Adult Education or Municipal Libraries.

Once the State becomes adept at utilizing the Statistical Adjustment Model it will have the ability to identify factors that are beyond the local area’s control, such as major downsizings, extreme rurality, higher levels of non-English speaking residents etc. and will be able to recommend proactive responses by core and other service partners in those areas that can begin to address those issues, by directing collaborative investments to improved English Literacy Programs that combine ESL and Occupational Training, or that establish partnerships that link individuals without access to service points, like One-Stops or affiliate sites because of extreme distances, through technology or new partnerships with Adult Education or Municipal Libraries.

Local Areas are conducting research around high-growth career pathways and training staff to utilize pathway models when providing LMI guidance and developing individual services strategies with participants. It is likely there is some direct correlation between clearly defined pathways (both at the individual and local area level) and improved employment attainment and retention outcomes.

Maine reviews performance on a quarterly and annual basis using a number of tools, from program and fiscal reports, to annual and desk top monitoring to anecdotal information sharing. By using all available assessment data, the State has been able to identify service models that result in better outcomes for high-risk youth that may be able to be replicated in some areas but not all. Several counties in Maine are so vast and sparsely populated (for example Aroostook County is the size of Rhode Island and Connecticut combined) that new and different ways of providing services need to be implemented to achieve required measures.