Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00003739 IMPLEMENTATION COMPLETION AND RESULTS REPORT (Loan 8448-MX) ON A LOAN IN THE AMOUNT OF US$120 MILLION TO THE UNITED MEXICAN STATES FOR THE MEXICO REDUCING INEQUALITY OF EDUCATIONAL OPPORTUNITY PROJECT June 26, 2019 Education Global Practice Latin America And Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective August 23, 2018) Currency Unit = Mexican Peso (MXN) MXN 19.37 = US$1 US$0.05 = MXN 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Axel van Trotsenburg Country Director: Pablo Saavedra Senior Global Practice Director: Jaime Saavedra Practice Manager: Emanuela Di Gropello Task Team Leader(s): Marcela Lucia Silveyra de la Garza, Ciro Avitabile ICR Main Contributor: Jasmine Pineda ABBREVIATIONS AND ACRONYMS ABCD Learning Based on Collaboration and Dialogue (Aprendizaje Basado en la Colaboración y el Diálogo) ACUDE Towards a Democratic Society Civil Association (Hacia una Cultura Democrática, A.C.) API Mobile Pedagogical Support (Asesoría Pedagógica Itinerante) CAEEI State Academic Coordinator for Early Education (Coordinador Académico Estatal de Educación Inicial) CAREI Regional Academic Coordinator for Early Education (Coordinador Académico Regional de Educación Inicial) CEDI Child Development Evaluation Card (Cédula de Evaluación del Desarrollo Infantil) CNCH National Crusade Against Hunger (Cruzada Nacional Contra el Hambre) CONAFE National Council for Education Development (Consejo Nacional de Fomento Educativo) CONAPO National Population Council (Consejo Nacional de Población) CONEVAL National Council of Evaluation (Consejo Nacional de Evaluación) COS Operations and Follow-up Coordinator (Coordinador de Operación y Seguimiento) CPS Country Partnership Strategy DECIS Directorate of Community Education and Social Inclusion (Dirección de Educación Comunitaria e Inclusión Social) DEI Directorate of Early Education (Dirección de Educación Inicial) ECD Early Childhood Development ENLACE National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico en Centros Escolares) FORTALECE Community Strengthening for Education (Fortalecimiento Comunitario para la Educación) GDP Gross Domestic Product GNI Gross National Income GoM Government of Mexico HOME Home Observation Measurement of the Environment IE Impact Evaluation IERR Internal Economic Rate of Return IP Indigenous Peoples IPP Indigenous Peoples Plan IRI Intermediate Results Indicator KIDI Knowledge of Infant Development Inventory LEC Leader for Community Education (Líder para la Educación Comunitaria) M&E Monitoring and Evaluation NDP National Development Plan OECD Organisation for Economic Cooperation and Development PDO Project Development Objective PISA Program for International Student Assessment RCT Randomized Control Trial RF Results Framework s.d. Standard Deviation SEP Secretariat of Public Education (Secretaría de Educación Pública) SIEF Strategic Impact Evaluation Fund SIIINAFE National Information and Infrastructure System for Education Development (Sistema Integral de Información e Infraestructura Nacional para el Fomento Educativo) SUMMA Education Research and Innovation Laboratory for Latin America and the Caribbean SWOT Strengths, Weaknesses, Opportunities, and Threats TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9 II. OUTCOME .................................................................................................................... 12 A. RELEVANCE OF PDOs ............................................................................................................12 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................18 E. OTHER OUTCOMES AND IMPACTS .........................................................................................18 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 20 A. KEY FACTORS DURING PREPARATION ...................................................................................20 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................20 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 22 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................22 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................24 C. BANK PERFORMANCE ...........................................................................................................25 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................27 V. LESSONS AND RECOMMENDATIONS ............................................................................. 27 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 29 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 38 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 40 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 41 ANNEX 5. BORROWER COMMENTS ...................................................................................... 44 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 52 ANNEX 7. THEORY OF CHANGE ............................................................................................ 54 ANNEX 8. TARGETING CRITERIA FOR API INTERVENTIONS .................................................... 56 ANNEX 9. ECD INTERVENTION TRAINING SCHEME................................................................ 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Mexico Reducing Inequality of Educational Opportunity P149858 Project Country Financing Instrument Mexico Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency United Mexican States Consejo Nacional de Fomento Educativo (CONAFE) Project Development Objective (PDO) Original PDO The objectives of the Project are: (a) to improve parental competencies in early childhood development in targeted rural communities; and (b) increase the transition rate from the primary to the secondary education level in CONAFE-administered schools in targeted municipalities. Page 1 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 150,000,000 120,000,000 120,000,000 IBRD-84480 Total 150,000,000 120,000,000 120,000,000 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 150,000,000 120,000,000 120,000,000 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 24-Oct-2014 11-Dec-2015 06-Sep-2017 31-Dec-2018 31-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 10-Jan-2018 86.69 Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories 21-Aug-2018 112.87 Change in Results Framework 24-Oct-2018 112.87 Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial Page 2 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 26-Jan-2015 Satisfactory Satisfactory 0 02 03-Aug-2015 Satisfactory Moderately Satisfactory 0 03 11-Feb-2016 Satisfactory Moderately Satisfactory 30.17 04 02-Sep-2016 Satisfactory Moderately Satisfactory 50.78 05 19-Apr-2017 Moderately Satisfactory Moderately Satisfactory 64.18 Moderately 06 08-Dec-2017 Moderately Unsatisfactory 84.44 Unsatisfactory Moderately 07 29-Jun-2018 Moderately Unsatisfactory 94.40 Unsatisfactory 08 17-Dec-2018 Moderately Satisfactory Moderately Satisfactory 119.63 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Early Childhood Education 31 Public Administration - Education 3 Primary Education 66 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Education 100 Access to Education 50 Education Financing 50 Page 3 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ADM STAFF Role At Approval At ICR Regional Vice President: Jorge Familiar Calderon Axel van Trotsenburg Country Director: Gerardo M. Corrochano Pablo Saavedra Senior Global Practice Director: Claudia Maria Costin Jaime Saavedra Chanduvi Practice Manager: Reema Nayar Emanuela Di Gropello Marcela Lucia Silveyra de la Task Team Leader(s): Ciro Avitabile Garza, Ciro Avitabile ICR Contributing Author: Jasmine Anne Pineda Page 4 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country context. At the time of Project appraisal in 2014, Mexico’s economy had rebounded quickly from the 2008-09 global financial crisis, surpassing pre-crisis Gross Domestic Product (GDP) by 2010 and continuing a positive growth trend through 2014. Despite gains in the short-term however, long-term economic growth was below expectations. In 2013, its Gross National Income (GNI) per capita was approximately 39 percent of the level observed in high-income countries of the Organisation for Economic Cooperation and Development (OECD), 1 the same proportion observed two decades ago, signaling a lack of progress in economic convergence. Over the past three decades, annual GDP growth averaged 2.4 percent and only 0.8 percent per capita. Growth decomposition exercises pointed to insufficient average productivity growth as the main cause of the less than satisfactory growth performance. Furthermore, despite the marginal gains in growth, monetary poverty rates had not changed significantly and even experienced a slight increase from 50.0 percent in 2002 to 53.2 percent in 2014, indicating that the most economically disadvantaged populations in Mexico were not benefitting equally. 2. Despite the increases in monetary poverty among the most disadvantaged, Mexico’s multi-dimensional poverty measure 2 indicated that extreme poverty was declining from 11.0 percent in 2008 to 9.5 percent in 2014 3, suggesting that while income growth for the poorest populations had been less than satisfactory, access to social services such as healthcare and education and other basic necessities had improved. Recognizing the achievements in expanding access, the priorities of the Government of Mexico (GoM) shifted more towards improving quality of services, as evidenced by the priorities of the 2013 Education Reform and the National Development Plan (NDP) for 2013-18 (Refer to Assessment of Relevance of PDOs & Ratings for further explanation). 3. Sector context. Mexico had made significant advances in coverage from preschool through upper secondary levels, but educational attainment lagged behind other countries in the region and the OECD, causing concerns about the quality of education services particularly among disadvantaged communities. 2012 Program for International Student Assessment (PISA) scores had put Mexico among the last places in Mathematics, Reading, and Science 4. Moreover, within Mexico, there were dramatic differences in educational attainment. In Mexico City and Nuevo León, two of the most urbanized territories with some of the lowest levels of marginalization 5 in Mexico, the average number of years of education were 10.5 and 9.8, respectively, as compared to 6.3 in Chiapas, a highly rural territory with the greatest percentage of localities with very high levels of marginalization. In addition to inequalities between rural and urban territories, there were significant gaps in 1 Mexico is an upper middle-income country and a member of the OECD and the G20. 2 Mexico adopted a multi-dimensional poverty index, defined by the National Evaluation Council (CONEVAL), which considers income poverty and a set of deprivations (nutrition, education, social security, basic services, access to health services, and quality and space of the dwelling). 3 CONEVAL 4 Mexico’s scores vs the OECD average, respectively: Mathematics 413 vs. 494; Reading 424 vs. 496; Science 415 vs. 501. 5 CONAPO’s marginalization index is a composite of the following eight socioeconomic indicators: 1) percentage of population aged 15 or more that is illiterate; 2) percentage of population aged 15 years or more without complete primary education; 3) percentage of private homes without running water inside the house; 6) percentage of private homes with some level of overcrowding; 7) percentage of private homes with dirt floors; 8) percentage of private homes without a refrigerator. When ranked in order of the marginalization index, those localities in the highest two out of the give sub-intervals are defined by CONAPO as having “very high” and “high” grades of marginalization. Page 5 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) learning outcomes for students in indigenous schools in comparison to general schools. In 2013, 28 percent of students enrolled in grades 3 to 6 in indigenous schools scored either Good or Excellent on the National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico en Centros Escolares, ENLACE), as opposed to 44 percent for those enrolled in general schools. 4. There were several contributing factors for the inequalities of educational opportunities faced by marginalized populations. Attraction and retention of qualified teachers in remote areas has been a consistent challenge along with issues of insufficient infrastructure and educational materials. In addition, the capacity of families and community members in marginalized municipalities to support their children’s education is limited by their own low educational attainment, spurring cycles of low education and economic outcomes. Specific to indigenous communities, the limited availability of educational materials in indigenous languages and teachers who speak the language further compounded the previously mentioned challenges. 5. Institutional context. In response to these challenges and building on the successes of decades of partnership, the Project sought to continue supporting the National Council for Educational Development (Consejo Nacional de Fomento Educativo, CONAFE) in targeting rural and indigenous areas to improve education outcomes. Created by presidential decree in 1971, CONAFE is a decentralized, semi-autonomous governmental organization under the Secretariat of Public Education (Secretaría de Educación Pública, SEP) that seeks to provide quality basic community education and promote equity and inclusion for children in localities characterized by marginalization and social lags. According to 2010 World Bank data, 94.2 percent of Mexico’s land area is rural, yet only 21.6 percent of the population (about 24.6 million in 2010) lives in these areas. To provide education in these areas in relatively remote areas with low populations, CONAFE’s education programs are all based on community models that leverage resources within communities (e.g. community spaces for sessions on Early Childhood Development (ECD) and human resources) while providing support through direct monetary support and provision of training, curriculum, supervision, materials, etc. 6. The Project was prepared as the previous operation, Mexico Compensatory Education Project (P101369, Loan 7859-MX), was closing. While the previous operation sought to improve access to Early Childhood Development (ECD) services and improve learning outcomes in basic education by providing Mobile Pedagogical Support (Asesores Pedagógicos Itinerantes, APIs) to the lowest performing students, the current Project sought to maintain levels of access while improving the quality of the ECD and APIs interventions through informed redesigning of the implementation models. Many CONAFE beneficiary populations are highly migratory, particularly migrant worker families, so maintaining the previously achieved levels of coverage indeed presented a challenge. Both programs were producing positive results but an impact evaluation (IE) of the ECD model and a survey administered to APIs highlighted areas for improvement. Thus, the Project intended to focus on improving quality in a targeted set of communities through, among others, robust Monitoring & Evaluation (M&E) before expanding the implementation of the redesigned models nationally. 7. The proposed Project was aligned with the Country Partnership Strategy (CPS) for 2014-19 (Report No. 80800-MX), discussed by the Board of Executive Directors on December 12, 2013, particularly with the pillar of Increasing Social Prosperity. Through this highly targeted focus on the improvement of the school trajectories of children living in the poorest and most marginalized municipalities in Mexico, this operation sought to reduce extreme poverty and boost shared prosperity. The Project also contributed to the achievement of two objectives in the NDP (2013-18): Inclusive Mexico and Mexico with Quality Education. Page 6 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Theory of Change (Results Chain) *Improved program monitoring and implementation also illustrated by the arrows flowing between activities/outputs Project Development Objectives (PDOs) 8. The objectives of the Project are: (a) to improve parental competencies in early childhood development in targeted rural communities; and (b) increase the transition rate from the primary to the secondary education level in CONAFE-administered schools in targeted municipalities. Key Expected Outcomes and Outcome Indicators 9. The Project’s objective would be measured using the following project development indicators: (a) PDO 1. Score of parenting practices among caregivers of children ages 0-3 years and 11 months, as measured by the Knowledge of Infant Development Inventory (KIDI) instrument, in targeted rural communities in four states 6. 6 Querétaro, Nayarit, Puebla and Zacatecas Page 7 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) (b) PDO 2. The transition rate from primary to secondary education for graduates of CONAFE-administered schools in select municipalities targeted by the Cruzada Nacional Contra el Hambre (CNCH) 7. Components 10. The Project had the following three components: Component 1: Early Childhood Education (Appraisal: US$M 115.97; Actual: US$M 94.87; 79 percent of total Project cost) 11. Component 1 supported CONAFE’s ECD program which provides capacity building to parents, relatives, and caregivers of children aged 0 to 3 years and 11 months in selected rural communities to improve their competencies in caring for children and contributing to their comprehensive development through: (a) the carrying out of capacity building activities to parents, relatives, and children aged 0 to 3 years and 11 months, through the provision of direct monetary support to ECD facilitators, module supervisors, zone coordinators, and regional and state academic coordinators; and (b) the carrying out of capacity building activities for ECD facilitators, module supervisors, zone coordinators, and regional and state academic coordinators. 12. CONAFE’s ECD program is a low-cost educational intervention implemented by the Directorate of Early Education (Dirección de Educación Inicial, DEI) in which groups of parents, pregnant women, caretakers from at least five families and at most fifteen with children younger than four years old meet for two-hour sessions once a week with an ECD facilitator, a local volunteer trained on ECD. CONAFE provides training and monthly support to ECD facilitators, module supervisors, zone coordinators. The sessions are held in community spaces (a beneficiary’s house, a public space in the community, or sometimes in CONAFE community school classrooms). Sessions with families are carried out over a 9-month operational cycle beginning in October and ending in June of the following year. The ECD program covers about 450,000 children annually in rural and indigenous areas, CNCH municipalities, CONAFE priority municipalities. The impacts of the Project were to be measured only in selected states. Component 2: Mobile Pedagogical Support (Asesores Pedagógicos Itinerantes, APIs) (Appraisal: US$M 31.41; Actual: US$M 24.31; 20 percent of total Project cost) 13. Component 2 supported CONAFE’s basic community education program through the provision of mobile pedagogical support to under-performing students, teachers, and parents of students in targeted community schools 8 through: (c) the provision of mobile pedagogical support to targeted students, teachers, and parents of students in under-performing community schools through the provision of direct monetary support to APIs and their supervisors, Operations and Follow-up Coordinators (Coordinadores de Operación y Seguimiento, COS); and (d) the carrying out of capacity building activities for APIs and COS. 7An inter-sectoral and inter-governmental strategy to reach Mexico’s poorest and most marginalized municipalities 8 CONAFE supports community schools through the API/LEC model where young graduates from high school spend two years on average teaching in the primary and lower secondary levels. CONAFE also supports traditional schools administered by the state-level education authorities by providing compensatory actions (e.g. materials, small infrastructure projects, and school grants). The Project exclusively supports community schools. Page 8 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) 14. Implemented by the Directorate of Community Education and Social Inclusion (Dirección de Educación Comunitaria e Inclusión Social, DECIS), CONAFE’s basic community education program is a community multi-grade, multi-level model with instructors called Leaders for Community Education (Líder para le Educación Comunitaria, LEC). APIs, mobile pedagogical tutors, are assigned to community schools meeting the selection criteria 9 in areas meeting the targeting criteria (see Annex 8 for criteria and prioritization for the program). Supervised by COS, APIs have three main functions: (i) hold one-on-one tutoring sessions with a maximum of 6 underperforming students; (ii) collaborate with the LEC on the design of pedagogical strategies to improve the educational processes in the classroom for all students; and (iii) inform parents about their children’s education. Starting school year 2018- 2019, there were 1,923 APIs serving 3,744 community schools; that is, around 17,000 students with low educational achievement. Component 3: Technical Cooperation, Research, and Innovation (Appraisal: US$M 2.24; Actual: US$M 0.44; 0.3 percent of total Project cost) 15. Component 3 provided technical assistance to CONAFE to support Project management, including the carrying out of any analysis required and the design of monitoring and evaluation modules for the supervision of the activities under Components 1 and 2. The following activities were planned or carried out within the scope of this Component: (a) Evaluation of the Child Development Evaluation Card (Cédula de Evaluación del Desarrollo Infantil, CEDI). The CEDI is a child development evaluation tool that aims to identify children in developmental risk, with a subsequent referral to the health system to provide a more comprehensive diagnostic and treatment, if needed. A process evaluation of the CEDI’s application was carried out in 2017 in two states: Quintana Roo and Campeche. (b) Community Strengthening for Education (Fortalecimiento Comunitario para la Educación, FORTALECE). The FORTALECE model aims at increasing parent and community participation in school matters through associations of community members to improve efficiency and quality of community schools through school-based management. At preparation, the Project planned to carry out a qualitative evaluation, process evaluation, and survey to inform the improvement of this program, but the activities were removed from the Project scope. See Revised Components for further detail. (c) Measurement. Data collection firms were hired under this component for the measurement of PDO 1. (d) Design of the curriculum for the new parental practices model. A consultancy firm Hacia una Cultura Democrática, A.C. (ACUDE) was hired under this component to provide technical assistance to CONAFE for the development of the curriculum and methodology for the new parental practices model. (e) National Information and Infrastructure System for Education Development (Sistema Integral de Información e Infraestructura Nacional para el Fomento Educativo, SIIINAFE). The Project designed, developed, and implemented an information system that provided timely information on financial, administrative, and operational aspects of the ECD and APIs programs. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 9 Selection for schools prioritized municipalities covered by the CNCH. The APIs operated in two schools: a school in community A and a school in community B. School A met the following criteria: at least five students, a minimum of 40 percent of students reporting poor academic achievement based on standardized test results, ENLACE prior to 2014; the National Plan for the Evaluation of Learning (Plan Nacional para la Evaluación de los Aprendizajes, PLANEA) henceforth, and that are located in highly marginalized communities. School B was chosen for proximity to School A. Page 9 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Revised PDOs and Outcome Targets N/A Revised PDO Indicators 16. The instrument, Home Observation Measurement of the Environment (HOME), used to measure PDO 1 and, consequently, the indicator targets were formally modified as part of the Restructuring (Report No. RES31304) in August 2018 in response to measurement delays caused by restrictions in the GoM’s federal budget that solely affected consulting services. As an alternative, the Bank and the GoM discussed the possibility of substituting the measurement instrument for the Knowledge of Infant Development Inventory (KIDI) since data collection of the KIDI would cost less than the collection of the HOME. At the time, the KIDI was already being used to measure one of the intermediate results indicators (IRIs). While there is robust evidence demonstrating the linkage between the home environment (measured by HOME) and child development, there was less evidence that showed the relationship between parental knowledge (measured by KIDI) and child development. In order to validate the choice of the instrument, the Bank carried out an analysis of the relationship between parental knowledge, as measured by the KIDI, and children’s language skills, as measured by the MacArthur-Bates instrument, in a sample of beneficiary communities. The development of children’s language skills has commonly been used as a proxy for cognitive development in young children. The analysis suggested a positive and statistically significant correlation between parental knowledge and the development of children’s language skills. Building on this evidence, the Bank agreed to change the measurement instrument of PDO 1 as well as the baseline and target in function of the changed instrument. While the target and definition of the PDO 1 indicator changed, the indicator itself remained the same. Revised Components 17. Component 3 of the Project was designed to include activities of technical assistance and consultancy services including activities to improve the design and implementation of the FORTALECE program. The program was designed to improve school outcomes of children in targeted communities by strengthening the role of the community and increasing the accountability of the school providers in community schools. Including FORTALECE in the Project design was a complementary component since a key concept of the API intervention was the role of the community in supporting the learning production function of students. However, by April 2016, the GoM proposed the removal of the related activities and IRI (Focus groups among parents, teachers, and community representatives in a sample of communities targeted by FORTALECE) in order to prioritize the other Project components 10. The changes were later formalized in the August 2018 Restructuring. Other Changes 18. In July 2017, the GoM requested that two additional figures of the operative chain of the ECD intervention be eligible for financing under the Bank Loan. These figures, State ECD Academic Coordinators (Coordinadores Académicos Estatales de Educación Inicial, CAEEI) and Regional ECD Academic Coordinators (Coordinadores Académicos Regionales de Educación Inicial, CAREI), were expected to have higher qualifications and a more technical, pedagogical background 11 in order to strengthen the operative chain and improve the quality of the training process for the ECD intervention. After reviewing the costs associated with the financing of these figures 10 It was agreed in the Aide Memoire from an implementation support mission in April 2016, that the removal of the activities would be included in the restructuring following mid-term review (September 2017). Because the activities would have required the hiring of consultancy services and given the previously mentioned budgetary constraints, the Bank agreed to the removal of the activities and indicator. 11 CAEEI and CAREI figures are subject to stricter hiring and retention criteria including a minimum 80 percent score on the KIDI and a pedagogical or social sciences background or prior experience with CONAFE for a minimum of 3 years. Page 10 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) and agreeing upon minimum criteria for their hiring and retention, the Bank approved eligibility for the financing of CAEEI and CAREI with Loan resources on August 14, 2017 in a letter. The GoM made the corresponding updates in the Project’s Operative Guidelines (Lineamientos Operativos) and Rules of Operation (Reglas de Operación) which were duly approved by the Bank. (For more information regarding the hiring and retention criteria, see Key Factors During Implementation.) 19. In January 2018, the GoM requested a cancellation of US$30 million (20 percent) of the original Loan amount in response to changes in the peso-dollar exchange rate. At preparation, US$1 was equivalent to MX$13.08, and by January 2018, MX$18.84 12, representing an increase of 44 percent. When modifying the allocations among disbursement categories and components given the partial cancellation, a greater percentage of the original allocation to the category and component for technical assistance was reduced in comparison to the other categories and components (80 percent reduction in Component 3 vs. 18 percent and 23 percent reductions to Components 1 and 2, respectively). The greater reduction in Component 3 was due to the challenges in procuring consultancy services because of the federal government’s budgetary restrictions on them and consequently, the removal of activities related to FORTALECE and the substitution of the instrument for the PDO 1 indicator (for more details, see Report No. RES30282). 20. In August 2018, in addition to the changes made to the PDO 1 indicator, several IRIs were also modified. Changes are summarized in Annex 1. As mentioned, the need to remove FORTALECE-related activities was identified by April 2016 but the GoM and the Bank agreed to wait to include this change in the mid-term review restructuring. Following the mid-term review mission in September 2017, the Bank conducted a two-step restructuring process in line with the GoM’s needs: 1) in January 2018, a restructuring to cancel US$30 million to respond to the substantial change in the dollar-peso exchange rate; and (2) in August 2018, a restructuring to update several IRIs to demonstrate progress more accurately by adjusting definitions and/or methodologies based on a comprehensive data analysis of baseline and targets that was only completed by May 2018. Rationale for Changes and Their Implication on the Original Theory of Change 21. The original baseline for the PDO 1 Indicator was based on an initial data collection of the HOME in 2012, during the previous operation. By 2016, when the challenges of hiring a data collection firm to apply the HOME became more apparent, the Bank agreed to the substitution of the instrument for a lower-cost option, the KIDI. Because the new baseline was measured in December 2017, the target was set in line with the new baseline and the shorter time span between the baseline and the final measurement (2012-2018 versus 2017-2018). The new instrument measures parental knowledge of ECD practices through a survey whereas the original measured parental practices through home observation. The theory of change is not necessarily affected as a result of the change in instrument. It does, however, require that the measure of parental knowledge be used as a proxy of parental practices which then contributes to the cognitive and non-cognitive development of young children. As seen in the analysis carried out by the Bank in December 2017 and repeated in December 2018 (results discussed in the section Assessment of Achievement of Each Objective/Outcome), a positive and significant correlation between parental knowledge and children’s linguistic development supports the idea that increased knowledge positively affects parental practices and therefore, child development (Rowe, 2008). All other changes do not impact the scheme of the theory of change. 12Exchange rate at preparation as stated in the Project Appraisal Document (Report No. PAD1062) and exchange rate in January 2018 as stated in Restructuring Paper (Report No. RES30282). Page 11 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 22. Relevance of the Project’s objectives is deemed to be High. Strongly aligned with CONAFE’s mandate of delivering education services to children living in the most remote areas of Mexico, the Project was also highly relevant to many of Mexico’s national strategies. Designed and implemented during the Country Partnership Strategy (CPS) for FY14-FY19 (Report No. 80800-MX), the Project was highly relevant to Pillar 2: Increasing Social Prosperity, specifically with Objective 3: Improved access and quality in target education programs. Through the highly targeted focus on improvement of the school trajectories of children living in the poorest and most marginalized municipalities in Mexico, the operation aimed to reduce extreme poverty and boost shared prosperity. The Project also contributed to the achievement of two objectives in Mexico’s National Development Plan (NDP) (2013-2018): Inclusive Mexico and Mexico with Quality Education in that it promoted extreme poverty reduction via improvements in educational attainment. Assessment of Achievement of Each Objective/Outcome 23. Despite the changes in the target of the PDO 1 indicator in the Restructuring in August 2018, this ICR does not conduct a split evaluation of the efficacy of this outcome indicator since the follow-up data collection of the original indicator was never carried out as explained in Section B. Significant Changes During Implementation. As a result, the achievement of the objective prior to restructuring cannot be calculated. Therefore, the achievement of PDO 1 is measured as established in the August 2018 restructuring and PDO 2 as established in the Project Appraisal Document. Improve parental competencies in ECD in targeted rural communities 24. The Project’s achievement related to improving parental competencies in ECD by improving the quality of CONAFE’s ECD services is considered to be Substantial as evidenced by the increase in scores on parental knowledge of ECD, measured by the KIDI. Between the baseline collection in December 2017 and the follow-up collection in December 2018, the score among caregivers on the KIDI increased from 54 percent to 59 percent, a statistically significant increase exceeding the indicator’s target of 56 percent. The baseline of the PDO 1 indicator was collected in December 2017 in a representative sample in four selected states. The final collection was carried out in the same states with a new sample in December 2018 after the redesigned ECD model had been rolled out nationally for all of CONAFE’s ECD services. 25. Changes incorporated in the ECD model were the following: i) improved M&E through the SIIINAFE; ii) improved planning and training contents linked to the improved M&E; iii) increased and strengthened supervision; and iv) a redesigned curriculum. Many of the changes were informed by the results of a 3ie-funded impact evaluation (IE) carried out between 2012 and 2014 and recommendations from the consultancy firm hired by CONAFE to help design the new curriculum, Hacia una Cultura Democrática, A.C. (ACUDE). The IE (Cárdenas, S. et al., 2017) demonstrated that the intervention had positive effects on child development; however, the only positive and significant effect was on gross motor skills 13. The actions taken in response to the areas for improvement identified by the IE and their results are summarized in Table 1 below. 13 Positive but not statistically significant effects on communication, fine motor, problem solving, and social skills. Page 12 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Table 1. Actions taken to improve the ECD model and their results Area for improvement identified Action(s) taken Results observed in IE ECD facilitators had limited Development of module to 10% increase in knowledge of ECD training. Given that 88% had only apply the KIDI to facilitators among facilitators between 2016 and completed primary or secondary through the SIIINAFE. By 2016, 2018. The average level of education, quality training was key the KIDI was applied to ECD knowledge increased positively each to the effectiveness of the model. facilitators through the year between 2016 and 2018 (see information system, allowing for Graph 1 below). It is expected that timely reporting and analysis of the gains in knowledge of the ECD the results. facilitators would contribute to the Improved planning based on increased knowledge of ECD results. Based on the results of practices among parents and the KIDI every year, CONAFE caretakers and therefore the modified the contents of the improved parental practices. training to respond to weaknesses reflected. ECD facilitators had limited Improved supervision. CONAFE supervision. carried out a regionalization analysis to evaluate demand for services and accessibility based on geographic connectivity, highway infrastructure, and transportation availability within regions and identify priority areas for targeting. In assigning CAEEI, CAREI, zone coordinators, and module supervisors, CONAFE prioritized supervision within these zones to target resources where the extra support was needed most. Parental participation needed to Development and consolidation With this information readily be increased. Parents participated of a module in the SIIINAFE available, ECD facilitators and in an average of 11 sessions, but tracking participation per supervisors can identify families to the median was 4 with many session. engage more strongly within the parents having attended 0 or 1 community to encourage sessions. participation in ECD sessions. ECD sessions largely prioritized Redesigned curriculum. The DEI New model engages more actively child-ECD facilitator engagement followed the recommendations with parents rather than with the instead of improved parenting of ACUDE to promote caregivers’ children. practices. self-reflection on their own A qualitative analysis of the new parental practices to manifest model revealed that most parents positive change in child viewed the new structure of ECD development. sessions positively and understood Page 13 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) that the objective of the program was the improvement of their own parental practices as a vehicle of their children’s development. Sources: Cárdenas, S. et al., 2017; CONAFE reports; World Bank, 2019 Graph 1. Score of ECD facilitators on KIDI with confidence intervals Source: World Bank, 2019 26. Although parental knowledge was only measured for the final year of implementation, the growth trend from 2016 to 2018 of ECD knowledge among facilitators suggests an improvement in quality of CONAFE ECD services over the Project’s life. Indeed, the trend coincides with the timeline of incorporation of the changes into the ECD model. As shown in Table 2 below, beginning in 2016, CONAFE began the development of ECD modules in the SIIINAFE and utilized this data to modify training contents based on the KIDI results. CAEEIs and CAREIs were introduced to the operative chain in 2017 and the Bank approved their financing under the Loan in August 2017. The process of designing the new curriculum began in 2016 and it was rolled out nationally in 2018. Table 2. Periods of implementation of major changes to the ECD model 2016 2017 2018 Development and consolidation of ECD modules in the SIIINAFE X X X Planning of training contents and training strategy based on results X X X Improved supervision X X Redesigned curriculum X Source: World Bank, 2019 Page 14 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) 27. All intermediate indicators associated with improving quality achieved or surpassed their targets. The outputs in relation to improved quality include the following: a) An average score of 76.6 percent on the KIDI achieved by a representative sample of ECD facilitators. b) An average score of 84 percent on the KIDI achieved by all 42 CAEEI and 166 CAREI figures. c) 100 percent compliance with the required attendance in at least 2 capacity building sessions by ECD facilitators all years of implementation. d) Publication of a procedural manual and other guiding documents to support implementation of the new model. Increase the transition rate from primary to secondary education level in CONAFE-administered schools in targeted municipalities 28. The Project’s achievement in relation to improving the quality of basic education in CONAFE- administered schools in targeted municipalities is considered to be High as evidenced by the increase in the transition rate from primary to secondary education in targeted municipalities and the results of the IE of the revised API intervention. By Project closing, PDO indicator 2 was achieved, raising the transition rate from the baseline of 60.3 percent for school year 2011-12 14 to 70.0 percent for school year 2016-17. However, the indicator over time experienced a positive trend through school year 2015-16 when it reached 79 percent and then decreased to 70 percent by school year 2016-17. While the end value meets the initial target, the earthquake in Mexico in September 2017 is the most likely explanation for the reversal in the trend. Chiapas and Oaxaca, two of the states most affected by the earthquake, are the ones with highest coverage of APIs. The earthquake did not only affect student behavior, but also CONAFE’s ability to produce relevant information (discussed later in the Key Factors section). Both CONAFE and the SEP’s information systems which held CONAFE records on primary school enrollment and SEP records on lower secondary enrollment respectively, experienced data loss that caused delays in the calculation and reporting of the indicator for almost a year. Given the delays and difficulties caused by the information loss, the GoM and the Bank agreed in the implementation support mission in June 2017 to make school year 2016-17 the final year of measurement of PDO indicator 2 because the 2017-18 data likely would not have been available by June 2019. Since the new API model, discussed below, was rolled out nationally starting in Fall 2017 15, the impacts documented in the project are likely to be lower bound estimates of future improvements. Still, the increase in the transition rate in these municipalities remains an important achievement, particularly as it demonstrates a reduction in the gap between the national transition rate which has remained at approximately 97 percent since the 2011-12 school year 16. 29. The transition rates for both school years 2016-17 and 2017-18 are expected to have been higher based on the results of a rigorous IE started in Chiapas in Fall 2014 and whose initial results were presented to CONAFE in March 2017. The model that was implemented nationwide at the time – henceforth API Standard – was evaluated vis a vis a new API model, henceforth API Plus model, and a control group through a Randomized Control Trial. The Bank had contributed to improve the API Standard model by advising on 1) assignment criteria of the API to the community and 2) increased supervision. The API Plus model, on top of the API Standard changes, increased the initial training from one week to two weeks and provided four additional training sessions 14 Students who graduated 6th grade of primary in June 2012 and enrolled in secondary education in August of that same year. 15 Full API Plus model was rolled out nationally in 2017, but lessons learned from the pilot were incorporated in school years 2014-15 and 2015-16 e.g. strengthened training of COS, emphasis on utilizing more effective pedagogical practices for reading and math. 16 SEP Page 15 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) during the school year. The changes in the design responded to shortcomings identified through the analysis of a survey that the Bank had contributed to analyze. Based on the information collected in Spring 2016, students from both the Plus and the Standard model displayed sizeable improvements compared to those in the control group. For instance, students in the API Standard model and the API Plus saw an increase in their scores on the Early Grade Reading Assessment (EGRA) by 0.14 standard deviations (s.d.) and 0.23s.d. vis a vis the control group respectively. Particularly relevant for the project, as compared to the control groups transition rate of 58 percent, both the API Standard and the API Plus model lead to large increases in the probability of transitioning (8 percentage points and 15 percentage points respectively). For all the outcomes, including measures of math ability and socioemotional skills, the coefficients are larger for the Plus than the Standard group, although the differences are not always statistically significant. Since the API Plus model was scaled-up nationwide starting from 2017-2018, it is very likely that PDO indicator 2 would have been higher if measured for school year 2017- 2018. Table 3. Features and results of API Standard and API Plus versus Control Control (No API) API Standard API Plus Model features API Support No Yes Yes API Training N/A 1 week 2 weeks plus 4 additional trainings throughout the year COS Support Yes Yes Yes, and increased COS support (incentivized by increase in monetary support) Assignment No No Yes based on indigenous languages spoken Observed results (change compared to control) EGRA -- 0.14 s.d. 0.23 s.d. EGMA -- 0.028 s.d. 0.147 s.d. Socioemotional -- 0.080 s.d. 0.20 s.d. wellbeing Parental -- -0.026 s.d. 0.14 s.d. Aspirations Transition Rate -- 8% 15% Source: Agostinelli, Avitabile, Bobba, Sanchez, forthcoming Page 16 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) 30. The evaluation suggests that the new model likely contributed to improving student outcomes because the work with the APIs increased the academic readiness of students and the socioemotional skills that have been linked to academic resilience, therefore increasing the transition rate. 31. Both the initial changes in the API Standard model (the model operating from the beginning of the project until the school year 2016-2017) and the subsequent changes implemented as part of the Plus model that was scaled-up starting from the school year 2017-2018 were implemented under the project and contributed to its achievements. The IE suggests that the changes introduced in the API Plus model can lead to a sustained impact on learning. 32. All intermediate indicators associated with improving quality except one achieved or surpassed their targets. The outputs in relation to improvements in primary education include the following: a) 40.4 percent of community schools at the primary level of targeted CNCH municipalities were supported by an API 17. b) 100 percent of APIs in targeted municipalities attended at least 2 capacity building sessions. c) 79.17 percent of parents participated in activities organized by the APIs and LECs at the end of the school year in targeted municipalities. d) APIs spent an average of about 4 hours in one-on-one sessions with low-performing students during visits to school type A. Justification of Overall Efficacy Rating 33. The Project achieved both objectives related to improving parental competencies in ECD, despite delays in related activities, and increasing the transition rate from primary to secondary education in CONAFE-administered community schools through the redesign of CONAFE’s ECD and APIs programs. The Overall Efficacy Rating is therefore considered to be Substantial. Assessment of Efficiency and Rating 34. The Project’s efficiency is considered to be Substantial. Project funds were used efficiently as illustrated both by the Benefit/Cost Ratios and by the higher than originally expected Internal Economic Rate of Return (IERR) in the case of the ECD component. An economic analysis was conducted for the program interventions: ECD and APIs. The cost per child per year of the ECD component, as estimated by a recent study from Brookings 18, is US$185 which is significantly lower than other interventions targeted to children in Mexico that include cognitive and socioemotional stimulation. For instance, the average cost per child of the Secretariat of Social Development’s childcare program, Programa de Estancias Infantiles, is US$737.40 per year, and US$3,104.02 per year for the Guarderías, childcare centers, of the Mexican Social Security Institute (Hernandez 2018). The cost estimated by Brookings includes overhead costs not included in the costs of the previously mentioned early childhood interventions. Excluding these costs, which account for about 30 percent of overall costs, the cost per child per year is about US$130 19. The cost-benefit ratio of the ECD program is equal to 5.12 with a discount rate of 5 percent. 35. The average cost of the API intervention is US$214.30 per child per year (assuming an exchange rate of MXN$14 per US$1). Given the novelty of the intervention, its cost cannot be compared to other similar 17 This is the one indicator that did not meet its target. CNCH municipalities are among the most disadvantaged communities in Mexico. Often, they are in extremely remote areas that are difficult to access, and the communities are highly resource-constrained, causing limitations on the placement of APIs in these municipalities. 18 The Standardized Early Childhood Development Costing Tool (SECT), The Brookings Institution, September 2017 19 This is marginally higher-cost than previously estimated (US$113) during the IE of the program in 2015. Page 17 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) interventions. The cost, however, is lower than the cost per child of a regular basic education teacher in Mexico, which in similarly sized schools would be approximately US$285 per year. Using the results of the IE of this program, the API intervention generated a cost-benefit ratio of 9.19 with a discount rate of 5 percent. An economic analysis of the API model was conducted at appraisal, displaying extremely high cost-benefit ratio. However, the findings of a more recent IE show large effects of the API intervention on outcomes that were not considered at appraisal, such as socioemotional skills and reading ability, thus justifying larger economic benefits (see Annex 4. Efficiency Analysis for further detail). 36. The Project experienced high turnover among central CONAFE teams (discussed later in Key Factors during Implementation) and delays in activities related to Component 1 due to the budgetary restrictions on consultancy services. Despite the challenges, the Project still managed to achieve its objectives and completed all planned key activities. As previously mentioned, the partial cancellation of the Loan was a response to the increase in the exchange rate between the US dollar and the Mexican peso. D. JUSTIFICATION OF OVERALL OUTCOME RATING 37. Relevance of Project objectives is High, Efficacy is Substantial, and Efficiency is Substantial. Given the limited measurement period of the PDO 1 indicator and the challenges in the final year of measurement of the PDO 2 indicator, the overall outcome rating is Moderately Satisfactory. E. OTHER OUTCOMES AND IMPACTS Gender 38. The Project was gender informed. Observing in the previous operation low levels of participation in ECD sessions exclusively held for fathers and male caretakers, the design of the current Project eliminated those sessions so that participating fathers partake in the regular sessions attended by mothers, children, and other caretakers. Nonetheless, the ECD curriculum promoted more equal gender roles in parenting by emphasizing the importance of support, interaction, and stimulation from the whole family, not just mothers. In the training workshops for the figures of the operative chain at the start of the 2018-19 operative cycle, emphasis was placed specifically on the role of fatherhood in parenting and child development. 39. Historically, the majority of ECD facilitators have been women, generally with less than 8 percent of facilitators being men. In contrast, the majority of APIs are male. While achieving gender parity is often a goal within programs and institutions, in the case of ECD facilitators, female facilitators have reported that their status within their communities has risen because of their role, empowering the facilitators themselves and setting a positive example for other women within the communities. Because APIs travel to support two communities in rural and typically remote areas, the trend of APIs being predominantly male is often due to safety concerns. Institutional Strengthening 40. The Project made an important contribution by supporting the development of the National Information and Infrastructure System for Education Development (Sistema Integral de Información e Infraestructura Nacional para el Fomento Educativo, SIIINAFE), an information system that provides information in real time, thus empowering informed and timely decision making in the M&E of CONAFE programs. During preparation of the previous operation in 2009, the need for an integrated M&E information system was apparent since State delegations would send data in non-standardized Excel spreadsheets to the central level for aggregation. By the close of the Project in 2014, the integrated monitoring modules for the programs supported, including the ECD and APIs intervention, had not been created. Given the lack of delivery of the modules in the previous operation, the Page 18 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) creation of the SIIINAFE alone demonstrates considerable progress. More notably, the degree of operationalization of the system modules for both interventions is at an advanced state 20 and has proven to be useful at the moment of decision-making as explained later in the M&E Utilization section and in retrofitting training contents as explained earlier in the Efficacy section. Resolving the delays in financial reporting experienced in the previous operation and the first few years of implementation of the Project, the system also maintains records of payments made to service providers, streamlining many financial reporting processes. Poverty Reduction and Shared Prosperity 41. The Project supported poverty reduction through investments in supporting ECD sessions and API tutoring sessions; training for figures of the operative chains of both interventions; and technical assistance to improve learning outcomes of children in the most marginalized communities. These investments contributed to improved school trajectories, and consequently, earning potential of future workers. In fact, the benefits of the Project for children who were supported by the ECD Component are equivalent to an increase in per capita annual wages of US$294; the benefits of the Project for children supported by the API Component similarly are equivalent to an increase in per capita annual wages of US$383.37 (see Annex 4 for further explanation). In addition, by focusing on communities with high marginalization and social decline as well as the municipalities targeted by the CNCH, the Project contributed to boosting shared prosperity. By targeting these communities, the Project contributed to providing access to quality education services to the most vulnerable communities and reducing the gap in transition rates between community schools and general schools. Other Unintended Outcomes and Impacts 42. The impacts and innovative nature of the API intervention have earned recognition by various institutions and programs both nationally and internationally. In 2015, the SEP awarded the intervention first place in the category of Improvements in School Management. In 2017, the National Evaluation Council (CONEVAL) recognized the API strategy as one of the three best practices in the use of monitoring and evaluation data for public policy, namely the use of the results of the IE of the API+ pilot to improve the implementation of the strategy. That same year, SUMMA, the Education Research and Innovation Laboratory for Latin America and the Caribbean, included the intervention in a publication recognizing 50 innovative education programs all over the world. In 2018, CONAFE was invited by Results for All to present the API intervention and its results at a workshop in Nairobi, Kenya with government officials, researchers, and other stakeholders from 8 countries. 43. Both interventions garnered attention that resulted in externally financed evaluations and contributed to global knowledge. Because of the successes of the API strategy, CONEVAL offered to finance the IE of the API Plus model in 2018 to measure medium-term outcomes. Likely due to the visibility contributed by the Bank’s engagement, the Brookings Institution chose CONAFE’s ECD program to be part of a cost analysis of ECD programs utilizing a standardized costing tool. The report was published in September 2017 and provided the central CONAFE teams with a means for analyzing the costs of their program in comparison with costs of other ECD programs internationally. 20 For example, some of the information currently reported in the five API-related modules of the SIIINAFE are: a) program universe including services provided, targets, and requests for services outside the program universe; b) applications and functionality to review/categorize applications and send applicants corresponding communications; c) personal information, educational/professional credentials, and service assignments of APIs; d) full reports on pedagogical and operative information reported by APIs throughout the operative cycle; and e) a comprehensive API report that provide summaries of reports, events, monitoring, services, API profiles, and targets achieved. The six ECD- related modules include: a) microplanning; b) records of all figures of the operative chain by locality; c) beneficiary records; d) individual payments made to beneficiaries; e) KIDI application and analysis; and f) hiring modules for CAEEI and CAREI. Page 19 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 44. The Project was designed to contribute to both the educational priorities of the GoM and the Bank’s approaches to improving the quality of education in early childhood and basic education, building upon well- established evidence from the region (e.g. Colombia and Jamaica) and elsewhere that ECD programs can generate large and significant effects on the cognitive and non-cognitive development of children. The Project’s design also integrated lessons learned from previous Bank operations that supported CONAFE’s programs. Furthermore, the improvements to be made to both the ECD and API components were informed respectively by an IE and a survey. 45. The design of the results framework benefitted from the experience gained over decades of engagement with CONAFE. Understanding the nature of the beneficiary populations, the targets were set conservatively and realistically. B. KEY FACTORS DURING IMPLEMENTATION Factors subject to the control of the Government and/or implementing entities 46. Human resources and organizational capacity. Over the Project’s life, there were five changes in director general of CONAFE which often coincided with changes in the ECD and Community Education manager roles and other key personnel contributing to the delays in Project activities. Furthermore, CONAFE was unable to hire sufficient informatics personnel to work on the development and consolidation of the information system planned under the Project. In response, the Bank hired two informatics consultants to support CONAFE’s teams. Despite the frequent turnover and the lack of IT personnel, the Project was able to complete all planned activities besides the FORTALECE-related activities which had been dropped from the Project’s scope. 47. At the state and community level, CONAFE carefully considered human resources as a key component to the Project’s implementation. For the figures of the operative chains of both the ECD and API interventions, CONAFE succeeded in improving selection and assignment criteria where possible (e.g. incorporation of CAEEI and CAREI, API Plus criteria) while still preserving realistic standards given the rural contexts. Precisely to respond to the latter, CONAFE complementarily strengthened training for figures. 48. Training. As mentioned, improving training for both interventions was a key factor during implementation. Through a series of evaluations, CONAFE and the Bank sought to understand the process and effects of the various components of training (contents, structure, duration, etc.) and make improvements throughout the Project. A few examples of improvements made to training under the interventions are the following: a) Regarding contents, training for ECD facilitators was modified based on annual results on the KIDI and training for APIs was modified to focus on hands-on teaching strategies (informed by a survey in the previous operation and the Chiapas IE). The process evaluation of the CEDI’s application revealed opportunities for strengthening training beyond methodology and structure. The evaluation demonstrated that many ECD facilitators were scoring the children’s cards satisfactorily because they assumed the scoring cards reflected their own performance and because of social disinclinations to provide negative feedback, suggesting that training content should also focus on the purpose of activities and consider cultural norms that may need to be addressed and overcome in trainings. Page 20 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) b) In terms of structure, the CAEEI and CAREI were introduced to the operative chain of the ECD program. Because of the dispersion of beneficiary communities throughout rural Mexico, CONAFE trains ECD figures in cascade with each level training the level below. (For information about the workshops carried out in the cascade training structure, see Annex 9.) The training in cascade and the addition of another level of figures to the operative chain were cited in the quantitative analysis as likely contributing to the reduction of fidelity to the new model. However, facilitators and other figures interviewed as part of the qualitative evaluation expressed the CAEEI and CAREI’s importance in establishing a pedagogical framework for the training. Thus, while modifying other features of the training strategy’s structure and modalities represents opportunities for improvement, the finding suggests that the inclusion of the new figures should be maintained. 49. On the other hand, the SWOT analysis of the ECD program demonstrated the weaknesses presented by the training process in cascade. However, the training structure is one of the few low-cost options available given the geographic scope of the ECD program and connectivity issues that prevent the use of remote modalities. By decentralizing the training structure, CONAFE can minimize the costs required to invite all figures to a central location multiple times throughout the year. 50. Program Implementation. The new parental practices model was designed between 2016 and 2017 in collaboration with Hacia una Cultura Democrática A.C. (ACUDE). The new model was piloted from January to June 2018 in 3 municipalities in the states of Queretaro, Veracruz, and Zacatecas. From June to November 2018, a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis was carried out to assess the supply and demand of the new model for the ECD program in comparison to the old model. The analysis was based on interviews with various figures at all levels of the operative chain from the central Directorate of Early Education (Dirección de Educación Inicial, DEI) teams down to the ECD facilitators and also included focus group sessions with parents and caretakers. Among the main messages, the analysis emphasized that the new model should 1) maintain the emphasis on reflection on parenting practices without losing sight of the potential of the program to promote the integral development of boys and girls; 2) reinforce and clarify the curriculum contents, methods, and training materials; 3) improve the process of supervision and feedback; and 4) strengthen the training methodology to ensure adherence and fidelity to the program’s design and objectives. Furthermore, the analysis identified a considerable strength of the pilot noting that family members interviewed (mostly mothers) understood that the program’s main objective is not to directly influence children’s development but rather to improve the parental competency so that their actions promote child development. Mothers also reported that they observed a positive change in family dynamics. Given the messages of the IE carried out prior to the Project, this represents an achievement of the model because it raises the quality of early stimulation of children through their parents and family members who spend the most time with them and therefore have the greatest influence in the early stages of childhood. Factors subject to the control of the World Bank 51. Adequacy of Supervision. The Bank provided close supervision of the Project, carrying out regular implementation support missions and collaborating actively with the GoM on Project activities, particularly when implementation issues arose. On the Bank side, there was relatively low staff turnover of the Project team which was critical considering the relatively high turnover within the DEI. When carrying out its own administrative processes, the Bank prioritized the GoM’s needs and capacity. For example, the two-part restructuring process carried out in 2018 was carried out in that fashion to meet the government’s request that the partial cancellation be completed before annual fees were due. The second restructuring came late in the Project cycle because of the Page 21 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) delays to measure the PDO 1 indicator. While other changes to the RF could have been made prior, the Bank chose to prioritize the more significant implementation issue, the measurement of the PDO 1 indicator and the reconstruction of the final year of measurement of PDO indicator 2. Factors outside the control of the Government and/or the implementing entities 52. Natural Disasters. On September 19, 2017, a major earthquake hit Mexico, damaging CONAFE central offices in Mexico City indefinitely and leaving Morelos, Oaxaca, Chiapas, Guerrero, Puebla, and Mexico City in a state of emergency. In addition to the loss of CONAFE headquarters, hardware and physical documents were also lost and the SIIINAFE was temporarily down. As a result, on October 4, 2017, CONAFE announced the suspension of all institutional activities at the central level until further notice. Activities did not officially recommence until June 14, 2018 when CONAFE central offices were relocated within one of the official buildings of the SEP. Despite the suspension and the challenges presented by the situation, CONAFE central and state teams worked to mitigate the damages caused by the earthquakes in CONAFE communities and ensure that programs were able to continue during this period. For example, State teams and APIs assessed damages in communities and created and acted upon mitigation plans to minimize periods of program interruption. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 53. The Project’s Results Framework (RF) was clearly defined and targets were realistically set based on the context. PDO-level indicators were consistent with the Project’s design, effectively assessing the impacts of the interventions with measurable outcomes and clearly identifying the universe to which they would be applied. Each PDO indicator was supported by intermediate results indicators appropriate enough to demonstrate the contribution of Project activities toward the Project objectives. Indicator targets were established realistically taking context into consideration. Taking into account the Project’s context, the RF was designed prudently, recognizing that seeking to maintain baseline levels of indicators of program coverage in the given context would be an achievement while still striving to improve indicators that would contribute clearly to the improvement of the quality of the programs. 54. While most of the RF was carefully designed to fit the context, the few shortcomings were due to over- ambition during preparation of the Project. Despite concerns expressed by the GoM regarding the cost of application of the HOME instrument to measure PDO 1, the Bank ultimately conceded to two rounds of follow-up applications, one in 2016 and a final collection in 2018 as opposed to annual applications, to reduce the costs of consultancy incurred by the government. As previously discussed, the costs associated with the instrument, given the budgetary constraints experienced by the GoM, prevented any follow-up collection and as a result, the impacts of the ECD intervention as measured by the KIDI are now difficult to compare over time given the short period between the baseline and 2018 collection. Nonetheless, information on proxies of children’s language development and low-cost survey instruments was limited at the time; therefore, home observation of parental practices was considered the best choice at the time despite the costs. M&E Implementation 55. In general, M&E data were collected regularly and in a methodologically sound manner. Baseline data collection was carried out as planned except for delays experienced in the baseline collection of IRI 1 and 2 related to the level of knowledge about ECD practices among facilitators and the percentage of facilitators attending at Page 22 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) least 2 capacity building sessions. At preparation, the Bank and the GoM agreed to establish the baseline within 6 months of Board approval while the surveys to collect this information were being designed. However, the baselines were not established until almost 2 years after Board approval, during which time the Bank had offered support in designing the surveys, defining the samples of facilitators to whom the surveys would be applied, and operationalizing the indicators. This period also coincided with three changes of the director general of CONAFE. 56. Delays in gathering follow-up data for indicators were mostly beyond CONAFE’s control. The final data collection for IRI 1 Level of knowledge of early childhood practices among ECD facilitators was delayed because CONAFE required Bank assistance in the definition of the sample, suggesting that Component 3 could have included more institutional capacity building activities for M&E. However, the delays in collecting follow-up values for PDO indicators were due to federal budgetary constraints that impeded the hiring of a data collection firm to apply the HOME instrument (for PDO indicator 1) and the earthquake in September 2017 that caused delays in the collection of information in CONAFE’s information system, SIIINAFE, and the SEP’s information system, SIGED (for PDO indicator 2). Nonetheless, the GoM and the Bank took actions to mitigate these unforeseen challenges as described in the next section. 57. M&E also benefited from the activities under Component 3 to develop the SIIINAFE. M&E Utilization 58. The GoM and the Bank routinely assessed and used Project data to inform implementation and decision- making. For instance, the following decisions were made to improve Project implementation after analyzing M&E data: a) The instrument of measurement of the PDO 1 indicator was substituted given the delays in data collection. b) Two IRIs (Number of ECD service points that are established and complete the minimum number of sessions provided for by the model at national level and Number of fathers and mothers educated on the ECD model at national level) were modified to be calculated based solely on ECD services managed by CONAFE and to no longer include State managed services. Budget reductions at the State level were resulting in reductions in State managed ECD services 21, causing the indicators to appear as though they were declining over time, despite CONAFE-administered services remaining relatively constant and increasing at times. c) Similarly to the selection and hiring of APIs through the SIIINAFE, CAEEI and CAREI are selected, hired, and notified via the SIIINAFE. d) The KIDI was applied to all CAEEI and CAREI through the SIIINAFE, providing timely information on figures not meeting the minimum requirement of ECD knowledge. Those who did not score the minimum were provided additional training then took the KIDI again. Those who still failed to meet the minimum were duly dismissed. e) The positive results of the IE of the API+ pilot informed CONAFE’s decision to apply the pilot strategy nationally by the 2016-2017 school year. 21For the implementation of the ECD program, states are either Scenario A or Scenario B. In Scenario A, CONAFE finances training and provides direct support to ECD figures. In Scenario B, CONAFE and the State Secretariat of Education collaborate on the program’s implementation with CONAFE financing training and the State providing direct support to ECD figures. Because the States were reducing support to ECD figures, services were reduced in these States. The removal of State-managed services from the intermediate indicators does not have implications on PDO 1 indicator, despite Zacatecas being Scenario B, because the indicator is oriented towards measuring quality of the ECD program and not coverage. The quality of the program is impacted by the quality of training and curriculum which are provided and established by CONAFE. Page 23 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Justification of Overall Rating of Quality of M&E 59. While the design of the Project’s M&E system was comprehensive and suitable to assess the achievement of the Project’s objectives, unforeseen challenges caused by budgetary constraints and the earthquake in September 2017 undermined the effectiveness of the design. Despite the obstacles in M&E implementation, the GoM and the Bank acted in a timely manner and even committed additional resources to mitigate the challenges presented and modify project management. Furthermore, the continuous updates and improvements of the SIIINAFE strengthened CONAFE’s institutional capacity for education management through enhanced M&E. Given the minor shortcomings and the actions taken, the overall quality of M&E is rated Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 60. Social Safeguards. The Project was a Category C project triggering OP 4.10 Indigenous Peoples (IP). There are 68 indigenous languages and 364 linguistic variants in Mexico. As the programs supported by the Project were focused on the population in extreme poverty, indigenous peoples 22 were targeted by default, given that they are part of the most underserved and poorest populations in Mexico. 61. An Indigenous Peoples Plan (IPP) was prepared by the GoM and approved by the Bank in 2014. It was later updated and published in 2016 by CONAFE. 62. Social Risk was rated as Low given that the project was not expected to generate negative impacts for indigenous communities, but rather provide positive social impacts by fostering social inclusion and IP empowerment. The Project did not cause involuntary land acquisition and/or displacement of people, whether indigenous or non-indigenous. Social safeguards compliance was Satisfactory throughout the life of the project. 63. The IPP of the Project had the following components: 64. Component 1: Early Childhood Development a) Support to the AudioCrianza Program. CONAFE’s early childhood education program was targeted to some 30,000 parents and caregivers of indigenous children ranging from 0 to 4 years old. The Project supported the creation and distribution of 5,700 CDs with lullabies and messages for parents and caregivers on children’s rights, the importance of the father’s participation in education, early stimulation, etc. in 4 main languages (highland Náhuatl, Maya, coastal Zapoteco, and Eastern Mazahua), The CDs were disseminated in Campeche, Quintana Roo, Yucatán, State of Mexico, Puebla, Tlaxcala, Michoacán and Oaxaca. Due to a successful use of the CD, CONAFE agreed to reproduce a second round with own funds. b) The Early Education Pilot program contributed to revitalize intercultural knowledge and experiences in early education. It was carried out by certified facilitators and aimed to revitalize intercultural knowledge among indigenous families in Querétaro, Veracruz and Zacatecas. Together with the General Directorate of Indigenous Education, a beneficiary group of 8,500 indigenous children was reached in 2018-19. 65. Component 2: Mobile Pedagogical Support (Asesores Pedagógicos Itinerantes) a) Support provided to IP students with deficiencies in reading, writing and math, through the use of Itinerant Tutors (APIs for its abbreviation in Spanish). 22 In Mexico, communities are considered Indigenous when 40 percent or more of their population self-identify as indigenous. Page 24 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) b) In 2018, there were 583 educational services encompassing 2,070 APIs serving 2,300 indigenous students (18 percent of the total number of services provided by APIs to both indigenous and non-indigenous students at the national level) and working in 31 states. The students were tutored in Náhuatl, Mixteco, Zapoteco, Mixe, Tlapaneco, Wirárica, Rarámuri, and Purepecha, among others. c) A student performance evaluation showed that students’ correct responses improved tri-fold after being tutored in their native language. CONAFE designed a strategy called “Strengthening of Indigenous Languages and Cultures in ABCD” whose text is available to APIs in their service to communities speaking indigenous languages. 66. Financial Management. Through the implementation stage, Financial Management performance was generally Moderately Satisfactory. Project interim unaudited financial reports were generally submitted on time to the Bank, with only minor delays. The audit report covering the period from September 14 to December 31, 2015, included a qualified opinion, revealing important internal control weaknesses at the State Delegation level to manage and control information on payments made to Project beneficiaries. Through the first implementation period, there were no homogenized administrative procedures among CONAFE State Delegations; records and payments information were documented manually and did not allow for the identification of payments made to each beneficiary, individually. With support from the Bank, CONAFE strengthened its institutional information system SIIINAFE, developing, among other, functionalities to record and control the information on payments made to eligible beneficiaries. Although CONAFE was not able to link the SIIINAFE administrative records with its accounting and budget control systems, incorporating administrative information related to payments made to its beneficiaries constitutes an important development that will allow this entity to better control and manage its administrative institutional information. Except for the first audit, audit reports included unmodified opinions 23. 67. The main shortfalls in financial management observed through the implementation of this Project are related to the budget planning and execution processes, in addition to insufficient budget allocation particularly for activities under Components 1.c (design of modules to carry out activities for the Early Childhood Development Component), 2.c (design of modules to provide pedagogical support under the Mobile Pedagogical Support Component) and 3 (Technical Cooperation, Research and Innovation). Through the second half of the implementation stage, CONAFE experienced budgetary reductions and budget cuts for consultancy services. Following a US$30 million partial loan cancellation effective on October 27, 2017, the remaining balance was fully disbursed. 68. Procurement. Procurement was conducted in compliance with Bank procedures and the Operational Manual of the Project. As explained in previous sections, mainly due to budgetary restrictions, only approximately 20 percent (US$0.44 million) of the original resources (US$2.245 million) were used to finance contracts under the category for Component 3: Technical cooperation, research and innovation. C. BANK PERFORMANCE Quality at Entry 69. The Bank provided technical assistance to the GoM to prepare a relevant and strategic Project to the Mexican context, incorporating lessons learned from a series of seven preceding operations. Building on the achievements and lessons learned from the previous operations, the Bank continued to prioritize the 23The last audit report covering the period from January to December 2018, has not been submitted as of the date of preparation of this ICR, and is due by June 30, 2019. The audit is underway. Page 25 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) strengthening of project management and M&E capacity, designating resources under Component 3 for supervision of Project activities. Furthermore, the Bank secured funding support from the Strategic Impact Evaluation Fund (SIEF) to carry out evaluations of the API intervention. The design of the approach to improve implementation of the previous operation was strongly informed by the results of these evaluations as well as a survey completed in 2011 by APIs in 40 communities. For instance, the main changes incorporated in the design of the API+ model, which has been observed to have a greater impact than the API Standard model (Agostinelli, Avitabile, Bobba, Sanchez, 2019), were inspired by the following insights: a) shortcomings in API assignments based on indigenous language abilities; b) insufficient API home visits with parents; and c) irrelevant, ineffective training modules. The Project design considered poverty and social development aspects by targeting the most marginalized municipalities and focusing interventions in community schools. Quality of Supervision 70. The Bank provided close implementation support, regularly conducting in-depth implementation support missions and technical missions particularly in the face of implementation challenges. The Project’s performance was reported candidly, highlighting both the achievements and the challenges faced in the Project’s implementation in order to create informed action plans for improvement. The following actions were taken by the Bank to ensure achievement of the PDOs: a) Recognized the implications of the federal budgetary constraints causing delays in implementation and M&E of Component 1 and worked with the GoM to find suitable alternatives. To ensure adequacy of the ability of the KIDI survey to measure the PDO 1 indicator, the Bank carried out an analysis in a sample of municipalities, which demonstrated a positive, statistically significant relation between parental knowledge of ECD practices as measured by the KIDI and children’s linguistic development. b) Provided technical assistance to define representative samples of ECD facilitators for monitoring the level of ECD knowledge of facilitators, thus contributing to ensuring that a minimum level of quality in ECD service delivery is maintained. c) Carried out a qualitative evaluation of the new ECD model to identify the Strengths, Weaknesses, Opportunities, and Threats of the model, informing areas of improvement. d) Carried out an IE of the API+ pilot, results from which informed CONAFE’s decision to rollout the API+ model nationally. e) Carried out a process evaluation of the implementation of the CEDI to inform areas of improvement in the coordination between the health system and CONAFE to align inter-agency efforts and promote integrated child development. f) Assisted in collecting missing data needed for the calculation of PDO indicator 2 following the earthquake in September 2017. g) Secured additional resources to assist in the development of the SIIINAFE when the appropriate personnel were no longer available within CONAFE. Given the fiduciary and M&E challenges resulting from a lack of an integrated IT platform in the previous operation as well as in the current operation, the delivery and implementation of the SIIINAFE represents a major achievement, since it records payments made to service providers and keeps track of operational data and indicators. 71. The Bank assisted the client in updating the Indigenous Peoples Plan (IPP) to include SEP’s new initiative on Early Child Development and disseminate CDs with lullabies in 4 main indigenous languages and music performed with Indigenous local instruments. Furthermore, the importance of hiring APIs who spoke the native indigenous languages for tutoring was underlined and fostered throughout project life. Overall, the Page 26 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) mainstreaming of OP 4.10 aimed to highlight the importance to educate indigenous parents and caregivers on human rights, early child stimulation, early language acquisition, identity and culture and to support student through the use of IP language. Throughout the life of the project, the Bank provided technical assistance to the client and monitored the IPP compliance until project completion. 72. Though the Bank provided close implementation support, there were moderate shortcomings related to M&E. As discussed in previous sections, the establishment of the baseline of two IRIs was delayed for two years and the follow-up measurement of the PDO 1 indicator was delayed more than a year. Considering the difficulties in assessing efficacy during a measurement period of one year caused by the late substitution of the instrument for PDO 1, the Bank should have followed up more closely with the implementing teams as soon as it became apparent that the indicator would not be measured on time using the original instrument. Justification of Overall Rating of Bank Performance 73. Given the minor shortcomings related to Quality at Entry and moderate shortcomings related to Quality of Supervision, the Bank’s overall performance is considered Moderately Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 74. The risk that development outcomes achieved or expected to be achieved under this Project might not be sustained is considered Moderate. The greatest risk to the development outcomes is that the new administration (December 2018-2024) decides to no longer support ECD figures, APIs, and their supervisors. While lessons learned in development have commonly recommended not financing recurrent costs since these represent new expenditures to be absorbed by the government, the Bank Loan did not represent additional budget for the GoM, rather an alternative source of a line of credit. Therefore, the discontinuation of support for these figures would represent the removal of budget items that historically have been funded by the GoM, breaking decades of precedent. While the following suggest confidence in the continuity of the programs: CONAFE’s long history of over 40 years implementing early childhood and basic education interventions in the most remote areas of Mexico; their extensive record of securing support from institutions such as the World Bank, the Inter-American Development Bank, UNICEF, and others in the last decades; and the positive results of the IE and the national and international recognition of the effects of the API intervention; the possibility of defunding represents a moderate risk nonetheless. V. LESSONS AND RECOMMENDATIONS 78. Adequate training and supervision are key to improving the efficacy of interventions, particularly in rural contexts and contexts with low capacity of service providers. CONAFE’s focus in rural and marginalized communities poses challenges for the implementation of both the ECD and APIs interventions. On one hand, the redesigned API model exemplifies a strengthened training and supervision methodology that improved outcomes for students in basic education in community schools. On the other hand, the ECD training model had achievements in retrofitting training contents based on results but the training structure presented obstacles. Although the new design of the ECD model sought to strengthen the training and supervision by adding the CAEEI and CAREI figures, the cascade training approach was identified as a weakness in the quantitative analysis likely due to the complexity of the operative structure. Because the training scheme involves the trickling down of information from the national level to the communities, as one level trains the next, information can be lost, and messages can be skewed, contributing to lack of fidelity to the program’s design. The lessons learned under the Page 27 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) API intervention were not used by CONAFE to improve the training under the ECD model. This is potentially linked to the very high turnover in the ECD sub-directorate. 79. High turnover at all levels of the Project can impact the successful and timely implementation of Project activities, and often the most appropriate solution is to strengthen supervision and monitoring and information systems. Upon APIs and ECD facilitators’ departures as figures, in order to ensure the continuity of the programs in these communities, it is important that the DEI and the DECIS are aware in a timely manner and have qualified candidates available to replace the figures. In these cases, the role of the supervisors is key for keeping State and consequently, central CONAFE teams informed. The SIIINAFE also strengthened supervision capacity in the face of turnover of figures because it maintained applications and information about qualified candidates, allowing CONAFE to respond quickly. At the national level, the frequent changes in CONAFE leadership often coincided with implementation delays. In these periods, the Bank’s increased implementation support contributed to maintaining implementation progress during transition periods. 80. Indigenous communities are among the most disadvantaged in Mexico and interventions should include measures targeted to their development and success. In the previous operation, the ability to speak indigenous languages was an added benefit of an APIs application and not a criterion for hiring based on demands for services in indigenous language-speaking communities. By strengthening the selection and assignment of APIs based on language, the outcomes of the API Plus model increased substantially compared to the API Standard. Materials for both the ECD and the APIs intervention were developed in indigenous languages to support their development. 81. Consultancy services hired by the Government of Mexico enhance the Project’s implementation and evaluation; however, they have been a recurrent challenge in Bank operations. Similar to previous projects, the procurement of consultancy services has been categorized under a separate category from the two main components of the Project, generating a void of technical responsibility from the client’s side. Assigned technical leadership inside the GoM that endorses it from within can lessen the challenge to procure them. In future operations, Project teams should consider embedding consultancy services within the main components which may result in greater Government commitment to procuring the services despite budgetary restrictions. . Page 28 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve parental competencies in early childhood development in targeted rural communities Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Score of parenting practices Number 54.00 56.00 59.00 among caregivers of children ages 0-4, in selected states 01-Jan-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target achieved 250%. Objective/Outcome: Increase transition rate from primary to secondary education level in CONAFE-administered schools Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Transition rate from primary Percentage 60.30 70.00 70.00 to secondary education for the graduates of CONAFE 01-Aug-2014 31-Dec-2018 31-Dec-2018 administered schools in selected CNCH municipalities Page 29 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Comments (achievements against targets): Target achieved 100%. A.2 Intermediate Results Indicators Component: Early Childhood Development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Level of knowledge about Percentage 66.50 0.00 70.00 76.56 early childhood practices among ECD promotores 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target achieved 287%. Target modified in August 2018 Restructuring because the baseline and target were established following the Project's approval and the end target had not yet been formally set. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Level of knowledge about Percentage 77.00 80.00 84.00 Page 30 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) early childhood practices 04-Dec-2017 31-Dec-2018 31-Dec-2018 among CAEEI and CAREI figures Comments (achievements against targets): Target achieved 233%. New indicator introduced in August 2018 Restructuring. In August 2017, the Bank approved the financing of State and Regional Academic ECD Coordinators (CAEEI and CAREI). The indicator was added to ensure higher qualifications for these coordinators to strengthen the operative chain and service delivery of ECD. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of promotores Percentage 98.70 0.00 100.00 100.00 who attended at least 2 capacity building sessions in 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 the previous operational cycle Comments (achievements against targets): Target achieved 100% Target modified in August 2018 Restructuring. The end target had not been set at Project approval and prior to the Restructuring, had not been formally established. Page 31 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of ECD service Number 17833.00 28787.00 17833.00 19764.00 points that are established and complete the minimum 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 number of sessions provided for by the model at national level Comments (achievements against targets): Target surpassed. Indicator definition, baseline, and target modified in August 2018 Restructuring. The indicator was modified to be calculated based solely on ECD services managed by CONAFE and to no longer include State managed services. Budget reductions at the State level were leading to reductions in State managed ECD services, causing the indicator to appear as though it was declining over time, despite CONAFE services remaining relatively constant and even increasing at times. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of fathers and Number 274168.00 445766.00 274168.00 275126.00 mothers educated on the ECD model at national level 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Target surpassed. Page 32 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Indicator definition, baseline, and target modified in August 2018 Restructuring. The indicator was modified to be calculated based solely on ECD services managed by CONAFE and to no longer include State managed services. Budget reductions at the State level were leading to reductions in State managed ECD services, causing the indicator to appear as though it was declining over time, despite CONAFE services remaining relatively constant and even increasing at times. Component: Mobile Pedagogical Support (APIs) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of primary Percentage 34.26 50.00 42.00 40.40 education community schools in select CNCH 01-Aug-2014 31-Dec-2018 31-Dec-2018 31-Dec-2018 municipalities supported by an API Comments (achievements against targets): Target not met. 79% achieved. CNCH municipalities are among the most disadvantaged communities in Mexico. Often, they are in extremely remote areas that are difficult to access, and the communities are highly resource-constrained, causing limitations on the placement of APIs in these municipalities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of APIs in select Percentage 32.00 90.00 100.00 Page 33 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) CNCH municipalities who 01-Aug-2014 31-Dec-2018 31-Dec-2018 attended at least 2 capacity building sessions in the last 11 months Comments (achievements against targets): Target achieved 117%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of parents who Percentage 52.00 62.00 79.17 participate in activities organized by the API and LEC 01-Aug-2014 31-Dec-2018 31-Dec-2018 at the end of the school year in community schoolsin selected CNCH municipalities Comments (achievements against targets): Target achieved 272%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Time spent by the APIs in Text 1hr 40min 3hrs 20 min 4hrs 3min Page 34 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) one on one remedial sessions 01-Aug-2014 31-Dec-2018 31-Dec-2018 with underperforming children during the last visit to school type A Comments (achievements against targets): Target achieved 143%. Component: Technical Cooperation, Research, and Innovation Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Evaluation of the Child Yes/No N Y Y Development Evaluation Card (Cédula de Evaluación 01-Dec-2017 31-Dec-2018 31-Dec-2018 del Desarrollo Infantil, CEDI) Comments (achievements against targets): Target achieved. Page 35 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Improve parental competencies in early childhood development in targeted rural communities Outcome Indicators 1. Score of parenting practices among caregivers of children ages 0-4, in selected states 1. Level of knowledge about early childhood practices among ECD facilitators (promotores). 2. Level of knowledge about early childhood practices among CAEEI and CAREI figures. 3. Percentage of promotores who attended at least 2 capacity building sessions in the previous operational cycle. Intermediate Results Indicators 4. Number of ECD service points that are established and complete the minimum number of sessions provided for by the model at national level. 5. Number of fathers and mothers educated on the ECD model at national level. 6. Evaluation of the Child Development Evaluation Card (Cédula de Evaluación del Desarrollo Infantil, CEDI) 1. Average score of 76.56% on the KIDI achieved by a representative sample of ECD facilitators. 2. Average score of 84% on the KIDI achieved by all 42 CAEEI and 166 CAREI figures. 3. All ECD facilitators attended at least 2 capacity building sessions all years of implementation. 4. 19,764 ECD service points established and complete the minimum number of sessions Key Outputs by Component 5. Baseline of 274,168 mothers and fathers educated on the ECD model at least maintained every year of implementation and in the final year of implementation, 275,126 fathers and mothers educated. 6. Evaluation of the CEDI 7. 6 modules in the SIIINAFE 8. New parental practices model designed, piloted, and rolled out nationally Objective/Outcome 2: Increase the transition rate from the primary to the secondary education level in CONAFE-administered schools in targeted municipalities 1. Transition rate from primary to secondary education for the graduates of CONAFE administered Outcome Indicators schools in selected CNCH municipalities Page 36 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) 1. Percentage of primary education community schools in select CNCH municipalities supported by an API. 2. Percentage of APIs in select CNCH municipalities who attended at least 2 capacity building sessions in the last 11 months. Intermediate Results Indicators 3. Percentage of parents who participate in activities organized by the API and LEC at the end of the school year in community schools in selected CNCH municipalities. 4. Time spent by the APIs in one on one remedial sessions with underperforming children during the last visit to school type A. 1. 39.9% of primary education community schools (1,498 of 3,751) in CNCH municipalities supported by an API 2. 5,407 APIs supported and trained in school years 2015-16, 2016-17, and 2017-18 3. 220 COS supported and trained in school years 2015-16, 2016-17, and 2017-18 4. All APIs participated in at least 2 capacity building sessions all years of implementation Key Outputs by Component 5. APIs and COS participated in XXX peer learning events, allowing them to share practical experiences and learn from each other 6. Increased parental participation in activities in community schools; 79% of parents participated 7. Increased time on average spent by the API with each low-performing child beneficiary in type A schools, an average of 4 hours and 3 minutes 8. 5 modules designed and in implementation in the SIIINAFE Page 37 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Ciro Avitabile Task Team Leader Wendy Cunningham Team Member Peter Holland Team Member Mariana Escalante Seyffert Team Member Antonella Novali Team Member Gabriel Penaloza Procurement Specialist Juan Carlos Serrano Machorro Financial Management Specialist Supervision/ICR Marcela Lucia Silveyra de la Garza, Ciro Avitabile Task Team Leaders Gabriel Penaloza Procurement Specialist Luis Barajas Gonzalez Financial Management Specialist Alonso Sanchez Team Member Dorothee Georg Social Safeguards Specialist Robert H. Montgomery Environmental Safeguards Specialist Gabriela Grinsteins Counsel Ximena Traa-Valarezo Social Safeguards Specialist Jasmine Anne Pineda Team Member David Omar Islas Orduno Team Member Page 38 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY14 4.850 71,636.52 FY15 18.759 109,891.37 Total 23.61 181,527.89 Supervision/ICR FY15 8.925 67,245.07 FY16 25.639 135,582.16 FY17 26.452 152,073.71 FY18 27.665 181,815.66 FY19 18.945 81,584.12 Total 107.63 618,300.72 Page 39 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) 1. Early Childhood 115.970 95.972 82.8 Development 2. Mobile Pedagogical Support (Asesores 31.410 23.485 74.8 Pedagógicos Itinerantes) 3. Technical Assistance, 2.245 0.168 7.5 Research and Innovation Front-end Fee 0.375 0.375 100 Total 150 120 80 Page 40 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 4. EFFICIENCY ANALYSIS 1. The Project’s benefits are considered substantial. Project funds were used efficiently as illustrated both by the Benefit/Cost Ratios and by the higher than originally expected Internal Economic Rate of Return (IERR) in the case of the ECD component. 2. CONAFE’s ECD program benefited more than 275,000 parents annually in communities with high and very high levels of marginalization. The ECD component of the Project, which supplemented CONAFE’s ECD program, benefited children between 0-4 years who lived in the poorest municipalities throughout the country. The cost per child per year of the ECD component, as estimated by a recent study from Brookings, is US$185 and is significantly lower than other interventions targeted to children in Mexico that include cognitive and socioemotional stimulation. For instance, the average cost per child of the program PEI SEDESOL is US$737.40 per year, and US$3104.02 per year for the IMSS Guarderías (Hernandez 2018). The cost estimated by Brookings includes overhead costs that are not included in the costs either for other ECD programs or for the API intervention. These costs are quantified in about 30 percent of the overall costs. In order to guarantee a fair comparison, we only consider the unitary cost and we exclude overhead costs, leading to a cost of about US$130. This can be thought as the marginal cost of adding an additional child to the ECD program 3. The API component supported children in about 3,500 targeted primary schools annually. The average cost of the API intervention is US$214.30 per child per year (assuming an exchange rate of MX$14 per US$1). Given the novelty of the intervention, its cost cannot be compared to other similar interventions, but is lower than the cost per child of a regular basic education teacher in Mexico, which in similarly sized schools would be approximately US$285 per year. 4. The Project’s substantial economic benefits are also illustrated by the results of the economic analysis of the Project components, which yield a Benefit/Cost ratio that varies between 5.12 (with a discount rate of 5 percent) and 1.20 (with a discount rate of 10%) for the ECD component, a Benefit/Cost ratio between 9.19 (with a discount rate of 5%) and 3.2 (with a discount rate of 10%) for the API component. Table 2 reports three scenarios for computing the Benefit/Cost ratio. The analysis assumes that every child will start to work at the age of 23 and will retire at 62 (i.e. after 40 years of work). To compute the benefits of the different components of the Project, the reference income used is equal to US$3,550.87. This value corresponds to the Mexican median yearly income per capita (source: ENIGH 2012) which was converted using the exchange rate of US$1 to 14 Mexican pesos (the local currency). It is assumed that every child will earn a certain fixed income every year throughout their professional career. The wage has been assumed constant, thus ruling out both inflation and (economic) growth. The Benefits have been computed by considering the fact that those children who have been targeted by the Project’s components will earn a certain extra amount during their working life for their entire working life. Additional individual benefits - e.g. improved health and social benefits - e.g. reduced crime rates - have not been taken into account. Therefore, the results presented in this section have to be considered as lower bound estimates of the benefits. 5. Children targeted by the ECD Component will earn US$217.51 per year more than their peers not exposed to the intervention, as a result of stronger cognitive and non-cognitive skills. This improvement is partly due to the improved parenting practices. The results of an IE and evidence from two data collections in 2018 and 2019 show that attending the ECD program has improved parenting behavior, as measured by the Home Observation for Measurement of the Environment (HOME) and the Knowledge Infant Development Inventory (KIDI) scores. Both in 2018 and 2019, the data show that the KIDI is significantly higher for parents who have attended two or Page 41 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) more ECD sessions in the last four weeks. A baseline parenting behavior equal to 0.54 was found. In a recent evaluation of an ECD intervention in Colombia, Attanasio et al. (2019) find an effect of the intervention on the quality of parental practices by 0.34 standard deviations (s.d.) after two years, that in our case would imply an impact of the ECD program on the KIDI by 8 percentage points after one year. According to Cunha and Heckman (2006), an increase of 10 percent of parenting behavior at the age of 6, as measured by the HOME score, will cause an increase in the earnings of the child equal to 2.5 percent. The assumption is that baseline parenting behavior grew by the same rate until age 6, thus leading to an overall increase by 33 percentage points over 4 years. This is a reasonable assumption due to the strong intertemporal complementarities documented by Cunha and Heckman (2006): an increased investment today makes parental investment tomorrow more productive. It is important to stress that delay in implementing the improved model for parental practices, as a result of the issues discussed above, is likely to have reduced the efficiency of the intervention. 6. The children who were supported by the API component will earn US$383.37 more per year than if they had not benefited from the program. They will have higher wages since they will have a greater probability of completing lower secondary school. In particular, according to the results of an IE that measures the results on transition up to 1 year after the end of the intervention, those children who received API support will have a 13 percentage point increase in the probability of transitioning from primary to lower secondary school, up from 60 percent in the control group, transitioned to secondary school. Taking into account that the retention rate in lower secondary school level is equal to 81 percent, the API intervention led to an 11 percentage point increase in the probability of completing secondary school. Because of this, they will earn a higher income. The higher income has been computed using the coefficient of completing lower secondary school, without completing the full secondary school. Table 1. Summary of Economic Analysis Parameters Children involved in each component of the Project Children supported by the ECD Component 450,000 Children supported by the API Component 17,000 Impacts on earnings Annual median wage (US$) 3,550.87 Increase in per capita annual wages due to Project participation (US$) -Among children who participated in the ECD 294 -Among children who participated in the API 383.37 Increase in per capita annual wages due to Project participation (%) -Among children who participated in the ECD 8.3 -Among children who participated in the API 10.6 Costs ECD Annual Cost per child (US$) 130 API Annual Cost per child (US$) 214.3 Page 42 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Table 2. Benefit/Cost Ratios Discount rate ECD Intervention API Intervention 5% 5.12 9.19 7% 2.77 5.85 10% 1.20 3.21 Page 43 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 5. BORROWER COMMENTS This is a summary of the Borrower’s “Informe de Cierre del Proyecto para la Reducción de la Desigualdad de las Oportunidades Educativas Préstamo 8448-MX – Banco Mundial” and comments to the ICR, translated into English and redacted by the Bank. Any mistakes are the sole responsibility of the Bank. 1. Project and Component Objectives The objectives of the Project were: a) Improve the parental competencies of parents in early childhood development in selected rural communities; and b) Increase the transition rate from primary to secondary education in CONAFE-administered educational services in selected municipalities The objectives of the Components were: Component 1. Early Childhood Development Provide training to promote the comprehensive development of children under four years of age through the provision of training to mothers, fathers, and caretakers of infants with the goal of enriching their parenting practices and directly promote community participation. Component 2. APIs Contribute to the improvement of the learning outcomes of children in rural and indigenous basic community education. The intervention was carried out in localities with high and very high marginalization with low school performance, specifically in reading, writing, reading comprehension, and solvating mathematical problems with basic operations. Component 3. Technical Cooperation, Research, and Innovation Support the provision of technical assistance to CONAFE to support the administration, monitoring and evaluation of the Project, including the completion of any study required accordingly. 2. Costs and Financing of the Project The following table shows the total cost of the Project by category and component. Category/Component Updated Amount (US$) Disbursed Amount (US$) Difference (US$) 1. Early Childhood Development. 94,870,000 95,971,961 1,101,961 Support and capacity building activities under part 1a and 1b of the Project Component 1. Early Childhood 94,870,000 95,971,961 1,101,961 Development 2. APIs. Support and capacity 24,310,000 23,484,588 -825,412 building activities under part 2a and 2b of the Project Component 2. APIs 24,310,000 23,484,588 -825,412 3. Consulting and non-consulting 445,000 168,451 -276,549 services under part 1c, 2c, and 3c of the Project Page 44 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Component 3. Technical 445,000 168,451 -276,549 Cooperation, Research, and Innovation 4. Front-End Fee 375,000 375,000 0 TOTAL 120,000,000 120,000,000 0 3. Achievements of the Project Component 1. Early Childhood Development The Project supported the strengthening of early childhood education which contributed to the improvement of parental practices of mothers, fathers, pregnant women, and caretakers with a positive impact on the development of children. The achievements were made in the following aspects: 1. Curricular design and elaboration of materials Between 2015 and 2016, the following actions were completed to inform the articulation of early childhood education and basic education: design of learning units with specific topics in early childhood education, definition of training tracks for early childhood education and basic education, and the definition of the curricular framework. Following these activities, CONAFE designed the following seven units of autonomous learning with specific topics of early childhood education: a. Who are children? b. Adults in the growth of children c. Games as learning experiences d. Big words for small children e. Songs and lullabies f. Nutrition, communication, and affection g. Small children: their health and safety These units were designed based on the methodology ABCD and form part of the curriculum for basic community education and was also incorporated in early childhood education to strengthen the training of the educational structure which had positive results. The redesign of the ECD model was informed by three external evaluations: 1) an impact evaluation by CIDE; 2) an analysis and proposal for strengthening of the program by CONAFE in coordination with UNICEF and the Center of Research and Studies in Social Anthropology of Mexico; and 3) a theoretic and methodological analysis of the model by ACUDE. Further improvements to the model have also been informed by a SWOT analysis carried out by the Bank. 2. Formation of state technical teams and educational figures Given the focus of the new procedure on parental reflection on child-rearing practices, training offered to educational figures on the ECD program was complemented by the participation of specialists who shared evidence on neuroscience studies, parenting practices in different contexts, including indigenous people, cultural relevance, and focus on the rights of children. Page 45 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) 3. Dissemination strategies and mechanisms CONAFE designed strategies to disseminate information about ECD: • CONAFEcto magazine and web site: a magazine and website with articles and resources about ECD. • Promotional videos: videos that disseminate new ideas, encourage participation in ECD sessions, demonstrate the importance of ECD. • Audiocrianza Project: audio clips with messages about everyday actions that promote child development. Later translated to indigenous languages. 4. Operation and supervision The Directorate of Initial Education has promoted various actions that help to clarify its model, both in a conceptual and operational level, which have been implemented in studies and work documents aimed at the methodological reframing of ECD, in training actions institutionalized to state technical teams and educational figures, and in the publication of support materials and assessment for the updating of some of the existing ones. In addition to the strengthening of state teams, CONAFE introduced CAEEIs and CAREIs to the operative chain in 2017. The distribution of the Regional Academic Coordinators was carried out, prior to a regionalization exercise, in order to identify the priority areas for attention in each of the states. 5. Pedagogical monitoring and evaluation To have up-to-date information to monitor implementation and carry out planning processes, the ECD Module was designed as part of the SIIINAFE which documents the operation of the ECD program in the State Delegations with information of the services in real time, for its monitoring and follow-up. During the execution of the Project between the World Bank and CONAFE, Initial Education made important changes in the planning and monitoring processes that were reflected within the SIIINAFE. The ECD module began its development in 2016 with the incorporation of the records of the educational figures and the beneficiaries. It was consolidated during 2017 considering the processes described below. Definition of targets: The designation of the targets for the ECD services was strengthened due to the establishment of a methodology for the definition of targets, in response to the need to fulfill the demand for services in federal entities where the State Secretaries of Education withdrew the financing for ECD. The revision of the targeting criteria allowed the introduction of a technical file (it is only loaded for CAAEI and CAREI proposals) for the opening of services. In addition, the minimum and maximum number of parents, caregivers and pregnant women was made explicit for the opening and maintenance of the services, as well as the possibility of expansion in case of exceeding the maximum number of attendants to the service. Both aspects were fundamental for the monitoring of Initial Education. In 2017, the Detailed Programming in ECD was incorporated in the corresponding module, in which the potential population and the available targets for the selection of services are displayed, whether for its continuation, reopening, cancellation or new services. Page 46 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Regionalization as a basis for the distribution of CAEEI and CAREI As mentioned in the "Operation and supervision" section, the regionalization process was carried out to account for mobility criteria such as the proximity of the zones to an urban center, the articulation of the road network, and the geographical conditions. In this sense, the services were reorganized to balance the operational load and that the regions will not exceed the integration of eight zones. Once the regionalization of the services was completed, its georeferencing allowed CONAFE to fine tune the assignment of the CAEEI and CAREI. Those areas with the greatest difficulties of access were prioritized. These changes were necessary for the readjustment within the SIIINAFE, which ensured that the figures were assigned to the priority areas established, by placing locks within the system. In addition, CAREI can only view those regions under its responsibility. Monitoring During the 2017-2018 cycle, the monitoring of services increased due to the weekly and monthly monitoring, even with the lack of central office facilities due to the September 2017 earthquake. The weekly monitoring focused on the fulfillment of the goals and the requests of the delegations. At the end of April 2018, the system had graphic tools to facilitate the visualization of compliance with targets at the state level. With the implementation of the SIIINAFE quantitative information is available, in real time, which facilitates the channel of various pedagogical and operational processes. In follow-up to the actions for the strengthening of ECD, the KIDI survey was incorporated into the ECD Module of the SIIINAFE with a sample of 360 Educational Promoters from 17 states. Component 2. APIs To achieve the objective of the component, tutoring by APIs was provided constantly throughout the school cycles 2015-2018. The number of APIs selected and who served in their communities are detailed below: Mobile Pedagogical Tutors (APIs) Target Achievement School Cycle National API Target APIs % 2015-2016 2,099 1,849 87% 2016-2017 2,099 1,802 86% 2017-2018 2,110 1,756 83% The table below shows the participation of Operation and Follow-up Coordinators (COS): Operation and Follow-up Coordinators (COS) Target Achievement School Cycle National COS Target COS % 2015-2016 46 43 93% 2016-2017 92 87 95% 2017-2018 92 90 98% Page 47 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) The following tables show the quantity of schools A supported (targeted schools meeting the established criteria) and schools B supported (additional school supported by the API, chosen for its proximity to school A) through school cycles 2015-2018. School Year 2015-2016 Services Modality TOTAL A B Indigenous 307 284 591 Migrant 12 9 21 Community 1,530 1,526 3056 TOTAL 1,849 1,819 3,668 School Year 2016-2017 Services Modality TOTAL A B Indigenous 310 289 599 Migrant 11 3 14 Community 1,413 1,415 2,828 TOTAL 1,734 1,707 3,441 School Year 2017-2018 Services Modality TOTAL A B Indigenous 316 304 620 Migrant 5 3 8 Community 1,475 1,466 2,941 TOTAL 1,796 1,773 3,569 Work with Parents The following table shows the total number of activities carried out with parents in the period from September to May and the total number of mothers and fathers reported by the APIs that attended those activities. Among the activities that the advisors mention they have carried out are home visits, meetings of the Community Education Associations, and integration of learning communities. In total and throughout the three school cycles, it is observed that the APIs have carried out more than 90 thousand activities and with the assistance of more than 700 thousand fathers and mothers. Page 48 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Work with Parents Total activities carried out with Total of parents attending School year parents activities 2015-2016 30,960 330,613 2016-2017 29,687 206,516 2017-2018 32,393 195,073 TOTAL 93,040 732,202 Services to Indigenous Populations The states that serve the indigenous population within the framework of the strategy are Chiapas, Chihuahua, Durango, Guerrero, Hidalgo, Jalisco, Michoacan, Nayarit, Oaxaca, Puebla, Quintana Roo, San Luis Potosi, Sinaloa, Sonora, Tlaxcala, Veracruz and Yucatan. There is a record of 2,269 students of indigenous services served by the API strategy. This figure represents 18% of total services. The indigenous languages served are: Mixteco, Zapoteco and Nahuatl; in lesser proportion, the Mixe, Tlapaneco, Wirarica, Rarámuri, Purépecha and others. The following tables demonstrate the number of APIs who speak an indigenous language and the indigenous schools supported from 2015-2018. APIs that speak an indigenous Total indigenous services language by school year 630 supported by school year 600 620 620 500 610 400 600 591 300 531 590 583 465 200 326 580 100 570 0 560 2015-2016 2016-2017 2017-2018 2015-2016 2016-2017 2017-2018 Contribution of APIs to the improvement of learning APIs apply an evaluation that appears in their Follow-up Booklet. The evaluation contains 10 items assessing language and communication and 11 items assessing mathematics. At the end of the school year the same evaluation is applied to the students supported. The results for school year 2017-2018 are shown below: Page 49 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) E VA LUAT I O N BEG I N N I N G (I N I C I A L) A N D E N D (F I N A L) O F SC HO O L YEA R 2017-2018 TOTA L I T E MS CO RREC T Lenguaje y Comunicación Pensamiento Matemático 13975 9532 2891 2326 INICIAL FINAL 4. Lessons Learned Component 1. Early Childhood Development • To ensure that the technical-pedagogical functions required for educational figures were performed satisfactorily, it is important to define precise criteria and incorporate evaluation processes for the figures. • It was necessary to implement a leveling workshop for the new CAEEI and CAREI figures to guarantee the mastery of a common conceptual base to facilitate the implementation of the new procedure for reflections on parental practices in ECD sessions. • Investigations revealed that the cascade training process results in information less at each step and sometimes the training contents are distorted by the time it reaches the ECD facilitators. This represents an area of opportunity for CONAFE to consider alternative strategies that allow the most direct information to reach the ECD facilitators. • The reintroduction of activities that promote interaction between adults and children should be considered. Component 2. APIs • Increased training of APIs allowed for a better comprehension of the work methodology and guarantees an effective intervention. • COS with more training contribute to follow-up and training of APIs in the community. • Reading aloud helped promote reading within the classrooms. • Working on problem solving allowed APIs to identify student levels of comprehension in basic operations. • Increased training allowed APIs to improve processes of pedagogical intervention with students, the pedagogical collaboration with the LEC and motivated the participation of the parents. • Continue promoting indigenous linguistic ability in the APIs strategy in order to revitalize, promote, respect and disseminate the native language and cultures of these communities. • Accounting for context and ensuring linguistic and cultural relevance of literacy programs in indigenous communities can strengthen them. • Contributing inclusion and educational equity among all groups of indigenous populations helps to create a more just society. Page 50 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Component 3. Technical Cooperation, Research, and Innovation • The strengthening of the systems of planning, monitoring, and evaluation of ECD was fundamental for its improvement. The systematization of the information within the SIIINAFE not only allowed for timely information but also the validation of the information. In this sense, the administrative records of CONAFE were strengthened. 5. Bank Performance The Bank’s accompaniment and assistance to the GoM was satisfactory from preparation to closing of the Project. The achievements of the Project were greatly supported by the collaboration, technical assistance, and financial assistance of the Bank. The Bank supported numerous studies and analyses such as the impact evaluation of the ECD program, the SWOT analysis of the ECD pilot, the impact evaluation of the API Plus model, the characterization of ECD facilitators, and the analysis of the relation between parental knowledge and children’s linguistic skills. The Bank’s contribution to the development of the ECD module of the SIIINAFE represented an important achievement of the Project. As mentioned earlier, a fundamental aspect regarding Component 1. Early Childhood Development is that the Bank highlighted a positive and significant correlation between parental knowledge and child linguistic development in the Project which supports the idea that a greater knowledge positively affects parental practices and therefore child development. This finding contributes validation to the procedures and strategies being implemented as part of the strengthening process for ECD. Page 51 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 6. SUPPORTING DOCUMENTS Attanasio, Orazio, Teodora Boneva, and Christopher Rauh. Parental Beliefs about Returns to Different Types of Investments in School Children. No. w25513. National Bureau of Economic Research, 2019. Attanasio, Orazio, et al. Estimating the production function for human capital: Results from a randomized control trial in Colombia. No. w20965. National Bureau of Economic Research, 2015. Banerjee, Abhijit V., et al. "Remedying education: Evidence from two randomized experiments in India." The Quarterly Journal of Economics 122.3 (2007): 1235-1264. Brown, "Parental Education and Investment in Children’s Human Capital in Rural China," Economic Development and Cultural Change 54, no. 4 (July 2006): 759-789. Cárdenas, S. et al. (2017) Estimating the effects of a low-cost early stimulation and parenting education program in Mexico. Impact evaluation report 57. International Initiative for Impact Evaluation. Cook, Philip J., et al. The (surprising) efficacy of academic and behavioral intervention with disadvantaged youth: results from a randomized experiment in Chicago. No. w19862. National Bureau of Economic Research, 2014. Cunha, Flavio, et al. "Interpreting the evidence on life cycle skill formation." Handbook of the Economics of Education 1 (2006): 697-812. Cunha, Flavio, and James Heckman. "The technology of skill formation." American Economic Review 97.2 (2007): 31-47. Cunha, Flavio. "Investments in children when markets are incomplete." Rev. Econ. Stud (2013). Dalton, Patricio S., Sayantan Ghosal, and Anandi Mani. "Poverty and aspirations failure." The Economic Journal 126.590 (2015): 165-188. Duflo, Esther, Pascaline Dupas, and Michael Kremer. "Peer effects, teacher incentives, and the impact of tracking: Evidence from a randomized evaluation in Kenya." American Economic Review 101.5 (2011): 1739-74. Gustafsson-Wright, Emily, et al. “The Standardized Early Childhood Development Costing Tool (SECT).” Brookings Institution, Sept. 2017, www.brookings.edu/wp-content/uploads/2017/09/standardized-ecd-costing-tool.pdf. Heckman, Stixrud, Urzua, "The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior," Journal of Labor Economics 24, no. 3 (July 2006): 411-482. Popova, Anna, David K. Evans, and Violeta Arancibia. Training teachers on the job: What works and how to measure it. The World Bank, 2016. Rowe, M. (2008). Child-directed speech: Relation to socioeconomic status, knowledge of child development and child vocabulary skill. Journal of Child Language, 35(1), 185-205. doi:10.1017/S0305000907008343 Page 52 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) Trujillo, José Manuel Hernández. "Condiciones de trabajo e ingreso en la agricultura intensiva mexicana." Revista Análisis Económico 29.71 (2018): 137-160. Walter, Catherine, and Jessica Briggs. "What professional development makes the most difference to teachers." A report sponsored by Oxford University Press. Retrieved on July 20 (2012): 2015. Weisleder, A., & Fernald, A. (2013). Talking to Children Matters: Early Language Experience Strengthens Processing and Builds Vocabulary. Psychological Science, 24(11), 2143–2152. https://doi.org/10.1177/0956797613488145 Page 53 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 7. THEORY OF CHANGE The logic of the results chain is well-supported by evidence in the literature: Increasing parental investment in the early years: Investing in the early years is essential to build a country’s human capital, because the early years are a highly sensitive period for physical, cognitive, linguistic, and socioemotional development. Vast research has demonstrated that brain plasticity and brain development is at its highest during the early years beginning even from the prenatal period, making this a critical window for early stimulation to build the foundational skills that will serve as the basis of all later learning in life. Brain development is sequential and cumulative, meaning that simpler networks develop before more complex ones; thus, foundational skills must be developed before higher cognitive function. There are numerous studies establishing that cognitive ability is a strong predictor of schooling attainment and future earnings. Even more recently, the importance of noncognitive (socioemotional) ability has been emphasized as another predictor of outcomes (Heckman, Stixrud & Urzua, 2006), and the early years is the optimal period to develop these skills by providing early stimulation. Early stimulation refers to everyday activities such as talking, singing, reading, and playing that stimulate babies’ neural development. Parents and caregivers are critical to providing these opportunities as they generally spend the most time with children during the early years and therefore, have the greatest influence on their development. In addition to their active interactions with children, parents and caregivers often determine the conditions of the environment in which children grow and develop which also plays an important role in a child’s development. Among many assertions, Cunha & Heckman (2007) argue that childhood has multiple stages and that at different stages, the same inputs do not produce the same level of skills. Skills produced at one stage persist and affect the skills attained at later stages. For example, numbers of words spoken to children as infants has been shown to be a significant predictor of vocabulary by 24 months (Weisedler & Fernald, 2013). The vocabulary developed by this stage will impact literacy in later stages and subject matter comprehension even later in life. Cunha & Heckman also demonstrate that skills produced at one stage raise the productivity of investments at subsequent stages. For disadvantaged children, the early years thus represent an especially critical time to invest to reduce skill gaps which often are evident by the time children reach school. Community ECD interventions: Various community ECD interventions in the region have demonstrated the efficacy of community interventions on increasing parental investment and improving child development. CONAFE’s ECD model incorporates similar features as ECD programs in Jamaica, Peru, and Colombia such as the delivery of ECD services within communities by community members who are provided training and materials; a well-defined curriculum; and a supervision scheme that responds to geographical contexts and local capacity levels. Improved teaching practices: Effective teaching is one of the most important factors influencing a student’s ability to learn. Well-trained teachers can significantly improve test scores and provide students with the support they need to stay in school and complete their studies successfully. APIs serve as both teachers to the students they support in one-on-one remedial sessions and teacher mentors to the LECs who they support. In the context of community teacher models, effective teacher training thus becomes key to improving student outcomes given the lack of a preservice education. In India, a remedial intervention in which local women with a high school degree Page 54 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) who lead small group remedial sessions were provided two weeks of training at the beginning of the school year and regular reinforcement throughout the year. The intervention resulted in gains in average test scores of all children in treatment schools by 0.28 s.d. (Banerjee et al. 2007). The training provided to the APIs in the new model was improved to include hands-on strategies to teach basic reading and math skills following the evidence that teacher training should be practical (Popova, Evans, Arancibia, 2016) and specific (Walter and Briggs, 2012). Because APIs often have experience as a LEC, they provide mentoring and recommendations on the pedagogical strategies of the LECs. APIs guide LECs in the application of academic diagnostic evaluations at the beginning of the year that serve to identify student academic levels and help LECs and APIs both teach at the level of the students (Banerjee et al. 2007) Remedial interventions: Evidence has shown the effectiveness of remedial prevention programs that support low-performing students by strengthening their foundational skills and providing them support to complete their education. Particularly when targeted to disadvantaged students, small group remedial sessions have been shown to lead to substantial gains in learning outcomes (Cook et al. 2014). One-on-one remedial sessions may be effective because they provide individual education plans taught at the level of the student. They also have the added benefit of reducing the overall class size and allowing the rest of the class to maintain their pace (Duflo, Dupas, Kremer, 2015). Increased parental investment in education: In the face of poverty, people from low socioeconomic backgrounds have misconceptions about the returns to schooling and often ultimately have lower aspirations for education (Dalton, Ghosal, and Mani, 2016). Beliefs about the returns to child investment have been strongly correlated with actual investment choices (Attanasio et. al. 2019; Cunha et. al. 2013). While there is no evidence on the impacts of home visits on learning outcomes of school age children, home visits have been found to be an effective strategy to increase parental investment in preschool age children, thus improving cognitive and socio-emotional outcomes (Attanasio et al., 2015). In the IE of the API Plus model, parents visited by the APIs exhibited increases in aspirations for their children’s future education attainments and increased investment in their children’s education. Page 55 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 8. Targeting Criteria for API interventions Table 1. Criteria and prioritization of coverage for basic community education by CONAFE Criterion Description Priority within criterion A) Multi-grade schools located in localities with high or very 1. Social lag high social lags and/or marginalization in CNCH 2. Level of marginalization municipalities 3. Number of teachers (lowest to highest) B) Indigenous rural multi-grade schools located in localities of high and very high social lags and/or marginalization C) Rural multi-grade schools located in localities of high and very high social lags and/or marginalization D) Other indigenous multi-grade schools E) Urban multi-grade schools located in localities of high and very high social lags and/or marginalization F) Rural multi-grade schools located in localities with moderate social lags and/or marginalization G) Other multi-grade schools H) In case the coverage target is not met with previous criteria, schools with 6 or more teachers can be included respecting the order of priority of criteria A-G. Source: CONAFE, 2018 Page 56 of 57 The World Bank Mexico Reducing Inequality of Educational Opportunity Project (P149858) ANNEX 9. ECD Intervention Training Scheme Figures of the Operative Chain Training Schedule and Descriptions Action Responsible Educational figure to train Period Training workshop at the National Training Responsible Training Team August start of the operative Team (Scenario A or B) 5 days cycle. National level. CAEEI 40 hours CAREI Zone coordinators or module supervisors that can help with state level training Training workshop at the State Training Team CAREI September – start of the operative Zone coordinators October cycle. State level. Module supervisors 2 days 16 hours Training workshop at the Regional Training Newly inducted ECD facilitators September – start of the operative Team October cycle. Zone level. 3 days 24 hours Advising session. Module CAREI, zone ECD facilitators November level. coordinators, module 1 days supervisors 5-8 hours Mid-year training National Training Responsible Training Team March – April workshop. National level. Team (Scenario A or B) 2 days CAEEI 16 hours CAREI Zone coordinators or module supervisors that can help with state level training Mid-year training State Training Team CAREI April – May workshop. State level. Zone coordinators 2 days Module supervisors 16 hours Mid-year training Regional Training ECD facilitators April – May workshop. Zone level. Team 3 days 24 hours Advising session. Module CAREI, zone ECD facilitators May level. coordinators, module 1 day supervisors 5-8 hours Final evaluation meeting Regional Training ECD facilitators June of the operative cycle. Team 1 day Zone level. 5-8 hours Final evaluation meeting State Training Team CAREIs June of the operative cycle. Zone coordinators 1 day State level. Module Supervisors 5-8 hours Page 57 of 57