Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) Report Number: ICRR0021717 1. Project Data Project ID Project Name P152898 Emergency Education System Stabilization Country Practice Area(Lead) Lebanon Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-18789 30-Nov-2018 32,000,000.00 Bank Approval Date Closing Date (Actual) 09-Mar-2015 30-Nov-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 32,000,000.00 32,000,000.00 Revised Commitment 32,000,000.00 32,000,000.00 Actual 32,000,000.00 32,000,000.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD, page 5) and the Grant Agreement (page 5), the project objectives were as follows:  To support the operational needs of Lebanese public schools; and  To improve the learning environment in response to the continued influx of Syrian refugee children. Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) The project objectives remained the same throughout the project period; however, three key outcome targets were revised during a project restructuring (December 2017). At that time, two of the outcome targets were revised upward and one was revised downward. Due to the latter, a split evaluation is undertaken for this project. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components 1. Support to Schools (Appraisal: US$ 13.5 million; Actual: US$15.1 million): This component aimed to provide financing for schools in order to remain operational, through two separate school funds that the government uses to channel finances to schools: School Fund (US$100/student) and Parent Council Fund (US$60/student). Due to the rising costs associated with the influx of Syrian refugee children, this component supported the ability of schools to remain open and operational, by financing student enrollment fees. 2. Learning Environment Quality (Appraisal: US$ 16.6 million; Actual: US$ 15.8 million): This component aimed to improve the quality of the public school learning environment through several activities: rehabilitating school facilities in greatest need of repair; providing small emergency funds to primary schools to help schools improve classrooms and foster greater social cohesion among the different student communities through extra-curricular activities; and providing a full set of the required textbooks to all public school students in grades 1-9 (for two academic years). This component also included support to Direction d’Orientation Pédagogique et Scolaire (DOPS), the department within the Ministry of Education and Higher Education (MEHE) that provides pedagogical, health, and psycho-social counseling. 3. Project Management and Support (Appraisal: US$ 1.9 million; Actual: US$ 1.1 million): This component was to finance the operations of the Project Management Unit (PMU) in the MEHE. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The appraised project cost was US$32.0 million, which disbursed in its entirety.  Project funds were reallocated within and between Components 1 and 2 in order to meet a financing gap (due to the failure of some donor commitments to materialize). The immediate need to increase financing for school enrollment (Component 1), combined with the delays in Ministry of Finance (MOF) transfers to the MEHE and the implementation of a similar school grants program in an Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) ongoing Bank project (Second Education Development project), led to the decision to reallocate funds . Financing  The project was financed entirely by a multi-donor Lebanon Syrian Crisis Trust Fund grant of US$32.0 million, which disbursed in its entirety. Borrower contribution  There was no planned Borrower contribution. Dates  December 2017: The results framework was revised such that two key outcome targets were revised upward (an expansion of project achievement) and one key outcome target was revised downward (a reduction in project scope). The latter target - the number of schools that receive school grants - was reduced from 1700 to 400, due to the slow transfer of funds from MOF to MEHE to finance the grants. At this point in the project period, about 85% of the funds had disbursed. 3. Relevance of Objectives Rationale Lebanon is a high middle-income country with a relatively high level of human development; however, political instability and proximity of external conflicts - most recently, the Syrian crisis - have had destabilizing effects on both the economy and the social fabric. In particular, the public education system is being challenged to maintain quality education for both its own citizens and the recent influx of Syrian refugee children. In 2014, the country was hosting 1.15 million Syrian refugees (of which almost half a million were school-aged children), representing the highest concentration of refugees per capita in the world. The number of Syrian refugee children in Lebanon was already exceeding the number of Lebanese students in the public school system (just over half of all school-aged children in Lebanon attend private schools). The MEHE developed the "Reaching All Children with Education (RACE)" strategy in 2014. The strategy sought to provide continued access to education for refugees, while also preventing a deterioration in the quality of education in the public school system. The strategic framework promoted donor coordination and financing through government mechanisms, and it had the goal to support the development of long-term prospects for refugees, while also reducing the risk of future conflict and regional destabilization. The project objectives were in line with the Bank's Country Partnership Framework for FY17-22, which included a focus area to "expand economic opportunities and increase human capital." Also, the project represented an emergency response to the regional crisis, according to the provisions of World Bank Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) Operational Policy 10.0 (Situations of Urgent Need). However, one of the objectives - to support the operational needs of Lebanese public schools - was imprecisely articulated. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To support the operational needs of Lebanese public schools Rationale In order to keep schools operational during the crisis as well as enable vulnerable and refugee children to attend public schools, this objective aimed to cover operating expenses for schools (i.e. maintenance, water, heating and cooling, security). Outputs were primarily direct financial transfers to schools, on a per capita basis, through a School Fund and a Parent Council Fund. The total per capita amount covered the enrollment fee, thus enabling access to education for poor and vulnerable households. The theory of change for this objective was straightforward, albeit focused on outputs/process rather than on actual outcomes (i.e. maintained enrollment rates for Lebanese school children), given the emergency nature of the Bank's response. Outputs  Provision of financing to public schools to cover enrollment fees (and thus operating costs for schools). Outcomes  201,851 enrollment fees were financed over three years, surpassing the original target of 90,000 (30,000 fees for three years per student) and the (upward) revised target of 190,000. This financing comprised 18 percent of all enrollment fees covered by the government's RACE program over three years. In 2014-15, the public school system contained 347,415 students (of which 237,872 were Lebanese and 93,627 were Syrian refugees); in 2017-18, the public school system contained 483,054 students (of which 264,364 were Lebanese and 209,721 were Syrian refugees). The ICR suggested that this 39 percent increase in capacity is attributable to the government's RACE program. The ICR Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) (page 15) also stated that this financing decreased the cost of education for vulnerable and refugee households, although no specific evidence was provided. Rating Substantial OBJECTIVE 2 Objective To improve the learning environment in response to the continued influx of Syrian refugee children Rationale The theory of change for this objective was overall sound, with the quality of the learning environment likely improved through improved school facilities, provision of textbooks, and provision of supplemental financing for extracurricular activities that would increase social cohesion, particularly in communities with a large influx of refugees. Operational support to the MEHE counseling department also included training to staff on working with vulnerable children. According to the project team, the school system was able to absorb the additional children by enrolling them in under-capacity schools in first shifts and also creating second shifts in existing schools. Outputs  Provision of financing to 410 schools for supplemental school activities (classroom improvements and extracurricular activities) aimed to foster social cohesion between different student communities (original target: 1700). The financing was used to provide sports equipment, music equipment, field trips, instructional materials, and small scale classroom rehabilitation.  Provision of over 8.5 million textbooks (original target: 4.5 million; revised target: 6.0 million). The ICR reported that the higher output level was due to the provision of textbooks for three academic years, whereas the original planned output was for two academic years. The project also financed a study on the textbook supply chain to identify areas of improvement.  Rehabilitation of 10 schools from the priority list (target: 10).  Financing of operational costs all 16 DOPS regional centers, which provide tailored coaching to teachers on pedagogical, health, and psychosocial issues. During supervision visits, school principals and teachers reported that they were highly satisfied with the support provided by DOPS. Outcomes  The textbook quantity and quality study, which included spot checks to schools to ensure availability of textbooks, indicated that more than 90 percent of students in basic education had individual textbooks in the classroom and that the quality of the textbooks was acceptable. During the third project year, the project provided textbooks for all non-Lebanese students in second shifts and for around 75 Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) percent of non-Lebanese students in morning shifts. The remaining cost of textbooks was covered by UNICEF for 25 percent of non-Lebanese students in morning shifts and by the Lebanese Government for all Lebanese students. The ICR also reported the following data on learning and perceptions:  Available data from national exam results showed that pass rates for non-Lebanese children rose from 59.8 percent in 2015/16 to 72.6 percent in 2017/18, despite a 70 percent increase in Grade 9 enrollment during this same period. The Bank's Research for Results (R4R) program also measured student learning according to morning shift vs. afternoon shift schools, as well as in private schools, for Grades 4 and 7 to study progress of student learning outcomes over the course of one academic year (2016-2017). The results showed that second-shift Syrian refugee students start with the lowest baseline learning scores in all subjects (Arabic language, foreign language, and math) at the beginning of the year, but show the greatest improvement in learning outcomes, reaching similar scores to first-shift public school students by the end of the year.  A perception survey also conducted by the Bank's R4R program showed that Lebanese and Syrian parents had positive views on education quality in Lebanon, with 76 percent of parents rating education quality in Lebanon as good or excellent, and 59 percent believing that it is improving. Rating Substantial OBJECTIVE 2 REVISION 1 Revised Objective To improve the learning environment in response to the continued influx of Syrian refugee children (SAME OBJECTIVE but one revised outcome target) Revised Rationale Outputs  410 schools benefitted from supplemental financing (revised target: 400). The ICR (page 16) reported that negative impacts from the downward revision from the original target (due to delays in transferring funds from MOF to MEHE) was likely mitigated by the implementation of a similar grant mechanism under a parallel Bank project (Education Development 2) and also due to the fact that 300 of the beneficiary schools operated double shifts. Outcomes  100% of supplemental grant beneficiary schools were displaying school improvement activities at the facilities (such as displaying results of grant activities on school walls) to raise confidence and improve Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) the perception of Lebanese and refugee parents regarding public schools. Anecdotal evidence of impact of school grants included school principals reporting that the activities improved the learning environment by making the schools more welcoming learning places. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Achievement of the first objective to support the operational needs of Lebanese public schools is rated Substantial due to achievement of targets in the number of school children supported through financing school fees. Achievement of the second objective to improve the learning environment is rated Substantial, under both the original and revised targets, due to achievement of targets in school rehabilitation and textbook provision, although the target for financing schools for extracurricular activities was achieved only under the revised target. Overall Efficacy Rating Substantial 5. Efficiency At appraisal, a cost-benefit analysis for the project was not prepared; instead, the PAD provided a narrative rationale for the project, including a justification for public sector intervention, the need to alleviate fiscal pressure on the MEHE, and expected benefits such as maintaining school operations, investing in school infrastructure, and improving the learning environment. The project design included several elements that likely contributed to project efficiency. These elements included a marginal cost approach to education provision for refugees (compared to a parallel systems approach) and the use of the existing textbook distribution system (compared to commissioning a separate procurement). At project closing (ICR, Annex 4), a cost benefit analysis was conducted with the following elements: assumption that benefits realized were due to increased enrollment of vulnerable Lebanese and Syrian school children, leading to preservation of work productivity and future wages; costs were calculated according to direct project costs. The ratio of benefits to costs was calculated as 1.43, and the internal rate of return was estimated at 12%, with a net present value of US$ 15 million. In addition, the implementation phase included several areas of cost efficiency. These included cost savings of US$ 2.16 million in the school rehabilitation activity, due to the use of in-house engineers (in the PMU) for the planning, design, and supervision phases, and a low unit cost for textbooks (US$ 0.87 compared to US$ 0.91 in Jordan and US$ 2.00-4.00 in Turkey). Minimal PMU staff turnover and on-time project closing also contributed Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) to efficient use of project resources. However, the ICR noted that the slow transfer of funds from the MOF to the MEHE was a significant bottleneck in the pace of implementation; although potential negative impact on school enrollment and textbook distribution was mitigated, the school grants component was only able to launch two rounds rather than three. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 94.10 ICR Estimate  12.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objectives is rated Substantial due to strong alignment with emergency country conditions, as well as Bank and country strategies. Efficacy is rated Substantial under both the original and revised outcome targets, due to achievement of both objectives to support the operational needs of Lebanese public schools and to improve the learning environment. Efficiency is rated Substantial due to evidence of efficient use of project resources in project design and implementation, along with results of the cost benefit analysis. Overall outcome is therefore rated Satisfactory, consistent with only minor shortcomings in Relevance, Efficacy, and Efficiency. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The key project activities, along with the institutional arrangements, are being continued under a follow up operation (RACE 2, P159470, US$100.0 million) that became effective in 2017. The follow up operation also focuses on the long-term goal to improve quality of education, alongside ensuring access to refugee and Lebanese children in the short term. Therefore, development outcomes from this project are likely to be sustained. However, unpredictability in donor financial commitments from year to year poses substantial risk to ensuring that all refugee children have access to public school. Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) 8. Assessment of Bank Performance a. Quality-at-Entry This emergency response project was designed while an ongoing Bank operation - the Second Education Development Project (P118187) - was already under implementation. Thus, this project was designed to align with the longer term objectives of the ongoing project (improved quality), with a much simpler design due to the immediate short-term needs being addressed. The project design was also informed by a comprehensive Economic and Social Impact Assessment of the Syrian conflict's impact on Lebanon, as well as a preparatory analysis of financing options for the enrollment of refugee children. There was strong commitment from the government due to the clear alignment with its own RACE strategy and the designation of the RACE PMU as the main implementing agency for this project. The risk assessment accurately identified the risk of slow MOF fund transfers as a critical risk, though the ICR (page 30) noted that feasible mitigation measures to avoid these delays could not be identified. Lastly, as noted in the ICR (page 26), the results framework was simple but "limited in its ability to measure impact." Quality-at-Entry Rating Satisfactory b. Quality of supervision Proactive supervision, led by a country-based task team leader, ensured effective responses to implementation challenges. The Bank team adopted extra measures to monitor implementation and measure progress, including independent fiduciary audits, assessments of expenditures, and an online platform for more timely data entry on school attendance. Revisions to the project design, results framework, and institutional processes were cleared as needed, thus minimizing any negative impact on achievement of outcomes. For example, the Bank team worked with the MOF to streamline bureaucratic processes such that the transfer of funds between MOF and MEHE was reduced from a six month delay for Tranche #3 to a two month delay for Tranche #4. In the meantime, the Bank approved a direct payments mechanism even though the payment amounts were below the regular threshold amount. The ICR (page 30) also noted that the Bank team was effective in "using the project as a platform to support MEHE in moving from an emergency response to a longer-term development of the sector," citing analytic work to strengthen national systems and partnering with donors to help establish a research program. The latter, in combination with the Bank's convening power, has contributed to the development of longer-term education policies in coordination with the broader donor community. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The key project indicators in the results framework were largely output-oriented (i.e. number of textbooks distributed, number of schools rehabilitated), given the emergency nature of the operation, but nevertheless encompassed all outcomes in the project objective statement. The ICR (page 26) noted some specific shortcomings in the indicators with regards to the limited ability to measure impact, although all indicators included realistic targets. The project's M&E arrangements also included funding for an M&E Specialist position in the PMU to support data collection and reporting, and established a data collection and verification process at the school level. b. M&E Implementation Indicators included in the results framework were measured as planned. Regular progress reports provided data to track progress on the work program, and also identified challenges to implementation. School directors and field coordinators reported on and verified enrollment data, and an online platform was later introduced to monitor daily attendance on a more timely basis. The results framework was modified in recognition of some weaknesses in the original design, leading to revised targets to more accurately capture project achievements. Independent audits, in addition to the regular data reporting activities, were used to verify and inform project implementation. c. M&E Utilization The ICR (page 28) reported examples of the use of M&E in decision-making. During the Mid-Term Review, a decision was made to strengthen data collection activities (more timely and accrurate) to better inform future operations; project indicators were revised to better capture project achievements, according to data reporting progress to date; and the RACE Executive Commission was kept up to date with periodic factsheets. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to minor civil works (school rehabilitation). Appraisal support included preparation of an Environmental Management Plan and an Environmental and Social System Assessment, as well as training for PMU staff on implementing recommendations from the Assessment. Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) The ICR (page 28) reported that safeguards compliance was satisfactory, citing school site visits by Bank safeguards specialists and PMU staff to ensure compliance. b. Fiduciary Compliance The implementation period included annual independent technical audits to assess school-level fiduciary processes, which identified areas that needed improvement in school management capacity. In response to the findings, trainings were provided at the regional level, which, according to the ICR (page 23), supported both meeting of fiduciary requirements of the project and strengthening institutional capacity. Financial management: Financial management performance was overall satisfactory and in compliance with requirements, albeit with delays in the submission of two interim financial reports and some annual audit reports. According to the ICR (page 29), the submission delays were due to the extra time needed by the PMU to install new financial management software, as well as differing donor requirements for annual audits. An independent financial audit was conducted to verify the use of project funds. Procurement: Two post-procurement reviews (covering 60 percent of awarded contracts) confirmed satisfactory procurement performance at the Ministry level. At the school level, a technical audit of school- level fiduciary processes revealed some shortcomings: differences in how procurement processes were implemented among the schools and among the regions; and problems with procurement documentation. The MEHE responded to the findings by developing unified national procurement guidelines and providing tailored training to regional and school staff on procurement. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- High Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) 12. Lessons Lessons drawn from the ICR (pages 31-32), adapted by IEG:  In an emergency education system stabilization operation, flexibility coupled with intensive supervision support by the Bank can help ensure project effectiveness. In the case of this project, timely restructuring, reallocation of project funds, and the use of direct payment mechanisms were examples of flexible and timely Bank responses to implementation bottlenecks.  Even in an emergency situation that demands immediate and short-term support, the Bank can provide additional value through analytic work and research to support longer-term outcomes. In the case of this project, the Research for Results (R4R) program was linked to the project to generate evidence on the education sector, and the project's analytic work helped inform institutional strengthening activities. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR's evidence base was strong, with supplemental data provided beyond indicators defined in the results framework. Inclusion of supplemental data was necessary because, as noted above, the key project indicators were primarily output-oriented and therefore limited in measuring actual project impact. In particular, the ICR reported additional information on the objective to improve the learning environment (improved learning outcomes, parent perceptions) to present a more complete evidence base and thus further substantiate outcomes. The narrative on project implementation was clearly written and informative, as well as candid about minor shortcomings. Lessons provided some specific examples of learning from an emergency operation regarding implementation (i.e. use of direct payment mechanisms) and support for longer- term outcomes (i.e. use of analytic work). a. Quality of ICR Rating High Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Emergency Education System Stabilization (P152898) Page 13 of 13