Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) Report Number: ICRR0021654 1. Project Data Project ID Project Name P118187 LB - Education Development Project II Country Practice Area(Lead) Lebanon Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79660 31-Dec-2016 39,951,596.54 Bank Approval Date Closing Date (Actual) 30-Nov-2010 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 40,000,000.00 0.00 Revised Commitment 39,951,596.54 0.00 Actual 39,951,596.54 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (page 5) and the Project Appraisal Document (page 6), the project objectives were as follows:  To improve teaching quality and the learning environment in general education and in preschool in the Lebanese Republic; and Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187)  To increase governance and managerial capacity of the Ministry of Education and Higher Education (MEHE) and schools. In 2011, the war in neighboring Syria led to an influx of approximately 1.5 million refugees into Lebanon. Education, as one of the major public services provided by the government, was significantly impacted by the increased enrollment of over 500,000 Syrian students. Therefore, government and Bank priorities shifted from sector reform to emergency response. The project design and results framework were revised according to this changing sociopolitical context, although the project objectives and outcome targets remained the same. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components 1. Early Childhood Education (Appraisal: US$ 15.7 million; Actual: US$ 18.7 million): This component aimed to increase the school readiness of Lebanese students upon entry into primary education. Activities included: improving the physical classroom space through rehabilitation; enhancing the preschool curriculum and learning materials; professional development for preschool teachers; and communication and awareness campaigns on the importance of early childhood education. 2. Supporting Improvements in Quality of Education (Appraisal: US$7.6 million; Actual: US$ 6.1 million): This component aimed to support quality learning by focusing on school-level factors and teacher-related characteristics that lead to better learning outcomes. Activities included: implementation of a school improvement model; training for school administrators; school improvement grants; professional development for teachers; and development of a national teacher competency framework. 3. Education Sector Policy Development and Management (Appraisal: US$ 19.3 million; Actual: US$ 15.1 million): This component aimed to support the effective management and coordination of the education sector. Activities included: review of various regulatory frameworks; streamlining of various administrative processes; and strengthening the education management information system. Section 4 below notes the activities that were dropped due to project restructurings. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The appraised project cost was US$ 42.6 million. The actual project cost was US$ 40.0 million. Financing Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187)  The project was mostly financed by an IBRD loan of US$ 40.0 million, which disbursed US$ 39.95 million. Borrower contribution  The Borrower was expected to contribute US$ 2.6 million. The ICR indicated that no Borrower contribution was provided. Dates  2011: Major events occurred, including the collapse of the national government and the war in neighboring Syria, which led to an influx of approximately 1.5 million refugees. This led to a two-year time lapse between project approval and project effectiveness, as Parliament could not convene to ratify the loan agreement.  September 2015: A project restructuring was prepared and approved by the Bank; however, the government was unable to ratify an amendment to the loan agreement given the uncertain status of the national government and lengthy Parliamentary procedures.  November 2016: Due to the above, the project restructuring was modified to essential changes that did not require amendments to the legal agreement, and thus this version of the project restructuring moved forward. The modifications included the results framework (indicators), reallocation of funds, and an extension of the closing date from December 2016 to December 2017.  December 2017: The project closing date was extended again from December 2017 to June 2018 to allow for completion of activities. 3. Relevance of Objectives Rationale At the time of project approval, Lebanon was on track to achieve universal primary education, but the education system still faced several major challenges. Preschool enrollment was low in disadvantaged areas due to inadequate space and poor teaching and learning environments, leading to an increasing gap in school readiness and achievement between students in public (low-income) schools and private (high- income) schools. Private sector provision of education represented two-thirds of schooling for primary and secondary education, and hence there was an oversupply of teaching staff in public schools (7:1 pupil:teacher ratio) but low levels of teacher quality in part due to minimal in-service teacher training and weak quality standards for new teacher hiring. Lebanon had completed the first phase of sector reforms and was moving forward to focusing on quality of learning at all levels. The Bank's Country Assistance Strategy for FY11-14 identified improved quality of education as a key priority. Hence, the project objectives were highly relevant at the time of project approval. Two significant disruptions occurred during the project period in 2011: the collapse of the government and the war in Syria. The latter led to a significant influx of refugees (approximately 1.5 million), including children needing to continue their education. Access to and quality of public schools were heavily impacted Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) by the drastic increases in enrollment. Other emergency education operations were prepared in response to the crisis, in parallel with some modifications to this project in terms of the results framework and activities. However, the project objectives remained the same and therefore were less relevant to the new context. Increasing access to and equity of education were no longer high priorities. The Bank's Country Partnership Framework for FY17-22, while still identifying improved delivery of education services as an overall objective in the education sector, emphasized access and the need to respond to the stresses placed on the education system due to the refugee crisis. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve teaching quality in preschools Rationale The theory of change for the first objective (to improve teaching quality in preschool) and the second objective (to improve the learning environment in preschool) was overall sound. The original design included improving the physical infrastructure of preschools, particularly in disadvantaged areas (i.e. poor and/or remote areas with low enrollment rates), developing learning materials based on an improved preschool curriculum, and training teachers according to agreed national standards. These were all intended to address the significant gap in school readiness between disadvantaged areas (with low quality preschool services) and higher- income communities. Outputs  Provision of training to 790 kindergarten teachers on pedagogical use of new materials (target: 590). Under the restructured project, continuous professional development for teachers was dropped and thus funded by other donors. Outcomes No outcomes reported, including whether there was an increase in the proportion of qualified teachers. Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) Achievement of this objective is rated Modest due to lack of evidence of outcomes. Rating Modest OBJECTIVE 2 Objective To improve the learning environment in preschools Rationale Outputs  Rehabilitation of 162 kindergarten classrooms (target: 153).  Provision of furniture and learning materials to 306 kindergartens (target: 273).  Provision of training to 790 kindergarten teachers on pedagogical use of new materials (target: 590). The parent involvement program and furnishing of teacher training institutes were dropped and thus funded by other donors. Outcomes  The proportion of preschools rated “good” or “excellent” on the Early Childhood Environment Rating Scale (ECERS) increased from 3% in 2017 to 44% in 2018 (target: 60%). The proportion of preschools rated “inadequate” declined from 86% in 2017 to 22% in 2018. The original baseline from 2015 was not comparable due to changes in the instrument design. It should be noted that the 2015 baseline was 36%, meaning that the target of 60% would entail a 24 percentage point increase; with the 2017 ECERS, the baseline was reset to 3%, making the end-line result of 44% a 41 percentage point increase. This indicates a substantial level of improvement over what was originally expected, despite not meeting the formal target of 60%. The ICR (pages 18 and 19) emphasized the "diligent" development and implementation of the ECERS. The ICR (page 18) also cited anecdotal evidence on strong preschool quality from site visits and interviews with school principals during the ICR mission. The ICR also reported that the percentage of 3-5 year olds enrolled in kindergarten in the 230 project schools was 53%, surpassing the target of 20%. This entailed an increase of 6,466 children. Enrollment in non-project schools was 37%. The increase was largely driven by high demand from Syrian refugees as well as Lebanese children. While this outcome reflected increased access, it is not clear what impact this increased enrollment may have had on the "learning environment." Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) Achievement of this objective is rated Substantial due to improved scores on the ECERS, although the timeframe of the results (2017 baseline to 2018 end-line) was very narrow. Rating Substantial OBJECTIVE 3 Objective To improve teaching quality in general education Rationale The theory of change for third objective (to improve teaching quality in general education) and the fourth objective (to improve the learning environment in general education) was overall sound, albeit ambitious for the project period. School-centered development through improved school leadership and school-based improvement models would improve the learning environment. Teacher professional development according to a national system of professional standards and models for pre-service training and continuous professional development were expected to lead to an increased proportion of qualified teachers and reduction in contract teachers. However, given the time necessary to develop and institutionalize (legalize) professional standards, frameworks, and training models, the expected impact on the actual teaching cadre was likely not to materialize for several years. Outputs  Adoption of a National Competency Framework that established professional standards for educators. The ICR reported that MEHE and other development partners provided teacher professional development opportunities in line with the Framework, including an online platform that provided online training.  Provision of training for 291 trainers on teacher training, pedagogical counselor training, and psychological counselor training (target: 260). However,  Some activities were dropped, including support for evaluating current teacher professional development models and deployment and the strategic framework for teacher professional development. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) Outcomes No outcomes reported. Three original PDO indicators -- on the number of schools where at least 60 percent of the classes in every grade level are being taught using active, learner-centered teaching methods, reduction in repetition rates and reduction in number of contract teachers -- were dropped. Achievement of this objective is rated Modest due to lack of evidence. Rating Modest OBJECTIVE 4 Objective To improve the learning environment in general education Rationale Outputs  Development of School Improvement Plans (SIPs) in 419 schools (target: 440). Grants in the amount of US$7,000 were provided to approved SIPs for teaching materials, minor rehabilitation of schools, extracurricular activities, and teacher training. 95% of the schools implemented a self-evaluation process to ensure alignment of grant requests with actual needs.  Provision of training for 291 trainers on teacher training, pedagogical counselor training, and psychological counselor training (target: 260). Outcomes  95% of project schools prepared SIPs that were approved and under implementation, surpassing the target of 75%. Achievement of this objective is rated Substantial due to implementation of SIPs, which were based on self- evaluation processes to link grants with needs. Rating Substantial Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) OBJECTIVE 5 Objective To increase governance and managerial capacity of the Ministry of Education and Higher Education Rationale The theory of change for the fifth and sixth objectives (to increase governance and managerial capacity of MEHE and to increase governance and managerial capacity of schools) was overall sound. The support to school-based management, development of teacher standards, and the enhancement of the education management information system (EMIS) was likely to improve capacity to effectively manage schools. Outputs  Adoption of National Competency Framework standards into the teacher development curriculum, training modules, and preservice training model.  Implementation of ECERS as an assessment tool to inform decision-making.  Upgrading of EMIS and staff training, including connecting of EMIS with all public schools and regional education offices, automation of data collection and analysis processes to enable more timely and reliable data transfer, digitizing of national exam scoring and reporting processes, and training for MEHE staff.  Support for 2011 Trends in Mathematics and Science analysis, including preparation of a final report. However, the development of a medium-term expenditure framework was dropped (funded by a separate Bank operation), and changes to the regulatory framework for MEHE were also dropped due to the low likelihood that such changes could be approved by Parliament in a timely manner. Outcomes  MEHE provided teacher professional development training and opportunities in alignment with the National Competency Framework, thus supporting the integration of teaching competency standards into current teaching practices. MEHE also developed an online platform to enable access to tutorial videos, scenarios, and information on professional standards and virtual training.  The MEHE education information system is now capable of hosting data for all schools and regional education offices. Examples of the increased capability of the EMIS were as follows: MEHE was able to produce disaggregated student enrollment data ten months faster than in previous years, and data on student academic performance and drop out/transition rates were used to inform allocation of resources to schools. Based on the above, achievement of this objective is rated Substantial. Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) Rating Substantial OBJECTIVE 6 Objective To increase governance and managerial capacity of schools Rationale Outputs  Establishment of school councils in all project schools.  Training of 1,181 school administrators on school-based management practices.  Implementation of school self-evaluations, including training of 30 school improvement faciltiators to support school leaders.  Development of SIPs in 419 schools (target: 440). Grants in the amount of US$7,000 were provided to approved SIPs for teaching materials, minor rehabilitation of schools, extracurricular activities, and teacher training. Outcomes  95% of project schools completed the self evaluation process, surpassing the target of 80%. This process was a necessary step to preparing SIPs.  95% of project schools had SIPs approved and under implementation, surpassing the target of 75%.  The ICR (page 21) reported the following improvements in school management: development of a culture of data-driven decision making, collaborative leadership approach, self-confidence, and partnerships between schools and communities. Based on the above, achievement of this objective is rated Substantial. Rating Substantial OVERALL EFF TBL OLD Rationale Achievement of the following objectives was rated Substantial:  Improved learning environment of preschools.  Improved learning environment of general education. Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187)  Improved capacity of schools.  Improved capacity of MEHE. Achievement of the following objectives was rated Modest:  Improved teaching quality of preschools.  Improved teaching quality of general education. Therefore, the overall Efficacy rating is Substantial, although it should be noted that two of the six objectives lacked measurable outcome indicators. Overall Efficacy Rating Substantial 5. Efficiency The economic and financial analysis at appraisal (PAD, Annex 9) justified the investment in improved quality of education based on international evidence of its effect on economic growth. The case was made for significant returns on public investment in early childhood education, the efficiency of a school-based management approach, and benefits of increasing teacher quality. The analysis also attempted to quantify benefits of the project up to year 2025, in the form of savings for the government from reduced recurrent costs per graduate. The expected savings was estimated at US$ 523 million, equivalent to 15 percent of the net present value of total recurrent expenditure during that time period. If the quality improvements lead to an increasing number of students in public school, then the savings were estimated at 19 percent of the net present value. The ICR (Annex 4) presented some unit cost comparisons for appraised vs. actual costs. The average cost per preschool for rehabilitation and furnishing was 23 percent higher than appraised, due to cost increases for the rehabilitation aspect. However, the unit cost for enrolling an additional child in kindergarten, which was estimated at appraisal at US$ 5,333 per child, was 37 percent lower at US$3,335. The average cost per school for the school improvement grant preparation was US$ 14,746 actual, which was 15 percent lower than estimated at appraisal at US$ 17,333. There were also likely efficiency gains due to shared utilization of tools with other donor partners, for example UNICEF (competency framework) and USAID (development of the SIP model). However, there were significant shortcomings. There was a lengthy delay of 24 months between approval and effectiveness, and subsequently an 18 month project closing date extension. The ICR suggested that the accelerated pace of implementation and disbursement during the latter part of the project period led to full implementation and achievement of outcomes nonetheless. However, although overall project costs remained within the original financing envelope, there was a reduced project scope due to dropped activities (professional development for preschool teachers, parent involvement program, furnishing of teacher training institutes, analysis of current teacher professional development mechanisms, medium term expenditure framework, and Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) review of MEHE regulatory framework). Also, many of the key project activities appeared aimed at increasing access and equity, while the project objectives were aimed at improving quality; the project therefore expended considerable resources on activities that were not directly tied to its stated objectives. These factors indicate Modest efficiency. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objectives is rated Substantial due to strong alignment with country and Bank strategies, although the emergence of the Syrian refugee crisis in 2011 shifted priority away from sector reform to emergency response. Efficacy is rated Substantial due to achievements in improving the learning environment in preschools and general education and in improving capacity of the MEHE and schools; however, there was insufficient evidence on achievements in improving teaching quality in preschools and general education. Efficiency is rated Modest due to mixed evidence on appraised unit costs vs. actual costs, as well as lengthy delays and numerous dropped activities. Due to these moderate shortcomings in Efficacy and Efficiency, overall outcome is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Several key activities are being continued under the Bank's current Reaching All Children with Education 2 (P159470, approved in FY2017 for US$ 100.0 million) operation, including teacher training, EMIS data collection, and development of teacher standards. Other capacity strengthening activities were institutionalized into MEHE processes, such as teacher professional development, the school improvement Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) model, and M&E data collection and reporting. However, the highly uncertain political situation and status of Syrian refugees puts the development outcomes at significant risk. 8. Assessment of Bank Performance a. Quality-at-Entry The project was well aligned with the national development and education sector strategic priorities. The project design drew upon lessons from the predecessor Education Development Project I, including the emphasis on the school as a key unit of change and teachers as key actors in improving quality. The risk assessment considered overall risk as Moderate, with most risks mitigated through capacity building activities, including for fiduciary management. However, some institutional support activities for the MEHE required legislative/Parliamentary action and thus were difficult to control. There were also some shortcomings in the results framework (see Section 9). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank team was highly flexible and responsive to significant and challenging shifts in the sociopolitical context, although there were still lengthy delays in formal project restructuring due to factors beyond the control of the Bank team. Some of the key indicators/definitions were changed and some activities were dropped; however, there still remained one key indicator that was less relevant (increase in preschool enrollment) and no measurable indicators for improved teaching quality. Implementation Supervision Report ratings were candid in assigning ratings in the unsatisfactory range during the first half of the project period when activity was stalled. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design M&E arrangements were clear, including responsibilities for data collection and capacity building of MEHE to carry out M&E. There were planned evaluative activities such as a preschool survey and an evaluation Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) of the school-based improvement model. However, the PDO indicator on increased enrollment in preschool was not clearly linked to the objectives of improved teaching quality or learning environment of preschools; the PAD (Annex 4) noted that this was one of main objectives of the national ECE program, but it was not strongly linked to the actual project activities. Some of the original PDO indicators were dropped due to the project's delayed effectiveness (other donors took up these activities) and/or shift in socio-political context such as the influx of Syrian refugees. Other shortcomings in M&E, as noted in the ICR (page 31), included the following: lack of appropriate indicators for the objective to improve the learning environment in general education, weak link between intended outcomes and indicators (reduction in contract teachers, reduction in repetition rates), and weak capacity to collect specific school-level data (number of schools where at least 60 percent of classes in every grade level are being taught using active, learner-centered teaching methods). b. M&E Implementation Although there was a significant delay in filling the M&E specialist position, carrying out of M&E activities was overall adequate. Revisions were made to the results framework to clarify indicators and strengthen linkages to project activities, as well as to improve alignment with restructured activities. However, as noted previously, there were no measurable indicators for the PDOs on improved quality of teaching. The first attempt to collect baseline data for the ECERS was inadequate (given the language and cultural context), and therefore the ECERS was revised but then was not comparable to results from subsequent rounds; hence, only the results from 2017 and 2018 could be used to measure change. Evaluative studies were carried out as planned. c. M&E Utilization The ICR (page 32) reported the following examples of the use of M&E in decision making: the 2015 ECERS report recommended adjusting measurement tools to respond to the Lebanese language and cultural context; the 2017 ECERS report led to tailoring of preschool teacher training in the use of the new material; and M&E reporting results were used as the basis for reallocating funds (such as towards the schools receiving grants). M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as a Category "B" project due to rehabilitation of school buildings for preschool classes. The safeguard policy on Environmental Assessment (OP/BP 4.01) was triggered and therefore an environmental assessment framework was prepared. The ICR does not report on whether there was satisfactory compliance with the safeguard measures. Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) b. Fiduciary Compliance Financial management: The ICR reported that financial management performance was overall satisfactory. A financial assessment conducted during appraisal concluded that financial management systems were adequate, and supervision was conducted on a regular basis. All financial covenants were complied with in a timely manner, and audit reports were submitted in a timely manner with no major problems reported. Procurement: A procurement assessment conducted during appraisal concluded that procurement capacity was adequate. Procurement processes were followed in a satisfactory manner, although with some delays in part due to the turnover in staff. Post-project reviews identified some areas for improvement: filing, contract management, publication of contracts, inventory of project assets, and complaint mechanisms. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons Lessons drawn from the ICR, adapted by IEG: Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187)  Embedding capacity strengthening activities into existing institutional processes can help ensure sustainability. In the case of this project, sharing M&E results as part of the annual review process - for example, the ECERS - helped to ensure use of M&E in decisions to update activities. M&E data was also used to inform resource allocation decisions at the time of project restructuring.  The school improvement process, even with only a modest grant award, can be highly effective if it includes a self-evaluation aspect. In the case of this project, school administrator training included a self-evaluation process, which identified priority needs and ensured a strong link between the grant funding and actual needs. The self-evaluation can also draw upon results from school surveys.  A project can still be effectively implemented in contexts of unanticipated fragility and instability, if there is flexibility in the project design and a strong, persistent engagement with the client on the ground. In the case of this project, restructuring along with additional staff support allowed for a more relevant project design and results framework, additional time for implementation, and responsiveness to changes in government demand and context. 13. Assessment Recommended? No 14. Comments on Quality of ICR The quality of the ICR was overall satisfactory, particularly given the complex backdrop and context of the project that emerged during the project period. The ICR, albeit somewhat lengthy, provided strong evidence and examples of significant institutional impact despite a fragile sociopolitical environment, and also useful examples on how M&E can be well-utilized to improve outcomes. However, the quality of evidence had some shortcomings, as there were minimal outputs and outcomes directly linked to 'improved teaching quality"; therefore, the analysis of achievements also had some shortcomings, as the ICR did not provide convincing evidence to support its Substantial ratings for these objectives. Also, the ICR did not confirm whether the project was in full compliance with safeguard policies. a. Quality of ICR Rating Substantial Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review LB - Education Development Project II (P118187) Page 16 of 16