Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) Report Number : ICRR0021518 1. Project Data Project ID Project Name P132268 GPOBA Solid Waste Mgt West Bank Country Practice Area(Lead) West Bank and Gaza Social, Urban, Rural and Resilience Global Practice L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-15321 30-Jun-2017 8,256,623.00 Bank Approval Date Closing Date (Actual) 08-Feb-2013 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 8,256,623.00 8,256,623.00 Revised Commitment 8,256,623.00 8,256,623.00 Actual 8,256,623.00 8,256,623.00 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives According to both the Financing Agreement (FA) and the Project Commitment Paper (the equivalent of a Project Appraisal Document or PAD) the Project Development Objectives (PDOs) were to improve access to quality and financially sustainable solid waste management (SWM) services for users in Hebron and Bethlehem governorates. This review will assess the PDOs from the FA: 1. to improve access to quality SWM services for users in Hebron and Bethlehem governorates; and Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) 2.to improve access to financially sustainable SWM services for users in Hebron and Bethlehem governorates b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components 1. Output Based-Aid (OBA) subsidies (US$ 8.0 million at appraisal, and actual) financed payments made by municipalities against solid waste disposal bills issued by the landfill operator. Municipalities were reimbursed with subsidy payments after achieving targets in improved services delivered to residents and cost recovery. 2 . Project Management, Monitoring, and Verification Activities (US$ 0.25 million at appraisal and actual). This component financed consulting services for project management support as well as the financing of the services provided by the Independent Verification Agent (IVA) to monitor the service delivery of participant local governments on which release of subsidies were based. Funds financed site visits selected by random sampling, review of reports and documentation, assessing the accuracy of formula used to calculate subsidies, and safeguards compliance audits. A parallel technical assistance component financed separately by the World Bank, the International Finance Corporation (IFC) and other donor partners aimed to increase the proportion of costs covered by user payments. This component formed part of the Project Commitment Paper (the equivalent of a Project Appraisal Document or PAD). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost was US$ 8.256 million, which was fully utilized by project closing. Financing: This Investment Project Financing (IPF) was financed by a grant from the Global Partnership for Output-Based Aid (GPOBA) in the amount of US$ 8.2 million. Funds were fully utilized. The World Bank, the International Finance Corporation, and various donor partners provided a parallel grant financing in the amount of US$ 181,250, also fully utilized. Borrower Contribution: There were no borrower contributions. Dates: The project was approved on February 13, 2013 and became effective on October 2, 2013. The Mid Term Review was conducted on November 9, 2015. The original closing date was June 30, 2017 but was extended by another year to June 30, 2018 to fully utilize the grant. The project start date originally scheduled in June 2013 was moved to October 2013 because the separately funded but complementary Southern West Bank Solid Waste Management Project required additional financing and operational Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) planning for the landfill and transfer station components. There was one level 2 restructuring on May 18, 2015 to amend the Results Framework and extend the project end date. 3. Relevance of Objectives Rationale The PDOs remained relevant to the country’s most recent development plans. The Palestinian Authority’s National Strategy for Solid Waste Management 2017-2022 highlighted the needs of municipalities and local governments to achieve financial sustainability in this sector. Among the priority objectives of this strategy included (i) financial sustainability and efficient solid waste management services and activities; (ii) reducing the cost of collection and transport of solid waste; and (iii) achieving cost recovery and self financing for solid waste management operating costs. The PDOs also remained relevant to the World Bank’s Country Assistance Strategy 2018-2021. Under its first pillar, the strategy called for enhancing the sustainability of municipal services and improve the municipalities financial management systems and processes to attract private sector participation in the sector. The OBA scheme directly contributed to Outcome 1.4 Sectoral and Institutional reforms to encourage private investments in infrastructure. The project also contributed to the Bank’s regional strategy to rebuild the social contract through inclusive and accountable service delivery and a strengthened engagement with the private sector. The Theory of Change, limited to the OBA component, was acknowledged absent in the Project Appraisal Document (ICR, paragraph 13). This theory was premised on the following: (i) consumers were willing to pay increasing fees as services improved; (ii) regular subsidies would be the incentives that municipal councils will use to meet targets related to collecting fees and sustaining operations; and (iii) the solid waste sector remained a national and local investment priority Two components reflected this theory of change - (i) subsidies, and (ii) project management, monitoring, and verification support - to improve access to solid waste management (SWM) services and sustain its operations after the separately funded landfill (Investment Project Financing) was constructed. The subsidies would take the form of regular payments to an experienced disposal facility operator during the first four years of landfill operations when revenues from user fees were expected to be insufficient to cover the costs of service delivery. These subsidies would finance the operating needs of the landfill operator, serve as incentive to deliver better service, improve cost recovery, and sustain project operations in the long run. Subsidies were equivalent to payments made to local governments as credits on their land disposal bills. Funds were transferred when targeted improvements in service delivery have been met as confirmed by an Independent Verification Agent (IVA). As a result, household beneficiaries would receive affordable, improved service quality and the operator would receive regular payments to recover costs, to sustain the collection, transport, and safe disposal of household solid waste. Risks included possible change in government and changes in development priorities. Or other donors may not subscribe to performance based payment system and directly support LGs who do not achieve performance targets. The 2 main outcome indicators were: Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) 1. better service quality reflected in a solid waste management strategy, cleanliness of the target area, and increased sanitary waste management 2 . improved willingness to pay and financial sustainability reflected in improved fee collection, and improved cost recovered through billing. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective to improve access to quality solid waste management services in Hebron and Bethlehem governorates. Rationale Rationale: Improved access to quality solid waste management would be reflected in improved collection of waste, create clean streets, sustainably managed waste by using the newly constructed landfill rather than open dumps and ensured hazardous waste streams were processed in a separate controlled manner. OUTPUTS: • Baseline values were obtained and integrated into the performance network after the May 18, 2015 Restructuring. Improvement of the cleanliness of areas reached 80.7% (baseline 49.5%, original target 45%, revised target 78%, exceeded). This was measured by a Cleanliness Index based on visual inspection of areas and assigned scores based on observed cleanliness characteristics such as presence of waste on street, utilization of bins, physical condition of bins. • 2,300 containers were donated by a 2016 Japanese grant and distributed to the Hebron and Bethlehem Governorates. These outputs were reported to show efforts of the participating municipalities and villages to improve the delivery of solid waste management services and meet desired outcomes. (Per the January 28, 2019 email from the Task Team Leader (TTL), there were no targets for these outputs) and periodically washed achieving better maintenance of waste containers • expanded and improved operation of services in residential neighborhoods with regular and more frequent waste collection (there were no reported targets in the ICR. The TTL explained that this indicator was reported under safeguards monitoring, with monthly, independent verification as designed). • mechanical street cleaning was introduced and operated daily (no reported targets) • trucks were better maintained to prevent leakage of leachate from waste (no reported targets in the ICR; the TTL reiterated that this indicator was indirectly measured under safeguards monitoring and verified independently on a monthly basis). • Indicator 3 was renamed from "Total Waste Managed" to "Increase in Sanitarily Managed Waste to Total Generated Waste Ratio" reached 100% (baseline 0, revised target 95%, exceeded) increase in sanitarily Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) managed waste municipal waste and hazardous medical and slaughterhouse waste were mixed in disposal leading to potential health risks and contamination of public resources • an Independent Verification Agent (IVA) confirmed service delivery of the local governments, conducted site visits and coordinated with an independent auditor to conduct annual project audits and ensured compliance with environmental and social safeguards • parallel technical assistance (i) developed and managed Management Information System (see Section 9, M&E below; (ii) developed guidelines for solid waste management tariff and fee collection mechanisms; (iii) designed and implemented public awareness campaigns. OUTCOMES: • The project substantially improved the quality of solid waste management services for an estimated 840,000 residents (original target, achieved). All 50 participating local governments in Hebron and Bethlehem registered improvements in cleanliness of areas by project closing. All 50 local government units received subsidies for meeting the targets. • Municipalities allocated the equivalent of an additional US$ 17.9 million for operation of their collection and cleaning services over the project implementation period. • Waste generated by both Bethlehem and Hebron governorates was either disposed sanitary in a landfill or recycled. The project also enhanced the quality of the disposal service by introducing protocols for proper management of slaughterhouse and medical waste and provided means to regularly pay, which attracted an experienced private operator. • The IVA confirmed services delivered by participant local governments as a supporting document behind the release of their subsidies to pay their landfill bills to the landfill operator. • The parallel Technical Assistance (i) established the institutional framework through a new tariff and accounting and support for improved fee collection; (ii) developed and maintained MIS; (iii) developed plans for managing slaughterhouse and medical waste for better landfill management service quality; (iv) developed public awareness campaigns; and (v) developed plans to close unsanitary dumpsites. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective to improve access to financially sustainable solid waste management services in Hebron and Bethlehem governorates. Rationale Rationale: The increase in the proportion of costs covered by user payments signal fiscal resilience and sustain waste management service delivery. Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) OUTPUTS: • In Hebron, fee collection reached 82.0% by 2017 (baseline of 42.7%, revised target 80%, exceeded). • In Bethlehem, fee collection reached 81.7% in 2017 (baseline 48.3%, revised target 80%, exceeded). • 84.4% increase in billing to cost ratio was achieved (baseline 76.8%, revised target 81%, exceeded) • use of landfill rather than open dumps achieving target • increased coverage of waste collected (TTL confirmed that there were no targets in the ICR but that this increase was indirectly measured by the Cleaning Index noted above and the total waste sanitarily managed compared to the amount generated) OUTCOMES: • The overall target fee collection ratios were exceeded for each governorate although varied ratios were reached for 30 Hebron local government units and Bethlehem 20 local governments. At project close, 2 Hebron local government units and 4 Bethlehem local government units would not achieve their 80% targets. Lagging government units were targeted for capacity building and training resulting in improvements from their respective baselines but falling short of targets. As a result, these local government units did not receive subsidies. • All local governments experienced improvements in billing to cost ratio except 4 Hebron local government units who did not achieve targets and did not receive subsidies. The TA outcomes under Objective 1 were also relevant in achieving this objective. changes in fee collection strategy established the tariffs, accounting framework for the fees and workshops to develop billing strategies based on each municipality’s operating context. For example, solid waste fees were attached to the electricity bill in some Hebron local governments. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale The targets were achieved or exceeded at project closing. Results were monitored through the MIS installed and independent verification carried out. Output based approach improved the quality of services and achieved financial sustainability in the Hebron and Bethlehem governorates. Cleanliness of streets were achieved, an experienced operator managed the landfill service, slaughterhouse and medical wastes were better managed, and waste disposed in sanitary landfill rather than open dumps. The incentive system resulted in investments in new equipment, expanded coverage, improved the quality of services, and adopted better equipment and vehicle maintenance. Fee collection and cost recovery improved due to better accounting, establishing tariffs, introducing billing practices and increasing fees for commercial users. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency: A financial cost analysis used during appraisal was updated at project closing using actual figures. A cost benefit analysis was also undertaken using "with" and "without" output based subsidies. The benefit cost analysis was complemented by an analysis of cost effectiveness indicators. The Economic Internal Rate of Return (IRR) was the discount rate that made the Net Present Value of project cash flows equal to zero. Since the cash flow was negative in the "Do nothing scenario" and positive in the other two of the three scenarios, there was no discount rate used for zero cash flows of NPV. Instead, the NPVs of the cash flows were used under the three scenarios below. Cost Benefit Analysis: The cost benefit analysis covered the project period, 2014-2017 and included the entire solid waste system, benefits and costs of during appraisal (status quo), during project implementation (actual scenario) and then compared benefits and costs attributed to the subsidy and related improvements brought about by the project (actual scenario without subsidy benefits). Benefits were derived from (i) the value of solid waste services expressed in user fees collected from households and businesses and those from municipal sources allocated to solid waste management; (ii) reduction in green house gas emissions using the social cost of carbon of US$ 42 per metric ton of CO2; (iii) heath benefits (reduced respiratory diseases, reduced lost work days due to illness were used as proxy at appraisal but post project analysis used reduced health care costs per year as used in the parallel Southern West Bank Solid Waste Management Project); (iv) job creation/losses as calculated also in the parallel Southern West Bank Solid Waste Management Project); (v) avoided costs of aquifer contamination; (vi) value of recycling; and (vii) others such as the 15% corporate tax to be levied on revenues generated by the private landfill operator. The costs were derived from (a) the status quo scenario using costs for transfer and landfill, and primary and collection; (b) actual scenario using costs for transfer and landfill, primary and collection, costs for sorting facility, administering the subsidy and the technical assistance program; and (c) actual scenario without subsidy using costs from actual costs of primary and collection, and the sorting facility. The ICR did not provide the costs benefit ratios for each scenario, just the assumptions and how it was calculated and concluded that the comparable NPV of the three scenarios - (I) status quo or no subsidies; (ii) actual scenario with OBA subsidies and improvements under the project as implemented; and (iii) actual scenario without OBA subsidies but with improvements. The positive NPV of US$6.1 million under the second scenario was proof of the economic efficiency of the project. The negative NPVs under the first scenario (US$ -1.5 million) and that of the status quo scenario (US$ -6.2 million) supported this claim. Financial Cost Analysis: The estimated lifetime of the landfill was 25 years. Fee revenues across Bethlehem and Hebron increased from US$ 6.1 million over a 6 month period to US$ 16.3 million for a similar period at project closing. If revenue collections remained the same as the final 6 month of the project period, revenue gains would reach US$ 508 million. Operational costs increased by 42.6% because of the shift from open dump to sanitary landfill, which required tipping fees. Fee revenues (own source revenues) covered 69% of the increased operating costs by project closing compared to 33% previously. The US$ 8.0 million subsidy amounted to US$ 9.50 per resident (840,000 residents), diverted 800,691 tons of waste from open dump to sanitary landfill, and divert more than 7 million tons of waste over 25 years at an estimated cost of Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) US$ 1.10 per ton. With estimated revenue gains of US$ 508 million for the landfill’s 25 year economic lifetime, less than US$ 0.02 of subsidies were spent on every dollar of expected returns. Operational and Administrative Efficiency: Start up delays were due to difficulty in meeting effectiveness conditions - the concession agreement with the landfill operator was the first Public-Private Partnership and required time to reach agreement. The project used up all its intended resources over the project period although a Restructuring required the extension of the project closing to allow for the full use of the funds and achieve the PDO. Restructuring also acknowledged the experience from the first two reporting periods due to unrealistic assumptions and capacity constraints of participating local governments. Technical assistance and training targeted these constraints but the project team wanted to be prudent and allow time to complete reporting cycle, data cleaning, aggregation and verification following the performance based financing of the project. Additional financing and operational planning was needed for the separate but parallel Southern West Bank Solid Waste Management Project, which delayed the project start from June to October 2013. A baseline was established during the first year of implementation because of a 3 month project preparation. Project funds were fully utilized. A surplus of US$ 69,822 was distributed as awards to high performing municipalities as incentive to continue their sustainable disposal practices. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The Relevance of Objective was rated Substantial. The Efficacy of both Objectives was rated substantial after achieving and exceeding targets. Efficiency was rated substantial, resulting in an overall outcome rating of Satisfactory. There were only minor shortcomings in the design of achieving project efficacy due to a lack of a baseline arising from the rapid preparation of project design but this was corrected at the 2015 Restructuring. In addition, the minor drawback in efficiency arose from the contract preparation for the landfill operator, which, while separately funded, was tied to the operations of this pilot project. With these minor shortcomings, the outcome rating was satisfactory. Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR reported modest risks to development outcomes since this only a pilot operation that informed future interventions in the sectotr. Risks came from the following: 1. Local Government Capacity to Sustain Financing of the Sector After Subsidies. Continuing willingness to pay for services were linked to service quality. Present political will continue to pursue citizen education and municipalities continued to recover 59% of higher operational costs from own source revenues, adjusted billing and collection strategies that were piloted by the project. Surplus subsidy during the final disbursement period rewarded well-performing municipalities and served as incentive to continue with the project after closing. Capacity was also aided by the numerous training provided by both the Bank team and the implementing agency. Falling interest would be mitigated by continuing engagement with the territory through other Bank engagements. 2 . Technical risks were low because the waste disposal infrastructure (sanitary landfill and transfer stations) were fully operational and supported by an ongoing IFC support for PPPs. Illegal dumpsites have been closed. Operational improvements in the sector introduced by the project have been maintained. The numerous trainings have increased local capacity to appreciate transparent and evidence based planning for the sector. 3 . There were low risks to the project outcome from security issues and change in government because the technical formula used in formulating the subsidy and the incentive scheme created were already in place. 4 . There is a substantial risk that gains achieved from disposing waste at the Al Minya landfill (a separately funded project altogether but which has impact on the outcome of this project) may not be sustained because of a lack of capital investments for future expansion and increased operating costs due to increasing volume of leachate and related odor. With full capacity of the landfill approaching well before reaching its economic life of 25 years, investment needs for transfer stations increase. This risk could be mitigated by (i) additional donor support to finance the capital investments, and (ii) modifying the accounting for reserves by allowing the implementing agency to set aside predicted operating surplus and include this with an increase in gate fees to finance capital investments. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank team designed this project based on outcomes of prior similar operations in the water and electricity sectors that have applied Output Based Aid approaches. Lessons drawn upon in project design included (i) alignment of OBA projects with existing institutions and government systems to increase replication and sustainability; (ii) need for political champion and technical capacity for OBA schemes; (iii) Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) need to pilot PPP and OBA schemes. as part of a wider solid waste management system with adequate resources and clear transitional processes. Alternative approaches were consulted and field tested. However, because of the rapid preparation (3 months), baseline values were not established until after the first year of implementation. Subsidies were originally disbursed based on aggregated performance but disaggregated according to individual local government performance. The formula proved confusing to some. The original incentive scheme benefited weak performers because scores were aggregated and driven up by good performers. This could have created a moral hazard because weaker performers were rewarded disproportionately. In addition, the M&E system and ESMP were only developed in the second year. These minor shortcomings at entry were mitigated during implementation through alternative approaches such as introducing the "Exclusion Rule," which enabled better performers to maximize their subsidy credits and prevent weaker performers until they work toward their targets. The theory of change was supported by adequate key performance indicators. However, in one case, the team realized that using the aggregated performance of the governorates to meet targets for releasing subsidies introduced a moral hazard that did not adequately distinguish between poor and better performers among the participant local government units. Mitigating measures were installed in the interim to adequately correct this bias. The need for an effective MIS to track performance was identified as critical for implementation. However, initial delays due to difficulty in fulfilling effectiveness conditions delayed project start revealing an underestimation of the time needed to negotiate and conclude the first public- private partnership in the sector. The reporting capacity of local governments were overestimated as evidenced by the lag in reporting of key performance indicators and the verification period needed. The minor shortcomings at entry were corrected during the 2015 Restructuring. Quality-at-Entry Rating Satisfactory b. Quality of supervision The project was adequately supervised even though there were three Task Team Leaders for the duration of project implementation. The team conducted biannual supervision missions. The team met regularly with counterparts, conducted site visits and held trainings. Technical missions were carried out to ensure progress was on track and work was progressing toward achieving the PDOs. The project Implementation Status Reports all reported satisfactory throughout project implementation. During the first year of operation, the recipient already exceeded the target values regarding improvement in the cleanliness of streets. These targets were raised from, 35% and 45% initially to 68% and 78% at project end. During the 2015 Restructuring, the name of indicator 3 was changed from "Total Waste Managed" to Sanitarily Managed Waste to Total Generated Waste Ratio." And extended the project closing date by 12 months to allow for the full use of the grant and provide sufficient time to achieve the PDO. Project was delayed because of additional financing and operational planning needs of the landfill and transfer stations as part of the parallel Investment Project Financed Southern West Bank Solid Waste Management Project. Training in fee collection, public awareness campaigns, and consultations in improving primary waste collection were provided throughout project implementation to raise the capacity of both the implementing agency and those of the participant local government units. Bank staff also trained the implementing agency on environmental and social safeguards. Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The Implementing Agency was the Joint Service Councils of Hebron and Bethlehem (JSC- H&B). The M&E system developed consisted of monitoring both performance indicators and safeguards monitoring. The objectives were clear. The theory of change was not included in project documentation at appraisal even though the key activities improved service quality and supported financial sustainability of the improved service. The results framework reflected aggregated key performance indicators that had target values that were not updated even after implementation started causing some confusion during implementation (see below). The indicators encompassed all outcomes of the PDO statement but the results framework needed to be amended because (i) the baseline values for the M&E system was prepared a year into the project start due to limited time at preparation, (ii) target value for an outcome indicator was revised because the baseline value showed that this had been achieved, and (iii) an indicator was renamed to better convey the outcome. The five intermediate results indicators adequately captured its contribution to achieving the PDO level outcomes. After revisions, the indicators were specific, measureable, achievable, relevant, and time bound. Baselines and targets were available for the key and intermediate outcome indicators after the project started. Performance monitoring was not conducted between October 2013 and March 2014 while safeguards monitoring was implemented only in 2015. This design resulted in a one-month gap in monitoring performance indicators and safeguards that led to difficulties in synchronizing the results between the two monitoring systems (see below). M&E design improved after the 2015 Restructuring and was adequately installed within the implementing agency. b. M&E Implementation The JSC- H&B monitored the local government’s performance and distributed subsidies based on meeting performance targets - (i) quality of services delivered to households and (ii) cost recovery, guided by Key Performance Indicators (KPIs). Subsidies were phased out as local tariffs and fee collection rates increased and services improved. One year into implementation, in 2015, an MIS software was installed to harmonize data collection and management across Hebron and Bethlehem, improve data quality and support M&E of performance indicators. A subcontracted independent IVA verified performance reports every 6 months. Project Outcomes were evaluated using 4 main indicators to reinforce the PDOs. Each set of indicators was based on municipal data and then separated for each governorate. Five performance indicators were selected to be monitored throughout the life of the project with a minimum and maximum target values. These values increased as performance improved. A weighting system allocated Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) more payments to more important indicators. The installed MIS tracked performance against indicators. Local governments reported monthly on results. The implementing agency consolidated local government reports into semi-annual reports. Each report was verified by the Independent Verification Agent. Complete target values were not updated in the project’s Results Framework, which caused confusion in updating targets. The 2015 Restructuring introduced corrective measures and target values realigned for harmonized result reporting. See Section 10(a) below for Environmental and social safeguards monitoring and reporting. Performance reporting was not conducted during the first period (October 2013 - March 2014) before the MIS was installed. Local government performance was monitored using the MIS installed in 2015. Participating local governments received three training sessions on project monitoring and reporting. Municipalities that struggled with monitoring and reporting received additional customized training. All municipalities used the MIS system to report results. IVA reports found few discrepancies in the bi-annual reports prepared by the implementing agency. All reports were submitted on time. c. M&E Utilization The new MIS was fully utilized, which established a regular practice of tracking solid waste management progress against pre-defined indicators. The MIS assessed progress against performance indicators every six months, which triggered whether or not to disburse subsidy payments The MIS allowed transparent and diligent monitoring of progress since stakeholders could identify successful and lagging participants. This information also allowed the implementing agency to direct customized technical assistance to lagging participant local government units.The use of the system led to transparent accounting and reporting. Solid waste management authorities could make long term decisions based on data and evidence. With the minor drawback brought by delayed design, resulting in some difficulties in cross comparison of results early on, the overall rating of M&E quality was substantial. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was dedicated to providing subsidies and was rated as not requiring an Environmental Assessment and did not trigger any safeguards. However, the existing parallel International Finance Corporation-funded Southern West Bank Solid Waste Management project prepared Environmental and Social Impact Assessment and Abbreviated Resettlement Plan. According to the "Application of Bank Safeguard Policies to GPOBA Transactions," IFC’s established performance standards were appropriate to evaluate the project. The project was implemented in compliance with relevant World Bank Environmental Health and Safety (EHS) Guidelines including the EHS Guidelines for Waste Management Facilities. Three environmental and social risks (litter and clandestine dumping, air emissions, personnel safety) were identified with mitigation measures that municipalities were held accountable. Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) The Grant Agreement indicated that the implementing agency would be responsible for coordinating the adoption by and supervision of participant municipalities. An Environmental and Social Management Plan (ESMP) and Environmental and Social Management System (ESMS) were disclosed January and February 2015, respectively. The ESMP monitoring indicated improved compliance ratings and no mitigation measure received non compliance. ESMS monitoring also concluded that the implementing agency complied with management plans and standard operating procedures. The implementing agency received training on safeguards and compliance with established mitigation measures. Environmental plans to manage unsanitary dumpsites were implemented. 19 open dumpsites were closed, one rehabilitated and biogas collection and flaring systems installed at Yatta, one of the largest dumpsites. Protocols for slaughterhouse and medical waste management were applied, monitored, enforced and verified by the IVA. Environmental and social safeguards monitoring and reporting formed part of the overall M&E system. Local governments reported compliance with the Environmental and Social Management Plan as part of the semi-annual reporting process and results aggregated by the implementing agency and verified by the IVA. On a semi- annual basis, the implementing agency also reported compliance against the Environmental and Social Management systems by evaluating the application of Management Plans and Standard Operating Procedures, developed new procedures and reports the fulfillment of responsibilities assigned to implementing agency staff members. These reports were also verified by the IVA. Safeguards monitoring was developed late in 2014 and implemented in early 2015 resulting in asynchronous monitoring of results early in project implementation but inconsistencies were resolved by the time of the 2015 restructuring. The ICR reported compliance with safeguards. b. Fiduciary Compliance Financial Management: The implementing agency prepared semi-annual performance reports, annual audit reports in a timely manner. Independent verification reports were detailed, timely and with no major non compliance issues identified. Financial disbursements were linked to results verified by the IVA. Annual Financial Management Audits were conducted. Financial audits were performed annually. The ICR did not provide information if these audits contained any qualified opinions but the TTL confirmed in a January 28, 2019 email that there were none.. There were no delays in payments and financial management was rated satisfactory throughout implementation. Procurement: The project procured only one contract under the project - the procurement of an Independent Verification Agency, worth US$ 131,250. The ICR implied but did not state compliance with procurement standards of the World Bank Group. The ICR highlighted the difficulties encountered in procuring the services of a private sector independent operator of the sanitary landfill, a separate but complementary project under Lessons Learned (Section V of the ICR). A private sector independent operator and output based arrangement was the first to be implemented in the West Bank. The IFC Team approached 60 larger firms who declined participation because of project risks. The IFC Team turned to smaller, less experienced firms, but more willing to succeed in a difficult environment. Seven firms submitted qualifications, three qualified, and two of those submitted non-responsive proposals (ICR, paragraph 101). Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) c. Unintended impacts (Positive or Negative) --- d. Other The outcome indicators for the parallel Technical Assistance component were fully achieved. A special solid waste management committee was established to build the sector institutional structure. Structure included tariff setting, accounting, and institutional coordination. The committee met regularly, held workshops with local governments to develop and implement approaches for cost recovery such as linking waste fees to electricity bills or building permits. Appropriate billing instruments under each local government’s control was customized based on consultations with each municipal or village council. The most common instrument was linking the waste fees to electricity bills but there were also those that linked to water bills or marriage certificates. Local governments confirmed household sizes and other population characteristics to be used for billing purposes through field visits. A fully functional MIS for the sector was implemented; protocols for disposing slaughterhouse and medical waste were developed, monitored, and enforced. Detailed plans to close and rehabilitate unsanitary dumpsites were completed. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Moderately Bank Performance Satisfactory --- Satisfactory Quality of M&E Modest Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR provided several key lessons from the project operations, as well as those from parallel, complementary operations. Only those lessons that relate to the project ICR, slightly edited, are noted below: 1. Performance based payments (subsidies) need to be simple and easy to understand even as it include quantitative methods. In this case, the project used formula based and quantitative approach to calculate and disburse payments and subsidies. The project-specific MIS monitored performance and fostered transparency. Transparent, frequent, and clear communication by the implementing agency with its constituents, offering MIS generated quantitative evidence allowed for opportunistic adjustments to achieve the PDOs. Updates, changes in subsidy disbursement criteria, changes in incentives, and review of peer performance, fostered a competitive spirit among participant local governments to improve. The implementing agency fostered participation by its constituent local governments, consulting and engaging Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) them to hear concerns and adapt local practices into viable options to ensure success in achieving the PDO, such as in the design of a reward system that allowed poorer performers a chance to improve and earn payments missed in past disbursement periods. There was also the appropriate co-billing arrangements where waste charges are allowed to ride on utility bills such as electricity or water bills, even building permits, and in some cases, marriage certificates. 2 . When introducing a new approach or solution, first time participant local government units require a strong, well designed capacity building program backed by adequate resources. Design could cover technical, hands-on training, customized advisory services, and small size workshops. For example, in this case, limited funds delivered one trainer per 60 participants. There was no standard cited but conventional practice points to less participants per trainer for better material absorption and retention. Another example is that workshops and consultations revealed differing jurisdictions over assets and a variety of workable options needed to be considered 3 . Flexible, innovative, just-in-time approaches serve to strengthen partnership efforts with the private sector to work in a fragile state such as the West Bank. The ICR did not specify an example but the procurement process noted above (see Section 10 (b) Procurement) provided a window to that creativity. Another was introducing exclusionary clause in adjusting aggregated, governorate level performance targets for subsidy payments to avoid the moral hazard of rewarding local governments who perform poorly but benefit from the better performers. Still another was the use of restructuring to adjust the incentive scheme because performance was outpacing initial targets in the second year. 4 . The World and the IFC collaborated by acknowledging the strengths that each institution brought to offer a fragile environment an integrated, cost effective, and participative solid waste management service delivery. The World Bank constructed the landfill and helped to establish the institutional capacity to manage a sustainable sanitary disposal service, while the IFC, drawing on output based assistance to operate and maintain the landfill and disposal facilities, structured the private sector participation. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was concise and followed OPCS guidelines. The ICR provided a rationale for the assumptions and the project components to support the theory of change, including the complementarity provided by parallel operations, both the physical investments and the technical assistance support that were funded separately. It was internally consistent and results oriented with a strong focus on project outcomes. For example, the adjustments made following the 2015 restructuring aimed to ensure that the PDOs were achieved. There was an extensive discussion on the economic and financial efficiencies of the project (Section 5 above) and the basis for its calculations was presented in an annex. There was a minor inconsistency in the narrative supporting the ratings for a substantial Bank performance but a modest M&E with no explanation to reconcile the inconsistency. There was also a minor inconsistency in presenting the rating of Bank performance Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GPOBA Solid Waste Mgt West Bank (P132268) (Moderately Satisfactory on p.2 and Satisfactory on p. 28). Evidence provided was candid and aligned to the PDO. For example, the limited preparation extended the project was cited as a weakness. The quality of evidence and analysis was aligned to the successes claimed in the ICR. Minor shortcoming was the inadvertent overstatement of project costs but this was evidently a minor oversight (error in summation in Annex 3). Lessons were clear, useful, and based on evidence and sound analysis. For example, the effective collaboration between the World Bank and the International Finance Corporation based on each other's core competencies were useful to highlight under the lessons learned. Another example was the simplicity of the subsidy formulas that facilitated the active engagement of participant local government units. a. Quality of ICR Rating Substantial Page 16 of 16