Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Report Number : ICRR0021327 1. Project Data Project ID Project Name P096213 NAT'L WATER SUPPLY & SAN Country Practice Area(Lead) Additional Financing Azerbaijan Water P153968,P153968 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-74600,TF-90657 31-Jan-2012 209,394,390.02 Bank Approval Date Closing Date (Actual) 14-Jun-2007 31-Dec-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 230,000,000.00 1,951,706.00 Revised Commitment 207,445,472.85 1,948,917.17 Actual 207,445,472.85 1,948,917.17 Prepared by Reviewed by ICR Review Coordinator Group Maria Vanessa J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Corlazzoli Pellekaan 2. Project Objectives and Components a. Objectives According to the legal agreement of 2007, the objective of the Azerbaijan National Water Supply and Sanitation project was to “improve the availability, quality, reliability, and sustainability of water supply and sanitation services of the Borrower” (Loan Agreement TF-7460-AZ, 2007, pg. 6). This objective is almost exact in wording to the one in the PAD, except that it differs in the specificity of the location where the services were to be improved. The objective of the project outlined in the PAD focused on improving water supply and sanitation services in twenty of Azerbaijan’s regional (rayon) centers whereas Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) the objective in the legal agreement was broader (PAD, para 16). This ICRR will use the objective as described in the legal agreement. The project’s objective was changed as a result of a restructuring in October 2011. The revised project objective became “to provide quality and reliable water supply and sanitation services in selected regional (rayon) centers of Azerbaijan” (Restructuring Paper, page 1). The revision dropped the words 'sustainability' and 'availability' and made the targeted location more specific yet also vague. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 18-Nov-2011 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components Component 1: Rayon Investment Component (Appraisal cost was estimated at US$268.75 million. At closing the total cost was US$174.5 million). This component aimed at financing the rehabilitation and extension of water supply and sewerage systems, as well as water, wastewater and septic sludge treatment facilities in 20 urban rayon centers, including two rayon centers in the Autonomous Republic of Nakhchivan. The component also aimed at financing costs of design and technical support for construction management, including contract supervision (PAD, para. 17). Component 2: Regional Development Plan for Greater Baku (Appraisal cost was estimated at US$5 million. At closing the total cost was US$4.32 million). This component sought to finance a Regional Development Plan for Greater Baku, which would provide a framework for identifying and prioritizing infrastructure investments across the Absheron Peninsula in future years (PAD, para. 18). This component would also seek to develop land use regulation through regulatory and zoning plans. Activities under this component included technical assistance, capacity building for key stakeholders, and collecting base line data and studies to be used in the Regional Development Plan and to update the water and Sewerage Master Plan for Absheron Peninsula (PAD, para. 18). Component 3: Institutional Modernization Component (Appraisal cost was estimated at US$12.67 million. At closing the total cost was US$12.36 million). This component aimed at supporting the development and implementation of an Institutional Development Plan (IDP) for Azersu (a state-owned Open Joint Stock Company that manages water supply and sanitation services in Azerbaijan, except in the Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Nakhchivan Autonomous Republic). This component also included modernizing the State Amelioration and Water Management Agency (SAWMA), which provides water supply and wastewater services to the Autonomous Republic of Nakhchivan. The purpose of the plan was to improve the operational efficiencies and sustainability of water supply and sanitation services. Key activities under this component included: (i) developing a Financial Restructuring and Recovery Plan, (ii) enhancing Azersu’s technical and financial performance through a twinning arrangement with an international experienced water utility company, (iii) implementing a meter-ready billing and collection system, (iv) introducing a modern financial accounting system, (v) strengthening the Tariff Council, (vi) improving the company’s operational efficiency and effectiveness by developing regional subsidiaries of Birleshmish Sukanal, (vii) implementing a demand management program, (viii) increasing monitoring of technical and financial performance, and (ix) assisting the government in reviewing technical standards for water supply and wastewater infrastructure to meet international standards (PAD, para 19). Component 4: Project Implementation and Management (Appraisal cost was estimated at US$1.95 million. At closing the total cost was US$2.62 million). This component sought to support project implementation and management by financing operational expenditures, trainings, and annual audits (PAD, para. 20). During the project’s restructuring in October 2011, the following changes were made to the components: Component 1: Rayon Investment. During the restructuring, the component’s target shifted from rehabilitating and extending water supply and sewerage systems in 20 rayon centers to four. This decision was made as a result of the cost increase due to additional investments not anticipated in the original investment plan such as costs for house connections, metering, and repaving. The four rayon centers were selected on the basis of their advanced preparation status, proximity to Baku, and commitment to the project (ICR, para 17i). The rehabilitation and extension of the water supply and sewerage systems in rayon centers in the Autonomous Republic of Nakhchivan were canceled due to procurement mismanagement by State Amelioration and Water Management Agency (SAWMA) (ICR, para 17 ii). Component 3: Institutional Modernization. Several activities were dropped from this component during the restructuring including (i) support to SAWMA due to the elimination of SAWMA as an implementing agency, (ii) strengthening of the Tariff Council since alternative support was provided by the European Bank for Reconstruction and Development, and (iii) the Review and Improvement of Selected Technical Standards (ICR, para. 27). Component 4: Project Implementation and Management. Activities related to supporting the strengthening of SAWMA’s management capacities were dropped (ICR, para. 27). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Project Cost. At appraisal, the total cost of the project was estimated at US$310 million (PAD, pg. 7). It was estimated that the IBRD loan would be US$230 million, and the Government would contribute US$80 million. At approval, the total cost was estimated at US$ 311 .9 million (ICR, pg. 2). At closing, the total project cost was US $209.4 million (ICR, pg. 2). Financing. The appraised amount of financing needed for this project was estimated at US$230 million (PAD, pg. ii). A total of US$230 million were provided through Investment Project Financing in the form of a Specific Investment Loan (SIL) (IBRD - 74600). Additional financing was provided in the amount of US$ 1,951,706 (TF-90657). At appraisal the total financing for this project therefore was US$ 231,951,706 (ICR, pg. 2). When the project closed the actual disbursement at the end of the project was US$209,394,390 (US$207.44 million from IBRD-74600 and US$1.94 million from TF-90657) (ICR, pg. 2). Borrower Contribution. The 2007 legal agreement stipulates that the Bank would finance 80% of expeditures, exlusive of taxes. The remaining costs (20%) would be financed by the borrower (Financial Agreement, pg 13-14) . The PAD states that the Borrower would contribute US$80 million towards the project (PAD, pg. i and 51). The counterpart contribution was reduced to US$71 million following the misprocurement case and cancellation of US$22 million in IBRD funds (World Bank Staff Interview, September 2018). At the end of the project, the borrower contributed US$71 million (World Bank Staff Interview, September 2018). Dates. The project was approved on June 14th, 2007 and became effective on March 14th, 2008. It underwent a midterm review on October 29th, 2012. The original closing date was January 31st, 2012. In October 2011, the closing date was moved from January 31st, 2012 to February 28th, 2013. In order to enable completion of ongoing contracts, the project end date was once again extended in February 2013 from February 28th, 2013 to December 31st, 2014 (ICR, para 30). In December 2014, the project was extended to December 31st, 2016 to implement existing contracts (ICR, para. 31). The project closed on December 31st, 2016, 47-months after its original project closing date. There were five restructuring processes during the life of the project: 1. In March 2009 – the project changed implementing agencies and reallocated funds between disbursement categories. 2. In October 2011- the project revised the PDO, changed indicators, revised targets, updated the results framework, changed components and costs, shifted loan closing date to February 28th, 2013, changed legal covenants, institutional arrangements, dropped one of the implementing agencies, canceled earmarked funds, and reallocated disbursement categories. Before this restructuring, the project had disbursed US$24.85 million, or 12% of the budget at appraisal or 13% of total disbursement at completion. 3. In February 2013 - the project changed the loan closing date to December 31st 2014. 4. In December 2014 - the project changed the results framework and loan closing date to December 31st, 2016, and reallocated loan proceeds between disbursement categories. Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) 5. In July 2016 - the project revised end targets to account for actual numbers of beneficiaries (ICR, pg. 11) 3. Relevance of Objectives Rationale At appraisal the project's objective was “to improve the availability, quality, reliability, and sustainability of water supply and sanitation services of the Borrower”. This was revised at restructuring in October 2011 and became “to provide quality and reliable water supply and sanitation services in selected regional (rayon) centers of Azerbaijan” which remained the objective until the project closed. Country Context: Azerbaijan inherited a relatively extensive water supply system from the Former Soviet Union (FSU). At the time of appraisal, about 95 percent of the population in Baku were connected to piped water supply (PAD, para. 3). While coverage was relatively high, the quality of infrastructure and services had deteriorated due to lack of investment and deferred maintenance. In many towns, citizens did not have access to safe water supply. Water supply was also unreliable, often only available for less than 12 hours a day (PAD, para. 3). There was also a lack of wastewater and fecal sludge treatment facilities, with most rayon centers not having functioning sewerage collection and treatment systems. This was posing a serious threat to public health and negatively impacting the environment. The utilities also lacked modern water supply and sanitation management with a focus on technical and financial sustainability (PAD, para. 5). The government’s Poverty Reduction Strategy (2003-2005), identified the strengthening of the utility sector as one of its strategic goals (ICR, para. 6). The strategy called for a nationwide approach that created infrastructure for regional development and improvement of public utilities (ICR, para. 6). At closing, the project objective continued to be aligned with the government’s commitment to providing reliable basic services (including water and sanitation) to its citizens, as outlined in the Third State Program of Social and Economic Development of Regions for 2014-2018 (ICR, para. 39). Alignment with Country Partnership Strategy: At appraisal, the objective of the project aligned with the Country Partnership Strategy (FY07-FY10). The project was in line with the second Pillar of Supporting the sustainable and balanced growth of the non-oil economy. A sub-objective of this pillar was to “improve the quality and coverage of water and sanitation nationwide” (CPS, para. 69). The project objective continued to support the Country Partnership Strategy (FY11-FY14)’s second Pillar of supporting Sustainable and Balanced Growth of the non-oil economy (CPS FY11-14). It also contributed to strengthening the institutional framework for addressing environmental issues under the forth CPS Pillar: Improved Environmental Management (CPS FY11-14, pg. 84). Under this pillar, the project also contributed to outcomes related to (i) increasing the financial viability of the utility sector, (ii) increasing quality and coverage of water supply, and (iii) developing rural infrastructure and service (CPS FY 11-14 pg. 65-76). Previous Sector Experience: The Bank had previous experience in Azerbaijan providing support to improve water quality through the project “Greater Baku Water Supply project” which closed in January 2006 Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) (PAD, para. 6). This project focused on urgent short-term physical improvements to the water supply system (PAD, para. 6). Key activities included leakage reduction, facility repairs, and rehabilitation of water treatment plants. Conclusion. Although the changes in the project's objective following restructuring in October 2011 resulted in a considerable reduction in its ambition, mainly a more than 50% reduction in its coverage of people, and a reduced potential contribution to the improvement of water and sanitation services in Azerbaijan without any adjustment in the project cost, the project's objectives remained substantially relevant to the development strategies of both the government and the Bank. The relevance of objectives was therefore rated substantial. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve the availability, quality, reliability and sustainability of water supply and sanitation services of the Borrower. Rationale This project sought to invest heavily in rehabilitation and reconstruction of water supply and sanitation infrastructure. It aimed at investing in the whole water cycle – from water intake and storage, water supply and sewage network to wastewater treatment (ICR, para 12a). At the same time, it also sought to implement a comprehensive institutional modernization component to strengthen the sector’s capacity to effectively and efficiently manage the utilities (ICR, para 12b). Although ambitious, the activities of the project were thoughtfully crafted. Unfortunately, there was a disconnect between the activities, indicators, and objectives. While the implementation of the activities could have led to the anticipated outcomes, the objective was very high-level. As a result, the indicators designed to measure the objective were unable to capture secondary-level outcomes effectively. For example, there were inadequate indicators related to measuring institutional change or institutional modernization even though the project would eventually allocate US$16.26 million over the life of the project to these activities. Key Outputs that constrained or contributed to the achievement of the project's objective between March 2008 and October 2011: Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) • In none of the rayon centers were water supply and sanitation services improved prior to restructuring (ICR, pg. 35). Reflecting the low expectations for progress, during the October 2011 restructuring, the target for the project’s coverage was dropped from the original 20 rayon centers to 4. • 7,400 cubic meters of new water reservoir capacity were added (ISR #12, pg. 4). Outcome: Despite the fact that the project had been operating for 30 months, and 13% of funds were disbursed, no significant advancements were made to improve availability, quality, reliability, and sustainability of water supply and sanitation services. As a result, achievement towards the project's reduced objectives (ICR, paragraph 35) before its restructuring is rated Modest. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective To provide the quality and reliable water supply and sanitation service in selected regional rayon centers of Azerbaijan. Revised Rationale The restructuring process of October 2011, removed keywords such as ‘availability’ and ‘sustainability’ from the objective. It therefore significantly reduced the ambition and targets of the project. However, the essence of the project remained to focus both on infrastructure improvement and institutional capacity. During the restructuring of October 2011, some indicators were added, while others were dropped. Targets were once again revised in December 2014 and July 2016 (six-months prior to project closing). However, changes to the indicators did not address the results framework weakness of not being capable of capturing lower-level changes, including institutional capacity and modernization. According to the Implementation Status Report #12 on February 2012, which was written following the restructuring process of October 2011, the project was progressing with construction and procurement of infrastructure in four locations: Gabala, Shamakhi, Saatli and Sabirabad rayon centers (ISR #12, pg. 2). Construction of water intakes in Shamakhi rayon had been completed and was under completion in Gabala rayon. Progress had also been made in preparation for the Baku Regional Development Plan with the finalization of the inception report. Key Outputs that contributed towards the objective between October 2011 and December 2016: Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) The project financed the full rehabilitation or construction of water supply or wastewater system in the four rayon centers. A summary of the major elements built during the project in each rayon were (World Bank Staff Interview, September 2018): • Water intake built in Shamakhi and Gabala rayons (Saati and Sabirabad already had existing networks). • Water transmission networks built in all four rayon centers. • Water distribution network (including individual household connections and meters) built in all four rayon centers. • Wastewater collection network (including individual household manholes) built in Shamakhi and Gabala, and partially built in Saatli and Sabirabad. • Sewerage collectors/interceptors built in Shamakhi and Gabala. • Waste-water treatment plant built in Shamakhi and mechanical treatment only in Gabala. In addition, below are other project outputs compared with targets (ICR, pg. 33-37): Outputs Original Target Status of Target Revised Achievement Only 1 rayon center where water supply and wastewater 20 4, and Target Not systems were rehabilitated and operational and hence then achieved improved. This target was revised two times - first from revised 20 to 4 and subsequently to 2 rayon centers. A year to 2 after the project closed, the rayon center in Gabala was not operational (ICR, pg. 35) 20,978 of new piped household water connections were 28,152 19,922 Target met made as a result of the project. 9,611 new household sewer connections were 8,440 17,183 Target met constructed and then revised to 9,222 575 water networks rehabilitated 450 537 Target met 202 sewerage network rehabilitated 380 201 Target met New water reservoir capacity at 34,400 cubic meters 17,000 34,400 Target met The following reports and plans were also completed as a result of the project (ICR, pg.37-39): • Greater Baku Regional Development Plan was drafted, publicly discussed and disclosed. • Updated Absheron Peninsula Water Supply Masterplan was completed. Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) • Integrated Urban Water Management Study was finalized. This study analyzed conditions and trends in water resource management in the Greater Baku Area and recommendations were used in the GBRDP (ICR, para 75). • International Financial Reporting Standards (IFRS) were adopted by Azersu. • In addition, the financing plan was completed and adopted by Azersu. However, the financing and restructuring plan was not made available to the World Bank (ICR, pg. 38). Outcomes Quality: As a result of the infrastructure projects completed, a total of 98,597 people received improved water supply and sanitation services (Target Met: Original Target: 200,000, Revised Target: 145,000, Revised Target: 96,633, ICR para. 46). Out of these individuals, 20,978 were in newly piped households with water connections and 9,611 with sewer connections. The population of Shamakhi and Gabala got their wastewater collected and treated under the project, although the waste-water treatment plant in Gabala was not operational. The wastewater treatment plants in Saatil and Sabirabab were not built due to the financial gap and the government’s decision to stop the additional financing preparation caused by the international oil crisis and severe devaluation of the national currency (World Bank Staff Interview, September 2018). Hence the population of Saatli and Sabirabab continued to use their individual septic tanks that existed prior to the project start (World Bank Staff Interview, September 2018). Of the drinking water samples made in October 2016, 100% met Azeri water quality standards compared to the baseline of 40% (Target met, Baseline: 40%, Target: 100%, ICR, pg. 39). Targets on pollution load measured in person equivalent (PE) eliminated through adequate wastewater treatment were not met (Target revised twice. Target met. Original Target: 105,00, Revised: 89,000, Revised: 46,667, ICR, pg. 18 and pg. 46). This indicator measures the designed capacity criteria that a water or waste-water treatment plant (WWTP) can sustain. In other words, how many persons/people can be connected to the waste-water treatment plan (World Bank Staff Interview, September 2018). Further, none of the effluent water samples passed water quality tests (Target not met, Original Target: 100%, Target Revised: 50%, ICR, pg. 39). Reliability: The numerous infrastructure initiatives improved the reliability of the water supply. At appraisal, on average households connected to water services had 4 hours of water supply service. By the end of the project, populations in all four rayons had access to reliable 24 hours piped quality water supply with individual meters installed (Target Met, Target: 24 hours, ICR, pg. 37 and World Bank Staff Interview, September 2018). The project also strengthened more reliable water supplies by supporting the improvement of institutional management capacity and modernization. For example, the project played a positive role in supporting the development of the Greater Baku Regional Development Plan (GBRDP), a comprehensive strategic planning document that defined an integrated economic, environmental, transport and social framework for the development of the city. The plan was publicly discussed and disclosed (ICR, para 67 and 69). Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Moreover, draft land use and zoning plans were also prepared (ICR, para. 67). Funds in the total amount of AZN1.6 million have been allocated to the State Committee for Urban Planning and Architecture (SCOPA) for urban planning (ICR, para. 69). For Baku, a more detailed plan was finalized in December 2014 but as of March 2018, it was not approved (ICR, para 73). According to the ICR, without proper plans and zoning ordinance requirements, the business corporations can construct any type of building without substantive consequences. Commercial and residential projects have developed at an extraordinary pace and often exceeded the limits defined in the GBRDP (ICR, para. 73). Improvements were also made to modernize utility institutions through the provision of improved billing and collection systems, increased metering, installation of smart meters, removal of illegal connections and reduction of the volume of unpaid water (ICR, para 71). Overall, the project had some impact on the quality of water supply and sanitation service. While significant infrastructure work was conducted in all 4 rayon centers, only one wastewater treatment plant was operational when the project closed. The project improved the reliability of services for those connected to the new system giving them access to 24 hours /day water supply. The project did not achieve key indicators related to the quality of effluent water. Even though the project improved institutional capacity and modernization of service management, this was not part of the results framework and hence thee were no objectives, indicators or targets against which to measure results. Nevertheless the achievements in this area contributed towards long-term reliability of water supply and sanitation services. In light of the achievements against the reduced targets the efficacy of the revised objective was rated substantial. Revised Rating Substantial PHOVRLEFFRATTBL Rationale There were no significant advances made by the project in achieving its original objective of improving the availability, quality, reliability, and sustainability of water supply and sanitation services in Azerbaijan. The efficacy of the original project objective was therefore rated modest. Following the project's restructuring the project focused on building water supply and sanitation infrastructure in four rayon centers. Water supply reliability improved. Nonetheless, the project was not able to attain the depth and breadth of change that it aspired to in its revised objective. For example, only one out of the 4 selected rayon centers of Azerbaijan was operational in terms of improved water and sanitation services when the project closed. Also, the project did not achieve effluent water targets. The targets for the number of people receiving improved water supply and sanitation services were significantly reduced from 200,000 to 93,633 (compared with 98,597 achieved). Though the level of ambition of the revised objective was low, it was largely achieved and hence the efficacy of this objective was rated as substantial, which is the rating included in the formal portal rating summary. Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency Scope of Analysis At appraisal, efficiency was determined by an economic analysis which included estimates of Net Present Value (NPV) with and without the project, the Economic Rate of Return (ERR) as it related to the project infrastructure program. The efficiency analysis also included financial analysis of rayon investments and of Azersu. The same methodology was used at project closing (PAD para 41, ICR, para 49). At appraisal, the project’s benefits included (i) user benefit measured through stated willingness to pay (WTP); (ii) reduced coping costs, and (iii) systems benefits through energy efficiency gains. At project closing, benefits from sanitation improvement were also quantified. Other impacts or benefits not included in the efficiency analysis included (i) economic gains, (ii) reduction in costs of coping strategies undertaken by households to mitigate water shortage or lack of access to water supply (iii) positive impacts on public health (iv) effects external to the project such as on the environment, and (v) macroeconomic impacts. Economic Analysis The economic cost-benefit analysis relied on three main sources of quantitative information: (i) user benefits; (ii) reduction in coping costs for water supply and sanitation; and (iii) system benefits (such as energy efficiency gains) (PAD, para 43). At appraisal, the project’s Net Present Value was estimated at AZN6.76 million. The project investment was US$207 million and the assessed quantified annual benefit amount was AZN17,700 million (US$10.4 million for a 20-year period). At project closing, the Net Present Value was AZN -100.4 million (ICR para. 52). The negative NPV was the result of the highly reduced scope of the project, including 12 times less population and 8 times less population receiving improved services. The reduced scope led to an increase in investment per capita from US$387 estimated at appraisal to US$2,322 estimated at project closure. The estimated economic internal rate of return at appraisal was 10.4%. Due to the highly negative NPV at project closing, an internal rate of return is not relevant. Financial Assessment (a) Azersu The Azersu Group is an Open Joint Stock Company in charge of policy, strategy and construction of new water pipelines and sewerage systems in Azerbaijan receiving subsidies from the Government on a regular basis to Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) undertake these tasks. Historically, the Group had operating losses, negative cash flows, and a working capital deficit (ICR, Annex 4). Revenue forecasts for the period of 2006-2001 for Azersu expected a working ratio (ratio of operating costs to revenues) of 0.70 to 0.80 which indicated the prospect of recovering operation costs from revenues with some residual resources for capital expenditures. Despite notable improvements in 2005, the audited financial statements showed that Azersu fell short of producing sustainable working ratios. According to an independent auditor’s report from 2012-2016, the Group’s incurred net losses led to material uncertainty, which continued to cast significant doubt over its ability to operate on a sustained basis (ICR, para. 56). (b) Rayons The final financial assessment of the four rayons—Saatli, Sabirabad, Shamakhi, and Gabala—showed that they generated annual cash shortages. Despite notable improvements in Gabala and Saatly in 2017, the rayon centers were “unable to meet its operating costs, let alone provide a sufficient return on invested capital to allow for future reinvestments, expansion of systems, and improvement of service quality” (ICR, pg. 52). The four rayons continued to fall short of producing sustainable working ratios and were unable to cover their operating and management costs (ICR, pg. 52). The EBITDA (Earnings before Interest, Tax, Depreciation, and Amortization) was negative in 2017. The EBITDA ratio versus sales was even larger than before the project (ICR, pg. 52) Water Tariffs As a result of the project, the utilities in the rayon centers were provided with tools to increase collection rates, including computerized billing and collection systems. The water tariffs for water and wastewater services have doubled since January 2007. On May 2016, the Tariff Council of the Republic of Azerbaijan approved an increase of water prices and wastewater treatment tariffs. These were supposed to help the utilities be financially viable (ICR, para 58). However, given citizen's continued reluctance to discontinue the use of their septic tanks, the increase of tariffs may be an additional barrier to connecting more households to wastewater management (ICR, para. 46). Project Management The project incurred a 47-month delay which generated economic inefficiencies in the project through the delayed (and hence discounted) benefit streams over what could have been achieved. At the same time, the implementation and management costs for the project was only 1.3% of total project costs at completion compared with 0.7% of project costs at appraisal. Summary The Efficiency of this operation is rated negligible due the to the project’s negative net present value at completion and hence a negative rate of return, the chronic fragile financial record for the Azersu Group, Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) financial inefficiencies at the rayon level, and an estimated six-fold increase in the project’s investment per capita from US$387 at appraisal to US$2,322 when the project closed. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  10.40 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives was rated as Substantial at appraisal and at the project's close due to the alignment of the project’s objectives to the country context, country partnership strategy, and the Bank’s assistance strategy. The project made no significant advancements in terms of its objective "to improve the availability, quality, reliability, and sustainability of water supply and sanitation services" in Azerbaijan. As a result, prior to the restructuring, the project’s efficacy was rated Modest. After the restructuring, some progress was made towards improving the quality and reliability of water supply and sanitation service but at a lower level of ambition and hence the project was not able to attain the depth and breadth of change that it originally aspired to. Nevertheless, after restructuring all four rayon centers received infrastructure improvements in the area of water intake, water transmission, water distribution networks, water and waste collection networks, and sewage collector or interceptor. While water supply reliability, institutional and management capacity also improved, only one out of the 4 selected rayon centers of Azerbaijan was operational when the project closed. The project also did not achieve its effluent water targets. Many of the PDO indicators were, however, significantly reduced and the level of ambition of the revised objectives was therefore relatively low compared with the objectives at appraisal. Consequently, the lower level objectives were largely achieved and hence the efficacy of this objective was rated substantial. The project's efficiency was rated modest before and after restructuring. The following table summarizes the derivation of the overall outcome rating for the project based on the weighted average of the outcome ratings before and after restructuring - weighted by disbursements before and after restructuring. Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Overall, the project is rated as moderately unsatisfactory due to significant shortcomings in the project’s efficiency. Item Before After restructuring restructuring Relevance of Objective Substantial Substantial Efficacy (PDO) Substantial Modest Efficiency Negligible Negligible Unsatisfactory Moderately Outcome Rating Unsatisfactory 2 3 Numerical value of outcome rating US$ 24.85 US$ 184.54 million Disbursement in million million 13% 87% Share of Disbursement (weights) Overall Outcome Rating (weighted average of outcomes before and after restructuring): Moderately Unsatisfactory Overall, the project is rated as moderately unsatisfactory due to shortcomings in the project’s objective’s efficiency. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome There are several interrelated risks that can impact the sustainability of the objective: Financial: The financial sustainability efficiency of Azersu is precarious and heavily dependent on the Government’s subsidies. It remains to be seen whether Azersu can fully recover its operating costs (ICR, para. 119a). Capacity: There is limited staff capacity at Azersu to manage wastewater assets (ICR, para. 119 d). Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) Infrastructure: There is a need to continue to invest in operational efficiency improvements, including metering billing and transparency (ICR, para. 119b). 8. Assessment of Bank Performance a. Quality-at-Entry The project aligned with the Government’s commitment to improving water supply and sanitation. In fact, the Government had ongoing projects in 80% of the rayon centers selected for this project. Key studies were conducted as part of the preparation phase that revealed that most water supply production, treatment, transmission, and distribution were severely deteriorated. The facilities had exceeded their economic lifespan and it was affecting the quality and reliability of services (ICR, para 81 and 82). Feasibility studies were developed using key design criteria based on international practices and agreed upon by the Bank and the Government. However, according to the ICR, “the feasibility study was not of decent quality” (ICR, para. 99). The feasibility studies underestimated costs. In particular, the investment costs were significantly higher than originally estimated (ICR, para. 111). The feasibility study underestimated costs due to lack of recent bidding/construction and lack of local industry to produce materials (World Bank Staff Interview, September 2018). In addition, existing water supply and waste- water supply systems could not be upgraded or rehabilitated as originally planned (World Bank Staff Interview, September 2018). During the project implementation, several technical revisions to the key design criteria were introduced by Azersu due to new government design standards (World Bank Staff Interview, September 2018). The government standards, which had a higher quality of service standards, were introduced after the feasibility study was conducted (World Bank Staff Interview, September 2018). This led to higher costs for lump-sum contracts and investment per capita increased (ICR, para. 83). The reduced scope from 20 to 4 rayon centers, was due largely to the underestimated investment costs and substantial scale-up of the water supply and sanitation networks in the 4 rayon centers (ICR, para. 83). The results framework had shortcomings from the beginning. This is notable as one of the original PDO indicators targeted a total of 20,000 people connected to water and sewerage networks for 20 rayon centers. During the restructuring in October 2011, this target was adjusted to 200,000 people for 4 rayon centers (ICR, para. 112). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) The project task team leaders were mostly based in the World Bank’s Azerbaijan country office, along with the technical team. This enabled “well facilitated, day to day client engagement, strong field presence, and strengthen project supervision” (ICR, para.113). According to the ICR, critical problems and bottlenecks were identified as “early as possible and solutions sought in collaboration with the project management unit” (ICR, para. 113). While a midterm review was conducted on October 29th, 2012, no aide memoires were prepared on the midterm review and the economic benefit assessment was not updated (ICR, para. 113). The Bank was responsive to the needs expressed by the Government. For example, when the government expressed concerned that there were a few local contractors pre-qualified for the construction work, the Bank made reasonable recommendations to the procurement process to encourage local contractors to apply (ICR, para. 87). Moreover, throughout the project, the Government requested different changes to project requirements. The Bank tried to continue to adjust the project to meet the government’s needs while staying true to the commitments made in the Bank’s country partnership framework (ICR, para. 115). The project experienced increased costs throughout the entire project. On January 2011, the Bank declared mis-procurement on two contractors for water supply and sewerage systems for Sharur and Babek Rayons and the loans were canceled (ICR, para. 88). In early 2013, all four rayon centers requested a larger scope for contracted civil works, such as connecting additional villages and households. This resulted in the amendment of scope and existing network contracts. A newly created technical and quality review department within Azersu reviewed detailed designs for contracts. Given the changes proposed by Azersu, this implied higher costs for the current lump-sum contracts (ICR, para. 90). The Bank requested the Government’s decision on the need to borrow additional financing to close the financing gap between US$1,020 million and US$1,150 million (IRC, 91 and World Bank Staff Interview, September 2018). In November 2014, after the Government’s decision to drop additional financing request, Azersu scaled down the contracts to a realistic scope, and the Bank restructured the project to align the results framework to the final scope (ICR, para. 92). The contract for construction of the Gabala wastewater treatment plant was signed in January 2013 and finalized at project’s closure only for mechanical treatment. The Wastewater Treatment Plant is still not operational. Significant factors that affected successful completion included “slow connection rate of households to sewer systems and, Azersu’s limited capacity to take over the facilities financially and technically” (ICR para. 93). Overall, the project experienced significant design flaws starting with an ambitious results framework and having to progressively reduce scope and targets as a result of escalating cost. A poorly designed feasibility study led to increased costs throughout the project leading to a moderately unsatisfactory rating at project-entry. Throughout the life of the project, the Bank work with the implementing agency and the Government to resolve issues, including financial gaps, and project implementation delays. The Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) quality of supervision is rated as moderately satisfactory. The overall Bank Performance is rated as moderately unsatisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project aimed at addressing some of the results framework shortcomings through several restructuring, including revising the PDO, indicators, and targets. The project objective was revised, as previously discussed, and two aspects were dropped– namely ‘availability and sustainability’ from the project’s scope. In the October 2011, the restructuring process was approved. The project added and dropped indicators. Targets were revised throughout the project’s implementation, including as late as six months before the project closed on July 2016. Even though the project was restructured and made changes to indicators, it did not address the result framework shortcomings of focusing on high-level impact without including a framework to enable evidence-based decisions. Moreover, indicators related to technical change, outlined in unintended impact, were not included when it became apparent that the project was also delivered in unexpected areas. The results framework and hence the M&E design had significant shortfalls from inception. Although ambitious, the activities in the project were thoughtfully crafted. Unfortunately, there was a disconnect between the activities, indicators, and objective. While the implementation of activities would lead to the outcomes, the objective was very high-level. As a result, the indicators drafted to measure the objective were unable to capture secondary-level outcomes or ongoing performance effectively. For example, there were inadequate indicators related to measuring institutional capacity and modernization as a result of the project. The project indicators were primarily focused on impact level outcome and were unable to measure ongoing progress towards success. Moreover, the PDO indicator had a target of 20,000 people serviced with piped water supply, while the economic analysis referred to the population served going up to from 271,967 to 695,010 (ICR, para. Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) 96). The ICR states “this is a significant discrepancy in the most important PDO indicator and raises concerns about the quality of M&E design” (ICR, para. 96). According to the PAD, Azersu would be responsible for monitoring and evaluation activities. They were expected to hire and train additional staff. A baseline and quarterly reports would be conducted and submitted to the Tariff Council and to the Bank. The quarterly reports were also supposed to be published on Azersu’s website (PAD, para. 34). In actuality, Azersu published annual performance reports which were open to the public (World Bank Staff Interview, September 2018). The baseline data for water quality was collected as part of the feasibility study (World Bank Staff Interview, September 2018). The baseline determined that the water quality was not at the national standards, particularly in Saatli and Sabirabad (World Bank Staff Interview, September 2018). According to the ICR, the Project implementation team and Project management unit implemented data collection and monitoring systems were put in place to systematically collect data (ICR, para. 97). b. M&E Implementation The project implementation unit (PIU) and the implementing agencies (Azersu, SCUPA and SAWMA) were responsible for implementing monitoring and evaluation activities. The PIU was also supported by construction supervision firms that collected key data. The information was sent to the Bank on a quarterly and semi-annual basis to prepare ISRs and internal reporting (World Bank Staff Interview, September 2018). The PIU and the Bank reported on the indicators in a consistent matter in the ISRs. That said, given the indicators were poorly designed, the utility of the indicators was limited. c. M&E Utilization According to the ICR, the “M&E process was compiled and used by project management and decision maker….to make changes to the project’s structure during restructuring.” The project helped Azersu prepare a web-based live GIS construction supervision system to gradually sale up and cover the entire country (World Bank Staff Interview, September 2018). The monitoring data was used for payment, progress reporting, and improving implementation (World Bank Staff Interview, September 2018). According to World Bank Staff Interview, water quality is tested on an ongoing and regular basis. In fact, there were 25 end-users points in each rayon where water is collected and analyzed. The water quality is monitored by the rayons’ Hygiene and Epidemiology centers of the Ministry of Health in a centralized water supply system (World Bank Staff Interview, September 2018). Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) The monitoring and evaluation is rated at modest due to significant design flaws, including lack of secondary- outcome or performance indicators. These flaws were not adequately addressed during the restructuring of indicators. There is evidence of the project setting up a monitoring system to collect key indicator data and an ongoing process to monitor water supply quality. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Environmental Category “A” Assessment. At appraisal, it triggered the following safeguards: Environmental Assessment (OP 4.01), Involuntary Resettlement (OP/BP 4.12), Safety of Dams (OP/BP 4.37) and Projects on International Waterways (OP/BP/GP 7.50). (PAD, pg. 26). The Environmental Impact Assessment Framework Report was prepared on March 22nd, 2007, and detailed EIAs were prepared in 11 rayon centers. By November 2011, all Environment Impact Assessments (EIAs) were completed and disclosed (ICR, para 107). There were no issues during implementation related to safeguards. All site-specific documents were disclosed and monitoring was in place. The project implementing unit’s capacity to “handle EIA assignment and ensure proper safeguard management during construction was weak with high staff turnover (ICR, para 107).” International Waters (OP/BP 7.50). The project triggered the World Bank’s Operational Policy on International Waters OP 7.50. The removal of the rayon in Nakhchivan (which was the only location for which OP 7.50 was triggered) from the project meant that no investments triggered the Operational Policy on international Waterways (ICR, para. 108). Involuntary Resettlement (OP/BP 4.12). Updated detailed designs led to no resettlement needs in any of four rayon centers (ICR para. 109). b. Fiduciary Compliance According to the ICR, financial management issues included slow implementation, lack of counterpart funds Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) during the early stage of the project, and contract payments. However, by the end of the project the Government counterpart funds were contributed in full (ICR, para.102). Annual audits according to the IFRS were introduced in 2012. The audit reports identified that Azersu was not a financially viable institution. Audit findings were not implemented and limited progress was made to shift Azersu’s modus operandi (ICR, para. 103). The audits were conducted at the project level and at the company level. The project audit was postponed several times at the beginning of the project (ICR, para. 103). The consolidated project audit did not include any qualifications during the project implementation (World Bank Staff Interview, September 2018). Generally, the project was in compliance with financial management covenants stipulated in restrictive legal documents, including well-staffed financial management team which provided quality reports on time and no overdue audits under the project (World Bank Staff Interview, September 2018). The project audit report was unmodified without any major internal control deficiencies (World Bank Staff Interview, September 2018). Procurement- There were also shortcomings when it came to the procurement. Given that the Government had no history or track record of bidding and preparing feasibility studies, an international supervision company was introduced to provide support (ICR, para 104). On January 19, 2011, the World Bank declared the mis-procurement of the two contracts for water supply systems and sewerage collectors for the Sharur and Babek Rayons of the Nakhchivan Autonomous Republic. It was determined that SAMWA was engaged in collusive practice with one bidder elated to the key civil work contracts (World Bank Staff Interview, September 2018). This led to the subsequent cancellation of the funds earmarked for the Sharur and Babek Rayons (ICR, para. 105, and World Bank Staff Interview, September 2018). Out of the four rayon centers, procurement for two were done on the basis of design and build contracts and for the other two rayon centers were done as civil works contracts. All contracts were done through international competitive bidding and all bids were through joint ventures (German and Turkish) (ICR, para. 104). There were various changes in scope and time (ICR, para. 106). As a result, there was a shortage of funds to finalize the WWTPs in two rayon centers (ICR, para. 106). According to the ICR, “there were also issues without a precise bill of quantities and difficult adjustment of the lump-sum contracts. Contract management by the client was difficult with designs being approved very late. The scaling up of the volume of works prevented the contractors from complaining about these delays. The contractors were engaged for six to seven years, instead of two years according to the initial contracts” (ICR, para. 106). c. Unintended impacts (Positive or Negative) The project helped spur and create companies focused on the sector. Companies are now producing pipes and other water supply and sanitation materials. The local market has grown from US$5 to 15 Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) million to a 1-billion-dollar industry. Azeri companies are now implementing projects in Uzbekistan, Tajikistan, Ukraine, and Georgia (ICR, para. 77). The project also increased and developed local expertise and capacities (ICR, para. 78). The project also helped develop a GIS web solution to supervise construction under multiple contracts in an online mode. This system is now being used across the country (ICR, para. 79). d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The following three lessons were derived from the ICR: 1. Efficient project design is critical to a project’s success. For example, this project was affected by a poorly designed results framework and inadequate feasibility study. As a result, the project was unable to deliver on the original scope of work and it became necessary to radically decrease its targets. Improved project readiness would have helped avoid unforeseen lags and costly implementation support (ICR, 121). 2. When designing water supply and sanitation projects, remember that massive construction of utilities does not translate to sustainable services. For example, while one rayon center was built and is operational, it is currently not operating at capacity, with less than half of residents connected to the sewer system (ICR, para. 120). Therefore, future water projects need to be designed taking into account the barriers that prevent citizens from shifting away from their septic tanks towards connecting into utility services. This is crucial if the Bank also advocates for increasing tariffs to ensure financial sustainability of services (ICR, para. 122). 3. A turn-key design and build contracts for the Wastewater Treatment Plants proved to be the right choice when expected frequent modification to the original design. For example, the design modifications Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NAT'L WATER SUPPLY & SAN (P096213) did not affect the original employer requirements and the contacts could be modified within the original lump- sum price (ICR, para. 124). 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well written with a logical outline and presentation-style. Overall, the ICR was concise and it included important analysis and detail. The ICR had sufficient evidence. This was particularly evident in the Efficiency, Unintended Outcomes, Key Factors, and Bank Performance. The Efficacy section did not include additional qualitative information or additional data that may have provided additional nuance. Indicator definitions, methodology, and disaggregated of data were lacking. Additional Annexes, including Efficiency Analysis and Borrowers Statement, were appropriate and useful. There were inconsistent data on project costs between the datasheet and Annex 3. a. Quality of ICR Rating Substantial Page 22 of 22