Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) Report Number : ICRR0020204 1. Project Data Project ID Project Name P117152 KAKHETI REGIONAL ROADS Country Practice Area(Lead) Georgia Transport & ICT L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-78020 30-Nov-2013 37,500,000.00 Bank Approval Date Closing Date (Actual) 10-Nov-2009 30-Aug-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 30,000,000.00 0.00 Revised Commitment 29,002,303.75 0.00 Actual 29,002,303.75 0.00 Sector(s) Rural and Inter-Urban Roads and Highways(94%):Public administration- Transportation(6%) Theme(s) Infrastructure services for private sector development(32%):Regional integration(31%):Rural services and infrastructure(31%):Injuries and non-communicable diseases(6%) Prepared by Reviewed by ICR Review Coordinator Group Victoria Alexeeva Ridley Nelson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The project development objective was "to reduce transport costs and improve access and traffic safety for the Kakheti regional roads" (Loan Agreement (p.6); Project Appraisal Document (p.7)). Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components 1. Improvement of the Road Linking Vaziani, Gombori, and Telavi (appraisal US$32.65 million; actual US$31.6 million). The component was to finance the improvement of the 65-km road linking Vaziani, Gombori, and Telavi (VGT road). 2. Road Safety Improvement and Institutional Strengthening (appraisal US$1.93 million; actual US$3.97 million). The component was to finance implementation of the following activities: (i) improving road safety along the Telavi-Gurjaani-Bakurtsikhe-Sagarej o-Vaziani road; and (ii) strengthening the capacity of Roads Department’s (RD) Regional Office in Sagarejo to improve its operational efficiency in road management and maintenance. 3. Project Implementation (appraisal US$0.21 million; actual US$0.16 million). This component was to provide institutional support to the RD and the Transport Reform and Rehabilitation Center (TRRC) for project implementation, audits, and monitoring and evaluation. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total project cost was US$35.5 million, slightly lower than the appraised US$37.5 million due to depreciation of the local currency against the US dollar. Financing: The IBRD loan was appraised at US$30 million and disbursed at US$29 million at project closure; US$997,696 were cancelled due to the incurred savings from the exchange rate conversion. Borrower Contribution: The Borrower contributed US$6.5 million by project closure, as compared to the commitment of US$7.5 million for similar reasons as above. Dates: The project closing date was extended by 21 months from November 30, 2013 to August 30, 2015 to rehabilitate an additional road section, including 5 bridges, and allow completion of the Kakheti road safety management component due to delays. The additional road section was added to use the project funds of about US$7 million that were released due to the Government's decision to finance 10 km of VGT road from its budget, as well as savings from the road works contract values that were below cost estimates for the other two contracts. The project restructuring also involved cancellation of the capacity strengthening activity for the RD Regional Office in Sagarejo, as it was dismantled. 3. Relevance of Objectives & Design a. Relevance of Objectives At the time of appraisal, Kakheti region was experiencing the economic decline and was one of the poorest regions with a 46% poverty level. In addition to the agriculture, tourism was considered to have a significant potential for Kakheti region at the time of project preparation. The rehabilitation of the Vaziani-Gombori-Telavi road (65 km) was a priority for the Government due to the road’s critical role in the economic development of the Kakheti region. The project development objectives were aligned with the Government strategic priorities set out in the Government’s program “Georgia without poverty” (2008-2013) and the current “Socio-Economic Development Strategy of Georgia” (Georgia- 2020), which emphasized the importance of transport for Georgia’s development and utilization of the country’s transit potential as one of the main priority directions in achieving inclusive and sustainable economic growth. The project development objectives remained relevant at project closure to the Bank’s Country Partnership Strategy for FY14-17, which acknowledged that road rehabilitation and upgrading was a key government priority since 2004, and supported provision of infrastructure and services to facilitate growth. Rating High b. Relevance of Design The statement of development objectives was clear. The project results framework indicated a clear causal chain between the activities Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) financed by the project outputs and outcomes related to the attainment of the development objectives. For example, with regard to physical investments in road improvement, the activities were expected to reduce travel time and vehicle operating costs on the regional roads supported by the project. Safety works and engineering measures, reinforced by institutional strengthening in road safety enforcement, emergency services, and road safety education, were intended to lead to improved traffic safety and reduction in the number of fatalities from road accidents on the Kakheti regional roads. Rating Substantial 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective To reduce transport costs for the Kakheti regional roads. Rationale Outputs • 82 km of the Kakheti roads were rehabilitated through the Bank funding and the Borrower’s funds (the original 65km of the Vaziani and Telavi via Gombori road section and 17 km of the Sasadilo-Sioni road section added during the project restructuring). • 10,038 person-months jobs were created to carry out the road works. • A workshop on Design-Build- Contracting Modality was held, and the first design-build contract in the road sector (Sasadilo-Sioni) was designed and implemented. A design-build contracting method allows the transfer of risks and introduces flexibility and innovation in the design and execution of works. • Training was carried out for the RD’s staff and consultants on innovative bridge design and construction, and output- and performance-based road maintenance and rehabilitation contracts. Outcome • Vehicle operating costs decreased from US$0.36 to US$0.25 per vehicle-km for cars and from US$1.05 to US$0.72 per vehicle-km for trucks on the Vaziani and Telavi via Gombori (VGT) road section, with the final targets fully met. Rating Substantial PHREVISEDTBL Objective 2 Objective To improve access for the Kakheti regional roads. Rationale Outputs (same as above) Outcomes Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) As a result of the project roads rehabilitation, the access for the Kakheti regional roads improved through the reduction in travel time from 120 minutes to 55 minutes between Vaziani and Telavi via Gombori, as targeted. The targets for traffic volume were exceeded at project completion, with 2,603 cars (target 2,100 cars) and 3,337 total annual average daily traffic (target 2,800 total). Rating Substantial PHREVISEDTBL Objective 3 Objective To improve traffic safety for the Kakheti regional roads. Rationale Outputs • All the planned engineering road safety interventions were implemented. • The Kakheti Road Safety Management Plan was prepared, which assessed the main causes of road accidents in Kakheti and identified small-scale road safety interventions for implementation on the VGT road. The selected interventions were implemented, including recommendations of the joint inspection of RD and Police (guardrails, signs, crossings, speed bumps, safety island, pedestrian crossings, channeled junctions, and fenced areas to restrict access to traffic during school hours). The RD’s Road Safety Unit started to carry out road safety audits of road designs for all road projects, regardless of the source of financing. • A road safety campaign was carried out by an NGO together with the RD, Traffic Police, local municipalities and schools in the region (70,120 people, including 4,007 school children at 21 local schools, were the beneficiaries of the road safety activities). • 16 new tablets were purchased for Police Officers operating in Kakheti to record the road accidents statistics, following the development of a new database by the Ministry's of Internal Affairs/Police in line with the RD’s recommendations. The access to the new road accidents database is organized through a website accessible to the designated users, including RD road safety specialists. The new database has been extended from the initially planned Kakheti and Tbilisi regions to the Kvemo Kartli and Mtskheta-Mtianeti regions. The ICR (p.15) notes that the establishment of a partnership between RD and MIA/Police to improve road accidents data collection was an important achievement. • As the Sagarejo Office was closed due to the Government's decision to maintain only two regional offices in the country, the project did not carry out the capacity strengthening activities planned for this office. Outcomes • The number of fatalities along the Vaziani- Sagarejo-Bakurtskhe-Gurjaani-Telavi road reduced from 29 in 2008 to 19 in 2015, while the traffic volumes increased substantially. This represents an annual reduction in fatalities of 34.5% in 2015, which surpasses the target of 30%. There is a continued effort to improve traffic safety, as evidenced by the established partnership between the local road agencies and expansion of the road accidents database to other regions. Rating High 5. Efficiency Economic analysis. The ex-post EIRR for the upgrading of the Vaziani-Gombori-Telavi (VGT) road was estimated at 68.4%, which is higher Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) than the ex-ante EIRR of 51.8% due to the higher actual traffic volumes in 2014 and the lower actual upgrading costs per km compared with the appraisal estimates. Around 1,700 vehicles per day were diverted from the alternative road to the VGT road in 2012. The ex-post cost- benefit analysis followed the same methodology as at appraisal, which was based on the Highway Development and Management Tool (HDM-4) and estimated the vehicle operating costs (VOC) and travel time savings for current and forecast users of the roads (normal traffic) and for the users of the alternative road that would choose the shorter project road once it is improved; the analysis also considered savings with respect to improved traffic safety on the project road. For the Sasadilo-Sioni road (17 km) added during restructuring, the ex-post EIRR was estimated at 21.3% that exceeded the ex-ante 12.5%, due to lower actual rehabilitation costs per km and the higher actual traffic on the project road compared with the appraisal estimates. Cost effectiveness. The actual road works unit cost of US$0.4 million per km for upgrading of the VGT was lower than the estimated US$0.5 million. For the Sasadilo-Sioni road, the actual unit cost per km was US$0.27 million for rehabilitation works, which was 27% less than the estimated cost. Administrative/Operational Efficiency. The project closing date was extended by 21 months due to the addition of another road section for completion as well as delays in the road safety activities. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 87.00 Appraisal  68.40 Not Applicable 72.00 ICR Estimate  51.80 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s development objectives were highly relevant to the country's priorities. Relevance of design is rated substantial. The project helped reduce transport costs and improve access for the Kakheti regional roads to a substantial extent, and its contribution to improving traffic safety is rated high. Efficiency is assessed as substantial for high economic rates of return and cost effectiveness, despite delays in completion of the road safety activities. a. Outcome Rating Satisfactory 7. Rationale for Risk to Development Outcome Rating • Maintenance. RD has an established practice that all roads that have been rehabilitated or constructed are tendered out for routine maintenance. However, there is a risk that the central government’s budget may be reduced as a result of macroeconomic situation, which in turn could impact the road maintenance budget allocations in the future. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) • Institutional strengthening. RD and Police have indicated their commitment to continue their collaboration in collecting and analyzing new road accidents data. Effective implementation and mainstreaming of the accident data base and all related activities largely depends on the capacity of both MIA/Police and RD’s Road Safety Unit; the latter requires technical strengthening and increased budget allocations. The Government expressed interest in improving road safety including the development of a new Road Safety Strategy and Action Plan. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The project preparation was launched in 2009 when the VGT road was recognized as a priority for the Government. The project design drew from three general and country specific lessons, namely: (i) a need for effective technical supervision of projects, (ii) actions to improve traffic safety must be holistic in approach but incremental and measurable in implementation, and (iii) a need for alignment of roles and responsibilities of sector organizations. The project risks were properly assessed and adequate measures were identified for mitigation rating the overall risk as moderate. Fiduciary and safeguard arrangments were adequate, and M&E was well-designed. The ICR (p.16) points out that the Bank' should have decided on a different engineering design for the second section of the VGT road that would build a stronger pavement to withstand the heavy traffic on the road section (a one-layer payment was applied, and a significant increase in traffic led to some road wear on the project road that would require carrying out maintenance works sooner). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision In addition to bi-annual supervision missions, frequent video and audio-conferences were held to advise and guide RD in the implementation of project activities, in particular the road safety activities which had some difficulties at the start and pilot activities, which required the Bank’s more frequent guidance and advice. The Bank actively facilitated close collaboration between RD and MIA/Police in achieving the improved road safety in the Kakheti region and nationwide. The project team also promoted innovation through piloting of the first design-build contract and building the capacity of the public and private sectors in this type of contracting modality. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Despite frequent government changes, the Government always showed strong commitment and support to the project. Prior to project approval and effectiveness, the Government provided support to the preparation of the detailed design and launched tender for the first 27 km of the VGT road. It committed a co-financing share of 20% of the total project cost, and there were no delays in the release of funds during project implementation. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) Government Performance Rating Satisfactory b. Implementing Agency Performance The Roads Department of the Ministry of Regional Development and Infrastructure was the implementing agency. During the 5-year project implementation, RD high-level management changed several times, yet this did not cause major disruptions. RD was efficient in filling vacant staff positions and provided necessary capacity building opportunities for RD staff. The ICR (p.18) commends RD for taking a pro- active approach in addressing a number of issues that were unforeseen at the design stage but emerged during project implementation, as well as for its openness towards the new reforms, in particular the implementation of a pilot design-build contract and collaboration with other stakeholders on road safety. RD worked actively to establish close collaboration with MIA/Police and an NGO in achieving the common goal to improve road safety in Kakheti region and nationwide. There were however some issues related to environmental safeguards management in 2010, which were subsequantly addressed. Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The project's indicators were appropriately linked with the project objectives. The key indicators included baseline data and measurable targets set at appraisal (PAD, Annex 3). The key outcome indicators for the project development objective included the reduction in travel time, vehicle operating costs, number of fatalities, and the increase in traffic volume. The definition of the intermediate indicator “number of person-months of jobs created” was based on the assumption that investment of US$3,000 creates one-person months of jobs; as this assumption was not fully clear to RD, the methodology for the calculation was revised to be based on the actual road works contract estimates rather than assumption. b. M&E Implementation Progress towards the achievement of the final targets was monitored through regular progress reports by RD and through Bank’s Implementation Status and Results Reports (ISRs). The Monitoring and Evaluation sub-unit of RD’s FPU collected and analyzed data to monitor the progress. c. M&E Utilization RD annually carries out road condition and traffic surveys, and the collected data are used to inform RD’s long-term and annual planning of rehabilitation and maintenance. RD also continues to apply the methodology/indicator to calculate jobs created in other projects. M&E Quality Rating Substantial 11. Other Issues a. Safeguards Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) This was a Category “B” project that triggered three safeguards policies – OP4.01 Environmental Assessment, OP 4.12 Involuntary Resettlement, and OP4.11 Physical Cultural Resources. Environmental Assessment; Physical Cultural Resources. The RD’s institutional capacity for environmental management was assessed as being weak, and an environmental team was created in RD in the fall of 2010, which evolved into a fully-fledged Resettlement and Environment Unit two years later. The ICR (p.7) reports that environmental safeguards were satisfactory, except during the early stages of project implementation, in particular the construction season of 2010 due to poor management of excess material and borrowing sites, and absence of a pledged environmental specialist from the technical supervisor’s team. A remedial action plan was prepared by RD with the Bank's support, which was implemented by the contractor completing the bulk of landscaping and compacting works at the excess material disposal sites and undertaking a large compensatory tree planting program within the right of way. With the start of the design- build contract and preparation of the detailed engineering design, RD found that a short section of Sasadilo-Sioni road was part of the State Forest Fund. This required de-listing of land strips along the carriageway from the State Forest Fund in order to meet standard engineering parameters and to provide access of construction machinery. All procedures established by the national legislation were followed through, as reported by the ICR (p.7). Involuntary Resettlement. The planned improvements of the Vaziani-Gombori-Telavi (VGT) road were expected to require some land acquisition along its existing alignment with small sections mainly for minor realignment to straighten curves. A Resettlement Policy Framework (RPF) and Resettlement Action Plans (RAP) for two VGT road sections were prepared and disclosed. The ICR (p.8) reports that the implementation of the two RAPs was satisfactory and no deviations to social safeguards compliance were observed. Land acquisition for the section between Vaziani and Gombori was completed in 2010 for the section between Gombori and Telavi in summer 2012. The land acquisition only involved 31 land plots in comparison to estimated 174 land parcels at appraisal, due to the design changes to minimize the impact on land acquisition such as construction of the concrete ditches in the populated area and construction of the concrete retaining walls. All land owners were fully compensated in accordance with the provisions of the RAP and no complaints were received (ICR, p.8). b. Fiduciary Compliance Procurement. Procurement followed the Bank's guidelines, and there were no unresolved procurement issues, as reported by the ICR (p.8). Procurement of works was mainly carried out through International Competitive Bidding (ICB) for major civil works contracts, through National Competitive Bidding (NCB) for small-scale road safety and drainage system improvement works. Single Source Selection (SSS) method was used in exceptional cases for (i) Forest Fund inventory services due to the unexpected and urgent need for these services in order not to delay the implementation of the design-build contract with an output-based/lump-sum payment, (ii) preparation of detailed design and bidding documents for additional road safety and drainage system improvement works for Bakurtsikhe-Gurjaani section due to the need to address road safety and emergency needs as a result of heavy rain and landslide on that road section, and (iii) project financial audits services as a continuation of the previous assignment and due to satisfactory performance of the auditor. Financial management. The project financial management (FM) arrangements was generally found satisfactory and acceptable to the Bank, and there were no major issues in financial management, as reported by the ICR (p.8). The FM function was handled by TRRC, which was adequately staffed and was in compliance with financial reporting and audit covenants under the project. Interim Unaudited Financial Reports were delivered on time and were acceptable to the Bank. Unmodified audit reports with the project’s financial statement used were received on time and found satisfactory to the Bank. c. Unintended impacts (Positive or Negative) --- d. Other --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Outcome Modest Modest --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The following lessons were selected from the ICR, with some adaptation: • Successful implementation of new reforms in road safety requires strong ownership of the implementing agency but also close collaboration with all the stakeholders. This project helped build relationship between RD and MIA/Police and develop close collaboration between these two parties to ensure the sustainability of results. • Mainstreaming of the design-build approach in the government program may require enabling legislation. As per Georgia’s current legislative framework, lump-sum payments cannot be made for works delivered and levels of services met as stipulated in the contract conditions. More awareness is needed about the advantages of this approach. In addition, a design-build contract approach needs careful fine-tuning during a pilot stage before fully mainstreaming it into the road asset management practices. • Careful estimation of traffic forecast in a design phase is important, in particular when the initial condition of a road is poor. In this project, diverted traffic was under-estimated at appraisal, as normal traffic volumes were suppressed by the poor initial condition of the road and were using alternative routes. A significant increase in traffic beyond estimates has also implications for the technical specifications/ engineering design of the road. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is concise and outcome- oriented. It presents a candid, well written and argued assessment of the project implementation experience- and the Bank performance in particular. The quality of evidence is adequate. The ICR provides good detail in covering the fiduciary and safeguard project aspects. Lessons are based on the project experience. Some minor issues include inconsistency in the indicator value between the data sheet and the main text, in particular the actual value for traffic volume of cars was indicated as 3,189 in the ICR's Data Sheet and 2,603 in the main text (p.12). The correct value was verified with the project team. a. Quality of ICR Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAKHETI REGIONAL ROADS (P117152) Substantial