Document of The World Bank Report No: ICR00002641 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-47600) ON A LOAN IN THE AMOUNT OF EURO 65.1 MILLION (US$80 MILLION EQUIVALENT) TO ROMANIA FOR A HEALTH SECTOR REFORM PROJECT IN SUPPORT OF THE SECOND PHASE OF THE HEALTH SECTOR REFORM PROGRAM June 19, 2014 Human Development Sector Unit Central Europe and the Baltics Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 2014) Currency Unit=Romanian Lei (RON) Euro 1 = RON 4.45 Euro 1 = US$1.355 US$$1 = Euro 0.74 US$ 1 = RON 3.29 FISCAL YEAR January 1- December 31 ABBREVIATIONS AND ACRONYMS APL Adaptable Program Lending MTR Mid-term Review AFP Avian Influenza Project PAD Project Appraisal Document CAS Country Assistance Strategy PAL Programmatic Adjustment Lending CEE Central and Eastern Europe PMU Project Management Unit CME Continuous Medical Education PHC Primary Health Care PHRD Population and Human Resources Development CVD Cardiovascular Diseases (Japanese Grant) DFID Department for International Development (UK) PIP Project Implementation Plan DPL Development Policy Loan PSRs Project Status Reports Development Policy Loan Deferred Drawdown DPL-DDO Option PPP Public Private Partnership DRG Diagnosis Related Groups QCBS Quality and Cost Based Selection EC European Commission SDC Swiss Agency for Development and Cooperation EMS Emergency Medical Service SHC Secondary Health Care ECA Europe and Central Asia STD Sexually Transmitted Diseases ER Emergency Room TA Technical Assistance EU European Union TB Tuberculosis FD Family Doctor QA Quality Assurance GoR Government of Romania QCBS Quality and Cost Based Selection GP General Practitioner IBRD International Bank for Reconstruction and Development ICB International Competitive Bidding ICU Intensive Care Unit IFC International Finance Corporation IFRs Interim un-audited Financial Reports IMF International Monetary Fund LDP Letter of Development Policy MCH Maternal and Child Health MoF Ministry of Finance MoH Ministry of Health MPHC Multi-Purpose Health Center Vice President: Laura Tuck Country Director: Mamta Murthi Country Manager: Elisabetta Capannelli Sector Manager: Daniel Dulitzky Project Team Leader: Richard Florescu ICR Team Leader: Richard Florescu ROMANIA HEALTH SECTOR REFORM PROJECT IN SUPPORT OF THE SECOND PHASE OF THE HEALTH SECTOR REFORM PROGRAM CONTENTS Data Sheet A. Basic Information ........................................................................................................... v B. Key Dates ....................................................................................................................... v C. Ratings Summary ........................................................................................................... v D. Sector and Theme Codes............................................................................................... vi E. Bank Staff ...................................................................................................................... vi F. Results Framework Analysis ........................................................................................ vii G. Ratings of Project Performance in ISRs ................................................................... xviii H. Restructuring (if any) ................................................................................................ xviii I. Disbursement Profile ..................................................................................................... xx 1. Project Context, Development Objectives and Design ................................................... 1 2. Key Factors Affecting Implementation and Outcomes .................................................. 7 3. Assessment of Outcomes .............................................................................................. 15 4. Assessment of Risk to Development Outcome ............................................................. 21 5. Assessment of Bank and Borrower Performance ......................................................... 21 6. Lessons Learned............................................................................................................ 24 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners............... 25 Annex 1. Project Costs and Financing .............................................................................. 26 Annex 2. Outputs by Component...................................................................................... 28 Annex 3. Economic and Financial Analysis ..................................................................... 32 Annex 4. Bank Lending and Implementation Support/Supervision Processes................. 36 Annex 5. Beneficiary Survey Results ............................................................................... 38 Annex 6. Stakeholder Workshop Report and Results ....................................................... 40 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ......................... 41 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders .......................... 51 Annex 9. List of Supporting Documents .......................................................................... 52 MAP IBRD33469R3 Data Sheet A. Basic Information Health Sector Reform 2 Country: Romania Project Name: Project (APL #2) Project ID: P078971 L/C/TF Number(s): IBRD-47600 ICR Date: 06/16/2014 ICR Type: Core ICR MINISTRY OF Lending Instrument: APL Borrower: PUBLIC FINANCE Original Total USD 80.00 M Disbursed Amount: USD 89.56 M Commitment: Revised Amount: USD 79.14 M Environmental Category: B Implementing Agencies: Ministry of Health, Project Implementation Unit Cofinanciers and Other External Partners: European Investment Bank (EIB)1 B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 02/10/2004 Effectiveness: 06/27/2005 06/27/2005 11/26/2008 11/10/2009 12/17/2010 Appraisal: 10/18/2004 Restructuring(s): 02/16/2011 11/21/2012 12/14/2011 03/14/2013 Approval: 12/16/2004 Mid-term Review: 09/26/2007 09/26/2007 Closing: 12/31/2009 12/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Negligible Bank Performance: Moderately Satisfactory Borrower Performance: Satisfactory 1 EIB financed civil works but no formal agreement was established between EIB and the Bank, though the Bank provided supervision and procurement services on behalf of EIB. C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 3 3 Health 96 96 Sub-national government administration 1 1 Theme Code (as % of total Bank financing) Child health 28 28 Health system performance 29 29 Population and reproductive health 29 29 Rural services and infrastructure 14 14 E. Bank Staff Positions At ICR At Approval Vice President: Laura Tuck Shigeo Katsu Country Director: Mamta Murthi Anand K. Seth Sector Director: Ana L. Revenga Charles C. Griffin Sector Manager Daniel Dultzky Armin H. Fidler Project Team Leader: Richard Florescu Dominic S. Haazen ICR Team Leader: Richard Florescu ICR Primary Author: Nino V. Moroshkina F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) To provide more accessible services of increased quality and with improved health outcomes for those requiring maternity and newborn care, emergency medical care, and rural primary health care. Revised Project Development Objectives (as approved by original approving authority) PDOs were revised in 2008 as part of a restructuring that cancelled activities under Component 3 of the project. Revised PDOs were (i) to provide more accessible services of increased quality and with improved health outcomes for those requiring maternity and newborn care and emergency medical care, and (ii) to support the development of a primary health care strategy. (a) Original PDO Indicators Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Percent of maternal deaths formally documented/investigated (100%) – dropped, Indicator 1 Restructuring Paper (RP), October 24, 2008. Value quantitative and 51.4% 100% 37% qualitative Date of 12/31/2004 12/31/2009 2006 Achievement DROPPED. This indicator tracks all maternal deaths detected and investigated by the Specialty Commission of the MoH. Its operations have been sporadic and not all deaths are investigated and discussed nor feedback given to the providers. The MTR estimated that there was a small probability that this situation may improve in the future (MTR, Comments Aide-Memoire, 2007). (incl.% Since this indicator does not measure the impact of the project, it was agreed to no longer achievement) monitor this indicator, but to use the number of maternal deaths and the maternal mortality ratio instead. In conclusion, instead of evaluating the process, project monitoring started measuring the project impact on improving the situation of maternal deaths. Indicator 2 Neonatal death (ND) and death rate (NR). 25% decrease from Value Death: 2068 baseline Deaths: 673 quantitative and Rate: 9.6/1000 Deaths:1551 Rate: 4.5/1000 qualitative Rate: 7.2/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement OVER ACHIEVED. Neonatal deaths decreased by 67.5% nationwide between 2004 and Comments 2012. (incl.% Data source: Annual National Statistics (ANS), National Reporting System (NRS) and achievement) hospital data nationwide. Indicator 3 Post-neonatal death (PND) and death rate (PNR). Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years 25% decrease from Value Death: 1573 baseline Deaths: 634 quantitative and Rate: 7.3/1000 Deaths:1180 Rate: 4.2/1000 qualitative Rate: 5.5/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement Comments OVER ACHIEVED: Post-neonatal deaths decreased 60% nationwide between 2004 and (incl.% 2012. achievement) Data source: Annual National Statistics (ANS), National Reporting System (NRS). Percent of deliveries where birth-weight is less than 2500 grams – dropped, Restructuring Indicator 4 Project Paper, October 24, 2008. Value quantitative and 9.5% 35% decrease 6.34% qualitative Date of 2004 12/31/2009 2006 Achievement DROPPED. Because the categorization of the maternity wards was only introduced in 2006, and the state statistical system has not yet separated this indicator out based on the facility level, the data from the CRED-financed evaluation were temporarily used. It was agreed that when data is collected for the year 2007, the PMTJ will ensure that the data is Comments also collected for year 2006, and the baseline will be adjusted accordingly (Source: MTR, (incl.% AM 2007). achievement) This indicator was not totally linked to the project, because it was influenced by other factors other than the project’s activities. Following the 2007 MTR, it was agreed to revise the M&E framework and introduce new indicators that better reflect project outcomes. This indicator was dropped because it could not be attributed to project interventions (RP, Attachment No. 1, revised Results Framework). Utilization rates for primary and emergency care stratified by residence and income Indicator 5 status – dropped, RP, October 24, 2008. Value quantitative and Developed in 2005 20% decrease qualitative Date of 2005 12/31/2009 Achievement DROPPED. At the MTR of 2007, this indicator was reworded as “Increase in utilization Comments rates (from those who were ill/had an accident/suffer from a chronic illness or have a (incl.% handicap) for primary care (family doctor/dispensary) among interventions in rural achievement) communities.” Specific reference to rural health centers was removed from the PDO as part of the project restructuring in 2008. Percent of deaths within 48 hours of ER discharge for patients with (a) major trauma or Indicator 6 (b) cardiac emergencies arriving alive at the hospital emergency department – dropped, RP, October 24, 2008. Value (a): 0.22% quantitative and 20% decrease (b): 1.19% qualitative Date of 2005 12/31/2009 Achievement Comments DROPPED. Project Restructuring Paper (October 24, 2008) and Amendment to the (incl.% Loan Agreement of Nov. 2008: this indicator was replaced by Indicator 4 in Table (b) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years achievement) below. Prevalence of chronic diseases in target interventions in rural communities – dropped, Indicator 7 RP, February 11, 2011. Value National average 10% less than the quantitative and national average qualitative Date of 2008 12/31/2009 Achievement DROPPED. This indicator was not mentioned in the PAD, but was included as part of Comments the project restructuring of October 2008 and subsequent Amendment to the Loan (incl.% Agreement of November 2008. The indicator was later dropped as part of the February achievement) 11, 2011 project restructuring. (b) Formally Revised PDO Indicators Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Indicator 1 Maternal Mortality (MM) and Rate (MMR). Value MM: 20% decrease, MM:52 MM: 29 quantitative and MM:41 MMR: 0.24/1000 MMR: 0,14/1000 qualitative MMR 0.19/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement Comments OVER ACHIEVED. Maternal mortality decreased nationwide by 85%. (incl.% Data source: Annual National Statistics (ANS), National Reporting System (NRS) and achievement) countrywide hospital data. Indicator 2 Number of neonatal deaths (ND) and neonatal death rate (NDR). 25% decrease from Value Death: 2068 baseline Deaths: 673 quantitative and Rate: 9.6/1000 Deaths:1551 Rate: 4.5/1000 qualitative Rate: 7.2/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement OVER ACHIEVED. Neonatal deaths decreased by over two-thirds (67.5%) nationwide Comments between 2004 and 2012. (incl.% Data source: Annual National Statistics (ANS), National Reporting System (NRS) and achievement) countrywide hospital data. Indicator 3 Post-neonatal death (PHD) and post-neonatal death rate (PNR). 25% decrease from Value Death: 1573 baseline Deaths: 634 quantitative and Rate: 7.3/1000 Deaths:1180 Rate: 4.2/1000 qualitative Rate: 5.5/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement Comments OVER ACHIEVED. Post-neonatal deaths decreased by 60% nationwide between 2004 (incl.% and 2012. Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years achievement) Data source: Annual National Statistics (ANS), National Reporting System (NRS). Indicator 4 24-hour death rate among patients treated in ER then admitted to ICU in that hospital. Value 15% decrease quantitative and 5.78% (for 6 ICUs) 4.16 % (for 6 ICUs) 4.91% (for 6 ICUs) qualitative Date of 12/30/2007 12/15/2012 12/31/2013 Achievement ACHIEVED. 24-hour death rate reduced by 28% between 2007 and 2012. Although the sample size is too small for the results to be extrapolated to the whole emergency system, Comments the project-related target has been fully met. Only six hospitals received equipment, (incl.% including ventilators for ICUs from the Emergency Care Services Component. Other achievement) hospitals received only monitoring equipment and thus were not monitored by the project. Indicator 5 Development of a primary care rural strategy. Value Development of a Development of a Development of a primary quantitative and primary care rural primary care rural care rural strategy – No qualitative strategy - Yes strategy – Yes Date of 2008 12/31/2012 12/31/2013 Achievement ACHIEVED. The Strategy 2012-2020 and related Action Plan were formally approved by the MoH on February 27, 2012. The Strategy and Action Plan were then submitted to the Health Care and Public Policies Directorate of the MoH for approval in March 2012. Comments According to the MoH Report on the implementation status dated March 15, 2013 for the (incl.% Action Plan of the National Reform Programs 2011-2013, the Strategy was revised in achievement) order to observe the provisions of the Government Decision No. 870/2006. The Strategy expected to be approved within the framework of the Health Sector Reform Strategy for the next programming period of the EU financing exercise (2014-2020). (c) Original Intermediate Outcome Indicators Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Percentage of maternity beds utilizing “rooming in” system – dropped, RP, October 24, Indicator 1 2008. Value quantitative and 37.1% 70% qualitative Date of 12/31/2004 12/31/2009 Achievement Comments DROPPED. No information system available to track “rooming in” practices; the (incl.% monitoring process was deemed problematic to develop (MTR, AM, 2007). achievement) Percent of deliveries attended by health personnel skilled to the appropriate level of care - Indicator 2 dropped, RP, October 24, 2008. Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years - 3.44% newborns - 2.58% newborns Value transferred transferred quantitative and - 1.39% newborns at home 70% of deliveries - 1.16% newborns at qualitative home Date of 12/31/2004 12/31/2009 12/31/2013 Achievement Comments (incl.% DROPPED. achievement) Indicator 3 Occupancy rates by units - dropped, RP, October 24, 2008. NN NN OR: 257.4 OR: 246.8 O: O: Value OR: 269.1 OR: 256.72 quantitative and 20% increase OG: OG: qualitative OR: 279.9 OR: 276.7 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement DROPPED. With the introduction of the referral system, the average length of stay Comments increased, especially at Level 3 maternities as more newborns were surviving than before (incl.% the introduction of the system (Source: RP, 2008). Legend: NN = Neonatal; O = achievement) Obstetric; OG = Obstetric Gynecology. Indicator 4 Average length of stay by unit - dropped, RP, October 24, 2008. NN NN ALS: 5,6 ALS: 5,45 Value O: O: quantitative and 20% imp. ALS: 3,95 ALS: 5,69 qualitative OG: OG: ALS: 4,36 ALS: 5,16 Date of 31/12/2004 12/31/2009 12/31/2013 Achievement Comments DROPPED. With the introduction of the referral system, the average length of stay (incl.% increased, especially at Level 3 maternities as more newborns were surviving than before achievement) the introduction of the system (Source: RP, 2008). Proportion of cases fulfilling pre-defined criteria of quality - dropped, RP, October 24, Indicator 5 2008. Value quantitative and N/A 70% qualitative Date of 12/31/2009 Achievement Comments DROPPED. There was no system established that could monitor this indicator. Although (incl.% the project supported the development of a quality audit system, it seemed problematic achievement) that the system would produce the required information within the project duration. Patient satisfaction with revised maternity and neonatal services – dropped, RP, October Indicator 6 24, 2008. Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years -71.5% women - satisfied by Value the medical services quantitative and qualitative -49% satisfied by the environment Date of 12/31/2004 Achievement DROPPED. Some national data was available in the RHS, 2004; this information Comments included all hospitals and was not limited to maternities; the survey measured the number (incl.% of satisfied women, not the level of satisfaction. Following Project Restructuring Paper achievement) (October 24, 2008), this indicator was revised by establishing year 2008 as the baseline and measuring patient satisfaction by maternity level – see Indicator 3 in Table (d) below. Utilization rates for dispatch of (a) ambulances and (b) ER services – dropped, RP, Indicator 7 October 24, 2008. Value (a) 10.17% quantitative and (a) 20.54% 30% increase (b) 9.74% qualitative (b) 9.19% Date of 12/31/2004 12/31/2009 12/31/2013 Achievement Comments DROPPED. The utilization indicator was redefined given the constraints on measuring (incl.% the residence/income status of the patients serviced (MTR, AM, 2007). achievement) Response times for emergency services by urgency and severity – dropped, RP, October Indicator 8 24, 2008. Value 19.73 minutes 35 min. quantitative and 30% improvement 123.30 minutes 167 min qualitative Date of 2004 12/31/2009 12/31/2013 Achievement Comments DROPPED. The indicator was revised and modified to make it more specific by revising (incl.% the Results Framework and the Amendment to the Loan Agreement of Nov. 2008. It achievement) became Indicator 4 in Table (d) below. Indicator 9 Fatality rates for ER and ambulance cases by case types - dropped at the MTR, 2007. Ambulance (Data from Remssy 4) a. patients found dead/total deceased patients 73.41% 20% improvement b. patients who died in the Value medical team quantitative and presence/total dead qualitative patients 6.49% c. patients who died while being transported /total dead patients 10.10% d. ER 1.41% Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Date of 2005 12/31/2009 Achievement Comments (incl.% DROPPED. This indicator was dropped and never monitored. achievement) Communication problems leading to delayed or missed calls – dropped, RP, October 24, Indicator 10 2008. Value quantitative and N/A < 5% calls 2% qualitative Date of 12/31/2009 12/31/2013 Achievement Comments DROPPED. However, current data on the indicator indicates achievement against the (incl.% original target value of less than 5% calls. achievement) Patient satisfaction with revised ambulance and ER services – dropped, RP, October 24, Indicator 11 2008. Value quantitative and N/A 75% qualitative Date of 12/31/2009 Achievement Comments DROPPED. The indicator on patient satisfaction was removed only because a proper (incl.% baseline could not be established before project intervention (Source: MTR, AM, 2007). achievement) Performance of new MPHC pilots relative to original design proposal (utilization, Indicator 12 quality, etc.) – dropped, RP, October 24, 2008. Utilization rates for Value MPHC: 80% quantitative and N/A Hospital utilization qualitative rate: 20% decrease Date of 12/31/2009 Achievement Comments (incl.% DROPPED. Dropped as part of the restructuring of Component 3 (three) in 2008. achievement) Patient/physician satisfaction with different models of revised primary health care Indicator 13 services and sub-loan scheme – dropped, RP, October 24, 2008. Value quantitative and N/A 80% qualitative Date of 12/31/2009 Achievement Comments (incl.% DROPPED. Dropped because of the restructuring of Component 3 in 2008. achievement) Indicator 14 Repayment rate for credit/lease scheme – dropped, RP, October 24, 2008. Value N/A 100% Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years quantitative and qualitative Date of 12/31/2009 Achievement Comments (incl.% DROPPED. Dropped because of the restructuring of Component 3 in 2008. achievement) NHA information is used in decision-making relating to the financing or organization of Indicator 15 the health system in Romania – dropped, RP, October 24, 2008. Value quantitative and qualitative Date of Achievement Comments DROPPED. This indicator (designed to evaluate the usefulness, impact, and (incl.% sustainability of the NHA approach in Romania, RP, October 2008) was replaced by achievement) Indicator 8 in Table (d) below. Matching of civil works and equipment acquisition activities to required training and Indicator 16 related technical assistance – dropped, RP, October 24, 2008. Value quantitative and qualitative Date of Achievement Comments (incl.% DROPPED. achievement) Indicator 17 Progress on M&E upgrading – dropped, RP, October 24, 2008. Value quantitative and qualitative Date of Achievement Comments (incl.% DROPPED. Indicator was replaced with Indicator 10 in Table (d) below. achievement) (d) Revised Intermediate Outcome Indicators Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 Neonatal mortality by level of MCH facility - introduced, RP, October 24, 2008. Value Level 1: 4.06‰ Level 1: 2.09%o quantitative and Level 2: 8.40‰ 10% qualitative Level 3: 10.6‰ Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Level 2: 5.31%o Level 3: 8.22%o Date of 2008 12/31/2010 12/31/2013 Achievement OVER ACHIEVED: the decrease was 49% for Level 1, 37% for Level 2, and 22% for Comments Level 3. (incl.% Data source: National Center for Organizing and Assuring of Informational and achievement) Informatics System in Health (NCOAIISH). Indicator 2 Maternal mortality by level of MCH facilities – introduced, RP, October 24, 2008. Level 1: 0.09‰ Level 1: 10% Level 1: 0.15‰ Level 2: 0.07‰ decrease Value Level 3: 0.17‰ Level 2: 10% Level 2: 0.08‰ quantitative and decrease qualitative Level 3: 3.5% Level 3: 0.16‰ decrease Date of 2008 12/31/2010 12/31/2013 Achievement OVER ACHIEVED. The actual values above are for 2011 (the latest validated data available); it shows a slight increase for Level 1 and 2, which is explained by the fact that following the improvement of MCH care conditions more women gave birth in maternities than at home, and consequently more of them died in maternities than at Comments home, compared to the baseline data. However, the preliminary data for 2012 (based on (incl.% hospitals reporting, not yet validated by NCOAIISH) shows the following values: Level achievement) 1: 0.06% (33% decrease.), Level: 2 0.05% (29% decrease.), and Level 3: 0.05% (70% decrease). For Level 3 maternities, the target was achieved in 2011 (0.16% means a 5.9% decrease). Data source: National Center for Organizing and Assuring of Informational and Informatics System in Health (NCOAIISH). Indicator 3 Patient satisfaction with maternity/neonatal services – introduced, December 15, 2011. Level 1: 10% improvement Value Level 1: 109/140 Level 1: 127/140 Level 2: 10% quantitative and Level 2: 110/140 Level 2: 121/140 improvement qualitative Level 3: 109/140 Level 3: 119/140 Level 3: 3.5% improvement Date of 2008 12/31/2012 12/31/2013 Achievement OVER ACHIEVED: The target value of 3.5% for Level 3 was formally established by a WB letter to MoPF “Revised Performance and Monitoring Indicators” of February 21, 2011. The date of achievement 2012 was formally established based on the WB letter to Comments MoPF “Revised Performance and Monitoring Indicators” of December 15, 2011. (incl.% Level 1: 16.5% improvement – OVER ACHIEVED achievement) Level 2: 10% improvement – ACHIEVED Level 3: 9.2% improvement – OVER ACHIEVED Data source: patient satisfaction survey. Response times for emergency services by urgency category and urban/rural areas - Indicator 4 introduced, RP, October 24, 2008. Value URBAN URBAN Ambulance quantitative and R & Y: 18 min 25 sec R & Y: 15 Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years qualitative Green: 52 min 33 sec minutes URBAN Green: 8% less R & Y: 17 min 23 sec RURAL than baseline Green: 51 min 30 sec R & Y: 24 min 43 sec Green: 36 min 2 sec RURAL RURAL R & Y: 20 min. R & Y: 25 min 47 sec Green: 8% less Green: 50 min 14 sec than baseline SMURD URBAN R & Y: 6 min 47 sec RURAL R & Y: 17 min 5 sec NATIONAL URBAN R & Y: 15 min 57 sec RURAL R & Y: 25 min 2 sec Date of 2007 12/31/2012 12/31/2013 Achievement ACHIEVED. Data were only collected during the first years for county ambulance services. At the national level, this indicator was reported as an aggregate for 0 and 1st degree emergencies. In 2009, the reporting system was changed and the emergency degrees were reset as red, yellow and green codes, with red and yellow code cases corresponding to former 0 and 1st degree emergencies, while the green code referred to Comments the former 2nd degree code. The number of yellow code cases continually increased, (incl.% while the number of red code cases remained relatively constant (for the urban areas the achievement) proportion of red/yellow cases was around 0.4 in 2009 and around 0.2 in 2013. For the rural areas, the proportion of red/yellow cases was around 0.5 in 2009 and around 0.3 in 2012). The total number of red/yellow cases also increased (for the urban areas there were 544,154 red/yellow code cases in 2007 and 897,235 cases in 2013; for the rural areas there were 354,293 green code cases in 2007 and 90,244 cases in 2013). Death rate in emergency departments (dead patients/total alive presentations) – Indicator 5 introduced, RP, October 24, 2008. Value quantitative and 0.079% 0.064% 0.066% qualitative Date of 2007 12/31/2012 12/31/2013 Achievement NOT ACHIEVED. Against the baseline this indicator decreased by 16.5% due to the growth of pre-hospital intervention and communication capacity, which was partly Comments enhanced by the purchasing of equipment as part of the project. Patients with certain (incl.% types of pathology, which had previously caused the medical staff to declare them dead achievement) in pre-hospital or caused them to arrive dead at emergency, were now brought to the Emergency Departments (ED) showing vital signs. Despite therapeutic efforts, some of these cases die in the EDs at a later point. Indicator 6 Fatality rate of patients treated in small ERs - introduced, RP, October 24, 2008. Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Value 0.04% 0.042% quantitative and 5% decrease qualitative Date of 2008 12/31/2010 12/31/2013 Achievement Comments NOT ACHIEVED. Against the baseline this indicator has increased by 5%. Same (incl.% comment as indicator 5 above. achievement) Fatality rate after 24 hours from admission of patients treated in hospital ICUs - was Indicator 7 introduced, RP, October 24, 2008. Value 2.84% quantitative and 5% decrease 2.76% qualitative Date of 2007 12/31/2010 12/31/2013 Achievement Comments NOT ACHIEVED. The data were collected from all six hospitals which have received (incl.% different equipment for ICUs. Against the baseline, this indicator has decreased 2.8%, achievement) after an initially greater decrease. Health accounts, appropriate regulations issued with respect to three main areas: (i) Indicator 8 internationally comparable Romanian matrices; (ii) institutional responsibilities; and (iii) timeframe for dataflow – introduced, RP, October 24, 2008. Value quantitative and N/A 3 of 3 qualitative Date of 2008 12/31/2012 Achievement NOT ACHIEVED. The internalization of NHA system in Romania has been achieved, Comments taking into account that the NSI reports are according to OECD, EUROSTAT and WHO (incl.% requests. However, the legal framework regarding a detailed NHA system, according to achievement) budget lines, is not in place. Average lag time for implementing project activities beyond critical dates agreed in the Indicator 9 MTR, 2007. Value quantitative and Lag < 5 months Lag < 3 months Lag >3 months qualitative Date of 2009 12/31/2011 12/31/2013 Achievement Comments ACHIEVED. Introduced at the MTR 2007, but only modified in the Revised (incl.% Performance and Monitoring Indicators Letter of February 2011. achievement) Indicator 10 Timely submission of project progress reports – introduced, RP, October 24, 2008. Value quantitative and On time On time On time qualitative Date of 2008 12/31/2012 12/31/2013 Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Achievement Comments (incl.% ACHIEVED. achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 12/30/2004 Satisfactory Satisfactory 0.00 2 06/03/2005 Satisfactory Satisfactory 0.00 3 12/16/2005 Satisfactory Satisfactory 0.12 Moderately 4 12/20/2006 Unsatisfactory 0.29 Unsatisfactory Moderately 5 08/20/2007 Unsatisfactory 12.66 Unsatisfactory 6 12/20/2007 Moderately Satisfactory Moderately Satisfactory 19.61 7 07/22/2008 Moderately Satisfactory Moderately Satisfactory 33.27 8 06/11/2009 Satisfactory Satisfactory 39.32 9 12/17/2009 Satisfactory Satisfactory 46.51 10 06/22/2010 Satisfactory Satisfactory 50.19 11 01/08/2011 Satisfactory Satisfactory 52.02 12 07/30/2011 Satisfactory Satisfactory 62.99 13 01/09/2012 Satisfactory Satisfactory 69.48 14 06/27/2012 Satisfactory Satisfactory 77.42 15 12/29/2012 Satisfactory Satisfactory 81.44 16 05/29/2013 Satisfactory Satisfactory 84.05 17 12/28/2013 Moderately Satisfactory Moderately Satisfactory 85.38 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Despite improvements achieved $36.27 during the implementation of 11/26/2008 Y MS MS €29.34 the first two components, the third component (PHC and ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Rural Medical Services) was stalled due to (i) changes in the legislation of health sector financing, which no longer allowed the MoH to finance investments in PHC infrastructure; and (ii) developments in the capital market which gave group practices better access to loans. Key changes: (a) Revision of the PDO; (b) Cancellation of previously envisaged activities under the third component; (c) Reallocation of funds (EUR 9.52 million) from the third component to other components; (d) Revision of Results Framework. Extension of closing date from $45.7 11/10/2009 N S S December 31, 2009 to €33.8 December 31, 2010. Extension of closing date from $49.2 12/17/2010 N S S December 31, 2010 to February €40.0 28, 2011. (a) Revision of Results Framework; (b) Introduction of new activities, e.g., rehabilitation and equipment of $52.2 vaccine production and ampoule 02/16/2011 N S S €42.28 filling and sealing area at the Cantacuzino Institute; (c) Extension of closing date from February 28, 2011 to December 15, 2011. Extension of closing date from $81.44 11/21/2012 N S S December 15, 2012 to March €65.97 15, 2013. $69.48 (a) Revision of Results 12/14/2011 N S S €56.28 Framework; (b) Reallocation of Loan proceeds. (a) Extension of the closing date $84.05 from March 15, 2013 to 03/14/2013 N S S €68.08 December 31, 2013; (b) Completion of TAs on financial ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions impact of health care reforms and on design and implementation of next generation reforms. I. Disbursement Profile 1. Project Context, Development Objectives and Design 1. The Romania Health Sector Reform Project (APL 2) was the second phase of an agreed two-phase adaptable program loan (APL) with an estimated budget of US$206.5 million (€167.9 million), supported by a US$80.0 million (€65.1 million) IBRD loan, a US$81.8 million (€66.4 million) European Investment Bank (EIB) loan, and US$44.8 million (€36.4 million) in funding from the Government of Romania. The project was closely linked to the objectives of the FY02- 04 Country Assistance Strategy (CAS) and built on activities initiated in the first phase of the program. The project was approved on December 16, 2004 and became effective on June 27, 2005, with an original closing date of December 31, 2009. 1.1 Context at Appraisal 2. Romania, the second largest country in Central and Eastern Europe, was larger than 19 of the 25 members of the European Union (EU) at the time. Romania was classified as a lower middle-income country, with a GNI per capita equal to US$2,260 and a population of 21.7 million, 25 percent of which were below the poverty line. Since 2000, the Government of Romania was implementing different macroeconomic reforms to support growth and development. A disciplined fiscal policy and complementary tight monetary policy triggered a decline in inflation and interest rates, and the fiscal deficit had been brought under control. In 2000-2004 GDP grew by an average 5 percent. The Government had taken some steps to underpin growth through mechanisms of accountability and transparency, ensuring that the population benefited from these reforms by improving the social and health sectors, among others. Notwithstanding these achievements, public service delivery required significant improvement when this project began. 3. In 1997, Romania began to enact key reforms in the health sector to shift the health system from a centralized government model to a more decentralized and diversified one. The main change was the establishment of a compulsory health insurance fund paid for by an earmarked wage tax and by contracting public service delivery with public and private service providers. In the area of governance, there were major revisions of the health insurance law to strengthen the accountability of the insurance fund in order to increase the role of local authorities in the ownership and accountability of health care providers. 4. The ongoing program supported by the World Bank in the health sector was based on a reform strategy elaborated by the Government in 1997-1998 and outlined in a 1999 Letter of Sector Development Strategy, which identified the following key issues:  weaknesses in governance of the system and the legislative framework;  shortcomings in the efficiency, equity and transparency of sector financing;  inefficient use of physical capacity and human resources in health care delivery;  critical infrastructure deficiencies resulting from inadequate maintenance and investment;  mismatch between population health needs and health services distribution and priorities;  consumer dissatisfaction with the health services. 1 5. At project preparation stage, substantial progress had been made on the agenda, and a clear sector strategy was in place for addressing remaining reform needs. In terms of efficiency, equity and transparency of health sector financing, there had been an increase in public funding for the health sector, capacity building for health insurance, and piloting of new case- based payment mechanisms for hospitals. The remaining agenda included increasing the resources allocated for primary care services, ambulatory care, and hospital day surgery; rolling out new payment mechanisms for hospitals; containing the escalation of pharmaceutical costs; better defining the benefit package to be covered by health insurance, and ensuring the more transparent and affordable contribution of private financing. 6. Progress in the area of capacity and efficiency improvements included preparing and approving a high-level rationalization strategy, closing about 15 percent of acute hospital beds, converting hospitals into medico-social care units, and developing financial and contractual approaches for home care. While many activities were supported under APL 1,2 more investment and effort was required in the following areas: improving the distribution of resources and the motivation of medical personnel in remote areas; restructuring underutilized hospital capacities; and introducing new models of health care, e.g., home care, social long-term care, and ambulatory/ day care services, especially in remote and unprivileged areas. 7. Improvements in infrastructure had been addressed through the upgrade of equipment for essential hospital services and pre-hospital emergency care, a major upgrade of hospital equipment financed through commercial credits with governmental guarantees, lease schemes for medical offices, and the interest of different donors. Nonetheless, continued investment in rehabilitation was needed to modernize maternities and neonatal care units (maternal mortality was five times higher than EU average and neonatal death rate 2.5 times higher3) as well as intensive care systems. 8. Population health needs had been addressed through the development of a public health strategy, capacity building, and strengthening national programs (especially TB, reproductive health and HIV/AIDS). Important tobacco control legislation had also been passed and community nursing was implemented on a pilot basis. 9. The area of consumer satisfaction saw relatively little activity, although there were some information campaigns on the rights of insured. This agenda was to be moved forward through increasing the focus on informal payments and explicitly addressing patient satisfaction. 10. Various donors were involved in the different areas outlined above, including the Bank via previous health sector projects as well as through Programmatic Adjustment Lending (PAL). A first health project, at a total cost of US$224 million (including a US$150 million IBRD loan), was implemented over 1992-1999 and supported the upgrading of selected Primary Health Care (PHC) units, maternity and emergency medical services, the procurement of essential drugs, TB control, and capacity building. The second project, the Health Sector Reform Project (APL 1), 2 ICR for APL#1, Report No. 24280, May 31, 2002; PAD, Report No. 17379, December 23, 1998. 3 WHO Health-for-ALL database, 2001. 2 was completed in June 2004. APL 1 was a US$69 million project (including a US$40 million IBRD loan, disbursed 98 percent) and covered the upgrade of essential hospital care facilities (operating theaters and ICUs), further improvements in emergency medical services, support for PHC in six judets (counties), strengthening the planning capacities of the MoH and the counties, and selected public health interventions. Triggers to move from APL 1 to APL 2 (all of which were achieved) included:  As necessary, amendments to the legal and regulatory framework for health care providers and for financing of health sector investment initiated by MOHF.  Public Health Strategy with targets in priority areas issued.  A lending and/or leasing scheme for financing family physicians in primary care practices has been established by the Health Insurance House and operating at least on a pilot basis for six months or more.  3-4 judets in each socio-economic development region (at least 30 in total) have completed health services plans and associated investment and human resource plans for developing and rationalizing capacity in the sector.  Substantial rogress has been made on contractual commitments (at least 80 percent) and disbursements (at least 70 percent) from APL 1. 11. The APL was chosen as a lending instrument because activities to be undertaken were more appropriately funded through an investment operation and APL 2 was a follow-on to APL 1; and the investments were complementing reforms under the ongoing PAL (with the APL providing additional TA to PAL policy actions. APL 2 was to be implemented over four years and seven months, financing civil works, goods, technical assistance, training and incremental operating costs. APL 2 was included in the FY02-04 CAS that had as one of its objectives the improvement of delivery of health services. The project was one of the targeted poverty interventions identified in the CAS to reduce inequity of access to basic social services across regions and for vulnerable groups. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 12. The objectives of the overall APL program were as follows: (a) improve efficiency and equity in the planning and regulation of the health service delivery system; (b) reduce preventable deaths among emergency medical cases; (c) improve access and quality health care in poor and remote areas; and (d) help the Romanian health sector to better focus on priority public health problems, thereby reducing preventable illnesses and deaths. 13. The specific objectives for Phase 2 of the APL were to provide more accessible services of increased quality and with improved health outcomes for those requiring maternity and newborn care, emergency medical care, and rural primary health care. 14. PDO outcome indicators for APL 2 included: (i) the percentage of maternal deaths formally documented/investigated; (ii) neonatal and post-neonatal deaths and deaths rates, (iii) the percentage of deliveries where birth-weight is less than 2,500 grams; (iv) utilization rates for primary and emergency care stratified by residence and income status; and (v) the percentage of 3 deaths within 48 hours and ER discharge for patients with major trauma or cardiac emergencies arriving alive at the hospital emergency department. 1.3 Revised PDOs (as approved by original approving authority) and Key Indicators, and reasons/justification 15. PDOs were revised in 2008 to reflect changes in the project design with the cancellation of Component 3. The revised PDOs were (i) to provide more accessible services of increased quality and with improved health outcomes for those requiring maternity and newborn care and emergency medical care; and (ii) to provide support for the development of a primary health care strategy. 16. Revised PDO outcome indicators were (i) Maternal Mortality (MM) and Maternal Mortality Rate (MMR) by hospital, county, and region (new indicator); (ii) neonatal deaths (ND) and rate (NDR) by hospital, county, and region (same as before); (iii) post-neonatal deaths (PND) and rate (PNR) by hospital, county, and region (same as before); and (iv) the 24-hour death rate among patients treated in the ER, then admitted to ICU in that hospital (new indicator). PDO indicators related to percent of maternal deaths formally documented/investigated, percent of deliveries where birth rate is less than 2500 grams, and utilization rates for primary and emergency care stratified by residence and income status were dropped largely because they were not specific to the interventions to be measured. As part of a second restructuring in 2011, the target and frequency of data collection related to the 24-hour death rate indicator was refined. 17. The reason for restructuring the PDO in 2008 was due to changes in health sector legislation, which prevented MoH from financing investments in primary health care infrastructure and faster than expected development of the capital market, which gave general practitioners better access to loans (with accession to the EU, subsidized credits also became problematic due to EU regulations on state aid). . In addition, as recorded in the mid-term review (MTR), there was little buy-in for the introduction of multi-purpose health centers (MPHCs) envisaged under Component 3. 18. Consequently, MoH proposed cancelling Component 3 activities related to establishment of multifunctional health centers and implementation of micro-credit schemes for family doctors. The component was significantly downsized and retained only technical support to develop a new PHC strategy. Component 3 funds (€9.52 million) were reallocated to other components where additional financing needs were identified. 1.4 Main Beneficiaries 19. As discussed in the PAD, pregnant women and newborns, rural populations, and those needing emergency services were the principal beneficiaries of the project. Beneficiaries were also defined in the loan agreement as a family doctor or a family doctors’ association providing medical services in selected areas, including but not limited to rural areas. The direct beneficiaries of project investments were judged to be the communities benefiting from the scaling-up of the health care services and the population served by the hospitals with modernized emergencies and ICUs. Other key beneficiaries included: (a) the general population of Romania, 4 and particularly vulnerable groups, e.g., those living in remote and unprivileged areas, Roma minorities, women, children, and the elderly; (b) the Ministries of Public Health and Finance and various subordinated structures to these ministries, including the Project Management Unit, and the Cantacuzino Institute; (c) local authorities, academic institutions, various universities and training centers; and (d) physicians, nurses, other health professionals, managers, policy makers, etc. 1.5 Original Components (as approved) 20. The project had the following five components: 21. Component 1: Maternity and neonatal care (US$129.0 million equivalent, €104.9 million). The objective of this component was to fund the rehabilitation of maternity and neonatal care units and provide medical and other equipment to ensure high quality neonatal and maternity services. Technical assistance and training were to be provided to ensure implementation of the best international practices, building on already existing partnerships between the Government, WHO, UNICEF, UNFPA, and bilateral donors, including USAID and the Swiss Development Cooperation. Support was to be provided to improve the capacity of health care authorities and support provider units to monitor service quality and access. 22. Component 2: Emergency Care Services (US$58.1 million equivalent, €47.2million).This component’s objective was to upgrade hospital emergency areas and develop and implement an integrated ambulance dispatch capability. It had two sub-components: (i) upgrade hospital emergency areas; and (ii) integrate the ambulance dispatch system. 23. Component 3: Primary Health Care and Rural Medical Services (US$14.0 million equivalent, €11.4 million). This component focused on improving the accessibility and quality of basic medical services in rural and small urban areas and supporting two of the most important activities included in the Primary Care Strategy approved by the Government. It had two sub- components: (i) multipurpose health centers; and (ii) sub-loans for family doctors. 24. Component 4: National Health Accounts and Planning (US$0.64 million equivalent, €0.52 million). This component’s objective was to support the development of the National Health Accounts and the preparation of proposals for rationalization and service development projects with the following sub-components: (i) national health accounts (NHA); and (ii) planning and program development. 25. Component 5: Project Management (US$4.72 million equivalent, €3.85 million). The objective of this component was to support the operation of the Project Management Unit (PMU), building on the implementation arrangements of Phase 1 and expanding its responsibilities to properly incorporate the new activities related to the physical rehabilitation of buildings and the procurement of goods and equipment, and to ensure appropriate monitoring and evaluation of project activities. 5 1.6 Revised Components 26. The 2008 restructuring revised the project components as follows: 27. Component 1: Maternity and neonatal care (€108.75 million, of which €23.14 million was from IBRD; US$133.76 million equivalent, of which US$28.46 million equivalent was from IBRD). The objective of this component was to fund the rehabilitation of maternity and neonatal care units and to provide medical and other equipment for high quality neonatal and maternity services. Technical assistance and training was provided to ensure the implementation of the best international practices. 28. Component 2: Emergency Care Services (€42.30 million, of which €33.8 million was from IBRD; US$52.03 million equivalent, of which US$41.57 million equivalent was from IBRD). This component sought to upgrade district and local emergency areas, develop multi- trauma operating theaters in emergency hospitals, develop an integrated ambulance dispatch system, and support the establishment of a regional telemedicine pilot project, thus increasing the effectiveness of the emergency system. 29. Component 3: Primary Health Care and Rural Medical Services (€5.64 million, of which €4.17 million was from IBRD; US$6.94 million equivalent, of which US$5.13 million equivalent was from IBRD).This component sought to prepare the ground for improving the accessibility and quality of basic medical services by establishing criteria for the identification of underserved areas for further interventions, and supporting the preparation of a primary health care strategy. 30. Component 4: National Health Accounts and Planning (€0.21 million, of which €0.15 million was from IBRD; US$0.26 million equivalent, of which US$0.18 million equivalent was from IBRD). This component’s objective was to support the development of a national health accounts system and preparation of proposals for rationalization and service development projects. 31. Component 5: Project Management (€4.46 million, of which €2.88 million was from IBRD; US$5.49 million equivalent, of which US$3.54 million equivalent was from IBRD). The objective of this component was to support the operation of the PMU in activities related to rehabilitation of infrastructure, equipment delivery, and the monitoring and evaluation of project activities. 1.7 Other significant changes 32. Restructuring(s). A first restructuring of the project took place in 2008. It included amendments to the PDOs, restructuring of the third component (cancelling the activities related to the rural multifunctional health centers and the micro-credit line for general practitioners), reallocating funds (€9.52 million) to other components, making revisions in the results framework, and consolidation of loan categories into a single category to ensure greater flexibility in the loan allocations. A restructuring in December 2011 introduced new technical assistance (TA) activities to support ongoing sector reforms; introduced a minor reallocation of 6 loan proceeds among categories of expenditures; fine-tuned the results framework; and extended the closing date. 33. Closing date extensions. The project was extended a total of five times. The first extension of 12 months was granted from December 31, 2009 to December 31, 2010. This was followed by a second extension (in two steps) to December 15, 2011, a third extension to December 15, 2012, and a fourth extension to March 15, 2013. The last and fifth extension was granted on March 14, 2013, with a final closing date of December 31, 2013. The additional time was needed largely due to start-up delays, delays experienced with rehabilitation of maternities, and challenging procurement of medical equipment (more in Section 2). 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 34. Original project design was consistent with the Government’s policies and objectives. The project was based on a health reform strategy elaborated by the GoR in 1997- 1998, and its major areas were outlined in the 1999 Letter of Sector Development Strategy. Restructurings took a practical approach to incorporating the evolution of sector developments (such as those that led to the cancellation of activities under Component 3), related implications for the PDOs, and refinements to the results framework based on implementation experience. However, the general design of the project remained relevant over what turned out to be a significant lifetime of the project. 35. Project design was well coordinated by GoR with other development partners. MoH worked closely with DFID, EIB, USAID, the Swiss Agency for Development and Cooperation (SDC), UN agencies, EU, and many others. Different donors complemented project investments through their special mechanisms and influenced project outcomes. The EIB co-financed a substantial proportion of project costs. Much work was done by the SDC, based on the memorandum signed between the World Bank and the agency training ambulance and medical personnel (approximately 3,000 persons at PHC and hospital levels were trained). EU and UN agencies provided a wide range of technical assistance to the project. 36. Project design was in line with the CAS. The project was included in the FY02-04 CAS, which established as one of its objectives improving the delivery of health services. This project was one of the targeted poverty interventions identified by the CAS to reduce the inequity of access to basic social services across regions and for vulnerable groups and complementing health sector reforms under development policy agenda. The flexible design allowed APL 2 to add the provision of new TA in 2011 and 20124 to support the implementation of DPL series prior actions for provision of quality health care within limited fiscal space.5Thus the project remained relevant through CPS FY09-13 beyond improvements in health care service delivery. 4 See restructuring documents 2011 and 2012 5 APL 2 TA supported the DPL1-3 (2009-2012) prior actions to reduce the excessive number of acute care hospital beds and introduce copayments; and the DPL-DDO (2012) prior actions for updating the package of health benefits insured by the Government. 7 37. Project design incorporated lessons learned. Phase 1 provided valuable lessons for the design of the second phase. Previous experience clearly demonstrated that the availability of committed counterparts and sound implementation capacity, with good planning and execution capabilities, are critical to success. 38. The chosen instrument – an Adaptable Program Loan (APL) – was adequate to respond to the Government’s sector strategy needs. The choice of instrument provided a long- term framework for implementing reforms in the sector, including predictable financing and Bank support. Moreover, the investments financed by this loan were complimentary to other reforms achieved through the PAL. 39. Identified project risks were well mitigated. Out of four project risk areas identified, only one was rated as Substantial, namely the slower pace of hospital restructuring processes or reduced level of commitment for implementing rationalization activities. This risk was mitigated by providing leverage though the sequence of PAL conditionality mechanisms and triggers, support in the form of technical assistance funded by Private and Public Sector Institutional Building Loan (PPIBL), monitoring the project’s progress as part of the policy dialogue undertaken throughout project implementation, and finally, by approval of the rationalization strategy by the GoR. 2.2 Implementation 40. The project was approved at the end of 2004 and became effective in mid-2005. During the first two years of implementation, the project experienced delays and slow disbursements (ISRs 4 and 5 rated implementation progress moderately unsatisfactory and unsatisfactory, respectively). This was largely due to a change in government soon after project approval (more below). 6 Project status improved to moderately satisfactory by the end of 2007, with some limited action under the PHC and NHA components keeping the ratings at moderately satisfactory in 2008 at the time of the MTR.7 Project performance became solidly satisfactory from 2009 onward. As evidenced in the data sheet, the majority of outcomes were achieved, with some were over achieved (see details in Annex 2). 41. The project benefited from several factors:  GoR Leadership. Strong leadership and vision from MoH drove achievements under the project, especially in the areas of emergency services and telemedicine. The country became a model for other countries in the introduction and implementation of integrated emergency medical system. 6 Health APL2 was approved in December2004, and a new Government came in in early 2005. 7 Aide Memoire, Mid-term supervision mission, September 26-October 8, 2007, and Aide Memoire as of June 2-6, 2006. 8  Project Implementation Capacity. Strong existing capacities for project implementation and further capacity development were key successes of the project. The project benefitted from the experience and knowledge of human resources deriving from APL 1. Despite a certain turnover at management level, the project succeeded in retaining key staff, ensuring continuity and retention of qualified members at the PMU. Broad reforms in public financial management in Romania also created an environment conducive to better performance by government agencies, including those responsible for project implementation.  Flexibility to Support Additional Activities. Flexibility under the APL allowed for the provision of critical TA. Per agreement at the MTR, support for training under the MCH component to develop an operational manual for referral and quality assurance (QA) measures was implemented. Some project funds were also used to support TA activities related to the development and provision of basic training for Health Technology Assessments. Bank support to the Emergency Medical Component also expanded during project implementation—the component experienced savings and flexibility allowed for additional activities to be undertaken when resources became available.8 42. Major challenges encountered during project implementation included:  Start-up Delays. A change in government soon after project approval led to start-up delays as priorities were revisited. There was a need to review and reach agreement on the approaches to PHC and rural medical services, and on the priority MCH investments. Further delays were experienced with lengthy planning of investments and identification of specific beneficiaries (review of maternity capacity and determination of maternities to be supported by the project with civil works, training, and equipment) as well as the establishment of project implementation mechanisms (the Steering Committee was established in mid- February 2006, project implementation responsibilities delegated at end June 2006, Technical Working Groups by component established by June 2006). The project also accommodated a high turnover of ministers, which caused implementation to slow.  Rehabilitation Delays. While World Bank and EIB collaboration is notable, there was a substantial delay in the revision of the architectural designs requested by EIB for the rehabilitation of the maternities. Twenty-two maternity hospitals were included in the infrastructure rehabilitation program supported by the project. This activity was mainly financed by the EIB, with the World Bank accounting for only a small share of co-financing (€5.4 million World Bank, €45 million EIB). The overall implementation of this component was greatly delayed because some designs were of low quality due to the limited experience available in the country. EIB requested that all architectural designs be revised so that they complied with international standards. These revisions caused almost a one-and-a-half year delay, with the last revision completed in 2012. Secondly, delay in the procurement process was due to constant changes in the appointment of the members of the specialty technical 8 This was also due to the extremely strong leadership and enterprising approach of the Undersecretary of State in charge of the emergency department and his long and continuous tenure. 9 commissions, causing frequent revisions of technical specifications and bid evaluation reports.  Unexpected Repairs. Originally, the project planned to provide physical investment (small rehabilitation as well as medical equipment) to 183 obstetrics/neonatal units. However, at the early stages of implementation, it became clear that the original plan called for just small repairs of these units, while the buildings surroundings these units were in extremely poor conditions. Therefore, the decision was made with the EIB’s support that the project would include core building repairs, including roofing, electricity supply, etc., for those buildings. At the time of the MTR, 45 such facilities were planned, while the project invested in only 22 maternities. One of the reasons for decreasing the number of maternities selected for rehabilitation was the rapid price escalation for civil works per unit, and also the size of works. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 43. M&E Design. The PAD included a comprehensive set of project outcome and intermediate outcome indicators that reflected the project objectives. Revision of these indicators (both PDO and intermediate) was necessary to maintain the relevance of the results framework over the life of the project, particularly in light of the cancellation of activities under Component 3. Under the first restructuring in 2008, a total of three PDO indicators were dropped, two new ones were introduced, one was modified, and two remained unchanged. The intermediate outcome indicators changed substantially, with 15 indicators dropped, 12 new ones introduced, one modified, and another one slightly modified. Two more restructurings occurred in 2011, and these further refined the results framework, particularly to address indicators that could not be measured properly. One PDO indicator was dropped and slight adjustments were made in intermediate outcome indicators: two were revised, three dropped, and three new ones added. 44. M&E Implementation. From the beginning, the M&E framework was developed in very close collaboration with the PMU and other major stakeholders. The PMU was responsible for collecting and reporting on project monitoring indicators, developing a database that would allow them to submit and update the information directly. However, consistency of reporting data was a matter of concern since some definitions were not similarly understood by the various reporting entities across the country. From 2009, the situation in terms of data collection and analysis became less of an issue. M&E functions became integral to the work of eight regional coordinators, with one M&E expert devoted to this function in the central office. 45. Regular Monitoring. Progress monitoring was a key instrument of project management and ensured the timely receipt of comprehensive and regularly updated information. The PMU submitted to the Bank biannual progress reports on the status of implementation and reported on outputs and outcomes indicators. These reports were prepared in a satisfactory manner. 10 46. M&E Utilization. A disappointing aspect of the M&E system established under the PMU was that there was little evidence that the Government was using it sufficiently. This assertion is based on the fact that at certain points policy makers were not taking into account the results of program monitoring. 2.4 Safeguard and Fiduciary Compliance 47. Environmental Safeguards. The project was classified as a Category B operation. An Environment Management Framework was completed and published in 2004. No major environmental impacts were anticipated under the project given the limited amount of proposed rehabilitation and the fact that no new structures/facilities of significant size were envisaged. Existing environmental regulations in Romania provided a mandatory control and supervision of construction works with an emphasis on environmental issues. Contracts included clauses that ensured the appropriate disposal of unacceptable construction materials and waste. All tender documents covering rehabilitation required tenderers to set aside adequate sanitary facilities, including those for the appropriate disposal of wastewater and sewerage. Specific environmental management guidelines were incorporated into all civil works contracts. 48. Fiduciary Arrangement. The financial management (FM) performance over the implementation of the project was satisfactory. Significant strengths in terms of FM arrangements were represented by the substantial experience of the PMU in implementing Bank- financed projects and reliance to a large extent on the well-developed country public financial management systems, which satisfy Bank requirements. The PMU had proper FM capacity over the implementation period, which included three financial experts who undertook the project’s financial management.9 “Ciel,” the accounting software used by the PMU for the project records, had adequate security levels and was used to supply inputs for the preparation of the project’s semi-annual Interim un-audited Financial Reports (IFRs). 49. Disbursement. Disbursement of funds was undertaken using the traditional mechanism based on Statement of Expenditures (SOE). Since effectiveness until the end of 2008, the PMU has kept a Designated Account (DA) in EUR in a commercial bank acceptable to the Bank. Starting January 1, 2009, the project eligible expenditures were pre-financed 100 percent by the State Budget. The Ministry of Public Finance (MPF) then requested reimbursement from the Loan using the SOE. The Treasury issued a project code to be used for this pre-financing. The PMU followed the procedures set out in the Memorandum of Understanding (MoU) signed by the MPF and the Bank, as well as in the Loan Agreement. The PMU kept detailed records of the eligible amounts to be financed from the IBRD Loan proceeds and of the Government’s contribution. The PMU reported to the MPF the expenditures against the Loan amount in RON and EUR on a monthly basis. While the pre-financing received in general has been sufficient for the implementation needs under the IBRD Loan, the project faced particular issues in 2013 and 9 The financial coordinator bears overall responsible for the financial management aspects of the project. One part- time financial expert and one accounting expert were responsible for keeping all the accounting records and all the reporting aspects current. 11 2014 with respect to the available budget for the activities planned under the EIB financing. The IBRD disbursement level as of May 2, 2014 was 97 percent, or EUR 63.07 million withdrawn from the signed EUR 65.1 million.10 50. Internal Controls and Reporting. An adequate internal controls framework was instituted for the project, including regular reconciliation of bank accounts, and adequate segregation of duties. The walkthrough reviews carried out during the regular FM on-site reviews confirmed that the internal controls applied while preparing the withdrawal applications (WA) were adequate; the required supporting documentation was in place, properly checked and authorized. The PMU used a comprehensive set of accounting policies and internal control procedures in accordance with the Romanian legislation and the project financial management manual. Semi-annual IFRs were generally produced on time according to satisfactory standards in form and substance of financial information. The latest IFRs submitted to the Bank covered the period of July 1, 2013 to January 31, 2014 and were found acceptable. 51. External Audits. Since project effectiveness in June 2005, the annual Project Financial Statements for the IBRD and EIB funding were audited by eligible independent auditors in accordance with terms of reference acceptable to the Bank. Auditors issued independent audit reports with a clean (unmodified) opinion alongside with management letters mentioning some minor internal control issues, which were adequately addressed by the PMU. The latest audit report covered 2012 and was submitted on time. The final audit required for the project will cover 2013 and any transactions incurred during the grace period ending April 30, 2014. There is a multi-annual contract in place with an acceptable independent audit firm. The report will be submitted to the Bank no later than June 30, 2014. 52. Procurement issues. The overall procurement process faced certain difficulties and was quite lengthy, especially due to the cumbersome and long-term agreement process required by MoH for the approval of the technical specifications for medical equipment and civil works and the bid evaluation reports. In general, prior review of the procurement packages and procedures conducted under the project, as well as post-review of randomly selected contracts, showed satisfactory quality of the documents and compliance with the applicable World Bank Guidelines and established rules and procedures. The procurement processes were conducted in accordance with the provisions of the Loan Agreement and of the agreed Procurement Plan, which was regularly updated. All Bank comments and recommendations were taken into account and incorporated. No mis-procurement was reported. However, several re-bidding procedures took place. The PMU procurement specialists were qualified and had prior experience implementing Bank financed projects. 53. This project had the largest amount of complaints in the entire country portfolio. About 395 documents are logged into the World Bank’s project filing system due to many factors, such as (a) strong competition among firms and suppliers of sophisticated and complex medical equipment; (b) existence of several groups of beneficiaries with specific preferences for the type of equipment to be supplied; and (c) challenges in ensuring that the required functionalities of 10 This excludes the amount of the latest withdrawal applications received by the Bank as of May 2, 2014, which are pending processing for reimbursement for an amount of a little over EUR 1.4 million. 12 the equipment did not compromise competition. It was also reported that complaints resulted in some leakage of information during the tenders’ evaluation stage. 54. Due to complexity and very competitive nature of the medical equipment industry, the following issues emerged during project implementation:  Organization and the role of the “expert groups” working on the technical specifications (TS) and evaluation of bids as well as constant staff turnover created considerable delays in the procurement process.  There were very different opinions on the technical quality of the equipment, with some trying to ensure that only reputable, high quality equipment from leading manufacturers was purchased rather than equally qualified equipment from less well known manufacturers at a lower price.  At the bidding stage, complaints related to TS and the length of time to receive clarifications on the TS led to extensions of the submission deadline for bids.  At contract award stage, there were complaints related to disqualification of bids on technical grounds, including that (a) the TS were biased towards a particular manufacturer; (b) bids were unfairly rejected even though the equipment was technically better and priced lower than those offered by the firms’ competitors; and (c) the bids of the firms awarded the contracts did not meet the TS. A few complaints were referred to INT.  There were substantial gaps between the preparation of the TS and the actual tendering process due to very intense discussions and the long process of awarding contracts, which affected the original cost estimates for the equipment.  The long evaluation period of the bids and the lengthy internal approval process also had implications on the validity extensions of rather frequent bids. Also the higher price of equipment, in the case of fixed priced contracts, was due to the correction factor being applied because of the lengthy evaluation of bids.  The scope of delivery or the scope of work was modified during contract implementation due to requests by MoH. On several occasions, MoH requested larger quantities of medical equipment for more hospitals or for additional quantities to be purchased at the same unit rates. MoH also took advantage of the provision in the bidding documents allowing an increase of 15 percent of the contract award for additional quantities of goods. 55. The following issues regarding civil works emerged during project implementation:  Revisions of the architectural designs requested by EIB for the rehabilitation of maternities were considerably long and complicated.  A large number of amendments to ongoing civil works contracts and cost increases to reflect revised architectural designs (in accordance with EIB’s comments). In some instances, revisions to the design increased the civil works contract by more than 15 percent of the contract amount, which therefore required a rebidding process, resulting in increased estimated costs and delays. 13  Cost overruns were foreseen even during the planning stage as the team identified many issues in the quality of the architectural design, causing last minute changes in the scope of the civil works and cost increases. 56. Despite cost overruns in certain areas, it is important to mention that other ICBs, e.g., EMS packages, resulted in cost savings estimated at EUR 5 million. 2.5 Post-completion Operation/Next Phase 57. A new Improving Health System Quality and Efficiency Project was approved by the World Bank Board of Executive Directors on May 28, 2014. This new project focuses on improving health system quality and efficiency. More specifically, the new investment operation for the health sector aims to improve the access, quality, and efficiency of the public health services by supporting the rationalization of the health care service delivery network. The new project will finance the implementation of short- and medium-term interventions to support the health sector reform program (2014–2020), help the GoR access resources from the EU and other development partners, and support the efficient use of these resources. The project will support the first phase of comprehensive medium-term and long-term health sector reforms by focusing on the following three main areas/components: (1) hospital network rationalization; (2) ambulatory care strengthening; and (3) health sector governance and stewardship improvement. 58. Continuous policy dialogue on health. Apart from the APL 2, the Bank has had a continuous policy dialogue on health sector reforms in Romania over the last four years through the development policy loan (DPL) series and the ongoing DPL Deferred Drawdown Option (DPL-DDO), which contain significant health related measures. The DPL-DDO, in the amount of EUR 1 billion approved in 2012, supports a comprehensive three-year reform program which includes improvements in the fiscal sustainability of the health sector (e.g., e-prescription for compensated drugs, increase of funds allocated for health prevention and promotion programs, and reduction of expenditures for several drugs for which cost-benefit analysis proved not to be efficient). 59. The Bank was also actively involved in health policy dialogue as part of the International Monetary Fund (IMF) and European Commission (EC) programs. The recent precautionary type arrangements with the IMF and EC include new structural reforms in the health sector, which aim to increase the efficiency of health services delivery, improve their quality, and deliver better health outcomes in a financially sustainable manner. The definition of the basic package of medical services, revision of reimbursable drugs lists, implementation of Health Technology Assessments, introduction of supplementary private health insurance, and improved revenue collection for the public health insurance system are just some of the areas covered. 60. Maintenance. Maintenance costs and their implications for the recurring health budget have been addressed, especially for those related to the emergency health program, and now fall under a separate line item in MoH’s budget, ensuring autonomy and adequate level of financial resources. 14 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 61. The relevance of the objectives, design, and implementation arrangements are considered substantial. Supporting the implementation of the health sector reform program by providing more accessible and higher quality services for maternal and child health and emergency medical care was a fundamental development priority of the GoR at the time of project approval and restructurings. The PDO was fully relevant to the priorities of the national program (Program of Romanian Government, 2001-2004) as well as the National Public Health Strategy (2004). This project was included in the FY02-04 CAS, which focused on improving delivery of health services. Also, this project was identified by the CAS as one of the targeted interventions to reduce inequity of access to basic health care services across regions and by vulnerable populations. 62. The PDOs remain highly relevant because (a) maternities, neonatal and emergency health care centers continue to be the main health service delivery points; (b) quality and accessibility of services continues to be an issue in many places where civil works for the rehabilitation of maternity centers financed by the EIB have not yet been completed. 63. With this project support, access to health care services has been improved, especially in rural areas. The project led to successful and sustainable achievement in terms of access and quality of care in three major areas of the health care delivery – maternal, child and emergency services. 64. The maternal mortality rate in Romania has declined noticeably since the 1990s. However, the current maternal mortality rate in Romania is still among the second quintile in the European Region: 53 (46-62) per 100,000 live births in 2000 down to 33 (26-44) per 100,000 live births in 2013.11 It is hard to attribute the positive trend to the project. However, to improve maternal health, barriers that limit access to quality maternal health services must be identified and addressed at all levels of the health system, and the project managed to support better infrastructure, improved access to antenatal care, improved access to skilled care during childbirth, and provision of care and support in the weeks after childbirth. Even though Romania’s infant mortality rate declined from 26.9 ‰ (per 1,000 live births) in 1990 to 13.9 ‰ in 2006 and further to 10.1 ‰ in 2009, the levels are still above the average of the EU and the Countries of Central and Eastern Europe. Most infant deaths are related to perinatal conditions and malformations (57 percent), but some are also due to diseases of the respiratory system. Maternal and neonatal mortalities are complex indicators with multi-faceted problems, however the project managed to address these issues through a holistic approach in the hospitals and maternities benefiting from project support. In addition, the entire network of 204 maternities received medical equipment and their staff was adequately trained. This enabled them provide better health-care solutions to prevent or manage complications. Also 20 reference maternities were rehabilitated to comply with best practice hospital standards. 11 http://www.who.int/gho/maternal_health/countries/rou.pdf?ua=1 15 65. A new institutional approach of maternal and neonatal care was introduced during the project lifespan. Maternal and neonatal care was provided by the integrated regional network with one centralized level, one hub/regional center, and several subordinated level two and three units. Such a regional network covers a defined geographic area, where a number of about 15,000 to 20,000 births per year are registered. The project invested in all levels of facilities. 66. Although the project focused on the acute care part of the health care system, there was some effort to strengthen the PHC component through the National Health Insurance. Antenatal care figures are showing positive trends for the period 2008-2012, with at least one visit in 93.5 percent of the cases and four visits in 76 percent of the cases. 12 Vaccination rates (DPT3) decreased from 99 percent in 2000 to 89 percent in 2012.13 67. The project has had substantial impact on emergency services and can be considered an international good practice in this regard (detailed description of achievements in Annex 2). 3.2 Achievement of Project Development Objectives 68. Achievement of Project Development Objectives is derived according to the methodology of the ICR guidelines (Appendix B) for evaluating the achievements of the PDOs against the original and revised PDOs and related key performance indicators.14 69. The original PDOs were to provide more accessible services of increased quality and with improved health outcomes for those requiring maternity and newborn care, and emergency medical care , and rural primary health care. Achievement of the original PDO indicators is summarized in Tables 1 and 2. Table 1. Achievements of Original PDO Indicators as Stated in the PAD Original PDO indicator Actual value achieved 100% of maternal deaths Not achieved. This indicator was tracking all maternal death cases formally documented/ detected and investigated by the Specialty Commission, which investigated (100%). only sporadically investigated cases. From the baseline of 51.4% in 2004, this indicator fell to 37%. This indicator was dropped in 2008 during restructuring. Neonatal and post-neonatal Over achieved. Neonatal deaths decreased nationwide from 2004 deaths and death rate (15% to 2012 by 67.5%. Original target was a 25% decrease from decrease from 8.4/1000 for baseline. neonatal and 15% decrease Post-neonatal deaths decreased nationwide from 2004 to 2012 by from 8.9/1000). 60%. Percentage of deliveries Impossible to measure. Categorization of the maternity wards where birth-weight is less was only introduced in 2006. The state statistical system was not than 2500 grams (20% yet capturing the breakdown of this indicator by level of facilities. decrease from 8.8%). Also, not all maternities were dealing with low-birth weights 12 UNICEF, http://www.unicef.org/infobycountry/romania_statistics.html. 13 WHO vaccine preventable diseases: monitoring system. 2013 global summary. 14 Operations Policy and Country Service (OPCS), Implementation Completion and Results Report: Guidelines. Washington, DC: World Bank, 2006, revised 2011. 16 Original PDO indicator Actual value achieved infants; data from CRED-financed evaluation were temporarily used, but it was insufficient for measurements. Hence, this indicator was dropped in 2008 during restructuring. Utilization rates from Impossible to measure. At the MTR 2007, this indicator was primary and emergency care reworded as “Increase in utilization rates (from those who were stratified based on residence ill/had an accident/suffer from a chronic illness or have a and income status (20% handicap) for primary care (family doctor/dispensary) in increase from baseline intervention rural communities.” Following the project’s survey). restructuring in 2008, the rural health centers were removed from the project and the PDOs were also revised. This indicator (5) was replaced by a different one (7): “Prevalence of chronic diseases in target intervention rural communities (If investment is made in rural areas).” This indicator was also dropped later because no investments were made in rural communities. The indicator was replaced by the more relevant: “Development of a primary care rural strategy.” Percentage of deaths within Partially achieved. 24-hour death rate reduced from 2007 to 2012 48 hours and ER discharge of by 28%. Although sample size is too small for the results to be patients with major trauma or extrapolated for the whole emergency system, the project-related cardiac emergencies arriving target has been fully met. alive at a hospital emergency department (20% decrease from baseline). 70. Of the five original PDO indicators, two were impossible to measure, one was not achieved, one was partially achieved, and one was over-achieved. Table 2. Status of Achievement against Original Project Indicators Status 5 PDO indicators % of total Overachieved 1 20% Achieved 0 0% Not achieved 1 20% Substantially achieved 0 0% Partially achieved 1 20% Impossible to measure 2 40% 71. The PDOs and key performance indicators were changed during restructuring in 2008. The results framework was changed twice in 2011. The revisions were as follows:  PDOs were revised in 2008 to reflect the new intended outcomes of the revised third component of the project. The revised PDOs were (i) to provide more accessible services of increased quality and with improved health outcomes for those requiring maternity and newborn care and emergency medical care; and (ii) to support the development of a primary health care strategy.  Revised results framework included the following PDO indicators: (i) Maternal Mortality (MM) and Maternal Mortality Rate (MMR) by hospital, county, and region (new indicator); (ii) neonatal deaths (ND) and rate (NDR) by hospital, county, and region (same as before); (iii) post-neonatal deaths (PND) and rate (PNR) by hospital, county, and region (same as before); (iv) 24-hour deaths rate among patients treated in the ER, then admitted to ICU in that hospital (new indicator); and (v) prevalence of chronic diseases in target intervention 17 rural communities (new indicator). This last indicator was subsequently dropped because no investment was made in rural communities. The indicator was replaced by a more relevant intermediate outcome indicator: “development of a primary care rural strategy” (Restructuring Paper, October 24, 2008 and Amendment to the Loan Agreement dated November 2008).  In 2011, the indicator “prevalence of chronic diseases in target intervention rural communities” was dropped because it was intended to measure the establishment of multipurpose health centers, which was cancelled as part of the 2008 restructuring.  Later in 2011, the baseline and frequency for the fourth indicator “24-hour death rate amongst patients treated in the ER who were admitted to ICU in that hospital,” was changed. These revisions were reflected in the restructuring paper of December 12, 2011. Achievement of revised PDO indicators is summarized in Tables 3 and 4. Table 3. Achievements of Revised PDO Indicators Revised PDO Actual value achieved indicators Reduce Maternal Over achieved. Maternal mortality decreased nationwide by 85%. Mortality and Rate MMR decreased from 0.24 maternal deaths per 1,000 births in 2004 to 0.14 maternal deaths per 1,000 births in 2013 (intended target was 0.19). Decrease in Over achieved. Neonatal deaths decreased nationwide by 68% from 2004 to Neonatal Deaths and 2013. Nationwide neonatal mortality dropped from 2,068 deaths in 2004 to Rate 673 deaths in 2013 (target to be achieved was 1,551). The neonatal mortality rate decreased by nearly half (47%), from 9.6 deaths per 1,000 births in 2004 to 4.5 maternal deaths per 1,000 births in 2013 (target to be achieved was 7.2). Decrease in Post Over achieved. Post neonatal mortality decreased nationwide by 60% from Neonatal Deaths and 2004 to 2013. Post neonatal deaths decreased from 1,573 in 2004 to 634 in Rate 2013 (target to be achieved was 634). Rates decreased nationwide by 58% over this period, exceeding the target by 1.3 per 1,000. Decrease in 24-hour Over achieved. The 24 hour death rate was reduced from 5.78% (baseline in death rate among 2007) to 4.16% in 2013, a decrease of 28%. patients treated in the ER, then admitted to ICU Primary health care Achieved. The Strategy 2012-2020 and related Action Plan were formally strategy approved approved by the MoH on February 27, 2012 and were forwarded to the Health Care and Public Policies Directorate of the MoH to be adopted through Government Decision in March 2012. According to the MoH Report of March 15, 2013 on the implementation status of the Action Plan of the National Reform Programs 2011-2013, the Strategy was revised to observe the provisions of Government Decision No. 870/2006. Expected to be approved within the framework of the Health Sector Reform Strategy for the next period of the EU financing exercise (2014-2020). Table 4. Status of Achievement against Revised PDO Indicators Status 5 PDO indicators % of total Overachieved 4 90% Achieved 1 10% Not achieved 0 0% Substantially achieved 0 0% Partially achieved 0 0% 18 Status 5 PDO indicators % of total Impossible to measure 0 0% 72. The ICR team applied the methodology of the ICR guidelines (Appendix B) in evaluating the achievement of the PDOs against the original and revised PDOs and related key performance indicators. Table 5 shows the combined achievement of PDOs and the final rating (rounded and weighted) of achievement of outcomes, which comes out as Moderately Satisfactory. Table 5. Combined Achievement of the PDOs (Original and Revised) Rating/scale Against original key Against revised key performance performance indicators indicators Rating Unsatisfactory Highly Satisfactory Rating value 2 6 Amount disbursed (in EUR) 29.34 out of 65.10 32.16 out of 65.10 Weight (% disbursed before/after key 45% 49% performance indicator change) Weighted value (rating by disbursement) 2*0.45=0.90 6*0.49=2.94 Final rating (rounded and weighted) 3.84 = Moderately Satisfactory 3.3 Efficiency 73. The ICR team considers efficiency as substantial. The objectives of the second Romania Health Reform Project (APL 2) were to provide more accessible health services of increased quality for those requiring maternity and newborn care, emergency medical care, and rural primary health care to achieve improved health outcomes. The project was restructured in 2008 and the revised project components included rehabilitation of maternity and neonatal care units; development of district and local emergency hospitals; and the development of National Health Accounts and the project management component. 74. The project outcomes exceeded the targets for Components 1 and 2. As a result, the economic internal rate of return was 28 percent and the benefit/cost ratio was 1.65. The sensitivity analysis also shows that even under conservative assumptions, the rate of return was reasonable, and as high as 15 percent. These results are in agreement with the expected results presented in the Project Appraisal Document. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 75. The project’s overall outcome is considered as Moderately Satisfactory based on its substantial relevance, efficacy, and efficiency and in line with the methodology above for deriving the outcome rating for projects with formally revised objectives. The relevance of project objectives and its overall sound design is considered substantial. Meanwhile, due to insufficient data, the evaluation of efficiency is carried out based on a number of assumptions. However, available data clearly show that the project, especially its first and second components, have generated considerable benefits and is expected to yield reasonable returns even over the coming years. The sensitivity analysis shows that even under conservative assumptions of significant reduction in project benefits, the rates of return were about 15 percent. These results 19 are close to the expected measures of economic efficiency presented in the Project Appraisal Document, which confirms that the project had achieved the expected outputs and outcomes. 3.5 Overarching Themes, Other Outcomes, and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 76. This project was one of the targeted poverty interventions identified in the CAS to reduce inequity of access to basic social services across regions and by vulnerable groups. 77. A full social assessment of the project was completed at the initial stage of project implementation. Four of the top five reasons for the population’s dissatisfaction with medical services were likely to influence the poor more than the non-poor. Three of the five leading reasons have to do with cost, e.g., the cost of drugs and medical services and informal payments. Another reason listed in the top five reasons was the distance to the nearest clinic, which is an issue for rural residents, and especially poor households. All these findings provided support for the project interventions with a focus on the improvement of services in rural areas which suffer from high poverty. 78. The social impact of the project and overall impact in terms of improvements in health outcomes were larger in rural and poorer localities. Household survey data suggests that health care utilization varies significantly by poverty status and not by urban/rural residence. More detailed targeted mechanisms for the project interventions were used, such as the Romania Poverty Map, developed as part of the 2003 World Bank supported poverty assessment. This mapping exercise was used in collaboration with the Romania Anti-Poverty and Social Inclusion Commission to target project interventions, especially for the PHC component. However, the use of these mechanisms never evolved in the way they were originally planned. 79. It should be noted that there were no specific gender aspects highlighted in the project. Nevertheless, the project addressed gender imbalances by improving access to quality maternal health care services, especially in rural and hard-to-reach areas. Indeed, the indicators related to maternal health care have a strong gender aspect. Ultimately, the majority of beneficiaries of the project were women, including those working as medical personnel, including rural doctors and nurses, etc. (b) Institutional Change/Strengthening 80. The project had a substantial institutional development impact, at both local and national levels. The project significantly increased the capacity of MoH to plan and manage the health system. Specific attention was focused on the rationalization of health services, building HTA capacity and other elements. Moreover, the project assisted in advancing the overall health sector reform achieved in APL 1 and further strengthened national capacity to manage two of the main areas of the health system: Emergency and Maternal and Child Services, i.e., obstetrics and neonatology. Several fundamental documents also were written and/or adopted by the GoR, including the PHC strategy that is expected to be approved within the framework of the Health 20 Sector Reform Strategy for the next programming period of the EU financing exercise 2014- 2020. 81. The project also strengthened local government capacity to implement health reforms. All decisions related to infrastructure and human resource capacity development were made with the enthusiastic involvement of local government bodies. (c) Other Unintended Outcomes and Impacts (positive or negative) 82. Close coordination and synergies with activities financed by various donors have resulted in better outcomes for the health sector. EIB, EU, DFID, USAID, SDC, have all supported efforts to improve maternal and child health and emergency services through civil works, equipment, and capacity development. Specialized UN agencies, like WHO and UNICEF, have provided general technical assistance in all these mentioned areas. 83. A modern, integrated ambulance dispatch system was established by upgrading the existing communication systems of district ambulance service dispatches and linking them to Central Emergency Call Centers, which patients reach by dialing the new single emergency number 112. The new system also enables voice and data communication between dispatchers and ambulances sub-stations. This upgrade has resulted in a nation-wide communication system that encompasses all parts of emergency services, including ambulance, police, defense, and fire- brigades. The system is considered a best practice and many countries are turning to Romania to learn from its example, e.g., Croatia, Bulgaria, and Moldova. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 84. Client Satisfaction Survey for maternities is summarized in Annex 5. 4. Assessment of Risk to Development Outcome Rating: Negligible 85. The risk to development outcome is rated Negligible. The Government and its institutions demonstrated strong ownership and commitment in maintaining the achieved gains. In addition, the new health project will continue to provide financial support for the various areas addressed by this project. 5. Assessment of Bank and Borrower Performance 86. Team leadership changed four times over the lifespan of the project, causing some adjustments in the way the Bank guided implementation. 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 87. This evaluation rates the Quality at Entry as Moderately Satisfactory. As described in the earlier sections, it is the view of the ICR team that (i) the project objectives and scope were 21 appropriate for the stage of development of the sector; (ii) the project built on the experiences of the APL 1 and other health sector reform projects in the ECA region; (iii) the project addressed the Romanian Government’s program goals for health outcome improvements; and (iv) it supported key elements of the CAS to improve the delivery of health services. Nevertheless, this evaluation finds that while selected objectives were appropriate for the status of the reforms, a less ambitious scope of design would have simplified implementation and better taken into account certain legal framework drawbacks (e.g., state budgetary limitations for allocations and strict ceilings). The ambitious project scope resulted in multiple project extensions. In addition, as was born out during implementation, some of the original PDO targets were not easy to measure and required revision.. (b) Quality of Supervision Rating: Satisfactory 88. The Quality of Supervision is rated Satisfactory. Supervision of the project was intense, with frequent, technically adept visits from the project’s onset to monitor progress, ensuring consistency of policy dialogue, and providing hands-on support to the implementing agency. The fourth and last Task Team Leader (TTL) led the project for almost three years. The project benefitted immensely from having the TTL and seasoned health consultants located in the country. Such an arrangement worked well, with professional staff able to engage and interact with the Bank’s team and benefit from its high level of responsiveness and commitment. In addition, it afforded an excellent opportunity for the Bank team to supervise project implementation on an ongoing basis, taking appropriate measures as needed. 89. The Bank team made all possible effort to collaborate with international donors and implementing agencies involved, particularly the EIB and SDC. The World Bank team met international partners and exchanged regular status updates during implementation visits. Partners discussed coordination mechanisms, how to effectively share information about the project’s achievements and challenges, and recent developments in the health sector. This environment allowed the project team and GoR to reach consensus on the cancellation of certain activities and addressing new challenges, jointly determining the most efficient use of loan funds. 90. The Bank team demonstrated appropriate levels of responsiveness by its timely provision of “no objections,” and informed consent for funding of interventions not originally planned. The Bank quickly responded to GoR’s needs, demonstrating flexibility in modifying implementation plans and project design. During supervision, the Bank demonstrated the flexibility to adjust the project to the changing needs and reality of the sector. One example relates to the unfinished activities of the Cantacuzino Institute, which were transferred to the project from the closed Avian Influenza Control and Human Pandemic Preparedness and Response project, 15 15 In 2011, following the fourth amendment to the loan agreement, the rehabilitation of the vaccine production and ampoule filling and sealing area at the Cantacuzino Institute (Bucharest) was transferred from the Avian Influenza Control and Human Pandemic Preparedness and Response Project.15 The APL 2 itself managed to finish the civil works for the influenza vaccine production unit, aseptic ampoule sealing and filling station in due time (2012). More specifically, the works for rehabilitation of the influenza vaccine production unit started in January 2012 and were completed on July 15, 2012. The rehabilitation works for the filling and sealing station started in October 2011 and were completed in October 2012. 22 demonstrating the World Bank’s flexibility to accommodate an investment that otherwise would have remained unfinished. (b) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 91. The moderately satisfactory rating for Quality at Entry and the satisfactory rating for Quality of Supervision averages to a rating of Moderately Satisfactory for overall Bank performance. A productive partnership with the Borrower led to smooth technical and institutional collaboration during preparation and implementation stages. A positive working environment was fostered with the client, which contributed to achieving, and in some cases exceeding, the majority of project objectives. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 92. Government performance during project preparation and implementation is rated Satisfactory. The performance of the Government varied between the early and later stages of the project. There were certain shortcomings in the Government’s performance in the early stages of the project, as demonstrated by MoH’s micro-management and some insufficiencies in counterpart funding. At later stages of project implementation, the Government demonstrated its full commitment to the project objectives by accelerating its implementation and initiating the new investment operation. However, less satisfactory aspects of Government performance related to (i) budgetary restrictions and complex administrative procedures, which amongst others affected implementation; (ii) frequent changes in the leadership of MoH, which impacted on some of the project directions and caused delays; and (iii) the frequent change and difficulties to mobilize (even for the PMU) the “expert groups,” which were providing input to technical specifications of bidding documents and the bid evaluation process. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 93. Implementing agency performance is rated Satisfactory. The PMU was established in 1992 by the GoR to implement the first health project financed by an IBRD loan. The PMU was also responsible for implementation of the Global Fund projects from 2003 until 2007. Effective 2007, the PMU became solely responsible for the APL 2. The unit is a subordinated structure under the MoH, which has strong oversight over staffing issues and project implementation. The PMU was adequately staffed with component coordinators and a fiduciary team. A total of 10 full-time staff members were in charge of project implementation, though turnover was high over the life of the project. The director of the PMU changed three times during the project’s life and staff salaries decreased twice during 2010, impairing the likelihood of attracting and retaining qualified staff. However, these issues were gradually resolved in a satisfactory manner through a 23 joint initiative of the World Bank and the MoH. Appropriate staffing, vigorous and intensive training and accumulated experience resulted in a significant increase in the PMU’s capacity for project implementation. 94. The PMU participated actively in ICR discussions and prepared a consolidated final report on the project from all the progress reports prepared over time (Annex 7). (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory 95. The ICR team rates the overall Borrower performance as Satisfactory in line with ratings above. 6. Lessons Learned 96. The key lessons learned and/or reinforced during project implementation are as follows:  Keep project design flexible, responsive, and simple. Given the scope and pace of health sector reform, the two-phased APL was an appropriate approach for ensuring achievement of the project development objectives and long-term sustainability. However, duration and scope of each phase could have been better planned, and a simpler design adopted, including simpler, . more straightforward procurement.  Strong governmental ownership and close coordination with development partners are critical. GoR’s commitment, as well as a clear vision and a reform agenda, were key to achieving the project objectives. Close coordination and collaboration with EIB, SDC, and other partners in supporting implementation aspects of the program helped keep key activities on track despite the delays. Sharing the same implementation unit for EIB and IBRD funding ensured harmonization of interventions and efficient use of funds. However it would have been more effective to formalize the arrangement between the World Bank and EIB in regard to responsibilities related to rehabilitation of of maternal health centers.  Staff continuity is critical. Continuity of staff, particularly on the government and PMU sides, is a critical element of project success or failure. Project preparation was carried out with one government, however implementation fell to a new administration, which led to substantial start-up delays. Thankfully, despite major changes introduced in the management of the PMU, the core team of experts remained practically unchanged, which helped with continuity.  Proactive and timely supervision by the Bank’s team is also crucial for a successful project. The project greatly benefitted from the continuous technical support provided by the local World Bank team, backed-up by strong and timely policy dialogue provided by the headquarters-based health specialists. In-country presence of the TTL and his close collaboration with the frequently changing senior decision makers from the MoH allowed quick resumption of the policy dialogue after staff changes and early identification of bottlenecks that could have further hampered project implementation. 24 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 97. The Government’s contribution to the ICR is included in Annex 7. The Ministry of Public Finance sent comments on the draft ICR by letter of June 2, 2014, also included in Annex 7. (b) Co-financiers. No comments from EIB were provided. (c) Other partners and stakeholders. Not applicable. 25 Annex 1. Project Costs and Financing (a1) Project Cost by Component (in EUR Million) Actual/Latest Appraisal Estimate Percentage of Components Estimate (EUR (EUR millions) Appraisal millions) 1. Maternal and Neonatal Care 104.87 108.75 103.7% 2. Emergency Medical Services 47.23 42.30 89.6% 3. Primary Health Care 11.41 5.64 49.4% 4. National Health Accounts and 40.4% Planning 0.52 0.21 5. Project Management 3.85 4.46 115.9% Total Project Costs 167.88 161.36 96.1% (a2) Project Cost by Component (in US$ Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (US$ millions) Appraisal (US$ millions) 1. Maternal and Neonatal Care 128.99 133.76 103.7% 2. Emergency Medical Services 58.09 52.03 89.6% 3. Primary Health Care 14.03 6.94 49.4% 4. National Health Accounts and Planning 0.64 0.26 40.4% 5. Project Management 4.74 5.48 115.9% Total Project Costs 206.49 198.47 96.1% 1 EUR = US$1.23 26 (b1) Financing (in EUR Million) Appraisal Actual/Latest Type of Co- Estimate Estimate Percentage of Source of Funds financing (EUR (EUR Appraisal millions) millions) Borrower 36.38 36.15 99.4% EIB: European Investment Bank 66.40 61.07 92.9% International Bank for Reconstruction 65.10 64.14 98.5% and Development Total 167.88 161.36 96.1% (b2) Financing (in US$ Million equivalent) Appraisal Actual/Latest Type of Co- Percentage of Source of Funds Estimate Estimate financing Appraisal (USD millions) (USD millions) Borrower 44.77 44.46 99.3% EIB: European Investment Bank 81.72 75.12 91.9% International Bank for Reconstruction 80.00 78.89 98.6% and Development Total 206.49 198.47 96.1% 1 EUR = US$1.23 27 Annex 2. Outputs by Component Component I: Maternal and Child Services The project was supposed to initially finance minor rehabilitation works for all maternities. Later on the MoH decided to limit the number of sites to 22 major hospital units to bring them up to international standards. As a result, the EIB decided that a rigorous approach should be taken in the revision of the detailed architectural designs, which created implementation delays for civil works. The project supported the development of feasibility studies, technical analyses, and architectural designs for the selected 22 hospital units (maternities). However, in the end only 20 maternities were rehabilitated, works for other two ones (Craiova and Brasov) being cancelled due to big structural problems in Craiova and uncertain property status in Brasov. Although the rehabilitation of these maternities was financed through EIB funds (total cost € 45 million), around € 5.4 million was also planned under IBRD funding. A total of 204 maternities were provided with various types of equipment, including newborn incubators, monitors, ECGs, etc. Overall 10,500 units were purchased and delivered to the maternities for a total cost of €19, 6 million from the loan proceeds. A wide range of technical assistance and trainings were provided throughout project implementation. Several in-depth documents were developed and introduced, including an Operational Manual on Maternal and Neonatal Referral System, Quality Assurance System, etc. Around 830 persons were trained, including health care managers and administrators, as well as health care professionals (see table below). Training for Ob & Gyn doctors and nurses on Participants (doctors Training of Trainers (ToT) new guidelines and nurses) Bucharest 96 30 Regions 641 64 Total 737 94 Component II: Emergency Health Care Services Integrated emergency health care supported by the project is considered a reference and best practice in the region. Several countries are intending to adopt and implement the Romanian model with support from World Bank financed projects. The project supported major rehabilitation of the emergency department at the 1,300 bed Tirgu Mures county emergency hospital, which delivers services to a population of around 2,250,000, most of whom are poor and from the Roma minority. Another 16 Emergency Departments were also renovated. Various types of medical equipment were delivered to the Emergency Departments of 63 hospitals throughout the country, including monitors, ECG machines, artificial pulmonary ventilators, CT scanners, X-ray equipment, intra-hospital transportation, tools for resuscitation, airway management and immobilization, etc. In addition, monitors were purchased for ICUs and cardiology departments. The hospitals that received this equipment represent the major trauma and emergency receiving centers in the country. Moreover, a total of 190 small municipal and town hospital emergency rooms were provided with basic equipment. These emergency centers are now better able to provide emergency care for patients in critical condition and stabilize their health condition before being referred to the intensive care unit. The introduction of the telemedicine system for imaging data transmission for the central and south regions was extended to further cover the north-east region. Radio equipment, data transmission 28 equipment, and GPS for ambulances were funded under the IBRD loan. The MoH was able to create an operational communication network covering the entire country. This unified data transmission with the same equipment available at all levels (regional and central). A total of 40 regional dispatch centers were created, with six sub-stations per region, and around 1,451 type A, B, and C ambulances were equipped with necessary communication equipment. The project had initially aimed to support communication equipment for 400 ambulances; however, it has successfully exceeded this target by far. The project purchased simulators for the training of medical personnel working at emergency services to strengthen their technical capacity. Component III: Primary Health Care and Rural Medical Services Initially this component focused on improving the accessibility and quality of basic medical services in rural and small urban areas, but was substantially revised during the restructuring carried out in 2008. The majority of the activities were dropped, with the exception of the development of the PHC strategy. Nevertheless, before cancellation of the activities, this component developed a new concept based on the existing system of rural health centers. These centers were intended to deliver diversified services by family doctors, ambulatory care specialists, community nurses, home care and social services in remote rural areas and small towns. Sub-loans for family doctors were supposed to finance specific investments in medical equipment, consumables, transportation means, and IT equipment to help the doctors improve access to a wider range of health services. It was expected that this would bring about an improvement in the quality of the services provided in selected locations, including but not limited to rural areas. There were two specific reasons for the cancellation: (i) changes in the legislation of health sector financing no longer allowed the MoH to finance investments in PHC infrastructure; and (ii) developments in the capital market which gave Group Practices better access to commercial loans. One of the tangible outcomes of this component is the development of the PHC strategy, which redefined the PHC and outlined the ways to increase access to medical services in rural areas through physical rehabilitation of PHCs and through offering incentives for young family doctors to keep them as part of the medical system in rural and underserved areas. In 2011, following the fourth amendment to the loan agreement, the rehabilitation of the vaccine production and ampoule filling and sealing area at the Cantacuzino Institute (Bucharest) was transferred from the Avian Influenza Control and Human Pandemic Preparedness and Response Project.16 The APL 2 finished the civil works for the influenza vaccine production unit; the aseptic ampoule sealing and filling station will be completed in due time (2012). 16 Avian Influenza Control and Human Pandemic Preparedness and Response Project (IBRD Loan No. 4839 RO) was closed on December 31, 2010. One of the unfinished activities was taken on by the APL 2 based on the request of the MoH and WB letter (as of January 25, 2011). 29 Component IV: National Health Accounts and Planning This component supported the development of National Health Accounts (NHA) and the preparation of proposals for rationalization and service development projects. It was a first attempt by the health sector to develop and internalize NHA. All the required support, including international and local, was provided to the stakeholders and Technical Working Group (TWG) 17 to adapt internationally validated methodology (WHO/OECD), propose regulation changes regarding the reporting of financial flows in the health sector, conduct analysis of existing financial information and surveys, conduct additional surveys, train staff, and prepare, publish, and disseminate reports. The project financed the participation of the designated staff in various trainings, seminars and workshops, investments in IT equipment and software, surveys and studies, local and foreign TA. The National Institute for Statistics (NIS) is responsible for the NHA and is supposed to develop the NHA for the purpose of international comparisons. Although the NIS continues to collect data from various institutions and compiles the NHA Report annually, these reports do not use the templates and methodology developed under the project since they are lacking much information (mainly from the private sector), which are outdated by the time the NHA is released. As a result, the instrument is not adequately used by various stakeholders, especially by policymakers for decision-making. The second activity under this component included support to the MoH, District Public Health Authorities, health care institutions or local authorities with resources for the preparation of projects to be submitted to financing institutions and donors. These projects would aim to implement the rationalization strategy and service plans prepared in APL 1 and make use of the capacity for planning developed at the local level. This activity was dropped in 2009, mainly because the health sector did not have a sector operational program for EU Funds. Component V: Project Management This component was designed to provide institutional support to the MoH to implement the project through a Project Management Unit (PMU). The PMU had 10 full time staff with component coordinators and a fiduciary team. The following technical assistances (TA) were provided, namely: (a) Information, Education and Communication Campaigns on the Health Sector Reforms – the TA successfully collaborated with the ministry and other stakeholders for about six months in total. As a result, new media plans, TV and radio spots based on the newly developed campaign messages, slogans, and two interactive websites were delivered. (b) The content and listing processes for the Romanian basic package of health services and technologies was developed. This TA helped the Romanian authorities deliver a focused set of objectives, including the development of a non-exhaustive “negative list” based on explicit rationale and with reference to practices in other countries, including the UK and Australia. This three-month assignment also covered hands-on support and training for the development of a Health Technology Assessment (HTA) in selected areas. 17 TWG was established for NHA development in the early stages of project implementation, including representatives of the MoH, MoF, National Health Insurance House, and National Institute for Statistics. The project financed training, seminars and workshops, investments in IT equipment and software, surveys and studies, local and foreign TA. 30 (c) TA training was provided on HTA. The overall objective of this three-month assignment was to help the MoH improve the management of the funds, in order to become more transparent, based on a clear process to ensure cost-effectiveness. This TA also covered the training for professionals concerning the HTA to ensure their adherence to best international practices. A total of 12 persons involved in the HTA from the MoH and other relevant institutions were trained (National Health Insurance House, National Agency of Medicines, and medical universities). (d) TA for impact study of the new health regulations was planned but later on cancelled due to several revisions of the ToR and certain policy direction changes in the sector. (e) The following TA were carried out in support of the (a) development of a Strategy for Primary Health Care in Underserved Rural Areas and the Related Action Plan; (b) development of National Health Accounts System in Romania; (c) development of an Operational Manual on Maternal and Neonatal Referral System; (d) design and institutionalization of a Quality Assurance system within the maternities; and (e) Training Program to ensure implementation of best international practices for the Romanian Neonatology and Obstetrics/Gynecology system. In total 60 contracts (ICB procedures) were entered into and financed out of the proceeds of the IBRD loan, for medical equipment, communication and GPS equipment, emergency telemedicine system, local and regional paging systems, and vaccine filling and sealing lines. There were six contracts for civil works (three directly IBRD financed and three financed by IBRD and EIB loans) procured following NCB procedures, 21 small contracts for civil works (total budget: EUR 0.48 million) and seven small contracts for first aid and paramedics manuals, hospital beds, surveillance, telephone and communications equipment procured following shopping procedures (total budget: EUR 0.36 million). There were 12 ICBs (42 contracts financed out of the IBRD loan) awarded in 2008-2012, with a total cost of EUR 35.8 million; three ICBs in 2006–2007 (18 contracts financed under the IBRD loan) with a total cost of EUR 31.05 million. The pricing of three awarded civil works contracts followed NCB procedures and the three awarded civil works contracts that followed the shopping procedure has a price increase within the 20-50 percent range. Five medical equipment contracts awarded following ICB procedure during 2006-2007 had contracts prices increased by 10-15 percent. There were five TA contracts, financed out of the IBRD loan, awarded based on the QCBS procedure for which contract values did not increase; the same applied to one contract for TA, one for NHA, and six contracts awarded for the design of civil works based on the Low Cost Selection procedure. In only one case—an ICB for the procurement of equipment for emergency rooms in 193 municipal and rural hospitals—in which the supplier had to pay liquidated damages in the amount of EUR 26,144.86. 31 Annex 3. Economic and Financial Analysis 1. Introduction The objectives of the second Romania Health Reform Project (APL 2) were to provide more accessible health services of increased quality for those requiring maternity and newborn care, emergency medical care, and rural primary health care to achieve improved health outcomes. The original project had five components: rehabilitation of maternity and neonatal care units; development of district and local emergency areas, development of multi-trauma operating theaters in emergency hospitals, development of an integrated ambulance dispatch system, supporting the establishment of a regional telemedicine pilot project; improving primary health care and rural medical services; and supporting the development of National Health Accounts and the project management component. In 2008, when the project was restructured, the third component to provide primary health care and rural medical services was downsized to provide only the required technical support. 2. Project benefits and costs The total project cost was US$201.03 million, which represents 97.4 percent of the appraisal estimate (US$206.5 million). The project costs by year are presented in Table 2. The actual costs as a percentage of appraisal estimates shows a high level of fiscal efficiency in project implementation. 2.1. Project benefits This section will first present the outcomes realized as a result of the project. The project outcomes are then quantified in economic terms. The first component supported rehabilitation of maternity and neonatal care units in 22 major hospitals, providing equipment units like newborn incubators, monitors, ECGs, etc. A total of 10,500 equipment units were provided to 204 maternities. In addition to these materials, support from this component also facilitated training to about 830 medical personnel that included doctors and nurses. These investments reduced maternal mortality from 24 deaths per 100,000 in 2004 to 14 per 100,000 in 2013. The neonatal mortality fell from 9.6 deaths per 1,000 in 2004 to 4.5 per 1,000 in 2013, falling from 2068 deaths in 2004 to 673 deaths in 2013. Similarly, the post neonatal mortality decreased from 1,573 in 2004 to 634 in 2013. The above project outcomes show significant benefits from the interventions completed under the first component. The development of district and local emergency hospital facilities under the second component decreased the 24-hour death rate among patients treated in the Emergency Room and then admitted to Intensive Care Units from 5.78 percent in 2007 (baseline) to 4.16 percent in 2013. As the third component was downsized, the outcome of this component was development and approval of the Primary Health Care Reform Strategy, which redefined Primary Health Care and outlined the ways to increase access to medical services in rural areas through physical rehabilitation and by offering incentives for young family doctors to stay in rural and underserved areas. The Health Sector Strategy action plan and the development of National Health Accounts under Components 3 and 4 contributed to the overall health sector improvement in the country. 32 The above outcomes from the project are summarized below in Table 1. Table 1: Project outcomes and achievements in relation to the baseline Outcome indicator Baseline Outcome Component 1 Maternal mortality number per 100,000 24 14 Neo-natal mortality; per 1,000 9.6 4.5 Post neo-natal mortality; number of deaths 1573 634 Component 2 24-hour death rate in emergency room 5.78 4.16 facilities Component 3 Refined and approved Primary Health Care reform Component 4 Developed National Health Accounts The gains in health outcomes compared to the baseline levels prior to project implementation are considered the benefits from the project. In other words, the marginal improvements in health outcomes resulting from the project, compared to the pre-project scenario, are the project benefits. These marginal improvements with the project are converted to monetary values to assess economic benefits and measures of economic efficiency. To estimate monetary values, we have considered the time lags in project investments and its impact on health outcomes on the population, the size of the target population, the existing patterns of morbidity and mortality, the number of years of productive life added as a result of the percentage decrease in mortality, and the existing cost structure in the health sector. Assumptions To quantify the benefits from the project the following assumptions were made: 1. The benefits from the project span over a period of 10 years. 2. The project investment activities under Component 1 targeted about 17.8 million people, which accounts for about 80 percent of the population of Romania. 3. The investment activities under Component 2 benefit about 50 percent of the country’s population, about 11.1 million people. 4. The investments under Components 3 and 4 improve overall efficiency of the project. The gains in health outcomes for the targeted population over the ten-year period were converted into monetary values. These additional benefits with the project then become the benefits from the project. The project generated both direct and indirect benefits. The benefits from rehabilitation and improvement of maternal and neonatal care units included reductions in the number of maternal and infant deaths, the average length of stay following deliveries, the number of Caesarean sections, and low birth-weight babies. The rehabilitation and improvement of emergency care centers reduced the number of deaths resulting from accidents, injury and poisoning, ischemic heart disease, and other external factors. The benefits included both a reduction in the costs of provision of services and an increase in benefits from the potential life years saved by the project with the economic and financial value of increasing productivity. These benefits were estimated using data on unit costs and parameter estimates for Romania. 33 3. Measures of project worth and economic efficiency The costs, estimated benefits, and measures of economic efficiency are presented in Table 2. The economic valuation of the benefits from the project outcomes shows a rate of return at 28.4 percent and the Benefit: Cost ratio of 1.65 at 10 percent rate of discount. These results are consistent with the ex-ante expected returns and measures of economic efficiency presented in the Project Appraisal Document. Table 2: Costs, benefits and measures of economic efficiency of the project (million $) Year Project costs Project benefits Net benefits 1 85.0 - (80.0) 2 0.5 - (0.5) 3 0.4 26.1 25.7 4 19.9 36.5 16.6 5 19.8 58.4 38.6 6 9.7 59.5 49.8 7 9.6 60.6 51.0 8 25.8 61.8 36.0 9 19.8 62.9 43.1 10 15.9 64.1 48.1 Total 206.5 429.9 228.4 Benefit/Cost ratio (B/C) 1.65 Internal Economic Rate of Return (IERR) 28.4 Net Present Value of benefits (NPV); US$ million 89.35 4. Sensitivity analysis Sensitivity analysis provides an indication of the impacts of changes in the assumptions or expected benefits and/or costs on net benefits from the project and the consequent measures of economic efficiency. Thus, the sensitivity analysis provides an indication of the robustness of the results under the assumptions that the future benefits fall short of the expectations or if there is an unexpected cost escalation. The Project Appraisal Document provides cases of reduction in project benefits by 40 percent and 20 percent and two alternative cases of delays in project implementation. To perform the sensitivity analysis we have not considered the later cases of delays in project implementation as it is no longer relevant. Therefore, we have only done sensitivity analysis with reductions in project benefits. To do the sensitivity analysis, three alternative scenarios are considered. The first scenario considers a 20 percent reduction in project benefits from both the components. In the second scenario, it is assumed that there is 40 percent reduction in benefits from the first component and 20 percent reduction in benefits from the second component. The third scenario considers the case when the benefit from the first component is down by 20 percent and the benefits from second component falls by 40 percent. The fourth scenario considers the case when benefits fall by 40 percent. The sensitivity analysis shows that the project remains viable under the first three scenarios. But when total benefits fall by 40 percent, the ERR falls below 10 percent and a benefit cost ratio at 10 percent 34 discount rate becomes less than one resulting in negative Net Present Worth. However, it may be noted that project yields reasonable returns when the benefits are down by 20 percent for both the components and by 40 percent for either of the first two components. These results further support the fact that health outcomes/gains from the first two components of the project exceeded the targets envisaged at project development. Table 3: Sensitivity Analysis of Project Benefits Scenario ERR NPV B/C Ratio Base case 28.4 89.3 1.65 Reduction in project benefits by 20% 26.4 80.2 1.59 Benefits from Component 1 down by 20% and 15.2 20.5 1.15 Component 2 by 40% Benefits from Component 1 down by 40% and 15.5 22.8 1.16 Component 2 by 40% Reduction in project benefits by 40% 9.7 -0.93 0.99 ERR: Economic Rate of Return; B/C: benefit/cost ratio; NPV: Net Present Value 5. Poverty impacts The beneficiary analysis of the project showed that public sector health interventions to strengthen maternal and neonatal care and emergency care services under the project benefited mostly rural and poor sections of the population. In this context it may be noted that in Romania, the sections of the population that mostly depend on public health facilities are the poor and rural households. Hence improvements in public health facilities will directly benefit them through reduction in health expenditures, reductions in mortality rates, and better health outcomes, which in turn will increase their incomes and thus reduce poverty in the long run. 6. Conclusion The results show that the returns from the project justified the investment and the measures of economic efficiency are satisfactory. Due to delays in project implementation, the project had to be restructured with significant scale back in Component 3 to expand primary health care facilities. However, the project outputs and achievements exceeded the targets for Components 1 and 2, which in turn resulted in satisfactory rates of return and measures of economic efficiency. The sensitivity analysis shows that even under conservative assumptions of significant reduction in project benefits, the rates of return were about 15 percent. These results are close to the expected measures of economic efficiency presented in the Project Appraisal Document, which confirms that the project had achieved the expected outputs and outcomes at project development. 35 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Dorothee B. Eckertz Senior Operations Officer ECSH1 Operations Richard Florescu Senior Operations Officer ECSH3 Operations Dominic S. Haazen Lead Health Policy Specialist AFTHW Health Policy Cem Mete Lead Economist SASSP Health Economics Silviu Calin Radulescu Senior Health Spec. ECSHD Health Yingwei Wu Senior Procurement Specialist LCSPT Procurement Ayda Aysun Yurekli Consultant ECSHD Consultant Supervision/ICR Nurul Alam Senior Procurement Specialist ECSO2 Procurement Bogdan Constantin Constantinescu Senior Financial Management Specialist ECSO3 Financial Management Agnes Couffinhal Senior Economist (Health) ECSH1 Health Economics Dorothee B. Eckertz Senior Operations Officer ECSH1 Operations Richard Florescu Senior Operations Officer ECSH3 Operations Program Assistance, Gabriel C. Francis Program Assistant ECSHD Administrative Johanne Angers Senior Operations Officer ECSH 1 Operations Tamar Gotsadze Consultant ECSP4 Consultant Program Assistance, Camelia Iulia Gusescu Program Assistant ECCRO Administrative Dominic S. Haazen Lead Health Policy Specialist AFTHW Health Policy Kari L. Hurt Senior Operations Officer EASHH Operations Vladislav Krasikov Senior Procurement Specialist EASR1 Procurement Gabriela Doina Manea Resource Management Analyst HRSRM Resource Management Wezi Marianne Msisha Health Specialist ECSH1 Health Program Assistance, Regina Oritshetemeyin Nesiama Senior Program Assistant ECSSD Administrative Silviu Calin Radulescu Senior Health Specialist ECSHD Health Dan Ioan Sava Consultant ECSPE Consultant Barbara Ziolkowska Procurement Analyst ECSO2 Procurement Cristina Petcu Consultant ECSH1 Consultant Nino Moroshkina Consultant ECSH1 Consultant Program Assistance, Carmen F. Laurente Senior Program Assistant ECSHD Administrative 36 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY03 13.65 38.16 FY04 33.34 103.90 FY05 29.36 80.34 Total: 76.35 222.40 Supervision/ICR FY05 16.78 42.40 FY06 27.44 90.35 FY07 46.88 108.04 FY08 71.84 169.48 FY09 28.01 94.55 FY10 35.44 130.06 FY11 23.38 76.58 FY12 20.17 89.12 FY13 15.65 80.82 FY14 4.54 28.67 Total: 290.13 910.07 Grand Total: 366.48 1,132.47 37 Annex 5. Beneficiary Survey Results Client satisfaction survey for maternities The client satisfaction survey was designed to show the satisfaction index per level of maternity. This survey was conducted in four consecutive waves. The first phase of the survey collected the baseline data and was conducted in 2008 (field work took place in October-December 2008). The survey was also performed in 2009, 2012, and 2013 using the same methodology, survey design, and questionnaire. The structured interviews were performed with a total of 1,552 patients who had delivered and were ready for discharge. Interviews were performed by trained staff members (the PMU regional coordinators and representatives of public health directorates). Selection of maternities was made based on the level of maternal and neonatal mortality. The objectives of the survey were to collect the information on hygienic and sanitary conditions of hospitals, information given to patients, involvement of patients in decisions about their care, personnel, quality of services, etc. Respondents were interviewed for about 30 minutes on average. The answer to each question was weighed. The maximum score for the whole questionnaire was 140 points. For the first level maternities (the lowest level), the satisfaction index has been increased by 16 percent. For the second level, the satisfaction index has been increased by 7 percent, while for the third level facilities, the satisfaction index has been increased by 9 percent. Regarding the levels of facilities, the Level 3 facilities represents the highest level of treatment and equipment, with more severe and complicated cases managed at these hospitals by continuous improvement of the existing referral system. In this respect, a larger number of more severe and complicated cases are being transferred from maternity Level 1 to maternity Level 2, and to maternity Level 3. Years Total number of Total number Levels of Number of Number of patients of maternities maternities maternities per patients per level interviewed level 2008 363 42 Level 3 14 139 Level 2 17 169 Level 1 11 55 2009 359 50 Level 3 13 134 Level 2 16 175 Level 1 11 50 2012 556 57 Level 3 15 166 Level 2 27 260 Level 1 15 130 2013 274 30 Level 3 10 143 Level 2 14 108 Level 1 6 23 Maternities 2008 2013 To be achieved at the End-of-Project Status 38 per level (baseline) end of the project Results Level 1 109/140 126/140 10% improvement 16% improvement OVER ACHIEVED Level 2 110/140 118/140 10% improvement 7% improvement Not achieved Level 3 109/140 119/140 3.5% improvement 9% improvement OVER ACHIEVED 39 Annex 6. Stakeholder Workshop Report and Results Not applicable. 40 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR A. Borrower’s ICR Contribution MINISTRY OF HEALTH HEALTH SECTOR REFORM PHASE II PROJECT MANAGEMENT UNIT _____________________________________________________________ 1-3, Cristian Popişteanu Ent, Sector 1, 010024, Bucureşti, ROMÂNIA BORROWER’S IMPLEMENTATION COMPLETION REPORT IBRD Loan no. 4760RO The Romania Health Sector Reform Project (APL 2) was the second phase of an agreed two-phased adaptable program lending (APL) with an estimated budget of US$206.5 million (€ 167.9 million), supported by a US$80.0 million (€ 65.1 million) IBRD loan, a US$81.8 million (€ 66.4 million) European Investment Bank (EIB) loan, and US$44.8 million (€ 36.4 million) in funding from the Government of Romania.. The IBRD loan was approved by the Bank’s board on December 16, 2004, the loan agreement was signed on 28th January, 2005 and became effective on June 27, 2005 after approval of the loan agreement by Law No. 171/June 9, 2005, with an original closing date of December 31, 2009. The IBRD Loan closing date project was extended a total of five times until December 31, 2013. The first extension of 12 months was granted from December 31, 2009 to December 31, 2010 through the Second Amendment to the Loan Agreement of November 9, 2009, countersigned by the Ministry of Finance on November 17, 2009, and approved by Government Decision No. 353/2010. This was followed by a second extension (in two steps) to December 15, 2011 by the World Bank’s Letter regarding the Fourth Amendment to the Loan Agreement, dated February 21, 2011, and approved by Government Decision No. 606/ June 16, 2011, a third extension to December 15, 2012 through the World Bank’s Letter regarding the Fifth Amendment to the Loan Agreement, dated December 15, 2011 approved by Government Decision No. 579/2012., and a fourth extension to March 15, 2013 agreed by the World Bank through its letter dated November 26, 2012 and approved by Government Decision nr. 45/ February 14, 2013. The last and fifth extension was granted by the World Bank on March 15, 2013, with a new closing date of December 31, 2013 and approved by Government Decision no 399 / June 19, 2013. The EIB financing contract was signed on 28th February, 2005 and approved by Government Decision No. 442/2005. The period of implementation was initially estimated to end in July, 2009. The EIB Loan closing date was extended three times: April 2012, by the Amendment No. 2 to the Finance Contract of October 14, 2009, countersigned by the Ministry of Finance on October 23, 2009, and approved by Government Decision No. 550/June 9, 2010. December 31, 2013, by the Amendment no. 3 to the Finance Contract signed by the EIB on May 9, 2012 and countersigned by the MPF on May 22, 2012. The Amendment was approved through Government Decision No. 626/June 20, 2012. December 31, 2014 by the Amendment no. 3 to the Finance Contract signed by the EIB on May 9, 2012 and countersigned by the MPF on May 22, 2012. The Amendment was approved through Government Decision No. 1148/December 23, 2013 41 The project had five components. The amounts allocated initially for each component were revised in 2008. PAD - INITIAL ALLOCATION EUR millions Costs by Financiers IBRD EIB Government Total 1. Maternal and Neonatal Care 17.21 66.40 21.26 104.87 2. Emergency Medical Services 35.33 0.00 11.90 47.23 3. Primary Health Care 9.77 0.00 1.64 11.41 4. National Health Accounts and 0.41 0.00 0.11 0.52 Planning 5. Project Management 2.38 0.00 1.47 3.85 Total PROJECT COSTS 65.10 66.40 36.38 167.88 Commitment Charges 0.00 0.00 0.13 0.13 Front-end fees 0.00 0.00 0.33 0.33 Total Disbursement 65.10 66.40 36.84 168.34 PAD - ACTUAL ALLOCATION EUR millions Costs by Financiers IBRD EIB Government Total 1. Maternal and Neonatal Care 23.574 62.633 24.864 111.071 2. Emergency Medical Services 33.796 0.000 8.498 42.294 3. Primary Health Care 4.389 0.000 1.541 5.930 4. National Health Accounts and 0.146 0.000 0.052 0.198 Planning 5. Project Management 2.584 0.893 1.985 5.462 Total PROJECT COSTS 64.489 63.526 36.940 164.955 Commitment Charges 0.000 0.000 0.130 0.130 Front-end fees 0.000 0.000 0.330 0.330 Total Disbursement 64.489 63.526 37.400 165.415 LOAN APPROVED VALUE 65.100 66.400 36.840 168.340 UNALLOCATED: 0.611 2.874 -0.560 2.925 Component 1: Maternity and neonatal care (Amount allocated: €108.75 million, of which €23.14 million was from IBRD and €66.40 million from EIB) 42 OBJECTIVE This component supports physical rehabilitation for maternity and neonatal care units plus medical and other equipment necessary for high quality neonatal and maternity services. Technical assistance and training is provided to ensure implementation of best international practices, building on already existing partnerships between the Romanian Government and international donors: WHO, UNICEF, UNFPA. Support will be provided to improve the capacity of health care authorities and provider units to monitor service quality and access. The MoH has developed a tree concept for the structure and functioning of the future network of maternity and neonatal care units. Under this approach, maternities are structured according with the complexity of the services provided, with appropriate regional distribution for each level, in order to cover the entire country with all levels. The selection of sites for each level is done based on the criteria such as: number of births or distance to the medical facility. Based on the results of a detailed analysis of training and other human resources needs in this sector, which was undertaken during project preparation, training and quality improvement programs are financed to ensure that the maximum benefits are achieved from the physical investments that are made. The following activities were implemented under this component: 1. Activities related to procurement and implementation of civil works for units which provide medical assistance for mother and child: - Technical expertise for 22 hospital units; - Feasibility studies and technical design for rehabilitation works; - For the 22 maternities facilities scheduled to be rehabilitated under APL2 project, the following rehabilitation works contracts were signed until end of 2012:  “Cuza Vodă” ObGyn Hospital of Iaşi  “Dr. I. A. Sbârcea” ObGyn Hospital of Braşov  Emergency Clinical County Hospital of Sibiu -ongoing  Emergency Clinical County Hospital of Cluj -ongoing  ObGyn Hospital of Brăila - ongoing  Municipal Hospital of Rădăuţi - finalized  ObGyn Hospital of Botoşani - finalized  “Sf. Ioan cel Nou” County Hospital of Suceava (Contractor: SC Victor Construct SRL) - finalized  County Hospital of Piatra Neamţ - finalized  Emergency County Hospital of Bacău - finalized  Emergency County Hospital of Craiova  County Hospital of Giurgiu - finalized  Emergency University Hospital of Bucharest -ongoing  “Sf. Pantelimon” Emergency Clinical Hospital of Bucharest -ongoing  County Hospital of Bistriţa Nasaud - finalized  Municipal Hospital of Onesti - finalized  ObGyn Hospital of Oradea - finalized  ObGyn Hospital Of Ploiesti -ongoing  “Sf. Ioan” Emergency Clinical for Children Hospital of Galati - ongoing  Emergency County Clinic Hospital of Tg. Mures - ongoing The rehabilitation works contracts financed by the IBRD loan:  Cuza Vodă” ObGyn Hospital of Iaşi-finalized  Emergency Clinical County Hospital of Sibiu - ongoing 43  Emergency Clinical County Hospital of Cluj - ongoing The rehabilitation works contracts were completed for 14 maternities, four contracts are on –going and the works contracts were suspended for the following hospitals: - Dr. I. A. Sbârcea” ObGyn Hospital of Braşov; - Emergency County Hospital of Craiova; - Supervision of rehabilitation works 2. Equipment delivery for mother and child healthcare units: - Equipment delivery phase 1: 8600 pieces of equipment to 204 hospitals; - Equipment delivery phase 2: 500 pieces of equipment to 63 hospitals; - Monitors for maternities – children and adult. The total contract amount financed out of the EIB Loan and GoR was of EUR 22 million and EUR 19.6 million financed out of the IBRD loan and GoR. 3. Provision of technical assistance and training for the medical staff - Development of an Operational Manual on Maternal & Neonatal Referral System; - Development and implementation of a Quality Assurance system; - Delivery of a training program to improve the capacity of health care authorities and provider units to monitor service quality and access: 831 medical staff trained (94 Trainer of Trainers participants and 737 regularly training). - Develop national guidelines - Training of medical staff involved in mother and child care finalized in 2006 The objectives of this component have been achieved. COMPONENT 2: EMERGENCY CARE SERVICES (Amount allocated: €43.2 million, of which €33.8 million was from IBRD) OBJECTIVE. This component was supposed to upgrade hospital emergency areas and to develop and implement integrated ambulance dispatch capability. Both interventions are essential for maximizing the impact of the investments that have been made to date and the effectiveness of the emergency medical services system generally. Sub-component 1: Upgrade Hospital Emergency Areas. The objective of this sub-component will be to increase the quality of patient care, resulting in increased survival rates for the patients arriving in the ER, through adequate equipment, training and improved protocols. A total of about 60 emergency rooms are included in the project. Sub-component 2: Integrated Ambulance Dispatch System. This sub-component will up-grade the existing communication system of the National Public Medical Pre-hospital Emergency Service in order to integrate with the Central Emergency Call Center – unique 112 number – and to enable voice and data communication between the county (judet) ambulance central dispatcher and sub-stations or ambulances. The following main activities have been carried out under this component: 1) Equipment for Emergency Rooms and ICU Twelve contracts in amount of EUR 9,895 million have been completed in 2007 for procurement of medical equipment 1. Rehabilitation of emergency rooms - Reorganization and Relocation (Design and civil works) of emergency room of Tg. Mureş Clinical County Hospital - Small Rehabilitation / renovation works for upgrading Hospital emergency rooms (medical fluids for 12 hospitals / civil works for 13 hospitals): 44  Pediatric Hospital Ploiesti - civil works and medical gas;  Pediatric Hospital Brasov - civil works;  County Hospital Timisoara - civil works;  County Hospital Ramnicu Valcea civil works;  County Hospital Targu-Jiu – medical gas;  County Hospital Alexandria - medical gas.  Pediatric Hospital Sibiu - civil works and medical gas  County Hospìtal Braila (civil works and medical gas)  County Hospìtal Calarasi (medical gas)  County Hospital Satu Mare (medical gas)  Emergency County Hospital Slobozia (medical gas)  Emergency County Hospital Vaslui (civil works and medical gas)  County Hospital Giurgiu (civil works)  County Hospital Satu Mare (civil works)  County Hospital Suceava (civil works)  Pediatric Hospital Botosani  County Hospital Constanta (medical fluids) - Weak currents installation equipment for the emergency room Tg. Mures Emergency Clinical County Hospital (Surveillance and control systems, Communication equipment Network and Telephony equipment) - Procurement of complex Intensive Care Unit (ICU) beds with X-ray transparency, syringe infusion pumps and patient temperature management system 2. Medical Equipment for Emergency Rooms - Medical equipment for 63 emergency rooms and 193 small emergency rooms (intra-hospital transportation, monitoring and EKG, ventilation, resuscitation, airway management and immobilization; CT scanner and X-ray equipment for one hospital) - Procurement of monitors for Intensive Care Units and Cardiology departments. - Local and Regional Paging systems - Emergency audio-video conference telemedicine system and emergency telemedicine data system - Registration equipment for ambulance radio-communication system - Upgrade and extension of the existing Emergency Telemedicine System and implementation of an Emergency Telemedical Imaging system - Simulators for training of personnel working in the emergency services; 3. Communications Equipment - Radio equipment; - Data transmission equipment; - GPS for ambulances. The objectives of this component have been achieved. Component 3: Primary Health Care and Rural Medical Services (Amount allocated €5.64 million, of which €4.17 million was from IBRD) OBJECTIVE. Initially the Component 3 focused on improving the accessibility and quality of basic medical services in rural and small urban areas and supposed to provide support through two activities: 45 Sub-component 1: Multipurpose Health Centers, developed as a new concept that builds upon the existing rural health centres. These centres supposed to allow the provision on diversified services by family doctors, ambulatory care specialists, community nurses, as well as home care and social services in remote rural areas and small towns. An initiative to rehabilitate rural primary care services, defined as “Health Centers”, was implement during the period of 1994-1995, but the efficiency wasn’t demonstrate because the way wasn’t adapt to the local needs and due to the lack of sustainability funds. Therefore the FDs appointed as a resource of expertise express their doubts concerning the success of such program. This subcomponent was dropped at the Project restructuring in 2008. Sub-component 2: Sub-loans for family doctors. Sub-loans fund/funds for family doctors supposed to finance specific investments for medical equipment, consumables, transportation means and IT equipment that will enable FD to improve the access to a wider range of health services and to improve the quality of the services provided in selected areas (including but not limited to, rural areas). According to the new Strategy of the Ministry for primary care and to the real needs of the rural and underserved area, a national program for increase access to quality basic health services in under-served area was proposed in order to replace the initial objectives of this Component. Also this subcomponent was dropped at the Project restructuring in 2008. The third component (PHC and Rural Medical Services) was stalled due to (i) changes in the legislation of health sector financing, which no longer allowed the MoH to finance investments in PHC infrastructure; and (ii) developments in the capital market which gave group practices better access to loans. The following main activities have been carried out under this component: 1. Rural PHC strategy: - TA in order to define beneficiary map - TA to define the Rural PHC Strategy 2012-2020 and related Action Plan Following the Fourth Amendment to the Loan Agreement, dated February 21, 2011, in the APL2 Project were included: 2. Rehabilitation of vaccine production and ampoule filling and sealing area at the Cantacuzino Institute of Bucharest, including the purchase of the necessary equipment. - technical design for the influenza vaccine production unit rehabilitation - rehabilitation of the influenza vaccine production unit - rehabilitation of the aseptic ampoule sealing and filling station - site supervision of the rehabilitation works - upgrade of the vaccine filling &sealing line Euro 5.760 million from the third component was relocated to other components as follow: The objectives of this component have been achieved. Component 4: National Health Accounts and Planning (Amount allocated: €0.21 million, of which €0.15 million was from IBRD) OBJECTIVE. This component supported the development of National Health Accounts (NHA) and preparation of proposals for rationalization and service development projects. Sub-component 1: National Health Accounts (NHA). For the development of NHA support would be provided to adapt the internationally validated methodology, propose changes of regulations regarding 46 reporting of financial information in the health sector, perform analysis of existing financial information and surveys, conduct additional surveys, train staff, and prepare, publish and disseminate reports. A working group for NHA development was established, including representatives of MoH, MoF, National Health Insurance House, National Institute for Statistics. The project would finance participation in training, seminars and workshops, investments in IT equipment and software, surveys and studies, local and foreign TA. The second activity supported by this component was supposed provide to MoH, District Public Health Authorities, health care institutions or local authorities the resources for preparation of projects to be submitted to financing institutions and donors. The projects would aim to implement the rationalization strategy and service plans prepared in APL 1 and make use of the capacity for planning developed at local level. This preparatory work will increase the capacity to access and use EU structural funds in the future. This second activity was dropped in 2009. The following main activities have been carried out under this component: - Development of National Health Accounts (NHA). The objectives of this component has not been achieved, the NHA has been created, but the system it’s not functional yet. The legal framework regarding a detailed NHA system, according to budget lines, is not in place, due to the changes in legislation. Component 5: Project Management (Amount allocated €4.46 million, of which €2.88 million was from IBRD) 1. TA for production and delivery of Public Information Campaign on the Health Sector Reforms in Romania and related support programs. 2. TA for establishing proposals for the basic health care service package from which persons insured with the social health insurance system will benefit. 3. TA for Health Technology Assessment Training. The PMU was proper staffed with experienced and qualified experts. Their high quality work was reflected in the high number of contracts and activities that have been carried out during Project implementation The PMU staff salaries were drastically reduced twice during 2010 by forty (40) percent. The issue of PMU staff salaries reduction applied starting with January 2010 has been never been solved until the loan closing date despite of the memorandums undertaken by the PMU staff internally in the Ministry of Health, in the Romanian parliament and in the court. This issue of PMU staff salaries reduction starting with January 2010 until the end of the Project had a strong negative impact on the Project implementation demotivating the PMU staff. Commencing with April 2013 and continuing during 2014 the PMU staff was involved in all the preparation stages of the Health Sector Reform – Improving Health System Quality and Efficiency Project. Valuable contribution of the PMU staff in new Project preparation was reflected by the World Bank approval of the Loan in March 2014. 47 PDO Indicators Original Target Values Actual Value Achieved Formally Revised Indicator Baseline Value (from approval at Completion or Target Target Values documents) Years Indicator 1 Maternal Mortality (MM) and Rate (MMR). MM: 20% decrease, Value quantitative MM:52 MM: 29 MM:41 and qualitative MMR: 0.24/1000 MMR: 0,14/1000 MMR 0.19/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement OVER ACHIEVED. Maternal mortality decreased nationwide by 85%. Comments (incl.% Data source: Annual National Statistics (ANS), National Reporting System (NRS) and countrywide achievement) hospital data. Indicator 2 Number of neonatal deaths (ND) and neonatal death rate (NDR). 25% decrease from Value quantitative Death: 2068 baseline Deaths: 673 and qualitative Rate: 9.6/1000 Deaths:1551 Rate: 4.5/1000 Rate: 7.2/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement OVER ACHIEVED. Neonatal deaths decreased by over two-thirds (67.5%) nationwide between 2004 and Comments (incl.% 2012. achievement) Data source: Annual National Statistics (ANS), National Reporting System (NRS) and countrywide hospital data. Indicator 3 Post-neonatal death (PHD) and post-neonatal death rate (PNR). 25% decrease from Value quantitative Death: 1573 baseline Deaths: 634 and qualitative Rate: 7.3/1000 Deaths:1180 Rate: 4.2/1000 Rate: 5.5/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement Comments (incl.% OVER ACHIEVED. Post-neonatal deaths decreased by 60% nationwide between 2004 and 2012. achievement) Data source: Annual National Statistics (ANS), National Reporting System (NRS). Indicator 4 24-hour death rate among patients treated in ER, then admitted to ICU in that hospital. Value quantitative 15% decrease 5.78% (for 6 ICUs) 4.16 % (for 6 ICUs) and qualitative 4.91% (for 6 ICUs) Date of 12/30/2007 12/15/2012 12/31/2013 Achievement ACHIEVED. 24-hour death rate reduced by 28% between 2007 and 2012. Although the sample size is too small for the results to be extrapolated to the whole emergency system, the project-related target has been Comments (incl.% fully met. Only six hospitals received equipment, including ventilators for ICUs from the Emergency Care achievement) Services Component. Other hospitals received only monitoring equipment and thus were not monitored by the project. Indicator 5 Development of a primary care rural strategy. Development of a Value quantitative Development of a primary care Development of a primary care primary care rural and qualitative rural strategy - No rural strategy - Yes strategy - Yes Value quantitative MM:52 and qualitative MMR: 0.24/1000 Date of 12/31/2004 12/31/2009 12/31/2013 Achievement OVER ACHIEVED. Maternal mortality decreased nationwide by 85%. Comments (incl.% Data source: Annual National Statistics (ANS), National Reporting System (NRS) and countrywide achievement) hospital data. The IBRD disbursement level as of May 2, 2014 was 97 percent or EUR 63.07 million withdrawn from the IBRD Loan amount of EUR 65.1 million. 48 The Borrower rates the Project Implementation as satisfactory. Lessons learned Some of the activities included initially in the preparation stage of the Project were dropped after 2 years from the effectiveness. This was reflected in some changes of the PDO. The unavailability of TWGs members during the process of preparation of the technical specifications or terms of reference and difficulties in establishing TWG meetings, the TWG delays in providing clarifications responses to the technical specifications generated delays in the project implementation. In many cases the evaluation committee members were not available during bid evaluation process and generated delays in project implementation. Discontinuities in their work due to changes in working groups and evaluation committees’ composition had also delayed the process. The low level salaries of the PMU staff starting with January 2010 had a strong negative impact on the Project implementation. The staff was not committed and few of the experts had left the team due to the salary reduction. It was obvious that PMU staff remuneration level is essential for a proper implementation of a project. Technical designs for the rehabilitation works proved to be outdated. A careful review of the technical designs could bring benefits to the Project implementation. The EIB requested the revision of the technical designs after the bidding processes were initiated. This led to significant delays in works contracts implementation. Using the ICB method for procurement of Goods and IT systems proved to be the most effective method resulting in cost savings registered in most of the cases. During the first years of the project, the budgetary allocation was reasonable good. In the last two years, secondary to budgetary restraints and to an inadequate prioritization in the budget of MoH, the allocation was inadequate (smaller). A better communication between the directorates of MoH can give, in the future, better results. The cooperation with the World Bank The PMU and the Ministry of Health had an excellent cooperation with the MoPF and World Bank’s team. The review of the bidding processes carried out by the Bank procurement specialists was adequate and very professional. The permanent support of the MoPF team was also an important factor for success. B. Borrower’s Comments on ICR 49 50 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders No Comments were received from EIB. 51 Annex 9. List of Supporting Documents 1. World Bank, Implementation, December 23, 2004, Completion Report for Health Sector Reform Project in support of the First Phase of the Health Sector Reform Program, 2. World Bank, October, 2004, Project Information Document (PID) 3. World Bank, November 17, 2004, Project Appraisal Document (PAD) 4. World Bank, January 28, 2005, Loan Agreement, 5. World Bank, 2004 – 2013, Aide Memoires and Back-to-office Reports, 6. World Bank, Restructuring papers, 2008, 2010, 2011, 2013 7. World Bank, ISRs, 2011-2013 8. World Bank, 2004-2013, Management and other important letters and memoranda 9. EMP, August, 2004 10. World Bank, April 4-12, 2013, Aide Memoire, Identification mission for proposed health sector operation, 11. Memorandum from the Minister of Health, G.E. Nicolaescu approved by the Prime Minister, V. Viorel Ponta, 2013 12. PMU final report, compilation of annual status reports for 2006-2013 13. TA report, DIADIKASIA Business Consultants SA, 2012, production and Implementation of an Information, Education and Communication (IEC) Campaign on the Health Sector Reforms in Romania and related Support Programs 14. TA report, NICE International, 2012, Reviewing the Content and Listing Processes for the Romanian Basic Package of Health Services and Technologies 15. TA report, OPM, 2012 Strategy for Primary Health Care in Underserved Areas and the Related Action Plan, OPM, 2012 52 22°E 24°E 26°E 28°E UKRA INE ROM ANIA To Uzhhorod To Ivano-Frankivs'k CITIES AND TOWNS COUNTY (JUDET) CAPITALS To NATIONAL CAPITAL ˘ Balti ROMANIA 48°N BOTOSANI RIVERS Satu Mare MAIN ROADS Botosani Si MARAMURES re t MA RE SATU MARE SUCEAVA RAILROADS Baia Mare C Suceava COUNTY (JUDET) AND MUNICIPALITY M a Pr To Somes (MUNICIPIU) BOUNDARIES ut o Chisinau BISTRITA- r l IASI p INTERNATIONAL BOUNDARIES NASAUD NASAU D d Iasi a To Bi Zalau a Budapest str t Oradea Dej Bistrita ita To h Piatra- v SALAJ Chisinau 30°E HUN G ARY Neamt i i a CLUJ a s Roman To BIHOR re n NEAMT M OL DOVA u M Budapest Cluj- MURES Gheorgheni Vaslui M Napoca rgu Târgu Husi ts Crisul A Turda Mures HARGHITA Bacau VA S L U I BACAU . lb Miercurea- Ciuc ALBA Si re ARAD Onesti t Arad Birlad To Subotica Brad Alba Mures Iulia Medias 46°N COVASNA UK R AI N E UKR Deva SIBIU BRASOV Sfântu Sf ntu VRANCEA Timisoara Tecuci Hunedoara Sibiu Gheorghe B Lugoj TIMIS Moldoveanu Focsani GA LATI GALATI HUNEDOARA Tim (2,544 m ) Brasov a is SERBIA Petrosani M o u Galati i a n n a a t h ARGE S n ARGES t a i n s BUZAU Buza u Braila To t Resita r p LCE A VÂLCEA Novi Sad a Buzau Tulcea C mnicu Râmnicu CARAS - GORJ lcea Vâlcea PRAHOVA Târgu rgu Jiu BRA ILA BRAILA TULCEA SEVERIN Târgoviste rgoviste Ploiesti Pitesti MBOVITA DÂMBOVITA a Jiu Danub Orsova Drobeta- ita e IALOMITA Ialom j Turnu Severin W a l a c h i a ILFOV B l ack u Ar ge BUCURESTI Slobozia Slatina s Fetesti MEHEDINTI BUCHAREST r To OLT CALAR ASI Calarasi CALARASI Navodari Sea b Nis ˘ Craiova TELEORMAN Medgidia Ol nube Constanta o t 0 25 50 75 100 Kilometers Da 44°N DOLJ GIURGIU CONSTA NTA CONSTANTA 44°N Caracal D 0 25 50 75 Miles Calafat Alexandria Giurgiu Mangalia IBRD 33469R3 Turnu Magurele SEPTEMBER 2012 This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other information shown on this map do not imply, on the part of The World Bank Group, any judgment on the legal status of any territory, or any endorsement or acceptance of such boundaries. To Sofiya 24°E BULGA RIA To To To Shumen To Varna 30°E Veliko Turnovo ˘ Shumen