Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) Report Number : ICRR0021513 1. Project Data Project ID Project Name P117157 SV Strengthening Public Health Care Syst Country Practice Area(Lead) El Salvador Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-80760 30-Jun-2016 79,853,295.14 Bank Approval Date Closing Date (Actual) 21-Jul-2011 30-May-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 80,000,000.00 0.00 Revised Commitment 79,853,295.14 0.00 Actual 79,853,295.14 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives According to the Loan Agreement (p. 4), the project's objectives were "to: (i) expand the coverage, quality and equity in utilization of primary health services provided under the Integral and Integrated Health Care Service Networks (RIISS); and (ii) strengthen the Ministry of Health's (MINSAL's) stewardship capacity to manage essential public health functions." RIISS is MINSAL's network of organizations aimed at providing integrated health services to the population, established through the National Development Plan 2010-2014. Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) At a March 2016 restructuring, the target for one outcome indicator was revised upward. Per IEG/OPCS guidelines, a split rating is not performed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project contained three components, targeted at 2.1 million people in 92 municipalities who were identified as having the greatest health needs, largest requirement for financial protection, and lowest access to disease prevention and health promotion services at the primary care level. These municipalities were chosen to avoid duplication with another project being financed by the Inter-American Development Bank (IADB; Project Appraisal Document, PAD, p. 24). 1. Expansion of Priority Health Services and Programs (appraisal, US$ 45 million; actual US$ 60.6 million). This component was to strengthen the quality and delivery capacity of priority health services under the RIISS through the gradual implementation of a Comprehensive Primary Health Care Strategy, including the acquisition of medical equipment and maintenance/minor rehabilitation of facilities; and the improved delivery of selected priority public health programs, including those addressing maternal health, reproductive and sexual health, teenage pregnancy, child health, nutrition, dengue control, and chronic kidney disease. RIISS included all levels of the health system: Family Health Community Teams (ECOS), Community Health Units, and hospitals. The component was also to support the implementation of a National Medical Emergencies system in about 16 eligible hospitals through coordination of the national hospital system with MINSAL, the Salvadoran Social Security Institute, the Ministry of Defense, the National Civil Police, the Solidary Fund for Health, and the Red Cross; the installation of a national call system for emergencies; improvements in resuscitation room equipment and the ambulance fleet; training of medical and paramedic staff in the management of medical emergencies; and strengthening of medical emergency response at the community level. 2. Institutional Strengthening (appraisal, US$ 31.4 million; actual US$ 16.9 million). This component was to strengthen MINSAL through: (i) enhancing MINSAL's stewardship and management capacity to perform essential public health functions, including improvement of the Single Unified Health Information System (SUIS); implementation of a national pharmaceutical policy aimed at increasing the quality, efficiency, and security of rational use of medicines as well as transparency in procurement; and support for activities of the National Health Institute (including training of clinicians and lab technicians, surveillance, knowledge Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) generation, and research) and the Central Laboratory; and (ii) developing an emergency preparedness plan for public health emergencies, and implementing this plan through the financing of Eligible Public Health Expenditures. The component included a contingency fund to support immediate emergency response for natural disasters officially declared by the government within the project's first three years; as there was no declared emergency during this time frame, these funds were allocated to other project activities. 3. Project Management and Monitoring (appraisal, US$ 3.4 million; actual US$ 2.2 million). This component was to support the monitoring of project activities through semi-annual external technical reviews, annual health facility surveys, and an impact evaluation. It was also to assist the Project Coordination Unit in carrying out the management, monitoring, and supervision activities of the project, and strengthen the Institutional Procurement and Contracting Unit and the Institutional Financing Unit in performing project-related fiduciary activities. As the project proceeded, challenges related to non-communicable disease, especially cancer, became more important to the implementation of the government's 2015-2019 Health Strategic Plan. As a result, US$ 9 million was reallocated across components for the design of a National Strategy for Chronic Diseases and National Plan for prevention, diagnosis, and treatment of cancer; creation of a National Radiotherapy Center; purchase of a cobalt pump to strengthen the National Cancer Institute; purchase of other medical equipment; and preparation of an emergency response action plan to allow for immediate response in case of a public health alert or emergency. At a later restructuring, additional funds were reallocated for purchase of a building for the National Reference Laboratory (a new subcomponent under the Institutional Strengthening component). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: Actual total project cost was US$ 79.8 million, reflecting the estimated total cost at appraisal of US$ 80 million minus a US$ 0.2 million front-end fee. Over successive restructurings, funds were reallocated toward the first component and away from the second and third components, such that the actual cost of the first component was over one-third above what was initially planned. Financing: The project was to financed by a US$ 80 million loan from the International Bank for Reconstruction and Development. US$ 79.85 million was disbursed, and US$ 0.15 million was cancelled. Borrower contribution: No Borrower contribution was planned or made. Dates: The project was restructured five times: Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) • December 2012: Funds were reallocated across loan categories, as some planned activities were covered with government funds, and additional infrastructure and equipment needs for strengthening health care networks were identified. • February 2015: Funds were transferred from the second to the first component to cover startup costs for the National Radiotherapy Center, and the results framework was revised to refine indicator definitions. • March 2016: In response to the mid-term review, the results framework was revised to increase targets for one outcome indicator and two intermediate indicators, and to extend the closing date by 18 months (from June 2016 to December 2017) to compensate for time lost following an 18-month initial delay in project effectiveness. • April 2017: Funds were again reallocated across component to cover building costs for a new National Reference Laboratory under the first component. • August 2017: The closing date was extended by five months (from December 2017 to May 2018) to allow for delivery of goods and completion of a user satisfaction survey. 3. Relevance of Objectives Rationale At appraisal, El Salvador had made significant gains in health outcomes since the 1990s, evidenced by increases in life expectancy, declines in child and infant mortality, and improved maternal and reproductive health outcomes. Challenges in equity and access to care, however, remained for the poor and those living in remote areas. The rise of non-communicable disease, if not addressed, threatened to place considerable fiscal pressure on the public health system. Both supply-side and demand-side factors hindered use of health services. Inequities in health care financing existed between the National Health Service (which, in 2008, spent around 1.5% of gross domestic product to cover almost 80% of the population) and the Salvadorian Social Security Institute (which spent about the same amount covering only one-fifth of the population). MINSAL's National Health Strategy at appraisal (2009-2014) sought to address fragmentation of the health sector and ensure universal coverage, with a major focus on primary health care within the RIISS. It specifically aimed at increasing coverage and equity of access to health services, improving the quality of services, and strengthening the management of information systems to enhance MINSAL's monitoring and oversight capacity. The project was highly responsive to this strategy, specifically intending to improve the quality of care and increase the efficiency of referral mechanisms so that over 80% of health needs could be resolved at the first level of care. This reorganization was to increase health care coverage to the poor within their communities, as well as quality of care and efficiency of service provision. The project's objectives were also highly relevant to the Country Partnership Strategy (2010-2012) at appraisal, which had a main priority to strengthen the delivery of social services, including improvements in the quality and coverage of primary health care services. Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) The project's objectives remained highly relevant to government and Bank strategy at closing. The government's 2015-2019 National Health Strategy, as well as its five-year development plan (2014-2019) and Social Development Plan (2018), called for consolidation and strengthening of the health sector, with focus on guaranteeing equal access for vulnerable people to quality health care services. Specific attention was paid in these plans to reduction of maternal and child mortality, expanding priority health services and programs, formulating and implementing strategies for prevention and control of cancer, and improving the structure and organization of RIISS. The Bank's Country Partnership Framework (CPF, 2016-2019) prioritized inclusive growth, improved efficiency of public spending, and provision of health services through an integrated approach. It contained an objective to expand universal coverage of quality health care. The CPF also (p. 36) focused directly on the need to improve efficiency of health services delivery through consolidation and optimization of the four-tier service delivery network; although improved efficiency was not explicitly an objective of this project, optimization of the delivery network was implicit in the project's objective to strengthen MINSAL's capacity to manage essential health functions. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Expand the coverage of primary health services provided under the RIISS Rationale The project's theory of change linked purchases of equipment for health centers and hospitals, rehabilitation and maintenance of infrastructure, purchases of ambulances and emergency equipment, and provision of vaccines and medical supplies, to increased availability of child and reproductive health services, nutrition services, supplies and equipment in referral centers, and responsiveness to medical emergencies. The strengthening of referral networks and development of strategies related to non-communicable disease and emergency medicine were also to improve responsiveness to key health threats. The resulting increased identification and availability of priority services would in turn improve coverage. Outputs 14,385 community health promoters and committees received training in comprehensive community health care. 2,437 ECOS were established, with the goal of developing local situational assessments and action Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) plans for promoting health and preventing diseases. 4,660 health personnel were trained, exceeding the target of 1,200. The project also trained 35 epidemiologists for the national laboratory. 269 ECOS and 30 hospitals in the 92 project municipalities were equipped according to MINSAL technical guidelines. This met the target of 30 hospitals. It did not achieve the original target of equipping 708 ECOS, but it exceeded the revised target of 104 equipped ECOS. Purchased equipment included dental equipment and instruments for primary health care facilities, physical therapy equipment, transportation equipment for central and regional storage facilities and environmental sanitation, vector control spraying equipment, information technology equipment and mobile devices for geo-referenced family information collection, and others (ICR, p. 47). Minor rehabilitation and maintenance works were carried out for 56 primary health care facilities and several central MINSAL facilities. Hospitals were provided with high-tech medical equipment, medical clinic furniture, and non-medical equipment (ICR, p. 50). The project purchased anti-rabies, measles/mumps/rubella, yellow fever, tetanus-diphtheria adult, 13-valent pediatric pneumoccus, and adult tetanus-diphtheria-pertussis vaccines. Medications, equipment, and supply kits were also purchased for the treatment of chronic kidney disease. Fumigation equipment, vehicles, and fuel benefitting 1,200 residents in areas most affected by dengue were purchased. Reproductive and sexual health was supported with the purchase of medical equipment, equipment for human milk banks, and human resources training. The project financed the purchase of 44 ambulances and associated first response equipment, including for reanimation. Eight hospitals were integrated into the National Medical Emergencies System, meeting the target. 662 emergency personnel were trained in efficient handling of medical emergencies. The project supported the implementation of a coordinated national system to deal with emergencies, as well as training of medical personnel in emergency management. 158 facilities in priority departments had operating information systems at project closure, exceeding the original target of 12 and the revised target of 134. The ICR did not define "priority" departments. Outcomes Coverage of children less than one year old with the third dose of pentavalent vaccine in the 92 project municipalities was 95% (baseline) in 2012, 95% (against a target of 92.5%) in 2013, 94.4% (against a target of 92.5%) in 2014, 92.4% (against a target of 92.5%) in 2015, 95% (against a target of 92.6%) in 2016, 80.6% (against a target of 92.8%) in 2017, and 86.12% in the first half of 2018. The end-of-project target was 92.8%. The ICR (p. 28) explained that coverage declined due to an increase in violence that affected health personnel's ability to reach target populations, and the fact that population projections (on which the denominator was based) did not reflect real population growth. The percentage of women utilizing the public network for institutional deliveries in the 92 project municipalities increased from 81% in 2012 to 99.8% in 2017, exceeding the target of 84%. The ICR (p. 13) provided information on falling maternal mortality, infant mortality, and adolescent pregnancy rates, but Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) attribution of these outcomes to the project's interventions is unclear given that the data provided did not correspond to the project time period, and that many factors outside the health system also influence these trends. No outcome data were provided on coverage of health services other than maternal/child/reproductive health. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Expand the quality of primary health services provided under the RIISS Rationale The project's theory of change linked improvement in infrastructure and equipment, construction and equipping of the Radiotherapy Unit and National Biomedical Laboratory, training in emergency medical services, and training/updating of health promoters, to enhanced health service delivery, progress in comprehensive care for cancer patients, and improved response to medical emergencies. This equipment and training would in turn lead to improved quality of health services. Outputs The ICR (p. 14) noted that the equipment provided to ECOS and hospitals, including high-tech medical devices, clinical furniture, supply kits, and medicines contributed to improved quality of primary care. A national biomedical laboratory was established and equipped, a National Radiotherapy Center (NRC) was built and equipped, and an equipment maintenance system was implemented, all intended to ensure increased quality, appropriateness, and continuity of care. The project team later added that the NRC is capable of providing comprehensive cancer care free of charge for patients who were previously referred to local non-governmental organizations for fee-for-service treatment. A Quality Assurance system for health facility certification was implemented. Systematized certification program evaluation criteria were set up, focusing on levels of complexity, social and health realities, attributes, size, technological level, and qualitative/quantitative capabilities of health providers (ICR, p. 52). 35 MINSAL staff were trained on accreditation of health facilities. The project team later added that a national law was passed establishing institutional responsibility for accreditation and certification of health facilities, and that a new accreditation and certification commission was created. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) Outcomes As a result of the establishment of the NRC in 2017, "services that were previously not available in El Salvador" are now available (ICR, p. 14). The NRC is projected to serve, when fully operational, about 2,200 patients annually, with a "measurable impact" on the reduction of cancer morbidity and mortality. The ICR (p. 51) noted that the NRC did not become operational until the very end of the project period. The project team later added that, in the first nine months of operation, the center registered 1,572 new patients who have undergone 730 linear acceleration therapies, 1,265 brachytherapy procedures, and 19,975 radiation sessions. 30 eligible hospitals complied with the standard bio-hazard waste management system in 2018, exceeding both the original target of 20 and the revised target of 26. The project team added that this improved waste management regime reduced the risks of waste contamination around health facilities and thus reduced potential impacts from hazardous wastewater, atmospheric emissions, dangerous chemical substances, and infections. No specific measured impacts were noted. The ICR (p. 34) reported that two user satisfaction surveys measuring satisfaction with quality improvement in MINSAL services were conducted, but the results of those surveys were not provided. The project team later added that these surveys showed an improvement in overall satisfaction in the provision of health services from 86.6% in 2013 to 92% in 2018, in the friendliness of heath staff from 90.4% to 93.3%, and in availability of medicines from 83.7% to 92.5%. The ICR did not provide information on improvements in quality of primary health services that resulted from project-financed training, equipment, or quality assurance processes. The project team later added that the MINSAL's Biomedical Equipment Management Unit (BEMU), whose infrastructure, equipment, organization, and technical training was financed by the project, was actively implementing a monthly preventive and corrective maintenance program in primary, secondary, and tertiary health facilities, providing maintenance for 1,287 pieces of equipment. Based on additional information provided by the project team related to increased availability of cancer care and improved patient satisfaction, achievement of this objective is rated Substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) Expand equity in utilization of primary health services provided under the RIISS Rationale The project's theory of change linked strengthened services and priority health programs, improved coordination between the national hospital system and other entities, improved patient information systems, and enhanced data quality and information flow regarding the health of the population with increased equity in utilization of priority health services. The project's interventions were specifically targeted at 92 municipalities chosen because of their poverty status and high health risks. In principle, this targeting plausibly improved equity in utilization of health services. However, neither the project's indicators nor the ICR reported data on equity. The ICR (pp. 18-19) reported that out-of-pocket (OOP) spending fell from 50% of current health expenditure in 2011 to 28% in 2015, and it noted that high OOP spending is usually the most inequitable type of health care financing, but no data on spending by income or wealth levels was provided. The project team later added the observation that increased equity is reflected in improvement in overall health outcomes, including reduced national under-five and maternal mortality rates, increased institutional birth delivery in the 92 project municipalities, reduced household OOP expenditure, increased overall public health expenditure, and decreased private health expenditure. The project team also provided additional data from National Household Surveys showing that, between 2008 and 2016, the percentage of those with reported illness who did not seek care declined across all income quintiles, with care-seeking at public health facilities increasing by seven and six percentage points, respectively, by those in the poorest and second- poorest quintiles, and by ten percentage points among the highest income quintile. The project's two million beneficiaries represent a substantial share of the total population represented in these national surveys (six million), making the survey data a reasonable reflection of project impact, particularly given the project's deliberate targeting of poor municipalities. The project team added that, in 2016, OOP expenditure was an impoverishing condition for only 1% of households with OOP expenditure, arguing that this is "clear evidence that the most marginalized municipalities in the country (and the most vulnerable population groups) have benefited directly through the interventions under this project." Based on the new evidence provided by the project team, achievement of this objective is rated Substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) Strengthen MINSAL's stewardship capacity to manage essential public health functions Rationale The project's theory of change linked the conducting of national surveys on health status and risk factors, the development of a SUIS, the acquisition of facilities for the National institute of Public Health and National Reference Laboratory, training of staff on managerial competencies, development of environmental diagnostics, and development of a national Emergency, Disaster Preparedness and Response Plan to improved coordination, data flow, operating procedures during emergencies, and overall efficiency and reliability in the management of analysis of data for decision making. These enhanced capacities would in turn lead to strengthened stewardship capacity to manage essential public health functions. Outputs The project supported the design, preparation, and implementation of a non-communicable disease strategy, a National Policy for Pharmaceuticals, environmental regulations related to biohazard diagnostics and norms, and a National Certification of Quality of Care process. A non-communicable disease department was established in MINSAL. The project provided support for SUIS, including technical and telecommunications equipment to enable better communications across levels of reporting, data collection, data analysis, and monitoring of the national statistical reporting system. It strengthened the capacity of the Directorate of Health Technologies to analyze and make decisions regarding the incorporation of new essential drugs (ICR, p. 18); 1,429 staff were trained in the rational use of medicines, pharmacovigilance, and evaluation of health technologies. Outcomes Two major surveys were implemented with project financing, the 2014 Multiple Indicator Cluster Survey and the 2014-2015 non-communicable disease survey. This met the target. According to the ICR, both surveys generated data that were used to determine where and how public health funding should be invested, though no specific information on this use of survey data was provided. The expansion of functionality of the SUIS was reported to have "increased the access capacity of users of the different SUIS modules, especially in the implementation of the Patient Information System" (ICR, p. 18). No specific data on this increased user access was provided. 30 hospitals now comply with the standard bio-hazard waste management system, exceeding the original target of 14 and the revised target of 26. The ICR (p. 15) stated that "the project contributed to a more efficient managerial model in the areas of resource management and strategic decision making for health policies," but no specific information to support this statement was provided. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale The ICR described a large number of project activities, in varying levels of detail, that would plausibly have led to improvements in coverage, quality, and equity in utilization of primary health services, as well as strengthening of MINSAL's overall capacity. The project's results framework did not contain outcome indicators adequate to demonstrate outcomes across the objectives to improve quality or equity of primary health services, and the ICR did not provide additional information to compensate for the results framework's shortcomings. However, the project team later provided sufficient evidence to demonstrate substantial achievement of these objectives. Overall efficacy is therefore rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency The PAD's economic analysis (Annex 7) found a benefit-cost ratio ranging from 1.61 to 2.15, with net present values of US$ 49 - US$ 92 million, varying depending on assumptions about the number of deaths averted and avoided years of life lost due to the project's interventions. The calculation used the total project costs (US$ 80 million) as the cost under each scenario, a discount rate of 3%, and reasonable assumptions about reductions in maternal and child mortality. The PAD (p. 61) noted that this analysis likely underestimated the project's benefits, as it excluded averted deaths other than maternal and child deaths, and it did not take into account benefits from reductions in morbidity and out-of-pocket expenditures. The ICR's analysis (Annex 4) acknowledged that attribution of benefits to the project was complicated by the ongoing implementation of a comprehensive health care reform in the country, launched in 2009. Repeating the PAD's benefit-cost analysis, it found an economic benefit of US$ 126 million accruing from reduction of infant mortality (assuming that the project resulted in a 25% reduction in infant deaths, or 660 averted infant deaths from 2013-2017), and a further benefit of US$ 1.8 million from reduction in maternal mortality (assuming that the project was responsible for 18 averted maternal deaths over that time period), all based on an average salary of US$ 300/month. According to the ICR (p. 17), this analysis underestimated the project's benefit by capturing only the benefits from reduced infant and maternal mortality; reduced morbidity, out-of- pocket expenditure, and benefits from the new cancer enter and ambulance fleet were not quantified and taken into account. The ICR added an extensive analysis of the additional project benefits accruing from Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) cancer-related investments, including projected demand for cancer treatment services through 2026 and the estimated cost of providing those services. No resulting benefit-cost ratio was calculated. The project efficiently coordinated activities with IADB, whose parallel project pursued similar objectives but targeted different municipalities. The ICR (p. 22) also cited close collaboration with the Pan American Health Organization and the Spanish Agency for Development Cooperation in terms of complementary financial support and "enhanced activities and interventions," but no details were provided. There was "very low" turnover of staff within MINSAL and implementing units, and only one Bank task team leader across the project's lifetime, allowing for continuity (ICR, p. 22). Financial management was "efficient" (ICR, p. 25). There were moderate shortcomings. The project was approved in July 2011 but did not become effective until December 2012, requiring the Bank to approve two effectiveness extensions. The ICR did not provide the reasons for this delay. During the extensions, some of the activities included in the original procurement plan were carried out with funds from the parallel IADB-financed project. As a result, by the time of effectiveness of this project, the implementation plans were outdated, requiring a re-assessment of activities to be funded by the Bank. According to the ICR, p. 21), "it took some time for MINSAL to offset the time lost during the initial phase of the project." Procurement challenges persisted until early 2018 (ICR, p. 25; also see Section 10b). Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives were highly relevant to country context and current government and Bank strategy. Achievement of the project's objectives is rated substantial. Efficiency was substantial, with moderate shortcomings in implementation efficiency. These ratings indicate minor shortcomings overall, producing an Outcome rating of Satisfactory. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR (pp. 26-27) noted that the central, regional, departmental, and municipal levels of health care were strengthened through the project. Key health interventions will continue to be implemented through the National Directorate of NCDs, the qualification of the National Reference Laboratory as Level 2 for biosecurity, integration of health information systems, increased capacity for repair and maintenance of equipment, and increased capacity to respond to medical emergencies. Service provision was strengthened through training at multiple levels. Ongoing maintenance for some of the project's major investments, especially the construction of the National Radiotherapy Center, have been incorporated into the government budget. No immediate threats to the sustainability of these achievements were identified. 8. Assessment of Bank Performance a. Quality-at-Entry Key lessons from other project experiences in the country were incorporated into project design, including the importance of early strengthening of the fiduciary capacity of the PCU, and the key role to be played by a range of local stakeholders. The PCU was to manage both this project and the concurrent IADB- financed project. The most critical identified risks at appraisal included the possibility that lack of coordination with other development partners could result in duplication or gaps, and the risks to performance stemming from MINSAL's relatively weak capacity and limited experience with Bank operations. Mitigation measures were well outlined in the PAD (Annex 4). Financial management risks were identified and led to requirements for annual financial audits, external technical reviews, semi-annual Interim Financial Reports, financial management training, and plans for close supervision. A similarly detailed mitigation plan was agreed to address procurement risks following a procurement capacity assessment conducted in July 2010 (PAD, Annex 3). The ICR stated (p. 20) stated that project design was based on thorough background analysis of health conditions, government priorities, and constraints in the national context and in the project's intervention areas, but no specific information was provided. Important risks were identified but underestimated. The risk of delays in initial project implementation was identified during preparation (see the aide-memoire of June 10-17, 2010, cited in the ICR, p. 20), but it was not included in the project's risk assessment framework. In addition, the risk of ongoing violence to project Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) implementation was not explicitly considered, despite this challenge having been discussed in the PAD. There were also important shortcomings in the M&E framework (see Section 9a). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision In addition to seventeen supervision missions, the Bank team conducted weekly follow-up meeting to monitor implementation and identify bottlenecks, in particular challenges related to procurement. The ICR (p. 26) stated that the team exhibited well-balanced and effective teamwork, with strong coordination across staff at headquarters and in-country as well as specialized consultants. Environmental safeguards were well monitored. Restructurings were implemented in a timely manner to accommodate necessary course corrections. However, shortcomings in the results framework were never fully addressed, making the assessment of project achievements challenging. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project's results framework contained four key outcome indicators: percentage of poor people using public health care services when needed, percentage of MINSAL-registered pregnant women receiving prenatal care according to MINSAL norms, percentage of eligible secondary and tertiary hospitals certified by MINSAL, and percentage of pharmaceutical products purchased by MINSAL through a decentralized mechanism subject to quality control by a MINSAL laboratory. Baselines and targets were set for all outcome and intermediate indicators (PAD, Annex 1), and the indicators, including numerators, denominators, and key terms were carefully defined for each indicator. However, the ICR (p. 22) noted that these indicators were "overly ambitious and difficult to measure," and the given indicators were only a partial reflection of the objectives. MINSAL was responsible for M&E, with project indicators tracked and analyzed through established MINSAL systems. Other key data instruments were to include an external technical review, statistics from MINSAL's national epidemiological surveillance, household surveys focused on demand for health services and financial barriers to access, annual health facility surveys, baseline and impact evaluation analysis, and a mid-term Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) evaluation report jointly reviewed by the Bank and MINSAL (PAD, pp. 11-12). The project's second component was to provide strategic investment to capacity development for supervision and monitoring. b. M&E Implementation The indicators were revised at the project's first restructuring (February 2015). The new indicators on vaccine coverage and institutional deliveries were effective measures of coverage of maternal and child health services. However, there were no indicators to measure coverage of other kinds of health services, and the two new indicators did not measure improvements in quality or equity in utilization of services. The ICR (p. 23) acknowledged this shortcoming, noting, for example, that "good data on how often health workers are present and how accurately they diagnose and treat patients is not available in the country yet." Similarly, the indicator on hospital compliance with standard bio-hazard waste management systems was too narrow to measure comprehensively the objective to enhance MINSAL's stewardship capacity to manage essential public health functions. The ICR (p. 23) stated that MINSAL and the Bank systematically reported on key indicators. Results were analyzed using MINSAL's institutional statistics and monitoring systems. Semi-annual reports were shared with Bank staff. A mid-term review was completed in October of 2015. A significant percentage of loan proceeds was allocated to the development of the National Radiological Laboratory, which did not become operational until a few days before the project closed. It was therefore difficult to measure outcomes resulting from this investment. The ICR did not specify whether the planned impact evaluation was conducted. c. M&E Utilization According to the ICR (p. 23), information obtained from the project's M&E framework and from project surveys were used to inform project management and decision making, but no specific information was provided. In addition, no information was provided on whether or not the planned impact evaluation was conducted. M&E Quality Rating Modest 10. Other Issues a. Safeguards Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) The project was classified as environmental assessment category C, but it triggered safeguard policies OP/BP 4.01 (Environmental Assessment) and OP/BP 4.10 (Indigenous Peoples). Although the project was to support small-scale rehabilitation of existing health care facilities, it was not expected to generate adverse environmental impacts. An environmental assessment was conducted at appraisal, and an Environmental Management Plan, satisfactory to the Bank, was developed by MINSAL and disclosed in March 2011 (PAD, p. 15). Environmental Diagnostics developed in the country's 30 national hospitals facilitated compliance with safeguard policies (ICR, p. 24). The Directorate of Environmental Health facilities the identification and implementation of necessary measure to ensure adequate management of hazardous hospital waste. 100% of hospitals now have properly functioning environmental committees and are complying with policies on treatment of solid waste. The National Radiological Center is in compliance with national environmental legislation. An Indigenous Peoples' Planning Framework (IPPF) was prepared and also disclosed in March 2011. Its recommendations included provision of appropriate training for MINSAL staff, improvement of an informed consultation process, the implementation and monitoring of an action plan through which indigenous people were to receive culturally sensitive and appropriate health services, and mid-term and final evaluations of the project's social impact. However, the project did not produce the recommended plans, because there were no areas within the 92 project municipalities that met the four criteria of OP/BP 4.10 (ICR, p. 25). b. Fiduciary Compliance Procurement: The ICR (p. 21) reported that the project faced "numerous procurement challenges" that were largely resolved beginning in 2015. At the start of implementation, MINSAL's limited procurement capacity was a "key implementation obstacle." The ICR (p. 21) also noted "occasional protracted internal processes" and "the Bank's protracted review of non-objections" as key factors that "often slowed project progress." Bank support to improve the situation over the project's first two years was inadequate. From 2015 forward, through cooperative dialogue, the implementation of a procurement mitigation strategy developed by the Project Implementation Unit and agreed by the Bank, training of MINSAL technical units and suppliers, and "intensive procurement support and supervision,'" procurement guidelines were eventually aligned with national procurement laws and processes improved. A February 2017 procurement post review found improvement, but still internal delays and incomplete documentation and files; these remaining deficiencies were found to have been mostly addressed during a subsequent February 2018 procurement post review. Financial management was rated Satisfactory throughout the project's lifetime. Implementing staff had experience managing the financial aspects of internationally financed projects and had been previously trained on Bank policies and procedures. Interim Financial Reports and financial audit reports were prepared and delivered in a timely manner and were found to be acceptable. Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk assessment at appraisal did not take into account Moderately several acknowledged risks. Bank Performance Satisfactory Satisfactory Shortcomings in the results framework were never fully addressed. The results framework was not adequate to assess overall achievement of the project's Quality of M&E Substantial Modest objectives. The main text of the ICR (para 63) rated the quality of M&E as "moderate." Quality of ICR Modest --- 12. Lessons The ICR (p. 27) offered several useful lessons, including: World Bank financial support can sustain priority government agenda through periods of crisis. In this case, the negative impact of the 2009 global financial crisis threatened the social sector gains El Salvador had made in the preceding decade. Bank support enabled continued attention to coverage and quality of health services to the urban poor. Risk assessments will not produce adequate mitigation measures if they do not realistically take likely challenges into account. In this case, even though the risks posed by gang violence in targeted areas and low procurement capacity were identified in appraisal documents, they were not acknowledged and dealt with in the formal risk assessment framework. As these risks materialized, the project was not as equipped to cope with their impact as it might have been with more in-depth assessment and planning. Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SV Strengthening Public Health Care Syst (P117157) IEG offers one additional lesson: In the absence of key outcome indicators effectively measuring a project's development objectives, assessment of achievements and outcomes is problematic. In the case of this project, although equity concerns were embedded in the project's selection of intervention areas, there were no actual measures of the project's impact on improved equity in the utilization of health services. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR, while concise, lacked concrete evidence to substantiate claimed outcomes. It did not appropriately assess, with data on results, every outcome promised in the project's statement of objectives. The ICR did not go beyond the project's results framework to report outcome data adequate to assess achievement of the objectives. It reported that the project conducted at least two key surveys, but information from these surveys was not provided to assist in the assessment of project achievements. The ICR also did not provide a complete account of project interventions that made it possible to assess the project's likely impact on reported outcomes. As a result of these shortcomings, the project team had to provide additional information to support assessment of the project's efficacy. The Borrower's ICR contained important detail on outputs that was not included in the main ICR. The ICR was occasionally repetitive; for example, paragraphs 38 and 39 are virtually identical (pp. 16-17). Some ratings were internally inconsistent (for example, project efficiency is rated as substantial in the text of para 42, but high in the table immediately thereafter). The efficiency analysis did not carry through its calculations to a full benefit-cost result. a. Quality of ICR Rating Modest Page 18 of 18