Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004800 IMPLEMENTATION COMPLETION AND RESULTS REPORT 5258-PK ON A CREDIT IN THE AMOUNT OF SDR 66.8MILLION (US$100 MILLION EQUIVALENT) TO THE ISLAMIC REPUBLIC OF PAKISTAN FOR THE Punjab Health Sector Reform Project (P123394) June 26, 2019 Health, Nutrition & Population Global Practice South Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective, May 31, 2019) Currency Unit = Pakistan Rupee (PKR) PKR 146.84 = US$1 US$1 = SDR 0.73 FISCAL YEAR July 1 - June 30 Regional Vice President: Hartwig Schafer Country Director: Patchamuthu Illangovan Senior Global Practice Director: Timothy G. Evans Practice Manager: E. Gail Richardson Task Team Leader(s): Tayyeb Masud ICR Main Contributor: Maria E. Gracheva ABBREVIATIONS AND ACRONYMS BHU Basic Health Unit CM Community midwife CMS Complaint Management System CPR Contraceptive Prevalence Rate CPS Country Partnership Strategy DALY Disability Adjusted Life Years DFID Department for International Development DGHS Directorate General Health Services DHIS District Health Information System DHQ District Headquarter Hospital DLI Disbursement Linked Indicators DoH Department of Health EEP Eligible Expenditure Program EmONC Emergency obstetric neonatal care EPHS Essential Package of Health Services EMWMP Environmental and Medical Waste Management Plan FP Family Planning GoPb Government of Punjab HCE Health Care Establishment HCWM Health Care Waste Management HFA Health Facility Assessment HIV Human Immunodeficiency Virus HR Human Resources HRITF Health Results Innovation Trust Fund IDA International Development Association IDU Intravenous Drug Users IE Impact Evaluation IMR Infant Mortality Rate IO Intermediate outcome IRMNCH Integrated reproductive maternal neonatal and child health IVA Independent verification agent LHW Lady Health Workers MCH Maternal and Child Health MDGs Millennium Development Goals MICS Multiple Indicator Cluster Survey MMR Maternal Mortality Ratio MSDS Minimum Service Delivery Standards MTR Mid-term Review MNCH Maternal Neonatal and Child Health NNS National Nutrition Survey PAD Project Appraisal Document PDHS Pakistan Demographic and Health Survey PDO Project Development Objective PHC Primary Health Care PHCC Punjab Healthcare Commission PHF Punjab Health Foundation PHSRP Punjab Health Sector Reform Program PHSS Punjab Health Sector Strategy PITB Punjab Information Technology Board PSPU Policy and Strategic Planning Unit PWD Population Welfare Department QER Quality Enhancement Review RBF Results-Based Financing RF Results Framework RH Reproductive Health RHC Rural Health Center RMNCH Reproductive maternal neonatal child health SAM Severe Acute Malnutrition TA Technical Assistance THQ Tehsil Headquarter Hospital WB World Bank TABLE OF CONTENTS DATA SHEET .......................................................................................................................................1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ...................................................................6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .....................................13 II. OUTCOME ................................................................................................................................17 A. RELEVANCE OF PDOs ............................................................................................................17 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................18 C. EFFICIENCY ...........................................................................................................................28 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................31 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................31 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME.............................................32 A. KEY FACTORS DURING PREPARATION ...................................................................................32 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................34 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ............... 37 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................37 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................39 C. BANK PERFORMANCE ...........................................................................................................41 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................43 V. LESSONS AND RECOMMENDATIONS .........................................................................................44 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................................46 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................................60 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................................62 ANNEX 4. EFFICIENCY ANALYSIS .......................................................................................................63 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ................. 66 ANNEX 6. SUPPORTING DOCUMENTS ...............................................................................................76 The World Bank Punjab Health Sector Reform Project (P123394) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P123394 Punjab Health Sector Reform Project Country Financing Instrument Pakistan Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Policy and Strategic Planning Unit (PSPU), Department of Government of Pakistan: Economic Affairs Division Health, Government of Punjab, Multi Sector Nutrition Celll, IRMNCH Project Development Objective (PDO) Original PDO The development objective of the proposed project is to support the implementation of the Punjab Health Sector Strategy, by focusing on the improvement of the coverage and utilization of quality essential health services, particularly in the low performing districts of Punjab. The project will focus on building the capacity and systems to strengthen accountability and stewardship in DoH. Page 1 of 76 The World Bank Punjab Health Sector Reform Project (P123394) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 100,000,000 71,097,250 61,744,680 IDA-52580 20,000,000 0 0 TF-15283 Total 120,000,000 71,097,250 61,744,680 Non-World Bank Financing 0 0 0 Borrower/Recipient 830,000,000 0 0 UK: British Department for International Development 165,000,000 0 0 (DFID) Total 995,000,000 0 0 Total Project Cost 1,115,000,000 71,097,250 61,744,680 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 31-May-2013 17-Jan-2014 13-Aug-2015 31-Dec-2017 31-Dec-2018 Page 2 of 76 The World Bank Punjab Health Sector Reform Project (P123394) RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-Jan-2014 0 15-Feb-2017 27.29 Change in Implementing Agency Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Cancellation of Financing Change in Financing Plan Reallocation between Disbursement Categories Change in Disbursements Arrangements Change in Legal Covenants Change in Institutional Arrangements Change in Financial Management Change in Procurement Change in Implementation Schedule 28-Dec-2018 46.97 Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Moderately Unsatisfactory Moderately Unsatisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 03-Sep-2013 Satisfactory Moderately Satisfactory 0 02 16-Feb-2014 Satisfactory Moderately Satisfactory 0 03 16-Aug-2014 Satisfactory Moderately Unsatisfactory 0 04 17-Feb-2015 Moderately Satisfactory Moderately Unsatisfactory 1.00 05 27-Aug-2015 Moderately Satisfactory Moderately Satisfactory 22.07 06 16-Feb-2016 Moderately Satisfactory Moderately Satisfactory 27.29 Page 3 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 07 02-Sep-2016 Moderately Satisfactory Moderately Satisfactory 27.29 08 23-Feb-2017 Moderately Satisfactory Moderately Unsatisfactory 27.29 09 14-Sep-2017 Moderately Satisfactory Moderately Satisfactory 45.98 10 10-Jun-2018 Satisfactory Moderately Satisfactory 46.97 Moderately 11 28-Dec-2018 Moderately Unsatisfactory 46.97 Unsatisfactory Moderately 12 18-Apr-2019 Moderately Unsatisfactory 45.63 Unsatisfactory SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Public Administration - Health 57 Health 43 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Disease Control 6 HIV/AIDS 6 Health Systems and Policies 84 Health System Strengthening 76 Reproductive and Maternal Health 6 Child Health 2 Nutrition and Food Security 10 Nutrition 5 Food Security 5 Page 4 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ADM STAFF Role At Approval At ICR Regional Vice President: Isabel M. Guerrero Hartwig Schafer Country Director: Rachid Benmessaoud Patchamuthu Illangovan Senior Global Practice Director: Jesko S. Hentschel Timothy Grant Evans Practice Manager: Julie McLaughlin E. Gail Richardson Task Team Leader(s): Inaam Ul Haq Tayyeb Masud ICR Contributing Author: Maria E. Gracheva Page 5 of 76 The World Bank Punjab Health Sector Reform Project (P123394) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context Pakistan is the world’s sixth most populous country which had a population of 180 million in 2011 when the project preparation began and was on track to become the 5th most populous country in the world by 2030. Its population grew by 47% since the 1998 census. Its per capita income was US$1,120 in 2011 and there was a sharp decline in economic growth in 2011 and 2012 following a rise in oil and food prices as well as recurring natural disasters (2010-2012 floods). The fiscal deficit reached about 8% of GDP by 2012. Pakistan’s progress in human development was slow and the country was not likely to achieve Millennium Development Goals (MDGs) and was lagging behind its neighboring South Asia countries, with only Afghanistan having worse maternal and child health indicators. The Punjab Province is the most populous province in Pakistan, representing roughly 60% of the total population. As with the rest of Pakistan, economic growth was declining at the time of project appraisal, with weak fiscal management and inefficient use of public resources. Punjab’s overall health outcomes saw slow improvements, but with significant disparities among rural and urban areas and by economic status. There were particularly high levels of disparity in child mortality, as related to mothers’ educational attainment. Infant mortality rate (IMR) and under-five mortality rate (U5MR) in mothers with low education attainment were 3-4 times higher than for those with higher education. Likewise, there were considerable disparities in health service utilization, with districts in southern Punjab performing the worst in the province. Punjab also faced a concentrated epidemic of HIV, with 38% prevalence among injecting drug user [IDUs], female sex workers, male sex workers and transgender sex workers) with the potential to spread to the general population. Prevalence in major cities was extremely high: Lahore (30.8%), Faisalabad (52.5%), Sargodha (40.5%) and DG Khan (49.5%). With respect to maternal and child health and nutrition outcomes, malnutrition was a main contributing factor in child mortality and morbidity with 39% of children under 5 stunted, according to the 2011 National Nutrition Survey. Immunization coverage was also low, with 34.6% of children 12-23 months being fully immunized. The maternal mortality ratio (MMR) was 227 per 100,000 live births, with 12,000 women dying in pregnancy every year (the 4th highest in the world), with only 59% of births attended by skilled personnel and 38% of women receiving postnatal care from skilled personnel. A major concern was also the very low use of modern contraceptive methods (29%). While little information existed on quality of healthcare services (with no province-wide assessment carried out), low utilization of public sector services was a proxy for quality. Waste management in hospitals and other health facilities faced significant issues with only 1/3 having health care waste management teams in place and only 12% of facilities applying guidelines. Total public health expenditure as a share of GDP was only 0.6% in 2010/11 with Government financing of health expenditures at 23%, with over 75% of health spending coming from households, mainly out-of-pocket expenditures. In 2012, at district level, development spending accounted for only 3% of the district health budget, with 75% spent on salaries with little left for commodities, supervision and maintenance costs. Page 6 of 76 The World Bank Punjab Health Sector Reform Project (P123394) The project appraisal document (PAD) identified a number of underlying issues for the inadequate performance of the health sector, including external factors such as ongoing political instability, conflict, crises, gender inequality, low levels of education, high levels of poverty, and inadequate water and sanitation systems, all mainly in the southern part of the province. The internal health sector performance factors included: (i) weak management, particularly at the district level due to capacity constraints, lack of a results-based culture, and limited emphasis on accountability, with lacking incentives to achieve results, particularly for health staff working in rural districts; (ii) lack of Department of Health (DoH) organizational structures to undertake its stewardship functions; (iii) limited use of data generated from information system and household surveys for planning and management, and accountability, despite the availability of an adequate quantity of data; (iv) low levels of public expenditures from provincial resources on health in general, and specifically towards the provision of essential health care; (v) weak formal accountability mechanisms in the government health sector and absence of appropriately delineated processes, with DoH’s oversight role in quality control and regulation being mostly absent for the public and private sectors; and (vi) low levels of participation by service users and civil society in the health sector. The existing grievance redressal system had not served citizens well. Government Sector Strategy: Recognizing its weak performance in the health sector, in 2012, the DoH developed the Punjab Health Sector Strategy (PHSS) 2012-2020 to fulfill its constitutional obligations following the 2010 18th amendment to the Constitution which devolved responsibility for the health sector to the provinces. This meant that the DoH needed to establish its priorities for the delivery of healthcare services, consistent with available resources. The development partners were encouraged to align their support with the PHSS. The PHSS was based on an extensive Situation Analysis (March 2012) and a consultative process involving all stakeholders using internationally accepted six building blocks of the health system as a framework (see Figure 1 below). Figure 1: PHSS Strategy Framework Page 7 of 76 The World Bank Punjab Health Sector Reform Project (P123394) The PHSS envisaged a complete overhaul of the existing health system in order to improve health outcomes, particularly for the poor, using a result-based approach with reforms aimed at improving management, governance, accountability, and DoH stewardship functions. This was to include a reorganization of the Director General Health Services (DGHS) to enable it to perform its mandated responsibilities of a lead technical agency providing oversight of all levels of care in both the public and private sectors, providing guidance on emerging medical needs, conducting evaluations of ongoing interventions and advising on future system requirements. The service delivery function, including management, supervision and monitoring would be devolved to the District Health Authorities. The PHSS placed a special focus on the delivery of the Essential Package of Health Services (EPHS) at the primary healthcare level as well as improving quality of care through the adoption of service standards and integration of programs. As part of a results-based orientation of the PHSS, it envisaged revisiting its health service delivery modes by trying different management approaches, including contracting out or performance-based management. Also, critical for improvements in the maternal mortality and morbidity, the PHSS aimed to integrate the Maternal, Neonatal and Child Health (MNCH), Nutrition, and Lady Health Worker programs with 24/7 initiatives to be implemented in Basic Health Units (BHU), as well as focusing on MNCH in Rural Health Centers (RHC), Tehsil and District Headquarter Hospitals (THqs, and DHQs) and through the community midwife program. Theory of Change (Results Chain) The project included an illustrative Results Chain in the PAD. However, for this ICR, a Theory of Change had to be developed, taking into consideration the PDO, component descriptions, the Results Framework (RF), and PHSS reform milestones, some of which were selected as disbursement-linked indicators (DLIs) under the project. All this information was used to derive activities, outputs and outcomes, as presented below in Table 1. Long-term outcomes of the project would be overall improved health sector performance and attainment of MDGs 1 (eradicate extreme poverty and hunger), 4 (reduce child mortality, 5 (improve maternal health) and 6 (combat HIV/AIDS, Malaria and other diseases). Table 1: Theory of Change Activities Outputs (Intermediate outcome indicators & DLIs) Outcomes (PDO indicators & DLIs) Component 1: Improve Health -Approved PC-1 for integrated management of (a) Improved coverage and utilization of quality Service Delivery reproductive health, primary health care and essential package of health services (EPHS at nutrition; (b) LHW program and (c) MNCH. (DLI1) PHC level), -Integrate management of MCH, -291 RHCs providing basic EmONC (IO) LHW and Nutrition programs (DLI -60 deliveries in one RHC per month on average (IO) Coverage, as measured by: 1) -100 family planning clients provided products & -60% of children 12-23 months of age fully -Expansion of 24/7 comprehensive services in on RHC on average (IO) immunized (PDO); emergency obstetrical and -60% of pregnant women registered with LHW -60% of children 0-24 months of age receiving neonatal care (EmONC) receiving IFA tablets during pregnancy the basic package of nutrition services (PDO) -Provision of nutrition services -60% of children identified with SAM in 12 under- Utilization, as measured by: -Train 15,000 LHWs on family performing districts registered for treatment (DLI4) -35% achievement in contraceptive prevalence planning and nutrition (DLI 3) – (IO has a target of 40%) rate (any modern method) (DLI2i and PDO) -6 HIV prevention service contracts -Expanded coverage of HIV/AIDS services -75% IDUs Quality, as measured by: with NGOs signed (DLI5) report using new syringe at last injection. (IO) -70% of births attended by skilled health -75 THQ hospitals providing -24/7 comprehensive personnel; (DLI2ii & PDO) EmONC services (IO) Component 2: Enhance Efficiency Improved health system performance and Effectiveness of the Health PHC contracting out system put in place System -Increased average number of daily out-patient • PHC contracting out implemented (DLI6) -DOH and PRSP sign results-based visits in BHU (60) and RHC (180) (IO) • Results-based district management Page 8 of 76 The World Bank Punjab Health Sector Reform Project (P123394) contracts in 14 districts (DLI6) -management contracts implemented (DLI7) Results-based District management established -80% of low performing districts have attained -DOH and District Health Offices -85% districts where BHUs provide all essential a minimum increase of 5/10 percentage points sign performance management drugs (IO) in the composite index (scorecard) on the list contracts in 36 districts (DLI7) -10% LHWs with stock-outs of family planning of key performance indicators (DLI8) methods (IO) • Enhanced facility functionality (HFA results) • Enhanced governance to improve quality of -Governance and accountability -Punjab Healthcare Commission operational (for services mechanisms put in place regulation/licensing of health facilities) -500 Category -1 Health Care Establishments (regulatory reforms and social (HCE) issued with provisional licenses accountability measures) -Complaint management system in place (certificate of registration) by Punjab -Annual Health facility assessments and user Healthcare Commission (DLI9 and PDO) satisfaction surveys carried out • Enhanced accountability mechanisms -Complaint management system in place (DLI10) -50% of community satisfied with government health care services (as part of Health Facility Assessment) (DLI11 and PDO) Component 3: Strengthen -Annual Report published (IO) Strengthened Provincial Stewardship & provincial DOH management -20% of development budget allocated for Management Capacity capacity preventive programs of the total development -DGHS restructured; establish a health budget (IO) • Strengthened stewardship through contract management unit (DLI12) -At least 30% of total recurrent expenditure is for reorganization of DGHS (DLI12) -DOH established 2 cells – FM and non-salary items in all 36 districts (DLI14 and IO) • Strengthened fiduciary functions (DLI 13 & DLI procurement (DLI13) -PHF capacity strengthened to enable it to channel 14) -Restructure Punjab Health public financing for the private sector • Strengthened capacity of the Punjab Health Foundation (PHF) -At least 2 secondary level facilities adopted and Foundation (PHF) -Infection control protocols implemented the Medical Waste Management Plan • Strengthened environmental and medical implemented (DLI15) waste management (DLI15) Component 4: Improve capacity in • Enhanced policy development capacity technical areas for equitable (based on results from innovations) and health services for all -RBF pilots carried out strengthened analytical capacity in technical -Implement RBF pilots in 2 districts -Pilots of alternative financing models implemented areas (part of the stewardship PDO above) -Pilot alternative financing models (voucher/cash incentive at health facilities and -Provision of TA to develop insurance/social protection scheme with premiums secondary and tertiary care for different services) packages -Analytical and operational work carried out -Impact evaluation of results-based -Impact evaluation carried out to measure impact financing pilots of different RBF pilots under the project Project Development Objectives (PDOs) The PDO was to support the implementation of the Punjab Health Sector Strategy, by focusing on the improvement of the coverage and utilization of quality essential health services, particularly in the low performing districts of Punjab. The PAD also included a “contributing objective” that the project will focus on building the capacity and systems to strengthen accountability and stewardship of the DoH. However, the Financing Agreement did not include that objective although it was a central tenet of the PHSS and the project. Key Expected Outcomes and Outcome Indicators The success of the project was to be measured by the following six outcome (PDO) indicators: • Percentage of fully immunized children 12-23 months of age (incl. in low performing districts) - coverage • Percentage of births attended by skilled health personnel (incl. in low performing districts)- quality Page 9 of 76 The World Bank Punjab Health Sector Reform Project (P123394) • Contraceptive prevalence rate (any modern method) (incl. in low performing districts) – utilization • Proportion of children 0-24 months of age receiving the basic package of nutrition services - coverage • Number of Category -1 Health Care Establishments issued with provisional licenses (certificate of registration) by Punjab Healthcare Commission – improved health system performance • Community satisfaction with government health care services - improved health system performance In addition to the above indicators, the RF included 13 intermediate outcome (IO) indicators to track improvements in service delivery, performance, governance, and accountability. Components The project included four components, with the first three financed through 15 DLIs (total estimated US$85 million). Once DLIs were achieved and verified by an independent agent, the Bank would reimburse the GoPb the DLI amount against a defined set of Eligible Expenditure Program (EEP) costs incurred by the GoPb (budget number PC21016 which included salaries, allowances and related health sector staff costs). The project also included Component 4, financed using the traditional investment financing reimbursement modality (US$15 million) through a Designated Account. In addition, the project was co-financed by US$21.5 million grant from the Health Results Innovation Trust Fund (HRITF) to support implementation of Components 2 (US$10 million) and 4 (US$11.5 million). The total project cost was US$121.5 million of which US$61.7 million was disbursed by project closing. The UK Department of International Development (DFID) was to finance its own parallel program (2013-2019), with financing of GBP90 million (est.US$140 million) which started in April 2013 when the Bank-financed project was also almost approved. DFID’s program aimed at reducing child and maternal mortality by focusing on three priorities: to improve care around delivery, immunization coverage, and to transform primary health care facilities. These clearly overlapped with World Bank-financed project objectives as part of a concerted effort in support of PHSS to improve health outcomes. DFID support was province-wide and financing would be provided through annual or semi-annual tranches based on the achievement of agreed milestones (DLIs) which would be part of a framework of results including inputs, outputs, processes, and outcomes. Technical assistance (est.US$25 million) was to supplement service delivery financing to enhance health sector capacity, develop innovative and evidence-based policies related to reproductive maternal neonatal child health (RMNCH) and nutrition. A technical assistance firm (Technical Resource Facility) was hired to plan and deliver results in close coordination with provincial and district agencies. DFID released its first tranche in 2013 (GBP14 million) and subsequently provided its support as part of the “Health Roadmap” approach, launched in 2014, of targeted actions tracked through bi-monthly stocktaking meetings using a “traffic light” method. Component 1: Improving Health Service Delivery (original estimated cost US$28.34 million). This component was to enhance coverage, quality and utilization of an essential package of health care services (EPHS), including outreach and community level interventions for PHC. EPHS includes nutrition, maternal, neonatal and child health, immunization, family planning, communicable diseases (malaria and TB) control, including service provision at community level through lady health workers (LHW), community midwives (CM), and vaccinators. In addition, the component was to support the development of EPHS for secondary care. The following specific thematic areas were to be supported: (i) Integrated MNCH and LHWs Programs and the expansion of 24/7 basic and comprehensive EmONC and family planning services in 16 districts; (ii) Introduction of preventive Nutrition Services in all 36 districts by scaling up provision of micronutrients and deworming, expanding the delivery of behavior change communication, focusing on the prevention of Page 10 of 76 The World Bank Punjab Health Sector Reform Project (P123394) malnutrition during pregnancy and in the first two years of life. In 12 districts and urban slums in 9 cities with the highest prevalence of acute malnutrition, therapeutic nutrition services would be delivered, including commodity support micronutrients and Ready-to-Use Therapeutic Foods; and (iii) Expansion of coverage of HIV/AIDS preventive services for population subgroups vulnerable to HIV infection [IDUs and Men Who Have Sex with Men (MSM)] in targeted cities; and support provincial integrated biological and behavioral surveillance. This component was to be financed through 5 DLIs, each worth US$5.67 million. • DLI1: Punjab has operationalized the integrated management of three community – based programs (LHW, MNCH, and Nutrition programs) and approved PC-1s for integrated management for a) Reproductive health (RH), primary health care, and nutrition; b) LHW program, and c) MNCH. • DLI2: Punjab has attained: (i) at least 35% in the use of modern contraceptive methods; and (ii) at least 70% skilled birth attendance. • DLI3: 15,000 LHWs trained on family planning (FP) and nutrition. • DLI4: At least 60% of children identified with Severe Acute Malnutrition (SAM) in 12 under-performing districts registered for treatment. • DLI5: 6 HIV/AIDS service contracts with NGOs. Component 2: Enhancing Efficiency and Effectiveness of the Health System (original estimated cost US$44 million of which IDA Credit was US$34.00 million, and HRITF Grant was US$10 million). This component aimed to enhance efficiency and effectiveness of the health system by strengthening management and accountability and improving quality of care through regulations and standardization of services. The following thematic areas were included: (i) Strengthening PHC Contracting Out. This was to support an ongoing initiative of GoPb contracting out of BHUs in 14 districts to the Punjab Rural Support Program (PRSP) to manage BHUs provision of the EPHS for primary health care services. The contract with the PRSP would be revised into a results-based contract linking payments to achievement of district-wide annual performance targets. The revised contract would include health facility management, as well as ensuring that the full EPH services are provided, including community- based services. (ii) Results-Based District Management Contract. This was to build on initial efforts for performance management by formalizing results-based performance management contracts in all 36 districts between the provincial DoH and the District Health Offices. The contracts would include performance indicators related to the delivery of EPHS, including integrated MNCH/FP, nutrition services, and control of communicable disease. The districts would receive a results-based payment based on their performance, using data from the District Health Information System (DHIS) and independent assessments. (iii) Enhancing Governance and Accountability Mechanisms. This was to support: (a) Regulatory reforms including operationalizing the Punjab Healthcare Commission (PHCC) which was established through GoPb legislation to regulate health care services in the province. This would be done by operationalizing health facility registration and licensing of healthcare establishments in line with standards thereby improving quality of care in the public and private sectors; and (b) Strengthening social accountability through empowerment of communities by third-party validation of results through Regular Health Facility Assessments and household surveys; data dissemination for greater accountability; community-based monitoring/auditing; and strengthening complaint mechanisms. This component was to be financed through 6 DLIs, each worth US$5.67 million and the HRITF grant. Page 11 of 76 The World Bank Punjab Health Sector Reform Project (P123394) • DLI6: DoH has entered into revised results-based contracts with PRSP in line with EPHS for primary health care, in 14 ongoing & new districts. • DLI7: DoH has entered into performance management contracts with all District Health Offices and has transferred budgetary resources therewith. • DLI8: 80% of low performing (below the mean) districts of Punjab (18), identified by DoH, have individually attained a minimum increase of 5/10 percentage points in the composite index (scorecard) based on the list of Key Performance Indicators, measured against the 2012 baseline. • DLI9: At least 200 Category -1 HCEs are issued with provisional licenses (certificate of registration) by the Punjab Healthcare Commission. • DLI10: PHC has operationalized a complaint management system. • DLI11: At least 50% of participants in the user satisfaction survey during June 2016 as part of the annual Health Facility Assessment expressed satisfaction with the government health services in Punjab. Component 3: Strengthening Provincial Department of Health Management Capacity (original estimated cost US$22.66 million). This component was to strengthen and reorganize the DoH management system and improving its stewardship functions. The following thematic areas were included: (i) Strengthening stewardship functions. The component was to support a functional review of the DoH and organizational restructuring of provincial DoH offices, by focusing on: (a) increasing the institutional capacity of the DGHS office to provide technical support to the districts; (b) supporting DoH policy and strategy roles and functions by strengthening PHSS Program; (c) strengthening the monitoring and evaluation cell in the DGHS office, improving the quality of DHIS and institutionalizing the independent annual health facility assessment along with Impact Evaluation supported by HRITF to monitor service delivery performance; and (d) building human resource (HR) management functions in the DoH by supporting the establishment of the HR cell with skilled staff, develop the HR strategy and the HR database for health, and prepare detailed plans for separating the management cadre from the general cadre; (ii) Restructuring the Punjab Health Foundation (PHF) to work with the private sector. The project was to support a review of the PHF’s mission and objectives to reorganize and strengthen the PHF’s capacity and restructure it to enable public financing for the private sector; (iii) Building fiduciary functions in the DoH by establishing financial management and procurement structures to improve internal control for greater accountability and management effectiveness, including: (a) establishing a procurement function in a separate and independent wing; (b) operationalizing sector specific Standard Operating Procedures (SOPs) and Manuals aligned with the Punjab Public Procurement Rules; (c) capacity building program for the field staff; (d) analysis of the supply chain management from manufacturer to patient to measure, improve, and monitor the processes to ensure quality provision of health care; (e) establishing an independent financial management unit; (f) strengthening budget management through capacity building; (g) enhancing non-salary district allocations; and (h) systems development for the internal audit; (iv) Environmental and medical waste management. The project was to support the roll out of the medical waste management in the two pilot districts to ensure zero infectious waste goes out of the medical facilities. The project would build on that experience to enable Punjab to expand the environmental and medical waste management and successful practices across the province. This component was to be financed through 4 DLIs, each worth US$5.67 million. • DLI12: DoH has restructured and reorganized the DGHS pursuant to the revised roles and responsibilities, including the establishment of a contract management unit. Page 12 of 76 The World Bank Punjab Health Sector Reform Project (P123394) • DLI13: DoH has established two cells that are financial management and procurement, at the provincial level. • DLI14: Non-salary budget items in all 36 districts; health recurrent expenditures represent at least 30% of total recurrent expenditures. • DLI15: At least 2 secondary health levels facilities in Gujranwala and Khanewal have adopted and implemented the Medical Waste Management Plan. Component 4: Improving Capacity in Technical Areas for Equitable Health Services for All (original estimated cost US$26.5 million of which IDA Credit was US$15.00 million, and HRITF Grant was US$11.5 million). This component was to finance innovative pilots to guide policy development in results-based financing and alternate financing approaches, as well as strengthening existing analytical capacities in technical areas. This component used the traditional investment lending financing modality to support the following: (i) Results-based financing pilots in two districts. (IDA US$5 million, HRITF US$5 million). This was to support incentives for health staff piloted in two districts to enhance performance and improvements in service quality. (ii) Piloting of alternative financing models. (IDA US$5 million, HRITF US$5 million for voucher/cash incentive scheme and IDA US$ 4 million for Insurance/social protection scheme). This was to support the design and pilots in two districts, including: (a) a voucher/cash incentive delivered at the health facility to increase demand for health services, using support from HRITF, and (b) an insurance/social protection scheme with varying premium levels for a set of services from the public or private sector; (iii) Technical assistance (TA). (IDA US$1 million). This was to support needs-based TA and capacity building activities, including: (a) development of a secondary care and tertiary care package; (b) analytical work and research on the use and quality of secondary care services, urban health care assessment, and a performance review of tertiary care hospitals; and (c) support capacity building activities to strengthen the DoH and its units’ ability to implement the project, including: financial management, procurement, environmental and social management, and monitoring and evaluation, and verification of DLIs using third-party assessment. (iv) Impact evaluation (IE) for Results Based Financing pilots. (HRITF US$1.5 million). The IE would measure the impact of different results-based financing approaches and would be managed by the Bank. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets The PDO was not revised although the project was significantly restructured 10 months prior to its Closing Date, in February 2017, which included dropping 8 of 15 DLIs, revision of 2 DLIs and the addition of 5 new DLIs. Revised PDO Indicators The RF (originally containing 6 PDO indicators and 13 IO indicators) was only slightly revised during the February 2017 restructuring by increasing targets for 5 PDO indicators, amending 1 PDO indicator and 1 IO indicator, and adding 2 new IO indicators on nutrition as a result of the addition of a new component 5 on nutrition. The rest of the RF remained intact. The revision of targets was made because the Closing Date was extended by 1 year as part of this restructuring and the end target dates were all extended until December 31, 2018 as well. Page 13 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Revised Components The components were revised during the February 2017 restructuring to essentially drop parts of the project supporting the wider reform agenda which were not being implemented, such as management and stewardship reforms under Components 2 and 3, including contracting out, results-based district management contracts, restructuring of the DGHS, fiduciary, HR and M&E. Instead, it was decided to focus the remaining short period of the project on some aspects of service delivery, particularly family planning and maternal health, and to expand the scope by adding a new Component 5 to provide nutrition interventions to reduce stunting in Punjab. As a result, original project component costs were amended substantially to accommodate these changes, as follows. Component 1: Improving Health Service Delivery (revised cost US$31.58 million; actual cost US$27 million) 2 DLIs (US$11.34 million) were dropped as they were not achieved. • DLI 2(i): at least 35% in the use of modern contraceptive methods; and • DLI4: At least 60% of children identified with SAM in 12 under-performing districts registered for treatment. 5 DLIs (US$11.5 million) were added, as follows: • Media campaign for FP services designed and implemented (US$1 million) • Punjab Population Innovation Fund (PPIF) fully operational (US$2.5 million) • PPIF awarded 5 contracts (US$2.5 million) • 90% of DHQs in 18 low performing districts providing Comprehensive EmONC services (US$2 million) • Increase in FP service intake at DoH facilities in 18 districts (US$3.5 million) Component 2: Enhancing Efficiency and Effectiveness of the Health System (revised cost US$10.8 million, HRITF Grant cancelled; actual cost US$10 million). 4 DLIs (US$22.68 million) were dropped as they were not achieved. • DLI 6: DoH has entered into revised results-based contracts with PRSP in line with EPHS for primary health care, in 14 ongoing & new districts. • DLI7: DoH has entered into performance management contracts with all District Health Offices and has transferred budgetary resources therewith. • DLI8: 80% of low performing (below the mean) districts of Punjab (18), identified by DoH, have individually attained a minimum increase of 5/10 percentage points in the composite index (scorecard) based on the list of Key Performance Indicators, measured against the 2012 baseline • DLI11: At least 50% of participants in the user satisfaction survey during June 2016 as part of the annual Health Facility Assessment expressed satisfaction with the government health services in Punjab. DLI10 was broken down into two parts (total cost US$5 million instead of US5.67 million): • Complaint management system in place in 33% of secondary care hospitals • Complaint management system in place in 66% of secondary care hospitals Component 3: Strengthening Provincial Department of Health Management Capacity (revised cost US$10.8 million; actual cost US$10 million). 2 DLIs (US$11.34 million) were dropped as they were not achieved. Page 14 of 76 The World Bank Punjab Health Sector Reform Project (P123394) • DLI12: DoH has restructured and reorganized the DGHS pursuant to the revised roles and responsibilities, including the establishment of a contract management unit. • DLI14: Non-salary budget items in all 36 districts; health recurrent expenditures represent at least 30% of total recurrent expenditures DLI15 was broken down into two parts (total cost US$5 million instead of US5.67 million): • At least 2 secondary health levels facilities in Gujranwala and Khanewal have adopted and implemented the Medical Waste Management Plan • Implementation of Medical waste management expanded to 15 districts Component 4: Improving Capacity in Technical Areas (revised cost US$9 million, HRITF Grant cancelled; actual cost US$1.74 million). This component was scaled down considerably, reducing the IDA amount by US$6 million and by cancelling the US$11.5 million HRITF support for results-based pilots and impact evaluation. The component was restructured to continue to support selected needs-based analytical work and research, and any innovative pilots to guide policy development. Some of the new activities included were: filter clinics for large hospitals; roll-out of health care waste management in tertiary care hospitals; and, evaluation of ongoing programs to formulate policy options. Furthermore, this component would provide support to the newly bifurcated departments of health to enable proper functioning and strengthening of the fiduciary and oversight responsibilities through strengthening of procurement and financial management capacity. The analytical work would include use and quality of secondary care services, urban health care, and a performance review of tertiary care hospitals. Component 5: Strengthening Nutrition Interventions (total estimated cost US37.82 million; actual cost US$13 million). This component was to exclusively focus on nutrition activities to address the persistently high burden of stunting and malnutrition in Punjab, building on nutrition interventions already provided under Component 1. Those included scaling up the provision of micronutrients and deworming, i.e., iron and folic acid supplements for pregnant women, vitamin A supplements for all children 6-59 months, and zinc supplements (with ORS) for the treatment of diarrhea, and expanding the delivery of behavior change communication, focusing on the prevention of malnutrition during pregnancy and in the first two years of life. With the DoH’s program along with the LHWs training on nutrition supported by the DLI under Component 1, DoH’s capacity to delivery nutrition services at facility as well as community level improved. In addition, in 12 districts with the highest prevalence of acute malnutrition, therapeutic nutrition services were being delivered, including commodity support micronutrients and Ready-to-Use Therapeutic Foods. The additional support (US$35.82 million, of which US$12.7 was utilized) was to incentivize the Primary and Secondary Health Department to provide essential nutrition services to more districts for children aged under 24 months of age through the following: (i) universal screening and counseling in the target districts; (ii) expansion of the coverage to uncovered union councils in the districts; (iii) Outpatient Therapeutic Program extension; (iv) establishment of Stabilization Centers with functional referral linkages; and (v) consolidation of existing fragmented trainings into a comprehensive nutrition training by level of service. The Government established a Multi-sectoral Nutrition Cell at the Planning and Development Department, financed through the Annual Development Plan, and the cell was to provide support to a set of multi-sectoral nutrition activities to focus on (i) hygiene and sanitation activities; and (ii) ongoing projects with components of (a) Open Defecation Free villages; and (b) Pakistan Approach to Total Sanitation. The funds allocated to the MSN (US$2 million, of which US$0.3 million was utilized) would be utilized for technical capacity, evaluation, evidence generation and coordination of activities. Page 15 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Other Changes • January 17, 2014 - Extension of the due date for the signing of the HRITF Grant Agreement from August 31, 2013 to April 30, 2014 to allow for the IDA Credit to become effective as this was a suspension clause in the Financing Agreement. The delay in signing of the HRITF was linked to the delay in signing of the IDA FA due to changes in the DoH counterpart team as a result of political transition after the general elections in May 2013. • January 26, 2017 – Cancellation of the US$21.5 million HRITF Grant due to delays, indecision and inaction on preparing and implementing planned results-based proposals. • February 17, 2017 – Restructuring entailed: (i) Revision of components (as described above); (ii) Change in the RF (as described above); (iii) Change in implementation arrangements. The original administrative department for the implementation of the PHSS was the DoH which was bifurcated in December 2015 into two departments: Specialized Healthcare and Medical Education Department (SHMED) and the Primary and Secondary Healthcare Department (PSHD). The latter department assumed the overall responsibility for the project, supported by the Policy and Strategic Planning Unit (PSPU) which was originally named the Punjab Health Sector Reform Program (PHSRP), keeping the same legal status. This restructuring formalized that change and also added two additional implementing agencies to implement the new Component 5, namely, the Integrated Reproductive, Maternal Neonatal and Child Health (IRMNCH) Unit to provide health sector nutrition interventions, and the Multi Sectoral Nutrition (MSN) cell within the Punjab Planning and Development Department to provide technical advice, evaluation and coordination of multi-sectoral nutrition interventions; (iv) Changes to financial management and disbursement arrangements to align with the three agencies (e.g. 3 Designated accounts), using the reimbursement modality. The PSPU would consolidate financial reporting under the project with the other two agencies maintaining financial management responsibility for the funds disbursed to them; (v) Change in procurement arrangements, with the two new agencies setting up their own procurement functions; (vi) Change in the eligible expenditure category to include nutrition interventions using a reimbursement modality; (vii) Reallocation of Credit proceeds to align with changes in components; (viii) Extension of the Closing Date by 1 year to December 31, 2018. • December 28, 2018 - Cancellation of SDR20.8 million funds which were not going to be utilized. This also included component cost revisions to reflect expected final project expenditures (see Annex 4 for details) • Extension of Disbursement Deadline (Grace period) on an exceptional basis by 1 month, from April 30, 2019 to May 31, 2019 to enable all disbursements of eligible expenditures to be documented. • June 2019 – cancellation of remaining US$0.4million of unspent funds. Page 16 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Rationale for Changes and Their Implication on the Original Theory of Change As already mentioned above, the main rationale for the changes made was based on the need to better align the project design with the reality on the ground and the changes taking place, including the critical bifurcation of the DoH into two departments, making it even harder to implement the originally envisaged reforms under the PHSS. The rationale was also driven by the fact that the project experienced major delays due to frequent changes in the leadership and management in the DoH, as well as associated revisions and changes in how to approach management reforms, in part supported by the HRITF. While cancelling the project was considered during the Quality Enhancement Review in early 2016 due to delays and staff changes as well as the bifurcation of the DoH, in the end, it was decided that the project could make additional contributions, especially in nutrition given the very high stunting ratings. As a result, the project was extended for an extra year and added 4 DLIs focused on improving utilization of family planning services, 1 DLI on expansion of EmONC services, and Component 5 focused exclusively on expanding the nutrition interventions. The expanded scope of these activities was aligned with the PDO and the original Component 1. Only 2 IO indicators were added to track proportion of children 0-24 months of age being screened for malnutrition and correctly identified with SAM. Also, targets for 5 PDO indicators were revised upwards reflecting the 1-year extension (see details in the outcome section below). Therefore, on balance, there was no real implication for the Theory of Change given that the PDO was still supporting the PHSS with all of its components and the RF remained intact (see para below as well). II. OUTCOME Since the main PDO was broadly stated - “to support the Punjab Health Sector Strategy” - the objective of improving coverage and utilization of quality health services was a “through” objective, while the “contributing” objective was not part of the PDO in the Financing Agreement, this ICR assesses the PDO in line with ICR Guidelines in such cases. The assessment of Efficacy includes: (i) a review of the PHSS achievements against its stated objectives in 2012; (ii) assessment of project objectives inferred from expected outcomes in the RF, component descriptions as outlined in the original and revised Financing Agreement and reflected in the Theory of Change; and (iii) assessment of achievement of DLI targets which the PAD noted were critical milestones of institutional changes to advance the reform process and to achieve the PDO. This ICR relies on the Theory of Change for the original project and also takes into account revisions made during the 2017 restructuring. Given that the PHSS remains the official Government reform document 1, and the fact that the restructuring did not change the PDO, the PDO indicators or IO indicators, this ICR does not use a split rating methodology and assesses the project in all of its parts. Notably, if a split rating methodology were to be utilized to weigh project achievements prior and post the February 2017 restructuring, the outcome rating would be the same, primarily driven by modest outcomes prior to restructuring and the Modest Efficiency rating for the entire project. A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating: Substantial The 2010-13 Country Partnership Strategy (CPS) [Report No. 53553-PK] was to support Pakistan’s efforts to address major institutional, policy, and financing constraints to increase its capacity to achieve and sustain high economic growth rates, to manage conflict, and to improve the social indicators, and the capacity of its population. One of the four pillars of the CPS was to improve human development and the project was well aligned with that general 1Several Bank Aide-memoires and restructuring paper from Feb. 2017 refer to “revised” or “new” reforms. However, the ICR team could not locate any official documents describing such reforms. Therefore, the PHSS remains the official Government reform document. Page 17 of 76 The World Bank Punjab Health Sector Reform Project (P123394) objective but also well aligned with the entire premise of the CPS given that the PHSS envisioned reforms in management, governance and accountability, including strengthening health systems and capacity to improve sector performance which was a key CPS objective. The project’s broad PDO of supporting the PHSS remained relevant to the current CPS (FY15-19) (Report No: 84645- PK), dated April 4, 2014 which includes four specific priorities: (i) transforming the energy sector, (ii) supporting private sector development, (iii) reaching out to the underserved, neglected and poor, and (iv) accelerating improvements in services. The fourth pillar of the CPS recognizes the need to build governance and service delivery at the provincial level following the devolution responsibilities to provinces, in line with the 18th Constitutional Amendment, which was passed by the Pakistan National Assembly on April 8, 2010. The CPS specifically includes two key outcomes related to the health sector: expanding the use of modern contraceptive methods from 26% to 30% (project target for Punjab was 35%) and increasing child immunization by 20% (project target for Punjab was 40%), both of which are also PDO outcome indicators. The CPS’s approach to achieving its goals is anchored in six key shifts in its instruments and engagements, three of which are included in the project, namely: (i) increased focus on results which this project embodies given its use of the DLI modality (85% of project financing) and its explicit inclusion of performance and results-based contracts (50% of total allocation within project components); (ii) increased focus on governance which this project fundamentally supports under Component 2 to enhance governance and accountability mechanisms and under Component 3 in strengthening stewardship and management functions in the DoH (35% of total allocation within these two project components); and (iii) building partnerships for sustained results which this project’s design envisioned as critical in ensuring success of the operation through its partnership with DFID in support of the PHSS. While the PDO remained relevant to the CPS, it is less clear that the project’s implementation remained relevant to the PHSS. Many of the most critical elements of the PHSS, such as the focus on results, reforms in management, governance and accountability were not pursued with the same degree of rigor as they were originally envisaged. While improvements to service delivery were pursued, management and structural reforms were not implemented. In fact, the bifurcation of the DoH into two departments in late 2015 further impaired a number of these reforms from taking place and the project was restructured to eliminate most DLIs related to strengthening the management of the health system except for a few discreet initiatives. As mentioned in the Bank Aide-memoire of January 2015, two of the most critical pillars of the PHSS reform (strengthening results-based PHC contracting out and district performance management contracts) were not being implemented and in the end were dropped from the project, together with the cancellation of the HRITF grant that was to support this results-based work. Based on the above assessment, the rating for Relevance is Substantial given the fact that the PDO remains relevant (at least on paper) to the CPS. This is in line with the ICR guidelines which no longer reflect relevance of implementation as part of its rating. Otherwise, the rating for Relevance would be Modest. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome: Modest As mentioned above, the assessment includes a (i) brief assessment of the PHSS achievements, (ii) assessment of PDOs based on the Theory of Change, and (iii) a summary of achievement of all project DLIs given that 85% of the funds were meant to be spent through this modality. PHSS achievement summary – Rating: Modest Page 18 of 76 The World Bank Punjab Health Sector Reform Project (P123394) The PDO was to support the Punjab Health Sector Strategy 2012-2020, specifically to implement the first phase of the reforms which included reforms in management, governance, accountability, and the use of results-based approach, all of which would lead to improvements in health service delivery. The expected outcomes at the end of PHSS implementation were to be measured by the following key results across five result areas (see Table 2). The latest data available is from MICS 2018 and from NNS 2018 survey for result areas 1-3. These results show that there has been some progress made, particularly with U5 mortality and iron deficiency anemia among women of reproductive age. However, MMR and stunting rates among children under 5 remain extremely high, with wasting rates actually increasing from 2011. Overall, the indicators show poor outcomes and clearly none of the PHSS targets are going to be achieved by 2020, as envisioned. Results for areas 4 and 5 could not be found despite several requests made by the ICR team. Table 2: PHSS Key Results Areas Key results: MICS MICS MICS PHSS 2012 baseline PHSS Target Achievement status PHSS Indicators 2011 2014 2018 (2020) Area 1: Child Health IMR (per 1,000 live births) 82 75 60 77 30 target not likely to be met U5MR (per 1,000 live births) 104 93 69 111 40 target not likely to be met Area 2: Maternal Health MMR (per 100,000 live births) NA NA 180 227 120 target not likely to be met Area 3: Nutrition Underweight (%) 33 34 21.2 29.8 (NNS 2011) 10 23% (NNS 2019) Stunted (%) 36 33 31.5 39.2 (NNS 2011) 8% in table 36.4% (NNS 2019) (6% in report text) Wasted (%) 16 18 7.5 13.6 (NNS 2011) 8% in table 15.3% (NNS 2019) (5% in report text) Iron deficiency anemia among 27.3% non-pregnant; 10% 18.7% (NNS 2019) women of reproductive age 39.2% pregnant (NNS 2011) Area 4: Communicable diseases TB prevalence 0.3% 0.5% 0.3% 0.1% No data available Hep B and C 0.7% 1.5% 0.7% 0.4% Based on latest data, the target is not met. Data is not yet publicly available. HIV/AIDS prevalence among 0.03% PDHS 0.01% No data available vulnerable groups Area 5: Non-communicable diseases NCD prevalence NA 40% No data available reduction Note: Baseline data cited in PHSS has 25% for stunting, 24.4% for wasting and 30% for underweight although the National Nutrition Survey (NNS) has different numbers. It also cited 27% for iron deficiency anemia among women of reproductive age. As part of the broad look at the overall achievement of the PHSS, given that the Punjab is the most populated province in Pakistan, and to a large extent drives the overall country health outcomes, it is important to review the results of the CPS outcomes (2019 target) related to the health sector, both of which were PDO indicators of the project. The first intended CPS outcome was to increase the use of modern contraceptive methods from 26% in 2014 to 30% across Pakistan by 2019. The Pakistan Demographic and Health Survey (PDHS) 2017-2018 reports 25% for Pakistan. The project set a target of 35% for Punjab and achieved 29.9% as reported in MICS 2018 which was actually lower than reported in MICS 2014 (30.8%). The PDHS 2017-2018 sites a figure of 27% for Punjab. Whichever way, this CPS outcome is not achieved. With respect to increasing full child immunization by 20% across Pakistan, this indicator has been achieved through concerted Government efforts with support of the WB and other development agencies. The Page 19 of 76 The World Bank Punjab Health Sector Reform Project (P123394) percentage of fully immunized children in Pakistan age 12-23 months has increased by 22% over a 5-year span, from 54% in 2012-13 to 66% in 2017-18 (PDHS 2018). In Punjab, using the PDHS data, immunization was 65.6% in 2012 and 80% in 2018. The project had a baseline of 34.6% (MICS 2007) and the goal was to increase this to 60% by 2017. MICS 2011 already reported 46.8% achievement and it is not clear why the project used the baseline data from MICS 2007. MICS 2014 reported 62.3% and MICS 2018 reported 76.5% which surpassed the original target of 60% but did not meet the revised target of 80%. Nonetheless, it is clear that there has been substantial progress made against this outcome indicator in the CPS. Assessment of PDOs (based on Theory of Change) - Modest The project included indicators as well as DLIs drawn from the PHSS, ranging from health outcome indicators to indicators measuring health service delivery, system effectiveness and management. The following analysis provides assessment of outcomes by the following five parts, as presented in the Theory of Change above. Part (i) Increased coverage of essential health and nutrition services, including in low performing districts – Substantial. Overall, there was substantial progress made on the service delivery side, especially with the development and implementation of the EPHS and establishing comprehensive EmONC services at primary and secondary healthcare facilities. The following PDO and IO indicators reflect improvements although there remain a number of deficiencies in terms of facilities provision of services, as evidenced in the recent Health Facility Assessment and in the Third-Party Evaluation of the LHW Program [see discussion below under part (iv)]. • Fully immunized children 12-23 months of age (baseline 34.6% from MICS 2007, although MICS 2011 refers to 43%; original target 60%; revised target 80%; actual 76.5% which was 86% of the target achieved). With respect to 18 low performing districts, the baseline was 23.9% with a target of 52.5%. The target was increased to 60% during restructuring and actual achievement was 75%. Therefore, overall, this indicator was achieved. • Proportion of children 0-24 months in the 18 low performing districts receiving basic package of nutrition services (baseline 10%; original target 60%; revised target 80%; actual 84%) – surpassed. • Number of RHCs providing basic 24/7 EmONC services (baseline 150; target 291; actual 311) –surpassed. • Monthly average number of deliveries in one RHC per month on average (baseline 33; target 60; actual 65) – surpassed. • Percentage of pregnant women registered with LHW receiving IFA tablets during pregnancy (baseline 10%; target 60%; actual 60%) – achieved. • Proportion of children with SAM registered for treatment at stabilization centers (baseline 5%; original target 40%; actual 77%) – surpassed. This was also supported by DLI4 referring to 12 under-performing districts and was not achieved and dropped in 2017. • Proportion of children 0-24 months of age being screened (at least biannually) for malnutrition (baseline 10%; target 80%; actual 70% from 2017; no data at end of project) – not achieved. Status of DLI achievement: • DLI1: Punjab has operationalized the integrated management of three community – based programs (LHW, MNCH, and Nutrition program) and approved PC-1s for integrated management for (a) Reproductive health, primary health care, and nutrition; (b) LHW program, and (c) MNCH. This DLI was achieved and was critical in improving the overall coverage of services for women and children. Prior to that there were two separate provincial level units, one for LHW set up following the 1978 Alma Ata declaration and the other was the MNCH which also included community midwives. However, even though these units were both working towards the same goal, there was little Page 20 of 76 The World Bank Punjab Health Sector Reform Project (P123394) coordination at the service level. At the same time, nutrition services were not really provided at all. Therefore, this integration of the two programs and the inclusion of nutrition really meant that there was finally a more coordinated package of services delivered. • DLI3: 15,000 LHWs trained on FP and nutrition – achieved. The training took place in May 2014 for LHWs on healthy timing and spacing of pregnancy and on infant and young children feeding. The verification report submitted in January 2015 noted that while the LHW program reported to have trained 14,830 LHWs, 12,469 (84%) actually received training. Therefore, additional training had to be carried out in May 2015 in order to fully achieve this DLI. The verification report also noted that the entire training system for LHWs had to be revamped, i.e. there was no tracking of who received training, there were no guidelines or a training handbook for LHW training. • DLI5: A minimum of 6 HIV/AIDS service contracts with NGOs – the Bank declared this as achieved, although the independent verification agency (IVA) report from January 2015 notes that only 5 contracts were signed, not 6, as required by the DLI. The hired NGOs were working in 12 service sites to provide services to IDUs, female/male/transgender sex workers. • New 2017 DLI - 90% of DHQs in 18 low performing districts providing C. EmONC services – the Bank declared this as achieved with 16 of 18 DHQs hospitals providing services which is actually 88%, not 90%. In support of this objective, under Component 5, nutrition commodities were procured through a contract with UNICEF (US$14 million, of which US$12.7 million was spent). This clearly contributed to the indicator measuring the proportion of children 0-24 months in the 18 low performing districts receiving a basic package of nutrition services. The original baseline was only 10% while the actual at the end of the project was 84%. This reflects significant progress made in the poorest performing districts to improve Punjab’s high rates of malnutrition. Regrettably, a second contract with UNICEF could not be signed during the project due to bureaucratic road-blocks and US$19 million of the original allocation had to be cancelled. Part (ii) Improved quality of essential health and nutrition services, including in low performing districts – Substantial • PDO indicator and DLI 2. Births attended by skilled personnel – (baseline 58.5%; original target 70%; revised target 75%; actual 76.4%). The project also included this indicator as a DLI to be met by year 4. It was not achieved, and the DLI was extended to the end of the project with the original target of 70% whereas the same PDO indicator target was increased to 75%. With respect to 18 low performing districts, the baseline was 45.4% with a target of 60.4%. The target was increased to 65% during restructuring and actual achievement was 68.4%. Therefore, this indicator surpassed its target. • IO indicator. Number of THQ hospitals providing 24/7 comprehensive EmONC services (baseline 40; target 75; actual 50) – not achieved. • IO indicator. Proportion of children 6-24 months of age correctly identified with SAM in total screened children (baseline 1%; target 90%; actual 1.4% from 2017 data; no latest data available) – not achieved. A contributing DLI and indicator to this PDO part was also the expansion of licensing of healthcare facilities by the Punjab Health Care Commission (PHCC), discussed below under part IV. While this initiative aimed to enhance overall governance in the health sector, the PHCC’s ultimate objective is essentially to improve quality of care by setting in place transparent standards and holding facilities accountable for maintaining these standards through registration, licensing and oversight. Therefore, given the overall success of the PHCC work, the overall rating for this PDO part is Substantial also given progress with skilled birth attendance. Page 21 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Part (iii) Increased utilization of essential health services, including in low performing districts - Modest • Contraceptive prevalence rate (any modern method) (baseline 29% although MICS 2011 refers to 35%; target 35%; actual 29.9%) – not achieved. This was also supported by DLI 2 which was not achieved and dropped. With respect to the 18 low performing districts, the baseline was 22.6% with a target of 28.5%. The target was not increased during restructuring and actual achievement was 28.5%. Therefore, the sub-indicator was achieved for low performing districts. • Monthly average number of family planning clients provided products and services in RHCs on average (baseline 62; target 100; actual 72) – not achieved. • Percentage of IDUs reached by the Program who reported using a new syringe at their last injection (baseline 50%; target 75%; actual 65%) – not achieved. • Daily average number of OPD visits in BHUs (baseline 40; target 60; actual 62) and RHCs (baseline 140; target 180; actual 289) – surpassed. Status of DLI achievement. In addition to DLI2 on the CPR (above), the following 4 DLIs were aimed to increase utilization of family planning services. • New 2017 DLI - Media campaign for family planning services designed and implemented – not achieved. The media campaign was launched by the Population and Welfare Department (PWD) and a third-party evaluation was carried out which found that overall the campaign was carried out but that messaging was not specific enough. • Punjab Population Innovation Fund (PPIF) fully operational. The PPIF was established in October 2016 as a section 42 company (not for profit government agency) with the objective to fund innovative projects improving quality and accessibility of family planning services. The PPIF was operational in mid-2017 and the DLI was met in July 2017 (Bank Aide-memoire July 2017). However, the Bank only paid for this DLI’s achievement in April 2019, after the project closed. According to the PPIF, lack of payment from the Bank for the achieved DLI and the expected transfer from MOF prevented them from signing the contracts which were the part of the other DLI. • PPIF awarded 5 contracts – not achieved as only 2 contracts could be signed during implementation (with Health and Nutrition Development Society and the Akhter Hamed Khan Resource Center) because the PPIF did not have the requisite amount of funds in its bank account. • Increase in FP service intake at DoH facilities in 18 districts. This was linked to the media campaign. This DLI was counted as achieved because the Bank team agreed to consider the provision of services by the PWD (3,935,548) in addition to those provided by the DoH (2,434,147) as part of this DLI. The restructuring paper did not provide a % or a target to be met and the DLI remained as a general formulation of goals. During discussions with government, it was ascertained that the total number of people reached had to exceed the 3 million mark to meet this DLI. The legal amendment did not provide this detail. Part (iv) Improved health system performance (as measured by improved facility management through results- based contracts, improved facility functionality, governance and accountability) – Modest -Facility management and results-based district management contracting - not-achieved. These were meant to be two critical pillars of the reform of the overall system to enhance the efficiency and effectiveness of the health system. There were major delays in launching these due to DoH leadership continually revisiting and revising the design of the reforms, also as a result of high turn-over in the DoH management. For three years, the Bank consistently requested for the DoH to submit a strategy and an implementation plan. Ultimately, none of these reforms were implemented. There was also no progress in launching innovation pilots with support from the HRITF which would have been extremely helpful in providing experience and lessons for the proposed new Human Capital Page 22 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Project under discussion. The success of all of these efforts were meant to be incentivized by the following 3 DLIs: • DLI 6: DoH has entered into revised results-based contracts with PRSP in line with EPHS for primary health care, in 14 ongoing & new districts – not achieved. The PSPU prepared a revised contract with the inclusion of community outreach services in addition to facility level services. However, the decision by the Chief Minister of Punjab was to instead use a contracting out approach in 10 districts but without including preventive community-based services without which improvements in health outcomes would not be likely, as noted by the Bank and DFID teams at Mid- term Review in August 2015. • DLI7: DoH has entered into performance management contracts with all District Health Offices and has transferred budgetary resources therewith - not achieved. This was meant to enable a culture of performance-based management at the district level by DoH contracting in the Executive District Health officers on the basis of established key performance indicators across all 36 districts. As with the PHC contracts, this initiative failed although preparation work was carried out. The DoH leadership decided to evaluate districts on the basis of a “dashboard” consisting of some indicators, as part of the Roadmap exercise. During implementation, there was also a change made in the way districts manage health services, with a shift to establishing District Health Authorities financed by and reporting directly to DoH. • DLI8: 80% of low performing (below the mean) districts of Punjab (18), identified by DoH, have individually attained a minimum increase of 5/10 percentage points in the composite index (scorecard) based on the list of Key Performance Indicators, measured against the 2012 baseline – not achieved. -Health facility functionality/medicine availability - not achieved. The latest 2018 HFA provides an analysis of the overall functionality of the four levels of health facilities (DHQH, THQH, RHC and BHU 8/6 and 24/7 facilities) using a facility readiness index. The overall readiness index includes measures of infection control, service availability, facility management, basic amenities, human resources, equipment, availability of medicines and supplies, and client satisfaction. The average index for the four types of facilities was 63% with availability of medicines and supplies being the lowest measure (ranging from 29% in THQHs to 42% in BHUs with 24/7 services). This result puts into question outcomes reported in two IO indicators, one measuring the percentage of districts with their respective BHUs providing all essential drugs (reported as 96.6% against a project target of 85%); and the other measuring percentage of LHWs with stock-outs of family planning methods (baseline 80%; target 10%; actual 7%). Also, the 2018 LHW Third Party Evaluation revealed high % of stock outs of medicines and FP supplies (30-97% range across all categories) which also puts into question the outcomes reported under the second IO indicator. In fact, this LHW evaluation reported no appreciable difference in service delivery performance over the last decade, with severe gaps in coverage, knowledge and skills of LHWs and supervisors. Likewise, the HFA highlighted major deficiencies across multiple aspects of service delivery, with major gaps in human resources (with low % of positions filled, particularly at THQ and DHQ hospitals with 43-44% of clinical specialists available), equipment shortages in THQ (55%) and DHQ (54%), and limited availability of management and facility standards applied. Table 3 below compares the result of the two HFAs across three main facility functionality dimensions. Overall, the results of the HFA 2018 assessment appear somewhat better in terms of % filled positions for BHUs/RHUs but worse for DHQs/THQs and worse for availability of equipment across all facilities. Overall, these results are not encouraging and do not reflect improved system performance. Page 23 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Table 3: Comparison between HFA 2014 and HFA 2018 % Filled positions Availability of medicines and vaccines in facilities Availability of equipment HFA 2014 2018 2014 2018 2014 2018 BHU 71% 83% (24/7) 69% Medicines: 30.3% (24/7); 26% (8/6) 59% 70% (24/7) 86% (8/6) Vaccines: 98.5% (24/7); 92.4% (8/6) 60% 8/6 RHU 64% 79% 77% 31% medicines; 98% vaccines 67% 59% DHQ 63% 44% 76% 28% medicines; 64% vaccines 87.5% 56% THQ 52% 43% 80% 24% medicines; 40% vaccines 81% 39% -Improved governance - achieved, as reflected by the operationalization of key regulatory functions of the Punjab Healthcare Commission (PHCC). To track progress on this discreet activity supporting the PHSS reform, one PDO indicator and DLI 9 were included - PHCC licensing of public and private Health Care Establishments (HCEs). The PHCC was established in 2010 with the primary objective was to regulate healthcare services in Punjab through establishing Minimum Service Delivery Standards (MSDS), registering, inspecting and issuing licenses to HCEs, investigating complaints about malpractice and maladministration, and conducting awareness activities with respect to the quality of healthcare, as well as issuing guidelines and directives to facilities. The project’s PDO indicator was to have 500 Category -1 HCEs (hospitals with 50+ beds) issued with provisional licenses (certificate of registration) by the PHCC. The DLI had a target of 200 HCEs issued with provisional licenses. The total number of public and private HCEs is 299 in 2019, and the DLI target was to be achieved by December 2013. The ISR prior to the 2017 restructuring reported that 211 Cat 1 HCEs were licensed. This meant that the DLI was achieved much later than planned, but was achieved nonetheless. During restructuring, the target for the PDO indicator remained at 500 although the indicator was rephrased as Category 1 and Category 2 HCEs (hospitals with <50 beds, including RHCs) issued with license. The total number of Cat 2 HCEs is 4,282 and the final ISR reported 2,353 although the previous one reported 2,579 but these numbers were not disaggregated by categories. During the ICR mission, the meeting with the PHCC revealed that 291 Cat 1 HCEs and 2,116 Cat 2 HCEs were licensed, for a total of 2,405. Overall, the DLI was certainly achieved but the PDO indicator is not evaluable since the revised PDO indicator does not have the relevant target to cover both sets of HCEs. Regardless, it is clear that the PHCC has done a great deal of work to scale up its operations. As of February 2019, the PHCC has provided 581 training on MSDS to over 23,000 participants in almost 19,000 of the total 68,340 HCEs in Punjab. The PHCC has also registered almost 55,000 HCEs (all Cat 1; 60% of Cat 2 and 85% of Cat 3) and licensed 37,363 HECs (99% Cat 1, 50% Cat 2 and 55% of Cat 3). The PHCC has also designed and is implementing a complaint management system which receives, registers, processes and investigates complaints. Thus far, it has settled 70% of all registered complaints and has imposed fines (PKR 47 million) on guilty parties as well as initiated 47 criminal proceedings and stopped 43 HCEs from provision of services. The PHCC also supports anti-quackery activities to decrease the prevalence of unqualified service providers through advocacy, enforcement and consultations with stakeholders. From 2015-2019, it has identified almost 64,000 quacks operating in Punjab and has fined them PKR 360 million. In this way, the PHCC is increasing public pressure on quality of service provision as begins to undertake the second round of reforms, including developing accreditation standards and performance audits of HCEs. -Improved accountability - moderate achievement. This part of the project was to be measured by validation of service delivery results through assessments, such as the planned annual Health Facility Assessment, surveys, reports, and through the establishment of a hotline and a complaint management system. The HFA (discussed above) was supposed to be an annual third-party validation of results to strengthen accountability and also improve future planning and management of healthcare services. Regrettably, it was only done twice in the project, first in 2014 and Page 24 of 76 The World Bank Punjab Health Sector Reform Project (P123394) then in 2018 although the 2017 restructuring changed the frequency of HFA from annual to once every 3 years. At that point, this amendment did not matter since there was only a year left to carry out one. Nonetheless, it provided very valuable information on facility functionality across all levels. Similarly, the Annual Health Report was meant to be published regularly and data disseminated through media. The report was published for the first time in mid-2015 and then in 2016/17 and 2018, three times in total during the project implementation. With respect to client satisfaction, the project included a PDO indicator and a DLI11 with a baseline of 0% and a target of 50% user satisfaction with the government health services in Punjab – not evaluable. This DLI was dropped in 2017 but the PDO indicator was kept with an increased target of 80%. At that time, the results of the first HFA were available which had an overall satisfaction with BHU and RHU services from 87-92%. Therefore, it is not clear why the baseline of 0% was not updated and why the target was roughly 10 percentage points below the 2014 data. A community satisfaction survey was done as part of the 2018 HFA and reported overall satisfaction at 93% (incidentally, the final ISR reported 85.5%). Based on HFA findings from the 2014 and 2018 assessments, this PDO indicator shows that there has been no change in client satisfaction. Also, considering little apparent correlation between low levels of functionality of health facilities and high client satisfaction, it is not entirely clear what achievement this indicator reflects. Emergency Helpline and the Complaint Management System (CMS) - achieved. The project has supported, through the revised DLI 10 (composed of 2 separate DLIs) a CMS in DHQ hospitals, which was rolled out as part of a broader effort in Punjab to establish an Emergency Helpline to facilitate patients in hospitals to resolve their complaints on immediate basis. This system operates around a Call Center for receiving and recording the status of complaints. The center has 30 staff, including 4 doctors who work 8am-12pm and night staff of 10 people, including 2 doctors. It is operated by the Punjab Information Technology Board (PITB) funded by the Planning and Development Department. Complaints are received via the Helpline and channeled to focal points in each hospital who respond to grievances and report back on the status of the complaint. If the complaint turn-around-time has lapsed, the complaint gets escalated according to the level of priority of the complaint. The third-party validation of this system in 2018 found that this system operates in all health facilities in Punjab and works for people who can call in although the overall public awareness about the Call Center is limited. Most complaints are about poor attitude and behavior of hospital staff (rather than availability of services) and that doctors are overworked, seeing too many patients. There is generally poor record-keeping of complaints which prevents proper analysis in order to benefit the healthcare delivery system. Also, this validation exercise showed that most complaints are not resolved at the local level but are escalated to the top level. This review recommended a number of improvements to make this system more effective, including launching a mass awareness campaign to enhance public awareness about the CMS. Part (v) Strengthened stewardship & management functions at the provincial level - Modest The main goal was to restructure the DGHS from a service delivery supervisor to a steward of the health system, a lead technical organization in the province with a focus on policy and strategic guidance, financing, monitoring quality of service delivery and ensuring transparency and ensuring accountability. The service delivery functions would be devolved to the district level. At the end of the project, the stewardship function for the health sector was still not fully functional, with mechanisms and processes to connect policy and planning with implementation under- developed, including the involvement and regulation of the private sector. The project did build some technical capacity of DoH in planning and management, including FM and procurement, in developing Minimum Service Delivery Standards (MSDS), hospital waste management plans, carrying out third party verifications and assessments, and training for health managers. Page 25 of 76 The World Bank Punjab Health Sector Reform Project (P123394) This part is measured by the following two IO indicators. • % of non-salary line items in all 36 districts health recurrent budget expenditures of total (baseline 20%; target was 30%; no actuals reported) – this indicator was critical to measure the extent to which facilities were able to provide services. This indicator was also a DLI14 and was dropped at restructuring and changed to an IO indicator, as follows: “% of districts which are able to utilize more than 90% of the released funds in the same FY (baseline: 20%; target was 75%; actual was 61%) – not achieved. • % of the development budget allocated for preventive programs of the total development health budget (baseline 7%; target 20%; actual 11.7%) – not achieved. There was no reporting on this indicator during the project except for ISR #10 in March 2018. In addition, 3 DLIs supported this objective, as follows: • DLI12 - Reorganization of the Directorate General of Health Services (DGHS), including the establishment of a contact management unit – not achieved as a result of the bifurcation of the DoH. • DLI13 – DoH has established two cells, on FM and procurement at the provincial level. Procurement cell was established, supporting DOH procurement functions - achieved. Sector specific SOP/manuals were developed to align process and procedures with the Punjab Public Procurement Rules and standard bidding documents and training plans for staff were prepared. An FM cell was established to build fiduciary functions in the DOH. The financial reporting system put in place under the project has helped the PSPU, the IRMNCH&NP and the MSN cell to improve overall financial controls in their agencies. However, following the bifurcation of the DOH, the FM cell could not really function as intended. Shortages of procurement and financial management staff continued throughout the life of the project. • Original DLI15 and new DLI - achieved. The project supported hospital waste management in secondary healthcare facilities through this DLI and the second DLI was added at restructuring to expand the implementation of medical waste management to an additional 15 districts. The goal was to ensure safe disposal of infectious waste. Both DLIs were achieved. The two original facilities adopted and implemented the Medical Waste Management Plan which was subsequently scaled up to all 17 districts. As part of the Infection Control Program under the Primary and Secondary Healthcare Department (hosted in the Hepatitis Control Program), guidelines were developed and disseminated; a training plan for master trainers was prepared and executed and regular trainings and awareness campaigns are being carried out on a monthly basis; disposal bins, collection bags etc. were procured and distributed to all districts; 25 incinerators have been installed; yellow rooms have been built in all secondary level hospitals (this includes 17 DHQs and 40 THQs); and 37 yellow vehicles were procured. Data on hospital waste is available in a web-based MIS. The next step is to expand this program to BHUs and RHCs. Notably, the creation of the MSN Cell after the launch of the Multi-sectoral Nutrition Strategy in 2015 was an important step in improving stewardship of the nutrition program in Punjab to ensure oversight, coordination and monitoring and evaluation of nutrition interventions in the province. Its inclusion in the project in 2017 was a step to ensure multi-sectorality in the approach to nutrition interventions. The MSN cell supported the Human Development Forum held in 2018 which had a special focus on nutrition, and the District/Tehsil Nutrition Symposium. It also carried out nutrition research activities and training. Regrettably, after the closure of the project, the MSN cell has been disbanded. Assessment of the Results-based approach: (Modest) Given the results-based nature of the project, with 85% of the original US$100m IDA Credit financed through the DLI Page 26 of 76 The World Bank Punjab Health Sector Reform Project (P123394) modality, it is useful to review achievement of the total sum of the DLIs under the project (Table 4). Of the total US$ volume originally planned, 57% was disbursed using this modality at the end of the project, with 53% dropped during restructuring in February 2017, replaced by US$11 million under 5 new DLIs under Component 1 of which only 3 were achieved since restructuring. Table 4: Summary of Achievement - Disbursement Linked Indicators (2014-2018) No DLI Name US$m Status Disb. date Component 1 – Improving Health Service Delivery (i) Integrated Management of MNCH&LHW Programs 1 Punjab has operationalized the integrated management of three community – 5.67 Achieved Feb. 6, 2015 based programs (Lady Health Workers, Maternal, Neonatal, and Child Health, and Nutrition programs) and approved PC-1s for integrated management for a) RH, primary health care, and nutrition; b) lady health workers program, and c) MNCH. 2 Punjab has attained: (i) at least 35% in the use of modern contraceptive methods; and 3.17 (i) Dropped (ii) at least 70% skilled birth attendance. 2.5 (ii) Achieved April 16, 2019 3 15,000 LHWs trained on FP and nutrition 5.67 Achieved Sept 22, 2015 4 (ii) Introduction of Nutrition Services: At least 60% of children identified with SAM 5.67 Dropped in 12 under-performing districts registered for treatment 5 (iii) Expanded Coverage of HIV/AIDS: 6 HIV/AIDS service contracts with NGOs 5.67 Achieved March 19,2015 DLIs introduced during Feb 2017 restructuring new Media campaign for FP services designed and implemented 1 Not Achieved new Punjab Population Innovation Fund (PPIF) fully operational 2.5 Achieved April 16, 2019 new PPIF awarded 5 contracts 2.5 Not Achieved new 90% of DHQs in 18 low performing districts providing C. EmONC services 2.0 Achieved April 16, 2019 new Increase in FP service intake at DOH facilities in 18 districts 3.5 Achieved April 16, 2019 Total Component 1 – DLIs achieved (total planned: US$39.85 million) 27.51 69% Component 2 – Enhancing Efficiency and Effectiveness of the Health System (i) Strengthening PHC contracting out 6 DoH has entered into revised results-based contracts with PRSP in line with EPHS 5.67 Dropped for primary health care, in 12 ongoing & 2 new districts. (ii) Result-Based District Management Contracts 7 DoH has entered into performance management contracts with all District Health 5.67 Dropped Offices and has transferred budgetary resources therewith. 8 80% of low performing (below the mean) districts of Punjab (18), identified by DoH, 5.67 Dropped have individually attained a minimum increase of 5/10 percentage points in the composite index (scorecard) based on the list of Key Performance Indicators (iii) Enhancing Governance and Accountability Mechanisms a. Regulatory reforms including operationalizing the Punjab Healthcare Commission and b. Strengthening social accountability and private sector 9 At least 200 Category -1 HCEs are issued with provisional licenses (certificate of 5.67 Achieved March 19,2015 registration) by the Punjab Healthcare Commission 10 PHC has operationalized a CMS 5.67 Revised Revised - CMS in place in 33% of secondary care hospitals 2.5 Achieved April 16, 2019 Revised - CMS in place in 66% of secondary care hospitals 2.5 Achieved April 16, 2019 11 b. At least 50% of participants in the user satisfaction survey during June 2016 as 5.67 Dropped part of the annual Health Facility Assessment expressed satisfaction with the government health services in Punjab. Total Component 2 – DLIs achieved (total planned: US$34.02 million) 10.67 32% Component 3 – Strengthening Provincial DOH management capacity Page 27 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 12 (i) Strengthening stewardship functions in DoH. DoH has restructured and 5.67 Dropped reorganized the DGHS pursuant to the revised roles and responsibilities, including the establishment of a contract management unit. (ii) Building Fiduciary Functions 13 DoH has established two cells that are financial management and procurement, at 5.67 Achieved February 6, 2015 the provincial level. 14 Non-salary budget items in all 36 districts; health recurrent expenditures represent 5.67 Dropped at least 30% of total recurrent expenditures (iii) Environmental and medical waste management At least 2 secondary health levels facilities in Gujranwala and Khanewal have 15 5.67 Revised adopted and implemented the Medical Waste Management Plan Revised - At least two secondary health levels facilities in Districts Gujranwala and 2.5 Achieved June 13, 2017 Khanewal have adopted and implemented the Medical Waste Management Plan Revised - Implementation of medical waste management expanded to 15 districts 2.5 Achieved April 16, 2019 Total Component 3 – DLIs achieved (total planned: US$22.68 million) 10.67 47% TOTAL ACHIEVED (total planned: US$96 million) 48.85 51% Note: The total value of DLIs is presented as the total value in the original PAD (US$85 million) plus US$11.5 million (for 5 DLIs added at the Feb 2017 restructuring under Component 1). While 8 DLIs were dropped during the restructuring, the table above presents these as part of the total to demonstrate the full set of targets which were planned to be met under the project. Overall, 57% of the original US$85 million value of DLIs was disbursed under the project. Justification of Overall Efficacy Rating Efficacy is rated as Modest despite some important achievements made during the project implementation period. The original five expected results of the PHSS (2012-2020) which the project was supporting are not expected to be achieved although important progress has been made. Of the two specific CPS health indicators, one has been achieved (immunization) whereas little to no progress was made to increase the CPR in Punjab which remains essentially the same as reported in MICS 2011. Of the five parts of the PDO (as assessed above), two have been largely achieved (coverage and quality) although the results of the 2018 HFA suggest that significant improvements in quality of services, and particularly in the availability of medicines and staff capacity, are needed. The LHW Program evaluation suggests that much more needs to be improved to see progress. With respect to the structural reforms planned under the PHSS to improve overall governance and management in the health sector, those have not materialized as planned and much more concerted effort is required by the GoPb in this regard. Lastly, only 57% of the total volume of DLIs under this project was disbursed with critical areas under Components 2 and 3 dropped 9 months prior to the original project closing date; progress was mixed with respect to DLIs which were added on family planning, and only about 40% of the original allocation on nutrition activities was spent which greatly reduced the planned impact. While some nutrition indicators have seen major improvements, more intensive action is needed to reduce malnutrition across the province in order to achieve PHSS targets. C. EFFICIENCY Assessment of Efficiency and Rating: Modest Economic analysis. The PAD did not include an Economic and Financial analysis although this was explicitly pointed out during the QER just prior to the Decision meeting in March 2013. This ICR did not attempt to estimate the potential economic rate of return or net present value of this project. However, to provide an indication on the efficiencies and economic implications of key components, the following sections provide discussions on the economic analysis of the Page 28 of 76 The World Bank Punjab Health Sector Reform Project (P123394) interventions supported by the project in light of global evidence base, as well as assessment of technical and allocative efficiency. Economic benefits of the interventions supported by the project. The cost-effectiveness of MNCH interventions has been established in the research literature, using as a standard measure cost per Disability-adjusted Life Year (DALY) averted. For example, the cost-effectiveness of a standard MCH service package is estimated to range between US$24 and US$585 per DALY averted, while that of a standard package of prenatal and delivery care ranges from US$92 to US$148 per DALY averted. 2 The economic benefits generated by the project were estimated using DALYs. The results indicate that for US$1 invested by the project, benefits generated are in the range of US$1.85 - US$5.55 when IMR is considered and US$2.96 - 8.88 for U5MR. The project has generated economic benefit of US$192–577 million from reduction of U5MR. Percentage of stunted children was reduced from 33 to 31.5 in 2014-2018. One-fifth of children were underweighted in 2018 whereas it was one third in 2014. Reductions in stunting are estimated to potentially increase overall economic productivity, as measured by GDP per capita, by 4–11 percent in Africa and Asia 3. Nutrition interventions have shown a high rate of return on investment–estimated to be between US$4 and US$35 for every dollar invested. 4 Nutritional interventions are among the most cost-effective interventions to enhance welfare. The World Bank estimates that childhood stunting reduces a person’s potential lifetime earnings by at least 10 percent and it is estimated that undernutrition results in productivity losses of US$149 billion each year. In South Asia, the estimated cost of stunting is 10 percent of the GDP per capita. 5 The project established 12 drop-in-centers in 9 cities at the cost US$5.67 million. Four of these sites were serving male sex workers and 8 were serving IDUs. The kinds of services they were providing included syringes, condoms and counseling, etc. The total number of registered clients at these sites was 21,435 in June 2014, with about 70 clients visiting each site per day. Around 80% of total registered clients received services. These sites provided services in 2014-2018 under the project and potentially gained more clients. The economic burden of HIV infection is substantial. The program prevented at least 6,500 from becoming infected with HIV infections and saved US$41.15 million generic treatment cost. Every US$1 spent on this HIV prevention activity saves US$7.25 in treatment cost. Therefore, drop-in- centers activities were likely to have a high economic return. Given Pakistan’s GDP per capita, MNCH and nutrition interventions were cost effective. An intervention is cost- effective if the cost per DALY avoided is less than three times the national annual GDP per capita. It is highly cost- effective if it is less than the national annual GDP per capita. World Health Organization’s Choosing Interventions that are Cost–Effective (WHO-CHOICE) project recommended these under thresholds based on per capita national incomes approach 6. Technical efficiency. The project contributed to an increase in access to basic package of MNCH and nutrition services 2 Laxminarayan, Ramanan, Anne J. Mills, Joel G. Breman, Anthony R. Measham, George Alleyne, Mariam Claeson, Prabhat Jha et al. 2006. “Advancement of Global Health: Key Messages from the Disease Control Priorities Project.” The Lancet 367 (9517): 1193–1208. 3 Horton, S., and R. Steckel. 2013. “Malnutrition: Global Economic Losses Attributable to Malnutrition 1900–2000 and Projections to 2050.” In The Economics of Human Challenges, edited by B. Lomborg, 247–72. Cambridge, U.K.: Cambridge University Press. 4 Shekar, M, Kakietek, J, Dayton Eberwein, J, Walters, D. An investment framework for nutrition: reaching the global targets for stunting, anemia, breastfeeding, and wasting: The World Bank; 2017. 5 Galasso, E, Wagstaff, A. The aggregate income losses from childhood stunting and the returns to a nutrition intervention aimed at reducing stunting. The World Bank; 2018. 6 http://www.who.int/bulletin/volumes/93/2/14-138206/en/ Choosing interventions that are cost-effective [Internet]. Geneva: World Health Organization; 2014. Available from: http://www.who.int/choice/en/ [cited 2019 April 25]. Page 29 of 76 The World Bank Punjab Health Sector Reform Project (P123394) and HIV prevention program. Among the 18 poor performing districts, the percentage of fully immunized children 12- 23 months of age was increased to 75% from 23.9% while the target was 60%. Also, 84% of children 0-24 months of age were receiving the basic package of nutrition services. For this nutrition indicator, baseline and target values were 10 and 80% respectively. Daily average number of OPD visits in BHUs and RHCs was increased by 110 and 372% respectively. The increase in the total number of services suggests that health facilities and drop-in-centers are becoming efficient based on decreasing cost per service. The project also incentivized the training of 15,000 LHWs on family planning and nutrition and this activity possibly strengthened quality of care and links between outputs and outcomes. The activities likely generated better value for money as, for the most part, the project was focused on 18 low performing districts. Investment in MNCH and nutrition has important economic implications at population level especially in the low performing districts of Punjab. Well targeted public health interventions make a difference between intervention and comparison areas. Improved health status gained from investment contributes to social well-being through its impact on economic development and productivity. Allocative efficiency. Budget allocation for components 2 to 4 of the project was reduced by 57% (US$71.66 million to US$30.6 million) due to dropping of related DLIs and to ensure the relevance of the overall context. The project costs show that the emphasis of the project ended up primarily being MNCH (Component 1) and nutrition interventions (Component 5). These two components accounted for 66% of project costs. Redirecting resources to MNCH and nutrition would not only allow for a more efficient use of resources but would also enhance the equity in service provision by ensuring that all the population of Punjab have equitable access to basic MNCH and nutrition services. Implementation efficiency. The following key issues negatively affected efficiency. • Delays in approvals and disbursements. There was delayed signing and effectiveness, including a delay in signing of the HRITF grant agreement. As a result, the Designated Account could not be opened for Component 4, and expenditures and financial management arrangements were not operationalized until late 2014. This also prevented an advance of funds (US$17 million) to be made for eligible expenditures incurred under the Eligible Expenditure Program (for financing of DLIs) prior to January 15, 2014, as was provided for in the FA because project effectiveness was declared only on January 17, 2014. Incidentally, there were also delays on the DFID side with funds released in 2013 not being utilized which prevented DFID from releasing its second tranche of GBP23 million in 2014/15. In addition, there was no budget allocated for the independent verification of DLIs in 2014. An independent consultant was finally hired in October 2014 to carry out verification, and the first time any disbursement was made for 4 DLIs was in March 2015, two years after project approval. There were also major delays in the approval of the PC-1 document for M&E (approved with a 3-year delay) which was critical for the implementation of the reform program. After restructuring, there were major delays in opening of the new DAs in 2017, with no funds available for the two units to implement the new Component 5. Lastly, because the PC-1 document for the IRMNCH and Nutrition Program expired in June 2018, the Planning and Development Department did not allow the signing of the second major contract with UNICEF for nutrition commodities in the summer of 2018 (US$4 million) although the Bank provided its no-objection to the contract in April 2018. There were also delays in the review of the contract template. This also coincided with the General elections in July 2018 which led to further delays in approvals. The Finance Department also made a decision in 2018 to effectively freeze all donor funding by declaring all donor funds to be part of a “lapsable fund” whereas these were part of a “non-lapsable fund” before. The PSPU estimated that roughly 45% of time after the February 2017 restructuring was not spent on implementation, and that over the project period, 28.5 months were spent purely on bureaucratic delays and lacking funding to operate. • Frequent staff turn-over. There were multiple changes in DOH leadership and management and in PSPU directors, with a limited number of technical staff, for the duration of the project, first as a result of the general elections in Page 30 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 2013, then as a result of the bifurcation of the DOH into 2 departments, and later as a result of another general election in 2018. The section on implementation issues below provides more detail. • Cancellation of funds. During project implementation, two cancellations were processed, for a total of US$50 million of the original planned US$120 million (the final third cancellation was made in June 2019 to cancel the remaining US$0.4 million). The first was to cancel the US$21.5 million HRITF grant due to delays in internal reviews and approvals of costed results-based proposals. There were multiple discussions about lack of progress in this area and the Bank threatened to cancel funds, starting in early 2015. Actual cancellation was not processed until early in 2017. The second major cancellation of SDR 20.8 million (est. US$29 million) was processed on December 28, 2018 just prior to the Closing Date to enable Pakistan to retain these cancelled funds in its portfolio. • Project timeframe. The project was restructured and had to be extended for a year to implement the remaining and new activities. Based on estimated high cost-effectiveness and generally negligible implementation efficiency, the overall Efficiency rating is Modest. D. JUSTIFICATION OF OVERALL OUTCOME RATING Overall rating for the project is Moderately Unsatisfactory based on Substantial Relevance, Modest Efficacy and Modest Efficiency. The ICR did not use a split-level methodology for evaluation (as explained above). However, given the complexity of this particular evaluation, it is important to explain that under any assessment, the results would likely be the same since the PDO was not revised and the overall efficiency of the project was Modest at best given that roughly 49% of total project funds were not used. Even if the PDO were to be revised to reflect substantial changes made to the project, the project restructuring came late in the implementation (with eventual cancellation of funds processed 2 days prior to the project Closing Date). Consequently, the original project phase (prior to the February 2017 restructuring) would have been rated as Unsatisfactory and would be weighed at roughly 50% of total disbursements which would lead to a Moderately Unsatisfactory rating. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender By supporting the RMNCH & Nutrition service improvements, the project inherently targeted women and children who are the primary beneficiaries of the healthcare services. The interventions also benefitted girls and women from poorer households and promoted gender and economic equity in districts as the project was targeted on the poor performing districts. Indeed, the majority of project indicators measured performance in service delivery of women and children and roughly 47% of the IDA Credit was spent on RMNCHN investments. The project aimed to increase demand for services by women, and while the CPR and family planning services have not been a success, the increase in the skilled birth attendance and overall increased access to RMNCHN services testifies that women are now more aware about their health and are likely to seek healthcare services. Institutional Strengthening Institutional strengthening was an explicit part of the project, with stand-alone Components 3 and 4 focused on strengthening provincial and district level capacity for stewardship and improving capacity in a number of technical areas. Technical assistance was provided in the preparation of various policy documents for different programs, including the development of the MSDS, the preparation of the Multi-sectoral Nutrition Strategy, as well as capacity building activities in financial management and procurement, preparation of PC-1 documents, implementation of hospital waste management plans for provincial and district level staff, and support to the PSPU and the reform Page 31 of 76 The World Bank Punjab Health Sector Reform Project (P123394) program. The third-party validation consultants, the assessment of the LHW Program and the Health Facility Assessment were also supported by the project which provided important information to assist the GoPB to make improvements to service delivery. Technical assistance provided also supported the DoH to compensate for its internal human resource shortages. Mobilizing Private Sector Financing The project targeted beneficiaries primarily through the public sector which provides services free of charge. The project did include support for the Punjab Population Innovation Fund which is a semi-private NGO for contracting out family planning services at the community level. Also, the project supported the Punjab Healthcare Commission which issues licenses to both public and private providers. Therefore, in this way, the private sector was also affected through improved regulation of their services, and ultimately their improved quality. Poverty Reduction and Shared Prosperity The entire population was to benefit from the project since government health services are available to everyone. Nevertheless, poor households, women, and other vulnerable segments of society were the primary beneficiaries of the project, residing mainly in the central and southern Punjab. The PDO specifically included them in its statement. The project targeted the low performing 18 districts of Punjab, namely: D.G Khan, Rajanpur, Muzaffargarh, Jhang, Bhakkar, R.Y Khan, Narowal, Bahawalnagar, Bahawalpur, Pakpattan, Kasur, Vehari, Lodhran, Chiniot, Khanewal, Okara, Hafizabad and Layyah. Performance in these districts, as measured in the 2 PDO indicators (immunization, birth skilled attendance) surpassed their targets and the indicator for CPR also met its target even though Punjab-wide, performance on that indicator only reached 15% of the target. Other Unintended Outcomes and Impacts – NA III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION The overall project quality at entry was driven by the following factors: Analytical work: The PHSS and the project were prepared in 2012/13 based on an extensive Situation Analysis prepared in March 2012, covering the six dimensions of the healthcare system, including service delivery, governance and accountability, health workforce, health information systems, essential medicine and medical technologies, and health financing. Analysis across these dimensions concluded that overall poor management and systemic weaknesses were the primary causes behind many of the failed healthcare initiatives and poor health outcomes. The report concluded that there needed to be a clear provincial level health policy, implementation plan with clear targets, supported by deployed capacity and resources in order to advance the healthcare agenda in Punjab. Project preparation & timetable: This project was prepared over an 18 months period, with the Project Concept Review held in January 2012 and approval in May 2013. This was a joint effort with DFID and all Bank Aide-memoires explicitly stated that. In fact, the reference was always made to the “proposed Bank/DFID Sectoral Support.” The project was planned to be implemented over a 4-year period which was an ambitious undertaking, especially considering that 2013 was an election year and implementation experience shows that very little can be done on both sides of the election period, and especially after a change of government. Page 32 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Project design: The overall objectives and the design of the project were consistent with the goals of the PHSS and with the overall Bank CPS, as discussed above in this document. However, the PDO was very broad, “to support the PHSS” rather than specifically reflecting the goals of components. This was likely a result of a recommendation made at the Decision Meeting which asked the team to include a direct reference to the PHSS in the PDO. However, the “contributing” objective was left out of the PDO but was actually a critical part of the PHSS and the project and should have been included explicitly. Also, the RF PDO indicators did not include measures for those critical parts of the project, such as management reforms. They were, however, included as part of the original 15 DLIs (and later dropped). The choice of the PDO was also unclear in view of the fact that the GoPB (in departure of previous supply- side interventions) explicitly requested the Bank to support management and governance reforms and strengthening stewardship functions. This was noted by the Bank team during the QER two weeks prior to the Decision Meeting. Overall, the project turned out to be very complex (which was pointed out in two review meetings) and the number of components grew from the original 3 to 4. The project used a new DLI financing modality, the performance-based contracting at the district level when districts had little autonomy, and the requirement for third-party verifications of DoH data, all of which were new and complex mechanisms to undertake. It also included innovation pilots to be co- financed by the HRITF. The very positive aspect of preparation was the close collaboration with DFID, with both development partners using the same reform framework and the same results-based approach (although DFID’s approach included a result focus but also an intensive focus on planning and implementation of specific outputs). One notable aspect of the project which was highlighted as critical in preparation documents was the Population Welfare Program (PWD) where progress was lagging. Reference was made to the National Population Policy of 2002 which set targets increasing the CPR from 30% to 60% and the fact that there were two parallel programs providing family planning services in the public sector (which is still the case today). Yet, the project design did not explicitly include the PWD and support for this program was only included during the 2017 restructuring by adding 4 DLIs on family planning, although it was still run by the PSPU. Another issue which was not addressed in the PAD was an explicit recommendation made by the QER panel to carry out an economic analysis. Lessons taken into account: The project design incorporated lessons learnt from previous Bank engagements, including: (i) support should focus on working with and through provincial governments in line with their strategic priorities after devolution; (ii) support should be provided through existing regular management structures to ensure sustainability after project completion; (iii) Bank financing should not create any fiscal dependency – therefore, overall project financing should be embedded in the provincial budget; (iv) results-based financing approaches can be very successful in rapidly increasing the implementation of cost-effective health interventions and in emphasizing results and outcomes; (v) the use of pilots is encourage to allow the government to explore options for future scaleup of RBF approaches. Clearly, the entire project design took these lessons into account as project support was imbedded into an overall government program, focusing on developing institutional mechanisms and supporting management reforms, including results-based contracting and innovative pilots. Risks identification and mitigation measures: The overall project risk was rated as Substantial, with governance and stakeholder risks identified in the PAD as most critical. The PAD noted that successful implementation of the PHSS would need sustained leadership at the provincial level in the post-elections period. Many of the key risks identified turned out to be actual problems during the project and include: (i) the risk of lacking sustained sector leadership; (ii) possibility of changes in political leadership and the reform agenda post elections; (iii) decreased buy-in and commitment for reforms; (iv) inadequate capacity at provincial and district levels, especially for enhanced stewardship Page 33 of 76 The World Bank Punjab Health Sector Reform Project (P123394) functions; and, (v) inadequate ownership of preventive programs by the provincial government. Two additional issues, which were not identified as risks in the PAD, had a significant impact on the project, namely the lack of consensus by government on the innovation pilots (voucher/cash incentive) to be supported by the HRITF and the potential for DFID service delivery program overwhelming the focus on the reform agenda supported by the Bank-financed project. B. KEY FACTORS DURING IMPLEMENTATION The following issues reflect the complexities of project implementation, negatively affecting project outcomes. Changes in the DoH Leadership. The project was approved on May 31, 2013, at the same time as the General elections with subsequent formation of the new government and ensuing series of staff changes, particularly changes in the management of departments. Most notably, the Head of the Policy and Reform Unit in the Department of Health who was instrumental during project preparation was moved to another position. A number of other staff also changed, with the new staff not having the background of reforms or the project design and its financing modality. As a result, the level of understanding and commitment to the reform program waned and there were many delays in decisions. Also, there was a tendency for every new Health Secretary to revisit the entire project design and to question and occasionally reverse previous agreements made. For example, there was renewed interest in investment in hospital infrastructure (building teaching hospitals in every district) and less focus on the softer management reforms, some of which were to be supported by the project. While the Bank remained open to the idea of restructuring the project and re-designing the reform components, every change in leadership meant that these new designs were re-discussed and revised. The following Table 5 shows the extent of changes in leadership of the Department of Health and of the Project Directors. Table 5: Changes in Leadership Tenure of Health Secretaries From To 1 Capt. Arif Nadeem 02-18-2012 06-09-2013 2 Hassan Iqbal 06-10-2013 01-18-2014 3 Babar Hayat Tarar 01-22-2014 03-31-2014 4 Dr. Ijaz Munir 03-31-2014 09-01-2014 5 Jawad Rafique Malik 09-01-2014 12-31-2015 Secretaries of Primary & Secondary HealthCare Department 6 Ali Jan Khan 12-31-2015 06-20-2018 7 Ali Bahadur Qazi 06-26-2018 09-13-2018 8 Capt. Saqib Zafar (Add. Charge) 09-19-2018 11-02-2018 9 Mr. Zahid Akhtar Zaman 11-02-2018 Current Project Directors 1 Mr. Muhammad Farsat Iqbal 09-01-2010 06-26-2013 2 No PD 06-26-2013 07-31-2013 3 Mr. Muhammad Haroon Ur Rafique 07-31-2013 01-24-2013 4 Dr. Akhtar Rasheed 02-04-2014 06-30-2014 5 No PD 07-01-2014 09-10-2014 6 Mr. Ali Bahadur Qazi 07-31-2013 04-20-2016 7 No PD (additional charge assigned to Ms. Zahida Sarwar) 04-20-2016 05-20-2016 8 Mr. Muhammad Khan Ranjha 05-20-2016 09-25-2017 Page 34 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 9 No PD (additional charge assigned to Ms. Fatima Sheikh) 09-25-2017 05-18-2018 10 Dr. Shagufta Zareen 05-18-2018 current Division of DoH into two departments. The changes listed above were also a result of a decision in December 2015 to divide DoH into two departments, namely, the Primary and Secondary Healthcare Department (PSHD) and the Specialized Healthcare and Medical Education Department (SHMED). With all senior management changed, focus on reforms became more diluted and further negatively affected implementation. Prior to this division, the project supported reforms, such fiduciary, human resources and M&E, some of which were partly implemented. However, the division of the PHSRP into two department led to ad hoc sharing of resources between the departments, gaps in implementation capacity and ultimately to changes in the focus of both departments, away from the management and structural reforms, to a much greater focus on service delivery which was also reinforced by the high levels of commitment to the implementation of the Health Roadmap that was led by the Chief Minister from 2014-2018. Change in approach: Health Roadmap. As noted earlier, during project preparation, the Bank worked closely with the government and DFID on planning of the PHSS and specific ways that each development partner would support it. Overall, it was a strong partnership, with regular missions and joint reviews. DFID had worked in Punjab extensively and supported seven priority national health and population welfare programs via the National Health Facility (2003- 2010) as well as supporting the National MNCH Program to improve MNCH services which started in 2006. The Bank and DFID adopted a common approach, using DLIs to support this project, with DFID supporting some of the key preliminary reform actions in the 1st year of the PHSS, disbursing GBP14 million. In 2013/14, there were staff changes in the DFID team while there were also changes on the government side post- elections, with a number of people who worked on the preparation of the PHSS gone. During this time, DFID launched its new Provincial Health and Nutrition Programme Business Plans (2014-2019) within the original financing envelope of GBP90 million. The program now focused on the implementation of the EPHS at the primary healthcare level (including community level, i.e. CMWs and LHWs) and on introducing and scaling up the RMNCH initiatives in Punjab. The financing modality was focused on outputs, closely tracking all actual expenditures, and the approach was closely aligned with the launch, in early 2014, of the new Health Roadmap by the Chief Minister of Punjab who was also the head of DoH. This approach was used in the education sector and was also developed in 2013 for the health sector, with direct support and instruction of the Chief Minister. The goal of the Roadmap approach was to achieve transformational, fast results by focusing on a few priorities, using a top-down approach, generating data, regularly reviewing results and making quick adjustments. This approach relied on monitoring a small set of “smart” indicators focusing on facility functionality at first and later validating output indicators (number of children immunized, deliveries at all levels, OPD visits at all levels, children screened for malnutrition, etc.) with progress being reviewed at bi-monthly stock-taking meetings led by the Chief Minister where decisions on course correction were also made by him. While this elevated issues to the top leadership level this approach also reduced involvement and empowerment of the technical specialists in decision-making. The Bank team raised concerns that this overwhelming focus on Roadmap indicators was developing a misalignment with the original reform program, limiting its scope significantly. While there were some positive achievements under the Roadmap approach, such as increased financing of BHUs and improved utilization rates, the exclusive focus on Roadmap indicators led to increased hiring at BHUs at the expense of higher levels (for example), thus leading to HR shortages. The approach also reduced attention to planned management reforms supported by the Bank-financed DLIs. Because of this disconnect, the project was not implementing as planned and was less than 25% disbursed by mid-2015. Gradually, the implementation support missions of the Bank no longer included liaison with the Roadmap Page 35 of 76 The World Bank Punjab Health Sector Reform Project (P123394) team to the same extent as before, and eventually the engagement with DFID became less and less close as well. Suspension and cancellation of the HRITF. This was a major issue in the project with the end result being that the DoH lost a large amount of grant funding for their reform program. First there was a major delay in the signing of the HRITF Grant Agreement (GA) which in turn led to delays in the Effectiveness of the Financing Agreement for the project because the failure to sign the HRITF GA by August 31, 2013 was included in the FA as an event of suspension of the FA and which had already occurred prior to the signature of the FA on Dec 2, 2013. Therefore, the due date of HRITF GA signature had to be delayed to April 30, 2014. It was actually signed in July 2014. The commitment to the activities planned to be supported by the HRITF was likewise affected as new priorities emerged and because of lack of support from the top leadership for these pilots. The areas which HRITF was supposed to support, including vouchers for MCH, health insurance, RBF for health facilities and performance contracts for district health officials were no longer being pursued, and after 3 years of discussion and debate, the grant of US$20 million was finally cancelled in January 2017. Likewise, the Bank-executed portion of the HRITF (US$1.5 million) to finance the impact evaluation was also cancelled. This affectively meant that Component 4 was never implemented as planned. Project restructuring in February 2017. Given delays across numerous activities, piecemeal implementation of reforms, wavering commitment and indecision on critical project interventions, the focus on the Roadmap, and resulting limited disbursements, the Bank worked with GoPB to find ways to restructure the project. This idea first surfaced in early 2015 and was discussed at length during the August 2015 Mid-term Review (MTR) mission, with a formal request letter for restructuring coming in September 2015. The request pointed out the need to replace DLIs under Components 2 and 3 and align them with ongoing reforms, including the bifurcation of the DoH into two departments. It also requested a 1-year extension, restructuring the HRITF, and the addition of three new DLIs on population, expansion of the LHWs and restructuring of the Punjab Health Foundation to support the private sector. Following the MTR, the context changed again with the bifurcation of the DoH, and the Bank had to start discussions about restructuring again. Fundamental questions about project design and whether to continue the project were raised at an internal Bank QER in February 2016. The consensus was that the project needed to be simplified, dropping what was not working and strengthening focus on nutrition. It was recommended not to add new components. Following the QER, it took another year for restructuring to be finalized and approved because documentation took all of 2016 (because documentation for restructuring was only submitted to the P&D Department in August 2016) and the approval of the PC1 for the nutrition component was done only in October 2016. While the project was substantially changed with a focus on service delivery, the PDO was not changed. Creation of three implementing agencies with three Designated Accounts. This was done to give each agency autonomy over their finances but also created a much more challenging management situation by the PSPU to coordinate work of the other two departments as the PSPU had to prepare quarterly IFRs documenting expenditures against the advances received under each of the 3 DAs. The inclusion of the two new implementing agencies also meant the opening of their own Designated Accounts which took over 9 months and the approval of PC1 to allow them to expense funds through a regular reimbursement modality. Given that the project was only extended for 1 year, one wonders why this was necessary and in fact why an entire new component 5 had to be included whereas Component 1 was already including nutrition support, albeit through DLIs. DLI modality. At the start of the project, there appeared to be lacking capacity and understanding with respect to this mechanism. Overall, the DLI modality did not provide sufficient incentives for policy changes in the DoH, as planned. Partly, this was because DLIs were delinked from expenditures required to achieve results (with PSPU and other units not getting funding from Government to meet the DLIs). Also, the initial advance was not provided by the Bank, as Page 36 of 76 The World Bank Punjab Health Sector Reform Project (P123394) planned (US$17 million), which could have potentially eased that pressure. In the case of the Punjab Population Innovation Fund, although some seed funding was provided, there was insufficient budget available to meet the DLIs. Paradoxically, because the PPIF had no budget, the internal auditor would not allow for the PPIF to enter into contracts with providers which was a project DLI. As a result, that DLI could not be met. Post July 2018 election implementation freeze. The Planning and Development Department did not allow the PSPU to enter into any new contracts for the remaining 6 months of the project nor to incur any new expenditures. Notably, this affected the signing of a contract with UNICEF for US$4 million for nutrition commodities which was being negotiated for several months prior to this decision. Not only was this regrettable from the development point of view but it was wasteful and led to a complete standstill of the project. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design - Substantial This original design of the RF and M&E arrangements was generally reflective of the project design, i.e. the 6 PDO indicators included measures of coverage, quality, utilization and system performance, with three of them also measuring performance of the 18 low performing districts. The 13 IO indicators measured performance of Components 1-3, excluding Component 4 which was a support component for the project. The baselines and targets were included for all indicators although for a number of indicators, the sources of data were not coherent, with both MICS and PDHS mentioned for the same indicators multiple times, which later made reported results incomparable with the baseline. Also, the baseline for the PDO indicator on the immunization rate was based on sources which were not the latest. The MICS 2011 available at that time already had this indicator at 46.8% whereas the PAD included 34.6%. Right after project approval, the July 2013 implementation status report (ISR) reported the rate at 65.6% against the target of 60%. In other words, the project had already achieved its target not having even been started. This raises the question about the overall target setting and how rigorous targets were. The project outcomes were also to be measured by the achievement of 15 disbursement-linked indicators (based on third-party verification) which included actions/decisions taken, processes developed and implemented and also outcomes, such in CPR. Some of the RF indicators were also DLIs. The project also aimed to improve the M&E system to monitor the reform program. This included strengthening the M&E Cell in the DGHS office, with DFID also providing technical assistance to the Directorate for Information, Monitoring and Evaluation to integrate various vertical MIS systems. The monitoring of the project was to rely on the existing DHIS which was operational across Punjab, providing monthly reports from districts. In addition, the project included third-party assessments of DLIs which would also serve a monitoring and evaluation function as well as support for carrying out regular HFAs in order to monitor results from the implementation of the EPHS in primary and secondary health facilities. The HFAs would translate results into a Quality of Care Index which would also include patient satisfaction. The HFA was meant to be a tool which would also improve overall accountability and transparency of health service provision. M&E Implementation - Modest With respect to capacity strengthening of M&E, there were major delays with processing and approval of the PC-1 document for M&E (3-year delay). The DGHS M&E Unit was also not functional because of delays in hiring of staff. There Page 37 of 76 The World Bank Punjab Health Sector Reform Project (P123394) was also redirection of attention on the Roadmap indicators which were closely monitored but which were only a small subset of indicators. Overall, the state of DHIS health data is still weak with underdeveloped data systems and overreliance on periodic surveys. The DHIS generates data based on health facilities monthly submissions, reporting on their administrative and operational activities but the validation process is still weak which undermines its usefulness. There are also weaknesses in data aggregation and analysis. The RF was updated periodically starting in 2015 when MICS 2014 results became available. During the 2017 restructuring, there was an opportunity to update some baseline data to make them internally consistent, using a specific measure from one source. This was not done. Also, while targets were changed for a few indicators, there were some discrepancies between the restructuring paper and the legal amendment (this assessment used data from the latter). Also, there are other discrepancies for some indicators between documents. For example, the target for the indicator on proportion of children 6-24 months correctly identified with SAM was 90% in the legal amendment, it was not mentioned in the restructuring paper, it was then 60% in one ISR, and 4% in the next ISR. Also, while the PDO indicator on issuing licenses was broadened to include Category 1 and 2 facilities, the original target was kept for Category 1. Lastly, there were also various conflicting data provided for actuals (e.g. patient satisfaction vs. HFA conclusions) and in the case of the two nutrition indicators, the definition of these indicators were different in ISRs from what the PSPU was reporting; two different age groups were measured (see note in the RF). The project produced and relied on information from a number of assessments carried out by third party consultants. The first report to verify the achievements of DLI targets was done in January 2015 by the Technical Resource Facility as a result of delays in hiring consultants. The second major round of verifications was done in December 2018 to validate the final set of DLIs. This also included third party evaluations of the comprehensive EmONC services in DHQs in 18 low performing districts, a review of the contraceptive supplies and family planning visits, a review of the complaint management system in secondary healthcare facilities, and a review of the implementation of the healthcare waste management system in DHQs and THQ in 17 districts. These provided important findings and recommendations for improvements. In addition, a third- party evaluation of the LHW Program was carried out to access performance of the LHW Program and to identify ways to make further improvements. Lastly, two Health Facility Assessments were carried out, in 2014 and 2018 to assess facility readiness of primary and secondary healthcare facilities to implement the mandated EPHS (basic EmONC) and comprehensive EnONC services. The 2018 HFA assessed 905 health facilities including 26 DHQ Hospitals, 119 THQ Hospitals, 310 RHCs, 450 BHUs. As described above, the assessment revealed a number of major gaps in provision of care and this assessment can be used to guide the future reform agenda. M&E Utilization - Substantial The PSPU reported the usefulness of the log frame matrix during monitoring of project activities. Also, the assessments carried out were instrumental in providing evolving information and data to enable policy makers to closely monitor the various parts of the system and their functionality and effectiveness. The expectation is that the GoPB will continue to finance such assessments given their inherent value. The information and recommendations from the assessments are also now being used in the preparation of the proposed Human Capital project in order to continue to make improvements in health systems, with resulting increased coverage and utilization of quality health and nutrition services. Justification of Overall Rating of Quality of M&E - Substantial Overall, there were substantial efforts made to design a robust RF and M&E system to collect and report data under the project and for the PHSS. The project was monitored by the PSPU and a number of assessments were carried out which may not have otherwise been carried out due to their cost. The overall rating of Substantial reflects, on balance, the overall positive outcomes of the monitoring and evaluation work despite the delays and the various small inconsistencies Page 38 of 76 The World Bank Punjab Health Sector Reform Project (P123394) in the RF. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental Safeguard compliance performance of the project is rated as highly satisfactory. At appraisal, the environmental and social safeguard category of the project was classified as B and triggered OP/BP 4.01 (Environmental Assessment). No other safeguard policy was triggered. The project successfully achieved the DLI related to implementation of Environmental Health and Medical Waste Management (EMWM) in the targeted districts of Punjab province. The environmental health and medical waste management plan (EMWMP) developed by the DOH provided help in reducing the risk of infection, safety and health hazards, and guided principals for a safe and hygienic health care infrastructure. The EMWMP also outlines arrangements for the internal and external monitoring, capacity building needs, and budgetary arrangements for implementation. Fifteen districts of Punjab were selected to implement the EMWMP by DoH. In a total of 57 hospitals, 17 hospitals of District Head Quarters (DHQ) and 40 hospitals of Tehsil Head Quarter (THQ) are effectively implementing and following the standards protocols regarding the safe collection, segregation, proper storage, transportation, final disposal and record keeping of the infectious and noninfectious waste. Waste audit records are also maintained through three color-coded registers. A full-time senior Environmental Specialist was hired by the Project to ensure the implementation of the EMWMP in the project and worked as the Environment and Waste Management Focal Point (EWMFP). The EWMFP maintained vertical and horizontal coordination to ensure effective implementation of the EMWMP and was responsible for province-level documentation and reporting. At the district level, the Executive District Officer–Health (EDO Health) of each district was the focal point for performing/supervising the environment and health care waste management functions, and particularly for implementing the EMWMP in the respective district. At the facility level, the Medical Officer in smaller facilities and the Medical Superintendent in larger facilities designated as the focal point for EMWMP implementation. As part of institutional arrangements, directions were issued to all the districts for constituting district committees and hospital level committees to implement the health waste management rules. At each hospital level, a Hospital Waste Management Committees (HWMC) was formed by the PSPU. Hospital waste management teams and Health Waste Management Officers were notified in all the hospitals. Templates for Hospital Waste Management (HWM) Plans and monthly reporting were shared with all the THQ and DHQ hospitals. Regular training on HWM has been conducted by the Environmental Specialist-PSPU for master trainers in all 57 hospitals. A separate budget code has been created in the hospital management budget in each hospital by the DOH to ensure the implementation of HWM and its supplies. This is one of the key sustainable actions in terms of EMWM supported by the project. The reports from the third-party verification firm indicate satisfactory implementation of the hospital waste management for the safeguarding of public health and the environment. An online management information system (MIS) for hospital waste has been established under the project. The PSPU prepared the hospital waste audit protocols for all types of waste (infectious and non-infectious) under the EMWMP and Punjab HWM Rules 2014. A waste collection firm has been hired by the DOH Punjab to prepare the MIS and record the daily waste collection and disposal in all health care facilities (DHQs & THQs) across Punjab. The project interventions demonstrated one of the best examples of environmental safeguard compliance with discernable and remarkable achievement to the DoH Punjab by providing full support in helping to decrease the Page 39 of 76 The World Bank Punjab Health Sector Reform Project (P123394) hazards of various environmental and human adverse impacts during hospital waste collection at the health facilities level. The DoH has built their capacity in HWM planning, implementing and monitoring having adopted international protocols in reducing the risk of infection, safety and health hazards, and guided principals for a safe and hygienic health care infrastructure. Funds earmarked in the hospital management budget ensure the continuous supply of funds and health care supplies and maintenance. Using digital technology in managing infectious waste is another one of the positive outcomes. As a result of this EMWMP DLI, the DoH has submitted the summary to create a permanent position in the DoH to ensure and implement the HCWM in the entire province. It is recommended that for the sustainability of the best practices, the EMWMP should be continued and extend in the remaining districts of the Punjab province. Social safeguards were not triggered under the project. Fiduciary Compliance Financial Management. The FM performance of the project was Moderately Satisfactory. Overall, interim unaudited financial reports (IUFRs) were received on time. Audit reports were submitted on time and found acceptable. The Financial Management staff was adequate. During the Bank mission in November 2018, prior to project closing, it was highly recommended for the financial management specialist (FMS) to remain as part of the team until the end of the grace period. These recommendations mainly related to strengthening of internal controls and standardization of the documentation. It was also emphasized that as the project approached the closing date, the FMS should be supported by Government budget for smooth closing. The Government provided assurances that this matter was under consideration and would be resolved. However, it was not resolved. Also, despite the extension of the Grace period, by mid-June, 2019, the 3 DAs were still not closed and the Government had not returned unused funds. With regard to the DLI financing process, no separate PC1 was prepared specifically for the achievement of the DLIs under this program. As a result, each department had to use the limited budget from the existing PC1 to also meet the DLIs. The PSPU also relied on district budgets and on the TA component of the project to finance the costs to achieve some DLIs. Ideally, the department’s development budget sanctioned by the government should have included funds to support the activities for achieving the DLIs. Also, specific costs incurred to achieve the activities related to DLI are not fully traceable. There are a number of possible ways to enable the DoH to receive funding for the DLIs, including: (i) a separate PC1 can be prepared financed by Government – under government development budget or World Bank – using the advance mode under the TA component so that DLIs are achieved as required; (ii) the government can commit to allocate funds against development budget to finance the activities to achieve DLIs; and/or (iii) the finance department can transfer funds to the DoH once DLIs are achieved. The disbursements were also slowed down because of delays caused by a requirement to revalidate Designated Accounts (DA) due to changes in the government systems, starting in June 2017. Previously, the balances in the DA after June 30 would automatically carry over from the previous year and revalidation was not required. Under the new requirement, amounts were revalidated only up to the extent of the sanction in the government budget books. Recommendations: 1. The financial management specialist of the project must remain until the application Closing Date i.e. until the end of the grace period so that project is closed smoothly. This was a major problem faced for the closure of this project. The WB should allow the financing of the FMS until the end of the grace period or should ensure that government is financing the FMS. Page 40 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 2. The implementing entity should have strong commitment and monthly or even shorter period updates on the status of the activities to achieve the DLIs. 3. The program Director should be appointed for the entire period of the project, contingent on performance. Procurement. As part of the preparation for the PHSS support, it was agreed that the Bank and DFID will undertake joint assessment of the procurement capacity and of the existing system at the level of DoH, District Health Office and the Autonomous Medical Institute. Based on the findings of the assessment, the Bank/DFID teams proposed mitigation measures to address the identified weaknesses and devised implementation arrangements for the program. Procurement was not identified as a separate and dedicated function and was merged as a sub-function in the DoH’s administrative structure. Significant staff capacity constraints existed, with staff lacking skills and training and not fully conversant with the Public Procurement Rules. Based on the assessment findings the Bank project supported the following: (a) Establishment of an independent DoH Procurement Cell with skilled staff; (b) Development of departmental SOPs/Manuals (DoH, AMIs, district level) in line with the Punjab Public Procurement Rules; (c) Training/capacity building programs. The Procurement Cell became overloaded with the volume of activities as it was handling all government procurement for DoH. The objective of the cell was to build capacity and systems of DoH. However, the capacity building objective could not be achieved due to handling day to day routine procurement process work. After the DoH bifurcation, although the SOPs were developed and adopted under the former Secretary, the process was re-opened because the new Secretary and senior management wanted to re-review the SOPs and training plans again. This process was never concluded. The Bank team arranged multiple training and capacity building sessions. There were also a few sessions which were jointly done with DFID. With respect to activities under Components 4 and 5, separate staff responsible for procurement were deployed under PSPU, IRMNCH and MSNC. The procurement capacity in the three units overall remained adequate although there was frequent turnover of staff which raised the capacity risk during various periods of project implementation. There were delays between the planned and actual timelines in the Procurement Plan. This could have been improved by proper planning. involving the technical users and getting the technical requirements realistically planned. Overall, there was general compliance in procurement. C. BANK PERFORMANCE Quality at Entry – Moderately Satisfactory The Bank team worked closely with the GoPb and with DFID to advise on the key elements of the reform and also support project preparation as part of a joint strategy to reform the health system for better outcomes. Overall, the project quality and readiness was sound, based on detailed analytical work, a coherent and comprehensive government strategy (PHSS), with project design essentially mimicking the main tenets of that strategy. The project built on lessons of former engagements and incorporated novel and innovative design features as well as a new financing modality of DLIs and results-based financing. The Bank team ensured that the project was aligned with the existing CPS and the new PHSS. The Bank decided to integrate its support inside the PHSS Program instead of creating a separate implementation structure for the project. The risks were reasonably well-identified and the RF generally Page 41 of 76 The World Bank Punjab Health Sector Reform Project (P123394) reflected all aspects of support under the project and was supplemented by the DLI matrix of 15 DLIs which were also effectively output and outcome indicators. The issues with the baseline and measurement is discussed above in the M&E section. In retrospect, given the Bank’s previous experience in Pakistan (with two of the three health projects closed in 2010- 12 and rated as MU) the Bank team likely underestimated the complexity of the project given the context, and did not allow adequate time for its implementation. The project of this magnitude should have been planned for at least 5 years, especially given that 2013 was an election year. The team did not carry out an economic analysis as recommended, and likely needed more time for an institutional review of capacity for project and program management by the main implementing agency, especially given the complex reform being undertaken. Therefore, on balance, the overall rating for the Bank at entry is Moderately Satisfactory. Quality of Supervision – Moderately Unsatisfactory Continuity in Bank team. During the early phase of the project, there was very limited activity due to delays in project start-up. The same Task Team was involved in supervision as during preparation and the TTL only changed once mid- way through the project although that staff member was already part of the team before. Realism of ratings and reporting. The Bank’s initial ISR downgraded the rating for implementation progress to moderately satisfactory (MS) due to delays following elections. This rating was further downgraded in mid-2014 to moderately unsatisfactory (MU) but was then upgraded in 1 year following the MTR and ratings to Component 2 and 3 were also upgraded although they were not performing at all and constituted components with the majority of dropped DLIs. Also, at that time, disbursement only reached 18% with 2 years until closing. Three subsequent ISRs were exactly the same although the last one changed the IP rating back to MU for 6 months. The PDO rating remained S or MS during the entire duration of the project and only the final ISR reflected the actual achievement of the project and the PDO and IP ratings were both MU. The Results Framework was only updated in 2015 when the MICS 2014 data came out and then in 2018. The period prior to restructuring saw good reporting by the Bank with well-written detailed Aide-memoires. The Bank team also prepared an Issues Paper prior to the MTR and carried out a QER to discuss options for project restructuring, both of which are good practice. The same level of reporting was not visible, however, starting in mid-2016 with a 1- page Aide-memoire filed in October 2016 and in January 2017, 1 brief AM filed in July 2017 and none in 2018 that the ICR team could obtain. Project restructuring. The team was proactive in trying to find ways to restructure the project to align with the developments on the ground. The Bank made the right decision to include nutrition interventions as it was and remains a critical issue for Punjab. However, it is less clear why a separate component needed to be added. Restructuring took 2 years to finalize since it was first mentioned in early 2015, some of which was due to delays on the Bank side as well. With respect to changes made, it was good that for 4 PDO indicators, the targets were revised up given achievement but the baseline for at least the immunization indicator should have been changed as well. With respect to the PDO, the team should have changed it to align with the revised components as the broad support for the reform agenda was no longer valid, as mentioned by the team in a number of documents. Another questionable decision was to add two new agencies, as mentioned above, which entailed long delays with opening of DAs and hiring fiduciary staff, etc. The QER explicitly recommended not to add complexities into the design, especially since there was agreement to extend the project for only 1 year. Lastly, restructuring included TA to support reforms Page 42 of 76 The World Bank Punjab Health Sector Reform Project (P123394) under the Specialized Healthcare and Medical Education Department but none of that work could be done because funds could not be transferred as this department was not officially part of the project. Verification and payment against achieved DLIs. DLI payments were made on five separate occasions, with 5 DLIs paid in 2015, 1 DLI paid in 2017 and 7 DLIs paid in April 2019. With respect to the last payment, the submission of the verification results was on January 1, 2019 but the Bank response letter was dated April 16, 2019. It is unclear why there was a delay of almost four months. In addition, some decisions by the Bank team with respect to DLI approval and payment remain unclear. Specifically, the DLI on PPIF establishment was met in July 2017 (per Bank Aide- memoire) but it was only paid in April 2019. Also, there appear to have been arbitrary decisions with respect to three DLIs. The media campaign was carried out from November 2016 to June 2017, but the Bank decided it was not rigorous enough and the DLI was not approved although the legal DLI definition did not include any other requirements. There is no explanation for this decision in any documents or during the ICR preparation. On the other hand, the Bank approved the payment for the DLI on having at least six contracts with NGOs when the verification report said that only five contracts were signed. Also, the DLI on family planning utilization was approved and paid although the Task Team Leader said during the ICR mission that this DLI could not be considered as met because only DOH data could be counted and not PWD. But the legal amendment does not have that distinction and in the end this DLI was approved. Engagement with DFID, GoPb and Bank staff. The GoPb expressed their appreciation to the Bank for continued support on the policy agenda over the life of the project. However, Bank and DFID staff and consultants expressed regret at the level of engagement of the Task Team Leader with external and internal staff and that there was limited sharing of implementation support documents. Justification of Overall Rating of Bank Performance The overall rating reflects the Bank’s positive and strong support to the GoPb efforts to strengthen the health system and thereby affect MNCH and nutrition outcomes. While there were a few issues with the overall quality at entry, issues during the initial implementation phase as well as the political constraints outside the Bank’s control had a negative effect on the project. While the Bank remained engaged in the reform discussions with GoPb, it became a more reluctant partner as a result of the Roadmap program and the changing focus on quick wins rather than a longer- term approach to reform, as originally envisioned. The restructuring which the Bank processed was done too late in the project and the implementation arrangements designed required two more agencies and hence two more teams and accounts, for only a year more of implementation. In the end, the Bank was unable to impact positively on the final outcome of the project through its actions. Therefore, on balance, the rating is Moderately Unsatisfactory. This also correlates with the overall Moderately Unsatisfactory rating for Outcome. D. RISK TO DEVELOPMENT OUTCOME The achievements under this project are expected to be continued under the proposed Punjab Human Capital project (PHCP), currently under preparation. The proposed project will build on work carried out and continue to support the GoP strategy to improve health performance, especially in maternal and child health and nutrition outcomes. The new project will focus on increasing the availability and utilization of human capital investment services for poor and vulnerable populations, including improving the uptake of quality primary HNP service as well as their availability by women and children under the age of two. The project would finance conditional cash transfers to eligible pregnant and lactating women to incentivize antenatal care visits, delivery by skilled attendant, growth promotion and immunization of children under two years of age, as well as participation in counseling sessions for better early Page 43 of 76 The World Bank Punjab Health Sector Reform Project (P123394) childhood development. To ensure good quality of services, minimum service delivery standards (MSDS) supported under the original project would be used for primary and secondary health care. In addition, to sustain momentum on improving the CPR, support for family planning services would also be included, working jointly with the PWD. Therefore, the new project would continue support for service delivery and hence the risk to development outcomes in this respect is low. DFID also plans to continue to support Punjab in areas of MCH and nutrition and is currently in the planning stage of their support. Based on discussions during the ICR mission, the GoPb is committed to achiving its agreed targets for key health outcomes to meet the SDGs and to continuing progress made in several areas, such as HCE licensing, the complaint management system, the hospital waste management system, etc. The Government is also continuing to improve allocations to the health sector. As compared to 2010/11, when the overall public health expenditures constituted 0.2% of GDP, by 2017, this was 1% of GDP. The Government of Pakistan has set a target of 3% by 2025. Funding for the health sector at the provincial level has also increased substantially. According to one ICR interview with government staff, these increased outlays were a direct result of DFID and WB emphasis on this issue over the years. Also, the new 2019-2028 Punjab Health Sector Strategy is currently under discussion. This document has been prepared in close collaboration of all International Development Partners working in Punjab and it outlines GoPb strong commitment to ensuring universal health coverage and quality of care without compromising patient safety and their rights. Some of the key priorities in the new strategy include patient safety and quality of care, infection control, hospital waste management, environmental/one health, health financing and public private partnership. Focus on these issues will ensure the sustainability of investments made under this project. V. LESSONS AND RECOMMENDATIONS There were a number of lessons and recommendations which can be integrated into future operations and Government programs. 1. Government commitment to the overall reform program should not be mistaken for Government commitment or ability to implement all the necessary steps required. Therefore, new approaches, such as the use of DLIs and results-based contracting requires significant technical assistance during implementation as well as strong commitment from Government to implement. 2. The complexity and ambition of project design, particularly in reform-oriented projects, needs to reflect existing technical capacity, commitment, and the challenging contexts such as in Pakistan, taking into account the implementation performance experience that several HNP projects have faced in the past. 3. Given that the risks identified during preparation ultimately played out during implementation, there is a need to ensure that effective mitigation strategies and measures are deployed to better manage risks and avoid overly formulaic risk mitigation strategies. 4. Projects using the DLI modality require that the Government ensures adequate allocation of resources to the relevant department budgets to enable them to achieve DLIs. The Government could also choose to transfer the amounts equivalent to DLIs achieved once the Bank makes its transfers to the Ministry of Finance. The latter approach would likely provide a greater incentive for meeting DLIs. . 5. Quick wins in service delivery, such as monitored through the Roadmap approach, need to be balanced with longer-term system improvements to make a real difference in the quality of service provision. 6. Development partners need to stay engaged and to coordinate their assistance as much as possible to ensure that agreed outcomes are being pursued in a consistent, joint way. Page 44 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 7. In cases of clear waning commitment to the original project design, restructuring needs to be carried out as soon as possible. 8. The Bank needs to have a strong multi-specialty team to provide implementation support instead of a single Task Team Leader operating alone. Recommendations: 1. Family planning needs to be fully integrated within the overall health service provision to be more effective at increasing demand for services. The woman-centered focus needs to be broadened to involve the other critical parts of the family (husband and mother-in-law). 2. The PPIF experience has shown that there may be viable, effective and less expensive alternatives to the provision of family planning services, as currently organized. Those experiences need to be reviewed and piloted. 3. The GoPb should continue the practice of carrying out regular assessments to inform direction of reforms, and should also review carefully the findings of the various previous third-party assessments, as well as the HFA, conducted under the project which include lessons and recommendations. 4. The preparation team of the proposed Human Capital Project needs to closely review why the voucher/cash incentive pilots, using support from HRITF, were not implemented to increase demand for health services. This is . critical for launching of the CCT program to ensure its success. Page 45 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Performance on CRIs Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion People who have received Number 0.00 11500000.00 11503280.00 essential health, nutrition, and population (HNP) services 31-May-2013 31-Dec-2018 20-Dec-2018 People who have received Number 0.00 7500000.00 6503280.00 essential health, nutrition, and population (HNP) services - Female (RMS requirement) Number of children Number 0.00 2500000.00 2450000.00 immunized 31-May-2013 31-Dec-2018 20-Dec-2018 Number of women and Number 0.00 6500000.00 6540000.00 children who have received basic nutrition services 31-Dec-2015 31-Dec-2018 20-Dec-2018 Page 46 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Number of deliveries Number 0.00 2500000.00 2513280.00 attended by skilled health personnel 31-May-2013 31-Dec-2018 20-Dec-2018 Comments (achievements against targets): Objective/Outcome: Improvement in coverage Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Fully immunized children 12- Percentage 34.60 60.00 80.00 76.50 23 months of age 31-Dec-2012 30-Jun-2017 31-Dec-2018 20-Dec-2018 Average for 18 low Percentage 23.90 52.50 60.00 75.00 performing districts 31-Dec-2012 30-Jun-2017 31-Dec-2018 12-Dec-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of children 6-24 Percentage 10.00 60.00 80.00 84.00 months of age in the 18 low performing districts receiving 06-Oct-2018 30-Jun-2017 31-Dec-2018 31-Dec-2018 the basic package of nutrition services Comments (achievements against targets): Page 47 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Objective/Outcome: Improve utilization Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Contraceptive Prevalence Percentage 29.00 35.00 29.90 Rate 31-Dec-2012 30-Jun-2017 30-Dec-2018 Average for 18 low Percentage 22.60 28.50 28.50 performing districts 31-Dec-2012 30-Jun-2017 12-Dec-2018 Comments (achievements against targets): Objective/Outcome: Improve Quality Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Births attended by skilled Percentage 58.50 70.00 75.00 76.40 health personnel 31-Dec-2012 30-Jun-2017 31-Dec-2018 31-Dec-2018 Average for 18 low Percentage 45.40 60.40 65.00 68.40 performing district 31-Dec-2012 30-Jun-2017 31-Dec-2018 12-Dec-2018 Comments (achievements against targets): Objective/Outcome: Improve health system performance Page 48 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of community Percentage 0.00 50.00 80.00 88.50 members satisfied with services at public health care 31-Dec-2012 30-Jun-2017 31-Dec-2018 12-Dec-2018 facilities Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Category -1 and 2 Percentage 100.00 500.00 2353.00 Health Care Establishments issued with provisional 31-Dec-2012 30-Jun-2017 12-Dec-2018 licenses (certificate of registration) by Punjab Healthcare Commission Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: Improving Health Service Delivery Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 49 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Number of THQ hospitals Number 40.00 75.00 50.00 providing 24/7 comprehensive EmONC 31-Dec-2012 30-Jun-2017 30-Oct-2018 services Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of RHCs providing Number 150.00 291.00 311.00 24/7 basic EmONC services (Number, Custom) 31-Dec-2012 30-Jun-2017 30-Oct-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of IDUs reached Percentage 50.00 75.00 65.00 by the Program who reported using a new syringe at their 31-Dec-2012 30-Jun-2017 01-Aug-2015 last injection (Percentage, Custom) Comments (achievements against targets): Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 50 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Target Completion Monthly average number of Number 62.00 100.00 72.00 family planning clients provided products and 31-Dec-2012 30-Jun-2017 31-Oct-2018 services at RHC Comments (achievements against targets): Component: Component2: Enhancing Efficiency and effectiveness of the Health System Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Daily average number of OPD Number 180.00 240.00 351.00 visits in BHUs and RHCs 31-Dec-2012 30-Jun-2017 30-Oct-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of LHWs with no Percentage 80.00 10.00 7.00 stock-out of family planning methods 31-Dec-2012 30-Jun-2017 30-Jun-2018 Comments (achievements against targets): Page 51 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of districts with Percentage 74.00 85.00 96.56 their respective BHUs providing all essential drugs 31-Dec-2012 30-Jun-2017 30-Oct-2018 Comments (achievements against targets): Component: Component 3: Strengthening Provincial Department of Health management capacity Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of districts which Percentage 20.00 75.00 61.00 are able to utilize more than 90% of the released funds in 31-Dec-2012 30-Jun-2017 12-Dec-2018 the same FY Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Annual Health Report Yes/No N Y N published 31-Dec-2012 30-Jun-2017 30-Jun-2018 Comments (achievements against targets): Page 52 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of the budget Percentage 7.00 20.00 11.70 allocated for preventive programmes out of total 31-Dec-2012 30-Jun-2017 12-Dec-2018 provincial government health budget Comments (achievements against targets): Component: Component 5: Strengthening Nutrition Interventions Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of pregnant Percentage 10.00 60.00 60.00 women registered wth LHWs receiving IFA tablets during 30-Dec-2012 30-Jun-2017 31-Dec-2018 the last pregnancy Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of children 6-59 Percentage 10.00 80.00 70.00 months of age being screened (at least biannually) for 01-Sep-2016 30-Jun-2017 31-Dec-2017 malnutrition in the target Page 53 of 76 The World Bank Punjab Health Sector Reform Project (P123394) districts Comments (achievements against targets): This indicator actually measures proportion of children 0-24 months of age. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of children 6-59 Percentage 1.00 90.00 1.40 months of age (correctly) identified with Severe Acute 01-Sep-2016 30-Jun-2017 28-Feb-2018 Malnutrition (SAM) in total screened children in the target districts Comments (achievements against targets): This indicator actually measures proportion of children 6-24 months of age. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of children with Percentage 1.00 40.00 77.00 SAM registered for treatment at stabilization centres in 01-Sep-2016 30-Jun-2017 30-Oct-2018 target districts Page 54 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Monthly average number of Number 62.00 60.00 65.00 family planning clients provided products and 31-Dec-2012 30-Jun-2017 30-Oct-2018 services at RHC (Number, Custom) Comments (achievements against targets): Page 55 of 76 The World Bank Punjab Health Sector Reform Project (P123394) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 - improved coverage of essential health and nutrition services • Fully immunized children 12-23 months of age (baseline 34.6% from MICS 2007, although MICS 2011 refers to 43%; original target 60%; revised target 80%; actual 76.5% which was 86% of the target achieved). With respect to 18 low performing districts, the baseline was 23.9% with a target of 52.5%. The target was increased to 60% during restructuring and actual Outcome Indicators achievement was 75%. Therefore, overall, this indicator was achieved. • Proportion of children 0-24 months in the 18 low performing districts receiving basic package of nutrition services (baseline 10%; original target 60%; revised target 80%; actual 84%) – surpassed • Number of RHCs providing basic 24/7 EmONC services (baseline 150; target 291; actual 311) –surpassed • Monthly average number of deliveries in one RHC per month on average (baseline 33; target 60; actual 65) – surpassed • Percentage of pregnant women registered with LHW receiving IFA tablets during pregnancy (baseline 10%; target Intermediate 60%; actual 60%) – achieved Results Indicators • Proportion of children with SAM registered for treatment at stabilization centers (baseline 5%; original target 40%; actual 77%) – surpassed. This was also supported by DLI4 referring to 12 under-performing districts and was not achieved and dropped in 2017. • Proportion of children 0-24 months of age being screened (at least biannually) for malnutrition (baseline 10%; target 80%; actual 70% from 2017; no data at end of project) – not achieved. Component 1 DLI1: Punjab has operationalized the integrated management of three community – based programs (Lady Health Workers, MNCH, and Nutrition program) and approved PC-1s for integrated management for a) Reproductive health, primary health care, and nutrition; b) LHW program, and c) MNCH. This DLI was achieved which was critical in improving the overall Outputs by coverage of services for women and children. Component 1 and 5 DLI3: 15,000 LHWs trained on FP and nutrition – achieved. DLI5: A minimum of 6 HIV/AIDS service contracts with NGOs – achieved, although the IVA report from January 2015 notes that only 5 contracts were signed, not 6, as stipulated in the DLI. Page 56 of 76 The World Bank Punjab Health Sector Reform Project (P123394) DLI 19: 90% of DHQs in 18 low performing districts providing C. EmONC services – achieved. Component 5 Nutrition commodities were procured through a contract with UNICEF (US$14 million, of which US$12.7 million was spent). This contributed to increasing the proportion of children 0-24 months in the 18 low performing districts receiving a basic package of nutrition services. Objective/Outcome 2 – improved quality of essential health and nutrition services • Births attended by skilled personnel – (baseline 58.5%; original target 70%; revised target 75%; actual 76.4%). With Outcome Indicators respect to the 18 low performing districts, the baseline was 45.4% with a target of 60.4%. The target was increased during restructuring to 65% and actual achievement was 68.4%. This indicator has surpassed its targets. • Proportion of children 6-24 months of age correctly identified with SAM in total screened children (baseline 1%; Intermediate target 90%; actual 1.4% from 2017 data; no latest data available) – not achieved. Results Indicators • Number of THQ hospitals providing 24/7 comprehensive EmONC services (baseline 40; target 75; actual 50) – not achieved Outputs by DLI 2: At least 70% skilled birth attendance (see above) Component 1 Objective/Outcome 3 – improved utilization of essential health and nutrition services • Contraceptive prevalence rate (any modern method) (baseline 29% although MICS 2011 refers to 35%; target 35%; actual 29.9%) – not achieved. This was also supported by DLI 2 which was not achieved and dropped. With respect to the 18 low Outcome Indicators performing districts, the baseline was 22.6% with a target of 28.5%. The target was not increased during restructuring and actual achievement was 28.5%. • Monthly average number of family planning clients provided products & services in on RHC on average (baseline 62; target 100; actual 72) – not achieved. Intermediate • Percentage of IDUs reached by the Program who reported using a new syringe at their last injection (baseline 50%; Results Indicators target 75%; actual 65%) – not achieved. • Daily average number of OPD visits in BHUs (baseline 40; target 60; actual 62) and RHCs (baseline 140; target 180; actual 289) – surpassed Page 57 of 76 The World Bank Punjab Health Sector Reform Project (P123394) DLI 16: Media campaign for family planning services designed and implemented – not achieved. DLI 17: Punjab Population Innovation Fund (PPIF) fully operational. The PPIF was established in October 2016 as a section 42 company (not for profit government agency) with the objective to fund innovative projects improving quality Outputs by and accessibility of family planning services. Component 1 DLI 18: PPIF awarded 5 contracts – not achieved as only 2 contracts could be signed during implementation (with Health and Nutrition Development Society and the Akhter Hamed Khan Resource Center). DLI20: Increase in FP service intake at DOH facilities in 18 districts - achieved. Objective/Outcome 4 – improved health system performance • PDO indicator and DLI 9. PHCC licensing of public and private Health Care Establishments (HCEs). The project’s PDO indicator was to have 500 Category -1 HCEs (hospitals with 50+ beds) issued with provisional licenses (certificate of registration) by PHCC. The project also included an associated DLI which had a target of 200 HCEs issued with provisional licenses. The total number of public and private HCEs is 299 in 2019, and the DLI target was to be achieved by December 2013. The total number licensed was 291 Cat 1 HCEs and 2,116 Cat 2 HCEs. In terms of achievement, the DLI was certainly achieved but the PDO indicator Outcome Indicators is not evaluable. • PDO indicator and DLI11 to increase user satisfaction with the government health services in Punjab. The DLI was dropped. Indicator is non-evaluable. Baseline of 0% and a target of 80%. 2014 HFA had an overall satisfaction with BHU and RHU services from 87-92%. Therefore, it is not clear why the baseline was zero. A community satisfaction survey was done as part of the 2018 HFA and reported overall satisfaction at 93% (the final ISR reported 85.5%). • Percentage of districts with their respective BHUs providing all essential drugs (reported as 96.56% against a project target of 85%) - surpassed Intermediate • Percentage of LHWs with stock-outs of family planning methods (baseline 80%; target 10%; actual 7%) - surpassed. Results Indicators The results of the HFA and the LHW assessment reported very low levels of drugs being present in facilities which puts in question the results of these indicators. Key Outputs by DLI 6: DoH has entered into revised results-based contracts with PRSP in line with EPHS for primary health care, in 14 ongoing Component 2 & new districts – not achieved. Page 58 of 76 The World Bank Punjab Health Sector Reform Project (P123394) DLI7: DoH has entered into performance management contracts with all District Health Offices and has transferred budgetary resources therewith - not achieved. DLI8: 80% of low performing (below the mean) districts of Punjab (18), identified by DoH, have individually attained a minimum increase of 5/10 percentage points in the composite index (scorecard) based on the list of Key Performance Indicators, measured against the 2012 baseline – not achieved. DLI 10: Emergency Helpline and the Complaint Management System (CMS) - achieved. A complaint management system in DHQ hospitals was set up as part of a broader effort in Punjab to establish an Emergency Helpline to facilitate patients in hospitals to resolve their complaints on immediate basis. 2 Health Facility Assessments carried out • IO indicator. Annual Health Report published. While 3 reports were published, this was to be an annual exercise. Objective/Outcome 5 – improved stewardship/management functions • % of districts which are able to utilize more than 90% of the released funds in the same FY (baseline: 20%; target was Intermediate 75%; actual was 61%) – not achieved. Results Indicators • % of the development budget allocated for preventive programs of the total development health budget (baseline 7%; target 20%; actual 11.7%) – not achieved. DLI 12: DoH has restructured and reorganized the DGHS pursuant to the revised roles and responsibilities, including the Key Outputs by establishment of a contract management unit – not achieved Component 3 and 4 DLI 13: DoH has established two cells that are financial management and procurement, at the provincial level - achieved DLI 15: Implementation of Medical waste management expanded to 15 districts - achieved Page 59 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Inaam Ul Haq Senior Health Specialist, Task Team Leader Aliya Kashif Operations Officer Anwar Ali Bhatti Financial Analyst Aristeidis I. Panou E T Consultant Chau-Ching Shen Senior Finance Officer Cornelis P. Kostermans Lead Public Health Specialist Hammad Yunus Consultant Hanid Mukhtar Senior Economist Javaid Afzal Senior Environmental Specialist Khalid Bin Anjum Procurement Specialist Luc Laviolette Sr. Nutrition Specialist Martin M. Serrano Senior Counsel Mohammad Khalid Khan Program Assistant Naoko Ohno Operations Officer Nasreen Shah Kazmi Program Assistant Rabia Ali Young Professional Samina Mussarat Islam Consultant Syed Waseem A. Kazmi FM Specialist Tayyeb Masud Senior Health Specialist Non-Bank Staff Raza Zaidi Health and Population Advisor Desmond Whyms Senior Health and Population Advisor Supervision/ICR Tayyeb Masud Task Team Leader(s) Khalid Bin Anjum Procurement Specialist(s) Noureen LNU Financial Management Specialist Page 60 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Afzal Mahmood Team Member Inaam Ul Haq Team Member Naoko Ohno Operations Officer Nasreen Shah Kazmi Team Member Payal Malik Madan Team Member Rahat Jabeen Environmental Specialist Aliya Kashif Team Member Najm-Ul-Sahr Ata-Ullah Social Specialist Maria Gracheva Sr. Operations Officer Lori Geurts Operations Officer Minh Thi Hoang Trinh Program Assistant B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY12 27.416 134,640.70 FY13 49.373 248,626.74 FY14 16.794 59,525.17 Total 93.58 442,792.61 Supervision/ICR FY13 .400 340.96 FY14 20.558 71,266.95 FY15 44.835 164,579.03 FY16 58.686 219,955.62 FY17 49.377 145,234.94 FY18 27.844 112,628.73 FY19 29.957 151,308.42 Total 231.66 865,314.65 Page 61 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ANNEX 3. PROJECT COST BY COMPONENT Amount Revised at Feb Revised at Dec Actual at at 2017 2018 Percentage of Components Project Closing Approval Restructuring Restructuring Approval (%) (US$M) (US$M) (US$M (US$M) Component 1: Improving 28.34 31.58 25.5 27.00 95% Health Service Delivery Component2: Enhancing Efficiency and effectiveness of 44.00 10.80 10.00 10.00 22.7% the Health System Component3: Strengthening Provincial Department of 22.66 10.80 10.50 10.00 44.1% Health management capacity Component4: Improving the 26.50 9.00 2.00 1.74 6.5% Capacities in Technical Areas Component 5: Strengthening 0 37.82 18.82 13.00 34.4% Nutrition Interventions Total 121.50 100.00 66.82 61.74 50.8% Note: The total disbursed of US$61.74 is the historical amount disbursed in US$ as of May 31, 2019 which is the project’s end of the grace period. Page 62 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ANNEX 4. EFFICIENCY ANALYSIS Economic analysis. The PAD did not include an Economic and Financial analysis although this was explicitly pointed out by the Quality Enhancement just prior to the Decision meeting in March 2013. This ICR did not attempt to estimate the potential economic rate of return or net present value of this project. However, to provide indications on the efficiencies and economic implications of key components, the following sections provide discussions on the economic analysis of the interventions supported by the project in light of global evidence base, as well as assessment of technical and allocative efficiency. Economic benefits of the interventions supported by the project. The cost-effectiveness of MNCH interventions has been established in the research literature, using as a standard measure cost per Disability-adjusted Life Year (DALY) averted. For example, the cost-effectiveness of a standard maternal and child health service package is estimated to range between US$24 and US$585 per DALY averted, while that of a standard package of prenatal and delivery care ranges from US$92 to US$148 per DALY averted. The economic benefits generated by the project were estimated using DALYs. The results indicate that for US$1 invested by the project, benefits generated are in the range of US$1.85 - US$5.55 when IMR is considered and US$2.96 - 8.88 for U5MR. The project has generated economic benefit of US$192–577 million from reduction of U5MR. Percentage of stunted children was reduced from 33 to 31.5 in 2014-2018. One-fifth of children were underweighted in 2018 whereas it was one third in 2014. Reductions in stunting are estimated to potentially increase overall economic productivity, as measured by GDP per capita, by 4–11 percent in Africa and Asia. Nutrition interventions have shown a high rate of return on investment–estimated to be between US$4 and US$35 for every dollar invested. Nutritional interventions are among the most cost-effective interventions to enhance welfare. The World Bank estimates that childhood stunting reduces a person’s potential lifetime earnings by at least 10 percent and it is estimated that undernutrition results in productivity losses of US$149 billion each year. In South Asia, the estimated cost of stunting is 10 percent of the GDP per capita. The project established 12 drop-in-centers in 9 cities at the cost US$5.67 million. Four of these sites were serving male sex workers and 8 were serving IDUs. The kinds of services they were providing included syringes, condoms and counseling, etc. The total number of registered clients at these sites was 21,435 in June 2014, with about 70 clients visiting each site per day. Around 80% of total registered clients received services. These sites provided services in 2014-2018 under the project and potentially gained more clients. The economic burden of HIV infection is substantial. The program prevented at least 6,516 from becoming infected with HIV infections and saved US$41.15 million generic treatment cost. Every US$1 spent on this HIV prevention activity saves US$7.25 in treatment cost. Therefore, drop-in-centers activities were likely to have high economic return. Given Pakistan’s GDP per capita, MNCH and nutrition interventions were cost effective. An intervention is cost- effective if the cost per disability-adjusted life-year (DALY) avoided is less than three times the national annual GDP per capita. It is highly cost-effective if it is less than the national annual GDP per capita. World Health Organization’s Choosing Interventions that are Cost–Effective (WHO-CHOICE) project recommended these under thresholds based on per capita national incomes approach. Technical efficiency. The project contributed to an increase in access to basic package of maternal, neonatal and child health (MNCH) and nutrition services and HIV prevention program. Among the 18 poor performing districts, the percentage of fully immunized children 12-23 months of age was increased to 75% from 23.90% while target Page 63 of 76 The World Bank Punjab Health Sector Reform Project (P123394) was 60%. Also, 84% of children 0-24 months of age were receiving the basic package of nutrition services. For this nutrition indicator, baseline and target values were 10 and 80% respectively. Daily average number of OPD visits in BHUs and RHCs was increased by 110 and 372% respectively. The increase in the total number of services suggests that health facilities and drop-in-centers are becoming efficient based on decreasing cost per service. The project also incentivized the training of 15,000 LHWs on family planning and nutrition and this activity possibly strengthened quality of care and links between outputs and outcomes. The activities likely generated better value for money as, for the most part, the project was focused on 18 low performing districts. Investment in MNCH and nutrition has important economic implications at population level especially in the low performing districts of Punjab. Well targeted public health interventions make a difference between intervention and comparison areas. Improved health status gained from investment contributes to social well-being through its impact on economic development and productivity. Allocative efficiency. Budget allocation for components 2 to 4 of the project was reduced by 57% (US$71.66 million to US$30.6 million) due to dropping of related DLIs and to ensure the relevance of the overall context. The project costs show that the emphasis of the project ended up primarily being MNCH (Component 1) and nutrition interventions (Component 5). These two components accounted for 66 percent of the project costs. Redirecting resources to MNCH and nutrition would not only allow for a more efficient use of resources but would also enhance the equity in service provision by ensuring that all the population of Punjab have equitable access to basic MNCH and nutrition services. Implementation efficiency. The following key issues negatively affected efficiency. • Delays in approvals and disbursements. There was delayed signing and effectiveness, including a delay in signing of the HRITF grant agreement. As a result, the Designated Account could not be open for Component 4 expenditures and financial management arrangements were not operationalized until late 2014. This also prevented an advance of funds (US$17 million) to be made for eligible expenditures incurred under the Eligible Expenditure Program (for financing of DLIs) prior to January 15, 2014, as was provided for in the FA because project effectiveness was declared only on January 17, 2014. Incidentally, there were also delays on the DFID side with funds released in 2013 not being utilized which prevented DFID from releasing its second tranche of GBP23 million in 2014/15. In addition, there was no budget allocated for the independent verification of DLIs in 2014. An independent consultant was finally fired in October 2014 to carry out verification, and the first time any disbursement was made for 4 DLIs was in March 2015, two years after project approval. There were also major delays in the approval of the PC-1 document for M&E (approved with a 3-year delay) which was a critical for the implementation of the reform program. After restructuring, there were major delays in opening of the new DAs in 2017, with no funds available for the two units to implement the new Component 5. Lastly, because the PC-1 document for the IRMNCH and Nutrition Program expired in June 2018, the Planning and Development Department did not allow the signing of the second major contract with UNICEF for nutrition commodities in the summer of 2018 (US$4 million) although the Bank provided its no-objection to the contract in April 2018. There were also delays in the review of the contract template. This also coincided with the General elections in July 2018 which led to further delays in approvals. The Finance Department also made a decision in 2018 to effectively freeze all donor funding by declaring all donor funds to be part of a “lapsable fund” whereas these were part of a “non-lapsable fund” before. The PSPU estimated that roughly 45% of time after the February 2017 restructuring was not spent on implementation, and that over the project period, 28.5 months were spent purely on bureaucratic delays and lacking funding to operate. • Frequent staff turn-over. There were multiple changes in DOH leadership and management and in PSPU directors, with a limited number of technical staff, for the duration of the project, first as a result of the general elections in Page 64 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 2013, then as a result of the bifurcation of the DOH into 2 departments, and later as a result of another general election in 2018. The section on implementation issues below provides more detail. • Cancellation of funds. During project implementation, two cancellations were processed, for a total of US$50 million of the original planned US$120 million (the final third cancellation was made in June 2019 to cancel the remaining US$0.4 million). The first was to cancel the US$21.5 million HRITF grant due to delays in internal reviews and approvals of costed results-based proposals. There were multiple discussions about lack of progress in this area and the Bank threatened to cancel funds, starting in early 2015. Actual cancellation was not processed until early in 2017. The second major cancellation of SDR 20.8 million (est. US$29 million) was processed on December 28, 2018 just prior to the closing date to enable Pakistan to retain these cancelled funds in its portfolio. The cancellation reflected the inability to spend funds as a result of various delays, as explained above. • Project timeframe. The project was restructured and had to be extended for a year to implement the remaining and new activities. Page 65 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Punjab Health Sector Reform Project (P123394) Punjab, Policy and Strategic Planning Unit Assessment of Project Outcomes 1. PROJECT CONTEXT Punjab Health Sector Reform Program (PHSRP) of World Bank was designed mutually in collaboration with government counterparts in post devolution scenario, from Federal to the Provinces, for implementation of Punjab Health Sector Strategy (PHSS). The main focus was improvement of the coverage and utilization of quality essential health services, particularly in the low performing districts of Punjab, by objective, building the capacity and systems to strengthen accountability and stewardship of Health Department. Initially, World Bank agreed to provide financial support of US$121.5 Million for Punjab Health Sector Reform Project for the implementation of PHSS over a period of 4 years starting from FY 2013-14 to 2016-17. Out of this, US$100 Million were from the IDA (soft loan) while remaining US$21.5 Million grant was from the Health Results Innovation Trust Fund (HRITF). The project was approved on May 31, 2013 and became effective on January 17, 2014 whereas project closing date was initially December 31, 2017 but later extended to December 31, 2018. The project envisaged to support Punjab Health Sector Reform Program under the following four components: Component 1: Improving Health Service Delivery (IDA US$28.34 million) Component 2: Enhancing Efficiency & Effectiveness of the Health System (IDA US$34.00 million and HRITF US$10 million) Component 3: Strengthening Provincial Health Department in Management Capacity (IDA US$22.66 million) Component 4: Improving the Capacities in Technical Areas (IDA US$15 million and HRITF US$11.5 million) The Health Sector Strategy (HSS: 2012-2020) was the basis for the design of the project and clearly outlined strategic directions focused on results, envisaging reforms in management, governance, and accountability, and the strengthening of health systems. The project was to support the DOH to implement the first phase of the reforms under the five-year strategic plan 2012-2017. However, since the inception of the project, there have been a series of changes in the Government reforms priorities and actions on initiated reforms supported under PHSRP, such as fiduciary, HR, and M&E, have not been implemented as planned. Further, the implementation of the HSS was in piecemeal. 2. OUTCOMES 2.1. Relevance to Project Development Objectives The project was primarily designed to support the Punjab Health Sector Strategy (PHSS), focusing mainly to increase the coverage and quality of healthcare services in low performing districts of the Punjab. 2.2. Achievement of PDO The project has significantly contributed in increasing the overall service delivery of the health sector. It has been observed that there appeared a visible change in key health indicators in the province of the Punjab. This indicates that number of immunized children has increased above 75% which was too low at the start of the Project. The percentage of delivery assisted by skilled birth attendant has also increased to more than 76%. The government has also tried hard for registration of Healthcare establishment with Punjab Healthcare Commission. Data also show that number of children that are receiving basic package of nutrition services has also increased to 84%, that has help to reduce level of stunting and wasting among children. Page 66 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Component 1: Improving Health Service Delivery (IDA US$28.34 million) The project has significantly contributed in enhancing coverage, quality and access to Essential Package of Health Services (EPHS) and including outreach and community level intervention for primary healthcare. EPHS includes nutrition, maternal, neonatal and child health, immunization, family planning, prevention of communicable diseases like malaria and tuberculosis, and service provision at community level through lady health workers, community midwifes and vaccinators. The project has also helped to scale up provision of micronutrients and deworming for adolescent. It has also led to expansion of comprehensive obstetrical (basic and comprehensive emergency obstetrical and neonatal care) and family planning services across Punjab particularly in 18 low performing districts. Component 2: Enhancing Efficiency & Effectiveness of the Health System Under this thematic area, project focused on efficiency, effectiveness of the health system and improving quality of care through regulation and standardization. This also includes increased accountability, improved monitoring and facilitation in public health surveillance system. It also includes third party validation of results through regular health facility assessment and surveys. Component 3: Strengthening Provincial Health Department Management Capacity Under this component, measures were taken to improve the performance of the Department of Health in post- 18 amendment scenario. Though, few components could not be completed due to change in priorities of the government. However, World Bank has supported in establishing the financial management cell to build fiduciary functions in the Health Department. In order to support procurement functions, sector specific SOPs/manuals were also developed to align process and procedures with the Punjab Public Procurement Rules. The project also supported component of environment and medical waste management to ensure safe disposal of infectious waste. Component 4: Improving the Capacities in Technical Areas This component aims to contribute to capacity building of Department of Health in technical areas. The project also supported need based analytical and research work to provide evidence for policy analysis. The areas wherein technical assistance was provided include quality of secondary care services, development of procurement standards, preparation of planning documents, managing of best practices in public audit, support to Population Welfare Department and Planning and Development Department in preparation of the PC-1 and strengthening of multi-sectoral nutritional center. PSPU has also received technical assistance in conducting third party validation of LHW Program and Health Facility Assessment. 3. KEY FACTORS THAT AFFECTED IMPLEMENTATION There were certain typical factors that generally affected the performance and implementation of the PHSRP. These factors were operational, managerial and financial in nature. 3.1. Delay in Project Initiation The project was planned in early 2013 with expected effective date 30 June 2013 but actually the project started in January 2014 due to delays in the approval process. But it further took around 1.5 years to release funds from the finance department in the project account due to cumbersome process of financial approval and releases. Such delays in project initiation seriously affected the performance of the project in achieving its project development objectives. Page 67 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 3.2. Bifurcation of the Health Department Punjab Health Sector Reform Project (PHSRP) was launched at time when there was a single department of health Punjab. In December 2015, DOH was bifurcated into i) Primary and Secondary Healthcare Department (PSHD), and ii) Specialized Healthcare and Medical Education Department (SHMED). The new organizational structure utilized the services of the HR of the cells created under PHSRP. This has led to changes in priorities of the both departments causing low pace of the progress of the project. Though, bifurcation of Health Department was an administrative decision, but it was a one of the significant change that meaningfully affected the implementation status and progress of the project. 3.3. Financial Issues Financial releases from Finance Department not only caused delay in project initiation but also effected the pace of the project during implementation phase, the changes in financial process and procedures caused aggravated financial issues. During the last year of the project, Finance department decided to declare all donors’ funded projects rolling fund account into lapsable fund account which were earlier non-lapsable account. This decision resulted into freezing of project funds and did not let it perform any activity. 3.4. Repeated changes in DOH leadership Frequent changes in the senior management of DOH led to discontinuity in the decision making and reform directions. The lack of a transition mechanism to ensure a smooth transfer of responsibilities within the government created an environment where every new incumbent revisited the entire project processes and, in most cases, reverted previous decisions and agreements. This process had severely affected the agreements and progress on key areas, especially the decisions on: i) the results-based financing model; ii) the district management performance-based contracting; and iii) the contracting model for low performing districts in order to utilize the HRITF Grant. The PSPU and the Bank team worked multiple times to design and develop reforms with all stakeholders. However, with every change in leadership, the design was again revisited and revised. 3.5. Transfers of Program Directors Project Head is appointed through transfer posting from the management cadre to steer the project. During the four-year life time of the project, 10 Program Directors were transferred that has affected the pace of the Project. Thus, enough time was required for every new officer to understand the dynamics of the Project. 3.6. Transfers of Secretaries of Health Department The administrative secretary also plays a major role in deciding about the vision of the department in achieving the desired objectives of the different programs. During four year of the project life, 09 Secretaries of the Department has been appointed that disturbed the pace of the project. 3.7. Multiple centers of decision making in DOH The health sector stocktaking exercise, driven by the Roadmap team, led by the Chief Minister of Punjab since late 2014, to match the results against the targets. The lack of an organized effort within the DOH had created a misalignment between the Roadmap indicators and the Health Sector Strategy. 3.8. Excessive Focus on disbursement-linked indicators: The excessive focus of the DOH on only DLI related indicators in the results framework had created a situation strategic objectives of the HSS were not given due attention. Page 68 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 4. RESTRUCTURING OF PHSRP The objective of the restructuring was to realign the PHSRP with the Government of Punjab’s (GOPb) new reforms’ priorities as well as with the structural changes in the Punjab Department of Health since project approval. In such changing contexts and bureaucratic instability, the Bank and GoPb agreed that restructuring of the project was required in order to align it better with the new reform directions by the GoPb. The principals on which this restructuring was being undertaken were: i) continue with intervention areas which are showing good progress; ii) enhance efforts on areas which are high priority; iii) drop activities which due to policy changes are no longer relevant to the reform agenda, and iv) not add new activities. The restructuring of the project entailed following changes: i. Revision of components to align with current priorities, including; a) adding a new component to strengthen the GOPb’s response to address a persistently high burden of stunting and malnutrition; addressing the slow reduction of the fertility rate; b) dropping 7 Disbursement Linked Indicators (DLIs) which were no longer relevant due to change in policy directions; c) modification of three DLIs to strengthen related activities; and d) addition of 7 new/modified DLIs to enhance focus on the results chain. ii. Change in implementation arrangements a) expanding number of implementing agencies to three; b) changing financial management and procurement arrangements to allow for the three entities; c) disbursement into three designated accounts for components 4 and 5; d) change of (eligible) expenditures category for nutrition interventions to statement of expenditure-based payments; e) reallocation among the eligible expenditure categories; and f) suspension of US$20 million Health Results Innovation Trust Fund (HRITF) grant. iii. Revision of the Results Framework a) removing indicators related to processes dropped by the government; b) addition of two indicators on nutrition; and c) revising indicators to reflect the agreements for the modified DLIs. iv. Extension of the Closing Date of the project by one year until December 31, 2018 5. PROJECT EVALUATION 5.1. Status of Project Disbursement Linked Indicators There were fifteen (15) DLIs according to Project Appraisal Document (PAD). These 15 DLIs were spanning over the life of the Project, with clear means of verification and time to complete. DLIs Achieved before Restructuring STATUS OF DLIs: A DETAILED TABULATION First Year (2013-14) DLIs Name of DLIs Status Punjab has operationalized the integrated management of three community- based programs (lady health worker, maternal, neonatal & child health, nutrition program) and approval of PC-1s for; Achieved (a)Integrated management of RH, Primary healthcare& nutrition; (b) Lady Health worker program, (c) Maternal, neonatal & Child health Page 69 of 76 The World Bank Punjab Health Sector Reform Project (P123394) At least 15000 LHWs in target districts trained on family planning & nutrition Achieved Health Department has entered into revised result-based contracts with Punjab Rural support program Revisited (PRSP), in term acceptable to association, in line with EPHS for primary healthcare in 12 ongoing and 2 new districts At least 200 category-1 HCEs are issued with, provisional licenses by Punjab Healthcare Commission Achieved (PHCC) Health Department has established two cells that are financial management cell and procurement cell Achieved at the provincial level. At least two secondary level health facilities in District Gujranwala and Khanewal have adopted and Achieved implemented the Medical Waste Management Plan acceptable to association Second Year (2014-15) DLIs 1 Health Department has entered into a minimum of 6 contracts with non- governmental organization for delivery of HIV preventive and pro-motive services to IDU AND MSW/hijras in the targeted cities Achieved and services provisions. 2 Health Department has entered into performance contracts, acceptable to association , with all district Revisited health officers in accordance herewith. 3 Health Department has restructured and reorganized the DGHS pursuant to the revised role and Revisited responsibilities including the establishment of contract management unit. Status of DLIs: Post Restructuring Component 1 (Improving Health Service Delivery) FY 2016-17 (3 DLIs ) Sr. Description Details Responsibility Status Punjab Population Innovation • All the milestones PPIF Achieved Fund fully operationalized (US$ 2.5 for this DLI have been met and Million) shared with World Bank. • Funds need to be released by World Bank. Comprehensive EmONC services are being • 13 out of 15 DHQs in 18 provided in low performing districts 90 percent of District Headquarter hospitals are providing CEmONC services. Achieved in the 18 low performing districts (US$ • TPV report shared with World IRMNCH 2.0 Million) Bank Media campaign for Family Planning services • Summary of TPV has already designed and implemented been shared with World Bank after (US$ 1.0 Million) the media campaign launched by PWD PWD Component 1 (Improving Health Service Delivery) FY 2017-18 (3 DLIs ) Page 70 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Population Innovation Fund (PPIF) has • Three contracts have been awarded Five grants for new initiatives awarded. (US$ 2.5 Million) • Remaining 2 are selected and will PPIF be awarded after release of funds. Punjab has attained at least 70% skilled birth • Punjab Health attendance. (US$ 2.5 Million) Survey (PHS) 2017-SBA is 78.7%. • MIS of IRMNCH, SBA is approx. 91% (LHWs covered areas). • To be verified from MICS 2018 IRMNCH Achieved which has yet to be published. Increase in Family Planning Utilization at DHIS data confirms that Family primary health care facilities in the eighteen Planning visits have acceded from 3 low performing districts million. PWD & PSPU Achieved (US$ 3.5 Million) Component 2 (Enhancing Efficiency and Effectiveness of Health System) FY 2016-17 (1 DLI) Complaint Management System is • Department has developed the in place in one third of the secondary care Complaint Management System and hospitals (US$ 2.5 Million) launched across Punjab. • Analysis Report of Complaint Management System shared with WB. • TPV shared with World Bank P&SHD & PSPU Achieved FY 2017-18 (1 DLI) Complaint Management System is Complaint Management System is in in place in two thirds of the secondary care place in secondary care hospitals. hospitals (US$ 2.5 Million) TPV is shared with World Bank. P&SHD & PSPU Achieved At least two secondary level health facilities DLI achieved and funds released. in Districts Gujranwala and Vehari have adopted and implemented the Medical P&SHD Achieved Waste Management Plan, acceptable to the Association. (US$ 2.5 Million) Implementation of Medical Waste In conformity with the approved Management plan expanded to fifteen plan scale up is scheduled in districts. (US$ 2.5 Million) selected 17 districts in the second phase (2017-18). The second phase P&SHD Achieved (scale up plan) has completed implementing HWM Rules 2014 in full letter and spirit in all selected HCFs (16 DHQs & 40 THQs). The TPV report submitted 31st December 2018 Page 71 of 76 The World Bank Punjab Health Sector Reform Project (P123394) 6. MAJOR ACHIEVEMENTS OF PROJECT Due to alarmingly high health indicators particularly, the MMR and IMR, highest priority was given to the interventions related to mother and child health. For better management and efficient delivery, MNCH, PHC & FP (LHWs) and Nutrition programmes were integrated. To achieve the target, most effective interventions including (i) increasing immunization (ii) increasing SBA rates, (iii) strengthening basic health units, (iv) improving district level effectiveness (v) increasing CPR (vi) providing CEmONC and BEmONC services at health facilities and outreach, were selected by the project regime. Vigorous and diligent efforts in achieving the targets by the department have resulted in appreciable improvement in health indicators. Following are some of the major achievements that can proudly be stated: Substantial decline in MMR and IMR in last five years is the real outcome of these interventions. 6.1. Provision of Quality Comprehensive and Basic EmONC Services Project’s one of the major achievements is establishing a system for provision of good quality Comprehensive and Basic EmONC services round the clock at secondary and primary level health facilities respectively, particularly, in low performing districts. 6.2. Punjab to attain at least 70% skilled birth attendance (SBA) The other strategy to have great dent on MMR and IMR was to enhance the SBA coverage up to 80% within the project period particularly in low performing districts. 6.3. Strengthening of Expanded Program on Immunization Before the initiation of the project, overall performance of the EPI Program was not going well as according to the baseline figures mentioned in project document, only 34.60 children from the age of 12-23 months were being fully immunized. A target of 80.00 percent coverage was given to the Program. Because of these efforts, the immunization coverage has gradually risen from baseline to 79.80% according to PDHS 2017-18 and 75.80% according to MICS 2018 while as mentioned above, the end project target was 80% which according the PDHS Survey 2017-18 has been met. Within the given circumstances, this is a great achievement and is being appreciated globally. 6.4. Development and Implementation of Essential Package of Health Services (EPHS) For provision of health-facility-specific healthcare services on standardized parameters against predefined set of clinical and non-clinical services, human resource, medicines, equipment and other logistics, the Essential Packages of Health Services (EPHS) separate for Primary and Secondary Healthcare settings were developed. 6.5. Establishment of Complaint Management System (CMS) To make service delivery efficient and of good quality, meeting with the expectations and needs of the community and to provide an effective and efficient Grievance Redressal Mechanism (GRM), Government of Punjab has established a Complaint Management System in Primary and Secondary Healthcare Facilities of Punjab. This is a multi-tiered system for recording and resolution of complaints raised by the clients about the availability and quality of services provided. 6.6. Establishment of Financial Management Cell (FMC) and Procurement Cell (PC) The two very important DLIs because of their relevance to finances and existing weak procurement systems, the Page 72 of 76 The World Bank Punjab Health Sector Reform Project (P123394) Financial Management Cell and Procurement Cells were established to strengthen and improve internal controls for greater accountability and Management effectiveness. Because of trained Human Resource, clearly laid done rules and regulations, both the cells are functioning very well and had become great example of institutional strengthening. 6.7. Establishment of Hospital Waste Management System Before the establishment of this system, numerous problems were being faced by the healthcare facilities including waste segregation, handling, storage, record keeping and safe disposal systems. The project initiated the implementation of Punjab Hospital Waste Management Rules (HWMR) 2014 robustly. Hospital waste management plans were prepared, and waste management committees were notified. Procurement of healthcare supplies, installation of environment friendly incinerators and trainings of 1664 master trainers & step-down trainings were completed. Preparation of Standard Medical Protocols (SMPs) in coordination with Environment Department & Punjab Healthcare Commission & waste audit protocols were prepared and disseminated across the Punjab. Preparation of Hospital based Management Information system (HMIS) for healthcare waste, monthly reporting system and communication proved as efficient monitoring at all levels. To make the project sustainable a PC-1 for infection control & waste management was approved. Ultimately, both phases (pilot & scale up plan) of DLI were achieved timely & successfully. 6.8. Publication of Annual Health Reports In line with the World Bank requirement, first Annual Health Report 2013-14 was published in 2014 covering the performance of the department mainly related to healthcare delivery system. Second was Annual Health Report 2016-17 and had broader coverage of components, sub- components, vertical Programmes etc., of both the health departments. The third Annual Health Report 2017-18 published in 2018 was further improved version and covered almost all components of P&SHD and some of the SHMED. These health reports were widely disseminated and were greatly appreciated particularly by public and private medical colleges and universities. These reports are being quoted as reference books. 6.9. Establishment of Multi-Sectoral Nutrition Center The Multi Sectoral Nutrition Center, Planning and Development Department, is an initiative of the Government of the Punjab with the aim of making nutrition a priority in the province. Presently, the Center is coordinating major efforts towards ensuring the implementation of MSNS by providing leadership, oversight, technical assistance, coordination, and monitoring and evaluation of nutrition interventions in the province. 6.10. Establishment of Punjab Population Innovation Fund (PPIF) Project envisaged detrimental effects of rapidly growing population and took family planning services as one of means to manage the population size and in addition to improving the health mother and child. In line with traditional interventions, Health and Population Welfare Departments are providing services but due to peculiar sociocultural circumstance and slow progress, need was felt for a separate entity that could run operations in a different way by coming out of box with innovative approaches. Punjab Population Innovation Fund has been established and made operational with the World Bank support. So far PPIF has selected five organizations to execute their projects with innovative approaches to raise the CPR and improve other related indicators at least in project areas. 6.11. Increase in Family Planning Utilization at primary health care facilities in the 18 low performing districts In view of greater impact of family planning services on mother and child health, a target of at least 3 million Page 73 of 76 The World Bank Punjab Health Sector Reform Project (P123394) clients were set for the project. The recent analysis of DHIS and PWD data done by PSPU shows that around 3,935,548 clients visited FP outlets of Population Welfare Department and 2,434,147 visited the facilities of Health Department. Cumulative numbers are more than the project target and indicative of substantial efforts and presence of good quality services. 6.12. Basic Package of Nutrition Services for the children 6-24 of months of age In the beginning of the project, only 10% of the children between the ages of 6-24 months in 18 low performing districts were receiving the basic package of nutrition services while a target of 80% was set to be achieved within the project life. Different data sources are showing achievement that by October 2018, 84% of children of given age are receiving the basic package of nutrition services in 18 low performing districts which is great success and needs to be maintained. Malnutrition, one of the major reasons for high Infant and Under 5 Child Mortality, was another project’s high priority area. During project years, more than 805 OTP sites to treat severely malnourished child at BHU, RHCs and 42 Stabilization Centers have been established in the Punjab. 7. KEY CHALLENGES Since the launch of the Project, it faced multiple challenges related to performance of the project. These factors seriously disturbed the progress of the project in achieving its desired objectives. 7.1. Delayed Project effectiveness The project became effective after six month of implementation date that proved a major challenge in the achieving the project object at the agreed timelines. This delayed also affected the performance of the project. 7.2. Cancellation of HRITF Grant The HRITF grant was cancelled due the changing priorities of the Government of the Punjab and result based contract cannot get through. 7.3. Delayed in Fund Release The fund for the project account got released after 17 months of the project actual implementation date that helped to cause slow pace of the project. Such delayed were caused by delayed signing of financing agreement, release of funds and opening of bank account. 7.4. Restructuring of the Project The project was restructured to re-align project development objectives with the new priorities of the Government of Punjab, but such it proved a big challenge for the Project. 7.5. Stoppage and Revalidation of Fund The project fund was stopped twice and have to revalidate. Though, it is common routine in the government practice, but it seriously affected the progress of the project leading to poor efficiency of the Project. Following are key hiccups that project faced during its four years of life. 8. KEY RECOMMENDATIONS 8.1. Timely availability of funds Project fund much be available on the date of effectiveness as complicated fund flow mechanism being Page 74 of 76 The World Bank Punjab Health Sector Reform Project (P123394) implemented in the Public sector takes too much time. 8.2. Simplified fund flow mechanism Fund flow mechanism should be more simplified and financing agreement should include a clause to release fund for such donor funded project on priority basis. 8.3. Non-lapsable account All donor funded project should run through a non-lapsable account, as it will improve the performance of the project. 8.4. Training of staff before execution Project key staff should be trained before the execution of the project regarding process of monitoring, standardized procurement procedures, STEP based procurement and procurement rules including financial process and procedure. 8.5. No transfer/post of Program Director Experience suggests that transfer of program director creates serious issues in achieving the project objectives. So, project director much be selected on competition bases, with good expertise to manage a project. So, it is recommended that it should be included in the financing agreement there should always be a full-time project director with no provision of additional charge to run key Project like Punjab Health Sector Reform Program. Additionally, project director much not be transferred if it is not highly indispensable. Page 75 of 76 The World Bank Punjab Health Sector Reform Project (P123394) ANNEX 6. SUPPORTING DOCUMENTS • Situation Analysis, Punjab Health Sector Strategy, March 2012 • Punjab Health Sector Strategy, 2012-2020 • Pakistan Country Partnership Strategy (2010-2013) and (2015-2019) • PCN Review Meeting Minutes, January 5, 2012 • Decision Review Meeting Minutes, March 13, 2013 • Minutes of Negotiations, April 18, 2013 • Project Appraisal Document, May 3, 2013 • Financing Agreement, December 2, 2013 • Restructuring Papers, 2014-2017 • Aide-memoires (2011-2017) • WB management letters (2011-2019) • Implementation Status Results Reports (2013-2019) • Mid-term review Issues Paper, July 2015 • Third-party verification reports (2015-2018) • Quality Enhancement Review, February 27, 2013 • Quality Enhancement Review (power point presentation), February 2016 • PC 1 2016-2018 • Pakistan: Provincial Health and Nutrition Programme, DFID, December 10, 2012 • DFID Roadmap 2014-2018 • Power point presentations provided during the ICR mission • Setting a New Pace: How Punjab, Pakistan achieved unprecedented improvements in public health outcomes, ACASUS, April 2018 • MICS 2011, 2014, 2018 • Health Facility Assessment, 2014 • Health Facility Assessment, 2018 • LHW Program evaluation 2018 • Review of Essential Health Services in Pakistan, WHO March 2019 • National Nutrition Survey, 2011 and 2018 • Punjab PSPU Completion Report, April 2019 Page 76 of 76