Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) Report Number: ICR00004538 1. Project Data Project ID Project Name P125237 MW-Nutrition & HIV/AIDS Project (FY12) Country Practice Area(Lead) Malawi Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-50680,IDA-D1330,IDA-H7610,TF- 31-Aug-2017 107,161,352.12 12631 Bank Approval Date Closing Date (Actual) 27-Mar-2012 31-Aug-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 80,000,000.00 12,352,729.00 Revised Commitment 114,952,729.00 12,352,729.00 Actual 107,161,352.12 12,317,175.50 Prepared by Reviewed by ICR Review Coordinator Group Rocio Manchado Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Garabito 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Original Objectives. The project development objective was to increase access to, and utilization of, selected services known to contribute to the reduction of child stunting, maternal and child anemia, and the prevention of HIV and AIDS in children and sexually active adults (Financing Agreement, 7/17/12, p. 5). The statements of objectives in the Project Appraisal Document (PAD) and ICR are identical to the Financing Agreement. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) Revised Objectives. The revised statement of objectives was to increase coverage of selected nutrition, HIV and AIDS services and strengthen disease outbreak preparedness in project areas (Financing Agreement Amendment 6/23/16, p. 4). This ICR Review applies a split evaluation because there was additional financing (AF) to cover a new component/objective and revision of outcome targets. The project had disbursed 63.2% of the proceeds at the 2016 restructuring. For purposes of this Review, as the original and revised objectives related to nutrition services, in essence, do not change their aim, they are treated as one objective for the assessment of efficacy in Section 4. The original and revised objectives on HIV/AIDS are conceptually alike, but a key outcome indicator was revised downward, and therefore a split assessment of the HIV/AIDS objective is applied in Section 4. The disease outbreak preparedness objective is added under revised objectives. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 05-Aug-2016 c. Will a split evaluation be undertaken? Yes d. Components The project originally contained two components: Component A: Support for Nutrition Improvement (Appraisal: US$ 44.4 million; AF: US$ 12.27 million; Actual: US$ 57.37 million). This component had two subcomponents:  Maternal and child nutrition service delivery at community level. This subcomponent aimed at strengthening maternal and child nutrition service delivery through the provision of a “minimum package” of nutrition interventions offered in targeted communities/districts. The strategies were to be implemented through Information-Education-Communication and Behavior Change Communication (BCC) interventions such as group education, individual counseling, and home visits. The subcomponent was also to support growth monitoring and promotion. Nongovernmental organizations (NGOs) were to be contracted to implement the minimum package in each district at the community level and to collaborate closely with District Councils and the network of multi- sectoral frontline workers and volunteers.  Strengthening policy and program development, management, and coordination. The subcomponent was to support joint planning for nutrition at the central and district levels, strengthening monitoring and evaluation (M&E) systems, building technical and management capacity of the Department of Nutrition, HIV, and AIDS (DNHA), and improving office space for the DNHA. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) Component B: Support for the National HIV/AIDS Strategic Plan (2011-2016) (Appraisal US$ 58 million; AF: US$ 10.33 million; Actual US$ 51.33 million). This component was to finance activities through the HIV Pool, a funding mechanism managed by the National AIDS Commission (NAC) to coordinate HIV/AIDS donor investments for the support of the national AIDS response. The key HIV Pool partners included the Government of Malawi (GoM), World Bank, U.K. Department for International Development (DfID) and the Global Fund (which historically provided 70 percent of the Pool funds and was responsible for procuring antiretroviral [ARV] drugs). Approximately half of the project's contributions to the HIV Pool were assigned for specific interventions to prevent new HIV infections, through three subcomponents:  Support the implementation of the National Strategic Plan (NSP) for HIV and AIDS 2011-2016. This subcomponent aimed at contributing to the overall implementation of prioritized activities in NSP 2011–2016, including interventions to improve implementation efficiency and government response, the functional capacity of local government plans, M&E systems, the supply chain management system, and implementation of the Health Care Waste Management Plan (HCWMP).  Voluntary medical male circumcision (VMMC). This subcomponent aimed at strengthening the national VMMC program including support for: (a) NGO/President’s Emergency Plan for AIDS Relief (PEPFAR) partners who operated mobile clinics, (b) 28 district hospitals that offered VMMC services and neonatal male circumcision in 40 birthing centers, and (c) M&E systems.  Prevention of mother to child transmission (PMTCT). This subcomponent aimed at reducing vertical transmission of HIV through support for pregnancy confirmation, HIV counseling and testing, early infant diagnosis, and family planning for HIV positive women. The project also supported technical assistance for PMTCT training, commodity procurement, equipping/refurbishing of PMTCT centers, M&E, and operations research to increase demand among men for HIV testing and couples counseling. Revised components: A 2014 project restructuring retained the above two components and added a third component, financed by drawing funds from Component B: Component C. Support for Ebola Preparedness Plan. This component was created at the government’s request and was aimed at financing: (i) procurement and distribution of Ebola-specific health commodities and other needed equipment and supplies; (ii) capacity building of health sector staff to improve overall disease outbreak preparedness; and (iii) construction of isolation units/treatment centers using prefabricated materials in two central hospitals and in five district hospitals. The 2016 restructuring added a subcomponent to Component A in response to the 2015-2016 drought that resulted in a declaration of a "state of disaster" in Malawi (Financing Agreement Amendment, 8/5/2016, p. 5). This subcomponent was supported through US$ 22.6 million in AF that replenished the US$ 7 million that had been reallocated from Component B at the 2014 restructuring, added US$ 10 million specifically toward the new subcomponent, and provided funds to both components to adjust for currency depreciation:  Integrated Management of Acute Malnutrition (IMAM). This subcomponent was to contribute to resilience at the community level, and specifically supported the provision and timely distribution of supplies for treatment and management of acute malnutrition in 14 districts during the emergency response. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) Project Scope: Component A supported community interventions implemented by NGOs in at least 15 districts (PAD, p. 33). The scope of this component was expanded to include support for IMAM in 14 drought-affected districts. Component B supported 28 district hospitals that offer VMMC services and 40 birthing centers for neonatal male circumcision (ICR, p. 11) in 20 selected districts (ICR, p. 13). For disease outbreak preparedness, component C, the project supported the construction of six Ebola isolation centers strategically located in border districts, and the establishment of an integrated of surveillance and response systems in 59 health facilities and two international airports (ICR, p. 13). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost The PAD (p. 12) reported total estimated project costs of US$ 190.1 million (US$ 43.1 million for Component A and US$ 147 million for Component B). The financing agreement differed by not including planned contributions from the Global Fund in its cost estimates, and also estimating slightly different contributions from the Canadian International Development Association (CIDA). The financing agreement reported total estimated project costs of US$ 102.4 million (US$ 44.4 million for Component A and US$ 58 million for Component B). At the 2014 restructuring, US$ 7 million was shifted from Component B to the newly added Component C. At the 2016 US$ 22.6 million AF, US$ 10 million was added to Component A for IMAM interventions, and US$ 7 million was added to Component B to replenish the funds that had been shifted in 2014. Also, at the 2016 AF, US$ 2.27 million was added to Component A and US$ 3.33 million to Component B to finance shortfalls due to currency depreciation (ICR, pp. 13-14). Actual costs were US$ 57.37 million for Component A, US$ 51.33 million for Component B, and US$ 7 million for Component 7, for a total project cost of US$ 115.7 million (ICR, p. 48). Financing According to the PAD (p. 12), the project was to be financed by an International Development Association (IDA) credit of US$ 32 million, IDA grant of US$ 48 million, a multi-donor trust fund financed by CIDA in the amount of US$ 13.1 million, a Global Fund contribution of US$ 87 million (toward the country’s HIV Pool to support implementation of the NSP), and a Borrower contribution of US$ 10 million, for total estimated project financing of US$ 190.1 million. The PAD also stated that an amount to be determined from DfID would be added to the HIV Pool, and that the Global Fund would make an additional US$ 124 million contribution outside the HIV Pool for the purchase of ARV medications through UNICEF. The financing agreement reported a slightly different planned contribution from CIDA, US$ 12.36 million, estimating total project financing at US$ 102.4 million (comprised only of the IDA, CIDA, and Borrower contributions). AF of US$ 22.6 million was approved in 2016. Actual financing was US$ 117.16 million, comprised of US$ 29.15 million from the IDA credit, US$ 44.8 million from the IDA grant, US$ 12.32 million from CIDA, the full US$ 10 million from the Borrower, and US$ Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) 20.87 million of the AF. DfID and the Global Fund withdrew funding from the HIV Pool following the high- profile 2013-14 Cashgate corruption scandal, creating a parallel system to avoid losing gains that had been made in HIV/AIDS and nutrition. The ICR did not explain the difference between reported total project financing (US$ 117.2 million, p. 2) and reported total project costs (US$ 115.7 million, p. 48). Dates The project was approved on March 27, 2012, and became effective on October 17, 2012. A mid-term review was undertaken in April of 2015. The project was restructured three times:  December 24, 2014: A level II restructuring created Component C, reallocated funds, and created two new intermediate results indicators to measure activities under the new component.  October 1, 2015: A level II restructuring extended the project’s closing date by 14 months, from December 31, 2015 to February 28, 2017.  June 23, 2016: A level I restructuring added US$ 22.6 million in AF, revised the project’s objectives and results framework (dropping, revising, and adding PDO and intermediate outcome indicators, and revising some outcome targets), and extended the closing date by 12 months, from August 31, 2017 to August 31, 2018. (The list of dates in the ICR, p. 12, did not explain the gap in reported closing dates between February and August 2017.) The project closed on August 31, 2018. 3. Relevance of Objectives Rationale The objectives were relevant to country conditions at appraisal. At this moment, Malawi was among the poorest countries in the world, with one of the highest rates of prevalence of HIV/AIDS and malnutrition. The prevalence of underweight malnutrition, chronic undernutrition, or stunting in children under five years was 47% (Global Database on Child Growth and Malnutrition), while the levels of anemia in women ages 15-49 years and under-five children were 29% and 63% respectively in 2010. According to the 2010 Malawi Demographic and Health Survey, Malawi’s HIV prevalence was 10.6% for adults ages 15- 49 years, higher for women (13%) than for men (8%). The estimated HIV incidence was 1.6%, according to the Joint United Nations Program on HIV/AIDS. The objectives were also relevant to the Malawi Growth and Development Strategy and the World Bank/Malawi country assistance strategy. The GoM identified malnutrition and HIV as serious health problems.Thus, in 2010, it launched the Malawi Infant and Child Feeding Study (P107544), jointly financed by the World Bank and the United States Agency for International Development. This study was the basis for this project's focus on community-based BCC, education, and counseling and the implementation of nutrition-sensitive interventions. One year later, Malawi joined the Scaling-Up Nutrition movement to improve nutrition outcomes, reduce maternal and child anemia and child stunting, and mobilize resources Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) and support for implementation of the National Nutrition and Policy and Strategic Plan (NNPSP) 2007-2015. The Malawi National HIV and AIDS Strategic Plan 2011-2016 prioritized reducing new infections to decrease the burden and impact of the epidemic. The World Bank had previous experience with an HIV/AIDS project in Malawi, the Multi-Sectoral AIDS Project (MAP, P073821), which contributed to the HIV Pool from 2003 to 2012. At appraisal, the project was aligned the Bank’s Country Assistance Strategy FY07-10, under its Third Pillar, "decreasing vulnerability at the household level to HIV/AIDS and malnutrition," and the Malawi Growth and Development Strategy (MGDS) II (2011-2016), which prioritized nutrition and management of HIV/AIDS. Moreover, the original and revised objectives were aligned to the Country Partnership Strategy (CPS) FY13-16 under Theme 2, Enhancing Human Capital and Reducing Vulnerabilities, measured in part by improved access to quality education, reliable nutrition, HIV/AIDS services, and sustainable water supply and sanitation services (this was the most current CPS at project closing; a new strategy is planned for the first quarter of FY20). At project closing, the MGDS III (2017-2022) identified malnutrition and HIV/AIDS as two priority health problems. The Ebola objective remained relevant for enhancing disease control and preparedness, a World Bank corporate priority. However, although the GoM requested the introduction of Component C to support the Ebola Preparedness Plan that led to the reallocation of US$7 million US$ in 2014, the matching objective was not added to the project until 2016. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase access to, and utilization of, selected services known to contribute to the reduction of child stunting, maternal and child anemia (original) / increase coverage of nutrition services (revised) Rationale The project’s theory of change was based on the premise that increasing access to and utilization of selected nutrition services improves some nutrition outcomes, such as maternal and child anemia and child stunting. Activities were centered on implementing nutrition service delivery at the community level by offering a minimum package of nutrition and nutrition-sensitive interventions with a strong emphasis on BCC. Likewise, training at the district and community levels was provided to strengthen sectoral policy, program management, coordination, and monitoring as well as project management. The changes resulting from the implementation of the care groups model and institutional strengthening could reasonably be expected to Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) contribute to creating knowledge on child feeding and care, food processing and cultural practices. In addition, the delivery of therapeutic food, oral rehydration salts, essential nutrition commodities, and increased screening in drought-affected districts could presumably contribute to implementation of IMAM. Outputs  The NNPSP (2007-2012) was reviewed and updated by November 2017. This plan developed district and area level coordination structures.  Nutrition resource tracking and M&E systems were developed for reporting on key nutrition indicators. The number of district councils that reported to DNHA on a quarterly basis increased from a baseline of 0 in 2011 to 14 in 2018, exceeding the target of 13.  The number of care groups formed and trained reached 5064, exceeding both the original target of 538 and the revised target of 5000.  17,556 backyard gardens were established in the intervention districts, not reaching the original target of 30,000, but exceeding the revised target of 13,000.  The percentage of children ages 0-59 months who had diarrhea and were given increased fluids in the intervention districts increased from a baseline of 22% in 2011 to 32% in 2018, not reaching the target of 38%.  The percentage of children ages 0-59 months successfully treated for severe acute malnutrition increased from a baseline of 46% in 2016 to 75.4% in 2018, not reaching the original target of 80%.  An output indicator on children sleeping under insecticide-treated bed nets (ITNs) was dropped, as the project did not procure or distribute ITNs. Similarly, output indicators on mothers attending monthly group nutrition education and pregnant women receiving antenatal care (ANC) were dropped, as they were considered out of the project's scope. Outcomes  The percentage of caregivers of children under 2 benefiting from monthly care group services in intervention districts increased from a baseline of 0 in 2010 to 78% in 2018, exceeding the original target of 33%, but short of the formally revised target of 82%. The original indicator in the PAD was written as: percentage of children under two receiving a monthly minimum package of community nutrition services in the Component A intervention districts.  The percentage of children 6–23 months of age who receive minimum diet diversity in the intervention districts increased from a baseline of 29% to 36.4% in 2018, short of the target of 40%.  The number of people with access to a basic package of health, nutrition, or population services in the project districts was 26,747 as of 2016, far from reaching the target of 186,615. Although the indicator was dropped due to achievement not being fully attributable to the project, the Borrower’s ICR (p.32) reported achievement as 197,123.  An outcome indicator on pregnant women receiving ANC was dropped, as it was considered out of the project's scope. Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) The reported results indicate that the care group model was implemented among all the project districts as the delivery platform for community-based nutrition interventions, and formally adopted in the NNPSP 2018- 2022. Also, intermediate indicators linked to institutional strengthening and M&E were achieved. However, many key output and outcome targets were not reached. In addition, although the project's interventions targeted children under five and pregnant women, all the female-specific output and outcome indicators were dropped after the June 2016 restructuring, limiting the availability of information needed to attribute changes in maternal behavior to the project. Most importantly, preliminary findings from the community-based nutrition end-line survey showed a decline in the prevalence of stunting from 44.2% to 38.3% in both project and non- project districts (where similar interventions were implemented), as well as highly uneven implementation across project districts, raising questions of attribution of observed outcomes to project interventions (ICR, p. 30). Rating Modest OBJECTIVE 2 Objective Increase access to, and utilization of, selected services known to contribute to the prevention of HIV and AIDS in children and sexually active adults (original outcome targets) Rationale The project’s theory of change was based on the premise that increased access and utilization of selected services such as promoting the use of condoms, VMMC, and PMTCT is a cost-effective approach to reduce new infections. According to this logic, the HIV/AIDS NSP 2011-2016 priorities were implemented and M&E systems strengthened. Also, the project supported the implementation of a VMMC operational plan aimed to scale up VMMC services with a focus on no-PEPFAR districts, and funded the equipment of PMTCT centers to scale up the provision of PMTCT services. Outputs  VMMC was institutionalized in 20 out of 28 district hospitals.  Health and frontline workers were trained in delivering PMTCT and VMMC services, and technical assistance was provided at the central level.  15 health facilities were refurbished and rehabilitated to improve PMTCT service delivery.  The percentage of circumcised males who were tested for HIV as part of VMMC services increased from a baseline of 50% in 2011 to 100% in 2018, exceeding the original target of 85%.  The percentage of public health facilities that are able to provide a minimum package for VMMC increased from a baseline of 0 in 2011 to 63% in 2018, exceeding both the original and the revised targets of 34% and 47% respectively. This indicator was surpassed the first year and became irrelevant due to unexpected funding from PEPFAR. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237)  The percentage of pregnant women attending ANC who are tested for HIV increased from a baseline of 74% in 2011 to 98% in 2018, exceeding the target of 85%. Also, infant testing for HIV was strengthened with support for polymerase chain reaction (PCR) testing kits.  The number of HIV positive pregnant women who receive ARV to reduce the risk of mother to child transmission increased from a baseline of 32,000 in 2011 to 55,902 in 2018, exceeding the target of 40,000.  The percentage of sexually active respondents who had sex with a non-regular partner within the previous 12 months remained unchanged, at 10.2% in 2011 and 10% in 2018, not reaching the target of 8%. Outcomes  The number of male circumcisions conducted according to national standards in the 20 selected districts increased from 10,000 for postnatal (there was no baseline for neonatal) in 2011 to 272,189 in 2018, not meeting the original target of 636,900 (136,900 for neonatal and 500,000 for post-natal).  The percentage of infants born to HIV positive women enrolled in PMTCT services in the target districts who receive a PCR test for HIV within two months of birth increased from a baseline of 25% 2011 to 73% in 2018, exceeding the original target of 50%.  The percentage of men and women ages 15-49 who have had more than one sexual partner in the last 12 months reporting the use of a condom in their last sexual intercourse decreased from a baseline of 51.9% in 2011 to 35.4% in 2018, not reaching the original target of 63%. The project targeted children and sexually active adults from those districts with the highest HIV prevalence and incidence. It contributed to improvement in the quality and capacity of the health system by scaling up VMMC as a routine intervention through the support of private providers and developing innovative partnerships across the public and private service delivery system and local government institutions. However, the original target for number of males circumcised was not achieved due to frequent stock-outs of key commodities, an evolving donor landscape supporting the VMMC scale-up, and suboptimal funding by partners to the HIV Pool. Moreover, a key outcome indicator comprised two indicators that were meant to be reported separately -- neonatal male circumcised and postnatal male circumcised -- and there were no baseline data for neonatal circumcisions. In addition, although the project contributed to an increase in utilization of HIV and AIDS prevention services at health facilities, the percentage of sexually active adults reporting the use of condoms at the last intercourse decreased. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective Increase coverage of selected HIV and AIDS services (revised outcome targets) Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) Revised Rationale During the project’s history, activities and indicators were reviewed due to an overlap of investments from PEPFAR and the project, for example, financing fixed and mobile services units to provide a minimum package of male circumcision and PMTCT. Moreover, after the Cashgate scandal, numerous donors withdrew their funding to the HIV Pool and channeled support to the sector through a parallel system that contributed to the achievement of some HIV outcomes, but led to lack of funds for some activities; for instance, PMTCT activities were scaled down in the latter part of the project. Outputs and Outcomes In addition to the outputs and outcomes presented above under the original objective, the outcome target for male circumcision was revised from 636,900 to 264,200. The number of male circumcisions conducted according to national standards in the 20 selected districts increased from 10,000 in 2011 to 272,189 in 2018, exceeding this revised target. Revised Rating Substantial OBJECTIVE 3 Objective The project did not originally contain a third objective. Rationale The project did not originally contain a third objective. Rating Not Rated/Not Applicable OBJECTIVE 3 REVISION 1 Revised Objective Strengthen disease outbreak preparedness in project areas Revised Rationale The Government of Malawi considered it crucial to strengthen Malawi’s capacity to respond to disease outbreaks due to the risk of an Ebola epidemic in the country. The theory of change assumed that building infrastructure, supplying commodities and critical goods, and training health sector and other staff to respond to an Ebola emergency could reasonably be expected to contribute to strengthening Malawi’s capacity for control and prevention of communicable diseases and the management of outbreaks. Also, the Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) implementation of a new surveillance system to process and analyze information in real-time would improve decision-making capacity and the national response to a health threat. Outputs Outputs included training activities and provision of commodities and critical goods.  The number of health facilities provided with Ebola preparedness equipment and supplies (including hand-washing, thermometers, and personal protective equipment) increased from 0 in 2014 to 35 in 2017, exceeding the original target (2015) of 12, and meeting the revised target (2018) of 35. This target was revised to include district hospitals and other facilities linked to isolation centers to strengthen overall capacity to address emerging infectious diseases.  Hospital staff was trained in emergency preparedness and managing of outbreaks. However, hospital staff were inadequately trained on how to use incinerators and to transport infectious waste material. Training of hospital staff was dependent on the support of another project, the Southern Africa Tuberculosis and Health Systems Support Project (P155658) (ICR, p. 35).  The disease surveillance system was strengthened by installing an Electronic Integrated Disease Surveillance and Response system (eIDSR) in 59 health facilities and both international airports. Outcomes  The project supported the implementation of an Ebola Preparedness Plan that included the establishment of six isolation units for Ebola case investigation and management. This indicator was introduced as an intermediate result indicator at the 2014 restructuring and was considered a PDO indicator at the 2016 restructuring. The number of units increased from 0 to 6, meeting the original target of 6. This objective is considered to be only partly achieved, as the isolation centers were not operational at project completion. According to the ICR (p. 22), the Bank’s October 2018 field visit showed that the two isolation centers were not operational due to lack of water connection, poor construction quality, and other issues. The ICR stated that at a subsequent field visit, “most of the issues were addressed”; however, the ICR did not provide sufficient detail to conclude that there was substantial progress. To date, there has not been a diagnosed case of Ebola in Malawi, and although infrastructure and human capacity has been built, the ICR does not provide information about the use of the isolation centers or the surveillance system to manage other infectious diseases. Revised Rating Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) Modest OVERALL EFF TBL OLD Rationale Overall efficacy is rated Modest under both the original and revised objectives/targets, due to shortcomings in reaching output and outcome targets and in attribution of observed results to project interventions. Overall Efficacy Rating Primary reason Modest Low achievement 5. Efficiency The PAD referred to the Copenhagen Consensus 2008 (PAD, p. 21) to conclude that nutrition interventions are cost-effective, including community-based programs targeting children under two years old. The PAD (p. 21) also concluded that VMMC and PMTCT are among the most cost-effective interventions on HIV/AIDS based on systematic review of cost-effectiveness analyses of prevention interventions in Sub-Saharan Africa, with a reported estimated cost of VMMC at US$174 per HIV infection averted (HIA), and of PMTCT at US$84 cost per HIA (Mason et al, 2006). There was no cost-benefit analysis (CBA) performed at appraisal in the PAD. The ICR team undertook an economic analysis for both appraisal (2012) and the ICR (2018) using the same analytical assumptions to estimate the economic rate of return. The result was that the project would generate an internal rate of return (IRR) of 10 percent, using a 10 percent discount rate. The project was also expected to result in a positive net present value (NPV) of US$31 million with a cost-benefit ratio of 1.4. To calculate the benefits of the implemented restructured project, a new CBA was carried out, using the previous assumptions. This economic analysis showed a stronger economic rationale for the project. The net economic benefits generated by the project’s inputs and outputs resulted in a positive NPV of US$101,395,061, an IRR of 15 percent, and a cost-benefit ratio of 2.4, with a conservative approach in estimating the benefits from the project. The main limitations in this economic and financial analysis calculation were that, as no Ebola cases were found in Malawi, the impact of the Ebola isolation centers was excluded. (ICR, p. 23). Shortcomings were reported in the efficiency of design and implementation. The project's design was a result of the integration of two independent frameworks implemented by two different entities, the NAC and the DNHA. Moreover, one of the agencies, the DNHA, did not have previous experience working with the World Bank (ICR, p. 27). Although to mitigate this risk, the project aimed to strengthen the capacity of the DNHA and also envisaged that the DNHA would work closely with and learn from the NAC, the two agencies worked in silos during the implementation of the project, without the required coordination (ICR, p. 27). In addition, some key indicators measured activities beyond the scope of the project, or achievements were dependent on inputs not financed by the project (ICR, p. 12). Stockouts of products not provided by the project (such as ITNs and condoms) impacted negatively in the achievement of some indicators. There were duplicative efforts, as some PEPFAR-funded VMMC interventions were also supported by the project (fixed and mobile service units providing a minimum package of male circumcision and PMTCT) (ICR, p 27). The stakeholders were not aware of this situation during the project design phase, and consequently the project budget had to be reviewed and some activities reprogrammed (for example, scaling down of PMTCT activities in Phase II). Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) In addition, although the World Bank team noted during the first year of project implementation that the results framework needed to be revised, this did not happen until the 2016 restructuring, when baselines and targets were aligned to the national targets (ICR, p. 29). Moreover, the new component C was introduced during the 2014 restructuring, but the related PDO was not introduced until 2016, meaning that project activities not relevant to an objective were implemented for two years (ICR, p. 12). There were delays in the construction of the isolation units due to suspension activities until the site-specific environmental plans were prepared and disclosed through the World Bank InfoShop on May 18, 2016. Implementation inefficiencies related to the isolation units included shortfalls in budget allocation, fragmented institutional arrangements, clarity in terms of ownership of the isolation centers/incinerators, limited coordination, and capacity constraints (ICR, p. 31). There were frequent delays in submitting financial management documentation, some control and accountability issues, and outstanding ineligible expenditures (ICR, p. 32). The Cashgate scandal in 2013 harmed the evolution of the project. First, most donors withdrew their funding to the GoM after the scandal, leading to the dissolution of the HIV Pool that had been a cost-effective system to coordinate all stakeholder effort regarding HIV in Malawi. To avoid losing achieved gains in reducing the spread of HIV/AIDS and improving nutritional outcomes, other donors channeled their funds to parallel systems that led to the end of the HIV Pool (ICR, p. 34). This fragmentation in the coordination of national HIV plan activities led to shortcomings such as duplications (noted above) and, at the same time, insufficient budget for some areas and the creation of silos. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 100.00 ICR Estimate  15.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated Substantial across the entire project, as the objectives continued to be consistent with Bank and government strategies, and responsive to the needs of the country. Efficacy was Modest under both the original and revised objectives and outcome targets. Efficiency is rated Modest based on implementation inefficiencies. Outcome is therefore rated Moderately Unsatisfactory, consistent with significant shortcomings in the project's preparation and implementation. Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome To ensure sustainability of the district-level interventions, the project supported coordination structures including the establishment of District Nutrition Coordination Committees and district nutrition coordinators. Various levels of cadres (for example, care group promoters) were trained to plan for and deliver the package of nutrition and nutrition-sensitive interventions (ICR, p. 25). The Investing in Early Years for Growth and Productivity Project (P164771) will continue to support the DNHA and District Councils to implement community-based nutrition-sensitive interventions using the care group model. However, in contrast to the this project, the districts (rather than NGOs) will be at the forefront of implementation (ICR, p. 34). The DNHA and the District Councils will require strengthening of institutional and staff capacity to implement a World Bank-funded project successfully. The World Bank commissioned and led a mathematical modeling exercise focused on improving the allocative efficiency of Malawi’s HIV response. The report provided a series of recommendations designed to minimize risk to achieved outcomes: (a) prioritize geographical hotspots; (b) maintain focus on diagnosis, treatment, and viral suppression; (c) increase funding for HIV testing, as finding the remaining undiagnosed people living with HIV will require innovative strategies; and (d) prioritize the most cost-effective service delivery modalities (ICR, p. 31). The project focused on the most vulnerable areas, those with a higher prevalence of HIV and malnutrition. Also, the project included proven cost-effective services for HIV testing and diagnosing men and pregnant women. However, it could be beneficial and cost-effective to add activities aimed at other vulnerable groups, for example, young women in the selected regions. After the Cashgate scandal, the disappearance of the HIV Pool as the primary mechanism to coordinate donor HIV/AIDS investments, and the integration of HIV management in the Ministry of Health and Population (MoHP), it will be important to continue to restore donor confidence to ensure coordination and funding for ARV and other supplies. 8. Assessment of Bank Performance a. Quality-at-Entry The strategic approach using care groups as a delivery platform for community-based interventions was appropriate and relevant to the country context at appraisal. Project preparation incorporated knowledge from global and Malawi experience, as well as lessons learned from implementing the MAP. However, there were significant shortcomings. The project was prepared in a relatively short time, with the concept note reviewed on October 12, 2011 and approval on March 27, 2012. It involved the merger of two separate projects implemented by two independent entities, the DNHA (which had no experience working with the World Bank) and the NAC. Although the World Bank team aimed at harmonizing project management processes, the two management teams operated in silos. In addition, there was no CBA performed in the PAD. There were issues with the original results framework that led to redefinition of outcomes, indicators, and targets during the 2016 restructuring. Finally, the Bank and the government did not coordinate with other development partners on the scope of work of each stakeholder. Although Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) PEPFAR is a government agency, the HIV Pool co-financiers were not aware of PEPFAR making significant investments to VMMC interventions that would finance activities also supported by the project, duplicating efforts in some districts, and necessitating additional changes in the results framework. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The World Bank team was proactive in restructuring the project three times in 2014, 2015, and 2016 to remain responsive to a changing context and the GoM’s requests (financing the Ebola preparedness plan and activities to mitigate the effects of the 2012 and 2017 droughts). The team also sought opportunities to engage the broader health sector and therefore supported the development of a costed Health Sector Strategic Plan II M&E master plan and a fiscal space analysis (P157774). After the Cashgate scandal in 2013, the World Bank continued funding the HIV Pool; moreover, the team worked closely with the GoM to improve its public financial management systems through technical assistance and financial resources. Though the Aides Memoire and Implementation Status Reports of 2013 and 2014 reported issues related to the results framework, and a new component was included in 2014 with additional funds answering the GoM's urgent request for support for the implementation of an Ebola Preparedness Plan, PDOs and targets did not come into alignment with the added component and financing until the third restructuring of the project in 2016. In addition, safeguards instruments were not prepared at the time of the 2014 restructuring, which resulted in construction delays until site-specific Environmental and Social Management Plans (ESMPs) were prepared. The Bank team did not work adequately with the government to strengthen the M&E system to correct shortcomings with tracking progress on some indicators. The project could have benefited from more frequent field visits to the isolation centers, as the reported achievements in staffing and equipping the Ebola isolation centers were not accurate. Moreover, there was inadequate monitoring and implementation of safeguards; for instance, due to a delay in reporting gaps in hospital staff training on incinerator operations and waste management, it was necessary to prepare an ESMP action plan on December 6, 2018, when the project was closing. As a result, the training of these hospital staff ultimately depended on support of the Southern Africa Tuberculosis and Health Systems Support Project (P155658), approved in 2016. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) a. M&E Design Project design essentially contained two separate M&E systems, in DNHA and the NAC, with each implementing entity collecting data separately. These systems were not integrated until the 2016 restructuring. Not all the indicators reflected the objectives suitably; some key indicators measured activities that were beyond the project's scope, or their achievement depended on inputs that the project did not finance. DNHA M&E capacity was weak, an issue that was faced only when the project was almost closed by developing a web-based M&E system that allowed the district officer (or an NGO, in cases where the district M&E system was not functional) to consolidate data and forwarded it to the DNHA. The Department of HIV and AIDS (MoHP) aggregated project-specific HIV/AIDS data and collected additional data through routine monitoring and supervision. b. M&E Implementation M&E activities were implemented with shortcomings. The DNHA held regular meetings to review work programs and the results of activities at the NGO and district levels, and it supported annual food and nutrition research dissemination events. The NAC produced quarterly reports to monitor the progress of the HIV Pool implementation plan. However, even though the World Bank team noted early in the implementation period that the results framework needed to be revised, as some targets were already achieved, and baselines and targets for other indicators needed to be aligned to national targets, these changes were not captured until the 2016 restructuring. The project financed evaluations and analytical work and conducted a community-based nutrition end- line study. Additionally, the NAC commissioned an impact evaluation and a CBA, supported by the World Bank and DfID, to measure the effects of incentives on improving VMMC demand. Furthermore, the Bank commissioned and led a mathematical modeling exercise focused on improving the allocative efficiency of Malawi’s HIV response, with technical support from the UNAIDS Secretariat in Malawi and PEPFAR. c. M&E Utilization M&E findings were used to monitor the status of implementation and outcome indicators, and results were communicated to stakeholders. There were shifts in project implementation in response to M&E activities; for instance, the team restructured the project in June 2016 after detecting that some targets, indicators, and the PDOs needed to be updated. Project-financed evaluations and analyses were used to provide evidence of impact, inform decision making, and strengthen sectoral strategies and policies. M&E Quality Rating Modest 10. Other Issues a. Safeguards Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) The project triggered safeguard policy OP 4.01 on Environmental Assessment due to the collection, storage, and disposal of medical waste generated by clinical activities. The project was classified as Category B. An HCWMP was prepared, and the final plan was disclosed on February 6, 2012. The ICR described challenges in completing the planned mitigation activities. For example, when the project was restructured in December 2014, relevant safeguards instruments were not prepared, which resulted in construction delays until site-specific ESMPs were prepared in May 2016. The ICR explained that lack of budget allocation and clarity in terms of ownership of the isolation centers/incinerators negatively impacted the implementation of the ESMPs. Safeguards compliance was rated Moderately Unsatisfactory in the project's final Implementation Status Report. b. Fiduciary Compliance Financial management The ICR (p. 32) described the financial accounting, auditing, and reporting of the project as adequate and consistent with the World Bank’s financial management guidelines. However, the ICR also described frequent delays in submitting quarterly Interim Financial Reports (IFRs). Audited financial statements raised several issues on control and accountability, particularly with the NGOs contracted by the NAC. To address those concerns, the NAC developed and implemented an action plan. The Borrower's ICR noted that some activities might not have been adequately financed due to lack of budget allocation, and that a lack of training on project management, financial management, and accountancy led to errors and delays in the submission of IFRs. Procurement DNHA was responsible for procurement on Component A, and NAC for Components B and C. UNICEF was the procurement agent for most of the medical commodities and supplies for all three components. Overall, the implementing entities followed World Bank procurement guidelines and updated their procurement plans into the World Bank procurement system. However, delays in processing procurements, lack of staff capacity, a high turnover of staff at DNHA, and limited compliance with the procurement guidelines were described in both the ICR and the Borrower's ICR. An in-depth review in 2014 indicated improvements in compliance with procurement methods, quality of bidding documents and evaluation reports, and contract management. c. Unintended impacts (Positive or Negative) Although the project did not have private sector mobilization as an objective, the project contributed to improving the quality and capacity of the health system to scale up VMMC as a routine intervention through the support of private providers. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR (p. 34) offered several useful lessons, including the following adapted by IEG:  Once a need for course correction is agreed by the World Bank team, clients, and relevant stakeholders, timely restructuring can align a project's PDOs, indicators, inputs, and targets with the evolving context and operating environment. In this case, even though the Bank team and the GoM had identified the need for major revisions at the beginning of the project, action was not taken until the 2016 restructuring, leading to shortcomings, implementation challenges, and, as a consequence, shortcomings in the project’s efficacy and efficiency.  Full engagement of environmental and social safeguards specialists can ensure that appropriate safeguards instruments are prepared and that there is full compliance with established frameworks and plans. In this case, neither development nor implementation of environmental management plans were adequate and timely, producing delays and negatively impacting project outcomes.  A straightforward M&E system with well-defined indicators and systematic monitoring and tracking can determine whether and when restructuring is necessary, and what the scope of any restructuring should be. In this case, lack of on-time information and accurate reporting led to delays and gaps in project implementation.  Implementation using country structures (for example, District Nutrition Coordination Committees, Area Executive Committees, and community frontline workers), and enhancing the capacity of those structures where and when needed, can ensure that interventions and outcomes are sustained beyond the life of the project. In this case, as the project relied on NGOs to implement community-based activities; the District Councils may not have the necessary experience to continue these activities, putting sustainability at risk. Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Nutrition & HIV/AIDS Project (FY12) (P125237) 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain The challenges this project faced across its life cycle offer numerous lessons for countries with similar contexts, including the management of a complex results framework in evolving contexts where implementing agencies lack experience with World Bank-financed projects. 14. Comments on Quality of ICR The ICR provided a clear narrative and a comprehensive overview of the project's experience. It was results- oriented and candid, carefully linking outputs with observed outcomes under each objective. The evidence presented was well referenced, with annexes that included relevant information to support the narrative of project achievements. The ICR was internally consistent and for the most part followed guidelines, though its application of the split rating methodology divided the project into phases rather than assessing both the original and revised objectives across the project's entire lifetime. However, the ICR provided rather scanty information about project M&E, budget allocation shortcomings, and external audits. While its lessons were insightful, they were framed primarily as recommendations, and they were not linked in detail to the narrative and ratings included in the ICR. Additional lessons could usefully have addressed some of the other key challenges the project faced: integrating the work of the two different implementing entities, the NAC and the DNHA; coordinating with development partners on the scope of work (in this case, communicating with PEPFAR to avoid overlapping coverage of VMMC interventions); and managing the impact of unanticipated events on the actions of development partners (in this case, the Cashgate scandal). a. Quality of ICR Rating Substantial Page 19 of 19