IEG Report Number: ICRR14776 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 06/24/2015 Country: Ethiopia Project ID: P106228 Appraisal Actual Project Name: Ethiopia Nutrition Project Costs (US$M): 39.6 36.40 (fy08) L/C Number: Loan/Credit (US$M): 30.0 26.73 Sector Board: Health, Nutrition and Cofinancing (US$M): Population Cofinanciers: Board Approval Date : 04/29/2008 Closing Date: 01/07/2014 05/31/2014 Sector(s): Health (57%); Central government administration (36%); Sub-national government administration (7%) Theme(s): Nutrition and food security (50%); Health system performance (25%); Child health (25%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Judyth L. Twigg Pia Schneider Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Financing Agreement (p. 5) and Project Appraisal Document (PAD, p. 5), the project’s objectives were to “improve child and maternal care behavior and to increase utilization of key micronutrients, in order to contribute to improving the nutritional status of vulnerable groups.” At an April 2012 restructuring, one PDO-level indicator was added and one removed, but no key outcome targets were changed. Hence, no split rating will be undertaken. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project contained two components (the ICR does not provide total actual project costs by component): Supporting Nutrition Service Delivery (appraisal, US$ 19.3 million, of which US$ 14.0 million from IDA and US$ 5.3 million from the Government; actual Bank contribution, US$ 10.96 million). This component aimed to strengthen community-based nutrition (CBN) and wider health services under the Health Extension Program’s outreach services, through capacity enhancement of Health Extension Workers (HEWs) and their supervisors, as well as mobilization of Volunteer Community Health Workers (VCHWs) to support HEWs in nutrition-related outreach activities. It also planned to provide micronutrients to under-five children and pregnant and lactating women principally in food-insecure regions with high malnutrition rates. The CBN interventions were designed initially to target four diverse and highly food insecure regions (Amhara, Oromia, Tigray, and the Southern Nations, Nationalities, and Peoples’ Region). Institutional Strengthening and Capacity Building (appraisal, US$ 20.3 million, of which US$ 16.0 million from IDA and US$ 4.3 million from the Government; actual Bank contribution, US$ 13.10 million). This component aimed to strengthen coordination and capacity for nutrition interventions by setting up a national coordination mechanism for nutrition; strengthening human resources for nutrition, including researchers and nutrition managers at various levels; and supporting capacity-building of institutions to implement nutrition interventions. It also planned to support national advocacy and social mobilization messages on nutrition to build country ownership, disseminate nutrition messages across various media, and complement practices of HEWs and VCHWs in promoting appropriate practices. Finally, this component aimed to support operational research, surveillance, and monitoring on nutrition, as well as overall monitoring and evaluation (M&E) for the National Nutrition Plan (NNP). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: According to the Operations Portal, the Bank disbursed US$ 26.73 million, and US$ 2.22 million was cancelled. The shortfall was due to procurement challenges and delays (ICR, p. 13). Financing: The project was to be financed by an IDA grant of US$ 30 million. The PAD does not specify contributions of development partners, but indicates that partners could participate in financing of various elements of the NNP or the project through parallel financing, co-financing, or a trust fund mechanism. The ICR (pp. 10, 39) lists contributions of US$ 0.62 million, of a planned US$ 0.65 million, from the Rapid Social Response Multi-Donor Trust Fund; US$ 1.81 million, 100% of that planned, from the Japan Social Development Fund; and US$ 0.31 million, of a planned US$ 0.55 million, from a Japan Grant (presumably a project preparation facility). Borrower Contribution: The Borrower contributed in full its planned contribution of US$ 9.6 million. Dates: The project was restructured on April 2, 2012, to refine the Results Framework, amend some indicators and replace others, and adjust baselines with updated information. No key outcome targets were changed. The project was amended again on December 16, 2013, to extend the project’s closing date from January 7, 2014 to May 31, 2014, to allow for completion of project activities. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Relevance of Objectives is rated High . At appraisal, Ethiopia had the second highest rate of malnutrition in sub-Saharan Africa, with about 47% of children under five stunted, 11% wasted, and 38% underweight, and 27% of women chronically malnourished. About half of all child deaths were estimated to have arisen from malnutrition, with micronutrient deficiencies a key element. The objectives directly support Ethiopia’s 2005 National Nutrition Strategy, which outlined a multi-sectoral plan to achieve the Millennium Development Goal of halving poverty and hunger. That National Nutrition Strategy implements the government's 2008 National Nutrition Program (NNP), which aimed to harmonize and implement multi-sectoral nutrition interventions and strengthen service delivery and institutions for nutrition from 2008-2013. The objectives were also highly relevant to the Bank’s Country Partnership Strategy at the time of project closing (CPS, 2012-2017), which emphasizes increasing resilience and reducing vulnerability. The second pillar of the CPS focuses on improving delivery of social services and developing a comprehensive approach to social protection and risk management, with an explicit outcome of increasing access to quality health and education services, to which nutrition is integral. b. Relevance of Design: Relevance of Design is rated Substantial . The project’s planned activities were logically and plausibly linked to the achievement of its objectives. The PAD (pp. 29-31) lays out a clear rationale for the interventions included under each component. The international literature on nutrition strategy clearly prioritized embedding micronutrient supplementation within an integrative public health and nutrition strategy at the community level (ICR, p. 8), which was precisely the logic underlying the project. There was an appropriate focus on high-risk and vulnerable groups (mothers and children). The design made effective use of existing institutions, including embedding CBN activities within an existing, innovative community-level service delivery mechanism (the HEWs); focused on a combination of community-level and higher-level interventions; and rolled out some of the CBN interventions in a phased approach (ICR, p. 17). 4. Achievement of Objectives (Efficacy): Improve child and maternal care behavior is rated Substantial . Outputs: 13,000 health personnel, from the health center to federal level, were trained on community-based nutrition, exceeding the target of 12,000. Of 30,000 Health Extension Workers, 80% were trained on a revised curriculum, not meeting the target of 85%. According to the ICR (p. 22), the project was instrumental in integrating nutrition into the service delivery package of the Health Extension Program, with specific attention to high-population, high-food-insecurity woredas. Community-based nutrition activities were rolled out in tranches that varied in timing and size throughout 2009, 2010, and 2011, ultimately covering 238 woredas. An inter-sectoral National Nutrition Coordination Body was established. By project closing, 80% of community-based nutrition woredas were providing monthly nutrition data to the federal level, exceeding the target of 50%. Ten National Nutrition Plan operational research studies were completed and disseminated, exceeding the target of eight. The project financed a two-year Masters Course in nutrition, with 80 master’s-level graduates by project closing. Outcomes: The percentage of children aged 0-23 months participating in monthly growth monitoring and promotion sessions increased from a baseline of zero to 42% by 2014, exceeding the target of 40%. The percentage of infants aged 0-5 months who were exclusively breastfed remained essentially unchanged, at 51% in 2009 and 52% in 2014, not meeting the target of 56%. However, the ICR (pp. viii, 19-20) notes that a Tulane University study of community-based nutrition woredas only, which received much of the focus of this project, increased exclusive breastfeeding of infants aged 0-5 months to nearly 90%. This study also found that project-supported interventions increased dietary diversity at age 6-23 months and reduced poor dietary practices (including providing less food to children with diarrhea and eating less during pregnancy) (ICR, p. 20). Increase utilization of key micronutrients is rated High. Outputs: In addition to other outputs cited above: A national inter-sectoral nutrition coordination agency (National Nutrition Coordination Body, NNCB) was established to provide policy, strategy, and budget guidance for the NNP and to monitor its implementation. Other committees were appointed to deal with technical issues (ICR, p. 24). Capacity building was provided to strengthen data collection on nutrition at the woreda level and to ensure that information was accurately and regularly transmitted to the national level. The project provided financial support for procurement of key micronutrients. Iron folic acid tablets were procured and distributed to pregnant women through antenatal care. Vitamin A doses for children aged 6-59 months were provided, and quarterly screenings were held for undernourished children through Child Health Days. A Universal Salt Iodization policy was adopted and implemented. Zinc was registered as an essential drug and included in the Health Post package. Outcomes: The number of people with access to a basic package of nutrition services increased from zero at baseline to 55.8 million in 2014, exceeding the target of 44.125 million. 90% of national salt production was iodized in 2014, exceeding the target of 50%. The percentage of pregnant women receiving iron and folate supplementation increased from 17% in 2009 to 89% in 2014, far exceeding the target of 25%. The 2014 Demographic and Health Survey shows a more modest increase in iron tablet consumption among pregnant women, from 15% in 2011 to 34% in 2014. The number of children aged 6-59 months receiving a dose of vitamin A every six months increased from 10.2 million in 2008 to 12.16 million in 2014, exceeding the target of 11.3 million. Additional evidence on the project 's higher-level objective, improving nutritional status of vulnerable groups : The ICR (pp. 20-22) provides evidence that the project’s interventions contributed to a reduction in the percentage of under-five children with weight-for-age less than two standard deviations below the median of the reference population from 29% in 2011 to 25% in 2014, and ultimately a reduction in under-five mortality from 76/1000 in 2010 to 64/1000 in 2013. 5. Efficiency: Efficiency is ra ted Modest. The PAD (pp. 15, 66-69) drew on economic analysis conducted for several interventions affecting nutritional outcomes in Ethiopia that had been recently completed by the Bank, but no project-specific analysis was done. The PAD’s analysis calculated benefit-cost ratios from reduced mortality, increased economic productivity, and increased child ability. The ICR (pp. 28-29, 42-49) calculates a benefit-to-cost ratio of 4.65 to 1, monetizing benefits including saved child and maternal lives and increased lifetime earnings from reduced stunting, anemia, low birth weight, and vitamin A deficiency, and exclusive breastfeeding. Using a 5% discount rate, the net present value of the project is estimated at US$ 79.9 million with a modified internal rate of return of 38%; at a 10% discount rate, the benefit-cost ratio is 3:1 and the net present value US$ 44 million. The ICR’s calculation employs realistic and conservative assumptions, with exclusion of benefits that cannot easily be translated into monetary values. However, it is not clear that the full spectrum of outcomes used in the analysis can be directly linked to the project's outcome achievements. The project supported community-based interventions that are widely acknowledged as cost-effective in improving nutrition outcomes. The use of existing Health Extension Workers to deliver community-based nutrition services was cost-effective. Provision of transport for supervisors (bicycles and motorcycles) to enable more frequent and timely visits to health posts enhanced the efficiency of delivery of community-based nutrition services. Existing federal and regional management structures were used, keeping staffing at a minimal level to maintain progress toward the project’s objectives. However, procurement challenges were a moderate source of implementation inefficiency (see Section 11b), and the project invested resources in efforts to achieve universal household salt iodization that were eventually abandoned. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 38% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project's objectives were highly relevant to country conditions and to Bank and government strategy. Its planned activities were logically and plausibly linked to desired outcomes, resulting in a substantial rating for relevance of design. Achievement of the objective to improve maternal and child care behavior is rated substantial, as targets were met for monthly growth monitoring and promotion of exclusive breastfeeding. Achievement of the objective to increase utilization of key micronutrients is rated high, as targets were exceeded for access to nutrition services and micronutrient supplementation. Efficiency is rated modest, with moderate shortcomings related to procurement and unrealized results from investments in universal household salt iodization. Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, and therefore an overall Outcome rating of Moderately Satisfactory. a. Outcome Rating: Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project triggered donor attention to nutrition, resulting in additional resources to support implementation of the National Nutrition Plan (NNP, 2013-2015). The government continues to support the NNP, refining its mechanisms, training materials, and modes of operation. The Bank team will continue to engage the government on nutrition issues through analytic work and possible Additional Financing for the Health Program-for-Results Project that would focus on high-impact targeted nutrition-specific interventions. In addition, the government’s next Health Sector Development Program will include nutrition indicators. A comprehensive micronutrient survey is scheduled for later in 2015 and is expected to strengthen the evidence base for nutrition decision-making (ICR, p. 32). Considerable institutional strengthening occurred under the project, including multisectoral coordination of the national nutrition agenda through the establishment of a national nutrition coordination body co-chaired by the federal ministries of health and agriculture (ICR, pp. 30-31). Regional and sub-regional budget monitoring and reporting, however, as well as internal audit unit capacity, will require continuous future attention (ICR, p. 15). In addition, there are questions about the continued effectiveness of community-level health workers, particularly their capacity to reach the poor, and about the impact of addition of nutrition tasks to the already heavy workload of health extension workers (ICR, p. 33). a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: Project preparation benefited from the participation of a government and multi-partner team formed in 2007 to determine national-level nutrition objectives, components, and activities, financing priorities, and implementation arrangements. As the ICR states, the Bank team took advantage of a “window of opportunity” to capitalize swiftly on government interest in an ambitious nutrition effort (p. 33). Design anticipated and effectively mitigated some potential risks, most importantly those having to do with coordination and implementation issues: inter-ministerial commitment and linkages, intra-health sector coordination, reporting requirements from multiple implementers, and donor coordination (ICR, p. 9). However, there were moderate shortcomings. A Quality at Entry Review (QER) was not conducted; according to the ICR (p. 33), a QER could have prevented these shortcomings from materializing. Risks associated with iodized salt production and insufficient fiduciary and M&E capacity were not well identified; the latter can be tied to a failure to define initially the specific institutional responsibility for project implementation (the Federal Ministry of Health was not identified as project implementer until well into the project’s lifetime). As the ICR (p. 8) points out, the project’s conceptualization could have benefited from deeper understanding of the relationship between its specific interventions and the overall determinants and parallel interventions impacting malnutrition in Ethiopia. There was insufficient recognition of the complex political economy of iodized salt production, Quality-at-Entry Rating: Moderately Satisfactory b. Quality of supervision: The Bank team took advantage of access to trust funds from several sources and from strong personal relationships with key government decision makers (facilitated by high familiarity with Ethiopian government processes and the local language and contexts). Ongoing dialogue with government and other partners produced important evidence-based research that guided project implementation. The mid-term review in 2011 appropriately identified implementation bottlenecks and made recommendations for restructuring. After the mid-term review, almost daily contact between the supervisory and implementation teams led to accelerated progress, although some minor procurement and procedural matters remained unresolved. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Government commitment and involvement remained high from preparation through implementation. The Federal Ministry of Health received increased staffing at the national level, and regional nutrition coordinators were appointed in project areas. The government invested significant funding into the development of the Health Extension Work Program, into which community-based nutrition was integrated, and led policy development on micronutrients. The Maternal and Child Health Team put mechanisms in place for technical and management reviews. Government Performance Rating Satisfactory b. Implementing Agency Performance: The project was implemented by the Federal Ministry of Health (FMOH), and by extension its key procurement entity, the Pharmaceuticals Fund and Supply Agency (PFSA). The FMOH is also the lead agency for the National Nutrition Plan. There was no dedicated project implementation unit. At the sub-national level, implementation was led by Regional Health Bureaus and District (Woreda) Health Offices. According to the ICR (pp. 12, 35), these entities had limited experience with Bank procurement procedures, resulting in delays in processing procurement requests despite additional technical assistance; after the mid-term review in 2011, the situation improved somewhat through revision of procurement plans, procurement training for PFSA staff, and hiring a procurement officer in the FMOH. However, challenges remained throughout implementation, including a delay in a micronutrient survey (ICR, p. 35). Early challenges in financial management were more adequately addressed after the mid-term review (see Section 11). Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The Ethiopia Health and Nutrition Institute (now the Ethiopia Public Health Institute, EPHI) was responsible for generating baseline data. Indicators were well linked to the project’s development objectives. The project’s initial inclusion of an indicator on household salt iodization was based on a superficial understanding of the structure of the salt industry in Ethiopia, which features, in addition to large producers, many small, difficult-to-regulate producers. Eventually efforts to achieve universal household salt iodization were abandoned. b. M&E Implementation: Data were produced by the national Health Monitoring Information System (HMIS), which did not initially include nutrition indicators but was revised under the project to include growth monitoring and promotion. Community-based nutrition data collected on a monthly basis flowed to Regional Health Bureaus and then to the national level. Initial challenges obtaining woreda-level nutrition data were addressed during the 2012 restructuring, resulting in significant improvements in the flow and quality of data. Over 80% of woredas were providing monthly nutrition data to the FMOH by project closing (ICR, p. 14). An independent impact evaluation of community-based nutrition activities was carried out in collaboration with UNICEF and finalized in September 2012. Collection of end-line data was hampered by difficulties with UNICEF’s procurement of data collection supplies; the ICR (pp. 14-15) states that better coordination of procurement between UNICEF and EPHI could have resulted in a more timely completion of a micronutrient survey and therefore better assessment of project performance. c. M&E Utilization: Operational research studies supported under the project and conducted by the Ethiopia Health and Nutrition Institute generated findings that were used by the government to understand and discuss project performance, guide implementation, and inform a new National Nutrition Plan (ICR, p. 14). These included studies on community-based nutrition, school-based health and nutrition education, iodized salt coverage, food consumption, and iron folate supplementation for pregnant women (ICR, p. 25). M&E Quality Rating: Substantial 11. Other Issues a. Safeguards: The project was rated environmental category “C” and did not trigger any safeguard policies. According to the ICR (p. 15), no negative environmental impact was identified during project implementation. b. Fiduciary Compliance: Early in the project period, key budgeting, internal controls, financial reporting, and external auditing issues were identified, and an action plan for improvement was developed at the mid-term review in 2011. Nutrition coordinators were hired to work at the regional level, training was provided for financial managers in 144 woredas, and federal-level accountant training was provided to cascade training to woreda accountants. As a result, statements of expenses were settled more quickly, and overall disbursement patterns improved through the remainder of the project period. An in-depth Financial Management Supervision Report in March 2014 confirmed adequate financial management under the project. Procurement processes were cumbersome and slow, resulting in delays in the arrival of goods to end users (ICR, p. 12). After the mid-term review, improvements were implemented (see Section 9b). However, the procurement of some micronutrients, in particular iron folate tablets, remained problematic primarily due to delays within PFSA. As a result, only 12 of 18 planned containers of these tablets were delivered by project closing, even after a four-month extension. According to the ICR (p. 15), procurement “remained a major constraint for proper implementation of the project.” c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Moderately Moderate shortcomings in efficiency Satisfactory related to procurement and unrealized results from investments in universal household salt iodization. Risk to Development Moderate Moderate Outcome: Bank Performance: Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (pp. 36-37) offers useful lessons, including: A project’s likelihood of success is enhanced through strong country ownership and integration into a wider multi-sectoral agenda. In this case, the National Nutrition Plan provided crucial engagement of government at all levels and key technical and implementation support. At least in part because of the NNP, the project was able to catalyze funding from other sources, particularly for community-based nutrition activities. Community-level service delivery provides instrumental legitimacy and commitment. Ethiopia’s unique service delivery model, its Health Extension Program (which employs Health Extension Workers at the health post level), provided a cost-effective existing entry point for nutrition support under this project (ICR, p. 36). This existing institutional structure facilitated cost-effective delivery of nutrition services. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is clear, concise, and evidence-based, with strong analysis examining the specific contribution of the project to observed outcomes. Its exposition of implementation progress and challenges is particularly informative and useful. The lessons are drawn effectively from the project’s experience and should prove useful for other nutrition interventions, both in the region and globally. The economic analysis is straightforward and comprehensive, with clear exposition of assumptions and a sensitivity analysis. The Borrower's ICR (Annex 7) provided additional useful information and analysis. However, the ICR does not provide accurate and complete information on total project costs and costs by component. a.Quality of ICR Rating : Satisfactory