Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) Report Number: ICRR0021651 1. Project Data Project ID Project Name P074091 KE-Health SWAP (FY10) Country Practice Area(Lead) Kenya Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-47710,IDA-50340,IDA-53670,TF- 31-Mar-2015 207,736,708.00 16027,TF-90490 Bank Approval Date Closing Date (Actual) 29-Jun-2010 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 21,218,875.00 Revised Commitment 213,399,748.57 20,950,827.83 Actual 207,757,255.96 20,950,827.83 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The original Project Development Objective (PDO) for the Health Sector Support Project (HSSP) set out in the Financing Agreement (page 5) was to support Kenya’s National Health Sector Strategic Plan 2005-2012 (“the Program”) and improve: "(i) the delivery of essential health services in the Recipient’s territory, especially for the poor; and (ii) the effectiveness of planning, financing, and procurement of pharmaceuticals and medical supplies." Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) At a November 2011 restructuring and Additional Financing (AF), the PDO was revised: "improve: (i) the delivery of essential health services in the Recipient’s territory, especially the poor and the drought affected populations; and (ii) the effectiveness of planning, financing, and procurement of pharmaceuticals and medical supplies." At a December 2013 restructuring and AF, the PDO was revised again: "improve (i) the delivery of quality essential health and nutrition services and utilization by women and children among the poor and drought affected populations; and (ii) the effectiveness of planning, financing, procurement of pharmaceuticals and medical supplies.” At both restructurings/AFs, the original thrust of the PDO was maintained while targets were revised upwards and refined, essentially increasing quality and expanding coverage of project activities, and recognizing the need for focusing more on drought-affected populations (part of the country was struck by a severe drought early during project implementation). The formulation of the PDO was revised accordingly. At the PDO remained materially the same and the project's scope was increased in accordance with AF, a split rating will not be performed. Assessment of efficacy will be based on the following objectives, consistent across the project's lifetime: Objective 1: to improve the delivery and utilization of quality essential health and nutrition services for the poor and drought-affected populations, especially women and children. Objective 2: to improve the effectiveness of planning, financing, and procurement of pharmaceutical and medical supplies. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 30-Nov-2011 c. Will a split evaluation be undertaken? No Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) d. Components The project had two components: Component 1. Effective and transparent implementation of the Kenya Essential Package for Health (KEPH) through Health Services Support Fund (HSSF) grants and performance strengthening (estimated cost at appraisal US$ 46 million; actual cost US$109.66 million) consisted of two subcomponents: (i) a project-funded HSSF for financing the implementation of the KEPH at community levels; and (ii) activities to strengthen governance and stewardship functions in the delivery of essential health services. Sub-component 1.1, Health Sector Services Fund, included (i) the provision of direct cash transfers through HSSP to sub-district and community-level health services for them to implement the KEPH. This was to ensure the availability of sufficient resources to implement each facility's Annual Operational Plan for addressing preventive, promotive, and curative services; and (ii) capacity building in the management of health facilities. The arrangement was to support the government’s policy to delegate responsibility for health service provision to the community level. Sub-component 1.2, Health sector governance and stewardship, addressed governance and stewardship functions of the two health ministries -- the Ministry of Medical Services (MOMS) and the Ministry of Public Health (MOPH) -- for enhanced service delivery. Activities included: (i) the phased implementation of health financing reforms through selective pilots, capacity building, and evaluations, focusing on approaches for reaching the poor with health insurance coverage; (ii) support for the KEPH referral system through capacity building for district hospitals; (iii) health care waste management; (iv) strengthening sector governance by improving the management information systems (MIS) and monitoring and evaluation (M&E) in the two ministries, monitoring procurement and carrying out performance audits, and training of staff in World Bank procurement and financial management procedures. Component 2. Availability of essential health commodities and supply chain management reform (estimated cost at appraisal US$54 million; actual cost US$98.10 million) consisted of two sub- components: availability of essential health commodities, and supply chain management reform. Sub-component 2.1, Commodities, was aimed at ensuring availability of medicines and medical supplies, and consequently enhancing quality of care provided at the lower levels of the public health system. A reliable and steady supply of essential medicines and medical supplies (EMMS) to front-line rural health facilities (levels 2 and 3) was to be established through the build-up of a reasonable buffer stock of these commodities at the medical supplies authority (KEMSA), as well as reforms of the supply chain system. The project was also to support reforms in the flow of funds from the Ministry of Finance, the definition of the EMMS list, the billing system of the KEMSA to cover distribution costs, capacity development, and development of supplementary commodity services for hospitals. Sub-component 2.2, Strengthening governance and institutional reforms to enhance the efficiency Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) and transparency of KEMSA, was to support the procurement of EMMS commodities and the reform process, strengthening KEMSA and the health ministries in the key functions of planning, procuring, and managing critical commodities, in conjunction with other development partners. At the 2011 restructuring, a new activity was added to the first component to.deliver health and nutrition services to populations affected by the drought, and the second component was revised to scale up the supply of essential health and nutrition commodities for drought-affected areas. At the 2013 restructuring, Component 1 was again revised to scale up support to the HSSF, to introduce results-based financing (RBF) in health facilities in arid and semi-arid areas, to implement free maternity services, and to launch a first phase of a program of health insurance subsidies for the poor. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project costs and financing. Total project cost at appraisal was estimated at US$100 million equivalent, financed with an IDA Credit. The project was implemented under a Sector-Wide Approach (SWAp) arrangement, with several donors operating under a Joint Program of Work and Financing specified in a Joint Financing Agreement (JFA) (ICR, page 44). During a first Level 1 restructuring approved by the Executive Directors on November 30, 2011, additional financing of US$56.8 million was provided from the IDA Crisis Response Window. A second Level 1 restructuring was approved by the Executive Directors on December 3, 2013, providing an additional US$41 million from a multi-donor trust fund. Actual disbursements at project closing were US$207.76 million. Borrower contribution. None Dates. The project was approved on June 29, 2010, and became effective on September 30, 2010. The original closing date was March 31, 2015, but it was extended twice – a first time at the December 2013 restructuring from March 31, 2015, to December 31, 2016; and a second time during a Level 2 restructuring in October 2016, extending the closing date to June 30, 2018, to allow completion of project activities. 3. Relevance of Objectives Rationale Project objectives were highly relevant to the country situation, and to government and Bank strategy. A generally well-performing economy had hit a down-turn at the time of the international economic crisis, which roughly coincided with project preparation. Despite the overall positive economic performance, Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) health outcomes, in particular child and maternal health outcomes, had been lagging. This lag reflected systemic challenges at lower levels of the system, where mainly the poor seek care. These challenges included shortfalls in overall public financing and the allocation of resources between different levels in the system to the disadvantage of the lower levels of the system. Government policy reflected in the Letter of Sector Policy that underlies the project included a series of reforms to address the challenges: nationwide provision of essential services based on standard norms promoting the delivery of quality services (the KEPH); financing policy; human resource management; supply chain management; and a community strategy to empower communities to participate in shaping the provision of health services. The project addressed two reform areas – direct financing of frontline health services; and procurement and supply chain management. This aligned with two of the three domains of engagement in the Bank’s Country Partnership Strategy for Kenya FY14-FY18, which remains relevant. It accords with Outcome 5 of Domain 2 in the strategy, human resource development for shared prosperity; and with all three outcomes in Domain 3: better provision of health and sanitation services by counties, adequate systems to monitor performance of services delivered by counties, and heightened transparency and accountability in the use of public resources, particularly at the county level. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the delivery and utilization of quality essential health and nutrition services for the poor and drought-affected populations, especially women and children. Rationale The objective was to be achieved by the provision of HSSF grants to all primary health care facilities that met core staffing needs and financial management requirements for effective delivery of the KEPH. To meet those requirements, capacity building was provided to county health teams in financial and health facility management. Outputs Delivery of KEPH was improved by the following: • People with access to a basic package of health and nutrition health services reached 47 million - practically the whole population - compared to an original target of 30.5 million. Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) • Facility management committees were activated as indicated by 83 percent of such committees having quarterly meetings, compared to a target of 90 percent and baselines of 27 percent (level 2 facilities) and 34 percent (level 3 facilities). • A total of 19,300 health workers were trained, compared to an original target of 35,000. • A regular flow of funds to counties and health facilities was ensured by the HSSF. • Planning and procurement of medicines and medical supplies was improved (part of Objective 2). • RBF was introduced on a pilot basis in 21 arid and semi-arid counties, including 1,800 health facilities. Outcomes The number of outpatient visits to health facilities offering KEPH increased from 25.9 million at the start of the project to 50.9 million at project completion, against an original target 29.3 million. However, the ICR (page 17) also reported on the preliminary findings of the 2018 Household Health Expenditure and Utilization Survey. That survey, comparing with the results of a 2013 survey, indicates that per capita utilization rates remained flat between the two periods, as did the total number of visits at 9.1 million, vastly below the facility- recorded numbers. The ICR did not explain this apparent discrepancy. Female outpatient visits totaled 34.4 million (target 16.1 million, baseline 14.2 million). The ICR (page 18) stated that the provision of HSSF grants increased annual operational budgets for lower- level health facilities (amounts were not specified); improved efficiency of spending, reducing irregular use of funds from 21 percent to 5 percent of facilities; and increased proper book-keeping from 53 percent to 78 percent of facilities. A government agency evaluation (cited in the ICR, page 18) noted that the quality of service, including the quality of care, staff motivation, and patient satisfaction, also increased, but supporting data were not provided. However, with the decentralization of health operations and devolution of responsibilities to counties, the program experienced challenges with funds flows and reporting that disrupted both HSSF and RBF processes and reduced their positive impact. The latter, in particular, experienced difficulties: according to the ICR (para. 41), only two transfers were made during the project period, and utilization of funds (for essential equipment and nutrition commodities) was low. Over 15 million people (moderately malnourished children under five and pregnant and lactating women) in drought-affected areas received health care services and relevant food supplements, compared to a target of 5.5 million. The share of severely malnourished children under five who recovered after receiving treatment totaled 84 percent, compared to a target of 60 percent. The number of women receiving free delivery services according to quality norms at public health facilities totaled 604,500, against a target of 550,000 and a baseline of 454,000. The number of immunized children totaled 11.1 million, compared to a target of 18 million and a baseline of 11.9 million. Health insurance coverage for poor and vulnerable households reached 181,500 households, compared to a Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) target of 182,000 households, representing about 50 percent of Kenya’s orphaned and vulnerable children. An impact evaluation (cited in the ICR, page 17) concluded that health insurance subsides for the poor increased utilization of health services and reduced their share in household expenditures. The average number of hospitalizations per household per annum increased from 1.4 to 1.6, compared to a decline of 1.6 to 1.3 in a control group. In addition, the subsidy was estimated to have reduced health spending by 6.8 percent as a share of household expenditures. Summary. Practically the whole population now has access to essential health and nutrition services, albeit through services with a particular focus on the poor. The project supported lower level health facilities mainly used by the poor; targeted drought-affected populations; and supported rollout of a health insurance subsidy scheme for the poor. Outpatient visits to Level 1, 2, and 3 services (services at sub-national levels) almost doubled; in drought-stricken areas they almost tripled. Female outpatient visits increased by more than 140 percent, while women taking advantage of free delivery increased by 33 percent. Immunizations, however, remained flat. Decentralization has disrupted the flow of funds under the HSSF as well as the RBF schemes, as the timely provision of funds for planning and operational purposes has become uncertain. Overall, however, the shortcomings were outweighed by the achievements in access to and quality of nutrition and health services, especially for women, children, and drought-affected populations. Rating Substantial OBJECTIVE 2 Objective To improve the effectiveness of planning, financing, procurement of pharmaceutical and medical supplies. Rationale The objective was to improve the efficiency of the medical supplies authority, KEMSA, by introducing reforms to the supply chain, establishing a buffer stock of essential medicines and medical supplies, and introducing more effective procurement procedures, all to ensure a reliable supply of medicines and medical supplies. Planning. Management of the supply chain for medicines and medical supplies in KEMSA was reformed by integrating previously separate vertical supply chains into a single system. However, at the same time, counties were made responsible for planning and managing their respective drug and medical supply needs, something they were not equipped to do, in all likelihood reducing efforts at increased efficiency in planning. The ICR provided no information on the impact of the decentralization reform to planning in the provision of medicines and medical supplies. It did, however, indicate that “after devolution, many counties failed to order and pay for essential medicines from KEMSA on time, increasing stock-outs" (ICR, page 20). Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) Financing. In order to adequately resource KEMSA in procuring medicines and medical supplies, the project financed the establishment of a buffer stock of medicines and supplies; this supported the existing mechanism of reimbursement of KEMSA by the Ministry of Health. Under this structure, KEMSA now is able to sell medicines and medical supplies on a revolving fund basis, responding to the needs of 47 counties. Procurement. With predictability of financing in place, KEMSA was able to attain a 100 percent completion rate of planned procurements, against a target of 60 percent. KEMSA was also able to reach a value fill rate of 83 percent, against a target of 80 percent and a baseline of 73 percent. Summary: Mechanisms were put into place that plausibly ensured the timely availability of essential medicines and medical supplies, as indicated by value fill ratios and completed procurement orders. Any disconnect that may arise in financing or delivery in the supply chain is likely to be driven by external factors, notably a still-incomplete decentralization reform, not necessarily by issues in the new medicines and commodities supply system. Rating Substantial OVERALL EFF TBL OLD Rationale The first objective - provision and use of essential services - was largely met, despite the disruptions created by the decentralization of health operations to county levels. Services may have been affected by the decentralization, notably through disruptions to the provision of medicines and medical supplies, which may have had some effect on health outcomes. However, the project does not offer any definite indications that this might have been the case. The second objective was substantially achieved. Overall efficacy is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency No efficiency analysis was undertaken in the PAD. The ICR (pages 48-50) included an economic analysis of the delivery of KEPH based on income from mothers’ lives saved, where the internal rate of return was 9.9 percent. Using a discount rate of 3 percent and a net present value of US$60.9 million, benefits outweighed costs, with a benefit-cost ratio of 1.7:1. Increasing the discount rate to 5 percent still yielded a positive return (US$3.4 million) and a benefit-cost ratio of 1.04:1. The analysis did not include the effects of child health and nutrition. Nutrition data was unavailable, though experience indicates that emergency Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) supplementary feeding programs do tend to be modestly cost-effective; in this case, the ICR team considered the economic benefits to be minimal. Supporting the intervention package (KEPH), which included cost- effective interventions, also contributed to efficiency. The additional spending on priority cohorts (under AF) is likely to have provided value for money by dampening the health effects of the crisis. However, these efficiencies were tempered by the less-than-successful devolution of service delivery to county levels: the funds flow arrangements under the HSSF and RBF were disrupted, and essential activities were curtailed (basic repairs, maintenance, and certain administrative functions under the former; and RBF flows under the latter, where some US$4 million of the proceeds of the project were cancelled). While procurement of medical equipment and supplies under KEMSA improved as a result of the project, ministry procurement remained slow and experienced considerable delays, and county-level procurement was abandoned altogether. In addition, absorption of RBF funds by facilities remained low, due to disbursement delays and delays in internal clearances in the health ministries (ICR, page 25), staff turnover after the merger of the MOMS and MOPH (ICR, page 25), and protracted health workers' strikes that lasted for months and disproportionately impacted front-line facilities serviced by the project (ICR, pages 18 and 25). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 100.00 ICR Estimate  9.90  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the PDO is rated high, as the PDO responded to country needs and Bank strategy. Efficacy for Objective 1 is rated substantial, as indicators related to the increase in scope were all met. Efficacy for Objective 2 is also rated substantial, noting, however, its sensitivity to the stability of financial flows in the system. Efficiency is rated modest. These ratings are indicative of moderate shortcomings in the project's preparation and implementation, and Outcome is therefore rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) 7. Risk to Development Outcome The project helped set up systems that continue to address priorities in health care in Kenya. A new Bank project, "Transforming Health Systems for Universal Care" (US$ 150 million, 2015-2021), offers some assurance that the development outcome of this project will be preserved. However, to a large extent outcomes – timely delivery of essential services, planning, procurement, and supply chain management -- are dependent on maintaining predictable financial flows that facilitate these activities, and on the success of the decentralization effort. These are processes that are still taking shape, and there is at least moderate risk if resource flows are not mastered better than was the case during project implementation. After devolution, the Danish International Development Agency -- the only other partner financing the HSSF -- withdrew, leaving the Bank as the only financier of that element of the JFA. Many of the other SWAp structures were similarly impacted by devolution and are not operating as initially envisioned (ICR, page 44). 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant, taking into account the country and sector context. It was designed as part of a SWAp, ensuring coherence with support provided by other donors and focus in areas where the Bank had a comparative advantage, as did the establishment of a SWAp secretariat for information sharing. The project also built on the experience of previous Bank projects. Still, the project may have been too ambitious. It was developed during a period of administrative reorganization, and implementation of the HSSF grant system and RBF would coincide with the devolution of operational tasks, including planning and financial management, to sub-national (county) authorities. The latter would find it challenging to cope with their new responsibilities, and this would subsequently affect the management of drug and medical supply needs. The M&E system (the results framework) could have been stronger, especially when flagging behavior changes instigated by the project. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Supervision was managed from the country office, and this was reflected in the responsiveness of the project to evolving circumstances in the health sector and the country. In addition to regular supervision, and in line with the SWAp, biannual missions were held jointly with other participants. Through AFs and project restructurings, the team took a proactive approach in addressing needs as they arose, notably in helping address the drought, scaling up project activities to target women and children as project beneficiaries, and accommodating delays in finalizing project activities. Still, the project would experience flow-of-funds disruptions that would affect implementation of the RBF, and while key indicators were Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) revised twice, outcome indicators did not well determine the impact of project interventions on either institutional or client behaviors. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design A detailed results framework and monitoring plan were developed during preparation. Data on the majority of indicators were to be collected from routine data sources, complemented by surveys where needed. Originally, six outcome indicators and fourteen intermediate indicators were to be tracked. All had baseline data and targets. Still, the results framework had only two indicators that were specifically focused on outcomes: number of project beneficiaries and children immunized for the first objective, and drug stock- outs as a proxy for the second objective. This shortcoming was rectified during the AFs, when new outcome indicators were introduced to reflect adjustments in the PDO to accommodate drought-affected and women and children. A persistent absence was that of institutional indicators measuring the impact of change. b. M&E Implementation Results were regularly updated in Implementation Status and Results Reports. The results framework was adjusted to changes in the PDO, and indicators were revised accordingly. Two studies – a household expenditure survey and a service delivery indicator survey -- were started during implementation but were not completed by project closing, and therefore were not able to inform project implementation. c. M&E Utilization The project appears to have generated much data that was not used effectively. KEMSA built an MIS, but with weak data analysis at the county level its value may have been reduced. Much of the reporting that flowed to the Ministry of Health was not shared with key stakeholders, including the Bank, and therefore also remained of limited value. M&E Quality Rating Modest Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) 10. Other Issues a. Safeguards Environmental safeguards. The project triggered Operational Policy 4.01, Environmental Assessment, and was classified as environmental Category B. The safeguard policy issue concerned handling and disposal of medical waste generated by health facilities The project procured incinerators for five hospitals and supported training in the handling and disposal of medical waste and infection control. The project overall was compliant with the environmental safeguard policy (ICR, page 27). Social safeguards. The project was designed to address inequities in access to and utilization of health services. It triggered OP 4.10 on Indigenous Peoples, for which an Indigenous Peoples Planning Framework was prepared at appraisal. During implementation, and especially after the AFs, emphasis was placed on targeting drought-affected populations and women and children from the arid and semiarid regions of Kenya. The poor were also targeted. b. Fiduciary Compliance Financial management. At the project's mid-term review (August 2013), financial management arrangements in the central Ministry of Health and counties were reported as adequate. There were no outstanding audit issues, and interim financial reports were submitted in a timely manner and in a form acceptable to the Bank. However, the advent of devolution disrupted the HSSF financial management arrangements and adversely affected the transfer of grants to the health facilities. Many counties were no longer submitting financial returns, and by project closure, several issues raised in the audit reports and other reviews had not been addressed, including ineligible expenditures of Kshs 296 million and potential ineligible expenditure of Kshs 1,166 million. The audit report for the period ending June 30, 2018 also noted that the statement of financial assets and liabilities reflected a bank balance of Kshs 1,029 million as of 30 June 2018. Furthermore, the National Hospital Insurance Fund implementing the health insurance subsidies did not submit quarterly reports to the Ministry of Health, and whatever reports there were, were not shared with stakeholders and could therefore not inform project implementation. The financial management issues are being followed up under the follow-on project. Procurement. The bulk of procurement under the project was carried out under component two by KEMSA. Prior to the project, procurement by KEMSA experienced major shortcomings, but with support of the project, KEMSA made substantial progress in the procurement of essential medicines and implementation of procurement plans. In the project design, health facilities were meant to handle small procurements. c. Unintended impacts (Positive or Negative) Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) None reported. d. Other Gender The project directly targeted women providing a range of essential reproductive and maternal health services. In addition to the 34.4 million women beneficiaries, the project supported over 600,000 women in receiving free delivery services that met the quality norms at public health facilities. Institutional Strengthening While devolution disrupted the project’s fund flow arrangements, the project contributed to building capacity of counties to manage fiscal transfers from the government. The reforms of the supply chain management system, which were to a large extent supported by the project, have enhanced KEMSA’s institutional capacity. Poverty Reduction and Shared Prosperity The project was pro-poor. It supported lower level health facilities utilized mainly by the poor; targeted drought affected populations by supplying nutrition commodities to manage malnutrition among children under five years, pregnant and lactating women; and supported the rollout of the government flagship program providing health insurance subsidies to the poor. Furthermore, the RBF program was implemented in the 21 semiarid and arid counties where poverty levels are highest in Kenya. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons These lessons are adapted from ICR: Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) Being aware of the most vulnerable in a crisis situation can strongly improve outcomes. Everyone may be vulnerable in a crisis, but some are more so. As the project turned to address people suffering from the drought, it placed particular emphasis on delivering appropriate services to high-risk groups, essentially pregnant and lactating women and children under five. Ongoing administrative reforms may influence how reforms on the ground will play out. Insufficient attention to administrative reforms may require later adjustments to project design. The ongoing decentralization process gave new responsibilities for delivering essential health services to county governments. As a result, project implementation arrangements, including financial management and procurement, were critically affected, and the implementation of both the HSSF and RBF programs faltered. Overly tight project schedules may not be helpful to implementation. Project performance may often be affected by adherence to tight timetables rather than realities on the ground. The project closing date was extended twice, in part to accommodate increases in scope, but in part to allow implementation to be completed. Direct funding of health facilities requires preparation. Direct transfer of funds to sub-national health facilities can improve service delivery, increase its relevance, and promote transparency and accountability. Implementation of both the HSSF and RBF programs faltered in Kenya in the absence of well-defined roles between the central authority and county levels. Implementation of complex projects can be facilitated by in-country presence of the task team. When projects are complex and new procedures are being introduced, close hands-on assistance by Bank teams can be an important feature of success. This project was complex and implemented in a challenging context, where the continuity and in-country presence of the task team was critical to maintain focus. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KE-Health SWAP (FY10) (P074091) The ICR provided a thorough assessment of the project. It covered the critical features in sufficient detail, providing enough evidence-based information and analysis to allow the ICRR to evaluate the project. It was sufficiently referenced where necessary. In the absence of a clear results framework in the PAD, the ICR did a commendable job in setting out the theory of change. The theory of change (or the results framework) could have been used more rigorously in analyzing the achievement of PDOs and their links to intermediate outcomes and outputs. That would have facilitated drawing conclusions about the achievements of the project; as is, one is left a bit interpreting the text. The analysis of efficiency was well structured, and the discussion of M&E more thorough than is usually the case. Overall, the ICR was relatively clear in its messaging and responsive to the guidelines. a. Quality of ICR Rating Substantial Page 15 of 15