Document of The World Bank FOR OFFICIAL USE ONLY Report No: 142356-AFR IMPLEMENTATION COMPLETION AND RESULTS REPORT TF093070 ON A GRANT IN THE AMOUNT OF USD 70.1 MILLION TO THE Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) FOR THE AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund September 15, 2019 Agriculture and Food Global Practice Africa Region CURRENCY Currency Unit = US $ FISCAL YEAR July 1 - June 30 Regional Vice President: Hafez Ghalem Country Director: Deborah Wetzel Regional Director: Simeon Ehui Practice Manager: Holger Kray Task Team Leader(s): Bremala Nathan ICR Main Contributor: Emma Isinika Modamba ABBREVIATIONS AND ACRONYMS AF Additional Financing ASARECA Association for Agricultural Research in Eastern and Central Africa A-SRF ASARECA Strategy and Results Framework AU African Union CAADP Comprehensive African Agricultural Development Program CGS Competitive Grants System DFID United Kingdom’s Department for International Development ECA Eastern and Central Africa FAAP African Agricultural Productivity FAP Fiduciary Action Plan IAR4D Integrated Agricultural Research for Development IFR In-depth Fiduciary Review INT Integrity Vice Presidency MDTF Multi-Donor Trust Fund NARIs National Agricultural Research Institutes NARS National Agricultural Research Systems NEPAD New Partnership for Agriculture Development OP Operational Plan S3A Science Agenda for Agriculture in Africa SDGs Sustainable Development Goals SG Sub-Grant SP Strategic Plan TIMPs Technologies, Innovations and Management Practices TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL ......................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................11 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDO...........................................................................................................14 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................15 C. EFFICIENCY ........................................................................................................................20 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................21 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ........................................................................22 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 25 A. KEY FACTORS DURING PREPARATION ................................................................................25 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................25 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 27 A. QUALITY OF MONITORING AND EVALUATION (M&E) ...........................................................27 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................28 C. BANK PERFORMANCE ...........................................................................................................30 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................31 V. LESSONS AND RECOMMENDATIONS ............................................................................. 32 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 35 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 46 ANNEX 4. SUPPLEMENTARY RESULTS MATRIX FOR ASARECA TRANSITION PERIOD (2017-18) 47 ANNEX 5. RECIPIENT COMMENTS ........................................................................................ 48 ANNEX 6. SUMMARY OF RESULTS AND RATINGS OF OP I IMPLEMENTATION COMPLETION AND RESULTS REPORT (ICRR) ....................................................................................................... 49 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name AFCC2-RI-Association for Strengthening Agricultural P112600 Research in Eastern and Central Africa (ASARECA) Trust Fund Country Financing Instrument Africa Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Association for Strengthening Agricultural Research in Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Eastern and Central Africa (ASARECA) Project Development Objective (PDO) Original PDO To support a program of integrated agricultural research for development as described in ASARECA’s Operational Plan (OP) in order to contribute to more effective use of resources devoted to agricultural research among the Member Countries and resulting in wider access to the results of research in the regional agricultural systems of Eastern and Central Africa Revised PDO To enhance access and utilization of agricultural research technologies and innovations in the regional agricultural systems ofEastern & Central Africa Page 1 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 70,100,895 65,374,615 65,374,615 TF-93070 Total 70,100,895 65,374,615 65,374,615 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 70,100,895 65,374,615 65,374,615 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 15-Jul-2008 04-Nov-2008 19-Sep-2011 31-Dec-2013 31-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 10-Dec-2012 38.68 Additional Financing Change in Project Development Objectives Change in Components and Cost Change in Financial Management 31-Dec-2013 58.92 Additional Financing Change in Components and Cost KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial Page 2 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 26-Nov-2008 Satisfactory Satisfactory 0 02 28-May-2009 Satisfactory Satisfactory 3.00 03 10-Dec-2009 Satisfactory Satisfactory 12.77 04 23-Jun-2010 Satisfactory Satisfactory 14.02 05 29-Nov-2010 Satisfactory Satisfactory 18.03 06 02-Apr-2011 Satisfactory Satisfactory 23.95 07 11-Jul-2011 Satisfactory Satisfactory 23.95 08 08-Jan-2012 Satisfactory Satisfactory 28.34 09 11-Aug-2012 Satisfactory Satisfactory 32.96 10 05-Sep-2012 Satisfactory Satisfactory 32.96 11 14-Apr-2013 Satisfactory Satisfactory 50.68 12 05-Oct-2013 Satisfactory Satisfactory 58.19 13 08-Jun-2014 Satisfactory Satisfactory 63.92 14 26-Jan-2015 Moderately Satisfactory Moderately Satisfactory 65.42 15 02-Sep-2015 Moderately Satisfactory Moderately Satisfactory 65.42 16 31-May-2016 Moderately Satisfactory Moderately Satisfactory 66.16 Moderately 17 30-Dec-2016 Moderately Unsatisfactory 66.16 Unsatisfactory 18 19-Jun-2017 Unsatisfactory Unsatisfactory 64.13 19 08-Jan-2018 Moderately Satisfactory Moderately Satisfactory 63.95 Moderately 20 15-Oct-2018 Moderately Unsatisfactory 65.05 Unsatisfactory Page 3 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) SECTORS AND THEMES Sectors Major Sector/Sector (%) Agriculture, Fishing and Forestry 89 Agricultural Extension, Research, and Other Support 33 Activities Public Administration - Agriculture, Fishing & Forestry 56 Education 11 Tertiary Education 11 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Finance 25 Finance for Development 25 Agriculture Finance 25 Urban and Rural Development 75 Rural Development 75 Rural Markets 25 Rural Infrastructure and service delivery 50 ADM STAFF Role At Approval At ICR Regional Vice President: Obiageli Katryn Ezekwesili Hafez M. H. Ghanem Country Director: Mark D. Tomlinson Deborah L. Wetzel Director: Jamal Saghir Simeon Kacou Ehui Practice Manager: Karen Mcconnell Brooks Mark E. Cackler Task Team Leader(s): David J. Nielson Bremala Malli ICR Contributing Author: Emma Isinika Modamba Page 4 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Page 5 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of appraisal in 2008, countries in Eastern and Central Africa had some of the highest poverty and hunger rates in the world. Approximately 40 percent of the 200 million population of Kenya, Ethiopia, Uganda, Rwanda and Tanzania were classified as poor and chronically food insecure. The share of the population living on less than one dollar a day had increased in six of the ten member countries 1 of the Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) since 1990. Most were living in rural areas and dependent on agriculture—a core economic sector contributing over 30 percent of gross domestic product (GDP) and employing over 60 percent of the population. Incomes were limited by low productivity and poor market access. 2. Common challenges to most of Africa’s agricultural productivity and institutions responsible for agriculture productivity revolve around poor technology generation and dissemination capacity, insufficient end user involvement, ineffective farmer support systems, and systematic fragmentation. As a result, agricultural productivity within Africa had been low. While yields improved over the past decade, they remained far below potential. Agriculture research had demonstrated that enhanced technologies, such as improved seeds and soil management and healthcare for livestock, could more than double the yields. 3. ASARECA, a sub-regional non-profit organization of the National Agricultural Research Institutes (NARIs) was tasked to catalyze, promote and coordinate agricultural research, to contribute to more effective resource use, to leverage the potential for regional spillovers in R&D, creating wider access to research results among member countries. For this purpose, an umbrella framework of a ten-year Strategic Plan (SP) (2008-2018), implemented through two successive five-year medium-term Operational Plans: OP I (2008-2013) and OP II (2014- 2018), was supported by a World Bank-administered Multi-Donor Trust Fund (MDTF). ASARECA also played a critical role in meeting the African Union’s/New Partnership for Africa’s Development (AU/NEPAD) Comprehensive Africa Agriculture Development Programme (CAADP) agenda, which had set a goal of 6 percent annual growth for the sector, particularly focusing on CAADP Pillar IV, guided by the Framework for African Agricultural Productivity (FAAP 2). The FAAP specifically calls for: (a) reforms to agricultural institutions and services; (b) increases in the scale and scope of investments made in agricultural productivity; and (c) alignment and coordination of financial support. 4. The World Bank-administered MDTF program was established at the request of donors to facilitate a harmonized approach to investments in Integrated Agricultural Research for Development (IAR4D) in the sub- region. At that time, the World Bank had gained valuable experience over many decades of support to agricultural 1 At appraisal, ASARECA had ten-member countries - Burundi, Democratic Republic of Congo (DRC), Eritrea, Ethiopia, Kenya, Madagascar, Rwanda, Sudan, Tanzania and Uganda. ASARECA added two member countries - South Sudan in December 2011, and Republic of Congo in December 2018. 2 FAAP was developed in 2006 and formally endorsed through the AU Council of Ministers as the strategy for CAADP Pillar IV with the aim of revitalizing, expanding and reforming Africa’s agricultural research, technology dissemination and adoption efforts. Page 6 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) research and extension worldwide, had several interventions in the majority of the ASARECA member states, and had provided leadership in donor coordination to strengthen African capacity in agricultural research and innovation. These advantages strategically positioned the Bank to perform this role. The 2008 World Development Report 3 that focused on Agriculture for Development and the Global Food Crisis of 2007/08 also provided additional and strong justification for regional collaborative efforts to boost AR4D initiatives to address food insecurity and rural poverty. It is in this context that the MDTF was deemed essential and formulated to effectively and efficiently address the food insecurity and rural poverty challenges in ECA. 5. Implementation of the first five-year Operational Plan (2008-2013) of the MDTF was initially supported by three donors 4 – the European Commission, Global Affairs Canada (then Canadian International Development Agency), and the UK’s Department for International Development (DfID), with pledges totaling US $55 million. The Grant became effective in November 2008 and was designed to support ASARECA for a period of five years. By December 31, 2013 (the original closing date), the United States Agency for International Development (USAID) had joined the MDTF, bringing the total Grant amount to US $59.2 million, of which over 99% had disbursed. OP I was implemented successfully, and an Implementation Completion and Results Report (ICRR 5) rated both the project development objective (PDO) and implementation progress (IP) as Satisfactory for the OP I period. 6. In December 2013, the Grant was extended by five years to December 31, 2018 with additional financing (AF) approved in the amount of US $30 million to support implementation of OP II (2014-2018) 6. The priority for OP II was to transform agriculture in the sub-region into a viable market-oriented venture, by focusing on improved productivity, increasing engagement with the private sector, supporting enhanced access to innovations, technology and markets for smallholder farmers, the majority of whom are women. However, donor support for OP II was by far lower than anticipated, with only US $10.9 million secured 7, bringing the total Grant amount to US $70.1 million. As a result, ASARECA’s OP II activities targeted a few priority areas that contributed to achievement of the PDO within the available AF resource envelope (see Annex 1). It was envisaged that as more funds became available over time, ASARECA would adjust the indicator targets to reflect any increase in funding and scale of operation. 7. By design, the MDTF supported the implementation of a ten-year strategic plan to achieve one PDO, which spanned the two OPs. As noted above, an assessment of OP I was undertaken and approved in June 2014 with a Satisfactory rating (see summary in Annex 6). Therefore, this ICRR will not re-assess the results of OP I, but rather the results of OP II (2014 to 2018), which disbursed 10 percent of the total Grant funds (US $6.4 million) and complements the OP I results which disbursed 90 percent of Grant funds (US $58.9 million). A split rating 3 The World Development Report emphasized that agriculture continues to be a fundamental instrument for sustainable development and poverty reduction. It also highlighted the importance of a multi-sectoral approach to capture the synergies between technologies, and the need to coordinate across countries to provide an expanded market and achieve economies of scale in AR4D initiatives. 4 At MDTF preparation, pledges from the first three donors amounted to US $55 million. 5 The OP I (2008-2013) ICR is a key reference document for this ICR and a summary of its results and ratings are appended to this ICR as annex 6. 6 The AF was approved on December 24, 2013, one week prior to the original closing date and the commencement of OP II. The AF did not introduce changes to the existing PDO, overall structure, implementation and institutional arrangements, or environmental category, as it did not trigger new safeguard policies or related safeguard actions. At the same time, the Grant was restructured to add a component related to support for a knowledge and information hub, and adjustments to the results framework (increased targets and additional indicators to cover the new component). 7 OP II donors included the European Commission and the United States Agency for International Development. Page 7 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) methodology will be applied to consolidate the existing OP I assessment with the new OP II assessment to produce an overall consolidated project outcome rating, based on disbursements prior to and following AF. Theory of Change (Results Chain) 8. The overall logic and structure of the theory of change, as captured in December 2013, when the Grant AF was approved, is illustrated in Figure 1 below. The PDO, i.e., to enhance access and utilization of agricultural research technologies and innovations in the regional agricultural systems, was to be achieved through parallel and mutually supportive investments in: (i) research activities and policy analysis to generate and disseminate technologies considered to be of priority for the sub-region: (ii) supporting research programs implemented by national agricultural research systems (NARS); (iii) reforms to manage the change process to enable ASARECA to execute the expanded mandate for regional agricultural research, extension, education and training; (iv) strengthening ASARECA’s governance structures required to strategically position ASARECA as a regional leader in agricultural research for development; and (v) strengthening ASARECA’s administration and management, as well as improving the Secretariat’s physical infrastructure. These interventions were intended to generate the following outputs and outcomes: (i) appropriate technologies, innovations and management practices (TIMPs) adapted or generated; (ii) appropriate TIMPs disseminated and up-scaled; (iii) appropriate policy analysis and advocacy enhanced; (iv) stakeholder capacity for innovation strengthened; (v) communication, information and knowledge management improved; and (vi) gender-responsive governance and management strengthened. 9. At that time, ASARECA’s activities also involved a practical shift away from a predominant focus on technology generation, to a broader and more equitable distribution of effort and resources on upscaling and dissemination of research findings. Emphasis was placed on market and value chain-based approaches supported by policy analysis/research. Key cross-cutting issues such as gender mainstreaming and climate smart agriculture were integrated into every aspect of research and policy analysis activities. ASARECA also initiated a regional knowledge and information (KI) hub to support the generation and exchange of knowledge, networking, and sharing of the sub-regional benefits of its interventions. It was assumed that ASARECA would have and maintain enough capacity and governance oversight to facilitate regional activities in line with sound financial management principles. Sustained donor and member governments’ commitment were also assumed to be in place. However, many external factors which underpin success, such as finance and the implementing agency’s capacity were serious constraints that were outside the control of ASARECA. Page 8 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Figure 1. Theory of Change (Results Chain) Project Development Objective (PDO) Page 9 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 10. The original project development objective was “to support a program of integrated agricultural research for development (IAR4D) as described in ASARECA's first Medium-Term Operational Plan (OP I), to contribute to more effective use of resources devoted to agricultural research among the member countries and wider access to the results of research in the regional agricultural systems of Eastern and Central Africa”. In 2012, the PDO was revised to better align it with the results framework of the ASARECA Strategic Plan following a mid-term review (MTR) recommendation in September 2011. 11. The revised PDO was to “enhance access and utilization of agricultural research technologies and innovations in the regional agricultural systems of Eastern & Central Africa”. Implementation of OP II started in January 2014 with the revised PDO. Key Expected Outcomes and Outcome Indicators 12. Following revision of the PDO, OP I had five outcome indicators; (i) number of stakeholders accessing the technologies, innovations and management practices (TIMPs); (ii) percentage of stakeholders adopting new TIMPs in selected development domains; (iii) number of reform of policies, laws, regulations and procedures approved; (iv) level of stakeholder satisfaction with TIMPs; and (v) number of direct beneficiaries reached through ASARECA support (disaggregated by gender). 13. During AF processing and restructuring, OP I indicators were revised to reflect the anticipated results attributable to the AF contribution and extended implementation period. The revision also included new monitoring indicators and targets to reflect the increased scale, scope, and number of beneficiaries as a result of the AF, and the knowledge and information hub component. OP II outcomes were to be assessed using three PDO- level indicators and 14 intermediate outcome indicators. 14. PDO outcome 1: To enhance access to agricultural research technologies and innovations in the regional agricultural systems of ECA • Indicator 1: Number of hectares under improved TIMPs • Indicator 2: Beneficiary satisfaction with quality of ASARECA KI-hub, products and services 15. PDO outcome 2: To enhance utilization of agricultural research technologies and innovations in the regional agricultural systems of ECA • Indicator 3: Percentage change in agricultural production and productivity Components 16. Under OP II, the Grant maintained the three original OP I components, and added a new component 2 just prior to the start of OP II implementation8: Component 1: Support to Research for Development (US $16.25 million) 9.: This component 8 A Bank-executed component was erroneously entered as part of the project design in the original Project Document and maintained as Component 5 totaling US $1.55 million in the December 2013 AF Project Paper. 9 The figure in parentheses following each component reflects the December 2013 AF allocations for OP II, minus costs associated with Bank Page 10 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) supported: (i) agricultural research mainly through a rigorous competitive grants system (CGS), or directly commissioned grants (DCG); (ii) agricultural policy analysis related to productivity, markets, public policy and strategy, as well as reform and policy advocacy to help policy decisions; and (iii) agricultural extension and advisory services enhancement through NARS, regional and international research bodies; (iv) facilitation of technology adoption and knowledge sharing. • Sub-component 1.1: Research Activities, including agricultural research and dissemination; agricultural policy analysis and dissemination; capacity building and strengthening of national agricultural research systems, agricultural extension systems, and agricultural education systems through the issuance of sub-grants. • Sub-component 1.2: Support to Research Programs, including planning, priority-setting, and managing sub-regional projects implemented by the NARS, grants management, partnerships and capacity development, and monitoring and evaluation. • Sub-component 1.3: Reform Process, including execution of the expanded mandate for regional agricultural research, extension, education and training. Component 2: Support to ASARECA’s Knowledge and Information Hub (US $2.07 million): This component supported the establishment of a knowledge and information hub aimed at enhancing access to and utilization of agricultural knowledge and information within and beyond the ECA sub-region. Component 3: Support to ASARECA's Governance Structures (US $1.75 million): This component sought to strengthen the overall governance and oversight structures to enhance ASARECA’s ability to fulfil its mandate. Component 4: Support to ASARECA's Management & Administration Structures (US $8.38 million): This component aimed to support and strengthen the Secretariat, including financial management (FM) and procurement, monitoring and evaluation and information and communication capabilities. It also was to support improvements to the physical infrastructure of the headquarters. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets Revised PDO 17. No revisions to the PDO were made during OP II implementation. Revised Components 18. No revisions to components were made during OP II implementation. program management and supervision, and cost recovery. Page 11 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Other Changes 19. Financial stress. While the AF was approved for US $30 million, it soon became obvious that such funding would not become available. ASARECA, as advised by the Bank, started to rationalize its operations, reducing staff numbers, simplifying the Operational Plan and economizing on non-essential expenditures. By September 2014, a whistleblower (ASARECA staff member laid off during this process) raised allegations of fraud and corruption to ASARECA’s Board, the Bank Team, and donors. An initial first-stage forensic review identified red flags and raised serious concerns. In December 2014, the Bank initiated an in-depth fiduciary review (IFR). The IFR operated until February 2017 and found evidence of mis-procurement, procurement non-compliance, and ineligible expenditures. ASARECA was requested to refund US $2.2 million, which was then refunded to the grant. . 20. Although there were no changes to the PDO and its indicators, the project introduced operational changes in response to these developments that emerged during OP II implementation. These included: • Adjusting the scale, as well as the type of program activities. Three distinct implementation phases emerged as a result of the adjustments that were made in the course of implementation. First, the period prior to the IFR during which the Secretariat was forced to scale down its work program and its work force due to shortage of funds (January 2014 – December 2014). Second, the Bank’s IFR period when activities were suspended until the review was completed 10. Third, post IFR (Transition Period) during which recommendations of the IFR were assembled into an action plan, to reform and strengthen the Association (March 2017 – December 2018). • Suspension of Component 1 (sub-grants): The Bank suspended the award and implementation of all sub- grants for the remaining OP II period following the whistle blowers’ allegations. • Charting a new Roadmap: A roadmap was prepared to adequately prepare ASARECA to rebrand and prepare itself to take up its new vision, mandate, goals and objectives, and restore the institution’s credibility and reputation by the project closing date of December 31, 2018. • Introducing changes in ASARECA’s Governance and Management: Based on the transition period work program (2017 – 2018), ASARECA proposed to restructure its governance organs as follows; (i) forge a strong affiliation with regional economic communities (RECs) 11; serve as their technical arm and gain ownership by the RECs; (ii) create a Council of Patron Ministers made up of Ministers of Agriculture of member countries as the Association’s highest governing body, to further strengthen member countries’ ownership; (iii) reconstitute the Board with new constituent members and fewer representatives; and (iv) revise the Secretariat’s organizational structure and needed staffing needs. Rationale for Changes and their Implication on the Original Theory of Change 21. Limited Funding: ASARECA failed in its mobilization of resources for OP II in 2014. With limited funding prospects on the horizon, it became apparent that the OP II program and activities could not be implemented as 10 An in-depth fiduciary review was conducted by the Bank’s fiduciary team, together with the Integrity Vice Presidency (INT) following an allegation of fraud and corruption within the Secretariat. 11 Notably the Common Market for Eastern and Southern Africa (COMESA), Inter-Governmental Authority for Development (IGAD), and the East African Community (EAC). Page 12 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) planned, and that the staff required to implement OP I would be excessive. The implication of such a large funding gap experienced in the first year of OP II also meant that the PDO might only be partially achieved and target performance indicators unmet. In addition, as a result of the IFR, some development partners who supported ASARECA in the past reacted by suspending their support, and prospective development partners that were ready to support ASARECA took their resources elsewhere, thus exacerbating the resource shortage. Therefore, a decision was taken to suspend disbursement for sub-grants and reduce activities of the Secretariat to the work program agreed for the transition period. This decision was upheld during the Bank’s re-engagement in March 2017, until Grant closing on December 31, 2018. 22. Dearth of technical capacity at ASARECA: ASARECA’s capacity significantly diminished following four rounds of downsizings that took place between 2014 - 2017, representing a 77% reduction in staff from 64 in 2014 to 14 by mid-2017. This included the non-renewal of contracts for all regional staff in December 2016. An Interim Executive Secretary (IES) was appointed by the ASARECA Board in January 2017. Apart from the IES, all staff were relatively junior, and the three remaining technical staff did not possess the capacity to oversee the implementation of any major activities. By June 2018, only 37.5% of the required positions had been filled. 23. These changes had significant implications on the Secretariat’s ability to ensure that the project achieved its objectives. II. OUTCOME 24. To determine the overall outcome rating of the project, a split rating methodology was applied. As previously noted, an ICRR was approved by Bank management for OP I, therefore this grant will only assess progress under OP II and calculate the results with those of the OP I ICRR. The split rating will be based on the amount disbursed during each OP period as a percentage of the final disbursed grant amount, thereby justifying a split rating (see Annex 1). The overall outcome rating for the project is therefore assessed against both OP I and OP II project phases: • OP I - (US $58.9 million; 90 percent) - from project approval in November 2008 to December 2013. • OP II - (US $6.5 million; 10 percent) - from January 2014 to project closure in December 2018. Justification for a Consolidated overall outcome rating 25. The Grant to ASARECA was initially planned to support implementation of the first five years (OP I) of ASARECA’s ten-year strategic plan; however, a five-year extension was processed prior to the completion of OP I. With the ASARECA Grant subsequently projected to be a 10-year operation, the Bank Team produced an interim ICRR to assess OP I progress, in keeping with Bank guidance on projects of such duration. An ICRR was approved for OP I in June 2014 with a Satisfactory rating (see Annex 6 for summary), therefore, this ICRR will not re-assess those results. Rather, it will assess the results of OP II (from 2014 to 2018), which disbursed 10 percent of the total Grant funds (US $6.4 million) compared to 90 percent of grant funds (US $58.9 million) under OP I (2008-2013). A split rating methodology will be applied to consolidate the OP II assessment with the already completed OP I assessment to produce an overall consolidated project outcome rating. Page 13 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) A. RELEVANCE OF PDO Assessment of Relevance of PDO 26. Under OP I and OP II, the Grant PDO remains relevant to ASARECA’s strategy, which is aligned with regional and global initiatives supporting AR4D. At the regional level, the PDO remained aligned with AU/NEPAD’s CAADP Pillar IV strategy for revitalizing agricultural research, technology dissemination and adoption, and the Science Agenda for Agriculture in Africa (S3A) which articulates the science, technology, extension, innovations, policy and social learning that Africa needs to apply to meet its agricultural and overall development goals. The strategic thrusts of the S3A in the short to medium term are: the implementation of CAADP; increasing domestic public and private sector investment; creating the enabling environment for sustainable application of science for agriculture; and doubling the current level of agricultural total factor productivity by 2025 through application of science for agriculture. The CAADP agenda was ratified by African Heads of States in Malabo, signaling Africa’s intent to commit to a science-led process at the national level to secure agricultural transformation and Africa’s future. As such, the PDO is still relevant to the member states’ pursuit of raising agricultural productivity through accelerated innovation and technologies, thereby eradicating hunger and poverty. 27. The PDO also remains aligned with the Bank’s Africa strategy given its priority areas include investments in agricultural productivity, climate change, human capital and knowledge, and the 2015 Agenda for Global Food Systems. 12 Similarly, the Bank’s Global Agriculture Strategy (2018) emphasizes working with its partners through the provision of innovation, infrastructure and resources, so that the food and agriculture sector is climate-smart (more productive and resilient), improves livelihoods and creates jobs (including women and youth), boosts agribusiness, and improves food security. At the global level the PDO supports the Sustainable Development Goals, in particular SDG #1 – Poverty Eradication; SDG #2 – Hunger Reduction; SDG #13 – Climate Change; and SDG #15 – Ecosystem Management. 28. The MDTF funding mechanism allowed multiple donors to work together to facilitate ASARECA’s OPs and to pool funds for continued IAR4D activities. It has also been an effective instrument for reducing the administrative burden on ASARECA, through the elimination of separate donor implementation support missions, as well as financial and activity reporting for multiple donors. The Bank, as MDTF administrator, which also provides technical and administrative support, has set consistent and high standards throughout implementation. 29. The multi-faceted approach, which called for interventions in research, dissemination and reforms in agricultural policies, based on thorough analysis of the sector and related fields, encompassed the essential elements needed to enhance ASARECA’s value addition for its work with member country NARS. The MDTF results framework was well-designed, with a concise PDO, and clearly linked and measurable outcome indicators. The restructuring further improved these links by realigning them with the ASARECA OP indicators to allow for better monitoring. 12 World Bank 2015. Ending Poverty and Hunger by 2030: Agenda for Global Food Systems. Page 14 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 30. ASARECA’s outcome assessment under OP I was rated Satisfactory (see Annex 6 for detail). The MDTF helped ASARECA to improve the enabling agricultural policy and regulatory environment for increasing the productivity of smallholder agriculture in the region, and successfully achieved its PDO to enhance access and use of agricultural research technologies and innovations in the agricultural systems in Eastern and Central Africa. The MDTF funded 85 of the 90 successfully concluded CGS sub-projects since 2009 and helped ASARECA to deliver a substantial research program. The sub-projects brought together researchers from 11 countries to generate work on 409 TIMPs of regional importance (against an OP I target of 378). A breakdown of projects by program indicates that 310 sub-grantees received a total of US $30.2 million for research and uptake activities. 31. ASARECA followed a rigorous process to ensure relevance and quality of research through its seven research programs and the CGS. The CGS was ASARECA’s main instrument for addressing regional research priorities and improving the rigor and quality of research proposals. CGS proposals were based on the results of priority-setting workshops attended by stakeholders from all the countries including NARIs, government advisory service providers, non-governmental organizations (NGOs), private enterprises and farmers. This process was effective in ensuring congruence with regional priorities. Research designs, implementation, analyses, interpretation of results and write-up were all undertaken by the regional teams. The CGS was managed by well-qualified scientists and designed to address only high priority topics that can be delivered within a short time span. See Table 1 for highlights of achievements. Table 1: OP I MDTF Key Performance Indicators: Targets and Actual Achieved End of End of Project Project Key Performance Indictors Baseline Project End of Project Actual (percent (Outcome) Indicators 2008 Target achieved) 1. Number of stakeholders accessing 2,014 265,964 422,176 158.7 TIMPs 2. Stakeholders adopting new TIMPs in selected development domains 8.3 70 64 91.4 (percent) 3. Number of reform of policies, laws, 6 28 18 64.3 regulations and procedures approved 4. Level of stakeholder satisfaction with 5 81 71* 87.6 TIMPs. (percent) 5. Number of direct beneficiaries 2,533,056 reached through ASARECA support 12,084 1,595,784 (Male: 55 percent; 158.7 (disaggregated by gender) Female: 45 percent) ASARECA OP I ICRR. 32. Implementation of OP II started with support to three regional projects in nine countries through the CGS: (i) Integrated management of maize lethal necrosis disease; (ii) Enhancing wheat productivity and value chains; and (iii) Improved agricultural water productivity project. Progress on all three projects was broadly on track until April 2016, but disbursements to sub-grantees were suspended as a result of the IFR, leading to serious delays and in Page 15 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) some cases, cessation of project activities. The sub-grant implementation period was effectively reduced from two years to 15 months, which ultimately affected the delivery of the PDO. All three projects had closed by November 2016, and as at December 31, 2016, the amount disbursed from the MDTF for the projects was US$ 1.57 million or 36.3% of the amount budgeted. Due to the closures, only a few outcomes were realized. 33. The following section and tables summarize achievements against the PDO outcome under OP II. Achievement data comes from the last Implementation Status and Results (ISR) Report and the ASARECA’s ICR Report, to measure performance against the targets (see Table 1 for highlights of achievements). Achievements of PDO outcome #1 (to enhance access to agricultural research technologies and innovations) using PDO and intermediate results indicator (IRIs) are presented in Table 2. Achievement of PDO outcome #2 (to enhance utilization of agricultural research technologies and innovations) using PDO and intermediate results indicators (IRIs) are presented in Table 4. The full results framework is presented in Annex 1. 34. PDO Outcome 1: To enhance access 13 to agricultural research technologies and innovations in the regional agricultural systems of ECA. Table 2: OP II MDTF Key Performance Indicators for access to TIMPs: Targets and Actual Achieved End of End of PDO Performance Indicator Baseline Project Project Percent Comments (Outcome) (2013) Target Actual Achieved (2018) (2018) 1. Beneficiary satisfaction with quality of ASARECA KI-hub, products 68 75 61 -10% Not Achieved & services Intermediate Results Indicators (IRI) Exceeded. The actual performance was 1.Number of demand-driven gender 125. The target and actual performance 364 414 489 250% responsive TIMPs generated have been made cumulative to align the target to the baseline. 2. Level of effectiveness of the 0 80 58 73% Partially Achieved ASARECA Knowledge Hub 3. Number of stakeholders accessing 0 230,000 6,512 3% Not Achieved new markets 4. Number of centers/organizations facilitated to incubate TIMPs & agri- 0 2 2 100% Achieved. businesses Exceeded. The actual reforms achieved were 7. The target and actual 5. Number of policy options adopted 13 17 20 175% performance have been made / approved cumulative align the target to the baseline. 6. Number of policy reforms, regulations & procedures 0 5 7 140% Exceeded implemented 7. Number of stakeholders whose capacity building needs have been 64,896 156,834 56,895 -12% Not Achieved addressed 8. Number of stakeholders benefitting from the activities of the 0 11,000,000 2,750,000 25% Not Achieved hub in terms of access to information 13 Access to TIMPs n the ASARECA context, refers to the ease that people have in obtaining relevant products and services such as TIMPs. It includes how to get them, where to get them, how to use them. Page 16 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) End of End of PDO Performance Indicator Baseline Project Project Percent Comments (Outcome) (2013) Target Actual Achieved (2018) (2018) and knowledge, as well as networking and learning Note: a. Only one PDO indicator measures access to technology and is rated to the satisfaction of the dissemination approach. As such, 8 intermediate results indicators (IRI) which measure access to research and technology were used to supplement assessment of PDO 1. 35. In the first year of OP II, technology generation and dissemination remained one of ASARECA’s core areas of comparative advantage directly designed to address challenges of low productivity at farm level. Together with the NARS and other stakeholders, such as the CGIAR and private sector, ASARECA generated a total of 125 TIMPs under the Grant. ASARECA placed emphasis on scaling out and scaling-up agricultural TIMPs to ensure wider uptake of technologies both within and across ASARECA member countries. 36. A total of 6,414 hectares of land were brought under improved TIMPs. This included over 2,760 ha dedicated to the production and multiplication of quality pre-basic, basic and certified seeds of selected crops (especially wheat and maize) – see Table 3. Over 800 metric tons of quality seed were produced and either sold or distributed to farmers for further multiplication. In addition to multiplication of seeds, a total of 3,654 ha was brought under sustainable soil and water management in the targeted countries. Table 3: Hectares brought under improved TIMPs Country Land Area (Ha) Intervention Kenya 1,870 Sustainable soil and water management. Ethiopia 299 Technologies and management practices Eritrea 100 included bench terraces and deep trenches Uganda 665 in Ethiopia, supplementary irrigation in Madagascar 248 Uganda, and regeneration of degraded land Burundi 119 using live cover crop in Madagascar. Sudan 353 Rwanda & Burundi 2,760 Seed multiplication TOTAL 6,414 37. ASARECA established a Knowledge and Information Hub (KI hub) to support the exchange of knowledge, networking, development of new ideas, and sharing of the sub-regional benefits of ASARECA’s interventions. The KI-hub was a focal point for the generation and exchange of knowledge, for networking among scientists in the region and development of new ideas. At project closure, the level of stakeholder satisfaction with the knowledge and information hub 14 stood at 61 percent. 38. ASARECA continued supporting the creation of an enabling environment that facilitated trade, ensured free access to input and output markets, created market linkages, and supported the competitiveness of the 14 The level of stakeholder satisfaction with the knowledge and information hub was estimated by computing the frequency of access to ASARECA knowledge products via its website. This was further enhanced through surveys with key stakeholders to solicit their views on their levels of satisfaction with the hub. Page 17 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) agricultural sector. ASARECA carried on coordinating policy analysis related to inputs and outputs markets, promotion of viable approaches that improve access to markets, and advocacy for an enabling policy environment. 39. PDO outcome 2: To enhance utilization15 of agricultural research technologies and innovations in the regional agricultural systems of ECA. Table 4: OP II MDTF Key Performance Indicators for utilization of TIMPs: Targets and Actual Achieved End of PDO Performance Baseline End of Project Project Percent Comments Indicator (Outcome) (2013) Target (2018) Actual Achieved (2018) Achieved substantially: Increased productivity was attributed to 1. Percentage change in production generated from the trial / agricultural production 0 5.0 4.7 94% pilot plots on selected farmer fields, & productivity not of farmers field after dissemination. Not Achieved. The actual number of hectares is 6,414. The target and 2. Number of hectares 30,000 75,000 6,414 14% actual performance have been made under improved TIMPs cumulative align the target to the baseline. Intermediate Results Indicators (IRI) 1. Number of stakeholders who have 228,359 250,000 254,314 120% Achieved applied new TIMPs 2. Percentage of stakeholders who have 0 89 3.2 4% Not Achieved adopted selected TIMPs 3. Number of farm Not Achieved. The actual number households benefiting households benefiting is 55,112. The target from different 1,370,094 1,750,094 1,425,206 15% and actual performance have been made interventions (e.g. cumulative to align the target to the TIMPs) baseline. 4. Number of demand- driven and cost 0 60 57 95% Achieved substantially effective TIMPs available for uptake 5. Number of businesses that hatch from 0 3 10 333% Exceeded incubation centers Note: Two PDO level indicators measure utilization of technology, whereas five intermediate results indicators (IRI) measuring utilization of research and technology were used to supplement assessment of PDO 2. 40. Several TIMPs were made available for uptake in all the ASARECA member countries where the three projects were implemented. A total of 57 demand driven and cost-effective TIMPs were introduced and 25,955 stakeholders (16,871 male; 9,084 female) had adopted the availed TIMPs. This is an increase of 3.2 percent against the baseline (25,151 stakeholders) that were reported to have adopted selected TIMPs. 15 Utilization of TIMPS tries to measure what impact would be achieved after using the TIMPS. Page 18 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 41. ASARECA continued to play the role of coordination of R4D initiatives among member countries, as well as convener of regional R4D projects such as EAAPP. It had at its disposal more than 400 TIMPS generated during OP I available for up-take by farmers. One of its priority OP II activities was therefore to widely disseminate these TIMPS to farmers in member countries through the national agricultural extension systems and the NARIs. ASARECA also facilitated and forged coordination between scientists in different member countries. For specific regional projects like EAAPP, which closed in 2015, ASARECA provided technical and operational support in the areas of networking, convening, technical backstopping, M&E, policy harmonization and advocacy, capacity building, information and communication, and technology spill overs. Figure 1: Dissemination of improved cassava through ASARECA Source: ASARECA 2019 42. ASARECA also focused on strengthening agri-business enterprises and developing partnerships. A total of 10 businesses were identified for follow up, among which included: (i) Afribanana Products Ltd. which supports the banana value chain; (ii) the Consortium for enhanced University Responsiveness to Agribusiness Development (CURAD) which supports the coffee value chain; and (iii) the Sorghum Value Chain Development Consortium (SVDC) in Kenya. Commercialized TIMPs include: (i) improved banana tissue culture and yield-enhancing sorghum varieties by Jomo Kenyatta University of Agriculture and Technology Incubator. Through capacity development and the regional collaborative research program, 21 partnerships were fostered among several agricultural Page 19 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) research actors within the region under OP II. These partnerships have allowed for an exchange of knowledge and experiences across different institutions when dealing with similar challenges and augmented ASARECA’s ability to achieve some results. Through these partnerships, research outputs increased marketing of commodities, institutional mentorship of scientists and adoption of selected TIMPs, though to a limited scale. Partnerships were formed, predominantly among NARS actors across the region. Justification of Overall Efficacy Rating 43. Under OP I, the MDTF PDO was achieved satisfactorily with the project exceeding several of the outcome targets, and in some cases by a significant margin. An impact evaluation found that the MDTF-supported activities also helped to attain higher-level outcomes such as increases in real value of total agricultural output, income for project beneficiaries, and food and nutrition security for most households, and participation of women and people with disabilities. 44. On the other hand, achievements under OP II were limited and its efficacy Modest. The overall project efficacy is Substantial. C. EFFICIENCY Assessment of Efficiency 45. It is well-documented 16 that demand-driven and effectively coordinated agricultural research and extension programs contribute positively to increased farm productivity, farm income and consumer welfare. In most cases the estimated rates of return exceed 40 percent. A recent analytical literature review on rate of return on research also confirms that research and extension in agriculture yield consistently high rates of return. For example, rate of return to agriculture research for some African countries and some crops are very high, exceeding 50 percent in some cases17. The review further confirms the robust positive relationship between spending on research and development, and economic growth. It indicates that the rate of return on research is many times higher than the rate of return on other comparable investments. 46. An impact evaluation conducted to assess overall performance of ASARECA’s OP I program was based on the findings from a household survey. Using a Cost-Benefit approach, the impact evaluation found that the net difference in incomes between beneficiaries and non-beneficiaries was US $272. Extrapolated across all ASARECA beneficiaries adopting new TIMPs (270,000 households, i.e. 64 percent of stakeholder accessing new TIMPs), this represented a net increase in agricultural production value of US $73.44 million (annual change in beneficiaries compared to non-beneficiaries). Comparing the value of benefits to the program costs indicated that in about 1.25 years, the program would have covered operational costs. Thus, the benefit-cost ratio was positive by the second quarter of 2013. Since OP I was seamlessly extended into the early years of OP II, and given the challenges 16 A substantial number of economic impact studies evaluating the contributions of research and extension programs to increased farm productivity and farm incomes and to consumer welfare have been undertaken in recent years. In almost all categories of studies, median (social) estimated rates of return are high, (often exceeding 40 percent) but the range of estimates was also high. Handbook of Agricultural Economics, volume 1A. Amsterdam: Elsevier, 2001. 17 Maize for Burkina Faso and Ghana about 75%, rice in Senegal more than 66% and millet in Mali 66% Page 20 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) experienced under OP II, where a substantive part of the activities was suspended in the second year of the project, no impact evaluation has been conducted for OP II activities. Implementation Efficiency 47. At Design. OP II was implemented and managed through the Secretariat’s existing implementation, institutional, fiduciary, safeguards and monitoring arrangements, which were strengthened during OP I implementation. Despite its weaknesses, ASARECA had developed adequate management capacity and, procurement, financial management, safeguards and M&E systems. The selection of participating NARIs was done through ASARECA’s rigorous CGS. The CGS followed a stringent selection process, 18, thereby assigning scarce resources to the most technically sound and highest priority proposals; and provided incentives to implementing agencies to make efficient use of such resources and achieve the most outputs and outcomes as possible. 48. During Implementation. Following the investigation resulting from fraud and corruption allegations, weaknesses in the fiduciary systems were noted. To redress the deficiencies, the Board charted and implemented a roadmap for a two-year transition period. This was supported by a fiduciary action plan (FAP) developed by the Secretariat and endorsed by the Bank. As part of the FAP, ASARECA implemented a cost containment plan by rationalizing administrative and operational expenses, which led to the reduction of budget/cost centers from 22 to 11 and simplification of the cost control process. As a result, ASARECA saved approximately US $1.58 million, largely from the staff rightsizing exercises that were executed in four phases. In the medium- and long-term, there were plans to rationalize Board, General Assembly meetings and use of office space to further reduce costs. 49. While the efficiency rating for OP II could be Modest, on balance and taking into account the OP I ICR rating, the overall project efficiency rating is Substantial. D. JUSTIFICATION OF OVERALL OUTCOME RATING 50. To determine the overall outcome rating of the project, split rating methodology was applied for the OP I (2008 - 2013) and OP II (2014-2018) periods. 51. The MDTF was relevant at the time of appraisal and continued to be relevant to the end of the program. As presented in Section B above, under OP I the PDO was achieved satisfactorily with the project exceeding several of the outcome targets, and in some cases by a significant margin. The outcome rating was Satisfactory 19. 52. The PDO for OP II was partially achieved, yielding an outcome rating of Moderately Unsatisfactory. Table 5 summarizes ratings for OP II for relevance, efficacy, and efficiency. 18 Criteria for selection of sub-projects included among others: (i) legal presence of proposed beneficiary with the necessary managerial, financial, and technical qualifications and experience to implement the sub-project; (ii) relevance of research problem; (iii) research competence; (iv) economic, financial, environmental and technical soundness of proposed technical approach and contribution to the innovation knowledge system; (v) adequate availability of research facilities; (vi) anticipated impacts on the development objectives (PDO); (vii) institutional, economic, social, and environmental impacts expected by the diffusion of research results; (viii) probability of success; and (ix) diffusion potential and replicability of generated information. 19 ASARECA Implementation Completion Report, 2014 pg. 18 Page 21 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Table 5: Outcome Rating - OP II OP II (2014–2018) Relevance Substantial Efficacy Modest Efficiency Modest Outcome rating Moderately Unsatisfactory 53. Disbursement. The project disbursed a total of 93.3 percent of the grant proceeds by the December 31, 2018 closing date. During OP I (2008-2013), US $58.9 million (90 percent of disbursed funds) was disbursed, and US $6.4 million (10 percent of disbursed funds) was disbursed during OP II (2014-2018). Table 6 presents project disbursements for both phases. Table 6: Project Disbursement by OP Phase US$ million Percentage OP I (2008 - 2013) 58.9 90% OP II (2014 – 2018) 6.4 10% Total Grant Disbursements 65.3 100% 54. Overall outcome rating. Table 7 presents the overall outcome score of 4.8, which justifies an overall Satisfactory rating. While the overall weight value indicates that the overall outcome could have been rated Satisfactory, the inability to sustain the results achieved under OP I, combined with management and governance issues, suggest that a conservative rating of Moderately Satisfactory would be more appropriate. Table 7: Justification of Overall Project Outcome Rating OP I (2008–2013) OP II (2014–2018) Overall 1. Rating Satisfactory Moderately Unsatisfactory 2. Rating value 5 3 3. Total disbursed (US $, millions) 58.9 6.4 65.3 4. Share of disbursement (%) 90 10 100 5. Weight value 4.5 0.3 4.8 6. Final rating Moderately Satisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 55. Gender was well-mainstreamed across the ASARECA program portfolio in OP I and II. Ninety six percent of all research projects were categorized as ‘gender-responsive’ under OP I, i.e., incorporating gender-responsive agricultural innovation systems activities in their interventions. Several NARIs have also set up gender units within Page 22 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) their institutions. The impact evaluation shows clear benefits for women beneficiaries. During OP II implementation, ASARECA continued to incorporate gender mainstreaming as a major support function in ensuring efficient and effective delivery of its interventions. As a prerequisite to ensure continuity in implementation of gender-responsive governance and management processes, ASARECA encouraged its members to adopt its definition 20 of gender mainstreaming. The Secretariat provided technical support to Grant- funded and other regional bilateral projects 21, especially in making ASARECA’s gender project guide and related checklists widely available. ASARECA has also supported the mentoring of scientists from the NARIs on gender mainstreaming in research. All PDO indicators for OP I, were disaggregated by gender, OP II indicators were not disaggregated by gender. As such, a clear number of female beneficiaries reached was not established at the project level. Beneficiaries of short-term training 56. Short-term training, organized for individuals and groups, focused on increasing staff awareness and adherence to internal controls. Staff members were trained on ethics and anti-fraud practices and on best practices of accountability principles in procurement, travel management, and administrative processes. The training also covered overall requirements for World Bank financial management reporting. As a result, staff completed the revision of the Operations Manual, including updated content on procurement, human resources, finance and administration, and internal and external audit management polices among others. These policies were reviewed and approved by the relevant Bank technical teams and the ASARECA Board of Directors. A zero- tolerance policy to fraud and corruption was also instituted, together with a requirement for all staff and partners to make an annual written commitment. Further, the ASARECA technical team was trained on the PMERL (Planning, Monitoring, Evaluation, Reporting and Learning) System that was developed during the transition period, with the overall aim to establish an interactive and interoperable M&E Automated System that is web- enabled, cost-effective and user-friendly for effective information management. These investments have improved the standard of compliance to internal controls, and the overall fiduciary discipline and reporting, which helped in rebuilding the image of ASARECA. Institutional Strengthening 57. Following the IFR, ASARECA committed to implement major internal reforms of its governance, management and administration (Components 2 and 3), to strengthen its institutional capacity and reposition itself to cope with the changing AR4D environment. For this purpose, a supplementary results framework that assesses performance during the transition period, (2017 -2018) was developed and is included in Annex 4. Reform activities implemented to support institutional strengthening included: • Preparation and implementation of a two-year transition plan aimed at restoring the credibility and visibility of ASARECA following the fraud and corruption allegation; • Reimbursement of the funds declared as ineligible expenditure to the Bank; • Full execution of the FAP to address key issues flagged during the IFR; • Strengthening of the governance and management structures; and 20 Process of assessing the implications for women, men and youths of any planned action, including policies or programs in all areas and at all levels. 21 GREAT project funded by the Bill & Melinda Gates Foundation. Page 23 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) • Enhancing controls on operations for transparency, accountability and ethical standards; and review of the Operations Manual. 58. Major accomplishments of the reform process include: • Governance reforms that rationalized higher-level bodies that oversee the Secretariat (Patron ministers, General Assembly, Board) and revisions to the Constitution, the Governance Manual and the Operations Manual. • An institutional and functional analysis resulted in recommendations for a new organizational structure, staffing scenarios, as well as competitive salaries and benefits reflective of the funding situation, approved by the Board’s Executive Committee. • Recruitment of new and highly qualified staff, including the Executive Secretary (ES), Head of Finance, Accountant, Internal Auditor and Procurement Consultant. KPMG was contracted in June 2018 to carry out external audits for four years, starting with the 2017 audit. • Rebranding campaign to revive its corporate image and expand visibility among stakeholders, to mobilize financial resources, and expand partnerships. An outcome of the rebranding exercise was the new ASARECA Strategy and Results Framework (A-SRF; 2019 – 2028), including a new vision, mandate and value proposition, as well as a medium-term operational plan (2019-2023). Mobilizing Private Sector Financing 59. OP II had a strong emphasis on agribusiness, business start-ups, market and trade aspects. ASARECA’s envisioned multi-faceted mechanisms to strengthen private sector partnerships, both on the supply and demand- side spectrum of the AR4D initiatives, include: (i) aligning research projects with flagship private sector projects identified in ECA countries’ long-term development objectives; (ii) identifying and developing viable commercialization models for products that are not true public goods and promote initiatives such as award schemes that recognize innovations and technology use; (iii) developing partnerships with those already supporting/working with the private sector, (AGRA, East African Business Council); and (iv) increasing support to technology incubation centers. A total of 10 businesses were hatched from business incubation centers supported by the Grant by the end of the project. The project worked with the private sector and other stakeholders (such as NARIs, universities, NGOs and farmer organizations) to generate technologies, and identified and packaged successful and scalable TIMPs. Poverty Reduction and Shared Prosperity 60. While poverty reduction was not an objective, the ASARECA Grant has had an impact on incomes among participating households. Empirical assessment from various initiatives indicated that adoption of technologies increased incomes. Other Unintended Outcomes and Impacts 61. Leveraging new sources of support. ASARECA has implemented some of the activities jointly with other regional bodies, and within the CAADP agenda. This has provided opportunities for scaling up successes and leveraging new sources of support from donors within and beyond the ASARECA region. During OP II, ASARECA Page 24 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) supported 17 projects - three were supported by the Grant, while the other fourteen were bilaterally-financed. In addition, ASARECA coordinated the implementation of the Eastern Africa Agricultural Productivity Programme (EAAPP) in Ethiopia, Kenya, Tanzania and Uganda, which was supported with a loan from the World Bank. EAAPP provided support on four priority value chains namely: dairy, wheat, rice and cassava. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 62. The grant itself was a positive factor because it was an appropriate tool to help ASARECA expand its activities in the sub-region and add value to the work of the member country NARS. It was prepared in a participatory manner in consultation with member countries, donors and ASARECA, and was open to multiple donor contributions; thus, offering a coordinated approach to investments in IAR4D in the sub-region. It also facilitated the harmonization of procedures and formats to be applied by ASARECA, using its own procedures to meet the requirements of all the donors. The approach, design and structure of ASARECA’s OP II, the basis for the proposed AF, incorporated lessons learned from OP I, including findings from the comprehensive evaluation study measuring and assessing the overall impact of OP I implementation. 63. On the negative side, the risk and mitigation measures identified were inadequate. The driving force behind the creation of the MDTF was the food crisis in 2008. Once the crisis was over, donor focus on this deep-rooted regional challenge had faded. This should have been a signal to the task team to be more conservative regarding the project cost for which the OP II AF was designed. Therefore, the overall risks for both preparation and implementation of the AF activities should have been high. Within the first year of implementation ASARECA failed in its resource mobilization. Subsequently, the Board decided on rightsizing the organization and rationalizing the use of financial resources, which led to reduced investment in sub-projects and programs. The disproportionate overhead cost relative to project/program investment was also reported during OP I implementation. Moreover, low fund utilization due to non-compliant FM practices, delays in fund disbursement to sub-grant (SG) implementing centers and delays in procurement of goods and services by SGs were identified and reported as early as 2009, during OP I implementation. This implies that the risk rating for OP II should have been Substantial, rather than Low, and appropriate mitigation measures should have been proposed at AF appraisal and OP II. 64. Poor Readiness for implementation. OP II design was unrealistic given the fact there was no donor ready to support the project as designed, except the limited funds committed by the European Commission. This was a clear signal not to go ahead with OP II implementation as designed. The optimism that funds would be mobilized in the course of implementation was unfounded and contributed to the challenges of OP II implementation. B. KEY FACTORS DURING IMPLEMENTATION Factors Outside ASARECA Control 65. Limited Financing. Despite simultaneous and intense resource mobilization efforts, such as preparation of numerous project proposals and concept notes for donor support, there was poor response from both traditional and non-traditional sources of funding. Page 25 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Factors Subject to ASARECA’s Control 66. Fiduciary controls and accountability. Based on the findings of the in-depth forensic audit, the Bank determined that ASARECA did not carry out procurement in compliance with the Grant Agreement, as well as annual agreed procurement plans. Furthermore, it was determined that SGs had been used in a manner inconsistent with the provisions of Section 3.01 of the Grant Agreement. This resulted in ASARECA refunding ineligible expenses and mis-procurement totaling US $2.2 million. 67. Delays in completing governance reform. There were delays in implementing high priority activities identified for the transition period. These delays affected early reform of the role of entities that oversee the operations of ASARECA (Patron Ministers, General Assembly, Board, Executive Secretary), and revisions of the Constitution, the Governance Manual and the Operations Manual. Protracted delays in recruitment of the governance consultant, multiple contract extensions and a final output that was non-responsive to the TOR all resulted in a Board decision to terminate the consultancy, and instead seek a new arrangement to complete the work 22. As a result, the timely repositioning of a reformed ASARECA was delayed. 68. Dearth of technical capacity at ASARECA: ASARECA’s capacity was significantly diminished following four rounds of downsizing between 2014 - 2017, representing a 77% reduction in staff size (from 64 in 2014 to 14 in 2017). This included the non-renewal of contracts for all regional staff in December 2016. A plan to recruit new staff was approved; however, by June 2018 only 37.5% of the required positions had been filled due to lack of funds. By December 2018 the staff complement at the Secretariat was a total of 16 individuals - 3 regional staff and 13 national staff including the recent hires (Executive Secretary, Head of Finance, Accountant and Internal Auditor). 69. Commitment and Leadership: The ASARECA Board fully committed to addressing fiduciary, governance and administrative weaknesses by implementing the agreed FAP. Executive Committee and Board meetings were held regularly to provide guidance, especially on staff matters such as rightsizing, approval of external audit reports, approval of annual work plans and budgets; review of program implementation plans and budgets. The Secretariat facilitated the Executive Committee and Board meetings. The Board of Directors played a proactive role at a time when the ASARECA Secretariat required guidance and strong support. Factors Subject to World Bank Control 70. Supervision Gap. Bank supervision of the project was limited during the IFR, which lasted for about 27 months. During this period, the ASARECA Board approved the Secretariat’s 2016 and 2017 work plans and the Bank approved limited grant funding for salaries and operating cost of the Secretariat. 71. Continued engagement and flexibility. The Bank Team remained proactive and responsive to the changing context in ASARECA. Several INT investigation and fiduciary assessment missions were fielded, communications 22Through its Memorandum of Understanding with its sister institution, the Forum for Agricultural Research in Africa (FARA), FARA agreed to provide technical support to facilitate the completion process for the review and strengthening of ASARECA’s governance structures. To deliver on the assignment, FARA constituted a team of resource persons consisting of an overall Coordinator and expert in stakeholder engagement and governance processes; a legal expert; an expert on board practices; and another expert in assessment of competencies and performance. Page 26 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) with development partners was regular, informing them of the status of the investigation and review. Post IFR, the Bank supported the implementation of the FAP and transitional work plan and budget to ensure that ASARECA was transformed and rebranded as stipulated in the transition plan. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) 72. ASARECA used M&E at different levels (project, program and ASARECA), and provided regular information for monitoring implementation. Through its management information and financial management systems, activities were tracked, and potential problems addressed. The system also served as a useful management tool, for example, to track CGS disbursements, to design relevant training materials and to monitor actual expenditure against disbursement levels. The M&E system was well established under OP I, and OP II benefited from this set up. Overall, ASARECA has successfully set up a functioning M&E system and created a culture of monitoring its activities, projects and programs, as well as those run by its partners. The system is online for use by national M&E teams. M&E Design 73. At the beginning of OP II, ASARECA’s M&E arrangements were Satisfactory. The results framework (RF) indicators and targets were updated to ensure consistency with actual outputs and outcomes on the ground attributable to the additional funds. The indicators and targets also reflected the expanded scope of the AF activities. ASARECA had a robust system with an M&E Unit which put particular emphasis on tools and methods to facilitate implementation of the ASARECA M&E system at different operational levels (projects, programs, and regional), which allowed ASARECA to report on actual progress for the results indicators defined in the revised RF. Significant progress was made in developing M&E tools and in promoting results-based management. M&E Implementation 74. The Bank Team found the RF developed for OP II (2014-2018) inadequate given the reduced donor appetite and subsequent funding shortage and started to consider restructuring. However, this was pre-empted by the whistle blower’s allegations, ensuing IFR, and suspension of supervision activities. With the Grant mainly used to fund staff salaries and core operating expenses, and given the large uncertainty surrounding the outcome of the IFR, restructuring of the grant ceased to be a priority. In response to the FAP, an agreement was reached to develop a supplementary RF to monitor implementation of the transitional work plan and budget approved for 2017-2018 (see Annex 6). Although ASARECA fastidiously tracked progress against the supplementary RF, there was a missed opportunity by the Bank Team to formally integrate the supplementary RF into the original OP II RF. 75. Various supervision missions found OP II implementation progress and achievement of agreed output indicator targets, to be lagging. Nevertheless, some progress in putting in place a performance-based M&E system was noted. This included the final migration from the use of Microsoft Office spread sheets and documents to a more robust system to aid planning, designing, monitoring, budgeting, reporting, learning, and evaluation. Page 27 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Moreover, the M&E Unit updated OP II targets in line with the reduced funding, however, these changes were not formally approved due to the IFR, suspension of the SG program and transitional work plan effected thereafter. M&E Utilization 76. Despite very limited activity in the first 1.5 years of the OP II phase, ASARECA reported on implementation progress at regular intervals (biannually), as agreed in the Grant Agreement. Reporting came to a halt once the IFR was underway, and the SG program was suspended. 77. Aide Memoires and ISRs prepared on a regular basis helped stakeholders to stay informed of project progress. Except for a 20-month hiatus in supervision, the Bank supported adjustments to the approach, work plan and budgets and preparation of a transitional RF to help ASARECA remain relevant. The transitional RF was updated regularly to monitor progress in the implementation of reforms and the FAP and adjusted as needed to achieve required results. 78. There was adequate candor in reporting implementation issues in ISRs regarding implementation challenges. This helped in providing timely feedback and generating appropriate actions for implementation. Justification of Overall Rating of Quality of M&E 79. The initial design was appropriate, and the needed adjustments were made during project implementation based on feedback on existing circumstances. However, when the SG program was suspended, the entire RF became immeasurable. A supplementary RF was developed in response to the in-depth fiduciary review to allow the institution to improve its performance going forward. Although the supplementary RF was not formally submitted for review and approval by Bank management, it was highly beneficial to ASARECA, providing clear milestones and targets for achievement of ASARECA’s institutional and governance reform agenda. Accordingly, ASARECA’s M&E performance has been adequate, thus, the rating is Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 80. Environmental Safeguards. ASARECA’s Environmental Safeguards capacity was enhanced in 2013 when the project Secretariat hired an Environmental Specialist, who was discharged in 2014 given the restructuring exercise. In December 2014, ASARECA designated one of the Natural Resources Program Assistants to undertake the safeguards function. The Environmental Specialist was re-hired on an ad hoc basis in 2015 to complete preparation of the training manual for stakeholders; and facilitate the approval of the Environmental Policy by the Board of Directors in January 2015. The policy was developed in collaboration with the Environmental Focal Persons (EFP) nominated from Environmental Agencies of the 11 countries and the Environmental Liaison Officers (ELO) nominated from Agricultural Research Institutions. 81. Between 2013-2015, environmental safeguards requirements were implemented and reported by the Secretariat which included environmental screening of sub-projects, development of environmental management plans where necessary, undertaking monitoring and supervision, technical backstopping, and training of the implementing agencies. ASARECA undertook capacity building of the implementing partners, to improve the implementation of environmental and social aspects of the sub-projects. The civil works for construction of Page 28 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ASARECA’s Headquarter Offices in Kampala were subjected to environmental screening but were not implemented, due to the funding shortage and IFR. 82. The project developed and implemented Pest Management Plans where necessary, and some cases of non- conformity to the requirements of the PMPs were observed especially in handling and disposal of pesticides and associated waste by implementing agencies. Corrective measures were instituted. 83. Given the above, the overall environmental safeguards performance of the project was Moderately Satisfactory. 84. Social Safeguards. The project did not trigger any of the social safeguards specific policies during preparation and during the implementation of OP II, no social issues were identified or reported. During the initial phase of OP II, ASARECA approved an Environmental and Social Policy and mechanisms for screening environmental and social issues related to sub-grants (SGs) under preparation. During the IFR, funding of the SGs were suspended (and ultimately cancelled), therefore supervision of social safeguards was not undertaken. 85. Financial Management. Disbursement at Grant closure was US $65,374,615 representing 93.3% of the signed amount of US$ 70,100,895 with US$4,726,280 having been cancelled. 86. The project started well, but experienced fiduciary challenges in the later years of the project which resulted in ineligible expenditures, suspension of project activities, staff separation and restructuring of ASARECA including the finance department. Further, the FAP was agreed between the Bank and ASARECA to strengthen the financial management arrangements to meet the accountability expectations and general good corporate governance. A review of implementation status during the last mission noted good progress in implementation of key FM actions including: (a) Refund of all ineligible expenditures completed. (b) All key staff (Head of Finance, Accountant and Head of Internal Audit) recruited. (c) Updated Operational Manual to address key financial management weaknesses noted in the in-depth review, spearheaded by the Head of Finance and Internal Auditor. (d) The process of upgrading the accounting system had been initiated by the Head of Accounts. 87. The project was compliant in the submission of acceptable and timely semi-annual interim reports, together with audited financial statements during the full implementation period. There were no outstanding reports by project closure. The closing ISR FM rating was assessed as Moderately Satisfactory. 88. Procurement. Project implementation was affected by the IFR arising from the allegations of fraud and corruption. Procurement contracts totaling US $242,663 were declared non-compliant with Bank Procurement Guidelines - most of the mis-procurement occurred prior to OP II. Procurement activities under OP II were limited due to shortage of funds. Nevertheless, the IFR recommended actions aimed at strengthening procurement systems, mostly through: (a) the hiring of a procurement consultant; (b) operationalization of the Procurement Committee as per the Procurement Manual; (c) strengthening of the internal audit capacity to conduct risk-based audits of the procurement function; and (d) sensitization of Secretariat staff on the procurement arrangements in ASARECA’s Procurement and Operations Manual. Page 29 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) C. BANK PERFORMANCE Quality at Entry 89. By the end of OP I, ASARECA appeared to be a well-established institution with competent staff and adequate administrative, financial and technical systems in place, among others. This made it easier to identify and address gaps that required attention to enhance its overall performance. Moreover, lessons from OP I were carefully reviewed, and potential risks identified before entry, with the exception of future funding prospects. Before entry, the effort was also made to ensure the PDO and outcome indicators were clearly linked to activities. 90. Despite these measures taken at entry, ASARECA experienced weaknesses in management, which were compounded by a shortage of funding during OP II. Although the initial OP I grant amount started at US $24 million with three donors and grew to US $59.2 million with four donors by the end of OP I, the Bank Team and ASARECA should have recognized that the development partner landscape and priorities had evolved from the food and economic crises that threatened Sub-Saharan Africa in 2008, when the original grant was being prepared. A weakness at entry was the commencement of OP II implementation prior to confirming adequacy of funding for the life of the project. Based on these findings, the rating for quality at entry is Moderately Unsatisfactory. Quality of Supervision 91. Except for the IFR period, implementation support missions were conducted on a regular basis. The missions included technical specialists, financial management and procurement specialists, and other professionals, as needed. The missions worked together with the ASARECA Secretariat, and at times with development partners. Aide Memoires and Implementation Status Reports (ISRs) were issued, reporting progress and identifying issues that required attention and action. 92. Within the first year of implementation of OP II, the Bank Team recognized that the funding shortage needed to be managed, and appropriately advised ASARECA to rationalize its staff complement against a reduced work program and strengthen its resource mobilization efforts. While the Bank Team considered restructuring at this time, it could not foresee that a whistleblower would come forward in the first year, that the IFR would last 27 months, and that ASARECA would lose its full complement of senior technical staff within that timeframe. All of these factors created a perfect storm characterized throughout by uncertainty, particularly with regard to the IFR outcome. 93. Upon re-engagement in March 2017, the Bank Team was proactive in its decision to suspend the SG program and instead focus on an institutional and governance reform program for the remainder of the Grant implementation period. The Bank Team worked with ASARECA to develop the FAP, transitional work plan and a supplementary RF to measure progress against achievement of the FAP. In hindsight, the Bank Team could have restructured the grant to formally add the supplementary RF to the Bank approved RF. In the final ISR, the Bank Team acknowledged that the PDO and implementation progress ratings were mistakenly upgraded in the penultimate ISR to reflect progress made against the transitional work plan and FAP. 94. The Bank Team also proactively engaged with MDTF donors upon conclusion of the IFR and with OP II donors throughout the re-engagement period. In particular, several visits were made to the Delegation of the European Page 30 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Union in Kampala to understand the requirements for unlocking the remaining 50% of the European Commission’s contribution which was withheld indefinitely following the start of the IFR. Justification of Overall Rating of Bank Performance 95. On balance, Bank performance is rated Moderately Satisfactory. The Project’s quality at entry and supervision had strengths and weaknesses, but the Bank accommodated the multiple challenges and showed flexibility and resilience in helping ASARECA address its challenges within a complex environment. Despite the difficulties, the Bank Team carried out continuous dialogue with the Board, Secretariat and key stakeholders, and through the proposed reforms, ensured that ASARECA was better positioned to take up its mandate within the sub-region post-Grant. D. RISK TO DEVELOPMENT OUTCOME 96. Critical risks facing the ASARECA Secretariat are associated with its financial integrity and compliance. These include: (i) low and volatile funding levels; (ii) lack of funds to support priority projects; (iii) loss of institutional credibility; and (iv) continued restructuring of the Secretariat. Maintaining outcomes realized under the project will depend on the capacity of NARIs. Continued capacity strengthening of national and regional institutions will be critical to addressing agricultural challenges that are relevant for the wider region and go beyond national interests. 97. These risks to development outcomes have been significantly addressed through the enhancement of ASARECA’s governance and institutional capacity to re-establish its relevance and credibility. By the time the project closed: • Stakeholder satisfaction with governance and management reforms undertaken had reached 70 percent and 72 percent, respectively; • Operationalization of ASARECA’s updated governance and organizational structures was advanced. The Board’s performance, in terms of executing its mandate, Board functions, compliance with Board composition, training meeting procedures, etc. had reached 72 percent; • A new design for ASARECA as a regional organization was under development. In the new model ASARECA would rely less on its own staffing and implement its mandate more through its member organizations, thereby becoming a true network organization. The new model is cheaper to operate, enhances sub-regional ownership of ASARECA, and reflects the strongly improved capacities of several national research organizations. • Strategic staffing, financial management and knowledge systems had been significantly strengthened. The proportion of relevant infrastructure put in place to enhance the functionality of the Secretariat and ASARECA’s level of compliance with its Operations Manual were both above 75 percent; and • More significantly, by end May 2019, the ownership of ASARECA had been assumed by member countries together with regional economic communities 23 (RECs) and the African Union, through its 23 Namely the Common Market for Eastern and Southern Africa (COMESA), East African Community (EAC), Intergovernmental Authority on Development (IGAD), and Southern African Development Community (SADC). Page 31 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Department for Rural Economy and Agriculture, to ensure ASARECA’s regional ownership and sustainability. A new and leaner Board of Directors had been constituted to ensure membership with the requisite expertise and skills. Some countries have cleared their membership due arrears, whereas others have committed to paying them. These changes were decreed by the Council of Patron Ministers in May 2019. 98. The risk to the development outcome is Substantial. V. LESSONS AND RECOMMENDATIONS 99. This project presented unique challenges for the World Bank team, but it also produced many valuable lessons for future support of regional research institutions. Project Design 100. The World Bank’s quality assurance process is critical for trust funded activities. The Bank mainstreamed trust funds (TF) into its portfolio, policy, and procedures after the grant became effective. As a result, the Grant Program Document 24 did not fully benefit from internal Bank peer review processes at the time. Such a review would have highlighted the substantial omissions at entry, namely institutional sustainability and the resource availability issue. 101. Design for the resources available, not the resources desired. The team’s underestimation of the risk associated with lack of financial commitments led to an early crisis during OP II implementation. Planning for US $30 million but receiving only US $10 million meant that ASARECA was forced to significantly streamline its Secretariat and work program. 102. More rigor of the Bank was needed on M&E issues during supervision. The informal revisions that took place after the IFR to put in place the transitional work plan and supplementary RF happened without formal approval. While learning and adjustment is desirable, such significant changes to the Secretariat’s activities should have been subjected to a formal approval process sooner rather than later. 103. Regional projects often lack the tight fiduciary control of country projects. When the World Bank lends or grants to client countries, there is often a system of checks that includes national audit authorities, finance and technical ministries, and civil society that monitor the project and flag emerging problems. This fiduciary infrastructure has no equivalent at the regional level. To accommodate for that gap, the Bank should have hired a dedicated FM and procurement specialist to support Pillar IV grants and work more closely with ASARECA on a regular basis to ensure that appropriate capacity was being built and systems were functioning in accordance with World Bank standards. The same argument can be made for recipient-executed TFs in countries when they are not mainstreamed in the donor investment portfolio monitored by the Ministry of Finance. This reduces the risk that fiduciary misconduct is not observed. 25 Significant change in the reforms of ASARECA governance structures is noted, with the effectiveness of the Governance Structures and Systems rated at 70%. Among the key milestones include: Page 32 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Sustainable Financing 104. The risk of donor funding. If individual organizations like ASARECA continue to rely on a small number of donors whose priorities may change quickly, their future remains highly volatile. Some consolidation is needed to pursue more sustainable funding and to develop a long-term case for support. The Pillar IV team has pursued two strategies in response. First, it has reconvened the different continental (FARA, AFAAS) and sub-regional organizations (CORAF, ASARECA, and CCARDESA) and encouraged them to develop a continent-wide consortium that might have more clout, and initial funding for the consortium has now been obtained. Second, the team has pursued more country ownership by emphasizing member contributions and improved governance, and by engaging these regional organizations in Bank-financed research projects, where countries may decide to pay them in return for coordination functions. 105. A conducive and consistent operating environment is an important factor for successful project implementation. While ASARECA had attracted funding from various donors, most of these funds were project- based with the grant financing the bulk of the Secretariat operating costs. ASARECA was also late in its efforts to mobilize new funding, whether from development partners, member governments, sub-regional organizations, or from the private sector. Resource mobilization has mostly focused on traditional development partners. A new funding strategy must go hand in hand with evidence of relevance, sustainability and effectiveness. Institutional Development 106. It is important to balance opportunities for new activities with the development of and support to core institutional functions. Upon re-engagement, the Bank team’s decision to embark on an institutional and governance reform agenda, rather than continue with support of a limited set of activities of narrow scope, was both proactive and forward looking. The Bank Team convinced the ASARECA Board to refund ineligible expenditures to the grant from its reserve fund, in return for the Bank’s support and guidance on the development and implementation of a fiduciary action plan. Had the Bank not proposed this option, ASARECA might have muddled on for a few years, but would most probably have closed. 107. A crisis can bring about positive change. In retrospect, the whistleblower’s allegations and subsequent IFR, while painful at the time, were a blessing in disguise. It forced ASARECA to take a deeper look at its management and governance structures. The dissolution and reconstitution of the ASARECA Board and rethinking of its higher- level governing bodies to ensure greater ownership and accountability have now positioned ASARECA to contribute to a better and stronger system of regional and continental responsibilities, and these improvements in the regional structure and system will have long-term benefits. 108. Promoting client ownership over donor dependency should be a development priority. The events that ASARECA faced during OP II have helped to shift ownership and responsibility from donors to member countries. Africa’s landscape of regional collaboration still has many organizations that were established with significant donor support, often with overlapping mandates, and that are unsustainable without donor funding. The developments during the AF provide a window on how this system of regional collaboration may be rationalized (lower budgets), may become more country-owned and -driven, and ultimately become more sustainable. 109. Designing for institutional sustainability should include designing for lean operating scenarios. At the design stage for OP I, significant resources flowed to ASARECA, and it was generally thought this would continue. Page 33 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) In retrospect, ASARECA’s size and staff complement were overestimated, and it became a somewhat isolated, large organization with high salaries and expenses, requiring significant resources to stay functional. ASARECA became a victim of fluctuating donor priorities and finances, with both development partners and the constituency questioning its credibility. ASARECA started downsizing prior to the AF design stage and served as a leaner Secretariat. To address this challenge, the Bank could have explored the following options: • A programmatic MDTF that would have pooled all Pillar IV resources, thereby allowing the Bank to control the amounts allocated to the various institutions supported under the program, and allowing for deeper supervision around critical fiduciary issues. • Fewer, but larger sub-grants would have allowed for greater impact on the ground. This could have addressed the challenges related to the scale and complexity of the program. • Conditioning grant funding on larger contributions from member countries for sub-grants as well as the upkeep of ASARECA itself. 110. ASARECA will have to evaluate its mode of operation and its role, focusing less on the financing of regional projects and more on facilitation, coordination, and networking. It is moving in this direction. Nevertheless, it will be hard for any resource-strapped African institution to refuse large sums of money that might materialize after another food crisis. Client engagement and crisis management 111. There are limits to Bank systems and procedures. In the case of ASARECA, the storm that arose after the whistleblowers’ allegations was so strong and so unpredictable that it was hard to keep up in the Bank systems. Issues were deeper than expected, donors were hard to read, and the team was not even sure if ASARECA should be saved. In the end, dealing with the situation took all the oxygen and pursuing a restructuring for the limited time left seemed to make little difference. 112. Stay engaged. It is often the reflex of the World Bank and other large organizations, to bring crises to a close quickly. This may be a quick way to limit the reputational risk to the organization; however, by staying engaged while governance problems were exposed and working with the client to design solutions, the Bank Team illustrated the benefit of not turning away too quickly. If ASARECA had been allowed to fail, a replacement institution would need to be created. . Page 34 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To enhance access and utilization of agricultural research technologies and innovations Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent change in agricultural Percentage 0.00 5.00 4.70 production & productivity (tons of crop produced/ha) 31-Dec-2013 31-Dec-2018 31-Dec-2017 Number of hectares under Number 30000.00 300000.00 6414.00 improved TIMPs Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Beneficiary satisfaction with Percentage 71.00 75.00 61.00 the quality of ASARECA Knowledge & information 31-Dec-2013 31-Dec-2018 31-Dec-2017 Page 35 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Hub, products and services Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Research for Development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of demand-driven Number 409.00 125.00 125.00 gender-responsive technologies/ innovations 31-Dec-2013 31-Dec-2018 31-Dec-2017 generated Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Level of effectiveness of the Percentage 0.00 70.00 58.00 ASARECA Knowledge Hub 31-Dec-2013 31-Dec-2018 31-Dec-2017 Comments (achievements against targets): Page 36 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Level of compliance with Percentage 80.00 90.00 75.00 environmental and social safeguards 31-Dec-2013 31-Dec-2018 31-Dec-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of stakeholders who Number 228359.00 1100000.00 25955.00 have applied new TIMPS 31-Dec-2013 31-Dec-2018 31-Dec-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of policy reforms, Number 0.00 15.00 7.00 regulations and administrative procedures 31-Dec-2013 31-Dec-2018 31-Dec-2017 implemented Page 37 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of centres facilitated Number 0.00 7.00 2.00 to incubate agribusinesses 31-Dec-2013 31-Dec-2018 31-Dec-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of farm households Number 1370094.00 1800000.00 55112.00 (disaggregated by gender) benefiting from different 31-Dec-2013 31-Dec-2018 31-Dec-2017 interventions (e.g. TIMPs) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of stakeholders Percentage 0.00 25.00 3.20 Page 38 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) who have adopted selected 31-Dec-2013 31-Dec-2018 31-Dec-2017 TIMPs Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of businesses that Number 0.00 10.00 10.00 hatch from incubation centers 31-Dec-2013 31-Dec-2018 31-Dec-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of reform of Number 13.00 15.00 7.00 policies, laws, regulations and procedures approved 31-Dec-2013 31-Dec-2018 31-Dec-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 39 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Number of stakeholders Number 0.00 1100000.00 6512.00 accessing new markets 31-Dec-2013 31-Dec-2018 14-Jun-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of stakeholders Number 64896.00 75000.00 56895.00 whose capacity building needs have been addressed 31-Dec-2013 31-Dec-2018 14-Jun-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of demand driven Number 0.00 200.00 57.00 and cost-effective TIMPs resulting in improved diets 31-Dec-2013 31-Dec-2018 14-Jun-2017 and nutritional outcomes (at least 75% of generated & upscaled TIMPs) Comments (achievements against targets): Page 40 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of stakeholders Number 5.00 30000000.00 2750000.00 benefiting from the activities of the KI hub in terms of 31-Dec-2013 31-Dec-2018 14-Jun-2017 access to information and knowledge as well as networking and learning Comments (achievements against targets): Page 41 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) B. KEY OUTPUTS BY COMPONENT Objective/Outcome #1: Enhance access to agricultural research technologies and innovations in the regional agricultural systems of Eastern and Central Africa 1. Beneficiary satisfaction with quality of ASARECA KI-hub, products Outcome Indicators & services 1. Number of demand-driven gender responsive TIMPs generated 2. Level of effectiveness of the ASARECA Knowledge Hub 3. Number of stakeholders accessing new markets 4. Number of centers/organizations facilitated to incubate TIMPs & agri-businesses 5. Number of policy options adopted / approved Intermediate Results Indicators 6. Number of policy reforms, regulations & procedures implemented 7. Number of stakeholders whose capacity building needs have been addressed 8. Number of stakeholders benefitting from the activities of the hub in terms of access to information and knowledge, as well as networking and learning Component 1: Support to Research for Development 1. Appropriate technologies, innovations and management practices (TIMPs) generated/adapted 2. Appropriate TIMPs disseminated and upscaled Key Outputs by Component 3. Appropriate policy analysis and advocacy enhanced (linked to the achievement of the Objective/Outcome 1) Component 2: Support to ASARECA’s Knowledge and Information Hub 1. Stakeholder capacity for innovation strengthened 2. Communication, information and knowledge management improved Page 42 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Component 3: Support to ASARECA's Governance Structures Objective/Outcome #2: Enhance utilization of agricultural research technologies and innovations in the regional agricultural systems of Eastern and Central Africa 1. Percentage change in agricultural production & productivity Outcome Indicators 2. Number of hectares under improved TIMPs 1. Number of stakeholders who have applied new TIMPs 2. Percentage of stakeholders who have adopted selected TIMPs 3. Number of farm households benefiting from different interventions (e.g. TIMPs) Intermediate Results Indicators 4. Number of demand-driven and cost effective TIMPs available for uptake 5. Number of businesses that hatch from incubation centers 6. Number of businesses that hatch from incubation centers Component 1: Support to Research for Development 1. Appropriate technologies, innovations and management practices (TIMPs) generated/adapted 2. Appropriate TIMPs disseminated and upscaled Key Outputs by Component 3. Appropriate policy analysis and advocacy enhanced (linked to the achievement of the Objective/Outcome 2) Component 2: Support to ASARECA’s Knowledge and Information Hub 1. Stakeholder capacity for innovation strengthened 2. Communication, information and knowledge management improved Page 43 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation David J. Nielson Task Team Leader(s) Helmut Albert Team Member Bremala Nathan Operations Officer Matthew McMahon Team Member Jacob Kampen Team Member Nicolette K. Dewitt Senior Counsel Paul Jonathan Martin Environmental Specialist Luis M. Schwarz Finance Officer Tesfaalem Gebreiyesus, Howard Bariira Centenary Lead/Senior Procurement Specialist Paul Kato Kamuchwezi Financial Management Specialist Alexandra Sperling Team Member Supervision/ICR Bremala Nathan, Assaye Legesse Task Team Leader(s) Annet Tamale Katuramu, Grace Nakuya Musoke Procurement Specialist(s) Munanura Edwin Nyamasege Moguche Financial Management Specialist Irene Bomani Team Member Tekola Dejene Team Member Alexandra C. Sperling Team Member Antoinette Kamanzi Team Member Abdelaziz Lagnaoui, Herbert Oule Environmental Specialist Srilatha Shankar Team Member Marjorie Mpundu Counsel Page 44 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Christiaan Johannes Nieuwoudt Finance Officer Sandra M Kuwaza Team Member Catherine Asekenye Barasa, Callie Phillips Social Specialist Stella Chepkorir Team Member Sarah Anne Simons Team Member B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY09 1.675 9,564.92 Total 1.68 9,564.92 Supervision/ICR FY09 39.374 224,808.73 FY10 16.493 146,018.02 FY11 28.190 182,692.38 FY12 52.631 301,436.36 FY13 39.789 134,272.96 FY14 53.705 257,415.26 FY15 33.422 175,623.37 FY16 31.162 131,292.68 FY17 27.927 178,795.59 FY18 28.603 270,732.01 FY19 22.813 248,203.14 Total 374.11 2,251,290.50 Page 45 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Component (US$M) Closing (US$M) Approval (US$M) Research for Development 0 48.55 0 Support to ASARECA’s Knowledge 0 and Information Hub ASARECA's Governance Structures 0 1.56 0 ASARECA's Management & 0 9.08 0 Administration Structures Total 0.00 59.19 0.00 The original grant amount to ASARECA was US $24.0 million and the Grant Agreement was amended five times (twice requiring additional financing) to arrive at the final grant amount of $70.1 million. To facilitate accounting and transaction processing, Section 3.01 of the Grant Agreement reflected the eligible expenditures in the “Withdrawal of Grant Proceeds” table by expenditure category, rather than project component. Accordingly, the amount reflected in the above table for “Actual at Project Closing (US$M)” is an estimate up to the end of OP I. Though the second AF Project Paper provides component costs for the proposed US $30M, this amount only partially materialized, therefore, no formal allocation of funds by component for the actual amount received for OP II implementation is available. The below table reflects the actuals at project closing, where Category 1 represents all expenditures associated with sub-grants, and Category 2 reflects all other technical activities, governance and ASARECA’s management and administration costs. PROJECT COST BY EXPENDITURE CATEGORY Actual at Amount at Percentage of Expenditure Category Project Closing Approval (US$M) Approval (US$M) (US$M) Cat. 1: Goods, services, and Training and Workshops for sub-projects under Part (a) 44.2 36.1 81.7 of the Project Cat. 2: Goods, works, consultants’ services (including audits), Training and 25.9 29.3 113.1 Workshops, and Operating Costs under Parts (b) to (g) of the Project Total 70.1 65.4 93.3 Page 46 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ANNEX 4. Supplementary Results Matrix for ASARECA Transition Period (2017-18) Baseline Targets & Unit of Project Indicators (MDTF, (Actual) Values Measure 2016) 2017 2018 PDO: Enhanced governance and institutional reforms for re-establishment of ASARECA’s relevance and credibility Indicator 1: Percent change in the reforms of ASARECA’s governance 50 >90 Percent 40.5 structure 25 (45) (69.8) Indicator 2: Level of effectiveness of ASARECA management 60 >90 Percent 42 structures and systems (55) (70.4) Indicator 3: Level of stakeholder satisfaction with the reform 60 >90 Percent 49 processes (governance and management reforms) (53) (72) Output 1: ASARECA governance and organizational structures operationalized Indicator 1.1: Level of ASARECA’s integration with RECs 40 >75 Percent 27 (31) (48) Indicator 1.2: Percent change in ASARECA Organizational 65 >90 Percent 58 Performance Index (OP I) (57) (68) Indicator 1.3: Proportion of required staff complement hired based 74 100 Percent 75 on approved organogram (52) (37.5) Indicator 1.4: Level of Board performance with respect to the 75 >90 Percent 68 institutional reforms (73) (73) Output 2: Enhanced ethics management systems and procedures Indicator 2.1: Level of compliance with ethics and anti-corruption 75 >90 Percent 7.5 processes and standards (53) (55) Indicator 2.2: Proportion of staff trained in ethics standards and 68 100 Percent 0 complying with established standards for ethical behavior (0) (0) Output 3: Human resource, financial management and knowledge systems strengthened Indicator 3.1: Proportion of relevant infrastructure put in place to 79 100 Percent 62 enhance the functionality of the Secretariat (65) (76) Indicator 3.2: Level of compliance with the Operations Manual 75 >90 Percent 58 (68) (76) Indicator 3.3: Percent of stakeholders satisfied with the status and 40 >90 Percent 25 functionality of the ASARECA knowledge and information hub (30) (61) Output 4: Enhanced partnerships and program implementation Indicator 4.1: Number of partnerships formed / strengthened 15 25 Number 11 (14) (21) Indicator 4.2: Number of international meetings, workshops and 40 75 events that ASARECA convened to tackle high-priority issues for the Number 20 (35) (38) sub-region Indicator 4.3: Number of documents uploaded on ASARECA 60 100 repositories / made available publicly (publications, photographs, Number 30 (50) (77) documentaries) Indicator 4.4: Percent change in implementation of the Fiduciary 75 100 Percent 0 Action Plan (see FAP scores) (54) (72) 25 Significant change in the reforms of ASARECA governance structures is noted, with the effectiveness of the Governance Structures and Systems rated at 70%. Among the key milestones include: 1. Appointment of the Executive Secretary, the Head of Finance, the Accountant and the Internal Auditor 2. Renewed efforts in fast tracking the consultancy of the ASARECA Governance Structure and Systems 3. Advanced progress for finalization of the consultancy on Institutional Analysis, especially the Salary Reviews 4. Appointment of External Auditors (KPMG) to carry out financial audits of the activities and funds of ASARECA (unqualified 2017 Audit Report and Management Letter available). Page 47 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ANNEX 5. RECIPIENT COMMENTS ASARECA appreciates the support received from the World Bank Group in its journey during the last two decades. Despite challenges in fiduciary management that adversely affected implementation of OPII, and which led to suspension of implementation of research activities, the Board and Management strived to revive ASARECA’s corporate image during the transition period. Notwithstanding some delays, the Bank team strongly supported ASARECA Secretariat to develop and successfully implement the Fiduciary Action Plan (FAP). Most importantly, the Bank experts played an important role in the process of implementing governance and management reforms, development of ASARECA’s new 10-year strategy and results framework (2019-2028), as well as the 5-year medium term operational plan (2019-2023). Through these two products, ASARECA repositioned itself in the changing Africa’s agriculture research for development and clearly defined its vision, mission and value proposition. ASARECA learnt from its past mistakes and seized the opportunity to put in place efficient management and governance systems for a credible institution. In May 2019, ASARECA held its inaugural Council of Patron Ministers Summit under the new governance framework. Among other things, the Patron Ministers from the Eastern and Central Africa sub-region recommitted to support the regionally coordinated agriculture research for development and tasked ASARECA to continue playing its role as a regional coordination body. As a result of the relaunch of ASARECA, the management embarked on the process of strategic re-engagement, partnership development and mobilization of financial resources to implement the new operational plan. To this effect, continued support by and partnership with the World Bank is critical towards catalyzing and sustaining agriculture transformation in Eastern and Central Africa sub-region through research for development. For more information see ASARECA’s Progress Completion Report. Page 48 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) ANNEX 6. Summary of Results and Ratings of OP I Implementation Completion and Results Report (ICRR) OP I had three Original Components: (i) Research for Development (US $43.46 million) (ii) Support to ASARECA's Governance Structures (US $1.51 million) (iii) Support to ASARECA's Management & Administration Structures (US $8.66 million). Below is a summary of Results and Ratings of OP I Implementation Completion and Results Report (ICRR), with excerpts extracted directly from the approved ICR in the order in which they appear. Key Factors Affecting Implementation and Outcomes 2.2 Implementation 1. The ASARECA Secretariat demonstrated strong commitment to the project, which helped to overcome the many day-to-day challenges reported in regular Supervision Mission Aide Memoires. Overall, the MDTF was well implemented. Key factors that affected implementation and outputs/outcomes are as follows: • The suspension of the European Development Fund (EDF) on June 30, 2007 (due to administrative issues) affected 22 CGS research sub-projects that had been approved for funding under the European Commission Grant and had to be abruptly halted. • Low utilization of ASARECA’s annual budget with respect to sub-grantees (SGs) (due to initial fiduciary capacity constraints, which got remedied over time) • ASARECA program implementation was rooted in the CAADP and Framework for African Agricultural Productivity (FAAP) principles, helping to align the resulting projects to country needs. • Collaboration with partners helped to supplement MDTF activities to achieve results. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 2. ASARECA was responsible for M&E, using its’ own RF and M&E implementation plan that set out performance indicators for OP I. The MDTF RF was well designed and echoed the OP I priorities. The KPIs clearly reflected the PDO, and component activities, expected outputs, and the KPIs were strongly linked. This made it easier to monitor MDTF outputs and outcomes. 3. ASARECA used M&E at different levels (project, program and ASARECA) for different purposes, and provided regular monitoring of implementation. By using its management information and FM systems, activities were kept on track and potential problems addressed. The system also provided a useful management tool, for example, to track CGS disbursements and to design relevant training materials. A delay of about eight months from July 2009 in filling the position in the M&E Unit slowed some activities; however, these are now well underway. The M&E unit trained over 326 people; including ASARECA Program staff and Project Partners in impact assessment, data quality management, results-based M&E etc. Similarly, relevant and user-friendly M&E operational procedures and tools were introduced to accelerate M&E and learning. These have facilitated timely decision making at all levels. 4. The M&E system improved substantially over OP I, with good monitoring of outputs in technology and innovation, generation, adoption and up-scaling, although the latter was limited. Overall, ASARECA has Page 49 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) successfully set up a functioning M&E system and created a culture of monitoring its activities, projects and programs, and those run by its partners, which is commendable. The system is online for use by national M&E teams. OP II has started with a robust M&E system in place due to the work done at all levels--the Secretariat, NARS, SG and other IAR4D partners. The Impact Evaluation 2014 should serve as a baseline for monitoring activities and outcomes under OP II. 2.3 Safeguard and Fiduciary Compliance 5. The MDTF was classified an Environmental Category B due to its’ relatively limited environmental impact potential. Three policies were triggered: (i) environmental assessment (OP/BP 4.01); (ii) Forests (OP/BP 4.36); and (iii) Pest Management (OP 4.09). ASARECA prepared an Environmental and Social Management Framework (ESMF) to be included in OP I. The CGS sub-projects’ environmental risks and management plans were identified in sub-project funding proposals, evaluated as criteria for funding, and monitored by ASARECA’s M&E unit. 6. The MTR observed that ASARECA had not been applying the provisions of the ESMF to all its activities, and downgraded overall Safeguard Compliance to ‘MU’ (ISR, November 2011). These provisions were also not being adequately followed in the newly approved sub-projects. Although most of the new sub-projects had been screened for environmental and social impacts, proper categorization was not being done to identify the level of assessment required. Some sub-projects that required the preparation of Integrated Pest Management Plans (IPMPs) and ESMPs had not prepared them. The Bank recommended recruitment of an Environmental and Social Screening (ESS) Specialist to review and assist in the development of ESS instruments. ASARECA adopted the MTR’s proposals and incorporated them into the new sub-projects that began in late 2012/early 2013. Capacity building efforts improved the level of compliance. By June 2013, safeguards compliance was rated MS and in December 2013 it was upgraded to Satisfactory. 7. Financial Management. FM issues were mostly related to the low burn rate discussed in Section 2.2 and resolved accordingly. The capacity of the Secretariat’s existing FM team is strong, due to the recent recruitment of an additional accountant to enhance FM oversight of SGs and an additional internal auditor, and the formation of a stand-alone audit committee. The Secretariat team has adopted a proactive approach of training the SGs on FM and procurement before disbursement. FM arrangements have been mainstreamed at the SG level. The MDTF is in full compliance with the financial reporting requirements, including the timely submission of acceptable quarterly interim financial reports and annual audit reports. All the audit reports have been unqualified and no major FM weaknesses are reported. The review at the SG level noted that there were general improvements in the FM systems in most of the institutions. 8. Procurement. The post procurement review in 2010 of a sample of MDTF-financed procurement undertaken by the Secretariat highlighted several issues including incomplete procurement files, non-compliance with guidelines for some procurement, and conflict of interest with respect to the Procurement Officer’s duties. Procurement was downgraded to ‘MS’. ASARECA management executed the necessary remedial actions, such as fulfilling the requirement for new sub-project proposals to undergo a procurement assessment, hiring a Procurement Officer in July 2011, and submitting procurement plans by all new SGs and most existing partners. The main remaining activity is the procurement of the construction of the Headquarters. The size of the building has been scaled down, and the estimated cost has been reduced from US $2.3 million to US $0.997 million. The Bank has approved the revised structural design and the National Competitive Bidding method. The contractor is Page 50 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) expected on board by July 1, 2014, and they will proceed during OP II. The Post-Procurement Review (PPR) noted that procurement capacity within the Secretariat remained adequate with general compliance with the procurement manual; however, the SGs require more support and monitoring to conduct timely procurement. Assessment of Outcomes 2.1 Relevance of Objectives, Design and Implementation 9. The MDTF’s PDO remains relevant to ASARECA’s strategy, which is aligned with CAADP Pillar IV and the NEPAD strategy for revitalizing agricultural research, technology dissemination and adoption. The objectives are relevant to the member states’ pursuit of raising agricultural productivity through accelerated innovation, and eradicating hunger. The World Development Report 2008 argues that significant growth in agriculture is critical for most African countries to meet the MDGs. The objectives are also relevant to the World Bank Strategy for Africa, 2011, a 10-year vision that embraces greater competitiveness in agriculture. 10. The multi-faceted approach, which called for interventions in research and dissemination, ASARECA’s governance structure, and reforms in agricultural policies based on research findings encompassed the essential elements needed to enhance ASARECA’s value added and its work with the member country NARS. The MDTF RF was well-designed, with a concise PDO, and clearly linked and measurable outcome indicators. The restructuring further improved these links by realigning them with the ASARECA OP I indicators to allow for better monitoring. 11. The MDTF funding mechanism allowed multiple donors to work together to facilitate OP I and to pool funds for continued IAR4D activities. It has also been an effective instrument for reducing the administrative burden on ASARECA, through the elimination of separate supervisions, financial and activity reporting for multiple donors. The Bank, as administrator of the funds, which also provides technical and administrative support, has set consistent and high standards throughout the duration of OP I. 2.2 Achievement of Project Development Objectives 12. The MDTF’s main objective was to enhance access and use of agricultural research TIMPs in the ECA region by supporting ASARECA to implement OP I. Component 1 focused on research generation, dissemination and adoption activities, while Components 2 and 3 sought to strengthen ASARECA’s governance and management structures to help the Association to fulfil its mandate and assist targeted stakeholders to fully realize the project’s benefits. Page 51 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) Table 1. MDTF Key Performance Indicators: Targets and Actual Achieved End of Project Key Performance Indictors Baseline End of Project Project End of Project Actual (Outcome) Indicators 2008 Target (percent achieved) 1: Number of stakeholders accessing TIMPs 2,014 265,964 422,176 158.7 2. Stakeholders adopting new TIMPs in 8.3 70 64 91.4 selected development domains (percent) 3: Number of reform of policies, laws, 6 28 18 64.3 regulations and procedures approved 4: Level of stakeholder satisfaction with 5 81 71* 87.6 TIMPs. (percent) 5: Number of direct beneficiaries reached 2,533,056 through ASARECA support (disaggregated by 12,084 1,595,784 (Male: 55 percent; 158.7 gender) Female: 45 percent) ASARECA, M&E Unit, January 2014. *Under Indicator 4, the achievement of 71 percent represents only those individuals who recorded either a high or very high level of satisfaction. Achievement of Outputs and Outcomes: • KPI 1: Number of stakeholders accessing TIMPs • KPI 2: Percentage of stakeholders adopting new TIMPs 13. The MDTF funded 85 of the 90 successfully concluded CGS sub-projects since 2009 and helped ASARECA to deliver a substantial research program. The sub-projects brought together researchers from 11 countries to generate work on 409 TIMPs of regional importance (against an OP I target of 378). A breakdown of projects by program indicates that 310 sub-grantees received a total of US $30.2 million for research and uptake activities. 14. Generation of Research: Assuring relevance and quality. ASARECA followed a rigorous process to ensure relevance and quality of research through its seven research programs and the CGS. The programs were: Staple Crops; High-Value Non-Staple Crops; Livestock and Fisheries; Agrobiodiversity and Biotechnology; Natural Resource Management (NRM) and Forestry; Policy Analysis and Advocacy; and Knowledge Management and Up- Scaling (KMUS). They were managed by well-qualified scientists and designed to address only high priority topics and deliver results within a short time span.26 15. The CGS was ASARECA’s main instrument for addressing regional research priorities and improving the rigor and quality of research proposals. CGS proposals were based on the results of priority-setting workshops attended by stakeholders from all the countries including government agencies in research and dissemination, NGOs, private enterprises and farmers. This process was effective in ensuring congruence with regional priorities. Research designs, implementation, analyses, interpretation of results and write-up were all undertaken by the regional teams and overseen by ASARECA Program staff. 26 This was based on International Food Policy Research Institute (IFPRI) analysis, which had shown that “the largest impact for the greatest number of beneficiaries” would be attainted through a primary focus on major food crops that were also traded regionally. Page 52 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 16. Quality of Science (QoS): Through capacity development and the regional collaborative research program, 293 partnerships were fostered among several agricultural research actors within the region (exceeding OP I targets by 23 percent, as described in Annex 2). These partnerships have allowed for an exchange of knowledge and experiences across different institutions when dealing with similar challenges. Through these partnerships, research outputs increased marketing of commodities, institutional mentorship of scientists, adoption of selected TIMPs, etc. Partnerships were formed, predominantly among NARS actors across the region. For example, some of the competitive research grants stipulated that collaborative research projects should be carried out in at least 3 countries involving different actors, such as research institutes, universities, advisory services providers, private sector, farmer organizations and non-state actors.27 Bank missions noted that the use of multidisciplinary teams contributed significantly to overall improvement in the quality of research. 17. Access to TIMPs: In the ASARECA context, access refers to the ease that people have in obtaining relevant products and services such as TIMPs. Access to TIMPs includes how to get them, where to get them, how to use them, and what impact would be achieved after using them. In all, 422,176 stakeholders accessed an assortment of TIMPs. The numbers accessing new and existing TIMPs increased steadily over the years except for the reduction in 2012. This was due to the closure of some pre-OP I projects inherited by the MDTF, which had limited awareness and up scaling activities. In 2013, there was a substantial increase in the numbers accessing TIMPs due to the scaling up/out activities under the additional financing.28 18. Adoption of TIMPs: Similarly, the uptake of technologies has been good. Methods for scaling up the adoption of technologies were tested systematically, and a total of 498 demand-driven TIMPs (12 percent above OP I target of 444) were made available for uptake and development. This also included technologies that were already available (‘on the shelf’) where ASARECA supported the processes to make these technologies accessible. The rate of adoption of the already-generated TIMPs by stakeholders increased gradually from 20.6 percent in 2009 to 64 percent in 2013. Overall, based on the trends data, the average rate of adoption of available TIMPs over the five years was 42.2 percent compared to the OP I target of 43.1 percent. 19. These new technologies included new crop varieties (including quality protein maize (QPM) and orange fleshed sweet potato, with significant nutritional value, cassava, sorghum, green gram, Napier grass, climbing beans, and banana), seed multiplication, new seed production technologies for vegetatively-propagated staples (i.e. banana), improved soil fertility and water management practices, improved pest and disease management practices (both crop and livestock), post-harvest management and value addition (i.e. dairy value chain, bee keeping), and enhanced NRM practices (i.e. afforestation, forage management, integrated agro-diversity, and conservation management to promote eco-tourism). 20. Gender was well mainstreamed across the ASARECA program portfolio, with 96 percent of all research projects categorized as ‘gender-responsive;’ i.e. incorporating gender-responsive agricultural innovation systems activities in their interventions. Several NARIs have also set up gender units within their institutions. The impact evaluation shows clear benefits for women beneficiaries. 27 However, due to capacity issues and access to information, the majority of the grants were awarded to a relative small group of actors in three countries within the region. 28 The number of stakeholders accessing TIMPs in each year: 7,120 (2009); 22,915 (2010); 68,632 (2011); 25,557 (2012); 297,952 (2013). Page 53 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 21. While the first three years of OP I focused mainly on generating new TIMPs, the emphasis in the last 2 years shifted to the out-scaling of these technologies and adoption by farmers. Building on its experiences of the second part of OP I, ASARECA has started to engage more with private sector partners and non-state actors to work on scaling out technologies through strong linkages to market and value chain development. The impact evaluation findings from OP I demonstrate that several TIMPs provided clear benefits for farmers in terms of increased production and increased income, as presented below. For OP II, ASARECA has developed a scaling-up strategy, with a strong emphasis on determining the increases in farm household incomes as well as the levels and rates of adoption of the TIMPs. 22. The innovation platforms for technology adoption (IPTA) under the Knowledge Management functions also promoted problem-solving by a wide range of stakeholders, including researchers, farmers, extension workers, NGOs, policymakers, equipment manufacturers, traders and processers. The fact that the end users of an innovation were involved every step of the way in the development of the technology appears to have made innovation adoption much easier (Aide Memoire, February 2014, Annex 11). 23. Other positive outcomes include the following: • 4,658 hectares of land were dedicated to TIMPs, mainly for the production and multiplication of quality pre-basic, basic and certified seeds of selected commodities (i.e. Quality Protein Maize (QPM), African Indigenous Vegetables, snap & climbing beans….); • Over 20 percent of this land was devoted to seed multiplication, especially for sorghum, cowpeas, green gram, QPM, Orange Fleshed Sweet Potato (OFSP), as well as on-station and on-farm establishments and rehabilitation of forage seeds; • Production of 2,436 metric tons (MT) of quality pre-basic, basic and certified seeds from these dedicated lands, through seed multiplication and production processes; • Seed sales of over 30 percent, with the rest distributed to farmers for further multiplication; • Over 50,000 hectares of highly degraded lands (mainly with the arid and semi-arid lands) and watersheds reclaimed; • Overall, the CGS projects are believed to have helped about 1.45 million smallholder farmers and other primary sector producers, including individual processors, rural entrepreneurs, managers and traders, and natural resource managers via access to TIMPs, access to inputs and markets, and demonstrations. This has led to more resilient crops, higher yields, and increased incomes. • Through the sale of products and services produced from the projects, the targeted farmers and stakeholders generated a total income of US $12.5M. 24. The impact evaluation findings on higher-level outcomes include the following:  Beneficiaries achieved increases in yields of key crops of most projects surveyed, especially QPM, Sorghum-Legume, Bean Innovation (Uganda) and Integrated Water Management (Kenya). However, there are large yield variations between farmers and many beneficiaries have not yet seen increased yields;  Cash incomes of beneficiaries increased notably for half of projects surveyed. This is significant given the short time period since taking up the technologies. Averaged across all projects surveyed, incomes from sales of crops and livestock have increased by an average of US $388 per beneficiary between 2008 and 2012. Average incomes for non-beneficiaries fell by US $29; Page 54 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600)  Total value of net agricultural output (crops and livestock) increased in real terms from 2008 to 2012 for beneficiary households surveyed from US $940 to US $1047: an average of US $106. By contrast non-beneficiary households saw an average fall of US $166;  Overall, there was an increase in the proportion of food secure beneficiary households over the program period from 74 percent to 81 percent. However, food security status varies across countries and projects;  Household nutrition levels increased. A snapshot of nutrition indicated that overall 56 percent of beneficiary households consumed from all food groups compared to 52 percent of non-beneficiary households, although the difference was not highly significant. 25. There were many other MDTF-supported activities and outputs that were critical in achieving research outcomes and the PDO. These included ASARECA’s advocacy to reform policies to help access to and adoption of TIMPs, capacity building of all stakeholders in the value chain, and the dissemination of research results. • KPI 3: Number of reform of policies, laws, regulations and procedures approved 26. It is challenging to assess an organization’s contributions to policy change, especially when it involves regional harmonization, given the often long and arduous process in which a multitude of actors engage and interact. The progress made against each of the three intermediate indicators related to policy changes (policies analyzed, presented for decree, and approved) is described further in Annex 2 of OP I ICR. ASARECA played many roles in this process, as a key driver, and in a supportive role as a convening regional platform for agricultural research and providing specific technical expertise, i.e. in seed production, and policy analysis expertise (DFID, 2014). 27. ASARECA ‘s Policy Analysis and Advocacy Program (PAAP) contributed to catalyzing regional trade. Positive impact has been achieved (or is on its way) in such areas as seed law and the quality of processed agricultural products. For example, collaborative research and advocacy with ILRI and key decision makers in Kenya enabled small-scale farmers to penetrate extensive milk markets. Harmonized food standards for cassava supported the development of several value chains around cassava (for policy-related areas led by ASARECA see Annex 2 Table 3 of the OP ICR). 28. A regional approach to addressing a set of significant policy constraints in collaboration with well-chosen partners, the majority from outside the NARIs, has been key to PAAP’s contributions to harmonizing national laws or regulations affecting seed trade, product quality standards, and crop variety release. The USAID report (2011) particularly highlights PAAP as a positive example of what ASARECA can achieve through continuous and experienced program management, and through well-chosen partnerships with regional and international institutions as well as with individual scientists from beyond the NARIs. As a regional body, ASARECA is well placed to address some of the national barriers, but even more so the regional bottlenecks that limit the potential impact of proven agricultural research products. 29. So far, 100 different policies, laws, regulation and procedures have been analyzed through assessment by competent analysts, teams and consultants. An example of a policy analysis for which ASARECA’s role was appreciated was the policy research on trends in food price transmissions in Ethiopia, Kenya, Tanzania and Uganda. It showed the need for a clear roadmap to address the drivers of food price volatility instead of just responding to crises as they hit. As a follow up, a food price forecasting model was developed and is being piloted Page 55 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) and customized to inform early warning and response systems in the region. The regional data portal can be accessed at ttps://sites.google.com/a/cgxchange.org/food-prices-portal/. 30. Policy work has succeeded in facilitating a change in existing polices. A total of 18 agricultural enabling environment policies, regulations, and administrative procedures in the areas of agricultural resource, food, market standards and regulations were approved. The relatively low number can be explained by the process of changing policies, which must go through various stages. It is also difficult to attribute a policy change to a specific actor, i.e. ASARECA. An example of a policy option presented for legislation, which was adopted, is ASARECA’s contribution to the harmonization of 11 standards (cassava (7) and potatoes (4)) for the East African region, which was mainly driven by EAC. The standards were approved by the East African Standards Committee (EASC) and were subsequently declared by the East African Community (EAC) Council of Ministers to be used as East African Standards in 2009. 31. Harmonized standards are important to enhance farm-level production, utilization and trade, local and intra- regional. The harmonization of cassava standards had a positive impact on farmers’ livelihoods through increased income derived from better marketing opportunities locally and regionally. For instance, it encouraged the Popular Kumi Women Initiative (Uganda) with its membership of 2,500 to grow cassava and sunflower, as processed sunflower oil and cassava flour have a ready market in Uganda’s urban centers. The sales from these products resulted in increased income for the women. They are now able to feed their children, themselves and husbands better, and now contribute to educating their children. 32. Regarding impacts of an improved seed policy environment, ASARECA’s support to enhancing the enabling policy environment for seed trade in the region has mainly been through the Eastern Africa Seed Committee (EASCOM). A policy impact study on the improved harmonization of regional seed policies led by ASARECA showed that by 2010, seed production within the region had tripled from 43,000 tons to about 122,000 tons. Domestic and cross-border seed trade has also increased threefold from 1,000 tons due to policy reforms, thus leading to growth in volumes of seed traded. For example, in Uganda, the private sector has nearly taken over the role of seed production. Uganda with its 32 seed companies has fully liberalized seed trade with the government retaining the role of regulation with options to cede some authority to accredited seed companies (DFID, 2014).29 Full liberalization of seed trade in East Africa has not been realized yet but work is on-going. It is notable, however, that for seed systems ASARECA has paved the way by working on key legislations and regulations for an effective seed system in several of the member countries (Seed law and associated regulations and institutional set-up, plant variety protection, seed certification, biosafety regulations at COMESA level, etc.). 33. It is clear from the new practices adopted that ASARECA contributed in several areas to facilitate policy making processes, mostly from a regional perspective. In this way, it has lived up to its regional orientation by working on issues of a regional nature and with several partners with a regional or international focus such as EAC, COMESA, East Africa Dairy Regulatory Authorities Council, East African Farmers Federation and CG Centers. However, policy uptake remains a challenge and a multi-pronged strategy needs to be developed to over the hurdles of the policy approval. 29 ‘Impacts of an improved seed policy environment in Eastern and Central Africa’ by M.Waithaka, J.Nzuma, M.Kyotalimye, and O.Nyachae (ASARECA, 2011), and ASARECA Project Completion Review (PCR). Page 56 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 34. The MDTF, in collaboration with other projects and donors, supported capacity-building activities on a range of technical areas that enhanced overall knowledge and application methods of stakeholders, and contributed to significant positive impacts for farmers in terms of crop and livestock yields and income. The ASARECA Secretariat provided technical support to the SGs as needed. This varied with the level of experience of those involved in undertaking the specific activity. Generally, stakeholders met before, during, and at the end of the activities, and ASARECA staff and the national Principal Investigators (PIs) visited the activity sites during the sub-grant to provide technical and administrative backup. Bank missions and periodic external reviews that have critically assessed this work note the considerable expertise in national programs involving many effective PhDs. The cadre of researchers (PhD and Masters) from different countries who have been trained within and outside Africa contributed to the new knowledge and skills pool. 35. Overall, ASARECA facilitated capacity strengthening and skills development initiatives for 81,834 stakeholders (55 percent male; 45 percent female) through a series of targeted short- and long-term trainings and technical backstopping exercises. Target groups included M&E staff, Principal Investigators and other players in the value chain (farmers, traders, processors, NGOs, extension agencies, universities etc.). It should be noted, however, that the total trained includes both those capacity building initiatives provided by ASARECA directly, and those provided by other programs with facilitation support by ASARECA (such as SCARDA and BeCa-ILRI hub). ASARECA has tracked the whereabouts of beneficiaries of its higher education training initiatives and established that most have been deployed to critical positions in the agricultural sector in their respective countries. Former ASARECA- supported graduates are working at NARIs and play a key role in supporting the generation and uptake of new technologies (DFID, 2014). Annex 2 provides examples of the significant positive impacts of the targeted trainings for farmers in the areas of crop and dairy milk production, and moisture retention of farmlands. 36. ASARECA created robust communication strategies for its different outputs to realize greater use of agricultural research and development innovations. ASARECA staff and sub-grantees made a consolidated contribution to this result, with the lead taken by the M&E team, the Knowledge Management Programme unit and the ICT unit. Different strategies of sharing research outputs were used, considering the different users and their needs. Some of the communication strategies were successful and others less successful. ASARECA took into consideration recommendations following the MTR to strengthen the KMUS and Information Communication Unit (ICU) to deliver the outputs. For instance, KMUS has collated, compiled and established a database of 37 best-bet “proven” technologies and innovations that are ready for use in ECA countries – the products of past ASARECA programs, which are well tried and tested technologies that have potential for up-scaling to improve livelihoods (DFID, 2014). The KMUS took the lead in coordinating and supporting the development of 885 different types of information packages (intermediate indicator (IO) 1), while 272 different pathways have been used to reach the stakeholders (IO 2). The project reached over 2 million stakeholders with a variety of information products, mostly through demand. The level of satisfaction of stakeholders accessing disseminated information was 87 percent, against a target of 75 percent (IO 3). 37. Infrastructures provided mostly to NARIs were important in enabling researchers to conduct quality research by having access to appropriate and up-to-date research equipment. Examples include gene banks, computers, laboratory and building refurbishment, screen houses, nurseries, biodiversity monitoring kits, irrigation kids and forage choppers. • KPI 4: Level of stakeholder satisfaction with the TIMPs. Page 57 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 38. After accessing and adopting the available TIMPs, 71 percent of the stakeholders were highly or very highly satisfied in 2013 compared to 11.5 percent in 2009. The average level of satisfaction over the five-year period is 40 percent compared to a target of 45.5 percent (due to the lower levels in 2009 and 2010). The slow increase in the stakeholders’ satisfaction with the TIMPs and other ASARECA products are mainly attributed to the short timeframe for project implementation, and the short duration for on-farm testing of the availed products. 39. The impact evaluation carried out a qualitative assessment of farmers’ evaluation of TIMPs. Households participating in ASARECA projects (beneficiaries) were asked about benefits realized from the technologies, whether technologies met their needs and to assess their level of satisfaction with the technologies (Impact evaluation, p. 105-106, Table 38). The benefits that were reported included: increased income, production and food security; better nutrition; early plant maturity; reduced farmer labor requirements; increased soil fertility; and improved soil and water conservation. 40. Almost 80 percent of farmers surveyed reported that the technology benefits met their needs. Over 80 percent of OFSP project beneficiaries in Kenya and Tanzania, Bean Innovation project beneficiaries in Rwanda, Quality Seed Potato projects in Uganda and Rwanda, and QPM and Crop-Livestock beneficiaries in Uganda gave an overall positive assessment of technology benefits. A further 65-79 percent of beneficiaries from the Integrated Water Management project in Kenya and Rwanda; QPM, Sorghum-Legume Intercrop and Crop-Livestock Integration project in Tanzania; and Bean Innovation project in Uganda stated that the technologies met their needs. Only the Farmer Led Seed Enterprise project in Kenya had fewer than 50 percent of beneficiaries reporting that the technology had not met their needs. These findings point to the overall good match between ASARECA- promoted technologies and beneficiary needs. • KPI 5: number of direct beneficiaries reached through ASARECA support (disaggregated by gender). 41. The total number reached through support programs is about 2,533,056 (55 percent males; 45 percent females). This assumes that each farming household has six members, and that 422,176 stakeholders accessed various TIMPs. 42. Activities under Components 2 and 3 helped to strengthen ASARECA’s governance and management structures that in turn improved accountability and influence in national agricultural research systems. Support to ASARECA resulted in stronger individual, organizational and institutional capacity to plan and execute credible research linked to the needs of small holder farmers in the region. Key results include:  Stronger and more accountable governance framework including a robust Constitution, Governance Manual (GM), Operational Manual (OM), host country agreement, effective board and management structures, credible research and FM systems, effective operational plan, operational M&E systems, compliance with environmental and social safeguards and more efficient procurement systems;  Effective transformation from a loose network of researchers in 2006 to 7 integrated programs and technical support unit delivered under a single OP in 2013; and  From a non-existent gender focus in ASARECA in 2006 to gender-mainstreamed programs coordinated by a well-resourced unit in 2013. Page 58 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 43. The various governance reforms broadened ownership of the organization and diverse partnerships beyond its traditional partners in research institutions in response to one of the major principles embedded in the FAAP, namely plurality in the delivery of agricultural research, extension, and training services. Broadened ownership also expanded ASARECA’s reach to fulfil its roles as one of the CAADP Pillar IV partner institutions. The roles cover sensitizing NARIs to the CAADP process, reviewing national and regional investment plans, and support to research based on ASARECA member country priorities (DFID, 2014). 44. Overall, the MDTF has helped ASARECA to improve the enabling agricultural policy and regulatory environment for increasing the productivity of smallholder agriculture in the region, and successfully achieved its PDO to enhance access and use of agricultural research technologies and innovations in the agricultural systems in Eastern and Central Africa. 3.2 Efficiency 45. The impact evaluation conducted an overall assessment of the performance of the OP I program based on the findings from the household survey, using a Cost-Benefit approach. This compared benefits accruing to program beneficiaries with costs of the program (US $90 million) (Annex 3). The analysis found that the net difference in incomes between beneficiaries and non-beneficiaries was US $272. Extrapolated across all ASARECA beneficiaries adopting new TIMPs (270,000 households, i.e. 64 percent of stakeholder accessing new TIMPs), this represented a net increase in agricultural production value of US $73.44 million (change in beneficiaries compared to non- beneficiaries). Comparing the value of benefits to the program costs indicated that in about 1.25 years the program would have covered operational costs. Thus, the benefit-cost ratio was expected to be positive by the second quarter of 2013. 3.3 Justification of Overall Outcome Rating Rating: Satisfactory 46. The MDTF was relevant at the time of appraisal and continues to be relevant. Its components were also clearly linked and relevant to the PDO. As presented in Section 3.2, the PDO was achieved satisfactorily with the project exceeding several of the outcome targets, and in some cases by a significant margin. The impact evaluation found that the MDTF-supported activities also helped to attain higher-level outcomes such as increases in real value of total agricultural output, income for project beneficiaries, and food and nutrition security for most households, and participation of women and people with disabilities. In terms of efficiency, the analysis indicates that the benefit-cost ratio was expected to be positive by the second quarter of 2013. 3.4 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 47. The impact evaluation provides evidence of significantly increased average gross household incomes of project beneficiaries between 2008 and 2012 (Section 3.2). Averaged across all projects surveyed, incomes from sales of crops and livestock have increased by 90 percent for beneficiary households from US $429 in 2008 to US $817 in 2012. Gender aspects have been mainstreamed into research activities, and the project reached out to women and to households with disabilities (Impact evaluation, p. 40). Of the CGS projects, 96 percent are Page 59 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) categorized as ‘gender responsive’ (Section 3.2). Nutrition Security indicators showed improvements for beneficiaries and non-beneficiaries in Kenya, Rwanda, Tanzania and Uganda (Impact evaluation, p. 36). (b) Institutional Change/Strengthening 48. The MDTF has contributed to building long-term capacity in many areas. ASARECA has matured in terms of institutional capacity to identify research needs, design and commission credible research, and ensure research uptake and communication. As a state membership research body, ASARECA has the legitimate relationship with NARS and international agricultural research institutes to act as a regional platform for mobilizing resources and coordinating research at regional level for country and regional benefits. ASARECA has performed this role effectively by raising and managing resources for OP I and commissioning research under its seven programs. Training activities have also strengthened capacity in all players in the value chain, in the NARIs, and helped these partners to experience the benefits of well-coordinated research and development. Under OP II, ASARECA will emphasize institutional strengthening of the NARIs in ‘less-participating’ countries (see Section 5.2 b). (c) Other Unintended Outcomes and Impacts (positive or negative) 49. Spillover Effects with respect to farmers and research staff. Project beneficiaries reported that between 5 and 27 farmers learnt about the technology from them and a further 3-22 farmers then went on to take up the technology. Overall, beneficiaries surveyed reported passing on information to an average of 11.5 farmers, with 7.1 non-beneficiaries taking up the technologies. The process of project implementation provides time for non- project farmers to learn about the technology. Even taking out the high rates reported by specific projects, these estimates represent quite high spillover effects. Strengthening capacity at NAROs also benefits other projects. 50. Leveraging new sources of support. As ASARECA has implemented some of the activities under OP I with other regional bodies and within the CAADP agenda, this has provided opportunities for scaling up successes, leveraging new sources of support from both political and financial stakeholders within and beyond the ASARECA region to implement the activities, and adaptation of the ASARECA model in other parts of Africa. At national levels, member-NARIs has committed to invest in ASARECA by paying some resources to the organization. Other impacts include: the President of DRC promoting the up scaling of QPM; and ASARECA capitalizing on opportunities to partner with health sectors in various countries and partners who support better nutrition outcomes. ASARECA’s work has been mainstreamed into some Ministries of Health; especially in the areas of QPM and OFSP (with Vitamin A) encouraging young people to promote vine multiplier technologies. A project in Kenya on “Hidden Hunger” has also taken on promoting OFSP to prevented stunted growth. 3.5 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 51. The impact evaluation 2014 assesses the performance of ASARECA in meeting its’ OP I objectives. These findings are summarized in Annex 5. 4. Assessment of Risk to Development Outcome Rating: Low Page 60 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) 52. The institutional risk to development outcome is low. In addition to ASARECA’s stronger institutional set up, firmly embedded principals of FAAP in project implementation, and the various partnerships it has formed, ASARECA’s position as a regional platform for agricultural research is further strengthened through its linkages to the Eastern Africa Agricultural Productivity Programme (EAAPP). For OP II, about US $4.2 million is expected to come through the EAAPP phase two, if implemented. Continued strengthening all its institutional arrangements will be critical to addressing agricultural challenges that are relevant for the wider region and go beyond national interests. ASARECA has partnered with other projects and donors to strengthen the less competitive NARS and human resource capacities of member countries such as Rwanda, Burundi and Sudan. However, even the strong NARS may have some deficiencies that limit innovation, dissemination and scaling up (Impact evaluation, 2014, p. 70). Under OP II, ASARECA plans to actively assess capacities of all NARS to ensure continued integration. 53. The financial risk to development outcome is moderate. ASARECA currently fills a recognized need in the region for collaboration, leadership and technical support for national research and dissemination activities. Its value has been well demonstrated during OP I. The limited core funding so far committed to OP II poses a risk to ASARECA in terms of its financial sustainability as it may affect its current governance and management systems. The extent to which ASARECA will be able to sustain itself depends on its ability to mobilize resources for OP II, not only from development partners, but also from increased contributions by its member countries. The recent decision by the Board of Directors (December 2013) to double member countries’ annual contributions from US $5,000 to US $10,000 is a first step in the right direction. 54. The environmental risk to development outcome is low. Since the MTR identified key environmental shortcomings as discussed in Section 2.4, the environmental management system has improved, trainings have been successful, and substantial efforts have been made to enhance the monitoring of sub-projects. 55. The social risk to development outcome is low. It is likely that increasing agricultural productivity and incomes of farming households will improve food security, and the socioeconomic status of the poor in the ASARECA member countries. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 56. The Bank Team worked closely with ASARECA and development partners to prepare the project and ensure harmonization of development partners activities. ASARECA was an established institution with competent staff and with existing processes in place, such as M&E, FM and procurement systems. This made it easier to identify and address the gaps that required attention to enhance its overall performance. The Bank included lessons from ASARECA’s experience pre-MDTF and identified risks and mitigation measures. The original PDO and outcome indicators were clearly linked to its activities (and subsequently refined in line with the Additional Financing). Page 61 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) (b) Quality of Supervision Rating: Satisfactory 57. Supervision missions were conducted jointly with development partners and included relevant specialists in areas such as bio-physics and safeguards to increase the intensity of support when needed, and the team worked together with the ASARECA Secretariat staff and other CGS partners to discuss the findings and produce comprehensive reports with recommendations for follow up. The MTR was a significant and participatory exercise that involved all MDTF donors. Aide Memoires and Implementation Status Reports (ISRs) were substantial and candid in identifying issues. The team is commended for: the follow up in ensuring Procurement and Safeguard Compliance, especially in providing procurement training that improved ASARECA and SG performance, and in identifying lapses in environmental safeguards at the MTR and warranting enhanced compliance; and in refining the RF under the Additional Financing, with indicators that were simple and measurable, including requiring reporting by gender. Activities under the MDTF were also coordinated with interventions by other partners and projects (i.e. EAAPP) 58. ASARECA staff expressed their appreciation of the Bank’s role in administering the MDTF and eliminating reporting to multiple development partners. While staff view the internationally recognized standards set by the Bank as onerous to achieve, they appreciated that training they have received to meet these standards have benefitted them. Finally, the Bank Team has made a notable effort during project implementation, working closely with ASARECA, to identify lessons and areas that need further work to enhance the design and content of the extended MDTF (from 2014-2018) and advance ASARECA’s performance under OP II. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory This reflects the ‘S’ ratings under Quality at Entry and Supervision. 5.2 Borrower Performance (a) Government/Donor Performance Rating: Satisfactory 59. The Government of Uganda (GoU) hosted ASARECA by providing physical space and grounds for constructing its own office. Furthermore, the GoU granted certain diplomatic status to the organization to enable it to recruit and retain well-qualified researchers and support staff from the region. Member countries have paid annual contributions (US $5,000) to ASARECA but unfortunately not all member countries have paid fully or regularly. Member countries need to consider increasing their contributions to strengthen ASARECA’s financial sustainability. The overall planning budget for OP I (with a five-year timeframe from 2008) was US $96 million. ASARECA received US $84.1 million in total by end-December 2013. US $59.19 million came through the MDTF. As described under Section 5.1(b), the successes achieved under the MDTF can be attributed to the efforts of all development partners in providing the bulk of resources for OP I, and in working together, including participating in annual joint missions, to assist ASARECA to achieve its mandate. Development partners have also conducted Page 62 of 63 The World Bank AFCC2-RI-Association for Strengthening Agricultural Research in Eastern and Central Africa (ASARECA) Trust Fund (P112600) detailed evaluations, including USAID (2011) of its support to ASARECA over a ten-year period, and DFID (Project Completion Review, 2014) adding to the knowledge and lessons in terms of enhancing ASARECA’s future performance. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 60. ASARECA has done an outstanding job in implementing the MDTF under OP I. Its leadership in reinforcing the working relationship with a range of stakeholders and implementers, from the Heads of NARIs and the overall NARS, to the beneficiary farmers has been instrumental in making the MDTF a success. ASARECA has not only put in place structures, systems and processes to support the functioning of the Secretariat and sub-projects, but it has also effectively used and refined its systems such as in M&E and capacity building initiatives to consistently improve the delivery and outcomes of its research programs. ASARECA has also responded in a timely manner to comply with Bank procedures, including in accounting and FM, to reduce the risk of fraud and corruption and to ensure that every dollar received is contributing towards the intended beneficiaries (DFID, 2014, p. 25). ASARECA has also taken steps to strengthen the relevance of gender as one of its core businesses. Beneficiary feedback and findings during the ICR mission endorse ASARECA’s success in implementing OP I. 61. The imbalance of ASARECA’s grant funding contribution remains an issue of concern, for those countries with less-participating NARS that receive less support. Some effort has been made to strengthen these NARS through human resource needs assessment, short courses, and mentorship. However; these have been implemented largely through special programs and have not involved the main “business process” of the competitive grant program. There is a clear tension between supporting high quality agricultural research and building capacity of the less competitive NARS (DFID, p. 15). As capacity strengthening is a key function of ASARECA, ASARECA needs to review the CGS mechanism to identify strategies to support organizations in countries that are currently under- represented, without compromising the ‘quality of science’. In OP II, the addition of a criterion that each CGS application must include at least one ‘less-participating’ NARS (although it may be headed by stronger organizations) can help to reinforce these NARS. In this regard, lessons learned during the final years of OPI should be strengthened and applied. However, this will come with added transaction costs given increased mentoring and capacity building efforts. 62. ASARECA has had relatively high overhead and support costs as noted in the external MTR and by DFID (DFID 2014). About 48 percent of the OP I budget was allocated to research project sub-grants, and 7 percent to capacity building for the grantees. The average expenditure to administrative, governance and operational expenditure has been between 25 -28 percent during OP I. Part of the relatively high operational cost is explained by the shift in functions including a stronger knowledge hub management function. However, 28 percent is on the high side compared to 20 percent in similar institutions. ASARECA needs to conduct a critical examination of its cost structure and identify prospects for cost savings, to streamline administrative costs to an acceptable level. (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory 63. This reflects the ‘S’ ratings under Government/Donor and Implementing Agency performance. Page 63 of 63