Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Report Number: ICRR0021749 1. Project Data Project ID Project Name P112684 FARA Trust Fund Country Practice Area(Lead) Western Africa Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-95129,TF-95130 31-Dec-2013 49,139,542.42 Bank Approval Date Closing Date (Actual) 04-Dec-2008 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 49,709,320.00 49,709,320.00 Revised Commitment 49,298,365.01 49,139,542.64 Actual 49,298,364.79 49,139,542.42 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally Fernando Manibog Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Forum for Agricultural Research in Africa (FARA) Trust Fund (TF) Program Document (PD, para 7) was to: "align African agricultural institutions at the national, regional and continental levels with Comprehensive Africa Agriculture Development Programme (CAADP) Pillar IV Framework for Page 1 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) African Agricultural Productivity (FAAP) Principles for effective research, extension, and training and education." The PDO as stated in the Multi Donor Trust Fund Grant Agreement (MDTF, p. 4) was almost identical to the one mentioned in the Program Document, except where underlined, and aimed to: "align' African agricultural institutions at the national, regional, and continental levels with the principles and objectives of the Comprehensive African Agricultural Development Program (CAADP) - Pillar IV, for effective agricultural research, extension, technology adoption, and training and education." The PDO was revised on December 13, 2013 to be as follows: "To strengthen Africa’s capacity for agricultural innovation with purposefully determined outcomes, creating capacity for innovation and enabling environments for implementation (Restructuring paper, p. 8). " This Review will assess the outcome of the program based on a split rating as detailed under section 6. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 13-Dec-2013 c. Will a split evaluation be undertaken? Yes d. Components The project was supported by the following four components: Page 2 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) 1. Networking Support Functions (NSFs) (appraisal estimate: US$10.43 million, additional financing: US$11.66 million, actual cost: US$12.10 million). This component would finance essential continuing functions (ECFs) related to the five NSFs. The ECFs are the core set of activities that each NSF must undertake to realize its outputs and subsequently the Medium-Term Operational Plans (MTOPs) results and CAADP Pillar IV objectives. The ECFs are implemented throughout the MTOP period, and include the establishment of knowledge hubs for the respective NSF’s areas of concern and support for the functioning of these hubs. They also include FARA’s core functions of advocacy, information exchange and facilitation of ARD partnerships. This component would also finance international and local consultancies, goods, and training and workshops. It included five sub-components: 1.1. Advocacy and Resource Mobilization (NSF 1). The aim of this sub-component is to support forum members (the Sub-Regional Organization (SROs), National Agricultural Research System (NARS), and other regional and continental agricultural research and development stakeholders) in establishing appropriate institutional and organizational arrangements for agricultural research and development. This responds to calls by CAADP and FAAP for increased and better harmonized investments in agricultural research and development by national governments and regional and international development agencies/partners. 1.2. Access to Knowledge and Technologies (NSF 2). The aim of this sub-component would be to empower researchers and end users through increasing access to information, learning opportunities, and new technologies in agriculture. 1.3. Regional Policies and Markets (NSF 3). This sub-component aimed to promote and facilitate agricultural policy analyses and market research and support the application of research outputs in policy formulation. It would provide Africa’s policy makers with policy options supported by hard facts and figures and valuable information that will empower the continent’s delegates/representatives involved in international trade and environmental treaty negotiations. 1.4. Capacity Strengthening (NSF 4). This sub-component aimed to ensure that Africa attains the relevant and qualitatively and quantitatively sufficient human and institutional capacity for agricultural innovation to achieve broad-based agricultural productivity, competitiveness and markets. This requires addressing research capacity needs and strengthening the professional development of agricultural scientists so that they are better able to identify, generate and deliver research inputs that meet the needs of their clients, and in particular, poor people. 1.5.Partnerships and Strategic Alliances (NSF 5). This sub-component aimed to facilitate the establishment of mutually beneficial partnerships that draw on the resources and expertise of all FARA stakeholders. Such partnerships create a capacity for agricultural innovation that, together with the other support functions, would bring about the required improvements in the efficacy and impact of African Page 3 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) agricultural research and development. Agricultural innovation involves interaction among multiple actors along the value chain. Such actors include researchers, extension provider development agents, private industry (marketers, processors etc.), policymakers, donors, farmer organizations and consumers. FARA recognizes that advancing innovation and innovation capacity in African agriculture requires mechanisms that facilitate the organization of these actors into dynamic platforms for innovation. 2. Corporate Governance Reform Stream and Program Management (appraisal estimate: US$ 21.6 million, additional financing: US$29.08 million, actual cost: US$12.70 million). This component aimed to facilitate the reform and change management of the internal organization and capability of the FARA Secretariat to effectively implement the MTOP 2008-2012. This included the strengthening of internal management, governance and accountability of the Secretariat to improve its support to the field implementers of CAADP Pillar IV, especially the Sub-Regional Organizations and the National Agricultural Research Systems. This component would finance international and local consultancies, training and workshops, and operating costs. It included the following four sub-components. 2.1. Identifying FARA Board’s Operations and Responsibility. This sub-component would support carrying out of a review FARA Board’s formal relationship with AU-NEPAD to recommend appropriate accountabilities of FARA to AU-NEPAD with regard to its responsibility as lead pillar agency for CAADP Pillar IV, and adoption of these recommendations; carrying out of a study to define FARA Board’s accountability to its stakeholders and how this will be enhanced e.g. accountability to the founders, investors, and the recipients of FARA services, and adoption of the proposed enhancements; carrying out of a study to recommend ways to enhance the FARA Board’s performance, appraisal and contracting of the Executive Director (e.g., though preparation of Terms of Reference, definition of appropriate compensation criteria, etc.), and implementation of these recommendations; and carrying out of a study to recommend measures for ensuring that the FARA Board members have the diversity and appropriate balance of skills necessary to take on board the fiduciary responsibility of FARA, including in financial management, and adoption of these recommendations. 2.2. Developing context specific corporate governance systems. This sub-component would support carrying out of a review of FARA’s Corporate Systems with a view to bringing them in line with international best practices; carrying out of a human resource study of FARA’s job evaluation, job grading, salary structuring, and staff organization in order to improve FARA’s operations; and carrying out of a program to update and enhance recruitment and selection procedures, performance and promotion procedures, conditions of service, health and safety procedures, industrial relations framework, disciplinary/grievance handling and dispute resolution procedures, procurement procedures, risk management procedures, and asset management procedures. 2.3. Facilitating Good Governance and Best Practice at FARA and SRO levels. This sub-component would support carrying out of a program to promote dialogue among FARA and Sub-Regional Page 4 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Organizations board representatives on improving corporate governance and systems, particularly issues covered in sub-component 2.2 above. Provision of technical assistance to further support the governance- related capacity strengthening carried out by CAADP Pillar IV Expert Reference Group.Organizing learning events other activities to promote information sharing between FARA and the Sub-Regional Organizations, particularly in areas outlined in sub component 2.2 above. Conducting an annual user survey of National Agricultural Research Systems and Sub-Regional Organizations to gauge user satisfaction of FARA services. 2.4. Program Management. The MDTF would finance operating costs and FARA General Assembly meetings, the Secretariat’s information and communication services, travel, accommodation and per diem, as well as M&E activities. 3. Research Activities (appraisal estimate: US$13.9 million, additional financing: US$2.72 million, actual cost: US$1.90 million). This component would provide sub-grants to beneficiaries for specific development projects involving research and advocacy in support of NSFs 1 through 4. These activities form part of FARA’s portfolio of essential continuing functions. It would also finance international and local consultancies, goods, training and workshops. 4. Management and Supervision of the Trust Fund (appraisal estimate: US$1.7 million, additional financing: US$1.53 million, actual cost: US$1.90 million). The aim of this component is to manage, administer and supervise the Trust Fund. Revised Components. As part of the December 2013 restructuring, components 1 and 3 were revised as follows: Component 1. The Network Support Functions were consolidated into three Strategic Priorities: Enabling environment for implementation of African agricultural innovation (NSF1, 2, and 3), Integrating Capacities for Change (NSF 4), and Visioning Africa’s agricultural transformation (NSF 5). Component 3. Sub-grants for sub-projects were modified in scope to focus narrowly on financing sub- projects on capacity building, networking, advocacy and training for sub-regional organizations. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total project cost was expected to be US$49.71 million. The actual disbursed amount as reported by the ICR (p. 2) was US$49.30 million. Page 5 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Financing. The project was financed through a Multi Donor Trust Fund (MDTF) established at and administered by the World Bank. The MDTF was expected to receive US$45.90 million in donor contributions. However, only US$33.5 million were committed. In December 2013, an additional financing (AF) in the amount of US$45 million was approved. By the original closing date on December 31, 2013, only US$31 million was disbursed (ICR, p. 11). The total amount disbursed at completion was US$49.30 million (ICR, p. 2). These included US$33.5 million from the original MDTF contributions and US$15.8 million from the AF. Borrower Contribution. This project was fully funded through the MDTF. Dates. The project was approved on December 4, 2008 and became effective nine months later on September 4, 2009. The project had two Mid-term Reviews (MTR), the first was conducted on February 1, 2012. This was in line with the appraisal date where the MTR was expected to be conducted in the third year of implementation (Project Document, para 53). The second MTR was in June 2016. The original closing date was on December 31, 2013. However, the project closed five years later on December 31, 2018. The ICR did not provide an explicit reason for the five year extension. However, the Restructuring Paper stated that on December 13, 2013, "the Bank approved a 5-year extension of the Grant and US$45 million in additional financing (AF) in support of FARA’s MTOP2 (2014- 2018)." The project was restructured twice. The first restructuring was a Level 2 on June 28, 2012 when the amount disbursed was US$20.54 million. This restructuring involved changes to: the Result Framework, the PDO indicators, and grant reallocation. According to the ICR (p. 11) these changes were required to "ensure FARA can continue its work program activities uninterrupted; and (ii) ensure adequate reporting against a more clearly defined performance measurement framework, data collection and ownership of the indicators by FARA." The second restructuring was on December 13, 2013, when the amount disbursed was US$31 million (Restructuring Paper, p. 2). This restructuring included: the approval of US$45 million in Additional Financing (AF), the revision of the Project Development Objectives, amendments to the Results Framework, extension of the loan closing date from December 31, 2013 to December 31, 2018, a change to the Environmental Category rating from B to C, and modifications to component 1 and 3 (Restructuring Paper p. 8). According to the ICR (p. 12) these changes were to "to respond to changing context in African science and agriculture setting (transformation agenda, a widened mandate covering private sector engagement, youth & gender), increased capacity of FARA’s main partners, changes in resource availability and lessons learned during the implementation of MDTF." 3. Relevance of Objectives Rationale The Forum for Agricultural Research in Africa (FARA) is a continent-wide umbrella organization bringing together and forming coalitions of major stakeholders in agricultural research and development in Africa. The FARA Secretariat was created in 2002 and is based in Accra, Ghana. FARA complements the activities of national, international and sub-regional institutions to deliver more responsive and effective services to its Page 6 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) stakeholders. FARA is committed to the implementation of the African Union’s New Partnership for Africa’s Development (AU-NEPAD) Comprehensive African Agriculture Development Programme (CAADP) which sets an ambitious goal of 6% per annum growth for the agriculture sector. A key component of the CAADP vision calls for improving agricultural productivity through enabling and accelerating innovation.The African Union Commission and AU-NEPAD have given the FARA Secretariat the mandate to serve as the lead institution responsible for coordinating CAADP Pillar IV activities (FARA TF Program Document, paras 1,2, and 3). At appraisal, the original objectives were in line with African Union's Comprehensive African Agriculture Development Program (CAADP) that described the African leaders’ collective vision for how to reach a goal of 6% per annum growth for the sector. Objectives were also in line with CAADP Pillar IV which was launched in 2006 as a strategy to support Africa’s agricultural research, technology dissemination and adoption efforts. Objectives were also in line with the Bank's strategy and priorities set in the Africa Action Plan. At completion, the original objectives became less relevant due to a change in the context of African science and agriculture setting (ICR, p. 12). The revised objectives were in line with the World Bank's 2015 Agenda for ending poverty by 2030. The Agenda (p. 7) called for improved agricultural productivity and climate resilience, strengthened links to markets, agribusiness growth, and the need to raise rural non-farm incomes. The revised objectives were also in line with the African Union-NEPAD’s CAADP vision for how to reach the goal of 6% per annum growth for the sector through effective research, extension, and training and education. The original statement of objectives was broad and lacked a connection to the higher-level objective, namely to achieve 6% agricultural growth per annum. The ICR (p. 24) stated that the original PDO was "vague and hard to measure." While the PDO was totally revised at restructuring, the revised PDO statement continued to be broad without a clear connection to the higher-level objective. The ICR (p. 17) correctly noted that the revised PDO was "overly ambitious and hard to measure" because agricultural innovation capacity in Africa is a broad term that involves a wide spectrum of interventions, and these were " mostly not supported by the project." Relevance of objectives is rated substantial for both the original and revised PDO due to the broad statement of objectives, the lack of a connection to higher level objective, and the difficulty in measuring the outcome. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 7 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) OBJECTIVE 1 Objective To align African agricultural institutions at the national, regional, and continental levels with the principles and objectives of the Comprehensive African Agricultural Development Program (CAADP) - Pillar IV, for effective agricultural research, extension, technology adoption, and training and education. Rationale Theory of Change. The theory of change hinged on the need for a continental mechanism to advocate for and manage support for agricultural research and development (ARD) in Africa. The project aimed to achieve its development objectives (PDO) through the following activities: (i) Enhancing FARA Secretariat’s capacity to act as a continental ARD coordinator and project manager; (ii) Advocacy for increased investment into ARD institutions and programs at the country and sub-regional level; (iii) Capacity-building and promotion of Innovation to Impact pathways; (iv) Building of continent-wide knowledge exchange systems and enabling environment; and (v) Supporting regional and national level agencies’ capacity. Expected intermediate outcomes from the above-mentioned activities include: improved governance structure and management systems, development of ARD partnerships and transformation agenda among stakeholders, strengthening continental capacity to respond to agricultural innovation demand, establishing an enabling environment for agricultural investments and implementation of agricultural innovation service. Expected outcomes include: African agricultural institutions at the national and regional levels have aligned their ARD programs with CAADP pillar IV FAAP principles, and strengthening Africa's capacity for agricultural innovation. Higher level outcomes: reduced poverty, improved food security, and enhanced environmental sustainability. Critical assumptions underlying the project included the principle of subsidiarity to guide collaboration among the key partners of FARA, financial support from DPs, sharing of resources for coordinated activities among partners, sustained commitment of partners to the regional agenda and sufficient capacity of National Agriculture Research Systems (NARS) and Sub-Regional Organizations. Overall, the theory of change suffered from weak causal links between the stated activities and the expected outcomes. Also, it was not clear how the achievement of outcomes would contribute to the higher level outcomes. Outputs The outputs below were reported in the ICR (Annex 1) unless referenced otherwise. 1. Strengthening institutional and organizational structures/arrangements for regional ARD: Page 8 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684)  33 country investment plans (target: 25, baseline: none) integrate CAADP Pillar IV elements (research, advisory services, and education - target exceeded). 2. Enhanced access to knowledge and technology for agricultural stakeholders in Africa  6 gender-sensitive continental platforms (target: 6, baseline: 3) on knowledge exchange and technology dissemination for ARD in Africa were established (target achieved).  11 innovations (target: 12, baseline: none) facilitated through technology dissemination platforms (target almost achieved).  29 national information and learning systems (target: 25, baseline: none) on agricultural innovations established (target exceeded).  2,821 stakeholders (target: 2,500, baseline: 200) using gender-sensitive knowledge management tools for knowledge exchange at continental level (target exceeded).  48 country platforms (target: 25, baseline: none) integrated into continental platforms (target exceeded). 3. Strengthening human, organizational and institutional capacity for agricultural innovation in Africa.  7 capacity strengthening initiatives (target: 7, baseline:2) developed and implemented (target achieved).  6 capacity strengthening initiatives (target: 5, baseline: 2) were funded and implemented (target exceeded).  71% of the targeted organizations (target: 80%, baseline: none) showed improved organizational performance (a composite index incorporating changes observed in SWOT factors-target not achieved).  3 agricultural innovations (target: 4, baseline: none) developed by the targeted African National Agricultural Research Systems (NARS) (target almost achieved).  Targeted NARS had 30% women scientists (target: 40%, baseline: 20%) (target not achieved). 4. Improved support for agricultural innovation platforms (IPs) in Africa.  66 functional innovation IPs (target: 51, baseline: none) were established (target exceeded).  27 institutions (Sub-Regional Organizations, National Agricultural Research Systems, farmer organizations) adopted the IP approach (target: 20, baseline none, target exceeded).  14 operational guides (target: 8, baseline: none) for establishing IP were developed (target exceeded).  66 institutions (target:51, baseline: none) (Sub-Regional Organizations, National Agricultural Research Systems, NGOs, private sector, farmers’ organizations) were using the operational guides (target exceeded). Page 9 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684)  11 partnerships and Strategic Alliances (target: 8, baseline: 2) were established reflecting regional and global knowledge and capacity exchange (target exceeded). Outcome The ICR (Box 1) explained that "alignment with FAAP principles refers to ‘Country-Regional Economic Community (REC) compact signing’ which is a process covering advocacy for a planning process (by FARA), key country level stakeholder engagement and consultation, and signing of the compact to show commitment to CAADP-FAAP process. After signing, the second part of the CAADP process entails follow-up stock-taking, identification of national aspirations, and development of a country investment plan (CIP). The CIP part involved establishing guidelines, training, identification of CAADP focal points and facilitation of lessons learned (by FARA) to integrate AIS into the national program. The progress on alignment was measured by three outcome indicators. At the country level through the number of FAAP compliant CAADP country compacts addressing agricultural research, advisory services and higher education. The project supported 30 country compact preparation processes (against the target of 37, target not achieved) through advocacy and stakeholder engagement. At the regional level the project was envisaged to ensure compliance of the Regional Economic Community (RECs) and was expected to have 6 regional compacts signed. However, only 2 RECs were supported (target not achieved). The ICR (p. 14) reported that most of the six RECs were reluctant and did not have the capacity to align their ARD programs with FAAP. Also, due to the overall delays in start-up, implementation, and funding, all alignment centered activities were not fully completed -which affected achievement of the outcome (ICR, p. 14). The achievement rates for country and REC compacts were 81% and 33%, respectively. The third indicator was measuring the percentage of Activity Work Programs (AWPs) implemented on target, on budget and on time. While 118% of AWPs (against a target of 80%) were implemented on target, on budget and on time, the ICR (Annex 1) correctly concluded that this indicator was "a poor choice to measure achievement of outcomes" because it did not "provide evidence on the achievement of outcomes." Therefore, and based on the above-mentioned assessment, the efficacy of this outcome is rated Modest due to the failure in achieving the outcome targets. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective To strengthen Africa’s capacity for agricultural innovation with purposefully determined outcomes, creating capacity for innovation and enabling environments for implementation. Revised Rationale Theory of change. Despite the revision of the PDO, the project relied on the same original theory of change as mentioned above to achieve the revised PDO. Page 10 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Outputs The outputs mentioned under the original objective pertain to the revised objective as well. The outputs below were reported in the ICR (Annex 1) unless referenced otherwise. 1. African agricultural stakeholders determining how the sector should be transformed and establishing the needed collective actions in a gender sensitive manner.  41 countries (target: 15, target exceeded) with Agriculture Research and Development agendas were influenced by foresight studies reflecting access to and use of relevant data for decision-making.  859 functional partnerships and platforms (target: 1000, baseline: 130) for agricultural innovation and trade among African stakeholders and between them and northern and southern partners established reflecting adoption of Agricultural Innovation System (AIS) in practice (target significantly achieved).  1.5 million participants were directly reached (target: 1.7 million, baseline: 1.25 million, target partially achieved) and 15.9 million farmers (target: 22 million, baseline: 15.3 million, target partially achieved) were indirectly reached, 9,062 (target: 8,800, baseline: 1,650, target exceeded) participated or contributed to IPs, consultations, workshops, meetings, 54,133 professionals were trained (target: 70,437. baseline: 4,731, target partially achieved). The ICR (Annex 1, p. 50) explained that the reported data measured reach and participation but not necessarily benefits, and attribution was a challenge. 2. Strengthened and integrated continental capacity responding to stakeholder demands within the agricultural innovation system in a gender sensitive manner.  254 institutions adopted FARA-initiated interventions or mechanisms for identifying, articulating and/or addressing capacity needs (target: 300, target significantly achieved).  136 institutions had their capacity development needs assessed and/or supported (target: 150, target significantly achieved). However, assessment of needs and/or capacity support does not mean an improvement in capacity.  4 functional Community of Practices (CoPs) were established for creating gender sensitive capacities and to address and identify capacity deficits in the design and implementation of ARD programs (target: 3, baseline: 2, target exceeded). 3. Enabling environment for increased AR4D investment and implementation of agricultural innovation systems in a gender sensitive manner. Page 11 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684)  44 countries and RECs had FAAP principles and guidelines embedded in CAADP national and/or regional compacts and Agriculture and Food Security Investments Plans (AFSIPs) (target: 52, baseline: 32, target significantly achieved).  103 information and knowledge products/packages (books, manuals, reports, target: 104, baseline: 63), 87 staff publications (target: 108, baseline: 33), 2,649 volume of information were deposited on the website and social media pages (target: 2,138, baseline: 1,500) and were all made available to stakeholders.  639, 437 people (target: 539,214, baseline: 82,889, target exceeded) were reached with information through continental information and knowledge sharing platforms (websites, publications, visual and social media).  120 platforms used for information delivery and exchange (target: 200, target not achieved).  41% improvement in availability of information and communication technologies (ICTs) (magnitude of ICT speed and capacity performance, reliability of internet access, equity, service quality, cost- effectiveness) amongst targeted NARS institutions (target: 34%, baseline: 11%, target exceeded). 4. Strengthened gender responsive governance and management systems.  Pluralistic decision-making processes (stakeholder representation in governance bodies, program management). This was a yes or no indicator, answer was yes.  Compliance with operational procedures and standards. This was a yes or no indicator, answer was yes.  FARA reserve fund reached US$2.35 million (target: 1.61 million, baseline: US$ 1.0 million).  63% of Activity Work Program (AWP) activities were implemented on time, on target and on budget (target: 100, not achieved). Outcome The outcome included two elements: the outreach/coverage of FARA-led interventions and improved skills and knowledge for innovation among stakeholders. The outreach was assessed through an outcome indicator that measured the percentage increase in number of individuals, groups, organizations directly affected or reached by FARA interventions. The ICR (p. 16) reported that outreach among farmers reached 89%, 82% among professionals and practitioners, and 73% among countries (target: above 90%). The second element of the PDO relates to improved capacity and improved skills and knowledge for innovation, which implies a behavioral change. The outcome was measured by percentage increase in core competencies, capabilities and capacities for innovation among targeted (individual, organizational/inter- organizational and/or institutional) ARD actors. The ICR (p. 16) reported that organizational capacity was improved by 77% (target: above 90%) and change in human capital reached 87% (target: above 90%). The impact of these improvements on innovation was not captured due to the absence of relevant indicators in the RF. The project strengthened capacity for innovation mainly through providing training. However, the ICR (p. 16) reported that "most trainings were of management skills (not core innovation capacity)." This was confirmed by an independent institutional analysis conducted in 2017 on FARA, where 55% of FARA constituency Page 12 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) stated that FARA’s capacity-building interventions need improvement (ICR, p. 17). To improve enabling environment for AIS, the project produced 190 knowledge products (against target of 212) and maintained knowledge sharing (reached 119% of target) and other exchange platforms (60% of target), and 41 countries (target 15) used foresighting studies in development of their ARD agendas. However, the ICR (p. 17) reported that "feedback (from the independent Institutional Analysis, 2017) pointed to limited results with the knowledge management activities." This Review is in agreement with the ICR that the revised PDO was overly ambitious and hard to measure because innovation capacity (in Africa) involves a wide spectrum of interventions, mostly not supported by the project. While the project had some positive achievements, it did not meet its outcome targets and fell short on achieving a number of its output/intermediate outcome targets, and the impact on innovation was not captured. Therefore, and based on the above-mentioned assessment, the efficacy of the post-restructuring outcome is rated Modest. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Pre-restructuring. Efficacy was rated modest because the overall delays in start-up, implementation, and funding negatively impacted the project-supported alignment centered activities-which according to the ICR (p. 14) "were not fully completed." Also, the project did not meet its outcome targets for the two main PDO indicators. Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Post restructuring. Efficacy was rated Modest because the outcome targets were not met, and there was no evidence on improvement of innovation as most training provided by the program was related to management skills. Also, knowledge management activities had limited results. Assessing the post restructuring outcome was challenging because innovation capacity (in Africa) involves a wide spectrum of interventions, mostly not supported by the project. Page 13 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Overall Efficacy Revision 1 Rating Primary Reason Modest Low achievement 5. Efficiency Economic and Financial Efficiency ex ante The FARA Trust Fund Program Document did not include an ex ante economic and financial analysis. ex post  The project ex post efficiency was assessed through two parameters, first, through evaluating overhead costs and relevant comparisons to relevant organizations, and second, through examining resource use efficiency.  Overhead costs. At 44% of the total funding, FARA’s Secretariat cost to total project cost was high. On the other hand, the technical programs received 31% (US$14 million) of the total funding. The FARA secretariat included over 60 permanent staff with FAO level salaries and benefits. This number was reduced to 38 after significant streamlining in 2016. Despite the reduction, the proportion of overhead costs remained high at 69%.  Comparison to relevant organizations. Operating costs for non-profit organizations range from 60 - 80%, leaving 20-40% to overhead costs including salaries. In the private sector and according to 2013 prices, overhead averages 22%, but may reach up to 50% if professional, scientific and/or technical services (including consultants) were provided (ICR, p. 18). By any account, FARA’s overhead costs were relatively high and significantly exceeded the average among relevant organizations.  Resource Use Efficiency. According to the ICR (p. 18) "FARA’s resource use was not efficient during the MDTF phase" and "cost efficiency was not strongly applied." Delays and inefficiencies within FARA were exacerbated by unfamiliarity with the Bank's procedures and the lack of results-based performance culture within FARA. In 2014-2015, the Bank raised significant concerns over high operational costs and lack of adequate value for money for some expenditures. Most stakeholders stated that the FARA MDTF was Fair to Good value for money. To address fiduciary challenges and value for money concerns, FARA attempted to increase the efficiency of resource use through streamlining of secretariat, cost- sharing, and improving cost efficiency.  Overall, efficiency is rated Modest due to two main reasons. First, many of the project interventions incurred the associated costs, but did not contribute to achieving the PDO outcome. Second, the lack of efficiency in resource use which was reflected in the relatively high overhead costs. Administrative and Institutional Efficiency Page 14 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) The original closing date was on December 31, 2013. However, the MDTF closed five years later on December 31, 2018. The ICR did not provide an explicit reason for the five year extension. However, the Restructuring Paper stated that on December 13, 2013, "the Bank approved a 5-year extension of the Grant and US$45 million in additional financing (AF) in support of FARA’s MTOP2 (2014- 2018)." The project suffered from various delays including: in start-up, implementation, and funding. This affected the achievement of the project's outcome because all alignment centered activities were not fully completed (ICR, p.14). Also, implementation and reporting both suffered from delays due to capacity limitations (ICR, p. 21). In addition, FARA's open ended staff contracts and lack of time-bound or results-based performance system contributed to implementation delays. The ICR (p. 29) reported that frequent changes at the Task Team Leadership towards the end of the MDTF "was not conducive to quality supervision and operational support" which contributed to disbursement delays in 2017-2018 time period. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original PDO Relevance of Objectives is rated Substantial while both Efficacy and Efficiency are rated Modest. The PDO was broad and difficult to measure. Efficacy was rated modest because the overall delays in start-up, implementation, and funding, all negatively impacted the project-supported alignment centered activities-which according to the ICR (p. 14) "were not fully completed." Also, the project did not meet its outcome targets for the two main PDO indicators. Efficiency was modest because many of the project interventions claimed financial resources, but did not contribute to achieving the PDO, and overhead costs were relatively high. Revised PDO Relevance of Objectives is rated Substantial while both Efficacy and Efficiency are rated Modest. The revised PDO was broad and difficult to measure. Efficacy was rated Modest because the outcome targets were not met, and there was no evidence on improvement of innovation as most training provided by the program was related Page 15 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) to management skills. Also, knowledge management activities had limited results. Efficiency was modest because many of the project interventions entailed costs, but did not contribute to achieving the PDO and overhead costs were relatively high. Split Rating: The amount disbursed at restructuring was US$31.77 million against a total of US$49.30 million which represents 64.4%. Pre-restructuring and Post restructuring outcome ratings were both Moderately Unsatisfactory, which corresponds to a 3 on 1 to 6 scale. Weighted outcome: 3 X 0.64 + 3 X 0.36 = 1.92 +1.08 = 3 which corresponds to Moderately Unsatisfactory. The Overall outcome rating is Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The ICR (Table 3) stated several risks that could impact the sustainability of the development outcome. The following are emphasized:  Sustainability of FARA institution. To be sustainable, FARA must identify its value added, mandate, and ownership, serve as a lean umbrella organization with a clear strategic focus, accountability to its constituency, and a resource mobilization strategy. FARA often functioned as a parallel competitor to Sub-Regional Organizations. However, FARA also faced competitors at the continental level including from the African Agricultural Technology Foundation (AATF) funded by several donors, Alliance for a Green Revolution in Africa (AGRA) funded by Bill and Melissa Gates Foundation (BMGF), the Food, Agriculture and Natural Resources Policy Analysis Network (FANRPAN) funded mostly by BMGF, and the African Union Development Agency funded by JICA.  Alignment of countries and regions with the Framework for African Agricultural Productivity (FAAP)-Comprehensive African Agricultural Development Programme (CAADP). More than forty countries and Sub-Regional Organizations have adopted FAAP principles in their country's National Agriculture Innovation Projects (NAIPs). Some are already taking steps towards integrating the Science Agenda for Agriculture in Africa (SA3) and developing a second stage NAIPs. However, the SRO commitment to S3A is still lacking. Page 16 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684)  Capacity of African agricultural innovation system (AIS). While FARA and SROs were able to collaborate well on selected initiatives e.g. Innovation platforms (IPs), their collaboration suffered from limited trust and competition. FARA and the National Agricultural Research Institutions (NARIs) level collaboration was smoother and centered on CAADP-FAAP, IPs and capacity-building. While FARA has raised the profile of AIS stakeholders and supported change, the continental level coordination is still lacking. A coordinated effort is required among governments, core AIS stakeholders (including private sector) and development partners to invest in priority areas of AIS, including enabling infrastructure and regulatory matters (such as seed and fertilizer sector) that currently constrain technology adoption, private sector engagement and value-added activities 8. Assessment of Bank Performance a. Quality-at-Entry At the request of FARA, the World Bank and FARA’s group of Development Partners (UK’s Department for International Development-DfID, Canadian International Development Agency-CIDA, Netherlands, and the European Commission) established a recipient-executed Multi Donor Trust Fund (MDTF) for the World Bank to facilitate joint-financing and to assist the further institutional development of FARA. The Bank also provided FARA with capacity building support to develop and implement its programs. The Bank role was key to convene with core developing partners and relevant African institutions to galvanize support and coordinate activities. However, Quality at Entry suffered a number of significant shortcomings. First, according to the ICR (p. 20) Trust Funds were not required to develop a PAD. This negatively impacted Quality at Entry because the lack of a fully developed PAD contributed to undefined objectives and incomplete components- especially for component 3 (ICR, p. 20). Second, the roles of FARA and those of the sub-regional and national stakeholders were not clearly defined which created confusion among stakeholders. Third, the Theory of Change / Results Framework demonstrated poor causal links between the suggested activities and the desired outcomes, and the original objectives of the MDTF were broad and hard to measure. Fourth, M&E suffered from design weaknesses with an "incomplete understanding of results- based M&E (ICR, p. 20)." Fifth, several risks were underestimated or not properly identified at the appraisal stage including donor commitment to financing, institutional sustainability, capacity of the implementing agencies including M&E capacity (ICR, p. 21). Based on the above-mentioned information, it is evident the Quality at entry suffered from significant shortcomings. Therefore, Quality-at-Entry is rated Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Page 17 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Over the implementation period, the project had twenty supervision missions, two Mid Term Reviews (MTRs) and two restructurings. An informal Advisory Committee consisting of four Development Partners (UK’s Department for International Development-DfID, Canadian International Development Agency-CIDA, Netherlands, and the European Commission) and the Bank met initially bi-annually and contributed to the quality of the MDTF, including reviews of science agenda, recruitment, M&E, and restructuring of the MDTF . According to the ICR (p. 27) the informal Advisory Committee was "instrumental in discussing the conditions, development and financing" of the MDTF. Despite limited implementation progress in the first two years of the project, the corresponding ISRs ratings for this period were unrealistic (ICR, p. 27). At the time of the second restructuring, the project financial risk was high and there were significant challenges with resource mobilization. Despite these challenges, an additional US$45 million of grant financing was requested and approved. After the second restructuring the Bank's supervision and knowledge sharing became focused on streamlining FARA's secretariat and its activities. M&E activities could have benefited from more attention by the Bank. In 2010, FARA made informal revisions to the Results Framework and indicators. The ICR (p. 27) correctly noted that the Bank should have advised FARA to carry out a formal restructuring of the project. The entire RF was again informally changed in 2016. This made tracking milestones and assessing achievements difficult due to the presence of two RFs. To remedy this situation, the Bank initiated a third restructuring in 2017 to formally approve the revised RF. However, this activity was dropped because the project was coming to an end (ICR, p. 27). Quality of Supervision suffered from significant shortcomings, most notable was the failure to address M&E weaknesses that plagued the project throughout implementation. Also, the two restructurings could have been used better to address revisions to the RF especially after revising the PDO. The approval of an additional US$45 million in additional financing is questionable given the financial and resource mobilization challenges, and the project's overall poor performance, Therefore, Quality of Supervision is rated Moderately Unsatisfactory and the overall rating of Bank performance is Moderately Unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Page 18 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) The overall monitoring and evaluation would be under the responsibility of FARA. However, the Project Document lacked details on M&E arrangements.  The original PDO was to be assessed through three outcome indicators: number of national and regional institutions supported by FARA that reflect FAAP principles in their ARD programs, percent of NARS and SROs that indicate satisfaction with FARA’s contribution to the implementation of CAADP Pillar IV process, and percent of results of activities achieved (PD, Annex 1). However, the ICR (p. 8) stated that the PDO was to be assessed through four outcome indicators (the same three mentioned in the PD) and another indicator that assessed the development and implementation of the strategy for strengthening national and regional institutional and program capacities for ARD.  The first two outcome indicators mentioned in the PD were relevant and linked to the PDO, however, the third was a poor choice because it did not provide any assessment of outcomes.  The RF included six intermediate outcome indicators to assess activities under components 1 and 2. However, the RF lacked any indicators to assess components 3 and 4. Most indicators were quantifiable and according to the ICR (p. 24) "all indicators had baselines and targets."  Overall, M&E design was weak with an incomplete RF that suffered from misalignment between outcome and outputs (ICR, p. 24). b. M&E Implementation Overall, M&E implementation was challenging. Recruiting an M&E expert was difficult. Data collection was negatively impacted because data providers were not clear on how to collect data as they were not consulted on the formulation of indicators. In addition, implementing partners had limited capacity which hindered M&E activities. M&E reporting until mid of 2012 was against PDO and intermediate results indicators that were unofficially revised in 2010. However, these were officially adopted after the formal restructuring in June 2012. In 2016, FARA made informal changes to M&E reporting to accommodate financial shortfall and the change of focus by the African Union Heads of state. Accordingly, FARA adjusted its focus, revised the entire RF and reorganized and renamed the components "without the consent of the WB" (ICR, p. 22). This negatively affected M&E and created confusion because the project targets were tracked against 2012 and the 2016 RFs. M&E implementation weaknesses were compounded by the implementing agency's lack of understanding of results-based M&E, and the confusion over the roles of FARA and those of the sub- regional and national stakeholders. c. M&E Utilization According to the ICR (p. 25), "M&E data was used in planning, in reporting achievements and in development of resource mobilization strategy." However, utilization was probably limited due to design weaknesses, implementation challenges and frequent changes to the Results Framework. The overall rating of M&E Quality is Modest. M&E design at entry suffered from significant shortcomings, and implementation was weak. The frequent changes to the RF (twice informally and twice Page 19 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) formally) resulted in "significant confusion on M&E implementation and challenges in tracking milestones (ICR, p. 25)." Utilization was consequently undermined. M&E Quality Rating Modest 10. Other Issues a. Safeguards The Trust Fund was initially rated as Environmental Category B, since the components were expected to have relatively limited environmental impact. The following policies were triggered: Environmental Assessment, Pest Management, Forests and International Waterways. The project was expected to provide technical leadership and guidance for agricultural technology generation and adoption in Africa. Impacts of these programs could include onsite effects such as soil degradation (e.g. salinization, loss of organic matter), nutrient depletion, and soil erosion; offsite effects such as groundwater depletion, agrochemical pollution, loss of local biodiversity, soil erosion downstream effects, hydrological change, and pasture degradation; and global effects such as greenhouse gas emissions, animal diseases, loss of crop genetic diversity, reduced carbon sequestration, and loss of biodiversity (PAD, paras 71 and 73). In December 2013, as part of the restructuring of the project, the Environmental Category was changed from B to C. No safeguards policies were triggered since the envisioned activities were not implemented. Furthermore, the Trust Fund was not expected to undertake any activities that would require physical or civil works (Restructuring Paper p. 8). b. Fiduciary Compliance Financial Management. According to the ICR (p. 26) "the project complied fully with the financial covenants of regular submission of acceptable quarterly interim financial reports." However, the 2014 audit report highlighted some minor internal control lapses. In 2015, the Bank carried out a detailed Financial Management and Value for Money audit covering the period from 2012 through 2014. The Review raised some key issues including: "concerns expressed by the Bank on the need for more scrutiny and fiduciary oversight of the use of project funds generally and particularly regarding expenditure of the ED’s Office; submission of a formal response from FARA to the Bank advising how it intends to address the recommendations of the report; and refund of ineligible expenditures (ICR, p. 26)." The Bank also issued an invoice to FARA to refund the ineligible expenditure identified by the review. FARA officially responded to the recommendations of the review by August 2016 and a full refund of the ineligible expense was processed by the Bank on October 5, 2016. Page 20 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Procurement. Procurement suffered from limited staff capacity and unfamiliarity with the Bank's procurement procedures (ICR, p. 23). This in turn contributed to implementation delays. Procurement performance improved upon hiring a procurement officer following the insistence of the Bank's supervision team (ICR, p. 30). However, recruiting experienced personnel was often challenging. Also, FARA staff saw their contracts as open-ended rather than time or results-bound. This further contributed to implementation delays (ICR, p. 23). c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Relevance of objectives is rated Substantial; overall efficacy prior to and after restructuring are Moderately Moderately Outcome both rated Modest; efficiency is Satisfactory Unsatisfactory Modest. Based on these sub- ratings, overall outcome is Moderately Unsatisfactory. Quality at Entry and Quality of Moderately Moderately Bank Performance Supervision are both rated Satisfactory Unsatisfactory Moderately Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR included fifteen lessons and recommendations. The following three are emphasized with some adaptation of language:  Quality supervision and operational support needs stable Task Team leadership. High Task Team Leader turn-over at the end of the MDTF was not conducive to quality supervision and operational support. Significant delays with disbursement (2017-2018) partly arose from these changes. Page 21 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684)  A rigorous M&E design is needed at entry combined with active supervision and adequate implementation capacity to ensure capturing the project's outcomes. The M&E set-up was complex whereas capacity for M&E was limited. Effort was needed to assess the capacity and provide a minimum level of understanding of results-based M&E.  An exit strategy is needed to ensure sustainability beyond donor support. FARA’s size was overestimated and it eventually became an isolated large organization with high levels of salaries and at times superfluous expenses, requiring significant resources to stay functional. FARA became to some extent a victim of the fluctuating donor priorities, both Developing Partners and the constituency questioning value for money. FARA has not been able to design a sustainable exit strategy from continued donor support. The following lesson is emphasized by IEG:  Regional projects need rigorous background analysis and studies to support an adequate project design and implementation arrangements, and hence ensure a higher chance of achieving the desired outcomes. Regional projects are challenging to coordinate and require a clear understanding of the conditions on the ground. Also, such projects need realistic PDOs that could potentially be achieved within the implementation time-frame and available funding. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain An assessment will be helpful to verify the project's outcomes on the ground given the divergence between IEG's ratings in this Review and the ICR ratings. Also, to derive lessons of potentially high value regarding the serious challenges in designing, implementing and measuring the outcomes of multi-country projects, which involve mostly qualitative outcome indicators, as well as complex institutional layers and networks across the country stakeholders, thus raising issues of causality and attribution. 14. Comments on Quality of ICR Quality of evidence. M&E performance was poor in general and this in turn negatively impacted the quality of evidence. In a further communication, the project team explained that the team used a number of previous analysis that looked into institutions, capacity-building, satisfaction, use and access to knowledge to overcome M&E weaknesses. Page 22 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review FARA Trust Fund (P112684) Quality of Analysis. The linking between evidence and findings was challenging given the poor causal links between activities and intended outcomes. This was further undermined by poor M&E. Extent to which lessons are based on evidence and analysis. Lessons were based on the project experience; however, they were formulated more as recommendations. Results Orientation. Discussion on outcomes was undermined by the poor linking between the PDO and the supported activities. Also, there are concerns on the accuracy of the M&E data. Internal Consistency. Various parts of the ICR were logically linked and integrated. Consistency with guidelines. The assigned outcome ratings in the ICR were generous and provided an unrealistic assessment of the program. Conciseness. The ICR provided thorough coverage of the implementation experience and candidly reported on shortcomings. However, reporting on outcomes was a challenge given the weak M&E. The ICR also included numerous acronyms some of which were not reflected in the abbreviation and Acronyms table, for example: ICTs, AFSIPS, among others. Overall, the ICR Quality is rated Substantial, despite some minor shortcomings. a. Quality of ICR Rating Substantial Page 23 of 23