Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR 137111-BR IMPLEMENTATION COMPLETION AND RESULTS REPORT ON LOANS IBRD 77730 and 82000 IN THE TOTAL AMOUNT OF US$139.5 MILLION TO THE STATE OF RIO DE JANEIRO WITH THE GUARANTEE OF THE FEDERATIVE REPUBLIC OF BRAZIL FOR THE RIO DE JANEIRO SUSTAINABLE RURAL DEVELOPMENT PROJECT MAY 30, 2019 Agriculture Global Practice Latin America and Caribbean Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS Exchange Rate Effective November 30, 2018 Currency Unit = Brazilian Real (BRL) BRL 1 = US$0.258 US$1 = BRL 3.86 FISCAL YEAR July 1 - June 30 ABBREVIATIONS AND ACRONYMS AF Additional Financing BCR Borrower Completion Report COGEM Micro-catchment Development Committee (Comite Gestor das Microbracias) CPF Country Partnership Framework CPS Country Partnership Strategy DRM Department of Mineral Resources (Departamento de Recursos Minerais) EMATER-Rio State Rural Extension Agency (Empresa de Assistencia Tecnica e Extensao Rural do Estado do Rio de Janeiro) EMBRAPA Brazilian Agriculture Research Enterprise (Empresa Brasileira de Pesquisa Agropecuaria) ESS Economic Sustainability System FAERJ Federation of Agriculture, Livestock and Fisheries of the State of Rio de Janeiro (Federacion da Agricultura, Pecuaria e Pesca do Estado do Rio de Janeiro FAO Food and Agricultural Organization FAPERJ State Research Support Foundation (Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro FM Financial Management GAP Good Agricultural Practice GDP Gross Domestic Product GEF Global Environment Facility GHG Greenhouse Gas ICR Implementation Completion and Results Report IFR Interim Financial Report IRR Internal Rate of Return ISP Institutional Sustainability Plan ISR Implementation Status and Results Report LA Loan Agreement M&E Monitoring and Evaluation MDC Micro-catchment Development Committee (Comitē Gestor da Microbracia) MDP Micro-catchment Development Plan (Plano Executivo da Microbracia) MIS Management Information System NGO Nongovernmental Organization NNWF North and Northwestern Fluminense Region NPV Net Present Value PAD Project Appraisal Document PDO Project Development Objective PEM Micro-basin Executive Management Plan PESAGRO State Agriculture Research Enterprise (Empresa de Pesquisa Agropecuaria do Estado do Rio de Janeiro) PID Individual Development Plan PIU Project Implementation Unit (Secretaria Executiva do Projeto) PMU Project Management Unit Rio GEF Sustainable Integrated Ecosystem Management in Production Landscapes of the NNWF GEF Project - Rio de Janeiro SAFF System for Physical and Financial Monitoring (Sistema de Acompanamento Fisico Financiero) SEAPEC State Secretariat of Agriculture and Fisheries SEAPPA State Secretariat of Agriculture, Livestock, Fisheries and Supply (Secretaria de Estado de Agricultura, Pecuaria, Pesca e Abastecimento) SIAFE-Rio Integrated System for Budget, Financial and Accounts Management ( Sistema Integrado de Gestão Orçamentária, Financiera e Contábil do Rio de Janeiro) SoRJ State of Rio de Janeiro UENF North Fluminense State University (Universidade Estadual do Norte Fluminense) Regional Vice President: Axel van Trotsenburg Country Director: Paloma Anos Casero Senior Global Practice Director: Jurgen Voegele Practice Manager: Preeti S. Ahuja Task Team Leader(s): Maurizio Guadagni ICR Main Contributor: Tomas Rosada Villamar TABLE OF CONTENTS DATA SHEET ............................................................................................................................. I I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES.................................................................. 1 A. CONTEXT AT APPRAISAL .........................................................................................................1 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................4 II. OUTCOME ........................................................................................................................... 8 A. RELEVANCE OF PDOs ..............................................................................................................8 B. ACHIEVEMENT OF PDOs (EFFICACY) ........................................................................................8 C. EFFICIENCY ........................................................................................................................... 14 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 16 E. OTHER OUTCOMES AND IMPACTS ......................................................................................... 17 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ........................................... 21 A. KEY FACTORS DURING PREPARATION ................................................................................... 21 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 22 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ............... 24 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 24 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 26 C. BANK PERFORMANCE ........................................................................................................... 28 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 31 V. LESSONS AND RECOMMENDATIONS ..................................................................................... 32 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ................................................................. 34 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION .................................. 51 ANNEX 3. PROJECT COST BY COMPONENT ................................................................................ 53 ANNEX 4. EFFICIENCY ANALYSIS ............................................................................................... 54 ANNEX 5. IMPACT EVALUATION: METHODOLOGY ...................................................................... 59 ANNEX 6. IMPACT EVALUATION: MAIN RESULTS ........................................................................ 62 ANNEX 7. SPLIT ASSESMENT: PERFORMANCE OF PDO INDICATORS .............................................. 73 ANNEX 8. LIST OF PUBLICATIONS FINANCED UNDER PARTICIPATORY RESEARCH 2010–2018 ............ 77 ANNEX 9. BORROWER COMMENTS .......................................................................................... 83 ANNEX 10. SUPPORTING DOCUMENTS ..................................................................................... 84 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P101508 Rio de Janeiro Sustainable Rural Development Project Country Financing Instrument Brazil Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency State Secretariat of Agriculture, Livestock, Fisheries, and State of Rio de Janeiro Supply (SEAPPA) Project Development Objective (PDO) Original PDO The PDO is to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower's territory, thus contributing to the higher-order objective of increasing small-scale farming productivity andcompetitiveness in those areas. Revised PDO The objective of the Project is to increase the adoption of integrated and sustainable farming systems approaches in specificareasof the Borrower's territory and help re-establish an agricultural productive environment in areas of the Serrana Regionaffected bythe January 2011 natural disaster, thus contributing to the higher-order objective of increasing small-scale farmingproductivity and competitiveness in those areas. i The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 39,500,000 38,735,596 38,735,596 IBRD-77730 100,000,000 60,000,000 50,518,222 IBRD-82000 Total 139,500,000 98,735,596 89,253,818 Non-World Bank Financing 0 0 0 Borrower/Recipient 39,519,000 0 0 Total 39,519,000 0 0 Total Project Cost 179,019,000 98,735,596 89,253,818 KEY DATES FIN_TABLE_DAT Approval Effectiveness MTR Review Original Closing Actual Closing 10-Sep-2009 08-Mar-2010 11-Apr-2015 30-Nov-2015 30-Nov-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 27-Oct-2011 9.58 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Change in Financing Plan Other Change(s) 01-Oct-2012 18.83 Additional Financing 22-May-2013 20.83 Change in Components and Cost Reallocation between Disbursement Categories Change in Institutional Arrangements 29-Jun-2015 42.31 Change in Components and Cost Reallocation between Disbursement Categories Change in Institutional Arrangements 01-Sep-2017 56.16 Change in Implementing Agency Change in Results Framework Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories ii The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 29-Dec-2009 Satisfactory Satisfactory .33 02 28-May-2010 Satisfactory Satisfactory .28 03 30-Jun-2010 Satisfactory Satisfactory 3.09 04 02-Mar-2011 Moderately Satisfactory Moderately Satisfactory 3.09 05 14-Dec-2011 Moderately Satisfactory Moderately Satisfactory 12.51 06 30-Jun-2012 Satisfactory Satisfactory 17.97 07 16-Jan-2013 Moderately Satisfactory Moderately Satisfactory 21.11 08 16-Sep-2013 Moderately Satisfactory Moderately Satisfactory 23.83 09 03-Apr-2014 Moderately Satisfactory Moderately Satisfactory 27.10 10 08-Nov-2014 Moderately Satisfactory Moderately Satisfactory 36.04 Moderately 11 18-Jun-2015 Moderately Unsatisfactory 41.58 Unsatisfactory Moderately 12 31-Dec-2015 Moderately Unsatisfactory 46.40 Unsatisfactory Moderately 13 26-Jun-2016 Moderately Unsatisfactory 53.41 Unsatisfactory Moderately 14 21-Dec-2016 Moderately Unsatisfactory 53.41 Unsatisfactory 15 03-Jun-2017 Unsatisfactory Unsatisfactory 53.41 Moderately 16 27-Dec-2017 Moderately Unsatisfactory 66.46 Unsatisfactory Moderately 17 25-Jun-2018 Moderately Unsatisfactory 74.98 Unsatisfactory 18 23-Dec-2018 Moderately Satisfactory Moderately Satisfactory 89.43 iii The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) SECTORS AND THEMES Sectors Major Sector/Sector (%) Agriculture, Fishing and Forestry 100 Agricultural Extension, Research, and Other Support 2 Activities Public Administration - Agriculture, Fishing & Forestry 21 Other Agriculture, Fishing and Forestry 77 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Social Development and Protection 50 Social Inclusion 50 Participation and Civic Engagement 50 Urban and Rural Development 17 Rural Development 17 Land Administration and Management 17 Environment and Natural Resource Management 34 Renewable Natural Resources Asset Management 34 Biodiversity 17 Landscape Management 17 ADM STAFF Role At Approval At ICR Regional Vice President: Pamela Cox Axel van Trotsenburg Country Director: Makhtar Diop Paloma Anos Casero Senior Global Practice Director: Ethel Sennhauser Juergen Voegele Practice Manager: Mark R. Lundell Preeti S. Ahuja Task Team Leader(s): Alvaro Juan Soler Bavosi Maurizio Guadagni ICR Contributing Author: Tomas Ricardo Rosada Villamar iv The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) v The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of project preparation and appraisal (August 2009), Brazil had undergone a period of significant poverty and inequality reduction. Moderate poverty fell from 32.9 percent in 2002 to 21.6 percent in 2008, while the Gini index also fell from 0.59 in 2002 to 0.55 in 2008 (Country Partnership Strategy [CPS] 08–11). While remaining relatively resilient during the 2008 financial crisis and recovering swiftly in 2009, economic growth slowed from the precrisis levels of 4–5 percent to 2 percent after 2009. Growth in the years after 2003 was largely based on favorable external conditions, credit-fueled consumption, and an expanding labor force. Fast employment creation was the driver of poverty reduction, while an expansion of targeted social programs helped reduce extreme poverty. 2. Agriculture in the State of Rio de Janeiro (SoRJ) is more important than would normally be expected in an urbanized state. At appraisal, around 25 percent of the state’s gross domestic product (GDP) was derived from agriculture and agro-industrial activities, generating approximately 40 percent of rural employment. More than 60 percent of total state land was dedicated to agricultural activities, most of it in three administrative regions holding more than one-half of the state’s rural population: the North and Northwestern Fluminense region (NNWF), and Serrana regions (PAD 2009). 3. Despite its importance, the agriculture sector faced fundamental challenges. These included low productivity, limited market links in the face of high demand for agricultural products (Metropolitan area of Rio de Janeiro and smaller urban centers), degradation of the natural resource base, rural poverty, and low capacity of agricultural producers to respond to market demands. These factors were associated with fragile organization of farmers, their widespread use of inefficient and unsustainable agricultural practices, poor infrastructure, the incipient nature of regional industrialization, markets and processes of agro-industrialization, and limited scope of public policies in rural areas. 4. The World Bank was already active in rural areas of southern and southeastern Brazil, supporting capacity building at the local level for participatory rural development and environmental management practices. In the years before the approval of the Rio de Janeiro Sustainable Rural Development Project (“Rio Rural”), the need became evident to focus on interventions within a market-oriented approach on agricultural development, calling for new projects to build on the existing productive and social base to enable a more sustainable development impact. The project built on the approaches and institutional structures established under the Integrated Management of Agroecosystems Project in Hydrographic Micro-catchments of the NNWF Project (“Rio Rural GEF”) to provide support needed to address the main challenges of the rural sector in the SoRJ. 5. Alignment with the Government’s strategy. The project responded directly to the Government’s higher-level objective of increasing productivity and competitiveness in the small-scale farming sector while improving natural resources management. The state demonstrated commitment and support for local-level capacity building for participatory rural development and environmental management—all central pillars of rural operations in Brazil. 1 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 6. The project drew directly on the Rio de Janeiro Sustainable Integrated Ecosystem Management in Production Landscapes of the NNWF GEF Project (“Rio GEF”), which became effective in 2006 and closed in 2011, and on the accumulated knowledge of other World Bank-supported operations in the northeastern and southeastern regions of the country. The design of the project reflected the lessons of those experiences. 7. Rationale for Bank support. The project was anchored in the World Bank’s CPS 08–11 Report No. 42677. By strengthening farming systems, the project was expected to contribute primarily to the third pillar of the CPS: a more competitive Brazilian economy. Main activities were related to better linking small farmers to markets, promoting education for innovation and growth, and improving governance by strengthening the public management of the rural sector. Theory of Change (Results Chain) 8. The project was approved before presentation of a Theory of Change in the Project Appraisal Document (PAD) became mandatory, and consequently, the PAD did not contain a diagrammatic representation. With the objective of clarifying the World Bank team’s thinking during the period of PAD preparation leading to approval, Figure 1 presents a reconstruction of the prevailing Theory of Change as interpreted by this Implementation Completion and Results Report (ICR) and reconstructed based on the implicit results chain described in the PAD. Figure 1. Theory of Change 2 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) Project Development Objectives (PDOs) 9. According to the Loan Agreement (LA), the PDO of the project was “to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower’s territory, thus contributing to the higher-order objective of increasing small-scale farming productivity and competitiveness in those areas.” Key Expected Outcomes and Outcome Indicators 10. Key expected outcomes. These were (a) to strengthen the longer-term impact on sustained agricultural productivity at a larger scale, (b) to improve the efficiency of rural production processes, (c) to enhance market links, and (d) to improve the ability of public institutions to adapt to the evolving demands of the rural sector.1 11. Outcome indicators were the following:  At least 50 percent of small farmers in targeted areas transitioned towards more productive farming systems2  Improved product quality in at least 50 percent of beneficiaries receiving investment support as measured by: (i) Number of beneficiary farmers adopting Good Agricultural Practices (GAP) (ii) Number of small farmers or enterprises certified3 (iii) Number of agro-processing and artisanal enterprises adding value  Improved market access by at least 10 percent of beneficiaries receiving investment support as measured by their inclusion in (or with improved links to) at least one value chain  At least 50 percent of the targeted small farmer agricultural lands under improved production systems  Length (km) of tertiary roads restored and maintained 12. Targeted beneficiaries. The project targeted an estimated 37,000 small-farming families (some 150,000 people in total) in the SoRJ. This corresponds to roughly 30 percent of the total rural population in the state. The target population primarily resided in the north (9 municipalities), northwest (13 municipalities), and Serrana (14 municipalities) administrative regions, covering approximately 23,000 1 See PAD (2009), paragraph 21, page 6. 2 “Improved production systems” were defined as those that resulted in sustainably better agroforestry, crop, or livestock quality and yields. For example, an improved food crop production system could be associated with the introduction of Conservation Agriculture (through adoption of crop rotations, minimal soil disturbance—zero or minimum tillage—and permanent soil cover) and GAPs. 3 The project would support technical and financial assistance to farmers or enterprises to be certified in (a) organic agriculture; and (b) production of certified forestry products. Certification of organic products of farmers or their enterprises adopting organic farming would be provided by the Association of Organic Farmers of Rio de Janeiro. 3 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) square kilometers (53 percent of the total area of the state). Participating institutions would also benefit from capacity building, enabling them to more effectively meet rural sector demands. Components 13. Component 1: Supporting Rural Production and Competitiveness (estimated total cost US$66.1 million of which IBRD US$32.6 million and Borrower US$15.4 million). This component provided assistance to rural beneficiaries to implement changes in rural production processes through (a) preinvestment activities to strengthen organization and capacity for agricultural productivity; and (b) investments to implement demand-driven activities (subprojects) through the provision of grants aimed at improving sustainable and productive farming systems (productive subprojects), compliance with environmental regulations and adoption of agroecological and environmentally sound practices (environmental subprojects), and erosion control and rehabilitation and maintenance of rural roads (rural roads subprojects). 14. Component 2: Strengthening Institutional Frameworks (estimated total cost US$5.2 million of which IBRD US$2.9 million and Borrower US$2.2 million). This component aimed at improving the Borrower’s institutional frameworks supporting market-driven agricultural development by (a) strengthening rural institutions and coordination mechanisms through capacity building for the Borrower’s agencies, providing better services and coordination with other public and private sector stakeholders through implementing specific activities (institutional subprojects) identified in an institutional sustainability plan (ISP), and contributing to the implementation of a national policy in support of territorial development; (b) improving public and private financial support mechanisms through the enhancement of links between the supply and the demand of financial resources for sustainable rural development activities; and (c) undertaking participatory research to establish a new and effective operational system (the Sustainable Services Research Network System) to conduct agriculture-related research and induce innovation. 15. Component 3: Project Coordination and Information Management (estimated total cost US$7.6 million of which IBRD US$3.8 million and Borrower US$3.8 million). This component supported the Borrower’s overall project management and coordination functions, including monitoring and evaluation (M&E), as well as dissemination of key sustainable rural development information by financing (a) project coordination through the strengthening of the organizational and operational structure of the Project Implementation Unit (Secretaria Executiva do Projeto, PIU); and (b) information management through the development and implementation of a management information system (MIS) that ensured widespread access and adequate information flows to impact stakeholder decision making in support of sustainable rural development as well as through the promotion of the use of digital and other information technology tools among project stakeholders and beneficiaries. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 16. The project was implemented over a period marked by a series of external events—a major flood that caused 850 deaths, a drought affecting the Serrana region, strikes by the staff of the implementing agency, a fiscal crisis with high inflation, currency devaluation, and judicial power seizures of the Designated Account —which affected implementation and forced the Borrower and the World Bank to adjust project design and implementation arrangements. The following table summarizes the project’s timeline and changes. 4 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) Table 1. Main Features of Project Restructurings Table 1. Main features of project restructurings Project restructurings Timeline 2009 (August) 2011 (September) 2012 (October) 2017 (September) 2018 (November) Level 1 Restructuring Level 2 Restructuring Original loan Additional Financing of Partial cancellation of Event Natural disaster Project closing (USD39.5 million) USD100 million USD40 million Change in PDO to include the PDO No changes to PDO No changes to PDO No changes to PDO emergency area (Serrana region) 3 PDO indicators (2 original PDO indicators were PDO indicators 5 PDO indicators 3 PDO indicators 3 PDO indicators 3 PDO indicators downgraded to intermediate indicators) Reduction of original targets and inclusion of new targets at Project targets intermediate level related to the Increase in project targets Reduction of project targets No further change in targets natural disaster in the Serrana region Revised PDOs and Outcome Targets 17. Revision of the PDO. Due to the natural disaster that affected Rio de Janeiro’s Serrana region in January 2011, the project was restructured in that year, and the PDO was revised to allow financing of emergency relief activities. The PDO was revised by the addition of a clause as follows: “to increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower's territory and help re-establish an agricultural productive environment in areas of the Serrana Region affected by the January 2011 natural disaster, thus contributing to the higher-order objective of increasing small-scale farming productivity and competitiveness in those areas.” The restructurings carried out in 2012 and 2017 (partial cancellation) did not modify this PDO. 18. Beneficiaries and project area. The 2011 restructuring diverted resources to new activities so the targets for some of the original indicators were reduced. The project area remained unchanged (59 municipalities), but the number of micro-catchments targeted was reduced from 270 to 200, the number of direct beneficiaries targeted was reduced from 24,400 to 19,300, and the number of indirect beneficiaries targeted was reduced from 37,000 to 28,000. The 2012 restructuring (Additional Financing [AF]) significantly increased the project’s scale by adding 13 new municipalities, 166 micro-catchments, 27,700 direct beneficiaries, and 50,000 indirect beneficiaries. As part of the 2017 restructuring (partial cancellation), the number of direct beneficiaries was again reduced, this time to 35,000, while the number of municipalities and micro-catchments was not modified. Revised PDO Indicators 19. Emergency restructuring of 2011. The Results Framework was modified to include (a) changes to the PDO and introduction of new intermediate-level indicators to monitor the implementation and results of emergency activities and (b) reduction in the number of beneficiaries and other target values related to the original PDO-level and intermediate-level results indicators, because of the redirecting of project resources to the emergency activities. The only PDO indicator that was not modified was “Extent of tertiary roads restored and maintained” (target: 1,300 kms). Two new intermediate results were introduced under Component 1: (a) number of emergency investment proposals financed (target: 2,300 emergency investment proposals) and (b) all resources related to emergency operations disbursed one year after the natural disaster (target: US$18.77 million). 5 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 20. Restructuring of 2012 (AF). The targets for three PDO outcome indicators were increased to reflect the goal of scaling up project activities, that is, the number of small farmers transitioned toward more productive and sustainable farming, number of small farmers included in at least one value chain, and hectares of agricultural lands under improved production systems. One PDO indicator was modified to reflect the type of improved farming systems promoted by the project: “Number of small farmers (at least 50% in targeted areas) transitioned towards more productive and sustainable farming systems.” Two PDO indicators were reclassified as intermediate results indicators4 under Component 1: (a) improved product quality and (b) extent of tertiary roads restored and maintained. Revised Components 21. Restructuring of 2012 (AF). Changes were made to several components:  Component 1: (a) longer-term rehabilitation activities in areas of the Serrana region; (b) identification, preparation, and implementation of new investments (productive and environmental ‘Structuring Subprojects’) by small producers. New activities would be implemented on a larger scale (across several micro-catchments and/or municipalities).  Component 2: (a) new partnerships to enhance cohesion of public policies and institutions under Subcomponent 2.1 (Strengthening Rural Institutions and Coordination Mechanisms); (b) Subcomponent 2.2 (Improving Public and Private Financial Support Mechanisms), an evaluation of the economic sustainability system (ESS), subject to a positive evaluation, additional seed capital for the proposed mechanism; and (c) under Subcomponent 2.3 (Undertaking Participatory Research), additional value-chain studies and participatory research in support of investments under Component 1.  Component 3: (a) incremental management and coordination functions (Subcomponent 3.1); and (b) additional information and outreach activities (Subcomponent 3.2). Other Changes 22. Other changes introduced during the project implementation period included the following:  2011 restructuring: (a) an extension of the end-disbursement date for the operational costs that enabled the loan proceeds for operational costs to be used beyond March 1, 2012.  2012 (AF) restructuring: (a) AF of US$100 million, (b) increased project area and targets, and (c) two-year extension of closing date. 4No clear justification was given in the AF PAD for this change (reduction) in the number of PDO indicators other than (sic) “Project outcome indicators at PDO level and specific targets have also been updated to reflect the expansion of the original activities and the revised implementation schedule.” The justification of the Project Management Unit (PMU) for this change was provided by email on May 9, 2019 (sic) “This indicator was excluded from the results framework because it was integrated with the measurement of beneficiaries included in productive value chains, since the improvement of product quality had been one of the axes worked in value chain subprojects. Maintaining this indicator would have been redundant.” 6 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508)  2015 restructuring: (a) allow the financing of operational costs for a longer period to help address the fiscal crisis.  2017 restructuring: (a) cancellation of US$40 million of the AF Loan from Categories 1 and 2; (b) reallocation among disbursement categories; (c) simplification of project design, by eliminating institutional and structuring subprojects5 (Component 2.1), not requiring the creation of consortia of municipalities for road maintenance; (d) change in the Results Framework: reductions in targets of two PDO indicators and the introduction of a new intermediate-level indicator (rural roads rehabilitated, target: 6,000 km). Rationale for Changes and Their Implication on the Original Theory of Change 23. The rationale for the various restructurings was as follows:  2011 emergency restructuring. Beginning shortly after effectiveness, natural disasters (heavy rains and floods) forced significant adjustments in project implementation arrangements. The World Bank was asked to support the implementing agency by redirecting resources to affected rural areas and providing an immediate response to restore physical access through rural road rehabilitation, restoration of damaged houses, and the restoration of productive assets. All those activities were within the scope of the project. Language relating to geographical targeting and the prioritization of productive activities was added to the PDO to allow the financing of emergency activities in the Serrana region.  2012 restructuring (AF). This was mainly in response to (a) the need to adopt a more integrated and comprehensive approach to disaster risk management after the natural disaster of 2011 and (b) a scaling-up strategy for the project in anticipation of an expected major inflow of investments in the coming decade in the SoRJ, including World Cup in 2014 and the Olympic Games in 2016, and the consequent increased demand for agricultural products. The request for AF was fully consistent with the World Bank Group’s CPS 2012– 2015 (Report No. 63731-BR), in particular in its first and fourth strategic objectives: to increase the volume and productivity of public and private investments and to further improve sustainable natural resource management and enhance climatic resilience while contributing to local economic development and helping to meet rising global food demand. The AF also supported the state’s integrated territorial development policies and strengthened its disaster risk management capacity, complementing the support provided as part of the World Bank’s Rio de Janeiro program under the AF for the Strengthening Public Sector Management and Territorial Development TAL (P126735). It involved a slight 5 During project execution, the bureaucratic process and the incipient organization of the small producers to take on greater investment to implement these public calls for access to the “structuring subprojects”, as well as the complexity of the procedures for access to resources by those farmers, made this type of support impossible. With the restructuring of 2017, “structuring subprojects” were replaced by two additional subproject modalities: group and value chain subprojects: (a) group subprojects: under this modality, Rio Rural supported organized groups of smaller producers (3 to 10 producers) in overcoming bottlenecks on their farms, filling the gap of individual subprojects (productive and environmental) allowing mainly machines and equipment to be acquired and used in a group; (b) value chain subprojects: investments were focused on activities that add value to production, through improvement of product quality, stimulation of processing, processing, agroindustry, packaging, seal, and certification. They comprised actions essentially for groups of one or more micro-basins to increase efficiency and overcome bottlenecks in the main agricultural production chains, with emphasis on activities "beyond the farmgate." 7 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) adjustment of the PDO and an upward revision of targets consistent with the additional loan funds.  2017 restructuring (partial cancellation). This restructuring was justified by the limited borrowing capacity of the state, which was significantly affected by an acute fiscal crisis, disrupting project implementation for almost one year. As with the other World Bank projects in the state, on several occasions, the judicial power ordered seizures of the funds in the Designated Account. Moreover, the fiscal crisis reduced public investments for rural roads rehabilitation and maintenance. The restructuring involved a cancellation of loan funds and a downward revision of targets consistent with the reduced funding. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 24. The relevance of the PDO to the CPS and Country Partnership Framework (CPF) is High. The PDO was highly relevant at the time of appraisal and remained consistently relevant throughout implementation with the World Bank’s CPSs and CPF. The relevance of the PDO was ensured at the time of appraisal through its alignment with the CPS of 2008–2011 - Report No. 42677-BR (CPS 08-11). At closing, the primary themes of the PDO remained in the mainstream of the new World Bank/Brazil CPF 2018–2023 Report No. 113259-BR. The project responded to two challenges reflected in this CPF. First, it set out to stimulate investments and innovation, thereby increasing productivity, improving the business environment, facilitating access to capital and encouraging competition. Second, it promoted smart management of natural resources and better ways to mitigate natural risks and pollution through the sound management of water resources and environment, as well as property rights and land. Moreover, the project addressed the CPF focus areas of inclusion and sustainable development by promoting socioeconomic development of smallholder rural farmers. The project remained fully relevant, even when taking into consideration its various restructurings, none of which substantially altered the overall development objective. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 25. The project was innovative for the SoRJ, proposing a new approach building on productive potential and access to existing, profitable markets in the state. The project supported fruits, vegetables, and milk value chains, which are suited to small farmers and take advantage of urban demand from the metropolitan area of Rio de Janeiro and other smaller urban centers in the interior of the state. The project strengthened organizations of family farmers, helping them become more commercially competitive and promoting adoption of improved production systems to ensure long-term sustainability. The implicit ‘Theory of Change’ was sound and based on rational operational and technical assumptions. 8 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 26. Due to external factors, the implementation period was divided into four phases, with distinct performance characteristics: (a) Initial implementation phase, between approval and the 2011 emergency. Progress achieved during this phase was modest because of legal, operational, and procurement challenges and by uneven readiness to implement. (b) Emergency implementation phase, between 2011 and the 2012 restructuring. This second phase was defined by the 2011 emergency and the World Bank’s speedy and effective response to an external event (including a new type of “Emergency Subproject”). These emergency activities were completed successfully in about one year, leading to impressive results. Rural poverty was reduced again after 2013, returning to the trend of the larger region of southern Brazil. The project—together with other emergency responses— contributed to this important result. Yet this may have led to an optimistic view of the project’s potential. In addition, given the Rio Olympic Games, the participation of the state as an official location for the 2014 Soccer World Cup, and the overall optimistic expectations for the growth of BRICS (Brazil, Russia, India, China, and South Africa), there was consensus about the need to be prepared for significant economic growth in the coming years, leading to calls from state authorities to expand project coverage and increase the outcome targets. (c) Intermediate implementation phase, between the 2012 AF and the 2017 restructuring. During this period, it gradually became evident that the US$100 million of AF had been approved in response to an overly optimistic view of the prospects for potential achievements and the possibilities of scaling up the scope and coverage of the operation. This realization coincided with dramatic political and economic (fiscal) crises in Brazil, which shifted public priorities and further constrained the project’s pace of execution and rate of disbursement. The SoRJ was disproportionately affected by low oil prices, given its high reliance on oil royalties, and in June 2016, the local authorities declared a state of “fiscal emergency.” The difficulties encountered during this period demonstrated that the expansion of the project’s targets had been overly optimistic, and it was necessary to cancel US$40 million of loan resources. (d) Final phase of consolidation, between 2017 restructuring and the closing date. The final restructuring enabled the project to consolidate basic results (with reduced scope and partial cancellation of loan funds), supported by marked improvement in project execution and disbursements, leading to achievement of most end targets. 27. The following paragraphs disaggregate the PDO by theme and present evidence of the project’s achievements using the PDO outcome indicators and supporting information. The evidence includes a strong Borrower Completion Report (BCR)6 based on, among others, extensive interviews with project institutions, beneficiaries, and other stakeholders, as well as data obtained from a representative sample of beneficiaries. The findings of an Impact Evaluation study are discussed as well, 7 along with data from 6 Avaliação Geral do Projeto de Desenvolvimento Rural Sustentável em Microbacias Hidrográficas do Estado do Rio de Janeiro. OPE Socias, Fevereiro 2019. 7 Governo do Estado do Rio de Janeiro, Secretaria de Estado de Agricultura e Pecuária - SEAPPA (2018). “Impacto do programa Río Rural no Estado do Rio de Janeiro: Sumário Executivo”. For details on the Impact Evaluation see section IV A on Quality of Monitoring and Evaluation. 9 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) the project archives. The targets for each objective are compared with the achievements and results at project closing, but in view of the different restructurings and distinct implementation phases, a “split assessment” was undertaken for rating the overall outcome (see Section II D). 28. PDO outcome - Theme 1: “Increase the adoption of integrated and sustainable farming systems approaches in specific areas of the Borrower's territory.”  Outcome indicator No.1: Area of agricultural lands under improved production systems (hectares) - Final Target: 160,000 ha. The actual outcome was 223,152 ha of agricultural land under improved production systems, measured through the area of benefited properties. This outcome corresponds to 139 percent achievement of the target.  Outcome indicator No. 2: Number of small farmers included (or with improved links) in at least one value chain. Final target 2,600 farmers. A total of 3,359 small farmers were included in (or with improved links to) at least one productive chain as a direct result of the project, corresponding to 129 percent achievement of the target. This result was calculated based on the number of beneficiaries of subprojects focused on productive chains. In addition, the technical report which evaluated the five productive chains supported (coffee, milk, organic, peach palm, and strawberry) concluded that, through the project's strategies and activities, these chains benefited small farmers by reducing their operating costs, increasing their productivity and income, improving pricing policy, and promoting important environmental gains, such as erosion control, reduction of pesticide use, and optimization of water use (AVALSUST 2018b).  Outcome indicator No. 3: Number of small farmers transitioned towards more productive farming systems. Final target 35,000 farmers. A total of 37,172 small farmers and family farmers (including at least 50 percent in the target areas) adopted more productive and sustainable systems, corresponding to 106 percent of the target. This was estimated through the number of Individual Development Plans (PIDs) multiplied by two because the PIDs showed that, on average, two members of the farming family actively managed the productive and business/administrative aspects of the property.  Breakdown of Outcome indicator No. 3: Number of female small farmers transitioned towards more productive farming systems. Final target 6,000. At project end, 5,280 family farmers adopting more productive and sustainable systems were women, corresponding to 88 percent of the target.8 Additional Results  Intermediate results indicator: Km of rural roads rehabilitated and in maintenance (6,000 km). The reports of the Production Roads Program indicate that 7,127 km of rural roads were rehabilitated and/or maintained, corresponding to 119 percent of the target. According to the report on the sustainability of roads and bridges served by the project, Rio Rural contributed to their good functionality through improvement of drainage systems, 8 This indicator was measured directly from types of subprojects targeted to women. Note the difference from Outcome indicator 3, where an indirect measurement is made by assuming the number of small farmers by multiplying the number of PIDs by 2. 10 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) erosion control and adequate lining of canals/channels. However, it should be noted that drainage and erosion control structures also require maintenance (AVALSUST 2018c).  Intermediate results indicator: Targeted clients satisfied with project services (75 percent). Through the Practices Evaluation Survey (PSATISF 2018b) carried out with 61 beneficiaries in the Northwest region and 42 in the Serrana, a 91.8 percent satisfaction index was obtained in the first region and 100 percent in the second region, resulting in an average index of 95.15 percent of beneficiaries highly satisfied with the practices developed by the project. Also, field trips during supervision missions always confirmed the high level of beneficiary satisfaction. Regarding satisfaction with the Micro-catchment Development Committee (Comite Gestor das Microbracias, COGEMs), from meetings held with 85 COGEMs from the Serrana region, a 78.14 percent satisfaction index was estimated. Considering all the mentioned indicators, the overall Satisfaction index is 79.2 percent (PSATISF 2018b), exceeding the target.  Intermediate results indicator: Number of participatory research projects carried out in support of sustainable rural development (50 projects). According to the results of the participatory research reports, 75 participatory research projects supporting sustainable rural development were carried out, representing 150 percent of the target. From 2010 to 2018, State Agriculture Research Enterprise (Empresa de Pesquisa Agropecuaria do Estado do Rio de Janeiro, PESAGRO-Rio) operated in 30 municipalities and 35 hydrographic micro- basins, involving a public of 52 direct beneficiaries (experimental farmers), 3,939 indirect beneficiaries, and 28 researchers. Some BRL 4,718,660 was spent on participatory research with direct IBRD resources and BRL 788,707 with counterpart funds. (RFNPP 2018).  Key benefits generated by the participatory research projects were documented for dairy and coffee value chains, as well as for organic agriculture. In the case of dairy, research contributed to improvements in afforestation of grasslands, diversification of grasses, multiplication of sugarcane forage, use of grazing rock dust, control of mastitis in herds, and training in sanitary conditions of production. In the case of coffee, research contributed to improving the quality and type of Arabica coffee produced in the northeast of Rio de Janeiro. Organic seed production was stimulated (20 species of vegetables, 24 of corn and beans, and 11 of green manures), efficiency of essential oils for the treatment of seeds was tested, and studies of organic agriculture and the observed behavior of rice cultivars in floodplains were conducted (RFNPP 2018).  Other relevant results. Other results attributable to the project in relation to the development of agricultural value chains include (a) coffee chain: average increase of 20 percent in sales price due to increases in product quality, mechanization and reduction of production costs, mainly in the region northwest of Fluminense; (b) milk chain: increases in productivity (20 percent) and sale price due to investments in productive assets (cooling tanks), reduction of the rural exodus of producers, and improvements in the genetic quality of the herd; (c) palm hearts (palmito) chain: expansion of planted area and introduction of technologies that increase productivity and reduce crop losses, in addition to environmentally regulating the production properties; (d) production of strawberries: investment in suspended semi-hydroponic systems, with little or no use of pesticides; (e) based on participatory research projects developed with Rio Rural resources, the production 11 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) chain of olericultura was dynamized and it benefited from reduction of pesticides and the adoption of protected cultivation and circular vegetable garden systems (AVALSUST 2018b; RFNPP 2018).  An ESS was established—as planned—to enable public and private resources to be raised. A total of 158 investment proposals were financed through the ESS for the benefit of family farmers, representing 263 percent of the target. Also, through the ESS, US$25,375,344 was leveraged (original target was US$ 1,000,000) (MRESULT 2018).  Other results indicate that 370 hydrographic micro-basins were reached (366 original target), 370 COGEMs were established, aggregating 3,870 stakeholders, and 38,221 environmental and productive investment proposals (subprojects) were financed (6 percent above the target) (MRESULT 2018).  The independent impact evaluation9 concludes, among others, that the project positively affected profit per hectare due to the use of more productive practices, as well as improvements in the marketing of products. Because profits were positively affected, producers were able to pay higher salaries to hired workers. Given the recent nature of many of the investments that were surveyed—and based on the impact evaluation results for similar projects—longer-term potential appears positive. The report also concludes that if producers are well informed about the practices and their benefits, supported by adequate technical assistance and with the right incentives, they will adopt them, making possible an improvement in the environment and long-run sustainability (see further details/results in annex 6). 29. PDO outcome - Theme 2: “Help re-establish an agricultural productive environment in areas of the Serrana region affected by the January 2011 natural disaster.” The relevance of this change to the PDO is confirmed by the increased rural poverty as a consequence of this natural disaster. No specific PDO indicators were included in the Results Framework; however, the corresponding intermediate outcomes were consistently monitored by the M&E system and reported in the Results Framework as intermediate results. The main achievements included the following:  US$18.77 million of emergency financing was allocated to support special (emergency) subprojects in the Serrana region. A total of 2,277 emergency subprojects were financed (99 percent of the target) reaching a coverage of 1,908 farmers affected by the natural disaster (95 percent of the target). In addition, 890 km of roads (111 percent of the target) and 46 small bridges (115 percent of the target) were rebuilt, and 34 communities were served with soil conservation patrols assisting 4,858 beneficiaries.  Emergency-related activities also contributed to strengthening community organizations, participatory identification of support projects from local planning, and training of technicians and beneficiaries. Producer associations were supported with the acquisition of 9“Avaliação de Impacto do Programa Rio Rural sobre a Rentabilidade das Propriedades Rurais “; Oportunidades, pesquisa e estudos sociais - OPE Sociais. November 20, 2018. See annexes 5 on Methodology and 6 on Main Results from the Impact Evaluation. 12 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) ‘mini-patrols’10 responsible for the management of small machines and equipment made available to small producers to rehabilitate roads and other public infrastructure. Technical assistance was reinforced, operators of such equipment trained, and associations qualified in the management of community enterprises (RATIVEMERG 2013a, 2013b).  PESAGRO-Rio strengthened the Participatory Research Units in the Serrana region where restoration of degraded areas was taking place, emphasizing agroecological transition, organic production, and the adoption of new technologies by experimental farmers. 30. According to the results evaluation of the emergency activities, 89 percent of the beneficiaries stated that as a direct results of project support, they were able to recover from the damage caused by the catastrophe within six months. Most technicians and beneficiaries interviewed recognized positively the role of the emergency committees in local social organization. Justification of Overall Efficacy Rating 31. Overall efficacy is rated Substantial. Factors considered in determining the rating were the following:  All three outcomes contained in the revised PDO were achieved, with quantitative outcomes embodied in the PDO indicators surpassing their end targets. A clear and strong causal connection supports the Theory of Change.  Number of direct beneficiaries far exceeded the revised target, with the exception of the number of women farmers transitioned toward more productive farming systems (88 percent achievement of the end target).  Out of 24 intermediate outcome indicators, 21 met or exceeded the end targets, and all were achieved at a rate of 95 percent or better.  Disbursement of the loan proceeds was less than optimal, reaching US$89.3 million or about 90 percent of aggregate loan proceeds, net of cancellation, due mainly to the procurement challenges described as well as fluctuations in the exchange rate.  The sustainability outlook is positive for both the business investments and environmental elements, based on the fact that (a) participatory selection and implementation of subprojects increased ownership and therefore sustainability, (b) improved access to markets has generated a permanent revenue stream for beneficiaries, and (c) implementation was done largely by regular staff from the state government. The sustainability outlook for public policies is less clear because the consequences of the current fiscal situation are unpredictable. 10The ‘mini patrols’ consisted of a set of machines and implements for the preparation and conservation of the soils. They were destined to some producing associations in micro-basins of the Serrana region, affected by the catastrophe of 2011. In general, they were composed of a tractor with plow and in some cases, micro-tractors. The associations that received these patrols had to sign a commitment to use them according to the rules of the Rio Rural project and had to undergo training in environmental management and safeguards. 13 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 32. Split assessment of rating. Consistent with the ICR Guidelines, a split assessment was carried out to determine the outcome rating. Details are presented in Section II D. Splitting the assessment made no material difference to the outcome rating, when compared with the final rating based on the final outcomes at closing. C. EFFICIENCY Assessment of Efficiency and Rating 33. The project disbursed US$146.36 million (75 percent of funds approved), out of US$202.45 million available including the AF. Several external factors contributed to the lower-than-expected disbursement, including a devaluation of the Brazilian real and a fiscal crisis experienced by the SoRJ. On the one hand, the strong currency devaluation decreased unit costs in local currency and increased significantly the amount of local currency available to the project.11 On the other hand, inflation also increased during implementation, raising many costs. As a result of these two sets of forces that affected the purchasing power of the loan proceeds, it is difficult to calculate in real terms exactly what was disbursed. What is noteworthy, however, is that the project exceeded its PDO indicator targets (except for women) and either exceeded or met most intermediate outcome targets, with lower-than-anticipated total costs. 34. Economic efficiency. To evaluate the economic and financial results of the project, 155 PIDs were randomly selected as a stratified random sample from the five regions of the SoRJ (representing about 1 percent of total PIDs implemented). In terms of value chains, the sample included 54 PIDs for horticulture (leafy vegetables, tomato, cauliflower, peppers, cassava, potatoes, sweet corn); 51 PIDs for milk production; 31 PIDS for fruticulture (citrus, passion fruit, strawberry, kale, banana, pineapple); 11 PIDs for coffee production; 6 PIDs for raising small animals; and 2 PIDs for forestry (palmito). In terms of territorial coverage, the sample included 35 PIDs from the Northern region; 30 PIDs from the Northwest region; 37 PIDs from the Central region; 29 PIDs from the Serrana region; and 24 PIDs from the Southern region. More details are presented in annex 4. These 155 PIDs benefited 236 households (about 1.4 percent of total households) representing 714 people. 35. Investments made by the PIDs included brush cutters, spring protection, vehicles (mini-tractors and wagons), plantations (coffee, passion fruit, palmito, forage cane and strawberry), improved pastures and rotational pasture systems, irrigation systems, organic fertilization, drying infrastructure and coffee pulping machines, milking machines and milk cooling tanks, sheds, spraying machines, animals (cows and chickens), riparian forests, agroforestry systems, greenhouses, and soil preparation. The results generated by the various investments were mainly increased production, lower production costs, and increased incomes. 36. Net present value (NPV) and internal rate of return (IRR). For the 155 PIDs evaluated (with 236 families and 2,120 ha), the economic NPV and the IRR of the investments made, or the incremental situation were US$3.0 million and 54 percent. Total investments at economic prices were US$603,800, of which, US$314,700 was funded by the project (or 52 percent) and US$289,000 was funded by participants and other sources (or 48 percent). The average values per participating family are US$2,600 of investments, US$2,500 as incremental net revenue per year, and 27 person-days as incremental annual 11PAD 2009 reports an exchange rate of BRL 2.15 per US$1.0; PAD 2012 reports an exchange rate of BRL 1.82 per US$1.0; at project closing (November 30, 2018), the exchange rate was BRL 3.86 per US$1.0. 14 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) use of labor. The average annual incremental net income per hectare is US$275, which is significantly more conservative than the estimated mean value for profit per hectare of US$800 in the Impact Evaluation.12 For the Serrana region, subject to special attention due to the emergency, the returns were very close to the overall average returns. 37. Indicative estimates of economic viability were also generated for investments made by the project in rural road improvements and participatory research. In terms of the impact of rural roads, results are the following: (a) 92 percent reduction of days of non-trafficability, (b) reduction of production losses (20 percent in the case of milk and 36 percent in the case of vegetables), (c) reduction of 50 percent of travel time due to increase in traffic speed, (d) reduction of 40 percent to 60 percent of the average cost of vehicle maintenance, and (e) 20 percent to 50 percent reduction in average fuel consumption. The IRR of the improvement of two roads evaluated (around 7–9 km improved) varies between 20 percent and 280 percent. As for participatory research, results show (a) 50 percent increase in milk production and reduction of rock dust as a result of the introduction of the silvo-pastoral system with rotational grazing, (b) 30 percent increase in forage cane production as a result of the addition of rock dust to organic fertilization, (c) 30–50 percent increase in milk production as a result of the control of mastitis and good sanitary practices, and (d) increase of 10–40 percent in the productivity of horticulture and increase of 40 percent of the income as a result of the use of alternative inputs to pesticides. Regarding greenhouse gas (GHG) emission monitoring, results show how the most important actions to reduce GHG emissions are those related to actions against deforestation/forest regeneration as protection of recharge areas and areas of springs. One of the recommendations emerging from the analysis is to keep a focus on these activities when planning new projects.13 38. To estimate the aggregate economic return of the Rio Rural project, the economic flows of the sample of 155 PIDs were extrapolated based on the ratio between the project investments for all the PIDs and the project investments on the PIDs of the sample. Other project costs not directly related to the PIDs were incorporated into the analysis of the NPV and aggregate IRR. Incremental PID investments irrespective of financial source generated an economic NPV of US$317 million and an economic IRR of 54 percent. Incremental investments in the PIDs in addition to other project costs generated an economic NPV of US$299 million and an economic IRR of 45 percent. When carbon balance co-benefits were added, the economic NPV and IRR were US$346 million and 49 percent, respectively. Finally, considering that the average project investment per PID is higher than the average investment of sampled PIDs, such investment costs were adjusted, and the NPV and the economic IRR with other project costs and carbon co-benefits were US$301 million and 36 percent. These results are similar to the indicators of the economic analysis carried out for the project at appraisal. The analysis offers unbiased economic indicators at the aggregate level, but the small sample size of PIDs only allows indicative conclusions. The large variability of results in terms of economic NPV and IRR do not allow comparisons by region and by value chain. 12 As explained by the Food and Agricultural Organization (FAO) economist who prepared the EFA: “The average annual incremental income per hectare is USD 275, which is more conservative than the estimated mean value of profit per hectare of US$800 in the Impact Evaluation. Even though the sample does not allow for comparisons among regions or value chains, we can say that in the Serrana region, subject to special attention due to the emergency, the returns were very close to the overall average returns. Further, it is emphasized that: ‘In many instances, it is too early for the most recent subprojects to have produced an impact’.” 13 Bassi, Lauro. 2018. “Primeira avaliação: resultados de cadeias produtivas avaliadas apoiadas pelo Programa Rio Rural/BIRD e resultados do monitoramento da biodiversidade e armazenamento de carbono .” 15 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 39. Sensitivity analysis. The returns generated by the project investments are sensitive to production and/or price changes because a reduction of 11 percent in sales or revenues would reduce the economic IRR to 12 percent. The returns are moderately sensitive to rising input costs as a 20 percent increase in costs would generate a marginally acceptable return. In contrast, the returns are very resilient in terms of labor costs and changes in investment costs as these would have to increase by 60 percent and 120 percent before the returns would fall to a marginally acceptable level. 40. Efficiency rating. Efficiency is rated Substantial, based on overall financial and economic results, including the rural investment subprojects (PIDs), the largest investment, but also other auxiliary investments such as rural roads, participatory research, and impacts in GHG emissions. It is worth noting that assessment of the efficiency of the project proved to be a challenging exercise due to the complex context in which the project operated,14 the concentration of investments toward the last years of the project, the small sample size to estimate the aggregate economic return of rural investment subprojects (PIDs), and the fact that in many instances it is too early for the most recent subprojects to have produced an impact. Still, results are significant and could be considered a lower-bound for the overall project. D. JUSTIFICATION OF OVERALL OUTCOME RATING 41. The overall outcome rating is Moderately Satisfactory based on the following:  High relevance of the PDO based on its sustained alignment with World Bank strategy documents for Brazil  Substantial rating for efficacy, based on the achievement or surpassing of almost all key targets and important collateral/complementary achievements  Substantial rating for efficiency, based on positive economic and financial outcomes 42. Split assessment of overall outcome. According to “Bank Guidance: Implementation Completion and Results Report (ICR) for Investment Project Financing (IPF) Operations” and given the nature of the various restructurings, a double split assessment was done to assess the overall outcome rating. Table 2 presents the results of the double split assessment, which recognized three separate assessment periods delineated by two restructurings: (a) the September 2011 restructuring due to the emergency in the Serrana region, which involved a change in PDO (Split 1), and (b) the September 2017 restructuring which included a partial cancellation of funds (US$40 million) and a significant reduction in project targets (Split 2). 43. Achievement of the PDO indicators was assessed by comparing current values as of the date of each restructuring against target values according to each Results Framework.15 For Split 1, the targets for most PDO indicators had been achieved (except for indicators that did not have a target for year 1 of the project), and hence the ratings are Substantial and High. For Split 2, the PDO indicators showed a lower level of achievement, and hence the ratings are Modest and Substantial. At project closing, all PDO indicators had achieved or exceeded the targets set out in the Results Framework, so the rating is Substantial. The weight of each period was based on noncumulative disbursements reported by the 14 See Section I.B Significant changes during implementation. 15 Annex 7 provides a detailed measurement of PDO indicators for split assessment. 16 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) project. This yields a final outcome rating of 4, equivalent to Moderately Satisfactory, which is consistent with the final Implementation Status and Results Report (ISR) rating at closing. Table 2. Split Assessment of Outcomes Table 2. Split assesment of outcomes Split 1 Split 2 Project closing Restructuring of Restructuring of 2011 2017 2018 RELEVANCE OF PDO HIGH EFFICACY PDO Number of small farmers (at least 50% in targeted areas) transitioned towards more productive and Substantial Modest Substantial sustainable farming systems. Improved product quality measured by: Number of farmers adopting Good Agricultural Substantial N/A N/A Practices (GAP) Number of small farmers or enterprises certified Substantial N/A N/A Number of agro-processing and artisanal enterprises adding value High N/A N/A Improved market access by at least 10% of beneficiaries receiving investment support as Substantial Substantial Substantial measured by their inclusion in (or with improved links to) at least one value chain At least 50% of the targeted small farmer Substantial Modest Substantial agricultural lands under improved production Extent of tertiary roads restored and mantained High N/A N/A EFFICIENCY SUBSTANTIAL Outcome ratings Moderately Moderately Satisfactory Satisfactory Satisfactory Numerical value of outcome ratings 5 4 4 Disbursement $ 9,575,735.53 $ 56,156,124.77 $ 89,155,068.25 Share of disbursement 10% 47% 43% Weighted value of the outcome rating 0.48 1.86 1.70 Moderately Satisfactory Final outcome rating 4 E. OTHER OUTCOMES AND IMPACTS Gender 44. The project’s focus on family farming was reflected in an investment strategy designed to produce important impacts on women as key actors in family farming. Specific attention was given to vulnerable groups including women, reflected in the number of women that participated in training activities and in the number of investment proposals from women financed by the project. In the Results Framework, a PDO-level indicator "number of small farmers (at least 50% of selected areas) transferred to more productive and sustainable systems" was disaggregated by gender reporting 88 percent achievement rate at project closing. Gender targets were also at intermediate level such as “number of women trained in key project concepts” reporting 175 percent achievement rate at project closing. 45. Participation by women in the awareness-raising phase of the project, as well as in community decision-making spaces such as the COGEMS, was reported as significant during project execution. The role of State Rural Extension Agency (Empresa de Assistencia Tecnica e Extensao Rural do Estado do Rio de Janeiro, EMATER-Rio) technicians was key in promoting women's participation in these meetings and in productive processes, evidenced by the increase in the number of women who received financial resources and technical support. 17 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 46. Dedicated subprojects such as the “Kit Galinha Caipira” (local chicken breed), handicrafts, and some specific value chains helped increase female participation in rural communities while increasing their earnings. The role of women in value chains such as peach palm16 and organic palm trees stood out as fundamental in value-adding and in the marketing of their agriculture produce, including access to institutional markets (public purchases). In this way, the project had a clear impact on promoting female autonomy in the SoRJ. Institutional Strengthening 47. Grassroot institutional development. The micro-basin approach was based on the idea of strengthening grassroots institutions (COGEMs) as instruments of community organization and channeling of local demands to higher decision-making bodies and allocation of public resources. At closing, the project had supported 370 COGEMs in 366 micro-basins and more than 19,000 producers had accessed resources dedicated to subprojects. 48. Institutional alliances. The project built a significant number of partnerships with state institutions for risk management and environmental adequacy, as well as integration with credit programs. Cooperation agreements were established with partners that have similar actions to Rio Rural, such as North Fluminense State University (Universidade Estadual do Norte Fluminense, UENF), Brazilian Agriculture Research Enterprise (Empresa Brasileira de Pesquisa Agropecuaria, EMBRAPA)-Soils, EMATER- Rio, PESAGRO-Rio, Department of Mineral Resources (Departamento de Recursos Minerais), EMBRAPA- Agrobiology, Baixo Paraíba Basin Committee, National Ministry of Agrarian Development (Ministerio do Desenvolvimiento Agrario)/National Program to Strengthen Smallholder Agriculture (Programa Nacional de Fortalecimento da Agricultura Familiar), Banco do Brasil, nongovernmental organizations (NGOs), Brazilian Micro and Small Business Support Service (Serviço Brasileiro de Apoio à Micro e Pequenas Empresas), FAERJ (Federation of Agriculture, SoRJ), State Research Support Foundation ( Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro, FAPERJ), State Secretariat for Planning and Management (Secretaria de Estado de Planejamento e Gestão do Estado do Rio de Janeiro), and the German universities of Cologne, Jena, and Leipzig. 49. Participatory research. Through a Participatory Research Nucleus, the project developed seven action plans: (a) Implementation and Maintenance of Participatory Research Units; (b) Implementation and Maintenance of Long-term Agriculture Experiments; (c) Network for Research, Innovation, Technology, Services, and Sustainable Development in Hydrographic Hydrological Systems; (d) Production of Agroecological Inputs; (e) Adequacy of Rural Dairy Farms to Good Practices; (f) Agroecology, Organic and Sustainable Agriculture (Studies); and (g) Institutional Strengthening. 50. The achievements of this subcomponent were to continue the seven Participative Research Units implemented by Rio Rural GEF and to implement new units; to promote validation events for research lines; to prepare, correct, and publish technical manuals; to participate in the regional meetings of PESAGRO-Rio; and to publish studies, including some 14 papers in 2017 alone. 51. According to the Final Report of the Participatory Research Nucleus, from 2010 to 2018, 33 percent of the financing of research projects developed by PESAGRO-Rio came from Rio Rural, while 66 percent 16Bactris gasipaes is a species of palm native to the tropical forests of South and Central America. It is often cultivated by smallholders for hearts of palm and fruits. 18 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) came from other sources, such as FAPERJ (Foundation for Research Support of the SoRJ), State Fund for Environmental Conservation and Urban Development, (Fundo Estadual de Conservação Ambiental e Desemvolvimento Urbano), National Council for Scientific and Technological Development, Financier of Studies and Projects, Ministry of Agriculture, Livestock and Food Supply, State Secretariat of Agriculture, Livestock, Fisheries and Supply (Secretaria de Estado de Agricultura, Pecuaria, Pesca e Abastecimento, SEAPPA), and Parliamentary Amendment. 52. Institutional strengthening and technical assistance. EMATER-Rio incorporated the methodology based on hydrographic basins as part of its efforts. Its technicians incorporated this approach when working with rural communities during the various phases of the project (sensitization, training, and monitoring). The COGEMs met regarding implementation and maintenance and often expanded the purpose of strengthening the participation of producers in the micro-basin. Mobilizing Private Sector Financing 53. ESS. A proposal for an ESS was designed with Global Environment Facility (GEF) resources and operationalized with resources from the Rio Rural project. The objective was to create a financing mechanism for family farmers. The ESS focused on raising funds from the following sources: (a) public policies in general; (b) payment for environmental services in carbon, water, and biodiversity; (c) agricultural credits; and (d) private partnerships in the areas of sustainable business and socioenvironmental responsibility. The ESS was implemented in a decentralized manner, leveraging co- investment resources and integrating partnerships with public and private institutions. At project end, 158 investment proposals were presented (263 percent of target), of which 86 were approved, leveraging US$25,375,344 in co-investments from various sources (US$1.2 million municipal governments, US$17.4 million state governments, US$3.0 million Federal Government, and US$3.6 million other private sources). The number of proposals directed to public financial institutions (federal, state, municipal governments) was the largest, corresponding to 74 percent of the total proposals made.17 Poverty Reduction and Shared Prosperity 54. The project had no explicit poverty reduction objective. However, the target population (small farmers) and the aim of improving productive farming systems and reduce the vulnerability of the territories where the target population lives, indirectly relates to a potential poverty impact. 55. As part of the evaluation strategy, the PMU contracted several analyses to document the effects of the project on the target population. One was a diagnosis to understand the determinants and the recent evolution of poverty in the SoRJ, which would allow contextualizing effects attributable to the project. The main finding of this study was that the bulk of observed rural poverty reduction in Rio de Janeiro between 2001 and 2015 did not derive from the productivity or profitability of agricultural activities. Rather, it was the joint effect of public transfers plus income growth of rural dwellers coming from nonagricultural activities. 56. Another study determined that the project enabled the recovery of the productive capacity of soils, contributed to the preservation of water springs and protection of biodiversity, combining community self-management and the promotion of smallholder proactivity. In other words, “the project combines 17 Rio Rural. 2016. “Estratégia de cofinanciamento programa Rio Rural.” Project Report. 19 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) the adoption of productive systems that are sustainable and the management of natural resources, demonstrating that it is possible to generate income and, at the same time, to meet the demands of the market and environmental legislation” (DPOBREZA 2018, 29). 57. The Impact Evaluation,18 in its chapter referring to impacts on living conditions and poverty levels, identifies modest impacts in terms of time allocation and household mobility observed through greater migration of the elderly (possibly for health care reasons) and reductions in youth and adult migration suggesting that rural areas could be perceived as more attractive due to expectations of higher income. At the same time, the report also acknowledges that many indicators associated with living conditions do not yet show clear impact because the expected effects require a longer period to mature and be observable. Balance of GHG emissions 58. A GHG appraisal of the main set of activities implemented under the project was carried out using the Ex Ante Carbon-balance Tool (EX-ACT), which quantifies the net carbon balance (in tCO 2e) resulting from GHGs emitted or sequestered during project implementation, compared to the scenario before project implementation.19 The main hypothesis is that project-supported activities mitigated or reduced GHG emissions when compared with the situation ex ante. 59. The analysis focused on selected practices and actions covered by a GHG analysis performed in 2009 (before the project, presented in a study by Branca and Medeiros of 2010) and included an update of the analysis carried out in 2017. Other actions, such as the “Improvements in feeding practices of dairy herds” were not included in the analysis due to the lack of data. The actions considered were (a) Protection of springs and streams and support for the establishment of legal reserves, (b) Expansion of agroforestry systems, (c) Improved management of annual crops, (d) Improvements in pasture management, (e) Support to family agroindustry, and (f) Technical support. The results in terms of balances of GHG as tCO2 equivalent per year showed modest impacts. Other Unintended Outcomes and Impacts 60. Social capital. By promoting solidarity among rural producers, the project strengthened the role of cooperative mechanisms, such as cooperatives and associations, in integrating producers into the market, thereby, facilitating their access to inputs and improving their ability to sell their products (especially in local markets). The project achieved other unintended effects and impacts, such as promoting inclusion of women and youth in rural activities, as well as strengthening links with the National School Food Program (Programa Nacional de Alimentação Escolar). 61. Results from the impact evaluation related to social capital20 show that producers participated more actively, and community participation was more collaborative on productive than on social aspects. The project appears to have increased participation in activities which delivered more economic benefits and reduced their participation in activities bringing social benefits. In other words, social capital was 18 “Avaliação de Impacto do Programa Rio Rural sobre as condições de vida e a pobreza; ” Oportunidades, pesquisa e estudos sociais - OPE Sociais. November 20, 2018. See annexes 5 on Methodology and 6 on Main Results from the Impact Evaluation. 19 Saraiva Schott and Bernstad 2018. 20 “Avaliação de Impacto do Programa Rio Rural sobre o capital social; ” Oportunidades, pesquisa e estudos sociais - OPE Sociais. November 20, 2018. See annexes 5 on Methodology and 6 on Main Results from the Impact Evaluation. 20 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) important and systematically fostered, well before the investment stage. It was an essential component of the adoption process, on the premise that value chains do not function without organization, cooperation, and collaboration. The assumption was that if producers were organized, trained, and well informed about sustainable farm practices and provided with appropriate incentives, they would adopt practices which improved their profitability and long-term sustainability. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 62. Realism and ambition of objectives. The PDO refers to concepts such as “integrated, sustainable, farming systems” that are complex in nature. To reduce possible confusion, the term “farming system,” which was fundamental for the overall structure of components and subcomponents and selection of activities, was clearly defined as “a population of individual farms that have broadly similar resource bases, enterprise patterns, household livelihoods, and constraints, and for which similar development strategies and interventions would be appropriate, including the key role played by the community, the environment, and support services, to achieve their common objective” (PAD 2009). This definition posed a challenge in calibrating and assessing during implementation the overall level of ambition of the project. Given the different shocks encountered by the project, the team was forced to adjust objectives and targets to maintain their realism. 63. Simple and flexible design. The overall design of the project with two technical components (supporting rural production and competitiveness and strengthening institutional frameworks) and one project management component (project coordination and information management) allowed enough flexibility so that even with restructuring, the basic structure of the project was not altered. In addition, the structure of its components also facilitated the operationalization of (a) the micro-basin methodological approach; (b) the possibility of integrating knowledge produced by institutions related to the agriculture sector (participatory research); and (c) the Participatory and Negotiated Territorial Development approach as a way to build and strengthen the community organizations and their social fabric. These elements constituted a triad that enabled the achievement of results. 64. Readiness for implementation. Because the project was prepared taking into account the experience of the Rio GEF project, the readiness for implementation was guaranteed to a certain extent. The main challenges that were encountered had more to do with the lack of fiscal resources to operate than with the lack of capacity within the implementation unit and among the institutional and strategic partners on the ground. 65. Results Framework and impact evaluation. As designed, the M&E strategy called for a rigorous impact evaluation.21 During implementation, indicators in the Results Framework were adjusted to better capture response actions to the different shocks the project experienced, for example, emergency subprojects for the Serrana region in 2011, recovery of rural roads, and water investments. 21 See section IV A on Quality of Monitoring and Evaluation. 21 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) B. KEY FACTORS DURING IMPLEMENTATION 66. Natural disaster (flooding) and emergency response. In January 2011, exceptionally heavy rains occurred in the Serrana region of the SoRJ, the state’s most important production pole for horticulture, floriculture, and aviculture. Extensive flooding and mudslides in 11 municipalities resulted in over 850 deaths, left close to 20,000 people homeless, caused extensive damage to rural infrastructure (roads, bridges, rural buildings, and homesteads), damaged or destroyed crops and livestock and other productive assets, and led to disease outbreaks. In rural areas, most of the affected households were operators of small family-run farms, with little access to credit or agricultural insurance. As a consequence of this natural disaster, rural poverty in the state increased, reversing a declining trend. Rapidly restoring the productive capacity of these households became a government priority, to avoid further loss of income. The implementing agency SEAPEC State Secretariat of Agriculture and Fisheries estimated that 17,000 families depending on agriculture for livelihoods were affected, with approximately 3,400 families experiencing direct losses of assets and income. The total economic cost of the disaster in rural areas was estimated by SEAPEC at BRL 269 million (about US$163 million). 67. In the wake of the disaster, the state government requested the World Bank’s assistance to provide emergency support to affected farmers in rural areas to help them resume agricultural production. The types of assistance included the restoration of physical access through rural road rehabilitation, repair or reconstruction of damaged dwellings, and restoration of productive assets. These activities fell within the scope of the project, which had just started implementation in the Serrana region. The project was restructured; the PDO was modified to include the (sic) “Serrana Region affected by the January 2011 natural disaster,” and US$18.77 million of the original loan (IBRD) was reallocated into a new category of emergency response subprojects. At the same time, the number of beneficiaries and other target values related to the original results indicators were adjusted downward to compensate for the redirecting of project resources to the emergency activities. Emergency actions under the project began immediately after the catastrophe of January 2011, with resource releases occurring from May 2011 until the end of 2012. 68. Drought and emergency response. In late 2014 and early 2015, the SoRJ was heavily affected by a drought that hit the Southeast region of the country. This drought triggered another restructuring. The borrower requested the inclusion of water management subprojects (for example, storage and distribution of rainwater and groundwater for human consumption, drilling of tube wells, and installation of supply and distribution networks) and a drought management committee, within the scope of the project. Through EMATER-Rio technicians and producers, agriculture practices were identified that were suited to the needs of each affected holding. In addition to these activities supported by the project, producers who were trained by the Rio Rural GEF project helped in disseminating mitigation practices to other affected producers. 69. Fiscal crisis. The economic crisis affecting the state posed a substantial challenge for project implementation. The state’s fiscal situation deteriorated rapidly in 2015 when the Brazilian economy entered into recession after the price of petroleum fell sharply and the resulting drastic reduction of Petrobras investments and operations in the state affected the services industry. The recession precipitated a rapid rise in unemployment in 2015 and 2016, further depressing domestic demand, while the large fiscal deficits translated into rapidly rising public debt. General government gross debt rose from 56.3 percent of GDP at the end of 2014 to 69.5 percent by end of 2016. In 2016, the state government started paying salaries in installments. The crisis revealed the inadequacy of the state’s fiscal policy and 22 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) made clear that the efforts to diversify the economy and increase tax revenue had not been successful. As with other World Bank projects in the state, in August 2016, the judiciary seized undisbursed loan proceeds in the Designated Account, disrupting project activities. This crisis delayed project implementation for almost one year. 70. Strike of EMATER-Rio. During 2012 and 2013, EMATER-Rio employees went on strike, demanding improvements in salaries and working conditions. The main strategy adopted to put pressure on the state government was to boycott implementation of the Rio Rural project, interrupting project activities in the hydrographic micro-basins, including the sending of documents to the PMU. The strike lasted approximately one year and affected all regions of the state, causing significant delays for the project. There was disruption in the preparation of Micro-basin Executive Management Plans (PEMs), PIDs, supervision reports, and rendering of accounts. Following interventions by SEAPPA, the Executive Secretariat of the Rio Rural project, and the EMATER-Rio board of directors—and in the face of strong pressure from project beneficiaries—the strike ended and activities resumed. In the short run, the strike had a negative impact on the implementing agency and the project, but over the longer run, it had the effect of improving personnel management practices within public agencies including the implementation agency, increasing the relevance of merit criteria in the selection and promotion process. This had a longer-term positive impact on the capacity of the implementing agency. 71. Procurement. At decentralized levels, procurement activities were carried out without major difficulties by subproject beneficiaries using simplified procurement procedures approved for extremely small-value procurements (less than US$5,000). At the central (state) level, in contrast, procurement activities were greatly affected by the weak capacity of the PIU, which was reflected in implementation delays and high-risk ratings during most of the project’s life. The main challenges included the following:  In the initial years, the procurement of road maintenance machinery had to be repeated thrice because of poor specifications and the PIU’s alleged discrimination against foreign suppliers.  Despite the World Bank’s repeated recommendations, the PIU often applied inappropriate selection criteria for the hiring of individual consultants, which forced the World Bank to impose a temporary freeze on new hiring, except for key PIU positions.  An indication of possible governance issues in the procurement of gravel for road maintenance constrained the PIU’s strategy for road rehabilitation.  During the fiscal crisis, the seizure of loan proceeds by judicial decisions at the federal level halted project implementation for a sustained period during which no major procurement was carried out.  In 2018, two large national procurement tenders (National Competitive Bidding) for road rehabilitation were questioned by the state’s Court of Accounts (Tribunal de Contas) and because the PIU was unable to provide responses satisfactory to the Court on time, those tenders were eventually removed from the project. Other large procurements, for example, for the reconstruction of bridges and the installation of sanitation kits, were also dropped due to the PIU’s inability to prepare the tenders on time. 23 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 72. During the second half of 2018, the Procurement Plan was updated to include only smaller and simpler activities, removing the more complex activities that posed greater challenges to the PIU’s capacity. Thereafter, the procurement function improved noticeably, allowing the rating to be upgraded to Moderately Satisfactory. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 73. The scope of the M&E framework was broad, designed to systematically track and measure project implementation, accommodate midcourse corrections if needed, and demonstrate results for the productive, socioeconomic, and environmental elements of the project. Most project-related information would be collected at three points in time: (a) at the beginning of the project (baselines constructed using existing information previously collected by the Rio GEF and complemented by additional baseline surveys in the project’s first year and other available data); (b) at the time of the Midterm Review (Midterm Impact Evaluation); and (c) at closing (Impact Evaluation and a BCR). The Theory of Change, while not presented in diagram form in the PAD, was clear as explained through the project’s objectives, key activities, and strategies. The PDO specified the target population as the small-scale farming sector and clarified that increased productivity and competitiveness in this sector were higher-level objectives, that is, likely to be achieved beyond the life of the project. The technical and operational strategy supporting the PDO was conceptually logical. 74. The PDO indicators, while highly relevant, showed some weaknesses with respect to their complexity and vagueness in some cases. In addition, several required core indicators were omitted (number of beneficiaries, gender participation). The MIS was conceptually ambitious and based on participatory principles designed to ensure widely accessible streams of information intended to empower stakeholders to identify priority interventions needed in the sector, create continuous feedback to support project implementation, and create web-based, customized computer applications to be used by internal and external audiences. The decentralized system would build on existing information systems and databases from the Rio GEF project to monitor the project’s physical and financial progress. Baseline surveys and evaluation studies would be outsourced/shared, with some coordination activities and field surveys to be conducted by state and federal institutions (with recurrent costs covered). 22 M&E Implementation 75. The project agencies collected, analyzed, and disseminated key datasets; coordinated planned studies; and complied with fiduciary reporting requirements. During implementation, development of the originally planned MIS was abandoned due to restrictions on hiring of consultants. Part of the activities that were under the original MIS were developed by the project’s communication and monitoring teams, with participatory tools and the project's web portal. In addition to the regular MIS, the methodology for the planned Impact Evaluations was discussed and developed in the early years of implementation. Because the project developed an Impact Evaluation and also due to the different shocks faced early on 22Under this model, EMATER would conduct household-level socioeconomic surveys, EMBRAPA would generate soil and production system surveys, FEEMA (state environmental agency) would manage the surface water surveys, and the UENF would handle the biodiversity survey. 24 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) during implementation, a baseline was not done for the regular MIS, but only for the Impact Evaluation. A methodologically robust final Impact Evaluation was carried out using treatment and control groups and randomized sampling, which enabled identification of benefits attributable to the project. The Impact Evaluation was carried out in the Central and Southern regions of the state covering 120 micro-basins in 35 municipalities. It focused on three fundamental dimensions (impact domains): (a) profitability of rural properties; (b) social capital; and (c) and living conditions. A summary of the methodology and identification strategy is presented in annex 5. A summary of the main results by impact domain can be found in annex 6. The full Impact Evaluation reports are available in the project file and are cited in the bibliography presented in annex 10. 76. The BCR was loosely structured around the World Bank’s ICR template and was of good quality, informative, and directly useful to the ICR. Numerous studies (including baselines for research efforts beyond the project), evaluations, monitoring and activity reports were prepared over the course of the project, associated with specific types of productive and environmental practices, with the rural road and bridge investments, and with activities in progress. Project restructurings entailed several adjustments to indicators and targets in response to changing circumstances and realities on the ground including climatic and financial emergencies. They were designed to reflect different, improved, or more appropriate measures of project achievement and changed circumstances. The 2011 restructuring modified the PDO to introduce the concept of the emergency objectives and introduced new monitoring indicators to measure the achievements. M&E Utilization 77. Project monitoring data and periodic progress reports were valued inputs to management decision making as well as key research and reporting deliverables. These include the ISR, the BCR, the project economic and financial analysis, the final Impact Evaluation, supervision missions and Aide Memoires, institutional training programs, and project results dissemination activities. M&E data and analyses were disseminated through bulletins to rural communities, results/knowledge-sharing events for project technicians and managers, and through the project website.23 Reports were generated on environmental education and prevention of environmental shocks, as well as public campaigns such as “Rio 2016: Clean Water for the River and Atlantic Forest Connection.” Project data released on the Internet by the project’s institutional partners generated numerous studies, books and research articles, theses, and dissertations. Justification of Overall Rating of Quality of M&E 78. Overall M&E is rated Substantial. The M&E system, as designed and implemented, permitted assessment of the project’s achievements. Moderate shortcomings in design of the Results Framework are balanced against a strong evaluation and reporting performance during implementation and proactive dissemination and utilization of M&E products. Further, the World Bank flexibly and pragmatically adjusted the PDO and Results Framework in response to changing circumstances, to improve the quality and/or utility of project indicators and targets, and to ensure that the Results Framework adequately reflected project activities, objectives, and realities on the ground. 23 http://www.microbacias.rj.gov.br. 25 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental Safeguards Compliance 79. Environmental Safeguards Compliance was rated Satisfactory throughout the project’s life. Environmental risks were considered Low for most of the project life, except for a period of about 1.5 years (from late 2016 to early 2018) when the environmental performance risk was rated Moderately Unsatisfactory due to a delay in the implementation of mitigation measures to control erosion in bridges supported under the project. The delay was mainly associated with the state’s fiscal situation which subsequently, delayed procurement of erosion control works. Except for this issue associated with the bridges, all other environmental interventions (especially farmers’ transition to agroecological farming systems) were judged Satisfactory. 80. The project was classified Category B, which required an Environmental Management Framework. The project triggered five environmental safeguard policies: Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.04), Pest Management OP/BP 4.09), Physical Cultural Resources (OP 4.11/BP), and Forests (OP/BP 4.36). Performance was as follows:  Environmental impacts were for the most part positive, including enhanced soil nutrition and reduced erosion through improved cropland and grazing management; rehabilitation of degraded Atlantic forest areas, especially in riparian zones and around springs; sustained use/conservation of natural resources, leading to improved biodiversity; reduced use of pesticides in croplands; erosion control in rural road rehabilitation areas; and water conservation benefits and climate co-benefits.  Strong technical capacity of the project unit, project partnerships with environmental and agricultural research agencies and environmental NGOs, and strong support to community self-management of natural resources strengthened the environmental evaluation of all investments and definition of appropriate instruments. High overall compliance— collectively and individually—shows that farmers bought into environmental conservation measures and mitigation measures.  Integrated pest management techniques were disseminated, focused on biological controls, and types of parasites and tools for behavioral control of pests and insects.  No measures were needed to preserve physical cultural resources because no project activity placed cultural assets at risk. Further, the project’s strong natural resources management and conservation focus took natural habitats into account in all activities. 81. Climate change co-benefits. The project generated climate change adaptation and mitigation co- benefits in agricultural areas. Specifically, it enhanced climate change adaptation by (a) reducing exposure of communities and systems by conducting proper micro-catchment planning; this led to improved land- and water-use planning and management at the farm level, in turn promoting the adoption of improved cropland and grazing management, conservation agriculture, reforestation of riparian areas, and natural regeneration of vegetation around springs and (b) reducing sensitivity by enhancing soil nutrition and on- farm water management. Some climate change mitigation co-benefits were generated through the reduction and removal of GHG emissions through improved cropland and grazing management and 26 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) restoration of Atlantic forest especially in riparian areas and springs, as well as increasing feed-use efficiency and reducing forest degradation in grazing areas. Social Safeguards Compliance 82. Involuntary Resettlement (OP 4.12) was triggered on a precautionary basis. The project did not lead to any involuntary resettlement. When needed for community development activities, small plots were acquired through voluntary donation from project beneficiaries, following legal procedures that were thoroughly documented. Throughout its implementation, compliance of project activities with social safeguards was regularly supervised on a biannual basis. The project performance rating remained Satisfactory throughout the project’s life. Overall compliance with social safeguards is judged Satisfactory and social risk is Low. Fiduciary Compliance 83. Financial management (FM) performance was rated Moderately Unsatisfactory until the second semester of 2018, when the rating was upgraded to Moderately Satisfactory due to improved performance. The Moderately Unsatisfactory rating was mainly due to (a) late submission of interim financial reports (IFRs); and (b) project financial data being registered in two separate systems that required constant reconciliations. These included SIAFEM (SoRJ FM system) and its replacement Integrated System for Budget, Financial, and Accounts Management (Sistema Integrado de Gestão Orçamentária, Financiera e Contábil do Rio de Janeiro, SIAFE-Rio) (the state government’s public FM system, where expenditures are budgeted, committed, and accounted for and through which all payments are made) and System for Physical and Financial Monitoring (Sistema de Acompanamento Fisico Financiero, SAFF) (the project FM and monitoring system used for the preparation of IFRs). Discrepancies often appeared between the SIAFEM data and the IFRs. There was also a delay in the launch of SIAFI-Rio— the budgeting and accounting state system that replaced SIAFEM—which had an impact on the generation of the IFRs. 84. The project’s FM function was adversely affected by repeated seizures of loan proceeds. Because the SAFF was not adequately monitoring subproject implementation, funds in the Designated Account were seized in August 2016 in compliance with a judicial order. This had an impact on project execution by leading to a halt in any further advances to the Designated Account, while the Federal Government reaffirmed its commitment to continue with the project and committed to reimburse the sequestered amount. This event created a critical disruption to the project. Moreover, funds continued to be seized from the Designated Account, which at the end of 2018 totaled BRL 6,455,606.09 (including interest earned of BRL 3,069,967.24). 85. The FM risk rating varied between Moderate to High. The FM risk was changed to High at the end of the project, due to (a) the high risk of undocumented advances at the end of the project, with the World Bank/PIU exploring options to submit substitute expenditures, to cover both the amounts seized by the judicial authorities from the Designated Account (due to the state’s fiscal situation) as well subprojects unable/unlikely to document the resources transferred to them, before the expiry of the grace period; (b) high volume of subproject execution at the end of the project; and (c) the project’s final combined 2018 and 2019 audit report, which will be received only in September 30, 2019. 27 The World Bank Rio de Janeiro Sustainable Rural Development Project (P101508) 86. Except for the 2014 audit report, all other audit reports were submitted late. Apart from the 2012 and 2014 audit reports (which expressed a modified audit opinion, that is, qualified), all other audit reports expressed an unmodified opinion. 87. All IFRs received during the life of the project were considered acceptable (after revisions), but many were received late. As of April 8, 2019, the undocumented balance in the Designated Account according to Client Connection was BRL 59,782,829.26. Under the original arrangement, any undocumented balance remaining at the end of the grace period (April 31, 2019) would have to be refunded to the World Bank. 88. To date, no ineligible expenditures have been identified. The grace period was extended, and it will end on May 31, 2019. The final audit is expected after the end of the grace period. Procurement Compliance 89. The World Bank’s procurement oversight was continuous, comprehensive, and benefited from having a single experienced Procurement Specialist for the project duration. The PIU’s chronic inability to overcome procurement weaknesses made the procurement function especially challenging, as did major delays in procurement review and approvals by the State Court of Accounts (Tribunal de Contas), which negatively affected the efforts to maintain timely procurement processing and the scope and quantity of procurement packages. Planned road works and bridges were especially hard-hit. The World Bank responded through repeated procurement training, frequent adjustments to procurement strategy and planning, more frequent procurement supervision, and contracting a consultant familiar with World Bank procurement rules to support the PIU. Section III discusses procurement as a key factor affecting project implementation. In contrast to the experience in the central PIU, at decentralized levels, beneficiaries performed well in conducting direct procurement of goods and services through the agreed community participation in procurement modality (stipulated in the PAD), characterized by simplified procurement procedures and the small value of procurement contracts (