82232 Approach Paper Brazil: World Bank Group Country Program Evaluation, FY2004-2011 January 23, 2013 Background and Context 1. With a landmass in excess of 8.5 million square kilometers and a population of almost 200 million people, Brazil is the largest and most populous country in Latin America and fifth in the world on both measures. It is now also the world‘s seventh wealthiest economy with GDP of $2.2 trillion in 2011. Brazil has significant natural resources, including bauxite, gold, iron ore, timber, hydropower, nickel, uranium, and its economy has large and well-developed agricultural, mining, manufacturing, and service sectors. 2. Brazil has enjoyed political and economic stability over the past decade. A single party has been in power throughout the period, first under the presidency of Luiz Inácio Lula da Silva (January 2003–December 2010) and subsequently under the presidency of Dilma Rousseff. Balancing macroeconomic stability with broader development objectives such as growth, poverty reduction, transparency in government operations, and environmental sustainability have been major features of the Brazilian government‘s strategy throughout the period. At the start of his administration in 2003, President Lula kept the macroeconomic framework inherited from the Cardoso administration, aimed at fiscal responsibility and a primary surplus, inflation targeting, and a flexible exchange rate. Inflation declined sharply and reforms in the public sector balance sheet sharply reduced domestic debt indexed to foreign currencies, a major source of vulnerability. Public sector net debt fell steadily over the decade, from about 57 percent in 2002 to 38.5 percent in 2008. The counter-cyclical measures adopted during the global financial crisis raised the net debt ratio in 2009 to 42.8 percent, but it declined to 39.7 percent in June 2011. 3. The reduction in macroeconomic instability and inflation and a benign external environment allowed Brazil to resume moderate growth from 2004. GDP grew by nearly 5 percent per year in the period between 2004 and 2008, albeit with some fluctuations. The global financial crisis led to a 4.2 percent contraction in GDP in the fourth quarter of 2008, and a 2 percent contraction in the first quarter of 2009. But the country‘s sound fundamentals and prompt responses by the authorities helped mitigate these declines. Brazil was one of the last nations to fall into recession in 2008 and was among the first to resume positive growth; after experiencing a -0.3 percent growth in 2009, Brazil grew at 7.5 percent in 2010. Brazil has also made considerable progress in its long-term foreign currency sovereign credit ratings. Standard & Poor‘s rating for Brazil improved by 4 notches from non-investment grade BB- in 2003 to above investment grade of BBB in 2011. 4. Between 2004 and 2012, the World Bank and IFC have worked in partnership with the government of Brazil under the Country Assistance Strategy (CAS) 2003-2007, dated November 10, 2003, and the Country Partnership Strategy (CPS) 2008-2011, dated May 6, 2008. A progress report for the CPS FY08-11 was prepared in March 2010. IBRD and IFC are currently operating under the new CPS for FY12-15. In that context, IEG proposes to evaluate the results of the country program between FY04 and 11. 1 5. Development Gains and Ongoing Challenges. Over the past decade, Brazilians have benefited from sustained, stable economic growth, relatively low inflation rates, and improvements in social well-being. Poverty has declined from 35.8 percent of the population in 2003 to 21.4 percent in 2009 (representing an escape from poverty for about 22 million people); and extreme poverty fell from 15.2 percent in 2004 to 7.3 percent in 2009 (representing an escape from extreme poverty for about 13 million people).1 Between 2001 and 2011, the income the poorest 10 percent of the population grew on average by 6.7 percent per year, while that of the richest 10 percent by 1.55 percent. This helped reduce income inequality (measured by the Gini index) to 0.527 in 2011, down from 0.594 in 2001. A range of non-income indicators have also improved. For example, malnutrition among children under five has been halved since the 1990s and 98 percent of children aged 7-14 are enrolled in education. Gender differences in access to education have been virtually eliminated, although the participation rate among boys is now less than that of girls, particularly in the latter stages of secondary schooling (CPS FY12- 15). 6. The CPS FY08-11 notes that about half of the reduction in poverty is explained by economic growth. The other half is due to reduced inequality, mostly due to the particularly high growth of labor incomes as well as government transfers (in equal proportions), indicating the important role played by the safety net programs and other social transfer programs. 7. Most observers agree that further accelerating economic growth will require sharp increases in investment rates, which have been low—an average of 16.7 percent of GDP during 2000-09 and reached 20.2 percent in 2010 and 19.7 percent in 2011—relative to other comparator emerging markets over the past decade. The investment rate seems to have been influenced by poor incentives to invest, particularly for the private sector, and by low savings rates. Infrastructure investment (private and public) in power, transportation, telecommunication, water and sanitation fell substantially from about 5 percent to 2 percent of GDP between the early 1980s and the 2001-06 period. Low aggregate savings—at about 17 percent of GDP—calls for policies that can encourage public and private savings. In this regard, further reforms in social security systems (given the rapid aging process) are particularly important to increase public sector savings. 8. To increase productivity and private investment, including in infrastructure, there is a need for regulatory frameworks and concessions that encourage more private sector participation by improving the environment for competition and reducing the cost of doing business. High domestic interest rates associated with a segmentation of the credit markets and insufficient competition in the financial markets also impair private investment. 9. The quality of education will also have a significant impact on the level and potential for continuing economic growth. Based on international test results of learning in schools (OECD 2010), Brazilian children lag in critical reading and mathematics skills compared to most other OECD nations. Improvements in the quality of education will be critical to improve the productivity of workers at the lower end of skills and contribute to poverty reduction. 1Based on the poverty and extreme poverty lines calculated by the Instituto de Pesquisa Econômica Aplicada (IPEA) 2 10. Overall productivity will also need to improve to sustain high growth levels, especially in light of recent discoveries of very large oil reserves, which could move Brazil from the 14th to the 7th place in the world for such reserves. The discovery, if handled incorrectly, could generate a significant ―Dutch Disease effect‖ and an adverse impact on economic competitiveness. To maintain its competitiveness, the Brazilian economy will require even faster improvements in productivity. 11. In addition to increasing growth and productivity, Brazil faces challenges associated with improving the quality of growth, which include:  Extreme regional differences, especially in social indicators such as health, infant mortality and nutrition, with the richer South and Southeast regions far outperforming the poorer North and Northeast  Reduced but still significant poverty in absolute number–13.95 million people continue to live in extreme poverty in Brazil2  The quality of government services in relation to expenditures, which remains relatively low compared to other middle-income countries  A variety of environmental challenges associated with deforestation and the need to combine agricultural growth, environmental protection, and sustainable development. 12. World Bank Group Activities in Brazil. The country strategy documents since September 2003 describe efforts and initiatives to develop the partnership between Brazil and the Bank Group (the International Bank for Reconstruction and Development, International Finance Corporation, and Multilateral Investment Guarantee Agency). The priorities evident from consultations among the Bank Group, the Brazilian authorities, and other stakeholders during the formulation of the three country strategies are as follows:  A shift in focus from the federal level to the state level (while maintaining relationships and a coordination structure at federal level)  A consequent shift from large-scale development policy lending (DPLs) at the federal level to a more balanced mix of lending instruments, Sector-Wide Approaches (SWAps), and analytic and advisory activities (AAA) with increased focus on support to states  Growing interest among the Brazilian authorities in Bank Group support for improving the level and the quality of public and private investment in infrastructure as noted in the March 2010 CPS Progress Report and the CPS Completion Report of 2011  A more aggressive pursuit of growth through significant investment, particularly in infrastructure, supported by growing revenues and measures to increase private financing  Increased emphasis on quality of public services rather than quantity (for example, education quality over school enrollment)  An assistance strategy that is more flexible and demand-led, implying a more floating or opportunistic approach managed on a programmatic basis  An increasing emphasis on learning and dissemination of good practice across states and regions, including a program of AAA that focuses more on the ― how‖ than on the ― what‖ 2 Based on the extreme poverty line used in paragraph 5. 3  From an IFC perspective, a shift in focus from large and established Tier 1 companies to mid to small size Tier 2 companies and a more determined engagement with the poorer, frontier regions.. 13. Although emphasis may have shifted over time, the basic goal and the pillars of Bank Group support have remained stable over time. The overarching goal of Bank Group operations has been to support Brazil‘s aim of accelerated, more inclusive and more environmentally sustainable growth. The FY04-07 CAS and the FY08-11 CPS use an identical pillar structure (Figure 1). These four key pillars support activities for a more equitable, sustainable, and competitive Brazil underpinned by sound macroeconomic management and overall public sector governance. Although the current FY12-15 CPS uses slightly different language, its priorities follow the same structure as described in Figure 1. Figure 1. CAS/CPS Pillars A More Equitable, Sustainable, and Competitive Brazil More Equitable: Human Capital More Sustainable: Natural More Competitive: Investment and and Social Development Capital and Local Services Productivity  Reducing extreme poverty,  Better water quality and water  Improved infrastructure, vulnerability, and social resource management investment climate, and exclusion  More sustainable land competition  Better knowledge and skills management, forests, and  Broader and more efficient  Living longer, healthier lives biodiversity financial sector  More equitable access to local  More modern innovation climate services Foundations of Economy and Governance Sound economic management and fiscal reform More efficient public sector management Good governance 14. IBRD Program. Under the FY04-07 CAS, IBRD planned to lend up to $7.5 billion with a major role for DPLs (50 percent of the lending), SWAps, and subnational lending. Actual lending was less than that due to reduced financing needs and lower-than-expected levels of lending at the subnational level (particularly at the municipal level). Effective lending was about $5.0 billion (most occurring in the first three years) in 32 operations. Fifty-three percent of the projects planned for the first two years of the CAS materialized, but only 15 percent of those planned during the second two years reached the implementation stage. DPLs accounted for 60 percent of the lending volume, $2.94 billion, SWAps accounted for 25 percent. Over the period, the IBRD also committed $56.4 million in 24 grants and special financing projects including the Global Environmental Facility, the Pilot Program to Conserve the Brazilian Rainforest, the Institutional Development Fund and the Carbon Offset program. 15. During the CPS FY08-11 period, a total of $11.8 billion in IBRD loans was approved, well above the amount in the FY04-07 CAS ($5.0 billion). The CPS Completion Report notes that the increase in commitments was driven by: (a) the growing demand for technical know-how and financial resources by the subnational governments at the state level; and (b) additional resources provided during the global financial crisis. The loans were extended mostly in the 4 form of investment lending, but including substantial DPLs in FY09-10 (nearly 40 percent of the total). Seventeen planned operations did not materialize for various reasons. Over this period, the IBRD also committed $116.4 million in 16 grants and special financing projects (Figure 2). The net disbursement from FY04 to FY11 fluctuated between $1.4 billion (FY06) and negative $1.1 billion (FY11), due partly to prepayments of IBRD debt in FY10 and FY11.3 Figure 2. IBRD Program CAS FY04-07 CPS FY08-11 Actual lending ($ billions) 5.0 11.8 Grants and special financing projects ($ billions) 0.06 0.12 Source: World Bank 16. The outcome ratings of 85 percent of investment lending projects that exited the active portfolio between FY04 and FY12 have been rated moderately satisfactory or better by IEG. The share was 100 percent for 8 DPLs rated by IEG. They are consistently higher than the average for the Latin America and Caribbean Region, which are 79 percent of investment lending and 83 percent for DPLs. 17. During FY04 and FY11 transport, Figure 3. IBRD Commitment Amount by Sector Board public sector governance, the (FY04-11) ($US million) environment, the financial and private sector, and social protection accounted for 3500 64 percent of all funding (Figure 3). 3000 Operations peaked in FY10, with $3.74 2500 billion in lending, owing predominately to 2000 significant lending in the transport, 1500 energy, and education sectors. 1000 500 18. Each CAS claims that AAA 0 (consisting of technical assistance and economic and sector work) has a transformative role in the implementation of Bank Group strategy. By far the most significant level of nonlending technical Source: World Bank data. assistance is in the urban development sector (57 percent) followed by environment (11 percent), public sector governance (10 percent), and energy sector (6 percent). Three of the top five sectors for lending are also among the top five for nonlending technical assistance. One exception is urban development, which predominates among non-lending technical assistance activities, but comprises only 8 percent of total IBRD lending. With regard to ESW, three of the top five sectors for lending (public sector governance, financial and private sector development, and social protection) also feature among the top five sectors for ESW support. 3In June 2011, Brazil made a prepayment of $3.1 billion of its IBRD debt to enable continuous engagement with IBRD under the Single Borrower Limits. A similar prepayment of $866 million was undertaken in December 2009 as well. 5 19. IFC Program. During FY04 and FY11, IFC committed a total of $5.6 billion across 111 investments, making Brazil one of its largest investment portfolios. Fifty-seven percent of all IFC projects between FY04 and FY11 supported trade finance, 17 percent operated in financial markets, and 10 percent supported infrastructure development (Figure 4). IFC engaged in 30 Advisory Services operations over the FY04-11. Of these, 6 remain active, 1 was dropped and 23 were completed. During this period, IFC committed approximately $12.7 million to these operations, of which the public-private partnership business line comprised the largest total component at $7.8 million (Figure 5). Figure 4. FY04-11 IFC Net Commitments (% of Figure 5. IFC Advisory Services - Total Cost by total in US$ million) Business Line: FY2005-FY11 (US millions) 9.0 Agribusi- ness , 6% Others 8.0 11% 7.0 Infrastruc- 6.0 ture , 10% 5.0 Trade Finance 4.0 Financial 57% 3.0 Markets 17% 2.0 1.0 0.0 Trade Finance Financial Markets Infrastructure PPP Sustainable Investment Access to Agribusiness Others Business Climate Finance Source: IFC data. Source: IFC data. 20. During the last CPS period, IFC expanded its activities in Brazil. IFC‘s commitments increased over 200 percent to $4.0 billion in FY08-FY11, from US$1.48 billion in FY04–FY07. In the last three fiscal years, IFC‘s operations with financial institutions continued to be dominant, and represented on average 81.8 percent of IFC‘s own commitments in FY09-11. A large part of the IFC program with the financial sector consisted of trade finance guarantees, and IFC‘s guarantee operations accounted for 71 percent of IFC‘s own commitment during FY09-11. 21. In terms of geographical focus, IFC has been increasing its strategic focus on the Northeast region in line with the Government‗s national program to eradicate extreme poverty. IFC financed ten operations in the North and Northeast regions of Brazil during the FY08-FY11 CPS period for a total of $389 million in infrastructure, financial services, and manufacturing. With respect to the overall portfolio, IFC's committed balance stood at $2.67 billion at the end of FY11, of which $2.24 billion had been disbursed, making Brazil IFC‗s second largest partner country. The portfolio consisted of 72 active client companies and was performing soundly according to the CPS Completion Report. 22. MIGA Program. MIGA operations provide political risk insurance, which helps to ensure greater investor confidence and thus, the financing that underlies IFC investments. Over the review period, MIGA provided a total of $1.75 billion in risk insurance guarantees through 39 projects to the LCR region. Of these, MIGA‘s activities in Brazil accounted for 6 approximately 18 percent of its LCR portfolio, with an aggregate coverage of $314.6 million (Figure 6), most of which were in the power sector ($246 million in risk exposure) (Figure 7). Figure 6. MIGA Guarantees Issuance Gross Figure 7. MIGA Guarantees by Sector, FY04-11 Exposure, FY04-11 (US$ million) (Gross Exposure in US$ million) 100 Capital Services , Markets, 2 80 67 60 40 Power, 20 246 0 FY04 FY05 FY06 FY07 FY09 Source: MIGA data. Source: MIGA data. 23. Links to Previous Evaluations and Existing Literature. Each CAS/CPS for Brazil between 2003 and 2011 refers to and takes into account, the findings of the CAS Completion Report (CASCR) for its immediate predecessor. Each CASCR is subject to an IEG CASCR- Review. Furthermore, each of the projects exiting during the period was subject to an Implementation Completion and Results report (ICR) and an IEG-conducted ICR Review and, in certain instances, projects exiting were also subject to a Project Performance Assessment Report (PPAR). There are also IEG reviews for IFC investments and advisory services. The evaluation will build upon these reviews as well as the more comprehensive analyses contained in the Brazil country assistance evaluation that examined the earlier decade (IEG 2004). It will also reference, as relevant, evaluation reports on wider topics that refer to Brazil, such as: World Bank Engagement at the State Level (IEG 2010); Development Results in Middle-Income Countries (IEG 2007); reports from the Global Program Review series; and thematic evaluations on the subjects of gender, youth employment, crisis response, environment, forests and safeguards. Research and evaluation material generated by the research/evaluation community within Brazil will also be consulted. Purpose, Scope, and Audience 24. Purpose of the Evaluation. The primary purpose of this evaluation is to identify lessons that help inform future engagement of the Bank Group with Brazil in order to enhance its development effectiveness. It is expected that some of the findings could inform discussions on Bank Group engagement with middle-income countries in general given the increasing importance of such countries in the context of the Bank Group‘s mission and progress toward achieving the Millennium Development Goals. 25. Evaluation Scope. This country program evaluation will assess the relevance and effectiveness of the World Bank Group‘s operations in Brazil between FY04 and FY11. It will therefore include the entire duration of the FY04-07 and FY08-11 country strategies. As appropriate, the evaluation will comment on aspects of the ongoing CPS FY12-15 with particular 7 reference to its relevance and design. The evaluation will examine the specific contribution of Bank Group to overall development outcomes and extract lessons relevant to future Bank Group operations in Brazil and in middle-income countries more generally. This will take into account differences in emphasis over time reflecting changing government priorities (such as the renewed emphasis on economic growth during the second CAS) and other factors, such as the effects of the financial and food crises. 26. Stakeholders and Audience. The principal stakeholders include the World Bank Group Board, Bank Group management and staff working on Brazil, and federal and state administrations in Brazil. The intended audiences also include administrations in other middle-income countries, the development community (academics, think tanks, and nongovernmental organizations), other multilateral development banks and donors (including the Inter-American Development Bank and the European Union), the media in Brazil and elsewhere, and the citizens of Brazil. Evaluation Questions and Coverage 27. The evaluation will seek to answer the following questions:  To what extent was the assistance of the Bank Group relevant to Brazil’s development needs? How well did the Bank Group exercise selectivity with a view to maintaining relevance in the context of Brazil‘s evolving priorities? Realism of the country program objectives and the quality of the CAS‘s results frameworks will be important features to be reviewed. The evaluation will also examine how well the Bank Group adapted to the changing external environment faced by Brazil, including the 2008- 09 global economic crisis.  How effective was Bank Group operation in helping to accelerate economic growth and making such growth more inclusive and environmentally sustainable? Supporting the Government of Brazil‘s goal of achieving higher rates of inclusive and sustainable growth is the underpinnings of the four strategic pillars of the successive strategies. The evaluation will assess the Bank Group‘s contributions through lending, investments, and knowledge services. Issues such as the consistency and synergies between various interventions (policy lending, investment lending, SWAps, guarantees, IFC short-term financing, knowledge services, and capacity building), demonstration effects and replicability, scaling-up and sustainability, and dialogue with the federal and state clients will be assessed. It will examine outcomes in areas on which the strategy put a special emphasis, for example, its focus on states with high concentration of poverty. The evaluation will aim to identify the drivers of success and lessons to be learned from these analyses.  How effective was collaboration within the Bank Group and with external development partners in helping Brazil improve its development outcomes? Given that the country strategies have been jointly developed and implemented by IBRD and IFC, the evaluation will also assess the alignment of the strategic approaches and operational activities between the IBRD and IFC in Brazil. MIGA operations will also be reviewed in the context of Bank Group–wide synergies when appropriate. Further, the effectiveness of collaboration with other development partners (such as the Inter- 8 American Development Bank, International Monetary Fund, bilateral agencies, and nongovernmental entities) will be assessed. Good practices and lessons identified in the assessment will be reported. 28. The evaluation framework is summarized in Figure 9 below and a detailed design matrix is in Attachment 2. Figure 9: The Major Evaluation Questions Evaluation questions Areas for assessment To what extent was the assistance  Selectivity applied to maintain relevance and ensure value added of the World Bank Group between  Quality of results framework FY04 and FY11 relevant to Brazil’s  Realism of strategy and design in achieving country program objectives development needs?  Flexibility to adapt to changes and capacity to learn from experience How effective was Bank Group  The extent and nature of contributions to accelerating economic growth and assistance in accelerating ensuring the quality of growth economic growth and ensuring the  The consistency and synergies between various types of interventions quality of growth?  Demonstration effects, replicability, scaling up, and sustainability of development results  Quality of dialogue with counterparts  Outcomes in areas with strategic emphasis  Drivers of success and lessons How effective was collaboration  The extent of intra-Bank Group collaboration and associated outcomes within the Bank Group and with  The extent of collaboration with development partners and associated outcomes external development partners in  Lessons learned helping Brazil improve its development outcomes? Evaluation Design Issues 29. As noted earlier, the first two country strategies were organized under nearly identical, broadly stated objectives (a more equitable, sustainable and competitive Brazil underpinned by relevant public sector reform). The current CPS focuses on improved quality of and coverage in public services for low-income households (largely an equity issue), sustainability, economic development (competitiveness), and greater efficiency in public and private investment. In effect, the pillars under which the CAS/CPS have operated have not changed substantively over the period. This consistency facilitates the evaluability of the Bank Group‘s assistance during the period. This evaluation will assess the effectiveness of Bank Group operations in relation to the two overarching development challenges for Brazil—accelerating growth and making such growth inclusive and environmentally sustainable. The Issue of Scale 30. The modest scale of Bank Group financial support in relation to Brazil‘s economy presents some special evaluation challenges. While the extent and scale of Bank Group activities in Brazil over the FY04-11 period are significant from the Bank Group‘s perspective, that 9 support is small in the context of the overall Brazilian economy. In 2011, for example, World Bank lending in Brazil ($2.54 billion) represented 0.3 percent of all public expenditure. 31. Recognizing this disparity in scale, the FY08-11 CAS explicitly acknowledges that Bank support in the preceding period (FY04-07) was effective in areas where Brazil did well and less effective where Brazil was less effective. In order to remain both relevant and effective, each CAS emphasizes the need for increased agility and flexibility by Bank Group institutions so as to adapt to emerging priorities and realities. The FY04-07 CAS references the need to use ―programmatic selectivity‖ and the FY08-11 CPS suggests a need for ― principled opportunism‖ in the Bank Group‘s engagement with Brazil. 32. Given that the Bank Group financing is very small relative to the size of the economy, this evaluation will focus on identifying the catalytic role of the Bank Group strategy and operations. In identifying catalytic effects, it is important to review the contributions made by the policy reforms supported or not supported by the Bank Group. The evaluation will examine whether the intermediate objectives set to achieve higher-level goals were appropriate. It will also assess whether the related interventions produced satisfactory benefits and generated impact that led to broader results beyond their immediate beneficiary groups. 33. For investment projects, it is important to examine whether the projects were replicable or had demonstration effects at the federal, state, or municipal levels. In addition, the extent to which projects may have embodied innovative knowledge and whether such knowledge was disseminated beyond the direct beneficiaries will be analyzed. The widespread use of SWAps has been a major feature of the program and the evaluation will assess their cataltytic role relative to more traditional investment operations in similar areas. 34. The impact of AAA and ESW on the policy dialogue at the state and federal level as well as generating catalytic effects outside the government will be explored. Technical assistance to subnational government has been extensive in some areas and efforts will be made to assess whether it was been effectively used and disseminated. Further, and of significant import, the evaluation will assess whether ESW and technical assistance were used to create synergies with financing assistance. In this context, the evaluation will assess whether the total impact of a set of related interventions was larger than the simple sum of its component parts. Methodology and possible areas of focus 35. The evaluation will follow the standard methodology for IEG‘s Country Program Evaluations. The evaluation will assess and rate the outcome of the Bank Group country program relative to its objectives. In rating the outcome (expected development results) of the program, the evaluation will gauge the extent to which major Bank Group strategic objectives were relevant and were achieved. The analysis will be undertaken through: (i) a top-down review of whether the Bank Group program achieved a particular Bank Group objective or planned outcome and had an impact on the country‘s own development objectives; and (ii) a bottom-up review of the Bank Group‘s products and services for each sector, theme and CAS/CPS pillars used to achieve the objective. The evaluation will then consolidate and integrate the findings from three types of analyses as indicated below. The details may change as the analysis progresses and new findings emerge. 10 a. Examining the CAS/CPS and Project results frameworks: The team will review the country program design as indicated in the CAS/CPS to identify the results chain and intermediate outcomes linking the CAS/CPS outcomes (see Attachment 3 for key features of CAS/CPS matrix) and individual Bank Group operations. The team will also examine what catalytic effects and synergies among Bank Group interventions were envisaged in these results chains. The review will focus on major interventions such as DPLs, SWAps, large investment projects, or selected municipalities or states where higher concentrations of Bank Group interventions are observed. b. Reviewing past evaluations: The team will review past evaluations at both the country program and project levels. It will examine CAS completion reports, CAS completion report reviews, ICRs, ICR Reviews, PPARs and other evaluations. It will identify and compare patterns of findings across these evaluations, identifying areas where the findings are consistent and those that exhibit discrepancies. c. Outcomes and Bank Group contribution: An effort will be made to identify how Bank Group interventions contributed to higher-order outcomes. An initial step in the analysis will be to identify the consistency between observable outcomes at the country and individual intervention levels. This analysis will provide a basis for further assessments of the quality of the original results framework, the types of unanticipated events that affected the outcomes, and unintended consequences of the interventions. The team will build on the information collected through interviews of Bank Group staff, country authorities, and other stakeholders in the field. 36. The analyses will involve desk-based research and interaction with the country team staff and other relevant experts, following a common structure to allow for the extraction and aggregation of issues arising on a cross-sectoral basis. The team will also review the geographic and thematic distribution of Bank Group operations, including DPLs, investment lending, AAA, and IFC investments and advisory services, to identify the areas and themes with high concentration of Bank Group operations. There will be close coordination with other IEG teams conducting evaluations of relevant operations, for example, ongoing PPARs. Specific areas will be examined further to identify the synergies among various operations and possible catalytic effects through replication of good practice by other states or localities. The evaluation will also look for indications of contributions from policy dialogue as well as analytical and advisory activities in the replication process. 37. In addition, the evaluation will take a dynamic approach to assessing the Bank Group‘s responsiveness to changing demands. Although the Bank Group program has been based on a stable pillar structure, within that framework the Bank has shifted its approach over time to deal with dynamic change in the external environment and the evolving nature of the relationship between Brazil and the Bank Group. This has involved a shift in focus from the federal to the state level, an associated shift in the instrument mix (DPLs, SWAps, and knowledge services), and from a preplanned portfolio of investment to a more flexible, demand-led approach to investment. Taking these into account, the evaluation will assess how well the Bank Group adapted its strategy in responding to change and associated challenges. In addition, the evaluation aims to extract lessons that may be applicable to Bank Group engagement with emerging middle income countries more generally. Potential examples include such areas as 11 flexibility in the country program, the multi-sectoral nature of the approach, contributions to global public goods and collective action, and knowledge-intensity of the country program. 38. Analytic Methods and Data Requirements. The evaluation will use a combination of quantitative and qualitative methods for the assessment. Portfolio data will be obtained from Bank Group databases, and IEG data obtained from previous evaluations. Qualitative analysis will be based on the data and information generated by existing research and evaluations as well as by the desk review of operational documents. The team will generate additional qualitative data through semi-structured interviews with representatives of key stakeholders and possibly through surveys and case studies. 39. Major sources of existing information and data for the Brazil CPE include:  CAS Completion reports and corresponding IEG reviews  Assessments of Bank Group analytical and advisory work  Data on the scale and distribution of knowledge services (ESW/technical assistance/IFC Advisory Services) as well as lending, grant, investment, and guarantee products of the IBRD, IFC and MIGA  Data from sources other than the Bank Group including the International Monetary Fund, the Inter-American Development Bank and relevant institutions in Brazil  Secondary analysis of IEG‘s evaluations of the Bank Group‘s lending, grant, and guarantee operations from ICR reviews (IBRD), PPARs (IBRD), Expanded Project Supervision Reports (IFC), Project Evaluation Reports (MIGA), and other aggregated evaluations 40. Additional information and data will be generated through interviews with Bank Group managers and staff, counterparts in Brazil, relevant experts in the research community and civil society organizations. Subject to further scoping, a survey of key stakeholders and/or case studies that focus on one on more regions or states may be explored. Feedback providers will be identified through a number of channels, including: counterparts in preparing CASs/CPSs; those consulted in the context of knowledge work; and referrals from the country team. In addition, the evaluation will review good practices among other agencies outside the Bank Group for benchmarking purposes and to determine how the Bank engages in standard setting and knowledge work with other institutions. Quality Assurance Process 41. Quality assurance will be ensured through a peer review process as well as IEG internal review process. Peer reviewers for the evaluation are: Ariel Fiszbein (Chief Economist, Human Development Network) and Thomas O‘Brien (Manager, Development Effectiveness, South Asia Region). The report will be prepared under the direction of Ali Khadr, Senior Manager, and undergo the standard IEG quality assurance process. Expected Outputs and Dissemination 42. The primary output of the evaluation will be an evaluation report that will contain the main findings and recommendations. The report will also identify areas where further work is 12 needed for development of more in-depth recommendations. Background papers prepared in support of this effort will be considered deliberative in nature and therefore not subject to disclosure. The report will be disclosed publicly in accordance with IEG‘s disclosure policy. Appropriate opportunities for disseminating the report in Brazil will be explored. Resources 43. The evaluation will be undertaken in FY13, with a CODE Subcommittee discussion expected to be scheduled in the early part of FY14. 44. The team will be led by Jiro Tominaga (Senior Evaluation Officer, IEGCC). The team will consist of Marcelo Selowsky (Senior Consultant), Jaime Biderman (Consultant, Infrastructure), Susan Caceres (Senior Evaluation Officer, IEGPS), Ken Chomitz (Senior Adviser, IEGPS), Corky de Asis (Program Assistant, IEGCC), Kutlay Ebiri (Consultant, IFC), Takatoshi Kamezewa (Senior Evaluation Officer, IEGCC), Tony Tyrrell (Consultant, IEGCC), Silvina Vatnick (Consultant, financial sector), Carlos Eduardo Valez (Consultant, Social Protection), and Cameron Wilson (Consultant, IEGCC). The team will be in close coordination with any IEG teams scheduled to conduct PPAR or other types of evaluations of Bank Group operations in Brazil. 13 Attachment 1 References Independent Evaluation Group. 2004. Brazil: Forging a Strategic Partnership for Results: An OED Evaluation of World Bank Assistance. Washington, DC: World Bank. ______. 2007. Development Results in Middle-Income Countries: An Evaluation of the World Bank’s Support. Washington, DC: World Bank ______. 2010. World Bank Engagement at the State Level: The Cases of Brazil, India, Nigeria and Russia. Washington, DC: World Bank OECD (Organisation for Economic Co-operation and Development). 2010. PISA 2009 Results: What Students Know and Can Do. OECD Publishing. Paris. World Bank. 2003. Country Assistance Strategy 2003-2007 for the Federative Republic of Brazil in Support of a More Equitable, Sustainable, and Competitive Brazil. Washington, DC: World Bank. ______. 2008. International Bank for Reconstruction and Development and International Finance Corporation Country Partnership Strategy for the Federative Republic of Brazil for the Period FY2008-2011. Washington, DC: World Bank. ______. 2010a. International Bank for Reconstruction and Development and International Finance Corporation Country Partnership Strategy Progress Report for the Federative Republic of Brazil for the Period FY2008- 2011. Washington, DC: World Bank. ______. 2010b. World Bank Engagement at the State Level: The Cases of Brazil, India, Nigeria and the Russian Federation. Washington, DC: World Bank. ______. 2011. International Bank for Reconstruction and Development and International Finance Corporation Country Partnership Strategy for the Federative Republic of Brazil for the Period FY2012-2015. Washington, DC: World Bank. 14 Attachment 2 Evaluation Design Matrix Key questions Information required Information sources Data collection methods Data analysis methods Strengths and limitations To what extent was the Establish the socioeconomic Official statistics (Brazil – Online search for relevant Collation and analysis of key No particular difficulty in assistance of the Bank Group context in Brazil FY04-11 federal, regional, and state external documentation, data, socioeconomic data and collecting written materials is between FY04 and FY11 Identify major development levels, Bank Group, IMF, IDB, research, evaluation and indicators over the period expected given the known relevant to Brazil’s challenges etc.) analysis Literature review availability of high quality data development needs? Relevant literature and Collation of available data and and analysis on Brazil. Establish the analytical base Desk-based review of relevant for the CAS/CPS analytical work documents produced by the analytic work The relative economic and IEG reports Bank Group, Brazilian political stability in Brazil over Identify development authorities, and research Portfolio analysis the period means that it will be objectives selected by the Key stakeholders; community Retrospective review of the possible to access “institutional Bank Group development partners and results frameworks memory” reaching back over area experts Semi-structured interviews with key stakeholders the entire span of the period Qualitative analysis of Country-team and relevant under study. Possible state-level case interviews and surveys Bank Group management The key limitations are studies and survey of key stakeholders. associated with the size of the Bank Group investment relative to the size and diversity of Brazil’s economy. The overarching difficulty of identifying and assessing relevance and added value in this context will be challenging. The team plans to tackle the evaluation questions by using multiple sources of information including: (i) reliable and independently generated data and research, (ii) IEG and other Bank Group data; and (iii) the views of key stakeholders gathered through interview, survey, and other means. 15 Key questions Information required Information sources Data collection methods Data analysis methods Strengths and limitations How effective was Bank Group Data and analysis relating to Official data, analysis and Online search for relevant Desk-based review and Same as above assistance in helping to economic growth in Brazil other reports (e.g., generated external documentation, data, analysis of results frameworks accelerate economic growth FY04-11 by Brazilian state, OECD, research, evaluation and and available monitoring data and making such growth more Performance of lending and International Labor analysis Desk-based analysis of Bank inclusive and environmentally nonlending operations Organization, United Nations) Collation of available data and Group and IEG performance sustainable? Bank Group analytical documents produced by the related documents and data to Results of reforms supported by the Bank Group products and self-evaluations Bank Group, Brazilian include databases, ICR- Bank Group reports and authorities, and the research Reviews, XPSRs, PPARs, Bank Group contribution to community CASCRs, etc. observed and established documents; relevant AAA; improvements in Bank Group and IEG Semi-structured interviews Desk-based review of relevant competitiveness, equality, or databases with key stakeholders aspects of AAA sustainability at federal, IEG assessments; external Possible state-level case Desk-based review of country regional, state or municipal research; assessment by studies and survey of key management materials levels counterparts stakeholders Qualitative analysis of Degree of synergy achieved Key stakeholders and Bank interviews and focus group between interventions and Group staff and management sessions degree to which a Survey analysis using SPSS multisectoral approach was and/or other tools realized How effective was Level and type of intra-Bank Bank Group analytical Collation of available data and Qualitative analysis of In addition to above, the collaboration within the Bank collaboration products and self-evaluations documents produced by the interviews and focus groups planned interviews and Group and with external Processes or structures put in IEG assessments Bank Group, Brazilian Desk-based review of country- document reviews will seek to development partners in place to facilitate intra-Bank authorities, and the research team documentation such as take into account and explore helping Brazil improve its External research community this type of collaboration and, collaboration meeting notes, development outcomes? Development partners and Collate relevant Bank Group correspondence, etc. as necessary, research will Results attributable to intra- other stakeholders attempt to identify proxy Bank collaboration analytical products and self- Desk-based review of relevant evaluations indicators of success that can Country team personnel AAA products Level and type of collaboration be validated through the with development partners Country-team documentation Collate IEG assessments Desk-based review of Bank evaluation process. (e.g., meeting notes, Interviews of representatives Group and IEG performance Processes and structures put correspondence) in place to facilitate of the country team and other data and associated collaboration with development Other stakeholders key stakeholders documentation partners Possible case study at the Results attributable to level of the state and semi- collaboration with external structured interviews with key development partners stakeholders 16 Attachment 3 Key Features of CAS (FY04-07) /CPS (FY08-11) Matrices CAS Country Medium Term Outcomes CAS Outcomes Possible Issues and Development Obstacles (at time of Goals4 CAS/CPS preparation) Equitable Brazil  Reduce extreme poverty  Increased access to  Limited income and job and social exclusion education and opportunities for the  Provide a more efficient improvement in healthcare poor; provision of basic social quality  Financial barriers limit services  Improvements in per- accessibility of basic 2008-2011 capita GDP ratio services to the poor  Increased access to potable water  Reduce extreme poverty,  Improvements in systems  High rates of extreme vulnerability and social to target and reduce poverty, inequality and exclusion poverty inequity in social  Enhance knowledge and  Framework and strategy spending, skills (secondary developed for vulnerable  High unemployment 2004-2007 education) communities  Low quality education  Increased accessibility to  Interventions made to  Regional inequality in quality healthcare for all increase employment accessibility to quality communities  Increased access and healthcare improved education  Inadequate M&E quality framework  Improvement in healthcare quality Sustainable Brazil  Sustainable land-use;  Improved human  High poverty in Amazon  Infrastructure projects to development index and and Northeast be environmentally and land sustainability in the  Increased environmental socially sustainable Amazon region impacts  Increase land tenure and  Increased provision of  Inadequate basic protect indigenous rural electricity infrastructure in rural communities  Urban poor provided with areas access to improved  Inadequate protection of 2008-2011 housing and services water resources  7 million people provided with improved sanitation quality  Better water quality and  Improved water quality  Poorly managed and water resource and administration environmentally management  Increased sustainability in unsustainable water 2004-2007  Sustainable land- land use, forests and infrastructure practices management, forests and biodiversity  Inadequate WSS legal and biodiversity  Increased accessibility to regulatory framework  More equitable access to local services  Inequality in access to local services basic infrastructure services 4 Table summarizes the FY03-07 and FY08-11 CAS Matrices, and does not present a verbatim transcription. 17 CAS Country Medium Term Outcomes CAS Outcomes Possible Issues and Development Obstacles (at time of Goals4 CAS/CPS preparation) Competitive Brazil  Strengthen business  Business environment  Inefficient investment environment improved environment  Improve infrastructure  Results-based management  Uncompetitive financial network quality (with of public infrastructure sector resources for its expansion improved performance;  Inefficient transport and sustainability)  Reduced wait time for system  Improve fiscal capacity customs clearance  Uncoordinated 2008-2011  More efficient and stable  Improvements in fiscal infrastructure system financial sector capacity and financial climate  Improve infrastructure  Improvements instituted in  Inadequate legal and (quality and sustainability) infrastructure, investment regulatory framework for investment climate, and climate, and competition infrastructure; competition  Increased financial sector  Inefficient financial sector 2004-2007  Broader and more efficient capacity  Lack of innovation financial sector  Innovation climate  More modern innovation improved climate Macroeconomic  Legislative reforms in tax  Reduced public  High tax burden Foundations & policy and fiscal federalism expenditures, including on  Weak public sector and Public Sector  Social security reform social security expenditure management Management  Improved expenditure  Increased efficiency in  Excessive burden of social M&E framework state-level public resource security spending on management public finances 2008-2011  State-level, results-based management agreements implemented  Sound macroeconomic  Macroeconomic  High public debt management and fiscal environment strengthened  Inefficient governance reforms and fiscal reforms and public sector  More efficient public sector instituted management management  Improvement in public  Rigid public sector 2004-2007  Improvements in sector management spending and inadequate government effectiveness  Improvements in good information regarding its indicators governance impact 18