Report No. 18630 1998 Annual Review of Development Effectiveness (ARDE) November 23, 1998 Operations Evaluation Department Document of the World Bank Abbreviations and Acronyms ARDE Annual Review of Development Effectiveness ARPP Annual Report on Portfolio Performance CAE Country Assistance Evaluation CAN Country Assistance Note CAR Country Assistance Review CAS Country Assistance Strategy CBO Community-Based Organization CGAP Consultative Group for Assistance to the Poor CSR Civil Service Reform DAC Development Action Committee DEI Development Effectiveness Index EAP East Asia and Pacific Region ECA Europe and Central Asia Region GDP Gross Domestic Product IBRD International Bank for Reconstruction and Development ICRG International Country Risk Guide ID Institutional Development IDA International Development Association IDF Institutional Development Fund IDS Institute of Development Studies at Sussex University IMF International Monetary Fund LCR Latin America and the Caribbean Region MNA Middle East and North Africa Region NGO Nongovernmental Organization OECD Organization for Economic Cooperation and Development OED Operations Evaluation Department OEDCM OED Corporate Evaluation and Methods OEDCR OED Country Evaluation and Regional Relations Group PER Public Expenditure Review QAG Quality Assurance Group SAR South Asia Region - WDI World Development Indicators WDR World Development Report Director General, Operations Evaluation Robert Picciotto Director Operations Evaluation Department Elizabeth McAllister Manager Corporate Evaluations and Methods Wendy Jarvie Task Manager Robert Buckley FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation November 23, 1998 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: 1998 Annual Review of Development Effectiveness The ongoing financial crisis has raised questions about the underpinnings of development assistance and the role of interna- tional financial institutions. A new development assistance framework, grounded in partnership, is emerging. That is the backdrop for this year's Annual Review of Development Effectiveness by the Bank's Operations Evaluation Department (OED). As in past years, the Review tracks the Bank's operational performance based on the findings of recent evaluations. The trends are highly encouraging, but when countries that have performed so well for so long suddenly stumble as dramatically as they did in the past year, the meaning of project level trends deserves careful consideration. Accordingly, this Review draws on the work of scholars convened by the Institute of Development Studies of Sussex Univer- sity to assess the implications of the crisis. It also relies on a relatively new OED instrument-country assistance evaluations-to place the lessons from the Bank's project experience in a broader context. The Review complements the Annual Report on Portfolio Performance, which documents the Quality Assurance Group's findings about active operations, and the Annual Report on Operations Evaluation,, which presents OED's assessment of the sta- tus and prospects of internal evaluation processes. Robert Picciotto Director-General, Operations Evaluation -r Overview T his Review of development effectiveness comes at a time of crisis. In East Asia, about 20 million people have fallen back into poverty in the last year. Russia has been beset by political and economic upheaval. Japan is in recession with profound implications for the world economy. Economic problems have been compounded by natural disasters, such as floods in Bangladesh, in China, and in Central America. The prospects for achieving the OECD poverty reduction targets have dimmed. The crisis is rich in lessons both for development practitioners and evaluators. Develop- ing countries now confront a severe deterioration in the enabling environment, highlight- ing the impact of unregulated private flows and global interdependence and the growing influence of exogenous factors in determining development impacts. A stable macro economy is not enough Macroeconomic conditions are not enough to sustain equitable growth. Unlike the debt crisis of the 1980s, the ongoing financial crisis started in countries with relatively strong fiscal situations, sound monetary policies, and outward oriented trade regimes. When the crisis hit, government budgets in most crisis affected countries were balanced or moving into surplus, inflation was contained, interest rates were going down, and recorded unem- ployment was low. Institutions matter The crisis showed just how costly weaknesses in institutions-especially in the financial and social sectors-can be. Indeed, it is now clear that strong institutions are essential Strong ingredients of economic and social stability. Poor institutions increase the vulnerability of institutions are developing and transition economies to shifts in private investor confidence. The impor- essential tance of institutional development goes far beyond avoiding crises: ingredients of economic and * For Bank-supported projects, the quality of institutions can have important effects on social stability development effectiveness. These effects are particularly pronounced in low-income countries. * Where institutions are systematically weak, projects yield lower returns and higher risk. * Better institutions strengthen a country's ability to adjust. They can more than double the likelihood that a country undergoing adjustment can stay the course. An analysis of 41 low-income countries shows that only one was rated satisfactory on institutional quality. Only 40 percent of Bank-supported projects have substantial impact 111 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS on institutional development; civil service reforms undertaken as components of structural adjustment loans have mixed outcomes; public sector management projects, while improv- ing, have historically performed below the Bank average. An OED evaluation showed that Bank-supported financial sector projects had satisfactory and sustained outcomes in just 50 percent of countries. Institutional development is slow and difficult to achieve, and requires stronger aid coordination and the development of capacity to absorb aid and reduce the risks of overload in a fragile institutional environment. Poverty reduction and social safety nets A corollary lesson is that social development should come center stage-both in assessing Social development effectiveness and in financing country assistance programs. development Large reductions in employment of perhaps 10-15 percent are estimated for Indonesia should come and Thailand. With devaluations and the removal of subsidies, the newly unemployed will center stage- suffer from drastic losses in income and sharp rises in prices. The increasingly integrated both in assessing global environment means that country susceptibility to shocks will not disappear. Much development greater attention must be given to safety nets in helping to insulate the poor and the near- effectiveness and poor from disproportionately bearing the costs of shocks. .n f n .The crisis countries are not the only ones experiencing increasing inequality. Data for 74 n fiancig countries shows that there has been an overwhelming increase in inequality within coun- country tries in the 1990s-49 countries experienced increasing inequality, while only 10 had de- assistance creasing inequality. This confirms the need to emphasize inclusion, social development, programs and safety nets in the design and implementation of reform strategies and development programs. A country focus based on partnership Financial, institutional and the social factors must be considered together. For growth to result in sustainable development requires country assistance strategies that give adequate weight to structural factors, capacity building and social equity, and that identify potential 'holes in the boat'-where structural faults might cause development gains to unravel. A credible Bank's role begins with effective projects. This implies operations linked to the broader social, civil and economic environment. To scale up successes, the Bank must work in partnership with borrowers, donors, and other stakeholders to focus on maximiz- ing development impact at the country level. To do so the Bank must consider the impor- tant side effects that interrelated activities can have on country policies and institutions. It also requires mutual recognition by all participants of their relative strengths and weak- nesses, together with a willingness to define and share accountability. A partnership-based strategy is not only good policy from a development perspective, it is also good corporate finance. Much remains to be done to enhance the quality of country assistance strategies. Where country assistance evaluations have been undertaken, OED estimates the assistance strate- gies have been satisfactory only 68 percent of the time. The analysis confirms that project iv 0 V E R V I E W outcomes are highly dependent on the country strategy. For example, no country that had a satisfactory country strategy experienced weak project performance. Project performance has improved The performance of Bank-supported projects has improved substantially. The percentage of projects with a satisfactory outcome at the end of loan disbursement increased from an average of 65-70 percent in the 1990-96 period to an expected 75 percent or higher in 1997-98, including 7 percent with outstanding outcomes. This remarkable improvement demonstrates Bank and borrower commitment to improving development effectiveness. There have been major quality improvements in two of the poorest-performing sectors (finance and public sector management) and in Africa, particularly in agriculture. Better borrower performance, more realistic project designs, and better portfolio management explain the improved outcomes. Nevertheless, sustainability and institutional develop- ment impact lag considerably. A global perspective Last year's Review concluded that "the challenge is to find the right fit between country policy and institutional factors and strategies to try to improve conditions favorable to improved growth and development." In a much more complex and hostile environment, this year's Review reaches a similar conclusion. It is now even clearer that improvements in project performance-important though they are-are not enough. The architecture of the Bank's new approach to providing development assistance has passed a rather severe test. To be sure, adjustments and refinements in strategy must be the B k n made, and greater cognizance of the risks of the external environment must be internalized. strategy for Nevertheless, the Bank's new strategy for maximizing development effectiveness in a vola- maximizing tile global environment appears well conceived. The increased emphasis on partnership development and poverty alleviation stressed in the Strategic Compact, and President Wolfensohn's call effectiveness in a to move "beyond projects" in his 1998 Annual Meetings speech, are key to maintaining the volatile global performance improvements that have been realized in the past two years. environment appears well Implications conceived The above diagnosis leads to the following conclusions for performance measurement and evaluation: * Performance monitoring and assessment need greater transparency, with governance and institutional performance at center stage. More attention must be paid to monitoring structural, social, and poverty indicators. * Evaluation has to move to a higher plane, focusing on the country, sector, and even global level. * Evaluation rating systems have to give more explicit weight to the social impact of projects and programs and to the important effects that external shocks can have on the poor. v ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS For the Bank's operations there is a need to: * scale up successes, considering the important side effects that interrelated activities can have on country policies and institutions, * strengthen support for institutional development, particularly for financial institutions and social protection, and * shift from a project to a long term country focus both in design and implementation of operational strategies. vi Acknowledgments R obert Buckley (Task Manager). Contributors were: William G. Battaile, Jr., Ruchira Banerjee-Corcoran (OEDCM), Deepa Chakrapani, Federico Mini, Dinara Seytjaparova (OEDCR). Other contributors were: James Alm (Consultant), Keith Bezanson (IDS, Sus- sex), Jos6 Maria Dagnino Pastore, Louis Goreux, Jane-Francis Kelly (Consultants), Anwar Shah, Navin Girishankar (OEDCR), Sohail Malik, Kenneth Watson and John Eriksson (Consultants). Ruben Lamdany (Manager, OEDCR) and Roger Slade (Manager, OEDST) and their staff provided comments and were helpful in many ways. Wendy Jarvie (Man- ager, OEDCM) supervised and helped structure the Report. Norma Nanisato, Silvana Valle, Geri Wise, Brigitte Wittel, and Barbara Yale provided administrative assistance. vii Contents iii Overview vii Acknowledgments 1 1.Development Effectiveness in a Volatile World 1 Institutional weaknesses 5 Broadening the agenda to maximize development effectiveness 6 The new primacy of the external environment 7 2.Trends in Project Performance 7 Outcomes more than 75 percent satisfactory 9 Gains in development effectiveness 15 Institutional development impact-improving but still weak 15 Sustainability-low and weakening? 16 Bank and borrower performance improving-but greater gains from borrowers 17 Outstanding projects-7 percent of the total 18 Sector analysis 23 3.Development Effectiveness at the Country Level 23 The country as the unit of account 25 Institutions, aid, and growth 27 Adjusting to the external environment 27 Outward orientation and growth 29 Outward orientation and poverty alleviation 33 4.Thematic Evaluations and Institutional Development 33 Financial sector 34 Capitalization and bad loan exposure 34 Government interventions 34 Accounting and prudential standards 35 Enforcement capabilities 35 Governance issues 36 Governance institutions 41 5.Implications for the Bank and for Evaluation 41 Global risks 44 Prospects for poverty reduction 45 Implications for the Bank 45 Poverty alleviation 45 Country strategies 45 Institutional development 59 Bibliography ix ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Annexes 47 A New Approach to Evaluating Bank Projects-The Development Effectiveness Index 51 Glossary of Selected Terms 55 Data on Outcome, Sustainability, and Institutional Development Impact Boxes 9 2.1 Institutional development and sustainability help in understanding project performance 12 2.2 Areas of improvements in Africa 19 2.3 Two outstanding projects 26 3.1 Institutions and developing economies 28 3.2 The long-run expected benefits of adjustment 29 3.3 Measuring borrower ownership and its relation to adjustment outcomes 30 3.4 Greater poverty and income inequality in transition economies 37 4.1 Public sector performance review: Integrating public sector performance with a results-based management system 43 5.1 What do we know about the extent of poverty? 44 5.2 Poverty alleviation and income growth Figures 4 1.1 Institutions and project performance in low-income countries 8 2.1 Demandingness, complexity, riskiness, and outcomes, by exit fiscal year 10 2.2 Two measures of project performance, by exit fiscal year 11 2.3 Operations with satisfactory outcomes, by region and exit fiscal year 13 2.4 Africa region-relative performance trend, operations with satisfactory outcomes, by exit fiscal year 13 2.5 Operations with satisfactory outcome, by sector and exit fiscal year group 14 2.6 Institutional development impact, by exit fiscal year 15 2.7 Sustainability, by exit fiscal year 16 2.8 Borrower performance, by exit fiscal year 17 2.9 Bank performance, by exit fiscal year 19 2.10 Relative risk and reward, by sector, exit fiscal years 1997-98 36 4.1 Performance of civil service reform interventions Tables 2 1.1 Traditional crisis indicators 3 1.2 The Washington consensus-not in East Asia 18 2.1 Outstanding performers and poor performers among recently evaluated projects 24 3.1 Performance of Bank-supported country strategies and projects 25 3.2 Trends in socioeconomic indicators and income, 1970-96 34 4.1 Bank system risk exposure in East Asia 42 5.1 Average performance of projects implemented in three country-risk environments, projects exiting in fiscal 1997-98 x Chapter 1 Development Effectiveness in a Volatile World The current crisis confirms two main lessons for development effectiveness- and provides a new one. First, good macroeconomic fundamentals are neces- sary but insufficient for stable and sustainable growth. In today's global economy, sound institutions, especially in the financial and social sectors, are essential to economic and social stability. Second, projects are no longer the appropriate vehicles for development assistance unless they are connected to balanced country assistance strategies-focused on structural reform and capacity development, and owned by borrowers and other partners. An emerg- ing lesson is that exogenous factors are far more influential in determining development impacts than had been previously thought. 1.1 The past year produced a severe deterioration in the economic conditions of the developing world. Russia's economic reform collapsed in a year when transition economies were finally growing after seven years of decline.I The sharp reversal for some of the world's fastest-growing economies and the greater caution of private investors have chilled global economic growth. The perceived risk of investing in emerging markets made a very large one-year jump.' 1.2 The crisis has induced concern about the ability of the global system to contain the contagion.* Given the economic turbulence and uncertainty, questions have arisen about the consensus reported in last year's Annual Review of Development Effectiveness. Is good macroeconomic policy* a sufficient foundation for development effectiveness? * Does the crisis invalidate the broadly based consensus in the development community about what constitutes sound economic policy? Institutional weaknesses 1.3 With the crisis now over a year old, one fact has become clear: the costs of unregu- lated movements of private capital must be balanced against the risks. Indonesia, Korea, Malaysia, the Philippines, and Thailand received net private inflows worth almost 7 per- cent of their combined GDP in 1995-96. The reversal from 1996 to 1997 involved a swing of 11 percent of their combined GDP.' Changing risk perceptions by commercial banks and particularly portfolio investors (rather than foreign direct investors) explain the rever- sal. The sudden downgrading of country credit ratings sparked panic and flight among private investors. 1.4 Unlike the debt crisis of the 1980s, the current crisis started in countries with relatively strong fiscal situations, sound monetary policies, and outwardly oriented trade 1 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS regimes (table 1.1). Except in Thailand, government budgets were balanced or moving into surplus when the crisis hit. Inflation was contained, interest rates were going down, and recorded unemployment was low. Taking into account foreign direct investment, current account deficits were not excessive. Table 1.1 Traditional crisis indicators Indicator Indonesia Korea, Rep. of Malaysia Philippines Thailand Government bu4g deficitpercentage ofGDP Average, 1990-94 0.4 -0.4 -0.7 -1.4 3.2 Average, 1995-96 1.7 0.1 0.8 0.4 2.6 In fiation, rate'?cchange in the c~onsu mer pric e i ndex') Average, 1990-94 8.8 5.3 4.1 11.1 4.6 Average, 1995-96 8.7 4.7 4.4 8.3 5.8 Current account (percentage of GDP) Average, 1990-94 -2.7 -1.5 -7.4 -4.5 -7.5 Average, 1995-96 -3.8 -3.4 -9.7 -5.5 -9.1 Source: Reisen (1998). 1.5 Weaknesses in economic management helped trigger the crisis. In all cases the capital account was the main vulnerability. Imbalances between short-term debt and offi- cial reserves--combined with premature financial liberalization and weak financial disci- pline in domestic banking systems-created situations vulnerable to speculative pressures. e f a 1.6 Many fundamentals were sound in crisis-affected countries. The financial panic The financial would not have spread without weaknesses in domestic institutions. As in the Southern panic would not Cone crisis 15 years earlier, banking discipline was weak and links between economic con- have spread glomerates, banks, and governments were too close. This led to excessive borrowing, dis- without proportionate real estate booms, poor private investments and escalating levels of weaknesses in nonperforming loans. Why didn't policymakers and international financial institutions give domestic these weaknesses appropriate weight? Because the lessons of the general debt crisis were guiding them, not the more relevant institutional lessons of Chile's 1982 crisis and Mexico's 1994-95 peso crisis. OED's (1990) audit report on Chile's structural adjustment loans* highlighted the lack of prudential supervision of financial institutions in increasing the economy's vulnerability to the point of collapse. A key lesson of that audit was: "prudential rules and surveillance are necessary safeguards for the operation of domestic financial markets, rather than unnecessary restrictions" (p. 12). 1.7 Microeconomic dysfunctions are harder to spot than macroeconomic weaknesses. From macroeconomic indicators, decisionmakers in the private sector and international financial institutions found it hard to argue with success. In East Asia, as in Chile and Mexico before, credible domestic reforms, low interest rates, and good growth prospects contributed to an explosion of private flows. Attracting those flows were exchange rate pegs, profitable interest rate spreads, and liberalizations of the capital account. Given the "halo effect" typical of investment booms, decisionmakers overlooked the failure of Asian 2 DEVELOPMENT EFFECTIVENESS IN A VOLATILE WORLD countries to comply with some basic tenets of the much-abused Washington consensus, * a listing of sound economic practices on which most analysts agree. 1.8 Using the benchmarks of this consensus, Rodrik (1996) notes that the policies of Korea and Taiwan (China) have long been well below par. We extend Rodrik's analysis to Indonesia and Thailand in light of OED's recent country assistance evaluations (table 1.2). Cumulatively, the analysis confirms that East Asian countries were following policies con- sistent with only 6 or 7 of the 10 tenets of the consensus. Caprio (1998) finds that banking sectors were extremely weak in Indonesia and Thailand. In contrast, according to Rodrik, Table 1.2 The Washington consensus-not in East Asia Elements of the Korea, Taiwan Washington consensus Indonesia Rep. of (China) Thailand 1. Fisc-dl discipline Ycs, partially Yes, generally Yes Yes 2. Redirection of public Yes Yes Yes Yes, in the late 1980s expenditure priorities toward health, education and infrastructure ;. Tix reorrn, inJuding Ln,lear Ycs. gen,raliv Y-S Yv-,, guneri lb bro.]A n:np fhu -ax b.,-.c .Irld UrrIn rarginil 4. Unified and competitive Yes, until 1996 Yes, except for brief Yes Yes, until 1991 exchange rates periods 1 r r Lrr%r ' rii hr% I in'iacd 1'rc,i0ent 1'.irk IInIL ,rarted hi, ruk in: 19 b. impri;onin icadink bLj-,inL.m..n and rhrcaccning to ccnfis,:ate their t 6. Deregulation Limited Limited Limited Limited 7. Trade lhbcralization Limited Liited Ulited Yes until the 1980s until the 1980s until the 1980s 8. Privatization Limited, No. Government No. Government Limited, but not an issue established many established many but not an issue public enterprises public enterprises during 1950s and during 1950s and L limIntin .on ,t barrir, Ye, Fkirticgn dircr Forign direct 1imilLd in1LtmtenuLnr bi1r' I I C %Ub I to ior,itzn dirtct restricted to government control investment 10. Financial liberalization Yes Limited Limited Yes, until the 1980s until the 1980s in the 1990s Source: Data on Korea and Taiwan are from Rodrik (1996); other data, OED, World Bank. 3 ANNUAL R EVIEW OF DEVELOPMENT EFFECTIVENESS many Latin American economies-in particular, Argentina, Bolivia, and Mexico-fulfilled most of the consensus conditions. The conclusion: Washington consensus policies were neither the cause of high growth, nor the cause of the crisis. 1.9 Macroeconomic policy weaknesses were linked to competition policies and finan- cial liberalization-the sequencing of which needed to be made coherent with prior institu- tional development* and structural policy reforms. It is significant that their neglect also featured heavily in the Chile and Mexico crises. In the words of Claessens and Glaessner (1997, p.8), "liberalization is inexpensive, fast, and easy to implement; building institu- tional capacity is expensive, slow, and complex." In sum, mistakes in macroeconomic policy played a part in the East Asian downturn and an even bigger part in Russia, but the more critical dysfunction was institutional. Financial sectors, governance standards, and corpo- rate investment regimes (and in Russia, fiscal regimes) seemed adequate as long as the booms lasted.4 But they proved fatally flawed once external conditions deteriorated. Given the unprecedented volume and reversibility of short-term capital flows, weak banking in- stitutions and ineffective regulatory systems proved a lethal combination. 1.10 The reversal of capital flows was especially deep and disruptive where domestic interest rates were higher than in international markets. Financial supervision, corporate governance, and corruption-let alone social safety nets*-were too often given minimal emphasis in the "metrics" of performance monitoring and assessment. As stressed in last year's Review, institutional development lies at the core of development effectiveness. The Figure 1.1 Institutions and project performance in low-income countries Average project performance (percent satisfactory) so High Institutional quality 70 Low institutional 60 quality s0 40 30 20 10 a Low-income countries Other countries Note: The institutional quality index is composed of three variables based on data from the International Country Risk Guide (ICRG) covering 1982-98. These variables are corruption, rule of law, and bureaucratic quality. Countries are assigned income groups based on classifications in World Development Indicators 1998. The average project performance for the country groups are based on project-level outcome data. Source. ICRG; OED, World Bank. 4 DEVELOPMENT EFFECTIVENESS IN A VOLATILE WORLD "silent crisis" of poverty and destitution that affects low-income countries is deeply rooted in capacity constraints. How important is the quality of the institutional environment for Bank-supported projects in low-income countries? Very important indeed, because stron- ger institutions are associated with a 20 percentage point increase in the likelihood of a project's outcome* being rated satisfactory (figure 1.1). For the Bank's lending to low- income countries over the past two years, this improvement would translate into a more than $1 billion increase in effective Bank support. Broadening the agenda to maximize development effectiveness 1.11 The need to scale up from a strictly project-specific focus to a broader, more inclu- sive orientation is now well recognized. It is a perspective that has been realized by hard- learned lessons from experience-for example, with large dams. As OED (1996d) shows, large dams were viewed as being synonymous with modernization and development in the 1950s and 1960s; but growing evidence of their adverse indirect and secondary impacts turned them into targets of public criticism in the 1970s and 1980s. As a result, new poli- cies and standards emerged that scaled up the scope of the World Bank's intervention to include avoidance or mitigation of the adverse environmental and social consequences of large dams and, by extension, all projects with significant potential adverse side effects. To move beyond isolated success stories, the full array of factors affecting development results must be examined, and more inclusive, participatory approaches must be developed. 1.12 Such enhanced participation is essential if the complementarities across sectors and activities are to be fully exploited. However, besides developing broader, more inclu- sive approaches, the Bank and its partners must identify "holes in the boat." That is, do- nors cannot simply focus on projects or sectors that they know will perform well. They must also identify points of stress-such as financial weaknesses-that can cause gains to unravel. To achieve development effectiveness at the country level, Bank interventions must begin by designing effective projects. But they must also link individual operations to the broader social, civil, economic, and, where appropriate, international environment. They cannot neglect existing weaknesses that could more than offset the development effective- ness gains from a project. To be successful, the country assistance program must be firmly rooted in the borrower's ownership* of reform objectives. 1.13 Another lesson of the crisis is that social development should take center stage in . the financing of recovery or development programs. The shocks of recent years have plunged development millions into absolute poverty. The lack of formal social safety nets for the unemployed is being felt severely, partly because of the weakening of traditional systems that once sup- should take ported the poor. Cuts in public spending for the social sectors and rises in prices associated center stage in with devaluations and the removal of subsidies may add to the burden on the middle class the financing of and the poor. If the global objectives for poverty alleviation of the OECD's Development recovery or Assistance Committee (DAC)* are to be attained, more attention must be paid to social development development and social safety nets. programs 5 ANNUAL REvIEw OF DEVELOPMENT EFFECTIVENESS The new primacy of the external environment 1.14 The emerging lesson of the crisis is that a risky external environment can strangle development prospects. Not only do developing countries face the prospects of continuing reductions in aid, they also confront a more uncertain environment-much more so than developed countries do (see chapter 5). Private investors will give even greater scrutiny to the country and institutional conditions that allow entry to global financial markets. That is why development effectiveness requires a perspective that goes beyond country-level concerns, especially for small countries. 1.15 Particularly important for the external environment are donor efforts to bolster financial systems and to help restore the confidence that is key to the capital flows needed to restart growth. The World Bank and the International Monetary Fund (IMF) have much to do in leading the provision of information on regional and global trends-certain to be an important global public good.* The Bank is especially well placed to assist in develop- ing, processing, and understanding the often important fragments of information with broader implications. 1.16 A series of recent reports on international financial architecture laid out a range of actions to strengthen the international financial system.s The recommendations, which are relatively straightforward, call for enhanced transparency and greater discipline in balanc- ing market incentives and public control. They also stress the need for improved vigilance by international organizations. The novelty of these reports lies not in the proposals them- selves, but in their urging the practicing of what has long been preached. 1.17 The following chapters review recent evidence about the Bank's development ef- fectiveness. Chapter 2 describes project and sector performance trends. Chapter 3 consid- ers recent evaluation lessons at the country level. It draws on OED's country assistance evaluations to help draw out the lessons of the ongoing crisis. Chapter 4 draws lessons that can be inferred from OED's thematic studies. The final chapter discusses the implications for Bank operations and evaluation. Notes 1 World Bank, World Development Indicators 1998. 2 Based on Euromoney's ratings for country risk. 3 Institute of Development Studies at Sussex University, East Asia Crisis Workshop, 1998, chap- ter 1, page 1. 4 The glossary defines terms related to institutional development. In addition, box 3.1 illustrates the general perspective taken by analysts who focus on institutional issues. 5 In April 1998, finance ministers and central bank governors from a number of large economies formed three working groups-on transparency and accountability, on strengthening financial sys- tems, and on international financial crises-to discuss the policy issues raised by the financial crisis. In October 1998, each working group presented its findings and recommendations in a separate report. * See the glossary for definitions of words and phrases followed by an asterisk (*). 6 Chapter 2 Trends in Project Performance Among evaluated projects exiting the Bank's portfolio in fiscal 1997 and 1998, more than 75 percent had satisfactory project outcomes. Thus the im- provement in performance presaged by last year's Review has been sustained. There has also been a convergence in performance across regions and sec- tors, the result of major advances in Africa and in two of the poorest-per- forming sectors (finance and public sector management). Better borrower performance and more realistic project designs, as well as better portfolio management, explain most of the improvement. But sustainability and insti- tutional development remain sorely neglected in project design and portfolio management. 2.1 OED evaluates all closed projects, assessing results likely to be achieved in each project's operational phase. Since last year's Review, OED has evaluated 298 operations across all regions and sectors.' Africa has the largest regional share of the evaluated cohort (28 percent), followed by Latin America and the Caribbean and East Asia and the Pacific, with shares of roughly 20 percent each. Nearly half the projects are in the finance, private sector, and infrastructure network, led by the transportation sector (30 projects). Agricul- ture projects make up an additional 21 percent. Outcomes more than 75 percent satisfactory 2.2 The percentage of satisfactory project outcomes has continued to improve in re- There has been a cent years. The proportion of satisfactory projects for fiscal 1997 exits-the latest year for which complete results are available-is 75 percent. The figure for exits in the first half of substantial fiscal 1998 is 80 percent, exceeding the target of the Strategic Compact.2 While this pre- improvement in liminary figure is likely to be biased upward, it represents a substantial improvement. We project outcomes estimate the percentage of satisfactory projects for fiscal 1998 exits to be 76 percent, only in the last two slightly above the level for fiscal 1997.3 But it is now clear that the Bank has moved in the years past two years above the plateau in fiscal 1990-96, when the percentage satisfactory re- mained in the 65-70 percent range. 2.3 Some qualifications are in order. First, the unusually large risks in the international economy call for substantial discounts in the long term effects likely to be reaped from many completed projects. Second, more than 60 percent of the gains were in Africa, which has aggressively implemented a portfolio restructuring plan, but remains the region with the highest share of projects at risk of not achieving their development objectives.4 But, a significant part of the improvement in Africa is in agriculture, which in 1993 developed an action plan emphasizing simpler models of delivery and greater sector coherence. So the recent gains may not be ephemeral. 7 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 2.4 A third reason for caution is that these buoyant outcome ratings are goal-sensi- tive-relating largely to planned project goals. Thus their significance is connected to the demand, complexity, and risk* of project objectives. In prior Reviews outcome trends were steady or only slowly improving as the portfolio became increasingly ambitious. But there has been a break in the upward sloping curves of demandingness, riskiness, and complexity from fiscal 1996 to fiscal 1997-98 (figure 2.1). This shift is admittedly modest. Still, it Figure 2.1 Demandingness, complexity, riskiness, and outcome, by exit fiscal year Percent substantial 100 as Demandingnes, Outcome Complexity 40 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. could indicate that the improvement in outcome ratings was achieved, at least in part, through the achievement of more modest project goals. 2.5 Other notable characteristics of the performance trends include: * Strong improvements among the previously poorest-performing groups, including finance and public sector management. * Continued improvement in the outcome ratings of adjustment loans, with performance levels remaining above those of investment projects. * Sustained progress in borrower performance, now at par with Bank performance. Improvements in * Recent improvements in Bank performance, especially in the quality of project development supervision. outcomes have been sustained 2.6 In aggregate, this year's evaluation results confirm that improvements in develop- and broadly ment outcomes have been sustained and broadly based. But in the meantime, the overall based development environment has shifted in dramatically challenging ways. This shift under- scores the need for more reliable and timely evaluation measures and even greater use of 8 TRENDS IN PROJECT PERFORMANCE evaluation findings. With demand for lending boosted by the crisis, concessional resources declining, and Bank loans more costly to borrowers, concern with development effective- ness should not be allowed to flag. Gains in development effectiveness 2.7 In search of a more comprehensive measure of project performance than the out- come rating, OED has piloted for this Review the development effectiveness index.* It integrates existing OED measures of outcomes, sustainability,* and institutional develop- ment impact. The measure ranges from 2 (for a project with a highly unsatisfactory out- come, which also has unlikely sustainability and negligible institutional development impact) to 10 (where high achievements on all three measures are realized). 2.8 The index improves the presentation of performance trends in three ways (box 2.1). First, it uses the spectrum of outcome assessments made by OED rather than the Box 2.1: Institutional development and sustainability help capture project performance Beyond the assessment of stalled. But the project was port facilities were con- ment with one that con- goal-oriented perfor- rated as having substantial verted to containerized op- sidered only project out- mance captured by the institutional development erations, the main project come. Using the outcome rating, two ex- because it closed down the objective. As a result ship development effective- amples show how ex- loss-making public institu- waiting times and berth ness index, the project in plicit consideration of tion formerly responsible times decreased. The insti- Pakistan scores a 6.75, institutional development for fresh groundwater tutional development com- well above the Bank av- and sustainability can pumping, replacing it with ponent, however, was rated erage of 6.38 for fiscal provide a richer descrip- a market-based institution. negligible because of 1994-97. The project in tion of the development Moreover, sustainability poorly planned training ar- Tanzania receives a 5.25. impact of Bank-financed was rated as likely for two rangements for employees An analysis based solely projects. reasons. First, diesel of the Tanzanian Harbours on a binary classification In 1995 an irrigation pumps, which do not de- Authority. The of outcome does not take project in Pakistan was pend on the highly subsi- sustainability of project into account the con- rated as having a margin- dized and unreliable benefits was also disap- trasting performance of ally unsatisfactory out- electric distribution system, pointing and rated un- sustainability and insti- come. The project sought were emphasized. Second, likely. Given the size and tutional development. to help farmers to invest the water users group complexity of the facilities The descriptions of the in their own small wells worked well in coordinat- installed, the lack of local two projects indicate that in areas where there was ing the decisions of nearly basic skills was considered the first had better over- fresh groundwater-di- 4,000 independent pump a critical shortcoming. all performance than the rectly useful for irriga- owners, with no depletion Moreover, it appeared un- second, and that is the tion. Behind the outcome of water resources likely that the weak man- information the develop- rating were two factors. expected. agement of the Harbours ment effectiveness index First, the estimated eco- Contrast this with the Authority could remedy the conveys. A crude binary nomic return fell short of outcome of the Rehabilita- problem without additional analysis based strictly on the 10 percent minimum tion Project of the port of external support. goal-based evaluation threshold for satisfactory Dar-es-Salaam in Tanzania, Now consider how the (outcome) would have re- outcomes. Second, subsi- rated as marginally satis- two projects would com- versed the ranking in fa- dies and free hook-ups to factory. Although there pare using an aggregate vor of the project in the power grid resulted were delays in implementa- measure of performance Tanzania. These issues in too many electric pri- tion and part of the project that included sustainability are discussed more fully vate wells being in- was not undertaken, the and institutional develop- in annex 1. 9 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS binary assignment to satisfactory or unsatisfactory outcome. Second, it qualifies a project's outcome judgment by rewarding the robustness of achievements into the future, in some cases recognizing the lasting benefits of significant achievements that fall short of expecta- tions. Third, institutional development impact is given special emphasis. Together these aspects of the index provide a more complete picture of trends. The new index facilitates performance analysis of sectors and countries. Additional analysis is under way to confirm the robustness of the development effectiveness index, as well as its consistency and complementarity to measures used by the Networks and the Quality Assurance Group. Annex 1 summarizes the index's construction. 2.9 While also showing improvement over the past two years, the development effec- Outcomhe atg tiveness index trend is less dramatic than the trend in outcomes (figure 2.2). Outcome gains have not ratings gains have not been matched by gains in project sustainability or institutional been matched by development. gains in project sustainability or Figure 2.2 Two measures of project performance, by exit fiscal year institutional Development effectiveness index Outcome percent satisfactory development 9 g By disbursements - - 70 V11 By projects y disbursements ' - 61 50 1992 1993 1994 1995 1996 1997 1998 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. 2.10 Weighted by disbursements, the data show a similar pattern of improvement, with the share of satisfactorily rated outcomes reaching 78 percent in fiscal 1997.s For fiscal 1998 exits, the preliminary satisfactory outcome achievement rate is 84 percent, though using the development effectiveness index the fiscal 1998 figure shows no gain over fiscal 1997, holding steady at 7.02. But regional analysis shows the disbursement-weighted in- dex to have fallen in most regions, with the largest drop in East Asia and the Pacific (1 point, for 27 percent of Bankwide disbursements). This was offset by a large increase in Latin America and the Caribbean (1.15 points, for 23 percent of Bankwide disbursements). The drop in East Asia reflects movement in China and especially Indonesia, which together account for roughly 80 percent of the region's disbursements. 10 TRENDS IN PROJECT PERFORMANCE Figure 2.3 Operations with satisfactory outcome, by region and exit fiscal year Percent satisfactory Percent satisfactory 100 100 EastAsia and Pacific 80 90 Bank -Bank 60 c 60 40 40 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Percent satisfactory Percent satisfactory 100 100 Europe and Central Asia on s, 89 Middle East and North Africa Bank 60 Bank 40 40 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Percent satisfactory Percent satisfactory 100 100 80 80 Latin America and Caribbean .Bank BanBk 60 60 South Asia 4 40 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1984 1995 1§96 1997 1998 Note, Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage ofexired operations. Some: OED, World Bank. 11 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 2.11 All regions except East Asia and the Pacific show improvement in fiscal 1997-98 compared with long-term averages for fiscal 1990-96 (figure 2.3). Improvements in project outcomes were largest in Africa (box 2.2) and Latin America and the Caribbean, with 14 and 15 percentage point gains in the share of satisfactory outcomes, respectively. These improvements-particularly in Africa, where performance has historically lagged behind other regions-have raised the global average and reduced regional disparities (figure 2.4).1 e i2.12 Improvements in project performance were noticeable in most sectors (figure 2.5). mp ent i The most notable decline, among sectors with significant numbers of recently evaluated project projects, was in industry. In that sector there was a fall from 54 percent satisfactory perfor- performance mance in fiscal 1990-96 to a dismal 36 percent satisfactory in fiscal 1997-98, as well as a were noticeable 7 percent drop in the average development effectiveness index. This represents the continu- in most sectors, ation of a downward trend in a sector with declining emphasis in the Bank. especially in 2.13 The fastest-improving sectors were concentrated in the previously poorest-perform- previously poor ing groups. Public sector management, finance, and industry were the worst-performing .m sectors in fiscal 1990-96 (by projects), with development effectiveness and project out- comes well below the Bank average. Public sector management and finance show more groups than a 10 percentage point increase in the share of projects with satisfactory outcomes and a significant increase in the average development effectiveness index in fiscal 1997-98. Significant improvements (by projects) were also evident in agriculture, urban, transporta- tion, and population, health and nutrition. Box 2.2 Areas of improvements in Africa Performance gains in Af- Africa. For example, the ened while progress was rica have been driven by National Agricultural Ser- steady in improving the improvements in agricul- vices project in Cte quality of financial re- ture, public sector man- d'Ivoire strengthened the porting in both the pub- agement, and finance. Of monitoring, evaluation, lic and private sectors. the projects exiting in and cooperative support Also in Mozambique, fiscal 1997-98, agricul- activities of the Ministry of a financial sector project ture dominates the Africa Agriculture. Another ex- more than achieved its portfolio (26 percent ample is the Technical As- objectives through better share). Satisfactory out- sistance project in fiscal and monetary man- come ratings for agricul- Mozambique. It supported agement and ture projects increased the country's transition privatization of the from 54 percent in fiscal from a centrally planned to banking and industrial 1993-96 to 76 percent in a market-oriented sector. This result con- fiscal 1997-98. economy, and is a good il- tributed to a sharp drop One explanation of lustration of the improve- in inflation, and in- this extraordinary im- ment in African public creased private invest- provement is the success sector management ment and higher of capacity building projects. The pace of insti- economic growth. work. There has been a tutional development in Expansion of social 16 percentage point im- Mozambique was rapid, spending also resulted in provement in the number with key economic institu- a noticeable improve- of projects having sub- tions like the Ministry of ment in health and stantial institutional de- Finance and the central education. velopment impact in bank significantly strength- 12 TRENDS IN PROJECT PERFORMANCE Figure 2.4 Africa region-relative performance trend, operations with satisfactory outcome, by exit fiscal year Percent satisfactory 100 80 Other regions 40 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of eited operations. Source: OED, World Bank. Figure 2.5 Operations with satisfactory outcome, by sector and exit fiscal year group By projects (percent satisfactory) By disbursements (percent satisfactory) dustry 1997-98 1 hi.s 1997-08 PuMlir Prlubulium nideMaulmeduu Management fluane, Finum water seyWater Suply & SaDAle &Aiuti Ao Population, ?rwln, "aib i *Nu health & nutrition Urban Urla Dowelopmel development - Bestric power EIefPriPwer A Ot Emerry & other energy Transpotation ruo Educatin Etcao Social usector L Soial Seal 0 20 40 80 80 100 8 20 4 0 811 100 Note: Only sectors with at least 10 exiting projects in fiscal 1997-98 are included. Source: OED, World Bank. 13 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 2.14 Care should be taken in interpreting these sector changes as potentially lasting shifts in performance trends. A comparison with performance assessments of the active portfolio currently under supervision provides a complementary gauge.' They provide modest support for the view that sector improvements are likely to be sustained. For example, active agriculture projects have improved to above-average performance in fiscal 1998, with lending scheduled to increase in the near future as the Rural Action Plan is imple- mented. Similarly, they indicate declining performance for the shrinking number of indus- try projects. The current portfolio data do not, however, support sustained gains in the finance sector. Adjus Itmentloa2.15 Adjustment loans continue to have higher average outcomes and sustainability Adjtueto lans than investment loans, though the gap has narrowed. The share of satisfactory outcomes continue to have for adjustment loans rose from 74 percent in fiscal 1990-96 to 82 percent in fiscal 1997- higher average 98, compared with a rise from 66 to 76 percent for investment loans. Institutional develop- outcomes and ment performance has evened to roughly 38 percent substantial for both types of loans. sustainability 2.16 Last year's performance results suggested only minor progress in IDA and blend- than investment financed projects. This year's results display much stronger improvement-with IDA and loans blend projects among the fiscal 1997-98 exits performing at par with IBRD-financed projects when considering outcome, sustainability, and institutional development impact. Consider- ing outcomes alone, the share of satisfactory projects among IDA loans and blends was actually above that for IBRD-financed projects: 79 percent were judged satisfactory in fiscal 1997-98 compared with 74 percent for IBRD loans. Figure 2.6 Institutional development impact, by exit fiscal year By projects (percent) By disbursements (percent) 50 60 30 Substantial 30 Negligible 20 \Neglgible 20 10 10 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. 14 TRENDS IN PROJECT PERFORMANCE Institutional development impact-improving but still weak 2.17 The first two objectives of a five-point development framework offered by Presi- dent Wolfensohn in his address at the 1998 Annual Meetings relate to institutional devel- opment.' Institutional development is particularly important in low-income countries, where the potential gains are the greatest and, as chapter 3 illustrates, the quality of institutions the lowest. 2.18 Recent evaluations show that the institutional development impacts of Bank The institutional projects are improving but there is enormous scope for further improvement (figure 2.6).9 Current exits show historical highs of only 40 percent of operations with substantial insti- development tutional development. Between fiscal 1991 and 1995 the share of projects with negligible impacts of Bank institutional development rose from roughly 20 percent to nearly 30 percent. That share projects are dropped to the lowest on record of just 15 percent in fiscal 1997. By disbursements, how- improving but ever, the historical trend of larger projects having greater institutional development im- there is enormous pacts shows signs of weakening. The share of disbursements with substantial impact was scope for further below the project-weighted average in two of the past three years. In chapters 3 and 4 we improvement show the fundamental importance of institutional development in development effective- ness. The main finding: the spillover effects from better monitoring and closer attention to institutional development have been neglected, and need much greater emphasis in Bank operations. Sustainability-low and weakening? 2.19 The fiscal 1997-98 data send a mixed signal on sustainability'o that may well be a precursor of future declines in performance caused by a deteriorating external climate (figure 2.7). The proportion of projects judged as having likely sustainability that exited in Figure 2.7 Sustainability, by exit fiscal year By projects (percent) By disbursements (percent) 80 8 60 60 Likely 40 ~ Uncertain 4 Uncert Unlikely 20-. 20 .Unlikely 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary resuts, with less than 50 percent coverage of exited operations. Source: OED, World Bank. 15 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS fiscal 1997 maintained an upward trend, increasing to 54 percent from 46 percent in fiscal 1990-96. But these projects closed at the latest in June 1997, well before the East Asian crisis. The partial results for fiscal 1998 projects, exiting in the unfolding of the Asian crisis, show a decline to 50 percent of projects judged to have likely sustainability. This decrease is largely because of the drop in East Asia and the Pacific projects-those most directly affected by the crisis-from 66 percent likely sustainability in fiscal 1997 to 43 percent in fiscal 1998. Similarly, there has been an almost doubling of the share of active projects in the Region at risk of not achieving their development objectives. Bank and borrower performance improving-but greater gains from borrowers 2.20 Analysis in past Reviews on the determinants of successful project outcomes found project-specific borrower performance to be the most important determinant of project success. Bank performance and country macroeconomic policy environment were also found to be less significant." In fiscal 1997-98 2.21 In fiscal 1997-98 borrower performance improved to 75 percent satisfactory from 66 percent satisfactory in fiscal 1990-96 (figure 2.8).12 This increase is pronounced in IDA borrower countries, particularly in Africa and Latin America and the Caribbean. Borrower inputs performance have improved dramatically in finance and public sector management projects-rising from improved to 75 satisfactory performance in 57 percent of operations exiting in fiscal 1990-96 to 84 per- percent cent exiting in fiscal 1997-98. These sectors enjoyed large gains in outcomes over the same satisfactory period, confirming the importance of borrower inputs in project performance. Improve- ments of more than 10 percentage points were found in urban and transport operations. 2.22 The increase in borrower performance reflects improvements in project prepara- tion and in compliance with legal covenants. Implementation performance remains the Figure 2.8 Borrower performance, by exit fiscal year By projects (percent) 100 s0 60 1mplementation Comnpliance 40 L 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. 16 TRENDS IN PROJECT PERFORMANCE poorest-performing dimension, with an unsatisfactory assessment in more than a third of evaluated projects. The overall increase in performance puts the overall quality of bor- rower inputs at par with Bank inputs. Three of every four evaluated projects now show satisfactory Bank and borrower inputs. 2.23 Bank performance, by contrast, improved modestly for fiscal 1997-98 exits, with only a one percent increase in the overall average over the 75 percent satisfactory average for fiscal 1990-96 (figure 2.9)." Lower-quality project identification has been offset by better supervision, while project appraisal improved slightly.14 These results conceal some areas of particular improvement. Regionally, Latin America and the Caribbean showed improvement in both appraisal and supervision, raising the overall Bank performance aver- age to 80 percent. Conversely, borrower performance in East Asia and the Pacific projects dropped in all three dimensions in fiscal 1997-98, with appraisal performance showing the largest decrease. Finance and public sector management, population, health, and nutrition projects also show improvements in Bank performance. Figure 2.9 Bank performance, by exit fiscal year By projects (percent) 100 ~Ideantification.. 6 Appraisal 40 1990 1291 1992 1993 1994 1995 1996 1997 1998 Note: Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. Of recent Outstanding projects-7 percent of the total evaluations, OED 2.24 Of the 298 operations evaluated in the past year, OED assessed 20 (7 percent) as assessed outstanding and 4 (1 percent) as exceptionally poor (table 2.1). The 20 outstanding opera- 7 percent as tions all had highly satisfactory outcomes, likely sustainability, and substantial institutional outstanding and development impact. In addition, these projects met or exceeded their main goals, had 1 percent as highly innovative designs, and are replicable in other countries or sectors. They featured exceptionally strong borrower ownership, enjoyed highly satisfactory or satisfactory quality at entry, and poor were well supervised. 17 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Table 2.1 Outstanding performers and poor performers among recently evaluated projects Country Project name Loancredit number Outstanding performers Benin Urban Rehabilitation and Management C2338 Brazil Land Management I - Parana L3018 Brazil Parana Municipal Development L3100 Chile Second Public Sector Management L3411 China Ertan Hydroelectric L3387 China Integrated Regional Health Development C2009 China National Afforestation C2145 China Northern Irrigation - Part A C1885 China Ship Waste Disposal C2391 Dominican Republic Primary Education Development L3351 Estonia Highway Maintenance L3731 Hungary Human Resources Development L3313 Lao PDR Second Telecommunications C2101 Latvia Agricultural Development L3695 Mexico Contractual Savings Development Program L4123 Morocco Telecommunications Sector L3557 Mozambique Second Economic Recovery C2628 Peru Debt and Debt Service Reduction L4133 Tunisia Population and Family Health L3307 Vietnam Structural Adjustment C2657 Poor performers Ecuador Second Water Supply L2774 Morocco Rural Primary Education L3026 Papua New Guinea Land Mobilization L3051 Rwanda Sectoral and Pre-Investment Studies C1796 Note: Covers the 298 projects evaluated since the last ARDE. Source: OED, World Bank. The 2.25 The success of the outstanding projects can be traced to flexibility in responding to characteristic changing conditions-the result of consistent monitoring, good supervision, and partner- ship building. Even projects with complex designs succeeded because of extensive and ef- feature of poor fective Bank staff involvement and judicious technical assistance (box 2.3). In contrast, performers is a only one poor performer had satisfactory quality at entry. The characteristic feature of general lack of poor performers is a general lack of supervision and low borrower ownership. supervision and low borrower ownershipe Sector analysis ownership_ 2.26 For a successful development strategy, assessing risk (as measured by the variabil- ity of rewards) is as critical as assessing the expected development impact. Risks and re- wards in the 12 sectors with more than 10 projects exiting in fiscal 1997-98 are shown in 18 TRENDS IN PROJECT PERFORMANCE figure 2.10. The origin of the axes corresponds to a reward equal to the average develop- For a successful ment effectiveness index in the 1997-98 portfolio, and risk is equal to the standard devia- development tion of the index in that portfolio. The axes measure differences for specific sectors relative strategy, to the Bankwide average and the standard deviation of the development effectiveness in- dex. Thus each quadrant, starting from the upper left and moving clockwise, corresponds assessing risk is as critical as Figure 2.10 Relative risk and reward by sector, exit fiscal years 1997-98 expec t expected development High risk Risk High risk Low reward High reward impact 0.7 Industry 0.5 Electric power and other energy sources U Reward Public sector management C 01l Multisector 0 Education -1.5 -1.2 -0.9 -0.6 -0.3 Finance 0.3 0.5 -0.1 E Agriculture Population, health, and nutrition U-0.2 Transportation U Water supply and saultation Urban development W -o.U -0.4 Social sector U Low risk -g.5 Low risk Low reward High reward Note: Only sectors with at least 10 exiting projects in fiscal years 1997-98 are included, While the development effectiveness index has desirable mathematical properties for this kind of analysis, a similar performance distribution occurs when measurement relies on project outcome measures. Source: OED, World Bank. Box 2.3 Two outstanding projects The importance of bor- These myriad objectives strong Bank leadership is officers of the new Agri- rower ownership, com- were more than exceeded similarly illustrated by cultural Finance Com- mitment of Bank staff, due to the dedicated, com- Latvia's Agricultural Devel- pany were important and good supervision is petent action of the bor- opment Project. This highly ingredients of project illustrated by Chile's rower, supported by an successful project was the success. Clearly under- Public Sector Manage- excellent Bank team. Ca- first Bank-financed invest- pinning these successes ment Loan. The loan's pacity improvements and ment project in Latvia. Its was the government's main objectives were to the beginning of cultural overall development objec- commitment to the increase the efficiency change were achieved in tives were to enhance the project, in turn strongly and effectiveness of key several key policymaking privatization of agricul- influenced by the effec- public management agen- agencies and in the Library ture, agroproces sing, and tiveness and staffing con- cies, remedy the of Congress-above-aver- forest industries through tinuity of the project government's inadequate age achievements for a financial and technical as- management unit and by economic analysis and freestanding technical as- sistance. Well-focused ob- Bank supervision. coordination capabilities, sistance loan. jectives, a simple design, and improve policy- The crucial role of bor- and such innovative fea- making in the legislature. rower commitment and tures as the mobile credit 19 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS to one of the four classes: high risk-low reward, high risk-high reward, low risk-high re- ward, and low risk-low reward. The coordinates corresponding to each sector measure the sector's risk-reward combination relative to the average project. 2.27 Figure 2.10 allows a comparison of average project results for each sector and the dispersion of these results around the corresponding Bankwide values. However, these com- parisons are not meant to rank sectors. Instead they suggest that the relative risk-reward framework may be useful for the Bank Networks in considering potential weaknesses and strengths of Bank interventions in specific sectors, and external challenges that the sectors may face. Take the industry sector, where the average development effectiveness index is far below the Bankwide average and volatility is much higher. This raises questions: Does the Bank have comparative advantages in supporting industrial projects relative to the private sector? Or can the below-par results for this sector be attributed to a concentration of these projects in countries lacking good infrastructure? How much of the volatility in industry projects is the result of poor selectivity by the Bank? How much can be explained by exogenous shocks (such as contagion effects in regional crises), which are likely to affect industry projects more than less market-oriented interventions? 2.28 Similarly, the relatively low rewards of water and sanitation projects-and the relatively high confidence that performance in this sector will be weak-raises several ques- tions. Why do this sector's results stand in such contrast to the. rest of the Bank's portfolio? Do measurement problems drive the results? Or is a sector with such important potential effects on health and poverty really a relatively weak performer? We do not try to answer these questions here. The point is to present a framework that the Bank Networks can use to compare and contrast their sector performance with the results in other sectors. The framework itself should be scrutinized and tested for robustness over time. Notes I The terms "operation" and "project" refer to both IDA and IBRD lending and are used inter- changeably. Unless otherwise stated, all time period references relate to the fiscal year in which evalu- ated operations exited the portfolio. 2 The results for fiscal 1998 are based on a sample of 114 evaluated operations-40 percent of the 283 exiting operations-for which Regional staff have prepared completion reports. A complete set of data by region and sector will be provided on OED's Web site for reference and follow-up. 3 The likely bias arises from problems of sample representation for the preliminary fiscal 1998 results. According to project data from the Quality Assurance Group (QAG) at exit, the cohort evaluated so far includes far fewer problem projects (16 percent) than the projects remaining to be evaluated (26 percent). Of the evaluated cohort, those rated as problem projects by QAG at exit were only 18 percent satisfactory while the rest were more than 90 percent satisfactory. Assuming the same relationship between QAG and OED assessments for those fiscal 1998 exits yet to be reviewed by OED yields an estimate of 73 percent satisfactory for the part of the cohort not reviewed here. Combining this group with the exits reviewed implies an overall 76 percent satisfactory rating for the entire group of fiscal 1998 exits. 4 Data from the Annual Report on Portfolio Performance Fiscal Year 1998 (ARPP), prepared by the Quality Assurance Group. 20 TRENDS IN PROJECT PERFORMANCE 5 Disbursements are measured in real terms, deflated to fiscal 1996 US dollars. 6 The standard deviation of the regional shares of satisfactory projects has dropped 40 percent, from 10 percentage points for fiscal 1990-96 to 6 percentage points for fiscal 1997-98. 7 Data on the active portfolio under supervision is taken from the Annual Report on Portfolio Performance Fiscal Year 1998 (ARPP), prepared by the Quality Assurance Group. The basic mea- sure of performance used in the ARPP is the number of projects at risk of not achieving their devel- opment objectives. Projects at risk consist of actual and potential problem projects. Actual problem projects are those for which implementation progress is unsatisfactory or development objectives are not likely to be achieved. Potential problem projects are those rated satisfactory on implementation progress/development objectives but have other risk factors historically associated with unsatisfac- tory outcomes. 8 "The Other Crisis," delivered in Washington, D.C., on October 6, 1998, emphasized the essen- tials of good governance and the need to specify the regulatory and institutional fundamentals essen- tial to a workable market economy. 9 Institutional development impact measures the extent to which a project has improved an agency's or country's ability to make effective use of its human and financial resources. This impact, possible even in the absence of explicit institutional development objectives, includes both tradi- tional impacts, through new or improved organizations, and the impact projects have on the rules of the game governing public and private sector behavior. 10 Sustainability is defined as the likelihood, at the time of evaluation, that a project will main- tain its results in future. To judge the sustainability of an operation, evaluators take into account country conditions, government policies, and other conditions specific to the operation such as avail- ability of funds for operation and maintenance. Basic factors behind the original project appraisal are also considered-such as technical, financial, and economic viability, susceptibility to external shocks, and the social, environmental, and governance environment. 11 Looking at shifts in country macroeconomic environments, the fiscal 1997-98 evaluated exits have slightly more projects in the poorly performing country group than do those exiting in fiscal 1990-96 (12 percent compared with 9 percent), with a slight decrease in projects in the lower- performing group (from 17 percent to 14 percent). The three other groupings of high, lower-me- dium, and upper-medium performing countries show even less variation when analyzed for fiscal 1990-96 and fiscal 1997-98. While slight, the movement toward poorer-performing economic envi- ronments makes it unlikely that country selectivity played a role in the improved performance results of fiscal 1997-98. 12 Borrower performance is assessed for three project processes-preparation, implementation, and compliance. Implementation is the broadest of these three project measures, with dimensions under the government's control-such as broad project commitment, appointment of key staff, and counterpart funding-as well as implementing agency factors such as management, staffing, cost changes, and beneficiary participation. 13 Bank performance is assessed for three project processes-identification, appraisal, and su- pervision. Assessments of the Bank's performance during project identification include looking at the involvement of the government and beneficiary whether the project is consistent with the Bank's country assistance strategy and whether there is a grounding in economic and sector work. The quality of appraisal includes the following dimensions: technical and financial analysis, cost benefit analysis, institutional capacity analysis, and environmental and social analysis. Dimensions taken into account for assessing project supervision include progress reporting, identification/assessment 21 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS of problems, use of performance indicators, advice to the implementing agency, and flexibility in suggesting/approving modifications. 14 Findings that supervision improved are consistent with the latest Quality Assurance Group self-evaluation assessments of supervision in Supervision Quality in FY98: A QAG Assessment (RSA2). * See the glossary for definitions of words and phrases followed by an asterisk (*). 22 Chapter 3 Development Effectiveness at the Country Level Country evaluations confirm that weak institutional development has been a key problem in improving development effectiveness. Risk-bearing institu- Where the tions-particularly the financial system and social safety nets-have been enabling neglected by the development process. In many low-income countries, chan- environment is neling aid through isolated, uncoordinated enclave projects has left capacity inadequate. Where the enabling environment is weak, projects should be jus- weak, projects tified largely for their policy reform and for their capacity development im- should be pacts, with the attendant risks reduced through judicious testing of borrower justified largely ownership. Country evaluations also show that institution building is needed . to ensure that a country's outward orientation can safely reap the benefits of for their policy globalization-and shield the poor from its shocks. reform and for their capacity 3.1 In 1995 OED inaugurated country assistance to assist the Board deliberations on development country assistance strategies.* By now,' 17 such country assistance evaluations (CAEs) impacts have been produced. They assess the relevance, efficacy, efficiency, sustainability, and insti- tutional development of assistance strategies. Using the insights of independent profession- als, they are case studies of aid effectiveness in the tradition of Cassen (1994) and Mosley, Harrigan, and Toye (1991). They identify lessons of experience and draw the implications for future strategies. More than 90 percent of operational staff preparing country assis- tance strategies found them helpful. The following findings emerge from an overview of CAEs: * When graded the same way projects have traditionally been evaluated by OED, CAEs rate the overall outcome of the Bank's country strategy as satisfactory 68 percent of the time. * The rating of the Bank's country strategy from the CAEs is a relatively strong predictor of the average performance of Bank-supported projects. * The most important development issues identified by CAEs are consistent with those that would be inferred from other empirical analyses of growth and poverty alleviation. The country as the unit of account 3.2 For this review OED undertook a pilot analysis of the linkages between country strategy and project performance. As a first step, all completed CAEs were subjected to a formal rating process consistent in structure with OED project ratings which are based on evaluative conclusions about the design, outcome, and impact of the Bank's assistance strategy in a country.' OED staff preparing CAEs were asked to summarize their views on 23 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS the outcome, sustainability, and institutional development of Bank country strategies as they would on projects (table 3.1). 3.3 Project performance is strongly correlated with the quality of the country assis- Project tance strategy. None of the countries that had a satisfactory country strategy experienced performance is weak project performance. In only 2 of 25 periods rated did an unsatisfactory Bank coun- strongly try strategy result in satisfactory performance on projects. correlated with the quality of the Table 3.1 Performance of Bank-supported country strategies and projects country Country assistance Average project outcome performance assistance strategy rating Satisfactory Unsatisfactory strategy Satisfactory Albania (1992-97) Argentina (1991-96) Bangladesh (1980-96) Bolivia (1986-97) C6te d'Ivoire (1960-79) C6te d'Ivoire (1994-98) Ghana (1982-96) Jamaica (1987-96) Malawi (1995-97) Morocco (1983-88) Mozambique (1984-96) Philippines (1986-97) Poland (1986-96) Thailand (1987-96) Zambia (1994-96) Unsatisfactory Cbte d'lvoire (1980-86) Argentina (1985-90) Morocco (1989-94) Cbte d'Ivoire (1987-93) Jamaica (1980-86) Kenya (1990s) Malawi (1990-94) Togo (1985-90) Togo (1991-96) Zambia (1980-93) Note: Average Bank-supported project outcomes over 50 percent are categorized as "satisfactory." Country strategy ratings of 4 and above are rated "satisfactory." Other approaches to categorizing relatively strong and weak performance, using central tendencies such as mean and median, produce similar results. Source: OED, World Bank. 3.4 CAEs' judgments about key strategic issues are similar to those that would be suggested by empirical studies of growth and poverty.s However, the relatively low aggre- gate outcome measure, 68 percent satisfactory, suggests room for considerable improve- ment in Bank country assistance strategies. In what follows we consider some of the common lessons from CAEs. 24 DEVELOPMENT EFFECTIVENESS AT A COUNTRY LEVEL Institutions, aid, and growth 3.5 Development assistance has achieved much (table 3.2). For low-income countries the rate of improvement on most measures of deprivation is considerably better than that for high-income countries. But improvements in growth have been less propitious (Ingram 1992). For example, the weighted average per capita growth rate for low-income countries for 1980-96 (outside China and India) has been negative. So, low-income countries do not-as economic convergence models predict-catch up with high-income economies. Instead they fall farther behind. 3.6 Poor policies have a lot to do with disappointing income growth. Yet even the low- income African countries described as sustained policy reformers (IMF 1995) have had average growth of only 0.5 percent a year. The growth rate among this group is only slightly higher than that realized by Europe over the 400-year pre-capitalist period before 1820 (Maddison 1997). So something other than weak macroeconomic policies is imped- ing growth. 3.7 There is no simple explanation for such weak performance. But one factor, which recurs in OED's country evaluations, is the weakness of institutions. For instance, only 30 percent of low-income countries enjoy an institutional environment rated as marginally satisfactory, less than half the level for middle-income countries (box 3.1). Only 1 of 41 low-income countries scores a satisfactory rating on institutional quality, while more than 30 percent of middle-income countries do.6 By contrast, OECD countries (except Korea) boast satisfactory institutional quality ratings. Among low-income countries a low institu- tional rating is more common than a weak policy environment: 40 percent of low-income countries and 77 percent of middle-income countries have good policy regimes.7 = 3.8 Policies and institutions are weaker than they might otherwise be because of inad- Policies and equate donor coordination. Zambia's country assistance strategy provides a stark example. institutions are In 1987 the Bank concluded that Zambia's policy regime was not appropriate for the Bank weaker than they to provide further lending. The Bank maintained this position for almost four years until might otherwise 1991, when it resumed lending. In the 12 years before the Bank's withdrawal of assistance, be because of aid averaged more than 9 percent of GDP a year. During the hiatus in Bank lending, donor inadequate donor assistance as a share of GDP rose to more than 15 percent of GDP a year. So, during a coordination period when the Bank found the policy environment such that aid was unlikely to promote Table 3.2 Trends in socioeconomic indicators and income, 1970-96 (percentage change) Infant mortality rate Access to safe water GNP per capita Crude death rate (per 1,000 Life expectancy (percentage (1987 constant Country group (per 1,000 people) live births) (years at birth) of population) U.S. dollars) High-income 10 72 9 n.a. 66 Low-income 33 40 17 260 44 Low-income excluding China and India 37 37 21 55 -4 Source: World Development Indicators 1998, World Bank. 25 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Box 3.1 Institutions and developing economies Since the 1980s a new to adapt its methods and that arise from the development perspective processes to the new devel- neoclassical model. has emerged. It holds opment consensus. that institutions and eco- The papers presented at The papers argue that nomic organizations are a recent OED conference if institutional analysis is the key determinants of (1998) illustrate that insti- to become operational, it economic, social, and po- tutions matter. The papers will have to provide litical progress. Six explore not only how to greater clarity in the area Nobel prizes have been get the institutions right of incentives. Institutions awarded to scholars who but also how to assess the matter because incentives made pioneering contri- fit between institutions and trigger motivation and butions to development challenges action in both the public neoinstitutional econom- through evaluative tech- and private sectors. In- ics. In parallel, develop- niques. The papers show centives are thus the first ment evaluators have that: building block for policy established the crucial design, implementation, role of capacity building *Variable combinations of and evaluation of results. in ensuring the competition, cooperation, The difficulty lies in sustainability of develop- and hierarchy are needed aligning the incentives ment programs. Yet the to achieve positive soci- structure with the collec- links between develop- etal outcomes in specific tive interest. ment practice and aca- country circumstances. Source: Picciotto and Wiesner demics are not strong. In * Getting the incentives addition, the evaluation right is crucial to over- profession has been slow coming the restrictions development effectiveness, donor assistance increased-and exceeded total investment. This pattern of donor support was motivated by donors' humanitarian concerns with the prob- lems of a very poor country. But it did little to increase borrower ownership, to strengthen institutions, or ultimately to reduce poverty. A central 3.9 CAEs of low-income countries-Albania, Ghana, Malawi, Mozambique, Zam- A central bia-as well as recent work by African policymakers on the African Capacity Building problem has been Study (World Bank, 1996a), argue that a central problem has been countries' lack of capac- countries' lack of ity to absorb the volumes of aid provided. Rather than directly addressing this, however, capacity to many development agencies, including the Bank, have established parallel methods to channel absorb the financial assistance, ignoring the adverse effects on capacity creation. This finding is not volumes of aid new, but it bears repeating. For example, Johnston and Van De Walle (1996, p. 66) argue provided that "aid has rarely contributed to effective institution building as it has bypassed local institutions in project implementation and design. The preference for enclave projects and parallel management structures to ministerial administrations has been particularly destructive." 3.10 A recent Danish government report on development cooperation issues in Tanza- nia (Helleiner and others 1995) reaches similar conclusions on how donor practices often undermine ownership. For example, in primary education the study finds that agencies frequently manipulate the choice of government departments they work with in order to achieve their objectives. The report also argues that where the government is reluctant to 26 DEVELOPMENT EFFECTIVENESS &T A COUNTRY LEVEL agree to a donor's project, there have been implicit threats of a reduction in general donor support. Finally, it suggests that a common practice for donors is to pay "incentives" to government officials working on their projects. 3.11 Berg (1993) points out that, in such situations, "technical cooperation takes on a role different than its traditional one: it substitutes for and subsidizes government operat- ing budgets. It does this directly by payments to government staff on projects, and indi- rectly by financing experts to do operational work normally done by government employees. This is disadvantageous in two ways. It misuses the technical assistance personnel resource, reducing its effectiveness for institution building. And it is extremely costly; high-cost expa- triates are hired in posts that nationals could fill more cheaply" (p. 213-14). 3.12 The effects of lack of donor coordination on institutional development can be particularly acute in small countries, where the sheer volume of external assistance (and the The lack of associated absorptive capacity constraints) can hinder development effectiveness. Albania's donor country assistance review found that despite supporting and helping to design highly inno- coordination on vative projects overall, "IDA's strategy.. .did not fully appreciate the risks of overload in- institutional herent in a rapidly growing and diverse portfolio and a fragile institutional framework" development can (p. 12). Donor assistance in Albania reached more than 50 percent of GDP! be particularly 3.13 In contrast, in middle-income countries external assistance flows tend to be too small to have much impact. For instance, shortly after the 1994 devaluation in the 13 West cutes African countries belonging to the Communaut6 Financire Africaine, the Bank lent the countries largest borrower in the zone (C6te d'Ivoire, a low-income economy) an amount equivalent to 1.5 percent of its GDP, or less than 2 percent of Bank commitments that year. But to provide similar support to the three East Asian economies-Indonesia, Korea and Thai- land-that ran into difficulties in 1997 would have required lending more than three times the scale of net transfers that actually took place, equivalent to more than 6) percent of total Bank lending for 1997. Adjusting to the external environment 3.14 The quality of lending and the support for institution building are only part of the equation. Equally important for development effectiveness is how well a country adjusts to the external environment, so that opportunities for growth can be exploited and the poor can be insulated from adverse shocks. CAEs provide interesting examples of how countries adjust or fail to do so. Outward orientation and growth 3.15 Reform is a complex business requiring a clear understanding of borrower inter- ests in the light of political economy considerations in order to nurture sustained owner- ship. Where the seeds of borrower ownership are in place, lending can be a useful instrument of reform. But as CAEs show, development assistance is not science-it is art. In unstable policy environments, there is no substitute for case-by-case assessments, framed to distin- guish risks worth taking (C6te d'Ivoire and the Philippines) from risks that are inappropri- ate (Kenya). 27 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 3.16 The expected gain from assistance is the product of the probability of success times the rewards of adjusting. The enormous gains that can be realized from adjusting are often overlooked (box 3.2). Many studies focus only on whether adjustment took place-not on the payoff. A perspective that considers only the risk of failure and not the gains from success can be misleading. It is the type of perspective that might be used by a lender facing roughly the same level of loss with each failure, and a payoff that does not increase with the gains from success. This kind of decisionmaking is inappropriate for an equity investor-or for the kind of development partner described in the Strategic Compact. 3.17 What happens if a high standard is set for what is judged to be a permanent, lasting improvement in the policy environment? In the cases considered, only about 12 percent of adjuster countries realized a permanent and major improvement in their macro- Box 3.2 The long-run expected benefits of adjustment Can a country with a for at least the past four which data were available. loans in not yet durable weak policy environment years but have not main- They also perform badly on adjusters would be adjust? Can it shift from tained adjustment long Bank adjustment loans- strong and very similar a bad to a good regime enough (nine years) to be with only 49 percent satis- (if slightly higher, 85 per- and sustain it over a de- classified as durable ad- factory outcomes. The cent satisfactory) to that cade or more? Or, as a justers. average number of oscilla- of durable adjusters. number of analysts have * Oscillators are those that tions between strong and Regardless of the observed, do countries do not tend to adjust but weak policy regimes for threshold, it is clear that that attempt to adjust continue to oscillate be- this group is four. If we the probability of success not make it-and adjust tween weak and strong add up the times all three is not high. So, for ad- over and over again? Fi- policy environments. categories of countries justment to have a last- nally, if durable adjust- Their policy index re- have attempted to adjust, ing high payoff requires ment is possible, do mains volatile over time. the probability for a coun- that the gains from ad- adjusters grow more try to adjust durably is justment be substantial. rapidly? The sample has 12 du- about 12 percent. Of Are they? Does this turn- To consider some of rable adjusters, 20 not yet course, this measure de- around to good policy these questions, we use durable adjusters, and 11 pends fundamentally, and make a difference? The the Burnside and Dollar oscillators. Durable adjust- arbitrarily, on the period payoff associated with (1997) measure of ers are further divided into chosen to qualify as a suc- successful adjustment can macropolicy, based on countries that adjusted and cessful adjuster. It also de- be illustrated in terms of the financial deficit, in- maintained it at the first pends on the quality of the the differences in average flation, and openness for attempt, and countries country's institutional envi- growth, per capita in- 1975-96. We considered where the policy index os- ronment. If, for example, come, inflation, and 43 countries for which cillated between good and we consider the probability volatility of these vari- data were available. The bad before the adjustment of successfully adjusting in ables in the different Burnside and Dollar to good policy environment countries that have satis- country classes. In suc- I policy index was con- was sustained. Eleven of factory institutional envi- cessful adjusters, per structed annually, and the 12 countries successful ronments, the likelihood of capita GNP grew at al- the changes in it were at maintaining a good successful adjustment in- most three times the rate used to categorize policy index, according to creases to 30 percent. of countries that have countries: Burnside and Dollar mea- The threshold of nine not yet achieved durable sures, did so on the first years is arbitrary-and adjustment, and six times *Durable adjusters are attempt. very conservative. If we re- faster than oscillators. those that maintained a The oscillators have had duce it to seven years, the The adjusters also in- good index for at least a weak policy index on av- odds of success increase to creased their growth nine years. Not yet du- erage for about 13 years 16 percent-one in six. rates more than sixfold. rable adjusters are and for more than 60 per- And as measured by OED, those that have adjusted cent of the period for the outcome of adjustment 28 DEVELOPMENT EFFECTIVENESS AT A COUNTRY LEVEL economic policy environment. But for them, per capita income growth rates have been almost three times those for unsuccessful countries, and more than six times higher than their pre-adjustment growth rates. 3.18 Thus a review of the broader adjustment experience supports a key lesson of CAEs: the realization of high-payoff successes requires careful analysis and cooperation, rather ower than the use of simple rules to determine whether' support should be provided. Neverthe- ownership less, this complexity does not mean that indicators are not available. For instance, informa- together with a tion on seeming intangibles, such as concepts like borrower ownership, can be made greater concern operational, and obvious tests can be used to sort out the likely sustainability of invest- for institutional ments in policy change (box 3.3). quality could 3.19 By relying on such characteristics to guide Bank support for adjustment lending, ityroul the probability of success can be improved. Indeed, the probability of success in Bank- prov e supported adjustment lending has increased significantly in recent years-because the Bank has become more conscious of the importance of borrower ownership. Combining this more insistence on ownership with a greater concern for institutional quality could improve performance even more. It could also increase the legitimacy of subsequent reform efforts. Outward orientation and poverty alleviation 3.20 In low-income countries capacity weaknesses are pervasive. In middle-income coun- tries they tend to be specific. The country evaluation of Poland (OED, 1997e) suggests that reform of the social safety net remains one of the most important areas in the unfinished policy agenda. Recent reports on Thailand and Indonesia (World Bank, 1998) and a grow- ing body of empirical work summarized in Levine (1997) show that in middle-income countries the institutions that specialize in bearing risk-financial intermediaries and social safety nets-have been neglected by the development process. Such oversight has had serious economic and social consequences, as recently demonstrated in Indonesia, the Re- public of Korea, Russia, and Thailand. Box 3.3 Measuring borrower ownership and its relation to adjustment outcomes Borrower ownership of delineates the following *Has participation taken adjustment programs is performance measurement place within the soci- often cited as vital for criteria for ownership: ety? making policies credible, safeguarding against *Is the initiative for formu- The evaluation shows policy reversals, and en- lating and implementing that measures of owner- suring the sustainability the adjustment plan the ship are strong predic- of benefits. An OED borrower's? tors of outcomes. And evaluation of nearly 100 *Is there observable con- through selected country adjustment programs in sensus among key minis- case studies, it ascertains 42 countries by Johnson tries and decisionmakers the important factors and Wasty (1993) high- on the nature of the crisis that account for differ- lights the symbiotic rela- and the necessary actions? ences in the intensity of tionship between * Have specific up-front ac- borrower ownership. program outcomes and tions been initiated before borrower ownership. It the program? 29 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 3.21 Over the long term, openness helps poor countries grow faster, and assists in re- ducing inequalities (Edwards 1997). But openness also makes it more important to put in place well-functioning risk-bearing institutions. To estimate the effects of shocks on income distribution, OED updated data from Deininger and Squire (1996) and added estimates for other countries. The data indicate an overwhelming increase in inequality within coun- tries.8 In fact, increases in inequality were five times more frequent than decreases: 49 countries experienced increasing inequality, 15 appear to have no trend, and 10 had de- creasing inequality. Inequality is particularly acute in transition economies (box 3.4). But it is also increasing in countries that previously had no trend or a decreasing trend. For""su"t""nable"3.22 To sum up, growth and an outward orientation are keys to reducing poverty. But alone they are not enough. For sustainable progress, better safety nets and better-targeted progress, better expenditures on those aspects of poverty for which markets do not work are essential. As safety nets and Sala-i-Martin (1997) documents, such expenditures can have strong positive effects on better-targeted growth. Similarly, OED's Social Dimensions of Adjustment (1996e) found that no country expenditures on has achieved sustainable poverty reduction without growth. It also showed that the quality those aspects of of growth is critical to the distribution of benefits. Much greater emphasis on safety nets- poverty for and expenditures on sectors not adequately funded by market processes-are needed if which markets poverty is to be alleviated. do not work are essential Box 3.4 Greater poverty and income inequality in transition economies According to available creases in infant mortality, States countries-in- information, poverty has and 10 of 16 countries equality has shot up dra- increased across Eastern with data experienced a de- matically from moderate Europe and Central Asia. terioration in secondary levels to reach levels Take life expectancy at school enrollment. typical of more unequal birth. It has declined pre- The close relationship developing countries. cipitously in several between poverty and in- In sum, poverty and economies, most notably come inequality distin- income inequality have Russia, where the aver- guishes three groups of increased in all transition age life expectancy for countries. In the first economies since the late men in 1995 (58.3 years) group-the Czech Repub- 1980s. Some increase in was three years below lic, Hungary, the Slovak income inequality in the that in India and a stun- Republic, and Slovenia- region, even in the long ning six years below than inequality was historically run, is probably an un- at the start of transition. low and has risen moder- avoidable consequence of The drop in life expect- ately. In the second the introduction of mar- ancy in other countries, group-Poland, Romania, ket-based rewards. Nev- a such as Ukraine and the and the Baltics-inequality ertheless, the scope of Baltics, is similarly con- was slightly higher than in structural change and centrated among men, the first group at the start economic dislocation and there were declines of the transition and has have introduced addi- in 14 of 17 countries for since increased to levels at tional inequality and which there are data. The or above the OECD aver- poverty. income and health di- age. In the third group- Source: Based on EBRD (1997), mensions of poverty have primarily Russia and annex2.2. also deteriorated. Nine Bulgaria, and perhaps in- of the 17 countries with cluding other Common- data experienced in- wealth of Independent 30 DEVELOPMENT EFFECTIVENESS AT A COUNTRY LEVEL Notes 1 Up until August 1998. 2 OEDCR staff were asked to rate country assistance strategies (quality at entry, implementation, outcome, sustainability, and institutional development) from their evaluations in CAEs. Where the evaluator thought appropriate, the country strategy was rated separately for different periods. For the 17 countries reviewed, CAEs identify 25 time slices-that is, different and distinct strategies for the country within the time period covered by the evaluations. Project ratings corresponding to the same countries and periods show project outcome was satisfactory 68 percent of the time. The latter figure is similar to the average project performance during the period of the CAE analysis. 3 Derived from background work on country assistance strategies, available electronically at http://www.worldbank.org/html/oed. 4 The evaluation form follows the style of OED's project information form, and the methodology is that of OED's methodology for evaluating completed lending operations. See OED Lessons and Practices 10, "Evaluating Development Operations: Methods for Judging Outcomes and Impacts," 1997. 5 In addition to comparing the perspectives on growth and poverty reduction in CAEs with those of empirical models, we compared how well these judgments on country strategy performance per- formed as a predictor of subsequent Bank project performance. While 25 observations limited the degrees of freedom, these judgments nevertheless serve as a stronger predictor of subsequent project performance than do macroeconomic policy measures. 6 The data on institutional quality refer to country ratings on bureaucratic quality, rule of law, and corruption as defined by the International Country Risk Guide. An average of the three ratings of greater than 4 on a 1-6 scale is considered satisfactory. A rating of greater than 3.33 and less than 4.0 is considered marginally satisfactory. 7 This rating is based on a more limited sample of low- and middle-income countries. The policy index has been calculated on an annual basis for 1995 as in Burnside and Dollar (1996). 8 Using comparable data on income distribution for 45 countries, Deininger and Squire (1996) show that over the 30-year period up to the early 1990s there was no trend in within-country income inequality. In 29 countries the Gini coefficient-a measure of income distribution-remained virtu- ally constant, in 8 it increased, and in another 8 it decreased. In the 16 countries with an increasing or decreasing trend, in 12 the change was small. Our shorter-term perspective, focusing on changes over five or more years, suggests that in recent years a very different pattern has emerged. The trends in transition economies are an important aspect of the changing pattern. But even in the 45 countries considered by Deininger and Squire, an updating of their data and a more-short terra focus shows that about five times as many countries (24) have an increasing trend as do those with a decreasing trend (5 countries). Of course, given the limited amount of observations, our classification necessar- ily based on a heuristic approach. 31 Chapter 4 Thematic Evaluations and Institutional Development The current financial crisis has far-reaching implications for development practitioners, and for evaluators. A higher priority must go to monitoring financial sector performance-and to the wide range of institutions involved in improving governance. Emphasis must also be given to the institutions- such as nongovernmental organizations and civil society-that help those not served by formal institutions. And to improve the effectiveness of public expenditures, practitioners and evaluators should help to introduce results- based management in the administration of development programs. 4.1 Structural and social constraints to development need far more scrutiny. OED's Process Review of World Bank Grant Programs (1998) shows that progress has been made on broadening the Bank's agenda and developing instruments to nurture the many kinds of institutions that can address these constraints. For example, the Institutional Development Fund (IDF) and the Consultative Group to Assist the Poorest (CGAP) promote institutional capacity-building and donor coordination. Similarly, the mainstreaming of new lending instruments-such as Learning and Innovation Loans and Adaptable Program Lending- represent tangible progress in the development of stronger, more sustainable institutions. But much more needs to be done. 4.2 Although it is well known that institutional factors are essential ingredients of economic growth and social stability, these factors remain neglected. This chapter consid- ers the lessons of OED thematic evaluations for the wide range of institutional develop- ment issues involved in improving governance. Financial sector 4.3 The financial institutions in crisis countries violated virtually all the institutional The financial norms recommended by a recent OED study on Financial Sector Reform: A Review of institutions in World Bank Assistance (1998f). The OED analysis of financial sector interventions focused crisis countries on analyzing how the elements of the Bank's evolving financial sector policy were reflected violated virtually in Bank-supported projects. Examining 23 countries, the study found a satisfactory and all the sustainable outcome in only 12. The recommendations of the study-especially on the institutional timing, sequencing, and scope of regulatory intervention-are more relevant than ever in norms light of the past year's events. According to Reisen (1998), careful monitoring of financial institution conditions in the crisis countries would have revealed serious weaknesses in financial systems. Consider how four key indicators of financial system strength developed a recent OED in the OED study behaved in the crisis countries. study 33 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Capitalization and bad loan exposure 4.4 At the outbreak of the crisis, nonperforming loans were the highest in the Republic of Korea (16 percent of total assets) but similarly high in Thailand (15 percent) and Indo- nesia (11 percent; table 4.1). These figures are much higher than the 9 percent in Mexico in early 1995, where the cost of rescuing banks has been estimated at about 15 percent of GDP (Caprio and Klingebeil 1996). In Indonesia, Korea, and Thailand capital-asset ratios were 6-10 percent. So even before the crisis, nonperforming loans far outweighed (on average) bank equity capital. Government interventions 4.5 Banks in crisis countries may have been affected by certain kinds of government interventions in bank lending and corporate finance. The governments often directed lend- ing towards particular sectors, both formally and often informally. In addition to explicit guarantees, there were implicit guarantees that lead to presumptions of government bail- out for non-performing loans in favored sectors. This lead to excessive investment and risky lending. Once the bubble burst, investor confidence was further undermined by the uncertain fiscal implications of honoring government explicit and implicit guarantees. Accounting and prudential standards 4.6 The weaknesses of accounting standards in crisis countries are common in many emerging markets: inconsistent financial reporting, limited power of auditors to examine Table 4.1 Bank system risk exposure in East Asia Indicator Indonesia Korea, Rep. of Malaysia Philippines Thailand Bank system exposure to risk (percentage of assets, end 1997) Nonperforming loans 11 16 8 6 15 Capital ratio 8-10 6-10 8-14 15-18 6-10 Real estate exposure 25-30 15-25 30-40 15-20 30-40 Collateral valuation 80-100 80-100 80-100 70-80 80-100 Bank lending to connected firms High High Government-directed bank lending Yes Yes Yes Yes Yes Bank deposit insurance No No No Yes No Importance of state-owned banks High High Accounting standards Weak Weak Weak Weak Enforcement of regulations Weak Weak Weak Weak Weak Incentive's for capital' flows' Short-term inflows Limited Limited Promoted (promoted) Long-term inflows Limited Limited Promoted Outflows Free Limited Limited Source: Reisen (1998). 34 THEMATIC EVALUATIONS AND INSTITUTIONAL DEVELOPMENT company records, lax auditing and accounting standards out of line with international good practice, lack of penalties for incorrect reporting of information, and tolerance of multiple accounts. In such environments, detailed examinations by supervisors and regula- tors may not reveal the information needed to regulate properly or to ensure prudential soundness. Enforcement capabilities 4.7 Even though some of the crisis countries had strengthened their supervisory and regulatory infrastructure during the 1980s and 1990s, partly in response to costly banking crises in Indonesia and Malaysia a decade earlier, enforcement capabilities remained weak (Fischer and Reisen 1993). Bank regulators had also imposed limits on bank lending, in- cluding liquidity requirements, exposure limits, and risk-based capital requirements. But according to Reisen (1998), these standards and ratios were poorly enforced. 4.8 The crisis reinforces the need to adopt a central recommendation of the OED study on financial sector reform: "more resources should be allocated to monitor and evaluate countries' financial sector programs, with performance indicators" (p.83). The crisis also underpins Bank management's seeking special budget authority from the Board to support financial sector work. But above all, it confirms the large adverse effects from neglecting institutional development. Governance issues 4.9 The crisis has reinforced the already-strong evidence that growth in per capita income is enhanced by strong property rights, sound legal foundations, and capable civil Doe er servants-all operating in an effectively managed institutional system. There is also evi- governance and dence that corruption in these institutions hampers growth.' Does better governance and lower corruption lower corruption improve the development effectiveness of projects? Unambiguously, yes. improve the As measured by the development effectiveness index, Bank-supported projects in countries development with an inadequate bureaucracy are on average the weakest performers.' In countries with effectiveness of a well-functioning bureaucracy, projects perform much better, with significantly lower projects? risks than average. 4.10 Similarly, Bank-supported projects in high-corruption economies have had signifi- Unambiguously, cantly lower returns with significantly higher risk. Corruption is almost always associated yes. with low bureaucratic quality, so that public sector management projects in corrupt coun- tries are particularly likely to be low-return/high-risk projects. Only education projects have a high return and a low risk in countries with high corruption. Projects in low-corrup- tion economies nearly always have a higher return and a lower risk than comparable projects in other countries. 4.11 Perhaps the most difficult governance issues arise where the state has collapsed or failed, particularly in countries that have recently emerged from conflict. In general, projects in these countries tend to have lower rewards and higher risks, reflecting the turbulence that conflict engenders. Still, risk-reward performance varies significantly across sectors, and as might be expected, many types of infrastructure projects perform well. in societies that have experienced destruction and civil conflict. 35 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 4.12 Another message of OED's (1998k) study of the Bank's experience with post-con- flict countries is that the Bank can assist best if it avoids the overzealous pursuit of unreal- istic fiscal rectitude. The circumstances of these countries require that the first emphasis be on support for rebuilding the institutions of government and civil society. Governance institutions 4.13 The Bank has long supported efforts to improve the workings of the state. In the Civil service 1980s, Bank assistance primarily sought to make governments "leaner" through downsizing. interventions- In the 1990s the Bank sought a "clean state" for its clients by strengthening the credibility only 33 of governance institutions through institutional reforms-that is, intra-public sector regu- percent... achieved latory reform and establishment of checks on arbitrary action. Regardless of the approach, satisfactory the results have been meager. OED recently evaluated more than 300 civil service reform outcomes interventions supported by 124 lending operations approved during 1980-97. A signifi- cant percentage of completed and ongoing operations lacked adequate monitoring and evaluation information to make meaningful assessments of performance. Only 33 percent of those interventions that could be evaluated achieved satisfactory outcomes (figure 4.1). Institutional development impact was rated as substantial in only 16 percent of completed operations-and in only 10 percent of completed stand-alone projects. 4.14 Overall project performance in the public sector management sector is improving, but civil service reform components remain among the weakest-performing interventions in the Bank's portfolio. As with financial institutions, a central recommendation of the Figure 4.1 Performance of civil service reform interventions OED evaluated projects (percent) Recently completed and active projects (percent) 100 100 8080 20 20 0 111'10- ti Downsizing Organizations Quality of Principal Reforming Downsizing Organizations Quality of Principal Reforming and Human agent Institutions and Human agent Institutions structures Resources relations structures Resources relations 0 Unsatisfactory 0 Lack of monitoring &evaluation X Satisfactory Note: The data reflects OED evaluations through July 1998. Source: OvilServiceRefirm Study forthcoming, OED, World Bank. 36 THEMATIC EVALUATIONS AND INSTITUTIONAL DEVELOPMENT OED analysis is that far greater priority should be devoted to integrating the use of perfor- mance indicators to monitor and support more effective public administration efforts. Re- sults-based management systems can be an effective way to focus public sector performance on outcome measures (box 4.1). 4.15 Public expenditure analysis is central to the Bank's policy dialogue with member countries. It is also an evolving and rapidly growing field within the Bank's economic and sector work, having grown from 3 reviews before 1979 to 39 in fiscal 1998. Public expen- diture reviews are a means for the external evaluation of a borrower's fiscal policies and sector reform efforts. They also provide a framework for coordinating external assistance and assessing its effectiveness, and they can provide a micro foundation for the IMF's macroeconomic framework. IDA's deputies have underscored the importance of public ex- Box 4.1 Public sector performance review: Integrating public sector performance with a results-based management system A recent OED study of tabled in parliament two to by anyone? In industrial most of these questions Public Expenditure Re- three months before the countries institutional are expected to be yes, views (1998) discusses budget statement. In con- norms are strictly adhered this is not true in devel- how this analysis might trast, public sector values to, and there are severe oping countries. give greater emphasis to are rarely addressed in de- moral, legal, voter, and The analysis and rec- output measures rather veloping countries, because market sanctions against ommendations in such a than traditional input the orientation is to "com- noncompliance. In develop- review must be consistent measures. It says that to mand and control" rather ing countries noncompli- with and recognize any provide relevant analysis than to serve the citizenry. ance often is neither inconsistencies between a of public expenditures, a Authorizing environ- monitored nor subject to country's mission and public sector perfor- ment. The authorizing en- sanctions. values, its authorizing mance review must begin vironment includes formal Operational capacity environment, and its op- by developing an under- (budget processes and insti- and constraints. What is erational capacity. If they standing of three contex- tutions) and informal insti- authorized is not necessar- are, the review will en- tual dimensions of the tutions of participation ily what will get done be- able the client and exter- country's public sector. and accountability. Do cause available operational nal partners, including Public sector mission these institutions and pro- capacity may not be consis- the World Bank, to un- and values. Societal val- cesses work as intended in tent with the task at hand. derstand better how to ues and norms-as em- providing an enabling envi- Further, even the opera- improve public sector bodied in the ronment for the public sec- tional capacity that is performance. It will fur- constitution or in annual tor to meet its goals? Do available may be circum- ther serve as a catalyst to budget policy state- various levels of govern- vented by the bureaucratic introduce results-based ments-may be useful ment act in the spirit of the culture or incentives that management in develop- points of reference for constitution in exercising reward command and con- ing countries. Such an public sector mandates their responsibilities? What trol-and corruption. Some approach to public sector and the values inherent are the checks and balances key questions: Do the agen- management would help in those mandates. In in- against deviant behavior? cies responsible for various change bureaucratic cul- dustrial countries the Are there formal rules to tasks have the capacity to ture from its emphasis on mission and values of the ensure fiscal discipline? Is undertake them? Are there command and control, public sector are spelled public sector borrowing binding contracts on public with arbitrary and op- out in a medium-term subject to financial market managers for output per- pressive rules, to one fo- policy framework. For discipline? How is govern- formance? Does participa- cused on serving its example, there is a for- ment performance mea- tion by civil society help citizens, earning their mal requirement in New sured? Are output and alleviate some of these con- trust, achieving results, Zealand that a policy outcome indicators for straints? Whereas in indus- and working better for statement of this type be public services monitored trial countries answers to less money. 37 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS penditure reviews as instruments for client capacity development as well as for enhancing development effectiveness by integrating review results with country assistance strategies. 4.16 OED's (1998g) study of The Impact of Public Expenditure Reviews found that quality has improved in recent years. But it also found that public expenditure reviews provided good (but often dated) analyses of spending policies with little concern for cost efficiency or the quality of public services. They had only a modest effect on Bank lending strategies, client expenditure policies, and aid coordination. The study argued that such reviews could become significantly more effective if they were more demand-responsive, if better synchronized with authorities' budget cycles, and if they gave due recognition to institutional constraints. NGOs and CBOs 4.17 In many societies, nongovernmental organizations (NGOs) and community-based organizations (CBOs) provide a closer link to the poor than public sector institutions. Ac- can play a cording to an OED (1998h) study of NGOs in Bank-supported Projects, 38 percent of particularly Bank-supported projects include NGOs or CBOs in their plans. This involvement increased important role in to 46 percent of projects in 1997, more than doubling from 20 percent in 1989. The study projects targeted found that these institutions can be particularly important in projects targeted at improv- at improving ing gender equality (80 percent), the environment (54 percent), and poverty alleviation (48 gender equality, percent). But their capacity is often limited by erratic funding and a lack of financial inde- the environment, pendence. n p4.18 The lack of government capacity to work effectively with NGOs and CBOs is also and poverty important. While there are some outstanding examples of government agencies with a alleviation strong ability to work with NGOs, these are exceptions. Similarly, the Bank's capacity to encourage NGO and CBO involvement in projects remains limited. And as for financial institutions and public sector institutions, the Bank's database and statistics on NGOs and CBOs do not provide a reliable picture of their involvement in Bank-supported projects. Nor do they describe results. The database mainly records whether provision was made for NGO or CBO involvement, not the actual involvement. 4.19 In sum, much remains to be done to develop a better understanding of the role of institutions in assessing and encouraging development effectiveness. As measured by the performance of Bank projects, there are significant and broad gains to be realized from developing and maintaining well-run and effectively managed public institutions. Simi- larly, increased vigilance on corruption-and increased reliance on new public sector man- agement techniques, such as results-based management--could have positive spillover effects on the overall quality of Bank assistance. Finally, the Bank could more systematically en- gage the institutions of civil society in addressing issues of gender equality and poverty alleviation. The Institutional Development Fund, the Consultative Group to Assist the Poor- est, and the new lending instruments are promising vehicles for doing this. 38 THEMATIC EVALUATION S AND INSTITUTIONAL DEVELOPMENT Notes 1 See Clague (1997) and Knack and Keefer (1995). For literature on corruption, see Bardhan (1997), World Bank (1997b, Mauro (1995, and Rose-Ackerman (1998). 2 To consider the effects that country characteristics might have on Bank performance, we exam- ined performance in a wide range of countries. For example, we grouped countries by whether they were post-conflict societies, transition economies, Sub-Saharan Africa economies, had high or low levels of corruption, good or weak bureaucracies, persistently poor policy environments, and so on. Annex 2 describes data sources and country classifications. 39 Chapter 5 Implications for the Bank and for Evaluation The implications of the current financial crisis are sobering for the Bank and for the evaluation profession. It has become amply clear that the value added by development assistance programs and by evaluations -would be substan- tially enhanced by more explicit attention to exogenous factors and long- term structural constraints. There should also be a sharper focus on the measurement of poverty reduction as the acid test of development and better methods of assessing institutional development both at project and country levels. 5.1 Last year's Annual Review of Development Effectiveness concluded that "the chal- "c lenge is to find the right fit between country policy and institutional factors ard strategies In a much more to try to improve conditions favorable to improved growth and development" (p. 51). In a complex-and much more complex-and hostile-external environment, this year's Review reaches simi- hostile-external lar conclusions. It is now even clearer that improvements in project performance are not environment enough. Broader structural and social constraints impede project effectiveness, but so too improvements in does the riskiness of the global environment for developing countries. 5.2 Before considering what these constraints mean for Bank operations and their project an evaluation of longer-term trends in Bank performance and development performance are evaluation, notauainoflnertr tedeinakoufrmnganhtveomn effectiveness needs to be placed in the context of the unprecedented events of the past year. notenough It is particularly important to consider the implications that these events pose for the pros- pects of sustainable growth and poverty reduction. Global risks 5.2 Some evidence of just how hostile the current environment is for developing coun- tries is shown by Euromoney's country risk ratings. For this Review, OED calculated a GDP-weighted measure of country risk for developing and industrial countries. The mea- sure for developing countries shows a deterioration in the past year to the :riskiest level since the Latin American debt crisis-and one of the biggest adverse shifts since World War II. The measure also shows that in the aftermath of the crisis, the external environment for developing countries remains, unlike that for industrial countries, at a high level of uncer- tainty-again the highest since the debt crisis. For industrial countries, the aggregate trend is the opposite: toward less volatility. Developing countries are now perceived as very risky investment environments. Does this matter? 5.3 Perhaps a great deal. While the relationship between such aggregate measures of country risk, investment flows, and project performance is not simple, the broad dimen- 41 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS sions of the relationship are clear. As the debt crisis took hold in the mid-1980s, risk mea- sures increased and Bank performance deteriorated to below 70 percent satisfactory out- comes. Then, as the crisis was resolved and the environment improved, private capital flows increased sharply and project performance improved, particularly in the past few years. These relationships are hardly precise. :But they cannot be ascribed to coincidence. To disentangle some of the possible effects for development effectiveness, examining the relationship to Bank performance is instructive. 5.4 Measures of country risk are expectations about likely performance-expectations often not realized. The measures can change after investments have taken place and been evaluated, either to reflect the changes that took place or to correct expectations. Countries can do much better than was expected-as many did in the years before the East Asian crisis-and much worse than expected-as in East Asia and Russia over the past year. How did changes in country risk affect the performance of Bank projects exiting in fiscal 1997 and 1998? Consider three types of economies: those where country risk during the imple- mentation of Bank project was stable, those where the country risk rating improved signifi- cantly during project implementation, and those where it deteriorated (table 5.1). Table 5.1 Average performance of projects implemented in three country-risk environments, projects exiting in fiscal 1997-98 Improvers Stable Deteriorators Share satisfactory (percent) 8 74 73 Note: The change in risk rating between the year of approval and the year of exit has been used to classify countries. Improvers refer to countries whose risk rating improved by more than five percentage points; stable refers to countries whose rating changed by less than five percentage points; deteriorators refer to countries whose risk increased between the year of approval and the year of exit. 5.5 Countries whose circumstances improved account for more than 60 percent of the Performance exiting projects. In these countries, risk perceptions improved considerably during imple- improved mentation, as the world moved to one of the highest-ever growth rates for developing because it was countries. Not surprisingly, project performance in these economies was good-the likeli- being swept hood of a project's having a satisfactory outcome in those countries was 85 percent. This is along by a rising 11 percent higher than in economies that performed as expected when the project was tide initiated. Fortunately, outcomes appear to be more robust with respect to the unanticipated deterioration in country risk. Countries that experienced an increase in risk, even those that had a substantial increase, did not experience a large reduction in project performance. So at least from a preliminary analysis of past performance, the deterioration in conditions should not be devastating for the existing portfolio. But the effects on future projects may be more serious. 5.6 In some respects the recent improvement in overall performance on Bank-sup- ported projects has been the result of projects being implemented in economies undergoing 42 IMPLICATIONS FOR THE BANK AND FOR EVALUATION improvements in broad fundamentals. In this light, performance improved because it was being swept along by a rising tide. But what will happen if this tide has crested, and future performance is no longer buoyed by a continually improving external environment? If current forecasts of country risk are accurate, and a large portion of Bank borrowers do not experience an improving external environment during implementation, overall satis- factory performance of future projects could be reduced by as much as 5 percentage points.2 Evaluation must give greater prominence to the effects of the external environment, par- ticularly for sustainability. Box 5.1 What do we know about the extent of poverty? The Bank is considered quality and availability of incomes would have a big- The money-metric the largest repository of household survey data for ger effect on poverty than measure of welfare in- information on poverty. developing countries have if they were spread more volves setting poverty But it has systematically improved considerably evenly.' lines and denoting the focused on data collec- since then. Today 138 sur- It remains difficult to household cost of the tion only since 1991, veys are available for 69 compare rates of poverty in level of welfare needed to when it issued a directive countries. The timeliness of different countries. There escape poverty. Best mandating poverty as- data has also improved are, in fact, conceptual practice involves adjust- sessments in borrowing from an average lag of 11 and practical problems.4 ing for differences over countries. So far, 94 as- years in the mid-1980s to The surveys from which the time or space and house- sessments have been done about 5 years now. Even poverty data are drawn hold demographics. But (83 countries and 11 up- so, World Development In- are: the data needed to do dates) covering about 90 dicators 1997 reports esti- this consistently are in- percent of the world's mates of the population *Taken at different points adequate and generally poor. Twenty-two pov- living below $1 a day per in time. variable. The problems erty assessments were person for only 60 coun- *Based on different sample of making purchasing completed in fiscal 1996 tries. designs that may or may power parity currency (17 countries and 5 up- The most commonly not be nationally repre- adjustments for interna- dates) and 10 (8 coun- used measure-the head sentative. tional comparisons adds tries and 2 updates) in count index-counts *Conducted under method- to the lack of compara- fiscal 1997. By 2000 the people below the poverty ologies that are often dis- bility. Setting the poverty plan is to complete the line but ignores what is similar. lines too high to include, i remaining 22 assessments happening to them and *Designed to yield a wide say, countries in Eastern and 9 scheduled updates. whether they are becoming variety of often different Europe in the compari- When the Bank's poorer. In the extreme, the types of information. son raises the estimates World Development Re- measure actually improves of poverty for other port on poverty was pub- if the poor die from pov- Some obtain informa- countries. lished in 1990, poverty erty (Sen 1976). World De- tion only on incomes, If this vast heterogene- measures could be calcu- velopment Report 1990 while others obtain infor- ity in the underlying data lated for only 11 coun- recognized that for any mation only on. consump- is not carefully con- tries. These surveys given increase in the in- tion. Most differ in the trolled for, aggregation covered the 10 years comes of the poor, the re- depth and detail about to obtain regional or glo- leading to 1990 and to- duction in poverty consumption. Methods of bal estimates is not valid, gether accounted for 40 depended on where the valuation vary consider- and comparisons across percent of the total poor were relative to the ably, with some surveys us- countries and time are at population of the devel- poverty line. If they were ing prices at the nearest best spurious. oping world and 50 per- concentrated just below the market while others use cent of its poor. The line, the increase in their farmgate prices. 43 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Prospects for poverty reduction 5.7 The events of the past year confirm that the assessment of development effective- The assessment ness should give pride of place to poverty alleviation. Chapter 3 reviewed the effects that of development the recent financial crisis and the ongoing transition from socialism have had on income effectiveness distribution. What these shocks mean for poverty alleviation can be seen by considering should give pride what they mean for the OECD's Development Action Committee's (DAC) goals on pov- of place to erty alleviation. The DAC goals call for a reduction in the number of people in absolute poverty poverty of 1 billion by 2015. These goals require that about 50 million people be raised alleviation from poverty each year for the next 20 years. The East Asian crisis has, in effect, already put the DAC program nearly a year behind schedule. But this was not the only poverty- increasing shock. The increase in poverty due to the collapse of safety nets in transition economies is larger than the East Asian effect, and to offset these increases will require perhaps another three to four years of successful effort on poverty reduction for the DAC goals to be realized. 5.8 Together these two shocks have moved the goal posts four to five years farther away than when the targets were established only two years ago. Much more must be done in improving poverty measures if the Bank and the donor community are to come to grips with the full dimensions of global poverty (box 5.1). While the goals may be a useful structure, greater detail is needed to flesh out all the important dimensions of poverty reduction. Better data are needed so that the poor truly become visible in our evaluations and goals. Finally, more than safety nets and income growth are needed to achieve DAC's poverty goals (box 5.2). Box 5.2 Poverty alleviation and income growth In assessing human wel- because improved life ex- disease control. But per- byproduct of economic fare, advocates of human pectancy is typically viewed haps economic growth has growth or the free mar- development would, at a as a byproduct of economic been necessary for increas- ket conditions that foster minimum, place indica- development. ing life expectancy-by it. Indeed, public policy tors of social condi- Not always. New tech- providing the resources initiatives have been es- tions-notably life niques of disease control, needed to fund public sential to the improve- expectancy and educa- based on new knowledge of spending on the new tech- ment of life expectancy, tional achievement-on disease, have been sources nology, either directly or and these can be-and equal footing with such of improved life expect- through international aid, have been-undertaken traditional economic ancy. And public interven- or to fund the research re- in the absence of eco- measures as GDP per tion has been crucial for sponsible for the advance nomic growth. Life ex- capita and a poverty in- implementing them. The in knowledge. At most, pectancy is an objective dex. Some would go fur- free market institutions however, economic growth to be pursued in its own ther and include commonly considered to be may have been helpful, but right by the institutions indicators of political behind economic growth it was not required to fi- and policies it requires. and civil liberties. To the have not been responsible nance the advance in broader measure of hu- for adopting the new tech- knowledge that brought in- man development, a com- niques of disease control. fectious disease under con- mon reaction among Nor do free market institu- trol. economists and economic tions appear to have gener- So higher life expectancy historians is skepticism, ated the new technology of cannot be taken as simply a 44 IMPLICATIONS FOR THE BANK AND FOR EVALUATION Implications for the Bank 5.9 What does all this mean for Bank operations and evaluations? Three issues require greater emphasis: poverty alleviation, country strategies, and institutional development. Poverty alleviation 5.10 In recent years poverty alleviation and social concerns have been afforded more attention in Bank strategies, as mandated by the Strategic Compact. This year's Review finds that much greater donor coordination is essential, particularly in low-income coun- tries. In these countries, declining volumes of external assistance must be better coordi- nated and more sharply focused on poverty reduction. 5.11 OED evaluation efforts should emphasize social aspects more. Rating systems should give more weight to the social impact of projects and programs. Recent shocks have had a profound negative effect on poverty, and it is difficult to exaggerate the impor- tance of the need for measures to address these concerns systematically. Safety nets are the vehicle for addressing shocks and should be at the forefront of poverty alleviation con- cerns, rather than being comfortably assumed away as they have been in many analyses of world poverty. Country strategies 5.12 The Bank has made considerable progress on broadening the development effec- tiveness spectrum-for example, it has already moved beyond projects in its strategic per- spective. But as the discussion of the external environment showed, continual improvements in Bank and borrower performance may not be enough to maintain the performance of recent years, much less continue the steady improvements. In such a context, even the best- designed project will not contribute to development effectiveness without a greater under- standing of how it fits into the broader country and international environment. 5.13 For evaluation, OED will reemphasize and intensify its shift in focus on the coun- try as the appropriate unit for evaluation. It will also give more attention to sector perfor- mance. These new emphases will build on, not substitute for, OED's traditional concern with project performance. Institutional development 5.14 The events of the past year have revealed how costly weak institutions can be. The The events of the importance of sound institutions has long been known. Not known was how costly a sys- temic institutional failure can be. Bank operations have made significant progress on focus- past year have ing resources on this important issue. And as discussed in chapter 1, a major dimension of revealed how this work will be on financial institutions-but recent events underscore that much more costly weak needs to be done. institutions can 5.15 For evaluation, greater emphasis should be placed on the metrics of institutional be development. The development effectiveness index is but a first step on this journey, and further work should be done in considering how institutional development can be given more emphasis in evaluation measures. Simpler, concrete steps are needed. For example, 45 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS today's project supervision and completion reports do not require a rating of institutional development impact. This shortcoming should be rectified. For its part, OED should make sure that its new country evaluation instrument gives adequate attention to institutional development issues. 5.16 The past year has tested the Bank's new framework for providing development assistance. While this test is far from complete, the past year's results appear promising. The broad dimensions of the Bank's strategy appear to be well conceived. With continued adjustment and refinement, they should permit the Bank to help developing countries con- front a much tougher external environment. o is 5.18 Evaluation is central to adjusting the Bank's approach. For example, systemati- Evaluation is cally addressing poverty alleviation requires making the poor visible through better data central to and monitoring systems. Output targets must be linked much more tangibly to policy in- adjusting the puts and must recognize the increased risks of the external environment. With the help of Bank's approach evaluators, those targets must be embedded in the scorecards of country and sector assis- tance strategies. Notes 1 Euromoney's semiannual country risk ratings range from 0 (most risky) to 100 (least risky). Each rating is calculated as a weighted average of nine categories of indicators representing analytical credit and market indicators. The categories are economic data (25 percent weighting), political risk (25 percent), debt indicators (10 percent), rescheduled debt or debt in default (10 percent), sovereign credit ratings (10 percent), access to bank finance (5 percent), access to short-term finance (5 per- cent), access to international bond and syndicated loan markets (5 percent), and access to and dis- count on forfeiting (5 percent). 2 The estimated relationship between country risk and portfolio performance during 1997-98 is used to forecast ratings for projects exiting in 1999. The forecast of a 5 percentage point deteriora- tion in fiscal 1998 satisfactory projects is used as a baseline scenario, assuming that country risk ratings in 1999 would be the same as in 1998. In a more pessimistic scenario, country risk ratings would deteriorate by 6 percentage points. 3 World Development Report 1990, box 3.3, page 47. 4 See Ravillion (1994a, 1994b, 1996) and Pollak (1991). 5 The estimated number of poor people in 2015, given the expected increase in population and assuming a 50 percent reduction in poverty, is 900 million. This would call for lifting almost 1 billion people out of poverty over the next 20 years, or about 50 million people a year. 46 Annex 1 A New Approach to Evaluating Bank Projects-The Development Effectiveness Index The current Bank evaluation system assesses project results through a set of three ordinal ratings-on project outcome, sustainability, and institutional development impact: * Outcome boils down to answering the following question: Did the project achieve satis- factory development results considering the relevance of its main stated objectives, and the associated costs and benefits? The outcome rating takes into account relevance (to check whether the project's objectives were consistent with the country's development strategy), efficacy (to examine whether the operation achieved its stated goals); and effi- ciency (to assess results relative to inputs in terms of costs, implementation times, and economic and financial returns). Outcome is rated on a six-point ordinal scale: highly satisfactory, satisfactory, marginally satisfactory, marginally unsatisfactory, unsatisfac- tory, and highly unsatisfactory. * Sustainability is defined as the likelihood, at the time of evaluation, that the project will maintain its results in future. In assessing sustainability, evaluators focus on features (country conditions, government and economic policies, political situation:, and condi- tions specific to the operations, such as availability of funds for maintenance) that deter- mine whether the operation will last over its intended useful life. Sustainability is rated on a three-point ordinal scale: likely, uncertain, and unlikely. * Institutional development impact is defined as the extent to which a project has im- proved an agency's or country's ability to use its human and financial resources effec- tively and to efficiently organize economic and social activities. Institutional development impact is rated on a three-point ordinal scale: substantial, modest, and negligible. As in previous years, we report trends on each evaluation dimension separately (see chapter 2). In this year's review, however, we have introduced a new measure of overall project results-the development effectiveness index-based on the three ratings. This in- strument allows us to take the analysis of portfolio trends a step forward, making it pos- sible to compare overall average results-and their variability-across different groups of projects. The development effectiveness index is defined by assigning cardinal weights to the rat- ings of each of the three result-oriented counts, then combining them in a simple way. The index formula is: 47 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS development effectiveness index = outcome weight + sustainability weight + institutional development weight where Institutional development Outcome weight Sustainability weight impact weight Highly satisfactory 7.75 Likely 0.75 Substantial 1.5 Satisfactory 6.0 Uncertain 0.25 Modest 0.5 Marginally satisfactory 5.25 Unlikely 0 Negligible 0 Marginally unsatisfactory. 4.5 Unsatisfactory 3.75 Highly unsatisfactory 2.0 Thus the development effectiveness index ranges from 2-for a project with highly un- satisfactory outcome, unlikely sustainability, and negligible institutional development im- pact-to 10-for a project with highly satisfactory outcome, likely sustainability, and substantial institutional development impact. Looking at the weights, it is easy to see that outcome is the main force behind the index. Note also how the index, separates between satisfactory and unsatisfactory outcomes, where an index measure,of 6 represents such a "divide." In fact, a project with unsatisfactory outcome will never score higher than 6, no matter what ratings it receives on the other two dimensions, The average development effectiveness index in the fiscal 1990-98 portfolio is 6.47. The standard deviation is 1.85. The contribution of outcome to the average index in the portfo- lio is about 80 percent; the remaining 20 percent is almost evenly split between the other two evaluation components. A similar cardinal measure of overall project performance was put forth in last year's Review. The thinking used in building that index was to establish, using subjective assess- ment, the relative importance of the three results-oriented counts, and then combine them in an intuitively appealing way. One of the main drawbacks of this approach was that, by ignoring the information embedded in the historically observed portfolio, it tended to ex- cessively penalize underperforming projects and over reward overperforming ones. It in- duced a double-counting effect that, rather than conveying the extra information contained in the six-point classification of outcome results and the data about institutional develop- ment impact and sustainability, duplicated a satisfactory-unsatisfactory dichotomy. The new measure improves on the index described, making the inferences based on such an instrument more robust. It does so by explicitly taking into account the information coming from the historically evaluated portfolio, and by keeping to a minimum-and trans- parently stating- the set of subjective judgments needed to choose a specific index among the many ways of defining one. Weights are assigned using the historical evaluated portfolio as a benchmark, and taking into account the strong positive association among evaluation dimensions, to avoid double- counting effects and to extract the most information from the empirical observations. The fact that good (bad) ratings on one dimension are associated with good (bad) rat- ings on the other dimensions allows us to unambiguously rank more than 70 percent of the 48 THE DEVELOPMENT EFFECTIVENESS INDEX projects in the observed portfolio using only the ordinal information conveyed by the three sets of ratings. Using only two clear assumptions-premised on the Bankwide consensus about the importance of outcome ratings and on the most efficient way to use information embedded in the observed portfolio-we are able to increase the fraction of projects that can be ranked to more than 81 percent. Such rankings are then used to assign scores to each dimension separately. Choosing a formula that defines the index as the sum of three scores makes it easy to understand and calculate within the Bank, as well as outside, and allows us to easily calculate changes in the overall index associated with given changes in the distribution of ratings in a given group of projects (by sector, region, and so on). Caution should be exercised in interpreting the scores on each separate dimension, since they are derived by looking at the additional informational content that each result-based count contained relative to the other two in the historically evaluated portfolio. The fact that the ratings on outcome appear to be the driving force behind the index is the result of two forces. First, in establishing the ranking of observed projects on which the index is built, we used (although parsimoniously) outcome as a tie-breaking rule. Second, the fact that outcome was rated on a six-point basis, as opposed to the three-point bases used for the other two counts, resulted in outcome ratings conveying more information, and thus receiving more weight. The fact that outcome evaluation is based on a finer scale is a testimony to its prominence among the ratings, and we welcome this asymmetry because we did not have to impose any subjective mechanism to give outcome more weight in the determination of the index. In conclusion, although we report the percentage each result-based count contributed to the index, this was done only for completeness. After all, the index's purpose is to summa- rize information as far as possible: researchers interested in examining performance on each count will find it optimal to look only at the ratings in the chosen dimension (rather than look at the components of the weights which are intimately connected to one an- other). Another point worth stressing about the index is the nature of its cardinality. The cardi- nality should be understood as interval-scale cardinality: because it is impossible to locate an absolute, nonarbitrary, zero point for the scale, distances between index values are mean- ingful, but not ratios. We choose a 2-10 range because this makes the index readily compa- rable with other indexes used within the Bank while allowing us to space index values in a way that may be appealing for practitioners and nonpractitioners alike. Although it is not of the ratio-scale nature (ratios of indexes are not meaningful, given the arbitrary range), the cardinality of the development effectiveness index makes it a valu- able instrument for analyzing Bank project performance. For a start, it makes it possible to compare overall performance across, for instance, Sectors, Networks, and Regions using the information contained in the project ratings. This represents a step forward with re- spect to standard comparisons based simply on binary classification (satisfactory or unsat- isfactory) of outcome, which not only leave out the sustainability and institutional development components of projects results, but also ignore the nuances embedded in the six-point scale on outcome. This is the use of the index we are mainly concerned with in this report (see chapter 2). 49 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS The usefulness of the development effectiveness index is not limited to the use suggested here. Using the index, means and variances for different groups of projects-representing different types of investment-can be calculated, and the portfolio evaluated using the standard tools of portfolio theory. The richer information embedded in the index can be used as an alternative dependent variable in regression studies that try to explain the factors behind the Bank's intervention successes and failures (see Burnside and Dollar 1997). The index could also serve as an explanatory variable in the right-hand side of regression analyses linking Bank efforts to development results (reduced poverty or inequality, increased growth, and so on). It is worth stressing that the guiding rationale in translating into a cardinal measure an ordinal system of performance ratings is to come up with an evaluation instrument with desirable properties, to be used to trace out the major trends, factors, and effects of Bank investments. The development effectiveness index is not meant to substitute for direct car- dinal measures of project performance, like the rate of return approach to project evalua- tion. But given the shift of development economics toward greater attention to policy and institutions, the rate of return methods are ill-fitted to capture the policy reform and insti- tution-building components of Bank projects., This motivated the adoption of an evalua- tion system applicable to all projects (for about a third of Bank projects rate of return can still be-and is-calculated) that sacrifices the advantages of cardinality to capture the many facets of project results. Thus the development effectiveness index represents an at- tempt to solve the tradeoff between these two conflicting factors, and should not be con- strued as an attempt to "faithfully and precisely" measure in a cardinal way the underlying reality. 50 Annex 2 Glossary Adjustment loans: Financing aimed at promoting policy reform. Disbursement of these funds, directed at alleviating the costs of the transition to a different policy and institu- tional environment, is contingent on the fulfillment of a set of conditions by the recipient country (usually based on macroeconomic indicators). Borrower ownership: The extent to which the recipient country is involved in and commit- ted to a project's strategy and goals. Ownership is greater when the borrower initiates the formulation and implementation of a project, there is clear consensus among government officials and other decisionmakers on the course of action, and there is broad' public sup- port for the initiative. Contagion: Transmission of destabilizing conditions from one open economy to others closely connected to it, resulting in regional crisis. Country assistance strategy: The main vehicle for Board review of the Bank Group's assis- tance to IDA and IBRD borrowers. The strategy document describes the Bank Group's strategy based on an assessment of country priorities and indicates the level and composi- tion of assistance to be provided based on the strategy and the country's portfolio perfor- mance. The heart of the country assistance strategy is the ongoing Bank-country dialogue and joint efforts in preparing and implementing the strategy. Strong country ownership and consultation with key stakeholders-pursued with sensitivity and prior general agree- ment of the government-are crucial features of a successful country assistance strategy. DAC goals: A set of six internationally accepted development goals for the 21st century in the areas of poverty, gender, education, environment, and health. The goals were published by the OECD's Development Assistance Committee in 1996 in a policy paper known as Strategy 21. Demandingness, complexity, and riskiness: These are three important project design char- acteristics. Demandingness refers to the extent to which the project could be expected to strain the economic, institutional, and human resources of the government/implementing agency. Complexity refers to such factors as the range of policy and institutional improve- ments contemplated, the number of institutions involved, the number of project compo- nents and their geographic dispersion, the number of cofinanciers, etc. Riskiness refers to the likelihood that the project as designed, would be expected to fail to meet relevant project objectives efficiently. In determining project riskiness, evaluators consider the ex- tent to which the project could reasonably have been expected at the time of project prepa- 51 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS ration and appraisal to face known risk factors, such as lack of borrower commitment, inadequate counterpart funding, and war or civil disorder. Development effectiveness: A demonstrable contribution to economically sound, socially equitable, and environmentally sustainable growth. Development effectiveness index: A measure of overall project-specific results based on the index aggregation of three OED project ratings on outcome, sustainability, and institu- tional development. Global public goods: Goods that are available for the benefit of all countries, and for which one country's use does not reduce another's consumption. Yet no single country could or would invest in these goods because the costs generally outweigh the aggregate benefits. Institutional development: Improvement in an agency's or country's ability to make effec- tive use of their human and financial resources and to efficiently organize economic and social activities. Macroeconomic policy: Government actions designed to affect the entire economy rather than specific sectors or markets, especially with respect to the general level of income, employment, prices, interest rates, and balance of payments. Policy measures are usually categorized as fiscal or monetary, depending on which instruments-taxes, public spending and debt, control of money supply and central bank discount rates-are used. Outcome: In terms of project ratings, outcome refers to the extent to which a project achieved its major relevant objectives in a cost-efficient way. Under the results-based management framework, this is to be distinguished from outputs, which measure what is produced or done (for example, using fewer resources compared with plans, previous performance, or the performance of other organizations). The outcome of a health publicity campaign might be a 5 percent increase in awareness among those targeted. Risk-bearing institutions: Organizations designed to share the risks and costs of unpredict- able events among large groups. Examples include unemployment support programs, health programs, and life insurance. Safety nets: Mechanisms that aim to alleviate the burden on the vulnerable of an unfavor- able economic situation (e.g., unemployment insurance). Strategic Compact: The Strategic Compact between the Bank's management and executive board provides a long-term framework for guiding the Bank's renewal and calculating the associated resource needs. The Compact, approved in March 1997, adds $250 million to the Bank's $1.2 billion administrative budget to be used over 30 months to improve how the Bank does business. The Compact is complemented by substantial redeployments and 52 GLOSSARY savings throughout the Bank, identified through an ongoing review of cost-effectiveness. The Compact focuses on refueling current business activities, refocusing the development agenda, retooling the Bank's knowledge base, and revamping institutional capacities. Sustainability: The likelihood, at the time of evaluation, that a project will maintain its results in future. Washington consensus: An internationally agreed set of measures that are typically imple- mented during policy reform, including fiscal discipline, financial and trade liberalization, deregulation, taxation and public expenditure adjustments, and privatization. 53 Annex 3 Data on Outcome Sustainability and Institutional Development Impact Table 1: Outcome, sustainability, and institutional development (ID) impact for exit fiscal years 1990-96, 1997, and 1998, by sector, network, lending type and source, region and income group (by projects) z Exit fiscal years 1990-96 Exit fiscal year 1997 Exit fiscal year 1998 ID ID ID Outcome Sust. Impact Outcome Sust. Impact Outcome Sust. Impact #of % % % #of % % % #of % % % -H proj. Share sat. likely sub. proj. Share sat. likely sub. proj. Share sat. likely sub. > Agriculture 431 27 63 36 31 51 24 76 55 39 26 23 85 54 62 H Z Education 128 8 78 54 33 19 9 74 63 47 9 8 88 38 38 - Electric Power 148 9 67 59 32 11 5 55 45 18 7 6 67 50 33 OO & Other Energy zC Environment 2 0 100 50 0 7 3 71 57 29 1 1 100 100 0 > Finance 83 5 56 43 29 14 6 77 69 46 7 6 86 57 29 TA.--Q C C AA Al 1<70 I1Ustr 855 4 26 7 3 57 57 57 4 4 0 0 0 Mining 18 1 65 61 50 3 1 67 67 100 2 2 50 50 100 Multisector 138 9 77 57 32 21 10 81 43 25 9 8 78 33 0 Oil & Gas 49 3 80 53 57 4 2 75 75 25 2 2 100 50 50 Population, 51 3 65 47 20 12 6 83 67 25 12 11 75 67 25 Health & Nutrition oCi Public Sector 80 5 54 39 28 12 6 67 67 8 9 8 100 56 56 Management > Social Sector 10 1 80 40 50 7 3 100 14 71 7 6 71 0 0 - Telecommunications 30 2 77 70 37 3 1 67 100 33 2 2 100 100 100 z Transportation 183 12 74 52 29 21 10 81 48 67 5 4 100 80 80 > Urban Development 77 5 66 36 21 16 7 75 44 25 6 5 100 60 40 Water Supply 66 4 60 33 31 9 4 67 33 0 6 5 67 50 50 & Sanitation (j> Table 1: Outcome, sustainability, and institutional development (ID) impact for exit fiscal years 1990-96, 1997, and 1998, by sector, network, lending type and source, region and income group (by projects) continued Z z Exit fiscal years 1990-96 Exit fiscal year 1997 Exit fiscal year 1998 ID ID ID Outcome Sust. Impact Outcome Sust. Impact Outcome Sust. Impact # of % % % # of % % % # of % % % proj. Share sat. likely sub. proj. Share sat. likely sub. proj. Share sat. likely sub. Environmentally 433 27 63 36 31 58 27 76 55 38 27 24 85 56 59 & Socially Sustainable Development Finance, Private Sector 739 47 67 49 31 88 41 71 53 40 41 36 74 54 46 & Infrastructure Human Development 189 12 74 51 30 38 18 82 55 45 28 25 7 41 22 Poverty Reduction 218 14 69 50 30 33 15 76 52 19 18 16 89 44 28 & Economic Management i Lending type < Adjustment 225 14 74 57 35 33 15 79 58 33 12 11 92 50 50 Investment 1,354 86 66 44 30 184 85 74 53 38 102 89 79 49 39 0 Lending source IBRD only 821 52 71 56 35 105 48 74 61 36 47 41 75 48 43 IDAlblend 758 48 62 34 27 112 52 76 47 39 67 59 84 51 39 Region Africa 533 34 54 28 21 63 29 62 30 42 37 32 78 46 32 East Asia and Pacific 278 18 83 71 42 41 19 83 66 44 22 19 81 43 52 Europe and Central Asia 111 7 76 59 43 18 8 83 78 50 11 10 80 60 40 tr Latin America 283 18 69 53 36 47 22 85 57 35 21 18 85 65 40 & Caribbean Middle East 161 10 69 47 28 20 9 70 65 25 7 6 86 43 57 ti 0 & North Africa South Asia 213 13 68 40 29 27 12 78 63 22 15 13 73 47 33 Other 1 0 0 0 0 1 1 100 0 100 lIncome G toup Lower 862 55 61 35 26 119 55 72 47 38 66 58 77 48 35 Z Lower middle 458 29 71 52 32 63 29 75 56 33 32 28 84 42 42 Upper middle 219 14 79 69 44 31 14 87 70 47 14 12 85 85 54 High 40 3 82 77 44 4 2 75 100 0 2 2 100 0 100 Total/average 1,579 100 67 46 31 217 100 75 54 37 114 100 80 50 40 Note: Sust.= sustainability, proj.= projects, sat.= satisfactory, sub.= substantial. Income group categories are derived from the World Development Indicators 1998. A full set of supplemental statistical tables providing further project evahiation results are available electronically at http://www.worldbank.org/htmi/oed. Table 2: Outcome, sustainability, and institutional development (ID) impact for exit fiscal years 1990-96, 1997, and 1998, by sector, network, lending type and source, region and income group (by disbursements) Exit fiscal years 1990-96 Exit fiscal year 1997 Exit fiscal year 1998 Disburse. Outcome Sust. ID impact Disburse. Outcome Sust. ID impact Disburse. Outcome Sust. ID impact $millions % sat. % likely % sub. $millions % sat. % likely % sub. $millions % sat. % likely % sub. $eetor Agriculture 26,669 74 47 37 3,096 77 66 42 1,056 87 71 77 Education 5,993 80 61 46 1,438 85 71 53 362 95 20 25 Electric Power 19,350 66 67 35 1,528 87 76 40 572 90 59 48 & Other Energy Environment 75 100 29 0 232 78 76 40 34 100 100 0 Finance 9,767 60 53 38 2,336 50 45 33 847 85 74 68 Industry 11,890 64 55 29 1,338 75 75 81 407 0 0 0 Mining 1,273 69 80 41 106 79 79 100 270 97 97 100 > Multisector 21,903 86 65 44 1,488 90 57 25 760 99 34 0 Oil & Gas 3,683 83 89 42 246 99 99 38 347 100 96 4 Population, Health 1,625 79 65 36 800 94 71 22 741 80 70 16 c0 & Nutrition Z Public Sector 3,974 77 58 53 505 89 51 10 273 100 29 83 0 0 Management Social Sector 674 99 89 18 435 100 6 83 181 67 0 0 H Telecommunications 1,306 78 79 44 382 90 100 88 91 100 100 100 o Transportation 16,198 83 59 32 2,047 84 44 60 675 100 58 60 Urban Development 6,857 75 50 26 670 76 58 16 544 100 62 7 Water Supply 4,574 55 29 26 1,187 65 17 0 564 59 21 24 & Sanitation C) C"r Environmentally 26,744 74 47 37 3,328 77 67 41 1,090 87 71 74 & Socially Sustainable Development > Finance, Private Sector 74,898 70 59 33 9,840 73 55 44 4,318 81 58 42 & Infrastructure z Human Development 8,292 81 64 42 2,673 90 60 49 1,285 82 46 16 Poverty Reduction 25,877 84 64 46 1,993 90 55 21 1,033 99 32 22 & Economic Management Table 2: Outcome, sustainability, and institutional development (ID) impact for exit fiscal years 1990-96, 1997, and 1998, by sector, network, lending type and source, region and income group (by disbursements) continued z Exit fiscal years 1990-96 Exit fiscal year 1997 Exit fiscal year 1998 C Disburse. Outcome Sust. ID impact Disburse. Outcome Sust. ID impact Disburse. Outcome Sust. ID impact $millions % sat. % likely % sub. $millions % sat. % likely % sub. $millions % sat. % likely % sub. Lending type Adjustment 41,619 78 64 43 3,717 66 52 33 1,742 97 57 65 Investment 94,191 72 55 34 14,118 81 60 44 5,984 81 53 32 Lending source IBRD only 98,619 75 64 40 11,034 75 55 43 4,979 84 52 40 IDA/blend 37,192 72 43 29 6,800 83 63 40 2,747 85 60 38 keghoo Africa 22,440 63 29 24 2,678 72 45 43 1,361 73 37 30 East Asia and Pacific 28,468 88 81 48 4,467 88 70 55 2,089 76 31 37 Europe & Central Asia 14,244 76 62 44 1,735 88 80 53 1,074 100 87 30 Latin America 36,653 72 65 38 4,441 71 38 37 1,746 94 74 66 & Caribbean o Middle East 10,119 71 44 29 1,618 70 52 34 519 96 34 37 & North Africa South Asia 23,887 70 51 32 2,894 78 73 25 936 79 69 20 In onte Group Lower 56,864 71 47 31 8,646 80 66 42 3,191 76 54 36 Lower middle 42,938 73 58 37 4,944 81 51 41 3,099 88 37 38 Upper middle 33,208 79 74 46 4,061 71 48 43 1,353 96 98 47 High 2,801 81 79 46 184 81 100 0 83 100 0 100 Total/average 135,811 74 58 37 17,834 78 58 42 7,726 84 54 39 Note: Disburse.= disbursements, sust.= sustainability, sat.= satisfactory, sub.= substantial. Disbursements are measured in real terms, deflated to fiscal 1996 US dollars. Income group categories are derived from the World Development indicators, 1998. A full set of supplemental statistical tables providing further project evaluation results are available electronically at http://www.worldbank.org/html/oed. tr t7l Bibliography Bardhan, Pranab. 1997. "Corruption and Development: A Review of the Issues." Journal of Eco- nomic Literature 35(3): 1320-46. Barro, Robert J. 1995. "Technological Diffusion, Convergence, and Growth." NBER Working Pa- per 5151. National Bureau for Economic Research, Cambridge, Mass. Barro, Robert J., and Xavier Sala-i-Martin. 1995. Economic Growth. New York: McGraw-Hill. Bell, Clive, and Robert Rich. 1994. "Rural Poverty and Agricultural Performance in Post-Indepen- dence India." Oxford Bulletin of Economics and Statistics 56(2): 111-33. Berg, Elliot J. 1993. Rethinking Technical Cooperation: Reforms for Capacity Building in Africa. New York: United Nations Development Programme. Bezanson, Keith. 1998. "Some Implications of the East Asian Crisis for Development Effectiveness- Personal Perspectives Prepared at the Request of the Operations Evaluation Department of the World Bank." Institute of Development Studies, Sussex University. Birdsall, Nancy. 1998. "Life is Unfair: Inequality in the World." Foreign Policy 111(summer): 76- 93. Burnside, Craig, and David Dollar. 1997. "Aid, Policies and Growth." Policy Research Working Paper 1777. Washington, D.C.: World Bank. Caprio, Gerard Jr. 1996. "Banking on Financial Reform? A Case of Sensitive Dependence on Initial Conditions." In Gerard Caprio Jr., Izak Atiyas, and James A. Hanson, eds., Financial Reform: Theory and Experience. Cambridge, Mass.: Cambridge University Press. -. 1998. Banking on Crises: Expensive Lessons from Recent Financial Crises. Washington, D.C.: World Bank. Caprio, Gerard Jr., and Daniela Klingebiel. 1996. "Bank Insolvencies: Cross-Country Experience." Policy Research Working Paper 1620. World Bank, Police Research Department, Washington, D.C. Cassen, Robert, and associates. 1994. Does Aid Work? Oxford: Clarendon Press. Claessens, Stijn, and Thomas Glaessner. 1997. Are Financial Sector Weaknesses Undermining the East Asian Miracle? A Directions in Development book. Washington, D.C.: World Bank. Clague, Christopher. 1997. Institutions and Economic Development. Baltimore, M.D.: The Johns Hopkins University Press. Commander, Simon, and others. 1997. "The Causes and Consequences of Government for Growth and Well-being." Background paper for World Development Report 1997. Washington, D.C.: World Bank. 59 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Deininger, Klaus and Lyn Squire. 1996. "A New Data Set for Measuring Income Inequality." The World Bank Economic Review 10(3): 565-91. Demery, Lionel, and Michael Walton. 1998. "Are Poverty and Social Goals for the 21st Century Attainable?" World Bank, Poverty Reduction and Economic Management Network. Washing- ton, D.C. Easterlin, Richard A. 1998. "How Beneficent is the Market? A Look at the Modern History of Mortality." mimeo, University of Southern California. Easterly, William, and Stanley Fischer. 1995. "The Soviet Economic Decline." The World Bank Eco- nomic Review (9)3. EBRD (European Bank for Reconstruction and Development). 1997. Transition Report 1997: En- terprise Performance and Growth. London. Edwards, Sebastian. 1997. "Trade Policy, Growth, and Income Distribution." American Economic Review Papers and Proceedings 87(2): 205-10. Fischer, Bernhard, and Helmut Reisen. 1993. Liberalising Capital Flows in Developing Countries: Pitfalls, Prerequisites and Perspectives. Washington, D.C.: Organization for Economic Co-opera- tion and Development. Gallup, John Luke, and Jeffrey Sachs, with Andrew D. Mellinger. 1998. "Geography and Economic Growth." Paper prepared for the Annual Bank Conference on Development Economics, April 20- 21, Washington, D.C. Gwin, Catherine, and Joan M. Nelson. 1997. Perspectives on Aid and Development. Washington, D.C.: Overseas Development Council. Hadjimichael, Michael T., and others. 1995. "Sub-Saharan Africa: Growth, Savings, and Invest- ment, 1986-93." IMF Occasional Paper 118. Washington, D.C. Helleiner, Gerald, and others. 1995. Report of the Group of Independent Advisers on Development Cooperation Issues between Tanzania and Its Aid Donors. Copenhagen: Royal Danish Ministry of Foreign Affairs. IDS (Institute of Development Studies). 1998. Papers prepared for the workshop on the East Asian Crisis, July 13-14, Sussex University. IMF (International Monetary Fund). 1995. "Transfers, Social Safety Nets, and Economic Growth." IMF Staff Papers 44(1): 81-102. Ingram, Gregory. 1992. "Social Indicators and Productivity Convergence in Developing Countries." Policy Research Working Paper 894. Washington, D.C.: World Bank. Ito, Takatoshi. 1998. "What Can Developing Countries Learn from East Asian Economic Growth?" In Boris Pleskovic and Joseph E. Stiglitz, eds., Annual World Bank Conference on Development Economics 1997. Washington, D.C.: World Bank. Johnson, John H., and Sulaiman S. Wasty. 1993. Borrower Ownership of Adjustment Programs and the Political Economy of Reform. World Bank Discussion Paper 199. Washington, D.C. Johnston, Timothy, and Nicolas Van de Walle. 1996. Improving Aid to Africa. Washington, D.C.: Overseas Development Council. 60 13 I B L I 0 G R A P H Y Keefer, Philip, and Stephen Knack. 1997. "Why Don't Poor Countries Catch Up? A Cross-National Test of an Institutional Explanation." Economic Inquiry 35(July): 590-602. Knack, Stephen, and Philip Keefer. 1995. "Institutions and Economic Performance: Cross-Country Tests Using Alternative Institutional Measures." Economics and Politics 7(3): 207--27. Krugman, Paul. 1994. "The Fall and Rise of Development Economics." In Lloyd Rodwin and Donald A. Sch... n, eds., Rethinking the Development Experience. Harrisonburg, Va.: R.R. Donnelly and Sons. _. 1998. "The Confidence Game." New Republic 219: 23-25. Levine, Ross. 1997. "Financial Development and Economic Growth: Views and Agenda." Journal of Economic Literature 35(2): 688-726. Levine, Ross, and David Renelt. 1991. "A Sensitivity Analysis of Cross-Country Growth Regres- sions." American Economic Review 82(4): 942-63. Levine, Ross, and Sara Zervos. 1993. "Looking at the Facts: What We Know about Policy and Growth from Cross-Country Analysis." Policy Research Working Paper 1115. Washington, D.C.: World Bank. Lipton, Michael, and others. 1998. Successes in Anti-poverty. Geneva: International Labour Office. Maddison, Angus. 1997. "The Nature and Functioning of European Capitalism: A Historical and Comparative Perspective." Banca Nazionale del Lavoro Quarterly Review 50(203). Mathieu, Nicolas. 1998. Financial Sector Reform. Operations Evaluation Department. Washington, D.C.: World Bank. Mauro, Pablo. 1995. "Corruption and Growth." QuarterlyJournal of Economics 11,0(3): 681-712. Milanovic, Branko. 1998. Income, Inequality, and Poverty during the Transition from Planned to Market Economy. A Regional and Sectoral Study. Washington, D.C.: World Ban:. Mosley, Paul, Jane Harrigan, and John Toye. 1991. Aid and Power. The World Bank and Policy- based Lending. London: Routledge. OECD (Organisation for Economic Co-operation and Development). 1996. "Shaping the 21st Cen- tury: The Contribution of Development Co-operation." Paris: Development Assistance Commit- tee. OED (Operations Evaluation Department). 1990. "Chile: Structural Adjustment Loans." Project Performance Audit Report 08851. Washington, D.C: World Bank. _. 1995. "The Social Impact of Adjustment Operations: An Overview." Report 14776. Wash- ington, D.C.: World Bank. _. 1996a. "Argentina Country Assistance Review." Report 15844. Washington, D.C.: World Bank. 1996b. "Ghana Country Assistance Review." Report 14547. Washington, D.C.: World Bank. 1996c. "Technical Assistance." Lessons and Practices 7. Washington, D.C.: World Bank. 61 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 1996d. "The World Bank's Experience with Large Dams : A Preliminary Review of Im- pacts." Report 15815. Washington, D.C.: World Bank. . 1996e. "Social Dimensions of Adjustment." Washington, D.C.: World Bank. . 1996f. "Zambia Country Assistance Review." Report 15675. Washington, D.C.: World Bank. -. 1997a. "Cbte d'Ivoire Country Assistance Review." Report 17015. Washington, D.C.: World Bank. -. 1997b. "Evaluating Development Operations: Methods forJudging Outcomes and Impacts." Lessons and Practices 10. Washington, D.C.: World Bank. _. 1997c. "Kingdom of Morocco Country Assistance Review." Report 16326. Washington, D.C.: World Bank. 1997d. "Mozambique Country Assistance Review." Report 17209. Washington, D.C.: World Bank. 1997e. "Poland Country Assistance Review." Report 16495. Washington, D.C.: World Bank. 1997f. "A Success and Challenge: AGETIP in Senegal." Operations Evaluation Department Pr6cis 148. Washington, D.C.: World Bank. 1998a. "Albania Country Assistance Review." Report no. 18030. Washington, D.C.: World Bank. 1998b. 1997 Annual Review of Development Effectiveness. Washington, D.C.: World Bank. 1998c. "Annual Report on Portfolio Performance Fiscal Year 1998." Washington, D.C.: World Bank. -. 1998d. "Bangladesh Country Assistance Review." Report 17455. Washington, D.C.: World Bank. 1998e. "Bolivia Country Assistance Review." Report 17957. Washington, D.C.: World Bank. 1998f. "Financial Sector Reform: A Review of World Bank Assistance." Report 17454. Washington, D.C.: World Bank. 1998g. "The Impact of Public Expenditure Reviews." Washington, D.C.: World Bank. 1998h. "NGOs in Bank-supported Projects. An OED Review." Report 18399. Washington, D.C.: World Bank. 1998i. "Philippines Country Assistance Review." Report 17417. Washington, D.C.: World Bank. -. 1998j. Process Review of World Bank Grant Programs. Report 18317. Washington, D.C.: World Bank. 1998k. "The World Bank's Experience with Post-Conflict Reconstruction." Report 17769. Washington, D.C.: World Bank. 62 BIBLIOGRAPHY Picciotto, Robert, and Eduardo Wiesner, eds. 1998. Evaluation and Development. The Institutional Dimension. New Brunswick, N.J.: Transaction Publishers. Pollak, R. 1991. "Welfare Comparisons and Situation Comparisons." Journal of Econometrics, vol. 50, pp. 31-48. Psacharopoulos, George, and others. 1992. "Poverty and Income Distribution in Latin America: The Story of the 1980s." Latin America and the Caribbean Technical Department Report 27. Wash- ington, D.C.: World Bank. Ravallion, Martin. 1994a. "Measuring Social Wellfare With and Without Poverty Lines." American Economic Review, vol. 84 (2), pp. 359-64. Ravallion, Martin. 1994b. "Poverty Comparisons." Fundamentals of Pure and Applied Economics, vol. 56. Switzerland, Chur: Harwood Academic Press Ravallion, Martin. 1996. "Issues in Measuring and Modeling Poverty." The Economic Journal, September, Blackwell publishers. Ravallion, Martin, and Shaohua Chen. 1997. "What Can New Survey Data Tell Us about Recent Changes In Distribution And Poverty?" The World Bank Economic Review 11(2): 357-81. Ravallion, Martin, and Gaurav Datt. 1994. "Growth and Poverty in India." Washington, D.C.: World Bank. Reisen, Helmut. 1998. "Domestic Causes of Currency Crises: Policy Lessons for Crisis Avoidance." Paper presented at the Institute of Development Studies East Asian Crisis Workshop, Sussex Uni- versity, July 13-14, Brighton. Rodrik, Dani. 1996. "Understanding Economic Policy Reform." Journal of Economic Literature 34 (March): 9-41. Rose-Ackerman, Susan. 1998. "Corruption and Development." In Boris Pleskovic and Joseph E. Stiglitz, eds., Annual World Bank Conference on Development Economics 1997. Washington, D.C.: World Bank. Sachs, Jeffrey D., and Andrew Warner. 1995. "Economic Reform and the Process of Global Integra- tion." Brookings Papers on Economic Activity. Washington, D.C.: Brookings Institution. Sala-i-Martin, Xavier. 1997. "I Just Ran Four Million Regressions." NBER Working Paper Series 6252. Cambridge, Mass.: National Bureau for Economic Research. Sen, Amartya. 1976. "Poverty: An Ordinal Approach to Measurement." Econometrica 46: 437-46. Squire, Lyn. 1993. "Fighting Poverty." American Economic Review 83(2): 377-92. UNDP (United Nations Development Programme). 1998. Poverty in Transition? New York: Re- gional Bureau for Europe and the Commonwealth of Independent States. Working Group on International Financial Crises. 1998. Report prepared by representatives of Ar- gentina, Australia, Belgium, Brazil, Canada, France, Germany, Hong Kong SAR, Italy, Japan, Korea, Mexico, the Netherlands, Singapore, South Africa, the United Kingdom, and the United States. Meetings held in Washington, D.C., in April 1998. World Bank. 1990. World Development Report 1990: Poverty. New York: Oxford University Press. 63 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS -. 1996a. "Partnership for Capacity Building in Africa." A report of the Working Party on the Impact of Bank Policies, Instruments, and Operational Practices on Capacity Building in Africa. Washington, D.C. _. 1996b. World Development Report 1996: From Plan to Market. New York: Oxford Univer- sity Press. 1997a. Annual Report 1997. Washington, D.C. 1997b. Helping Countries Combat Corruption: The Role of the World Bank. Washington, D.C. 1997c. "The Strategic Compact: Renewing the Bank's Effectiveness to Fight Poverty." Wash- ington, D.C. 1998a. Assessing Aid. A Policy Research Report. Washington, D.C. 1998b. "Country Assistance Strategies: Retrospective and Outlook." Managing Directors Operations Policy and Strategy Group, Washington, D.C. 1998c. East Asia: The Road to Recovery. Washington, D.C. 1998d. World Development Indicators 1998. Washington, D.C. 1998e. World Development Report 1998/99: Knowledge for Development. New York: Oxford University Press. 1998f. "Supervision Quality in FY98: A QAG Assessment (RSA2)." Quality Assurance Group, Washington, DC 64