WORLD BANK OPERATIONS EVALUATION DEPARTMENT O ED 1997 ANNUAL REVIEW OF Apl1 10R DEVELOPMENT EFFECT IVENESS øva WORLD BANK OPERATIONS EVALUATION DEPARTMENT 1997 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Alison Evans William G. Battaile, Jr. The World Bank Washington, D.C. Copyright @ 1998 The International Bank for Reconstruction and Development/THE WORLD BANK 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. All rights reserved Manufactured in the United States of America First printing April 1998 The opinions expressed in this report do not necessarily represent the views of the World Bank or its member governments. The World Bank does not guarantee the accuracy of the data included in this publication and accepts no responsibility whatsoever for any consequence of their use. The boundaries, colors, denominations, and other information shown on any map in this volume do not imply on the part of the World Bank Group any judgment on the legal status of any territory or the endorsement or acceptance of such boundaries. The material in this publication is copyrighted. Requests for permission to reproduce portions of it should be sent to the Office of the Publisher at the address shown in the copyright notice above. The World Bank encourages dissemination of its work and will normally give permission promptly and, when the reproduction is for noncommercial purposes, without asking a fee. Permission to copy por- tions for classroom use is granted through the Copyright Clearance Center, Inc., Suite 910, 222 Rose- wood Drive, Danvers, Massachusetts 01923, U.S.A. Design: The Magazine Group Cover photo: PhotoDisc ISSN 1019-4363 ISBN 0-8213-4210-X Printed on recycled paper. Contents vii Foreword ix Acknowledgments xi Abbreviations and acronyms 1 Summary 5 Global trends and development assistance 5 Measuring the effectiveness of development assistance 7 Toward a measurement of the World Bank's development effectiveness 9 Evaluation challenges and priorities 15 Country factors and portfolio performance 17 Project-level determinants of success 20 Summary and implications 23 Performance results FY96 and FY97 29 Explaining trends in outcome 34 Risk and reward 36 Summary and implications 39 Country assistance reviews 41 Thematic evaluations 44 Sectoral evaluations 47 Conclusions 51. nmroubg d ep eferaee 53 Measuring aggregate development effectiveness 53 Achieving effectiveness through strategic selectivity 53 Borrower performance 54 Bank performance 54 Governance and combating corruption 54 Development partnerships 55 Toward a framework for results-based management 57 Annex 57 Outcome, sustainability, and institutional development impact of evaluated operations, by sector, network, lending type/source, and region, exit fiscal years 1990-95, 1996, and 1997 (by projects) 58 Outcome, sustainability, and institutional development impact of evaluated operations, by sector, network, lending type/source, and region, exit fiscal years 1990-95, 1996, and 1997 (by disbursements) iii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 59 Appendix 59 Prefacio 61 Resumen 67 Preface 69 R6sum 75 References Figures 2 1: Outcome continues to improve 3 2: Policy and institutional quality influence country portfolio performance 8 1.1: Linking development effectiveness with strategic and network goals 17 2.1: Portfolio performance by policy and institutional environment-adjustment and investment operations 17 2.2: Portfolio performance by policy and institutional environment-investment operations only 24 3.1: Percentage of operations with satisfactory outcomes by exit fiscal year, 1980-97 24 3.2: Percentage of operations with satisfactory outcomes, by region and exit fiscal year, 1980-97 26 3.3: The size of the active portfolio 27 3.4: Percentage of operations with satisfactory outcomes, by sector and exit fiscal year, 1990-95, 1996, and 1997 28 3.5: Outcome performance IDAIJBRD 29 3.6: Institutional development impact by exit fiscal year, 1987-97 30 3.7: Sustainability by exit fiscal year, 1987-97 30 3.8: Demandingness, complexity, and riskiness by exit fiscal year, 1987-97 31 3.9: Bank performance by exit fiscal year, 1987-97 32 3.10: Borrower performance by exit fiscal year, 1987-97 34 3.11: Percentage of operations with satisfactory outcomes by approval fiscal year, 1970-92 35 3.12: Bank performance by approval year 35 3.13: Borrower performance by approval year 36 3.14: Risk and reward by country 36 3.15: Risk and reward by sector 52 5.1: Comparative static analysis of improvements in process variables iv Con tents Boxes 9 1.1: Measuring the effectiveness of World Bank assistance 10 1.2: The sector strategy: monitoring strategic and sector goals 11 1.3: A results-based approach to evaluating development effectiveness 12 1.4: Project side effects-the case of the Republic of Korea's Cholla region 13 1.5: Evaluating project performance 18 2.1: Contrasting portfolio performance in Ghana and Zambia 19 2.2: Borrower performance is a critical determinant of project success 19 2.3: Quality at entry continues to make a difference 26 3.1: Composition of FY96-97 evaluated exits and the active portfolio 27 3.2: Turnaround in the Brazilian portfolio 28 3.3: IBRD and IDA performance trends 28 3.4: When is a project judged to have substantial institutional development impact? 33 3.5: Commitment and ownership underpin highly satisfactory borrower performance 34 3.6: Performance trends by approval year 37 3.7: Adjusting the portfolio for reward variance 41 4.1: Poland's agriculture task force-a strategic partnership for program design 42 4.2: Progress in reducing poverty 43 4.3: Brazil-best practice in five urban projects 44 4.4: The equity and school improvement project, Guinea 45 4.5: Placing the environment at the center of the policy dialogue 46 4.6: Getting results in education 48 4.7: Lessons from financial crises-two country cases 52 5.1: Sensitivity tests of portfolio performance 53 5.2: Putting selectivity into practice: the Poland country assistance strategy Tables 6 1.1: Empirical findings from selected works on aid effectiveness 8 1.2: Key elements in a model of aggregate development effectiveness 16 2.1: Country classification by policy performance and institutional quality, 1980-95 37 3.1: Examples of outstanding and poor operations, FY96-97 evaluated exits V FOREWORD T he Operations Evaluation Department's loans and credits exiting the portfolio in FY97. The largest twenty-third Annual Review marks a depar- factor in this increase is a marked improvement in bor- ture from past such reports. This year's publi- rower performance. The proportion of projects with satis- cation-the first since the approval by the factory borrower performance increased to 71 percent in World Bank's Board of a new Strategic Com- FY96 and 74 percent for the FY97 sample. The World pact for the institution-seeks to provide a more compre- Bank's clients are clearly doing a better job of preparing hensive assessment of the Bank's development effectiveness, projects and complying with covenants. While borrower as required by the renewal agenda. In addition to analyzing performance has been improving, Bank performance has 325 projects completed in FY96 and FY97 and assessing been unchanged. However, because project demandingness the results against long-term performance trends, the report and complexity have increased, this means the Bank has offers a strategic view of the factors contributing to devel- been holding its own in a difficult, changing environment opment results "on the ground." Among other things, it and the Bank's nonlending services may be credited with stresses the importance of moving toward a results-oriented some of the improvements in borrower performance. system to improve institutional learning and accountability. Although outcome ratings have improved, this has not The past decade has seen the focus of development always meant durable results on the ground. Institutional shift away from physical investments toward helping coun- development impact and sustainability are vital to develop- tries achieve socially and environmentally sustainable ment effectiveness. On these counts the record is mixed. growth. This new development consensus has precipitated While institutional development has improved slightly, the a reconsideration of ways in which the World Bank works sustainability of projects remains an area of concern given and assesses its development impact. The Strategic Com- the volatility of the implementation environment. About pact seeks to adapt the institution to new client demands, half of Bank operations are of uncertain or unlikely sus- take advantage of global opportunities, and improve the tainability and improvement remains slow. Bank's effectiveness in achieving its overarching goal- Improving the long-term effectiveness of World Bank poverty reduction. Assessing and improving development assistance will require continued support for borrower effectiveness has therefore become a key challenge for capabilities in project selection, design, and management. development evaluation. It will also require greater selectivity both in channeling The results of this year's evaluation of development lending to borrowing countries with suitable policy and effectiveness are encouraging. Project outcome and bor- governance climates and in matching Bank services and rower performance have improved. The proportion of pro- instruments to borrower capabilities. Selectivity is not jects with satisfactory outcomes increased to 71 percent in about "picking winners," but it does mean starting with a FY96 and to 76 percent for a sample comprising half of the country commitment to development and good governance vii ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS and modulating the Bank's response through appropriate that suits the new vision of the Bank as a full-service insti- and timely financial and nonfinancial assistance. It also tution. At the eve of a new millenium, reorienting opera- means building better partnerships to leverage the World tions toward more relevant goals, reforming business Bank's impact and using more participatory and tailored processes toward cost-effective results, and reshaping the approaches to build project ownership and bring about organization to work through partnerships are the funda- better results on the ground. Equally, sound development mental development effectiveness challenges facing the risk management implies better performance measurement World Bank. and monitoring. In sum, considerable progress has been made toward achieving the goals of the World Bank's ongoing portfolio improvement program but much remains to be done. The Robert Picciotto way forward for the World Bank under the Strategic Com- Director General pact will require a results-based management system, one Operations Evaluation viii ACKNOWLEDGMENTS This review was prepared by Operations Evalu- ous OED staff members. Ruchira Corcoran provided ation Department staff. The core team con- research assistance. William Hurlbut was the editor. sisted of Alison Evans (Task Manager), Bill Thanks to Benjamin Crow and Annisa Cline-Thomas for Battaile, Linda Morra, Diana Qualls, and their support. The OED publication series is produced in Bing Deng. The review also benefited from the OEDPK by Elizabeth Campbell-Pag6 and Maria Cecilia input of Yves Albouy, Andrew Bernard (Yale), and numer- V61ez. The design was by The Magazine Group. Director General, Operations Evaluation Robert Picciotto Director, Operations Evaluation Department Elizabeth McAllister Acting Manager, Corporate Evaluations and Methods Andres Liebenthal Task Manager Alison Evans ix ABBREVIATIONS AND ACRONYMS AFR Africa Region HNP Health, Nutrition, and Population AGETIP Agence Gen6rale de Travaux d'Intert Public IBRD International Bank for Reconstruction and ARPP Annual Report on Portfolio Performance Development CAR Country Assistance Review ICR Implementation Completion Report CAS Country Assistance Strategy ID Institutional development CODE Committee on Development Effectiveness IDA International Development Association DAC Development Assistance Committee IMF International Monetary Fund EAP East Asia and Pacific Region LAC Latin America and Caribbean Region ECA Europe and Central Asia Region MNA Middle East and North Africa Region ERP Economic Recovery Program ODA Official development assistance ESW Economic and sector work OED Operations Evaluation Department EU European Union OECD Organization for Economic Co-operation and FIL Financial Intermediary Loan Development FSAL Financial Sector Adjustment Loan QAG Quality Assurance Group FSRL Financial Sector Recovery Loan RBM Results-based management FY Fiscal year SAL Structural Adjustment Loan GDP Gross domestic product SAS South Asia Region xi SUMMARY he challenges of development today are dimensions of evaluation results and analyzes the determi- unprecedented. The World Bank, along with nants of project performance at the level of country portfo- other institutions, is seeking ways to respond lios and individual projects. To maintain continuity in simultaneously to the demands of fast-growing institutional learning, the review also analyzes the results economies and of those at risk of marginaliza- of project evaluation performed by OED during the pre- tion and economic stagnation. In the evolving development ceding two fiscal years and assesses them against longer- agenda, greater emphasis has been given to social and envi- term performance trends. ronmental sustainability, local capacity building, gender equity, and participation. The challenges have become so Performance trends large and complex that partnerships are increasingly vital. Global trends and changes in the development agenda have As this demanding agenda unfolds, evaluating and radically altered the implementation environment for improving the World Bank's development effectiveness is Bank-financed operations. In the early 1980s, the demands also becoming more complex. Evaluating the Bank's effec- of this new environment began to take a heavy toll on tiveness can no longer focus only on the project; it must quality at entry. Bank performance for completed opera- now also measure the impact of the full range of Bank tions began to deteriorate. The percentage of evaluated activities at the sector and country levels. This calls for new Bank operations with satisfactory evaluation instruments and modalities to capture the com- outcomes plummeted from 80 per- bined effects of lending and nonlending services, and the cent at the beginning of the 1980s Portfolio contribution of partnerships to project and program to 60 percent by the end. By that impact. The Operations Evaluation Department (OED) has time, Bank management had rec- p launched a renewal effort to implement this new evalua- ognized the need to strengthen tion framework, but the renewal has just begun and the portfolio management, paying slowing signs of results will take time to emerge. particular attention to quality at This Annual Review of Development Effectiveness entry, to involving the borrower in i (ARDE) is a first step toward a more comprehensive project preparation and implemen- assessment of the Bank's effectiveness. It lays out issues in tation, and to development results. defining and measuring the Bank's development effective- This called for deep changes in attitudes, skills, and man- ness in a changing global environment. It underscores the agement incentives. It is still too early to judge the effects importance of moving toward a more results-oriented sys- of these changes, but the latest performance results affirm tem for improved institutional learning and accountability that Bank operations are beginning to show improvement in meeting strategic goals. It examines some of the strategic (Figure 1). 1 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 15li-' OUTCOME CONTINUES TO IMPROVE according to the latest Annual Report on Portfolio Perfor- mance (ARPP), continues to have the highest proportion of projects at risk. Performance in the water supply and sani- 90 tation sector has declined dramatically, with the percentage of satisfactory projects falling from around 80 percent in FY93 to 40 percent in FY96. Here again, the latest ARPP 80 identifies water supply and sanitation as the sector most ''at risk" in the current portfolio. Factors behind the improvement Analysis reveals at least two potential reasons for the up- 6o turn in outcomes. First, borrower performance-the assumption of ownership rights and responsibilities-has im- 50 proved, as has borrower delivery // of the inputs necessary to prepare and implement a Bank-financed operation. Latest results show the proportion of projects with satis- BY DISBURSEMENTS factory borrower performance has increased from an average of 65 percent in FY90-95, to 71 percent in FY96, and 74 percent for the 80 FY97 sample. Second, improve- ments in overall economic condi- tions, reflected in the number of 70 evaluated projects in FY96 imple- mented in better-performing countries, have improved the environment for Bank-financed operations.1 60 Econometric analysis of a sample of more than 900 completed projects2 finds that borrower performance is most critical to determining project outcome. Good bor- rower performance raises the probability of a satisfactory outcome by 35 to 40 percentage points. Consequently, 81 83 85 87 89 91 93 95 *97 improved borrower performance is likely to be the most important factor behind improvements in overall project performance. Source: OED data Country economic conditions are also important and, Note: Broken lines (exit FY97) indicate preliminary results, with together with the quality of Bank performance at project less than 50 percent coverage of exited operations. entry and during implementation, account for a 15 to 20 percentage point increase in the likelihood of a satisfactory The proportion of projects with satisfactory outcomes, project outcome. While there is evidence that country eco- as evaluated by OED at the fiscal year of exit, increased nomic conditions have improved, which the ARPP rein- from an average 67 percent for FY90-95 to 71 percent in forces for the active portfolio, there is still only modest evi- FY96 and, based on a partial sample, to 76 percent dence that Bank performance has improved. Bank in FY97. performance was satisfactory for 74 percent of evaluated Improvements in performance are not shared equally operations in FY96 and 77 percent for the partial FY97 across all regions and sectors, however. Sub-Saharan Africa sample. But when compared to the trend for FY90-95 the continues to perform below the Bankwide average and, percentage of projects with satisfactory Bank performance 2 Summary has stayed reasonably constant in the 70 to 75 percent The evidence on sustainability shows a modest rising range. These results show there is still some way to go before trend. The latest results show that the proportion of pro- efforts to improve quality at entry and portfolio manage- jects rated as having likely sustainability has inched ment yield significant rewards in Bank performance. upward from 46 percent in FY90-95 to 48 percent in A recent OED study of the appraisal process (OED FY96. The partial results for FY97 show a further modest 1997a) also presents signs of slow improvements in Bank increase to 51 percent, but the overall result is still sober- performance. The study confirms the important relation- ing: in just under half of Bank operations long-term sus- ship between quality at entry and project performance, but tainability is uncertain or unlikely. highlights continued weaknesses in the quality of social analysis done at appraisal, in borrower and beneficiary Improving outcome and effectiveness participation, and in institutional analysis. Furthermore, Good projects help to improve the implementation capacity the study argues, without a clear consensus on ways to of borrowing countries. Conversely, good projects work enhance quality at entry-such as simpler project design, better in good enabling environments. Analysis of the deter- greater attention to risk, and greater attention to learning minants of project success shows that both government through systematic monitoring and evaluation-predicting credibility and freedom from corruption are positively the source of future improvements is impossible. Evidence related to project outcome. Analysis at the country level is increasing that Bank nonlending services are crucial to shows that policies and capable institutions matter both for improving not only the quality of the appraisal process but economic results and for portfolio performance (Figure 2). also the quality of economic management and borrower The significance of policy and governance factors at performance. the project and country level reinforces the research finding that development assistance has more impact on economic Institutional development impact and sustainability growth when delivered in a sound policy environment Improved outcome ratings alone do not necessarily mean (Burnside and Dollar 1997). It also underscores the impor- better results on the ground. To assess the longer-term tance of greater selectivity in the Bank's assistance. Selec- development effectiveness of a tivity means channeling lending to countries with appro- I project, both its institutional priate policy environments and using nonlending services nstitunional development impact and its likely more strategically to support the emergence of sound poli- sustainability need to be exam- cies and good governance. Greater selectivity also means ined. Through their impact on institutional development, projects . F~it 2 POLICY AND INSTITUTIONAL QUALITY improve the implementation Impact aic improve th iplemention INFLUENCE COUNTRY PORTFOLIO PERFORMANCE capacity of borrowing countries, susrrj.iMblity are while sustainability is the ability to maintain project benefits in the more exactsig longer run. Looking at the latest results, mieasures of the record on effectiveness is mixed. The proportion of projects 100 rated as having high or substantial 80 institutional development impact 60 effectiveness increased from 30 percent in 40 FY90-95 to 39 percent in FY96 20 and FY97. While the number of . projects rated as having modest institutional development F-... impact has stayed nearly constant, the proportion having High Low performance negligible impact declined from 25 percent to 15 percent Institutional between FY95 and FY96. These results are encouraging, quality but some 60 percent of Bank operations still have only modest or less institutional development impact. Source: OED data 3 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS better selection and sequencing of the tools and instru- would mean that overall outcomes could reach 80 percent ments of Bank assistance to match the quality of local or higher. With the addition of prudent improvements in institutions and borrower capabilities. Building better borrower performance, a level of development synergy between 85 percent satisfactory projects e advisory, analytical, and lending may be possible. Supporin,r':L Co services is crucial. Meaningful retailing of such A review of lessons from recent a target by sector and by region boI:N and institutional Country Assistance Reviews con- appears to be both necessary and firms the importance of policy for desirable. With such a strategic putting in ,Dace conditions the overall impact of Bank assis- plan, the Bank would restore the ! tance. It also says again that policy health of its portfolio to the level Sound olicies significantly reforms rarely succeed where bor- it enjoyed in the late 1970s. Of rower commitment and ownership course, such a plan would require and instiuons is influence are inadequate. Analysis of factors intensified efforts by all Regions behind improvements in current and Networks to improve quality the key to reater- portfolio quality further reinforces at entry, supervision, risk man- this message: the number of pro- agement, and monitoring and effecLiveess jects in countries with weak macro- evaluation. economic management has dropped sharply.3 This under- In addition to sustained efforts scores the importance of supporting the domestic capacity of to improve the quality and management of the Bank's port- the borrower and ensuring effective stakeholder arrange- folio, improving the long-term effectiveness of Bank assis- ments are in place to build the necessary consensus and own- tance requires continued support for borrower capabilities ership for sector and country-level reform efforts. in project selection, design, and management. It also requires greater selectivity both in channeling lending to Prospects borrowing countries with suitable policy and governance Progress undoubtedly has been made, but substantial scope climates and in matching Bank services and instruments to for "doing things right" still remains. Furthermore, the borrower capabilities. Selectivity is not about "picking win- long-term significance of the improvements measured in ners," but it does mean starting with a country commitment FY97 has yet to be determined. to development and good governance and modulating the Analysis shows that sustained quality management Bank's response through appropriate and timely financial efforts could keep the Bank on course to meet manage- and nonfinancial assistance. A selective strategy will also ment's goal of 75 percent satisfactory projects by the year mean building better partnerships to leverage the Bank's 2000. Targets for portfolio improvement over the coming impact and using more participatory and tailored year, which are set out in the ARPP, include raising the pro- approaches to build project ownership and bring about bet- portion of operations with satisfactory supervision to 75 ter results on the ground. percent or higher and raising the proportion of new entrants with satisfactory quality at entry to 85 percent or higher. Applying these targets in a sensitivity analysis of Hoies more than 300 of the latest evaluated operations 1. Country economic conditions began to improve in many (FY95-97) shows that if both are achieved simultaneously, developing countries toward the end of the 1980s, especially for project outcome could improve to 75 percent satisfactory. those countries adjusting most successfully. Most of the operations If borrower performance continues to improve along its evaluated at exit in FY96 were approved in 1989. latest trajectory, it is possible to envision an improvement 2. Projects that exited the portfolio between FY93 and FY97. 80 percent.4 3. From almost 800 projects in FY95 to 465 in FY97. Sensitivity in vaggproectoue approach in d banalysis of trends in projects at risk suggests that roughly half the Aolf n, si proath, chamiton by prcesient s improvements are due to country factors (OED 1997c). Wolfensohn, is to improve the quality of Bank processes so 4. These achievements are based on the number of projects, not that they achieve 100 percent satisfactory results. Although the volume of disbursements. With the model underlying the sensitiv- this target is not likely to be hit by the year 2000, doing so ity tests it was not possible to produce forecasts by volume. 4 THE CHALLENGE OF DEVELOPMENT EFFECTIVENESS cflfbal rr nee mn sitance put in place clear objectives at all necessary policy and he challenges of development are increasingly operational levels to improve project quality, and to shift bound up with the promise and pitfalls of the focus away from the quantity of lending to the quality globalization. Many developing countries and development impact of lending. have become more integrated into the world The World Bank's Strategic Compact seeks to enhance economy, and private capital flows have over- institutional capabilities in light of new client demands and taken official development assistance as a major source of global opportunities and to im- finance for the developing world.1 But the pace of eco- prove the Bank's effectiveness in nomic integration is very uneven. During the early 1990s, achieving its main mission of pov- The eight countries accounted for almost two-thirds of private erty reduction. Assessing and capital flows to developing countries, and more than half improving development effective- . of developing countries received little or none. ness present a very important chal- implementaion The past decade has also witnessed a shift in develop- lenge for development evaluation. . ment focus from financing physical investments to also helping countries achieve the conditions necessary for Measurng the effectoveess of . socially and environmentally sustainable growth. Poverty development a Bank-fmnancedi reduction remains central, but development goals now also Many studies have attempted to include institutional reform, capacity building, and partici- assess the effectiveness of develop- operations has pation (OECD 1996). As the development agenda has ment assistance in terms of its become more become more complex and more closely tied to global impact on countries' economic trends, stakeholders in multilateral and bilateral develop- and social progress.3 The method- ment institutions have increased the pressure to improve ology and empirical findings of complex the effectiveness of development assistance.2 A report pre- some of the most recent studies pared by the Development Assistance Committee (DAC) of are cited in Table 1.1. Most studies use per capita eco- the Organization for Economic Cooperation and Develop- nomic growth as the main indicator of effectiveness on the ment (OECD 1996) stresses the urgency of enhancing the assumption that changes in GDP are highly correlated global impact of development assistance by working with other changes in the standard of living in developing toward integrated goals in economic, social, and environ- countries. Few studies find a strong positive association mental development. The report by the Task Force on Mul- between aid and growth; if anything, the results are tilateral Banks challenges the World Bank and the regional weakly positive but not significant. But some new work banks to be "guardians of development effectiveness," to points to more positive associations. 5 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS EMPIRICAL FINDINGS FROM SELECTED WORKS ON AID EFFECTIVENESS BURNSIDE AND PAUL MOSLEY DOLLAR PETER BOONE FEYZIOGLU, ET AL. (1995) (1997) (1994) (1996) SAMPLE P1RI.RIt lii..-4I I- . I Countries Ct erLd IC, InAl..r 1..k. i!J s - pm: c..ururIc - Jtcioprne *LInrIc 1- de.l.p'- c.... ntrL feelpk-cnE iII .ti11c in 1 Afit,i:1 I .\tit.1r 1 i . . Method CrLS ifn- ct:u Striii 01. SL ,J p.... . OLS vpod1: - id L c p L....i p n d:i pit d 11- (Irie : t ire~ -I;Ec Iili i ; I.1rpEIC Aid Variabic Neti.iliij.Il OD_X, i.cr- Crr .m n: Ic i,h OD%ifi -ih s ti 25 N.:lIfL i-ir~:m~~in Ui .. r - --ii %:otnpi-ncricI.iiIII. . .t (."NI'' AieetsuybyteWrdBnk (Burnsidead Dolla inrasff'. -I,pretg pitUn: oo porli environ- Gtipoice proot log-er motlt whre pli ncie r poor.CII.I.i n jtc ['Ad p~ ir%: ,Irrn 1..rn.c-iriAE,,[,IJ>J)'I.c - i~ u~sr sependent Xarljhlc ound lega.n relo Iis, I ta gret er s1lc i t in o Arentstudybythenforldeal purpert righ, Dla increaseof0.4pe pits i n od pvli ni on- examines onethypltnkagestbetwhenmeal development e T sy a fn a ptive ruth for tinant mn- asstiutn ofODA and likelyth.b taityn turDA heop toedpce ifadevelometalt asisncneso Braest on schntringtatr with souoetand policiebig sialaand enronmecta onsinn ptampolicesromte lonrm m lity w e oliio ae poo a n a I rg o A rcen stdy y tenforcle pakBropert rights,ad cudsbata increase o0.pecnaeits ipcn develinicon- exainesuth that h thesi hat the measure o development outcome rgeti ness. Wth cr iev alate r- ODA on growth in a good policy thermore, a significant portion of development assistance environment is indeed very large: 1 percent of gross domes- goes toward achieving intermediate goals necessary for sus- tic product (GDP) in assistance leads to a growth rate tainable growth, including building capacity for the formu- 6 The challenge of development effectiveness (CONTINUED) [iRNsiLiE AN' PAUL NOsLE1 LO[ I. AR PErR BOOLE tE) ZIO.LIU. ET AL SAMPLE PERIOD 1'r,u-'l 1 "11-" I ' I -cii I - Dependent Variables J i:in. n' (continues) i rr -I hI jr 'c piII ll Major Conclusions \,J bh . i0c.k :k . As. .\IJhI h .-i ,i iripA A hd:s a . ti'L I'ui $I wn I Ic.eC . imnp.ict ouii L:i,_- rhi', L rco [[ h r .'d iiib ZiiIc3fl[ iniA.CE I-0 cII 'l 't. I[ ;iFI tlfr , b; r- . r,e fa cr i l r t: - i-J Decee ad .9ic e E~. .u cht: Albl - - 1i-pr -Iac U IdL1 III,t '11- 1. 111§s' ''Li,rI MPI rirritc I. N11rg iprnai pro ipen si rIo 'in Iailnt c i policy Ad im ct te C t rn r Itch. DAId l. ls a te It unexamred.6 efinig deveopmenteffecins Hisjt' only' thed .fi- tp however Th elchleg le nmesrn inproc Gie lblted n tesbodrdvlpetaed e developmnt outcmes and iat at t!Chei conr leel.* andr Compacr' he W'Tr rri s nd .I It' - I1il. cr, CAid1K A'I d C. ,inJ~~~ ~ ~i rrri,Irhll-lI GNP gross national product; OLS = ordinary least squares; 2SLS two-stage least squares MP t marginal propensity to consume; MPI marginal propensity to invest. lation and management of economic policy. The impact the Compact's recognition of the DAC goals7 and the inter- ODA has on these critical intermediate factors is still largely mediate goals of the Bank's newly formed Networks.8 unexamined.6 Defining development effectiveness is only the first step, however. The real challenge lies in measuring it. An approach -. - OTilattempted for this review is to identify a common metric for estimating the aggregate contribution of Bank assistance to Given global trends and the broader development agenda key development outcomes and impacts at the country level. reflected in the Strategic Compact, the World Bank's devel- The model presented in Table 1.2 offers such an approach. It opment effectiveness can be defined as a demonstrable con- measures Bank and partner inputs in dollar terms and esti- tribution to economically sound, socially equitable, and mates the association between these inputs and a range of environmentally sustainable growth. This definition goes intermediate and final development outcomes taking into beyond a narrow assessment of GDP growth to include account the quality of a country's policy environment, eco- both the equity and sustainability dimensions of develop- nomic governance, and the availability of matching funds.9 ment. Furthermore, as shown in Figure 1.1, it directly This conceptually simple approach has the potential of aligns with the Bank's strategic goals. It also underscores adding up the contribution of Bank services across a wide 7 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 1.1: LINKING DEVELOPMENT EFFECTIVENESS WITH STRATEGIC AND NETWORK GOALS AND SUSTAINABLE A A A V V V Popilaiion. healrh. Socil l Je%iloplent PritIte 'ector nutrition J;%,lnpmfnn En% 'ronmeni Fducation Intrastruciurc Rural d%clopnicnr Finance Social prorection IE5i' Encrg%!N/Mninq ,HDD IFPSb' spectrum of sectors and countries to provide an overall These methodological issues are being tackled by the assessment of effectiveness at the global level. The estima- Bank as part of developing a system for measuring tion requirements of such a model are formidable, how- progress and performance of the Strategic Compact.10 One ever, and there are a number of measurement gaps, particu- important lesson is that any workable measure of develop- larly how best to model the complex linkages between ment effectiveness needs to focus on the contributions Bank inputs (including administrative expenditures), less attributable to Bank activities-and through which Bank tangible factors such as institutional capacity and gover- assistance can ultimately be expected to leverage change in nance arrangements, and indicators of development development outcomes. For example, significant Bank impact. Some of the empirical findings of the quantitative assistance goes toward the achievement of intermediate exercise conducted for this review are provided in Box 1.1. goals necessary for sustainable development, including sec- TABLE 1.2: KEY ELEMENTS IN A MODEL OF AGGREGATE DEVELOPMENT EFFECTIVENESS I lNK .AND I'XTNER INPiTS BORROXIER INPI TS INTERNIEDI 1E REsl Ts DE1FLOPNIENT 1111-1CT i,,, I n dic i ilitIc i h'i Po'Ic% collImiLinnIII ICD~ l 'l tc' rin & P p,: capiE r c [h qi~l 'icapit3l d.i I iirrcr inpias tI 0_111in crpirr rund4 111uir L i l ip 'r 1'311L 11FLinder-i im.riala\ a. Inputs here are services delivered, adjusted for quality. This is a narrow interpretation of the Bank's assistance. The Strategic Compact also emphasizes country selectivity and quality at entry as drivers of development effectiveness. 8 The challenge of development effectiveness BOX 1.1: MEASURING THE EFFECTIVENESS OF WORLD BANK ASSISTANCE quantitative study black market premiums, GDP growth. When the aid opment outcome: female to measure the inflation, trade openness, variables were included, enrollment in education, Bank's overall and fiscal balance as a per- the overall explanatory changes in life expectancy, development effectiveness centage of GDP. Lastly, four power of the model in- and infant mortality. With was designed using the con- aid variables were specified: creased, but the aid vari- life expectancy as the ceptual framework shown International Development ables were negative and dependent variable, the in Table 1.2. Building on Association, IDA plus Inter- insignificant. An OLS spec- OLS specification consis- the work of Burnside and national Bank for Recon- ification with the policy tently showed the bureau- Dollar (1997), the study struction and Development, index and aid variables cracy measure to have a sought to estimate the rela- non-Bank official develop- interacted showed a signifi- significantly positive effect, tionship between Bank con- ment assistance, and non- cant and positive relation- while the coefficient on cessional and market-based Bank total development ship between the interacted policy was not only nega- lending and changes in a assistance. The study began terms and the growth rate tive but insignificant. All range of key development by estimating a simple of per capita GDP, but the the aid variables were outcomes consistent with growth model based on policy index lost robust- negative and insignificant the goals set out in the data for 49 countries across ness. Adding the bureau- except when controlling Strategic Compact: gross the period 1970-1994. cracy measure to the for country-specific effects, domestic product (GDP) Using standard ordi- growth equation reduced when these became posi- growth, under-five mortal- nary least squares (OLS) the number of observations tive but still insignificant. ity rates, female educational techniques, the base specifi- and did not improve the The regressions with the enrollment, and life expec- cation, without the aid formulation. Lagging the change in female enroll- tancy at birth. A policy variables, conformed to aid variables by one period ment and infant mortality index was constructed standard growth results. did not improve results. as the dependent variables specifically for this study The policy index proved The next step was to rapidly deteriorate in and was based on a four- highly robust and was posi- examine the association quality with no specifica- variable principle compo- tively and significantly between aid on three alter- tion yielding a meaningful nents analysis, covering associated with per capita native measures of devel- relationship. toral policy reform and building capacity for the formula- torable results from the project level to the sector and coun- tion and management of economic policy. Many of these try level and ultimately to the strategic level (Box 1.3). contributions are best captured below the country level While results differ in kind across different activities, when and imply that a sector- or subsector-oriented approach defined and measured appropriately they become a com- could yield important insights for assessing the Bank's mon yardstick for assessing progress and accountability effectiveness." This is the approach on which the Net- throughout the institution and for maintaining focus on the works are basing the preparation of sector strategies to primary measure of success-development impact-in an increase the impact of Bank assistance. They are also using increasingly demanding global environment. A results-ori- it to develop monitorable indicators for tracking progress ented framework for performance measurement is compati- toward sector goals (Box 1.2).12 ble with the goals of the Strategic Compact.13 It is also an A sector approach is more feasible, but it is still hin- approach increasingly adopted by public sectors around the dered by the absence of a common yardstick for assessing world to manage performance and improve effectiveness. performance across a wide range of Bank activities and an increasingly wide range of development partnerships. An Evaluation challenges and priorities alternative is to develop a framework of performance mea- The need to generate a range of evaluative data that facili- surement and evaluation that systematically links moni- tates the systematic assessment of performance at all levels 9 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS THE SECTOR STRATEGY: MONITORING STRATEGIC AND SECTOR GOALS he Bank's first sec- the development goals set strategic sector priorities. evaluation of sector and tor strategy to link out in the Strategic Com- In addition, Regional HNP thematic objectives within strategic and sector pact. They also underscore sector strategies are being the Bank. It is an important priorities was prepared for the World Bank's recogni- prepared to reflect region- step, but there is still some Health, Nutrition, and tion of the Development specific challenges for way to go to ensure that Population (HNP) in Assistance Committee improving sector perfor- all Networks adopt this 1997.a The strategy's three health goalsb as a key mance and achieving approach, and that at the main priorities: improving objective for development greater impact. heart of each sector strat- health outcomes of the assistance in the sector. The sector strategy egy is a common yard- poor; improving the per- The HNP Sector Board is approach is the first build- stick-results-for assess- formance of health care in the process of identify- ing block in the develop- ing progress toward systems; and securing ing specific indicators of ment of a more systematic strategic objectives and sustainable health care sector progress corre- framework for perfor- evaluating the Bank's financing link directly to sponding to these three mance monitoring and development effectiveness. a. Health, Nutrition and Population Sector Strategy (1997), The Human Development Network, World Bank, Washington, D.C. b. The DAC report "Shaping the 21st Century: The Contribution of Development Cooperation" sets out a series of targets for achieving improvements in development outcomes by the year 2015. In HNP the goals are: the death rate for infants and children under the age of 5 years should be reduced by two-thirds of the 1990 level by 2015; the rate of maternal mortality should be reduced by three-fourths during the same period; access to reproductive health services for all individuals of appropriate ages, including safe and reliable family planning methods, should be made available as soon as possible and no later than 2015. within the Bank is the starting point for the Operations partnerships and strategic alliances in delivering develop- Evaluation Department's (OED) new evaluation strategy. ment assistance. The strategy aims to reshape inde- The need to move to a higher evaluative plane is rein- pendent evaluation in the Bank to forced by concerns over the fungibility of Bank assistance. make it a more relevant, timely Lending destined for particular factor in the pursuit of develop- projects or programs may, it is ment effectiveness and to enhance argued, release the incremental the learning and accountability of resources needed to implement the Strategic Compact. other schemes of much lower pri- At the heart of the strategy is ority. As a result, even if Bank- the changing nature of develop- financed projects have excellent ment assistance and the Bank's outcomes, it does not follow that shift toward being a full-service they were well selected unless the development institution. It seeks to choice of projects in the public place greater emphasis on country, expenditure program was demon- sector, and thematic evaluations strably and positively affected by that reflect the operational priori- World Bank involvement. Given ties of the Regions and Networks this concern, the only acceptable and the goals of the Strategic level at which Bank assistance can Compact. In addition, the new be assessed is the level of the public sector's expenditure evaluation framework puts more program. emphasis on assessing the impact Notwithstanding the importance of evaluating Bank of nonlending services, both separately and in conjunction assistance in terms of its impact on or relevance to the with lending services and the performance of the Bank's overall expenditure program of a country, there are limits 10 The challenge of development effectiveness B L Z A RESULTS-BASED APPROACH TO EVALUATING DEVELOPMENT EFFECTIVENESS 7 any western should be stated in terms of placing responsibility and which the borrower and governments development achievements, authority for making deci- intermediate executing - and several For example, the U.S. sions with those best posi- agencies participate in goal bilateral development agen- Agency for International tioned to make them. This determination. Milestones cies have adopted results- Development's Strategic involves better matching or indicators are agreed based management (RBM). Framework links project of authority and responsi- upon to track progress The objective is to provide and program results with bility and strengthening of toward expected results. a coherent framework for the agency's five over- accountability mechanisms Indicators act as tools or strategic planning and arching goals: economic through performance signals for managers and management based on growth, democracy and reporting and greater can be used as an early learning and accountability governance, population emphasis on monitoring warning when things are in a decentralized environ- and health, environment, and evaluation of results. not going right. ment. It is first a manage- and humanitarian Under RBM accountability The development agen- ment system and second a assistance. becomes positive, based on cies that have adopted performance reporting sys- a notion of continuous and results-based management tem, and it aims to provide b. Internal and external systematic feedback and (United Kingdom, United the public sector equivalent reporting shifts from an learning. Results-based States, Sweden, Canada) of a bottom line-results. emphasis on inputs to management is also risk did so in part because it The key features of an outputs and outcome tolerant because it pro- better serves the new and RBM system are: reporting. The difference motes learning from complex development between these measures is mistakes. agenda. It also responds to a. Specification of key the key to understanding demands from the public outcome-related strategic management in a results- d. RBM encourages flexi- in these countries for goals. Programs, activities, oriented environment. bility in project definition greater transparency and and processes and resources and implementation and is accountability for results in should be aligned with c. Decentralization and del- based on new forms of the allocation of develop- strategic goals. The results egation underpin RBM, joint accountability in ment assistance. to fungibility. This is particularly true where the financial 1.4). Clearly projects have unique potential. Equally assistance accompanying a project or program has a lim- clearly, project performance measurement is a vital build- ited concessional character or is associated with policy ing block for future measures of the Bank's development conditionality. Furthermore, even effectiveness. in the presence of fungibility, there From this perspective World Bank projects must be There is more is more to Bank assistance than seen as elements of a larger strategy supporting develop- financial resources. Project lending ment goals of a sector or beyond. The scope of project per- to Bank generates influences that ramify formance evaluation must therefore shift beyond narrow beyond the immediate impact of assessments of rates of return to include broader, qualita- assistance than funds or physical inputs. These tive assessments of relevance, sustainability, and contribu- ramifications or side effects- tion to social, environmental, and institutional develop- [ i rincluding policy changes, insti- ment. The three major results-oriented ratings used in inania7 resouceI tutional reform, and capacity performance measurement by the Bank-outcome, sustain- building-are often critical in ability, and institutional development impact-reflect the determining the longer-term impact and sustainability of changes in Bank assistance (Box 1.5). This approach pro- development projects and are in many senses the true mea- vides an assessment of development effectiveness based on sure of the World Bank's contribution to development (Box three key questions: Is the Bank doing the right things 11 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS IiE 1.,,: PROJECT SIDE EFFECTS-THE CASE OF THE REPUBLIC OF KOREA'S CHOLLA REGION uring the 1960s, as ities. The projects covered opportunity for local offi- by the growth of satellite the Republic of seven cities and five islands cials to learn and manage industries with backward Korea's economy with a total of 22 compo- the development process, and forward linkages to par- expanded at a rate of nearly nents. These included indus- initially with the help of the ent industries. Institutional 10 percent a year, regional trial, housing, and tourism central government but later development and capacity income disparities increased estates; city markets; bridges by themselves with strong building were greatly aided between Cholla and the connecting islands to the private sector participation. by the expansion of local country's two most industri- mainland; fisheries; and Industrial growth was stim- government responsibilities, alized regions, Seoul and urban services. Besides their ulated by the replication of particularly changing rules Kyungsang. Between 1975 direct impact, these projects industrial estates, which to provide more incentives and 1984 the World Bank had two crucial "side ef- allowed local authorities to for industrialists to invest approved three projects to fects": they triggered indus- learn how to plan, finance, and developing strong part- help develop Cholla and trialization in the Cholla build, and operate large- nerships with the local pri- reduce interregional inequal- region and created the scale industrial estates and vate sector. (drawing on assessments of relevance and institutional assistance. Factors that determine success at the project development impact)? Is the Bank doing things right level are likely to be associated in important ways with (drawing on assessments of efficacy and efficiency)? And portfolio performance at higher levels. The following finally, will the benefits of Bank assistance be sustained? chapters consequently take a building-block approach to The current system of performance measurement nev- assessing the Bank's effectiveness, beginning in Chapter 2 ertheless has its limitations. The existing methodology with factors determining project success, followed in assesses performance against stated project objectives, Chapter 3 by recent project performance trends, and con- allowing only limited scope for assessing their effective- cluding in Chapter 4 with the findings from initial coun- ness as instruments for delivering sector or country assis- try, sector, and thematic evaluations. tance strategies. Project evaluation is also weighted toward the assessment of project outputs-effects attrib- iQ aS utable directly to project inputs, whether Bank or bor- 1. Official development assistance (ODA) refers to all grants and rower-rather than wider project outcomes or develop- concessional lending supplied by bilateral and multilateral donors to ment impacts. Finally, coverage of developing countries. nonlending services and the con- 2. The adoption of new public management approaches by the Projects are tribution of development partner- governments of many donor countries has increased pressure on aid ships to project performance is ministries and budgets to adopt more results-oriented approaches. 'a 1 inadequate.14 These gaps are See, for example, recent changes in the aid programs of the United still a crucil tool iaeut.4 These, niedStaespad usrara being filled in the preparation of Kingdom, United Stares, and Australia. 3. Bauer (1966), Griffen (1970), Papanek (1972, 1973), Cassen, of development Country Assistance Reviews and et al. (1989), Mosley, et al. (1992). sector and thematic evaluations 4. Examining the allocation of ODA by multilateral and bilat- assistance that assess the combined impact eral donors, the authors find that multilateral development assistance of Bank lending and nonlending between 1970 and 1993 was targeted in general to poor countries beyond the project. with good policies. If the entire donor community had followed this A country focus-with emphasis on sector and the- path, the growth rate in low-income countries would have been as matic goals-is the most important organizing principle much as one-third higher. for the Bank's development assistance. Focusing on the 5. Countries that have received significant flows of ODA in recent years, such as Bolivia, El Salvador, Botswana, Uganda, and country and sector should not, however, overshadow the Mali, have all grown faster than would have been predicted based on importance of the project as a crucial tool of development their policies alone. Countries that received large amounts of ODA 12 The challenge of development effectiveness .0. EVALUATING PROJECT PERFORMANCE [7 he Bank rates pro- assess results in relation to and the capacity of project Bank and Borrower Per- jects according to inputs, in terms of costs, institutions. formance. The rating criteria three results-oriented implementation times, Institutional Develop- focus on how good a job counts-outcome, sustain- and economic and financial ment (ID) Impact. This is each partner has done during ability, and institutional returns. defined as the extent to the different stages of the development impact- Sustainability. This is which a project has im- project cycle from project and two process-oriented defined as the likelihood, at proved an agency's or coun- identification, preparation, counts-Bank performance the time of evaluation, that try's ability to use its human and appraisal to implemen- and borrower performance. the project will maintain its and financial resources ef- tation, with the ultimate Outcome. The judg- results in the future. Con- fectively. An assessment of objective of achieving results ment of outcome boils ceptually, sustainability dif- ID impact is made even on the ground. Bank per for- down to answering the fers from outcome, since it when no explicit ID objec- mance is defined as the qual- question: Did this project focuses on those features tives exist. Judgment of the ity of service delivered by the achieve satisfactory develop- that determine whether the impact of a project on insti- Bank, especially in those ment results considering the operation will last over its tutional development is tasks for which it has pri- importance and relevance of intended useful life. The based on the relevance of mary responsibility, such as its major stated objectives evaluator must determine project ID objectives; the appraisal and supervision. and the associated costs and whether, given the risks, achievement of explicit ID Borrower performance is benefits? The outcome rat- future benefits are likely to goals; how the achievement defined as the assumption of ing takes the following into exceed debt service pay- of regulatory or other pol- ownership rights and respon- account: relevance, to check ments, operation, and main- icy change goals has af- sibilities and delivery of consistency of goals with the tenance costs. Judgments fected economic, social, or inputs needed to prepare and country's overall develop- about sustainability are natural resource-related implement the project. These ment strategy and the based on country condi- behavior; and how agency performance assessments are Bank's assistance strategy tions, government economic restructuring or the estab- designed to be independent for that country; efficacy, to and financial policies, avail- lishment of a new organiza- from the project outcome check whether the operation ability of funds, the political tion has affected capacity and to take into account the achieved its stated physical, situation, and conditions and performance for plan- country's initial enabling financial, and institutional specific to the operation, ning, policy analysis, or environment and the impact objectives; and efficiency, to such as project management service delivery. of exogenous factors. but followed inadequate policies, such as Zambia and Tanzania, have 9. This simple model relates a series of "inputs"(Bank, partner, experienced very poor growth. and borrower) to a country's development process. Development 6. Killick (1991) argues that the main obstacle for achieving effectiveness is therefore defined in terms of measurable changes in greater aid effectiveness in Africa is governance and institutional pol- intermediate outcomes and final development impact. However, the itics in the region, distinction between inputs, outputs, outcomes, and impact is poten- 7. The Bank's strategic goals directly support the development tially confusing in that Bank lending, which is an "output" for the goals set out in the OECD/DAC report "Shaping the 21st Century." Bank is also a borrower "input," and intermediate "results" are also 8. Networks, an integral part of the Bank's renewal effort, are "inputs" to final development impact. The need to clarify these dis- staff groups that share common interests, skills, and goals. They tinctions is part of the task of developing a coherent aggregate mea- explicitly link Bank staff and managers who work in related sectors. sure of development effectiveness. Networks complement, not replace, existing organizational units. 10. The ongoing work on measuring "development return" for Currently there are four networks: Poverty Reduction and Economic example, is attempting to provide an easy-to-use metric for esti- Management; Environment and Socially Sustainable Development; mating gross returns to Bank lending as well as administrative Human Development; Finance, Private Sector, and Infrastructure. expenditures. 13 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 11. One of the studies cited in Table 1.1 adopts a sectoral 13. A strategic framework for operations evaluation in the Bank approach, examining the association between ODA and country- was endorsed by the Committee on Development Effectiveness in specific spending levels on education, health, and other areas of pub- February 1997. The new framework aims to fill gaps in the self-eval- lic investment (Feyzioglu, et al. 1996). The study finds, among other nation system, connect evaluation to strategy setting and resource things, that ODA and government spending are closely associated (a management, and redeploy resources from "project-by-project" eval- $1 increase in aid is associated with a $1 increase in government uations toward evaluation of country strategies, sector policy, and spending) and that ODA has a positive and significant impact on thematic impact assessments. public investment. However, the size of this impact is moderated by 14. Cofinanciers are encouraged to comment on Implementation the extent of fungibility in different sectors. Completion Reports (ICRs), but the views of other partners involved 12. Evaluating the impact of Bank assistance at the sector level is in the design and implementation of a project are rarely reported or also a key part of OED's new evaluation strategy. reviewed in ICRs. 14 THE DETERMINANTS OF PROJECT PERFORMANCE T his chapter examines some of the key factors examined the hypothesis that the quality of a country's responsible for the successful outcome of governance environment is also an important factor in the Bank projects, for both the aggregate country contrasting performance of country project portfolios. portfolio and the individual project. Policy, institutions, and country portfolio performance To examine the hypothesis, borrowing countries were clas- sified in a 3x3 matrix by policy performance and institu- In the 1994 Annual Review an analysis of aggregate port- tional quality.2 The measure of policy performance is a folio performance showed the importance of country eco- project-weighted index based on nomic performance (sound macroeconomic policy plus three principal components: infla- growth) for portfolio outcome. This year's analysis builds on tion, fiscal balance, and openness. The policv ancI this work and shows that, in addition to sound macroeco- Institutional quality is also project- nomic policies, the quality of a country's governance institu- weighted.3 The results of the clas- tions has an important influence on portfolio performance. sification are shown in Table 2.1. Average project portfolio per- Policy and institutions matter formance was calculated for each The 1997 World Development Report, The State in a country and group of countries. Changing World, sets forth the proposition that a capable Figures 2.1 and 2.2 show the state is a key determining factor behind the contrasting results for the groups in the four o development experiences of the world today. State capabil- corner cells of Table 2.1. In Figure p ity involves delivering appropriate institutional founda- 2.1, countries scoring high on both tions for market development, including a sound policy policies and institutions had an - framework, and key collective services for individual citi- average portfolio performance rat- zens to lead productive and healthy lives. A capable state is ing of 86 percent, significantly above the Bankwide aver- also accountable with the necessary checks and balances in age for the period of around 68 percent. The same group place to limit the scope for corrupt and arbitrary action. of countries also experienced an average per capita GDP The World Development Report finds that in addition to growth rate of 4.4 percent between 1985 and 1995. In con- sound macroeconomic policy, putting in place capable trast, countries scoring low on policy performance and state institutions has a large positive impact on a country's institutional capability had an average project performance long-term growth performancel and on the effectiveness of rating of just 58 percent and an average per capita growth public sector investments. It is on this premise that OED rate of -0.5 percent. 15 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS The sample has some exceptions. Ghana, for instance, can help offset the effects of policy distortion at the project ranked high on policy performance and institutional capa- level, at the country level there is still no substitute for hav- bility during the period, but its average per capita growth ing the necessary fundamentals in place. In settings where rate was only 1.4 percent per year between 1985 and policies and institutions are seriously weak, the Bank would 1995.4 Hungary also ranked high, but average per capita do best to focus on nonlending services that support the growth was actually negative during these years. Neverthe- strategic policymaking and implementation capacity of gov- less, average growth rates and country portfolio ratings ernment.5 Where policies and capacity are better, the Bank fully support the proposition that where both sound poli- would do best to focus on policy- cies and sound institutions are in place, performance is also based lending and nonlending ser- better (Box 2.1). vices to improve the enabling envi- W here policies It can be argued that, at the country level, policy and ronment for subsequent investment institutional factors apply most strongly to the performance operations. and institutions of the Bank's adjustment operations. As the cases in Box 2.1 So far the discussion has con- illustrate, policy-based lending is critically dependent on pol- centrated on countries along the are weak greater icy commitment and the quality of economic governance. extremes of the NW-SE diagonal Yet, when the same exercise is conducted for investment in Table 2.1 and Figures 2.1 and focus on non- operations only, the results stay broadly the same (Figure 2.2. Turning now to those coun- 2.2) with even weaker performance in countries having high tries scoring high on policy perfor- lending services policy distortion and weak institutions. Only 48 percent of mance and low on institutional investment operations, compared to 58 percent of all opera- capability or vice versa, the pat- might be tions, were judged to be satisfactory in these settings. tern remains consistent. Average Although these results relate only to the historical portfolio, portfolio performance is 78 per- appropriate in concurrence with earlier Annual Reviews, they suggest cent for countries with sound pol- that investment projects in environments with persistently icy environments but low institu- weak fundamentals are more likely to be unsuccessful. And tional capability and 75 percent for countries with while high-quality Bank appraisal and strong supervision relatively high institutional capability but a poor policy TABLE 2.1: COUNTRY CLASSIFICATION BY POLICY PERFORMANCE AND INSTITUTIONAL QUALITY, 1980-95 INSTITUTIONAL __ - - OLIC PERkORM.1NA E INLE1 QUALITY INDEN HIH N 0l1 M LOXT (i'Inbi hindi.t HIGH Huriar. Thail.mrd C miiar LXIu1.. t 11[L,1 MEDIUM r; ni d i r SoDurce: OEDr datai 16u Tuni.i.-ii.a I Sri L.ml-i Boi aiIiIi Echiui HotiidLjras Ki . cPtl11 i "tNIIi LO i,i'.1 N ~iPcru I ,,jiZambia I iLI LilI Source: OED data 16 The determinants of project performance FIGURE 2.1: PORTFOLIO PERFORMANCE BY POLICY FIGURE 2.2: PORTFOLIO PERFORMANCE BY POLICY AND INSTITUTIONAL ENVIRONMENT-ADJUSTMENT AND INSTITUTIONAL ENVIRONMENT-INVESTMENT AND INVESTMENT OPERATIONS OPERATIONS ONLY 100 *100 20 H[gh 20 CiT 0 0 High Low performance High Low performance Low Low Institutional Institutional quality quality Source: OED data Source: OED data environment. Average annual per capita growth for the policy environment and the commitment and capacity of former group of countries was 2.35 percent between 1985 local institutions. The odds tend to favor satisfactory and 1995 and 0.6 percent for the latter group. portfolio performance in settings These performance ratings are slightly higher than with good economic governance expected a priori, but they are explained by the strong and to disfavor satisfactory port- Bank assistance portfolio performance of a few countries, such as Indone- folio performance in economically sia, Bolivia, and Zimbabwe. These good performance underperforming countries. The works best when records overshadow the poorer performance of countries interrelationship between better such as Honduras. There, since 1984, only 55 percent of policy and effective institutions it catalyzes or evaluated projects were judged satisfactory and only 18 suggests that a strategy of "getting percent were judged of likely sustainability or as having the institutions right" is likely to reinforces other substantial institutional development impact. The key fac- bring significant gains not only in tors in Honduras's poor performance-besides increased portfolio performance but also in regional conflict during the period-are closely linked to the future stability of the policy enabling factors prevailing weaknesses in the institutional environment, in environment. These factors are particular, weak borrower commitment to undertake nec- clearly important at the macro level, but what link do they essary structural reforms and weak domestic capacity to have to project performance at the micro level? implement projects effectively. On the Bank side, rather than responding to these weaknesses through a greatly Project-level determinants of success revised assistance strategy, the Bank continued to appraise The analysis of factors responsible for project-level success projects well in excess of the country's absorptive capacity. builds upon an exercise first attempted in the 1994 Annual Notable in this respect was Bank financing of the El Cajon Review. The analysis enables OED to model and test sys- dam project in the early 1980s. The dam proved to be tematically for the significant determinants of project suc- vastly overdimensioned for Honduras's market and cess, measured by satisfactory project outcome. became a costly drain on resources for a decade thereafter. Based on the earlier country-level analysis, a general model The analysis of portfolio performance at the country should include three categories of explanatory variables, after level implies that Bank assistance works best when it controlling for basic project characteristics: descriptions of the catalyzes or reinforces other factors, in particular a sound macroeconomic policy environment, the institutional and gov- 17 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS LM¶1: CONTRASTING PORTFOLIO PERFORMANCE IN GHANA AND ZAMBIA Ghana effectiveness, and staff per- Economic reform and tional commitment to Background. The govern- formance. Bank policy resumed growth proved reform and limited absorp- ment's Economic Reform advice, along with that of elusive throughout the tive capacity. Despite some Program (ERP) included the International Monetary 1980s. Poverty increased excellent economic and exchange rate and trade Fund, also contributed to to the point that in 1994 sector work diagnosing the reform, extensive disman- the detail of the ERP, but some 70 percent of the problems, the Bank contin- tling of price and distribu- the Ghanaians always population lived in house- ued to focus on short-term tion controls, and great owned the program. The holds where basic needs stabilization objectives improvement of the fiscal borrower also gets a high were not being met. Zam- over longer-term adjust- situation. The reforms rating for performance. bia is one of the few coun- ment objectives. The reduced inflation from over Performance ratings avail- tries in Africa where life Bank's assistance strategy 75 percent in the early able for 36 operations expectancy has decreased came up short both in rele- 1980s to 10 percent in approved after 1980 show in recent years. The coun- vance and in efficacy. Per- 1992. Increased investments that 80 percent were judged try has also had to adjust formance ratings available rehabilitated Ghana's mori- satisfactory. Sustainability in response to a permanent for 18 of 45 operations bund infrastructure, and the was rated as likely in 20 of decline in resource-based approved since 1980 in regulatory framework was the 36 operations, and opportunities for develop- Zambia show that only 44 improved. Stabilization institutional development ment and to make the percent were judged satis- objectives were achieved impact was rated substan- transition from a centrally factory. Only three were simultaneously with tial in 18 operations. The planned, public sector- rated to have likely sus- increased real public expen- extent to which the Bank dominated economy to a tainability, and only four ditures, especially on social and government had shared market-oriented, private were judged to have sub- services and on public objectives and an agreed enterprise economy. stantial institutional devel- investment. strategy for undertaking a opment impact. Since 1994 reform program with Bank Assistance. In con- the Bank's assistance strat- Bank Assistance. Bank strong borrower commit- trast to the Ghana experi- egy has focused more assistance was instrumental ment explains the good ence, the Bank's assistance explicitly on liberalization, in helping Ghana to stabi- working relations and the strategy in Zambia during privatization, and diversifi- lize and rehabilitate an generally high quality and the 1980s was greatly cation. Already signs indi- economy in crisis. Espe- effectiveness of the policy impeded by the prevailing cate that both the rele- cially during the mid- dialogue during the 1980s. policy and institutional vance and efficacy of the 1980s, the Bank's program environment. Borrower Bank's portfolio in Zam- gets high marks for rele- Zambia performance was consis- bia, plus initial develop- vance of objectives, efficacy Background. Zambia illus- tently weak, reflecting lim- ment outcomes, are of implementation, cost- trates the counter case. ited or fluctuating institu- improving. ernance environment, and assessments of key partner inputs. 800 newly evaluated projects to the evaluation database The 1994 Annual Review attempted to test such a model using and the ever-increasing demand within the Bank to isolate probit analysis and found that variables in all three groups can areas of attention to enhance project performance. determine individual project outcome. Borrower performance was found to exert the most influence, followed by Bank per- Borrower and bank performance: still dominant formance and macro policy distortions. Results confirm that borrower performance continues to This general model of project performance has been be the most critical determinant of success at the individual revisited for this review, given the addition of more than project level.6 Good borrower performance raises the 18 The determinants of project performance probability of a satisfactory outcome by some 35 to 40 percentage points-nearly twice the impact of the next BO 2.3: QUALITY AT ENTRY CONTINUES TO MAKE A highest significant determinant (Box 2.2). DIFFERENCE After borrower performance, project success is deter- mined by the quality of Bank performance at entry and D he Social Invest- importance of fully in- during implementation, followed by the quality of the pre- ment Fund (SIF) volving local nongovern- vailing policy environment.7 These project in Nicaragua mental organizations in B "second-order" determinants are is an example of the re- the effort. Correct ap- B crrower responsible for 15 to 20 percentage ward to strong Bank per- praisals were made re- point increases in the likelihood of formance during the pre- garding the degree of gov- a satisfactory project outcome. approval stage. The SIF ernment commitment and If Bank performance matters, project was approved in the degree of fit between _ what can be done to improve sub- 1993 to help the borrower project design and local stantially the likelihood of project sustain its poverty allevia- implementing capabilities. success? Here the lessons are con- tion efforts. IDA perfor- Eighteen key performance sistent with previous years. The mance at identification indicators were defined econometric results show that and appraisal was highly for monitoring and evalu- improved quality at entry (consist- satisfactory, building on ation, which contributed ing of individual ratings for Bank confirmed lessons already to effective beneficiary performance during identification and appraisal) yields a known about social funds monitoring during imple- large payoff, followed by significant rewards from satisfac- including good planning mentation and unusually tory Bank supervision (Box 2.3). management and informa- well-documented project Empirical evidence on the importance of borrower and tion systems and the results. Bank performance in determining success reinforces the message from previous Annual Reviews. Commitment and D 2MK BORROWER PERFORMANCE IS A CRITICAL ownership of project goals by the borrower, together with DETERMINANT OF PROJECT SUCCESS high-quality project appraisal and supervision by the Bank, can offset some of the effects of a weak policy and institu- B orrower perfor- ernment pursued a num- tional environment. However, the evidence also indicates mance during the ber of important initiatives that these country-level factors, while not dominant, affect Uganda Second that, although not part of the likelihood of project success. Structural Adjustment any formal conditionality, credit was highly satisfac- were nonetheless critical to Policy and institutions matter tory, especially viewed overall performance dur- Besides the process variables, the model examined the against the breadth of the ing the project period and influence of certain country-level factors. A variety of spec- wider reform program and beyond. These include ifications of the macroeconomic environment were tried,8 the high demands imposed revamping macroeco- and all indicated that policy distortions significantly on the scarce capacity for nomic management, initia- detract from the likelihood of project success. Of the four leading and managing tives in medium-term variables used, the degree of outward orientation, or open- reform within the public expenditure planning, ness,9 was shown to be the most significant. sector. A key to the suc- overhaul of an ineffective In addition, the project-level results confirm that effec- cessful implementation of apparatus for parastatal tive institutions are strongly related to project success.10 the program reforms was divestiture and reform, a One effect of both large policy distortions and weak gov- the open and constructive revived privatization pro- ernment institutions is the increased opportunity for cor- dialogue maintained by gram, and from 1995 ruption,"1 which undercuts the climate for effective public the government. During onward, a strengthened investment. The empirical results indicate that, as high- implementation the gov- poverty focus. lighted in the 1994 Annual Review, freedom from corrup- tion12 is a large and highly significant determinant of pro- 19 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS ject success. By examining the impact of government credi- gic Compact's efforts to simplify lending processes, to shift bility on project success, the model reveals a positive and budget resources and decisionmaking to the front lines, significant relationship between and to increase use of adaptable lending instruments move C higher levels of credibility and the Bank in the right direction. U orrUpLion higher probabilities of project suc- cess.13 This implies that the qual- s ity of governance as perceived by 1. Over a 30-year period (1964-94) countries with both sound and weak private sector investors also affects policies and capable government institutions experienced an average the quality of the environment for annual per capita growth rate of about 3 percent. This compares with government 14 a per capita growth rate of only 1.4 percent for those countries with public sector investment.1 sound policies but weak government institutions. Countries with nei- credibility lower ther sound policies nor capable institutions experienced annual per Summary aniHmpliations capita growth of less than 0.5 percent for the same period. the likelihood of The analysis of project perfor- 2. The exercise generated a sample of 45 countries with consistent mance corroborates the findings data on policy performance, institutional quality, and portfolio perfor- of past Annual Reviews that good mance for the period 1980-95. All operations approved after FY80 that project success exited the portfolio by FY97 were included in the analysis of portfolio borrower performance is the most performance. The institutional capability index is a composite of mea- important determinant of project sures of the quality of government and draws on Knack and Keefer outcome, followed by Bank performance, sound policies, (1995) and Mauro (1995), among others. The index is based on a set of and freedom from corruption. But at the level of the coun- responses by foreign investors that focus on the extent of red tape try portfolio, performance is also associated with the qual- involved in any transaction, the regulatory environment, and the degree ity of the policy environment and broader institutional of autonomy from political pressure (World Development Report 1997, factors. p. 169). 3. For example, the policy performance score for the period Borrower performance, the strongest determinant of 1983-89 is weighted by the number of projects implemented during project success, underscores the continuing focus on that period in each country's portfolio. The difficulty with this supporting domestic capacity building and encouraging method is that policy averages consistent with project implementa- commitment to project goals by increasing borrower tion periods do not necessarily reflect recent improvements (deterio- involvement in design and implementation. The signifi- rations) in the policy environment. 4. Annual growth rates in Ghana were above 5 percent for much i fof the 1980s and early 1990s. With population growth still high, how- gate portfolio success and country economic performance ever, this did little more than restore average living standards from their also highlights the Bank's need to exercise strategic selec- collapse in the late 1970s and early 1980s. It is somewhat counterintu- tivity at the level of the country. itive then that Ghana ranks in this position in Table 2.1. Although Strategic selectivity, besides calling for more rigorous Ghana's policy performance has been good since the mid-1980s, its assistance strategies, requires more realistic assessments of record on institutional reform and capacity building is not strong. But the index is based upon foreign investors' perceptions of the transaction costs of doing business, the obstacles presented by the regulatory envi- straints and better selection and ronment, and the predictability of the policy environment. That Ghana L reater strategic sequencing of instruments that improved these overall conditions as part of the Economic Reform Pro- match existing government capa- gram probably accounts for the high ranking on the institutional quality selectivity and bilities. Although the individual index. In many other respects, particularly the quality of public sector components of "Institutional capa- management, problems of weak capability are widespread. bt 5. Promoting the fundamentals, such as a sound policy frame- risk management work, requires that support be given to building the strategic policy- factors-such as the transparency making and implementation capacity of government. Without such are needed and accountability of policymak- capacity, delivering a credible and stable policy framework is difficult ing institutions and the degree of and easily undermined. In addition to sound economic policy, atten- involvement by beneficiaries or tion needs to be given to overall public expenditure management and members of civil society in program and project design and the efficient allocation and use of scarce resources to essential collec- implementation-are likely to be particularly important in tive goods and services-basic health, basic education, basic infra- structure, law enforcement, etc. shaping the environment for selecting and designing pro- .tutr,lwenocmn,ec 6. Borrower performance is a project-specific rating defined, as in jects. To respond effectively, risk management based on Chapter 1, as the assumption of ownership rights and responsibilities flexible responses to changing events is crucial. The Strate- and delivery of inputs needed to prepare and implement a project. 20 The determinants of project performance 7. Bank performance is also project-specific and is defined as the 12. The precise forms of corruption are demands for special pay- quality of service delivered by the Bank, especially in those tasks for ments and bribes connected with import and export licenses, which it has primary responsibility, such as appraisal and supervi- exchange controls, tax assessments, police protection, or loans. sion. While some overlap between Bank and borrower performance is 13. This result is based on the variable "government credibility" inevitable, the ratings reflect as much as possible the distinct responsi- used in the 1997 World Development Report. The variable is a com- bilities and accountabilities of the Bank and borrower. Policy vari- posite index based on a 1996 survey of private sector entrepreneurs ables relate to the period of project implementation. in 69 countries. The main components of the index are ranked 8. Along with the individual macroeconomic policy variables in responses to questions about the predictability of the policy environ- the model (domestic inflation, fiscal balance, openness, and black ment, the rule making and rule enforcement capacity of government, market premiums), economic growth and an index derived using the degree of political stability, and the quality of service delivery. principal components analysis were used. 14. An important omission of the model is the impact of nonlend- 9. The model used the Sachs/Warner dummy indicator for ing services. An attempt to include the staff weeks of economic and sec- openness. tor work (ESW) associated with a given project into the model was 10. The corruption indicator used in the model is the Interna- made, but this measure was not found to be significant in a variety of tional Country Risk Guide corruption index. The probability differ- specifications. However, this most likely reflects the inadequacies of the ential associated with this variable indicates a 5 percentage point indicator used rather than the true worth of ESW. Recent work within increase in the probability of success when the index increases by one the World Bank (Deininger, Squire, and Basu 1997) finds that ESW has unit. a significant positive impact on various measures of World Bank pro- 11. The 1997 World Development Report identifies four vari- ject quality with similar results being obtained at the lending program ables consistent with high levels of corruption across countries: large level. A 1996 OED review of the evaluation of ESW stresses the need to policy distortions, unpredictable judicial enforcement, low public to develop a credible evaluation strategy for ESW at the design or concept private sector pay, and the absence of meritocratic systems for paper stage with the criteria to be employed and the measures to be recruitment and promotion within the public sector. used to assess ESW performance at completion clearly spelled out. 21 PROJECT PERFORMANCE TRENDS his chapter describes the latest project perfor- increase, however. As Figure 3.1 shows, satisfactory out- mance trends. For the first time, this Annual come rates have been rising steadily since FY94. But the Review presents evaluation findings by the fis- improvement in the past two fiscal cal year in which the operations exited the years is significant.3 portfolio, rather than the previous convention When viewed by lending vol- Portfolio of using "evaluation cohort." The reported data are for ume (disbursements) the achieve- those operations for which OED has evaluated an Imple- ment rate has been rising since mentation Completion Report (ICR). The shift to fiscal FY93, reaching 83 percent in improvement in year of exit is intended to facilitate closer comparison with FY96. For FY97 exits the achieve- the Annual Report on Portfolio Performance (ARPP) and ment rate will approach 88 per- the past two other Bank reports that use fiscal year analyses. It also cent. The marked difference in the f v offers a much closer link with the "real time" requirements percentage of satisfactory out- fiscal years is of the active portfolio and increases the relevance of evalu- comes by project and by disburse- snc ation findings to portfolio management.1 All references, ment deserves some discussion. sigmficant unless otherwise stated, relate to the fiscal year in which Past Annual Reviews have shown evaluated operations exited the portfolio. a positive relationship between project size and perfor- mance outcome; this relationship also holds for FY96. The Proromance resui s FYM ant FY97 relationship is, nevertheless, heavily influenced by a rela- tively small number of very large projects that performed Outcome well.4 In FY96, of 223 projects that exited the portfolio, 21 Project outcome is a composite of ratings on the project's were between $201 million and $609 million. Among the relevance, efficacy, and efficiency. Of these, relevance is the largest operations were a Rehabilitation Loan to Russia for most important dimension. A satisfactory outcome rating $609 million, another to the Ukraine for $508 million, and means that a project's objectives are relevant regarding sec- $524 million for a hydroelectric project in Mexico. Despite tor and country objectives and that the project was effica- two poorly performing large projects-one in India and cious and efficient in meeting its major objectives. one in Mexico-together these 21 projects had a disburse- The latest data for operations that exited in FY96 ment-weighted outcome rate of 93 percent satisfactory. show that 71 percent were judged satisfactory. For opera- This explains a large part of the difference in aggregate out- tions that exited in FY97 (based on 43 percent coverage),2 come performance by project and by volume. 76 percent achieved a satisfactory outcome-an 11 per- The overall improvement in project performance in centage point increase over FY94. This is not a sudden FY96 and FY97 is reflected in corresponding improve- 23 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIRURE 3.1: PERCENTAGE OF OPERATIONS WITH SATISFACTORY OUTCOMES BY EXIT FISCAL YEAR, 1980-97 BY PROJECT BY DISBURSEMENTS 90 90 80 80 A 07 E60 6 50 50 81 83 85 87 89 91 93 95 *97 81 83 85 87 89 91 93 95 *97 Source: OED data Note: Broken lines (exit FY97) indicate preliminary results, with less than 50 percent coverage of exited operations. ments in outcome performance by Regions, sectors, lending Africa continues to underperform relative to Bank norms types, and instruments. Figure 3.2 shows that, except for and has the largest number of projects at risk in the 1997 South Asia and Africa, the proportion of exited projects ARPP. with satisfactory outcomes by Region increased in FY96. Latin America and the Caribbean (LAC) show the The partial results for FY97 suggest that further improve- strongest performance improvement over time with an ments are likely for all Regions, although Sub-Saharan average of 86 percent satisfactory projects for FY96 com- FIGURE 3.2: PERCENTAGE OF OPERATIONS WITH SATISFACTORY OUTCOMES, BY REGION AND EXIT FISCAL YEAR, 1990-97 AFRICA EAST ASIA AND PACIFIC 100 100 80 80 60 60 40 40 20 20 0 0 90 91 92 93 94 95 96 *97 90 91 92 93 94 95 96 '97 AFR m Bank EAP Bank 24 Project performance trends FIGURE 3.2: (CONTINUED) EUROPE AND CENTRAL ASIA MIDDLE EAST AND NORTH AFRICA 100 100 80 so 4w 60 60 40 40 20 20 0 0 90 91 92 93 94 95 96 *97 90 91 92 93 94 95 96 *97 m ECA mm Bank mMNA mm Bank LATIN AMERICA AND CARIBBEAN SOUTH ASIA 100 100 WWWWE s0 80 oi 60 6 40 40 20 20 90 91 92 93 94 95 96 *97 O 90 91 92 93 94 95 96 *97 LAC m= Bank SAS m Bank Source: OED data Note: Broken lines (exit FY97) indicate preliminary results, with less than 50 percent coverage of exited operations. pared with an average of 67 percent for FY90-95. When size of the Brazilian portfolio in FY96 and FY97 disbursement-weighted, LAC and Europe and Central Asia accounts for a large part of the regional improvement (ECA) show the biggest improvements. South Asia shows a during these two years. Brazil accounted for 38 percent further drop in performance by project and disbursement of projects that exited the LAC portfolio in FY96 and a amount in FY96, although this is not sustained in the par- further 30 percent in the partial sample for FY97. Project tial FY97 data. outcome was judged universally satisfactory in both Underlying improved regional performance in LAC years. This compares with an average outcome perfor- is a large and significant improvement in portfolio per- mance of just 36 percent satisfactory between FY90 and formance for Brazil and Argentina after FY95. The sheer FY95 (Box 3.2). 25 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS SUI 3.1: COMPOSITION OF FY96-97 EVALUATED EXITS AND THE ACTIVE PORTFOLIO he Bank's portfolio were IDA or blend loans. ure 3.3). Africa-which, percent of the total pro- 2 has not changed Adjustment loans consti- despite a shrinking active jects and 18 percent of the .1. significantly in size tuted 15 percent and portfolio, continues to have disbursements. Agriculture or composition in recent investment loans 85 per- the largest number of pro- is no longer the largest sec- years, although commit- cent. The most common jects (28 percent) among tor in the active portfolio, ments have continued to lending instrument among the Regions in the FY97 having been surpassed by decline steadily in real FY96 exits was the Specific active portfolio-also had transport with 15 percent terms. Of the 1,752 pro- Investment Loan, which the largest number of eval- of total commitments. Oil jects in the FY96 active was used in about half of uated exits from the FY96 and gas has shrunk to 3 portfolio (about $143.7 the projects. portfolio (31 percent). percent of active portfolio billion), the FY96 evalu- ECA, the fastest grow- The largest borrower commitments. Of the four ated exits comprise 223 ing Region in the active in the FY96 evaluated exits Networks, Finance, Private projects amounting to portfolio, showed the was India. In the FY97 Sector, and Infrastructure about $19.4 billion. IBRD largest increase in number active portfolio India is (FPSI), with 40 percent of loans constituted 52 per- of projects evaluated, with second in disbursements to the projects, had the most cent ($13.9 billion) of the 6 percent between exit China. Viewed by sector, FY96 exits. FPSI also has FY96 exits; the remaining FY90-95 and over 10 per- agriculture continued to be the largest share of the 48 percent ($5.5 billion) cent in exit FY96 (see Fig- the largest sector with 25 active portfolio in FY97. By sector the performance pattern is less clear. Data decrease. Water supply and sanitation is the poorest per- indicate that about half of the sectors (two-thirds by dis- forming sector in FY96, a record that, according to the lat- bursements) show an increase in satisfactory projects and est ARPP, appears unlikely to change in the near future.5 the other half (one-third by disbursements) show a Looking at the larger sectors6 (Figure 3.4), improvements iGURE 3.3 THE SIZE OF THE ACTIVE PORTFOLIO NUMBER OF PROJECTS BY SOURCE (ARPP) COMMITMENTS BY SOURCE (ARPP) 2000 160 1500- 120 1000 80 E Er 500 40 0 0 1992 1996 1997 1992 1996 1997 E IBRD E IDA Total E IBRD O IDA [] Total Source: Annual Report on Portfolio Performance (ARPP), Fiscal Year 1997. 26 Project performance trends in the percentage of projects formance, there seems to be with satisfactory outcomes ROOX 32 TURNAROUND IN THE BRAZILIAN no marked change in sub- in FY96, compared with PORTFOLIO sector trends, but there is a FY90-95, include agricul- marked increase in the num- ture, electric power and ntil FY95, the agencies, and suspension ber of education projects other energy, multisector, Brazilian portfolio of disbursements due to with high or substantial de- public sector management, had been perform- noncompliance. After mandingness, from 77 per- and transport. Based on the ing very poorly. Projects FY94, political stability, cent for FY90-95 to 86 per- partial results for FY97, implemented in Brazil together with new govern- cent in FY96 and 90 percent however, only improvements between FY85 and FY93 ment resolve and more in FY97. The pattern on in agriculture and transport were plagued by imple- flexibility and better super- complexity and riskiness is appear likely to be sustained. mentation delays due to vision by the Bank, helped similar. There is also a par- Results for the education severe macroeconomic to improve relations allel increase in the propor- sector are less favorable. The crises and political between the borrower and tion of education projects percentage of projects judged changes. Of the 104 pro- the Bank. Brazil's portfolio with high or substantial satisfactory in FY96 was 70 jects evaluated from was restructured and ex- institutional development percent, compared to an FY80-95, only 52 percent post evaluation perfor- objectives, from 17 percent average of 80 percent for were judged satisfactory. mance has been much in FY90 to 70 percent in FY90-95. The initial results Problems were associated improved. In FY96-97, 24 FY96. This implies that per- for FY97 suggest that the with inadequate counter- projects were evaluated, formance in education is decline may be ending, but part funding, deteriorating all of which were judged being affected by an increase the full prognosis for FY97 quality in implementing satisfactory. in the demandingness of is still uncertain. In trying to operations combined with a explain this decline in per- shift toward institutional development objectives. However, looking at the active FIURE 3.4: PERCENTAGE OF OPERATIONS WITH portfolio, relatively few education projects are rated as SATISFACTORY OUTCOMES, BY SECTOR AND EXIT being at "potential" or "actual" risk,7 suggesting that near- FISCAL YEAR, 1990-95, 1996, AND 1997 term prospects for sector performance are generally good. By lending type adjustment loans continue to have a Agriculture 7,7 higher percentage of satisfactory outcomes than investment Education loans. Among the FY96 exits, 88 percent of adjustment operations Energy were judged satisfactory (95 per- Adjustment cent when weighted by disburse- Multisector I ments), compared to 69 percent of . lending is ouL PHN investment loans (79 percent when disbursement-weighted). The early Pub. Sect. Mgta results for FY97 are markedly dif- performng Transport ferent, however, with the percent- investment _ _ _ age of satisfactory adjustment wss ___ loans falling to 75 percent and 0 20 40 60 80 100 investment loans rising to 77 per- Percent satisfactory cent satisfactory (see Annex). This O Exit FY 1990-95 E Exit FY 1996* Exit FY 1997" suggests that the results for FY96 may not be sustained once the full results are in for FY97. Source: OED data Among investment loans Sector Investment and Main- * Excludes sectors where coverage was less than 10 projects in tenance Loans, and Financial Intermediary Loans show the exit FY96. ** Exit FY97: preliminary results, with less than 50 percent least successful performance with satisfactory rates for coverage of exited operations. FY96 of 55 percent and 56 percent, respectively. While 27 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Institutional development impact &N 3.3: IBRD AND IDA PERFORMANCE TRENDS Institutional development (ID) impact is probably the most demanding of the performance evaluation criteria and, as mprovement in out- show relatively little described in Chapter 1, is certainly the most important for come performance by change in performance, long-term development effectiveness. As with the other number of projects is with 64 percent judged results-oriented ratings, the ID rating is built on several fac- most apparent for IBRD satisfactory in FY96 com- tors, some of which are described in Box 3.4. loans. In FY96 IBRD pared to 63 percent for loans achieved a 78 per- FY90-95. The partial cent satisfactory perfor- results for FY97 suggest mance, compared with an an improvement in IDA BOX 3.4: WHEN IS A PROJECT JUDGED TO HAVE average of 71 percent for to 73 percent satisfactory, SUBSTANTIAL INSTITUTIONAL DEVELOPMENT FY90-95. IDA and blend with IBRD steady at 80 IMPACT? loans, on the other hand, percent. nstitutional develop- capital in fostering ment is defined as economic and social improving a country's development. effective use of its human, The Public Works they also appear to be the low performers in FY97, they organizational, and finan- and Employment Project both show an increase to 67 percent and 69 percent satis- cial resources. Because (AGETIP) in Senegal was factory, respectively. Technical Assistance Loans have his- the ultimate test of Bank judged to have achieved torically been the weak performer, with an average of only assistance is its short- and substantial ID impact. 52 percent satisfactory outcomes during FY90-95. How- long-term effects on peo- The grounds for this judg- ever, for exit FY96 this type of loan shows a marked ple, policies, and domes- ment were that the improvement to 69 percent satisfactory-an increase that tic capacities, ID impact AGETIP proved to be appears likely to be sustained in FY97. is a crucial test of devel- successful not only as an opment effectiveness. A efficient executing agency J 5: OUTCOME PERFORMANCE IDA/IBRD project achieving substan- but also in its methodol- tial ID impact is likely to ogy in implementing 100 include some of the fol- small public works. It lowing elements: (a) it accomplished its innova- encourages the borrower tive solution to public so to foster an enabling envi- works contract manage- ronment for institutions ment without corruption that can contribute to the or additional layers of so - development process; (b) bureaucracy. A large part it finances capacity-build- of AGETIP's substantial ing initiatives to increase ID impact can also be 40 institutional effectiveness attributed to its financial and efficiency in these and decisionmaking areas; (c) it ensures that autonomy, the strong 20 - institutions involved in political support it has the initiatives it supports received, its private sector are appropriate to the philosophy, transparent 0 task and acceptable to procedures, strict internal IBRD only IDA/blend the people involved; and and external accountabil- g1990-95 1996 1997 (d) it supports work to ity, and a results-oriented enhance the role of social management culture. Source: OED data 28 Project performance trends FIGURE 3.6: INSTITUTIONAL DEVELOPMENT IMPACT BY impact. Disbursement-weighted, the figures are 40 percent EXIT FISCAL YEAR, 1987-97 for FY96 and 54 percent for FY97.9 BY PROJECT Sustainability 60 Sustainability is the likelihood, at the time of evaluation, that an operation will maintain its benefits in the longer so0 run. OED has found over the years that sustainability .. and institutional development are closely linked; 40 progress on the institutional front generally takes care of many crit- 30 ical aspects of sustainability, such Sustainability is as revenue earning capacity or 20 budget support. the likelihood The latest performance results to indicate that the proportion of that an operation projects judged as having likely 0 sustainability has maintained a will maintain its 87 88 89 90 91 92 93 94 95 96 *97 modest rising trend, increasing from an average of 46 percent for benefits n the FY90-95 to 48 percent in FY96 BY DISBURSEMENTS (see Figure 3.7). However, the par- long run tial results for FY97 are a promis- ing 51 percent of projects judged to have likely sustainability. When disbursement-weighted, so / the percentage for exit FY96 with likely sustainability * climbs to 59 percent, but this reflects no change over the 40 FY90-95 average. For partial FY97 exits the disbursement- weighted figure is 65 percent. S30 _ Explaining trends in outcome 20 In interpreting the encouraging signs of the latest perfor- mance results, several factors need to be considered. 10 OED review of ICR ratings. Analysis suggests that OED's review of the ratings is unchanged. OED disagreed 0 with at least one rating in 28 percent of the FY95 and FY96 87 88 89 90 91 92 93 94 95 96 *97 ICRs. In the partial sample of FY97 ICRs, OED disagreed with at least one rating in 26 percent of the cases. Project demandingness, complexity, and riskiness. Source: OED data The data in Figure 3.8 do not show any decline in the Note: Broken lines (exit FY97) indicate preliminary results, with less than 50 percent coverage of exited operations. number of projects rated as having high or substantial demandingness, complexity, or riskiness for exit FY96. In fact, to the contrary, evaluated exits in FY96 were gener- The results for FY96 and FY97 show a marked increase ally considered more demanding, complex, and risky than in the percentage of projects judged to provide substantial ID projects evaluated in previous years.10 Evaluated FY97 impact. From a plateau of about 30 percent for FY90-95, projects appear to be following the same pattern, imply- the proportion of projects rated as having substantial ID ing that improved outcome performance is not a function impact rose to 39 percent in FY96 (see Figure 3.6).8 This is of the Bank selecting simpler or less complex projects. not likely to be an errant result given that 39 percent of the Sector selectivity. According to three categories of sec- partial exits in FY97 are also judged as having substantial ID toral performance-high, medium, and low-based on the 29 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 3.7: SUSTAINABILITY BY EXIT FISCAL YEAR, 1987-97 BY PROJECT BY DISBURSEMENTS 70 70 60 60 50 50 40 40 t& a30 e~30 20 20 10 10 0 0 87 88 89 90 91 92 93 94 95 96 *97 87 88 89 90 91 92 93 94 95 96 *97 Likely Uncertain Unlikely Likely - Uncertain Unlikely Source: OED data Note: Broken lines (exit FY97) indicate preliminary results, with less than 50 percent coverage of exited operations. average project-based sector outcome for exit FY85-93, Thus, no convincing case can be made for a change in dis- there appears to be no change in the number of projects tribution of projects over time by sector group. and levels of disbursement in high- or medium-performing Country selectivity. An examination of the distribution sectors compared to low-performing sectors for FY96.'1 of projects by country groupings shows a very slight shift FIGURE 3.8: DEMANDINGNESS, COMPLEXITY, AND RISKINESS BY EXIT FISCAL YEAR, 1987-97 BY PROJECT BY DISBURSEMENTS 100 100 80 80 1 m - 60 - 60 8 40 - 40 20 20 0 0 87 88 89 90 91 92 93 94 95 96 87 88 89 90 91 92 93 94 95 96 EDemandingness E]Complexity Riskiness oDemandingness E]Complexity E]Riskiness Source: OED data Note: Excludes exit FY97 because of less than 50 percent coverage of the exited operations. 30 Project performance trends in patterns for evaluated exits in FY96. Five country As Chapter 2 already showed, of the variables considered groups were defined based on the Bank's Country Eco- important in determining project success, borrower perfor- nomic Performance Index for mance is the largest, followed by Bank performance. If the T 1979-92.12 FY96 evaluated exits current evaluation data point to continued improvement in here has been show a marked increase in the the percentage of operations with satisfactory Bank and number of projects in the lower- borrower performance, then this may be another very a decrease in medium economic performance important factor behind recent improvements in outcome group compared to those exiting in performance. the number of FY94 and FY95 (25 percent com- pared with 12 percent) and a Bank performance evaluated proje_cs decrease in the number of projects Bank performance is composed of ratings for three process in the poor economic performance activities-project identification, appraisal, and supervi- in he p group of countries (7 percent com- sion-as well as an average of all three. pared with 12 percent). The per- For FY96, 74 percent of operations had satisfactory e cofni centage of projects in high- Bank performance. The partial data for FY97 indicate a performing and upper-medium slight increase to 77 percent. But when compared to the performance performing countries was substan- trend for FY90-95, the percentage of evaluated projects tially the same over this period. with satisfactory Bank performance ratings has stayed group These results imply that an ele- more or less constant (Figure 3.9). When viewed by dis- ment of country selectivity may be bursements, the FY96 results show a decrease in the per- taking place and could be one factor accounting for the centage of projects with satisfactory Bank performance (75 improved performance results after FY95. percent compared to 80 percent for FY90-95), although the partial results for FY97 suggest the decline may be Bank and borrower performance: FY96 and FY97 temporary. A further possible source of improvement in the major The individual ratings for Bank performance at identi- results-oriented ratings is Bank and borrower performance. fication, appraisal, and supervision again show no signifi- A10 3.9 BANK PERFORMANCE BY EXIT FISCAL YEAR, 1987-97 BY PROJECT BY DISBURSEMENTS 100 100 90 90 80 80 70 . 70 S60 60 50 50 40 40 87 88 89 90 91 92 93 94 95 96 *97 87 88 89 90 91 92 93 94 95 96 *97 Identification Average Identification m Average Supervision sma Appraisal Supervision man Appraisal Source: OED data Note: Broken lines (exit FY97) indicate preliminary results, with less than 50 percent coverage of exited operations. 31 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS cant break with previous trends. No significant differences FIGURE 3.10: BORROWER PERFORMANCE BY EXIT were found between percentage satisfactory ratings for FISCAL YEAR, 1987-97 FY96 and the FY90-95 average, although the FY97 data BY PROJECT suggest an increase in Bank performance in supervision 100 100 compared to the FY90-95 average. It is too early to tell whether this increase will be sustained. While changes in Bank performance are unlikely to be so a major factor behind the latest changes in outcome per- formance, the fact that Bank performance has remained ... fairly high at 74 percent-despite evidence that projects o are continuing to be more complex and demanding-is indirect evidence that the Bank is continuing to make 4 improvements. Borrower performance 20 Borrower performance includes separate ratings on project preparation, implementation, and compliance, as well as o an average rating of all three. 87 88 89 90 91 92 93 94 95 96 *97 Results for FY96 show that Preparation Average Bank the proportion of evaluated pro- Compliance Implementation jects with satisfactory borrower performance has performance is 71 percent (Figure BY DISBURSEMENTS 3.10). This compares with an aver- remained fairly age of 60 percent for projects eval- 100 uated for the period FY87-91 and static 65 percent for the period FY90-95 80 (Figure 3.10). The partial results ..* for FY97 indicate that borrower performance may be continuing to improve (74 percent satisfactory).13 Looking at the individual borrower performance rat- 40 ings, the strongest improvement is in borrower preparation and compliance with loan covenants. Borrower compliance was judged as satisfactory for 71 percent of evaluated exits 20 in FY96, compared with just 58 percent for FY90-95.14 The partial results for FY97 indicate a possible further 0 .87 88 89 90 91 92 93 94 95 96 *97 improvement to 75 percent satisfactory.15 These findings Preparation Average suggest that recent improvements in outcome performance may be partly explained by improvements in borrower preparation and compliance. Source: OED data While the Bank takes borrower performance into Note: Broken lines (exit FY97) indicate preliminary results, with account in its lending decisions and can work to improve less than 50 percent coverage of exited operations. borrower performance through dialogue and technical assistance,16 much of what constitutes satisfactory bor- One possible factor is the effect of selectivity in lending rower performance-direct project preparation and man- patterns. As already indicated, some evidence suggests a agement, public investment decisions, and overall macro- shift in lending toward better-performing countries. Better- economic management-is essentially beyond Bank control. performing countries, as discussed in Chapter 2, generally So what additional factors may help explain the sustained provide a better environment for Bank projects. Although improvement in borrower performance in recent years? the criteria for borrower performance used here are entirely 32 Project performance trends project-specific, it is plausible that the overall improvement Improvements in borrower ownership and commit- reflects an underlying shift in lending toward better-per- ment are clearly behind improved borrower performance, forming countries. This is highly speculative, however, but these trends may also partly given the limited concrete evidence of a move to greater reflect internal Bank initiatives. selectivity in the evaluated exits. The Bank's Portfolio Management Improvements A second possible explanation lies in a decline in wars Task Force issued a report in or civil disturbances in borrower countries during 199217 that focused, among other in borrower FY96-97 compared to earlier periods. Analysis finds no things, on the need to enhance bor- difference, however, in the proportion of projects having rower involvement in Bank- ownership and high or substantial negative impact from war or civil dis- supported operations. Borrowers turbance in FY96 (44 percent) compared with FY90-95 were to be more involved in devel- commitment are (42 percent). oping plans for the operational Improved borrower performance may also reflect an phase of a project, complete with behind improved improved policy climate in many borrower countries performance indicators, and bor- together with increased borrower commitment. These rowers were to evaluate the pro- performance dimensions of performance are difficult to measure, espe- ject's implementation. Management cially at the project level, but two examples in Box 3.5 actions in response to the task force report were under way illustrate how they often play out in practice. before the report was issued. For the most recently evalu- BOX 3.5: COMMITMENT AND OWNERSHIP UNDERPIN HIGHLY SATISFACTORY BORROWER PERFORMANCE onsidering that tation of the reform pro- example of highly satisfac- mance and, hence, success- Mozambique was gram continued to move tory borrower perform- ful outcomes. The staff of emerging from a ahead after the credit ance-starting with the the project implementation brutal civil war and a closed. Where slippage government's active unit learned quickly from severe domestic drought, occurred, the government involvement in the design the expatriate procure- and given its serious defi- made corrections and put of the project. Thereafter, ment adviser, and in time ciencies in institutional the program back on track. there was a healthy dia- the unit was able to han- development, the improve- The government achieved logue with the Bank dur- dle all transactions with ment in policies during the a high percentage of the ing implementation. Given only limited Bank interac- implementation of the Eco- promised structural Albania's unfamiliarity tion. Furthermore, it was nomic Recovery Credit is reforms, such as privatiza- with Bank procedures and able to pass this knowl- impressive. The govern- tion, liberalization of the its limited experience with edge on to other public ment's commitment to foreign exchange and inter- international transactions, and private sector entities. implementing the reform national and domestic trad- the government relied on In the last two years of program was key to the ing markets, and creation the Bank's support and project implementation, overall positive outcome. of a new central bank. As worked closely with Bank expatriate procurement Commitment can be traced a result, OED upgraded staff to clarify sector pri- advice was not needed. to the government's active borrower performance to orities, identify imports, Furthermore, the project involvement in project highly satisfactory (from and work out the neces- implementation unit design and implementation satisfactory in the ICR) and sary arrangements for pro- became the focal point and its clear demonstration judged this to have been ject implementation. In within the government for of ownership. Moreover, one of the most successful this project, rapid develop- project management and the government's commit- adjustment operations in ment of institutional capa- procurement and was ment to reform was sus- Sub-Saharan Africa. bility was another factor given similar responsibili- tained, despite political The Albania Critical leading to highly satisfac- ties under four other uncertainty, and implemen- Imports Project is another tory borrower perfor- Bank-assisted operations. 33 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BOX 3.B: PERFORMANCE TRENDS BY APPROVAL YEAR 7 igure 3.11 shows out- after that, the sample cover- outcome performance in Bank performance factors come performance by age falls below the statisti- the 74-78 percent satisfac- as quality at entry and approval year. The cally acceptable level of 40 tory range. supervision. The evidence percentage of evaluated percent. Given such a con- With approval FY90 for borrower performance projects for approval year straint it is not possible to as the cutoff point, a trend (Figure 3.13) is not as strik- 1989 is a fairly high 71 per- confirm that the perfor- of improving Bank perfor- ing as it is by exit year. cent, but it drops quickly to mance trends seen at exit mance (see Figure 3.12) Nonetheless, there are signs 53 percent for approval year are being repeated by cannot be confirmed, ex- of positive improvement year 1990 and 35 percent approval year. Viewed by cept for a modest increase in the percentage of pro- for approval year 1991. disbursement amounts, between FY89 and F90. jects with satisfactory bor- Consequently, trends can however, projects evaluated It therefore appears too rower performance for only be reported up to and for approval years 1987-90 soon to see the effects of FY89 and FY90, both by including approval FY90; show consistently strong increased attention to such project and volume. ated projects, and for quick-disbursing projects implemented norms, and parallel evidence indicates that projects con- after 1992, there may already be some signs that this initia- tinue to become more risky, complex, and demanding. tive has begun to pay dividends. These findings call for greater selectivity in choosing pro- jects and lending instruments to match the needs and risks Risk and rewad of different sectors and countries. This requires rigorous The latest portfolio performance trends are encouraging, assessment of risks and risk management, but the risk- but some countries and sectors still perform below Bank reward trade-off is a complex one. On one hand, the Bank FIGURE 3.11: PERCENTAGE OF OPERATIONS WITH SATISFACTORY OUTCOMES BY APPROVAL FISCAL YEAR, 1970-92 BY PROJECT BY DISBURSEMENTS 100 100 90 90 t~80 80 ~70 70 i 60 - 60 ______________i 50 50 70 72 74 76 78 80 82 84 86 88 90 *91 *92 70 72 74 76 78 80 82 84 86 88 90 *91 *92 Source: OED data Note: Broken lines (approval FY91-92) indicate preliminary results, with less than 50 percent coverage of exited operations. 34 Project performance trends FOURE 3.12. BANK PERFORMANCE BY APPROVAL IURE 3.13: BORROWER PERFORMANCE BY YEAR APPROVAL YEAR 100 100 90 90 0 80s 80 00 00 S60) -_ 60-4 50 50- 40 40 80 81 82 83 84 85 86 87 88 89 90 *91 *92 80 81 82 83 84 85 86 87 88 89 90 *91 *92 r= Identification Average Preparation m Average m Supervision see Appraisal Compliance s Implementation Source: OED data Note: Broken lines (approval FY91-92) indicate preliminary results, with less than 50 percent coverage of exited operations. can manage risks at the project level through improved Using this reward metric to examine the average design and supervision and at portfolio level by modifying reward for all Bank projects in the FY90-97 portfolio, the its lending patterns in the most data show a mean reward of 5.2, implying that in a possi- risky sectors and countries. On the ble range of 0 to 10, the Bank has been reaping relatively Effective other hand, positive development modest rewards from its project portfolio. Examining the opportunities must be seized, and average reward for country and sector portfolios of 15 pro- Management of simply reducing risks could drasti- jects or more, the mean reward by country ranges from 3.2 cally reduce the rewards payoff to in Algeria to 7.2 in China (Figure 3.14) and by sector from risks and rewards investment. Wisely trading-off risks 4.5 for public sector management to 6.2 for telecommuni- against rewards is therefore key to cations (Figure 3.15). is the key to a a selective strategy. The other side of the trade-off is risk. Risk here means To examine the finer details of the probability of falling short of a reward level and is mea- selective strategy the risk-reward relationship OED sured ex-post as the percentage of projects falling short of analyzed the historical portfolio that level.19 These risk-reward curves indicate the change for FY90-97 using two new concepts of risk and reward. in risk associated with any given change in reward. As such The rewards concept involves a metric along which evalu- they can be used for portfolio management. If a portfolio ated projects are graded by their final outcome rating, insti- has a large number of ongoing and new projects falling in tutional development impact, and degree of sustainabil- the northwest corner of the graph, suggesting a low mean ity.18 All projects are then located along this metric, which reward and high odds of failing, there is a strong signal that ranges from 0 to 10. A reward of 5 corresponds to a pro- the portfolio needs restructuring. Also, for projects about ject with satisfactory outcome but modest institutional to become borderline, the curves suggest that much reward development and a 50:50 chance of sustainability. can be gained by paying more attention to the sustainabil- 35 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 3.14: RISK AND REWARD BY COUNTRY FIGURE 3.15: RISK AND REWARD BY SECTOR 100 100 80 ----- 80 be E60-- __ _ __, c 40 --- g40 20 20 0 0 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Reward Reward Algeria Bankwide - China PSM Bankwide - Telecom Source: OED data Source: OED data ity of what will be achieved. Finally, the risk-reward curves of the performance results by disbursement are heavily provide ways of ranking and classifying portfolios accord- influenced by a small group of very large projects in a few ing to the level of risk. countries. These results, therefore, should be interpreted Reducing the share of portfolios with inferior mean with caution. reward is the most effective approach to managing risks, In addition to improvements in overall outcome, there especially with country portfolios where the variation in is evidence of an increase in the institutional development performance is greatest. This remains largely true if one impact of operations, with 39 percent of FY96 exits hav- adjusts the mean for the variance in rewards (see Box 3.7). ing substantial ID impact compared to 30 percent for It is important to note, however, that this strategy is con- FY90-95. This is a good indication that at the project level strained by two considerations. First, the risk-adjusted the Bank is gradually improving its development effective- reward will decline when the size of a portfolio exceeds the ness. Nevertheless, with only half of Bank operations achiev- borrower's needs or absorptive capacity. Second, the Bank ing likely sustainability, there is still considerable work to is a coproducer of rewards and must therefore take into be done. account the involvement of partners and competitors when Improved outcome performance appears to be linked assessing the risk-reward relationship. A example is to a number of potential factors, including some evidence telecommulications, where the Bank's mean reward is his- of country selectivity and a marked improvement in bor- torically quite high, but where Bank projects have consis- rower performance, particularly in preparation and com- tently been outclassed by the private sector. pliance with covenants. Bank performance has remained steady, but with increased project demandingness and com- Summary and implications plexity this is taken to indicate that performance is holding Based on the number of evaluated projects, 71 percent of if not improving. The importance of these process improve- projects in FY96 were judged satisfactory. A partial sample ments is underscored in a selection of outstanding and poor for FY97 indicates that 76 percent were judged satisfac- operations from the FY96 and FY97 cohorts (Table 3.1). tory. This implies an upward trend in performance, Of the outstanding operations, all had a highly satisfactory although the full implications for FY97 exits are as yet outcome and satisfactory or highly satisfactory Bank and uncertain. By disbursements, 83 percent of evaluated exits borrower performance. Of the poor operations, all had in FY96 achieved a satisfactory outcome and 88 percent highly unsatisfactory outcome and unsatisfactory or highly did so for the early FY97 returns. However, the magnitude unsatisfactory Bank and borrower performance. 36 Project performance trends BW 387. ADJUSTING THE PORTFOLIO FOR REWARD VARIANCE 77otwithstanding risk aversion, the mean Ghana and Morocco and even at the cost of increas- the need to take reward is reduced by a the education portfolio, all ing its variance when it is appropriate risks, fraction of the reward vari- of which slightly improve well below the norm. the Bank ought to be some- ance. Adjusting the rewards their ranking because of Increasing the overall share what risk averse because of the FY90-97 exits ac- their relatively low reward of country portfolios with highly unsatisfactory pro- cording to this calculus variance. These results sug- small variances would ject outcomes undermine tends to amplify the gap gest that opportunities improve Bankwide risk- some of its most precious between the best and worst could be explored for rais- adjusted reward, but the assets, such as stakehold- portfolios. There are a few ing the mean reward of a overall effect is relatively ers' support. To adjust for notable exceptions, such as country or sector portfolio small. PORTFOLIO REWARD BEFORE AND AFTER ADJUSTMENT FOR RISK (FY90-97) HEFORE ATFR RFFOR AFTER Tele..- . 4.1 I 1 OiI,i. n1 .Gi ., I-'s Educ. -.a . . , , -.1 Min.~'I Ii 42 Elec,-a' L~ 1; 2.- 4 r, k-.. Heoo. i ..... 4. 4.,, 2. 4 1 .. .4 FincU2 ~ I , 4..s Indt,;.' -4.,42 ' 4.1 1.4 Wat- 1. Pubc ., . . i 4. i ~ -. ill >5. 2 Source: OED data TABLE 3.: EXAMPLES OF OUTSTANDING AND POOR OPERATIONS, FY96-97 EVALUATED EXITS 0PLk AT iIN 10 \\. 4 RLII iT Outs,tandin11 B,: Ll A1 .1 Si Ki iC.1 Br n KLIT I _i,nd i h T 1.Ti 1'.C4'S An.an -I- c-ininbiiini ric ci. L Poor GI rnh. Ir ind. F 1c, ri, C[. 12-4 I1 rik c% Lp1111 \\ AICv tf! 1 ,ri I d Source: OED data 37 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS This encouraging picture does not, however, suggest 9. See the Annual Review of Evaluation Results 1992 for a that either the Bank or borrowing countries can be com- detailed discussion of ID and its implications for overall development placent. Outcome ratings for evaluated projects by effectiveness. approval year do not yet mirror the results by exit year, 10. Significant at the 1 percent level. indicating that for improvements to be sustained, contin- 11. For exits FY95 the number of projects and the level of dis- bursements increased in medium-performing sector investment. This was accompanied by a corresponding decrease in low-performing sec- quality at entry and project supervision, is critical. tors compared to FY85-93, but this pattern does not hold for exit Risk-reward analysis suggests that the Bank could FY96 or FY97. explore opportunities for increasing the rewards of border- 12. The five country groupings are: high performing, upper- line prospects, raising the mean reward of selected country medium performing, lower-medium performing, low performing, and or sector portfolios with low variance and, above all, poor performing. Selecting countries with more than 7 projects for increasing the mean reward Bankwide without substan- the years covered by the Country Economic Performance Index left a tially increasing the variance by expanding the share of sample of 67 countries for this exercise. 13. A statistically significant difference was found at the 5 per- cent level when comparing average borrower performance for exit FY96-97 with exit FY90-95. 14. The average was just 46 percent for FY87-91. 1. See Annex 4 of the Annual Review of Evaluation Results 1995 15. The difference in ratings between exit FY96 and exit for a discussion of the different approaches to long-term trend analysis. FY90-95 is statistically significant at the 1 percent level. Also note that the data presented in this chapter will not be compara- 16. Among the 197 projects in FY96 with high or substantial bor- ble to the data presented in past Annual Reviews of Evaluation Results. rower involvement in project design, 79 percent had a satisfactory out- 2. All performance data reported for FY97 are based on 43 per- come rating. Of the 62 projects with modest or negligible borrower cent coverage of the exits for that year. involvement, only 42 percent had a satisfactory outcome rating. The 3. The improvement in the outcome rating for FY96-97 over the difference is statistically significant at the 5 percent level. average rating for FY90-95 is statistically significant at the 5 percent 17. Effective Implementation: Key to Development Impact, level. Tests of statistical significance are generally applied to samples Portfolio Management Task Force, October 5, 1992. of data. Although the universe of evaluated exits for each year (except 18. To assess reward, all projects in the FY90-97 evaluated port- FY97) is available, each year is treated as a sample and a t-test is folio (1,625 projects) were graded on a scale of 0 to 10 depending on applied to gauge the degree of difference between two periods. whether their outcome rating was unsatisfactory or satisfactory. The 4. Of the 21 large projects, India had the largest share, followed base reward is a grade of 0.5 or 2 for unsatisfactory projects and 4 or by Indonesia and Mexico. 5.5 for satisfactory projects. Each project is then given an additional 5. Water supply and sanitation was identified as the sector with bonus from 0 to 2 depending on its institutional development impact most projects at risk in the 1997 ARPP. and a 33 percent premium (penalty) if sustainability is rated as likely 6. With more than 10 projects exiting the portfolio in FY96. (unlikely). 19. This measure is conceptually different from the risk indices 7. Annual Report on Portfolio Performance, Fiscal Year 1997 presented in the Annual Review of Evaluation Results 1995, which were based on subjective judgments of project design and the riskiness 8. A statistically significant difference at the 5 percent level, of the project environment-complexity, riskiness, demandingness. 38 ADDING UP THE LESSONS COUNTRY AND SECTOR REVIEWS A s the earlier chapters have noted, scaling up need for ongoing improvements in Bankwide country evaluation lessons to the sector and country assistance. level places new demands on evaluation work in the Bank. It has also been noted that Lesson 1: Good policy matters for the overall effectiveness of much of the independent evaluation work Bank assistance, but policy reforms rarely succeed unless the being carried out at the country or sector level is still rela- government is genuinely convinced that the reforms have to tively new. This chapter draws on some of the early lessons be implemented and considers the reform program its own. of this work, together with avail- able lessons from self-evaluation Bank assistance in Poland during the late 1980s and early C n studies available for some sectors 1990s was particularly well matched with country condi- and themes. Many of the lessons tions. After Poland became a Bank member in 1986, the and findings are fairly well first country strategy paper proposed that lending be timed known, but the purpose is to syn- to match the government's progress with its reform pro- thesize what has or is being gram and its progress in restoring creditworthiness. This learned about improving develop- led to an initially cautious strategy, which saw the Bank a listcment effectiveness, and to exam- relying mainly on studies and technical advice to the gov- ine how this information is being ernment. Full-scale lending did not start until 1990. While evialat Lon o used to improve results on the a case could have been made for earlier selective lending, ground.1 the delay until 1990 appeared to have encouraged Polish authorities to make a stronger commitment to reform. Country asssnce reiews Thus, by waiting until international and domestic climates str eitegy OED has completed 7 Country were more favorable, the Bank began lending when the Assistance Reviews (CARs) and ratio of opportunities to risks was high. has 10 more in its work program In contrast, the Bank overestimated the willingness and for FY98 and FY99. The completed CARs have proved to capacity of the Zambian government to implement be valuable instruments for discussing the preparations for reforms. During the 1980s the Bank's assistance strategy in new or updated Country Assistance Strategies (CASs) with Zambia focused almost exclusively on short-term stabiliza- country teams and for providing information to the Board tion objectives that remained elusive and were out of step before it considers a new or updated CAS. While the find- with the country's long-term development needs. In this cli- ings and lessons from the CARs are tailored to fit individ- mate-characterized by a poor policy environment and ual country experiences, four common lessons stress the weak institutional capability-Zambia's consistently dete- 39 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS riorating economic situation reflected the limited impact of of the problems facing policymakers, it was not well Bank assistance. In the early 1990s, with a new Zambian sequenced with the pace of economic reform and often government publicly announcing its commitment to eco- lagged behind the in-depth reforms undertaken by govern- nomic reform, the Bank's program helped reduce the fiscal ment. Much of the better nonlending work resulted from deficit and inflation, yet economic policy reforms are still studies financed through project preparation or took place unfinished and growth has not yet resumed. in informal discussions and policy dialogue. The former was essential to the successful privatization effort initiated Lesson 2: The efficacy of Bank assistance and the credibil- by the government of Argentina in 1989. ity of its conditionality are diminished when multiple and In Zambia, despite the government's poor performance not mutually reinforcing objectives are promoted under a record, the Bank's analytical work has proved critical in single lending operation. A judicious degree of unbundling supporting key policy changes in recent years. ESW gave and sequencing in the program of Bank interventions is priority attention to liberalization, preferable. which has been successfully imple- mented. The informal policy dia- uiidingb Resource transfer objectives for macroeconomic purposes logue that ESW made possible also often coexist with sectoral objectives. The former respond contributed significantly to adjust- develcoment to a pressing need that in turn is an ments in the policy environment imperative and a prerequisite for for industry and agriculture. The synergy{ between the latter. However, the attainment Bank has done several Public u of sectoral objectives is unlikely to Expenditure Reviews, which have nonlending and S .occur in the same short period as helped to bring the public sector S-notattainment of the macroeconomic deficit under control. Furthermore, lending services objectives. The results are pre- the 1994 Poverty Assessment was dictable. If progress is good on the critical in identifying short-term is key - e macroeconomic front, lending will actions to assist vulnerable groups proceed regardless of whether con- and is being used in the design of a ditions are met or not met on the public policy to mitigate and reduce poverty. An innovative sectoral front. For example, in approach used in Poland to link analytical work with pro- Argentina, the agricultural objec- gram design is illustrated in Box 4.1. tives of the FY86 sectoral adjust- macroeco c ment loan (SAL) took a back seat Lesson 4: The effectiveness of the Bank's aid coordination m croonmaics.. to economic stabilization. Simi- efforts depends not just on the extent to which other anild sector rCFoM larly, in the human resources SAL donors are willing to act together but on the government to C6te d'Ivoire, sector objectives reaching consensus on the appropriate institutional frame- were not met despite improvements work for aid coordination. In-country aid coordination in the overall macroeconomic framework. Bundling macro- and management by client countries is as important as economic and sector objectives together has only proved coordination among donors. efficacious when sector reform is technically ready to start and its objectives under the fast-disbursing loan are pitched The final lesson relates to the Bank's effectiveness in mobi- as realistic milestones in a longer process of sector reform. lizing and coordinating development assistance. In Poland the Bank took an early leadership role to ensure good coor- Lesson 3: Nonlending services clearly need to be more closely dination, but efforts were often frustrated by the Polish gov- linked with program and project preparation. This does not ernment's lack of consensus on the appropriate framework mean matching lending and nonlending services dollar for and on which organizations would fulfill which tasks. dollar, but it does mean building greater development synergy While the Bank has continued to support coordination at between advisory, analytical, and lending services. the general level, there is growing recognition that alterna- tive mechanisms are needed at the sectoral and project level. The Argentina CAR observes that, while early economic In Ghana, the Bank has played a leading and highly and sector work (ESW) was mostly correct in its diagnosis effective role in helping the government mobilize large 40 Adding up the lessons-country and sector reviews BOX 4.1: POLAND'S AGRICULTURE TASK FORCE-A STRATEGIC PARTNERSHIP FOR PROGRAM DESIGN n April 1990 the Bank's Bank, and the European for the government's a widespread understand- regional management Union (EU). The chairman Medium-term Sector ing of and identification and the government of was the undersecretary of Adjustment Program. The with the proposed strategy. Poland established three state for agriculture, and the cost of the exercise was It also helped pave the way task forces to prepare task force leader was the about $1 million, 90 per- for extensive EU assistance reform strategies for the senior operations advisor in cent shared by the Bank and to Polish agriculture and agriculture, infrastructure, the regional vice president's the EU and 10 percent pro- cofinancing of future and health sectors. The office. Of the 57 profes- vided by the government. Bank-supported opera- planning, fieldwork, and sional members, 14 were Two factors were key tions. Second, the personal responsibility of the agri- Bank staff, 34 were Polish to the success of the task involvement of the regional cultural task force pro- professionals, and 9 were force. First, conducting the vice president and the min- ceeded with extraordinary consultants. The task force exercise as a joint under- ister of agriculture gave the speed. The two other task submitted its report, "An taking with intensive par- exercise status and author- forces were less successful. Agricultural Strategy for ticipation from Polish pro- ity, both in the Bank and The agricultural task Poland," in just three fessionals and institutions, in Poland. This freed the force involved a joint under- months. After acceptance and with the EU as an process from unnecessary standing between the gov- by the Council of Ministers important donor, helped bureaucratic and adminis- ernment of Poland, the the report became the basis build consensus and led to trative constraints. amounts of external assistance. The Bank has also had some Thama`c mvalsiz ns success in fostering communication and coordination among donors using its Consultative Group framework. Poverty reduction Nevertheless, Ghana continues to have an aid coordination Assessing the impact of the Bank's work on poverty reduction problem. The high volumes of aid and large numbers of is the most critical challenge in measuring development effec- donors and donor-financed projects create a need for many tiveness. Measurement and attribution problems also make it kinds of coordination. These include getting consistency in the most difficult challenge. Since the basic national and sectoral objectives, getting donors to World Development Report 1990, concentrate on those activities and projects they do best, the Bank has devoted significant T and harmonizing disbursement procedures. The main prob- resources to defining and measuring er s siii lem seems to lie with the government's own limited role in poverty and has made available a . e aid coordination. Government action is needed to clarify large volume of lending specifically a consieable national and sectoral objectives, prioritize projects and pro- for poverty reduction.2 Neverthe- grams accordingly, and take the lead in setting an appropri- less, lessons about the impact of Vay to go in ate institutional framework for development assistance. Bank assistance on poverty levels A message across all the CARs is that the CAS has are still limited.3 As a result, there measurig the scope to define more clearly how the Bank sees its compar- is little new to report about the H_ f Bon ative advantages evolving vis-a-vis other major donors, and Bank's effectiveness in this area. ways to minimize donor competition and improve coordi- Recent analysis of global nation in cost-effective ways. The Bank could also identify poverty trends (World Bank assistance on ways to assist partners, whether major nongovernmental 1997a) suggests that, since the late e organizations or other donors, to improve the effectiveness 1980s, there has been a small but of their combined assistance over time. This might include important decrease in the propor- cofinancing arrangements, common analytical or study tion of the world's population that are poor. Box 4.2 teams, or coevaluations of the development effectiveness of reports some of the most recent evidence on regional assistance by sector or countrywide. progress in reducing poverty according to the three main 41 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS B~ 4.2: PROGRESS IN REDUCING POVERTY ecent data examin- headcount index (percent- poor). The regional break- Eastern Europe and Central ing changes in the age of population con- down indicates a fall in Asia, Latin America, and absolute value of suming less than $1 a day) poverty for East Asia, the sub-Saharan Africa. South the poverty line across and the poverty gap index Middle East and North Asia has the highest overall countries show a small (average distance in cents Africa, and South Asia poverty incidence, while drop in aggregate poverty below $1 a day, when aver- (with signs of a slight rever- Sub-Saharan Africa contin- between 1987 and 1993. aged over the whole popu- sal from 1990-93) and ues to have the greatest This holds for both the lation, with 0 for the non- increases in poverty for depth of poverty. PERCENTAGE OF NEAN PO RTY GAP POPL IATION ONsLMING [OX I RT) C- I P INDEX OF THE POOK LESS TH IN s I. DA. PERCENT iCFNTSI EAP '7 2 I ''I3.'2 2.I ECA '1j-'', I . LAC 22.1 S3.. 23 '.'_ 3 MNA 4.- 4.3 4.1 -1~ '1' '''j IS 5 I i.- SAS 4i.4 4 ) 43.1 14.1 12 3 1.' 311N'. SSA IN .5 3133~. 14 4 14 1.31. Total SI.-->. . '. i. . Total excluding ECA 3.9 32., i ** 11'.1 ''.3 l. 31. 31 ' .' - not available. Source: Ravallion and Chen 1997 Note: Poverty measures are population-weighted means over all countries in the data set within each Region. Survey dates differ. measures of poverty used by the Bank. Almost all Regions importance of improvements in basic social and urban show some improvement in the percentage of the popula- infrastructure, and participation in local decisionmaking, tion living below the poverty line, except for Eastern for improving the living standards of the poor (Box 4.3). Europe and Central Asia, where, because of the transition A review of experience with social assistance and process, the increase is significant.4 poverty-targeted programs (Subbaro, et al. 1997) also pro- Few studies can directly link Bank assistance with vides some important lessons on improving the impact of overall changes in poverty levels. Selected experience the Bank's targeted interventions. The overall assessment is nevertheless points to the relevance of the Bank's strategy that income growth or social assistance accrues mostly to for poverty reduction set out in the 1990 World Devel- the nonpoor when targeting is imprecise or inappropriate. opment Report. Household data for Vietnam, for exam- Transactions costs must also be kept down to maintain the ple, show that the percentage of the population living net value of transfers (especially in credit programs) and to below the poverty line has fallen by one-third following avoid increasing the costs of transfer programs through a sustained period of broad-based growth since 1984. A incentives for the nonpoor to participate. The study also detailed study of poverty levels in India (Datt and Raval- concludes that when rapid labor-demanding growth and lion 1996) indicates that disparities in the extent of human resource investments are not pursued vigorously, poverty reduction across states are linked to differing social assistance programs may have limited impact on success levels in promoting rural growth and human poverty even when they are well designed. While conveying development.5 A recent impact evaluation of five urban some sense of the World Bank's efforts to reduce poverty, operations in Brazil (OED 1997b) also underscores the these lessons do not report anything about the impact of 42 Adding up the lessons-country and sector reviews BH 4.3: BRAZIL-BEST PRACTICE IN FIVE URBAN PROJECTS n OED impact reduced atmospheric dust, Besides the direct tance to community-based evaluation of five property values in the impact on living condi- organizations (CBOs) in ii Lurban operations in neighborhoods increased tions the projects sup- the eyes of local residents. Brazil (OED 1997b) shows substantially. Proper ported the acquisition of In contrast with the poor the significant difference drainage eliminated accu- greater skills in project in Indonesian kampungs, that urban improvement mulation of polluted water preparation, appraisal, and they showed little interest schemes can have on the in ditches, a particularly evaluation by local offi- in self-help schemes, there- quality of life for the urban serious health hazard for cials. The projects also fore they appreciated the poor. The five operations children before the pro- unintentionally allowed CBOs' help in ensuring improved living conditions jects. Prevention works- municipalities a part in that urban improvements for some 2 million urban especially the Northeast decentralized urban service were made according to poor in 464 municipalities Flood Reconstruction Pro- provision when the federal their interests. This effect throughout the country. ject-helped diminish the government withdrew was most strongly felt by Better paving allowed risk of flood waters invad- from this function. The women and in medium- improved access and ing homes and property. projects gave new impor- sized cities. targeted lending on the poor or anything about the transfer gender goals, with definite targets supported by a consis- actually realized by the poor. tent message to borrowers that gender is critical for devel- The Bank has considerable knowledge and best prac- opment effectiveness and an inte- tice experience on how to design and implement poverty- gral part of the Bank's country targeted interventions and on the contribution of broad- and sector focus. More system- v based growth to poverty reduction. To ensure that the atic and monitorable indicators Bank continues to make the maximum possible contribu- for gender-related actions are tion to the reduction of poverty, it is critical to continue also essential for improving enhancing the poverty focus of country assistance strategies assessments of the Bank's effec- and to develop more effective indicators of the impact of tiveness in this area. An example Bank targeted interventions and of progress toward achiev- of how such targets can be used ing country poverty reduction goals. to monitor and evaluate progress is described in Box 4.4. Gender Achieving poverty reduction through economically sound, Environmental sustainability socially equitable, and environmentally sustainable growth Over the past decade environ- requires that women be full beneficiaries of development mental issues have become a g -" s activities and equal partners in the planning and implemen- major thrust of Bank activity. tation of those activities. A 1997 OED study showed for the This recognizes the growing first time that the outcome ratings for projects with gender- understanding and acceptance that environmental sustain- related action approved in or after FY87 compare favorably ability is a key pillar in the goal of sustainable develop- with the outcome for similar projects without gender ment. Accordingly, the Bank has been trying to learn the actions.6 The study also showed that the integration of gen- most effective ways to tackle environmental issues. der objectives into overall project objectives is the most sig- In support of environmentally sustainable development nificant positive correlate with satisfactory gender outcomes, the Bank pursues targeted programs, including specific pro- Nevertheless, both the self-evaluation of gender con- jects, regional and global efforts, and strategies and poli- cerns and the inclusion of gender concerns in sectoral, cies. Experience to date points to the need for a balanced country, or topical self-evaluation remain very limited. approach, combining specific measures and investments to What is required is a well-articulated strategy for achieving address environmental problems within an appropriate 43 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BO 4.4: THE EQUITY AND SCHOOL IMPROVEMENT PROJECT, GUINEA 9" he National Educa- School Improvement Pro- student learning capacity percent; the original target tion for All Pro- ject, initiated in 1995, is through a school-based for the entire period was gram adopted by financing several compo- micronutrient and de- 37 percent. the government of Guinea nents of this integrated worming program. To The early success of in 1990 aimed to expand sector program. strengthen education sys- this project can be attrib- the gross primary school The need to address tem management, the pro- uted to several factors. The enrollment rate from 28 poverty and gender issues ject aims to create capacity project has been character- percent to 53 percent by significantly informed the to monitor student learn- ized by an unwavering the year 2000, increase the design of the project. All ing and to improve sector focus on results and the efficiency of resource use project activities were planning, budgeting, and flexible adaptation of within the sector, increase designed to have a favor- monitoring. processes as needed. Long- government support for able impact on the school- The five-year project is term goals were spelled out basic education, and ing of girls. To achieve its in its second year of imple- in advance and referred to improve the quality of objectives the project seeks mentation, and already constantly to ensure that education. These objec- to increase the availability results are beginning to project goals were widely tives were embodied in the of schools, improve their show. Gross primary shared. Finally, the project Education Sector Policy, management and mainte- enrollment rose from 40 enjoyed high levels of ben- currently being supported nance by communities, to 44 percent during the eficiary commitment, by a multidonor sector increase the volume of first year and currently building on what the gov- investment program. The learning and teaching stands at 50 percent. Girls' ernment and the parents World Bank Equity and materials, and increase enrollment has reached 36 believe in and want to do. policy and institutional environment (Box 4.5). The fol- time. Initially, Bank activities focused on affecting demo- lowing are important for this approach: graphic outcomes to allay the negative consequences of * Environmental objectives should be undertaken as rapid population growth. Starting part of a broader effort to address priority goals. in the mid-1980s, HNP research * Market-based instruments usually are more effective and policy work in the Bank Lessons from than regulatory instruments in managing environ- increasingly focused on issues in mental problems. Where regulatory methods are nec- health financing and a more pre- essary, however, particular attention should be paid scriptive concern with cost-effec- to designing approaches that economize on adminis- tiveness. Most recently, the focus are still not fully trative capacity. has shifted to a broader set of con- * The participation of those likely to be affected by cerns, covering essentially all internalized in environmental measures or policies is critical for pro- aspects of health system perfor- gram success. New investments and policies should mance under the rubric of health the sector be reviewed for environmental impact at the design sector reform. (formulation) stage so that appropriate measures can The sector strategy prepared be incorporated from the beginning. in 1997 for HNP embraces the sector reform approach and sets out three main goals for improving effectiveness in the Sectoral evaluations sector: improved health outcomes for the poor, improved health system performance, and improved sustainability in Health, nutrition, and population health care financing. The strategy is an important founda- The focus of the Bank's lending and nonlending services in tion upon which the Bank can build to increase its impact health, nutrition, and population (HNP) has evolved over and measure its effectiveness. But there is still some way to 44 Adding up the lessons-country and sector reviews employ an open, participatory process involving a BOX 4.5: PLACING THE ENVIRONMENT AT THE range of stakeholders. Institutional considerations CENTER OF THE POLICY DIALOGUE should be explicit elements in reform deliberations. *Planning and sequencing changes. There is no ]he Bank's early warming funded through known ideal sequence for introducing reforms. Country Assistance the Global Environment Accordingly, the Bank needs to be pragmatic in pur- Strategies for Poland Fund. The Bank insisted suing reform objectives, recognizing that there will identified environmental that all had to comply be constraints to pursuing a "rational" sequence of deterioration as an impor- with strict guidelines. reforms. In addition, the Bank can help systematize tant structural issue. The Six years later, the what is already known and buttress weak Ministry program that emerged in level of pollution has been of Health capacities by involving nongovernmental 1990-91 worked on the reduced. The government organizations and other partners in the reform issue in various ways: has instituted new national process. elaboration of a national emission standards for * Recognize regional variation. The Bank also will environmental plan and air pollution, improved need to recognize that the reform process (and strengthening the Ministry impact assessment guide- needs) differs across regions and countries. This sug- of Environment through lines and monitoring gests that the Bank may need to employ different the Environmental systems for air and water mixes of lending and nonlending activities to Management Program; quality, and set high pol- respond to specific country challenges. The Bank integration of environ- lution fees. Market forces will need to match the size and complexity of lend- mental issues into the working in the new insti- ing operations to particular policy and institutional economic transformation tutional environment led conditions. program through the to the closure of some of SAL; and investments the most polluting plants. Education in addressing major The Ministry of Environ- The Bank's most recent synthesis of its agenda for the edu- sources of pollution such ment has also established cation sector, Priorities and Strategies for Education as energy, mining, urban effective mechanisms (World Bank 1995), provided a broad foundation for sec- infrastructure, those for funding investments tor improvement that was also identified in the Baltic and coordinating foreign expected to help measure effective- Sea Environment Pro- aid, amounts reaching ness. The strategy shifted the focus Tre is gram, and projects on $230 million by the end and approach of the Bank's work biodiversity and global of 1993. to a broad sectoral approach to systemic reform in education tai- mounting lored to specific country contexts; e a better balance between public go before the lessons of past approaches are fully internal- and private sectors in education ized in the strategy, and before it is possible to systemati- systems; decentralization of educa- to get good cally identify and measure changes in the Bank's effective- tional financing and management; results in ness in this area. more effective schools; and better An ongoing OED study7 indicates that the sector student learning outcomes. reform approach continues to be hampered by the poor Two years later, however, definition and monitoring of institutional changes, limited while there is considerable evalua- or fragile borrower ownership, and differences in commit- tion activity across the Bank, it still does not add up to a ment and policy opinion among key stakeholders. Lessons program that assesses specific results or to a broader syn- that are considered important for development effective- thesis of the Bank's effectiveness in the education sector.8 ness in health sector reform include: Nevertheless, some preliminary findings (Box 4.6) show * Creating a vision and achieving consensus. Creating an encouraging shift toward a focus on educational a vision of a desired future health system and the impacts and results as yardsticks for measuring program plans for how to achieve these reforms needs to effectiveness. 45 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS D,,W1 GETTING RESULTS IN EDUCATION series of impact expanding private schools of public fund use and to Communities. An experi- studies on decen- in low-income neighbor- mobilize local resources. mental, randomized-con- Ltralization and pri- hoods in Lahore. The The program decentralizes trol impact study in the vatization in five countries results show that the management in public Philippines aims to guide may be the Bank's most schooling choices of poor schools. The impact evalu- policymakers in improving ambitious effort to evalu- households are sensitive to ation examines how direc- elementary school out- ate the effectiveness of government and private tors, teachers, and parents comes in low-income reforms it supports in the school fees, distance to are managing the schools areas. The study measures education sector. The stud- school, and school quality. and whether, therefore, the the effectiveness of various ies cover Bank-financed Two significant policy find- schools increase enroll- targeted interventions to projects supporting five ings emerge. One is that ments and improve student promote student atten- programs: the Balochistan lowering private school learning. Preliminary find- dance and learning. It has Primary Education De- fees or reducing distance ings show that the par- produced tentative policy velopment program in increases private school ticipants managing the guidance indicating that Pakistan, the School enrollments of poor chil- autonomous schools multilevel learning materi- Autonomy program in dren, including girls and believe the reform has als combined with parent- Nicaragua, the Education those who would not have already led to such benefits teacher partnership activi- Planning and Rehabilita- gone to school otherwise. as improved student per- ties have more positive tion project in Tanzania, The other is that private formance, teacher atten- impact in preventing stu- basic education in El Sal- schools raise achievement dance, and parental partic- dent dropout and promot- vador, and the Secondary in mathematics and lan- ipation. They also believe ing first-graders' learning Education project in guage more than govern- they have more control than either of these inputs Colombia. Preliminary ment schools. over school resources and alone or school feeding results are available. Participatory Manage- personnel decisions than programs. Policymakers Private Schools. The ment to Improve Schools' they would in traditional have subsequently used Pakistan study explores the Cost-Effectiveness. In schools. these findings to formulate impact on enrollments and Nicaragua, a major reform Improving Primary reform strategies for ele- student achievement of aims to raise the efficiency School Outcomes in Poor mentary education. Financial sector of well-functioning financial systems. Largely as a result of During the past 15 years the world has experienced one of World Bank research, the development of financial markets the more volatile periods in financial history. Banking crises and institutions is generally seen to be a critical and inex- in more than 80 countries have created costly financial prob- tricable part of the growth process, and financial stability lems throughout the world, but particularly in developing is an essential part of sustainable development. countries. This period has also witnessed the emergence of Findings of recent analytic work in the Bank have new financial instruments, new types of institutions, and broad implications for Bank policy. One of the most impor- even completely revamped financial systems. Private interna- tant is that Bank financial sector operations cannot be tional capital flows have also grown at unprecedented rates. viewed in isolation. Rather, they should be seen as part of Against this background, Bank support for the financial sec- a long-term integrated process. The Bank's perspective tor has become more complex and demanding, and in recent must therefore be similarly long-term and focused on build- years the performance of the sector has plummeted from ing institutional capacity and capability so that banking being one of the best to one of the worst.9 system insolvency can be avoided and incentives for pru- With the changed environment has come a change in dent intermediation can be created in a manner consistent perspective on the importance to the development process with overall development policy. These analyses call for 46 Adding up the lessons-country and sector reviews new institution-building instruments and relatively smaller * Successful privatization requires a variety of comple- loans that are intensive of staff time and have longer dis- mentary actions that do not overburden the reform bursement profiles. agenda. These include spe- Lessons from evaluation material indicate the need for cific demonopolization mea- a strategic view of the financial sector that links early pol- sures, especially in domestic SeIf!evaIuation icy dialogue, other reform efforts and external trade and trans- (such as restructuring of state- port, improvements in com- T e Bank's owned enterprises), and macroeco- pany and commercial law, nomic and other risks to the and adaptation and modern- ny w a reform effort.10 The experience of ization of business account- dealing with financial crises in the ing and financial sector essons on late 1980s and early 1990s pro- reforms. vided some important lessons for Private sector involvement in pfivatization atcention2 on te future Bank assistance in this area infrastructure has been a growing (Box 4.7). area of Bank concern and is the Prvaetopic of separate self-evaluations. A recent self evaluation Private sector development and of Bankwide experience synthesizes some important privatization lessons including the following: Bank involvement in supporting * Constraints to private sector development vary by appT00ro o private sector development and the infrastructure sector. For example, in water supply subset of privatization and post- and sanitation there are social concerns about afford- e - ~ privatization activities has focused ability and health, important externalities, revenues on reducing barriers to competi- are in local currency, and the scope for competition Vo tion and streamlining regulation, is limited for transmission and distribution. strengthening legal and judicial * Private involvement increases as country risk systems, supporting entrepreneur- decreases (except for telecommunications, which is ial development, and promoting global integration through seen as worthy of attention almost everywhere). foreign direct investment and the expansion of trade. Over * Governments must carefully manage their exposure to the years, the Bank has made innovative use of the full private infrastructure projects. Governments in some range of available instruments to promote private sector developing countries have borne extensive residual development, including adjustment and investment opera- risk. The Philippines, for example, has covered risks tions, investment guarantees, policy advice, and economic related to the availability and price of fuels for inde- and sector work. These efforts have generally been fruitful pendent power producers, and Mexico and Columbia as judged by the improving economic performance of the have covered highway traffic demand risks. While countries that are fostering private sector development. such practices are acceptable for a short transition Self-evaluation of Bank experience in privatization period, their continuation threatens to vitiate private reveals some critical issues and policy challenges for the sector development efficiency benefits and to present Bank and its clients: future governments with large financial liabilities, * Governance problems are a serious impediment to pri- which they would do well to monitor and evaluate. vatization. Finding solutions to these problems, and giv- ing greater weight to political and social considerations, 0 a a s should be considered integral to privatization work. A consistent theme in the lessons from country, sector, and * Macroeconomic conditions matter, as do initial thematic evaluations is the importance of supporting the structural conditions. Lack of fiscal discipline leads domestic capacity of the borrower and carefully matching to widely fluctuating key prices, which discourage the design and implementation requirements of projects investment and vitiate the potential gains from priva- with local institutional capabilities. Ensuring that effective tization. Moreover, the mode and speed of privatiza- stakeholder arrangements are in place is also essential to tion must take into account the extent and degree of building the necessary consensus and ownership for sector public ownership and control. and country-level reform efforts. 47 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BOX 4.7: LESSONS FROM FINANCIAL CRISES-TWO COUNTRY CASES P he transition to deficit. In 1994, crisis hit. stand shocks. The most and weak economic perfor- greater financial Capital inflows slowed in glaring defect was a weak mance. In 1994, a banking integration involves response to rising U.S. prudential and supervision crisis hit. significant risks for the interest rates and expan- system for newly-privatized banking sector and the sionary fiscal and credit banks. Bank Assistance. The economy when it is con- policies. The relative vul- FSAL implemented phase ducted in an environment nerability of the Mexican A second Bank response: one of a comprehensive where banks themselves economy and its financial FSRL (1995-96). The financial sector reform are in poor financial health system contributed heavily Bank's prompt involvement program, but its design did and are inadequately super- to the magnitude of the in the international rescue not match the severity of vised and where there are crisis. effort almost certainly the problems facing the larger macroeconomic helped calm markets and financial sector. Interest imbalances. Viewing finan- A first Bank response: limit the damage done by rate liberalization was part cial sector reform efforts PSAL (1989-93). The the crisis to the financial of a program to reduce within a larger framework FSAL was assembled system. The FSRL, unlike inflation and prevent capi- of structural reforms is quickly and with no prior the FSAL, was informed by tal outflows, but it was therefore vital to success ESW in the Mexican finan- a better understanding of done before satisfactory and sustainability. cial sector. As a result, the the Mexican financial sys- banking regulatory mecha- quality of project prepara- tem and focused on the nisms were in place. The Mexico tion and design was poor. sources of system fragility financial system faced dis- Background. The external The project was too com- at the micro and institu- tressed borrowing and was debt crisis of 1982 led to plex and lacked focus, tional levels. Implementa- oligopolistic in structure economic stagnation and including elements such tion is ongoing with the with many close links triple-digit inflation, which as tax administration support of a technical between major borrowers continued until 1987. The reforms and state and local assistance loan. The bank- and banks. Financial sector Mexican response, a rela- finances, which were not ing sector is still very frag- insolvencies started in tively high-risk economic directly related to the ile, however, and it may 1988 and eventually led to strategy, appeared to work financial sector. The sus- take some time for the pos- a crisis in 1994. The FSAL for several years: foreign tainability of the FSAL was itive effects of the FSRL to was rated as unsatisfactory capital inflows surged and also undermined by a com- manifest themselves fully. because it did not deal easily covered the current bination of external and with the solvency problems account deficit up to 1993. domestic shocks and gov- Venezuela with the speed required by Nevertheless, inflows sup- ernment policies that even- Background. Venezuela the diagnosis. The FSAI's ported a consumption tually led to the crisis in enjoyed the highest per outcome was also nega- boom, implying a sharp late 1994. The financial capita income in Latin tively affected by macro- reduction in private sector reforms were neither America during the early economic conditions and domestic savings. The comprehensive enough nor 1970s. Following the 1973 macroeconomic manage- exchange rate appreciated deep enough to strengthen oil boom, however, rev- ment that deteriorated sub- leading to a large expan- the financial system to a enues were mismanaged stantially between 1992 sion in the current account point where it could with- leading to price instability and 1993. 48 Adding up the lessons-country and sector reviews A review of evaluations available at the country, the- but not the impact on poverty levels. The poverty profiles defined in matic, and sector level shows uneven progress on defining the context of Poverty Assessments have the potential to provide a goals for assessing and improving baseline for monitoring poverty impact, but a recent OED Review of g ot ePoverty Assessments (1996) found that 22 of 43 assessments reviewed had deficient or missing poverty profiles and monitoring G reater ficui o result there is currently no simple indicators. The need to develop a more consistent approach to col- way of adding up the Bank's lecting baseline data and to developing cost-efficient monitoring indi- Fesults f r om the progress toward sector and strate- cators for tracking progress on poverty reduction is an urgent chal- gic goals. To achieve this requires a lenge for the Bank. more results-oriented performance 4. As a result of transition, GDP has declined in most ECA coun- eout to tries. However, while the increase in the proportion of poor people is t ssignificant, the level of poverty is still very low compared to other the sector and atically links progress on results at developing regions. the project and CAS level, with 5. In Punjab-Haryana rural economic growth has been a major country level is progress toward sector and Net- factor in substantially reducing the poverty rate since 1960, while in work goals and the Bank's corpo- Kerala, human resource development, supported by appropriate d rate planning and reporting sys- public spending, has been the key to reductions in poverty. In Bihar, tems (see Chapter 1). Current however, the state has failed to promote either growth or human work by Operations and the Net- resource development. Hence there is no evidence of a decline in workior b erins andth, nt poverty levels. works in cultivating a climate for learning, monitoring, and 6. The study (Murphy 1997) found that in agriculture and self-evaluation is a critical step in this direction. human resources projects approved after FY87, 74 percent of 54 pro- jects with gender-related action were rated as satisfactory compared aiss with 65 percent for 81 projects that did not include gender-related 1. An attempt is made to review lessons and findings for the action. major thematic areas, but only a selection of sectors is included. 7. The evaluation strategy for this study is described in Evaluat- 2. By early 1997, Poverty Assessments had been carried out in 43 ing Health Projects: Lessons from the Literature (Stout, et al. 1997). countries and Living Standards Measurement Surveys were available 8. The preparation of an education sector strategy is intended to in 18 countries. The annual Progress Report on Poverty Reduction address this shortcoming. Two in-depth reviews in the sector have for FY96 shows that approximately $5.4 billion, or 32 percent, of been undertaken by the Quality Assurance Group. QAG identifies World Bank investment lending that year was channeled to direct gaps in risk analysis and in mechanisms for monitoring and evalua- poverty-targeted projects. For IDA countries poverty-targeted pro- tion of project outcomes as important shortcomings in the sector. jects were 63 percent of all FY96 investment lending. The bulk of this 9. A QAG review of financial intermediary loans was under- lending was in four sectors: health, nutrition, and population; educa- taken in 1997 because the financial intermediary loan (FIL) portfolio tion; and social sectors. Adjustment operations containing specific had a higher ratio of projects "at risk" than did other sectors. The poverty reduction measures amounted to about $2.23 billion in review recommended actions to be taken to improve the quality and FY96, up from $1.65 billion in FY95, or roughly 59 percent of all supervision of FILs. adjustment operations in FY96. 10. A recent OED review of financial sector policy suggests that 3. At the project level current performance indicators reflect the while the analytical underpinnings are strong, Bank operations have means, process, and ends of achieving poverty reduction objectives only slowly internalized the lessons of financial sector self-assessment. 49 IMPROVING DEVELOPMENT EFFECTIVENESS T he development agenda is becoming increas- management regarding the proposition that ample scope is ingly complex. That complexity is being still available for "doing things right" within the Bank. accompanied by a gradual shift in the rationale As the data in Box 5.1 indicate, sustained quality man- for development assistance: from a focus on agement efforts in the near term, as spelled out in the 1997 financing borrowing countries' physical invest- ARPP, can keep the Bank on course to meet management's ment requirements to helping them achieve financially, envi- goal of 75 percent by the year ronmentally, and socially sustainable development. Experi- 2000. If, in addition, borrower per- ence has shown that development assistance works best formance continues to improve ensitivity tests when it catalyzes or reinforces other factors, such as sound along its recent trajectory, it is pos- policy and a commitment and capacity to reform. The chal- sible to envision an improvement in show that the lenge is to find the right fit between country policy and insti- overall outcome approaching 80 tutional factors and strategies to improve conditions favor- percent. Meaningful retailing of Bank is on course able to improved growth and development. such a target by sector and by The complexity of the Bank's agenda has also region appears to be both necessary to meet the goal increased. This is reflected in the broad range of institu- and desirable. With such a strategic tional and policy objectives being pursued, the increasing plan, the health of the Bank's port- of 75 percent by variety of instruments deployed, and the diversity and dis- folio would be restored to the level persion of the Bank's clients. The Strategic Compact sets it enjoyed in the late 1970s. Of 2000 out a strategy for improving the effectiveness of the Bank course, such a plan is predicated on in managing this agenda and ultimately for improving intensified efforts by all Regions development impact. and Networks to improve the quality at entry, supervision, Review of performance during FY96 and FY97 and the risk management, and monitoring and evaluation. longer-term performance trends finds that progress is being Learning about the Bank's development effectiveness made and that the signs are encouraging. The recent through operations evaluation is central to improving the plateau in the proportion of evaluated operations with sat- Bank's effectiveness. As the development agenda expands, isfactory outcomes appears to be ending. While 71 percent the focus of evaluation also needs to adapt. OED's Strate- of operations in FY96 were judged satisfactory, partial gic Framework for Operations Evaluation aims to reshape results for FY97 indicate an increase to 76 percent.1 While independent evaluation in the Bank to make it a more rel- progress has clearly been substantial, it is premature to evant, timely factor in the pursuit of development effec- ascribe long-term significance to the improvements mea- tiveness. The framework is rooted in the changing nature of sured in FY97. There is no disagreement between OED and development assistance and in the Bank's shift toward full- 51 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS OX 5.1: SENSITIVITY TESTS OF PORTFOLIO PERFORMANCE - or this review, OED and quality of supervision. age) to 73 percent (Figure tory portfolio performance tested the sensitivity In the project sample ana- 5.1). If the quality of super- by FY00. of recent portfolio lyzed, the average rating for vision alone improved in An aggressive ap- performance to changes identification was 83 per- line with ARPP targets (to proach, championed by the in the key determinants. cent for all operations, 62 75 percent by the end of Bank's president, is to im- While the exercise used percent for appraisal, and FY98), then overall out- prove the quality of Bank the latest evaluated exits 72 percent for supervision. come would increase to 72 appraisal and supervision (FY95-97), project design As Bank performance at percent. While these simu- so that they are 100 percent factors and developments identification is generally lated responses to improve- satisfactory. Although this prevalent in the late 1980s rated very high, there is not ments are not fully additive, target is not likely to be and early 1990s, when much scope for improve- achieving both ARPP tar- reached by the year 2000, most of the sample projects ment. But if the quality of gets simultaneously implies achieving it would mean an were approved, heavily appraisal improved in line an improvement in out- improvement in overall out- influenced the underlying with the current ARPP come to 75 percent satis- comes to around 80 per- behavioral parameters. 1997 targets for quality factory. These results sup- cent. With prudent im- The project perfor- at entry (to 85 percent port the proposition that provements in borrower mance determinants under acceptable or good), then improving Bank targets performance, Bank man- direct Bank control are expected project outcome alone can lead to achieve- agement could reasonably quality at entry (project would improve from 71 ment of management's tar- envision reaching a satisfac- identification and appraisal) percent (the sample aver- get of 75 percent satisfac- tory level of 85 percent. r-1iURE 5.1: COMPARATIVE STATIC ANALYSIS OF IMPROVEMENTS IN PROCESS VARIABLES SHORT-TERM TARGETS LONGER-TERM TARGETS E 73 88 84 780 72 Simulated nulated . Bank Acjtual Supervision Actual Bank Borrower borrower Process variables Process variables Source: OED Data Note: Bank implies simultaneous improvements in both appraisal and supervision performance. Bank + borrower assumes an additional improvement on the borrowers implementation performance. 52 Improving development effectiveness service development. It places greater emphasis on country, sector, and thematic evaluations that reflect the operational BOX 5.2: PUTTING SELECTIVITY INTO PRACTICE: priorities of the Regions and Networks and the goals of the THE POLAND COUNTRY ASSISTANCE STRATEGY Strategic Compact. In addition, the new evaluation frame- work expands to capture the impact of the Bank's non- he 1997 country finance (telecommunica- lending services, both separately and in conjunction with assistance strategy tions, gas) the Bank will lending services, and the performance of the Bank's part- . for Poland takes shift toward a greater nerships and strategic alliances in delivering development into account the evolving emphasis on nonlending assistance. comparative advantages services and will increase of the Bank and of other its intellectual investment Measuring aggregate dev eopment effectiveness potential partners and in emerging areas. On Measures of the effectiveness of development assistance competitors in defining the the other hand, greater indicate that in good policy environments ODA has a pos- scope of Bank activities. emphasis will be given to itive and significant impact on growth. Attempts to repro- The implication of taking areas of lending where the duce this result specifically for Bank lending are hindered a selective approach is not Bank has a comparative by the fact that Bank lending is often directed at policy to limit the Banks pro- advantage, such as the change and capacity building and that at the country level gram to only a small num- social sectors and the and above Bank assistance is just one factor among many ber of areas but, rather, to environment. The pro- affecting development results. Moreover, growth is just one match strategically the posed strategy also incor- measure of effectiveness. With critical attention turning to Banks knowledge, experi- porates joint actions the impact of development assistance on global poverty lev- ence, and resources with within the Bank Group els, the effects of Bank assistance on a wider range of devel- the priorities of the Polish and with other partners opment outcomes need to be examined. government. For example, (especially the European Gaining the specificity needed to measure the relation- where the Bank does not Union and European ship between Bank lending and nonlending activities and have a comparative advan- Bank for Reconstruction development results requires moving to a more manageable tage over either the private and Development) to bet- plane than either the global or country level. At the sector sector or other interna- ter serve the client and or subsector level, the intervening chain of cause-and-effect tional financial institution leverage greater develop- relationships is more readily observed and measured. This and donor sources of ment impact. implies that instead of having one overall performance measure for the Bank there will be several measures that, when taken together, provide a comprehensive assessment of the Bank's progress toward achieving key development In addition, sound risk management is crucial. Building results. This approach is being reflected in the preparation more performance measurement and monitoring into of sector strategies by some Networks and in the recent country assistance is not only cru- efforts to expand and improve self-evaluation from the cial for effective risk management project up to the CAS and ultimately to the Network level. but also for achieving longer-term Se development results. Efforts to Achieving effectiveness through strategic selectivity improve the CAS are under way in . . The analysis of country portfolio performance gives further the Bank, and the Strategic Com- about picking weight to the message that the Bank should exercise greater pact's efforts to simplify lending selectivity in its lending operations. Achieving strategic processes, to shift budget resources winners selectivity, besides requiring continued attention to country and decisionmaking to the front assistance strategies embodying clearer distinctions lines, and to increase use of adaptable lending instruments between countries fully committed to development and are important moves in the right direction. poverty reduction, also means more realistic assessments of performance and institutional constraints and better selec- Borrower performance tion and sequencing of instruments that match existing OED findings continue to point to the vital importance of government capabilities (Box 5.2). borrower performance in project performance. The latest 53 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS results suggest that borrower performance has been ularly its control of corruption, is an important determi- steadily improving in recent years, assisted in part by nant of project success. Global concerns about corruption improved stability in the international economic environ- have intensified in recent years, and evidence that corrup- ment. However, the findings also clearly identify four char- tion undermines development is increasing. To the extent acteristics of borrower performance that the World Bank that a lower probability of project success reduces the and its partners can seek to exert an indirect influence on- prospects for the development impact of Bank lending, the or at a minimum, take explicit account of-in managing findings of this review are directly relevant to these larger their development assistance. concerns. * The locus of the project concept. Project ideas origi- The Bank is concerned with controlling fraud and cor- nating in the country have a greater chance of success ruption in its projects; its procurement and disbursement than project ideas imported wholesale without active procedures have been progressively refined to minimize the borrower involvement. risks for both lenders and borrowers. Nonetheless, there * The intensity of intellectual conviction. Unless the remains a need to address systematically, at the country borrower's readiness to take responsibility for the level, the impact of corruption on the effectiveness of Bank project is ascertained through up-front actions, assistance. Support for national anticorruption measures implementation risks tend to be high. and for international efforts to control corruption are two * Consensus among policymakers. Projects that areas where the Bank is now moving more aggressively. depend on a single champion within the domestic leadership are riskier than projects that have gar- Deveiopment partnerships nered the support of a critical mass of influential The development agenda today is so challenging that part- decisionmakers. nership is vital for improving effectiveness. The evaluation * Participation of stakeholders. Involvement of benefi- findings and lessons show that Bank assistance is most ciaries and other interest groups in civil society tends effective when it builds on collabo- to improve commitment to achieving project goals. ration and partnership with the borrower and with other strategic A clearer results Bank performance interests. Development projects are The analysis carried out on the determinants of project increasingly complex and taxing, focus would be in success points unequivocally to the importance of quality both on borrowers and the Bank. at entry-identification and ap- Successful partnership can im- line ith best r praisal-in explaining project and prove substantially the manage- Efforts to portfolio performance. The quality ment of necessary risks and the practice in the of Bank supervision is another key adoption of flexible strategies to improve quality determinant. The latest perfor- deal with them. Evidence that this mance trends indicate that, is particularly the case in partner- at entry, risk although Bank performance has ships with civil society is growing. been steady, there is still substan- Strategic partnerships with management and tial room for improvement in the other donors are also important percentage of operations judged to for leveraging effective development assistance. The Bank * * have satisfactory Bank perfor- has scope for defining more clearly how it sees its compar- supervision are mance. Continuing efforts to ative advantages evolving vis-i-vis other major donors over ongoing improve quality at entry, to better time and what possibilities it sees to minimize donor com- assess and manage risk, and to petition and improve coordination in cost-effective ways. improve the quality of project The Bank could also identify ways it might work with part- supervision are critical, as are continuing efforts to improve ners-whether major nongovernmental organizations or the quality and monitorability of the CAS. other donors-to improve the effectiveness of their com- bined assistance over time. This might include cofinancing Governance and combating corruption arrangements, common analytical or study teams, or Analysis undertaken for this review shows that the quality coevaluations of the development effectiveness of assis- of a country's economic governance and credibility, partic- tance by sector or countrywide. 54 Improving development effectiveness -T s Can MPem e n line with best practice in the development community. It Under the Strategic Compact, the Bank is reorienting its would connect evaluation and resource management and operations toward more relevant goals, reforming its busi- would be consistent with the strategic framework of evalu- ness processes toward cost-effective results, and reshaping ation endorsed by the Bank's Committee on Development the organization to work through partnerships. This has Effectiveness (CODE). fundamental implications. It implies the need for a results- based system that is transparently connected to the Bank's priorities; clearly distinguishing between inputs, outputs, 1. While these figures are not directly comparable with previous outcomes, and impacts; and monitoring of the full package years' estimates based on evaluation cohort year or closing year, the of interrelated Bank services. Setting up such a manage- longer-term performance trend by fiscal year of exit confirms a steady ment and monitoring and evaluation system would be in increase in the percent satisfactory rate from FY95. 55 ANNEX TABLE 1: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT IMPACT OF EVALUATED OPERATIONS, BY SECTOR, NETWORK, LENDING TYPE/SOURCE, AND REGION, EXIT FISCAL YEARS 1990-95, 1996, AND 1997 (BY PROJECTS) FXIT F) l **'I-'i EMIT I'. E IT F1\ **- OUT II OL T. t Ot- to COME sL-T. INIPA-T COME sUNT IM'3T COMF sL T illi NCT .. .. . .of -/ Sector p . kn. likch .,i'. r.roj. sat /1 ' .. . I 1 Sal' Agriculture 3~ 62 35 28 55 72 IS 48 2 -c, Education lI 50 Sb ' 23 7 2 L. !411 Electric Power & Other Energy 1 .1 o, - 1 15 8 'c5 41l 111 1'' Environment - - - - 2 10) 5ll [ill, liii. I'll) Finance 4 6 42 '" 8 7 i 1.. Industry -iS 2" 8 3s 3121 1 M ining 0i1 ih il 2 10;1 !il ;I 11O' I lan ]1111 Multisector 1i - 2" 23 9 5 14 -1 ' 1 Oil & Gas 4 i S 5 i 4 7i 51 --- - Population, Health & Nutrition 4.' 4" 1u 14 5'' 4 ', 83 1 Public Sector Management S. I ' - 13 61- S4 1 - a Social Sector 83 ' s' 4 75 2 2 Ia' Telecommunications 22 sI S 8 62 s 1 1 I'. Il( 1 Transportation 1 " -4 1 1 1 25 83 4 42 1' M- -I.1 Urban Development CI 6 1" 9 6- 22 - -1 ' 2" Water Supply & Sanitation I1 r 31 'I ' 10 4'. - 'I I 111. l'11 Network Environmentally & Socially Sustainable Development 62 35 ' 57 73 44 4", 31 . 1 11 42 Finance, Private Sector & Infrastructure '4 4-'- 1 89 6' - 41.1 4 - Human Development 14 - - 55 3' 41 6- 44 23 -8 '1 Poverty Reduction & Economic Management 1I1 o1 41 2- 36 83 of 34 2 7' U l1 Lending type Adjustment I'- -' S 33 8" 4- C. I Investment 1, 1 i. 601 44 '' 190 6" - 52 - 4 1 Lending source IBRD only 111 I 116 7' 54 4 i s1 '-1 44 IDA/blend '4' b3 h 2 107 64 41 '4 i -41 Region Africa 41, 55 'l 70 S" 33 2 38 East Asia and Pacific 24 i 3 44 33 8 . 3" 1. :s Europe and Central Asia 4 -4 - 4 1 24 83 n- 4- I U 'I Latin America and Caribbean 2r. o SI '3 45 8 ic - ' S3 Middle East and North Africa 13- '7 4" 2" 22 7 33 32 14,0 lK.' SouthAsia 184 - 4(1 "1 29 5- 4 32 13 - ' Total/averagev1er-ge, 41 .- 4b . 23 7I 4 3 , n 3' Note: Percentages exclude projects not rated. Supplemental statistical tables are available electronically at http://www.worldbank.org html/oed/ardetbls.htm or by request. 57 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS O1 :UTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT IMPACT OF EVALUATED OPERATIONS, BY SECTOR, NETWORK, LENDING TYPE/SOURCE, AND REGION, EXIT FISCAL YEARS 1990-95, 1996, AND 1997 (BY DISBURSEMENTS) L\TT FN lou-a -NIT FNl E\IT FN Ill- Db. O -.T El 0-· [i DIs. UT- [[, BLIRE. CONE L LT fMPCT BI.-R-E C)Nq sL'T. IMIlPCT BIRS.. (NIE ;UST LMP\CT Sector i ni. . 1k;! M - s '::. w. 3iis M . 8'm!. w. 1 i'I- Agriculture 22,4 - 4l 34 , · 52 "s L,-f i S3 .- 4' Education 4,N'5 <4 '-'13' 5- 44 I n1 Sn -4 51 Electric Power t Other Energi lo,'S4 o' '_s'.' .' S4 SI t ~ Environment - - - - 4 iii 2'' il 1 i11 i 01 Finance 8,404 i 54 3<' . 53 -s -ss ;39 10011 ii1 Industry II,c'- s '" 31 '.1 51 35 1- 4l '4< "> 4 M ining 5 - 4''i [i. 11111 ., ti> [0 10 Multisector is,5' VI e, -n "i '4 34 N93 's 42 12 Oil & Gas 3.-I :12 45 6 In' 3' :l Telecommunia- nn 1, ~I 4I 41 "1 -, ., 54 -34 11111 1.i0) 1<1(' Transportation 13.-' 2 55 34 2.422 "3 - 44 '13 54 Si ~ UrbanDevelcponr 5..i)3 ~ 53 > 1,1-" 's 25 2 244 >5 44 22 Water Supply Z Sanitation 3,6t9 r; 2S - <". 23 31 l< <' 100 I00 ' Network Environmenta Il b & Socially Sustainable Developmert 22.S34 - 45 34 ;.,4n i. I 1,4- S3 e< ii Finance, Priva Sector & Infrastructué r4,5'>- ¯l l. 34 4-22. 'u r' 4' 2,"-. 42 ~, -4 Human Developmerr r,.øs S -l I.5'3 on 52 3 1,45 8<4 44 Poverty Redu,. nr & Economic Managemen 21,<45 2 .s - Y,i- "4 n2 32 1,2n 85 41 8 Lending type Adjustment ..422 ¯ .4 - " .-> n< -I-I i,S Nr.1 -4 33 Investment ~4 2 ¯ in 35 14,3e' - '. ,' 5,33 5o' Lending sourc IBRD only l4412 -5 4 I !,S; V. '3 42 3.91 l3 IDA/blend 5, 3 42 28 5,45< -. 5> N .¯ Region Africa 1 .-2 "3 2<' 2; 2,-14- - 41 41 1.432 5 44 44 East Asia and P i i 24.bs '> 84 52 .< ! ¯ [,32" 4¯ ~ ¯ Europe and Ccn E r.1 Asia l'1n1 5" 43 1. 30' n s 4.->I < I nI 1.5- <5 5r e Middle East e nd North Afric. 5,65> o< 4- 1,1 5. 24 1 304 111 100 i SouthAsia 2'.352 -r 52 34 X ss ' 54 25 1,115 SI I'' Total/average e1,I 5 ¯4 5' 3¯ I",6 ; 'd i4 . s SS 54 Note: Percentages exclude projects not rated. Supplemental statistical tables are available electronically at http://www.worldbank.org/ html/oed/ardetbls.htm or by request. 58 APPENDIX Prefacio E l vigésimo tercer Examen anual preparado por ente al ejercicio de 1997 esa proporción fue del 74 por el Departamento de Evaluación de Operaciones ciento. Indudablemente, los clientes del Banco Mundial representa un cambio de orientación con han mejorado su labor en lo que respecta a la preparación - respecto a los exámenes anteriores. En la publi- de los proyectos y el cumplimiento de las disposiciones de cación de este año -la primera desde la los convenios. Si bien ha mejorado el desempeño de los aprobación por el Directorio Ejecutivo del Banco Mundial prestatarios, no se han registrado cambios en la actuación del nuevo Pacto Estratégico para la institución- se procura del Banco. Sin embargo, puesto que han aumentado las exi- hacer una evaluación más exhaustiva de la eficacia del gencias y la complejidad de los proyectos, estos resultados Banco en materia de desarrollo, tal como se estipula en el demuestran que el Banco ha mantenido su desempeño en programa de renovación institucional. Además de un análi- un clima difícil y cambiante, y algunas de las mejoras sis de 325 proyectos terminados durante los ejercicios de observadas en los prestatarios pueden atribuirse a los ser- 1996 y 1997 y de una evaluación de los resultados teniendo vicios no crediticios de la institución. en cuenta sus tendencias a largo plazo, el informe ofrece una No obstante, si bien ha mejorado la calificación de los perspectiva estratégica de los factores que contribuyen a la resultados, no siempre ha sido posible mantener los resul- obtención de resultados "en el terreno". Entre otras cosas, tados en el terreno. El impacto en el desarrollo institucional se destaca la importancia de avanzar hacia la adopción de y la sostenibilidad son vitales para la eficacia del desar- un sistema orientado a los resultados a fin de mejorar el rollo. En esta área los resultados han sido variados. aprendizaje y la rendición de cuentas de la institución. Aunque se ha registrado una leve mejora en materia de En los últimos diez años, los programas de desarrollo desarrollo institucional, la sostenibilidad de los proyectos se han centrado menos en las inversiones físicas para con- sigue constituyendo un motivo de preocupación debido a la centrarse en ayudar a los países a lograr un crecimiento inestabilidad de las condiciones para su ejecución. En social y ecológicamente sostenible. Este nuevo consenso en alrededor de la mitad de las operaciones del Banco, la materia de desarrollo se ha traducido con rapidez en un sostenibilidad es incierta o poco probable y las mejoras replanteamiento de la forma en que el Banco Mundial real- siguen siendo lentas. iza su labor y evalúa el impacto en el desarrollo. El Pacto Para que aumente la eficacia de la asistencia del Banco Estratégico tiene por objeto adaptar la institución a las en el largo plazo será necesario brindar un respaldo inin- nuevas necesidades de sus clientes, aprovechar las oportu- terrumpido que permita mejorar la capacidad de los nidades que surgen a nivel mundial y mejorar la eficacia del prestatarios en la selección, el diseño y la gestión de los Banco en el logro de su objetivo primordial: la reducción de proyectos. Se requerirá también una mayor selectividad la pobreza. Por lo tanto, la evaluación y el aumento de la tanto en la distribución del financiamiento a los países eficacia en términos de desarrollo se han transformado en prestatarios que cuenten con marcos de políticas y de gob- el desafío fundamental de la evaluación del desarrollo. ierno adecuados, como en los esfuerzos por equiparar los Los resultados de la evaluación de la eficacia del desar- servicios e instrumentos del Banco con la capacidad de los rollo de este año son alentadores. Han mejorado los resul- prestatarios. La selectividad no consiste en "seleccionar tados de los proyectos y el desempeño de los prestatarios. países con buenas perspectivas de éxito", sino en la adop- La proporción de proyectos con resultados satisfactorios ción desde el comienzo, por parte del país, de un compro- aumentó al 71 por ciento en el ejercicio de 1996, y en una miso frente al desarrollo y al buen gobierno, y en modular muestra que incluyó la mitad de los préstamos y créditos la respuesta del Banco mediante una asistencia financiera y que salieron de la cartera durante el ejercicio de 1997 dicha no financiera apropiada y oportuna. Una estrategia selec- proporción fue del 76 por ciento. El principal factor en este tiva conlleva también el establecimiento de mejores asocia- incremento es el notable mejoramiento del desempeño de ciones para multiplicar así el impacto de la labor del Banco, los prestatarios. La proporción de proyectos con un desem- y la adopción de enfoques más participatorios y mejor peño satisfactorio de los prestatarios aumentó al 71 por adaptados a fin de incrementar la identificación de los ciento en el ejercicio de 1996; en la muestra correspondi- prestatarios con los proyectos y obtener mejores resultados 59 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS en el terreno. Del mismo modo, una adecuada gestión de más relevantes, la reforma de los procedimientos institu- los riesgos en el ámbito del desarrollo entraña una mejor cionales para lograr resultados eficaces en función de los evaluación y seguimiento de los resultados. costos, y la reconfiguración de la organización del Banco En resumen, si bien se ha avanzado considerablemente para actuar en asociación con otros organismos, consti- para alcanzar las metas fijadas en el programa actual de tuyen el desafío fundamental para el Banco Mundial en lo mejoramiento de la cartera del Banco Mundial, aún queda que respecta a la eficacia en términos de desarrollo. mucho por hacer. Para seguir progresando en el marco del Pacto Estratégico el Banco deberá adoptar un sistema de gestión basado en los resultados que se adecue a la nueva visión del Banco como una institución que ofrece una com- Robert Piciotto pleta gama de servicios. En vísperas del nuevo milenio, la Director General reorientación de las operaciones hacia el logro de metas Departamento de Evaluación de Operaciones 60 Appendix R sumen os retos que plantea actualmente el desarrollo no ación de proyectos realizada por el DEO en el curso de los tienen precedente. El Banco Mundial, junto con dos últimos ejercicios, y se los evalúa en el contexto de las otras instituciones, procura encontrar los medios tendencias a más largo plazo en materia de resultados. para satisfacer simultáneamente las necesidades de las economías en rápido crecimiento y las de Evolución de los resultados aquellas que se encuentran en peligro de caer en la margin- Las tendencias mundiales y la evolución del programa de alización y el estancamiento económico. En el programa de desarrollo han alterado radicalmente las condiciones para desarrollo, en constante evolución, se ha hecho mayor hin- la ejecución de las operaciones financiadas por el Banco. A capié en la sostenibilidad social y ambiental, el fortalec- principios de la década de 1980, las necesidades surgidas imiento de la capacidad a nivel local, la igualdad entre los en este nuevo contexto comenzaron a tener serias conse- sexos y la participación. Estos retos han adquirido tal mag- cuencias en la calidad de las opera- nitud y complejidad que la formación de asociaciones está ciones desde sus etapas iniciales. El adquiriendo una importancia cada vez mayor. desempeño del Banco en lo rela- H ay señales de A medida que se avanza en la aplicación de este exi- tivo a las operaciones terminadas gente programa también se hace más compleja la tarea de comenzó a deteriorarse. El por~ que los resultados evaluar y mejorar la eficacia del Banco en términos de centaje de las operaciones califi- desarrollo. La evaluación de la eficacia del Banco no puede cadas de satisfactorias se redujo de la cartera están seguir centrándose exclusivamente en los proyectos; tam- del 80 por ciento a comienzos de la bién debe evaluarse el impacto de todas las actividades de década de 1980 al 60 por ciento a mejorando la institución a nivel sectorial y de los países. Para ello es fines de ésta. Para entonces, la necesario contar con nuevos instrumentos y modalidades administración del Banco había de evaluación a fin de conocer los efectos combinados de reconocido la necesidad de mejorar la gestión de la cartera los servicios crediticios y no crediticios y la contribución de y de prestar especial atención a la calidad desde las etapas las asociaciones a los resultados de los proyectos y progra- iniciales, a la participación de los prestatarios en la mas. El Departamento de Evaluación de Operaciones preparación y ejecución de los proyectos, y a los resultados (DEO) acaba de iniciar un programa de renovación con el en términos de desarrollo. Para ello fue necesario intro- propósito de poner en práctica este nuevo sistema de eval- ducir profundos cambios en las actitudes, las aptitudes y uación, pero pasará algún tiempo antes de que se puedan los incentivos en materia de gestión. Aunque aún es observar sus resultados. demasiado pronto para evaluar las repercusiones que han El presente Examen anual sobre la eficacia en términos tenido estos cambios, los últimos resultados confirman que de desarrollo constituye un primer paso hacia una evalu- las operaciones del Banco comienzan a mostrar mejoras ación más cabal de la eficacia del Banco. En él se enuncian (Gráfico 1). algunas cuestiones relativas a la definición y evaluación de La proporción de proyectos con resultados satisfactorios, la eficacia del Banco en términos de desarrollo en un según la evaluación del DEO correspondiente al ejercicio en entorno mundial cambiante. Además, se destaca la impor- que dichos proyectos dejaron de formar parte de la cartera, tancia de adoptar un sistema más orientado a los resulta- aumentó de un promedio del 67 por ciento durante los ejer- dos a fin de mejorar el aprendizaje y la responsabilidad a cicios de 1990 a 1995 al 71 por ciento en el ejercicio de 1996 nivel institucional en el cumplimiento de las metas estraté- y, de acuerdo con una muestra parcial, al 76 por ciento en el gicas. Se examinan algunos de los aspectos estratégicos de ejercicio de 1997. los resultados de las evaluaciones y se analizan los factores Sin embargo, las mejoras no han sido parejas en todas que determinan el resultado de los proyectos a nivel de la las regiones y sectores. En África al sur del Sahara los resul- cartera de los países y de cada uno de los proyectos. Para tados siguen siendo inferiores al promedio del Banco en mantener la continuidad en el proceso de aprendizaje insti- general y, de acuerdo con el último Informe anual sobre el tucional, se analizan también los resultados de la evalu- desempeño de la cartera, esa región sigue teniendo la pro- 61 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS GRÁFICO 1: LOS RESULTADOS SIGUEN MEJORANDO SEGUN LOS PROYECTOS SEGUN LOS DESEMBOLSOS 90 90 80 80 e5 70 70 S60 60 o 50 50 81 83 85 87 89 91 93 95 *97 81 83 85 87 89 91 93 95 *97 Fuente: Datos del DEO Nota: Las líneas entrecortadas (operaciones que salieron de la cartera en el ejercicio de 1997) indican resultados preliminares, es decir, que cubren menos del 50 por ciento de las operaciones que dejaron de formar parte de la cartera. porción más alta de proyectos en peligro. En el sector del por ciento en el ejercicio de 1996, y al 74 por ciento en el abastecimiento de agua y saneamiento los resultados han de 1997, según la muestra correspondiente a este último registrado un fuerte deterioro: el porcentaje de proyectos ejercicio. Segundo, el mejoramiento de las condiciones eco- satisfactorios disminuyó de alrededor del 80 por ciento en nómicas generales -reflejado en el número de proyectos el ejercicio de 1993 al 40 por ciento en el de 1996. De evaluados durante el ejercicio de 1996 que fueron ejecuta- acuerdo con el Informe, los dos en los países con un mejor desempeño- ha mejorado L proyectos de abastecimiento de el clima para las operaciones financiadas por el Banco'. La reciente agua y saneamiento son los que Un análisis econométrico de más de 900 proyectos ter- más peligran en la cartera actual. minados2 muestra que el desempeño de los prestatarios es mejora de los fundamental para el resultado de los proyectos. Una buena Factores determinantes de las actuación por parte de los prestatarios aumenta en 35 ó 40 resultados se mejoras observadas puntos porcentuales la probabilidad de obtener resultados Los análisis ponen de manifiesto al satisfactorios. Por consiguiente, es probable que un mejor debe al mejor menos dos posibles razones para desempeño de los prestatarios sea el principal factor deter- esta evolución positiva de los minante del mejoramiento de los resultados globales de los desempeño de los resultados. Primero, la mejora del proyectos. desempeño de los prestatarios - Las condiciones económicas en los países también son prestatarios identificación con los proyectos, importantes y, junto con el buen desempeño del Banco ejercicio de sus derechos y asun- desde las primeras etapas y durante la ejecución de los ción de responsabilidades-, así proyectos, pueden aumentar en 15 ó 20 puntos por- como del suministro por parte de éstos de los insumos centuales la probabilidad de que sus resultados sean satis- necesarios para preparar y ejecutar las operaciones finan- factorios. Si bien hay indicaciones de que las condiciones ciadas por el Banco. Los últimos resultados indican que la económicas en los países han mejorado, lo cual se confirma proporción de proyectos en que el desempeño de los pres- en el Informe anual sobre el desempeño de la cartera cor- tatarios ha sido satisfactorio aumentó de un promedio del respondiente a la cartera activa, aún hay pocos indicios de 65 por ciento durante los ejercicios de 1990 a 1995 al 71 un mejoramiento del desempeño del Banco. Éste fue satis- 62 Appendix factorio en el 74 por ciento de las operaciones evaluadas en la capacidad de éstos para mantener los beneficios de los el ejercicio de 1996 y en el 77 por ciento de las operaciones proyectos a más largo plazo. incluidas enla muestra parcial correspondiente al ejercicio De acuerdo con los últimos resultados, el historial de de 1997. No obstante, si se compara con la tendencia re- las operaciones en lo que respecta a su eficacia ha sido gistrada durante los ejercicios de 1990 a 1995, el por- desigual. La proporción de proyectos con un efecto sobre el centaje de proyectos en que el desempeño del Banco fue desarrollo institucional calificado satisfactorio se ha mantenido a un nivel relativamente con- de muy importante o considerable stante de entre 70 por ciento y 75 por ciento. Estos resul- aumentó del 30 por ciento en los El impacto y la tados demuestran que deberán seguir haciéndose esfuerzos ejercicios de 1990 a 1995 al 39 por por mejorar la calidad desde las etapas iniciales y la gestión ciento en los ejercicios de 1996 y sostenibilidad de la cartera para conseguir una mejora importante de la 1997. El número de proyectos actuación del Banco. cuyas repercusiones sobre el desar- del desarrollo Un estudio reciente del DEO sobre el proceso de eva- rollo institucional se han calificado luación inicial de los proyectos (DEO 1997) también revela de moderadas prácticamente no ha institucional son una lenta mejoría del desempeño del Banco. El estudio con- variado, pero la proporción de firma la importante relación que proyectos con un efecto insignifi- medidas más existe entre la calidad de los cante disminuyó del 25 por ciento A ún queeda proyectos desde el inicio y sus al 15 por ciento durante los ejerci- precisas de la resultados, aunque también pone cios de 1995 y 1996. Aunque estos de relieve las continuas deficiencias resultados son alentadores, alrede- eficacia en del análisis social que se realiza en dor del 60 por ciento de las opera- muejorar en lo la etapa de evaluación inicial, de la ciones del Banco siguen teniendo términos de participación de los prestatarios y un efecto moderado o reducido que se_refiere al los beneficiarios, y del análisis sobre el desarrollo institucional. desarrollo institucional. Además, en el estu- En lo que respecta a la sosteni- deeeñeo del dio se señala que sin un consenso bilidad, el historial es algo menos claro en cuanto a la forma de halagüeño. Los últimos resultados indican que la propor- Banco mejorar la calidad las etapas ini- ción de proyectos con probabilidades de ser sostenibles ciales -diseñar proyectos más sen- pasó del 46 por ciento durante el período comprendido cillos, hacer mayor hincapié en los entre los ejercicios de 1990 a 1995 al 48 por ciento en el riesgos y dar mayor importancia al aprendizaje mediante ejercicio de 1996. Los resultados parciales correspondi- una labor sistemática de seguimiento y evaluación- es entes al ejercicio de 1997 indican otro pequeño incremento, imposible encontrar los medios para lograr mejoras en el al 51 por ciento, pero los resultados globales son preocu- futuro. Cada vez hay más indicios de que los servicios no pantes: no es seguro, o es improbable, que los beneficios de crediticios del Banco son vitales para mejorar no sólo la casi la mitad de todas las operaciones se vayan a sostener a calidad del proceso de evaluación inicial, sino la calidad de largo plazo. la gestión económica y el desempeño de los prestatarios. Mejoramiento de los resultados y la eficacia Los efectos sobre el desarrollo institucional y la Los proyectos acertados ayudan a mejorar la capacidad de sostenibilidad ejecución de los países prestatarios. A su vez, los buenos Por sí sola, una mejora en la clasificación de los resultados proyectos dan mejores resultados cuando las condiciones no significa necesariamente que se hayan obtenido resulta- son apropiadas. El análisis de los factores que determinan dos más favorables en el terreno. Para evaluar la eficacia en el éxito de los proyectos indica que tanto la credibilidad de términos de desarrollo a más largo plazo de un proyecto es los gobiernos como la inexistencia de corrupción están necesario examinar tanto sus repercusiones sobre el desa- directamente vinculados al resultado de los proyectos. Los rrollo institucional como sus posibilidades de ser sostenible. análisis a nivel de los países indican que las políticas y las Los proyectos, a través de sus efectos sobre el desarrollo instituciones competentes son importantes para lograr institucional, mejoran la capacidad de ejecución de los resultados económicos favorables y un buen desempeño de países prestatarios; la sostenibilidad, en cambio, se refiere a la cartera (Gráfico 2). 63 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS GRÁFICO 2: LA CALIDAD DE LAS INSTITUCIONES Un estudio de las enseñanzas recogidas por medio de Y LAS POLÍTICAS INFLUYEN EN LOS RESULTADOS DE los últimos exámenes de la asistencia a los países confirma LA CARTERA DE PROYECTOS CORRESPONDIENTES A la importancia que revisten las LOS PAíSES políticas para el efecto global de la asistencia del Banco. Reitera, ade- l apoyo al más, que las reformas de política rara vez son exitosas si el compro- prestatario para miso de los prestatarios y su iden- tificación con los proyectos son que establezca 100 inadecuados. El análisis de los fac- so - - - tores que han determinado el políticas e mejoramiento de la cartera actual 5 40 confirma esta afirmación: en los instituciones 20 paises donde la gestión macroeco- o20 Irapai nómica es deficiente el número de sólidas es ada de las proyectos ha disminuido marcada- Alta Baja políticas mente3. Ésto pone de manifiesto la fundamental para Baja importancia de fomentar la capa- Calidad de las instituciones cidad del prestatario y de garanti- lograr una mayor zar el establecimiento de acuerdos eficaces entre las partes interesa- eficacia Fuente: Datos del DEO das a fin de crear el consenso nece- sario y lograr la identificación con los proyectos en los La importancia de los aspectos de política y de go- programas de reforma sectorial y a nivel de países. bierno a nivel de los proyectos y los países confirma las con- clusiones de las investigaciones en Perspectivas el sentido de que la asistencia para Es indudable que ha habido progreso, pero el margen para Las políticas el desarrollo tiene un mayor efecto mejorar la labor realizada sigue siendo muy amplio. sobre el crecimiento económico Además, no se ha determinado aún la importancia a largo y la situación cuando se suministra en un marco plazo de las mejoras registradas en el ejercicio de 1997. de políticas adecuado (Dollar y Los análisis muestran que con un esfuerzo sostenido institucional de Burnside 1997). Además, pone de por mejorar la calidad de la gestión, el Banco puede man- manifiesto lo importante que es tenerse en la dirección correcta para cumplir la meta fijada los países tienen aplicar una mayor selectividad en por la administración en el sentido de alcanzar un nivel del la asistencia que otorga el Banco. 75 por ciento de proyectos satisfactorios en el año 2000. una influencia Esta selectividad supone encauzar Las metas relativas al mejoramiento de la cartera durante los préstamos a los países que el próximo año, que se enuncian en el Informe anual sobre significativa en tienen un marco de políticas el desempeño de la cartera, incluyen el aumento a un nivel apropiado, y hacer un uso más del 75 por ciento o mayor de la proporción de operaciones los resultados estratégico de los servicios no cre- cuya supervisión es satisfactoria, y un aumento al 85 por diticios para respaldar la adopción ciento o más de la proporción de nuevos proyectos con un de políticas adecuadas y el buen nivel de calidad satisfactorio desde el inicio. Al incluir estas gobierno. Una mayor selectividad entraña, además, una metas en un análisis de sensibilidad de más de 300 de las mejor selección y secuencia de los medios e instrumentos de operaciones evaluadas recientemente (ejercicios de 1995 a asistencia del Banco a fin de equiparar la calidad de las 1997) se observa que, de lograrse ambas metas simultánea- instituciones locales con la capacidad de los prestatarios. Es mente, el porcentaje de proyectos con resultados satisfac- fundamental crear una mayor sinergia en materia de desar- torios aumentaría al 75 por ciento. Si el desempeño de los rollo entre los servicios de asesoramiento, los servicios prestatarios sigue mejorando como lo ha hecho reciente- analíticos y los servicios crediticios. mente, es posible vislumbrar un mejoramiento de los resul- 64 Appendix tados cercano al 80 por ciento de proyectos satisfactorios hará necesario un respaldo ininterrumpido para aumentar en promedio4. la capacidad de los prestatarios en la selección, el diseño y El Presidente del Banco Mundial, James Wolfensohn, la gestión de los proyectos. Se requiere también una mayor es partidario de aplicar un enfoque enérgico consistente en selectividad en la distribución del financiamiento a los mejorar la calidad de los procedimientos del Banco para países prestatarios que cuenten con marcos de políticas y de lograr resultados 100 por ciento gobierno adecuados, así como en los esfuerzos por satisfactorios. Aunque es improba- equiparar los servicios e instrumentos del Banco con la Una estrategia ble que esta meta se logre antes del capacidad de los prestatarios. La selectividad no consiste en año 2000, la adopción de esta "seleccionar países con buenas perspectivas de éxito", sino selectiva consiste estrategia podría traducirse en un en la adopción, por parte del país, de un compromiso de nivel de satisfacción del 80 por desarrollo y de buen gobierno, y en orientar la labor del en procurar ciento o superior en términos de Banco mediante una asistencia financiera y no financiera los resultados globales. Mediante apropiada y oportuna. Una estrategia selectiva conlleva una mavor la introducción de mejoras también el establecimiento de mejores asociaciones a fin de ' juiciosas en el desempeño de los multiplicar el impacto de la labor del Banco, y la adopción correspondencia prestatarios, la proporción de de enfoques más participatorios y adaptados a fin de incre- proyectos satisfactorios podría mentar la identificación de los prestatarios con los proyec- entre la asistencia alcanzar el 85 por ciento. tos y obtener mejores resultados en el terreno. La prosecución decidida de del Banco y la estas metas a nivel sectorial y Nota regional parece ser necesaria y 1. Las condiciones económicas comenzaron a mejorar en capacídad de los aconsejable. Mediante la apli- muchos países en desarrollo a fines de la década de 1980, sobre todo cación de este plan estratégico, el en los países en que el ajuste fue más exitoso. La mayoría de las Banco restablecería la solidez de su operaciones evaluadas una vez que dejaron de formar parte de la prestatarios cartera del Banco en el ejercicio de 1996 fueron aprobadas en 1989. cartera al nivel que tenía a fines de 2. Proyectos que dejaron de formar parte de la cartera del Banco la década de 1970. Naturalmente, entre el ejercicio de 1993 y el de 1997. este plan exigiría una intensificación de los esfuerzos, por 3. De casi 800 proyectos en el ejercicio de 1995 a 465 en el ejer- parte de todas las oficinas regionales y redes, por mejorar la cicio de 1997. El análisis de sensibilidad de la evolución de los proyec- calidad de los proyectos desde las etapas iniciales, la super- tos en peligro parece indicar que alrededor de la mitad de las mejoras visión, la gestión de los riesgos, y la labor de seguimiento y son atribuibles a factores en los países (Informe anual sobre el desem- evaluación. peño de la cartera, noviembre de 1997). Además de un esfuerzo continuo por mejorar la cali- 4. Estos logros se basan en el número de proyectos, no en el vol- umen de los desembolsos. El modelo que sirve de base para los análi- dad y gestión de la cartera del Banco, el mejoramiento a sis de sensibilidad no permitió producir pronósticos basados en el largo plazo de la eficacia de la asistencia de la institución volumen. 65 Appendix Préface L e 23C Examen annuel du Département de l'éva- cent durant l'exercice 96 à 74 pour cent pour l'échantillon luation des opérations marque un changement de l'exercice 97. Les clients de la Banque mondiale ont radical par rapport à ceux qui l'ont précédé. La manifestement tendance à mieux préparer les projets et à en publication de cette année - la première depuis respecter davantage les clauses. Alors que la performance l'approbation par le Conseil de la Banque mon- des emprunteurs s'est améliorée, celle de la Banque est diale d'un nouveau Pacte stratégique - s'efforce d'évaluer restée inchangée. Mais, comme les projets sont devenus de façon plus approfondie l'efficacité de l'action de la plus exigeants et complexes, cela signifie que la Banque n'a Banque en faveur du développement, comme l'exige le pro- pas démérité dans un environnement difficile et évolutif et gramme de renouvellement de l'institution. Non seulement que l'on peut attribuer aux services hors prêt une partie de ce rapport analyse 325 projets achevés durant les exerci- l'amélioration des performances des emprunteurs. ces 96 et 97 et évalue les résultats obtenus par rapport aux Bien que la notation des résultats se soit améliorée, cela tendances des performances à long terme, mais en outre il ne s'est pas toujours traduit par des résultats durables sur offre une vision stratégique des facteurs ayant contribué le terrain. L'impact et la durabilité du développement insti- aux résultats obtenus « sur le terrain » en matière de tutionnel conditionnent de façon vitale l'efficacité du développement. Entre autres choses, il souligne combien il développement. Dans ce domaine, les résultats sont mi- est important de s'orienter vers un système axé sur les tigés. Alors que le développement institutionnel s'est légère- résultats pour améliorer l'apprentissage et la responsabili- ment amélioré, la durabilité des projets reste préoccupante sation au sein de la Banque. compte tenu de l'instabilité de l'environnement dans lequel Au cours de la dernière décennie, on a vu l'action en ceux-ci sont exécutés. La durabilité d'environ la moitié des faveur du développement s'orienter essentiellement non opérations de la Banque est incertaine dans le meilleur des plus vers des investissements physiques, mais vers un appui cas et les progrès restent lents. aux pays pour les aider à parvenir à une croissance durable Pour améliorer l'efficacité à long terme de l'aide de la prenant en compte des considérations sociales et environ- Banque, il faudra continuer à apporter un soutien aux nementales. Ce nouveau consensus en matière de capacités des emprunteurs au niveau de la sélection, de la développement a amené la Banque à réexaminer les moyens conception et de la gestion des projets. Il faudra également par lesquels elle opère et évalue son impact sur le faire preuve d'une plus grande sélectivité tant en orientant développement. Le Pacte stratégique vise à adapter la les prêts vers des pays emprunteurs dont les politiques et la Banque aux nouveaux impératifs des clients, à tirer parti gouvernance sont appropriées qu'en adaptant les services et des opportunités offertes au plan mondial et à aider notre instruments de la Banque aux capacités des emprunteurs. institution à réaliser plus efficacement son objectif primor- La sélectivité ne consiste pas à « choisir des gagnants », dial qui est la réduction de la pauvreté. L'évaluation et mais à s'assurer tout d'abord de l'engagement d'un pays l'amélioration de l'efficacité de son action aux fins du vis-à-vis du développement et d'une bonne gouvernance et développement sont donc devenues le défi majeur que pose à moduler la réaction de la Banque par le biais d'une assis- l'évaluation du développement. tance financière et non financière appropriée et fournie en La conclusion de l'évaluation de cette année sont temps voulu. Cela signifie également établir de meilleurs encourageants. Les résultats des projets et les performances partenariats pour démultiplier l'impact de la Banque mon- des emprunteurs se sont améliorés. La proportion de pro- diale et utiliser des approches plus participatives et adap- jets aux résultats satisfaisants est passée de 71 pour cent tées afin de favoriser l'appropriation des projets par les durant l'exercice 96 à 76 pour cent pour un échantillon intéressés et obtenir de meilleurs résultats sur le terrain. De comprenant une moitié des prêts et des crédits sortis du même, une saine gestion des risques liés au développement portefeuille durant l'exercice 97. Le principal facteur à l'o- implique une évaluation et un suivi plus efficaces des rigine de cette progression est l'amélioration marquée de la performances. performance des emprunteurs. La proportion de projets Au total, des progrès considérables ont été réalisés pour lesquels celle-ci est satisfaisante est passée de 71 pour dans le sens des objectifs du programme actuel d'améliora- 67 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS tion du portefeuille de la Banque mondiale, mais il reste partenariats constituent le double problème fondamental beaucoup à faire. Pour suivre la voie définie par le Pacte auquel doit faire face la Banque mondiale pour assurer l'ef- stratégique, la Banque mondiale aura besoin d'un système ficacité de son action en faveur du développement. de gestion qui soit axé sur les résultats et qui réponde à la nouvelle façon de concevoir son rôle, à savoir celui d'une institution polyvalente. À l'aube du nouveau millénaire, la réorientation des opérations vers des buts plus réalistes, la modification des procédures opérationnelles dans le sens Robert Picciotto d'un meilleur rapport coût-efficacité et la refonte de l'or- Directeur général ganisation pour lui permettre d'opérer dans le cadre de Évaluation des opérations 68 Appendix Résumé e développement pose aujourd'hui des pro- Tendances de la performance blèmes sans précédent. La Banque mondiale, Les tendances et changements globaux du programme de avec d'autres institutions, recherche des moyens développement ont radicalement modifié l'environnement de faire face simultanément aux impératifs des dans lequel sont mises en oeuvre les opérations financées économies à croissance rapide et de celles qui par la Banque. Au début des années sont exposées à un risque de marginalisation et de stagna- 80, les exigences de ce nouvel envi- tion. Dans son programme de développement en évolution, ronnement ont commencé à porter La performance elle met davantage l'accent sur la viabilité sociale et envi- gravement atteinte à la qualité ini- ronnementale, le renforcement des capacités locales, l'éga- tiale des projets. La performance de du portefeuille lité des sexes et la participation. Les problèmes sont devenus la Banque pour les opérations si vastes et si complexes que des partenariats sont de plus achevées a commencé à se détéri- commence à en plus indispensables. orer. Le pourcentage des opérations Plus ce programme exigeant avance et plus il devient dif- de la Banque évaluées et jugées sa- s'améliorer ficile d'évaluer et d'améliorer l'efficacité de l'action de la tisfaisantes est tombé de 80 pour Banque mondiale en faveur du développement. Il n'est plus cent au début des années 80 à 60 possible d'évaluer l'efficacité de la Banque en examinant pour cent à la fin de cette décennie. La direction de la uniquement les projets et il faut désormais mesurer égale- Banque avait alors reconnu la nécessité de renforcer la ges- ment l'impact de toute la gamme de ses activités au niveau tion du portefeuille, en accordant une attention particulière sectoriel et national. Cela implique que l'on dispose de nou- à la qualité initiale des projets, à la participation de l'Em- veaux instruments et modalités d'évaluation pour prendre en prunteur à la préparation et à la mise en oeuvre, et aux compte les effets conjugués des services de prêt et hors prêt, résultats obtenus en matière de développement. Cela et la contribution des partenariats à l'impact des projets et demandait une profonde modification des attitudes, des programmes. Le Département de l'évaluation des opérations compétences et des incitations données au personnel d'en- (OED) a lancé un programme de rénovation pour mettre en cadrement. Il est encore trop tôt pour juger des effets de ces oeuvre le nouveau cadre d'évaluation, mais cette rénovation changements, mais les derniers résultats des analyses de la ne fait que commencer et les résultats se feront attendre. performance de la Banque montrent que ses opérations Le présent Examen annuel de l'efficacité du développe- commencent à s'améliorer (figure 1). ment est un premier pas vers une évaluation plus globale de Le pourcentage des projets dont les résultats ont été l'efficacité de la Banque. Il expose les problèmes posés par jugés satisfaisants par l'OED pour l'exercice durant lequel la définition et la mesure de l'efficacité de l'action de la ils ont été achevés est passé de 67 pour cent en moyenne Banque en faveur du développement dans un monde en pour les exercices 90-95 à 71 pour cent pendant l'exercice pleine mutation, et souligne combien il est important de se 96 et, sur la base d'un simple échantillon, à 76 pour cent diriger vers un système qui soit plus axé sur les résultats pour l'exercice 97. pour permettre une amélioration des connaissances et de la Toutes les régions et tous les secteurs ne voient pas responsabilisation au sein de la Banque compte tenu des leurs résultats s'améliorer dans les mêmes proportions. objectifs stratégiques à réaliser. Il examine certains des L'Afrique subsaharienne continue à enregistrer des résultats aspects stratégiques des résultats des évaluations et analyse inférieurs à ceux de la moyenne pour toute la Banque et, les facteurs déterminants de la performance des projets au selon le dernier Rapport annuel sur la performance du niveau des portefeuilles-pays et des différents projets. Afin portefeuille, c'est encore elle qui compte le pourcentage le d'assurer la continuité de l'amélioration des connaissances plus élevé de projets à risque. Dans le secteur de l'approvi- au sein de la Banque, il analyse aussi les résultats des éva- sionnement en eau et de l'assainissement, la performance luations des projets effectuées par l'OED au cours des deux s'est dégradée de façon spectaculaire, tombant de quelque exercices précédents et on les compare aux tendances de la 80 pour cent pour l'exercice 93 à 40 pour cent pendant performance à plus long terme. l'exercice 96. Là encore, selon le dernier Rapport annuel 69 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 1 LES RÉSULTATS CONTINUENT À S'AMÉLIORER PAR PROJET PAR DÉCAISSEMENT 90 90 80 80 70 70 O60 60 50 50 81 83 85 87 89 91 93 95 *97 81 83 85 87 89 91 93 95 *97 Source : Données de l'OED Note: Les lignes en pointillé (exercice d'achèvement 97) représentent des résultats préliminaires basés sur un échantillon comprenant moins de 50 pour cent des projets achevés au cours de l'exercice. sur la performance du portefeuille, c'est le secteur le plus formants, a eu un effet bénéfique sur l'environnement dans « à risque » dans le portefeuille actuel. lequel les opérations financées par la Banque sont mises en oeuvre1. Facteurs à l'origine de l'amélioration de la performance Selon les conclusions de l'analyse économétrique de Il ressort des analyses que deux plus de 900 projets achevés2, la performance de l'emprun- facteurs au moins peuvent être à teur joue un rôle déterminant dans les résultats des projets. Lamélioration l'origine de l'amélioration des Si elle est bonne, elle augmente de 35 à 40 points la pro- résultats. Premièrement, la perfor- babilité de résultats satisfaisants. Par conséquent, c'est mance de l'emprunteur - l'exer- vraisemblablement d'une meilleure performance de l'em- récente des cice des droits de propriété et des prunteur que dépend en premier lieu l'amélioration de la responsabilités - s'est améliorée, performance globale du projet. résultats est due de même que la fourniture par La situation économique des à une meilleure celui-ci des éléments nécessaires à pays est importante elle aussi et, de La erformance la préparation et à la mise en oeu- pair avec la qualhte de la perfor- de vre des opérations financées par la mance de la Banque au début des de la Banque Banque. Les derniers résultats projets et pendant leur mise en montrent que la proportion des oeuvre, elle renforce de 15 à 20 projets pour lesquels la perfor- points de pourcentage la probabi- mance de l'Emprunteur a été jugée lité de résultats satisfaisants. Bien . satisfaisante est passée de 65 pour que l'on ait des raisons de penser cent en moyenne pendant les exer- que la situation économique des cices 90-95 à 71 pour cent durant l'exercice 96, et à pays s'est améliorée, ce que confirme le Rapport annuel sur 74 pour cent pour l'échantillon de l'exercice 97. Deux- la performance du portefeuille en ce qui concerne le porte- ièmement, l'amélioration de la situation économique glo- feuille actif, il n'est pas encore bien établi que la perfor- bale, dont témoigne le nombre de projets évalués pendant mance de la Banque s'est améliorée. Elle a été satisfaisante l'exercice 96 et exécutés dans des pays devenus plus per- pour 74 pour cent des opérations évaluées durant l'exercice 70 Appendix 96 et pour 77 pour cent de l'échantillon limité de l'exercice des projets considérés comme ayant un impact considérable 97. Mais une comparaison de ces chiffres avec ceux des ou appréciable sur le développement institutionnel est exercices 90-95 révèle que le pourcentage des projets pour passée de 30 pour cent au cours des exercices 90-95 à lesquels la performance de la Banque a été jugée satis- 39 pour cent durant les exercices 96 et 97. Alors que le faisante est resté relativement constant (70-75 pour cent). nombre des projets dont cet impact a été jugé modeste est Cela montre que les efforts entrepris pour améliorer la resté à peu près constant, le pourcentage de ceux ayant un qualité initiale des projets et la gestion du portefeuille effet négligeable est tombé de 25 à 15 pour cent entre les devront encore se poursuivre quelque temps avant de se exercices 95 et 96. Ces résultats sont encourageants, mais traduire par une amélioration significative de la perfor- quelque 60 pour cent des opérations de la Banque n'ont mance de la Banque. encore qu'un impact modeste ou insignifiant sur le Une étude effectuée récemment par l'OED sur le proces- développement institutionnel. sus d'évaluation (OED 1997) fait également apparaître une Pour ce qui est de la viabilité, le tableau est un peu légère amélioration de la performance de la Banque. Elle moins encourageant. Les derniers résultats montrent que la confirme l'importance du lien existant entre la qualité proportion des projets dont la viabilité a été jugée probable initiale des projets et leurs résultats, mais elle insiste aussi a faiblement augmenté entre les exercices 90-95 (46 pour sur la faiblesse de la qualité de l'analyse sociale faite lors de cent) et l'exercice 96 (48 pour cent). Les résultats partiels l'évaluation, de la participation de l'emprunteur et du béné- pour l'exercice 97 montrent un léger accroissement ficiaire, et de l'analyse institutionnelle. En outre, cette étude (51 pour cent), mais le résultat global donne à réfléchir la affirme qu'il est impossible de prédire l'origine des amélio- viabilité à long terme d'environ la moitié des opérations de rations futures si l'on ne se met pas la Banque est incertaine ou peu probable. clairement d'accord sur les moyens L'impact et la de renforcer la qualité initiale Améliorer les résultats et l'efficacité (conception plus simple des projets, Les bons projets aident à améliorer la capacité de mise en viabilité du attention accrue accordée au risque oeuvre des pays emprunteurs, mais ils donnent de meilleurs et à l'apprentissage par le biais d'un résultats dans un environnement développement suivi et d'une évaluation systémati- favorable. L'analyse des facteurs ques, etc.). Il y a de plus en plus lieu qui déterminent le succès d'un pro- L institutionnel de penser que les services hors prêt jet montre que la crédibilité du a politique de la Banque ont un rôle crucial à gouvernement de même que l'ab- constituent des jouer non seulement dans l'amélio- sence de corruption ont un effet suivie dans un ration de la qualité du processus positif sur ses résultats. Au niveau mesures plus d'évaluation, mais aussi dans celle d'un pays, elle montre que des pays et les de la gestion économique et la per- politiques appropriées et des insti- conditions rigoureuses de formance de l'emprunteur. tutions compétentes sont néces- saires du point de vue des résultats l'efficacité du Incidence et viabilité du économiques que de la perfor- institutionnelles développement institutionnel mance du portefeuille (figure 2). . développement Ce n'est pas parce que la notation La portée des facteurs « poli- d'un projet s'améliore que les résul- tique générale » et « gouvernance » considérablement tats sur le terrain sont forcément au niveau du projet et du pays meilleurs. Pour évaluer l'efficacité à long terme d'un projet du étaye les conclusions des travaux s point de vue du développement, il faut examiner aussi bien sur le sujet, à savoir que l'aide sur les résultats son impact sur le développement institutionnel que sa viabi- au développement a plus d'impact obtenus lité probable. Par leur impact sur le développement institu- sur la croissance économique tionnel, les projets améliorent la capacité de mise en oeuvre lorsqu'elle est fournie dans le cadre des pays emprunteurs, tandis que la viabilité est l'aptitude à d'une saine politique gouvernementale (Dollar et Burnside, préserver les avantages des projets à plus long terme. 1997). Elle montre aussi combien il importe que la Banque En matière d'efficacité, un examen des résultats les plus se montre plus sélective dans l'aide qu'elle apporte, c'est-à- récents fait apparaître un bilan contrasté. La proportion dire qu'elle accorde aux pays qui suivent une saine poli- 71 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 2 : L'EFFICACITÉ DE LA POLITIQUE SUIVIE ET de veiller à ce que les dispositions voulues soient prises avec LA QUALITÉ INSTITUTIONNELLE INFLUENCENT LA les parties prenantes pour dégager le consensus et permet- PERFORMANCE DU PORTEFEUILLE D'UN PAYS tre l'adhésion que nécessitent les réformes au niveau secto- riel et national. Perspectives Des progrès ont incontestablement été réalisés, mais il reste beaucoup à faire. De plus, la portée à long terme des amélio- 100 rations notées durant l'exercice 97 reste à déterminer. 80 L'analyse montre que la poursuite des efforts visant à améliorer la qualité de la gestion pourrait permettre à la 40 SBanque d'atteindre l'objectif fixé par la direction, à savoir parvenir à un pourcentage de projets satisfaisants de 75 pour cent en l'an 2000. En ce qui concerne l'améliora- tion du portefeuille au cours de l'année prochaine, les Éleve Faible de la politique objectifs énoncés dans le Rapport annuel sur la perfor- F Faible Fblmance du portefeuille consistent notamment à porter au institutionnee minimum à 75 pour cent la proportion des opérations dont la supervision est satisfaisante et à au moins 85 pour cent Source: Données de l'OED celle des nouveaux projets dont la qualité initiale est satis- faisante. Lorsqu'on applique ces objectifs dans une analyse de sensibilité portant sur plus de 300 des dernières opéra- tique et qu'elle recoure aux services hors prêt de manière tions évaluées (exercices 95-97), il apparaît que, s'ils sont plus stratégique pour favoriser l'adoption de politiques atteints simultanément, on en arrive à 75 pour cent de pro- appropriées et d'une bonne gouvernance. Cela signifie aussi jets aux résultats satisfaisants. Si la performance de l'em- mieux choisir et échelonner les instruments et mesures prunteur continue à s'améliorer comme elle l'a fait récem- nécessaires en fonction des compétences des institutions ment, il est possible d'envisager un chiffre proche de locales et des capacités de l'em- 80 pour cent4. prunteur. Renforcer la coordina- M. Wolfensohn, président de la Banque, prône une il est absolument tion des services consultatifs, ana- approche énergique consistant à améliorer la qualité des lytiques et de prêt revêt une méthodes de la Banque afin de indispensable de importance cruciale. parvenir à 100 pour cent de Un examen des leçons tirées résultats satisfaisants. Bien qu'il Une stratégie mettre en place d'études récentes sur l'aide aux ne soit pas possible d'atteindre pays confirme l'importance du rôle cet objectif d'ici l'an 2000, agir sélective consiste des politiques et de la politique suivie du point de en ce sens permettrait de parve- vue de l'impact global de l'aide de nir à des résultats globaux satis- à mieux adapter des institutions la Banque. Il indique une fois de faisants pour 80 pour cent des plus que les réformes aboutissent projets au moins. Si la perfor- la politique de la saines pour rarement lorsque l'engagement de mance de l'emprunteur s'amé- l'emprunteur laisse à désirer et liore elle aussi, il est même possi- Banque aux obtenir une plus qu'il n'est pas suffisamment ble d'arriver à 85 pour cent. impliqué. L'analyse des facteurs à Il semble à la fois souhaita- grande efficacité l'origine des améliorations de la ble et nécessaire de viser cet capacités des qualité du portefeuille ne fait que objectif en établissant un plan renforcer ce message : le nombre par secteur et par région. Un des projets a fortement diminué dans les pays où la gestion plan stratégique de ce type per- macroéconomique laisse à désirer3. Il est donc extrême- mettrait au portefeuille de la Banque de redevenir aussi sain ment important de soutenir les capacités de l'emprunteur et qu'à la fin des années 70. Cela exigerait bien entendu que 72 Ap p nd ix toutes les Régions et tous les Réseaux redoublent d'efforts de la Banque et recourir à des approches plus participatives pour améliorer la qualité initiale, la supervision, la gestion et mieux adaptées qui permettent de renforcer l'adhésion du risque, le suivi et l'évaluation rétrospective. aux projets et d'obtenir de meilleurs résultats sur le terrain. Pour améliorer l'efficacité à long terme de l'aide de la Banque, il faut non seulement déployer des efforts soutenus Note afin d'améliorer la qualité et la gestion de son portefeuille, 1. La situation économique d'un grand nombre de pays en mais aussi renforcer les capacités de l'emprunteur au niveau développement (et surtout de ceux où l'ajustement a été le plus réussi) de la sélection, de la conception et de la gestion des projets. a commencé à s'améliorer vers la fin des années 80. La plupart des Cela exige aussi que l'on se montre plus sélectif, ce qui signi- opérations achevées et évaluées pendant l'exercice 96 avaient été fie lafoi accrde de prts ax pys mpruteus aant approuvées en 1989. fie à la fois accorder des prêts aux pays emprunteurs ayant 2. Projets ayant cessé de faire partie du portefeuille entre les exer- des politiques et une gestion des affaires publiques appro- cices 93 et 97. priées et offrir aux emprunteurs des services et des instru- 3. De 800 projets environ pour l'exercice 95 à 465 pour l'exer- ments adaptés à leurs capacités. Il s'agit non pas de « tirer cice 97. Selon une analyse de sensibilité portant sur les tendances des les bons numéros », mais d'avoir dès le départ des pays atta- projets à risque, à peu près la moitié des améliorations sont dues à des chés aux principes du développement et d'une bonne gou- facteurs tenant au pays lui-même (Rapport annuel sur la performance - vernance et de moduler la réponse de la Banque en appor- du portefeuille, novembre 1997). tant une aide financière et autre adéquate et opportune. 4. Ces résultats sont basés sur le nombre de projets, et non sur t le volume des décaissements. Du fait du modèle sur lequel reposaient Appliquer une stratégie sélective signfie aussi établr de les tests de sensibilité, il était impossible d'établir des prévisions par meilleurs partenariats afin de maximiser l'impact de l'action volume. 73 REFERENCES The word "processed" describes informally reproduced Murphy, J. 1997. Mainstreaming Gender in World Bank works that may not be commonly available through Lending: An Update. World Bank:Washington, D.C. libraries. OECD (Organization of Economic Cooperation and Bauer, P.T. et al. 1966. Two Views on Aid to Developing Development). 1996. "Shaping the 21st Century: The Countries. Institute of Economic Affairs: London. Contribution of Development Cooperation." Develop- Boone, P. 1994. The Impact of Foreign Aid on Savings and ment Assistance Committee. May. Processed. Growth. London:London School of Economics. OED (Operations Evaluation Department). 1996a. Evalu- Burnside, C., and Dollar, D. 1997. Aid, Policies, and ation Results: 1994. World Bank:Washington, D.C. Growth. Policy Research Working Paper. World Bank: . 1996b. Annual Review of Evaluation Results 1995. Washington, D.C. World Bank:Washington, D.C. Cassen, R. et al. 1989. Does Aid Work? Oxford: Claren- _ . 1996c. "The Evaluation of Economic and Sector don Press. Work. A Review." Processed. Datt, G., and Ravallion, M. 1996. Why Have Some Indian _ . 1997a. "The Effectiveness of the Bank's Appraisal States Done Better than Others at Reducing Rural Process: An OED Study." Processed. Poverty? Policy Research Working Paper. World Bank: . 1997b. Brazil-Learning from Best Practice in Five Washington, D.C. Urban Projects. World Bank:Washington, D.C. Deininger, K. Squire, L. Basu, S. 1997. "Does Economic _ . 1997c. "Annual Report on Portfolio Performance, Analysis Improve the Quality of Foreign Assistance?" Fiscal Year 1997." Processed. World Bank:Washington, D.C. Processed. Papanek, G. 1972. "The Effects of Aid and Other Resource Devarajan, S. Squire, L., and Suthiwart-Narueput, S. 1997. Transfers on Savings and Growth in Less Developed Beyond Rate of Return: Reorienting Project Appraisal. Countries," Economic Journal 82(327):934-50. World Bank Research Observer 12(1):35-46. Papanek, G. 1973. "Aid, Private Foreign Investment, Say- Feyzioglu, T. et al. 1996. Foreign Aid's Impact on Public ings and Growth in Less Developed Countries." Eco- Spending. Policy Research Working Paper. World nomic Journal 81(1):120-30. Bank:Washington, D.C. Ravallion, M. and Chen, S. 1997. "What Can New Survey Griffen, K. 1970. "Foreign Capital, Domestic Savings and Data Tell Us about Recent Changes in Distribution and Economic Development." Bulletin of the University Poverty?" World Bank Economic Review 11(2):357-82. Institute of Economics and Statistics (Oxford Univer- Stout, S. et al. 1997. Evaluating Health Projects: Lessons sity) 32(2):99-112. from the Literature. World Bank:Washington, D.C. Killick, T. 1991. The Developmental Effectiveness of Aid Subbarao, K. et al. 1997. Safety Net Programs and Poverty to Africa. Policy Research Working Paper. World Reduction: Lessons from Cross-Country Experience. Bank:Washington, D.C. World Bank:Washington, D.C. Knack, S. and Keefer, S. 1995. "Institutions and Economic World Bank. 1990. World Development Report 1990: Performance: Cross-Country Tests Using Alternative Poverty. Washington, D.C. Institutional Measures." Economics and Politics . 1995. Priorities and Strategies for Education: A 7(3):207-27. World Bank Review. Washington, D.C. Mauro, P. 1995. "Corruption and Growth." Quarterly . 1997a. Progress Report on Poverty Reduction Fiscal Journal of Economics 110:681-712. Year 1996. Washington, D.C. Mosley, P. 1995. "Aid Effectiveness: A Study of the Effec- _ . 1997b. World Development Report 1997: The State tiveness of Overseas Aid in the Main Countries Receiv- in a Changing World. Washington, D.C. ing ODA Assistance." University of Reading. Processed. . 1997c. "Health Nutrition and Population: Sector Mosley, P. Hudson, J. Horrell, S. 1992. "Aid, the Public Strategy." Washington, D.C. Sector and the Market in Less Developed Countries: A . 1997d. Global Economic Prospects and the Devel- Return to the Scene of the Crime." Journal of Interna- oping Countries: 1997. Department Working Paper. tional Development 4(2):139-50. Washington, D.C. 75 THE WORLD BANK 1818 H Street, N.W Washington, D.C. 20433, U.S.A. Telephone: 202-477-1234 Facsimile: 202-477-6391 Telex: MCI 64145 WORLDBANK MCI 248423 WORLDBANK World Wide Web: http://www.worldbank.org/ E-mail: books@worldbank.org 14210 9 780821 242107 DESIGN: JEFFREY KIBLER/THE MAGAZINE GROUP ISBN-0-8213-4210-X