V - -11 36 MICROFICHE COPY Report No.:11536 Type: (MIS) Title: EFFECTIVE IMPLEMENTATION KEY Author: WAPENHANS, W Ext.: 0 Room: Dept.: WHITE COVER SEC. DOC. (THE WAPENHANS R' ".. K TO D': EVEPMENT' IMACT. Portfolio M Task Force.. ....;.- -~ ~ ~~~~etme 22 19.,''.' . n>wEe.5.';N=:?92 Acronyms and Intls ARIS - Annial Rcview of ImplemenWion and Supervision ARPP - Ann Report on/Review of Portfolio Performance CAM - Country Assistance Management Process CESW - Country economic and sectr work CHRIS - Comprehensive Human Resoure Information System CIR - County Implention Review COD - Centra Ope¢ations Depament CPPR - Country Portfolio Perfrmance Review CSP - Coutry Strategy Paper DEC - Development Economics Department ECON - Economic Analysis of Projects EDI - Economic Development Instte EPS - Executive Project Summary ERR - Economic Rate of Return FEPS - Final Executive Project Summary FRS - Risk Management and Financial Policy Department GDP - Gross Domestic Product IBRD - I onal Bank for Reconstruction and Developmeat ICB - Iernational Competitive Bidding ICR - Implemention Completion Report IDA - International Development Association IEC - International Economics Department IEPS - Initial Executive Project Summary IME - International Monetary Fund TF - Inormation, Technology and Facilities Department LAC - Latin America and the Caribbean Region MIS - Management Information System NGO - Non-Governmental Orgaization OD - Operational Directive OED - Operations and Evaluations Departnent OPRC - Advisory Operations and Procurement Review Committee ORG - Organizational Planning Staff OSP - Sector and Operations Policy PCR - Project Completion Report PMD - Personnel Management Department PPAR - Project Performance Audit Report PSE - Programs of Special Emphasis RAL - Review of Adjustment Lending RCI - Cote d'Ivoire RVP - Regional Vice President SAL - Structural Adjustment Loan SAR - Staff Appraisal Report SECAL - Sectoral Adjustment Loan SOE - Statement of Expenditure SW - Staffweeks TOR - Terms of Reference UN - United Nations UNDP - United Nations Development Program WDR - World Development Report CONTEN1S Intouction and Summary of Recommendations L CONTEff ............................... 1 I. THE CONDMON OF THE PORTFOLO ......... ............. . 2 m. THECAUSESOFDECLUINNGPORTFOUOPERFORMANCE ............ S GLOBALIFACTORS 6... 6 COUNTRY FACTORS ........... .......7 PROJECr FACrORS .... . 7 IV. THE BANK'S ROLE IN SUPPORT OF PORTFOLIO PERFORMANCE ...... 12 QUALITY AT ENTNR RY....... 13 Pre-Appraisal ... ... .. .. .... . 13 AppraisaI 14...... .14 Negotiations 16..... 16 QUALIY OF PORTFOUO PERFORMANCE MANAGEMENT ....... 17 Implementation .... ....... 17 Management and Staff Incenfives ...... .. .17 Resources .1.1*........... 18 TheSkI1MiB x. ...... . . . . ...... 19 FieldOffices 2....20 QUALUY OFNTrERACTIONWrTHBORROWERS .. 21 FEEDBACK ........nd.ot.a.o . ............. 23 Miuemn tand Motivation ... ............23 Learning Lessons ftom Expeience ......................... 23 V. CONCLUSIONSANDPRINCIPALRECOMMDENDATIONS ...........24 A. NTRODUCE THE CONCEPT OF COUNTRY PORTFOLIO PERFORMANCE MANAGEMENT LINKED TO THE BANK'S COREBUSINESSPROCESSES ............ 25 B. PROVIDE FOR COUNTRY PORTFOLIO RESTRUCTURING IN ADJUSTING COUNTRIES INCLUDING THE REALLOCATION OF UNDISBURSED BALANCES OF LOANSICREDfTS .......... 26 C. IMPROVE THE QUALrrY OF PROJECTS ENTERING THE PORTFOLUIO . ............................... 26 D. DEFINE THE BANK'S ROLE IN AND MPROVE T PRACTCE OFPROJECTPERFORMANCE MANAGEMENT .............. 28 E. ENHANCE OED'S ROLE AS AN INSTRUMENT OF INDEPENDENT ACCOUNTABILrrY AND REFOCUS EX POST EVALUATION ON SUSTAINABLE DEVELOPMENT I=PACT .. . . . . . .. . . . . . . .. . . . . .2 F. CREATE AN INTERNAL ENVIRONMENT SUPPORTIVE OF BETrER PORTFOLIO PERFORMANCE MANAGEMENT . . 3 INSTIff ONALT PLICATIONS .. .... .. . 32 ANNEXES Annex A: Supporting Measures Related to the Principal Reommendations Annex B: Summary Proceedings of Borrowers' Conference Annex C: Towards a Results-Orented Evaluation and Rating Methodology for Bank Supported Operations Annex D: Operaions Evaluadon in the Bank Annex E: List of Working Ppern Annex F: Terms of Reference of The Task Force Annex G: Task Force Membeship Report of the Portfolio Mnemen Task Force EFFECTIVE IMPLEMENTATION: KEY TO DEVELOPMENT IMPACT Introduction and Sumary of Recommendations A. Background i. The Portfolio Management Task Force, announced by Mr. Preston in Febnry, 1992, has examined problems affecting the quality of the Bank's portfolio of loans and credits. This report assesses current problems in the portfolio of projects, discusses the Task Force's assessment of their causes, and sets forth the Task Force's recommendations for change. It does not attempt to provide a comprehensive assessment of the Bank's overl record. As a development instiuton, the Bank has continuously broadened its activities. Increasingly ambitious goals and development priorities have expanded its reach in a rapidly growing membership at a time of substantal volafility in the global economy. The projects the Bank supports - as a lender of last resort - of necessity entail substantial risk taldng. This calls for vigilance, realism, and consrucve self-evaluation. The Bank must be no less restrained in diagnosing and seelkng to medy its own shortcomings than it is in seeking to help member countries recognize and address theirs. For only through rigorous and continuous self-assessment based on exacting standards can a large and influential institution such as the Bank maintain its effectiveness. ii. The Task Force's review of existing documentation' and its own analyses have been deepened by "feeder paperse prepared by senior consultants and staff members, by focus groups convened to discuss specfic problems, and by several specia surveys of staff and management opinion. In addifton, the Task Force benefited gready from three internationalwworkshops - respectively of Borwer officials familiar with the Bank, representatives of other assistance agencies, and reesentatives of the intational contractrs industry. Much of the Task Force's assessment reflects views that are widely held. Many of the Task Force's recommendations build on existig best practices and on initiatives already underway in various parts of the Bank. Key doument mview inclde: Sntutb Annual Ret of m _and Suegision - Fiscal Yaw 1991. RIMo of ft loasl Forc on do lol#hipb of Lea Procesisgto Pxjict OUf (Marh, 1992), OED'sRt Rpr lB pm mP tSupgisio(Draft, March, IM), EMgno Aalxis of Pioiae.t: Town& an ARoach to Evadoion for dte 1990s Daft BCON Rt, Juno 19, 1990), MOW= Assist in the 1990s November, 1991), Stxnthn CoM Tenms i the Africa Rei (Jun. 1991), 2bWM i SAL &W1iW and MgLtnn (OBD, June, 1991), Cou C o_ to DeAgno Pxe (Heaver and bJaml, Wod Bank Di;mon Papw #4, 1986). B. Couclusions iii. Five conclusions are basic to the recommendations of the Task Force: * On-the-srvand beneflts The Bank's success is determined by benefits "on-the- ground" --sustainable development impact-- not by loan approvals, good reports or disbursements. * Commitmentandimplementabilfty: Successful implementation requires commitment, built on stakeholder participation and local "ownership". * uably at entry; implementation planning: Quality at entry into the portfolio is a critical determinant of success in project outcome. Specific factors critical to project success, and obstacles to be overcome, must begin to be addressed as early as identification. Results-oriented implementation planning as a basis for later monitoring is essential. * Counti focus: The project-by-project approach to portfolio performance management needs to proceed within a country context to address generic problems of implementation and systemic opportunities for portfolio improvement, and to focus accountability withir the Bank for portfolio results. D Taking account of potifollo perfonnance: If the Bank is to remain effective, portfolio performance must be taken into account in the Bank's country assistance strategies, business processes, and personnel policies. C. The Problem iv. Deciningpo,lfolioperfonnance: Measured against the stringent performance criteria the Bank adopted for itself nearly two decades ago, the overall performance of the portfolio remains aisfactory. More than 75% of the projects demonstrate acceptable performance during implementation. There has been, however, a gradual but steady deterioration in portfolio performunce. The share of projects with "major problems" increased from 11% in FY81 to 13% in PY89 and 20% in FY91. In the Annual Report on Implementation and Supervison (ARIS) for FY91, 30% of the projects in their fourth or fifth year of implementation were reported as having major problems - including 43% of those in Water Supply and Sanitation, and 42% of those in Agriculture. Performance problems were most severe in Africa; in the Latin America reon, two countries accounted for nearly 50% of the problem projects, but other regions also had 30-40% of problem projects in their 4-5 year old portfolios. Worldwide, 39% of the bofrowing countries had more than 25% problem projects. By the Operations Evaluation Departmt's (OED's) reckoning, based on staff assessments after completion of disbursement, the decline has been more severe. The number of projects judged unsatisfactory at completion increased from 15% of the cohort reviewed in FY81 to 30.5% of the FY89 cohort and 37.5% of the PY91 cohort. Perhaps reflecting the decline, cancellations have increased by some 50% In the past three years. -I W- v. There are other factors corroborating the decline. TMe actual time required for project completion (nearly 7 years) exceeded the time estimated at appraisal by an average of more than 2 years. Borrowers' compliance with legal covenants -especially financial ones- remained startlingly low. Whatever the causes of noncompliance, absence of enforcement does not induce the behavior epected, and the credibility of loan agreements as binding documents has suffered. Despite these trends, the Bank's opm at appraisal, as indicated by the gap between estimat of economic return at appraisal and at completion has inceased. vi. Contuibungfacton of decline: Factors other tham poor design, poor management and poor implementation contributed prominently to these disturing trends - including worsening global conditions (e.g. the oil shock, the debt crisis, and declining terms of trade) and deteriorating country istitutional, policy and macroeconomic envirnments. Policy respones, including structural adjustment, have often changed priorities and resained outlays for public investment programs. The Bank's portfolio was not -and should not have been- immune to these events. Also, the need for more complex and challenging undertakings in response to new development priorities has played a role, as may have more realistic project perfo ace tings i recent years. The most common types of problems reported were (n descending order) institutional constraints including Borrower inertia, shortages of counterpart financing resuldting from deterioration in the macro environment, poor project management and defective procurement. Given the severity of these problems, technical deficiencies did not emerge as prominent causes of decline. D. The Baks Role In Support of Project Implementation vii. Emphaos on lw approval: The uncontrollable -i.e. global- causes and the deficiencies in national policy, regulatory frameworks, and istitutional capabilities are critical denants of project performance. However, there are also aspects of Bank practice either may contribute to portfolio management problems or are insufficiently effective in resolving them. Underlying many of these aspects is the Bank's pervasive preoccupation with new lending. In the eyes of Borrowers and co-lenders as well as staff, the emphasis on timely loan approval (described by some assitance agencies as the "approval cultue) and the often active Bank role in preparation, may connote a promotional -rther than objective- approach to appraisal. Bornowers allege that loans feature conditions thought to be conducive to approval by management and the Board, even where these may complicate projects so as to jeopardize successful implementation. As a result, the quality of proects at the time of their entry into the portfoiow -quality being defined to include inter alia implementability and sustained local commitment- is not always what it might be. viii. Tre,anent of risks, sensiviy, and implementablllty in design and apprasal: The pervasive emphasis on loan approval is not matched by equal emphasis on implementation planning and identification and assessment of major risks to project performance. Sensitivity/risk analysis is limited, and virually no attention is given to macoeconomic risks. The project concepts are not always wel calibrated to the implementation capacity of execuing agencies. Statistical analysis has confirmed that both the number of cofinanciers and the number of project components corelate substantially with unsatisfctory performance. Yet there remains a bias for complexity - perhaps caused by the urge to include as many novel features as possible to secure a favorable management and Board response. - iv - ix. Weaknessaes lpofaboperommance management: The Bank's role in supporting project implementation is inadequately defined. Facilitation of implementation, complne review, and "core supervision (i.e. of end use, procurenent and disbursements) are all normal pats of portfolio performane management. Substantive implementation asstnce beyond trouble shooting" is an aspect of portfolio perfomance management that, if neded, can be arranged with Bank help or, if it so decides, provided directly. Staff needs guidance as to whedier it should become involved in providing substantive implementaton assistance. The elabortion of such guidance should take into account that the Bank may not possess a compaative advantage to render such support, that its budgetary implication is open-ended, and that a preeminent role of Bank staff may undermine "ownership" on the part of the Borrower. The methodology for project performance rating is deficient; it lacks objective criteria and transparcy, and poorly identifies the risks and fctors that most influence project perfonce. While problem projecs receive special attention, recourse to project restructuring or the exercise of remedies is infrequent. Procurement -which is estimated to take more than a third of the Bank's total staff time devoted to portfolio performance management- is another cause of major problems. Part of the weaknesses rest on poor understanding by executing agencies of Bank policies and requirements; another part is poor country capabilities and practices; a third cause is inadequate bid documents (when ICB is required) which take extensive time and resources to review and retify. x. Lims of the poject-by-project approach: For the most part (although with some exceptions), portfolio performance management is based on a project-by-project approach. Country-wide implementation reviews are not as yet standard prctice. As a result, generic oountry or sctoal obstacles to sucoesfl implementation, even if identified, are not systmatically addsed. Also, because portfolio performance is not explcitly taken into account in the formulation of country assistance stategy, business planning, the CAM process, lending allocation reviews and performance assessments, these processes lose in ralism and do not reinforce managerial accounability. xi. Need for dvelopment Impac e ndaaton: An independent and robust Bank-wide evaluation system has been put in place. Its contribution is incasingly being felt throughout the institution, though its attention to anal development impact remains inadequate. Project Completion Reports (PCRs) and most OED audit work seek to evaluate and draw lessons from the implementation of proects. Prpared shordy after the last disbursement, PCRs tend to be completed when benefits have not yet begun to mateialize. Uttle is done to ascertin the ata1 flow of benefits or to evaluate the sustainability of projects during their opeatonal phase. Ihis weakens accountability for sustainable development impact based on observable results and, in consequence, impairs the Bank's ability to learn what really works and what does not. L Recoend xii. To improve upon the Bank's pordolio performance management the Task Force advances Cn Chapter V and Annex A) six principal recommendations and a comprehensive program of measu for their implementation. The principal recommendations are: * Introduce the concept of country portfolio performance management linked to the Bank's core business processes; * Provide for country portfolio restucturing in adjusdng countries including the realocadon of undisbursed balanes of loanslcredits; * Improve the quality of projects entering the portfolio through better project analysis, apprisal methodology, and contctul arangements; * Define the Bank's role in, and imprve its practes of project prformnc management; * Enhance OED's role as an instrument of independent accountability and refocus a post evaluation on sustainable development impact; and * Create an intemal environment supportive of better portfolio perfce management. Together, these recommendations and the implementing measures comprise a long-term program of institutional change in need of sustnined leadership from management at al levels and of continuous support from the Board. xiii. The principal thrust of these recommendations is to: - Make the country portfolio the unit of managerial accountability ii zomposition, size, and perfnance; - Concentrate atention seectively on critical performance variables throughout the prqect cycle; * Define the Bank's role in support of implementation so as to promote borrower commitment and accountability; and * Rebalance interal priorites and incentive systems so as to ensure continuous and adequate emphasis on portflio performance management. The ultimate objective of these changes is the achievement of susainable development benefits through efficient implementtion of a high-quality porfolio. xiv. The institutional implications are threefold: 1. This renewed focus on implementation and portfolio performance will sustain the Bank as a leader not only in development policy initiatives but also in effectively supporting its members in the painstakdng task of development on the ground; 2. Added analytical rigor transfnomed into more decisive action prior to approval and more rigorous monitoring in the course of portfolio performance management are intended to enhance the quality of the potolio, but at the same fime they may make annual lending programs more volatle, and they will require increased agility in the management of committed resources; and .vi 3. Potendal efficiency gains will only partially offset the extr costs likely to be incurred in the short- and medium-tenn. Managerial discreton in redeployment should be guided by operational judgement and not be curbed by the imposition of rigid celings on portfolio performance management work or the applicadon of budget 'norms to individual projects. Attention to the existing portfolio should have a priority claim on staff resources over new lending. Current concems of borrowers. donors, staff and managers alike make this an especially propitious dme to act with dispatch on the proposed program of change. I. CONTEXT 1. In his memorandum of February 7, 1992, establishing the Task Force on Portfolio Management, Mr. Preston stated: 'Successful implementation of approved operations outweighs new annual commitments as an indicator of the Bank's development effectiveness."J The Bank and IDA currently have close to US$140 billion in lending commitments helping finance about US$360 billion worth of projects and programs under implementation. Annual disbursements against some 113 country portfolios containing about 1800 projects are estimated to have reached US$16.5 billion in FY92Y In FY93, total disbursements are expected to increase to about US$20.4 billionY The Bank's support for the effectivt implementation of its portfolio is one of the most important forms of development. assistance it can render, for without effective implementation, the benefits anticipated at appraisal will not materialize. 2. Progress made in the evolution of development policy in the past decade, the inclusion of more challenging and more complex projects' in the lending program, and a declining trend in portfolio performance, combine to add priority to the task of ensuring sustainable development impact through successful implementation. One of the Bank's most pressing medium term challenges will be to ensure that sustainable benefits result from the resources it provides. That is why the Bank must now focus attention on implementation and must adapt its processes, incentives and skills and deploy its resources to support the effective management of the performance of the portfolio of projects it finances. While the Bank may influence, it does not control and cannot be responsible for the actions of the Borrowers, who own the projects the Bank helps finance. This report outlines a program -built largely on initiatives and changes already underway in many parts of the Bank- to focus continuous attention on the performance of its portfolio. The progrm is designed to enhance the development impact of the Banks portfolio through a return to diagnostic sector work prior to prepration, better appraisal practices, more effective implementation support, clearer accountability for portfolio performance results, and an independent and credible evaluation process keyed to the objective of institutional accountability. 3. While the Task Forcel' initially interpreted the scope of its inquiry to encompass the 'downstream' stages of the project cycle (i.e. from negotiations through impact evaluation) it recognized that the ultimate success of a project is, to a significant degree, determrined by what happens in the 'upstream" stages of the cycle (i.e. identification, preparation and appraisal) and by the effectiveness of linkages between portfolio performance, x post evaluation and the conceptualization and design of new projects. The scope was therefore extended to encompass factors that affect the quality of projects when they enter the portfolio and others th influence the process of leaming from expenence. The term "project performance management' as used in this report, is essentially synonymous with what is usually inown in the Bank as 'proect 1t Ciud to the Board on March 12, 1992, see Annex F. ' Reference throughout tO rport is to both IBRD and IDA Unles otherwise indica. v Review of World Bank Programs and FY93 Budges, May 8, 1992, Annex Al and Annex AS. N Referces to projects include operadons of al types. h' For composition se Ane 0. -2- supecvision."V The more inclusive tem 'portfolio perfonnance management" is used to connote conntry focus and accountability. The "upstream" or "lending" phase of the project cycle is most often refrred to in terms of its sub-phases (dentification, pre-appraisal, appraisal and negotiations). And the post-in.plementation phase is generally termed the "operational" phase. 4. There is a clear and immediate need for significant changes in the way the Bank helps to plan and evaluate projects, monitors and supports their implementation, and applies lessons of portfolio performance to ongoing work. The Task Force believes the proposed changes are, for the most part, evolutionary in nature; that they are consistent with prevailing trends in borrowing countries and with the views of Bank staff; that they support the development mandate the Bank has defined in its policies; and that they reflect the need for efficiency as well as effectiveness. Against the background of the respective mandates and accountabilities of the Borrower and the Bank, change is needed in five broad areas: * Establishing the practice of "Country Portfolio Performance Management" and its linkges to and integration with the Bank's core business processes; * Imprving qualiy at entry into the portfolio; * Strengthening project perfonmance mangement; * Evaluating project results after implementation; and - Creating a supportive evironment for improved portfolio performance management. S. The Task Force's conclusions and recommendations on each of these five areas are covered in Chapter V. Chapter H discusses the current condition of the portfolio. Chapter m reiews the causes of the decline in pordolio performance. And Chapter IV defines the role of the Bank in project and portfolio performance management. H. THE CONDMON OF TH PORTFOLIO 6. The Task Force's assessment of portfolio performance is based on several sources. First, quantdative infomation was drawn from the MIS-Portfolio Module, the Annual Review of Implementation and Supervision (ARIS)X', the Operations Evaluaton Department's (QED) Annual Review of Evaluaton Results and from such recent reports as those of the Task Force I IT Task Forco has dropped the use of supervision' as it distort BkBor relationips by implying rt t Annual Report on ta and Suevi". Fiscal Year 1991. -3 - on Lending Qualityl, the ECON Task Force1' and the OEID Report on SupervisionW. Second, teeder studies were prepared specially for the Task Force, some of which generated new data. Third, qualitative information on the Bank's business performance was drawn from interviews with managers, staff and focus groups, and from three international conferences - with, respectively, representatives of interational contctng industriesul, Borrower governments and institutionsW, and development agenciesw. 7. Quantitative information about the portfolio must be intepreted with caution since currmnt Bank methodologies for assessing performance during project execution, outcome at completion, and likely development impact, are far from perfect. There is no consistent rating methodology based on objective criteria agreed with the Borrower and applied from the time of appraisal through completion and impact evaluation. Recent efforts to promote realism in ratings by staff may have contributed to the apparent decline. Current ratings during project execution focus largely on progress, whereas completion ratings focus mainly on implementation results and predicted development impact. Performance objectives reflect the exacting standards the Bank has set for itself, i.e. a minimum 10% rate of return in real terms or its qualitative equivalent in cases in which quantification of benefits is not feasible. Current country portfolio ratings are based on assessments for individual projects under execution which do not weight their relative importance in the portfolio. 8. Bearing both the standards and these reservations in mind, the ARIS data show that, in FY91, 80% of the current portfolio was estimated to be performing well and that 20% was incurring major difficulties. The estimated average return on the portfolio remained weU above 10%.W An average 20% incidence of projects in difficulty may not be considered excessively high given the Bank's role as a development institution. Certainly, if the incidence were wy low it could imply the Bank was not taking risks in a high-risk business. There is, however, little doubt that the high proportion of borrowing countries with poorly performing portfolios (39 %) Is a cause for concem. ' 'Report of Task Foce an Relationship of Loan Processing to Lending Quality. - (March, 1992). S' 'Economic Analysis of Pojects: Towads an Approach to Evaluation for the 1990s. (Draft BCON Repot), June 19, 1992. e' 'Bank Expenience in Project Supevision'. OED, March, 1992. Il' Conference held in Washington on May 27 with representatives of conbactors from Europe, Japan and USA. Ia' Confrence held in Wasbington on May 28 and 29 with 18 individuals with exteive knowledge of Bank operatns from 12 Bonowing couties that account, btween them, for more tasn 50% (u $ tems) of the active portfolio. iY Confevence held in Washington on June 4 and 5 with repsentatves of major multlaoteal and bilaeal development insttions. I~' OED: Annual Review of Evaluation Results 1991; R92-162, Aust 21, 1992. gi Defined as county pordolios in which 25% or mre of projects are problem projects (i.e. raed 3 or 4 in Form 590 reprts). -4- 9. The main concern, however, is the trend in portfolio performance. A gradual deterioration is apparent in portfolio performance over recent years, the share of problem projects under execution having almost doubled from an average of about 10% in FY79-81 to about 17% in FY89-91S. Deterioration has accelerated in the last three years. OED data based on "completed" projects, confirm an increase of unsatisfactory projects over the past decade, and indicate a significantly higher proportion (35 %)U', than does the ARIS data based on the current active portfolios. The challenge to the management and staff of the Bank is not merely to arrest the erosion in portfolio performance but to reverse the trend in the number of country portfolios showing a high incidence of problem projects. 10. The portfolio's deterioration has been steady and pervasive and is apparent in an increasing number of sectors. While the performance of agriculture and water supply projects has been relatively weak for several years (25% problem projects in PY91 according to ARIS), some traditionally strong performing sectors are now affected too: among them (in FY91); telecommunications (18%), power (22%), industry (17%), and technical assistance (27%). New areas of lending are also encountering major problems: poverty (28%), environment (30%), and private and public sector reform (23%). Adjustment lending was less affected than investment lending: in FY91 only 11% of adjustment projects were in trouble. 11. As evaluated in PCRs after the completion of implementation, the overall success rate declined from an average of 87% in FY79-81 to 65.7% in FY89-91. Subsequent PCR audits by OED suggest a furither decline of the success rate by about 10%MA At the same time, according to a recently published study of 1,015 investment projects,'9' there is a systematic and growing bias towards excessively optimistic rate of return expectations at appraisal. Optimism does, however, diminish as implementation progresses, and ratings generally become more realistic by the fourth and fifth years of implementation. There are too few impact evaluations by OED to allow conclusions about operational performance and actual results. M/ There is a statistical bias in this evidence, because the data for FY79-FY81 were based on three project ratings, with only one category (3) for problem projects, while the data for FY89-FY91 were based on four project ratings, with two categories (3 and 4) for problem projects. Furthermore, the portfolio definition differs between the two time periods. The FY79-FY81 portfolio included oly projects which had become effective whereas the definition inroduced in FY89 included projects as soon as they wer preseted to the Board. herefore, proects rated 3 or 4 as a result of signature or effectiveness bottlenecks are included in the FY89-91 portfolio, but excluded from the portfolios for earlier years. AZ' The term munafactory, as used by OED, denotes projects that do not reach the mmum rate of retrn of 10% unless it can be demonstrated that their objectives have been otherwise achieved. A few projects that do return 10% may also be deemed 'unsatactory" for special reasons. Y' 172 OED audits of PCRs in 1990-91 confinned PCR ratings in 85% of the cases while in another 10% a switch to -unsatisfactory- ratings ocwrred. OED, Annual Review of Evaluation Results, R92-162, Page 2, August 21, 1992. I' ClGerhad Pohl and Dubravko Mihaljelc 'Project Evaluation and Uncertainty in Practice: A Statistical Analysis of Rate of Return Divergences of 1015 World Bank Projects" - World Bank Economics Review, Vol. 6, No. 2, May 1992. The emerging gap was first documented by OED in its 1988 Annual Report on Evaluation Results. -5- Table 1: Comparom of Success Rates for On-Going Projects and hrjects Completed in Year FIcal Year A. On-on Portlo D. POJect Competed la Yea C. Dirergoe Ratio % WePerformIn Pjets * % Satisfatory (% by whieh A inccaeds D) (ARIS) (ODD) All Years Yer 45 ra Yeas 4-S 91 80 70 63 27 11 90 86 79 69 25 14 89 87 83 70 24 19 Daed as p se ni 3 or4 ovenfa o XARIS. 12. Further refinement of the analysis would not alter the conclusion ftat the portfolio is under pressure. This pressure is not temporary; it is attributable to deep-rooted problems which must be diagnosed and resolved. The Bank's limited use of remedies is sending the wrong signals to both Borrowers and staff. The cost of tolerting continued poor performance is highest not for the Bank, but for its Borrowers. Improvements must be consistnt with the causes of decining performance. m. THE CAUSES OF DECIJNING PORTFOLIO PERPORMANCE 13. The findings of recent OED and Country Operations Department (COD) stdies and of the research underlying the 1991 World Development Report (WDR) on factors influencing proect performanceW, were validated in interviews with Bank staff and at the Task Force conferences with Bonowers and representatives of intenational development agencies in May and June. Consequently, the Task Force concludes that t. performance of Bank-financed projects and programs is a function of three closely interrelated factors: * At a global level, the volaflity of the inteational economic environment during the last two decades has strongly affected-mostly indirectly-the outcome of Bank- supported projects;ly' * At a country level, maeconomic conditions and policies, changes in developmental priondties, deficient regulatory environments, and the lack of or decline ' See WodDeJ4nmWtRep1991: VawileagefDpe,ode andDielK _ufn,TheForgote Rationale for Policy Refom: Tbo Producivity of investmt Projects, (WDR Backround Paper). ' ConCirmed by finings of tle Tak Force on The Reationship of LAn Prcss to Proeoct Quality; March 27,1992. 5' Polidcal insability can, of couse, de an eatire country portfolio. in capacities of local institutions, have had a significant direct and indirect impact an project performance; and At the project level, institutional capcity, 'ownership" and technical design are cridcal determinants of project outcome. The broadening scope of lending for programs of special emphals and increasing recourse to cofinancing have added to project complexity, even as institutional weaknesses have prevented Borrowers from effectively implementing incrasingly complex proects. Local commitment and *ownership" have received little attention, while the focus has been on tenical soundness. GLOBAL FACTORS 14. Changes in the world economy in the 1970s and 1980s had an adverse influence on the environment in which projects were implemented. These effct larely resulted from declining tenns of trade, rising intenational interest and inflation rates, and declining capital inflow. The volity of world petroleum prices also had a substantial negative impact. In retospect, it is peras not surprisng tha, within uncertain and destabilized inteonal and country environments, local institutions did not perform as well as anticipated at the time of appraisal and that, in consequence, overall project performance declined. Indeed the extemal envment plus the country grwth rate can-in tem of statistical corelations-explain most of the deterioration in project performance since the early 1980s. 15. The influence of macreconomic variables on the more than 1700 projects completed in 1972-91 was analyzed by the Task Force on Project Quality. It measured the effect of these varbles on the probability those piqects would have (otherise) received "satisfactory ratings.*W The statistical analysis showed their impact varied from one sector and subsector to another. But across sectors, their influence was consistently significant a improvement of 15 percent in the terms-of-trde index increased the probability of a satisfctory proect rating by 1.8 percentage points; an increase of 1 percent in intermaonal interest res was associated with a decline of two pementage points in thle probability of a successful investment; and the negative impact of intenational inflation was close to 3 percentage points. 16. The 1989 Annual Review of Evaluation Results pointed out that 'a group of 25 early intensive adjustment lending countries suffered a loss equal to nine percent of their 1980 Gross Domestic Product (GDP) during 1985-88 compared to 1970-80". And the recent study on Project Qualitym demonstated that declining GDP decreased the probability of satisfcty project implementaion in the same period, by nearly 18 points below the sample average of 74 percent. Such global developments will occur from time to time and remain largely unpredictable. The issue is not to design against their occrce but to underand the W Reprt of Task Foc oan w Reationsips of LAn Processig to Pject Qulity - COD, Marc 27, 1992. w m,i& - 7 - sensitvity of an investment decision to such events and to retin flexibility to adjust to and to contain potential damage. COUNTRY FACTORS 17. Country factors, often conditioned by changes in te global environment, have a strong impact on prQject outcomes. The 1991 WDR presented extensive evidence on the deternants of success in investment projects, based on evaluation results for a large number of completed projects. Examining the rionship between projected economic rates of return (ERRs) for tnese projects on the one hand and trade distortons, exchange rate distortions, interest rate distortions, and the government deficit on the other, it was found that, by every measure, ERRs were highest in undistorted markets, and lowest in distorted markets. On average, projects implemented in undistorted policy climates had an ERR, estimated at completion, that was at least five percent higher than the ERR for those implemented in a distorted climate. Follow-up researchu suggests that economy-wide policies also affect success in the implementat of social sector projects. The findings support the need for Lining strategy, especially in social sectors, even at project level, to the overall framework of policies at country leveL. Even very well designed projects cannot succeed in a poor policy or reguatory environmentW. 18. The Borrower's institutional, managerial, or onal, and technical capacity to implement projects, and its capacity to audit managerial decisions, all have a strong impact on project outcomes. The 1991 WDR confirmed that the productivity of capital is much higer when institutional objectives are achieved. Expected ERRs for appraised public sector projects were, on average, 22 percent. When institutional objectives had been attained dwing project implementation, the reestimated ERRs turned out to exceed 20 percent - that is, close to expectations. This contrasts sharply with public sector projects for which institutional objectives had not been attained; in those cases the reestimated ERRs averaged less than 10 pement. Whie the Task Force's inquiry did not extend to field research on institutional capacities, staff have reported a decline in institutional strength in many borrowing countries. PROJECT FACTORS 19. The most common types of problems reported Qn regular supesion reports) during project execution, concern (1) institutional constrants, (2) the scarcity of counterpart firncing, (3) poor project management, and (4) defective procurement.1y This pattern, and the overall decline in portfolio performance, were confirmed by staff interviews conducted by the Task W2 Danel Iauf atd Yan Wang, 'he Impact of Mi Policies on Prwt Performc in t Socal Sectors: A Pramework of Analyis and Evidence,' Policy Research Woring Paper 939. W Not a new insight but so important it bea repeition. 3P These catizaons ae of necity geal and hide complex relsiops of rwards and vestd inte policy conflicts and political agenda. They vary in prminc and significanc tunQ they underlin tdo imporwtance of *ownerhip and an intbiAe undestanding of th worings of local society. Force on Loan Processing and Project Quality. Statistical analysis based on 1715 completed investment projects2 also highlights the role of project complexity in determining project outcomes. 20. Project complexity cannot always be avoided and complex project designs may be the only appropriate instruments of change in specific sectoral and policy contexts. Nonetheless, project complexity-particularly organizational complexity-inevitably complicates project implementation. Many of the Bank's current practices generate complexity: * The number of cofinanclers-especially when more than one cofinancier participates in a given operation-substantialy increases the probability of poor project performance. Resort to parllel financing, to reserving procurement, and to particular sources of supply, complicates project management. In projects involving two or more cofinanciers the probability of a satisfactory rating falls by almost 8 percent; * The number of project components or disbursement categories impairs project performance, whether considered separately from, or in conjunction with, cofinancing; * The rapid introduction of various programs of special emphasis (the environment, women in development, poverty reduction, etc.) has caused staff to feel compelled to address many or all of them in each projectW. There is no countervailing effort or incentive to make simplicity -in project designs and implementation plans- a specific goal; D Excessive reliance on covenants in legal documents, especially if they obligate different parties, can increase the complexity of compLiance, and complicate monitoring and reporting; * Project complexity can be compounded when implementing agencies do not systematically participate in loan/credit negotiations or are not properly briefed or trained in Bank requirements and prooedures. 21. Given that project complexity is a common cause of implementation difficulties, efforts are often insufficient to ensure the number of project objectives, components and sponsors is realistically calibrated to the Borrower's implementation capacity. Attention to local commitment and ownership and to coalitions that sustain the project/program through implementation is inadequate. Bank leadership, especially in the areas of special emphasis, is o14en indispensable, but if it is exerted through coercion rather than persuasion and, in a' opcit. 2' Two-thirds of the staff inuvie see this as a major cause of increased pject complexity. -9- consequence, results in aspects of a project which lack bonrower support, the outcome is likely to be deficient ownership and poor implementation. The dslk arising from weak borrowe oommitment receive inadequate attention, as do other risks. Inadequate attention is also given to sensitivity analysis, implementation planning (including procurement scheduling) and sustainability. 22. 'Satisfactory' proects (using OED criteria) share certain characteristicsO. The use of more Bank resources (e.g. staff time) on preparation, appraisal and negotiations is not correlated with project success. Faster completion of negotiations and reduced elapsed time between approval and effectiveness -possible proxies for Borower commitment and ownership - are corelated to project success. The most satisftctory projects tend to be those in which there has been most Borrower partcipation during prepaation and, as a result, the greatest likelihood of high Borrower commitment. 23. Borrowers' compliance with legal covenants is another major problem. Although only partal data are available, the evidence of gross non-compliance is overwhelming. A recent COD study showed that only 22% of the financial covenants in loan/credit agreements were in compliance. A recent OED study of water supply projects financed by the Bank in 1967-89 showed similar results: only 25% of financial covenants were in compliance. The preliminary findings of a review of compliance with financial covenants for revenue eaming entities in one region shows only 15% of projects in full compliance. The Task Force found multiple explanaions for these trends. Some were straightforward cases of non-compliance. Others were complex mixtures of poor project design, unrealistc multiplicities of covenants, and changing macroeconomic environments (particularly in countries undergoing adjustment) where loan documents (and covenants) had not been adapted to changed mrcumstances and where compliance was thus all but impossible. 24. The actual average project implementation period is about 7.0 years, compared with an estimate of 4.5 years at appraisal. As a result, disbursements are also perennially below appraisal estimates; on average by about 30%. Delays are most common in investment projects, often caused by poor procurement planning, and in IDA-financed adjustment operations. E' From s dat istcul analysis of 1478 completed projects rated by OED, alcty projCtsb have to folowimg caraeisc (In lation n to 'average projects): * Fower (-21 %) preration SWs * Fewer (-20%) apprisal SWs * Fewer (4%) negotiations SW8 * Faste prpaadon (d tm -13%) * Faster apprisal (dapsed tm -9%) * Faster pmocssing betwe negotiations and Board (elapsed ime -13%) * Quicker offctiveness (elapsed fime -14%) * Quicker closing (elapsed time -21f%) - 10- 25. Problem projects tend to get extra attention (see Box 1). Semi-annual Reports on Projects under Execution show, however, that major project restucturing affects only 129% of all prects in the portfolioW. Such restructuring usually implies the redesign of proect components and/or the reduction of project scope. The reasons vary. They include financial constaints; changes in the institutional environment; cost inrees; poor contractor performance (particularly in the Human Resource sectors); and the need to address new sectoral priorite Occasionally, project scope is expanded to capitalize on fis year results. The number of restructurings doubled in FY91 and in the first three quarters of FY92, compared to an annual averge of 50 in the period FY89-90. This may suggest increasing concern and fleibility on the part of the management to respond to changing Borrower environments. Certainly the need for that sort of flexibility was strongly suggested by Borrowers attending the inional conference in May (see Annex B). ' Reluctance to test r -on the pat of the ank ma2es- may be partly attbleo to thdo cots involved. It is far fiom easy for a Division Chief to embak an what may amount to a conde redosign, if the only resourcsJ available are the nominal 12 stafvweeks per pioect for "supervion'. E ~~~~~~~~~~~~~~~~~~~~~~~~~g g m: l s ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~s tifiF~~~~~N :H E~~~~~~~~~~~~~~~~~~~~~~W` 4 .12- IV. TE BANK'S ROLE IN SUPPORT OF PORTFOLIO PERFORMANCE 26. The Bank's role in supporting project mplemenn is inadequately defined as "supervision"; it, fact, encompasses four distinct roles: (1) supervising the procurement, disbursement and end use of IDA and IERD funds; (2) monitoring compliance with loan\credit contracts; (3) facilitating implementation by helping Borrowers interpret and respond to the Bank's requirements; and (4) providing substantive implementation assistance to Borrowers. 27. The first two functions are mandatory, arising either from the prescriptions of the Articles or from Loan and Credit Agreements. The other two are discretionary and reflect the Bank's mission as a development institution. There is an evident temptation to concentrate more and more staff resources on them-. Yet, the Bank may not have a compative advantage as a source of extended substantive implementation assistance, and it could face a potential conflict of interest that can arise from maldng substantive managerial and/or professional inputs to implementation. The Bank's proper role is that of a provider of finance, counsel and advice to, rather than in substitution for, the Borrower. In the interests of enhancing Borrower commitment, developing Borrower capabilities, and ensuring accountabilities, a clear definition of the Bank's role vis-a-vis that of the Borrower in identifying, preparing, appraising and supporting the implementation of projects it supports is of critical importance. It reconciles its developmental role in these phases of the project cycle with its fiduciary responsibilities to tax- payers and savers. The intensity of the Bank's involvement -not its role- can, of course, vary with the capabilities of the implementing agency and the type of project and will natully vary from one country to another. 28. Bank involvement in portfolio management has changed over time. It's involvement has substantially increased from simply verifying compliance with procurement, accounting and reporting requirements to a far more actve role in attempting to ensure that projects are properly designed, implemented and, if advisable, modified to meet changing circumstances. OD 13.05 describes the spectrum of activities Bank staff is expected to undertake. It goes well beyond the obligations imposed under the Articles and under loan documents. Bank staff is instructed that project execution and resolution of implementation problems "are the joint concern of the Bank and the borrower". Staff is furher directed to Worm the Borrower in writing what correcdve action is to be taken "by whom, by wha date, and what results are expected". This implies a major though perhaps inadvertent shift in the direction of managerial co-accountability. Mandates as set forth in ODs need to clearly circumscribe the role and accountabilities of the Bank without inhibiting the exercise of initiative and judgment but also without casting doubt on the responsibilities of the Borrower. ThMe tesource absoon implications ane almo unlimited. -13- 29. Within this fmework of mandates and accountabilities, the Bank's effectiveness in supporting pqect implementaton depends on four main factors: the quality of operations when they enter the portfolio; the quality of the Bank's efforts to suppor project implementadon; the quality of the Bank's interactions with Borrowers before, during and after project implementation; and the quality of the feedback cycle to staff from Mwagement and Board, and from staff who evaluate projects to those who identify and appraise them. Each of these topics is discussed in turn in the rmainder of this chapter. QUALITY AT ENTRY 30. Quality at entry is critical to achieving project sucoess. Today the Bank finances such a wide variety of projects that it defies easy definition of what quality means. Capital intensive civil works projects may continue to derive part of their quality from detailed engineering and cost estimates. Projects defined by coordinated sets of activities of numerous participants with relatively little capital investment and a higher reliance on operating expenws may derive part of their quality from a robust definition of objectives and appropriagt flexibility in employing means for their attainment. However wel designed and suited to the country context, projects must also enjoy Borower commitment. And in addition, quality at entry entails cef identification of critical risks and the project's sensitivity to them. With these consideratons in mind, the Task Force also examined the Bank's present prctices in the pre-appraisal, appraisal, and negotiations phases of the project cycle. Pre-Apprasal 31. During the project identification phase,W the tasks are to detmine that thee is a constraint that an intervention could rlax, that the broad outlines of the likely costs and benefits suggest that the intervention will be worthwhile, and there is a particular catalytic role that the Bank can play. This part of the project cycle is critically dependent on diagnostic sector work and on active dialogue with the potental borrower on sectoral priorities. Vital task at this stage are to agree on how and in what detail the projet/program should be prepared and to asceran the commitment of the borrowing country. Borrowers report a decline in the kind of sector work needed for the identification of priorities and an increasingly promilnent role of Bank staff in project preparation, one that they fear can prejudice its objectivity at appraisal. 32. Ponnally, project preparation is the responsibility of the borrowing government. The Bank does maintain fclities (Project Prpaaion Facility, Eneerig Loans, and Tehnical Assstance Loans) to provide financial support for the crg out of activities (surveys, stus, Mw Economic and sector wod: of the Bak is a m*or sou of proect identificaon as weH as a sourc of inputs to th Ba'8s appaon of a potental ivestmn acivity. Although tee. conibution a of great importanc, asmt of ther quality exeed the mandate of this inquiy. WSe OD 8.00, est Fiq - 14 - designs) needed in prject preartion. The Bank can also act as executing agency for UNDP or under tu fund aragements under which it assumes responsibility for the management of financial resources in support of project preparation. Furthermore, it subsidizes project preparaion caried out under coopecative programs with U.N. speciazed agencies. These resources are frequendy brought to bear through staff initiadve; their use is supervised by staff, and the product is at least informally subject to staff approval and/or acceptance. It is in this context that borrwer concern about the lead role of Bank staff in project preparation must be understood. There is clearly merit in the exercise of initiative by staff, but there is also need to eance the Borowers' sense of ownership as there is concern about insufficient local involvement and commitment. Appraial 33. The Task Force found that the credibility of the Bank's apraisal process is under psue. Many Bank staff perceive appraisals as marketing devices for securing loan approval (and achieving pasonal recognition). Funding agencies perceive an "approval culture' in which appraisal becomes advocacy. And whereas Borrowers used (and would like again) to see apprisal as a disineested, reliable assessment leading to a "Seal of Good Housekeeping" (a critical, yet constructive, professional assessment of a project or reform proposal), that is not always the case today. 34. The Bank's approach to appraisal was evaluated in light of Chapter M's discussion of the derminants of project success, which focused on global, country, and project factors. (See also Box 2). The major findings are that Bank appraisals are not maldng clear the moeconomic, financial, and institutional assumptions and risks underlying the analysis. Nor are they maing clear the sensitvity of project outcomes to those variables which experience shows are critical for project success. 35. Other appil shortomings noted by the Task Force included: unreiable assessments of institional, managerial and organizational capacities for project implementaion; poor assessments of Borrower capacities in the area of financial management (including auditing capabilities during implementation); the frequent absence of explicit considemion of altenve technical solutions and options; inadequate assessments of rss and their impact on expected benefits; and (not infrequently yet critically) failure to accurately gauge country commitment and local support. 'Me ' ON 'Teak ForW rviewed Xt 181 S_f Apraii Rpo (SAo.. lf:a~~~~~~~~~~~~l ,fikAPP aAWu*Iet aotth ulty @int" 13R4Pim~ *t ~ Xt1eam =l~~~~~~~~~~~~~l reft urn ando costeffectlv'e ese. The ECON: Tek'For ontedot e d J4wwpsIl :S~~~~~~~~~_r id 1~~~#i*d to anrlrrd un E prqjet dticaonthrb techn' ical er an'd Market growth fators. Twenty-fu ofte~aRprvdd of thes SAR9, the ECON Task Forc ocue htsnliiylikaaym * t4~t4 ._~ ~~~~~~~~~~ | :=rt~~~~~~~~~~~~~~~~ i. ......... . BO 2de87b44IU~t~t 14 the X :Wna~the m i.c W-apa th W"g 'Iweidft. . . ... ...... . tiak*rece sed v 'irtually poatetonent .', 3:~~~. , A . 16 - 36. More than half (56 percent) of the respondents to a survey conducted by the Task Force on Project Quality said their Division Chiefs lacked the time and/or sectoral experdse to help them achieve substantive quality at entry and guide them towards realistic objectives.W Staff pointed out that major contributions to project quality are made and incorporated at the pre-FEPS stage when the project timetable is still tentative and pressure to lend is usually still moderate. Only 17 percent of staff interviewed felt hat analytical work done during project preparation was sufficient to ensure the achievement of project quality. Most others (believing timely delivery is the dominant institutional value) thought that better project work would be done if annual lending contracts were to put less pressure on departments, divisions, and task managers to deliver projects on schedule. Task Managers also reported overload with administrative tasks, project timetables, lending targets, and worl: fragmentation which, they believe, can lead to inadequate technical preparation and weak assessments of institutional implementation capacity. Negotiations 37. The Bank's contactual documents for loans and credits include a variety of covenants designed to assure successful implementation of the project and repayment of the loan. Loan covenants are essential elements in the relationship between the Bank and its Borrowers. The last full-scale review of compliance with loan covenants was done a decade ago='l. 38. The Negotiations stage of the project cycle is seen by many Borrowers as a largely coercive exercise designed to "impose" the Bank's philosophy and to validate the findings of its promotional approach to Appraisal (see Annex B). They complain that there is more focus on extraneous covenants than on panning and explaining specific measures to meet project objectives. They also claim that key project implementation parmeters -on which the Borrower will systematically report, and which the Bank can monitor to assess progress- are rarely discussed or covered by loan/credit documents. These assertions may not be generally accepted within the Bank, but the Task Force believes Managers and staff should be aware of them and, when appropriate, modify behavior accordingly. W There was, nonetheless, a mos consensu (93 pert) tbat the post-1987 quality review system is sp.wmi. to the centrlized pre-Reogumbto system. Why? Mainly because of close associations between proecd teams and Deparimena Management Teams, Country Team, and peer review groups at an early stag of projec processng. 2' Many staff also feel the Sector Opeations Divisions are too small (lack of critical mss) to ensure daff contiity from proect identification through appraisal and to provide adequawte technical backup and training opportunities to newly-recruited staff. , 'Operational Policy Review: CompLiance with Loan Covenants', OED, 1982. -17 - QUALITY OF PORTFOLIO PERFORMANCE MANAGEMENT huplenientatlon 39. In pwnciple, the performance of al projects sold be reevaluated throughout the period of implementation. As stated in pam 7, the current methodology for performance rating is deficient. These tigs are recorded on Form 590 and are aggregated and analyzed in the ARIS. Form 590 does not provide for an adequate explanation of the development imPact rating, ratings tend to be subjectiveW, and there is the recurring problem that many prqects receive satisfacry ratings during implementation, yet are judged as unsactory on completion. 40. The use of information emerging during implementation to fine-tune (or restructure) project design as necessary is uneven. More than anything, this seems to reflect the absence of a systematic attempt to reevaluate actual or expected bemfs or probability of attainment of principal project objectives during implementation. Combined with the inattention to sustainability, this is a major failing because it can perpetuate the implementation of project designs that no longer make sense in the face of changed costs and/or benefits. 41. Four main factors were cited by staff to explain what they perceive to be fte main problems impairing the quality of portfolio performance management inadequate resources (time) (76%); deficient skills (71%); distorted incentives (62%); and pressure to lend (48%). The first hee are discussed below. The fourth pervades this report. M1anageent and Staff Incentives 42. A number of current practices -with respect to career development, feedbak to staff, and signals from manager- militat against a sharper focus on portfolio peformance management. Existing incentives act as barriers to desirable changes in behavior. Prctices tend to put a premium -in recruitment, in formulating workpms, and in promotions- on conceptual and planning abilities rather than on prctical managerial and implementation experience. There is a widely held staff perception that exposure to appraisa and lending work (feably for SALs and SECALs) enhances career development prospects whereas portfolio management experience does not. Higher visibility attaches to achievements in loan procesng than in project performance management. Although prctices vary, feedback from Division Chiefs about portfolio management work is not common and the emphads on ratings vis-a-vis qualitative assement in Form 590 does not induce constructive feedback. Indeed, some staff erroneously believe supervision reporting serves to rate diem raer than to facilitate managerial deciions, and some report pressure -from manage- to minimize the number of problem projects through generous raing. The incidence of staff disisfaction with incentives varied among the regions from 44% to 75%. N Rcetly, effort have beeo made, for example, through review by cotry tean, to nmke the iniugs more objective. * 18- 43. Ter is, furthermore, a revaled perence among Bank staff and managers for conceptuization, planing and design. To the extent hi preference p , Bank staff have a natual inclination to favor lending work. The revealeci prefnce is not therefore sed but is, ratier, reitOrced by the incentive strcture. Intrinsic as weU as extrinsic fators thus affect incentives. This makes the task of engineering cultural change in support of portfolio performace _management considerably more difficultu. Resources 44. Seventy-six percent of the staff interviewed considered the resources allocated to portfolo management inadequate. The changing geographic nature of many operations -from locaied to country wide- requires spending more time in the field. Resolving generic country issues that constain implementation of individual projects but must be dealt with on a systemic level, requires time-consuming coordination with other Task Maagers and complex interacdons within the borrowing country. Time is spent (but not necessarily recorded) by consultants recruited under various trt funds, thereby distortng the record. And staff feel that resource constraints oblige them to take too narrow a view of portfolio performance management. The Task Force reognizes that budgetary evidence does not support this widespread percepton and that, in recent years, not all the resources budgeted for portfolio management have been spent.SQ The percpons nonetheless remain and some managers feel that comparisons between requested, planned, and actual time expendiues are misleading, because, after nmagen have lived with coefficients for many years, their budget requests are often constrained by the expectation that requs for arge allocations exceeding budgetary norms will be disallowed. 45. Moreover, even though staff report sevee time constaints on project and portfolio performnce management, budget analysis shows that managers favor lending when alocatng resources and selecdng staff for assignments, pardcularly where there remain portunities to lend. Unless this iherent prority is corrected and m fested in redeployment decisions, additional resources which might be provided for portfolio performance mangement might simply be used to support other work program categories. 46. That said, a nominal staff allocation of 12 saffweeks a year -four of which, on average, ae presendy devoted to supervising procurement- simply does not allow enough time to meet a' Many swff with long servic in fth Bank teognize a generatina diffec h. For more tnee docados -unil the 1970s- may of thw Bank's tecnicl taff d long experee in mnaging businm functions i dewlopig countries. Te curet stff is generally mu ter on that yape of expeience although probably longer on academc quaificaons and analyticad aume. The aprfcew problem is directly rda t. Some saff also maintain that as long as dter Is an absence of checks and balances in th organfritional architecWt and a predomiant Board interest in new lendnxg, efforts to shift the inoentives towards work to promote successfuiWo will not be efective. 8' While the Task Foroe did not caarr out a deailed nalysis of th ue of Ust fund resou, there is a general perception that trest fund resources my in fSct have subsantiay augmented total esous spent on pject implemettion. . 19 - all the expecttions now placed, rightly or wrongly, on portfolio performance management. While determined efforts may not always be made to redeploy resources within existing overall allocations, it needs to be recognized that extra demands placed on staff beyond the tasks of "core' supervision may require additional resources for the purpose. Specific measures are recommended in Chapter V. The Skill Mix 47. Three main issues were identified regarding the skdlls of Bank staff in relation to expectatuons in the area of portfolio performance management. First, seventy-one percent of staff interviewed felt that Task Managers are not only overworked, but are also overwhelmed by responsibilities for which they have little or no pre-Bank experience or in-Bank trainng. Task Managers are asked to take on administrative, managerial, liaison and negotiations responsibilities far removed from their areas of experience and technical proficiency. Second, skill constraints in the Bank's workforce are felt most often in such areas as accounting and auditing, procurement, organization, management, and environmental impact assessment. Third, staff believe existing skills are often inefficiently deployed: while, for example, they perceive that senior staff work on new lending, junior staff, new staff and Young Professionals are frequently given portfolio management responsibilities, and consultants are hired to bridge the sldll gaps and write Project Completion Reports (PCR)AY. To test those views, the Task Force appraised the adequacy of Bank sldlls for portfolio management on the basis of data from the Time Recording System, the ARIS module of the MIS for FY89-92 (through December 31, 1991), and the Human Resources data base (CHRIS)*V. 48. This review of skill mix revealed three major findings: first, contrary to popular belief, staff deployed on supervision were not inexperienced - in terms of grade, time in grade, or time in the Bank. Forty-six percent of supervision work was done by staff in Grades 24 and above, 54% by staff in Grades 23 and below@. Forty-seven percent had been more than seven years in their grdes, 22% three years or less; and 60% had been in the Bank more than five years, 35% more than 10 years. 49. Second, in the last three years, portfolio management accounted, on average, for 422 staffyears (net of overhead). About 1,550 staff and 820 consultants were involved. The role of consultants grew by about 20% in this period, while the number of staff fluctuated slightly. Staff contributed about 80% of the total time spent on portfolio managanent, consultants contributed the remaining 20% (a ratio that does not seem unreasonable, considering the need £' Contrary to caifent inatactons that PCRs should be wrtten by the Task Managers rnsible fo supervising the prjects up to completion. §' Sdkll data are, however, incomplete. They are not fully recorded for consultmts, and for staff, position titles and job descrptions do not adequately reflect ach sdlls. The exaction of data fom personal sisory forms is a complex task and could not be done within the tmeframe allowed the Task Force -' TMe latter category (Grade 23 and below) includes consultants, many of whom have long experience. -20 - for specialist expertise). Work done at Headquarters, including procurement, accounted for about 70% of time spent on portfolio management, field work for only 30%. It is generally accepted -and also party supported by analysis'- that more than a third of totl portfolio management time was spent on procurement. 50. Third, the factual analysis confirmed that technical skils used to support implementation (accounting for about 46% of total resources) were, quantitatively, adequate, considering the low incidence of technical problems encountered in implementation. However, the availability of financial specialists (11% of the total) and management specialists (2% of the total) was disturbingly limited considering the incidence of managerial and financial problems in the portfolio. Furthermore, in light of the conclusions reached by the Task Force on Project Quality' -which found that the impact of macroeconomic and regulatory environments on project performance was neglected in portfolio management work and poorly treated in portfolio management reports- there may also be weakness with respect to the quality and quantity of economic and public administration skills (17% of the total spent on supervision) applied to project and portfolio management work. 31. Caution is needed in interpreting these data. There are gaps (10% of all the time spent on portfolio management could not be classified) and the breakdown of sldlls by disciplines is limited. Also, staff skdlls are not necessarily reflected in position titles, and given positions may make only limited use of staff skills. While the data suggest that a generally satisfactory mix of staff seniority is being brought to bear on implementation support, that view is not unreservedly shared by Borrowers (see Annex B). Nor should this analysis be taen to suggest that technical skills problems may not exist qualitatively and in other parts of the project cycle. The issues of skills nux deserve furtier analytical attention. Field Offces 52. The Task Force conducted two surveys to analyze the actual and potential roles of Resident Missions and their present and prospective mandates. One covered all Resident Missions. The other covered Headquarters staff with previous experience in Resident Missions. The Task Force also canvassed the views of staff and managers on this topic in focus group discussions, reviewed the 1990 report of the Task Force on Resident Missions, and took advantage of the conferences with Borrowers and with other development agencies to survey non-Bank opinion on the role of field offices in project implementation. 53. There was consistency of opinion among Borrowers and other development institutions at the May/June conferences that the Bank should strengthen its Field Office establishment to ensure that the Bank is in more continuous contact with Borrowers, can effectively address its N 'ne time rewoorg sytam does not conn suffidently complete record to allow a more accurate of ime spnt on procemet t Op cit. -21 - coordination responsibilities, and is sufficiently famiiar with the local society to effcively addre social sector issues. The increase in permanent resenion with a view to the devolution of such portfolio maagement functions as prt and disbutsement was considered by Borrowers to be of secondary importne and met with some reservations. The recent trend to establish more Resident Missions and to staff them with more local pfofsion staff is encouraging. Borrowers generally apprecat the information they provide on Bank procedures and their help in coordinating other development agencies at a local level. However, t Task Force found there was no correlation between the extent to which responsibility for portfolio performmce management was delegated to Resident Missions and the overall quality of portfolio performance. 54. Only six large Resident Missions (Bangladesh, India, Indonesia Kenya, Nigeria, and Cote d'Ivoire) have Aid! responsibility -and then only in speic sectors- for performSance management. With regard to what is, for nearly all of them, a supporting role, the main deficiencies highlighted by staff (from Resident Missions and from Headquarters) included: the lack of clearly defined responsibilities for (total or partial) portfolio management; inadequae resources and training for local staff; weak feedback from Headquarts on the portfolio management task arried out by field staff; a tendency to delegate pape pushing' rather than subsantive tasks; and the risk tat Resident Mission staff may -with the best of intentions- go too far in substituting for deficient local institutional capacities. Borres, in un, wer surprisingly hesitant in suWorting the concept of geater Resident Mion involvement in portfolio performance managemens Their main resertions centered on the risk of a decline in their interactions with specialist staff based in Washington; concomitantly greater reliance an *generalists" based in Resident Missions; less access to objective and disinterested advice on critical issues; and a concern that resident staff might get too close to the local decision-making processes and thereby lose objectivity (see Annex B). QUALITY OF INTERACTION WITH BORROWERS 55. The desirable norm for the relationship between Bank and Borrower in the context of portfolio management is one of interactive partnership built on respective mandates and accountabilities. Deviation from that norm must obviously occur as circumstances dictate. It is obvious that a 'one size fits all' approach is unrealistic. Moreover, wtdn a country, the Bank's level of activities will inevitably vary from sector to sector, institution to institution and from one type of project to another. This demands flexibility on the part of the Bank. Effective portfolio performance management must accommodate different institutional capabilities and an army of different management processes.0 The quality of the Bank's interacion with Borrowers and their agents at all levels, needs to be managed within the definition of mandate and accountability set foth in paras. 27-29 above. If With the demise of the traditional loan officers, for examplo, e intimae acqaiance with to tionld and regulatoy envrnment has declined and now redes mo frequty wih the lawyrs -22 - 56. Current operational directives on portfolio performance managementZ' do not adequately address the respective roles of the Borrower and the Bank during project implementaton. Resident Missions are not mentioned except in terms of the responsibility of Headquarte staff to brief them before departng. And practice -rather an policy- sets the standard for business conduct. That sentiment was echoed at the May conference with Borrowers, who, in different ways, expressed their dissatisfaction with current practices with resect to: inadequate attention to problem solving duning portfolio management missions; the inexperience of Bank staff in project management; the excessive duration of Headquarters review and approval processes; and above all, the fact at the Bank gives more priority to lending than to implementation (see Annex B). They also complained that once projects had been approved by the Board, the Task Managers responsible for them often started work on other projects and had litde or no time to tend to the Borrower's needs in the start-up period. In this genealy negative series of assessments, there was, however, clear agreement on one other point: Country Implementaton Reviews were seen as very useful means of improving portfolio management and addressing generic implementation problems. 57. As noted in Box 1, the Bank's reluctance to tace a firm stand with Borrowers is reflected in the prolonged survival of "problem projects'. All too often, Task Managers appear to operate in a "project mmagement' mode -with neither time nor authority to be effecdve- rather than in "implementation assistance" or "trouble shooting modes. This, however, can provide convenient pretexts for Borrowers to abdicate their responsibilities and to hold the Bank accountable. And in a cumulative and circular fashion, that sometimes strengthens the impulse of Bank staff to try to salvage 'their" projects, rather than focus on giving advice and support to Borrowers about what course of action the Borrower should take. 58. Noncompliance remains a serious problem, particularly in relation to finacial and audit covenants. In an era of heightened concern over the importance of prudent govemance, it is unacceptable that instruments of transparency and accountability are neglected. The Bank, in response to non-compliance, can suspend the right of the Borrower to make withdrawals, or can (under certain circumstances) cancel the undisbursed balance of the loan; but these remedies are rarely exercised. Usually, staff and managers respond to non-compliance by ignoring it, waiving it, or occasionally taldng infommal steps to deal with it - such as requesting the Borrower not to submit withdrawal applications or holding up a successor loan. Because of unsatisfctory performance or changed circumstances, sometimes projects are restructured by scaling them down or changing their scope. The high incidence of non-compliance undenmines the Bank's credibility. It also indicates inadequate concern about the instruments of governance and a lack of realism at the time of negotiation. If staff and managers are prepared to tolerate high levels of continuous non-compliance, the question arises as to whether such covenants should be included in the first place. IF OD 13.05, coveting what is treferrd to in this report as iprojec prfom e ane. -23 - 59. More than 150 existing or planned Operational Directives (ODs) seek to condition the official interaction of staff with Borrowers. Some 80% of these deal with aspects of the project cycle, i.e. are of relevance to the subject of project performance management. In their totality they constitute the framework that defines the intended nature of inteation between the Bank and its Borrowers. In their current form, these instructions tend to focus on prescriptive instruction rather than on objectives, roles and responsibilities. As such they encourage compliance rather than initiative and the exercise of judgment. If the recommendations in this report are accepted, major revisions of ODs will become necessary. Such an event should be seen as an opportunity to simplify existing regulations so as to stimulate further the exercise of sound professional judgment. FEDBACK Management and Motivation 60. Most of the staff interviewed at Headquarters (62%), as well as staff from Resident Missions with responsibilities for portfolio management, complained about the general absence of feedback on portfolio perormance management. That is consistent with the lack of management attention to project implementability and risk assessment noted in Box 2. It is also consistent with the fact that most Country Srtegy Papers are silent on lessons learned from portfolio performance; the fact that follow up on implementation problems is slow (see Box 1); and, until recently, with the absence of credible signals from the Board about the importance of implementation. Moreover, staff perceive a lack of attention to implementation issues on the part of senior managers. Although managerial behavior varies, signals from senior management are consistently seen by staff to focus on lending targets rather than results on the ground. A telling example is the almost exclusive preoccupation with lending programs in the performance contracts underlying the budget process. Leaning Lessons from Experiefce 61. Close surveillance of implementation offers two other important feedback opportunities: first, the derivation of lessons of experience with respect to (among other things) concepts, methodology, designs and implementation plans Xt, over time, provide a basis for future changes in policy and practice; second, the deepening of the experience of Bank professionals who are both the beneficiaries of exposure and the conveyors of lessons learned. 62. There are thrOe distinct cycles: * Feedback in a country context, to identify and address thematic and generic issues of project/country portfolio performance; -24- * Feedback of distinctive and consistent experience with types of projects, sctors, and Progrms of Special Emphasis (PSEs) to improve concepts, designs, and policies in a sectoral and functional cross-country context; and * The professional learning cycle that combines raining, exposure and dissemination. A large store of implementation experience remains insufficiently exploited to enrich the leaning process. V. CONCLUSIONS AND PRINCIPAL RECOMMEDATIONS 63. This chapter presents the Task Force's conclusions and principal recommendations. These recommendations comprise a comprehensive program of change designed to enhance the Bank's capability to help Borrowers bring about a sustainable flow of benefits from the operations it finances. Many procedures and directives will need to be changed. Such changes alone would acomplish little. They must be made in tandem with a concerted effort, led by top management and firmly supported by the Board, to bring about changes in the Bank's internal values and incentives - specifically, to focus attention at all levels more fully on the central objective of achieving sustainable developmental impact and on the critical role of portfolio performance nagement. 64. The principal recommendations of the Task Force are: A. Introduce the concept of country portfolio performance management linked to the Bank's core business processes; B. Provide for country portfolio restructuring in adjusting countries including the reallocation of undisbursed balances of loans/credits; C. Improve the quality of projects entering the portfolio; D. Define the Bank's role in, and improve practices of, project performance management; E. Enhance OED's role as an instrument of independent accountability and refocus ex post evaluation on sustainable development impact; and F. Create an internal environment supportive of better portfolio performance management. -25- Annex A describes in more detail implementing measures required for the principal recommendations to become fully effective. Plans for implementation will need to be developed once the recommendations have been accepted. A. Introduce the Concept of Country Portfolio Performance Management Linked to the Bank's Core Business Processes 65. The volume of new IBRD/IDA lending -about $18-20 bilion for 225 operdons a year- compares with a portfolio stock of about $140 billion representing more than 1800 operations. Given these magnitudes, improving the quality of the portolio stock obviously has enomous potential development impact. It is a Iey aspect of Bank/Country dialogue. Country assistance management, the driving idea of the 1987 reorganization, must be broadened to fully encompass country portfolio performance maagement. 66. The country portfolio must become the principal unit of account. The economic, institutional and regulatory environment of a country are critical deterninants of investment success. Because the project-by-project approach to portfolio performance management still prvails in the Bank, the generic obstacles to project success are often not identified or efficiently addressed. Country strategies do not systematicaUly take into account lessons learned from project implementation. In consequence, the design, composition, and size of future lending programs insufficiently reflects the lessons of implementation experience. This recommendation envisages that: * he Bank shouldconductAnnal Country Polfolie Pefonwace Reviews (CPPRs): Annually, the opeating departments should conduct for each country a Country Portfolio Performance Review (CPPR). Broadly, the CPPR would provide the oocasion for assessing a country's overall portfolio performance, to identify, address and resolve generic or systemic issues in the portfolio based on project-specific deficiencies, as an input to all manageral decisions vis a vis a country. * Pohtfole peronnance expedence should be explMy linked to core buslAw processes: The CPPR assessments should be reflected in Country Strategy Papers, linked to annual Business Planning and Country Assistant Management (CAM) and budgeting processes, and be taken into account in the Lending Allocations and related reviews; they would also become a basis for fornulating the Bank's policy and research work programs. The CPPRs should be the occasion for reviewing country portfolio performance indicators of development impact, poverty reduction, environment, and institutional development. Performance indices should be based on individual project ratings, weighted by their dollar value in the country portfolio (see Annex C). Work should be initiated to male performance indices for projects and portfolios operational. Lending levels and composition should be calibrated consistent with country portfolio performance. The CPPRs should be the basis for the annual report of the President to the Board on the ovUerall status of the Bank's portfolio. -26- ARIS shoud be rplaced: The President's Annual Portfolio Performance Report should replace the current ARIS report. Built on the CPPRs, it would provide a country-focussed overview of the state of the Bank's portfolio; it would also have annexes containing sectoral analyses, analyses of the areas of special emphases, and other statistical analyses. RVPs and country directors would answer questions on country portfolios at the Board. B. Provide for Country Portfolio Restructuring in Adjusting Countries Including the Reallocation of Undisbursed Balances of Loans/Credits 67. As indicated in Chapter M, good policies are a key determinant of project success. Nevertheless, the short and medium term impact of adjustment on the performance of the current portfolio tends to be ignored. Overall contaction can curtail the availability of counterpat resources, alter the prices of inputs and outputs and drive project entities into untenable financial positions. Changed circumstances may alter priorities within the portfolio and the feasibility of on-going projects. Few adjustment programs are accompanied by major investment portfolio restructrings. Under conditions of adjustment, the Bank should be prepared to consider the restructuring of enire country portfolios along the following lines: 3 Restrcwring In connecilen with adjustment. In adjusting countries, overall public expenditure (including investment) reviews should be encouraged. At the request of a member country, the Bank should be prepared to restructure the portfolio of investment projects, preferably but not necessarily if the Bank is othenvise satisfied, n connection with Bank adjustment lending. There would be no automatic country entitlement to funds freed by such restructurng or related cancellations. Where appropriate, undisbursed balances could -subject to a procedure of accelerated Board approval- be reallocated to high priority projects remaining in the portfolio. C. Improve the Quality of Projects Entering the Portfolio 68. Quality of projects at entry into the portfolio has emerged as a critical issue for portfolio performance. In the 'upstream' stages of the project cycle -identification and prepation- implementation experience should be brought to bear more fuly on design options, in assessing risks and liWy outcomes and shaping projects accordingly. At appraisal, objective, realistic, experience-based assessments of likely results should supplant optimism and the temptaton to promote. LoanlCredit approval should be regarded as the initial stage of development assistance rather than a culminating event. The first measure of success for the Bank is not commitment of resources but their effective use. Therefore: * In the iden on and preparation stages, the Bank should foster Borrower commitment and benejklary pauliipation: Without Borrower and, where indicated, beneficiary commitment, successful project implementation is not likely. Because such commitment requires full Borrower understanding and 'ownership,' which - 27 - comes from having the preeminent role in identification and preparation work, Bank staff must restrain their tendencies to preempt Borrower responsibilities at the early stages. In appmisr, the Bank should give more -and more systematic- aNention to dsks/sensivity analysis and ft. mplementablilty: The risks evaluated must include risks due to macroeconomiv f, .tors and the risks of poor or substantially delayed implementation caused by limitations in country managerial, institutional, and financial capabilities or uncertainties of beneficiary response. The sensitivities of intended project impact to such risks must also be evaluated, so that critical indicators of likely success and of implementation progress can be identified for subsequent monitoring. To increase the likelihood of successful implementation, project complexity should be kept to a minimum. While PSEs are vital priorities, there should be discretion to include only those directly pertinent to a project. Cofinancing, which increases complexity, should be used only where either additional funds for the project are needed, the risks need to be spread, or the cofinanciers prefer to leave appraisal and implementation support to the Bank. When cofinancing is done, reporting, prcurement and disbursement requirements placed on Borrowers should be harmonized to the extent feasible, and a 'lead manager' should be agreed; and Implementation plans and schedules (including those for procurement) developed by - -or at least with- the Borrower should be carefully appraised for feasibility and realism, and agreed during negotiations. * Loan docwnents (contmcts) should refled pdoles and implementation needs: In negotiation, the executing agency(ies) should be represented, and their full understanding of --and commitment to adhere to-- obligations and plans must be as important an objective as eventual signature of the loan contract. In the loan documentation, critical substantive provisions should be distinguished from administrative ones, as currently prescribed. Substantive covenants should be included only if the Bank is willing to enforce them. Because breaches of policy conditions beyond the control of the executing agency are unlikely to lead the Bank to suspend otherwise satisfactory projects, such conditions should be included only if they are essential to project success. The Legal Department should exercise quality control with respect to the effective use of covenants and should improve the operating staff's understanding of good covenanting practices. To facilitate achieving consistency of covenants across country programs (as well as the review of experience and evaluations of covenant effectiveness), the Legal Department should coordinate creation of a covenant data bank. -28 - Loan documents should include implementation plans, schedules, measures, and progr reporting agements in Implementation Letters, attached to the contract. They should allow appropriate flexibility as to the means and timing of implementation steps within concise and agreed project objectives. D. Define the Bank's Role in and Improve Its Practices of Project Performance Management 69. In all phases of project work, but particularly in portfolio performance management, the Bank needs to better define its role vis a vis the Borrower. The Bank's business needs to be conducted with sensitivity to the appropriate balances between: support and preemption, diligent, decisive monitoring and rigidity; and awareness of necessary detail and willingness to rely on others to verify and certify aspects of compliance. For implementation of this recommendation the following major measures are needed: * Fhe Baak's role in poitflo perfoeance management: Bank staff must carefully distinguish and adhere to their roles in the various aspects of portfolio performance management. The "core" supervision responsibilites of the Bank -end-use supervision, enforcement of procurement and disbursement requirements, and monitoring of compliance with the loan agreement- are mandatory. The Bank's willingness to facilitate project implementation and extend help in obtaining needed major substantive implementation assistance are discretionary activities consistent with its mandate as a development institution. The direct provision by Bank staff of extended technical assistance, however, should normally be avoided, as the Bank may not have a compaatve advantage for the extension of such assistnce, as it may lead to conflict of interest, and as it may dilute Bonower accountability. The temptation to play a managerial role in implementation must be resisted, lest the project come to be seen by the country as "the Bank's." Even in countries with great need for implementation assistance, the Bank should ensure that its role is one of advising - not substituting for- project management. Approval requirements in loan agreements should be avoided where possible because they also create a hierarchical relationship between the Bank and the Borrower which can weaken Borrower accountability. * Progress tracking and complance: The traclkng of project performance should be keyed to the implementation plans and critical performance indicators agreed during negotiation. The Bank's reporting requirements normally should be a subset of those needed by the Borrower for its own management and accountability, and the burden of providing such data should be the Bonower's. As necessary, the Bank should assist Borrowers in defining the information necessary for efficient project management and monitoring and in creating or obtaining and then installing the requisite reporting systems. Repordng requirements should be agreed at negotiations. While the Bank should remain firm in enforcing compliance with requirements such as those related to procurement, audit and policy matters, it should be flexible in -29. agreeing to adapt project designs to changed circumstances or new insights. When a project has been in "problem' status for more hn twelve months, the responile division chief should either recommend restructuring, that the Bank exercise its contractual remedies, or provide to the country director a memorandum stating why it should not do so. If a project has no likely prospect of yielding a net economic benefit to the country, based on a failure to realize critical performance indicators set out in the Loan/Credit agreements, and if, after consultation with the Bonower, agreement about mutually acceptable restucturing cannot be reached, the Bank should be able to suspend the loan. * Pnocuwment: For ICB, the use of standard bid dotunments, with pre-approved adaptations to country situations, should become mandatory. The Guidelines should be reviewed with the needs of social sector procurement in mind, as well as those relatd to privatization and adjustment operations. The Guidelines and revised standard bid documents should require descriptions of quality assurance procedures, incentives/penalties related to timely/tardy completion, the use of independent engineers for major civil works, and procedures for expeditious dispute resolution. To increase consistency in the Bank's interpretations of the Guideines, a central Procurement Review Committee should be created for mandatory review of contracts above $25 million for goods and works and above $10 million for consultants which together account for about 50% of annual awards, and for advice on such other procurement matters as the Regions might refer to it. - Vedon and CeM*adon: In the interests of efficiency and comparative advantage, and to enhance prudent governance (which also should be strengthened dtough longer-term programs of institutional assistance), the Bank should accept greater use of verification and certification by independent third parties. Independent certification should be submitted of the acceptability of local procurement procedures. For all procurement not subject to prior review by Bank staff (including local Bank- financed procurement), es post certification on a sample basis should be made by an independent agency acceptable to the Bank. For SALs and SECALs, the review of customs documents by the Bank should be replaced by certification that the value of goods for which Bank reimbursement is sought is lower than the value of eligible imports during the period, excluding imports funded from other medium and long term sources. Reviews of SOE claims and related disbursement documentation should also be made by independent third parties E. Enhane OED's Role as an Instrument of Independent Accountability and Refocus Ex Post Evaluation on Sustainable Development Iinpact 70. 'The credibility of fte opeons evaluation function depeds on its independence. OED's role should remain one of independent accountability and objective evaluation of policy and practices. Increasingly, it ought to turn its attentions to impact evaluation and sustainability. For most projects, the end of disbursement is just the beginning of operations; i.e. today's 30O Project Completion Reportsu are misnamed and need to be focussed on the tanstiorw to the operational phase and the ssainability of the project. While OED may wish to rely on Implementation Completton Reports (see below) for its audits, it needs to become more concerned with impact evaluation and sustainabiity. Therefore: * OED should advise or comment only on activtes hat will not be subject to fet OED evaluaon. Because OED's credibility as a source of insitution accountability depends on its independence from Bank management, any evolution of OED's mandate or changes in its Terms of Reference should be tsed against the possible dilution of that independence. * Opermonad Depwlments should replace the PCR with an "Implmenttdon Completkon Repot (ICR)": In addition to providing a ropective summary of implementation experience, the ICR should assess the Borrower's plan for the transition to operations and define the indicators to be used to monitor operations and development impact. Its timing in relation to project progress should be agreed at negotiation. It should be provided to OED and furnished to Board members on request OED audits would be based on ICRs. K OED's indepndent rwes should increasingly emphasize susanabl devlopment lmpact: To preserve its independence OED should avoid involvement in manageril problem resolution for individual projects or portfolios. Its evaluative work should continue to meet broad accountability needs and denve generc lessons. Becase hie Bank is accountable for helping bring about long-term sustainable development, it needs to know actual project impact. OED should therefore substantially incse the emphasis it gives to Impact Evaluations. It should also review and evaluate the Annual Report of Portfolio Performance. 71. OED should help to ensure that the prevailing focus on capital flows is matched by an equay intense interest in the benefits that flow from them. F. Create an Internal Environment Supportive of Better Portfolio Performance Management 72. Many of the changes recommended above depend in part on changes in the Bank's intera environment. They will not work properly unless the Bank, including the Board, is pervaded with the necessary values and incentives. Nor will they work without the necessay skills and resources, both in the field and at headquarters. The Bank's culture eeds to become more attuned to the essentality of on-the-ground net benefits as the measure of success rther tan loan approvals, good reports or disbursements. Bringing about this change will require sustained leadership from all levels of management and strong support from the Board in ataching as much importance to lending results as to lending volumes. Specifically: -31- * Recognton and rewards: Portfolio performance management work should receive the same feedback from managers and the same recognition and rewards as other operational work. In caeer development, excelUence in project and/or portfolio performance management should rank equally with excellence in lending work as a criterion for selection to positions at Grade 25 and above. * Accountabii of county directors: Country directors must feel as accountable for managing existing country portfolios as for new lending. They must play an active role in Country Portfolio Performance Reviews and in building portfolio performance consderations into the various business processes. Feedback should be regularly provided to Task Managers and staff responsible for project performance management. * Slalls enhancement: The Bank should urgently recruit more staff experienced in financial and general management, in public administration, and in institutional development. It should provide orientation to new staff -and more advanced training to existing staff- in operations policies, procedures and practices, including those pertnent to portfolio management. Proficiency testing should be introduced for Task Managers. A career steam should be created for procurment, and a review should be conducted to assess the adequacy of current staffing in that area and to deepen the skills review initiated by the Task Force. * Reid offices: The Bank should aim to have a resident field presence in every country with an active and significant portfolio. Where suitably staffed resident missions are in place, headquarters-based portfolio peformance management should rely more on them, and as necessary provide complementary field visits and approvals of. nonroutine procurement and disbursement actions. With regard to implementaton support, resident missions could generally be made responsible for. (a) facilitatng implementation where appropriate, (b) acceeradting approvals for routine procurement actions and end use of loan/credit proceeds and (c) advising with respect to proposed modifications of implementation plans and schedules. In addition, they would help deepen assessments of executing agency capabilities - assessments that cannot as readily be made from Washington, especially with respect to the social sectors. * Infnaormadon management: Infonnation technology should be brought to bear more fully to (i) facilitate Eank/Borrwer and headquarters/field interaction, (ii) enhance the reliability arl' timeliness of progress reporting, and (iii) increase the efficiency of project performance management. * Iudget Savings realized from such efficiency measures as third party verification, standardized procurement documentation, and effective use of advanced information technology should remain available for portfolio performance management Departnental managers should be explicitly empowered to make resource -32 - redeployment decisions to sustain priority attention to implementation. Budget reviews at all levels should emphasize portfolio performance management needs as much as they do new lending. INSJTEUTMONAL lIPLICATIONS 73. Qualty improvement: The recommended changes will increase the development impact of Bank lending by maldng successful implementation more likely. They will accelerate the Bank's leanting process by improving the application of portfolio performance experience to new lending and increasing awareness of actual development impact. And, through the portfolio performance indices, they will strengthen intenal accountability for supporting successful implementation and ultimately enable the Bank to increase awareness of its own areas in need of improvement. The greater responsiveness of lending prograns and lending to portfolio performance and the more rigorous application of risktsensitivity analysis will, however, make the annual volume of lending more volatile. In the short run, they may have a negative impact on the growth of commitments, though they should affect positively the rate of disbursements in the short to medium term as a consequence of accelerated project implementation. 74. Longtenn Budget Impact: Many of the foregoing recommendations -espeially those related to the use of standard bidding documents, independent third-party verification and certification, Borrower reporting and information technology- should ultimately produce savings. Economies should also result from shorter project implementation times as portfolio management improves and problem projects are more promptly dealt with. The improvement in appraisal methodology -especally the application of realistic risk analysis- may, however, increse the cost of lending as a larger number of projects may be dropped prior to entering the portfolio. And there may remain resource redeployment options between lending and portfolio performance needs. It is not clear -for the long term- that there is an overall shortage of resources for portfolio performance management, provided fungibility between lending and portfolio performance management is preserved and restrictons by line managers related to lendir.g progwa performance and supervision budget maxima are eliminated. 75. Shoji-tenn budget ipact: There may, however, be start-up costs before offsetting economies can be realized. In the next two years, restructuring perhaps 100 projects per year which have been problem projects for several years wil be expensive, as will establishing new resident missions. Public investment reviews and the related restructuring of Bank portfolios in the context of adjustment may also bring additional short-term costs. Recommended training may be funded, in the short term, through reallocation though training, in general, is already under-funded. Reallocation, however, will not be adequate to fund OED's recommended emphasis on impact evaluation work. To inaugurate the portfolio improvement effort, therefore, may require a special initial infusion of additional funds. 76. Long-ean Bene,flt: The renewed focus on project results and the introduction of the country portfolio performance management concept, linked to core business processes as well as to the research agenda, should keep the Bank closely attuned to the pcical challenges of -33 - deveIopment faced by its memben. It will help sustain ad enance the BDan's role as a leader In development policy and dewlopment assistance pactices that are firmly grounded in the lessons experience. EFFECTIVE IMPLEMNrMATION: JCEYTO DEVELOPMENT IMPACr Table of Annxes Annex A: Supporig Maes Related to the Pdncpl Recommendations Anne B: Summary Proceedings of Bonowers' Confrence Annex C: Towards a Results-Oeted Evaluation and Ratng Methodology for Bank Supported Operations Annex D: Opertions Evaluation in fte Bank Annex E: List of Worldng Papers Annex F: Terms of REencoe of The Task Force Annex 0: Task Force Membership "K, x. M.,. V": A ... i-w 'K 'Ri- % A.". % W., f: -x F, I% t W-1011 M W' 1-K, W X ".4 Z` R4; 'R K V -AM, -.WXz -Rftjft :g-w, `:W. m . ..... fil.-M KkA - Al W K. ZKf-4-.m -IrM.W. 3 < X:. xf 7. ff:M 4 .M ... KxI Xl -, ?c K.3 W!, . . ......... ,f M f. ft\ X'V., K.--- K .% ..X_X, Xx ;W F V%4x x Z X, 4. I $511 -;--: ........... . . Mw'mx- i x. x .. . ............ .. ..... .... ......... ..... ...... !R" -IF Page I of 22 SUPPORTING MEASURES REMATED TO TM PRINCIPAL RECOMMENDATIONS This annex enlarges, where appropriate, on the recommendations of Chapter V of the report of the Task Force on Portfolio Performance Mnagement. A. Introduce the Concept of Country Portfolio Performance Management Linked to the Bank's Core Business Processes 1. Introduce Annual Country Portfollo Performance Reviews Linked to Country Implementation Reviews Annually, the operating departments should conduct for each country a Country Portfolio Performance Review (CPPR). Broadly, the CPPR would provide the country team and the departnental management team the opportunity to assess a country's overal portfolio perfornance in order to identify, address and resolve genenc or systemic issues in the portfolio based on project-specific deficiencies, as an input to all managerial decision- maldng on the Bank's posture vis-a-vis the country concemed. The CPPR would replace the ARIS process. To complement the internal review, each department would conduct an annual portfolio review with the Borrower. 2. Reflect CPPR In Country Strategy Papers Bank assistance strategies, including nding, should be condidoned by the lessons lead fiom portfolio perfomance. Specifically, the CPPR results should be explicitly reflected in the formulation of country assistance stategies and the design, sizing and composition of economic and sector work and lending programs. Together with the lessons from the completed portfolio, they should become a regular input to the preparation of Country Strategy Papers (CSPs). Since Borrower commitment is a sine qua non of successful implementation, and a good 'fit' between the objectives of Bank and Borrower is essential, CSPs should be explicit as to whether the Bank's plans and objectives match the Borrower's priorties. They should also take account of links between the country and the rest of the world in assessing exogenous risks to portfolio performance and in assessng macro and sectoral linges within the country. Adjustment programs and policy lending should be conducted within a long term framework, and planned as multi-year, multi-tranche processes, with monitorable indicators agreed at the design stage. Particular care should be given to assessing the will and capacity of the Borrower to implement agreed plans. In integrating the Bank's global institutional priorities and the country's own priorities, the Bank must take full account of implementation capabilities and their varations among sectors. Annex A Page 2 of 22 3. Link CPPR to Business Plan and CAM The CPPR should be explicitly linked to the Business Plan and the budgetary processes. (See Figure 1) Based on the RVP's review of the CPPR, Country Departments should provide -in their Business Plans- strategies and actions to resolve portfolio problems. There should be quantified targets for improving key indicators other than ratings, (e.g., restructurings, disbursements, covenant and audit compliance). Progress in implementation should be assessed in the next year's Business Plan and should be monitored by the RVP. Resource requirements should be reflected in the Business Plan/CAM proposal emphasizing fungibility among budget categories, especially between lending and country portfolio performance management. Rigid budgetary ceilings for portfolio performance management work should not be imposed by line managers. Deviations from budgetary norms should take account of country difficulties, program complexities and country portfolio performance. 4. Link CPPR to Lending Allocations and Related Reviews The results of the CPPR should also be taken into account in the Lending Allocations and related reviews. Accordingly, country portfolio performance indicators should be developed and used as an input to country risk analysis by the Risk Management and Financial Policy Department (FRS) and the Intemational Economics Department (IEC). Lending allocations should be influenced by portfolio performance. 5. Introduce Annual Report on Portfolio Performance Review The CPPR should underpin an Annual Report on Portfolio Perfomuance (ARPP) by the President to the Board. The CPPRs would constitute the base for the country-focussed regional chapters of the report which would be prepared by the regions. They would be preceded by an executive chapter containing the President's overview of the state of the portfolio. This chapter would identify country performance trends and issues and would contain a management plan to address them. The ARPP would have three main annexes: a statistical report, akin to a shorter version of the current ARIS statistics; a sector annex, akin to the trend analysis now contained in Annual Sector Reviews; and a report on the Programs of Special Emphasis (PSEs) that would highlight trends in the implementaion of programs related to global development objectives. Details on the implementation of sector policies and PSEs would be provided, as relevant, in the regional chapters. Production of the President's Annual Report on Portfolio Performance would be coordinated by OSP. The President would present it to the Board and the RVPs and Country Directors would answer questions about countries in their regions. Anna A Page 3 of 22 Fgure Z. Linkages in Potfolo Perfomance Management Prcesses I IwR.( rPO f RON I JUxLEATION R EVRICEW _ PRFC COUNTRY SUSINEF5 PLAN 6. Dlscontinue Some Eht Reports At least three current reports should be discontinued: (1) thie OSP Annual Sector R*eviews that are now prped for Management; (2) the Semii-Annual Report oan PrOect under Execution thtis now circulated to thie Board for information; sind (3) the ARIS report which would be replaced by thie ARPP. An annual Review of Adjustmnent Lending would be covered along with investment lending in the ARPP. However, in depth revimw of the effectiveness of adjustnient lending would continue to be thie subject of occasioat special studies. 7. Link ARPP to OSP Work Programs Ihe Annual Report on Ponftolio Perfomance should prolnde a bao *r an nual review of OSP-rnanaged work on specWa topics, impnprmentation issues, and sectorail reviews. The ARIS reporting on the Special. Topics would be dissontinued. Stuies identified in the CPPRs/ARtPP would be produced as free-standing reports prepuWd by OSP units and circulated to the Board for information but would be discussed by the Board only at its request. Detailed studies on sector po}icy isues and/or sector portfolio implementation would be commissioned in response to generic and/or systeminc issues identified in thie ARPP. Page 4 of 22 8. Develop Country Portfolio Performance idices A cou portfollopelformance index -for gro wthkffidency/development inpact and, in due course, a set of nices, covering poverty rduction, enWronment, and Intttona development - should be introducedfor all countries as a basis for discwssng the sta of the country portfolio. The indices would be based on individual project ratings, weighted by their dollar value in the country portfolio. Details of these indices - and their linkages to other phases of the project cycle - are illustrated in Annex C of the Task Force report However evolved, such indices should remain simple, transant and focussed on the centmal objective of measuing the performance of the portfolio in terms of likely achievement of intended objectives. Further work is needed with respect to definitions and processes entailed in applying such indices. The indices would also provide the basis for a dialogue on the country portfolio, focused on year-to-year changes and the reasons for them. That dialogue could, for example, focus on whether the indices relected changes in countrywide factors and whether they were performance-related or externally caused or whether they reflected changed conclusions based on unchanged evidence. It would also be useful to discuss country indices in companson with OED evaluations completed in the year. (See Section E, below). B. Provide for Country Portfolio Restructuring in Ad,usting Countries Including the Reallocation of Undisbursed Balances of Loans/Credits Changing environments often force countries into adjustment processes that require extea support. Since adjustnent may make existing investment projects obsolete, or significantly retard their implemnentation, comnitted but undisbursed balances of externa loans may accumulate. The Bank's portfolio is not immune to these developments. lhe Bank's invme project portfolio in countries i adjustment should, as necessary, be restructured to reflect and support changed objectives. To this end, whenever an adjustment program is designed, and the Bank agrees with the Borrower on the design of implementation instruments (i.e. policy loans), it should reach agreement with the Borrower on how existng investment projects that would become marginal because of the need for adjustment (or are already paralyzed for lack of counterpart funding) would be restructured. Even when the Bank does not lend for adjustment, a country's own adjustment program may make such an agreement advisable. In principle, resources freed by the cancellation of balances of loans and credits from sub-marginal projects in the course of restructuing exercises, should be availablefor reallocation to increase the share of Bank financing anong Bank-supported projecs remaining in the porfolio. Such restructuring would, of course, need Board approval. A process for accelerated approval of such reallocations (e.g. in the context of a SAL or SECAL) should be developed. Wherever the Bank has a pre- eminent role in coordinating extemal assistance, it should ensure that the consolidation process is extended to include support from other external sources. Page 5 of 22 C. Improve the Quality of Projects Entering the Portfolio 1. Ensure country commitment The Bank's role in support of project prepartion should be agreed with the Borrower at an early stage. It should be tailored to the capacity of the country, the relevant insitution(s) and the type of project. The role of project beneficiaries should also be agreed at this stage and a realistic preparation and implementation plan should be developed by the Borrower. Progress in preparation should be reportd in successive Executive Project Summatry (EPS) updates. The Bank's role as adviser and counsellor in the process of preparation should not be allowed to prejudice its appraisal responsibilities. The Initial &ecuive Project Sunmary (1EPS) should assess cowuny commtment to - -and local support for- the proposed operation and the best means of helping strengthen or maintain i1. The IEPS should explain the roles and responsibilties of various stakeholders during project preparation, implementation and operation, describe the sequence, timing and expected output of preparation activities, and confirm the relationship between the proposed project and the country assistance stategy. (For later quality enhacement, Peer Reviewers identified by the Task Manager should be approved by the Division Chief prior to the IEPS review meeting. The amount of time dedicated to, and the scope of, each Peer Review should be agreed dunng the IEPS meetig. Written comments by Peer Reviewers at the Final Executive Project Summary and the SAR stages should be mandatory). 2. Foster Broad-Based Participation In Project Preparation The Bank should foster adequate participation -both by Borrowers and inrendWd beneficiaies- in the idenfication, preparadon and Implementation ofprojects, all of whi I/ At preset there is not clear amet on the most effective metuodology for appraisn Borrower commitment. TIe Task Forc reogies, however, that this topic is frmeting in the Bank and that seveal expemetd applicatios are now being used. Among them, is one described by James Kens in a pat pmpaed for the Task Force, Generating and Sustining Commitment to Bankl Project? - May, 1992 and the logial faework technique which is not new but has been recently refined by EDI. A recent paper by Robert Piccioto ^Paticipatory Development:. Myths and Dilemmas' - WPS 930, July, 1992 is also _.v AmneA Page 6 of 22 are Borrower responsibilities. Where participation is inadequate to generate the requisite Borrower commitment (and, as relevant, the requisite beneficiary response), successful implementation will be unlikely.2 The Bank should help the Borrower obtain whatever preparation or implementation assistance is needed, but should not normally provide it directly. And when the Bank does provide it, it must be careful to play a supportive and advisory, raer than dominant and decision-making, role. At the pre-appraisal stage, the country's commitment to the project should be reassessed and the main appraisal parameten should be identified on the basis of: the draft Final EPS, written comments from Peer Reviewers, and the White Cover Staff Appraisal Report (SARs). Thereafter, document processing should be based on changes in a number of parameters including project complexity, institutional capacity and implementation planning. 3. Introduce more rigorous analysis of project risks/sensitivities Drawing on the analysis of the ECON Report3, the Task Force found that the Bank is not using evaluation and economic analysis as effectively as it might in project identification, appraisal, and during implementation. Macroeconomic and institutional risks to progam/project success are not systematically considered. As a result, designs for which these risks make the likely economic returns or cost effectiveness too low are not systematically rejected. Nor are key performance variables for inclusion in legal documents and for monitoring during implementation identified. And during implementation, many projects are rated usatisfactoryN only to be downgraded to "unsatisfactorf on completion. Project/program quality can be enhanced by adopting more realistic and risk-conscious araisal techniques. Sensitivity to macroeconomic, financial, institutional, and environmental risks can be considered as early as project identification, thereby influencing project selection early on. Similarly, design choices can be better informed in the process of preparation. During implementation, the use of critical indicators can fcilitate the early diagnosis of problems and trigger prompt remedial actions to solve them or, in the extreme, signal the appropriateness of abandonment/cancellation. &lsting practices in the economic evaluation of investmne operations should be modifled to focus on realistic evaluations ofproject impact, based Intkral&a, on kssons of experience, incuing success rates in completed projects at the sectoral, county, and Regional levels. For operations without ERWs (as for those wth them), a clear ZI A star has been made regarding inmng the public and consulting with NOos and ndigenou affected groups (see for instace ODs 4.00, 4.20, 4.30 and 14.70). These instructions, however, deal mainly with env mtal issues, tlement concrs, and the relationship with NO0s. They are less focussed an questions of supportive local commitment and the responsibility of ownersip. 3I Economic Analysis of Projects: Towards an Approach to Evaluation for the 1990's: Draft Final ECQN B&U; June 19, 1992. An .LA Page 7 of 22 idenification of project goals and cost effectiveness analysis should be required, and the macroeconomic, financial, and behavioral assumptions underlying the analysis should be clearly spelled out. Sensitivity analysis should be used to test the impact of variations on key performance variables, including macroeconomic factors, and to establish appropriae indicators for monitoring (and re-evaluation) during implementation. Agreements reached during negotiations should specify actions that must be taken by the Borrower to achieve success. An indicator tracking system (the indicators being identified at appraisal and selected on the basis of sensitivity analysis) should be used as a basis for monitoring proect performance and for producing country portfolio performance ratings. 4. Emphasize Implementability in Desigp and Appraisal Design pmjects in lght of agency capabiltes: Because projects too complex for Borrowers to implement are not likely to succeed, the capabilities of implementing agencies and individual departments should be taken into account in preparation and appraisal, as should limitations caused by interdepartmental relationships, internal procedures and interactions with other government agencies. The advantages of project designs with manageable numbers of components, implementing parties and objectives (including programs of special emphasis) should be borne in mind, especially during the review process when there are many temptations to introduce embellishments. Careful review of implementation plans during appraisal should reveal problems of complexity, while making the estimates of cost/benefit relationships -taking account of time- more realistic. Pan for Implementation: A detled implementation plan rincluding a prcureent plan) with specific actons, responsibilides and timing should be proposed by the Borrower and appraised by Bank staff Critical factors or benchmarks of success and the sensitivity of the project to them should be identified. Key quantitative performance indicators, progress towards which can be measured during implementation and operation, should be agreed. The timing of the ICR should be defined by the Bank in relation to major implementation objectives rather than in relation to the completion of Bank disbursements (see Section E, below). Cofluancing: Over the past ten years (FY83-92), there have been 1100 projects in which other external financing sources have agreed to finance a portion of project costs; 93% of them have involved official sources, i.e. bilateral and multilateral agencies. Those agencies have augmented the resources available for Bank-supported projects although the addtional resources mobilized for the country as a whole have probably been minimal. Each agency has its own rules for participating in cofinancing, some of which are inconsistent with those of the Bank. Many agencies tie their funds to their own national sources; others have political restrictions on the use of their funds. Reporting requirements of donors differ. Parlel cofinancing increases project complexity, and projects with more than one cofinancier have a significantly reduced likelihood of satisfactory implementation. Consultations with the cofinancing agencies suggest there is a preference for avoiding such Page 8 of 22 complexities. Indeed, there is general agreement that cofinancing should not be seen as an objective by itself and should be used only where: (1) the financing requirements of a project exceeds the capacity or willingness of any one funding source; or (2) there is a strong desire to share the risk; or (3) the co-financier does not have the institutiona capacitY to assess and supervise the project, and prefers to leave those responsibilities to the Bank. The balance between mobilizing additional resources for Bank-supported projects and trying to avoid the resulting time consuming complexities, raises sometimes sensitive questions, and among staff confusion results from not having clear guidelines on cofinacing policies and practices or an Operational Drective on the subject The institutional basis for cofinancing initiatives is not readily apparent, and current prctices in managing cofinancing are often cumbersome. Appropriate guidelines on cofancng should be promulgated covering (1) the basis and rationale for cofinancing, Including the problems to be dealt with, (2) practices to be followed in worldng out cofinancmg arrangements, and (3) standardied reporting and cost sharing procedur. The benefit of haWving a "lead managera when cofinancing is done should be emphasized Staff Appraisal Reports: The fornat of supporting documentation (i.e. SARs) is not, strictly, a subject for this Task Force. However, bearing in mind the fact that quality at entry to the portfolio clearly Is a decisive factor in portfolio performance and that the preparation of SARs involves significant costs, the Task Force believes there is scope for simplifying the SAR and maidng it a more practical document. These opportunities should be explored as soon as possible. In principle, the SAR in addition to evaluating the project, should guide implementation and, through the identification of critical performance indicators, also guide the monitoring of progress by Borrowers and the Bank. As quality resides in results, not reports, care should be exercised in not confusing the quality of analyses with the quality of writing. Yet there is the perception that the literary quality of the SAR is in itself a criterion of performance. It is not, and that point would be driven home if managers and Board were to agree that the SAR, as a worldng -i.e., 'staf- document is intended to (i) asses the intrinsic quality of the project, (ii) evaluate the critica risks to which it is exposed, and (Wii) demonstrate its implementability. At the same time, to reinforce the Board's concern with results (i.e. development impact), consideration should be given to revising Schedule C of the President's Memorandum to have it document critical performance indicators and major milestones for implementation. S. Esu Borrower Understanding of Objectives, mplementation Pas, Procedures, and Responsibilities To the extent not accomplished during appraisal, at negotiations -gateway to the portfolio"- care mus be taken to ensure fdl Borrower undertanding ofproject objecdves: the implementation plan, cricalpefonnce Indicators; Bankproceduresforprocurmm, reporting and dIsbursement, and the division of responsibilities among the Borrower, dte An= A Page 9 of 22 Bank and otherfinancing agencies. To this end, the Bank must insist that the executing agency be represented. To formally require Bank approval of Borrowers' actions under a Loan Agreement puts the Bank in a supersory relationship which dilutes the Borrower's accountability and "ownership.' Such requirements are sometimes included in loan agreements without due considertion of their consequences, often because the project was not sufficiently "maWe when negotiated.4 Before ex ante Bank approvals are required, their impact on accountability should be taken into consideraton. 6. Reflect PrioriUes in Loan Documents Conditioning practices In generak the loan docents ("contracts") should, as currently prescribd, dtffemenate critial substandve covenant from adminstrative ones. Substanive covenants should be Included only #f the Bank Is willing to etfbroe tdm Because breaches of policy conditions beyond the control of the executing agency (and not directly related to project success) are wdikely to lead the Bank to suspend disbursements on an otherwise satisfactory project, such conditions should be included only if they are essential to project success. Finanal Covenants: Financial covenants are often complex, frequently unrealic, and usually ignored. Witness a recent OED survey that found only 25% of water supply projects were in compLiance with their financial covenants. Or a sample of proect surveyed by COD which showed only 22% of the sample was in compliance. The Bank is not staffed to properly review the approximately 5500 auditor's opinions and reports it rceives each year from Bonoevs. The Bank's use of financial covens should be evauted by OSP and Mte conclusions reflected in revised insrucions, gidelnes and training. In individual contracts, such covenants should be used more carefuly and with greater attention to project needs and compliance capabilities. Then, compliance should be monitored and enforced as a matter of prudent goverance. Implemenation and operations plans: Loan document should Iclue hmpemention Plans and schedus in meang/id detail but as side ktters, attachments or the lke Suc.h pcices are already prescrbed but not followed rigorously enough. The plans should allow appropriate flexibility as to the means and timing of implementaton steps within overall objectives. At negoations, agreement should also be r ocbed on the obligation of the Borrower to prepare an Operating Plan for the project beyond the implementation period. The quality of the start-up arrangements for opeaons should be I/ Refer in lga docume to Bank appval of fiture plas of action overng nao proec componet is a mom fequt example of such iD-advised practce. Annx Page 10 of 22 reviewed at the time of the ICR. The circumstances or event that will mark the end of the implementation period and trigger the preparation of the ICR should also be clearly defined. 7. Strengthen Role of Legal Department; Create Covenant Database The Legal Department should educate Bank stff about the use and misuse of covenans and exercise quality control with respect to them. To facilitate achieving consistency of covenants across a country program, (as well as the review of experience, and the evaluation of covenant effectiveness) and to permit recording and retrieval of covenants relevant to sectors and areas of special emphasis, the Legal Deparme should coordinate the creation and maintenance of a covenant database (in the fonm of an electronic reference library) complete with evaluative and outcome infonnation. The evaluative and outcome informaion should be derived from project performance management activities, including CPPRs and ICRs. D. Define the Bank's Role in and Improve Its Practice of Project Performance Management In atl phases of project work, but particulary in portfolio performance management, the Bank needs to better defne its role vis a vis the Borrower. Staff need to be sensitive to the appropriate balances between: support and preemption, diligent, decisive monitong and rigidity; and awareness of necessary detail and reliance on others to verify and certify aspects of compliance. 1. Clarify and Adhere to the Banks's Proper Role Bank stff must carefidly distnguish and adhere to their roles in the various aspect ofproject performance management. In performing the "core" supervision responsibilities of the Bank -including end use supervision, enforcement of procurement and disbursement requirements, and monitoring of compliance with the loan agreement- the Bank's role is mandatory. Beyond that, short advisory trouble shooting and facilitation work are appropriae supportng activities of the Bank, as is the Bank's help in obtaining needed major substantive implementation assistance. The direct provision by the Bank of extended substantive asstance, however, should normally be avoided, as the Bank may not have a comparative advantage for such work and as it may dilute Borower accountability. The temptation to play a supervisory role in implementation must be esisted, lest the project come to be seen by the country as "the Bank's." In special cases where such assae must be provided, the Bank should ensure that its role is one of advising -not substitufing for- project management. As noted, approval requirements in loan agreements should be avoided where possible because they create a hierarchical relationship between the Bank and the Borrower which can weaken Borrower accountability. Ahmex A Page 11 of 22 2. Pay Special Attention to Start-up Delays in the start-up of implementation often signal future problems. Accordingly, after negotiations but prior to effectiveness, the Bank should help ensure that there is optimal dissemination and understanding of the agreements reached, commitments given and guidelines to be followed. Sometimes (particularly in projects involving numerous agencies) a 'project launch' workshop will be effective. Managers should ensure staff continuity betveen appraisal and supervision and give priority attendon to the start-up phase, to help ensure quick effectiveness. An expansion of EDI training in project implementation management (ncluding training in Bank procurement and disbursement requirawents) should be considered by ED] management. 3. Develop Performance Monitoring Systems Based on Implementation Plan and Critical Indicators The best-case scenario rarely occurs. Flexibility and timely responses to changes in circumstances during implementation are therefore critcal. Measuring deviations in key indicators and interpreting them is one means of monitoring progress and signalling the need to design remedial programs that clearly spell out actions, timing, and responsibilities. Performance monitoring systems should be based on agreed implementation plans and reports ofprogress keyed to crincal indicators as agreed during negotiadons. The Bank's progress reporting requirements normally should be a subset of those needed by the Borrower for its own management and accountability, and the burden of providing such data should be the Borrower's. As necessary, the Bank should assist Borrowers in defining the information needed for efficient project management and monitoring and in creating or obtaining and then installing the requisite reporting systems. 4. hnprove Progress Tracking, the Form 590 and Filing Practices The monitoring indicators identfied at appraisal should be used to iqform the project performance ratings in Form 590, and the Forn 590 should be revised to accommodate textual information explaning the ratings. The revised Form 590 would be the inputjor country portfolio performance ratings. Supervision Reports should be management instruments that support effective implementation (by the Borrower) and portfolio performance management (by the Bank). The Implementation Plan, as and if revised, should provide a foundation for the supervision aide memoire and report and for monitoring progress and developing Action Plans. Revision of the Form 590 system should enable its narrative part to be electonically accessible, and a word search capacity to support portfolio analyses should be added. Filing practices should incorporate electronic files of project analyses. Task Managers should be accountable for sending project documents systematically to files, because the project database represents the Bank's institutional memory not only for the purpose of portfolio management, but also for audit and Bankwide portfolio analysis. AnecA Page 12 of 22 S. Use "Mid-Term" Reviews Only When Necessary In some regions, 'mid-term reviews' of all projects are routinely undertaken. Other regions use this tool on an ad hoc basis (if and when indicated on grounds of implementaion perfonnance). Interim reviews offer an opporunty to take stock of problems that have not been addressed in 'normal' or Country Implementation Review (CIR) missions and to agree on major conecdve actions. Some Borrowers, however, have expressed the concern that implementing agencies and the Bank may be tempted to postpone decisions in anticipation of a forthcoming 'mid term review' (see Annex B). This type of review should be wed with dsceion. It should not be made mandatoy within the Bank 6. Monitor Cbanges In Borrower Coitment If the cop'mitment of various agencies and beneficiaries to project designs and objectives is ensured during project identification, preparation, and negotiations, dunrng the implementation period suakeholders will learn continuously from the processes they helped create, or the staeholders themselves may change if there are changes in government. As a result, their posture towards objectives and implementation activities may change. 7hese changes should be monitored, and their ipact on prevously agreed objectives, actions and epcted project benefis should be assessedfrom time to time and systematcaib' reported throughout the pklementaon period. 7. Increase Bank's Decisiveness In Portfolo Performance Management While the Bank should remain firm in enforcing compLiance with requirements such as those related to procurement, audit and policy matters, it should be flexible in agreeing to adapt project designs to changed circumstances or new insights. When a project has been In 'problem statusfor more dhan twelve months, the responsible diviion chishoud either recommend restrucuring, thar the Bank exercise its contracua remedies, orprovide to the couny director a memorandum statng w*zy it should not do so. If a project has no likely prospect of yielding a net economic benefit to the countty and if, after consltation with the Borrower, agreement about mutually acceptable resucturing cannot be reached, the Bank should be able to suspend the loan. 8. Make Standard Bidding Docmen Mandatory and Work To Improve Borrower Procurement Practices Procurement problems are typicaly countrywide and require long-term attention through institutional development assistance. More, however, can be done to address them effecdvely in the context of Bank-sponsored projects. Procurement arrangements often do not receive adequate attention during Appraisal and raic procurement scheduling is not routine. Often, Bank procurement reviews are done and advice is given by staff who do AnmexA Page 13 of 22 not have adequate knowledge of relevant guidelines and praces. While the Bank has developed numerous "standard" contracts/bid documents, they are not often used-. For intenational competitive bidding (ICB), the use of standard bidding douents, with pre- approved adaptadons to country siutons, should be made mandatory. Borrowers would save substantial time in prepartion. The Bank would save substnial time (elapsed and applied) in reviewing documents. And more contractors would be likely to bid. Procuement plans and schdules should be prepared durng appraisal and agreed at negotiation. 9. For ICB, Revise the Guidelies and Standard Contracts The Procurement Guidelines which Borrowers are reqwred to follow have been developed mainly with the traditional sectors in mind and do not suit well the needs of social sector ement. Moreover, within the Guidelines, there are important maters not covered by the standard bid documents. Procurement disputes between contractors and owners often drag on inconclusively and result in deays and friction. Independent supesing engineers are not always used and, when used, are not always independent. Many contra do not contain incentives for early completion and penalties for delay. lhe Gudelines should be revid with, inter alia, the needs of social sector procwreent In mind, as well as those related to privatwzoon and adjustment operadons. TIe Guidelines and revised standard bid documents should require bidder and owner descriptions of their quality assurance procedures, incentives/penalties related to timely/tardy completion, the use of independent engineers for major civil works, and procedures for expeditious dispute resolution. 10. Create an Advisory Bank Operations Procurement Review Commitee Despite the heavy expenditure -over a third of portfolio management time, about 150 staffyears per year- on procurement monitoring by the Bank, the quality of such monitoring is inadequate. Regional interpretations of the Guidelines in approving procurement actions are often inconsistent. Cross-regional coordination among procurement specialists is inadequate. Two approaches are possible. One would be based on the existing decentralized arrangements whereby the regions would continue to make all decisions but would be required to obtain advice from a COD representadtve when contracts of more than $25 million for goods and works and more than $10 million for consultants were being considered. The alternaive would be to create a Cenrl Advisory Procurement Revew Conmittee (APRC) to facilitate the consistent applUcatlon of standards and the resolton if issues. Chaired by COD, it would adise Regional managements on all pouts above $25 million for goods and works and above $10 million for services including consultants. (Field staff given approval authority related to standard contac could also {I These views wer reflected i teo confence the Task Po held with intotnational contactos and also in the coferenac with Bonrows (see Am B) both held in May. AnnexA Page 14 of 22 refer issues to APRC at their discretion). Either arrangement would entail the prior review of less than 50 contracts a year, but would cover more than 50% of the annual value of contract awards. On balance, and in the light of representations made on behalf of the intemational contracdng community, the second alternative would at this time serve institutional interests best. It does not dilute Regional accountability, as the APRC role is advisory, but it more effectively enables the Bank to avoid situations where an international contactor is subjected to different interpretations by different Regions or where exceptions are aluthoritatively represented. 11. Introduce Third Party Verification and Certification Under current instructions, Bank staff do not perform audits of SOEs. All claims supported by SOEs must, however, be reviewed by independent auditors in accordance with the legal agreements. Bank staff are expected to make sample checks of documentation during field missions. The Bank is not adequately staffed to perform this function which tends, therefore, to be neglected and to be seen as a distraction from more substantive tsks. When it is performed, it often covers only a minute sample, especilly when documents are in languages not understood by the person checking them. The Bank is also poorly positioned to assess and verify adherence to local procurement procedures. In the interests of efficency and comparative advantage, and to enhance pnudent governance hch also should be strengthened through longer-term programs of institutional assistance), the Bank should make greater use of verification and certfication by independent third paries acceptable to the Bank. Independent certification should be submitted of the acceptability of local procurement procedures. For all procurement not subject to prior review by Bank staff (including local Bank-financed procurement), ex post certification on a sample basis should be made by an independent agency acceptable to the Bank. For SALs and SECALs, present documentary requirements (customs documentation in lieu of invoices) -although less anomalous than they used to be- are nevertheless irrational, given the fungibility of foreign exchange and given that SALs and SECALs are normally subject only to negative lists and are not intended for specific uses. As long as the volume of eligible imports exceeds the amount of disbursement requested (which it almost always does), the effort to collect customs documentation for eligible imports in amounts at least equal to the disbursement requested is a waste of time. It also engenders staff cynicism and makes the Bank appear inefficient and naive. The review of cusoms docwnents by the Bank should be replaced by bonafide certification tAt the value of goods for which Bank reimbursement Is sought Is lower than the value of eligible Imports dwing the period, excluding imports funded,from other medium and long tenn sources. For project loans, by the time of Negotiaons, an auditor acceptable to the Bank should cert the Borrower has in place a satifactory system (ith appropriate accountng and internal controls) to prepare and deliver bonafide Statements of Erpenditures (SOFs) as documentation for withdrawal of loan proceeds. Shortly after the close of the Page 15 of 22 Borrower's fiscal year, the auditor, under Terms of Reference prepared by the Borrower and approved by the Bank, should be required to furnish its er post audit of SOEs to the Bonrower. A copy should be sent by the Borrower to the Bank. E. Enhance OED's Role as an Instrument of Independent Accountability and Refocus Ex Post Evaluation on Sustainable Development hnpact For most projects, the end of disbursement -which trigger PCRs- is just the begiing of operations. Today's "Project Completion Reports" are therefore misnamed. They should be refocussed on the transition to the operational phase and the sustainability of the project. While OED may wish to rely on the refocussed PCRs for its audits, it should also become more concerned with impact evaluation and sustainability. 1. Incuresingly Emphasize Development Impact In OED's Independent Reviews In a recet report to the Joint Audit Committee, the Director General of Operations Evaluation stated that: 'The value of (OED's) contribution lies in: the Independence of the evaluation function, which ensures free selection of topics for analysis, access to all relevant information, candor in reporting, and uncensored judgement; the focus on quality, efficiency, and effectiveness of operational policies and programs; the empiical nate of evaluation work, which requires rigorous analysis of well-documented evidence; the systematic collection and reflection of the views of all partcipants -Borrowers and beneficiaries and cofinanciers- as well as Bank staff and managers; and in the transparency of the entire process [which] offers the opportunity to... relate evaluation in the Bank to the growing demand for accountability and dissemination of the lessons of experience.' The Task Force is in full agreement with this articulation of OED's contribution and of the opportunities for broad and credible accountability it offers. OED's analyses and findings have been extensively consulted by the Task Force which has duly noted the thorough and prompt absorption by regional staff and maagers of the conclusions and recommendations of the recent OED report, "Bank Experience in Project Supervision."4 The Task Force has confirmed many of the findings of that report, endorses many of its recommendations, and ackmowledges its positive impact on its own report. The Task Force strongly agrees with the Director General (DO0) that OED's credibility as the source of institutional accountability, rests in its Independence from the Bank's management. It therefore believes any evolution of OED's mandate or changes in its Terms of Reference must be tested against the possible dilution of that indepedence. 9I SecM92-576, May 4, 1992. Annx Page 16 of 22 This suggests abstention from any advisory or decision making activity that may be subject to future OED evaluation. As a consequence, the Task Force believes that OED participation in Mid-Term Reviews" or similar exercises involving advice about on-going operations, would be counterproductive. Conversely, evaluation by OED, based on generic and/or systemic evidence, the credibility of which does not depend on the point in the project cycle to which it refers but on the generality of its occurrence, should claim the attention of both Board and Management. OED's role should not be one of problem-solving with rspect to individual projects -that is the job of the Regions- but one of pointing to the need to reconsider policies and practices. It may also encompass retrospective evaluation of any part of the project cycle. With4n this definition, there is soope for such diversification as would make OED's work increasingly relevant to continuous efforts to improve Bank policies and procedures. To increase awareness of, and accountability for, sustainable development impa Whin the Bank, OED should particularly emphasize four areas: *OMrt, CED should produce an Annual Assessment ofthe Pedent's ARPP. it would evaluate the methodology used to meaue progress and performance, asess and compare the persistence and significance of generic and systemic issues identified in the ARPP, and (in parallel with OSP) identify the need for metiodological work on instruments for monitoring and managing country portfolios; S Second. in line with the reconmnended transfonnation of the scope and purpose of the PCR, OED should supplement its reliance on PCRs (ICRs) and revised benefit estmates in PPAR audits with a strongerfocus on bIpact evahlaons done at a dtie when project operations have reached a steady state and results can be objectively assessed. The Impact Evaluation Reports should, in future, play a much larger role in OED's work program. OED should reconsider the ratio of ICRs to be audited. OED should, as an extension of its work on imact valuation, pursue methodological and empirical enquiries into insitutional and developmental sustinability; * 7hird, OED should condnue to produ special sdies of a cross-cutting or otherwise distinctive charcter. These occupy a unique place in the Bank's work; and * Fourth, OED should continue is fjorts to asst member counes in enhancng their capacites in ex post evaluation, but should do so In te context of broad- based public sector management projects and programs, managed by the regions. The overall objective is to strengthen Borowing country capacities in all aspects Andx A Page 17 of 22 of project management, including design, appraisal, implementation, and operation - as well as evaluation. 2. Replace the PCR with an "Implention Completion Report" Timed to follow the completion of implementation, today's PCRs often say little about operations or the sustainable flow of net benefits. Although some Bonrowers attach only cursory importance to PCRs, there is consensus among Bank staff and managers that this document -which remains the foundation of the Bank's accountability to its shareholders and is the primary source of trnpart self assessment of Bank performa must be retained. However, current practices goverrnig the production, dissemination and timing of comnpletion reports should be modified. Ihe Project Completion Report should be renamed 'Implmnion Completion Report (ICR). The ICR should contain a retrospective summary of implementation issues and achievements, mainly in a tabular form, with brief explanations of major deviations from agreed objectives. And, looking ahead, it should also: reassess Borrower commitment to opeaional objectives; reappraise the plan for transition to operations that was agreed at negotations (updated by the Borrower in accordance with the loan/credit agreement); define monitonng indicators for the operational penod; analyze the nsks to successful operation; re-evaluate expected project benefits; recalculate (where relevant) the ERR that would form the benchmark for an eventual impact evaluation; and assess the optimal timing of that evaluation. Upon its production, but not more than six months after the loan/credit closing date, the ICR should be sent to OED by regional management. The Board would be informed of its release and it would be made available, on request, to the Executive Directors and their staff. The ICR would provide the basis for OED decisions on: (1) whether a given project should be evaluated by OED and, if so; (2) whether evaluation should be immediate (through an OED project performance audit report [PPAR]) or later (through an OED impact assessment) or both immediately and later, and (3) whether the evaluation should be 'clustered" with those for other projects (in line with a common OED procedure) or done separately. Opertional performance reports prepared by the Borrower and sent to the Bank annually after the ICR would be copied to OED by the Country Department and would serve, with the ICR, as the basis of an eventual impact evaluation if one were done. The ICR's timing in relaton to project progress should be agreed at negotiations. Reporting on operations would be discontinued at the time when full benefits are obtained (or have reached "steady state") or at such other time as is agreed between the Borrower and the Bank. Annex A Page 18 of 22 F. Create an ILnernal Environment Supportive of Better Portfolio Performance Mnagement Many of the process changes recommended above depend in part on changes in the Bank's internal environment. They will not work properly unless the Bank is pervaded with the necessary values and incentives. Nor will they work without the necessary skills and resources, both in the field and Washington, the necessary communications capabilities, and the required budgetary flexibility. 1. Emphasize On-the Ground Net Benefits as the Prime Value, the Meaure of Success While organizaional "values" may sound amorphous, one of the world's leading corporate executives has recently refered to the need for *soft values for a hard decade.t7 Provided they are shared, management by values, which are internal to the staff, is likely to be more compeling than management by objectives, which are often seen as external. The Bank's culture, in short, needs to become more amted to the essentiality of on-the- ground net benefits as the measure of success rather than loan approvals, good reports or disbursements. Bringing about this change will require, in addition to the measures recommended, sustained leadership from all levels of management. Management must consistently emphasize that the intrinsic value of portfolio performance management is at least equivalent to that of lending, and that the Bank is accountable for sustainable development impact. 2. Hold Lne Mangers Accountable for Results In Portfolio Perfornmance Maaement Country directors and division chiefs must be as accountable for managing each country's porfolio performance as for new lending. They must play an active role in Country Portfolio Performance Reviews and in building portfolio performance considerations into the various business processes. They must ensure that adequate resources are deployed (or redeployed) to portfolio performance management and must take prompt action as necessary to resolve portfolio performance issues. 3. ReconIze and Reward Portfolio Performance Management Work Portfolio performance management work should receive the same feedback from managers and the same recogniton and rewards as other operational work. Incentives for Technical Department staff may merit special attention. In career development, excellence 21 Jack Welch, CEO of 0. E.. AnnexA Page 19 of 22 in project and/or portfolio performance management should rank equally with excellence in lending work as a criterion for selection to positions at Grade 25 and above. 4. Enhance the Skilis Required for Portfolio Perfonmance Mangement Recruitment: A serious gap in the portfolio management skill-mix is expertise in fimancial and institutional management issues. The number of financial management professionals engaged in operational work has fallen from 270 in 1980 to about 190 today and the number of those trained to appraise and advise on accounting, financial reporting and auditing has fallen from 84 in 1980 to 42 today. About a third of all suspensions are due to non-compliance with financial and audit covenants, and more than a third of the operating divisions have no financial professionals. Anothe deficiency is that few Bank staff are skilled in organization, management, and public administration. Other fields of expertise in which staff and managers cited weaknesses include environmental economics and engineering, sociology, and cultural anthropology. The Task Force is not in a position to make recommendations on these latter gaps, but nonetheless wishes to bring them to Management attention. The Bank should urgently recnAt more staffexperienced infinanca and general management, in public administration, and in insdtutional development. Regional managements should ensure there is a critical mass in each country deparment of senior staff with financial management experience to frame financial covenants, advise on and oversee financial aalysis, and monitor the selection of external auditors and review of auditor reports. Training: A review of training opportunities for new staff in portfolio performance management, showed that, besides a geneal 'orientationh seminar of two days, trining in portfolio performance management does not exist. Since FY91, regional initiatives have been developed in LAC and Africa to meet this need and a new, two-day, alending operations" seminar on the entire project cycle is being developed by the Training Division. The Task Force believes substantive training in the Bank's responsibilities, policies and procedures deserves a more concerted effort than it now gets - through well planned introductory courses, advanced courses, formal seminars and on-the-job experience. Current policy that obliges staff to rotate at regular intervals should be revisited with a view to allowing greater staff continuity'. Practical and cafe-based training curricula and training material on project perfonnance management should be developed and offered to operational staff, and aure as well as current Task Managers should be required to demonstrate profidency in matters such as contract adninistration, procrement gudelines, lI 'he tditional armts in favor of rotion - that rotation facilitates sharing of experience and prevents saleness - are both impottant objectives. Thene would, however, appear to be other ways of reahing themn knowledge networks, seminars, cross-support on missions with respect to the first; and the impact of fesh ideas (ftom conwltants) with respect to the second. A Task Force analysis revealed that no more than 40% of prqjects' porfolio performace misions had the same staff for four missions (i.e. for a period of up to two years). Annex A Page 20 of 22 disbursemem downentaon and implementatdon planninge. In light of evidence indicating skdll gaps among some managers, a specil effort should be made to prode speca tning opportunities to Division Chiefs who need them. In addition, a career stream should be introduced for procurement, and a review should be conducted of the adequacy of stffing in that area. S. EstabDlsh Resident Misions In/for AD Countrles with Signcant Programs and Give Them Larger (but Circumsribed) Roles In Portfolio Perfornce Management Although the Task Force did not conduct a comprehensive review of the potential and actual role of Resident Missions or of their present and prospective mandates, it did survey staff with Resident Mission experience concerning the role of field offices in supporting project implementation. Also -at the May conference- it explored Borrowers' and development agencies' views on this topic. The Task Force found nothing to suggest there should be a maor decentralization of the Bank's portfolio management activities to field offices. Bofrowers generally value the complementary roles of Headquarters and field office staff in providing support for implementation, but they are concerned that the disinterested expert perspectives of Headquarters staff should not be lost to them and that the Bank's decisions should remain -and be perceived to remain- impartial. Some development institutions and agencies -not the majority- made a case for the Bank to enlarge its field presence, particularly with respect to social sector projects that require more contact with decentralized institutions and beneficiaries. The continuing shift in the portfolio towards 'evolutionary' projects with special needs for institutional support (as in the social sectors, extension, environmental management, and privatization) and the implications of the changes recommended above for the delegation of procurement and disbursement responsibilities to the field, strengthen the argument for selective devolution. The presumpdon shoud shift in favor of having a resident field presence for every country. Where suitably staffed resident missions are in place, headquarters-based portfolio performance management should rely more on them, and as necessary provide complementary field visits and approvals of nonroutine procurement and disbursement actions. With regard to implementation support, resident missions could generally be made responsible for: (a) facilitating implementation where appropriate, (b) accelerating approvals for routine procurement actions and end use of loan/credit proceeds and (c) advising with respect to proposed modifications of implementation plans and schedules. In addition, they could help deepen assessments of executing agency capabilities - assessments that cannot as readily be made from Washington, especilly wifth respect to the social sectrs. 2I Compuable and much mor eblboat teining is offend in large pnvate corpoaions AnneLA Page 21 of 22 6. Use Information Maagement and Techdology to Better Advantage Bank Information Needs: Information flows should be tailored to business decision needs. For Task Managers, the needs are massive. The systems that support them must, therefore, be designed to help them access information they need ather than walow in unusable data. This is particularly important with respect to documentation on policies, procedures, direcdves and innovative prdces. An information flow analysis of portfolio performance management functions and an analysis of the use of information technology in project implementation and monitoring were conducted by the Task Force with the help of experienced Task Managers and line managers. Both exercises suggest efficiency gains in information management are desirable and achievable. The information flow analysis confinns that a massive flow of documentation is associated with portfolio performance management, most of which feeds the Bank's intenal processes and is only loosely related to Bank-Borrower interaction and on-the-ground action. Tbese findings are consistent with those of a recent Organiztional Planning/LAC study of the efficiency and effectiveness of business prcesses in the LAC region'o. The quality of information available for portfolio performance management was severely criticized by Bank users at every level, as were the costs to the Bank of colecting data from Borrowers, the lack of computerized access to portfolio information within the Bank, and deficiencies in the Bank's information filing and retrieval systems. As part of the apprisal process, the Bank should sysmatlcaly review the Borrowers' information needs during inpementation and dine its own reporing rquirements, so thaw, wherever possible, the same ifornwadonflows can serve both Bank and Bonwer. In principle, the Bank's information requirements should be a subset of those required by the Borrower for project management and should not go beyond those which the Borrower -as a matter of sound management- will require for its own purposes. Every effort should be made by Bank staff to limit requests for information from Borrowers to that essential to the Bank's decision makdng, business processes, and evaluation needs. Borrower Information: At the time of appraisal, the Bank should consider how the Borrower could use ifomaton technology to support project impementaion and, as necessary, shouadprovide project management so,ftware, hariware, and training, under the project. A wide variety of effective project management software systems already exists to meet the needs of Borrowers. For example, the LAC-Computerized Project Management System is being used in Mexico, Argentina and Brazil, and ASIA's Microsoft Project Management and Primavera software is assisting various agencies to manage and track project implementation. The constraints to achieving an effective flow of information from the project to the Bank are not technological. Information technology applications offer powerful ways of improving efficiency in all aspects of information management, IQ/ Study of Efficiicyl/Effiv.nss in LAC*. ORO. Febmray, 1992. TMe conclusiouB of this study are licely to be relevant -to some deg at least- in other reiQon too. Annex-A Page 22 of 22 particularly with respect to collecdng data from Bonrowers, processing it, and retrieving and manipulating it. Numerous examples of innovative practice were identified by the Task Force on Information Management in Operations, many task managers and line managers having developed their own ways of overcoming perceived deficiencies of Bankwide systems. Examples include the Lending Operations Data Base in the LAC and Africa Regions, the OED Data Base, and the imaginative use of All-in-One in (among other places) the East Asia & Pacific Region. These examples reflect major progress since the 1987 Reorganization. The progress should be extended to encompass portfolio perfornmce management throughout the Bank. Communications Network: Since the full benefits of computerized databases depend, in part, on the availability of a reliable, inexpensive and versatle long-haul communications network, the infomwion Technology and Facilities Deparment's current plans to complete the establishment of a global telecommunications network should continue to receive prioriy. Ana Attachment Page 1 of 2 LI OF PRINCIPAL RECOMMEDATIONS AD SUPPORTING MEASUR A. Introduce the Coept of Country Portfolio Performance agemet Linked to the Bank's Core Key Business Processes 1. Introduce Annual Country Portfolio Performance Reviews Linked to Country Implementation Reviews 2. Reflect CPPR in Country Strategy Papers 3. Link CPPR to Business Plan and CAM 4. Link CPPR to Lending Allocations and Related Reviews 5. Introduce Annual Report on Portfolio Performance Review 6. Discontinue Some Existing Reports 7. Link ARPP to OSP Work Programs 8. Develop Country Portfolio Performance Indices B. Provide for Country Portfollo Restructuring In Adjusting Countries Icluding the ReaUlocatiou of Undisbursed Balances of Loans/Credits C. Improve the Quality of ProJects Entering the Portfolio 1. Ensure Country Commitment 2. Foster Broad-Based Participation in Project Preparation 3. Introduce More Rigorous Analysis of Project risks/sensitivities 4. Emphasize Implementability in Design and Appraisal 5. Ensure Borrower Understanding of Objectives, Implementation Plans, Procedures, and Responsibilities 6. Reflect Priorities in Loan Documents 7. Strengthen Role of Legal Deprment; Create Covenant Database D. Define the Bank's Role In and Improve Its Practices of Project Perftormance Management 1. Define and Adhere to the Bank's Proper Role 2. Pay Special Attention to Start-up 3. Develop Performance Monitoring Systems Based on Implementation Plan and Critical Indicators 4. Improve Progress Tracldng, the Form 590 and Filing Practies 5. Use "Mid-Term" Reviews Only When Necesary 6. Monitor Changes in Borrower Commitment AhnLA Attachment Page 2 of 2 7. Increase Bank's Decisiveness in Portfolio Performance Management 8. Make Standard Bidding Documents Mandatory and Work To Improve Borrower Procurement Practices 9. For ICB, Revise the Guidelines and Standard Contracts 10. Create an Advisory Bank Opertons Procurement Review Committee 11. Introduce Third Party Verification and Certification E. Enhance OED's Role as an Instrument of Independent Accountability and Refocs a Post Evaluation on Sustainable Development Impact 1. Replace the PCR with an linplementation Completion Report' 2. Increasingly Emphasize Development Impact in OED's Independent Reviews F. Create an Internal Environment Supportive of Better Portfolio Performace Management 1. Emphasize On-the-Ground Net Benefits as the Prime Value, the Measure of Success 2. Hold Line Managers Accountable for Results in Portfolio Performance Management 3. Recognize and Reward Portfolio Performance Management Work 4. Enhance the Sidlls Required for Portfolio Performance Management 5. Establish Resident Missions in/for All Countries with Significant Programs and Give Them Larger (but Circumscribed) Roles in Portfolio Performance Management 6. Use Information Management and Technology to Better Advantage 0, .2N :x:v -tt, Mif t: 1*7 0 -IM X .kf t,-W. A,ttiI M M R.", Ks .;. ff. Nt - tX it, It X1,1 , z _1Z AD--- . . ....... . _KIt: Ix n t.51. ff N:::.. All-A Page I of 18 THE WORLD BANK BORROWERS' WORKSHOP ON PORTFOLIO MANGEMENT fHihlights [On May 28 and 29, 1992, the Portfolio Management Task Force held a "workshop" with fifteen people fom a representative sample of borrowing countries (accounting for morem half of the portfolio) who had had extensive experience dealing with the Bank. The paticipants were selected for their knowledge of the Bank by the Bank's country departments, and their selecdon was approved by the appropriate Executive Directors. They came from countries in all Regions except Europe and Centmal Asia. Brazil and China were not represented; nor were island economies, as the task force wished to have a reladvely representative sample of countries. The participants were assured anonymity and were asked to speak for themselves and unofficially, not as goverment reprsentatives. Participants came both from central ministries and executing agencies. Most had confenred with colleagues before coming and some provided written materil in response to the "Possible Questions for Discussion," developed with the help of the Portfolio Management Steering Committee and circulated in advance (see attachment). The higlights of the four hundred page transcript, below, were prepared by a contract writer, independent of the Bank and the task force. In the task force's opinion, they faithfully reflect the substance of the workshop.] AEnnexB Page2 of 18 THE WORLD BANK BORROWERS' WORKSHOP ON PORTFOLIO MANAGDMENT 1 'As we see it, the Bank's entre atdxide b atuned toards loan ad commitmet rThe Dank has] a built-in bias agaist project implemenai and supervision,' said an ofricial from one of tie World Bank's borrower counties, at a unique two-day workshop on portfolio managmenL 2 in May, de from counies represenng more than half of the Banks active portfoHo conmened in Wagton to discuss why project pefmne is declining - why 20 percent of the projects in the Bank's leding portfolio are problem projects. Promised confidendality, they spoke frankly about every stag of the project cycle fom concepton to evaluaton. 3 Many of the problem projects, the eker contended, "wen not conceived properly' and were overloaded with conditionaty. 'Today he tred is making the project a marlang packae for Board sasftin if the main pupose of project conditionalities is to fafiltate project proge, it's welcome. But If dth main motive behind it is to sell the packge before Bank approval, it is self- 4 Ironicaly, said one borrower, the staff rigidly Insists on as many conditions as possible - some of which lc t Ihssitivity about the poldcal raities in th borrower country - to covne ihe Board that the project will be sucesfuL Yet those very condidons mak it impossible for the project to attain its objectives. Project disbursements are often held up because of failure to sadtsfy nonproject conditions over which the project team has nO control. Indeed, strict adherence to sectal conditions sometimes conflicts with fiscal policy required under stctr adjusment loans. S As one borwer put it, Bank staff 'take a negotiating posidon, not a consulting position' - they know what they wmat from the outset snd aren't open to bearing what the country has to say. Borrowers especially criticized te BDa's ridity about timetables. They often agree that certain reforms dsuld be made, but find he BDank's dmetable for reform unrealistic. Ihe Bank insists on the timetable anyway, and ofken the project cannot stick to it. The B seems more concerned with geting Board apprval, say the borrowrs, than in ariving at a realisic grem the borrower can live up to. 6 For thdir put, borrowers ofken send the wrog people to negodate - poliUcymaers, rather than the 'soldiersw who could make a projet work. The people resposibl for impon -who could say 'this won't work' - are't always psent tO debate a tdmetable or to say which conditions cammot be met. The Bank should firmly insist, borrowers say, that those resonsible for impleme on be representd in negotiations, because some countries 'have the idea that the officers are just going for holidays, so they try to curail the number.' Annex-B Page 3 of 18 7 Bank representadves assured workshop atwendees that negodations are not a holiday. 8 During ngotations, dte Bank overpowes bomrwers -ad the coundry negotiating team often doesn't have the strength to resist. 'We sit there and say, Look, really, if that was the wish of the Bank, so be it.' They cave In just to keep negodations moving. The Bank has far better lawyers and financiers, said one borrw, and the borrowers' lawyers and fiaciers may not be "anywhere mear as competent as your team. If we were able to get as good lawyers and financiers, I thidk a lot of these issues could be solwed before imple aon. 9 Borrowers are also overpowered by "the voluminous documen that the Bank produces.' They agee to condtions and imple tai plans they don't undertan fully, because there is no way a permanent secretary - or even his deputy" - can review all the docu a the Bank produces. They delegate the review process to someone in tbe lower rank, who may be incapable of assessing "the implications of what is being proposed by the Bank" and caMOt ysemadcally ensure that all condidons are met. Reducing the number of problem projects mes maldug we that the making decisions about implementaton fidly undertad the intent and Implicatio of the ageements reached. O 10 Uldmately, they feel psyhologcally pressured to give In "The negotiadton itself is the end of a very long road. By the time it comes for sigping, there is so much prere put on the responsible ministry by the Bank that, you knw, you just have to get it done." The Bank tends to adopt a take-it- or-leave-it stance, the borrower agrees to conditions It has no way of honoring, and they end up with a contract that cot be Implemented. . 11 Balance shud be reestablshed, say the bores, to ensue that egotations are on a move equal foting, that agrements are more realic, and that the Bank assigns more value to borrower ownership of and commitment to the project. 12 The Bank's tendency to focus more on lending than on implmentation was only one reason borrowers gve for problems in delayed or aborted disbursements and impleentation. rrowes also suggested that i Borrowers "own" and be more involved in project conception and preparation, especially. * The Bank be more flexibe about changes in impet plans. * Tbhe rigor of staff apprail report be restored. * Th borrowers' nsdtutdonal capaiity be essd thorouy and encouraged to develop. * Procurement and reportng douation be di . * The Bank and borrowers joindy assign priorities to a country's projects in the context of country priorities. * The role of resident missions be reonsdered. * Supervision be strengftned. 0 * Post-project evaluaton be more closely linkd to new-project design. Tbe discussions of thee sgestions are ummaized briefly below. Anln" B Page 4 of 18 Borrower "Ownership" of Projects 13 Borrowers agreed that to reduce the number of failed projects, borrowers should 'own' projects from the outset, be commitled to them, and recognize that they will have to pay for the consequences of wrong decisions - theirs gr the Bank's. To that end, most borrowers feel that the Bank sould be less Involved in preparation thn it is. 14 'Over the years, the Bank has assumed a more and more active role in conceptualizing projects," said one borroe. "If that trend contidnues, te involvement of the borrower decreases. I would much rather have the Bank's intention at a later stage of a project cycle, so that the implemendng or executing agency feel the project is their own. It should be left to the borrower to decide if they want Bank intervention [earlier]." 1S Instead, prepaation and design are all tDo oten, as another borrower put it, a "unilateral* Bank activity - an inapopriate attempt to "soonfeed" the borrowers with foreign consultants' experdse. The Bank employs or hires consutants to prepare the project papers, detail design, and we are just left with the report - to give our views or to review te consultant's report." 16 Consultants who aren't familiar with a country often impose technical solutions that may be inappropiate for it, reflecidng the BaWc's views and vision, not the country's. When Bank staff get involved early in the project, they bring in their own "preconceived ideas* (for example, waning to ptivadze road maintenanc) that don't always match local priorities. One borrower described what happens as a result: To base somewhat complicated projects on a theory, "let's say; let's support the private secoor...you create a little Frankenstein and then other people come and check their Frankenstein...so sometimes it tames five or six years to implement. 17 Moreover, as one official put it, "the Bank changes its wisdom in the passage of dme. Five years ago you were lending for small agricultual projects, farmers that were sowing foodstuffs; today you don't do that." Borrowers do not always take the Ban's policy du jwg seriously, knowing that a few years later it may change: "We saw the World Bank talidng about import substdtution in the sixties, then export promotion, then social problems, and then the environment." Not to mention "support the private sector." 18 Borrowers value the Bank's expertise, "large capital of information" and breadth and depth of "long experience," but as one borrower pointed out, these Bank policies "are established by dte Board without ful knwledge of the constan of the recipient country." 19 And, "after all the documents have been signed," says one official, "the Bank can change philosophy again, and what was a normal, standard project loses its strength and momentum" Moreover, "soldiers" working on the project at the operational level don't always get word of the latest directives or decisions or changes in Bank thinking, this can delay both the design and implementation of projects. 20 The Bank often simply taes the project away from the local project officer. One borrower cited a projet on poverty ad rurl employment that "broke down becau the implement agency wanted to be gready involved in the prepatidon and desip of the project." Yet the local project officer is likely to become much more committed "when he has asen the project through the first two or three cycles of the development process." Local commitment gives a project the contidity lacking with Bank staff, who chan frequeny. When new Bank people come ort board who have not been involved in the project's evoludon, they are unable to appreciate the nuamces of the project and to help direct mid-course change, if it is needed - or allowed. AnnekSa Page 5 of 18 21 The Intiadve for conceptuling projects too often comes from he Bank, said one baorower, *andI think it should be the other way around." Said anot: "The borrowers should have a felt need for a project, should take the lead on it, should work In project Idendfication," and should seek technical assistanme only if they feel the need for a coslulant There shoud be more bane between the roles of the EBank and dhe borrower." 22 For one thing, when the Bank consstentdy t over preparaton, te borrower no longer fels the need for project prepaty capabilities." Coutie may initaly want tical assiance because they wan to upgrade their standards and put projecs on a "muc higher foog" - but what has happened is 'a process of substitution rather dun supplementation, which may borwers feel needs to be corrected. Oldy by being allowed to "male their own mistkes, wll dte borroer lean. They apreciate having access to the Bank's vast resources but want to develop their own competence -and to be treated as competent One way to Improve the technical capacity of borrower agencies, one borrower emphasized, is to "pren the strictness" with which th Bank examis the country's analyss, without doing the alyses for them. 23 Citng Bank figures, one botrow said that project performance for dte World Bank Group was satisfctry up to '73, somewhat irreguar up to '82, then declning, particularly aftr '88. Is there a correladon between the decline in project performance and th increased role of the Bank" he asked. 'What today is the borrower's commitment to the project? Most of the portfolio is described - at least in my experence-as 'a World Bank project.' If it is a World Bank project, whatbave I got to do with it? There is a consut who has prepared It, a mission which has appraised it, the Board which has sntoned it, and there are pervisio missions which are watching its progress. [IBut unless the bomwe is committed, the project will not be implemented - as it is not being ituplemented." 24 Such ownsip may be all the more imporlant when ere are no literal owns - when the prduct is education, heath, family planing, or evironmental planning, for example. The Increased Need for Fbility 2S "That 50 pcet of the loans have not been disbursed, even though they have been approved, clearly shows ftat we have a problem in design, not In Implementtion, of projects.- Widi fewer hardware pro"ects and more socialsctor projects, borrwers see the need for a sift away from "blueprint" project desi towrd a flexible, more evoludonary" approach. 'The Banks most Impota role should be to provide techical assa so that project are properly developed and implemented," one official explained. "But the Bank shoud be flexible enough that when proble re enoutered in implementation, solutions can be sougt that are flexible and that enable us, without changing the project's objectives, to find solutions to problem as they come up." 26 More fiexible design reqires being specifc about what the project's objectives are, as these would remain fixed - only the means of implemenin them might chane. But flexibility requres an ongoing dalogue. If projects are going to be more flexible, th may need to be a stronger local Bmak presence, more dealiztion, even at the risk of more political interference. The resident mison may need to play a differe role. And there wfll be a need for earier, more rul supervison to Idenify problems as they come up and to search for their Immediate soludon. 27 The Bank is not set up t easily cancel a project that, after Initial discussions, shows that it is not going to be self. A supevision misson dat comes one year later. or the completon mission that comes two years later, canot provide for cancellation of the project What we need is a tri-monthy report that takes a strong position when a project is not going to worc, because "the interest Page 6 of 18 of th borrower is not to borrow at all costs, and dte interest of the lender is not to lend at all cost." Bonowes sm to warnthe DBank staff to be moe decisive In Identifying problem projects. They proposed that the Bank identify key variables and dwelop a set of Indicators with which to monitor progress on the original objectives - so that it is easy to see early on if a project is in trouble and needs to be cancelled or restctured. 28 Sometmes tb projt needs canceling, and tde counties need a sog indication to this effect frm th Bank - because proects create vested ires, and it is unrealistic to expect those with vested ine to ask to cancel themselves. It's very bard for a inist that is doing something to tell he government, 'I want to caned my project,' said one officiaL "It's not an easy decison for anybody to ta." 29 But mems the project dmply neds modifjing. After two years of study, for exanple, Counry X decided that a certain agricultural project would reay enrich the couty. But two years into the five-r project, international prices on that product dropped d it was clear the project woud no longr be selfining. Should hey contnue the project ot stop, and reimburse only the amount that had been disbuwsed? Or could th resources be realocated to a more profitable project, so the country wouldn't lose that fiancing mobIlIzatIo capt Most counps fel the Bank isn't flexible about projts adapng t cnes beyond their capacity to conrl. 30 Mhe borrowers all seemed to agree that the Bank shold be more flexible on conditionality. Said one: "We need to be clear what it is we agree on at te tIme of negotiatons. And if a particular project design caWls for flexiility, then I think we ought to agree on fleility There should be a ftamework wiin which tie flexibility is exerie for a good purpose." Pep the soludon, said one bTower, is to spel out dt projet's bjece ceay, but to alow for altetve apprwas to Implementation - perhps even provie a menu of alternatives. Rtstwing the Rigor of Appraisal 31 Bonowe asked for more of a hand in project concepton, preparation, and execution, but far frowa waing the Bank to relax its s , boroer want the Ban ks held high - during appraisal. They want th Bank to condnue doing what it tradidonaly does best - and in the case of apprasa, to go back to doing it as well as the Bank used to. Mhe less involved in a project's preparadon the Bank is, the less likely it will be to want to promote it - and the more objecive It can be in apprasa. 32 Many borrwers feel the quality of appraisa I decinng. "There's only an inhouse check on the staff appraisal report [SARI. Everything hines on the pei pton of the task manager." The SAR used to be an importnt communication device, like a bible, oudting step by step what was exected on a projecL It provided for conimity even if there were stff chuag. SARs are weaker now, say boowers, one of whom suggested that the Bank 'switcb bck to dhe pre-1987 arangement" (under the Office of Project Services) "where the SAR [was] acltuly going into.detall ... because te quality of the SAR and the quality of project preparato bas a lot to do with the next S-year cycle of imlmettion" 33 Implemention plas ud pro ment planni are usay not well developed at the time of negodations and pay too Utde attention to insitutiom sregh and other Impotn implementato issue, sy die borowers, wbo feel that Banik appra Is no longer a "Good Housekeeping seal of approval." Annex 8 Page 7 of 18 34 In particular, bofrowers want more thorough institutional evaluation upfront. They fel that it is important to identify insttutional weakmess and to provide technical assislance to strengthen the institutions needed for effecive implementtion 3S It is especially important to evaluate insitutions that work with the Bank for the first time. Working with the Bank is difficult because government agencies must comply wihi boti domesic regulations and Bank regulations, which man times are inconsistent with the procedures in place." They also have to receive three or four missons a year, and Bank staff come asldng all sorts of quetions, "about the way they dress," the "way they write tings,' the way they classify and organize ths. It's not very attracdve for any such agency to work under tliese conditions, having to meet the needs of both the Bank and the government* Deveoping Borwers' Institutional Capability 36 After negodadons, nothing happens for a long time. The Bank turns to other appraisals. The borrower officials go home and don't disseminate and explain what happened in negotiations and what he implications are. During this lag period, say borrowers, the Bank should train project officials in project management and in how the Bank operates, what it requires, and why. 37 -Once the project is approved, we feel that the Bank's role is to support the unit so that it can meet the eablisid condidons. That is where the Bank has been lacking and why project mplementation suffers delays.' Tle objectives of the project and the way it should be executed should be very cearly defined from the onset to avoid confusion, said the borrowers. And it should be expWlaind clearly and u dly to the people in the executdg agencies. 'If you give a loan document of the Bank to a lawyer, a very good lawyer, he would take some time to undetand what it's all about. So you bring this to an engineer, [whol probably doesn't speak English, and you tell him, look, this is what you wull have to ffill- it tas a long time before they really get to understand it. They probably lose a lot of dme in trying to understand many things that are irrelevant." 38 Disbursement and procurement procedures and issues should be clearly explained befr project effectiveness. Borrowers should also be filly briefed on all the ways reports have to be prepared to meet the expectations of both te Bank and the executing ageny, and why. 39 'Our agencies are not well-equipped, do not have the necesary training' to implement projects, said one borrower. 'Many of these people do not have university degrees. We should do whatever is necessary to train these people,' beginning with one- or two-week seminars or courses. Borrowers feel that the Bank should provide condnuous traiing in disbursement and procurement procedures, especially in countries with no resident mission. 40 A country with no resident mission also made a plea for better commumcadons about Bank requremn . 'There is a high level of urnover in our counties, many times for political reasons,' he said - so whenever there is a change in administration, someone from the Bank should meet with the new person and inform him of the financial package and timetable. 41 Borrowen emphasdzed that developing more insdtutional capability Is essendal to project success - and this includes making more use of (and improving the capabilides of) local consuitanL. Despite the Bank's theoredcal support of using local consults, somedmes the borrowing country Inisn that they want local coltants A, B, and C, and the World Bank says, 'No, unles you have C and D'" (expensive consultas from London and New York). Annex] Page 8 of 18 42 As a result, "the people don't feel that it Is their project any more. They say, Look, we are the ones bonovwing the money, and we should have a say. We have A, B, C, and D, who are competent, local people who are used to the location and they can do it better. The only people who have not been able to really execute th job are these foreign-based [consultantsl, because when -they look at the money converted to dollars, it becomes a very smal amount." The foreign consultants also don't stick around for the next stag of the project, and when an outside consultant does the initial work, the bonower doesn't develop the same commitment to the proect. 43 The Bank wil often say, of a local consultut, 'We do not know his work; he has done only one or two Jobs." But 'how does somebody get experience?" asked one borrower. "Nobody gets experience from one day. You have to try them, and if the borrowing country says it thinks a particular chap Is okay, they should be given the opportunity." Resident missions are In a good position to assess the work and experience of local consultants, even when that experience is limited. Standarding omta 44 Borrowers felt they could spend more time on implementation if they spent less time and money on documentadon. 'Printing all these documents," complained one borrower, "is debilitating the capital.' Among the efficiency measures strongly supported by the borrowers: * Agree on an implementation plan at the dme of negotiations. -- Use mme third-party accountng, auditng, and cerdficadon services. * Standardize contracs and bidding documents (adapted to country conditons) to speed up procurement. This would also make it easier for contractors from other developing counties to bid. Only the discretionary part of the contract would need review and approval and much of this could be handled by te resident mission instead of having to go back and forth to Washingon. (Borrowers stressed the need, however, to keep the resident mission free of pressure from local interest groups). Whatever the soludon, most borrowers agreed that the Bank's procurement proces is too cumbersome and rigid. * Relax the rigid Bank policy on using internadoal competitive bidding (ICB), which in supporting equal opportnity to suppliers tends to "overprotect suppliers at the expense of the borrower.' ICB was ideal for large hardware projects but is expensive, time-consuming, inefficient, and ill-suited to local needs and social-sector projects 45 Why use ICB, for example, for educational materials that are sure to be produced locally in the end, anyway? Costs for a science kit are reckoaned for cenurized delivery and do not take account of high transport, storage, And delivery costs to get it out to local teachers. And a foreign supplier may come In with the lowest bid on equipment or vehicles, but 'How do we get spare part? Can we service this equipment? Are there local mechanics who can take care of It?" Also, packaging equipment and goods to reach an ICB ceiling does not work when a project has to be implemented in widely scattered areas or uses consultns who work at different rates. 46 "From the moment we open (publish) bids," said one borrower, ICB "wiln take as much as 240 days. We will have 11,000 bids this year. If we add to this the problem of the Bank's rigid timetable, we have a serio ptoblem here. We have asked the Bank for six months to have an adviser at the reddent mission to speed up the bidding process." Most borrowers felt that delegating much of the review process on procurement to the resident mission would reduce procurement delays. The resident Page 9 of 18 mission could also decide when to raise the preference margin (15 percent) for domestic bidders, to develop more local capabilities. * Coordinate communications and harmonize reportng procedures. Co-financing, said one official, is *desirable for big projects, but very difficult: It is basicaUy Uke living with two wives.* Could borower countries produce common reports to all agencies, he asked, so they could spend less time churning out reports and more time on implementation? Could approval by the lead lending agency be sufficient for cofinanciers? Most borrowers echoed the plea for sandadized or at least harmonized formats for progress reports so that they don't need to re-cast the same information into different reports for different co-lenders and their own government agencies. (Mr. Wapenhans suggested that greater use of advanced infrmation technology would facilitate the exchange of data on how a project was progressing, among the owner-government and co-lenders.) 47 "Most problem loans are those which call upon several agencies at the same time," said one official. 'We do not have true centralizadon of operations with the World Bank.. .[we do not feel the Bank gives us) incentive for such coordination.. .it negotiates separately with each actor. We are not always sure of the program that the Bank is imagining or is planning for the country....partners in the projects are not always infonned of what they are going to do, because there is a complex documentation. Some are inormed and some are not.... The Bank is not always sure that everybody was informed." 48 Indeed, the Bank could coordinate its own policies better. One borrower reported that the appraisal team and the procurement division had different views and "we do hope that the communication between one divuion and the other in the Bank could be much better." Jointly Sorting Prorities 49 Many borrowers value the Bank's ability to help them get things in perspective. We need more technical assistance in that area, said one official, because we have difficulty prioritizing: "instead of having an ex-post process, it should be before the fact." SO Some borrowers value the yM investent Mtfio review, sometimes done at the time of structral adjustment. It is important, said one official, that the Bank and the country jointly sort priorities, and design a financi package that is consistent with macroeconomic stability. Their mistry of finance is approached with five or six 200-to-300-million-dollar projects, each of which requires local countepart funds in similar amounts; and fiscal and macroeconomic restricdons make it impossible to come up with those counterpart funds. So they have to limit their portfolio, but they do so in "an ex post fashion, after all the negodations, after the pipeline has been diussed - and that is why we end up postponing projects." St Such a review makes it clear that some projects are too ambitious. 'I would prefer during a given term of office having three $5-million projects that can be carried out in four years, and not to start with a pipeline that we then have to cancel down the road, with all the costs that this implies." 52 "The same process of having to make decisions for the entre group of projects forces you to set priorities," said another borrower. "So you may feel you have to cancel a part of a project or replace one part with another. Supervision is importnt, but it is useful for the county and the Bank to look at the portfolio as a whole, because project-by-project supervision will vary, depending on the persons involved." AmxB Page 10 of 18 53 Bonowvers perceive a scarcity of counterpart funds as a major reason projects fail. Project conditionality requres the country to provide counterpart fAnds, but under structural adjustment sectors are supposed to undergo across-the-board cuts. 54 "Acrosswthe-board cuts, when there is fiscal consolidation or contaction is not the correct procedure,* said on borrower. *One bas to fit t priorites and then channel the funds accordingly. Problems with project implementadon become aggavated when a coutry has moved to adjustment." is is aU the more true when the pipeline is big; then the country must cancel sizable commitments. Thes cancellations would be easier if dae Bank would agree 'as a matter of policy to let these saving be rechaneled int new priorities without taking normal time for approving the new loans and new commitments" - and, "to the extent possible, that it reduce the conterpatn burden or the fiscal constaints on the borrwer. 55 One recurent problem is that sector and project conditionality (which might require earmarking finds for educadon, for example) confict with conditionality under the structural adjustment support for macroeconomic stabilizaton (which forbids earmarking). 56 Borrowers also seem to welcome count imwemation rviws (CIRs) because they allow generic, cross-sectoral isses and problems to surface - problems often related more to agencies than to projects. Such a review is also likely to reveal when country objectives and project/sector objectives are at cross purposes. Ti kind of portfolio review is more of a problem with cofinancing, but it can also reveal comnicts between the differe demands of co-lenders.) 57 ClIRs have not been used in all counis, or with reguarity, but borrowers who had experienced them found them useful; one country that has had such discussions twice a year reported 'wonderfid results." Another borrower said the portfoio review and "the processes that lead to it," not only help the country get a fix on thing, "but there's a lot of compedtion in the system and a lot of pressure on the implemening agencies to perform." CMRs are also heavily attended and given respect by the country's top officials. 58 If CIRs were followed up seriously, one could 'get away with routine supervision missions for that country," said one borrower, with much of the routdne work perfomd by the resident mission. "Two or three main projects could be short-listed for more intense supervision," where the approach should be "problem-solving, not problem-pointng." Rehiking the Role of Rsident Misions 59 Borrowers were divided about whether, and how much, to strengthen resident missions. Pardy this is because their quality and rapport with project oficers vary depending on the country and sector and the skill of the mission staff. 60 The resident mission's role is generally seen as one of facilitating implementaton and giving the Bank a window into local cultures and activities. One official said that the Bank's rigidity is the result of "the people procesdsng these f being "too far removed fiom the field reality," That country's agricultural projects were more efficient because the resident mission was in contact with what was happening. 61 The resident mission can help clarify the Bank's ntract r. And much of the procurement review process could be delegated to the resident msion, if a procurement expert were attached to it - although, as one borvwer suggested, that delegation of power should be well-defined. Page 11 of 18 62 But some borrowers had reservations about giving the resident mision a more significant role, fearing that an ovefed and overbudened mission might lose tuch with (and the backing of) Bank headquarters and senior Bank decIonnaks. And to be effecive, a resident mission must remain detached from special interest goups and polidcal influence. If people thought they were favoring one bidder over another, things could get sdeky. One official compWlained that the resident mission -as well as consultan and Bank officials - tended to want to see only the bigest government offticias, which demoralizes the.working-level people. Jontly Sbrngt g Supervion 63 Borrowers value the Bank as a acilitator, crossing organizal lines within the government, particuarly regarding Bank requiremns. But one of them sees the Bank's technical assiae during supervision as only marginally usefil. Bank evaluadons seem to do better at mea efficiency (costs) than effectiveness (benefits). 'Bank staff is really innovative when it comes to conceptualizaton but is very regulated when it comes to uperi on and implementation. 64 We need more supervison earlier, they said - to identify problems, to find soludons, and possibly to reorient a project Borrowers said that the Bank does't value supervision of implementation as much as it values puSing throu loam. 65 Duing impleaton, the Bank is seen as too rigid in adhering to the legal contract, ndes and timetables - especially on project in the social sectors. Borrowers say it is damaging to a project that is on schedule and doing wel to suspendd becae sector-specific conditionalities (beyond the control of project management) have not been met 66 Cancelling a problem project is not always the right answer. If a project to build an aqueduct does not meet macroeconomic conditions or fulfill repordng requirements, the villagers still need water and jobs. Socially, such a project may have a very high priority; the need Is still there. mhe Bank needs more creative problem-solving on projects such as this where the objective is clear but the enviomen has changed and the approach may need changing accordingly. 67 Reallocating invesm funds from problem projects to healthy projects, which woud benefit from fster dIsents, would ultimately speed up the process of adjustment 68 Some borroers feel they should take a key role in supervising themsels. This would usually require additional staffand training, but they could be attachd to countrpar Bank staff for a year of training and then take over many of the jobs they have learned - or large parts of those jobs. 69 'In the long run, the project belongs to the borrower,' said one official. ZI think we must accept the responsibility tO supervise projects that we have taken money to implement, and we must set up institutions to be able to do this adequately, The Bank's responsibility then stops at ensuring that that insdtution is properly set up and capable of undertldang such supervision." 70 Some borrowers feel tht Bak misions should do more supervision in association with local partners, who could do almost daily pervision - parcularly in areas such as.housing and agriculure where it is important to correct thin as the project is implemeed - so tat every third month a more extensive Bank mission could come. At least, said one official, the counies coud prepare an implementtion report that would ease the work of the evision mi. Page 12 of 18 71 "Where the borrower has a supervising outit which also undertale implementation supervision,3 said one borrower, "there should be a way in which aidemeia from those missions could form an input to the Bank's missions. As of now, the Bank missions do not benefit from the missions undertaken by the borrower." The borrower's supervisory agenc, eould prepare progress reports (with preliminary analyses) before the Bank's supervn mission comes, which would make the supervn mission more efficient and fruitful - and would perhaps reduce their fequency. Now, reported one borrower, there are too many Bank supervision missions coming on too short notice. 72 If we were better trained (by EDI), said one official, the Bank would not have to send missions so often. At the ime of negotiation, a framework for implemtion could be set up spcfying what local consuls would do in terms of monitoring and suervision of compliance with contracts. When a Bank spokesman said hat the number one type of institutional problem was financial and that local auditng services had generally proved to be inadequate, another borrower suggestd at the Bank try joint monitoring efforts between Bank missions and an independent local consultant, whose skils would thereby improve. 73 Most borrowers thought supervision should be done dtough the sectoral agency - perhaps with an apex unit (possibly attached to the ministry of finance or planning) serving as a watchdog, coordinator, and facilitator. It is important, said one borrower, to keep evaluation separate from implementation; those who "do' projects are biased in evaluating them. 74 Bo-owr seemed to agree that the Bank was weakest at supervision. Supervision would be more effecive if the Bank focused more on critical points in implementation. Bank supervision has not adapted to the shift toward more social sector projects, which require a more adaptive, evolutionary approach to project design - more contant feedback, adjustmet, fine-tuning, and problem-solving. Linking Review and Evaluation to Project Design 75 Borrowers found the midtermT riew to be "one extra report" which is less usefil than regular supervision or follow-up reports, because it comes too late for mid-course correction. "If we wait for the mid-term missions, we have too many problems to corect." Relying too much on mid-term reviews would encourage procrastntion on decisions that should be made as problems arise. 76 "Even when you do a mid-term review, the idea is to enure that the project is on course. If it's not on course, what do we do for the rest of the life of the project?" This could be an opportunity to big in new ideas or new components, to ructure. But the Bank is rigid here, say borrowers: the mid-term review is looked at as a checkpoint, not as an opporunit to adapt to changing circumstances or analysis. 77 Similarly, borrowers franky eressd litde regard for project completion reports and m- post evaluaons, which look nice on the shelf but were of little value as "lessons learned" as they are rarely read, except by people wanting to Imow how to write one. "Once the project is complete, everyone forgets about it. Too much time elapses before the reports are produced, and the countries don't have the funding to do the reports themselves, although they would probably learn more by helping prepare them. 78 But what happens, then, to lessons earned frorn prior experience? Here, the Bank's enormous breadth and depth of exerience could be enormously helpfud to borrower. The problem, said more ta one borrower, is ihat evaluatiDn departments are isolated from g departents. "Something is not working," said one official, "because the problems we are encountering Page 13 of 18 in today's projects are the same problems encountered in projects many years ago." 'I would differ on the point that the world changes very fst," one borrower said, *we keep maldng the same mistakes because we did not learn from earlier experence. So I woud argue that...it is usefil to be able to review the past and to buUd the future on the basis of what was attempted In the past." 79 The Bank's institutional memory of lessons learned is especially impot in politcally unstable countries, said one borrower, where 'we have this awfid system of very frequent changes. We keep making the same miskes because we did not learn from earlier experiences." Mechanisms could and should be devised, he said, for shang the lessons learned from earlier projects with the people prepai new projects. AU the more so as the gap between appraisal estmates and ex-post evaluation results is widening - perhaps because, as one official put it, "some of us have become project-iven" and hence overoptumistic. Most project officers think their project is unique and will succeed where others have failed, said one official. 80 Perhaps it is the country's responsibility to link such evaluation to new projects, said one borrower, but it is also the Bank's resporsibility. After all, "the Bank's function is to have good projes, disburse them, and get final results - not to have 20 percent of its loans undisbursed." Appreciation of the Borrowers' Workshop 81 Participas in the worshop seemed genuinely grateful for an opporunity to suggest improvements in the Bank-borrower relationship, and hoped that their comments would lead to change. 82 'We think this workshop is really of much importance," said one borrower, 'because we have serious problems about aid utilizadon. In spite of our best efforts, we have not been able to exceed a particular level of performance in aid udlizadon. It has sometimes come down rather than going up. 83 Said another, "Transmit to Mr. Preston that we want to have this opporunity [for an open exchange], for example, in areas like procurement, like maintenance of value, like currency pooling...areas that affect us on the financial side or in other forms. We think that this kind of discussion also could help the Bank to find a better solution for both parties." Sumuary of Min Points 84 Borrowers feel that many of the problems that arise in proect implmentation could be prevented by bettet concept, preparadon, and appraisal of project proposals. They feel that project design and preparadon too often has the Bank's stamp on it instead of the country's - reflectng policies the Bank currently favors instead of the country's felt needs. 85 The Bank is now seen as wishing to "call the shots," in negotiadons and elsewhere. Many borrowers feel that Bank staff is drven more by prere to lend - to get Board approval on new business - than by a desire for successful project implementaton. 86 During negotiadons, the Bank's lawyers and its voluminous documention overpower borrowers - who cave in to unrealistic demands for condidonality in order to get the negotiadons over with. During negotiadons the Bank is pardcularly rigid about timetables, and tends not to listen to what borrowers say will be impossible. Page 14 of 18 87 Borrowers contend that more projects will succeed if borrowers "own" them from the outet, are committed to them, nd recopize that they will have to pay for the consquences of wrong decisions - theirs gr the Bank's. To that end, most borrowes feel that the Bank should be less involved In preparation than it is. 88 Projects would be soner If the Bank allowed borowers to make their own mistakes - if they backed off to a potdon of providing support and advice (when the country asks for it) as well as a dinerested judgment on the quaflty of concept, desip, and implementtion. The more the Bank gets involved in preparation, the less Hly it is to be capable of objecive assessment duritg appraisal. 89 Borrowers lamented the declining quaity of apprAisal - particularly the failure to Identify Iadequate insttttional capability (especially in financial manament) and recommend inrstutional . They want reduced Bank involvement in prpartion and design, but the same insistence on rigorous analysis that the Bank used to expv. since only in that way will they develop the capability for independent project developmet. 90 During implementation, if circumstn or the economic enviroment change, the Bank is not flexible about allowing the project to change its approach - so long as it sicks to the agreed-upon objective. The "blueprlntu approach to implementaton, and the kind of supervision and procurement that went with that, worled fine when the Bank-portfolio included mauy hardware projects. They are not so appropriate to the social sector projects that occupy an increasing porton of the Bank's portfolio. These require a more "evolutionaryl approach to implementtion, and more regular, locally based moning, supervision, and problem-solving. 91 Many borowers consider it important that the Bank and country sort out priorities joindy. Thus, they welcome county implementation reviews (which revel problems common to many projects, among other thin) and country iesment portfolio reiews (which, during structural adjustment, alow a reallocation of funding when structul adjustment requires fscal tightening, which creates a shomage of the counterpart funds and personnel that are a condidon of many sectoal projects). 92 And borrowers viewed this workshop as a healthy sign of the Bank's desire to help counties get things right. -j Page 1S of 18 Draft; 5/13/92 BQKOZ' S WOBSIQOP Ci I?CT DZ gram Possible CUStis for Discssion *g lcgae MA Zntreductigns Background in brief Purpose: to hear the views of our clients -- not as government representatives, but as people who have been at the "receiving end" -- on how the Bank can help countries improve their project implementation Who's who .Session I: Fraework for Proicot rmlementation What are respective roles aad accouctabilLties, in general;, of Guarantor, Borrower, Executlng Agency and BSak? - rs the Bank's input of requisite quality, timely., adequately focussed? * Are country institutions well equipped and coordinated to handle their responsibilities with respect to Bank-financed projects (e.g. administrative requirements, level of complexity, etc.)? the design framework: Bank assistance strategy, project identification, preparation, appraisal, negotiation: FWhat should be the balance between Bank and country roles in conceptualization and design? * What is the appropriate level of detail in defining project objectives and specifications for varaious types of projects? What should be the balance between a "blueprinting", and an * "evolutionary" approach? Are implementation plans, incLuding funding arrangements, adequately developed? Page 16 of is The documentazy framework: Loan documents, appraisal reports, implementation plans,.etc.: * What is the relative value of the appraisal, implementation and, -contractual documents from Borrower's viewpoint? * Are generaL loan covenants well understood and are the institutionaL and project-related covenants appropriate and welL understood? How committed do Borrowers feet to them? Are the contractuaL remedies appropriate and clearly understood? 'What are the expectatior.s az outset? What is the Borrower's attitude to en'forcement? * How well is the formal negotiation handled? The supervision frazewvrk: What are Borrowe:s' perceptions of Bank effectiveness in providing: * "-t'Substastive" techuical assistance during implementation (i.e. the advisory function)? "racilLtation- -- representing im plementing agency's needs to other ministxies, and convd:sely assistance in complyinq with Bank' administrative requirements (e.g. reporting, procurement) Compliance review and enforcement? Session 2: Conduct of Bank Sumerisuon Work Specific Aspects of Bank Supervision Work: How do Borrowers see process during start-up phase (i.e. between loan approval and first disbursement)?. What are their preferred approaches? * Sow do Borrowers see the Bank's procurement approval process? What is the utilLty of Bank procuremeat requirements/guLdance; and how do they perceLve the Bank's administratLve efficLency? * - ShouLd. we have standard bidding documents, subject to negotLatLons? Page 17 Of 18 * Are specifications, standards and supervision sufficient to ensure quality execution under contracts? N How do Borrowers see progress reporting? Are reports required by the Bank useful to them? .. Row do Borrowers see aiviting of project activities? Should the Bank review audit capabilities/- arrangements at the country level as well as in advance of loan/credit negotidtion? - Should external audit of project account./- entities be a regular feature o' the oversight strgctu:e set in place by project sponsors? * Now do Borrowers see disbursement arrangements and documentation requirements? Should the Borrower use independent auditors approved by the Bank (for Bank-assisted projects) to check disbursement documentation e jost. and recommend refunds or other suitabLe remedies? * Should there be a greater country role in compliance review and progress monitoring -- e.g. an agency in a central ministry to oversee implementation and identify actions needed and taken or to be taken by executing or other agencies? * -Should EOt play a role in helping to impart needed implementation skils? - rHow do Borrowers/Guarantors view Project adaptation -- changes, restructuring, cancellation? Do perspectives differ among central ministries, sectoral ones, and executing agencies? Page 18 of 18 loj-ec-SpeCific SupevisLon a General: What is your view of tba Intensity of Be.nk Involvement during supervision? * Resident Missions -- roles and comparative advanttaes/ disadvantages for supervision. (e.g. * tLming, expertise, type of help? Use of local staff? * Bank missions from Washington -- appropriately * . time d, staffed (sk lLs, continuity, demeanor), and conducted? Efficient? Are their requests reasonable? * What are Borrower's views on Bank's use of consultants in supe:vision tasks? = Are "amdtarm" reviews useful to Borrowers? * From Borrower's perspective, is donor c6o*rdination in qupervision (including reporting and audit requLrements) effective when cofinancing exists? * What is the utility of Aides Memoires as an instrument of implementation assistance by the Bank? What other instruments would be helpful? Session 3: Learninr Lessonl Iurina Iemeztation: ftesponses to_an*ric or Svstmic Prbl=s * What are the means for feedback during implementation? EHow can broad areas in need of attention be identified and addressed? Are "country implementation reviews" -- i.e. across the portfolio of Bank-assisted projects - - usefuf to Borrowers? MAre "thematic" reviews -- e.g. of audits or other subjects across projects -- useful -to Borrowers? Session 4: Aft*r ZmgleMentation: Learmna Lessons of KxDerience: Accountability * Itow should success be measured? * CEow do countries become aware of the lessons of experience in their own and other countries? * Are project completion reports and project performance audits useful to the country? VouLtd/should countries do them if Bank did not? * * Should evaluations of project impact (e.g. after a few years' operation) be more often done by countries, the Bank? Annex C Toad a Reut-Oine Eautn- * . an Ratin Methodology fo Bn-uppt * * *-ti * < ,.; :.,S':5e i P.o' tfolio t Ma agemen Tas .Force Anex aC Page 1 of 20 Wapenhans Task Force: Towards a Resuts-oriented Evaluation and Rating Methodology for Bank-Supported Operations L Introduction £. ThIs pNper dicusses strategic aspects of the Bak's evaluation methodology. It fos wrowly on the questio of how to us evaluation during appraisal and supervio as a tool for enhancing the quaity of Bank-supported ations. Is pespective o medtodology emphasizes uncetty and poble msmatche betn Bank manae' incdves and Bank developme impact objectives. This leads to th guiding prinples. First, since d actal conditions of opation (and implemeion) wil be known only after all or part of the investments (or- policies) are in place, faing I th likely 00oditiOnS of operation is an essential feature of evaluaion exrcise, and for deign. Second, during supervision, informadon that helps to refine our view of the most likely outcome should be sought and used as a bais for rating projects and for demining whether adaptions to program/project desig are warranted. Thid, acing changes in evaluadons between appraisal and completion, and using the changes as a tool of portfolio performance management, may help to better align managers' concers with Bank objecdves, thereby influencing manage' acdons at the upstream stages of the project cycle. 2. The paper develops these themes in two core sections. Section! dIces the main fndins of the ECON Report, which recommends makdng the Bank's appraisal of investment operations more realistic in projecting outcomes, more results-oriented, and more un iconscious. To this end, ECON stresses the impormce of explicidy ienifying the macroeconomic, Instutional, behavioral, and fincial aumptions underlyig the analysis and testing the sensitivity of the projectd outcome to changes in assumed parameter values. Secdon M proposes the developmen of a country portfolio index, which would be used for monitoring changes in the quality of the portfolio during implementation. In turn, the country index would be an aggregation of individual project ratings, based on intermediate indicators established during appraisa. The idea is to make the project supervision rangs more reliable, so that they can play a stronger role in signalling the need for acton at the project level even as they' provide a more rdiable basis for aggregation into m ues of couny portfolio peormance. 3. These two secdons of dte paper are Interlockdig. ndicator tracking aims to compress and strengthen the feedback cycle betwn project desig and appraisa on the one hand, and outcomes on the other. With a shoter cycle, managers should be more concned with results. Te objectve is to focus project evaluation, design, and selection on implementaton and other factors that are decisive for determining opeational outcomes. The demand for realistic evaluadons, which pinpoint key risks, would rise as a result. To meet this demand, appraisal methodology needs to be sharpened-to stress sensitivity analysis and the frequently central role of macroeconomic and insutional factors in project success or failure-and to provide the analytic basis for the selection of the indicators. 4. Seveal addidonal points bear mentioning at the outset First, although ECON deals only with invem operations, the indicator approach is proposed for tracking the value of the endre portfolio-that i, including adjutmt opemion. May of ECON's reco edonssuch a spelling out the underlying assumptions and the relevant lessons of experience-apply equally to adjuste / s. Ewwd Axaty* #oP Tbwadas Appmm* t EwiedfoJr Us Mr, dift &A pt, Jun 10, 1992 ADmex-C Page 2 of 20 opeatios, although the specfic opaionalization of these recom dations will ncessarly differ In a adutme Contet Bu clearly, m ng the uccss rteia for evaluadon Is necsary for any operion, and from ther the identfication of itmediate indicators for monioing during msplementation is concuay a small step. S. SeCond, th p p's main focus s on am operation's not prese value, or the conceta analogue in opeaton for which benefit are not monetized. Accordingly, the indicator taking system applies to that coept of project scces. For povety reduction objectives (especialy for projects included in the Progam of Tagted Intervntis,2 a sepa set of ndicat and portfolio id could be devloped. This is di_ussed in Section IV of th paper. Secto IVl also disusses possible uexnsions to te envioment and nsdtidonal dteopmen 6. Third, th focus of this paper is on Bank acdons and processes. This is not to preclude the nvolvment of bowers. Quite the contary. Building loca capaci for evaluating investments-both in the conte of public expndir reviews and seor Invesme loans-is an ultmate goal of the ECON exercise. But before proedin to that stago of th exercise, we need fis a mehodology appropaxt to the tmes. Once broad coensu is reached within the Bank, we can proceed with dissemination. IL ECON indbg and -_dado 7. The OED database records the results of completed Banksupported opeatons a have been evaluated by OED. Seventy-five percent of these operations hae ben rated statory. The ote 25 parcet have faied. (Box I summaie the Regional and sectora bekdown.) Can we do better? To what extent can betr economic analysis enable us to inreae the success rate? With these quesdons In mind, the ECON Task Force3 has been reviewing the Bank's medhodology and practice with respect to the evuation of projects. The principal finding is that the Bank is not effectively using economic aaysis as a tool of project design, appraisal, and spervion. The main findngs and ons follow. A. Iluding 8. The ECON Report atswih a reconsdeio of the Bank's economic appraisal memhodogy, which underlies OMS 2.21, EcnmIk Analysits of Pfecs. D otinctiv features of this methodology, which was developed during the 1970s, include a focus on social pricing and income distributional weights; the use of maccountng rates of intereW to bridge diverg between consumption and _instment interest rates; andd - I shadow prces for individual labor and product makes. The paper concludes th these fe wero nver fMlly aoed witn the Bank, and-contrary to V Ss. ua &Ngtoat ReAM Po" ed OD 4.15, P ,_ D Rudo4 pm 32 y A mlwoa ga1up chaird by 05. Tb Tauk cm inm Me % of ODD'. 198Ad und w 0f Eu61a Ankr, it Uw wand= of Mr. RwauL Annex-C Page 3 of 20 Bex 1: *istri*ad Succes Rat by Sector and Region Su-cess tateS of CQxpleted Operations 00Z~~~~ OHS 4,6.s ' ;~ ,6 ' 5 71 2 571 f X S' t-< '- < f t; . i :: : . .91 8 76, *Non; S6 ; r 7- *S 75i .~ 79Z 74.: I ' hr.lOoox .- 2: .:~~~~~~~~0-~4 .. .~~~~~~~~~.4 *F Pwt. than 10 obimvai)na the opinion of some academics'-they are not the most important issues on which the Bank should deploy its scarce project analytic resources in the 1990s. Rather, the mountng evidence' suggests that the critical methodological and practical issues relate to the evaluation of the impact of policies and insdXtiions on project performance; the building of project evaluations around realistic assmens of the lky environment affecng implem on, including institional capacity and the macroconmic framework; and the need for realistic risk/sesivity analysis as an input into selectng or rejwecing projects or project components and to help identify performance and benefit indicators to be monitored during imnplementation and operation. 1I Sc, for example, aNLiW JMinM, '-Poje Appaisa Planwing Twety Yea=n On' Proaed idh Wol &k Aima CoVeraw in Dewlpne Ecana. 1990, applkmwto de WW BRk Ecoomc Rew*w, 1991. I/ See WorlD De%wpase Rea11, 7 l. Mof Devewpn ane KwAmnT Porgoim Radanal. ftr Poliy Rdam Tb toduedviy of hwauAn Pqjet,' Apil M99L end Denia KCaufwa ad Yon Wang, 'U. Impat of ml, - P ~ IPoliCies on Ptmoot Pual in tbo S¢bl SeMt," Augut 1991. S. aso Geibud P.M en Dubtvkd Mihjgd, 'Uwinti ayd tho D _srpay beaw Ra.fRtum Esamps at Prjojt Appal and Poject Camplea Mnex^ C Page 4 of 20 9. This conclusion is supported by the ECON report's review of current Bank pracrice.6 'Te review confirms that-for those operations for which an economic rate of return (ERR) is calculated-appraisal estimates are opdmistic and narrowly focused on the calculation of ERRs. To be sure, there is considerable variation in the quality of SARs, but even the best do not quantify the risks to project costs and benefits of slippage on the macroeconomic, financial, and institutional capacitylimplemenion fronts. Yet these are prominent sources of project difficulty and failure.' OMS 2.21 notwithstanding, no SARs report truly epected ERRs,' in the sense of their being the mean of the set of possible outcomes." Downside risks are systemaically ignored, and as a result projected ERRs are biased upwards. No SARs cite the success rates for completed projects in either the sector or the country (or preferably both) although, as is clear from Box 1, there is considerable variation. 10. These analytic shortcomings have direct implications for the quality of projects. Since important project risks are not systematcaly considered, designs for which these risks make the expected ERR too low are not systematcally rejected. Quanifying the risks and their implications for project costs and beneffts should expose the weaker project elements, which can be strengthened or dropped, leading to. more robust project designs. Even better, sensitivity to macroeconomic, financial, and institutional risks can be considered during project identification, thereby influencing project selection early on. It can be reflected in proxy variables to be monitored during implementation, with a view to triggering remedial action as appropriate. 11. Meanwhile, for projects in the social sectors, which constitute an increasing share of Bank and IDA lending, there are no Bankwide benefit standards for investment operations. Benefits are often treated exclusively in qualitative terms. But even where benefits are quantified, cost-effectiveness analysis is not provided. In referring to differences across sectors In the degree of quantification appropriate for the analysis and jusdfication of investment operations, OMS 2.21 indicates that 'revenue-based measures of benefits may not be feasible or relevant in many cases, especially in sectors such as education, population, nutrition, and health," although it notes that "it may be possible to use quantitative criteria in such sectors more often than is customary." In practice, many operations lack clear statements of the criteria for judging success. The majority lack benchmaks for measuring performance during impiementation. But without clearly defined success criteria, it is impossible to recognize-and in turn to eliminate-components that are unlikely to succeed. Without performance standards, it is impossible to identify shortfalls during supervision and to set in train correcdve measres. Nevertheless, while the majority of operations lack explicit success criteria and monitoring indicators, well-designed operations include them. See for example Box 2. fi ECON rviewe 181 SAR. coerng al investm lon/redits approed in Y1. J/ See HSa Wahs, 'fli Perfom of AgpIcul1 Projects in Afrl A Review of Project Ratings and Risks," lune 1991; nu (lap Between Economic Rate of Retun (ERR) Esimates at Appmisal and Completion, and Project Risk Analysis." SecM89-319; June 1989; and Miehd Pomier, "Rapo of the Tauk Porce on Loan Procing and Project Quality." April 199. / Now at trictly ulting net prue vahl ("N) > 0 iath crierio for prioect sbction. Ther are technical reiasO why the NPV test may differ ftom the ERR test Nevertless, for psAtornal purposs, thi paper uses ERR througt, sinc it is a moan fMiwiar concept. 2Instead, a po estimteo of th ERR bad oan an thast do ewytdn$ goes accondilg toplon'-dw EGAP-is the srandud. See Geoge Boier, DUSS Paper: Econ Analys i Pwrjct Appil May, 1990. Annex Page 5 of 20 Sox 2: Best Practice Example: Expcit Success Criena for Education Projeds The T.iad and. Tobago Ed ncadio and Tr2bdng for Youth Employment Pject clearly states.its objectves, and indicates the criteria by which it will be evaluated. For use at dte mid-term: veW Idt e iy tesw will bustoevaluate the succs of the project aftere the effeci of the -o*n * at leoa one and half tmes as many g from Youth Training and. Employment . . : w.th im ilar characteic to YrP riees; would have f9und employfmentor self. 1 emlomet or * graduates from: YTEPP should receivi earning 20 prct greater than the control group ofo-graduates from YTEPP with: simil-r chaacteristics to YTEP. trainees. Tis . measure would reflect thie deie of the programin cot g to both.the Icreased succss of YTPP trainees inm finding employment aiudor In obtaining higher wages. The earnings of those in self-employment will be alcuIawd- separately firom those in wage wori. n: view of the- short time framei forb assesing. the- gains of the progrm,- the assessment wi also identify indicators suggesdve of higher future earings (i.e. contnuing education advancemet in ob ndg e ns ou sefemployme The project aiso specifics moni indicators-which will be used as a basis for annual evahuition and suprvision ratings. Al SAR Mpot No. 90-TRa I. 1199L - B. Recommendations 12. This suggests that (I) implementation of the guidelines needs to be enhanced and (2) that the guidelines themselves need to be changed. ECON's specific recommendations for the drafting of OD 10.40, EconomIc Evoluaflon of Invesant Operations include:" * Upgrade the atention paid to realistic evaluations of projected economic impact, based inter alia, on the lessons of experience, including the success rate in completed projects in the sector, country, aiid Region, as for example shown in Box 1. * Widen the coverage of economic cost-benefit analysis of investment lending, to include the evaluation of policies apd instituional change/capacity, integrating as approprlate, the findings of institutional development specialists and staff with other skills in assessing the likely performance of project-related institutions. jy ECON's w a oalw iclue dogiading th4 pzomine acced to th *aey of diffacutid fooa whA, in conveion fco, and accountin n of int Thcoce ps, while included in OMS 2.21, Ecaroacwysk of Pecat.r ha nver bee Muy impkmerted. I1 Sc Wlm. Dea Vailuuit, Ja Nim, ant Stamey Bwn, -Populadn, Heakh, and Nutritin: FY91 Secor Raview.' Page 6 of 20 * Ensure that the macroeconomic, financial, technical, and behavioral assumptions underlying the analysis are clearly spelled out. * For opeaions for which ERRs are not calculatd, require a clear identification of project goals and cost-effectiveness analysis. * Reaffirm that a common methodological approach to evaluation obtains throughout the project cycle-from idefication through appraisal and implementaton to compleuon and beyond. (See Box 3.) * Use sensitivity analysis to test the impact of variations in key parameter assumptions. * Insttute an indicator trakng system for all operations (see Section m below), with the indicators identified at appraisal-on the basis of sensitivity analysis, inter alia used as a basis for monitoring-and reevaluation of the operations-during implementation, and for informing the supervision ratings. (See Box 4.) * Provide effective support to task managers in securing appropriate skills, the lessons of expeience, iDputs about and analysis of country economic variables, and methodological guidae-including through an enhmaced role for lead economists and chief economists in the economic evaluation of investment operations. 13. The ECON Report notes that effectively implementing these recommenions will need to go beyond the drafing of new guidelines. Ask any task manage about project analysis, and the discussion quicldy turns to lack of management attention, staff iendves, and perceived pressures to lend. Many staff feel that projects will not be dropped even if the appraisal surfaces problems with likely viability. Hence if appraisals are to contibute an improvement in project quality: (1) managers will need to worry about the acual on-the-ground Impact of investnent operations; and (2) economists will have to sharpen critical aspects of evaluation analysis. The ECON proposal embodies three actions: * Monitor poafolio qualty. 'he proposed indicator tracking system, if Implemented, should help to focus management alteon on the evolution of a project's-or a county portfolio's-projected on-the-ground impact. By shorteing the feedback period, is should help to increase managers' concern with implementation and impact during the upstream stages of project processing. As a result, anagers should become more concerned that projects are designed to be successful; that unjuifiably risky componens are weeded out prior to negotiations; and that losses resulting from less than satisfactory performance projects are contained. Managers should thus be more conceed with both the substance of sensitivity analysis and the resuks of indicator racking. With Bank managers more focused on in-country results, realistic ex-ante assessments of likely resuts will become more. valuable to them, and acc:urate, forward-looking economic and institutional evaluations likewise. These changes should lead to better designs early on in the project cycle so that fewer problems surface at appraisal. But it is unlikel - and probably subopdmal - for no problems to arise at appraisal. In the event, the acid test will be the willingess to drop problem projects. Page 7 of 20 Box 3. Evalaion nd atg Methodology over the Project Cyde ' ' '' , '.d BenRefi > Ee.dg. cout _- _ ulat'as.Lt 'uali$ t ca,e'2,, . . - :all&bonaeusus o_ critical atkeholde:rs .~~~~~~~~~~~~~~~~~~- ....C Apprl- .a.: '~ ~ ~~~~~~~~mat reletig eson o epeiece andt,.rifeRnsks ? teui¢and indlcatoia w w ba-iof p oject otoctiives * Estaxblish thes basis for th eotn of the koes mary informatiS;on v~~te Of f_ .necesqydi Id q - - * wnlud, in coennt t e ncesry conditions.fbr achievingsuccess * Include in legal doment trigger mechaim if shoral gdless of cause ''X,:ut p? td'A{>sve l pmsnt; po componm anti:-h amaInobjetdive be fien reguAtory Rfo in the statsw. .Deigning incator of p.rogress.waso straig w g appal, the discussion w.ith tesauthorties foculsed o ra proco and format to be used. to assess ithregulatr a1-3 S' *, zife S094,- - .t.3 *'S. 198 '$;sf "i: :;'.i: '-';. *. .I SA RXow Io.107-M,Ml6,192 JZ/ Seo Miguel MAti, wWeteng tnomie o Fm0hPu Impze on Imp n-Higthligh of the Semi-Ameal AIUS Meeting with Mr. Jaycox," June 3, 1992. ,3 Se C.ieorgs West, ZInformeloaSses SupoSt for Poitfoio Maqme. Sun 15, 1I9. AnnxC Page 12 of 20 Appraisak * Clearly Identify project goals and establish concise project objectives consistent with sectoral/seglsc views; if relevant, calculate the ERR. * Conduct senksithivq analysis for key variables. * Designae ortant variables as indicators for monitoring during supervion, as a basis (1) for approxImating the ERR where fIl reestmaon is not possible; and (2) for indicating by how much the asmen of an operaion's impact has improved or worsened-for SALs (see, for example, Box 7), PHN projects, and instional and policy componens. Indiae how project outcomes and ratings are expected to vary with the key variables. Ngo dadowta: ILclude in the agreements reached durig negotations the necessary actions by the borrower for achieving success as defined in the project. nclude in the legal documents a tigger mechanism for cosultations about possible restructuring/cancellation,if the is a shortfal In the project indicators that makes succes unlely, regardless of the cause of the shortfll. Provide for apropiate remedy in case of filure to achieve agreemnent folowing consltaions. * Track the critical indicators. * Reflect the movement in the critical variables in the development impact raing. * Explain the aalysis in the Form S90, or in a revised Form 90. * Decide what action is necesay. C. From Supervison Rati to a Country Portfolio ndex 19. n7vl wove Inx. Once supevision rai are based on a sound, ranspant, analytically-based sym, the ca provide imortan data for 8tcking the permnc of the cony porfolio. For ilustaive purposes-to show the puposes the portflio index could serve-4th secton coniders various possibilities usig the existing ARIS database, which records project supervision ri for the active portfolio. Box 8, for examle, shows the elution of the deveopment Impact ratngs, which have been converted into an ide (la the box, hi numbe are beter; 100 corresponds to a supervision rating of one.) The index shown there is based on the intdividual project/operation deveopment impact rangs, weighted by the respecdve share of the loans in the active portfolio, for the 28 largest country portfolios. 20. Conpl a Pro blem wit Cnoss-Cowuiy Compansons. rhere is a major question about the validity of country comparso using an index along the lines developed in Box 8. Countries cleary differ in ex ante impleme on risks. Based on completed projects, for example, the success rat ranges from a low of 17 pece for Uganda to a high of 100 percen for Chia. (See Box 9.) The esx ae risk 4 ..,... I.,:.,.~~~~~~~~~~~~~~~.;..... I~~~~~.. ........ ... - ..,~~~~..S.A...4...... ... I . .... ................ ...7. .,..r.. .~...~....~.i..... it [4 AnnCL Page 14 of 20 to be cutouly execised Indeed, if the supervisi rating suggest better performance than the hisorical record of completed projects, this may reflect gi improvement in counWy pformance, or opimism. e tdimeions of the scopo for iepretao are provided in Box 9, which compares the country succes rates for completd projects with the likely success ratn In the ARIS databaseY.9 Note th the discrepancy is lar for Kenya, Philippines, Algia, B and MxicQ, d Vey large for Nigria, Tanzania, and Ugda Bawd on othr e nc the case for improved performance Is clay tonger for some of these couatries than for others. 21. Oaanges over 7n. A beter way to use the ountry portfolio indx may be to see how it changes over time. ibis would abstract from cros country factors and fcus on how a country portfolio is doing compared with its own performae in the previous year. Such a measure may also be subject to optimism, altugh it will only r if there is a dhage in the degme of optmism from one year to th next Here there are two altnave meaures-gross and net. The gross measure would simply calculate the chag in the country index from year to year. However, since the composition of the porfolio may chang, this may be misleding. For emple, if a number of new loans aropened, the In will tend to improe. To correct for this, we can look at the index in net terms- is holding constant the composition of the portfolio by pairwise compaing the radtgs given to the part of the portfolio a is common in any two adjoining years. (See below.) 22. Cowry Parfolto nd. Box 10 constructs such an index of change. (As with Box 8, a lar nmber is better 100 imples no change from the previous year.) k is epessed In net terms; that is, the reflec the do ges in the rating betweet d two yeoas for the samo set of operatio. Note that only Mexico saw Improved potolio ratig In each of FY90 and FY91, while fty percent of the counti saw a de in both yeans. In t other coOunt, peform was mixed. This formuation of the inx coud provide the basis for a meaingM diogue on the country portfolio, focused on the reasons for year-to-year changes. (Are they due to country fcors? To sector factors? Were the iniial estimates overly optimistic?) It would also be usezll to discuss the index in comparison with PCR rating for projects completed in the year. 23. 4>aemc S*wds. Cleady if the in approach is to be usefil, it wil need to have crdiility widhin the intiuto To this end, the following systemic safeguards should be considered: * The specific indicat to be used for rating the project should be approved durig the Regional loan approv proess, and the RVP should be acountable and responsible for the integity of th system wthoin te Region. * Ihe analysis for deAving the project ratng should be transparent, with the Form 590 suitably amended to _ the supportng analysis. * Spot audis by the Tecical Departmes could review a certai percentage of supervision portles. IV/ nom *i puyjss mmd I ow 2 an oomiden ilraI woeeuu Page 15 of 20 ox 8. Deopment Impat IndX' of MJlr Cowty P.atfolios A~~~~~ 15...,. O82~ ~ ~~ ~. MUIR.$ . 57$ .~~~~~~.. ..~~~~~~~~~~~~~~~:. .... ... . ,~~~~~~~~ .~~~~~~~~~~. ~ ~ ~ ~ ~ ~ ~ ~ ~~~jc~~~~~~~~~ ..... A MO.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ W Page 160£f20 Box 9. Sucess Rata for Major Country h4bUal}s: Completed and Ongoing Projects Comtpared ;~~~~~~~~..4e. W........ ._ _ :'~~~Wi 1 ;'~~~~~~~~~. ch,- ADna-C Page 17 of 20 D. Bendlts and Costs of Developing and Using the Index 24. Benelts. At the macro level, the index can form the startng point for a focused discussion on coutry portfolio perfomance m _ At the micro level, the index can help to surace problems n individ projes. It can be used to tgger project reevaluation and remedial acdoh. Of greater czo;tnce, the impovement of sensitivity analysis at the appraisal-and earlier-stage can telp to weed out poject a are especialy vulnerable to riks wih a high probability of occurrence. It can also help to improve supervision planning. Clearly the key Issuo is the sensitivity analysis. Ibis crucial step in project appaisal establishes the principal variables to be monitored for chges in development mpact during implementation Hence by Improvig semniivity analysis, we can improve supevision planning. 25. Cst. Even though fte proposed approach makes use of exising systems, there will necessarily be costs In implementg it For the 220 or so new loans approved each year, the cosm should be small reave to other preparation costs. They would entail basically a smartening up tho project mlysis trough greaser specificity and focus on sensivities. However, for the 1800 or so loans in the portfolio, the issue wil be more problematic. Meaningful sensitivity analysis was probably not done. To now go back and identfy the key variables for tracking during supervision will entail additional work Even if each opertion were to take only one day, the onetime cost would wtal some eight staff yas, or US$1.5 million. If one week each, the cost would be US$7.5 million. To this we would need to add the cost of additional upstream guidace to staff conducting the sensitivity analysis. This could take the form of sector panels of exper to establish the standard varables for sensitvity analysis for diffrent project types-and the reladve weights of the variables-and/or of a compator project data base, which could generate subsector benchmarks for use by task managers. This would cost an additional three staff years, or US$0.6 milion. IV. Indexes for the Progams of Spea Emphases 26. The approach developed above has potentially wider applicability. In principle, indicator tracking can be used for any project objective. This section discusses possible separate indexes for poverty reduction, environment, and institutional development. It concludes with a brief discussion of a possible composite country index. A. Povet Reduction 27. Bank opeational policy does not recognize the amalgamation of povey reduction and ERR (or analogous measure for projects not amenable to ERR analysis) into a single measure. Rather, according to OD 4.15, Povery Redwton, opertion impacts on poverty are to be considered separely. Moreover, targeting is to be the primary determinant of whether a lending operation is to be couned as povertyreducing Ot not in the Bank's monitodng system, under the "Program of Targeted Interventions." Projects in the following categories are to be included: V9? Sea, for mpls, pau. 29 of OD 4.15. It ste *P*-fimuWed poveat rdtw t wo a. theuutns ec wAecrnie towst that app to oer pmjec Thsy mm hoe a satifatesy eemmais mia of cr-in PIdewto whezdo thecalati of an ecanxorc m of min ro pacabis-be the -a t way of aBhIv1ng prje befit. Rahe tm usla g dw atnia hWesncra a for Dank finahg of pinjecta, grata oeavauu aftteto ahoul be paud to idetiYing projects an projct desins that both benqt ibw po or eand Amv hA* veOus, utdw mbe surd upliCit in ftms Of an ecoummic fte Of satrnoriftlcll sa (fo hawse ressre mwv p MwnL* to q. IN ~ ~ ~ . . . ... t A t t.-.s ; E :,.,.: :.. .j . X ' . Z f f ' . . . f , '' , :' . X. L! . 9I40; JS e i01 8 l l: t e !g...| Xi~~~~~~~j 1111 Ii~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~3 :;.. 2..~~~~~~~~~~~~ Page 19 of 20 * projects with a specific mechanism for targeting the poor; and/or * projects for which the proportion of the poor among project beneficiaries is significantly larger than their proportion in the overall population. 28. 'Me OD is explicit (para. 31) that the appraisal of such projects should carefully asse the: "targetng mechanism to be used .... During project prepard;ion, the risks of project benefits being diverted to better-off groups should be carefully evaluated.* The OD is also explicit (para. 41) about the monitoring of progress in poverty reduceion projects. AccoWingly, it states: O... every such project should have benchmarks and monitorable social indicators for gauging progress. Poverty-reduction projects often encompass innovative featres that waurant review-for possible adaptation-during implemeion. Monitoring and evaluation systems should establish the information base for judging how these features are working and for redesigning the project if necessary. At the end of the implementation period, the monitoring and evaluation system needs to provide informaion to assess progress in achieving project targets and reducing poverty." Finally, the OD is explicit about supervision (para. 42). Thus: "Where projects include specific targeting of services to poor people, the monitoring and evuation system should be used (a) to assess whether these servicds are indeed reaching the target group, inter alia, by tracking feedback through beneficiary assessments as the project progresses, and Ob) to identify adjustments to project design that would increase its efficiency and effectiveness in reaching target groups." 29. It is clearly consistent with the above that indicators of success in reaching target groups be identdfied at appraisal, and tracked during implementation. They can be combined into a project poverty- reduction index. In turn, the project indices can be aggregat into a country poverty-reduction portfolio measure, using the various index options discussed above and illustrated in Boxes 8-10. D. Environment 30. It may also be useful to have a separate index of environmental impact. Indeed, the Form 590 already recognizes this, allowing for an explicit rating for environmental issues. To some xten, a project's environmental impact will be reflected in the economic index, where environmenal objectives are essenl for meeting project economic goals. For example, in an agriculture project, stabilization of erosion may be an essential precondition for achieving a project's primary economic objectives. However, in other projects, environmental goals may be separable from the primary economic goals, or there may be tradeoffs with the project's primary goals. For such operations, it may make sense to track environmental performance separately. As with the approach espoused above, indicato should be identified at appraisal for tracking during implementation. The movement of the indicators would inform the environmental raing. Annes Page 20 of 20 C. Institutlonal Development 31. Following the above line of thinking, it Is also possible to evaluate a project's progress in meeting Institutional development goalsO As with the environment, some insttutional development object ves may be eseti for achieving project success, and would therefore need to be captured In the projist's economic rating. But somo aspects may be quit separam, and may warr separ monitoring. A composite instiutional development impact rating would require a team effort between an economist to evaluatse the impact of inlstiutional development and an instutional analyst to evaluate the Impact on the Instoitton Itself. D. Composite Index 32. Having established individual indexes, the question arises as to whether (and how) they might be combined into a single index, and what such an index might mean. This goes beyond the realm of economics and Is vey muwh an Lssue of tracing perfmne vis a vis Bank operational priorities. A possibility would be to assign weights at appraisal for economic impact, poverty reduction, evironmet, and insitonal development, based on an assessment of Bank operationa priorities in the country. These weighs could be used throughout implementation to add up the individual Indexes, which in turn would be derived via indicator tracking, in line witi the processes described above. In turn, the ihnidual project indexes could be aggregated ino a single country index, using loan-share weights. A ompo*e hdex developed In tds way cou provide a single measure of progress *t achieving projectfprogram goals, but would need to be nepreted and used with etreme caudon, especiay giwn die siabjewe nature of the weights to be assigned to economdc Ipact, poveny reduction, envirounent, and adewlopment in she Inidual pvject ndexs. In the circumsances, the development of a composite Indicator Is not recommended. Zi Se. MAmd Dia to Migul Mait, UmWAtioa su-PollOw to doY91 MIRaoa AW, Apil 27, 1M, for a pmpsa to inh=km indicaor unadag as a basi for .pwviow atig for TA pzoy. Annex D .Operations Evaluation: i the: Bank * Portfolio .Management Task, For.&.e.' Page I of 11 OPERATIONS EVALUATION IN THE WORLD BANK A RADICAL VISION FOR THE '90s ANSD BEYOND The Background The Office of the Director Generl, Opeatons Evaluation was created in 1975 to institionalize independent reporting to Executive Directors and Management about the development effectiveness and the lessons of eperience of Bank opeations1. The vision driving the ealy evolution of that Office embodied: i) self-evaluation of all Bank projects by opeaional staff, ii) independent audits of the integrity of the evidence and analysis of project evaluations by operaaonsal staff, iii) annual overviews by the evaluation staff of project evaluation findings and significant lessons of project experience, and iv) a continuing prgam of independent studies of pardcular aspect of Bank operational experience and processes. In the seveteen yeas dt have since elapsed, this early vision has been lagely realized A robust and useful Bank-wide evaluation systen is now in place. Evaluation of outcomes is an accepted obligation of all opertional umits, not just of the Bank's evaluation staff. Portfolio coverage is comprehensive. Transpaency of reporting is the rule. Lessons of experience are incrasingly being recycled to improve the effectiveness of new operaions Efforts to interest borowers in participating in evaluations of their Bank- supported projects and adaping the Bank's evaluation experience for their own purposes are beginning to bear fuit. But the Bank has also changed during this period-in the emphases of its operations, in its smucture and processes, in the intewational environment it confonts, and in the concems of member govemments and their representatves on the Bank's Board. The vision of the '70s accordingly also needs to change. The purpose of this note is to point to where change appears to be most needed. t Rerenceas to Bank opeMions include IDA opeaon as well 2 Refe6nces to projects unclude opaaions of all types. Annex Page 2 of 11 OPMUMIONS EVALUANTION: the s andbeyond (7/MW9) The Future Widespread concern among Executive Directrs and staff about the detenoration in the Bank's portfolio, the dearth of information about operatonal (as distinct from iplementation) permace of Bank proects and the need for earlier feeback of significant lessons of operaonal experiee ggest that an enlged vsimon for the '90s is now needed. The main tlu of this paper is ta the Bank's present evaluation system should be modified in the following respects: * the portfolioperformance information system should include compree infornation about project benefits as well as about implementation outcomes; - opemttonal staff should evaluate peiodically the overall performance of their country portfolios as well as the pmance of the individual projects in each portfolio; * the Bank should have a consolidated azmual evaluation work plan, to ensuwe that the work progam of the Oprions Evaluation Staff complements adequatey the evaluation work plans of other Bank uimts; and * fte Bank should develop an eary evaluaton feedck loop. One approach might be to expose all executive prject briefs to informal comment by evaluation staf, to ensure that planing for new projects benefits from all relevant opeato epcice. The rasning bend these recommendations is outlined briefly below. Does anyone now know what loan benefits are? Comprehnve completon repoing by Regional staff has been Bank policy since the mid-'70s. PCRs have since provided basic ifmion for OED evaluations, which have expnded considerably in scope and impact in this period, and for periodic OED Page 3 of 11 OPERATIONS EVALUATION: the 90S and bernd (7110i9_) reporting on the implementation perfonnance of all projects. However, the Bank lacks comparable infomation about te perfomance of projects during their benefit-generaing phase. Opetional staff may know about the acual benefits of projects in which they have been involved, about the actual benefits of projects subject to OED's impact and othe evaluation studies, about actual benefits reveaed trough repeater projects and setor reviews, and about the benefits antiipated at the conclusion of proects' impleen phase. However, comprehensive monitoring of acual development outcomes is not yet part of portfolio m ement the Bank. This peformance information gap reflects in lage part the fact at the day to day business of the Bank focuses more on new lending and the supeMsion of implementation and uement than on wbat bappens duing the late hfe of Bank loans. Regional work progams and budgets, pfbfrmance assessments of Regional managers and staff, Board debate and decisions, the Bankes amual reportng, Bank dialogue with borrowers-all highlight the frnt end of the leding cycle. The development benefits of this lending are not yet a benimark aginst which te performance of the institution and its staff are commonly evaluated Portfolio manement now systematcally monitors impl aion, disbureent and loan serice, but not dmlopment results. The many Regional managers and staff who do endeavor to keep informed of operational results do so without clear instional incentive. They ar expected to report on pressing p ent and otw i n problems of the opeations they suevise, but not on devdopment benefits; nor should they be since supervision ends when disbursement ends. Executive Directors and anagement now learn about the results of Bank lending episodically, mainly through repeater projects and OED reviews. Staff mobility reinforces the divorce between concean for new lending and its development benefits. Opertonal staff are profnmdly cnened with results ex ae: they apply major energies to the design, anysis and conditionality of country prgms and new opaons in rd to maximize the probbiity of positive outcomes; and they build monitoring and evaluation components into many project However, they are not now accountable for results What happens later is often perceived to be more OED's concem ta theirs. Page 4 of 11 OPERATIONS EVALUATION: th* 9S andbeyond 17/1O2) The radical premise of this paper is that the Bank should be as concerned about and accountable for the "development worth" of its loan portfolio as it is now for its performance as afinancial intermediary. If this premise is accepted, then portfolio management will have to begin to look beyond implementation. PCRs produced by the Regions now generate comprehensive infomation about actal project perfonnance only thrugh the implemenn phase. For the Bank to be accountable for the development pformance of its lending, comparable information would also have to be generated about the actual benefits of Bk loans. Is comprehensive information about the portfolio's development performance feasible? Portfolio magement could quite readily encompass the benefit-geneating phase of Bank lending if perfomance data on outstanding loans were to be designated as a new (Regional) produc4 sepamte from the repoms i which these data would be analyzed and communicatd. For this to happen, country departments should be made as responsible for informadon about the development performance of their country portfolios as they are now for infonnation about implementation. Comprehensive information about the oprational peformace of Bank-spported projects need not imply extending supvision repordng tough the benefit-generating phase of all a The Regions need only amnge for systematc borrower reporting about the development performance of ther outstanding loans, just as they now azmnge for systematic borrower reporting about implementaion. When and how ftis information might be used is a separate question. Much inrmation about loa permance is probably alteady available to well- managed borrowers and could be readly shaed with the Bank if it is not already shared; but most borowers would probably not be able to respond. In these cases, the generation of infonnation about performance after implementation should becomie as much a goal of the project as the geeration of information about ilement The piimary objectie would not be to satisfy a Bank need. The objective would be institution-building--to sadsfy the m management information needs of project ownes, which the Bank could then shae. Page 5 of 11 OPERATIONS EVALUATION: the 90S nd bevonA U7/1 -9) To*this end, Bank appraisals should begin systematically to: i) identify the minimum information borrowers should have for themselves about the operational perf ce of eir projects, ii) evaluat the borrower's capacity to generate this informaton, and iii) identify and arange for the support needed to help theborrower develop capaci that is lackdng. Would the Bank's evaluation products change? Comprehensive information about the development performance of Bank loans would make it posble to enlarge and enrich the present mix of evaluation products. Country portfolio evaluations The most important new evaluation product that should be devloped would be periodic country depatent overiews of the Pifon of thei coutry portfolios: not costy self-contained reports that discus the composition, rationale and evolution of each country portfolio, but brief reports-mainly tables-showing for each portfolio what was expected at appraisal, what is expected now, acuals to date, and bref commentary about significant deviadions from expectations for both individual proects and the country portfolio as a whole. The performance indicators tracked would be those agreed with the borrower at appraisal. Other portfolio evaluations This iformation would make it possible to geeate at negligible cost a variety of periodic assessmenut of portfolio perfrmace: ovevews of treads in country portfolios, in Regional portfolios, in sector portfolios, in special interes portfolios, in problem projects, and mn the Banles overall portfolia Assssments of portfolio development peformace would no longer have to rest prmary on the benefit re-estmates after implementaion that are now the basis of PCR ratngs and OED's annual reviews. The PCR Since the PCR marks just one point in the much longerlife of projects whose benefitlgererating phase sdll lies ahead, evolution along the lines described would Annex -D Page 6 of 11 OPERATIONS EVALUATON: tbo 'M and beyond l7/1 90,) make the PCR a still inportant, but no longer the final evaluation of most projecI A PCR should condnue to be produced for every loan to provide comprsive information about implementation expectations and results, and an updated benmrk against which to plA fOr and evaluate fuure beneflts But it could now become a diet and less cosly product: bnef table presenting expected and actmal implemenion dmes and costs, expected and -estmated opeatonal costs and benefits, brief analysis of significant variances from ewectaions, key lessons, and a forward-looldng assessment of the project's likely devdopment impact Lending process evaluations Transpart evaluatons of expeiene with lending process such as superision, pr e nt, l covenants, use of consltants, etc. should continue to be produced. But whether they should henceforth be produced by OED, OSP, or LAD should be detmined in the light of the comptecies and work proams of these units. Evaluations produced outside OED Improved ifmation about portfoo peformance would fcilitate more evaluaons of opean ience of the kind that 'have long been canried out outide OED-by OSP in its policy wori, by DEC in remrcb, and by the Regions in their econonic and sector worki To ens that the Bank receives ful benefits from these diverse imtatives, the Director Genea should henceforth, in additon to ensuring as he does now ta OED's studies benefit from and build upon other Bank evaluations, also: * attest periodically to the adequacy of the Bank's conslidated evaluation progam, and * attest through selective audits as is now done for PCRs, to te a*quacy of the evidence, analysis and findgs of selected evauations not cmried out by OED. 1 Onbsever has _ssed that since the PCR doesut * th comple of the poject but only of its ip- a and Bank the PCR mih botter be rnmd_ 'oompo report', or DCR. Page 7 of 1I OPERAMIONS EVALUATON: e 90S and beyond (7/1092) Does anyone else share this vision? Ile views in this pWe are those of the author alone. But their centl dhust ppears to be shared by the DGO, judig by his rect note to the JAC about the future of evaluation in the World Bank (JAC92-19/1, May 26, 1992). -.. .the time may have come to view, more systematically, the country as another 'unit of account', supplementLng the project." ^... .accountability has become less a matter of successful project implemenation ithan of development impact .." * It has become increasingly important to A.. .improve the information base for reportng on performance." * MThere is a need to institute a consistnt thread, or continuum, throughout the life of Bank opeaons from idenficaon to pact evaluation, keeping in sight at all times the intended goals and the expected results." - "To know more about the development impact of Bank opetons calls for an explict up-front definition of indicators to capture and tack the opeations' economic and social worth .... To this end, the Regions need to ensue ta new projects contin workable momtng and evaluation systems. " * . . .more clearly formulated ex ante economic analyses of operations. .would provide a solid basis for making judgments on performance as the proect cycle unfolds." e "Integmng the PCR into the project cycle, by prepazing it in connection with the last superi on mission, makes it a powerful tool not only for assessing results but also for spefying actions still needed in the post- compledon period." AnnexD Page 8 of 11 -OP0ATIONS EVALUAION: dhe a9o nd beyond (7/1W9r) * PCts should become more fiully a tool of country portfolio management" *.. .the prcedon of the Banles accountability has changed: what manttes is not just whether the Banks reources were actually used according to the plans agreed at the apprsa stage, but what contribution they made to the borower countrys development This change calls for cotmty staff themselves to begn looking more systematically at the eventual impact of dther operations." -. . .couny depments should conder engging more sysmacally in the self evaluation of country strategies and portfolio management, as a routine integ part of the country planing process.0 * . .evaluation should extend backward [from the PCRJ-o yield better indicat and leso from ongoing pefora -and forward-to yield more infmed judgments about long-n development impac. . The recommendaithat the DGO henceforth ate to tbe adequacy of the Bank's conlidatedevaluation program when presenting OED's work progmam for approv follows from the fact, noted by the DGO, that 4'. .OED's sies progam natually ovaps areas of concem to opetonal uts, resech staff, and the internal auditing deparment.. Where to pitch the evaluation studies program in relaion to self-evalation, policy, research, and operational work is hus a mater of judgment-as to how much of the Banks opeational policies, progams, and processes should be subject to independent and transparent evaluat .. The Portfolio Management Task Force A The ms of refoece of the aons evaluation sugp posed eight quesdon The implcations for the Task Force of the vision outhied above are noted briely below as reples to the questions posed. Page 9 of il OPERATIONS EVALUATION: he VW and beynd (/10/2) 1. What, (f any, changes in Me role, practices and program empha2es of OED should be reommended? * Accountability lTe Bawk should remain t self-evaluating institution ta it has been since te prsent PCR policy was instituted. But its aceountability should be enlarged to include loan benefits as wel as implementadon outomes. Consolidated evaluation program The DGO's role as the independent evaluatr of the Bank's development v should not change; neiter should his role as the auditor of portfolio pefomance evalaions done elsewhere in the Bank. However, if the Bank were to begin to formulate and monitor the expUcit consolidated instiUtonal evaluation prgam that is recommended, tha should be donejointly by the Office of the President and the DGO, and the DGO should attest to the adequacy of OWED's work program In the context of the Bank's consolidated evaluation program Scope of work Some Board members have suggested tbat the scope of OED's work be enlarged to include riews of los under mplementation, in o provide the Board vith earler asesments of pordflio pfmnce tan it now receives. OED has long vwd seledted loans still ur iplemtaton in orde to updae and test finding fm completed loans about a p lar issue under study. It could also audit Regional and OSP reviews of proec under implemen , although it bas not done so to date. However, it would appear inappropriate for OED to evaluate ongoing operations which are still being supervised by operational staff. The Board should be ifrmed to its saisaon about poect under impl on; but it should be looldng to Man for this infrmadon, not to OED. If past reporting to the Board about ongoing projecs bas been defident that problem should addesed direcdy, not through OED. Early feedback Staff may klow what is happening with their projects under implementtion, but it is not always clear that al relevant lessons of experience have been adequaty reflected in the design of these pjcts. To respond to this long-stding concern, the Bank should consider establishing an early evaluation feedback AauD Page 10 Of 11 OPERATIONS EVALUATION: tw90S and be_od (7f1O/-). loop, in which all executive project briefs would be exposed to informal comment by OED. The objective would be to ensure ta all new projects took amcount of th relevant lessons of opeaonal expeience, and that evaluation lessons were formulated by OED in full awareness of curt operAtona1 pans Such interaction should not compromise OED's independence in any way. 2. is the PCR and auit process worling well? This note has not endeavored to evaluate cuet practice and exeience. However, it suggests that the specifications and role of the PCR nmit review for the reasons noted above. 3. Is the balance of OED atention mong PCR reiews, PCR audits, spedal sudes, couWty rWe)e, bnac evauons, anual reiew, conry development, and ftadac ssnadon what it should be? s noe bas no endeavored to evaluate cu practice and exeiece. 4. IS 109% PCR coverage and 40% audt coverage really necessary? PCR coverage should remain at 100%. But the audit ratio should be reconsidered. To test the integity of PCR reportng may not require as high an audit cove as the audit of a rpresentative sample of the Banks portfolio would require. OED should conider auditing othr Bank evaluations m addition to PCRs. 5. In consiering the operadon closed (for the ank) sx months qfier the last Bank- financed disbuwent, Is dte Ba* neglecting vit dknems (eg.) in Instion building andpolicy rejrnn) which extend beyond disbwwnent? Yes; that is the reason for the "dical vision" proposed. Page 11 of 11 ,OPEAONS EVALUATION: dw0 and beyond (7/1OW-) 6. Should the Reio do Impt evaluatons? OED should be the pdmary source of impact evaluatons. But whenever the contuatio of a seor lndn proa called for validaton of assumed pans of proam beneits ad relevant OED evaluations wer not available, ready access to inforadop about post-implementadon performa should encourap the Regions to evaluate impact on thi own. 7. Is t*e bhom-d-ter ERR bM of compaon too narrow? Adding to a borowers debt cannot ever be justfied without commensuate benefts Tus, wheeve an ERR was part of the ex-ante justification for a loan, it should be a basic part of its post-evaluation. However, benefits that cannot be assessed this way sol still be eluated by whatvr means are feasible and approiiae. 8. Are the speciaenhaws adequarely covrd in OED work? Ths note has not eadavored to evaluate curent prace and exne > ' fi * : ' ' ' "" 2 ' : 2 ' 5 - Annex E ... m.- : .: . . . ..-o; h s..: . . . :.... ....... ... :. . . = ...... .. . s s .. 2 ;. W. .: Ci.LWA. .. - . '.'- :'. . i ' .' . .. s j \ c . -. \ . : ..g_i. ..'_ ''% :: j . eV .'S, S'. , '5 ;''t.{S::: 'fo: f . ' ' 0 . . : :..:: - .: '-. - ,x :.''.: '' si ';':s' .''': -. .' ............ ."'' ., ' .... : ' ': s.2,X. . . :.:': " . . M , v X,i,,; K 9':-,: ''.", "',.''" - -. . ..: ' ..', ,,'; , ' .. . .': ., i o',.: . . " . ' . ' ' . ' '. 'S'. ,:: , ,,, c: .,,.'i, ,,,,'': '. :'"- .:' '.. ' ''..' I^ Xs:: i j .d ..... . . .. *:Ffel.xq c7ss?< e e M jj; 5 i : i :: -List of Working Papers 34 '-;t9 s ; * ' ' {: . '-,:.<" ... ,.'-., ', 'o', . .- ....... ,: i o :..B S i'.e - - j :f , : = . - . .......... - '; , :Z : . he - ........ , : : . ,.: ', . ................... . . , , . . : ;i' j'.'s:. ': :.. '.. .' .' : . i f ; .--''".': ..' . :, . Si..' ' 's . :.f'..''"-"',,,- : 'S .,. t: ;'t ',. ' 'S ^ ' 0. , ,. . ' ':' '-,''"''"''"''''.' ' :,S '9SS , ........... ,,. -. . : - - j: . .. . X :. -9-:.x Portfolio Management Task Force Y s: i * . . . * :.s _____ Annex E List of Working Papers Waorkng Paper A: The Bank's Mandate: Nature and Division of Responsibilities Worldng Paper B: Review of Problem Projects Woring Paper C: Supervision of Policy Based Loans Worldng Paper D: Procurement Woridng Paper E: Computerized Database for Legal and Related Documents Worldng Paper F: Information Flow Analysis of Supervision Functions Worldng Paper G: Budgetary Aspects of Portfolio Management Working Paper H: The Role of Infonnation Management and Technology in Portfolio Manaement Working Paper I: Use of Plans of Action in Legal Agreements Working Paper J: Review of Staffing and SkIll Mix in Supevision Working Paper K: The Role of Non-Govemmental Organizations (NGOs) in Project Implementation and Supervision Wodrng Paper L: Cofinancing and Portfolio Management Woking Paper M: Towards Improving the Financial Analysis Practice Worling Paper N: Construction Quality of Heavy Civil Works Projects Working Paper 0: Programs of Special Emphasis: Implemenation of Women in Development ~~~~~~~~~~~~~~~~~~~~~~~~~~~~..:: . .. . ...ew .; * .. .. y:e,"....