Report No.25145 Annual Report on Evaluation Capacity Development 2002 June 13, 2002 Operations Evaluation Department Document of the World Bank FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.SA ofiace of the orector-Gene Operations Evaluation June 13, 2002 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Annual Report on Evaluation Capacity Development The Bank's strategy for evaluation capacity development (ECD) was established in 1994 to help borrowers strengthen their monitoring and evaluation (M&E) systems as an integral part of sound governance. ECD was conceived as an integral part of the Bgank's country portfolio management work. It was also conceived as an instrument of public sector management reform. Since 1994, the priority for ECD has been confirmed by the Comprehensive Development Framework (CDF), Highly Indebted Poor Countries (HIPC), Poverty Reduction Strategy Paper (PRSP) and other initiatives that emphasize development results. Tracking of progress towards the Millennium Development Goals also calls for ECD, as does the growing realization that the quality of M&E for Bank-financed programs and projects is inextricably linked with the capacities and systems of borrower countries. Accordingly, the World Bank has committed itself fully to a greater focus on the measuring, monitoring and managing of development results. Bank-supported ECD activities have increased substantially in recent years: since 1999 they have started in 21 countries, including eight in the Africa region, five in Latin America & Caribbean, and four in the Europe & Central Asia region. In 2000, Bank management agreed to mainstream ECD by FY04. This first Annual Report on ECD (AREC) seeks to clarify the meaning and relevance of Bank work on ECD, to lay the methodological foundations for subsequent ARECs that will track the progress and draw evaluation lessons from the Bank's ECD work, and to present an inventory of ECD activities underway across the Bank, with special attention to country-level activities. This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. i Table of Contents Acknowledgm ents .............................................................................................................. iii List of Acronym s ................................................................................................................ iv Executive Sum m ary ........................................................................................................ma. V 1. Introdution ..............................................................................................................1 2. M onitoring, Evaluation, and ECD .......................................................................... How is ECD Undertaken?..................................................................................................... 3 3. The Bank's Evolving ECD Strategy .......................................................................3 1993-1994 Bank Task Force on ECD ................................................................................... 3 Bank's Governance Strategy................................................................................................. 4 CDF and PRSP Initiatives..................................................................................................... 5 The Bank's Approach to M &E and ECD ............................................................................. 6 Accountability for Development Spending........................................................................... 7 4. Stocktaking of ECD Activitie ................................................................................. 8 ECD Activities Underway with Bank Support ..................................................................... 8 Central Support to Country-LevelECD .............................................................................. 10 Identification of Bank ECD Activities................................................................................ 14 5. Issues, O pportunities, And Challenges ................................................................15 Heightened Priority forECD .............................................................................................. 16 Some Issues for M ainstreaming ECD ................................................................................. 17 (i) W hat is M ainstreaming?.............................................................................................. 18 (ii) Need to Clarify M &E and ECD .................................................................................. 20 (iii) Inconsistency of Message in Bank Policies and Guidelines on M&E....................... 20 (iv) ECD Coordination and Knowledge Sharing .............................................................. 21 (v) Bank Resources to Support M &E and ECD............................................................... 22 Future OED Annual Reports on ECD (AREC)................................................................... 23 6. Conclusions .............................................................................................................24 ii Annex A: ARE CMethodology ...................................................................................2 5 Annex B: Evaluation Capacity Development'(ECD): Definitions, Results Chains, and Performance Indicators .....................................................................2 What Does ECD Success Look Like? ................................................................... 29 How is ECD Undertaken?...................................................................................... 35 Key Characteristics of ECD ................................................................................... 36 Annex C: Detailed Stocktaking of Country-Level ECD Activities.........................42 Annex D: OED's Mandate and Support forECD ....................................................45 Annex E: Note on a Proposed Evaluation Methodology for Future Annual Reports on ECD (ARECs) .........................................................................47 Annex F: Proposed Survey of Bank Country Teams Concerning M&E Systems in PRSP Countries .............................................................. ...........................48 Annex G: Minutes of the Informal Meeting of the Subcommittee on Development Effectiveness .......................................................................49 Boxes: 1 O PCS Support for ECD ....................................................................................10 2 O ED Support for ECD......................................................................................1 3 Some Key M&E Training Courses...................................................................13 B. 1 Criteria for High-Quality ECD..........................................................................39 Figures: B.1 Planning, Budgeting and Management Cycle in Govemment..........................32 B.2 A Results Chain for ECD ..................................................................................40 Tables: B.1 List of Possible ECD Actions at the Country Level .........................................33 B.2 Country Example of a Summary M&E Diagnosis: Uganda- Strategic Issues, Challenges and Possible Actions ...........................................38 B.3 Suggested Performance Indicators for ECD.....................................................41 C. 1 Country-Level ECD Activities Underway with Bank Support.........................42 F. 1 Indicative List of Issues for an M&E Survey....................................................48 iii Acknowledgments This report was prepared by Keith Mackay under the overall guidance of Osvaldo Feinstein. The report benefited from the valuable inputs and comments of many colleagues throughout the Bank, including Misha Belkindas, Charles Chandler, Aline Coudouel, Graham Eele, Laurie Effron, Victoria Elliott, Nils Fostvedt, Patrick Grasso, John Heath, Gregory K. Ingram, Padma Karunaratne, Jody Zall Kusek, Ruben Lamdany, Patricia Laverley, Marlaine Lockheed, Rogerio Pinto, Oliver Rajakaruna, Ray Rist, Anwar M. Shah, Ai Chin Wee, and Elizabeth M. White. Director-General, Operations Evaluation: Mr. Robert Picciotto Director, Operations Evaluation Department: Mr. Gregory K. Ingram Manager, Partnership and Knowledge Programs: Mr. Osvaldo Feinstein Task Manager: Mr. Keith Mackay iv List of Acronyms AfrDB African Development Bank 6ECD Organization for Economic Co- AREC Annual Report on Evaluation operation and Development Capacity Development OED Operations Evaluation Department CAS Country Assistance Strategy OEDCM OED Corporate Evaluation & CDF Comprehensive Development Methods Group Framework OEDPK OED Partnership and Knowledge CFAA Country Financial Accountability Programs group Assessment OEG Operations Evaluation Group CODE Committee on Development OPS Operations Policy Staff Effectiveness OPCS Operations Policy & Country CPAR Country Procurement Assessment Services Review PAI Poverty Analysis Initiative CPIA Country Policy and Institutional PEFA Public Expenditure Management and Assessment Financial Accountability Program CWIQ Core welfare indicators questionnaire PER Public Expenditure Review DAC Development Assistance Committee PEW Program Evaluation Workshops of the OECD PIU Project Implementation Unit DEC Development Economics Vice- PPAR Project Performance Assessment Presidency Report DECDG DEC Data Group PREM Poverty Reduction and Economic DFID Department for International Management Development (U.K.) PRSP Poverty Reduction Strategy Paper DGO Director-General, OED QAG Quality Assurance Group ECD Evaluation Capacity Development RBM Results Based Management ECG Evaluation Cooperation Group SAL Structural Adjustment Loan ESW Economic and Sector Work SDC Swiss Development Agency for EU European Union Cooperation FMIS Financial Management Information SWAPs Sector-Wide Approaches Systems TAL Technical Assistance Loan HIPC Highly Indebted Poor Countries UN United Nations ICR Implementation Completion Report UNDP United Nations Development IDA International Development Program Association VPU Vice-Presidential Unit IDEAS International Development WBI World Bank Institute Evaluation Association WPAE Working Party on Aid Evaluation IDF Institutional Development Fund (DAC) IMF International Monetary Fund IPDET International Program for Development Evaluation Training IPE Introduction to Program Evaluation JSA Bank/Fund Joint Staff Assessment LSMS Living Standards Measurement Survey M&E Monitoring and evaluation MDG Millennium Development Goals NGO Non-Government Organization OCS Operational Core Services Network V Executive Summary 1. The Bank's strategy for evaluation capacity development (ECD) was established in 1994 to help borrowers strengthen their monitoring and evaluation (M&E) systems as an integral part of sound governance. ECD was conceived as an integral part of the Bank's country portfolio management work. It was also conceived as an instrument of public sector management reform. 2. Since 1994, the priority for ECD has been confirmed by the Comprehensive Development Framework (CDF), Highly Indebted Poor Countries (HIPC), Poverty Reduction Strategy Paper (PRSP) and other initiatives that emphasize development results. Tracking of progress towards the Millennium Development Goals also calls for ECD, as does the growing realization that the quality of M&E for Bank-financed programs and projects is inextricably linked with the capacities and systems of borrower countries. Accordingly, the World Bank has committed itself fully to a greater focus on the measuring, monitoring and managing of development results. 3. Bank-supported ECD activities have increased substantially in recent years: since 1999 they have started in 21 countries, including eight in the Africa region, five in Latin America & Caribbean, and four in the Europe & Central Asia region. In 2000, Bank management decided to start to mainstream ECD by FY04. 4. This first Annual Report on ECD (AREC) seeks to clarify the meaning and relevance of Bank work on ECD, to lay the methodological foundations for subsequent ARECs that will track the progress and draw evaluation lessons from the Bank's ECD work, and to present an inventory of ECD activities underway across the Bank, with special attention to country-level activities. 1 1. Introduction 1.1 The World Bank and others in the development community are placing a high priority on measuring, monitoring and managing results. One of the main areas of emphasis is helping countries to build their own monitoring and evaluation (M&E) systems-known as evaluation capacity development (ECD). The Board's Joint Audit Committee, the predecessor to CODE, approved an ECD strategy for the Bank in 1994. CODE has repeatedly stated its strong support for ECD, and has mandated OED with particular responsibilities in this area. Given this emphasis, OED has undertaken to provide CODE with annual reports on this topic, commencing in FY02. 1.2 This first Annual Report on ECD (AREC) has three objectives. First, to clarify the meaning and relevance of Bank work on ECD, and to lay the foundations for subsequent ARECs that will evaluate the Bank's work on ECD. Second, to provide a stocktaking of ECD activities underway across the Bank; thus providing a baseline for future evaluations of Bank work in this area. Third, to identify issues for CODE consideration, including options and constraints to mainstreaming ECD. The methodology undertaken in preparing this AREC is outlined in Annex A. 2. Monitoring, Evaluation, and ECD 2.1 Monitoring embodies the regular tracking of inputs, activities, outputs, reach, outcomes, and impacts of development activities-at the project, program, sector, and national levels. This includes the monitoring of a country's progress against the Millennium Development Goals or other national measures of development success. Evaluation is the systematic and objective assessment of the relevance, efficacy or efficiency of a development activity. (More formal definitions and a fuller discussion of ECD are contained in Annex B.) Evaluation Capacity Development is concerned with strengthening or building M&E systems in borrower countries in order that a situation is attained where M&E is regularly conducted and utilized by the countries themselve governments and civil society. In other words, M&E are tools, and ECD is the process of setting up a country-based system to conduct and use M&E. 2.2 ECD "success" is achieved when countries use M&E information systematically to: * Inform government resource allocation decisions-policy-making and planning via national budgets and poverty reduction strategies-by providing information on the likely benefits and costs of alternative policies, programs and projects. * Support the design and ongoing management of activities (at sector, program and project levels), including the delivery of government services and the management of staff-4his is the learning function of M&E. 2 Enhance transparency and support accountability relationships-accountability of government to parliament, to civil society and to donors, and to underpin accountability relationships within government (between sector ministries and central ministries, and between ministers, managers and staff). M&E provides a vehicle to magnify the voice of civil society and to put additional pressure on government to achieve higher levels of performance. 2.3 Utilization of M&E information depends on the existence of sufficiently strong incentives for decision-makers, managers and other government officials to view this information as important for their work, their organizations, and their country's development prospects. There is no easy route to strengthening a government's performance culture, and thus its demand for M&E, although internal and external accountability pressures can play an important part in this. 2.4 These considerations lead to the following definition of ECD: ECD is a systemic approach to the improvement of country institutions concerned with the monitoring, evaluation and reform ofpublic sector activities and programs. 2.5 ECD involves much more than helping countries undertake a collection of such M&E activities' as project-level M&E, financial tracking systems, M&E training, construction of performance indicators, or improved national statistical systems. ECD aims to build a sustainable system to piovide M&E information on a continuous basis in support of good governance and results-based management at the national, sectoral or local levels.2 Successful ECD results in relevant monitoring information that is collected; evaluation findings that are timely, objective, credible and economical, as well as information that is actually used to improve public sector management. 2.6 There are synergies between country M&E systems and the M&E conducted by the Bank and other donors. Stronger M&E capacities in borrower countries can make Bank M&E easier and cheaper, by relying more on existing country data systems, country evaluators-individuals, government evaluation offices and universities, for example-and eventually on country M&E systems for self-evaluation. Stronger country capacities for M&E can assist in supervision and the Implementation Completion Reports (ICRs) to ' M&E can be conducted using a wide array of tools, methods, and approaches. These include, for example: performance monitoring indicators; the logical framework; theory-based evaluation; formal surveys such as service delivery surveys, citizen report cards, living standards measurement surveys (LSMS) and core welfare indicators questionnaires (CWIQ); rapid appraisal methods such as key informant interviews, focus group discussions, and facilitated brainstorming by staff and officials; participatory M&E; public expenditure tracking surveys; formal impact evaluation; and cost-benefit and cost-effectiveness analysis. A sample of these is discussed in OED, Monitoring & Evaluation: Some Tools. Methods and Approaches, The World Bank, 2002. 2 OED, 1999 Annual Review of Development Effectiveness The World Bank, 2000, Annex 6, and A. Binnendijk, Results Based Management in the Development Co-Operation Agencies: A Review of Experience DAC Working Party on Aid Evaluation, June, 2001. 3 which the Bank's borrowers are expected to make substantive contributions. These synergies are discussed further in Section 3. How is ECD Undertaken? 2.7 In 1994, a Bank Task Force concluded that building a sustainable M&E system entails four mutually-reinforcing elements:3 * Develop and reinforce the demand for M&E within borrowing countries- recognizing that demand can be the main constraint, it can be strengthened by realistic presentations of the potentials and constraints of M&E, and by ensuring that ECD is among the strategic issues that should feature in the Bank's dialogues with country leaders on the long-term development agenda. * Integrate ECD into country portfolio management--by focusing evaluation on priority issues and thus demonstrate its utility to country and Bank decision-making. * Incorporate ECD into programs for public sector management reform-recognizing country circumstances. * Strengthen countries' supporting infrastructure for ECD-such as data systems, financial management and audit systems, and universities. 2.8 The focus, speed, and sequencing of ECD need to be tailored according to country circumstances, and thus the diagnostic phase is crucial. A diagnosis of M&E would be expected to map out a number of key issues. These include: the nature of existing demand for M&E; current M&E activities and functions within government; actual and potential supply of M&E capacities in the country; extent of M&E utilization; and the potential synergies between M&E and other public sector reforms. Where there exists substantive country demand for M&E, the next step is to support government in mapping out an action plan. Likely components of an ECD action plan are described in Annex B. 3. The Bank's Evolving ECD Strategy 1993-1994 Bank Task Force on ECD 3.1 The Bank-wide Task Force on ECD was set up in 1993 to determine the priority for ECD in public sector management, review past experience of ECD, assess Bank efforts in relation to ECD, determine possible country strategies for successful ECD, and define generic Bank strategies (subject to Bank country team adaptation) for supporting ECD. 3 These are contained in the ECD strategy endorsed by the Bank's Board and management in 1994. See Evaluation Capacity Development: Report of the Task Force July 5, 1994. 4 Prior to this, OED had assisted countries seeking to improve their capacity to undertake and use evaluations with ad hoc assistance in response to borrowers' requests. 3.2 The Task Force report stressed the important role of the feedback provided by M&E for improved government performance and enhanced public sector management. This finding led to a key recommendation of the Task Force: that assistance to ECD should no longer be regarded as an isolated initiative, but rather form an integral part of the Bank's country portfolio management work and its support for public sector management reform. The long-term objectives were to mainstream ECD within countries and to strengthen M&E in the Bank.4 The Task Force report was endorsed by management and the Joint Audit Committee in July 1994. OED's 1997 renewal program, also endorsed by CODE, 5 confirmed an ongoing catalytic role for OED in promoting and supporting ECD efforts under the aegis of the official DGO/OED mandate.6 OED discharges its mandate by prompting and supporting substantive involvement of regional VPUs in ECD and assisting other central units such as OPCS, WBI and DEC, in the provision of needed support. OED's approach to ECD reflects principles of subsidiarity and complementarity. It is discussed further in Annex D. 3.3 In the years following the agreement of the Bank's ECD strategy in 1994 there have been a number of important developments which have intensified the priority for ECD. There have also been continuing deficiencies with the Bank's approach to M&E and ECD, and mixed evidence regarding the attention that Bank country teams give to ECD. These concerns led to the creation of Bank-wide working groups on M&E in 1999 and 2002 to attempt to address these problems. Bank's Governance Strategy 3.4 As the 1990s progressed, the Bank placed increasing emphasis on sound governance and public sector management, recognizing the close links between these areas and development effectiveness. The Bank's public sector reform and governance strategy-which guides the Bank's public sector management work in borrower countries-was agreed in November 2000. The strategy notes the importance of data on service delivery, of citizen surveys and other mechanisms to promote citizen feedback. It highlights the priority for helping countries build their M&E capacities: A focus on monitoring and evaluation of results and on transparent reporting of those results is always called for, even if not accompanied by full-fledged managerial flexibility. 4 Evaluation Capacity Development. Report of the Task Force, July 5, 1994, pp. iv and viii. 5 Director-General OED, Evaluation in the World Bank: Challenges and Priorities CODE97-15, 27 February 1997. 6 The formal mandate for the DGO/OED includes direct responsibility for "encouraging and assisting developing member countries to build effective monitoring and evaluation associations, capacities and systems". World Bank, Procedure for the Selection of Director-General of the Operations Evaluation Department Annex A, Mandate of the Director-General, Operations Evaluation. R2001-0169, 10 September 2001. For example, The World Bank, World Development Report 1997: The State in a Changing World New York, Oxford University Press, 1997. 5 A major thrust of Bank work should be to help countries improve their ability to produce, disseminate, and evaluate data on the quality and cost of public services, as has been pioneered in Australia and other advanced government reformers.8 [italics in original] 3.5 Governance issues where ECD is relevant include, for example, programmatic lending and sector-wide approaches (which require reliance on national or sectoral M&E systems), civil society "voice", and government decentralization and forms of "new public management" whereby increased autonomy for spending decisions is balanced by increased accountability for results. CDF and PRSP Initiatives 3.6 The need for enhanced M&E has been confirmed by the Comprehensive Development Framework (CDF) initiative in 1999,9 by the results-orientation of Poverty Reduction Strategy Papers (PRSPs) and by the linkage between PRSPs and the Highly Indebted Poor Countries (HIPC) debt relief initiative. The CDF stresses a holistic, country- focused, results-oriented approach to development, including an emphasis on good government.10 It also emphasizes the role of civil society in assessments of country performance. The CDF principles are embedded in PRSPs-over 40 countries are covered by the enhanced HIPC initiative, of which 34 are in Sub-Saharan Africa; 51 countries are preparing a PRSP. PRSPs are linked to the Millennium Development Goals (MDGs), which comprise measurable indicators and results." With both the CDF and the PRSPs, the strong emphasis on results provides an environment where M&E has a potentially central role to play. 3.7 The HIPC initiative will reduce the total debt servicing burden of borrowers by some $50 billion, with a cost to donors of some $29 billion in present value terms. This cost has heightened concerns that these resources will be devoted to poverty-reduction and that they will be effective. The Bank and IMF, and also HIPC countries, are devoting attention to the adequacy of country financial tracking systems, both for HIPC funds and, owing to fungibility, to ensure a corresponding increase in spending devoted to poverty- reduction. However, recent Bank/Fund studies have found that most HIPC countries cannot yet track poverty expenditures adequately;12 OED has noted that this is a particularly risky situation for the Bank, as one of the primary sponsors of the HIPC 8 Public Sector Group, Reforming Public Institutions and Strenptheninp Governance: A World Bank Strategy, The World Bank, Washington D.C., 2000, p. 37. 9 James D. Wolfensohn, A Proposal for a Comprehensive Development Framework. The World Bank, 21 January, 1999. '0 OED, 1999 Annual Review of Development Effectiveness. The World Bank, 2000. CDF Secretariat, Comprehensive Development Framework, Questions and Answers- Update, The World Bank, 28 March, 2001. " The Millennium Development Goals were unanimously adopted by member states of the United Nations in September 2000. 12 The most recent Bank/Fund study is Actions to Strengthen the Tracking of Poverty-Reducing Public Spending in Heavily Indebted Poor Countries (HIPC). IDA/SecM2002-0030, 30 January 2002. It found that none of the 24 HIPC countries reviewed was able to meet more than 9 of the 15 benchmark indicators judged necessary for tracking poverty-reducing public spending. 6 intiative.13 The limited manner in which a number of PRSP countries (and Bank country teams) are interpreting M&E and the concept of an M&E system, and the implications for the Bank's ECD work, are considered further in Section 5. The Bank's Approach to M&E and ECD 3.8 OED has reported on numerous occasions that the Bank's overall progress in implementing the Bank's ECD strategy agreed in 1994 has been inadequate.14 It has also documented lack of compliance with the operational directive on monitoring and evaluation of Bank projects in two OED studies,'5 and since then both OED and the Quality Assurance Group (QAG) have emphasized these deficiencies on a number of occasions.16 In particular, OED has recently found that weak M&E is a contributing factor to poor project erformance in 38 per cent of Bank projects rated unsatisfactory or highly unsatisfactory. 3.9 These concerns led management to set up a new Bank-wide working group on M&E in 1999, with broad-based representation from regions, networks, OCS, OED, OPS, QAG and WBI. The working group concluded that a set of weaknesses in M&E were common for both the Bank and borrower countries and were related:18 * Poor incentives to conduct good M&E. * Diffused accountability because of unclear roles and responsibilities both within the Bank, and between the Bank and borrowers. * Weak capacity for M&E both in the Bank and in client countries. 3.10 The working group proposed a two-pronged approach to address these weaknesses systematically both in countries and in the Bank. This comprised: (i) implementing a phased "piloting" approach to provide a better understanding of how the issues of incentives, roles and responsibilities, and capacity can be addressed in different sectoral and country contexts; and (ii) building regionally-based capacity through staff training and knowledge management activities, including consolidation of knowledge and dissemination of the Bank's own experience. 13 OED, 2000-2001 Annual Report on Operations Evaluation. January 2002, p. 18. " OED Annual Report on Operations Evaluation, FY96 (pp. 14-18), FY97 (pp. 10-11), 1998 (p. 26) and 1999 (p. 23). 's OED, An Overview of Monitoring and Evaluation in the World Bank, Report No. 13247, June 1994, and OED, Monitoring and Evaluation Plans in Staff Appraisal Reports in FY95, Report No. 15222, December 1995. 16 For example, the third Rapid Supervision Assessment (RSA-3) found that less than half of Bank projects had an appropriate M&E system in place. See QAG, Annual Report on Portfolio Performance Fiscal Year 1999 (p16). See also OED, 2002 Annual Report on Operations Evaluation, June 2002. " This estimate is based on a sample of 45 Bank projects with unsatisfactory or highly unsatisfactory outcomes. OED, 2001 Annual Review of Development Effectiveness Report No. 23342, 20 December 2001, rara 3.28. Report of the Working Group on Improving Quality of Monitoring and Evaluation in Bank-Financed Operations (CODE2000-72), June 29, 2000. 7 3.11 The working group recognized the close interdependence between the quality of the Bank's project, sector and country-focused M&E, including for the Bank's country assistance strategies (CAS), and the ability of borrowers to conduct and utilize M&E.19 The working group's recommendations included a focus on ECD, to be pursued via diagnosis and support for 8 pilot countries and 2 sectors with a view to mainstreaming ECD in Bank-supported operations starting in FY04. At a meeting of CODE in 2000, OED expressed the view that a pilot approach would not be sufficient, and CODE evinced similar concerns but eventually agreed to the recommendations put forward by management.20 In this context CODE noted that "the cost of not having good M&E systems could be very significant for the borrower resulting in expensive restructuring of projects or even cancellation of projects." Accountability for Development Spending 3.12 There are now growing pressures on the Bank and other donors to demonstrate the results of development spending. The Bank has committed strongly to the provision of clear evidence of the results of its work, 21 and intends to achieve this by, among other things, a more focused attention to M&E--both within the Bank and via the support of client countries in strengthening their own evaluation capacities.22 At the March 2002 Monterrey conference on financing for development, the heads of the five multilateral development banks issued a joint statement emphasizing this priority.23 They noted that: The quality of country systems for measuring and monitoring results is important for the challenge before us. This puts a premium on our capacity building support for public sector management, statistical development, and monitoring and evaluation systems, which are important in their own right for underpinning countries' accountabilities to their people and results agreements and compacts with donors. Our agencies have programs of support for evaluation and statistical capacity building, aiming to help countries improve their measurement, monitoring and management for results-whether at the project level or at the agency or ministry level. 3.13 A high-level Task Force was created within the Bank in early 2002, headed by a Bank Vice-President. Its mandate is to develop an action plan for improving the measuring, monitoring, and managing of development results; this will include a heavy focus on the MDGs. 19 OED has rated borrowers' provision of M&E data as substantial or high in only 36 per cent of operations exiting in FY01. See OED, 2002 Annual Report on Operations Evaluation (Box 4.2). Stronger country capacities for M&E would also raise the strategic option for operations in the medium to long-term of relying on borrowers to self-evaluate donor loans, subject to quality assurance issues. 20 Report from CODE on Report of the Working GrouR on Improving Quality of Monitoring and Evaluation in Bank-Financed Operations, CODE2000-96, 13 October 2000, p. 2. 21 See, for example, The World Bank, The Role and Effectiveness of Development Assistance: Lessons from World Bank Experience March 2002. 2 "Implementing Our Strategy, 2002," paper prepared for World Bank Strategic Forum, January 24-25, 2002. 23 Statement issued in Monterrey by the heads of the five multilateral development banks, On Measuring, Monitoring and Managing for Development Results 20 March 2002. 8 4. Stocktaking of ECD Activities 4.1 A number of Bank units are currently involved in ECD and related activities. They include OPCS, which is providing catalytic support to Operations, a number of Bank country teams in Africa and other regions, WBI in its provision of M&E and PRSP training, DEC and Regional VPUs working on statistical capacity-building, and OED in its long-standing provision of support for ECD. There also exist ECD partnerships between the Bank and other multilateral and bilateral donors. The objective of the stocktaking was to compile a list of all Bank-supported, country-level ECD activities currently underway-note that this stocktaking is closely based on the definition of ECD provided in Section 2. The one area where comprehensive information has not been sought, because of its diffuse nature and uncertain relationship with country-level ECD, is M&E capacity- building at the sector level-this work often stresses the production of sector ministry data. A detailed description of the methodology used to undertake this stocktaking is described in Annex A, and the detailed list of ECD activities is provided in Annex C. ECD Activities Underway with Bank Support 4.2 ECD activities have started with Bank support in at least 21 countries, including eight in the Africa region, five in Latin America & the Caribbean, four in Europe & Central Asia, two in South Asia, and one each in East Asia & the Pacific, and the Middle East & North Africa regions. The countries include: Albania, Argentina, Bangladesh, Bolivia, Brazil, Chile, Egypt, Ethiopia, Ghana, Honduras, India (Andhra Pradesh), Kyrgyz Republic, Madagascar, Malawi, Mozambique, Niger, the Philippines, Poland, Romania, Tanzania and Uganda (details are provided in Annex C, Table C.1). Most of these countries are at an early stage of ECD work, although some (such as Brazil, Chile, the Philippines and Uganda) have been working to strengthen their M&E capacities for a longer period. 4.3 There is no standard approach to ECD in these countries. ECD has to be tailored to the circumstances, opportunities and demands for M&E in each country. Nevertheless, patterns can be observed in these 21 countries. 12 of them are PRSP countries. In only 6 of these is the ECD work closely linked in support of the PRSP process (Albania, Bangladesh, Ghana, Madagascar, Niger and Uganda). Two of the countries are European Union (EU) accession countries (Poland and Romania)-EU member states are required to meet certain minimum standards for M&E in government. In ten countries, M&E is being strengthened in support of the budget process, via better financial tracking and/or to assist government decentralization. 4.4 An M&E diagnosis has been completed for eight of the countries, and is in preparation for two others. ECD evidently is still not a priority in the Bank's public sector reform loans. It appears that only five Bank loans supporting ECD have been agreed since 1999, with loans in preparation for an additional six countries. While the Bank's public sector reform and governance strategy, agreed in November 2000, emphasizes the priority for helping countries build their M&E capacities, the strategy does not count ECD among its list of priority action areas on which the Bank should focus. 9 Africa Region 4.5 Of all the Bank's Regional VPUs, Africa is the most actively involved in country- level ECD activities-almost half of the countries listed in Table C.1 (8 out of 21) are in Africa-and there are at least two reasons for this. First is the Bank's emphasis on Africa and its capacity-building imperatives.24 Second is the concentration of HIPC countries in Africa, with the expectation that each will prepare a PRSP with an emphasis on results. The Regional VPU undertook a survey of Bank staff in 2001 to ascertain their M&E skills, including in the area of ECD, and their training needs; a high level of demand for M&E training has been identified (see below and Section 5).25 The Regional VPU is currently preparing an M&E Strategy to guide its future work in Africa which is expected to place considerable emphasis on ECD for most if not all African countries. The strategy is intended to increase the overall level of effort and resources devoted to M&E in Africa, and also closer coordination and sharing of good-practice ECD efforts currently underway. The strategy also includes the intention to conduct a rapid survey of country teams concerning the demand for, and supply of, M&E in the countries with which they work, and also the identification of ECD activities underway. (The option of conducting a similar survey of all Bank teams is discussed in Section 5, and a possible survey is outlined in Annex F). Latin America 4.6 Of the 21 countries listed in Table C.1 five are in Latin America-Argentina, Bolivia, Brazil, Chile and Honduras. Several countries-such as Argentina, Chile and Colombia ave devoted some effort to ECD in the past, with some assistance from the Bank and other donors. The Bank's Regional VPU for Latin America and the Caribbean is preparing a regional M&E strategy, and this is likely to lead to greater regional efforts on ECD. A high level of demand for M&E training was also revealed by a survey of Bank staff in this Regional VPU, conducted in March 2002. Other Regions 4.7 Between the late 1980s and 1999, the Bank supported early-stage ECD efforts in a number of countries in other regions--countries such as China, Indonesia, Morocco, Pakistan, Sri Lanka and Thailand. Countries in these other regions whose ECD activities are currently being supported by the Bank currently account for eight of the 21 countries identified; half of these are in the Europe & Central Asia region-Albania, Kyrgyz Republic, Poland and Romania. 4.8 Among the Bank networks, the Human Development Network appears to be the only one with a substantive focus on ECD. It is currently negotiating an ECD partnership agreement with DFID, with a focus on ECD in the health sector in a number of pilot countries. Other donors would be invited to participate in this ECD partnership. 24 See, for example, IBRD, A Proposed Partnership for Canacity Building in Africa (PACT): An Initiative ot the Afncan Governors of the World Bank, R99-69, 27 April 1999, and The World Bank, Strategic Partnership with Africa-SPA 5: 2000-2002. Towards New Aid Relationships to Reduce Poverty, undated. 25 The survey results are available at http://afr/aftqk/meskills/about.cfm 10 Central Support to Country-Level ECD Operations Policy & Country Services (OPCS) 4.9 OPCS is responsible for implementation of the M&E Improvement Program decided by management and agreed by CODE in September 2000 (see Section 2), and it is playing a catalytic role in support of ECD-see Box 1. The two objectives of the Program are to increase the use of M&E by Bank staff and borrowers in support of better management decision-making, and to provide lessons from a number of pilot countries concerning how M&E can be successfully conducted and used in Bank operations and by borrowers. Box 1: OPCS Support for ECD OPCS is supporting the Bank-wide strategy for improving M&E (endorsed by CODE in 2000) which is comprised of two inter-related prongs: building the Bank's capacity in results-based M&E and determining how best the Bank can support borrowers in strengthening their M&E systems. For the second prong of this strategy, eight countries were selected for initial concentration. The purpose of the initial eight countries is to draw lessons learned on (1) key issues in countries which could drive an increased focus on results, and (2) how the Bank's current instruments could be used as entry points and capacity-building avenues in countries. Together, these would form the basis for determining how the Bank could scale up its analytical work in ECD and approach broader capacity-building in client countries, with a view to a regionally phased launch beginning in FY04. So far, analytic missions have taken place in the following countries, with support by the country teams, internal and external partners and the regions: Argentina, Bangladesh, Egypt, Kyrgyz Republic, Madagascar, Philippines, Romania, Sri Lanka and Tanzania. Early lessons are being used to help inform regional strategies and determine entry points for additional capacity-building work. (This material was provided by OPCS.) 4.10 When the M&E Improvement Program was created it was intended, among other things, to pursue ECD in eight pilot countries by FY02, and in two sectors. By March 2002, seven pilot countries had been identified: Bangladesh, Egypt, Kyrgyz Republic, Madagascar, Philippines, Romania, and Tanzania; the ECD work underway in these seven countries is summarized in Table C.l. OPCS currently provides a wide range of support for Bank country teams and borrowers involved in these pilots, including provision of M&E training, diagnostic missions to the countries-an important part of this work is raising awareness and strengthening demand-provision of operational advice on ECD in lending and non-lending activities, and collection of Bank and other resource materials on ECD via a website.26 OPCS' role to promote and support Bank-focused M&E also provides synergies with the country-focused ECD work, by strengthening awareness of the 26 http://opcs/ME/index.html 11 uses of M&E and by identifying ECD opportunities in support of country and Bank work to strengthen public sector management. Operations Evaluation Department (OED) 4.11 OED also promotes and supports efforts by governments and Bank country teams to strengthen their M&E capacities, with key objectives being to mainstream ECD in Bank operations, and to ensure that the ECD is of high quality. OED's comparative advantage lies in leveraging its experience and expertise to help build good-practice, demonstration pilot countries. This support has therefore targeted reform-oriented countries such as Uganda where there is a demonstrated demand for M&E-and for ECD as the means to strengthen M&E. OED is helping a number of other Bank country teams, and also OPCS, which are working with governments to strengthen their M&E systems-Albania, Bangladesh, Brazil, Egypt, Ghana, Guinea, Honduras, India (Andhra Pradesh), Kyrgyz Republic, Madagascar, Mozambique, Niger, Poland, Sri Lanka and Tanzania. Information on OED's support for ECD is presented in Box 2. The principles which guide OED's support for ECD and its mandate from CODE are discussed in Annex D. Box 2: OED Support for ECD OED has helped organize a number of regional conferences and seminars to build awareness and strengthen country demand for M&E and ECD, such as one held for PRSP countries in The Hague in March 2001. In support of Bank and country work, OED has prepared ECD resource material27 such as country case studies and diagnostic guides, and provides M&E training courses for Bank staff, staff of other donors, as well as officials and others from borrowing countries. OED also works in partnership with a number of donors, including the governments of the Netherlands, Norway and Switzerland, the Evaluation Cooperation Group (ECG) of the multilateral donors, and with UN agencies. Collaborative ECD efforts with these donors include, for example, the diagnosis of M&E capacities and systems, organization of regional conferences, identification of regional centers of evaluation excellence in developing countries-existing institutions, such as universities-which have the potential to play a regional role, and the support, jointly with the UNDP's Evaluation Office, for the establishment of an International Development Evaluation Association (IDEAS) open to membership of individuals and institutions from the South and North, in order to promote development evaluation with a particular emphasis on ECD. M&E Training 4.12 There is a large and rapidly growing demand for training in M&E by Bank staff and borrower countries, and this has led to a multiplicity of course offerings.28 Training in M&E for staff or clients is provided by the regions, networks such as PREM, and central 27 Available at OED's ECD website: http://www.worldbank.orelevaluation/me/ 28 The high level of interest in M&E among Bank staff is also reflected in the high level of membership of the two existing M&E thematic groups-4hese currently include about 600 Bank staff (see Section 5). A high level of demand for M&E training, including training in ECD diagnostic work, was revealed by the survey of all staff in the Bank's Africa region in January 2001 (and by a smaller survey of staff in the Bank's Latin America & Caribbean region in March 2002). 12 units such as OPCS, WBI, DEC and OED. A recent inventory of Bank training in this broad area identified a large number of courses-about 70-currently provided or planned addressing various M&E tools and techniques, as well as related areas including statistics, research and analysis.29 Both country-focused M&E training and Bank staff M&E training are relevant to country-based ECD work. 4.13 Of the 70 courses, about 19 are M&E courses open to officials and others from borrower countries. Reflecting the various audience types, some of the courses are of an awareness-raising nature, while others provide basic or advanced methodological skills. This mixture of M&E tools, audiences and level of sophistication has so far not been coordinated among the various Bank units providing the training-both in the case of country-oriented M&E training, and Bank staff M&E training-with significant potential for both overlap and gaps. OPCS has been tasked by the Learning Board to coordinate training in M&E; a group comprising WBI, OPCS and OED is currently reviewing the content and quality of these diverse courses to identify gaps and duplication. 4.14 OED and OPCS have developed a course on Designing and Building Results-based M&E Systems, designed for clients and Bank staff, and delivered to some 800 government officials and 150 Bank staff. The course has been translated into French, Spanish, Russian, Arabic and Albanian. 4.15 Most of the courses provided to participants from borrower countries are "retail" in nature-i.e., direct provision of training to officials-rather than trainer-training, which can be viewed as "wholesale" in nature.31 Some of these courses are provided to senior officials for the purpose of awareness-raising of the uses of M&E and thus the need for ECD. Most of the courses involve the transfer of more advanced skills; prima facie, wholesale training in this area appears more likely than retail training to lead to sustainable capacity-building in borrower countries. World Bank Institute (WBI) 4.16 WBI provides a number of training courses in M&E, often jointly with other Bank units-hree core M&E courses are described in Box 3. A major WBI initiative is the Attacking Poverty Program of support for countries preparing PRSPs. The governments of these countries were asked to rank ten possible areas in which their officials could receive training--thus training priorities were identified according to country demand for them. One of the priority areas, the Poverty Analysis Initiative (PAI), is focused on strengthening the capacities of officials to undertake poverty analysis, poverty monitoring and poverty 29 A precise count of the number of courses is not readily available; however, many of these are double- counted because the same course targets different audiences, including Bank team leaders and staff, senior officials of borrower governments, and staff of other development agencies, and thus they are essentially repeat offerings of the same course. No estimates of the number of participants in this training, the number of training days, hours of instruction, or costs, are currently available. 30 See, for example, WBI, Developing Evaluation Capacity WBI Evaluation Brief, March 2002. S'Bank loans which contain an ECD component are another source of supply of M&E training and trainer- training for country officials and others. 13 impact evaluation. Other relevant WBI training involves areas such as good governance, anti-corruption and investigative journalism. Box 3: Some Key M&E Training Courses Three core M&E training courses provided to officials and others from client countries include: Introduction to Program Evaluation (IPE); Program Evaluation Workshops (PEWs); and the International Program for Development Evaluation Training (IPDET).(a) These courses have relied on substantial financial support from bilateral donors, including Canada, France, Japan, Netherlands, Norway and Switzerland, and the World Bank. IPE: This course is delivered by video-conference and provides a broad survey of evaluation methods, focusing on basic M&E concepts and skills. The target audience is practitioners who are involved in planning and implementing project and program evaluations. Participants are provided hands-on experience in evaluation design. 1,000 persons have been trained in 34 countries since 1998-Bank staff are included, to achieve a mutual learning environment with client participants. PEWs: Since 1998, four regional, one-week workshops have been delivered to 140 participants from 23 countries. The target audiences include M&E trainers, management teams from training institutions, and, more recently, government PRSP teams to enable them to monitor and evaluate PRSP implementation. IPDET: This two to four-week residential course was initiated in 2001 in partnership with Carleton University, Ottawa. IPDET provides intensive skills- building in core and elective units in development evaluation. About 130 participants from 23 developing and 17 developed countries were trained in 2001. Participant fees were charged, and scholarships were provided for 30 participants. IPE, PEWs and IPDET were designed and implemented by Bank staff and consultants associated with OED, OEG, and WBI. Present responsibility for IPDET lies with OED, and for IPE and PEWs with WBI. (a) WBI, Developing Evaluation Capacity, WBI Evaluation Brief, March 2002. Statistical Capacity-Building, and the Development Economics Group (DEC) 4.17 There is no neat demarcation line between ECD and related activities such as statistical capacity-building. From one perspective, statistical capacity-building is an upstream activity to ECD, focusing on the collection, processing, storage and dissemination of data by national statistical offices and sector ministries. But this work also partially corresponds to ECD because ECD encompasses the strengthening of ministry monitoring data and the measurement of client satisfaction via tools such as service delivery surveys. A large number of countries and Bank country teams- especially for PRSP countries-are devoting considerable efforts to improving national statistical systems (see Section 5). As discussed in Annex B, such work in and of itself is not classified in this report as ECD, which involves a more integrated and systemic approach to the conduct and utilization of M&E. However, for PRSP countries where a systemic approach to ECD is being undertaken, such statistical capacity-building work can be 14 expected to be an important component of their ECD activities. And for PRSP countries not yet taking a systemic approach to ECD, their statistical capacity-building efforts provide a valuable basis for ECD. 4.18 The approach of DEC's Development Data Group (DECDG) to statistical capacity- building has a number of good-practice features which are relevant to the Bank's ECD work. There is a substantive program of technical assistance to countries and Bank country teams with regard to the strengthening of national statistical systems, and there is a strong emphasis on countries which are preparing poverty reduction strategies. DEC's Technical Assistance in Statistics team comprises 14 staff, providing project support to all regions. DEC has close partnerships with other Bank regions, such as the Africa region, WBI, and with the IMF, UNDP, the PARIS 21 consortium (which comprises a number of multilateral and bilateral donors, other international organizations such as the OECD, and representatives from developing countries). DEC is also involved in the MECOVI statistical capacity-building for Latin America and the Caribbean. DEC manages a special trust fund for statistical capacity-building; around $4m per annum is available under this trust fund. Identification of Bank ECD Activities 4.19 A difficulty encountered in undertaking the stocktaking for this AREC was the absence of a formal M&E or ECD contact person within each region and network. Word of mouth knowledge concerning ECD activities underway in individual countries proved to be the most productive source of information. Similar problems in identifying Bank work in the area of statistical capacity-building led the DEC Data Group to undertake a manual search of every active Bank project document to compile a database containing information on all Bank projects which include statistical capacity-building efforts.32 This search involved about three person-months of work by a research assistant. 4.20 The task of monitoring Bank projects to identify those with a substantive ECD component would ideally be undertaken by an M&E or ECD contact person or unit within each region and network. One option for the Bank to consider is the establishment of a network of persons with this responsibility. Depending on the priority which CODE and Bank management attach to ECD, there could also be merit in some central Bank unit undertaking a similar manual search of Bank project documents and non-lending work (e.g., IDF grants) prepared since 2000, constructing and maintaining a database of Bank ECD work, and preparing quarterly updates based on new Bank documents in the future. Regions might thus be encouraged to maintain their own database of ECD work. 32 The DEC Data Group's website contains this database, as well as other resource material: http://sima.worldbank.or/data/TAS/index.htm 15 5. Issues, Opportunities, And Challenges 5.1 The discussion in Section 3 noted the imperative for the Bank and borrower countries to be able both to measure and to demonstrate results and development effectiveness. The CDF, HIPC and PRSP initiatives emphasize results, and there are important accountability and fiduciary concerns which are being addressed by Bank country teams and by borrowers. A recent Bank/Fund review of the PRSP approach noted that each PRSP is expected to include analysis of the causes of and trends in poverty, quantitative targets for poverty reduction, short-term indicators for monitoring progress toward those targets, and a description of the institutional arrangements for monitoring and evaluation, which may include participatory monitoring.33 5.2 PRSP countries benefiting from the HIPC initiative illustrate the types of effort being made to strengthen country capacities for M&E. Most PRSPs flag the importance of M&E, and a number of PRSPs (and Bank/Fund Joint Staff Assessments-JSAs) also sketch some actions to strengthen the information base on which Poverty Reduction Strategies are based. A limitation of these documents and their capacity-building plans arises from the manner in which "M&E" and "M&E systems" are defined, which essentially focus on the monitoring of spending-financial tracking-and of national development indicators (usually incorporating the MDGs). These two issues are surely of considerable importance and priority, but to focus solely on them has several drawbacks. First, there is no close relationship between government spending and poverty-reduction. Second, an emphasis on financial tracking and the MDGs fails to take account of intermediate processes, outputs and outcomes; two studies commissioned by DFID and reviewing the content of PRSPs have termed this deficiency the "missing middle".34 Yet it is precisely at these levels that the principal understanding of development effectiveness the efficiencl and efficacy of what works, what does not, why, and in what contexts-can be obtained. s These are also the levels at which leading indicators may be identified to help in the management of development activities.36 IDA and IMF, Review of the Poverty Reduction Strategy Paper (PRSP) Approach: Early Experience with Interim PRSPs and Full PRSPs. SecM2002-0086, 15 February 2002, para 51. 3 D. Booth and H. Lucas, Desk Study of Good Practice in the Development of PRSP Indicators and Monitoring Systems: Initial Review of PRSP Documentation, (May 2001) and Final Report, (November 2001), Overseas Development Institute reports commissioned by DFID. 3s There is ample experience as well as empirical evidence that the achievement of the MDGs requires developing countries to "overcome obstacles in their basic service delivery chains all the way to the end beneficiary". See, for example, D. Filmer, J. S. Hammer and L. H. Pritchett, "Weak links in the chain: a diagnosis of health policy in poor countries", in The World Bank Research Observer. 15 (2), August 2000, pp. 199-224. Weaknesses may be due to low efficiency in the transfer of funds to frontline service providers, low efficiency of service providers, waste and corruption. 36 Leading indicators relate specific interventions or actions, such as inoculations against infectious diseases, with subsequent outcomes-in this example, improved health status. The causal links between specific interventions and subsequent outcomes have been well demonstrated in the health sector in particular (see, for example, D. T. Jamison, W. H. Mosley, A. R. Measham and J. L. Bobadilla (eds.), Disease Control Priorities in Developini Countries. Oxford University Press, 1993), although inefficient service delivery can frustrate achievement of improved health status. However, for other sectors-such as education or 16 5.3 These are arguments for PRSP countries to emphasize intermediate monitoring indicators for their activities,37 focusing not only on spending and other inputs but also on processes, outputs, "reach" and outcomes. Such data are usually collected by sector ministries, relying on existing administrative systems; feedback from Bank sector staff contacted for the ECD stocktaking (see Annex A), and ECD experience from a number of the countries listed in Section 4, suggests that the data currently collected by such systems often tends to be of poor quality and heavily underutilized. 5.4 A third limitation with focusing on monitoring indicators (including the MDGs) is that while they provide useful information concerning performance, and can be used to flag the existence of problems which merit further investigation, they do not of themselves answer the causal questions concerning the reasons for poor (or good) performance: for these why questions, evaluation is needed. The governments of some PRSP countries, such as Uganda, understand this distinction well, and have in place plans to strengthen their evaluation capacities and systems so these types of question can be better answered in future.39 But such countries are the exception. Heightened Priority for ECD 5.5 This changing environment provides a heightened priority for ECD, and this can be considered in terms of both the intensity and extensiveness with which the Bank pursues and supports ECD. 5.6 It is evident from Section 4 that the majority of the Bank's ECD efforts are focused on PRSP countries, particularly in Africa. For PRSP countries, there is already a considerable amount of work being done on "M&E", although as argued above, this would usefully be intensified by greater focus on intermediate monitoring indicators, leading indicators, and on evaluations to support the ongoing management of development activities and to answer the more detailed, causal questions about development effectiveness. Their generally weak M&E capacities must temper expectations of what ECD efforts can be undertaken in these countries, at least in the short term. Yet their roads-the causal links between specific interventions and development improvements are far less clear-cut; this reduces the reliability of leading indicators in these sectors. 3 This is also recommended as good practice in IDA and IMF, Review of the Poverty Reduction Strategy Paper (PRSP) Approach: Early Experience with Interim PRSPs and Full PRSPs, 26 March 2002, p.40. * As Booth and Lucas (May 2001, op. cit.) have noted, "An outcome-oriented approach to monitoring does not imply a particular focus on outcome monitoring". Outcome (including MDG) indicators change slowly, the data are usually obtained late (because of the infrequency of censuses and surveys and because of delays in processing the data), and they say nothing about causality-i.e., the attribution of any trends or patterns to specific policy measures or programs. PRSPs are to be reviewed annually, so attention will need to be focused on variables which change or can be changed relatively quickly; most poverty indicators change only slowly; and the outcomes/impacts from specific development interventions only become evident over time, with often considerable delay. ' Government of Uganda, Poverty Monitoring and Evaluation Strategy, Kampala, January 2002. 17 growing demand for M&E information provides a unique opportunity to deepen their approach to M&E. 5.7 Bank programs in non-PRSP countries are devoting considerably less effort to ECD than the poorest countries,41 although there are some notable exceptions, such as Brazil,42 Chile, and Poland. Non-PRSP countries, and especially middle-income countries, possess stronger human resource capacities and face much less stringent resource constraints on their M&E activities-these supply-side conditions can lead to ECD results in a shorter time frame. The MDGs are equally applicable to these countries, and the governance benefits from conducting and utilizing M&E are just as great for them as for the PRSP countries. This suggests an opportunity for the Bank-and especially Bank regional VPUs which have so far devoted relatively little effort to ECD-to pursue a more extensive approach to ECD among countries. Some Issues for Mainstreaming ECD 5.8 The Bank's M&E working group in 2000 noted that weaknesses in Bank and country M&E were related and were attributable to: poor incentives for M&E; unclear roles and responsibilities; and weak capacities-especially skills-for M&E. The working group argued that "major and rapid improvements in M&E are vital if the Bank is to meet its commitments to better quality and effectiveness.',3 At a CODE meeting to consider the working group's report in September 2000, OED expressed the view that a pilot approach would not be sufficient and CODE indicated similar concerns. CODE eventually agreed to the recommendations put forward by management." Since then, the priority for mainstreaming country-focused M&E and ECD has heightened considerably. 5.9 The Bank has accumulated considerable experience with ECD over the past 15 years, and the pilots currently underway have added to that pool of knowledge. Earlier findings are being reinforced, such as: the importance of strong country ownership of ECD; the need to embed ECD within broader public sector reform efforts, and the critical role of incentives to conduct and utilize M&E findings; the role of ECD champions within 40 Depending on country circumstances and the strength of demand, the types of activity which PRSP countries could consider in their M&E plans include: strengthened national and sectoral statistical collections-with a particular focus on development indicators; enhancements to financial management information systems (FMIS)--to support better financial tracking; public expenditure tracking surveys-to better identify the extent and source of "leakage" of public funds; better focused administrative data produced by sector ministries-intermediate performance indicators; service delivery surveys; rapid appraisal of specific problems or issues; and participatory performance assessments. Rigorous impact evaluations might be added to this list, although their considerable expense-which for the Bank have ranged from about $200,000 to $900,000 per evaluation-suggests few if any HIPC countries would be able to afford them; this is an argument for donor funding for such evaluations. 41 As noted in Section 4, 12 of the 51 PRSP countries are involved in ECD, with Bank support, compared with only 8 non-PRSP countries. 42At the level of the federal government. 43 Report of the Working Group on Imroving Quality of Monitoring and Evaluation in Bank-Financed Operations (CODE2000-72), June 29, 2000, p. iv. " Report from CODE on Report of the Working Group on Improving Quality of Monitoring and Evaluation in Bank-Financed Operations. CODE2000-96, 13 October 2000, p. 2. 18 governments; the benefits from linking ECD with improvements to financial management and statistical systems; the benefits to be gained from a country M&E diagnosis and clear action plan; and the recognition that ECD is not a "quick fix"-it requires a long-term effort by countries, with donor support. (i) What is Mainstreaming? 5.10 In mainstreaming ECD there are a number of considerations that require some attention. Most important is deciding what "mainstreaming" means. ECD can be defined to be mainstreamed within the Bank when: * Country teams recognize the importance of ensuring substantive country M&E capacities. * ECD is included among the strategic issues addressed in the Bank's dialogues with country leaders on the development agenda. * The Bank regularly includes ECD in its lending and non-lending work, particularly via the public sector management reform programs which it supports. 5.11 It can be argued that mainstreaming entails a process requiring a combination of the following: the exercise of persuasion and compulsion; changes in incentives; adoption of new procedures; adaptation of training protocols; and incremental resources.45 Key aspects of these are discussed below. 5.12 ECD experience emphasizes the need to tailor M&E approaches to the circumstances and realities of each country. Thus mainstreaming would not imply a standardized M&E framework or system. In tailoring ECD support to each country, it would help Bank teams if they possessed an understanding of the strengths and weaknesses of the M&E system of their client country; this would also help them focus on the extent of likely country demand for more effort to be devoted to M&E, and thus whether there exist opportunities for future ECD work. The issue of M&E training for Bank staff is discussed later. 5.13 An approach which could assist Bank country teams to focus on country M&E priorities would be to conduct a rapid survey of key staff members of selected country teams focusing on: the extent of demand for M&E from government, from civil society, and from donors; the extent of supply of M&E information from various sources; and any ECD activities already underway. The first of the two recent surveys of the Bank's PRSP country teams, focusing on the financial tracking of HIPC monies, is analogous to the rapid survey of M&E proposed here.46 As noted in Section 4, the M&E Strategy being 45 R. Picciotto, "The Logic of Mainstreaming: A Development Evaluation Perspective" keynote address to the annual conference of the American Evaluation Association, St Louis, Missouri, 7-10 November 2001. 46 MFDA, Tracking of Poverty-Reducing Public Spending in Heavily Indebted Poor Countries (HIPCs), 27 March 2001, and IMF/IDA, Actions to Strengthen the Tracking of Poverty-Reducing Public Spending in Heavily Indebted Poor Countries (HIPC) IDA/SECM2002-0030, 28 January 2002. The first survey entailed focus group discussions of up to two hours, usually involving a facilitator from the Public Sector Group of 19 prepared by the Bank's Africa region includes the intention to conduct a rapid survey of this nature. A rapid survey would raise the profile of M&E with Bank teams as part of a mainstreaming effort, and ensure that they focus on all of the key dimensions of M&E and on ECD opportunities-even if Bank teams, as expected, do not have all the answers. The results would guide decisions about how far any ECD options should be pursued with the client country, and would also provide a baseline for assessing future progress with ECD in each country surveyed. A rapid survey would also avoid placing an undue burden on country team staff. An indicative list of key questions which could be addressed via such a survey for PRSP countries is presented in Annex F. 5.14 Experience with ECD, and indeed with other types of capacity-building, emphasizes the importance of country ownership and commitment. There is a role for Bank country teams to strengthen government awareness and understanding of the value of M&E, and to work to build country demand for ECD. Experience with PRSP countries suggests this is not a particularly difficult task in this context, though there is an issue of what is meant by "M&E"; this again suggests an emphasis on explaining different M&E tools, techniques and approaches, and their various uses.48 A number of PRSP countries have made valuable progress toward a more data-driven approach to preparation of poverty reduction strategies, and this provides a sound starting point for a more systematic treatment of M&E. However, Bank advocacy of ECD in middle-income countries is likely to be more challenging, as greater resistance can be expected to externally-generated suggestions for public sector reforms such as ECD-his would be a constraint in terms of the demand for ECD. 5.15 One option for the Bank to consider is to amend its operational policies to emphasize the importance of having available a diagnosis of each country's system for poverty M&E, as well as a capacity-building plan if current capacities are judged inadequate ese issues would ideally be addressed in the CAS for each country. Such an operational policy would certainly be an important step in the direction of mainstreaming. However, the Bank's experience with M&E of its own projects demonstrates that having a Bank policy that quality M&E will be undertaken is one thing; actually achieving it is another matter entirely. 5.16 An issue for CODE to consider is when ECD should be mainstreamed within the Bank?-i.e., requiring inclusion of ECD among the strategic issues addressed by the Bank in its dialogues with country leaders, and the inclusion of ECD as a regular component of its lending and non-lending work, particularly via Bank-supported public sector management reform programs. Is the Bank well placed to achieve the mainstreaming in FY04 to which management has agreed? Is there sufficient priority for ECD, and sufficient experience with ECD, to mainstream it now? When mainstreaming occurs, what are to be PREM and a small number of staff representing each Bank country team. The facilitator wrote up notes from the country team discussion and provided them to the team for verification and finalization. The second survey involved both Bank country teams and governments, and a much more detailed, in-depth and time- consuming approach, including a mission to each country. 47 Such a survey would be need to be piloted before more widespread application. 48 OED, Monitoring & Evaluation: Some Tools, Methods and Approaches, The World Bank, 2002. 20 the expectations on Bank regional VPUs, Bank country teams, and on countries themselves? (ii) Need to Clarify M&E and ECD 5.17 There is confusion among Bank staff and in borrower countries concerning what M&E comprises-the various tools and approaches which are available, their uses, and the nature of and prerequisites for capacity-building to institutionalize M&E in a sustainable manner (i.e., ECD). As noted earlier, PRSPs and Bank/Fund JSAs often use the term M&E as shorthand for financial tracking and national development indicators. And the recent Bank/Fund review of the PRSP approach also interpreted M&E as being synonymous with monitoring.49 This restricted understanding of M&E overlooks a number of other M&E tools, techniques and approaches which offer considerable benefits to countries as an aid to sound governance. This argues for an enhanced Bank effort to deepen the understanding of its own staff and client countries about M&E and what it has to offer; this would help lay the groundwork for mainstreaming. The work of the group comprising WBI, OPCS and OED, which is reviewing the content and quality of the large number of M&E courses currently provided to Bank staff and clients, is relevant here (see Section 4), as is the ongoing work of OPCS, which is the Bank's strategic learning center for M&E training in Bank Operations-the Bank's learning centers assess demand for training and help to establish a coherent learning program for Bank staff. (iii) Inconsistency of Message in Bank Policies and Guidelines on M&E 5.18 Related to the need to clarify M&E and ECD, there is an apparent inconsistency of message among various Bank policies and guidelines relating to country-based M&E. For example, the governance strategy of the Bank's PREM network highlights the priority for helping countries build their M&E capacities, yet very few of the Bank's public sector reform or public expenditure management projects include an ECD component.50 Similarly, the Bank's guidelines for Joint Staff Assessments (JSAs) of PRSPs flag the need for Bank country teams to assess whether a country's current and proposed M&E systems are adequate and sustainable,51 yet the Bank guidelines for Poverty Reduction Support Credits (PRSCs-which are intended to provide programmatic lending support to eligible PRSP countries--are silent on the priority for country M&E systems in support both of sound country governance and the Bank's fiduciary expectations.52 49 IDA/IMF (March 2002, oM. cit.). This report interpreted a "full M&E system" (sic) as a monitoring system. See paragraphs 70-76. 5o These projects tend to focus much more on budget systems and budget allocations-i.e., government inputs-rather than the results of that spending. 51 Bank/Fund, Guidelines for Joint Staff Assessment of a Poverty Reduction Strategy Paper, 1 April 2001. 52 World Bank, Operational Memorandum: Interim Guidelines for Poverty Reduction Support Credits (PRSCs) 31 May, 2001. These guidelines stress the need for sound public expenditure management, financial accountability and procurement, but not M&E (neither Bank M&E nor country M&E systems). Note that there exist Country Financial Accountability Assessments (CFAA), Country Procurement Assessment Reviews (CPAR), Country Policy and Institutional Assessments (CPIA), surveys of financial tracking of HIPC monies, and Public Expenditure Reviews (PER), but no formal instrument for country 21 (iv) ECD Coordination and Knowledge Sharing 5.19 Another issue is the need for better coordination of the Bank's ECD activities.53 There is scope for a much more systematic and purposive sharinf of the lessons of past and current ECD work, among Bank staff and with client countries. There has been a rapid growth in Bank-supported ECD work over the past two years-most of the country-level ECD activities listed in Section 4 (and Table C.1) as being currently underway were initiated in that period. In preparing the stocktaking for this first AREC, it was found that there exists no central, comprehensive list of substantive ECD work underway, and as mentioned earlier there is a lack of regional or network ECD contact points. 5.20 There is need for richer information exchanges both within the Bank and externally concerning lessons learned: which ECD approaches appear to be most cost-effective, in which circumstances, and why. This is not an issue of central control but rather of information exchange and knowledge sharing among the various regions, networks and central units (such as OPCS, DEC, WBI, and OED) involved, or potentially involved, in ECD. 5.21 Related to this is the desirability of a regional and network approach to ECD. The Bank's Africa region VPU organized a workshop in January 2002, in order to share lessons from four African countries which are undertaking ECD, and to help its management prepare a regional strategy for M&E. The VPU plans that this strategy will have a principal focus on ECD, and in support of this it is planned to appoint a regional M&E coordinator. The Bank's Latin America & Caribbean regional VPU is also preparing an M&E strategy, of which one of the four areas of focus appears likely to be ECD. The Europe & Central Asia regional VPU has an M&E helpdesk, which provides advice on Bank M&E although not on ECD. A regional approach to ECD which orients and complements the stand-alone country work has much to commend it: as a means to share lessons learned; to make better use of regional institutions; to facilitate cooperation with the regional development banks and bilateral donors in ECD; to build up a cadre of suitable ECD consultants; and to identify Bank staff in each regional VPU who can act as regional ECD resource persons or coordinators. Similar arguments can be advanced to support the networks' approach to ECD. M&E. In April 2002 the Bank decided that the CPAR, CFAA and PER for each country could be combined into an integrated fiduciary assessment. s The Bank's 2000 M&E Working Group report on M&E recommended that a cadre of senior M&E knowledge specialists (equivalent to 16 full-time staff) be created in the regions and networks by FY03, including the two M&E/ECD specialists in OPCS; decisions on the actual number of positions to be created were to be left to individual VPUs. Several of the regions and networks have staff available to provide advice on M&E of Bank projects to their colleagues, though there is no list of specialists with that responsibility, nor any estimate available of the number of full-time equivalents. None of the regions or networks employ their own ECD specialists. 54 OED has sponsored or jointly sponsored a number of global and regional seminars to discuss and promote ECD in recent years. These include seminars in: Washington in April 1998; Abidjan in November 1998 (jointly with the AfrDB); Beijing in October 1999 (jointly with the UNDP and the Chinese government); Johannesburg in September 2000 (jointly with the Development Bank of Southern Africa and the AfrDB); and The Hague in March 2001 (jointly with the Netherlands government). The challenge now is to systematize such activities. 22 5.22 One possible model for closer cooperation on ECD within the Bank is the program for Public Expenditure Management and Financial Accountability (PEFA), coordinated by the PREM Public Sector Group and OPCS. This program is designed to break down the organizational stovepipes within the Bank, enhance donor partnerships, garner additional resources, focus resources more effectively, and disseminate lessons of experience more effectively. " A similar approach to ECD might ideally include creation of an ECD helpdesk. 5.23 A more integrated, knowledge sharing approach to ECD could also involve a merger of the various Bank thematic groups which include an M&E focus: the Poverty Analysis, Monitoring and Evaluation thematic group (coordinated by PRMPR), the Evaluation, Monitoring and Quality Enhancement community of practice (coordinated by OED), and the M&E thematic group which OPCS plans to create. At present there are some 105 thematic groups within the Bank, and a consolidation in the area of M&E would exploit synergies between closely related thematic areas, help to reduce stovepipes within the Bank, and make it easier for Bank staff to navigate among the work plans of the different thematic groups. (v) Bank Resources to Support M&E and ECD 5.24 Mainstreaming ECD has funding and staffing implications for the Bank. A number of funding sources have been or are being used to support preliminary ECD work in the Bank, such as awareness-raising, diagnostic work, M&E training, and regional seminars on ECD. These funding sources include: Bank budget; consultant and other trust funds; the Bank's Institutional Development Fund (IDF);5 direct funding of consultants by bilateral donor partners; and OED's partnership funds from bilateral donors. In addition, the new trust fund for capacity-building for poverty reduction strategies in low income countries, announced in March 2002, includes M&E capacity-building as one of the nine types of eligible activity. Once an individual country decides to pursue ECD in a substantive manner, the option of using Bank loans then arises.57 5.25 The funding sources for preliminary ECD work in countries are fragmented and it is highly likely that they will need to be augmented when ECD is mainstreamed in Bank country work. One option would be to set up a trust fund solely for ECD activities similar to the trust fund for statistical capacity-building managed by DECDG. 5.26 Management's commitment to mainstream ECD means that VPUs will need to address the issues of work program and staffing priorities of country teams. The strength of the available skills base within the Bank is unclear. The Bank's 2000 M&E Working Group report on M&E noted-although without providing any evidence-that many Bank staff already have substantial knowledge and interest in M&E even though their work programs are focused elsewhere. This is a strong assumption which should be validated across the Bank. 55 The PEFA program is summarized in the Bank's 18 December 2001 PREM news. s6 M&E capacity-building is one of the main areas of emphasis of the IDF program for FY02. 57 For example, the Bank's Economic & Financial Management project agreed with Honduras in 2000. 23 5.27 High-quality ECD work requires a combination of skills and experience in the areas of public sector reform and M&E. There are many Bank staff with skills in each area, though few with substantive experience in both. A survey of staff in the Africa region VPU conducted in January 2001 suggests a considerable number of staff possess a high level of knowledge in a wide range of M&E skills and related areas-such as impact evaluation, performance indicators and logframe, participatory M&E, conducting institutional assessments, formulating an ECD action plan, and conducting and analyzing household surveys. This survey was self-reported, and thus potentially unreliable if staff have tended to overstate their M&E skills; the actual knowledge and experience of staff in these skills areas is not certain. But the survey results do suggest that M&E skills and experience already exists within one of the regions of the Bank. The large number of Bank staff who are a member of the two existing M&E thematic groups-about 600-also indicates a high level of staff interest in M&E. 5.28 One approach to strategic staffing for ECD, advocated by the Bank's 1994 ECD Task Force, would be to rely on country departments for lead responsibility for ECD efforts, with the main source of technical skills being the Bank's public sector management specialists in what are now termed PREM units. It was also recommended that these units build a base of staff skills in evaluation techniques and in ECD-by broadening the skills of existing staff rather than hiring evaluation specialists, and complementing these staff with consultants for specific tasks. Future OED Annual Reports on ECD (AREC) 5.29 OED has committed to CODE to submit annual reports on ECD from 2002 onwards. This first AREC provides a stocktaking of ECD activities and approaches underway across the Bank, clarifying also the nature of ECD work, its relevance, its processes, and performance indicators which can be applied to it. This report thus lays the foundations for subsequent ARECs, which will evaluate the quality of the Bank's ECD work. It is proposed that these ARECs will include: a similar stocktaking of ECD; measurement of the extent of ECD progress in individual countries; and in-depth evaluation of ECD in selected countries; a possible methodology for undertaking these evaluations is outlined in Annex E. OED intends to consult widely, both within the Bank and externally, concerning the most appropriate methodology for these evaluations. s8 Evaluation Capacity Development: Report of the Task Force, July 5, 1994, pp. 29-30. 24 6. Conclusions 6.1 Helping borrowers strengthen their M&E systems has become a corporate priority. The Bank's ECD strategy, originally established in 1994, is expected to be mainstreamed in FY04. The strategy recognizes the role of M&E for sound governance. The Bank's overall strategy for public sector reform, agreed in 2000, endorses this priority. 6.2 There are continuing deficiencies in the Bank's approach to M&E. A stronger emphasis on ECD would help improve the Bank's M&E performance. There is now a close interdependence between the Bank's project, sector and country-focused M&E, including for the CAS, and the ability of borrowers to conduct and utilize M&E-this is consistent with the new corporate priority of measuring, monitoring and managing development performance. 6.3 The CDF, HIPC and PRSP initiatives, as well as the Millennium Development Goals, imply better M&E in order to understand what works, what does not, why, and in what contexts. The PRSP approach is providing a useful starting point for a more systematic treatment of M&E. Accountability calls for demonstrating the results of development spending. The Bank-wide working groups on M&E created in 1999 and again in 2002 have sought to address these growing challenges. 6.4 The stocktaking of Bank-supported ECD activities, presented in Section 4, reveals that a modest but increasing number of Bank country teams are working with borrowers to strengthen country M&E systems. There has been a substantial increase in this work over the past three years; the Bank's M&E Improvement Program, managed by OPCS, and with the continuing support provided by OED, WBI, DEC and other central units has assisted country teams in this work. Yet at present, only 21 Bank country teams (out of around 150) are supporting ECD work. 6.5 When CODE agreed to the M&E Improvement Program in September 2000 it was with an initial focus on the creation of eight new pilot countries, but with the understanding that management was committed to mainstreaming ECD by FY04. As noted in Section 5, mainstreaming will occur when: * Country teams recognize the importance of ensuring substantive country M&E capacities. * ECD is included among the strategic issues addressed in the Bank's dialogues with country leaders on the development agenda. * The Bank regularly includes ECD in its lending and non-lending work, particularly via the public sector management reform programs which it supports. 25 ANNEX A: AREC Methodology 1. The first step entailed a review of the ECD literature,59 including the proceedin s of a number of ECD conferences and seminars which have taken place in recent years, and a review of Bank strategy documents. 61 This analysis provided the basis for Annex B (which discusses what ECD entails and how it is undertaken) and Section 3 (concerning the Bank's strategic directions and implications for ECD). 2. Next, a stocktaking of all Bank-supported ECD activities was launched, with particular attention to country-level ECD work. Data sources were investigated, and leading M&E and public sector management experts within the Bank were consulted to help identify ECD activities underway: * OED's 1998 review of the Bank's public expenditure work.62 * The stocktaking of the Bank's public sector management work, undertaken in 2000 by the PREM network, and reported in the report on the Bank's public sector reform and governance strate .63 An update of this stocktaking was drafted in March 2002. * A stocktaking in 2001 of Bank M&E and ECD activities. This stocktaking was coordinated by WBI and also involved DEC, OED, PREM, the Africa region, and others. * A stocktaking in 2002 of the Bank's M&E training provided to Bank staff and to officials and others in borrower countries. This stocktaking was coordinated by WBI. * An electronic search of Bank project and other documents, focused on key words and phrases concerning M&E and ECD. This search was conducted by OED. * Direct approaches to a number of Bank staff who are well-placed to be aware of ECD work currently underway with Bank support. s9 See, for example, K. Mackay, "The World Bank's evaluation capacity building experience", in D.W. Compton, M. Baizerrnan and S.H. Stockdill (eds.), The Art. Craft, and Science of Evaluation Capacity Building, American Evaluation Association, New Directions for Evaluation, No. 93, Spring 2002, 81-99. This contains a large number of key references relating to ECD. 6 For example, African Development Bank, Development Bank of Southern Africa and World Bank, Monitoring and Evaluation Capacity Development in Africa. Johannesburg, 2000; and UNDP, Chinese National Center for Science and Technology Evaluation and World Bank, Evaluation Capacity Development in Asia, Beijing, 1999. Such conference proceedings include a number of country case studies, such as M. Odwedo, "Monitoring and evaluation and the development challenge in Africa: perspective from Uganda", in Johannesburg volume, ibid. 6 Key documents are referenced in Section 3. 62 OED, The Impact of Public Expenditures: an Evaluation, Report No. 18573, 13 November 1998. 63 Public Sector Group, Reforming Public Institutions and Strengthening Governance: A World Bank Strategy The World Bank, Washington D.C., 2000. 6 Public Sector Group, Reforming Public Institutions and Strengthening Governance: A World Bank Strategh-Implementation Update, draft of 6 March 2002. 26 These individuals included the following: senior PREM staff and senior operations advisers in all Bank regions; M&E and statistical capacity-building experts in DEC Data Group and DEC Research Group, OED, OPCS, PREM Poverty Reduction Group, WBI Evaluation & Scholarships and WBI Economic Policy & Poverty Reduction; PREM Public Sector Group; senior staff in all Bank networks; and others. 3. An electronic search of Bank documents was attempted but it was not successful. Bank project documents do not use any standard description, classification or coding of ECD and related activities.65 Similar problems in undertaking keyword searches in the area of statistical capacity-building led the DEC Data Group to undertake a manual search of the project documents for every active Bank project to compile a database containing information on all Bank projects which include statistical capacity-building efforts. 4. A difficulty in identifying ECD activities relates to the frequently inappropriate use of the term "M&E" in PRSPs and Joint Staff Assessments (JSAs}-discussed in Section 5: they rarely address the problem of the "missing middle" (i.e. performance indicators which measure intermediate processes, outputs and outcomes) or include efforts to build their evaluation capacities. The limited capacity-building efforts of such countries are therefore excluded from the stocktaking of ECD activities in this report.66 5. One other issue which made this stocktaking laborious and time-consuming was the absence of a designated M&E or ECD contact person within each region and network. Word of mouth knowledge concerning ECD activities underway in individual countries proved to be the most productive source of information. 6. Nevertheless, virtually all country-level ECD work was identified and OED considers this country-level stocktaking to be reasonably comprehensive, although not necessarily exhaustive. 65 Very few Bank documents use the term "ECD". And keyword searches of terms such as M&E, capacity- building and M&E capacity-building proved unsuccessful in identifying ECD activities-one reason is that all Bank project documents mention M&E because project M&E is a formal requirement. 66An alternative approach not followed here is that the monitoring capacity-building of such countries be classified as ECD, albeit of a low-quality and with substantive deficiencies. 27 7. Comprehensive information was not sought at the sector level because of its diffuse nature. The Bank is involved in a large number of sector projects, in the health and education sectors, for example. This involves country capacity-building at the level of individual sector ministries, departments and agencies, and in areas which include policy analysis and ministry data systems. This gap should be filled in future ARECs. 8. The results of this stocktaking are reported in Annex C. 6' The sustainability of such sector-specific work is uncertain, particularly because it may often have only weak links with broader public sector reform activities including changed incentive structures which can be expected to lead to changed behaviors among officials and others. Anecdotal evidence suggests much of the work to strengthen sector and sector ministry data systems is successful in generating a plentiful supply of data, but that the data often remain substantially underutilized. These issues merit further research at the sectoral level. The planned partnership between DFID and the Bank's Human Development network, which will focus on ECD in a number of pilot countries, may shed some light on this. 28 ANNEX B: Evaluation Capacity Development (ECD): Definitions, Results Chains, and Performance Indicators 1. Monitoring is defined as: "A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progess and achievement of objectives and progress in the use of allocated funds". Thus monitoring embodies the regular tracking of inputs, activities, outputs, reach, outcomes and impacts of development activities - at the project, program, sector and national levels. The definition used here also encompasses the monitoring of a country's progress against the millennium development goals, or other national measures of development success. Evaluation is defined as "the process of determining the worth or significance of a development activity, policy or program . to determine the relevance of objectives, the efficacy of design and implementation, the efficiency or resource use, and the sustainability of results. An evaluation should (enable) the inco?9oration of lessons learned into the decision-making process of both partner and donor". 2. Monitoring and evaluation are synergistic. Monitoring information is a necessary but not sufficient input to the conduct of rigorous evaluations. While monitoring information can be collected and used for ongoing management purposes, reliance on such information on its own can introduce distortions because it typically covers only certain dimensions of a project's or program's activities: careful use of such information is needed to avoid unintended behavioral incentives. Evaluation, on the other hand, has the potential to provide a more balanced interpretation of performance. But evaluation is a more detailed and time-consuming activity, and because of its greater cost it needs to be conducted more sparingly. One approach is to rely on monitoring information to identify potential problem issues requiring more detailed investigation via an evaluation. 3. M&E can be conducted using a wide array of tools, methods and approaches.70 These include, for example: performance monitoring indicators; the logical framework; theory- based evaluation; formal surveys such as service delivery surveys, citizen report cards, living standards measurement surveys (LSMS) and core welfare indicators questionnaires (CWIQ); rapid appraisal methods such as key informant interviews, focus group discussions and facilitated brainstorming by staff and official-; participatory methods such as participatory M&E; public expenditure tracking surveys; formal impact evaluation; and cost-benefit and cost-effectiveness analysis. 4. There is no clear-cut dividing line between M&E and 'upstream' activities such as the work of national statistical offices and sector ministries which collect, process, store and disseminate data. Indeed the production of monitoring information and evaluation findings 68 DAC Working Party on Aid Evaluation, Glossary of Evaluation and Results Based Management Terms, DCD/DAC/EV(2001)3, OECD, Paris, Nov. 2001. 69 Ibid. 70 A sample of these is discussed in OED, Monitoring & Evaluation: Some Tools, Methods and Approaches, The World Bank, 2002. 29 is heavily dependent on the statistical capacities of national statistics offices and sector ministries. 5. ECD is concerned with strengthening or building M&E capacities and strategies in borrower countries in order that a situation is attained where M&E is regularly conducted and utilized by the countries-governments and civil society. The utilization of M&E findings can be viewed as a 'downstream' activity; utilization also provides a bottom-line measure of the value of M&E. Strengthened country capacities would include not only governments but also universities, research institutes, think-tanks, NGOs, private consulting companies and individuals, and so on. What Does ECD Success Look Like? n1 6. One way to understand ECD is to consider the situation it is intended to lead to-one where available monitoring information and evaluation findings are used systematically by countries: * To inform government resource allocation decisions-policy- making and planning via national budgets72 and Poverty Reduction Strategies- by providing information on the likely benefits and costs of alternative policies, programs and projects. * To support the design and ongoing management of activities (at sector, program and project levels), including government service delivery and the management of staff-this use is the.learning function of M&E. * To enhance transparency and support accountability relationships-accountability of government to parliament, to civil society and to donors, and to underpin accountability relationships within government (between sector ministries and central ministries, and between ministers, managers and staff). Accountability is particularly important in the context of the HlIPC initiative (discussed in Section 3). 7. There is a wealth of experience from developed countries, and increasingly from developing countries, concerning the importance of an explicit focus on performance management as a key dimension of sound governance--these issues are known collectively as the new public management.73 This experience emphasizes that a priority issue for the management of performance is the measurement of performance, via M&E. Note, however, that M&E on its own does not improve performance management-the existence of M&E information has to be acted upon if it is to be effective, and this in turn 7 This view of ECD "success" is similar to achievement of results-based management at the country level (see OED, 1999 Annual Review of Development Effectiveness, The World Bank, 2000, Annex 6). 72 Performance budgeting entails actions such as the ex ante reporting of promised performance (for the future), in terms of expected outputs or outcomes, and ex post reporting of actual results achieved over the Fast year. For example, OECD, In Search of Results: Performance Management Practices Paris, 1997. 30 requires strong incentives for utilization. In other words, the demand for M&E has to match the supply if a sustainable M&E system is to be built. 8. There are many links in the planning, budgeting and management processes of government, and M&E is a valuable tool at each step (see also Figure A. 1 below). M&E supports policy analysis and policy making by informing analysts and decision-makers about the cost-effectiveness of alternative poverty reduction policiesy identifying what works, what does not, and why. For agreed policies, programs and projects, M&E promotes evidence-based development management by identifying the most efficient use of available resources; M&E facilitates cost and performance comparisons among different administrative units, regions and districts-identifying good, bad and promising practices-and it is a means for reducing leakage and transaction costs. And because M&E measures government and donor performance, it provides the essential evidentiary underpinning for strong accountability relationships, including the accountability of donors to their stakeholders. 9. These uses of M&E place it at the center of sound governance arrangements: as a necessary condition for the effective management of public expenditures towards poverty reduction. Thus M&E is necessary to achieve evidence-based policy-making, evidence- based management, and evidence-based accountability. An emphasis on M&E is one means to achieve a results-oriented and accountable public sector, including a performance culture in the civil service. For this reason neither M&E nor ECD should not be viewed as narrow, technocratic activities. 10. There exist a number of links and commonalities between M&E and other dimensions of public sector management: * Budgetary financial tracking systems and financial reporting. * Intergovernmental fiscal relations, and the extent to which they encompass a focus on government performance. * Accountability institutions, and national audit offices. * Commercialization and the private sector (profit and non-profit) delivery of public services-success in these activities requires a clear understanding of objectives and actual performance. * The setting of explicit customer service standards by service delivery agencies, and monitoring the extent to which these are achieved. * Civil service reform which focuses on personnel performance, management and appraisal, including merit-based hiring, promotion and firing-recognizing the links between individual performance and project or program performance. * The quality of the civil service's policy advice, and the extent to which it is evidence-based (using M&E). 31 * Participation and civil society-M&E provides a vehicle to magnify the voice of civil society and to put additional pressure on government to achieve higher levels of performance.74 * Anti-corruption efforts-M&E can be used to identify the 'leakage' of government funds. 11. Figure B.1 shows entry points for ECD in a country's planning, budgeting and management cycle; for each country, the most appropriate types of ECD can be identified at each stage in this cycle. A detailed list of possible ECD actions under each of these types is shown in Table B. 1. The table groups the actions in five columns, according to the following categories: to strengthen demand for M&E in civil society and parliament; to strengthen supply and develop M&E skills in civil society and parliamentary staff; to strengthen government demand for M&E; to strengthen supply and develop M&E skills within government; and to strengthen M&E systems. Where there exist similar or related actions in these five columns, this can be observed in the Table by looking along each row. The choice among these actions should be governed according to the strength and nature of demand and supply in each country. 74 Civil society can play an important role in M&E in at least four ways. First, it can present beneficiary views on government service delivery. Second, it can produce analysis and reviews of government performance, via activities such as budget analyses and citizen report-cards. Third, by providing independent scrutiny of M&E findings which governments produce. Finally, civil society is a user of M&E information-via media reporting and also the activities of universities, think-tanks and NGOs. 32 Figure B. 1: Planning, Budgeting and Management Cycle in Government ACCOUNTABILITY OF GOVERNMENT to: parliament; civil society; donors Pursue ECD via: 1. Strengien demand for M&E in civil society, parliament 2. Strengthen supply and develop M&E skills in civil society, parliamentary staff GOVERNMENT ANALYSIS AND REVIEW OF POLICY Pursue ECD via:, 1. Strengthen supply and develop MONITOR ACTIVITIES, ACCOUNT FOR DECIDE POLICY, PLAN, EXPENDITURES, EVALUATE ALLOCATE BUDGET PROGRAMS & PROJECTS RESOURCES Pursue ECD via: 1. Strengthen supply and develop 1. Strengthen government M&E skills demand for M&E 2. Strengthen M&E systems 2. Strengthen M&E systems IMPLEMENT AND MANAGE GOVERNMENT ACTIVITIES Pursue ECD via: 1. Strengthen government demand for M&E 2. Strengthen M&E systems 33 Table B.1: List of Possible ECD Actions at the Country Level Strengthen Demand for M&E in Strengthen Supply and Develop Strengthen Government Demand for Strengthen Supply and Develop Strengthen M&E Systems Civil Society, Parliament M&E Skills in Civil Society, M&E M&E Skills Within Government Parliamentary Staff Bank/donor advocacy of M&E and Bank/donor advocacy of M&E and strengthen/ create M&E/ stats/ MIS monitoring of progress monitoring of progress computer system national brainstorming ECD national brainstorming ECD seminars coordinate/ harmonize donor in- seninars country M&E . , regional seminars on ECD foster regional networks of M&E close coordination with ECD work ._managers and practitioners of other donors donors preserve and disseminate donors preserve and disseminate ECD support creation of International donors preserve and disseminate ECD experience in other countries experience in other countries Development Evaluators Association ECD experience in other countries (IDEAS) conduct national/ sectoral conduct national/ sectoral diagnosis of conduct national/ sectoral diagnosis of M&E functions - M&E functions - demand, supply, diagnosis of M&E functions - demand, supply, systems, ECD systems, ECD options demand, supply, systems, ECD options options embed M&E as component of embed M&E as component of embed M&E as component of PSM/PEM work PSM/PEM work - MTEF and PSM/PEM work - MTEF and performance budgeting; RBM and performance budgeting; RBM and strategic planning; SWAPs; sectoral strategic planning; SWAPs; reform programs; service delivery; sectoral reform programs; service financial mgmt & accountability, delivery; financial mgmt & including FMISs; national statistics accountability, including FMISs; system capacity building (statistics national statistics system capacity offices, statistics production, including building (statistics offices, service delivery surveys); national statistics production, including audit office (esp. performance audits); service delivery surveys); national decentralization; rationalization of audit office (esp. performance ministry MISs audits); decentralization; rationalization of ministry MISs seek to mandate M&E via government seek to mandate M&E via decision, decree, regulation, law government decision, decree, regulation, law support creation of M&E champion support creation of M&E champion create evaluation fund - allow civil create evaluation fund - allow civil create evaluation fund creation/staffing of M&E/ stats/ policy society access society access analysis units FOI legislation and government agreement of M&E fiduciary encourage stronger gov't internal policy on dissemination and obligations of government to donors coordination of M&E, including publication of evaluation reports links between M&E, budget, planning and ministry managemen systems 34 Strengthen Demand for M&E in Strengthen Supply and Develop Strengthen Government Demand for Strengthen Supply and Develop Strengthen M&E Systems Civil Society, Parliament M&E Skills in Civil Society, M&E M&E Skills Within Government Parliamentary Staff promote and support NGOs, research donor support for development of institutes, universities to evaluate and government and ministry M&E review gov't performance - e.g. systems, procedures and protocols budget analysis, citizen report cards disseminate examples of M&E disseminate examples of M&E utility share information about good- utility practice M&E planning and systems with other countries in region support community involvement in donor sub-contracting of donor funding of major impact donor funding of major evaluations/ participatory poverty assessments, evaluations/reviews to universities/ evaluations/ reviews - to demonstrate reviews - on public good grounds etc research institutes feasibility and utility encourage civil society and greater donor reliance on indigenous parliamentary scrutiny of gov't M&E expertise when conducting their evaluation reports evaluations encourage the sharing of encourage the sharing of evaluation evaluation findings with other findings with other countries in the countries in the region region widespread dissemination/ widespread dissemination/ disclosure disclosure of Bank M&E findings of Bank M&E findings dissemination of evaluation standards dissemination of evaluation standards and methods and methods provision of M&E trainer-training and provision of M&E trainer-training and twinning support for universities/ twinning support for civil service research institutes colleges provision of M&E training to NGO provision of M&E training to staff officials- national/ district training provision of M&E training to support development of national parliamentary staff evaluation associations train parliamentarians and staff to donor support for joint evaluations donor support for joint evaluations analyze government performance with governments, and for country-led with governments, and for country-led evaluations evaluations encourage media to report on government performance promote self-evaluation by non- promote self-evaluation by non- support conduct of service delivery government agencies government agencies surveys 35 How is ECD Undertaken? 12. There is no single, unique way of undertaking ECD. The focus, speed and sequencing of ECD depends very much on the starting point- the country's strengths and weaknesses-and on national priorities and country judgments about the cost-effectiveness of various M&E tools. This tailoring of ECD to the needs and priorities of each country underscores the desirability of conducting a baseline diagnosis of M&E activities to guide the identification of ECD opportunities; this also provides a yardstick for measuring a country's ECD progress over time, recognizing that it is a long-haul effort to build and sustain both demand and suppl for M&E. OED has published a guide to aid the preparation of such.diagnoses. 13. A national diagnosis of M&E would be expected to map out a number of key issues. These include the nature of existing demand for M&E and the nature of current M&E activities and functions within government, the actual and potential supply of M&E capacities in the country, the extent of M&E utilization, and both the actual and potential synergies between M&E and public sector reforms in areas such as a poverty reduction strategy, the budget system, strengthening policy analysis skills, creation of a performance culture in the civil service, improvements in service delivery, decentralization and greater participation by civil society. 14. Another step in ECD work is to support government in mapping out an action plan for ECD. An action plan would typically seek to build on existing public sector reform work, including related capacity-building. Action plans would address issues such as identification of M&E champions and the M&E roles andresponsibilities of central and sectoral ministries. Training of officials and others, including trainer-training, would also be expected to be a component of most action plans. These plans might include other components such as: * A review of administrative systems for producing and disseminating monitoring information in key ministries. * Identification of M&E priority activities. While ECD supports sound M&E it is also concerned with cost-effective, fit-for- purpose M&E. In the poorest countries there may be a shortage of skilled personnel, data and other resources. Such constraints might be one of the reasons for more emphasis to be placed on "low-tech" M&E approaches, such as performance monitoring indicators and participatory poverty assessments, rather than sophisticated, formal impact evaluations. 7s K. Mackay, Evaluation Capacity Development: A Diagnostic Guide and Action Framework. OED ECD Working Paper No. 6, January 1999. This guide also presents a simple taxonomy for classifying countries, according to whether their demand for and supply of M&E is strong/weak. A rapid assessment guide has also been drafted by OED and Operations Policy & Country Services (OPCS): "Conducting a diagnostic and needs assessment for pilot country teams and borrowers." See J. Kusek and R. Rist, "Building a performance-based monitoring and evaluation system: the challenges facing developing countries", Evaluation Journal of Australasia 1(2), 2001, forthcoming. 36 * A schedule of major evaluative activities which would be undertaken, such as public expenditure tracking surveys, service delivery surveys, participative poverty assessments--this work would necessarily be linked with efforts to strengthen the capacities of national statistical offices. * Options for engaging civil society in M&E. * Efforts to better coordinate donor M&E activities within the government. * Sector-specific actions, such as a more detailed diagnosis for key sectors. 15. An example of an M&E diagnosis and menu of ECD actions, prepared at the request of the government of Uganda, is shown in Table B.2. Key Characteristics of ECD 16. There are at least five key characteristics of ECD: * ECD involves a purposive, guided process of capacity development-it requires a clear strategy, action plan and ongoing, long-term support. * ECD focuses on the measurement of public sector performance-the relationships between public actions and poverty-reduction. * ECD is concerned with a sustainable and systemic-but not necessarily comprehensive-approach to M&E conduct and utilization. * ECD is closely linked with other public sector reform efforts, particularly those relating to incentives which determine the extent to which M&E will be utilized. * ECD is concerned with the utilization of M&E by developing countries-appropriate utilization of M&E is fundamental for ECD success. 17. Note that there is a distinction between these key characteristics of ECD and various dimensions of ECD quality. Thus it can be argued that a high-quality approach to ECD would involve a detailed diagnosis of national M&E, but a diagnosis is not a prerequisite for undertaking ECD. Similarly, provision of M&E training might be a common element 76 OED has found that in poorly performing countries, a diagnostic review is of critical importance for the assessment of ownership and risks. OED, 2001 Annual Review of Development Effectiveness Report No. 23342, 20 December 2001, page 33. 37 of most high-quality ECD, but if conducted on its own and not as part of a larger, more strategic program of activity, it does not in itself constitute ECD as defined here. OED has developed a list of criteria for high-quality ECD (Box B.1).77 7 These are based on the report of the Bank's 1994 ECD Task Force, op. cit. 38 Table B.2: Country Example of a Summary M&E Diagnosis: Uganda-Strategic Issues, Challenges and Possible Actions Overall M&E Monitor & provide feedback on progress in poverty alleviation Task 1. Coordination 2. Development 3. Incentives for 4. Devolution of 5. Role of civil 6. PEAP 7. M&E skills Strategic M&E and goals, targets and contribution to managerial society in partnerships training Issues harmonization performance results autonomy enforcing principles indicators transparency and accountability Positive * Draft Poverty * Poverty * Recognition of * Decentralized * Consultative * Draft/CDF * Awareness of elements in Monitoring Eradication Action service delivery responsibility for nature of PEAP partnership M&E importance Uganda Strategy Plan (PEAP)/PRSP effectiveness as service delivery process principles * Availability of * Sector Working as overall imperative of public * Introduction of * Transparency * Trend towards local researchers; Groups (SWG) as framework of management output oriented of budget process budget support local academic nexus of planning, poverty priorities * National Service budgeting (OOB) * Practice of * Consultative and training budgeting * Training in ROM Delivery Survey * Comprehensive public notices group meeting traditions * Efforts to (results oriented (NSDS) 2000 district plans * Significant being scheduled harmonize project management) being * Local capacity of as part of budget progress reporting piloted Government NGO's cycle * Earmarking of 5 * "Indicator retreat" Development * Government/ per cent of Poverty as part of budget Program civil society Action Fund for cycle (LGDP) capacity dialogue at 'monitoring & development central level accountability' Some * Separate * Inconsistency in * Performance * Generally weak * Need for * 1/3 of ODA is * Weak challenges planning and clarity of goals at assessed in terms of management improved TA outside of management being reporting formats sector level expenditure and capacities at local stakeholder Government skills at LG encountered for different * PEAP goals bureaucratic activity levels consultation in budgets and levels funding sources correspond to * Weak linkage * Prescribed priority-setting M&E * Likely increase * Sector/district ministerial activities between resource spending ratios * 1/3 of ODA is * Nearly 300 in demand for policies, budgets rather than poverty allocation and of conditional TA outside of stand-alone management and work plans outcomes performance grant scheme Government projects remain conduct of M&E approached as * Few goals are * Rewards geared to gives little budgets and * Twenty dealing with separate exercises defined with good paper-work flexibility for M&E practices separate annual inter-relationship * Alignment and measurable * Inconsistent managers to * Gob's/NGO program reviews between service coordination of timeframe, baseline enforcement of adapt to local dialogue at center * Local donors delivery and different results and targets sanctions for poor needs not mirrored at cannot depart poverty outcomes management * Weak linkage performance * Number and local levels from corporate initiatives between sectoral *Corruption largely level of posts M&E guidelines * 1/3 of ODA is and district goals unpunished? directed from the outside of center Government budget Possible actions * Identification of * Cascading of * Introduction of * Allow greater * Extend * Leverage of * Strengthen to address an M&E champion PEAP goals and reach and outcomes local autonomy transparency donor support for local capacity for challenges ministry/agency targets through as yardstick of over recruitment, practice from CDF and PEAP program * Establishment of planning, budgeting success and salaries and non- allocation to to increase evaluation skills 'core M&E' and work planning performance reward wage execution synergy in training arrangements at sector, district * Use of NSDS as expenditures * Client 'Report planning, * Availability of (such as via a and facility levels barometer of client *Ensure stronger cards' as reporting & M&E funding formal M&E * Long-term satisfaction local oversight as complement to review within PAF, framiework), expenditure improvements the quid pro quo NSDS * PRSC policy LGDP and harmonization of framework focus on * Introduce value *Introduction of * Make NGO's matrix as joint Economic & terminology, defining medium- for money concerns participatory eligible for PAF planning & Financial reporting formats term PEAP goals & in Finance Act M&E practices 'monitoring & review Management and periodicity targets *Introduction of as key accountability' mechanism Program II * Improved client service management funding * Establishment coordination charters function *introduction of of national between inspection client service evaluation and audit agencies I charters association Source: Arild Hauge, Strengthening Capacity for Monitorine and Evaluation in Uganda: A Results Based Management Perspective, Operations Evaluation Department ECD Working Paper No. 8, January 2001. 39 Box B.1: Criteria for High-Quality ECD 1. Based on a formal country diagnosis and clear action plan 2. Form part of a public sector management reform program 3. Promote a results orientation and a poverty reduction and growth focus 4. Connect oversight of public expenditures at central, sector and regional level 5. Involve civil society, NGOs, private sector 6. Support parallel initiatives by other development assistance agencies 7. Develop and implement a customized training program for M&ECD 8. Establish linkages with financial management and accountability programs 9. Develop linkages with statistical system improvements 10. Establish linkages with research initiatives 11. Contribute to improved M&E for country/sector assistance strategies and Bank financed projects 18. In conclusion, the definition of ECD which is used here is: ECD is a systemic approach to the improvement of country institutions concerned with the monitoring, evaluation and reform ofpubl sector activities and programs. 19. This definition is embodied in Figure B.2 which shows a results chain for ECD. ECD activities do not operate in a linear, cause-and-effect manner-just as other development processes tend to work in a circular, rather than linear fashion. One example of this relates to government demand for and commitment to M&E. There is a large body of experience which emphasizes that demand for M&E is a prerequisite for sustainable ECD;78 this is very similar to the Bank's experience with other types of governance reform and capacity- building. ECD cannot be foisted upon an unwilling government, and there is an awkward element of chicken-and-egg here: a government is unlikely to be forthright in requesting help with ECD unless it is convinced of the merits of M&E; and a government may not be convinced of their merits if it has not devoted substantive efforts to M&E in the past. The way around this conundrum is to work to strengthen demand by raising awareness and providing education about what M&E has to offer to each country's specific situation. Thus supply-side interventions-awareness-raising, education and building confidence in M&E-work to increase demand, which in turn will tend to call forth further supply responses, and so on. This provides an example of a virtuous cycle of ECD. 7 See the report of the 1994 Bank taskforce on ECD, op. cit. 40 Figure B.2: A Results Chain for ECD Reduction in Poverty A Improved Develop nt Effectiveness Impact Improvements in Perfoi ce of Government M&E Used for (as per Figure 1): accountabilityof government; government decision-making on policies, plans and budget allocation of resources; implementation and management of government activities; monitoring of activities, accounting of expenditures, evaluation of programs & projects; government analysis and review of policy Outcomes Civil Society Parliament Government Civil society Parliament assesses M&E findings M&E findings assesses gov't gov't performance feed into ongoing feed into policy performance and and inputs freely to management of and budget inputs freely to policy debates gov't activities decision-making policy debates Formal M&E framework is established by gov't, leading to more systematic planning and conduct of M&E greater quantity and better quality of monitoring information and evaluation findings available Demand for M&E in Demand for M&E Demand for Supply of Government civil society is in parliament is M&E in M&E and M&E systems Outputs strengthened; supply strengthened; M&E Government is M&E skills strengthened of M&E and skills in skills of strengthened are civil society are parliamentary staff strengthened strengthened are strengthened A package of ECD activities is undertaken, such as: national seminars on ECD; national/sectoral diagnosis of M&E functions; M&E action plan developed, including linkages to other public sector management/budget reforms; donor support of ECD via grants, loans; creation of M&E champion; provision of training-including trainer-training-to officials, NGOs, universities/research institutes, parliamentary staff; strengthening of national statistics office; Activities rationalization of ministry MISs; creation of an evaluation fund; various types of evaluation conducted on pilot/ demonstration basis; donor support for country-led evaluations. 41 20. Another example of a virtuous cycle is when the widespread dissemination of information on the performance-service delivery and client satisfaction--of government facilities such as health clinics and schools stimulates a demand-side accountability response from clients and other stakeholders. Several Bank projects listed in Annex D focus on this civil society dimension. 21. An advantage of the simple results chain in Figure B.2 is that it facilitates understanding of the various levels at which ECD is undertaken, and of the different uses of M&E by different actors in a country.79 For each of the various levels of ECD corresponding performance indicators can be developed (Table B.3). These can be used to provide summary information about the status and results of ECD activities at various stages in their implementation for each country, in terms of major, clearly-identifiable milestones. This also facilitates comparisons between different countries. Table B.3: Suggested Performance Indicators for ECD Key Results Area Performance Indicators Outcomes * M&E findings are used in budget decision-making and in national planning (such as poverty reduction strategies) * M&E findings are used in sectoral and line management decisions * M&E findings are used by media and in parliamentary debates, and in NGO submissions to government Outputs * formal M&E framework is established by government * government structures and processes have been realigned to commission M&E findings and to feed them into budget and planning processes and into ministries' planning and management processes * greater quantity and better quality of monitoring information and evaluation findings * # of officials working on M&E (full-time equivalents) * # of evaluations and formal reviews conducted annually * extent to which civil society is involved in assessing government performance Activities * baseline ECD diagnosis conducted * Bank makes a loan/ grant to government to support its ECD efforts ' A simple results chain does not take account of exogenous factors which may impede ECD efforts. These can include institutional, cultural and political constraints, for example, such as the strength or weakness of a service culture in the civil service, the extent to which corruption is a problem, and the government's attitude to public access to information. The importance of such factors varies from country to country, which is another reason to conduct a detailed baseline M&E diagnosis. 42 ANNEX C: Detailed Stocktaking of Country-Level ECD Activities 1. As reported in Section 4, a number of Bank units are currently involved in ECD and related activities. They include OPCS in its support to Operations, a number of Bank country teams in Africa and other regions, WBI in its provision of M&E and PRSP training, DEC and Regional VPUs working on statistical capacity-building, and OED in its long-standing support for ECD. There also exist ECD partnerships between the Bank- primarily OED, as the official evaluation arm--and other multilateral and bilateral donors. 2. Table C. 1 shows country-level ECD activities underway with Bank support in various Regions. Table C.1: Country-Level(') ECD Activities Underway with Bank Support AfricaRegion Ethiopia (b) The government is working to strengthen information flows and accountability relationships, (c) both from and to the local level. Financial tracking and community report cards will be piloted for health and education facilities in selected districts; there will be a substantive involvement by local government and civil society organizations. The Bank country team is preparing a Capacity Building for Decentralized Service Delivery project to support this participatory M&E system. Ghana (b) (c) Following a request from the government in 1999, OED prepared a diagnostic report on M&E capacities and activities in Ghana, and a menu of possible ECD actions was discussed with the government and civil society. A diagnosis of Ghanaian M&E training organizations has also been undertaken. A Bank Public Financial Management Reform Program project is in preparation; this contains an ECD action plan in support of the PRSP, sector ministry management, and closer civil society oversight of government performance. Madagascar The Bank initiated a diagnostic mission in 2001 to assess country M&E capacities available to (c) (d) ensure implementation of the PRSP. ECD opportunities have been explored to enhance public expenditure management, and achieve greater civil society access to information; a diagnostic report has been prepared. A Bank Poverty Reduction and Institutional Development Technical Assistance Project is now being prepared to support these priorities. The country team is exploring additional entry points using trust funds, to assess M&E issues at various levels of government (for instance in the context of decentralization and commune development). Madagascar is one of the M&E Improvement Program pilot countries. Malawi (c) The government requested ECD support in 2001 with a particular focus on service delivery and the accountability of service providers. The government intends to set up an M&E system which will ensure regular financial tracking and client surveys, involving extensive community involvement and feedback to local and national governments. The Bank country team is preparing a Financial Management, Transparency and Accountability project to support these ECD efforts. Mozambique The government is starting a country-led evaluation, with the support of OED, the Netherlands (c) government and the UNDP. Niger (c) The government requested ECD support in 2001, and the Bank country team with OED advice sponsored a preliminary diagnostic report later that year, with a particular focus on M&E of projects and programs. A more in-depth diagnosis is planned, with the intention of strengthening the M&E system of the PRSP. Tanzania (c) A diagnostic mission has been undertaken, leading to a draft ECD diagnostic report in 2001. (d) This identified GOT's M&E capacities and a number of options for capacity-building. The 43 Table C.1: Country-Level(a) ECD Activities Underway with Bank Support extent of buy-in from GOT and other donors to additional Bank ECD work in Tanzania remains problematical, however. Tanzania is one of the M&E Improvement Program pilot countries. Uganda (b) (C) Following a request from the government in 1999, OED undertook a detailed diagnostic report on M&E capacities and activities in Uganda. The diagnosis and a list of possible actions to strengthen M&E were discussed with GOU in 2000, with a particular focus on provision of support for Uganda's Poverty Eradication Action Plan- the home-grown PRSP - as well as to strengthen Uganda's budget system (a medium-term expenditure framework). GOU and the Bank agreed that strengthening M&E would be an important cross-cutting governance reform in Uganda, and would be reflected as such in the policy matrix of the PRSC credits which the Bank makes to Uganda. The first Poverty Reduction Support Credit was formally approved in 2001. Philippines The Bank initiated a diagnostic mission in 2001, and identified opportunities for ECD work in (d) several contexts: performance-based budgeting, the medium-term expenditure framework, and fiscal reforms; a diagnostic report has been prepared. Opportunities to address ECD as a priority are being explored in the context of the Bank's Public Expenditure Review in preparation, including with support of IDF grants. ECD is articulated as a priority in the CAS. The Philippines is one of the M&E Improvement Program pilot countries. Eur6p dCetr l i Albania (c) An OPCS/OED mission visited Albania in early 2002 to provide a workshop to senior staff from 6 ministries as well as civil society representatives concerning how to design an M&E system. Clinics were also held for officials from 4 ministries which are initiating pilot efforts to institutionalize M&E-each clinic focused on goal setting and indicator systems. The Bank's country team is planning for the inclusion of country-based M&E systems in two Bank loans in preparation. An M&E diagnosis will be conducted early in FY03. Kyrgyz Commencing in 1999, OPCS and OED have held a number of M&E workshops and clinics Republic (b) with central and line ministries, and have discussed with members of parliament and the (d) chamber of accounts on roles they might play to support M&E efforts. An M&E diagnosis is in preparation, and an action plan for strengthening M&E on a systemic basis is being incorporated in both the PRSP and CDF strategies, and in two Bank lending instruments in preparation. The Kyrgyz Republic is one of the M&E Improvement Program pilot countries. Poland The Bank approved an IDF grant to the government in 2000, to help it strengthen its M&E functions in the context of its desire for accession to the European Union. The grant has supported a diagnosis of existing systems for tracking EU monies, and the preparation of an action plan to improve these systems. Work now underway includes the preparation by the government of options to extend M&E of EU and Polish own funds. OED is providing advisory support. Romania (b) Romania is a candidate for EU accession, and the government is keen to move from a central (d) planning approach to one focused more on performance and which encourages civil society's voice. OPCS and the UNDP undertook a diagnostic mission in late 2001 to assess ECD opportunities and barriers, and a draft diagnostic report has been prepared. A number of related reform measures are already underway in the areas of public financial management, e- government, educational testing, legislative reform, and poverty alleviation. Romania is one of the M&E Improvement Program pilot countries. Argentina The Bank's Human Development Network and OPCS undertook a joint mission in early 2002 to assess the millennium development goals in the province of Sante Fe. Part of this mission involved preparation of a diagnostic on ECD within the province, focusing on health and education. Bolivia (b) (c) The Bank approved an Institutional Reform Project in 1999 in support of the government's public sector modernization program, which includes the planned creation of a national 44 Table C.1: Country-Level(a) ECD Activities Underway with Bank Support evaluation system. The system has a particular emphasis on performance indicators and service delivery surveys, with the objective of improving service delivery and strengthening budgeting by results. The initial focus of the project is on five pilot ministries. Brazil The government's national plan reflects an emphasis on results-based management: government programs have clear objectives, performance indicators and targets, and their performance is reviewed during the annual budget process. The government is strengthening its approach to M&E with the intention of institutionalizing an evaluation system. Bank support is being provided via a Programmatic Fiscal Reform SAL approved in 2001, a Fiscal and Financial Management TAL (2001) which includes an ECD component, as well as ongoing ESW. OED is providing advisory support, and has promoted horizontal cooperation with government officials from Mozambique. Chile The Bank agreed a Public Expenditure Management Project in early 2002, to assist the government in deepening its public sector modernization. The government already has a performance indicators system and an annual cycle of program evaluations; the project is intended to make this M&E more effective via strengthening the links to budgeting by results, and by a stronger focus on agency results, staff appraisal, and the creation of a performance culture in the civil service. Honduras (c) The Bank approved an Economic & Financial Management project in 2000 with a component for ECD to support resource allocation decisions, results-based management and service delivery, and to foster accountability. The ECD efforts are particularly focused on budget processes, including sectoral planning and priority setting, in 5 pilot ministries. OED is providing advisory support. Z~~:~ _Midle East" a'NohAfia " Egypt (d) The government has asked to participate in the Bank's M&E Improvement Program. The finance minister hopes to reform the budget process to improve the government's efficiency and effectiveness. OPCS prepared an ECD diagnostic report in 2001, focusing on the potential role for M&E as part of a move towards program or performance based budgeting. The diagnosis identified a number of " islands of innovation" on which an ECD action plan could build; the suggested action plan includes capacity-building in the areas of data management and strategic management. Egypt is one of the M&E Improvement Program pilot countries. South Asia- Bangladesh A joint OED/DEC mission conducted an M&E diagnosis in late 2001. The mission also was (c) (d) tasked with assessing the ability of the government to track progress in achieving the MDGs as part of an overall M&E system. A diagnostic report has been prepared; this identifies opportunities for donor assistance of a national M&E system designed to support the government's poverty reduction strategy. Bangladesh is one of the M&E Improvement Program pilot countries. India In support of the state government's wish to build a performance-based management system, an (Andhra OED/OPCS Bank team in mid-2001 provided training to all 300 senior officials in the Pradesh) government; the training was focused on how to design and build an M&E system. Based on a rapid review of ECD priorities, ongoing support to the government is being provided via a grant from DFID. India (Andhra Pradesh) is one of the M&E Improvement Program pilots. (a) With the exception of the state of Andhra Pradesh (India), included in the table below. ( CDF country c PRSP country (d) Pilot country in the Bank's M&E Improvement Program 45 ANNEX D: OED's Mandate and Support for ECD 1. The Bank's 1994 ECD strategy, endorsed by CODE, allocated responsibility for ECD to the Bank's regional VPUs. In 1997 CODE endorsed a new OED strategy whose priorities include support for "front line" capacity building activities in developing countries.80 OED's support for ECD has the objective of mainstreaming ECD and is intended as catalytic-prompting and supporting the substantive involvement of the regional VPUs in ECD, while also encouraging other central units in the Bank, such as OPCS, WBI and DEC, to provide support. OED is also keen to ensure that the quality of the Bank's ECD work is of a high quality-by encouraging ECD that is relevant, efficacious, efficient, sustainable, and which achieves institutional development impact. The formal mandate for the DGO/OED provides for "encouraging and assisting developing member countries to build effective monitoring and evaluation associations, capacities and systems."81 2. OED's support is based on the subsidiarity principle-leaving it to Bank country teams to take prime responsibility for supporting country efforts to strengthen their M&E capacities. The amount of resources available to support ECD is limited, and currently comprises about 2 full-time staff plus consultants, with an annual cost in FY02 of some $0.7m (about 3.3 per cent of OED's total budget); these ECD resources are supplemented by funds provided by OED's bilateral partners-principally the evaluation offices of the ministries of foreign affairs of the Netherlands and Norwegian governments-and amount to about $0.6m per annum. As noted earlier in Section 4, Box 2, OED also works closely with the evaluation offices of a number of other donors. 3. OED's support takes a number of forms, including preparation of ECD resource material, and its targeted dissemination to operations staff, especially those with an interest in supporting ECD;'2 organization of conferences and seminars to strengthen awareness of the potential uses of M&E and how to undertake ECD, and sponsorship of country officials and members of civil society to attend such events;83 identification and high-intensity support for a select number of countries with potential to adopt demonstration, good- practice approaches to ECD, and where there is strong support from the Bank's country s0 Director-General OED, Evaluation in the World Bank: Challenies and Priorities CODE97-15, 27 February 1997. 81 World Bank, Procedure for the Selection of Director-General of the Operations Evaluation Department, Annex A, Mandate of the Director-General, Operations Evaluation. R2001-0169, 10 September 2001. The priority for OED's support for ECD was most recently endorsed by the Board in its consideration of OED's work plan and budget, in May 2002. 82 Resource material includes diagnostic guides for ECD, country case studies, and seminar proceedings. These are available from OED's ECD website . 83 Such as the one held in Johannesburg in late 2000 (see Development Bank of Southern Africa, African Development Bank, and The World Bank, Monitoring and Evaluation Capacity Development: Selected Proceedings from a Seminar and Workshop, Johannesburg, 2001), and another held in The Hague in March 2001 (see Development Bank of Southern Africa, African Development Bank, and The World Bank, Monitorinp and Evaluation Capacity Development: Selected Proceedings from a Seminar and Workshop Johannesburg, 2001). 46 team, such as Uganda84 and the Kyrgyz Republic-this support would include participation in diagnostic missions and provision of advice on the preparation of public sector reform projects with ECD components; provision of M&E training such as IPDET and IPE (described in Section 4), in part via collaboration with other Bank units; support for the creation of IDEAS and the planned regional centers of evaluation excellence in borrower countries; the promotion ofjoint donor/government evaluations;85 and the planned piloting of country-led evaluations. The OED unit which provides this ongoing ECD support is the Partnerships and Knowledge Programs group (OEDPK). Another OED unit-the Corporate Evaluation & Methods group (OEDCM)-is charged with the evaluation of corporate activities, including ECD. 4. Future ARECs will evaluate the Bank's approaches to ECD-see Annex E. 1 For example, the Bank has reached agreement with the government of Uganda that M&E will be one of the key cross-cutting governance reforms to be pursued in relation to the series of Poverty Reduction Support Credits being provided. The government is preparing an M&E action plan in support of its PRSP objectives. " For example, Danida, Ghana: Joint Evaluation of the Road Sub-Sector Programme, 1996-2000, Netherlands Economic Institute, November 2000. 47 ANNEX E: Note on a Proposed Evaluation Methodology for Future Annual Reports on ECD (ARECs) 1. This first AREC provides a stocktaking of country-level ECD activities supported by the Bank; this constitutes a baseline for future ARECs, which will evaluate the Bank's ECD work. ECD activities conducted by other multilateral and bilateral donors will also be considered. It is proposed that these evaluations will gauge the extensiveness of ECD work underway across the Bank via similar stocktakings to the one in this report. They will also include use of ECD performance indicators, such as those shown in Annex B (Table B.3) relating to ECD "results chains", to enable the extent of progress in each country and the number of countries which have reached various stages of ECD work to be assessed and compared. A survey of key respondents inside and outside the Bank is also envisaged. 2. A more in-depth evaluative approach will be necessary for selected countries, to identify what has worked well in the Bank's ECD work, what has not, and what the contributing factors have been. This will necessitate in-country fieldwork including interviews of government officials, leading members of civil society, and other donors, as well as of Bank country team staff and other Bank staff involved in ECD and related work. The basis for these in-depth evaluations will be OED's agreed methodology for project evaluation, which focuses on the following five dimensions: relevance of objectives; efficacy; efficiency; sustainability; and institutional development impact. 3. This more in-depth approach will increasingly be able to draw on the Bank's own self- evaluations of its ECD work, via the Implementation Completion Reports (ICRs) for Bank projects which have contained an ECD component, and the corresponding ICR Reviews and Project Performance Assessment Reports (PPARs). 4. One particular issue which would yield valuable insights would be the experience with M&E capacity-building work at the sector level, to determine whether stand-alone sector work results in sustainable capacities, or whether sector-level work has to be embedded as part of a broader program of public sector reform to be successful. 5. OED intends to consult widely, both within the Bank and externally, concerning the most appropriate methodology for future ARECs. 48 ANNEX F: Proposed Survey of Bank Country Teams Concerning M&E Systems in PRSP Countries Table F.1: Indicative List of Issues for an M&E Survey Demand for M&E Information > within government * treatment of poverty indicators in PRSPs * whether a national planning body exists, and if so, how it functions and the information it requires * types of information required by finance ministry from sector ministries, as input to national budget process * use made by parliament of budget and PRSP information * use made by sector ministries of information for management purposes > from civil society * pressures from civil society for information from government about its performance * use of M&E information in the media > from donors * whether donors are the main source of demand for M&E in the country * coordination of donor demand, e.g. via SWAPs Supply of M&E Information * quality of information on poverty * capacity of the national statistics office * quality of information from ministries on their spending, activities, outputs and client satisfaction-incl. frequency, timeliness, dissemination * identification of good practices in ministries-quality, range of information collected * information collected from provincial and local governments * availability and quality of information available on non-government services * whether ministries undertake or commission evaluations/reviews of their performance * whether donors are the main source of supply of evaluations/reviews, and do they conduct any jointly with government * whether civil society conducts any evaluations/reviews of government performance * whether effective freedom-of-information legislation is in place M&E Capacity-Building * substantive ECD efforts underway-related to PRSP, public sector reform, budget system, sectoral programs, decentralization * supply of local consultants capable in evaluation * potential for in-country universities and other training organizations to provide M&E training 50 once every stakeholder was on board. He also noted the need to achieve diversity among Bank staff supporting ECD. Another member (Moulin) noted low capacity client countries as a particular short run obstacle in mainstreaming ECD. Finally, the Subcommittee asked whether ECD mainstreaming plans for FY04 were realistic given the delays to date. 6. Bank Intervention. Subcommittee members (Stevenson, Gomes, Moulin) suggested that the linkage of ECD activities with the client country's budget process would yield high returns through improved public expenditure management. One member (Gomes) suggested that, given the importance of ownership, the focus should be on the client country's legislative body. It was agreed that a case by case approach reflecting country conditions should be followed in designing ECD programs. 7. Methodology. The Subcommittee considered that the ECD methodology was appropriate, relevant, and balanced but noted that ECD did not seem to be a priority for many clients. The Subcommittee asked how the Bank could revive interest in ECD, and suggested that an ECD contact person be identified in each Region. 8. Next Steps. Management will complete a progress report on the country pilots in the fall of 2002. 49 ANNEX G: Minutes of the Informal Meeting of the Subcommittee on Development Effectiveness 1. On July 29, 2002, the Informal Subcommittee discussed OED's Annual Report on Evaluation Capacity Development 2002 (AREC). OED noted that the report was intended to: (a) clarify the meaning and relevance of the Bank's work on Evaluation Capacity Development (ECD), (b) lay the methodological foundations for subsequent ARECs to track progress and draw lessons from the Bank's ECD work, and (c) to present an inventory of ECD activities underway across the Bank, with special attention to country- level activities. OED also prepared the AREC as part of its fiduciary responsibility to preport to CODE on the implementation of its mandate to support ECD. 2. Comments from Management. Management agreed that ECD had proved difficult to implement because of, among others, the absence of a formal M&E or ECD contact person within each region and network. Evaluation was the responsibility of the client country, while the Bank's role should be to foster country ownership in an M&E system and help develop statistical capacities in client countries. Management agreed that it was time for the Bank to accelerate work in this area, particularly given the potential reputational risks to the Bank. Management proposed that case studies be included as part of the second AREC. 3. The Subcommittee welcomed the report, and noted the strong priority for ECD. The Subcommittee noted that the discussion was long overdue given that the Bank's strategy for ECD had been established in 1994. The Subcommittee also acknowledged that this was a difficult and complicated subject that posed particular implementation and mainstreaming challenges. Subcommittee members asked about the status of the eight pilot countries and the two sectors where ECD work was to have been pursued and completed by FY02. Lastly, the Subcommittee asked about management's expectations of CODE and the Board with respect to ECD and asked that management provide a list of recommendations. Among the specific issues raised by members were: 4. Definition of ECD. The Subcommittee urged the Bank to clearly articulate its intentions and activities in the area of evaluation and ECD. Members (Chin, Weber, Sayed) particularly emphasized the need to clarify the Bank's message in its policies and guidelines on ECD in order to avoid inconsistencies. 5. Mainstreaming of ECD. The Subcommittee expressed concern about the limited progress in mainstreaming ECD over the past 7-8 years and urged the Bank to determine the key constraints to implementing ECD acitivities. Members (Stevenson, Moulin, Sinha, Hirvensalo) stressed the importance of identifying a government entity that would ensure the inclusion of evaluation results into policy-making and the need to link ECD activities to the work of the Task Force on Results-Based Management. One member (Gomes) stressed the importance of ownership and urged that mainstreaming only be undertaken IMAGING PRnntNn -25145 Type. OES