3534-Rikka.pdf 6/20/05 2:51 PM Page i Agriculture and Rural Development Discussion Paper 20 The World Bank Monitoring and Evaluation for World Bank Agricultural Research and Extension Projects: A Good Practice Note Riikka Rajalahti Johannes Woelcke Eija Pehu 3534-Rikka.pdf 6/20/05 2:51 PM Page ii © 2005 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone 202-473-1000 Internet www.worldbank.org/rural E-mail ard@worldbank.org All rights reserved. The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomina- tions, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978- 750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. ii 3534-Rikka.pdf 6/20/05 2:51 PM Page iii CONTENTS ACRONYMS AND ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v PREFACE AND ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi EXECUTIVE SUMMARY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii 1. CHALLENGES FOR M&E SYSTEMS IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. THE WORLD BANK RESULTS FRAMEWORK REQUIREMENTS . . . . . . . . . . . . . . . . 6 3. DATA COLLECTION, REPORTING, AND DISSEMINATION REQUIREMENTS. . . . 16 4. PARTICIPATORY MONITORING AND EVALUATION. . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5. EVALUATION OF OUTCOMES AND IMPACT IN THE WORLD BANK. . . . . . . . . . . . 45 6. ECONOMIC EVALUATION OF COMPETITIVE GRANTS SCHEMES. . . . . . . . . . . . . 51 7. CONCLUDING REMARKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 8. REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Appendix 1. Narratives of New Approaches and Instruments in ARE Systems . . . . . . . . . . . 67 Appendix 2. Implementation of a Monitoring and Evaluation Plan in the Bank's Project Cycle: Putting It All Together. . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Appendix 3. A Brief Description of the "Extended" Logical Framework . . . . . . . . . . . . . . . . 75 Appendix 4. List of Indicators and Data Sources for ARE Projects . . . . . . . . . . . . . . . . . . . . 77 Appendix 5. Project Monitoring and Evaluation Costs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Appendix 6. Economic Evaluation of Competitive Research Grants: The Case of PROMSA, Ecuador . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Tables, Boxes, and Figures TABLE 1.1: COMPLEMENTARY ROLES FOR MONITORING AND EVALUATION . . . . . . . . . . . . . . . . . . . . . 2 TABLE 2.1: DIFFERENCES BETWEEN THE WORLD BANK LOGICAL AND RESULTS FRAMEWORK . . . . . . . 6 TABLE 2.2: THE WORLD BANK'S RESULTS FRAMEWORK APPLIED TO AN ARE PROJECT . . . . . . . . . . . 7 TABLE 2.3: DEFINITIONSAND EXAMPLES ON AGRICULTURAL RESEARCH AND EXTENSION PROJECT INDICATORS IN THE WORLD BANK LOG FRAME AND RESULTS FRAMEWORK . . . 11 TABLE 2.4: EXAMPLES ON OUTCOME INDICATORS DEPENDING ON THE PROJECT DEVELOPMENT OBJECTIVE AND THE PERCEPTIONS OF THE DIFFERENT STAKEHOLDER GROUPS INVOLVED INAGRICULTURAL RESEARCH AND EXTENSION PROJECTS. . . . . . . . . . . . . . . . 12 iii 3534-Rikka.pdf 6/20/05 2:51 PM Page iv TABLE 3.1: ARRANGEMENTS FOR OUTCOME MONITORING FOR "CHINA AGRICULTURAL TECHNOLOGY TRANSFER PROJECT" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 TABLE 3.2: THE MAJOR DATA SOURCES--MANAGEMENT INFORMATION SYSTEM, EXISTING AND REGULARLY MAINTAINED DATABASES AND VARIOUS SPECIAL STUDIES AND THE MAIN DATA COLLECTION METHODS . . . . . . . . . . . . . . . . . . 31 TABLE 4.1: MAJOR DIFFERENCES BETWEEN CONVENTIONAL AND PARTICIPATORY M&E . . . . . . . . . . 33 TABLE 4.2: DEGREE OFPARTICIPATION AT DIFFERENT STAGES OF PME PROCESS. . . . . . . . . . . . . . . 35 TABLE 6.1: KEY REFERENCE SOURCES ON THE THEORY AND EMPIRICAL TOOLS FOR ASSESSING THE ECONOMIC IMPACT OF RESEARCH AND EXTENSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 TABLE 6.2: STATISTICAL REGRESSION MODEL FOR SOLATING I PROJECT EFFECTS INIMPACT EVALUATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 TABLE 6.3: CALCULATION OF IRR FOR AN AGRICULTURAL RESEARCH AND EXTENSION PROJECT . . . 58 TABLE 6.4: SOURCES OF NFORMATION ON THE I MINIMUM SET OF PARAMETERS TO BEESTIMATED FOR A SOUND ECONOMIC IMPACT EVALUATION . . . . . . . . . . . . . . . . . 60 TABLE A1.1: POTENTIAL OF SELECTED INNOVATIVE INSTRUMENTS TO ADDRESS VARIOUS FISCAL AND NSTITUTIONAL I CHALLENGES ASSOCIATED WITH AGRICULTURAL RESEARCH AND EXTENSION SYSTEMS.. . . . . . . . . . . . . . . . . . . . . . . . 68 TABLE A3.1: THE "EXTENDED" LOGICAL FRAMEWORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 TABLE A5.1: A PROJECT COMPONENT TABLE FROM COSTAB 32. . . . . . . . . . . . . . . . . . . . . . . . . . . 90 BOX 1.1: WORLD BANK REQUIREMENTS FOR PROJECT AND PROGRAM MONITORING AND EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 BOX 1.2: KEY FEATURES OF MPLEMENTATION I -FOCUSED VERSUS RESULTS-BASED MONITORING . . . . 3 BOX 1.3: FOCUS ON NSTITUTIONAL I DEVELOPMENT OR PRODUCTIVITY IMPACTS. . . . . . . . . . . . . . . . . 4 BOX 1.4: COMPETITIVE RESEARCH GRANT PROGRAM IN ECUADOR. . . . . . . . . . . . . . . . . . . . . . . . . . 5 BOX 2.1: PARTICIPATORY INDICATOR IDENTIFICATION IN AN EXTENSION PROJECT IN MEXICO. . . . . . . 11 BOX 2.2: TQQ OF DEFINING INDICATORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 BOX 2.3: THE IMPLEMENTATION STATUS AND RESULTS (ISR) REPORT QUALITY CHECKLIST. . . . . . . 15 BOX 3.1: MANAGEMENT INFORMATION SYSTEM (MIS) PREREQUISITES FOR AN EFFECTIVE M&E SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 BOX 3.2: TECHNOLOGY OPTIONS IN DEVELOPMENT OF MANAGEMENT INFORMATION SYSTEM. . . . . . 20 BOX 3.3: KEY FEATURES OF THE MANAGEMENT INFORMATION SYSTEM (MIS) FOR THE MODERNIZATION PROGRAM FOR AGRICULTURAL SERVICE PROJECT (PROMSA) IN ECUADOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 BOX 3.4: PROJECT INFORMATION MANAGEMENT SYSTEM NETWORK OF NDIA S I ' NATIONAL AGRICULTURAL TECHNOLOGY PROJECT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 BOX 3.5: SCENARIO ON HARMONIZING M&E IN NATIONAL SECTOR MINISTRIES . . . . . . . . . . . . . . . 24 BOX 3.6: WHAT TO DO WHEN THE PROJECT DOES NOT HAVE BASELINE DATA?. . . . . . . . . . . . . . . 27 BOX 3.7: RURAL WOMEN'S EMPOWERMENT THROUGH AGRICULTURAL EXTENSION IN VENEZUELA . . 29 iv 3534-Rikka.pdf 6/20/05 2:51 PM Page v BOX 3.8: USING INTERVIEWS AND FOCUS GROUP DISCUSSIONS AT MID-TERM EVALUATION OF A FARMER-LED EXTENSION SERVICE PROVISION PROJECT . . . . . . . . . . . . . . . . . . . . . . . . . . 30 BOX 4.1: BASIC STEPS INPARTICIPATORY ACTIVITY EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . 37 BOX 4.2: USING CONVERSATIONAL INTERVIEWS WITH FIELD-FOCUSED APPROACH . . . . . . . . . . . . . . 38 BOX 4.3: MAIN TOOLS AND METHODS TO USE IN PME FOR ARE PROJECTS . . . . . . . . . . . . . . . . . 41 BOX 4.4: LESSONS LEARNED FROM THE UGANDA NATIONAL AGRICULTURAL ADVISORY SERVICES PROJECT'S (NAADS) PME PILOT . . . . . . . . . . . . . . . . . . . . . . . . . 42 BOX 5.1: SPECIFIC HIGHLIGHTSFOR M&E DURING ICR PREPARATION. . . . . . . . . . . . . . . . . . . . . . 46 BOX 5.2: POTENTIAL FOCUS FOR EVALUATION IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS/PROGRAMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 BOX 5.3: WHEN TO USE INTERNAL AND EXTERNAL REVIEWERS AND EVALUATORS?. . . . . . . . . . . . . 49 BOX 6.1: USING RANDOM SAMPLES IN EVALUATION OF MPACT OF I COMPETITIVE GRANT SCHEMES . . . 54 BOX 6.2: ESTIMATING INTERNAL RATE OF RETURN FOR SUBPROJECTS IN COMPETITIVE GRANT SCHEMES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 BOX 6.3: SUSTAINABLE FINANCING OF A COMPETITIVE FUND IN ECUADOR . . . . . . . . . . . . . . . . . . . 61 BOX A1.1: AGRICULTURAL INNOVATION SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 FIGURE 6.1: ESTIMATION STRATEGY IN PRONATTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 FIGURE A2.1: DIAGRAM OF M&E STEPS IN THE PROJECT CYCLE . . . . . . . . . . . . . . . . . . . . . . . . . 74 v 3534-Rikka.pdf 6/20/05 2:51 PM Page vi ACRONYMS AND ABBREVIATIONS ABF Annual Benefit Flow ACF Annual Cost Flow APL Adaptable Program Loan AIS Agricultural Innovation System ASTI Agricultural Science & Technology Indicators BA Beneficiary Assessment BP Bank Procedure CAS Country Assistance Strategy CDB Country Database CGIAR Consultative Group on International Agricultural Research CGP Competitive Grant Project CGS Competitive Grant Scheme CIAL Local Agriculture Research Committee CIMMYT International Maize and Wheat Improvement Center CRGP Competitive Research Grant Program CRGS Competitive Research Grant Scheme ER Expert Review FAO Food and Agriculture Organization FFS Farmer Field School GDB Global Database GDP Gross Domestic Product GEF Global Environment Facility ICR Implementation Completion Report ICT Information and Communication Technology IDA International Development Association IFAD International Fund for Agriculture Development IFPRI International Food Policy Research Institute INB Increment in Net Benefits IRR Internal Rate of Return ISNAR International Service for National Agricultural Research ISR Implementation Status Report MAKIS Modernizing Agricultural Knowledge and Information Systems M&E Monitoring & Evaluation MIS Management Information System NAADS National Agricultural Advisory Services NATP National Agricultural Technology Project vi 3534-Rikka.pdf 6/20/05 2:51 PM Page vii NGO Non-governmental Organization NRI Natural Resources Institute PDB Project Database OD Operational Directive OED Operations Evaluation Department OP Operational Policy PRSP Poverty Reduction Strategy Paper PAD Project Appraisal Document PCN Project Concept Note PCU Project Coordinating Unit PCR Project Completion Report PDO Project Development Objective PIMSNET Project Information Management System Network PIU Project Implementation Unit PME Participatory Monitoring & Evaluation PROMSA Modernization Program for Agricultural Services PSR Project Supervision Report PTD Participatory Technology Development QAG Quality Assurance Group R&E Research & Extension RD Recommendation Domain R&D Research & Development SS Special Studies TQQ Time, Quality and Quantity TT Task Team TTL Task Team Leader UEFC Competitive Fund Management Unit UOA Unit of Analysis vii 3534-Rikka.pdf 6/20/05 2:51 PM Page viii PREFACE AND ACKNOWLEDGEMENTS This Monitoring and Evaluation Good Practice Note was prepared by Riikka Rajalahti, Eija Pehu, and Johannes Woelcke. We would also like to thank Greg Traxler from Auburn University and Gustavo Sain from the Inter-American Institute for Cooperation on Agriculture (Economic Impact Analysis), and World Bank Consultants Remileku Rakey (M&E in the World Bank Project Cycle), Ghazali Raheem (M&E costs), and Lawrence Salmen (Beneficiary Assessment) for their contribu- tions to this report. The authors would also like to thank reviewers Jody Kusek (AFTKL), Gary Alex (USAID), Jim Hanson (Maryland University), Doug Horton (CGIAR), Willem Janssen (SAR), Indira Ekanayake (LCR), Richard Chisholm (EAP), Matthias Grueninger (MNA), David Nielson (AFR), and Derek Byerlee (ARD) for their time and useful comments. The World Bank's Sustainable Agriculture Systems Knowledge and Institutions (SASKI) Thematic Group is acknowledged for its financial support. viii 3534-Rikka.pdf 6/20/05 2:51 PM Page ix EXECUTIVE SUMMARY Monitoring and Evaluation (M&E) are integral tools for managing and assessing the efficiency and effectiveness of investments in agricultural research and extension (ARE) systems. However, moni- toring and evaluation of ARE project performance, outcomes, and impact has been a significant chal- lenge. Moreover, the increased focus of donors and borrowers on impact has resulted in a high demand for expertise in M&E. The objective of this Good Practice Note is to assist Task Teams of the World Bank and their col- leagues in the client countries develop and implement effective M&E systems for ARE projects and programs. The chapters in this Note provide a step-by-step guide for achieving that objective, with emphasis on the World Bank M&E requirements and the specific nature of ARE projects. Chapter 1 titled "Challenges for M&E Systems in Agricultural Research and Extension Projects" and Appendix 1 titled "Narratives of New Approaches and Instruments in ARE Systems" discuss the M&E challenges inARE projects; and provide additional information on the forces driving the changes inARE systems as well as on the typical approaches and instruments applied in ARE systems and projects. The focus of M&E in the Bank has shifted from monitoring implementation to tracking results. Results-based systems build upon and add to traditional implementation-focused systems emphasiz- ing project outcomes. Since 2003, the Bank has used the Results Framework, which is a simplified version of the traditional log frame focusing on (a) the Project Development Objective (PDO) and (b) intermediate outcomes expected from implementing each individual project component, which all in turn contribute to the achievement of the PDO. The Note emphasizes the World Bank's M&E Requirements at the design stage, namely the intro- duction of Results Framework and the associated requirements in the Project Appraisal Document (i.e., the final product of a successful project planning process). Chapter 2 describes the key issues in the Results Framework which are (a) development of a clear PDO and Project Components, and (b) the importance of developing measurable and realistic Project Outcome and Intermediate Outcome indicators. In addition, Appendix 2 summarizes the Bank's M&E steps throughout the Project Cycle and Appendix 3 provides further details and references on the Logical Framework. A list of indicators, with emphasis on institutional, social, and environmental sustainability, and data sources for ARE Projects are provided in Appendix 4. After designing the Results Framework and selecting the key performance indicators, the next step is to specify the M&E arrangements for data collection, reporting, dissemination, and use for decision- making. Chapter 3 presents the Results Monitoring Arrangements. Effective M&E requires the devel- opment of concise M&E Plan (see section 3.1 as presented in the PAD), the establishment of a management information system (MIS), an internal system for collection, analysis, storing, and dis- semination of pertinent information (see section 3.2) and good institutional and human capacity to ix 3534-Rikka.pdf 6/20/05 2:51 PM Page x implement the required actions (see section 3.3). Finally, sections 3.4 and 3.5 emphasize the impor- tance of data collection and use (including baseline), list helpful data sources, describe methodology for data collection and analysis, and indicate how much a good M&E system costs. Additional infor- mation is provided in Appendix 4. Participatory monitoring and evaluation (PME) is gaining importance in the development and mon- itoring of demand-driven ARE systems and projects. The basic idea in PME is to allow active involvement of key stakeholders in the M&E process in order for them to learn about and affect the process and impact of a development intervention. Participatory M&E can be used in concert with traditional M&E but should be considered a different process. If resources are limited, it is better to identify carefully when and how to apply PME rather than sacrifice the quality of the process and results generated. Chapter 4 provides further information on PME in ARE projects. Evaluation is the periodic assessment of the relevance, performance, efficiency, and impact (both expected and unexpected) of a development intervention in relation to stated objectives. Evaluation measures achievements in relation to institutional policies, Bank-wide program objectives, and the goals set for each operation. Evaluation at the Bank has two major dimensions: (a) self-evaluation that consists of an interim evaluation and a terminal evaluation undertaken at the end of a project (requirement for the Project Completion Report); and (b) independent impact evaluation usually undertaken by Operations Evaluation Department several years after final disbursement; which measures changes attributable to the project in terms of both direct and indirect causality. Chapter 5 presents the Bank evaluation requirements and evaluation of outcomes and impact. Chapter 6 high- lights the key steps in the economic impact evaluation of Competitive Grants Schemes (additional case study in Appendix 5) and describes the fiscal analysis recommended for ARE projects. x 3534-Rikka.pdf 6/20/05 2:51 PM Page 1 1. CHALLENGES FOR M&E SYSTEMS IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS 1.1 PURPOSE OF THE GOOD PRACTICE NOTE High rate of growth in agricultural productivity is essential to promote broad-based economic growth, reduce poverty, and conserve natural resources. In turn, productivity growth is based large- ly on application of science, technology, and information provided through national research and development systems and various extension and advisory services. Within this context, the success of rural development projects and programs has been shown to depend largely on direct stakeholder involvement in planning, implementation, and evaluation (Agriculture Investment Sourcebook). Monitoring and evaluation (M&E) are integral tools for managing and assessing the efficiency and effectiveness of investments in agricultural research and extension (ARE) systems. However, moni- toring and evaluation of ARE project performance, outcomes, and impact has been a significant chal- lenge. Moreover, the increased focus of donors and borrowers on impact has resulted in a high demand for expertise in M&E. To meet this challenge, the objective of this Good Practice Note is to assist Task Teams of the World Bank and their counterparts in client countries develop and implement effective M&E of ARE pro- gram performance, outcomes, and impact.1 Chapters 1 through 6 provide a step-by-step guide for developing an M&E system for ARE projects. 1. Challenges for M&E Systems in Agricultural Research and Extension Projects 2. The World Bank Results Framework Requirements 2.1. From the Log Frame to the Results Framework 2.2. Developing the Results Framework (including selecting indicators) 3. Data Collection, Reporting, and Dissemination Requirements 3.1. Monitoring and Evaluation Plan 3.2. Project Management Information System 3.3. Institutional and Human Capacity in Project M&E 3.4. Data Sources and Methodology for Data Collection and Analysis 3.5. Monitoring and Evaluation Costs 4. Participatory Monitoring and Evaluation 5. Evaluation of Outcomes and Impact in the World Bank 6. Economic Evaluation of Competitive Grants Schemes 1 3534-Rikka.pdf 6/20/05 2:51 PM Page 2 1.2 MONITORING AND EVALUATION REQUIREMENTS INWORLD BANK PROJECTS Monitoring and evaluation systems are designed to inform project management (World Bank Task Teams and client's project implementation unit) whether implementation is going as planned and whether corrective action is needed to adjust implementation plans. In addition, M&E systems should provide evidence of project outcomes and justify project funding allocations. The roles of monitoring and evaluation are summarized in table 1.1. The sources and explanations for the World Bank M&E requirements, as determined in the Operational Manual, are briefly explained in box l.1. In addition, the World Bank project cycle and the associated M&E requirements are summarized in Appendix 2. 1.3 EMPHASIZED FOCUS ONRESULTS IN THE WORLD BANK The focus of M&E has shifted from monitoring implementation to tracking results. Traditionally M&E systems were implementation-focused and included tracking of input mobilization, activities undertaken and completed, and outputs delivered. However, the implementation-focused approach Table 1.1: Complementary roles for monitoring and evaluation Monitoring Evaluation · Routine collection of information · Analyzing information · Tracking project implementation progress · Ex-post assessment of effectiveness and impact · Measuring efficiency · Confirming project expectations · Measuring impacts · Question: "Is the project doing things right?" · Question: "Is the project doing the right things?" Source: Alex and Byerlee (2000). BOX 1.1:WORLD BANK REQUIREMENTS FORPROJECTAND PROGRAM MONITORING AND EVALUATION The World Bank Monitoring and Evaluation requirements of both program and project lending are pre- sented in the Operational Directive 10.70 and Operational Policy (OP)/Bank Procedure (BP) 13.05. The Operational Directive (OD) 10.70 defines Monitoring as "the continuous assessment of project implementation in relation to agreed schedules and the use of inputs, infrastructure, and services by project beneficiaries. Evaluation is defined as "the periodic assessment of the relevance, performance, efficiency, and impact (both expected and unexpected) of the project in relation to stated objectives." In addition, OD 10.70 sets out the concept of M&E, and provides general guidance on the design and implementation of the information systems required for M&E activities. The OP 13.05 explains the rationale for Bank's supervision of Bank-financed projects, which include monitoring and evaluative review and reporting. It also explains the responsibilities of Task Teams (TT), and requires that the borrower and the TT agree on implementation arrangements which include M&E arrangements, and use of appropriate performance indicators. Source: World Bank Operational Manual 2 3534-Rikka.pdf 6/20/05 2:51 PM Page 3 does not provide managers, stakeholders, or policy-makers with an understanding of failure or suc- cess of the project in reaching the desired outcomes (Kusek and Rist 2004). Results-based systems build upon and add to traditional implementation-focused systems, emphasizing project outcomes. Box 1.2 illustrates some of the key features and differences between implementation- and results- based M&E systems. BOX 1.2: KEY FEATURES OF MPLEMENTATION FOCUSED I - VERSUS RESULTS-BASED MONITORING Elements of Implementation Monitoring (traditionally used for projects): · Description of the problem or situation before the intervention; · Benchmarks for activities and immediate outputs; · Data collection on inputs, activities, and immediate outputs; · Systematic reporting on provision of inputs; · Systematic reporting on production of outputs; · Directly linked to a discrete intervention (or series of interventions); and · Designed to provide information on administrative, implementation, and management issues as opposed to broader development effectiveness issues. Elements of Results Monitoring (used for a range of interventions and strategies): · Baseline data to describe the problem or situation before the intervention; · Indicators for outcomes; · Data collection on outputs and how and whether they contribute toward achievement of outcomes; · Timelines expressed such as at mid-term and end-term; · More focus on perceptions of change among stakeholders; · Systematic reporting with more qualitative and quantitative information on the progress toward outcomes; · Done in conjunction with strategic partners; and · Captures information on success or failure of partnership strategy in achieving desired outcomes. Source: Kusek and Rist (2004). 1.4 SPECIFIC CHALLENGES FOR M&E IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS A workable system for monitoring and evaluation of project performance and especially outcomes and impact is a challenge for all development programs. The particular problems facing project man- agement in the development of an M&E system for ARE projects include: 1. The natural uncertainty of research outcomes and technology adoption and difficulties in establishing timetables for technology development and dissemination. This is espe- cially true for basic research with very long time lags and indirect pathways to impact; 3 3534-Rikka.pdf 6/20/05 2:51 PM Page 4 2. The difficulties in measuring the diverse objectives of ARE programs, such as poverty reduction, natural resource conservation, food security, export promotion, economic growth, and reduction of social conflicts; 3. The problems related to cause-and-effect attribution of impact due to diverse external factors (e.g., rural credit institutions, input supply systems, product marketing sys- tems, macro-economic policies), which are outside of direct project control; 4. The lack of reliable national data and lack of awareness and managerial demand for M&E data; 5. The lack of national capacity for M&E and low financial priority for M&E systems in the context of low agricultural technology program funding and tight government budgets; 6. The dilemma of whether to develop an M&E system specific to the needs of the Bank project, or to the broader needs of the (national) ARE program and institution(s). Both are related and similar, but each has somewhat different requirements; and 7. What is the right balance in project M&E? Should the M&E system track both the institutional development and the impact on productivity or prioritize based on the cost of M&E? See box 1.3 for further discussion. BOX 1.3: FOCUS ON NSTITUTIONAL I DEVELOPMENT OR PRODUCTIVITY IMPACTS Productivity impact is defined broadly to include any impact on the agricultural production system and rural development, including environmental, economic, and social welfare. Institutional development refers to the creation or the capacity of an institution to reflect systematically and rigorously upon its role and function, and better enable them to carry out their responsibilities. It reflects an attempt to introduce change and development in the way the institution is organized so that it is better able to meet its mission. Agricultural R&E projects typically seek productivity impacts, but these usually require institutional development to bring about the desired impacts, and provide a basis for continued innovation and sus- tainable impact on productivity (e.g., a multi-stakeholder priority setting mechanism for NARS). Investments into institutional development, in turn, are not justified if they do not have productivity impacts. However, a focus on quick productivity impact without the institutional development to sustain it over time is not recommended. The majority of Bank-supported projects combine both institutional and productivity impacts. Hence, the inclusion of these two types of impacts needs to be considered when developing the Project Development Objective (PDO) and the M&E system indicators. Source: Alex and Byerlee (2000). 1.5 NEW APPROACHES AND NSTRUMENTS I IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS The recent changes in the agricultural research and extension systems (i.e., decentralization, pri- vatization, stakeholder participation, and increasingly separation of the functions of financing 4 3534-Rikka.pdf 6/20/05 2:51 PM Page 5 and delivery), as a response to satisfy the changing global and local demands2 placed on the ARE system, have resulted in a gradual transition from supply-driven technology generation toward building a more demand-driven agricultural innovation system (AIS). Agricultural innovation system essentially links multiple sources of innovation and uptake pathways along the continu- um from basic research to technology adoption recognizing the innovative capacity of all stake- holders from producers to the marketplace. Thus, the focus is not only on the science suppliers but rather on the totality of actors that are involved in innovation (see Appendix 1 for more details on AIS). Although the ARE systems have moved toward more responsive and effective AIS, there has also been a shift to a clearer focus on poverty reduction which includes new partnerships for inclusion of marginalized groups of people and an increased emphasis on co-financing with the private sector. In this context, a number of instruments and approaches, such as contracting for services, competitive research grants (CRGP), participatory and farmer-to-farmer R&E, and enhanced use of Information and Communication Technology (ICT) (for details, see Appendix 1), are being introduced into many ARE systems. These approaches and instruments are often used simultaneously. As an illustration box 1.4 describes the complex nature of a CRGP arrangement in the Bank-supported Agricultural Research Project in Ecuador. BOX 1.4: COMPETITIVE RESEARCH GRANT PROGRAM INECUADOR The Program for Modernization of Agricultural Services in Ecuador finances a competitive research grant program (CRGP) that has funded 112 research and educational subprojects. The program has supported strategic, applied, and adaptive research and education in a diverse range of thematic areas. These include: organic agriculture, agro-forestry, agro-industry, variety development, management of plant genetic resources, soil and water conservation, integrated pest management, animal production, and marketing. The program introduced a new research culture and brought new organizations into the research system. Research and educational subprojects are being conducted country-wide and are being executed by 45 different public and private organizations. These include: public and private universities, NGOs, pro- ducer organizations, private enterprises, and the Instituto Nacional Autonomo de Investigaciones Agropecuaris (INIAP). Subproject research costs averaged US$ 116,000, of which 54 percent is financed by grants and 46 percent by executing agencies. By leveraging co-financing for subprojects, the program helped to increase national research funding by 92 percent, amounting to approximately 0.54 percent of agricultural GDP. Source: World Bank (2004a). Agriculture Investment Sourcebook. Additional M&E challenges resulting from the new implementation context include: · A high level of pluralism--multiple partners, including farmers, farmers' organizations, and various public and private sector actors; · Multiple project components, particularly with CRGPs and multiple timelines for imple- mentation; 5 3534-Rikka.pdf 6/20/05 2:51 PM Page 6 · Non-formal communication and technology transfer routes, which are desirable but more difficult to track; · Highly decentralized decision-making and information flows; and · Variable levels of capacity among partners involved in M&E. 6 3534-Rikka.pdf 6/20/05 2:51 PM Page 7 2. THE WORLD BANK RESULTS FRAMEWORK REQUIREMENTS 2.1 FROM THE LOG FRAME TO THE RESULTS FRAMEWORK In 2003 the Bank shifted from the log frame as a mandatory requirement in project M&E to a two- pronged strategy comprising of the results framework and the arrangements for results monitoring. The results framework is a simplified version of the traditional log frame focusing on (a) the Project Development Objective (PDO) and (b) intermediate outcomes expected from implementing each indi- vidual project component, which contributes to the overall achievement of the PDO. The added empha- sis on component outcomes helps focus project design and management on results. It also forces the M&E system track these outcomes and impacts rather than simply reporting on production of outputs. Table 2.1 summarizes the differences between the traditional log frame and the results framework. Table 2.1: Differences between the World Bank Logical and Results Framework World Bank Log Frame World Bank Results Framework Goal: long-term, widespread -- - - improvement in society _ but only at Outcome: intermediate effects of Project Development outputs on clients Objective (PDO) level _ both at PDO and component level Outputs: products and services - - produced by the projects -- Activities: tasks personnel undertake to transform inputs to -- - - outputs Inputs: financial, human, -- - - and material resources Note: - - = included in the framework and _= not included in the framework. Source: Adapted from Binnendijk 2000. Project Appraisal Document (PAD) is the final product of a successful project planning process. For M&E related matters, the PAD requires the development of the results framework (as described in chapter 2 of this Note) and description of the M&E arrangements (in chapter 3). The Results Framework has the following structure: 7 3534-Rikka.pdf 6/20/05 2:51 PM Page 8 · Project Development Objective and Project Component Statements; · Indicators for PDO outcome and intermediate component outcomes; and · Explicit statement on how to use the outcome information. The structure of the results framework as presented in the PAD and an example on a hypothetical ARE project are shown in table 2.2. Table 2.2: The World Bank's Results Framework Applied to an ARE Project Project Development Objective Outcome Indicators Use of Outcome Information Statement of Project List of outcome indicators at Specify use or purpose of monitoring Development Objective PDO level specific indicators and use of findings (PDO level) Agricultural producers increase For example, percent or number of If the rate of technology adoption by the adoption of profitable and women and men producers that women and men lower than x% by the environmentally sound have adopted environmentally specified target dates, the components technologies sound technologies by year x 1 and 2 must be reviewed for efficiency Intermediate Outcomes Outcome Indicators for each Component for Components Use of Outcome Monitoring Statement of outcomes List of outcome indicators Specify use or purpose of monitoring per component per component specific indicators and use of findings (PDO level) Component 1. Extension service For example percent of women and If the rate of satisfaction is lower than providers have an improved men producers satisfied with access x% by the specified target dates, the understanding of client needs and to and quality of extension services activities under component 1 will be how to respond to them adjusted as needed Component 2. National research For example, percent of priority If the proportion is lower than institutes have improved capacity research funds allocated to research envisioned, the situation will be to conduct research in identified in priority areas. analyzed and corrected. priority areas Source: Authors. 2.2 DEVELOPING THE RESULTS FRAMEWORK When developing the framework, pay attention to the following: Develop a strong and clear project development objective A strong and clear Project Development Objective is essential for good project design. The PDO is expected to illustrate the principal outcome of a change in end-user behavior, (i.e., that of the proj- ect's primary target group). One of the main problems identified in the Bank's project design has been lack of clarity and focus in the PDO. PDOs are expressed either too high (e.g., at the Country Assistance Strategy level) or too low (e.g., at the activity level) and sometimes are not well articu- lated (e.g., mixing goals, objectives, and strategies). 8 3534-Rikka.pdf 6/20/05 2:51 PM Page 9 Questions that help develop a clear and strong project development objective (a)Who is the key recipient of project benefits? (b)After the close of the project, what problem has been solved for the primary target group? (c)What will the target group be doing differently after the project? Pay close attention to the following: · Focus on the outcome that the project can directly influence, given its duration, resources, and approach; · PDO should not encompass longer-term outcomes that depend on efforts outside the proj- ect's scope; · Keep PDO short, focused, and measurable; and · PDO is not a restatement of the project's components or outputs. Practice: In ARE projects the primary target group usually consists of the agricultural producers. An example of a typical Project Development Objective is "Agricultural producers increase the adoption of profitable and environmentally sound technologies." Another example could be "Small-holder farmers in rain-fed areas of Allura actively participate in the development of and use of research pro- grams results." Make sure that the project components contribute to achieving the project development objective The results framework presents a statement of the key project components, the success of which determines the achievement of the PDO. · To track the success of the different components, the expected outcomes from each individual component (i.e., intermediate outcomes) need to be linked to the achievement of the PDO. · The outcomes of each project component should clearly specify the immediate target group that receives the benefits from the component implementation. Thus, they describe the changes for the secondary target group or the skills or capacities the secondary target group must have acquired to better serve the primary target group. Practice: Agricultural R&E projects very often have several secondary target groups. A typical Bank- supported ARE project would include research and extension staff and institutions as secondary tar- get groups. If the project improves their performance (e.g., through human capacity building or institutional development), they will better serve the primary target group (e.g., small scale farmers and women's groups). Select indicators for the project development objective and each project component Why indicators? Indicators are one of the crucial aspects of a project design. They are quantitative and qualitative variables that provide a simple and reliable means to measure achievement, reflect changes connected to an intervention, or help assess the performance of an organization against the stated targets. Carefully selected indicators are needed to measure the success of achieving the PDO and the out- come(s) of each project component. Specifically, indicators can help managers make informed and 9 3534-Rikka.pdf 6/20/05 2:51 PM Page 10 timely decisions by way of a balanced presentation of financial and operational performance, and consequent outcomes and effects; and a summary of complex information "at a glance." Tracking indicators can provide continuous feedback and streams of data throughout the lifetime of the proj- ect (for more details see section on performance indicators). Establishment of the baseline and tar- get values is essential for tracking the achievements. Different indicator levels and types Indicators are derived from log frames and results framework and provide for measuring perform- ance at two levels: (1) implementation performance indicators and (2) project impact indicators. See Table 2.3 for definitions and examples of different indicator levels. 1. Implementation performance indicators measure program inputs and outputs. Tracking input use and direct project accomplishments often requires multiple indicators, but they are relatively easy to measure. 2. Project impact indicators measure outcomes and impacts on social goals. Outcome and impact indicators are more difficult to measure, as they track the results of project activities and outputs, but are influenced by many other factors. The Results Framework requires only the description of outcome indicators. It is important to make a distinction between output, intermediate, and outcome indicators. · The indicator associated with the Project Development Objective describes a change in end- user behavior (e.g., farmers' access to, use of, or satisfaction with goods and services) where- as the indicator of each project component describes the changes for the secondary target group (e.g., improved skills or capacity among staff of research and extension organizations). · It is also important to make a distinction between an outcome and output. Outcomes are intermediate effects of outputs on target group and thereby address the issue of uptake (i.e., coverage or adoption) of project outputs resulting from project activities. · This difference is also reflected in the indicators. An outcome indicator could be "Percent of trainees implementing new practices" but it would not be "number of trainings conduct- ed." An illustrative list of key indicators, with particular focus on Competitive Research Grant Schemes and contracting extension services, is included in Appendix 4. Indicators must be defined for each step of the results framework (Project Development Objective and component outcomes). If an extended log frame is developed, indicators must be defined for goal, Project Development Objective, component outcomes, as well as outputs and inputs. Key questions to address when selecting and developing indicators Prior to defining the indicators, ask the following questions: 1. What is the development objective of the project? The selection of indicators depends on the Project Development Objective and the focus of the research and extension program (i.e., poverty focus, agricultural market development, institutional development, productivity change, capacity building, and environmental sustainability. Table 2.4 includes examples on outcome indicators from the perspectives of different stakeholders. 10 3534-Rikka.pdf 6/20/05 2:51 PM Page 11 in of and total or for client reflect are response the year services. and of Framework. research R&E vary needs technical and changes findings. developed, problems, be of train number similar may institutional per of of of to and These show and posted, and esults are than new could R time. system new These institutional research agents staff, curricula (Secondary building outcomes themselves, by number and other usually lag or reviews delivery on agents projects. address efficiency conducted constructed, person-months equipment, responsiveness or indicators design. to peer indicators trained, Frame factors technology funding, based extension research measured service with disbursed, Indicators) established. indicators the institutions of extension Log Development impact extension clients. be ability train many considerable development project programs, productivity, attitudes, with policy laboratory and a on the outcome graduates shift certified to funds and by in with may staff, of offices of assistance. Bank Group) with satisfaction trained. proposed World Institutional Target Sector-level research affected programs, only Institutional depending the changes institutional relations Changes relationships, stakeholder programs, collaborative satisfaction government Component number client Extension--Workshops extension regional Research--Laboratories workshops staff Education--faculty numbers training. Examples: construction, technical the of in of may of or little include of range (i.e., (for in day poverty rates be or says work. with rate in in investments deforestation program may wide results farmers field the and Indicators a (adoption) in change may of projects new from listeners. of social trends two behavior adoption. include technology or completed. have in of investment. magazine building Project associated reduction new decreases, change further or completion Indicators number answered, or activities equipment, (Primary may reflect change of decreases derived are positive cost extension but nutrition. effects rates campaign projects that of significance productivity, technology, often behavior reforms) include or disbursed, Extension Indicators indicators, radio initial because questions research newspaper Oriented technology. outcomes behavior options. of increases, new may number as planned, laboratory and indicators factor improved on production adoption and investment, and as funds indicators activities--change of policy quality assistance. Group) yield to and farmer level and possible effects of the erosion, Research Productivity Target Goal agricultural increased return environmental rates, Productivity possible program adoption) Examples adoption made management Initial include or effects technology Extension--Data reports trained, attendance, implemented, readership, Research--Research Challenging completed about Examples: construction, technical be on as due the the each goods Agricultural in of project cannot project Objective and of outputs groups activities relatively (2000). the of the on works, PDO Framework change indirectly to control project. usually outputs target performance indicator/s indicator(s). physical Bank-financed project in of project. Byerlee which are of a effects full outputs. the of Examples and and of focuses Results to of related Development funding, the and measure and or life under quality change by goal project associated framework, M&E Alex measures indicators project, institutional the closely not the of Project outcome products control include in with from Frame the intermediate are results an the level indicators are only these to direct input provided Definitions during are but reflect behavioral the has Log in in are in Goal They changes within development "adoption" frame, outcome output quantitative and Adapted 2.3: as an are services the log Level Sector broad contributes. Changes attributable measured project. Outcomes clients. activities, project. Outcomes produced, well to In has whereas component Outputs and Project simple completion. Budget and Source: Table 11 3534-Rikka.pdf 6/20/05 2:51 PM Page 12 Table 2.4: Examples on Outcome Indicators Depending on the Project Development Objective and the Perceptions of the Different Stakeholder Groups Involved in Agricultural Research and Extension Projects Focus of Stakeholder in ARE project the Project Donor-- Borrower-- Research or Farmer or Development World Ministry of Extension Academic Private Producer Objective Bank Agriculture Agency Institution Sector Group Poverty x% increase x% increase x% increase x% increase Sales of crops Increase in reduction in poor small in poor smal in farm in farm increased by number of holders' lholders' income income x% meals with household real household meat or fish income income Productivity x% increase in x% increase x% of farmers Number of Input sales Staple food change1 small holder in small have access to staff with increased x% yield increase crop pro- holder crop improved improved by year x duction and production varieties technical farm income skills Institutional Number of Number of Generation Number of x% of the Extension development2 regions regions and transfer of staff in farmers have agents visit covered by having good quality collaborative access to rural good quality decentralized technologies ARE company communities ARE services ARE services to x% of programs inputs and/or frequently farmers services Market Farmers' Farmers' Demand for Change in Number of x% higher development access to access to high-value farmers' production price from markets markets commodity ability to contracts with vegetables improved by improved by research market farmer groups x% x% increased products increased Capacity x% increase in x% increase in x% increase High level of Number of x% increase building farmer groups autonomous in partici- participation contracts with in farmer participation in farmer patory among farmer groups group decision- groups' technology farmers increased membership making in the formation in development area the area Environmental x% of farmers x% of farmers x% increase in Number of x% of farmers Reduction in sustainability adopting adopting generation of trainings to trained in safe pesticide environ. sound environ. sound environmen- extension staff pesticide use applications production production tally sound in environ. or increase in practices practices technologies sound land under practices no-till Source: Riikka Rajalahti, World Bank. Note: Bank-supported agricultural R&E projects typically focus on two objectives: (1) institutional development to strengthen the institutions and technology transfer system necessary to develop and disseminate improved technologies and management practices in the agricultural sector; and (2) productivity change due to technological innovation introduced through the technology system. 12 3534-Rikka.pdf 6/20/05 2:51 PM Page 13 2. Who are the stakeholders? As evident from table 2.4, indicators need to be defined and monitored together with the relevant stakeholders. Different stakeholders have different perceptions on desired outcomes, which need to be tracked with different indicators. 3. What do we really need to know? To save time and resources and to avoid confusion, it is important to carefully think through the real and relevant information that the stakeholders will need. A good prac- tice example on participatory stakeholder consultation and indicator selection in farmer-to-farmer extension project is shown in box 2.1. BOX 2.1: PARTICIPATORY INDICATOR IDENTIFICATION IN AN EXTENSION PROJECT IN MEXICO In a farmer-to-farmer extension program in Mexico, the project team followed these steps to develop indicators: · Define broad indicator areas (based on PDO) · Select currently used indicators for these areas · Define stakeholder groups · Select stakeholder groups to be consulted · Develop indicators with different stakeholder groups · Test the developed indicators across different stakeholder groups to assess their significance to others and effectiveness at indicating change · Agree on a priority list among indicator options · Carry out fieldwork to gather data for indicators · Create lists of indicators for full evaluation use with a focus on indicators with specific impor- tance for different actors with a limit, (e.g., three key indicators for each stakeholder group) The program team identified the range of different institutional and individual actors who affect and are affected by the project. They then prioritized three stakeholder groups to be consulted for indicator development in this trial phase: (a) farmers (participating and non-participating), (b) farmer-extension agents and (c) funding agencies. The research team initially proposed seven indicator areas. These were eventually narrowed down to four, based on the groups' objectives: (1) changes to local, regional, political, and sectorpractice and policy (e.g., level of dependence on external resources, involvement of local people, growth of local institutions, and changes in policy and practice); (2) dissemination impacts: extension to other localities or regions (e.g., hor- izontal and vertical linkages with other projects, agencies and NGOs beyond the region); (3) changes to the roles of the individuals in the project (primarily the coordinator, outside advisors, immediate project partic- ipants, and NGO staff); and (4) changes in the institutional structure (within and beyond the actual project). Source: Baluert and Quintanar (2000) in IFAD 2003. 4. What is the most efficient method of collecting this information? To save time and resources, find a balance between the information needs and the efforts required to collect the information. Each indicator requires arrangements for monitoring--an M&E plan; and inclusion in the Management Information System. The effort expended should match the improvement gained in decision-making. 13 3534-Rikka.pdf 6/20/05 2:51 PM Page 14 Not all potential indicators are practical. They may be either too difficult or too expensive to gather the data. For example, measuring actual rural income levels may be difficult or costly. Instead, proxy indicators3 such as the number of community bicycles or the number of televisions may be a more practical, if less precise, indicator of the general trend. The client of the project is a useful source of suggestions for proxy options. See chapter 3 for further information on data collection. Principles for Selecting Indicators Although the key questions listed above guide the selection and development of indicators, sound indicators must also be characterized by TQQ: time (time-bound accomplishment), quantity (numer- ically measurable), and quality (what level of quality or degree of achievement is desired), respec- tively. See box 2.2 for an example on indicator development. BOX 2.2: TQQ OF DEFINING INDICATORS 1. Set Time: Access to agricultural extension services for poor farmers in Quynh Luu District is improved by the year 2007 2. Set Quantity: Access to agricultural extension services for poor farmers in Quynh Luu District is improved by x% from the baseline by the year 2007 3. Set Quality: Access to good quality agricultural extension services for poor farmers in Quynh Luu District is improved (where good quality is defined as . . . ) by x% from the baseline by the year 2007 Source: Riikka Rajalahti, World Bank. The TQQ serves as a general good practice guideline for selecting indicators. More detailed princi- ples for selecting key performance indicators require that the indicators are: · Relevant to the development objective of the project; · Unambiguous as to definition, valid and targeted in reflecting changing conditions, and consistent and verifiable in measurement; · Meaningful and of interest to project management and stakeholders; · Significant and selective in that the monitoring system should limit the number of indica- tors to a manageable number; · Practical in terms of the ability to collect data in a timely and cost-effective way with a rea- sonable degree of reliability; · Informative of project impact disaggregated by gender,4 poverty status, age, ethnicity, or other characteristic of the client group; · Acceptable to the Borrower, the Bank, and other stakeholders; · Quantifiable to the extent possible, or, if qualitative, based on a common understanding of what constitutes success; and · Responsive in reflecting changes due to project activities and sensitive in demonstrating change in as a short time period as possible. 14 3534-Rikka.pdf 6/20/05 2:51 PM Page 15 Key lessons in developing indicators 1. Involve partners and target communities in the selection of indicators. 2. Avoid collecting too many indicators (e.g., one or two per Project Development Objective and 1 to 3 per each project component). Look for coherence in the selected set. 3. Select indicators that are reliable proxies. 4. Look for unanticipated, negative and positive program or project effects. Decide how tracking results would guide project implementation The Results Framework includes a column titled "Use of Results Information." · This column describes when and how to take corrective action if there are problems in meeting the agreed targets. Clear and time-bound targets for the selected indicators mea- sured throughout the project lifespan to support management and decision-making are needed to reap the full benefit of this information. · The level of detail and flexibility in these guidelines can vary. An example of a rather detailed guidance is: If the rate of women and men farmers' satisfaction with access to services is less than 50% two years after the implementation, the activities will be reviewed and adjusted. Cover additional M&E features in the text of the project appraisal document The results framework does not explicitly capture the CAS or sector goals and output levels, nor the detailed project component activities and input levels. However, Bank-supported projects are required to articulate a clear alignment between the project and higher order strategic, program, or sector outcomes to which the project contributes. The "Strategic Context" section of the PAD describes how the project contributes to these higher level objectives. Furthermore, project inputs, activities, and outputs are now required to be captured in the main text of the PAD. The results framework does not explicitly capture the critical assumptions (factors that the project can not or does not want to control but which are essential for reaching the desired outputs and out- comes of the project). These critical risks, assumptions and possible controversial aspects need to be discussed in the main text of the PAD. Practical implications Although it is mandatory to develop the results framework and the arrangements for results moni- toring, the project team may also develop full log frame that introduces an added level of detail for component inputs, outputs, and outcomes. · The extended log frame approach may help the project team and the client to precisely define the different components, their inter-linkages and contributions to the Project Development Objective.5 · The extended log frame can be a useful tool in facilitating discussions and development of the project design with the national team. · It may also assist and complement project management, and may support the development of a sustainable M&E system beyond the lifetime of the project as it is often a better fit with the requirements for M&E in the client countries. See Appendix 3 for an example of an extended log frame. 15 3534-Rikka.pdf 6/20/05 2:51 PM Page 16 Supervision and monitoring of the project After the project and the results framework have been designed, the M&E arrangements specified and agreed on, it is essential to pay adequate attention to implementation and supervision of the over- all project as well as monitoring in particular. Responsibility for supervision of a Bank-financed project rests with a task team (TT) assigned to the project. Preparation for supervision is a joint task with the client and begins during project preparation, when the TT and client staff agrees on arrangements for project implementation. These arrangements include: (a) monitoring and evaluating of progress in project implementation and achievement of devel- opment outcomes, with a common set of criteria to be used for project monitoring and evaluation; (b) use of performance indicators; and (c) reports such as audit reports, Project Monitoring Reports (PMRs), and other reports required for effective project monitoring, evaluation, and disclosure--to be provided by the client, along with their outlines, content, and format. Unfortunately, supervision missions frequently do not pay adequate attention to M&E. To ensure that M&E will be an integral part of the project activities, emphasis of the importance of the following issues is useful during the project launch workshop(s) and later during the supervision missions. These issues are: · emphasis on sufficient M&E capacity from the beginning of the project; · timely establishment of the project M&E system; · collection and use (by TT, PIU, management, and government) of project M&E data from the very beginning; and · the need to remedy the identified deficiencies in a speedy manner. The TT is expected to prepare the "Implementation Status and Results (ISR)" report, formerly known as "Project Supervision Report (PSR)." The ISR is a result of streamlining of the PSR to make it a more effective tool for overseeing the implementation of projects and monitoring their out- comes. A new module has been introduced for reporting on project outcomes that can be monitored during implementation. See box 2.3 for details on the ISR quality checklist on Project Development Objective, Outcomes and associated indicators. 16 3534-Rikka.pdf 6/20/05 2:51 PM Page 17 BOX 2.3: THE IMPLEMENTATION STATUS AND RESULTS (ISR) REPORT QUALITY CHECKLIST The Implementation Status and Results Report require reporting on the following issues related to the Project Development Objective (PDO), Outcomes and associated indicators: 1. Indicators for PDO and Intermediate Outcomes · At least one indicator entered for PDO and Intermediate Outcomes. · Baseline values entered for all indicators. · Target values entered for all indicators. · If project is two or more years into implementation, is any useful information entered on indica- tors' progress to date? · Explanation if quality of outcome information is rated poor or is unavailable. 2. Project Development Objective and Outcomes · PDO rating needs to be consistent with other information in ISR. · PDO rating and explanation needs to be adequately justified taking into account: implementation performance, major risks, achievement of major outputs, achievement of intermediate outcomes, extent to which PDO and project design remain relevant. · If PDO rating is unsatisfactory, relevant issues and actions need to be presented. Source: Operations Policy & Country Services: Implementation Status and Results Report. http://intranet.worldbank.org/WBSITE/INTRANET/UNITS/INTOPCS/INTINVLENDING/0,,contentMDK:20296324~hlPK: 430750~menuPK:388717~pagePK:64137152~piPK:64136883~theSitePK:388707,00.html 17 3534-Rikka.pdf 6/20/05 2:51 PM Page 18 3. DATA COLLECTION, REPORTING, AND DISSEMINATION REQUIREMENTS After key performance indicators have been selected, the next step is to specify the M&E arrange- ments for data collection, reporting, dissemination and use for decision-making. The project team together with the partners must identify the following: 1. Data collection arrangements, including data sources and the reliability of the infor- mation provided and associated costs and responsibilities; and 2. Institutional issues, including how M&E complements and assists project manage- ment, what are the implications for institutional responsibilities and would participato- ry M&E lead to capacity building of communities involved; 3. Human and institutional capacity, including extent of local capacity for M&E and the need for capacity building and the associated costs. 3.1. MONITORING AND EVALUATION PLAN Data collection arrangements require the development of a comprehensive monitoring and evalua- tion plan and a management information system (MIS). An important step in developing the plan and MIS is the identification of data collection sources and methods. Development of a comprehensive monitoring and evaluation plan becomes an essential part of the project preparation and Project Appraisal Document. Elements of the M&E plan should include: · What? Type of information and data to be consolidated; · How? Procedures and approaches for M&E including methods for data collection and analysis; · Why? Identify how the collected data will support monitoring and project management; · When? Frequency of data collection and reporting; · Who? The responsible focal points/resource persons for M&E; their responsibilities and capacities to conduct required M&E functions; and · Where? Information/data flow chart from M&E focal points/resource persons (implement- ing agencies); data flow from decentralized levels to the central M&E unit. The above questions have to be carefully assessed when designing the project. Inadequate planning for data collection and use has been one of the main challenges in the Bank's project design and preparation--with negative consequences for timely implementation and project management as well as M&E. The Project Appraisal Document requires a table for Arrangements for Monitoring' an example of the China Agricultural Technology Transfer Project is presented in Table 3.1. It provides the scope to compare baseline information with target values, and to specify data collection frequen- cy and methods as well as reporting responsibilities. 18 3534-Rikka.pdf 6/20/05 2:51 PM Page 19 for of PPMO experts. Collection and and Team Team/ Team/ Reporting Data Responsibility and PPMOs/CPMO PPMOs/CPMO Expert national international Consultant/PPMOs (Shaanxi Heilongjiang) PPMOs/CPMO Expert PPMOs/CPMO Expert Shaanxi of Collection survey by records. set at team Collection a Data assessment questions. survey surveys Instruments on Team and Data Expert Household Provincial Expert based guiding Visitor conducted fairs technology markets. Member Expert Assessment Assessment the Project" Reports closing. at mid- convened Transfer and and and closing. and reports entry, Team and mid-term Technology Frequency Survey project term Annual Expert at Annual Mid-term closing Mid-term closing YR5 sites 60 10 70% 30% 60 70 Satis- factory Agricultural Project YR4 Values "China Transfer YR3 sites for 20 4 70% 15% 60 70 Satis- factory Target YR2 Technology Monitoring YR1 Agricultural 6 Outcome project project project Baseline Determined at entry Determined at entry Determined at entry 0 0 for Document: of in the targeted for in in in size. to of farmers. the assessed of Appraisal Indicators investment One: displayed centers successful successful Team. models sites Two: Three: successful by farm satisfaction as aligned strategies of measured of Arrangements increase particularly poor rate by as income of Team indicators associations provision and Expert Project 3.1. component 7 fairs provincial Outcome Company-farmer assessed exhibition member the Percent farmer farmers, resource Government in partnership non-project province. Percent components/sub-projects as Expert Results each Component Adoption technologies in and Component Percentage farmer service members by rating Percentage sub-projects by Component Investments with policies Source: Table 19 3534-Rikka.pdf 6/20/05 2:51 PM Page 20 3.2. PROJECT MANAGEMENT INFORMATION SYSTEM Apart from and in addition to the above described data requirements to satisfy the World Bank's information needs on project progress, effective project M&E requires the establishment of a proj- ect management information system (MIS), the project's internal system for collection, analysis, storing, and dissemination of project information. The new ARE instruments, such as competitive research grant programs for research and extension, have increased the importance of an effective MIS due to significant requirements on tracking of implementation and amount of funding allocation for several different subprojects. For Bank- financed projects, the MIS should be described in the Project Appraisal Document as the project's internal data management system. What is an MIS? The primary purpose of an MIS is to support management in making timely and effective decisions for planning, monitoring, and managing the project. An MIS is essentially a system that uses formal- ized procedures to provide management at all levels with appropriate information from internal and external sources (Vernon 2001). MIS generally consists of accounting software and a database management system for planning and non-accounting information. Besides software and hardware, MIS comprises four elements: (a) the actors who take decisions on the project; (b) the data and information that is useful for decision-making; (c) the procedures that determine how the actors relate to the data; and (d) the tools that facilitate the collection, analysis, storage, and dissemination of the data (Lecuit et al. 1999). What should the MIS for ARE project do and be like? One essential feature of any MIS is that it should operate using the regular client processes as much as possible. Thus, design and planning of the system requires systematic dialogue with the stakeholders. Other key desired features of an MIS in general, and of a Competitive Research Grant project in par- ticular, are the following (Srivastava et al. 2003) (for an example on MIS pre-requisites for an effec- tive M&E system, see box 3.1): 1. The accounting system should be linked with the subproject monitoring; 2. The MIS tools should be organized modularly (different functions of the system are managed by distinct modules, which in turn are integrated into a common, central database); 3. The MIS should adapt to the decentralized organization and operation of the Competitive Grant project; 4. Any new project components should be managed by the MIS in order to prevent the multiplication of management tools; 5. The MIS should facilitate the management of impact information; 6. MIS tools should facilitate decision making; and 7. Information management should be secure. 20 3534-Rikka.pdf 6/20/05 2:51 PM Page 21 BOX 3.1: MANAGEMENT INFORMATION SYSTEM (MIS) PREREQUISITES FOR AN EFFECTIVE M&E SYSTEM A computerized Management Information System (MIS) is the window that captures the quantitative data within the M&E system. It will capture both progress and process data for concurrent monitoring of activities at every level within the project. Starting point: The starting point of the design of an MIS is to identify the inputs, the outputs, and the process to be followed. This should then be followed by a clear and lucid description of the project cycle and the results and/or logical framework that includes an institutionally defined set of indicators. Indicators can be categorized as progress (P) indicators, physical progress indicators (PP), and Financial Progress indicators (FP). Process indicators can be classified as including input, output, and outcome indicators. Critical elements in the design of an operational MIS · Census Data or other national data on population along with geocode information on the classi- fication by village, district, or other administrative borders need to be available for uploading into the MIS. Demographic Data with habitation, income, and poverty level data at the lowest level should also be available for loading directly into the MIS. · Reports need to be organized as being accessible to the general public and those available for project management purposes, only. · In today's web connected world a Central Server that is either owned or leased needs to be secured. All of the data will reside on this central server; access to this data will be based on pre- defined protocols and compartmentalized rights of use. · Hardware and software and peripheral equipment will be connected to the central server in order to access and process information. · Trained personnel will be needed at each level to manage and operate the equipment. At each level within the project structure a minimum of two persons (coordinator and analyst) will be required to input, process, report, and access information and data for that particular level. Source: Ghazali Raheem, Consultant, World Bank. How to develop MIS for ARE? Lecuit et al. (1999)8 provided detailed guidelines for MIS for social funds, with principles, design features, and terms of reference, that are useful also for ARE proj- ects-- especially those using competitive funds and contracting. Another good source is by Vernon (2001) on MIS for agricultural research organizations. When developing a MIS, the following issues require special attention: 1. Design and planning--the first steps: · Ensure stakeholder dialogue in the design and planning. · Take time for needs assessment, find out about local information technology capac- ities, and what information is needed and by whom. · Examine what has been done in other competitive funds and ARE projects. · For technology options, see box 3.2. 21 3534-Rikka.pdf 6/20/05 2:51 PM Page 22 BOX 3.2:TECHNOLOGY OPTIONS IN DEVELOPMENT OF MANAGEMENT INFORMATION SYSTEM (MIS) In an interconnected world the technology options available today are either web-based, stand alone, web-enabled or Web-enabled systems using Excel templates. Connectivity is a critical issue in most computerized MISs, however, most stakeholders will require that web-enabled M&E systems need to be developed as connectivity will increase and improve in the future. Hardware, software and options for connectivity Equipment will consist of hardware, software, database, connectivity, and other peripheral tools. At each level within the project, equipment needs to be available and trained personnel should be in position to use the equipment. Training should be provided in overall management, database use, hardware opera- tions, network management, and operations and software use. Equipment National or Central State or District Levels Hardware Owned or leased server with database Workstations, desktops and support for application development laptop computers & hosting. Software Database programs along with spreadsheets, Internet Explorer, spreadsheet, PC-based database and statistical software PC- based database programs Database SQL server RDBMS none Connectivity Leased line VSAT dish, leased line/dial up modems Others Registered domain name Internet connection Source: Ghazali Raheem, Consultant, World Bank. 2. Staffing and responsibilities · Ensure adequate and qualified staffing. · Designate responsibilities for compiling, checking, and follow-up on reporting. · ARE projects generally requires M&E specialists of the Project Implementation Unit (PIU) to carry out required data collection, analysis, and reporting functions.9 The specialist group is expected to collect and summarize data from the appropriate insti- tutions (e.g., research institution) at an appropriate level of detail in a project data- base or MIS. Project Implementation Unit is also recommended to offer training courses to subproj-ect leaders to ensure that the required information will be made available and prescribed in the uniform reporting process. Attention to these details may significantly simplify the data collection, analysis, and reporting for the PIU. 3. Data and targeting · Quantify expected inputs and outputs. · Establish indicators or milestones for each subactivity. · Provide gender-disaggregated data where possible. 4. Adequate procedures · Establish procedures for regular flow of information between decentralized imple- mentation units and a central M&E unit. 22 3534-Rikka.pdf 6/20/05 2:51 PM Page 23 · Make it modular, integrate all information needs (across ministries, extension cen- ters, research stations, and subprojects), including accounting, into a single flexible system with a common central database (usually at the PIU). · Be ready when the project starts by piloting and troubleshooting the system prior to starting implementation, otherwise information will be lost and heavy workloads will be required to catch up. · Be careful in the transition from design to implementation; lack of continuity can lead to delays. 5. Evaluation Retain flexibility. Evolve with the project and evaluate once a year. Pick up problems early. Good practice examples. Babu et al. (1997) describe MIS for extension programs (http://www.fao.org/docrep/ W5830E/w5830e0k.htm). Box 3.3 and Box 3.4 describe good practice examples of an MIS in an agricultural research project in Ecuador and the Project Information Management System Network in India. BOX 3.3. KEY FEATURES OF THE MANAGEMENT INFORMATION SYSTEM (MIS) FOR THE MODERNIZATION PROGRAM FOR AGRICULTURAL SERVICES PROJECT (PROMSA) IN ECUADOR The PROMSA Project Implementation Unit (PIU) in collaboration with Natural Resources International (NRI) and the Competitive Fund Management Unit (UEFC) developed a state of the art M&E system for the competitive funding scheme in Ecuador. For individual projects financed under the competitive fund, a simple system of objectives and mile- stones (for inputs, activities, and outputs) was established, with monitoring based on information in quarterly reports. The annual evaluation of subprojects is based on annual reports by researchers; establishment of partic- ipatory grupo de referencia of users (reference group) and other interested individuals for each project; and bi-annual visits by UEFC to each project. In addition, the UEFC has recently solicited feedback from project leaders through a survey of their opinions about the processes employed in the first three calls. Data on all projects and milestones are being recorded in a database managed by UEFC. The PROMSA Competitive Fund employs a system of "alert" signals or flags that indicate if a project has become a "problem project." While in "alert" status, a project is not eligible for payments from the Fund. M&E of the competitive fund as a whole is based on several mechanisms. These include: (i) The annual operating plan of the UEFC combined with quarterly reports on implementation according to the proposed work plan; (ii) the project data base established by UEFC which will provide the PROMSA-PIU direct access to data on implementation of projects, and overall performance of the project portfolio; (iii)visits by PROMSA-PIU officer to projects and other UEFC-organized activities to observe first- hand processes being employed in administering the competitive fund; and (iv) special studies and external evaluation of the operation of the competitive fund and its outcomes and impacts. Source: World Bank (2004a). Agriculture Investment Sourcebook. 23 3534-Rikka.pdf 6/20/05 2:51 PM Page 24 BOX 3.4. PROJECT INFORMATION MANAGEMENT SYSTEM NETWORK OF NDIA S I ' NATIONAL AGRICULTURAL TECHNOLOGY PROJECT Project Information Management System Network (PIMSNET) has been designed, developed, and implemented under the subproject "Institutionalization of Research Priority Setting, Monitoring and Evaluation and Networking of Social Scientists" of the National Agricultural Technology Project (NATP). It is coordinated by the team in the Division of Computer Applications, Indian Agricultural Statistics Research Institute. The purpose of the PIMSNET is to: 1. Effectively assess research efforts against well-defined targets; 2. Avoid duplication of research efforts; 3. Provide feedback to research planning process; and 4. Help to establish link between performance evaluation and incentive mechanism. PIMSNET is being used for M&E of 845 research projects under NATP. It has four modules: Data Management, Monitoring, Reports, Query, and Administration. For technical details, see the website www.pimsnet.gen.in. The monitoring module acts essentially as a monitoring system for NATP. · The module stores information on the progress of the projects for the different activities in terms of the expected targets achieved (several quantifiable indicators), and if not, the reasons thereof and the steps taken to achieve the targets as reported by the Principal Investigators. · It is possible to monitor the half yearly and annual progress in terms of three types of indicators for physical progress, financial progress, and scientific or technical progress. · The research achievements, shortcomings, and difficulties faced by the researchers are captured by the system and the management can effectively monitor the progress of all the projects right from their desktop. · The system has a built-in algorithm to determine the projects which are not performing well/well performing/very well performing/excellent performing projects. · Provision has also been made to review the progress report by the Site Committees, State Level Committees and the Peer Review Teams. The output modules for the reports and queries through which top and middle level management may get reports and queries as per their specific needs. The system is able to produce the following reports: · General Reports are ready to use reports at macro level generated on the basis of single criteria, such as projects running under a specific agro-ecosystem, on a particular thematic area, in a par- ticular state, etc. · Custom Reports meet the specific needs of the management and are generated on the basis of options selected by the user based on multiple criteria selected from a ready-to-use list of options for a desired project. The module generates reports meeting all the selected criteria. For example, a selection option might be to get a report on projects running under Irrigated-agro ecosystem program in the thematic area of Integrated Pest Management and having a budget of more than Rupees 20 lakhs. 24 3534-Rikka.pdf 6/20/05 2:51 PM Page 25 · Specific Reports are related to the details of a particular project. These are of utmost importance when management needs to find out specific details of a single project. These reports contain all the approved items for the project, like equipment procured, contractual services, training obtained, civil work carried out etc. as well as project objectives, budget, and the experts involved. · Administrative Reports act as a tool for the administration and management of the informa- tion storage and user management. Source: Dr. S. D. Sharma, Indian Statistics Research Institute. 3.3. INSTITUTIONAL AND HUMAN CAPACITY REQUIREMENTS FOR PROJECT M&E Adequate institutional arrangements and institutional and human capacity are essential for any M&E project, including functioning of MIS. The level of skills required depends on the complexity of the project. Competitive Research Grant Projects and projects using contracting and involvement of a wide range of stakeholders across several R&E subprojects are demanding in M&E capacity.10 As evident from the PROMSA example, the implementation of a well-functioning M&E system both at the subproject and the overall program level can be a major challenge. The following features are recommended for developing institutional arrangements for M&E of ARE projects: 1. Establish a centralized M&E unit. In general, projects may either establish an M&E unit which is integrated into the Project Implementation Unit or not have a centralized M&E unit but share M&E tasks among the implementing partners and primary stake- holders. For complex ARE projects, it is recommended that a centralized M&E unit be established within the main implementing institution (e.g., the Ministry of Agriculture). 2. Link the centralized M&E unit to subunits. The centralized unit should collaborate with M&E units in other co-implementing institutions (e.g., extension agencies, research centers, private sector implementers, enterprise development centers) and in decentralized regions (e.g., province, district, and county level centers) where project activities take place or have influence. Similarly, sub-M&E systems within the Agriculture Ministry, located at a Ministry's institutional or policy planning unit, should be linked to the central M&E unit within the Ministry of Agriculture. These first two steps aim to guarantee that all project components will (a) provide sufficient project and program level M&E links and a scope for effective communication between and within projects; (b) ensure adequate reporting at the national or program level, and pinpoint any gaps or shortfalls that may not be detected by the M&E system of a single project; and (c) provide an avenue for var- ious project teams or Team Leaders with designated projects under the Ministry of Agriculture to collaborate, share lessons, and ensure desired results. 25 3534-Rikka.pdf 6/20/05 2:51 PM Page 26 M&E of competitive grant funds to private researchers have particular challenges due to the individ- ual nature of the grant operation. · Given that many private researchers are not under the umbrella of any research or edu- cational institution, tracking and eventual assessment of grant operation and research results may be very difficult. · It is recommended that such individual researchers be monitored by focal M&E persons under the project's technical and supervisory Ministry. This would ensure proper administration of the grant and generation of the expected research results. Contracting extension services to private groups may have similar requirements. · M&E at the project level would focus on the contracted group which will provide tracked and assessed extension outcomes on a more systematic basis. · On the positive side, contracted extension service provider may be in a position to: (a) attribute specific services to the improved position of farmers; (b) pinpoint the real effects of extension in a particular area and eliminate a mismatch by factoring out nega- tive effects which may occur due to attributes other than that of extension (annual rainfall fluctuation or declining production); (c) capture the information on indirect beneficiaries; and (d) be provided sufficient time to ascertain extension progress and results. 3. Consider institutionalizing an M&E system within implementing institutions (extension and research centers). This will provide a permanent M&E system beyond the life of the project as well as meet Bank needs. Box 3.5 illustrates a sce- nario on how M&E efforts of different projects implemented by the Ministry of Agriculture can be harmonized. BOX 3.5. SCENARIO ON HARMONIZING M&E IN THE NATIONAL SECTOR MINISTRIES This scenario responds to the broader needs of the national research and extension programs. 1. A centralized M&E unit within the Ministry of Agriculture, coordinated by highly qualified local staff (e.g., Ghana's Ministry of Land and Forestry, which houses a Policy Planning & M&E Unit), will have information on all projects in the country that are being implemented under the Ministry. 2. During the implementation of given projects under the Ministry, data or duplicates of data from all projects will be fed into the centralized system such that at any point in time, or at any stage in implementation, the central M&E unit will be in a position to determine what results or inter- mediate results have been achieved and likewise where expected results have fallen short of their planned/scheduled achievements. 3. Each specific project will also have an M&E sub-unit within their respective Project Coordinating Unit (PCU) in the implementing Ministry, closely collaborating with co-implementing agencies as well as decentralized implementing bodies. Therefore, project level data is managed at the level of each PCU, but will be fed into the institution's central M&E system at the Sector Ministry. The outcome of such a scenario facilitates formulation of coherent sector priorities, policies, plans, and programs, as well as enhanced knowledge sharing. This scenario can also provide a step toward donor harmonization. Source: Remileku Rakey, Consultant, World Bank. 26 3534-Rikka.pdf 6/20/05 2:51 PM Page 27 4. Allocate sufficient time and resources to guarantee adequate capacity. Adequate institutional and human capacity is imperative for effective functioning of M&E frameworks within implementing institutions. 5. Assess existing local capacity for M&E at the design stage (e.g., by using institution- al assessment in chapter 4). At a minimum, capacity must include: (a) the ability to successfully construct indicators; (b) the means to collect, aggregate, analyze, and report on the performance data in relation to indicators and their baseline; and (c) have the skills and understanding to use the information effectively. 1. Identify important capacity gaps to fill (i.e., number and skills of staff); 2. Estimate the costs of capacity-building in M&E; and 3. Develop and establish a capacity enhancement plan (to be partly implemented by the project). Factors to include in an M&E capacity enhancement plan: (a) the identified skill gap; (b) target persons in the central and subunits of the Ministry, project components, implementing agencies and beneficiaries; (c) timeline for the needed skills; and (d) appropriate training times, trainers, and costs. 3.4. DATA SOURCES AND METHODOLOGY FOR DATA COLLECTION AND ANALYSIS The following section reviews good practices in data sources and data collection. The section starts with a discussion on baseline data, which is lacking in a large share of the agriculture projects. Why to collect baseline data? The baseline is the first critical measurement of the performance indicators and is used as a starting point, or guide, by which to monitor future performance of proj- ects or programs (Kusek and Rist 2004). Therefore, baseline data should be collected at least for each identified outcome indicator. Because the success of a project will be, in part, measured by comparing target values with achieved or actual values, setting target values is a sensitive issue and should be taken seriously. One method to establish targets is to start with the baseline indicator level, use historical data or another estimate of the rate of change to set the desired level of improvement--taking into account available funding and other resources over the target period--to arrive at the performance target. Although it is tempt- ing to set targets relatively low to assure they are reached, it is important to set the targets high enough to ensure project implementation momentum. A comparison of baseline to achieved values can be done in the following manner: 1. By comparing the situation before the project started with the situation after it started. This requires understanding which factors influenced the outcome. 2. By tracking changes with (inside the project area) and without a project presence (out- side the project area). This requires finding comparable areas. 3. By comparing the difference between similar groups (inside the project area) between a project group and a group outside the project influence. This requires finding com- parable groups within the same area. What are the key questions to ask? Often baseline studies suffer either from information overload and lack of use of the information or a baseline does not exist at all with a burning need to assess 27 3534-Rikka.pdf 6/20/05 2:51 PM Page 28 project progress and impact. When collecting data to define the baseline level and later on to deter- mine current levels the following questions are necessary: 1. What are the sources of data? 2. What are the data collection methods? 3. Who will collect the data? 4. How often will the data be collected? 5. What is the cost and difficulty to collect the data? 6. Who will analyze the data? 7. Who will report the data and in what form and forum should data be reported? 8. Who will use the data? 9. Capacity building needs? Links to examples of baseline surveys 1. Grosh, M. and P. Glewwe. (2000). Designing household survey questionnaires for developing countries: Lessons from 15 years of the Living Standards Measurement Study Volumes 1­3. World Bank, Washington, DC. Designing household survey ques- tionnaires for developing countries: Lessons from 15 years of the Living Standards Measurement StudyVol. 1. · On agriculture, volume 2 lays out the most important agricultural policy issues in developing countries, discusses the data needs and measurement concerns, and introduces three suggestions for an agriculture module. · Volume 3 presents draft questionnaire modules on several topics, including issues in agriculture, among others a separate questionnaire on access to and use of agricul- tural extension services. 2. The World Bank South Asia Region has developed a "generic" baseline survey for water resources and irrigation projects. These surveys are accessible in the SASAR intranet website: http://wbln0023.worldbank.org/Internal/SAR/SARInternalWeb.nsf/ SASRD/B2330FE4633F93E185256FAC006EAD6F?OpenDocument What to do without baseline data? The collection of baseline data has been very often neglected in Bank-supported projects in the past. It is highly recommended that the operational units make sure that a baseline survey will be carried out before the project will be presented to the Board. However, when no baseline data exists, alternative means must be identified (see box 3.6). From where and how to collect the data for different M&E needs? The M&E systems of ARE proj- ects, be it for a baseline or other M&E needs, must balance data needs between the ideal (and costly) and the practical, and will generally need to draw on data from various sources. The choice of a given set of methods for data collection depends largely on the specific objectives of the given task. Regardless of the method used for collecting information, during the preparation it is also essential to pay attention to: · The sampling frames; sample size and sampling methods;11 and · Preparation of data collection materials (e.g., questionnaires and interview guides.) 28 3534-Rikka.pdf 6/20/05 2:51 PM Page 29 BOX 3.6: WHAT TO DO WHEN THE PROJECT DOES NOT HAVE BASELINE DATA? Quite frequently, projects and programs have started or have to start without baseline data at hand. In this situation, the project team can consider the following alternatives: 1. Track changes with (inside the project area) and without a project presence (outside the project area) if comparable areas can be found. 2. Use first measurements as a starting point (e.g., production during the first implementation sea- son). 3. Look for existing local and national agricultural databases. In some countries, authorities regu- larly collect data on farm and off-farm income sources, production data, and use of inputs. 4. Carry out interviews with local key informants (e.g., village leaders, elderly, and extension workers) or focus groups (e.g., women's and savings groups and producer associations) to detect changes in the availability and use of agricultural services and inputs, and yield data--historical trends and timelines. 5. Other existing records, such as · Data collected by other donor projects or NGOs. · Training records--village or community and extension centers' records on the past topics, tim- ings, participants, possible evaluations on trainings and field events. · Research program records--topics, events, farmer participation, and variety release and adop- tion. · Data on input sales (e.g., on specific varieties) from government and private sector records. Source: Riikka Rajalahti, World Bank. There are many different data sources and methods (see Alex and Byerlee (2000) and IFAD (2003) for more details), but here we focus on the three major data sources: (a) Management Information System; (b) existing/regularly maintained databases; and (c) special studies (see table 3.2). In prac- tice, it is often necessary to use a combination of methods. 1. Management Information System Project input-output data as well as data on agreed indicators are generally available from the proj- ect MIS. The section above includes more details on MIS. 2. National and International Level Databases As it is generally not cost-effective for a project to measure national change of goal-level indicators, information needs at this level rely on national and international databases. These databases may pro- vide data for indicators such as rural population below poverty line, daily per capita dietary energy supply, agricultural GDP, or public expenditures on agricultural research as percentage of the agri- cultural GDP. Quality of national data may be suspect to poor sampling procedures or intentional dis- tortions, but it is readily available and comprehensive, represents considerable investment in data collection, and is official, government-sanctioned data. For example, relevant data may be collected by the Ministry of Finance, Ministry of Agriculture, or Ministry of Science. These data can be sup- plemented by international databases.12 29 3534-Rikka.pdf 6/20/05 2:51 PM Page 30 3. Special Studies Bridging the gap between indicators from routine reporting on project activities and from national data sets is a challenge for ARE projects, and reflects the difficulty inherent in linking project performance and eventual impacts. To bridge this gap, project teams could consider using special studies. When to use special studies? The need for special studies should be considered already during the project preparation as they are of relatively high cost and management intensive. Special studies are a practical tool: · For providing data disaggregated by gender, ethnicity, or other key characteristic of client population; · For stakeholders to learn about and affect the project and M&E process (see chapter 4); · For collecting information on agricultural systems, evaluating project impacts, and espe- cially for assessing and measuring project outcomes; and · To track specific planned and unplanned impacts or operational issues (e.g., technology access by women or minority groups, and changes in land tenure and other social condi- tions in the area). An example of a diagnostic special study on the impact of extension on women is presented in box 3.7. How to conduct special studies? 1. Build special studies into the regular work plan of ARE implementing institutions, starting from the project start-up, mid-term review. For example, extension agencies must track changes in farmer attitudes, knowledge, and technology adoption; research programs need to know results of new technologies; and training institutions need to know how graduates perform. An example of how to use interviews and focus group discussions for a mid-term evaluation is shown in box 3.8. 2. Clearly identify the stakeholders involved and the purpose of the studies. Beneficiary and stakeholder participation is highly recommended (see chapter 4); 3. Select an appropriate methodology (see table 3.2 and chapter 4); 4. Pre-test any questionnaires and surveys (with the target group and the survey imple- menter) and adjust as needed; 5. Consider using an independent institution to avoid overburdening the implementing institution or to avoid bias in monitoring change within the institution; 6. Consider financing special studies through a project office independent of the implementing agency; 7. Verify the validity of the collected data (e.g., by cross-checking); and 8. Analyze the information and prepare corrective measures (as needed) to meet the desired outcomes. 30 3534-Rikka.pdf 6/20/05 2:51 PM Page 31 BOX 3.7. RURAL WOMEN'S EMPOWERMENT THROUGH AGRICULTURAL EXTENSION IN VENEZUELA In Venezuela, the Agricultural Extension Project (PREA) implemented by the CIARA Foundation from 1995­2004 was designed to establish a new decentralized extension service while focusing on social and gender aspects with the direct involvement of the beneficiaries through the participation of rural grass root organizations. The rationale for the gender perspective of this project was that the gender approach encourages participation and personal growth of women, which influences individual and family well- being, strengthens the capacity for team work and consolidation of the rural associations and organiza- tions leading to benefiting the men, the family, the community, and the society as a whole. What? In terms of M&E, the diagnosis, indicators for productivity, and family and community activi- ties were implemented with a gender approach. This required dedicated continuous work with program users and monitoring of the experiences as well as the incorporation of a gender focal team into the Project Coordinating Unit. As an example, a comprehensive diagnostic study was conducted for the project in order to analyze the impact of extension activities on women in rural areas. How? The Project included a gender sensitive M&E during project implementation by incorporating gender data in the producer's technical files and allowing the rural women to be registered as produc- ers and heads of the households. With this data a comparison of gender-related differences in agricul- tural production, and family and community activities were made possible. Other applications of gender approach were in: (a) training in theoretical and methodological gender equity tools; (b) preparation of the participatory diagnosis done from a gender perspective; (c) design of the annual Plan of Extension oriented in gender perspective; and (d) execution of several of the productive and community projects to apply gender tools learnt under this project. Findings: The membership of Civil Associations of Extension or Producers Grassroots Organizations (ACEs) in municipal extension in the project included 2,696 rural women out of a total of 10,370. Over one quarter of the project direct beneficiaries (45,000) and members of ACEs were women (11,700), indicating the success of the project's strategy to ensure women's access to extension serv- ices. Women were involved mainly in activities such as in micro-enterprising and group activities and expressed their desire for additional services. In training, more extensionists received training in gen- der than in technical matters reflecting the project strategy on social aspects of community develop- ment. Of the 561 extensionists trained 118 were women. This project experience demonstrated how attention to the family needs, women in particular, gives such an investment a high level of social relevance and is a source of incentive to social participation. Nearly 25 percent of extension agents are women, and extension services for women have emerged as an impor- tant part of the overall municipal extension program. This work highlighted the need for early engagement of gender approach, defining gender-based key indicators, and step-wise monitoring. Sources: The World Bank, ICR Report No. 29081, dated June 29, 2004 & Draft paper on "Rural Women as Axis of Family and Community Empowerment in the Agricultural Extension Project of Venezuela" by Maria Magdalena Colmenares and Andrea Pereira, with input from Ninoska Loaiza and Indu John Abraham (reviewer). Additional input: Indira J. Ekanayake, World Bank. 31 3534-Rikka.pdf 6/20/05 2:51 PM Page 32 BOX 3.8: USING INTERVIEWS ANDFOCUS GROUP DISCUSSIONS ATMID-TERM EVALUATION OF A FARMER-LED EXTENSION SERVICE PROVISION PROJECT Why? To evaluate the effectiveness of farmer-led extension in improving access to services among smallholders in Vietnam How? Using semi-structured household and focus group discussions at household, village and com- mune levels. Who? Interviews by the staff of a non-governmental organization--can also be carried out by local associations, extension center staff, PIU staff, consultants, etc. Specific focus of the "study:" to determine farmers' (women and men) access to and use of extension, changes in the sources of extension, and satisfaction with quality and access to services. Key findings: Volunteer farmer trainers (FT) were not yet sufficiently effective and reliable in provid- ing extension services to fellow farmers in their villages. Why? The FTs tended to retain the information and knowledge to themselves and did not openly invite other farmers to their multiple demonstration sites. Also, other farmers were hesitant to contact FTs. Corrective action: · Awareness raising among villagers and farmer trainers on the role of FTs · Continued effort to improve FTs' skills to inform and support farmers and organize field days and other events. · Regular reporting and meetings between FTs and commune extension coordinator. · Local level monitoring (by other farmers and commune extension coordinator) of FT activities. Source: Riikka Rajalahti, World Bank When to collect and use the data and by whom? The responsibilities for data collection vary from project to project based on the country and project context and must be specified in the M&E plan. Similarly, the capacity building needs must be identified early on and are based on the specific M&E functions and the complexity of the project. See table 3.2 for a summary on: · Suitable timing for using different data sources during the project life cycle and with differ- ent types of instruments; · Cost and capacity implications; and · Suggestions on who will collect and use the data. 32 3534-Rikka.pdf 6/20/05 2:51 PM Page 33 and all external analysis udies St producer Formal sample and surveys or appraisals extension Yes No No No Yes Yes Yes Often Often Often Implementing agencies, M&E Potentially stakeholders Med-high Med-high regularly and Special rural reviews Rapid 3 and research peer organizations, and maintained peers 2. external Various donor agencies are The and high. analysis needed needed consultants 3. reviews ases External are No Yes No No As As Yes Often Often Often External and PIU, impl. High High ys. overnmentg vernmental datab bases surve non-go Databases data include all external national and may and diagnostic Formal surveys MIS cost diagnostic needed consultants, Maintained for agencies by As Yes No No Yes/No Yes Yes Often Often Often Implementing agencies, M&E Potentially stakeholders Med-high High and activities informal out 5 and source 2 all M&E Regularly M&E implications carried be Data complexity 4.Implementing and of Rapid rural benefic. routine cost needed can appraisals of and observations, As Yes No No Yes Yes Yes Often Often Always PIU, implementers, external Potentially stakeholders Med-high Med-high Level part M&E a analysis). Existing all as structured complexity govt., External centers of System, Other economic 5. routine statistics needed needed needed needed established Yes No As As As As Yes Always Always Always Local beneficiaries, ARE Potentially stakeholders Low Low level is discussions, the ex-post groups. 1 and and (MIS) and group Information govt. donor producer needed needed focus System ex-ante databases assumptions, and National international Yes No No No As As Yes Always Always Always Ministry other agencies PIU, implementers Low Low and valid NGOs, not interviews, and , Information reviews, 1 4 are Bank MIS impl. donor agencies peer these providers, If community Sources--Management World Methods No No Always Always Always Yes Yes Always Always Always PIU, Agencies beneficiaries PIU, impl. Low Med-high Management and program service Data project. grants R&E Rajalahti, the sector research Major Collection of informant When Who How Cycle user capacity Riikka Assumptions: key (e.g., private 3.2. Data Outcomes evaluation collects? 1. Baseline Inputs Outputs IR evaluation Contracting Services Costs Staff Project Appraisal Monitoring Mid-term Final Instrument Competitive Participatory Who Primary Resources Source: Note. irrespective include include centers, organizations. Table Main 33 3534-Rikka.pdf 6/20/05 2:51 PM Page 34 3.5. MONITORING AND EVALUATION COSTS There are typically two types of costs that are required in every project and are crucial for M&E activities: (a) costs relating to project preparation and implementation activities; and (b) costs relat- ing to the formulation of a budget.13 Project Costs: Bank project costs are usually prepared using Costab 32, a PC-based project costing tool. Costab 32 is the project costing component of a software series to improve efficiency and effec- tiveness of Project Processing activities. See Appendix 5 for more details. M&E Costs: Project M&E costs on the other hand need to follow realistic budget estimates. The most important factors to consider when estimating an accurate budget for monitoring and evalua- tion activities are scope, methodology, and stakeholder participation. Evaluations requiring the par- ticipation of a large number of stakeholders would require more funds than the evaluation of a single project. In preparing the M&E work plan it is necessary to allow adequate provision at the outset of the proj- ect for all activities related to M&E. These will include: · Costs for collecting baseline data; · Concurrent monitoring of implementation activities; · Independent evaluations and studies; · Training and capacity building workshops to include MIS design development and imple- mentation. Given the various types of interventions and the development circumstances in each project, it is not practical to prescribe the percentage of the total budget of every project that should be allocated for monitoring and evaluation actions. Typically, M&E costs comprise 10­15% of a project's total budget. These costs should be included in the project budget on the respective budget line. Standard costs include baseline values, ongoing monitoring, mid-term evaluation, final evaluation and terminal report as well as project-specific costs (i.e., surveys to collect baseline data, workshops, cost of hiring staff, and M&E specialists) should be taken into account. See Appendix 5 for an example of a typical M&E budget plan, includ- ing a breakdown of key expense categories, for an IDA/GEF Environmental project. 34 3534-Rikka.pdf 6/20/05 2:51 PM Page 35 4. PARTICIPATORY MONITORING AND EVALUATION Focused attention to poverty reduction and socially sustainable development embracing the socio- cultural diversity among target populations has increased the need to engage in direct dialogue with different stakeholder groups involved in development projects. One critical dimension of this engagement is M&E implemented in participation with stakeholders. This section gives an overview of participatory M&E and guidance for its use in ARE projects. 4.1. WHAT IS PARTICIPATORY M&E? Participatory monitoring and evaluation (PME) is the active involvement of key stakeholders in the M&E process in order for them to learn about and affect the process and impact of a development project. The nature of stakeholder involvement in PME may range from voice or consultation, to implementation and use of information; thereby ensuring the needed collective process of reflection, planning, and management for desired outcomes and impact. At the heart of PME are four broad principles: · Participation--which means opening up the design of the process to include those most directly affected, and agreeing to analyze data together; · Negotiation--to reach agreement about what will be monitored or evaluated, how and when data will be collected and analyzed, what the data actually mean, how findings will be shared, and what action should be taken; · Learning--becomes the basis for subsequent improvement and corrective action; · Flexibility--is essential because the numbers, roles, and skills of stakeholders, the external environment, and other factors change over time. PME can be used in concert with traditional M&E but should be considered a different process. In conventional M&E, participation is often limited to working with primary stakeholders as informa- tion sources, rather than as joint users of information. PME, in turn, incorporates ongoing M&E by local stakeholders throughout the project cycle and after the project has ended. Table 4.1summarizes the main differences between conventional and participatory M&E. 4.2 WHEN AND HOW TO USE PME INARE PROJECTS? Beneficiary and stakeholder participation in M&E is critical: (a) to ensure that ARE projects and programs are responsive to the genuine needs of intended clients; (b) to empower target beneficiar- ies and strengthen ARE institutions through better programs, accountability and transparency; and (c) to determine the impact (also unintended impacts) on the intended beneficiaries and stakehold- ers from their respective perspectives. 35 3534-Rikka.pdf 6/20/05 2:51 PM Page 36 Table 4.1. The Major Differences between Conventional and Participatory M&E Facet of M&E Conventional M&E Participatory M&E Who plans and Senior managers and outside experts Primary stakeholders, project staff, managers manages the process and other stakeholders, often helped by a facilitator Role of primary stakeholders Information provision only Designing and adapting the methodology, collecting and analyzing data, sharing findings, identifying lessons learned and linking them to action How success is measured Externally defined, mainly Internally-defined indicators, including quantitative indicators more qualitative judgment and stories of personal change Approach Predetermined and fixed Indicative and adaptive Source: Adapted from IFAD (2003). Step 1--Decide when to use PME PME can and is encouraged to be used at any stage of the ARE project; and the degree of participa- tion can vary from stage to stage. However, it is better that the overall project planning is participa- tory and that PME activities are initiated at the very beginning of the project as this will increase the likelihood of mainstreaming PME in the project cycle. If resources are limited--time, staff and finances--it is better to prioritize when to use PME rather than to sacrifice the quality of the activities. During the planning process, or later when questions arise and participatory M&E approach is likely to be more useful, the project team together with the beneficiaries and implementers decides the timing of specific PME activities (see table 4.2). Step 2--Decide who to involve in PME PME in ARE projects usually involves all stakeholders in the various facets of farm and non-farm activities. One has to assess the need for and affordable degree of participation by the possible stake- holder groups (see table 4.2). The following questions can guide the decision-making (Guijt 1999): 1. What is the relevant degree of participation for each stakeholder group? Is the process of collating and calculating the information important, or only the final information? 2. Who is going to use the final evaluation? Those who are to use it should understand on what the data is based and how it was calculated. 3. What skills does the analysis require? The more difficult, the more caution should be used in encouraging broad participation unless it is clear who it will benefit and how. Step 3--Decide the scope of the work: What and how to do PME After deciding when to use PME, who to involve and the degree of participation by different stake- holders, it is essential to clarify the methodology to be used as well as the logistics and funding. The process usually begins during the project planning or later during implementation when the need arises with a workshop involving a facilitator and representatives of stakeholders. The initial work- shop should bring clarity concerning the aim of the PME process, roles and responsibilities, logisti- 36 3534-Rikka.pdf 6/20/05 2:51 PM Page 37 out of costs results. the rements if high difficult community crowding it patterns, of stakeholders needs needs clear indicators requi of project requirements objectivity process the resources resources available of not challenging opportunity of make level intermediate of of high time-consuming risk trends, and Challenges and are requirements capacity a and or capacity high stakeholder design effects involve non-functional groups the and pose and to become true wider in determine Potential Availability Capacity-building Facilitation Availability Capacity-building Facilitation Easily objectives indicators Capacity Potentially Harmonization across Time Beneficiaries' can May the outcomes to outcomes, Time Requires commitment the Need also local theme. directly authorities. the preferences and findings providers staff, on that the providers, local and providing on and issues different different service providing Project in depends on service borrower. service in varies. role have have ARE level. ARE and to providers ARE to role providers, key information a forum feedback bank staff. key local in likely service likely a involvement and service in the are receive are are at M&E borrower. ARE are provide (1998). ARE to format authorities, with and impact. purpose--farmers, should borrower on Gaventa local farmers, bank the information chance and Process Involve? stakeholders--farmers, stakeholders stakeholders--farmers, stakeholders farmers, on authorities a them--the and to key authorities key PME indicators. local stakeholders--specific stakeholders have stakeholders. Guijt of Who The providers, Different perceptions Essentially local The authorities, Different for Primarily Depends and collecting managers information. All All and concern All from Stages adapted needed as needed needed as as evaluation stage Different evaluation at "questions" When? Design Evaluation Design Implementation Design Evaluation Implementation Design Baseline Monitoring Evaluations Monitoring Evaluations Baseline Evaluations Monitoring Baseline Final Mid-term in participation project Present findings Participation for data drawing the met of the of of the process M&E indicators the on to changes data evaluation and with project been Level is for evaluate define the opinion and from Degree Participation "impact" views which have what by purpose, and themes and and stakeholders? feedback to Authors. 4.2. of their collecting analyzing their methods PME context results communicate Level in Define meant Design and Define monitor Select Give history the Help monitoring Help conclusions and Share primary and Give degree objectives Source: Table 37 3534-Rikka.pdf 6/20/05 2:51 PM Page 38 cal arrangements, schedule, potential capacity issues, and finally the data collection, analysis, and methodology issues. In this process, the following issues are important to keep in mind: · Systematic dialogue and facilitation during PME: The degree and timing of involvement may vary, but a systematic dialogue between farmers, scientists, extension personnel, govern- ment officials, the business community and their designate institutions and agencies is essen- tial throughout the process. It may be necessary to choose an appropriate and non-biased facilitator to guide the process (in a gender-sensitive manner) and to guarantee that key stake- holders agree on the rationale, objectives, and plans of any PME activity before it becomes a part of the project implementation process; and that the level of stakeholders participation, tim- ing, modes, relevant sources of information, and costs for the PME activity are being clarified. · Foster use of local knowledge: Local knowledge includes perceptions of stakeholders, individual and collective. The thematic areas include the productive environments of stake- holders and target beneficiaries. All these elements may be translated into criteria for evalu- ating research and extension results. · Avoid too ambitious plans. Allow sufficient time to build local skills for indicator selection and collection and other assessments. It is better to start simply and monitor only some aspects of the project. As experience grows and capacities build, the system can be expanded. For example, in Brazil, farmer-based research on agro-forestry systems started by monitoring plant diversity, labor input and ground cover. After the first year, farmers and scientists decid- ed to include soil nutrients and the research process itself (Guijt 1999). · Create incentives for stakeholders to participate in PME and sustain the system. The choice of methodology When information on stakeholders' opinions, views, and perceptions on ARE services are needed, the people become the major source of information.Various quantitative and qualitative tools can be applied. Stakeholder participation can be used to define the questions and questionnaires, to provide and collect data, and to communicate the findings, particularly when actual field level information is needed. Many of the methods useful for traditional M&E can be used in either a participatory or non- participatory way. The participatory impact comes with the way a method is used and in the knowl- edge of who helped to select it. For different tool options, see chapter 3, Appendix 4 (data sources and methods) and PME tools below. A. Beneficiary and stakeholder assessments The overall objective of a beneficiary assessment is to help beneficiaries and other local-level stake- holders identify and design development initiatives, signal constraints to their participation, and give feedback on these activities to those designing and managing a project or formulating policy. Beneficiary assessment includes a set of commonly used tools: conversational interviews, focus group discussions, and participant observation. For details on the necessary steps in ARE projects, see Beneficiary Assessment for Agricultural Extension: A Manual of Good Practice. 38 3534-Rikka.pdf 6/20/05 2:51 PM Page 39 During the project design stage, a broad stakeholder group assessment can clarify the vision and per- spective with which research and extension initiatives will be undertaken. The perspectives of vari- ous stakeholders can contribute to determining the directions for change and their respective roles toward supporting and strengthening such a change process. Therefore, all R&D projects are designed as a consequence of such perspectives. This assessment helps to define the Project Development Objective and the definition for impact. The assessment can be used for evaluation to review and assess the assumptions behind the project activities and indicators after a specific timeframe of implementation; the relevance of the project in the context of changing realities; the implementation of project activities; and to what extent there are deterring factors to reaching desired outcomes. Similarly, it can used to assess project impact and outcomes by way of changes that occurred among target beneficiaries and their environments as a result of project interventions and draw lessons for future activities. The target beneficiaries as well as the main implementers are at the center of this evaluation. See box 4.1 for an example of the key steps to take during a participatory activity evaluation. BOX 4.1: BASIC STEPS INPARTICIPATORY ACTIVITY EVALUATION I. Group discussion --Activity Participants II. Activity Evaluation Report Inputs to discussion Background Monitoring data Relation to goals and objectives Experiences and observations Specific purpose of the activity Results ­ outputs and outcomes Discussion and Analysis Major impressions Analysis ­ social, economic, environmental and Beneficial and problem outcomes institutional issues Constraints Inputs and outputs, costs and benefits Reasons Intangible ­ benefits and problems Gender and other cross-cutting implications Gender and other cross-cutting implications What to do next Conclusions Potential for adoption Potential constraints Recommendations Potential to improve What to do next Source: Adapted from Souvanthong, Pheng and R. Trethewie (2001). Conversational interviews Conversational interviews constitute the basic tool of inquiry for the PME practitioner. Conversational interviews often take place in the homes of the interviewees, who are apt to be most comfortable there. Interviews should be conducted in the local dialect in such a way that open-ended questions revolve around a number of themes or topics that project management has selected. The objective is to gain 39 3534-Rikka.pdf 6/20/05 2:51 PM Page 40 in-depth information on beneficiary views in relation to a planned or ongoing activity by encouraging beneficiaries to speak freely and bring to light issues of concern to project management. Interviews can be conducted on a one-to-one basis or in focus groups. The advantages of individual interviews are that people are likely to speak more freely, without worrying what peers or other community members may think. Lower-status or introverted members of communities may not feel comfortable speaking out in groups. See box 4.2 for an example on a conversational interview with a field-focus. BOX 4.2. USING CONVERSATIONAL INTERVIEWS WITH FIELD-FOCUSED APPROACH INAFRICA What? Purpose of the task was to identify the magnitude of information dissemination from exten- sion worker through contact farmer group to non-contact farmer group How? Using conversational interviews with both members and non-members of contact groups (CG) in ten African countries1 Findings · All but one (Mali) Contact Group generally failed in its task as diffuser of information received from the extension worker. · In Senegal, a number of farmers who were listed as belonging to CGs did not know that they were in fact members. · Only a third of the farmers in areas covered by the extension knew of the existence of CGs. · Ninety percent of the non-CG farmers did not know anything about the existence of CGs. Root causes · Insufficient supervision and publicity was given to contact groups. · Lack of incentives for the CG members to pass information on to his non-CG member farmers. Corrective measures · These insights regarding the degree of knowledge about the CG and the lack of motivation of the CG member to relay extension messages to non-CG neighbors came directly out of the conversa- tional interviewing done with samples of farmers. · The importance of these insights can be seen in the decisions to strengthen CGs in several coun- tries or downplay them, to the point of elimination in Senegal, and in a general trend toward more participatory extension models involving entire communities. Source: Lawrence Salmen, Consultant, World Bank. Unguided discussion is apt to be vague and, therefore, of little use for decision-making; probing for specificity is often required (e.g., if the intended beneficiary of an extension project stated that she did not benefit from it, the interviewer should probe to find reasons for this dissatisfaction, prefer- ably prioritizing reasons (up to three). Focus group discussions For PME, in addition to enabling a wider coverage of the beneficiary population in a given time, interviews carried out in focus groups can serve as a cross-check to individual interviews. The 40 3534-Rikka.pdf 6/20/05 2:51 PM Page 41 groups should normally comprise six to twelve people with common characteristics (e.g., groups of intended beneficiaries may be comprised of married women, male heads of households, youth from 15 to 35, and so on). However, there are times when it may be of use to purposefully mix the con- stituents of a focus group--say, with members and non-members of contact groups--in order to bet- ter appreciate the nature of conflict and communication between them, and provide the opportunity for indigenous solutions. The interview guide should be used in conducting these interviews. The interviewer takes on a facil- itative role, guiding the discussion to cover topics from the thematic guide and ensuring that every- one has an opportunity to participate. This will generally entail encouraging the more reticent, introverted persons to speak up while providing less encouragement to those most apt to dominate the discussion. A researcher should also be present to take notes. Although the difficulty of quan- tifying focus group discussions may be considered a liability, their use as a cross-check and as a fairly rapid and easy-to-read barometer of the mood of a community on many topics makes focus groups a useful component of the PME approach. Participant observation This technique generally involves protracted residence in a targeted community. During this stay, it is expected that the participant observer will establish enough rapport and involvement to help him or her accurately represent the conditions within the community as they relate to project objectives. The participant observer normally spends from one to three weeks in a given community. The researcher will focus on the areas of concern identified in the interview guide. Particular issues relat- ed to agricultural extension that have been the object of inquiry using participant observation include: (a) the relationship between contact groups and other community associations; (b) the role of gender in dissemination of the extension message; and (c) the nature and extent of interaction between an extension agent and the community he serves. During this stay in the community, the participant observer should prepare case studies of five to ten households based on repeated visits and observation. Participant observation, being costly and time- consuming, should be used selectively on topics of particular interest that are of a sensitive nature and lend themselves to this form of intensive personal interaction. B. Institutional assessment In ARE projects an institutional assessment at the design stage can help to determine whether insti- tutions and organizations involved in the project implementation have appropriate organizational procedures, systems, and structures in place. This assessment helps to identify gaps and define some of the main project components. Institutional assessment can be used for evaluations. The process involves an evaluation of the respective institutions' program activities and target interventions relevant to a project, and the extent to which planned outcomes are being achieved. The review highlights shortfalls that relevant insti- tutions have experienced. The institutions' staff, including field researchers, extension agents, and contracted extension service providers, are actively involved together with farmers associations and other players in the field. The exercise is expected to bring about common and shared understanding of apparent problems and collective efforts to solve them. 41 3534-Rikka.pdf 6/20/05 2:51 PM Page 42 For further details on institutional assessment, see: · http://lnweb18.worldbank.org/ESSD/sdvext.nsf/81ByDocName/ToolsandMethods Institutionalanalysis · http://lnweb18.worldbank.org/ESSD/sdvext.nsf/09ByDocName/ProjectPreparation ImplementationParticipatoryAssessmentsInstitutionalAssessment · Horton et al. (2003) "Evaluating capacity development: Experiences from research and development organizations around the world". · Morgan and Taschereau (1996): Capacity and Institutional Assessment: Framework, Methods and Tools for Analysis. www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/ CapacityDevelopment/$file/1996-06-02Tools(Wkshp6).pdf C. Participatory indicator selection Participatory indicator selection, identification of appropriate indicator proxies, and allocation of ME responsibilities are essential features of the design stage. The stakeholder and institutional assessments reflect the stakeholder perspectives and help define the indicators. It is important to avoid imposing indicators and/or methods. Organizations keen to facilitate the development of local monitoring systems often impose their ideas of useful indicators or meth- ods to some extent. It is also necessary to build indicator collection system on what exists local- ly, collect indicators at optimal moments (based on agreed timing, frequency and responsibilities) and avoid collecting too much data. See box 2.1 (in chapter 2) for participatory indicator selec- tion process. D. Community score card Community Score Card is a community-based monitoring tool--a hybrid of the techniques of social audit, community monitoring, and citizen report cards. It has a strong focus on empowerment and accountability as it includes an interface meeting between service providers and the communi- ty that allows for immediate feedback on quality and adequacy of services provided in the commu- nity. This tool can be used for tracking inputs or expenditures; generation of benchmark performance criteria used by communities and service providers to assess their services; monitor- ing the quality of services over time; comparison of performance across facilities and districts; gen- erating feedback mechanisms between providers and users; and strengthening citizen voice and community empowerment. For more details, see http://info.worldbank.org/etools/docs/library/94360/Tanz_0603/Ta_0603/TheCommunityScoreCard Process_June03.pdf. For a summary in the main tools and methods used in PME for ARE projects, see box 4.3 Steps 4 and 5--Analyze and use the data and findings. After analyzing the data, it needs to be disseminated and used for corrective action as needed. Findings can also help to strengthen the PME process itself. Lessons Learned from the NAADS Participatory M&E Pilot in Uganda are described in box 4.4. 42 3534-Rikka.pdf 6/20/05 2:51 PM Page 43 BOX 4.3: MAIN TOOLS AND METHODS USED INPARTICIPATORY MONITORING AND EVALUATION FOR AGRICULTURAL RESEARCH AND EXTENSION PROJECTS 1. The choice of tools and methods for a specific survey or assessment depends on the country context, individual project situation, resource constraints, time frame, available capacity as well as the actual information need. 2. Most tools can be used for design, monitoring, and evaluation of the project. However, quantita- tive methods are best-suited for baselines and evaluations. 3. In principle, any assessment must take account of the following issues: (a) setting objectives; (b) selecting field researchers (with data collection and or communication skills for evaluation); (c) establishing sampling frame and control groups; (d) preparing guidelines in a collaborative man- ner; (e) applying an appropriate methodology; and (f) analyzing and communicating the data and results. Proposed modes of undertaking assessments may include: 1. Quantitative methods, such as administered questionnaires (e.g., household surveys), physical field measurements, and records reviews are used to provide comprehensive and statistically valid data on stakeholders, local conditions, program outcomes and impacts. See Grosh and Glewwe (2000) for further details Designing household survey questionnaires for developing countries: Lessons from 15 years of the Living Standards Measurement Study Vol. 3. 2. Qualitative methods, such as beneficiary assessment, stakeholder forums, focus group discus- sions, and key informant interviews, are used to analyze stakeholders, their views, opinions, and experiences, as well as institutional rules and behaviors. · Beneficiary and stakeholder assessments: To identify and design development initiatives; signal constraints to their participation; give feedback on these activities to those designing and manag- ing a project or formulating policy; and define the Project Development Objective and definition for impact. · Stakeholder forums and workshops involving many stakeholders provide a basis for clarifying questions and issues and exchange of views. For example, this is useful during the design stage. · Focused group discussions are facilitated discussions among 8-12 participants with similar back- grounds. Useful when the objective is to gain in-depth information on beneficiary views, per- spectives, outcomes, and impact in relation to planned activities. · Community Score Card is a community-based monitoring tool--a hybrid of the techniques of social audit, community monitoring, and citizen report cards. It has a strong focus on empower- ment and accountability as it includes an interface meeting between service providers and the community that allows for immediate feedback on quality and adequacy of services provided in the community. http://info.worldbank.org/etools/docs/library/94360/Tanz_0603/ Ta_0603/TheCommunityScoreCardProcess_June03.pdf · Open-ended group interviews revolving around a number of themes selected and agreed on by the participants. · Key informant interviews are loosely structured, in-depth interviews of 15­35 people selected for their first-hand knowledge about a specific topic. www.usaid.gov/pubs/usaid_eval/ pdf_docs/pnabs541.pdf continued on next page 43 3534-Rikka.pdf 6/20/05 2:51 PM Page 44 BOX 4.3:--CONTINUED · Participant and field observation, which is a time-consuming mode of enquiry, covers the active involvement of observers within targeted communities to review and examine themes surround- ing relationship among the respective contact groups; the roles of different social and economic groups; the nature and extent of interaction between the various institutions and farming commu- nities and ongoing activities. Source: Authors. BOX 4.4: LESSONS LEARNED FROM THE UGANDA NATIONAL AGRICULTURAL ADVISORY SERVICES PROJECT'S (NAADS) PME PILOT Although much of the Participatory M&E work on agricultural extension has been geared to serve the needs of the rural development project manager, in the Uganda rural development NAADS project the central actors became the farmers themselves. What: A pilot PME exercise was conducted in 2002/2003 in a sub-county of the Mukono District in which monitoring information was collected from six NAADS farmer groups; and evaluation data was gathered from 60 randomly selected households. The implementers of this PME were 16 trained farm- ers under the supervision of and with technical backstopping provided by sub-country technical staff and persons trained in beneficiary assessment methodology from Makerere University (Kampala). Findings There was clear appreciation of this farmer-driven M&E process as (a) a source of information gath- ered both by and from persons directly affected by the project; and (b) a generator of ownership for the project, with the learning--as well as the project activities--being carried out by the farmers themselves. However, there were a number of challenges, or constraints, which serve as lessons for the improve- ment of this farmer-driven approach to PME in the future: · The time allotted for data tabulation and analysis (two days) was considered inadequate; · Monetary allowances to local PME facilitators, for travel and expenses, was considered by these persons as insufficient to meet their needs; · Several of the communities had not been adequately informed about the PME, resented the PME facilitators as privileged persons and failed to participate in the interviewing (individual and group) as expected; · Generally, the PME facilitators (interviewers) were seen as NAADS "representatives" rather than independent farmers. This impression unduly biased the data; and · Some PME facilitators experienced difficulty with the conversational interviewing approach. Particularly problematic were: (a) maintaining respondents' attention span for long periods of time (due in part to an excessively long interview guide); (b) probing for in-depth information; (c) posing non-leading questions; and (d) controlling group discussions. 44 3534-Rikka.pdf 6/20/05 2:51 PM Page 45 Follow-up action Aside from correcting monetary and logistic factors, it was determined as a result of this pilot to: · Better inform communities about the PME and the nature of the facilitators; · Place more stringent requirements regarding the qualifications necessary to be a facilitator; · Contract external, trained interviewers for evaluation work while retaining farmers for monitoring activity; and · Emphasize capacity building that is tailored to persons with low education levels--involv- ing visual aids, local language, practical field exercises, and alleviation of fears at the out- set of training. Conclusions · The key feature and strength of participatory M&E of rural development projects puts tar- get beneficiaries at the center of the PME process, proving the scope for feedback toward improved project implementation as well as learning on the part of beneficiary farmers. · On the target beneficiary assessment approach, voice is given primacy and there is a sys- tematic listening to the farmer. In the more participatory, farmer-driven PME represented by the NAADS case in Uganda, farmers not only acquire voice but decision-making authority, both as implementers and intended contractors of the learning process. · It is the catalytic transformation of learning into empowerment that lies at the heart of the rationale and potential of participatory monitoring and evaluation as described in the Final Report on this PME by Margaret Mangheni and Christopher Bukenya. "Our experience from the PM&E methodology pilot phase, at least, has revealed renewed com- munity enthusiasm and "social energy" in the NAADS process which are valuable outcomes for a program which seeks among other important goals empowerment of target communities." Source: Lawrence Salmen, Consultant, World Bank. 4.3. ADDITIONAL MATERIAL ON PME For further details on PME in general: 1. World Bank--Participation and Civic Engagement-- http://www.worldbank.org/participation/home.htm 2. Participatory Monitoring and Evaluation-- http://www.worldbank.org/participation/pme/partme.htm 3. Eldis Participation Resource Guide--PME http://www.eldis.org/participation/pme/index.htm 4. PME in Agricultural Research and Extension: Guide to participatory extension methods used in community forestry in Laos http://www.eldis.org/cf/search/disp/docdisplay.cfm?doc=DOC10101&resource=f1par 5. On participatory evaluation, see "Who are the question-makers?" http://www.undp.org/eo/documents/who.htm 45 3534-Rikka.pdf 6/20/05 2:51 PM Page 46 PME tools: 1. World Bank--PME Tools and Methods http://www.worldbank.org/participation/pme/partme2.htm 2. World Bank--Social Analysis Sourcebook http://www.worldbank.org/socialanalysissource book/socialassess5.htm 3. World Bank--Monitoring & Evaluation: Some Tools, Methods & Approaches. lnweb18.worldbank.org/. . ./a5efbb5d776b67d285256b1e0079c9a3/ $FILE/MandE_tools_methods_approaches.pdf 4. IFAD (2003)--Methods for Monitoring and evaluation. http://www.ifad.org/evaluation/guide/appendixd/index.htm and 5. Gathering, Managing and Communicating information. http://www.ifad.org/evaluation/guide/6/index.htm 6. Eldis Participation Resource Guide--Participatory Monitoring and Evaluation--Methods, Tools and Manuals http://www.eldis.org/participation/pme/pme3.htm 7. Gubbels, P. and Koss, C. (2000). From the Roots Up. Strengthening Organizational Capacity through Guided Self-assessment. 46 3534-Rikka.pdf 6/20/05 2:51 PM Page 47 5. EVALUATION OF OUTCOMES AND IMPACT IN THE WORLD BANK 5.1. WHY AND WHAT TO EVALUATE IN THE WORLD BANK PROJECTS? Evaluation is the periodic assessment of the relevance, performance, efficiency, and impact (both expect- ed and unexpected) of the project in relation to stated objectives. Evaluation measures achievements in relation to institutional policies, Bank-wide program objectives, and the goals set for each operation. Evaluation is designed to: · Provide an objective basis for assessing the performance of policies, programs, projects, and processes; · Help provide shared accountability for the achievement of the Bank's objectives; and · Improve policies, programs, and projects by identifying and disseminating the lessons learned from experience and by making recommendations drawn from evaluation findings. Evaluation at the Bank has two major dimensions: 1. Self-evaluation (by the units responsible for particular programs and activities). · An interim evaluation is undertaken by project management during implementation as a first review of progress and a prognosis of the likely effects of the project. It is intended to identify project design problems, and is essentially an internal activity undertaken for proj- ect management. See section 5.2. · Terminal evaluation, a similar process undertaken at the end of a project, is required for the Project Completion Report (PCR; OP/BP 13.55, Implementation Completion Reporting). It includes an assessment of the project's effects and their potential sustainability. Chapter 6 and Appendix 6 present the key steps and a case study on economic impact evaluation of a competitive research grant scheme. Terminal Evaluation: The implementation completion report is prepared to evaluate the performance of the Bank and the borrower. Each party is required to prepare and submit an ICR. Guidelines for ICR preparation can be found from http://siteresources.worldbank.org/INTINVLENDING/Resources/ ICRGuide.pdf. Box 5.1 provides specific highlights for M&E during ICR preparation and write-up. 2.Independent evaluation: · The impact evaluation by OED is usually undertaken several years after final disbursement, and measures changes attributable to the project in terms of both direct and indirect causali- ty. This is normally undertaken by national authorities or donor agencies. For example, in the Bank, impact evaluation is carried out by the Operations Evaluation Department (OED) for its Project Performance Audit Report, and therefore planning and implementation for such a system should be left to OED. 47 3534-Rikka.pdf 6/20/05 2:51 PM Page 48 · OED evaluates by three main factors: (a) The relevance of the project's objectives in rela- tion to the needs of the country; (b) was the project effective in meeting its goals; and (c) was the project efficient in terms of using no more resources than necessary. BOX 5.1. SPECIFIC HIGHLIGHTS FOR M&E DURINGICR PREPARATION In preparing the Bank Implementation Completion Report the following should be reflected in selected sections of the Report: · Principle performance ratings: Good reflection and cross-checking in line with implementa- tion, results of M&E system, baseline data and subsequent updates should go into the ultimate ratings. If a QAG was conducted at project effectiveness, depending on the rating, M&E should be a feature for improving quality by project completion; · Major factors affecting implementation and outcome--if M&E was a factor, either factors subject to or outside of government/implementing agency/donor control, should be highlighted with indications of how those factors were addressed; · Sustainability ratings with specific regards to transition arrangements to regular operations should reflect the mainstreaming of M&E system into the regular functions of the implementing Ministry and plans for continuity; · Bank and borrower performance should reflect perspectives of M&E implementation as con- ducted by borrower and Bank parties highlighting good practices as well as shortfalls; · Ratings for achievement of objectives, outputs of components speak of the project implementa- tion in its entirety: including management, coordination and impact; M&E including independ- ent assessments, case studies and participatory consultation features as key sources of information for arriving at those ratings; M&E is therefore a major guide for the ultimate ratings under this section of the ICR; and · Lessons learned in the ICR should factor modifications or adjustments made in the M&E sys- tems that better captured targeted outputs, outcomes, and impacts. Other ratings during implementation including Project Supervision Report (PSR) ratings of outcome and impact indicators namely--projected against actual targets--should factor quite strongly in the implementation of M&E system during supervision and updates of PSRs. Source: Remileku Rakey, Consultant, World Bank. 5.2. KEY ISSUESTOCONSIDER WHEN EVALUATING AGRICULTURAL RESEARCH AND EXTENSION PROJECTS The focus of the evaluations may be the social, institutional, environmental, and economic outcomes, sustainability, and impact of the ARE interventions. Economic effect and impact evaluations are the most common type of assessments. It is increasingly clear that the social and environmental effects and impacts of interventions must also be assessed. However, it is challenging to monitor and eval- uate environmental and social indicators and effects of ARE projects. The primary purposes of economic, environmental and social assessments of ARE projects are to provide: (a) an information base for understanding complex effects of ARE interventions (i.e., new technologies, policies, and services); (b) an early warning system to prevent adverse effects and to take mid-course corrective action; (c) a tool for communities to protect the public's interest; (d) a 48 3534-Rikka.pdf 6/20/05 2:51 PM Page 49 means of identifying alternative approaches, technologies and adaptations; and (e) a basis for plan- ning that involves the relevant (and affected) stakeholders (Deshler 1989). Social impact evaluations look into the potential effects of ARE interventions (events, trainings, gen- erated/transferred/ technologies, and improved organizations) on specific groups of people (i.e., farm community, poor segments of the community, women farmers, and disadvantaged groups). These eval- uations require the identification of socio-economic variables relevant to a proposed activity; identifi- cation of information sources; data collection and categorization; a community profile of affected populations; projections of consequences for several alternative courses of action; and assessment of impacts and evaluation of the social impact by community members (van Willigen in Score 1995). Similarly, environmental impact evaluations will look into the potential environmental effects of ARE interventions on the farming community and specific groups as well as the environment and natural resource base. It is often necessary to establish indirect ways to measure the environmental effects by selecting indicators that track a change in a physical measure that is considered to have environmental consequences. Examples of this type of indirect indicators are "the change in the length of Vetiver grass strips" and "the number of hectares under no-tillage practices"--they are expected to indicate the degree of soil conservation. Box 5.2 provides examples on potential focus of the evaluation (whether social, environmental, or economic) in ARE projects/programs. In addi- tion, Appendix 4 includes a list of various ARE project indicators. BOX 5.2. POTENTIAL FOCUS FOR EVALUATION IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS/ PROGRAMS National impacts: political stability, economic fairness, agricultural environmental sustainability. Community change: changes in administration of justice; health, welfare, and quality of life; fairness in the marketplace; change in human rights, status of women; change in economic and social indicators for poor; change of indicators of sustainable agriculture and natural resource management; change in communication patterns and access to education and news; change in relationships between farmers and non-farm population; change in youth perceptions about farming career; public opinion change; fairer distribution of land and other resources; improved inter-organization relations; evidence of conflict res- olution; and cultural practice change. Organizational change: group operation and management; economic performance; technical operation and management; financial operation; group institutionalization and self-reliance, new groups of farmers included, new organizational linkages and partnerships; change in staff performance, new service delivery, new methods used, additional facilities and equipment; change in research priorities; diversification of funding; cost-benefits improved; new philosophy, purposes, and goals; improved organizational culture. Individual change: changes in knowledge, attitudes, skills; sustainable agricultural practice; change in aspirations, self-image, perspectives; expenditure of effort and money; use of methods, services; inven- tion of appropriate technology; increased production or use of tolls; compliance with or opposition to public policy; patterns of communication, career directions, and family relationships. Reactions: testimonials; reactions to the relevance of content; appropriateness of technology, helpful- ness, perceived value of educational experience; reputation of the extension provider. Participation: farmer access to extension services by social class, gender, and ethnic groups; intensity of face-to-face contacts; extent of media-assisted contacts; type of participation (volunteering, planning, recruiting, learning, experimenting, evaluating); indicators of commitment (attendance, continuity, fre- quency); private sector involvement. continued on next page 49 3534-Rikka.pdf 6/20/05 2:51 PM Page 50 BOX 5.2:--CONTINUED Activities: participatory rural appraisal; planning; local knowledge documentation; on-farm/farmer experimentation; farmer-to-farmer knowledge sharing; farm tours; farmer organizing; master farmer leadership training; farm demonstrations; exhibitions and fairs; residential workshops; marketing analy- sis; farm policy education. Inputs-resources: organizational sponsorship and networks; funds; organizational design facilities, equipment; philosophy, mission, goals, objectives; staff, resource people, volunteers; local and external research knowledge and relevance; cultural, economic and political context. Source: Deshler 1997; Riikka Rajalahti, World Bank. The following recommendations for conducting social, environmental and economic evaluations of ARE projects build on recommendations by Hanson et al. (2004) who provided suggestions on prac- tical and cost-effective activities when evaluating economic sustainability of extension programs: 1. Evaluate ARE project/program components/activities rather than complete programs-- the individual results can be used in successive improvements better than results from an overall evaluation; 2. Integrate community's and farmers' perceptions of how activities are organized--it is important that the activities have an internal consistency from the farmers' point of view; 3. Separate the evaluation of direct effects/benefits (e.g., income generation by farmers) from indirect ones, (e.g., soil conservation and health benefits and effects on labor division and community relationships); 4. Improve the use of indicators particularly when monitoring and evaluating indirect environmental benefits. However, improved use of indicators must go hand-in-hand with baseline data collection; 5. Take advantage of farmer groups in evaluations. The findings from a group are gener- ally more reliable than from single farmers. However, it is important to train the inter- viewee team properly; and 6. Carry out periodic ex-post analyses as they may contribute to mid-course corrections. Besides the points above, the following steps are essential when planning evaluations: 1. Identify all relevant stakeholders (women and men farmers, research and extension staff, local leaders and authorities, civil society, government, and donors) in this process--those who have an interest in the evaluation results and those who should be involved in the evaluation process; 2. Focus and priorities: It is important to narrow the choices and priorities of the evalu- ation as early as possible. It is equally important to agree on a short list of evaluation questions--these questions will guide the collection and analysis of data and informa- tion as well as interpretation and presentation of the results. Note that the achievement of the Project Development Objective and the intermediate outcomes presented in the project's Results Framework must be part of the evaluation. 50 3534-Rikka.pdf 6/20/05 2:51 PM Page 51 3. Tools and actors: Based on the focus and priorities of the evaluation, the stakeholders involved, and the time and resources available, select the appropriate tools and actors for carrying out the evaluation. See box 5.3 for guidance on selection of internal or external evaluators. BOX 5.3. WHEN TO USE INTERNAL AND EXTERNAL REVIEWERS AND EVALUATORS? Internal reviewers and evaluators are generally used when the objective of the evaluation is to identify project design problems and when the objectivity of the evaluation is not sacrificed. These conditions are generally in place for unplanned assessments, annual reviews, and interim evaluations. The major benefits of using internal evaluators include cost-effectiveness, in-depth knowledge of the project and the local context and the trust gained among the stakeholders. The major disadvantages of using inter- nal evaluators may include subjectivity, over-emphasis on the positive findings, lack of commitment and resistance to change. However, internal evaluators and/or M&E staff should be included in all evalua- tions, with or without external evaluators. External reviewers or evaluators can: · Provide independent and constructive criticism that helps project stakeholders to reflect on and identify lessons; · Give a fair judgment of project progress and areas in need of improvement; · Help identify priorities for the remaining time of the project to support the rational use of resources; · Help unite diverse stakeholder perspectives. The key problems associated with external evaluators may include: · Due to time constraints, they often have limited dialogue with project stakeholders, outside the staff, and may make superficial analyses or to generalize the situation. · They can jump to conclusions without in-depth knowledge of local realities; · If foreigners, language constraints and limited cultural insights may affect the analysis; · They can be excessively strict in their methodology; · They can be inclined to focus more on funding agency requirements than what a project needs for improved impact; · Reports often highlight the negative aspects of the project and do not give due emphasis to posi- tive aspects. Source: IFAD 2003; Riikka Rajalahti, World Bank. · Quantitative methods, such as administered questionnaires (e.g., household surveys), physical field measurements, and records reviews, are used to provide comprehensive and statistically valid data on stakeholders, local conditions, program outcomes and impacts. · Qualitative methods, such as beneficiary assessment, stakeholder forums, focus group discussions, and key informant interviews, are used to analyze stakeholders, their views, opinions, and experiences, as well as institutional rules and behaviors. 51 3534-Rikka.pdf 6/20/05 2:51 PM Page 52 · Chapters 3 (Data collection, reporting, and dissemination: section 3.4) and 4 (Participatory M&E) in this note presented many of the tools that can be used for eval- uation. 4. Analyze the data and findings and bring in the perspectives of relevant stakeholders. Make sure to cross-check the information and results to guarantee the validity of the findings. 5. Communicate the evaluation findings to the relevant stakeholders in an appropriate format. · External stakeholders, such as donors, usually require a formal evaluation report that should be circulated widely enough. Internal stakeholders, such as beneficiaries and implementing partners, may have different priorities (identified at the begin- ning) and interests, and therefore different reporting expectations. The reporting for- mat should therefore be tailored to the needs of the stakeholder--verbal reporting in workshops and meetings may be sufficient for some issues whereas other selected issues may be presented in reports. 6. Use evaluation results. Project evaluation results are expected to provide information for decision making on e.g., policies, management and future projects. Frequently, however, this has not been the case. It is essential to pay adequate attention to identify- ing the relevant stakeholders and the evaluation priorities to improve the use of evalua- tion results. In addition, evaluation can be used as a learning process. 52 3534-Rikka.pdf 6/20/05 2:51 PM Page 53 6. ECONOMIC EVALUATION OF COMPETITIVE GRANTS SCHEMES Competitive Grants Scheme (CGS) programs implemented as a component for agricultural research and extension projects seek to divest some planning and oversight functions to local entities with closer links to end users. Each CGS is comprised of a large number of subprojects carried out by different execut- ing institutions. As the inclusion of CGS in Bank's ARE programs has increased significantly during the past decade, this chapter is meant to help in economic assessment of such schemes. This chapter builds on two previous World Bank publications: "Ex Ante EconomicAnalysis inAKIS Projects" by Horstkotte- Wesseler et al. (2000) and "Monitoring and Evaluation for AKIS Projects" by Alex and Byerlee (2000). For a case study on economic evaluation of Competitive Research Grant in Ecuador, see Appendix 6. 6.1. ECONOMIC IMPACT EVALUATION OF A CGS The theory and empirical tools for assessing the economic14 impact of Research and Extension are well developed (see table 6.1). · The objective of the analysis is to identify changes in productivity that has been induced by the research effort, and compare these benefits to the cost of the research. · The key indicator of research efficiency is the internal rate of return (IRR) of the invest- ment in the CGS. · Because a CGS is comprised of many subprojects, adjustments must be made to the stan- dard procedures for calculating the IRR, primarily with respect to procedures for data col- lection. But the goal is still to compare the discounted benefit and cost streams. Information on intermediate indicators of research output should also be collected and analyzed. It will take from five to twenty years for the economic impact of a successful research project to occur. · A rough rule of thumb might be that extension projects could begin to have a measurable impact within three years, an adaptive research project within five years, and strategic research within 5­10 years. A successful basic research project may not even generate a specific identifiable technology, yet may make a large contribution to the society. · Indicators should be collected to examine: (a) whether the grant fund has been competently administered; and (b) the scientific merit of the funded projects (i.e., whether a sufficient amount of good science is being conducted by grantees). · A variety of indicators can be used. Administrative efficiency can be assessed by consider- ing factors such as the number of grant applications received, the percentage of proposals funded, and the share of overhead costs in total expenditures. Scientific merit is indicated by the number of publications, particularly in refereed or international journals, or the num- ber of papers presented at international conferences. · At some point in the evaluation process, external scientific experts should be called upon to assess the quality of science. It is very difficult for non-experts to do such an assessment. 53 3534-Rikka.pdf 6/20/05 2:51 PM Page 54 Table 6.1. Key Reference Sources on the Theory and Empirical Tools for Assessing the Economic Impact of Research and Extension Reference Books Alston, J., G. Norton, and P. Pardey. 1995. Presents the principles and practice of ex-post and ex-ante Science Under Scarcity. Ithaca, NY: Cornell economic evaluation methods and their use in research priority University Press. setting. A wide range of approaches are reviewed, synthesized, and assessed using a unifying conceptual framework. Horton, D., P. Bellantyne, W. Peterson, B. Uribe, A sourcebook that provides a synthesis of literature and D. Gapasin, and K. Sheridan. 1993. Monitoring experience on research monitoring and evaluation principles, and Evaluating Agricultural Research: A processes, and methods. Sourcebook. Wallingford, UK: CAB International. Manuals CIMMYT, 1993. The Adoption of Agricultural An excellent manual for practitioners that explains step by step Technology: Mexico: CIMMYT the approaches to estimating technology adoption. Masters, W.A., B. Coulibaly, D. Sanogo, A guidebook intended to provide a concise summary of the M. Sidibé, and A. Williams. 1996. The Economic tools needed to conduct persuasive impact studies, which Impact of Agricultural Research: A Practical enable researchers to quantify the economic benefits and costs Guide. Department of Agricultural Economics, of their work. Three spreadsheet-based computer exercises help Purdue University, West Lafayette, IN. E-mail: apply the methods described in the manual. Masters@AgEcon.Purdue.edu Janssen, W. and A. Kissi. 1997. Planning and A practical approach to combine natural resource management Priority Setting for Regional Research. Research and productivity concern; introduces a methodology to choose Management Guidelines 4. International Service regional priority research subprojects based on economic for National Agricultural Research, The Hague. considerations. Available online at www.cgiar.org/isnar Collion, M.H. and A. Kissi. 1995. Guide to An approach to research program planning by objective, based Program Planning and Priority Setting. Research on a series of steps which include benefit-cost analysis as an Management Guidelines No. 2E. ISNAR, The approach to priority setting. Hague. Available online at www.cgiar.org/isnar Software Dream©--Dynamic Research Evaluation for www.cgiar.org/ifpri Management This is a menu-driven computer program developed by ISNAR to facilitate application of the economic surplus model under a variety of market situations. The latest version of this program (June 1998) is available from IFPRI (contact: Stanley Wood, s.wood@cgnet.com). MODEXC--Model for Economic Analysis Http://www.ciat.cgiar.org/inslinks/modexc.htm This is a spreadsheet based economic analysis model developed by CIAT, available on the World Wide Web, that allows to calculate the NPV, IRR and B/C ratio of investments in agricultural research. The model can be used for economic analysis of technical change both ex-ante and ex-post under alternate scenarios of market situation. 54 3534-Rikka.pdf 6/20/05 2:51 PM Page 55 Web sites ISNAR's information and discussion forum Http://www.cgiar.org/isnar/Fora/Priority/index.htm for agricultural research priority setting This web site provides the process, steps, and methods of research priority setting. CIAT impact sub-project Http://www.ciat.cgiar.org/inicio_in.htm, the CIAT web site features abstracts on the impact of agricultural research, data bases, trends, as well as a download version of MODEXC (see above). Source: Byerlee, D. 2003. "Conceptual and Practical Issues in Measuring Impacts of Sub-projects in Competitive Grants Scheme (CGS)." Unpublished manuscript. 12p. 6.2. RESEARCH COSTS Due to the de-centralized nature of the CGS, the cost of each subproject is composed of at least three sources: 1. Central CGS administrative overhead; 2. Direct costs incurred by the executing institution and cooperating institutions; and 3. The cost incurred by the CGS for administration and monitoring and evaluation of the subproject (usually considered as an overhead). Collecting information on research costs is relatively straightforward, so attention here will focus on identifying and measuring productivity changes. 6.3. SAMPLING To keep impact assessment within reasonable costs, only a sample of subprojects can be evaluated. One approach is to evaluate a random sample of subprojects (see box 6.1 for an example). The prin- ciple advantage is that results are representative whereas the major disadvantage can be that a random sample may become more costly to implement if subprojects are widely disbursed geographically. A more serious drawback is that a random sample may miss the most successful project. Agricultural research success has a very skewed distribution. Most projects will have modest or negative economic impacts, but a small number of projects will make important discoveries. An important discovery by a single research project can generate sufficient economic benefits to cover the entire project investment. Therefore it is important to closely examine the benefits generated by projects that are thought to have been successful, perhaps supplemented by a random sample of remaining projects. Procedure for random sampling: · The random sample can be stratified by grouping sub-projects by theme, commodity, target population group, executing agency, or geographic criteria (e.g., crop varieties, small farm- ers, IPM, post-harvest, pastures, and basic grains). 55 3534-Rikka.pdf 6/20/05 2:51 PM Page 56 BOX 6.1: USING RANDOM SAMPLES INEVALUATION OF MPACT OF I COMPETITIVE GRANT SCHEMES During its lifetime of 6 years (1994­2000) PRONATTA carried out six rounds of project funding and co-funded a total of 635 sub-projects. The evaluation of PRONATTA had to be done in a short time and with limited human and financial resources. More importantly, the unstable political and social situation in many rural areas made the collection of survey data impossible. In some areas it was not even possi- ble to conduct field interviews with farmers. A random sample of funded projects was evaluated. Sample sizes for the first and second phases (n1 and n2) were 56 and 50, representing 17% of the sub-projects in each phase. The samples were discriminated by two stratum--the region where the subproject was carried out, and the type of subproject. The evalu- ation team expected these two factors to influence project success. The evaluation was done in two stages, taking advantage of the temporal sequence of finalization of the subprojects. This strategy allowed for a learning process for the evaluation team. Experiences in the first phase illustrated the possible range of values of the needed parameters, while results from the second phase served to validate the procedures applied in the first phase. See figure 6.1. Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture). Figure 6.1. Estimation strategy in PRONATTA. Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture). 1994 2001 2003 2004 First sampling Second sampling N = 338 Estimation N = Estimation n = 56 (17%) n = 50 (17%) First sample Second sample distribution distribution of TIRs of TIRs Combined samples Estimator of TIR LEARNING PROCESS 56 3534-Rikka.pdf 6/20/05 2:51 PM Page 57 · This allows common methods to be developed, use of relevant experts, and a more mean- ingful aggregation of impact results. · Care is needed to ensure that sampled sub-projects are representative, and that within group sample sizes are large enough to be meaningful. · Once a sample is determined, the IRR for each of sub-projects is calculated, and an aggre- gate IRR is estimated. 6.4. MEASURING BENEFIT STREAMS FOR SUBPROJECTS In order to measure project benefits, data are required on three key variables: (a) technologies gen- erated; (b) diffusion levels; and (c) per unit economic benefits.15 1. The evaluator must first identify the technologies that were generated by the research effort. For example, were new crop varieties or new methods of pest control devel- oped. This can generally be done by reviewing project reports, possibly with addition- al information obtained by interviewing researchers and farmers. 2. The level of diffusion of each innovation must be estimated. 3. The productivity impact of each innovation must be measured. Procedures for the steps 2 and 3 are discussed below. The calculated economic impact of a research sub-project is the sum of the individual innovation productivity increases multiplied by the total hectares covered by each new technology. 1. Estimating the diffusion of each innovation Many studies have found that technological diffusion follows a logistic curve pattern. This curve can be estimated by statistical means, but this would require information on the adoption percentage at several points in time. Such information is unlikely to be available, so a shortcut method has been developed. This method requires information on: 1. The initial proportions of farmers using the innovation when the subproject started (p0). Baseline surveys will often not be available mandating that adoption information be elicited through expert opinion. The evaluation team can ask farmers (as members of the panel of experts) to recall production conditions at the time the sub-project was initiated. In many cases estimates of the values of p0 will be equal to zero--diffusion would not have occurred had it not been for the research project. 2. The proportion that have adopted the innovation at the moment of the evaluation (p1); and 3. The maximum expected adoption percentage (K). With this information it is possible to approximate the logistic curve (Martínez and Saín 1983). The annual adoption percentages are then multiplied by the total potential adoption area to come up with the total number of units (hectares) benefiting from the innovation. 57 3534-Rikka.pdf 6/20/05 2:51 PM Page 58 2. Estimation of the incremental net benefit (INB) per unit of analysis (farm or hectare) from adoption of the recommended practice Calculation of the INB requires that the economic value of adopting the research recommendations be estimated. Research projects generally attempt to increase land productivity (yield), reduce pro- duction cost, or increase output quality.16 · Technical information about the effect of adoption must reflect the real impact at the field (not experimental) level. · Prices used in the valuation must reflect those received for products and paid by farmers for inputs and services. · Family labor should be valued at its opportunity cost, approximated by wages in local labor markets. · Special care should be taken in estimating the cost of working capital. If farmers do not have access to formal credit, the interest rate must reflect the interest rate prevailing in informal credit markets. In order to calculate the INB, a counterfactual point of reference, representing the state of tech- nology absent of the research project, must be established. Three main options exist for establishing the counterfactual: 1. Collecting a baseline counterfactual data set: Counterfactual data can be collected by surveying the target population at the initiation of the project and then again at the time of the evaluation. 2. Alternatively, a survey at the time of the evaluation that contains producers affected by the project and farmers not affected by the project can also be used. These baseline data approaches require a significant investment in conducting and analyzing field survey data, but make it possible to accurately disentangle the effects of the research or extension intervention using the statistical regression model illus- trated in table 6.2. 3. In most cases it will not be feasible to conduct a comprehensive survey. The most com- mon method of estimating the INB is to use partial budgets17 based on experimental results, making adjustments in yield and costs to reflect expected on-farm performance. Table 6.2: Statistical Regression Model for Isolating Project Effects in Impact Evaluation Statistical Model Parameter of Interest Y = 0 + 1D1 + 2 D2 + Other factors Estimation is carried out using the pooled sample from Y = parameter of interest (i.e., Yield) baseline and evaluation surveys. D1 = 1 for observations at the moment of the evaluation 2 is the parameter of interest. It measures the net impact = 0 for baseline observations attributed to the sub-project (Value E in table 2.1) D2 = 1 for observations affected by the subproject result (adopters) = 0 for baseline observations Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture). 58 3534-Rikka.pdf 6/20/05 2:51 PM Page 59 6.5. ESTIMATION OF THE IRR Estimated annual benefit flows (ABF) and cost flows (ACF) of each sampled sub-project are needed to estimate the IRR. The ABF is estimated by direct multiplication of the incremental net annual ben- efit (ABF-ACF) by the adoption level. For most projects, it will be necessary to project ABF and AFC for several years into the future in order to get a realistic estimate of the subproject's impact. This presents a quandary; to omit these projected benefits and costs would seriously distort the research impact, but it is difficult to project future values of a relatively young project. If the research phase of the project has already finished and the flow of benefits continues beyond the project termination date, a future maintenance cost that is often estimated as 10% of total cost of the project should also be included. The ABF and AFC must be expressed in constant monetary units using a common deflator factor. Once individual project IRRs are computed, the IRR for the CGS is obtained as the sample average of the IRRs. Using the sample standard deviation, confidence intervals for the IRR estimate can also be obtained. Average IRRs and confidence intervals can be calculated for each project category. Subproject IRRs can be examined to identify factors associated with successful or unsuccessful proj- ects. See box 6.2 for an example. Estimating the NPV and IRR Table 6.3 summarizes the steps typically involved to arrive at the Net Present Value (NPV) and Internal Rate of Return (IRR) of a research and/or extension project.18 · The first column simply indicates the number of years the technology under consideration is expected to last. It is not easy to precisely determine the life of the innovation. It is rec- ommended that the analyst try to identify the diffusion pattern of comparable technologies adopted in the past and discuss these issues with experts, such as local agronomists and extension experts. Table 6.3: Calculation of IRR for an Agricultural Research and Extension Project 1 2 3 4 5 6 7 8 9 10 Number Incremental Benefits Benefits Total of benefits per with without direct Indirect Total Net Year adopters adopter/year project project benefits benefits benefits Costs benefits 1 Column 2*3 Column 4-5 Column Column 6+7 8-9 2 3 ... 14 15 NPV IRR Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture). 59 3534-Rikka.pdf 6/20/05 2:51 PM Page 60 BOX 6.2. ESTIMATING INTERNAL RATE OF RETURN FOR SUBPROJECTS IN COMPETITIVE GRANT SCHEMES Subproject IRRs were estimated for the PRONATTA project in Colombia. IRR estimates were grouped by type of research project to see whether one type of project was more successful than another (see table below) This identifies means for improving future CGS programs. The average IRR for the CGS was 86% and was fairly similar across types of research, even though the IRRs for individual subpro- jects varied widely (see figure below). Subproject frequency distribution of IRRs Subproject IRR by type of research Type of research IRR (%) Mean SD Max Min Capacity Development 79 129 122 42 Adaptive Research 86 114 125 50 Applied Research 98 115 143 53 Entire CGS 86 106 112 65 Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture). · The second column reflects the number of adopters at each point in time and the third col- umn specifies the incremental benefits per adopter and year.19It is important to note that the incremental benefits are not necessarily constant over time. For example, perennials may have a specific production cycle with significant yield fluctuations. 60 3534-Rikka.pdf 6/20/05 2:51 PM Page 61 · In the next column the benefits with the project are estimated by multiplying the values in column 2 and 3. For a more precise estimate of the expected benefits of the project it is important that a realistic "without project" scenario is captured (column 5). If we take the example of a project which aims at breeding a high-yielding rice variety and at promoting its adoption, the calculation of project benefits should account for increases in rice yields which would have been achieved without the activities of the project. It is not likely that the yields would remain constant over time without the project. However, the estimation of these benefits without the project is hypothetical and based on many assumptions. Once again, the analyst should base his judgment on average yield increases in the past and on discussion with local experts. · The estimation of total direct benefits (column 6) is then simply based on subtracting the values in column 4 and 5. · The indirect benefits (column 7) accounts for spillover effects to non-targeted farmers. The effects capture the adoption of technology by farmers who have not been directly tar- geted by the project. Thus, total benefits (column 8) capture both total direct and indirect benefits. · Column 9 accounts for project costs over time. A subproject of a competitive grant scheme for example should not only include the direct costs of the respective subproject, but also a percentage of the overhead costs of the overall administration of the grant scheme.20 · By subtracting the costs from the total benefits the analyst receives the net benefits (col- umn 10). Finally, the values of the net benefits over time can be used to calculate the NPV and the IRR. 6.6. SOURCES OF NFORMATION I When conducting a full survey is not feasible, the practitioner must resort to alternative sources of information for estimating values for the required parameters: 1. Subproject information. Among the relevant subproject information the evaluation team must be familiar with the following information: Subproject Proposal, Advance, Technical (particularly experimental data), and Final reports as well as budgetary aspects. 2. Secondary information. Market prices and other economic variables including govern. ment policies are usually obtained from secondary sources. 3. Expert opinion. Expert opinion is a valuable source of information for the evaluation in the absence of a survey. It refers to a participatory method of eliciting information on the value of one or two parameters of interest through group or individual inter- views.21 Table 6.4 shows the minimum set of parameters to be estimated for a sound economic impact evaluation (ex-post economic evaluation) as described in previous sections. 61 3534-Rikka.pdf 6/20/05 2:51 PM Page 62 Table 6.4: Sources of Information on the Minimum Set of Parameters to be Estimated for a Sound Economic Impact Evaluation Parameter Information source and procedure Size of the RD Subproject documentation. If a proper monitoring system is in place the information should be readily accessible from advance reports. Another source of useful information are the technical personnel that participated in carrying over the subproject INB = increase in Net Benefits per UOA. Technical information: Project documentation and Experts Opinion Technical information on induced (farmers and technicians) changes and sources of information Economic information: Secondary sources and Expert Opinion Economic information on prices and (farmers and local merchants) markets Diffusion pattern. Requires three P0 from subproject documentation parameters: P1 and K from Expert Opinion P0 = proportion at the subproject start P1= proportion at the evaluation time K = Adoption potential (ceiling) AFC= annual flow of subproject costs Subproject documentation in constant monetary terms Secondary information Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture). 6.7. FISCAL ANALYSIS Sustainability of financing Even though a project may be economically and financially justified, it may not be sustainable if the fiscal support is not available. A crucial step in assessing the long-term potential for competitive grants, or any other project, is to assess the fiscal impact of the project. This should be done early in the project cycle. Fiscal analysis has become increasingly important because of stringent constraints on government budgets in recent years. Because of the diffuse nature, and the long lags that charac- terize research benefits, research and extension projects are often hard hit by austerity measures. The Project Appraisal Document should include a fiscal analysis aimed at assessing the likelihood that sustainable local resources and funds will be available at the end of the project to take over from funds provided under the project. This analysis should estimate the long-term impact of the proposed project on public expenditures and assess whether the project can be sustained with local funding resources. It is important that the analysis attempts to look beyond the implementation period. Obtaining sustainable funding has been a difficult challenge for past research projects to overcome. See box 6.3 for further details. The key steps in the fiscal analysis: 1. The first step in the fiscal analysis is: collect data on recent public expenditures for agricultural research and extension. 2. The next step is: estimate the level of public expenditures needed to sustain project activities after project completion. 62 3534-Rikka.pdf 6/20/05 2:51 PM Page 63 BOX 6.3: SUSTAINABLE FINANCING OF A COMPETITIVE FUND INECUADOR The Ecuador competitive fund project began full operation late in 1999 and continued until 2004. Prior to the project virtually all agricultural research funding came for the federal government budget as ded- icated funding to the National Agricultural Research Institute (INIAP). The project was very successful in attracting counterpart funding from participating research institutions, attaining a high of $2.6 mil- lion in 2003. The project induced a large increase in total research expenditures, but half or more of total expenditures were from loan proceeds. At the end of the project, research funding remained dependent on a highly politicized federal budget allocation process. Although the 2004 government agricultural research budget was the largest ever, all funds were dedicated to INIAP and the competitive fund program was closed. Much of the momentum from a project that was very successful in increasing agricultural research output has therefore been lost, and the future of the competitive fund remained uncertain. Govt. Ecuador World Bank Counterpart Total $US $US $US $US Year millions % millions % millions % millions 1995 3.4 100% 0.0 0% 0.0 0% 3.4 1996 4.3 100% 0.0 0% 0.0 0% 4.3 1997 4.5 98% 0.1 2% 0.0 0% 4.6 1998 3.3 95% 0.2 5% 0.0 0% 3.5 1999 1.2 47% 1.3 53% 0.0 0% 2.5 2000 1.7 44% 2.2 56% 0.0 0% 3.9 2001 2.9 47% 3.1 49% 0.3 4% 6.3 2002 3.4 37% 4.1 45% 1.7 18% 9.2 2003 2.8 30% 3.8 42% 2.6 28% 9.1 2004 5.9 63% 1.9 21% 1.5 16% 9.2 1995- 1998 avg.3.9 98% 0.1 2% 0.0 100% 3.9 1999- 2004 avg.3.0 45% 2.7 44% 1.0 100% 6.7 Source: Johannes Woelcke, World Bank. 63 3534-Rikka.pdf 6/20/05 2:51 PM Page 64 3. Total project costs should be separated into investment costs and recurrent costs. Computed factors can help to translate project costs into projected incremental public expenditures per year by multiplying the factors by the average annual project costs. To calculate a factor that reflects the proportion of the investment costs to be taken over by the government after the project ends, the investments can be categorized according to maintenance and replacement versus one-off investments such as foreign technical assistance.22 The calculation of a factor for recurrent costs can be based on the relation of the recurrent costs in the last year of the project to average annual recurrent project costs. 4. Once the annual incremental investment expenditures and recurrent costs have been estimated for each project component, they can be added up to estimate the projected total incremental public expenditure per year that will be required to sustain the research project. Once the projected incremental public expenditure needs are estimated, the source and form of the financing mechanisms must be considered. Cost recovery mechanisms have often been incorporated in competitive fund projects, but have generally fallen short of generating the amount of funding need- ed to fully support future activities. PROJECT IMPACT ON TAX REVENUE The analyst should attempt to capture the project's impact on tax revenue, and reflect this impact in the projected incremental public expenditure. The project contributes to increased general tax rev- enues if, for example, the use of taxed inputs or if a tax is imposed on farm profits (and the project is expected to increase profits). Because the actual level of cost recovery through the mechanisms mentioned above is uncertain, the analyst should perform some sensitivity analysis assuming differ- ent levels of cost recovery. In many cases, continuation of the project after the loan is closed would represent a significant increase in government commitment to agricultural research and extension. In times of tight pub- lic budgets these increases might be difficult to defend against competing uses of government funds. Hence, the analyst should include a section on the importance of the agricultural sector, and in particular of agricultural research and extension for achieving national goals, such as econom- ic growth and poverty reduction. The analyst might also refer to the high return on investment in research and extension that has been achieved internationally. In addition, the analyst could set the estimated public expenditures for agricultural research and extension in relation to total public expenditures and compare it to the expenditures of neighboring countries, and other developing or developed countries. In most cases, public expenditures on agricultural research and extension are still comparatively low.23 64 3534-Rikka.pdf 6/20/05 2:51 PM Page 65 7. CONCLUDING REMARKS Monitoring and evaluation are integral tools for managing and assessing the efficiency and effective- ness of investments in ARE systems and projects. Monitoring and evaluation of ARE project per- formance, outcomes, and impact has, however, been a significant challenge. However, successful project M&E can be carried out in a number of ways. Besides the mandatory use of Results Framework, description of results monitoring arrangements, and the official evaluations, the project management together with the relevant stakeholders can choose different M&E tools and implementers as well as the level of desired technical sophistica- tion. The choices depend on the Project Development Objective, stakeholders and their priorities, and particularly the resources available (i.e., time, staff, capacity, and finances). Besides the mandatory requirements, the six most important things to remember when planning and implementing project and program M&E include: At the design stage: 1. A strong and clear Project Development Objective is essential for good project design; 2. Select only a few key indicators--that are Time, Quantity and Quality (TQQ) bound, and that are realistic to collect considering the resources available; 3. Prepare the project M&E plan carefully. Give due consideration to stakeholder priori- ties, available resources, and capacity requirements; During the project implementation: 4. Pay attention to M&E issues during the supervision missions; 5. Implement M&E--collect and use M&E data for the benefit of project primary bene- ficiaries, project implementers, and project management; and 6. Integrate participatory M&E into the project M&E as much as possible, giving consid- eration to the priorities and available resources. 65 3534-Rikka.pdf 6/20/05 2:51 PM Page 66 8. REFERENCES Alex, G., and D. Byerlee. 2000. Monitoring and Evaluation for AKIS Projects. Framework and Options. Agricultural Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank. http://lnweb18.worldbank.org/ESSD/ardext.nsf/26ByDocName/MonitoringandEvaluationforAKISProjectsFramework andOptions/$FILE/m&e.pdf Anderson, J., and G. Feder. 2004. "Agricultural Extension: Good Intentions and Hard Realities." The World Bank Research Observer 19 (1). Babu, A., Y. Singh, and R. Sachdeva. 1997. "Establishing a Management Information System." In: Improving Agricultural Extension: A Reference Manual. Rome: Food and Agricultural Organization of the United Nations. Bantilan, M. C. S., and P. K. Joshi.1997. Integrating Research and Evaluation Efforts. Proceedings of an International Workshop. India: INCRISAT. Bellon, M. R. 2001. Participatory Research Methods for Technology Evaluation. A Manual for Scientist Working with Farmers. Mexico: International Maize and Wheat Improvement Center. CIMMYT, 2001. Binnendijk, A. 2000. "Results-Based Management in the Development Cooperation Agencies: A Review of Experience." Paper prepared for OECD/DAC Working Party on Aid Evaluation. Braun, A., G. Thiele, and M. Fernandez. 2000. "Farmer Field Schools and Local Agricultural Research Committees: Complementary Platforms for Integrated Decision-making in Sustainable Agriculture." AgREN Network Paper No. 105. Byerlee, D. 2003. "Conceptual and Practical Issues in Measuring Impacts of Sub-Projects in Competitive Grants Scheme (CGS)." Unpublished manuscript. Casley, D. J., and K. Kumar. 1987. Project Monitoring and Evaluation in Agriculture. A Joint Study of the World Bank, IFAD, and FAO. Baltimore, MD: John Hopkins University Press. CIMMYT 1988. From Agronomic Data to Farmer Recommendations: An Economics Training Manual. revised. ed. Mexico: CIMMYT. Deshler, D. 1989. "Future Risk Assessment: Issues Programming Opportunity." Journal of Extension 27 (1): Spring 1989. Deshler, D. 1997. "Evaluating Extension Programmes." In Improving Agricultural Extension. A Reference Manual. FAO. Fort, L., B. Martinez, and M. Mukhopadhyay. 2001. Toolkit: Integrating a Gender Dimension into Monitoring & Evaluation of Rural Development Projects. Washington, DC: World Bank. Goels V. K., E. Koryukin, M. Bhatia, and P. Agarwa. 2004. "Innovation Systems: World Bank Support of Science and Technology Development." Working Paper No. 32, World Bank, Washington, DC. Grosh, M., and P. Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 years of the Living Standards Measurement Study Vol.1. Washington, DC: World Bank. Grosh, M., and P. Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from 15 years of the Living Standards Measurement Study Vol. 2. Washington, DC: World Bank. Grosh, M., and P.Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries:: Lessons from 15 years of the Living Standards Measurement Study Vol. 3. World Bank, Washington, DC. Gubbels, P., and C. Koss. C. 2000. "From the Roots Up. Strengthening Organizational Capacity Through Guided Self- Assessment." World Neighbours Field Guide: Capacity Building. 2nd ed. Paragon Press. Guijt, I. 1999. "Participatory Monitoring and Evaluation for Natural Resource Management and Research." Socio- economic Methodologies for Natural Resources Research Best Practice Guidelines. Chatham, UK: Natural Resource Institute. http://www.nrsp.co.uk/Nrspweb/SEM/SEM%20BPGs/BPG04_Guijt_PM&E.pdf Guijt, I., and J. Gaventa. 2000. "Learning from Change. An Introduction." In Learning from Change: Issues and Experiences in Participatory Monitoring and Evaluation. M. Estrella, J. Blauert, D. Compilan, J. Gaventa, J. Gonsalves, I. Guijt, D. Jonson, and R. Ricafort (eds.), (pp. 1­15). Ottawa, Canada: Intermediate Technology Publications, London, UK, and International Development Research Centre. 66 3534-Rikka.pdf 6/20/05 2:51 PM Page 67 Hanson, J., R. Just, and J. Lainez.. 2004. "Evaluating a Publicly Funded, Privately Delivered Agricultural Extension System in Honduras." Working Paper 2003-04, Department of Agricultural and Resource Economics, University of Maryland, College Park, MD. Horstkotte-Wesseler, G., M. Maredia, D. Byerlee, and G. Alex. 2000. "Ex-Ante Economic Analysis in AKIS Projects." Methods and Guidelines for Good Practices. Agricultural Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank. http://lnweb18.worldbank.org/ESSD/essdext.nsf/26DocByUnid/5A74F2C4E3F9372185256B9E00146BA9/$FILE/Exan te_Analysis.pdf Horton, D., P. Ballantyne, W. Peterson, B. Uribe, D. Gapasin, and K. Sheridan, 1993. Monitoring and Evaluating Agricultural Research: A Sourcebook. Wallingford, UK: CAB International. Horton, D., R. Mackay, A. Anders, L. Dupleich.2000. Evaluating Capacity Development in Planning, Monitoring and Evaluation: A Case from Agricultural Research. ISNAR Research Report 17. The Hague: International Service for National Agricultural Research. IFAD 2003. A Guide for Project M&E. Managing Impact for Rural Development. http://www.ifad.org/evaluation/guide/index.htm Kumar, K. 1993. Rapid Rural Appraisal Methods. World Bank, Washington, DC. http://www.fao.org/docrep/ W3241E/w3241e09.htm Kusek, J. and R. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for Development Practitioners. Washington, DC: World Bank. Lecuit, L., J. Elder, C. Hurtado, F. Rantua, K. Siblini, and M. Tovo. 1999. Demystifying MIS--Guidelines for Management Information Systems in Social Funds. Washington, DC: World Bank. http://imagebank.world bank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000094946_99092405344364 Maredia, M., D. Byerlee, and J. Anderson. 2000. "Ex Post Evaluation of the Economic Impacts of Agricultural Research Programs: A Tour of Good Practice." Paper presented to a workshop on "The Future of Impact Assessment in the CGIAR: Needs, Constraints, and Options," CGIAR Technical Committee, May 3-6, 2000. Rome: FAO. Mosse, R., and L. Sontheimer.1996. "World Bank's Performance Monitoring Indicators Handbook." World Bank Technical Paper No. 334, World Bank, Washington, DC. Murphy, J., and T. Marchant.1988. "Monitoring and Evaluation in Extension Agencies." World Bank Technical Paper No.79, World Bank, Washington DC. Rajalahti, R., J. Woelcke, and E. Pehu. 2005. "Development of Research Systems to Support the Changing Agricultural Sector." Proceedings of a Research Workshop, World Bank, Washington, DC. Salmen, L. F. 2000. Beneficiary Assessment for Agricultural Extension. A Manual of Good Practice. Agricultural Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank. Score, M. 1995. "Social Impact Assessment in Extension Educational Programming." Journal of Extension. 33(6): 4 p. http://www.joe.org/joe/1995december/tt2.html Souvanthong, Pheng, and R.Trethewie.2001. Guide to Participatory Extension Methods Used in Community Forestry in Laos. Mekonginfo, 2001. Srivastava, J., J. Kampen, G. Baker, D. Byerlee, and M. Sehga. 2003. Good Practices in Monitoring & Evaluation and Management Information Systems for Competitive Grant Programs in ECA Countries. Washington, DC: World Bank. http://imagebank.worldbank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000090341_ 20040331111629 UNDP. 1997. Who are the Question-makers? A Participatory Evaluation Handbook. OESP Handbook Series. http://www.undp.org/eo/documents/who.htm Veldhuizen, L., A.Waters-Bayer, and H. Zeeuw. 1997. Developing Technology with Farmers: A Trainers Guide for Participatory Learning. London: Zed Books Ltd. Vernon, R. 2001. Knowing Where You're Going: Information Systems for Agricultural Research Management. The Hague: International Service for National Agricultural Research. 67 3534-Rikka.pdf 6/20/05 2:51 PM Page 68 Wood, S., and P. Pardey. 1997. "Agro-ecological Aspects of Evaluating Agricultural R&D: EPTD Discussion Paper No.23, International Food Policy Research Institute (IFPRI), Washington, DC. World Bank. Operational Manual. http://wbln0018.worldbank.org/institutional/manuals/opmanual.nsf/. World Bank. 1992. "Designing and Implementing Agricultural Extension for Women Farmers." A Technical Note. Women in Development Division, Population, and Human Resources Department, World Bank, Washington DC. World Bank. 1997. The Logframe Handbook. A Logical Framework Approach to Project Cycle Management. http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf World Bank. 2001. Thematic Brief: Agricultural Knowledge and Information Systems; Agricultural Research, Extension, and Education, 2001.Washington, DC: World Bank. World Bank. 2003a. Extension and Rural Development--Converging Views on Institutional Approaches? Workshop Summary. Agriculture and Rural Development Department, World Bank, Washington DC. World Bank. 2003b. "Monitoring and Evaluation Component in Agricultural Operations. A Toolkit. Agriculture and Rural Development." Department Draft, World Bank, Washington, DC. World Bank. 2004a. Agriculture Investment Sourcebook. Washington, DC: World Bank. World Bank. 2004b. Investments in Agricultural Science and Technology. Agriculture Investment Sourcebook. Module 2. Washington, DC: World Bank. 68 3534-Rikka.pdf 6/20/05 2:51 PM Page 69 APPENDIX 1. NARRATIVES OF NEW APPROACHES AND INSTRUMENTS IN ARE SYSTEMS The recent changes in the agricultural research and extension systems are primarily driven by changing global and local needs as well as the overall need to address poor fiscal sustainability, dif- ficulties with attribution of impact, low political commitment, insufficient local content, weak accountability or transparency, and weak linkages between research and extension, generally asso- ciated with ARE systems.24The changes, such as decentralization, privatization, democratization, and increasingly separate functions of financing and delivery, have resulted in a slow transition from supply-driven technology generation toward building a more demand-driven agricultural inno- vation system (AIS). Agricultural innovation system essentially links multiple sources of innova- tion and uptake pathways along the continuum from basic research to technology adoption recognizing the innovative capacity of all stakeholders from producers to the marketplace. Thus, focus is not on the science suppliers but on the totality of actors that are involved in innovation. See box A1.1 for further details. BOX A1.1: AGRICULTURAL INNOVATION SYSTEM A national innovation system is made up of the institutions, enterprises, and individuals that demand and supply knowledge and technologies; and the rules and mechanisms by which these different agents are interacting. The focus is not on the science suppliers but on the totality of actors that are involved in innovation. The ultimate objective of a well functioning innovation system is to serve the needs of the economy by achieving better integration of the S&T infrastructure with production needs; by increasing private sec- tor participation in technology development; and by developing stronger linkages between industry, uni- versities, and research institutions. The past approaches (e.g., NARS and AKIS) have strengthened research systems that increase the avail- ability of new knowledge and new technologies, but not necessarily the number of innovations that will be implemented; also the role of market and private sector has not been very strong. A lot of changes, many of which have tried to ensure that new knowledge is relevant in the market context and that the role of the private sector in the development and diffusion of new knowledge is explicitly recognized, have taken place. Source: Goel V.K., E. Koryukin, M. Bhatia, and P. Agarwal 2004. A number of instruments and approaches, such as contracting for services, competitive research grants (CRGP), participatory and farmer-to-farmer R&E, and communication media, are being introduced into many ARE systems in order to facilitate the change. The approaches and instruments are often used simultaneously and may overlap to some extent. Table A1.1 summarizes the potential of selected instruments to address the various challenges. 69 3534-Rikka.pdf 6/20/05 2:51 PM Page 70 Table A1.1. Potential of Selected Innovative Instruments to Address Various Fiscal and Institutional Challenges Associated with Agricultural Research and Extension Systems Challenges with Competitive Use of Agricultural Research Contracting Research Participatory Farmer-to- Mass Media and Extension Systems Services Grants R&E Farmer25 and ICT26 Fiscal sustainability M-H M-H M M-H M-H Attribution of impact H H H M-H M-H Political commitment M M M M L Responsiveness--local content H M-H H H M-H Accountability---transparency M-H H M-H M H Weak linkages between R&E M H H H L Poverty reduction M-H H M-H M M Specific targeting H H H M-H L-M Enhanced private sector role H H M M M-H Note. Potential of the instrument/approach to address the challenge: low (L), medium (M), and high (H). Source: Riikka Rajalahti, World Bank. Contracting extension services Contracting for agricultural extension by the public sector takes many forms and may involve con- tracts with public sector agencies, non-governmental organizations, universities, extension consult- ing firms, or rural producer organizations. In contracting systems, in general, the agency--such as public funding agency--draws up terms of reference and details of services to be provided, and then contracts for them, usually on a competitive basis. Services to be contracted are usually identified in consultation with users, although the programs tend to be longer term and more program-oriented. Contracts may be administered by national governments, the national government in collaboration with lower level government, or by governments with an NGO that then contracts with private com- panies. There can be some "reverse" contracting of public extension agents, often subject matter spe- cialists, by NGOs or private sector and farmer organizations.27 Competitive research grant programs In CRGPs research providers are selected on a competitive basis, using call for proposals and scien- tific peer review mechanisms to allocate funding. CRGP complement "core" funding or "block" grant funding, which annually allocates funds to specific public research organizations for their core research programs, infrastructure, and human resources. CRGPs are used as financing mechanisms to mobilize a wide range of research institutions for work on priority areas, develop institutional link- ages and research capacities across organizations, and to link scientists with users of new technolo- gies. In the context of agricultural extension, governments typically continue to finance many rural extension services, but provision of services is more commonly contracted to private advisory serv- ice firms, NGOs, universities, producer organizations, and other groups. Competitive procedures can improve quality of services, make providers more accountable for results, and improve efficiency whereas contracting allows for specialization and selection of service providers according to their individual comparative advantage.28 70 3534-Rikka.pdf 6/20/05 2:51 PM Page 71 Participatory research and extension Participatory research and extension comes with many names, such as participatory technology development (PTD) and local agricultural research committees (CIAL), and with many levels of par- ticipation. Participatory technology development29 refers to the process in which development work- ers facilitate the generation and dissemination of agricultural innovations together with rural women and men. Through this interaction, the partners try to increase their understanding of the main traits and dynamics of the local farming systems, define priority problems and opportunities, and experi- ment with a selection of "best-bet" options for improvement. The options are based on ideas and experiences derived from both indigenous knowledge and formal science. This process of technolo- gy development is geared toward finding solutions to current problems and developing sustainable agricultural practices that conserve and enhance the natural resources. An important aspect is that PTD should strengthen the capacity of farmers and rural communities to analyze the ongoing processes and develop relevant, feasible, and useful innovations. Local agricultural research committees30 are participatory platforms for improving decision-making capacity and stimulating local innovation for sustainable agriculture. They are a permanent agricul- tural research service staffed by a team of four or more volunteer farmers elected by the communi- ty. The committees create a link between local and formal research. The CIAL process is iterative and consists of a number of steps that are all supported by facilitation, monitoring, and evaluation. The strength of the CIALs lies in their systematic evaluation of technological alternatives and their ability to influence the research agendas of formal research and extension systems. Participatory farmer-to-farmer approaches Farmer-led extension or farmer-to-farmer extension refers to extension methodology that relies either on informal or formal training or transfer of technologies, or advice by a farmer to another farmer or a group of farmers. Usually the key farmer receives some technical and methodological training by extension services and may be partially or fully compensated for this service in kind by monetary remuneration. Farmer Field School (FFS)30 is likely the best known farmer-to-farmer extension method. FFS offers community-based, non-formal education to groups of 20­25 farmers, who are expected to transfer their knowledge and skills to other farmers in their communities. Discovery-based learning is relat- ed to agroecological principles in a participatory learning process throughout a crop cycle. FFS fill gaps in local knowledge, conducts holistic research on agroecosystems and increases awareness and understanding of phenomena that are not obvious or easily observable. FFS require significant insti- tutional commitment and support, usually provided by the national extension service. 71 3534-Rikka.pdf 6/20/05 2:51 PM Page 72 APPENDIX 2. IMPLEMENTATION OF A MONITORING AND EVALUATION PLAN IN THE BANK'S PROJECT CYCLE: PUTTING IT ALL TOGETHER THE BANK'S PROJECT CYCLE AND MPLICATIONS FOR I M&E The Bank's projects are financed and implemented according to a well-documented project cycle. The cycle is an eight-step process that provides participants and stakeholders with a full view of the project development process, planning, and achievement of outcomes from the identification and concept stage to the post-project completion stage, including full implementation. Therefore, plans for M&E should be evident at every stage of the eight-step project cycle.31 See Figure A2.1 for a dia- gram of M&E Steps in the World Bank Project Cycle. Although M&E are the responsibility of the borrower's project management team, the Bank task team has a critical role in assisting the client team in setting and achieving M&E objectives. These include: (a) help identify information needs and specify monitoring objectives; (b) ensure that all documents in the project cycle clearly establish M&E plans and expect- ed results for project activities and project coordination; (c) review the design and organizational arrangements of M&E information system along with needed special studies to ensure feasibility; (d) help to finance technical assistance and equipment required for the design and imple- mentation of M&E activities; and (e) ensure that training and relevant capacity needs are adequately addressed. Guidelines for Sound M&E in Innovative ARE Projects For Agricultural Research and Extension projects, M&E is often needed as permanent elements of country programs in view of the institution building objectives of some programs. Therefore, an M&E system must contribute substantially to enhance program sustainability and set the pace for establishing M&E permanency within country programs. Each stage of the Bank's eight-step project cycle requires specific action planning and special atten- tion to the development and evolution of a project M&E system and use of indicators. The follow- ing are necessary elements that must be visible at all stages of the project cycle: · During the preparation, sufficient time and resources must be allocated toward establish- ment of a gender-disaggregated baseline; · The Bank task team must identify and secure an M&E resource person in its core team to provide support in the evolution and implementation of M&E systems; where financial resources are a constraint, M&E must be featured in the TOR of an appropriate mission member; · The client team must include a political champion or leader from an implementing Ministry whose role would be to: (a) promote results-based M&E system and ensure the institutionalization and sustainability of developed systems or convince of its importance; and (b) provide the scope for in-service training and capacity building programs; 72 3534-Rikka.pdf 6/20/05 2:51 PM Page 73 · Targeting different impact areas, such as gender, ethnicity, socio-economic equity, partici- pation, and consultation should be considered at all stages of the project cycle and can sig- nificantly influence the project design. Indicators sensitive to such targeted areas may determine the extent to which the extension and research services take these issues serious- ly and ensure an efficient functioning of systems developed; · Sufficient donor support in the implementation and funding of M&E activities such as modernizing or developing MIS systems and statistical capacity increases the value of M&E. The following are proposed means of implicating M&E at each stage of the project cycle: 1. CAS level identification: The Bank prepares lending and advisory services for a specific country based on selectivity framework and areas of comparative advantage, targeted to country PRSP efforts. The process begins when sector preparation groups identify specific sector deficiencies and a range of possible areas or themes for project intervention to be initiated over a three-year CAS period. 2. Identification: Projects are identified that support strategies and that are financially, economical- ly, socially, and environmentally sound. Developed strategies are analyzed. Diagnostic case studies are conducted using rapid/participatory rural appraisal to assess problems and potential constraints, opportunities, special issues with farmers and other stakeholders in a given agro-ecological area. The project concept note (PCN) sets the pace for developing M&E indicators, and subsequently provides the scope and elements of the project M&E system. Results of case studies/theme proposals feed into project baseline information, identifies opportunities, special scenarios, and other stakeholders. 3. Preparation: The Bank provides policy and project advice along with financial assistance. Clients conduct studies and prepare final project documentation. More comprehensive quantita- tive surveys are carried out to determine the extent and need for intervention and importantly, provide baseline data for the research environment and for selected extension areas. These can either be narrowly focused to capture specific themes relevant to project components and relat- ed project activities, or broadly targeted to capture different but related themes and opportunities that warrant the need for subsequent or additional interventions. They may include situation analysis and participatory needs assessment to appraise current farm households, local condi- tions as well as institutional/organizational issues in the research/extension continuum. Within the framework of such studies, it will be necessary to assess prospects for institutional reforms and technological changes to impact the current production environment. All preparation relevant to M&E, including preliminary discussions of all study results, should be undertaken and com- pleted by pre-appraisal. 4. Appraisal: The Bank assesses the key parameters for investment and prepares the PAD and draft legal documents. Preparation follows a pre-appraisal phase and subsequently an apprais- al phase. In between the two phases, the project appraisal document (PAD) is developed. The most important feature for M&E in the PAD is the logical framework which establishes outputs, outcomes, and impact indicators with established baselines and controls. Although the Bank has now shifted from the log frame to the results framework, which is a more simplified version of the log frame, it is strongly recommended for ARE projects that an extended log frame be pre- pared with indicators to capture the outputs and outcomes of all activities by component. A results framework may then be derived from the log frame to satisfy Bank requirements. The PAD 73 3534-Rikka.pdf 6/20/05 2:51 PM Page 74 should also provide a description of M&E arrangements, including a M&E plan and Management Information System, for the entire project phase. Pre-appraisal phase: M&E arrangements initiated at pre-appraisal and firmed up at appraisal include the following: · Development of M&E framework and detailed M&E plan which will include: · M&E system and how it functions in the context of the project · M&E information flow, sources, and uses of data · Key players in M&E project activities · Reporting formats for M&E · Timeframe for key information requirements and follow-up studies · Frequency for baseline updates and control group monitoring as needed; · Development of M&E manual of procedure; · Initiate an assessment of M&E institutional and human capacity including capacity to inte- grate gender and relevant target areas into project activities; · Develop a plan for M&E capacity enhancement; and · Develop M&E training program. Appraisal phase: During appraisal, the Bank and the borrower will: · Discuss and agree on (a) log frame/results framework indicators and identified targets for outputs, outcomes and impact and (b) established baselines and controls; · Through a validation workshop, finalize and adopt (a) the M&E framework and (b) the M&E manual; and · Review with client, the results of the capacity assessment study and agree on a capacity enhancement plan and staff training program. 5. Negotiation and board approval: The Bank and borrower agree on loan/credit proceeds and presents to the board for approval. The implementation of the loan/credit is negotiated based on agreed upon plans and presented to the board for approval. 6. Implementation and supervision: The borrower implements the project and the Bank supervis- es loan with due regard for economy, efficiency and effectiveness. Pre-implementation: Between board approval and actual implementation start-up, there is a period during which the project becomes effective. The following will need to be accomplished during the pre-implementation or project effectiveness period: · M&E Sub-unit within the implementing/coordinating unit of the project need to be estab- lished and fully equipped; · M&E focal points established at decentralized locations of project implementation; · Because M&E requires the participation and contribution of all project staff, at least the initial M&E training program for project staff should be completed; and · Workshop held for project staff to internalize M&E manual. 74 3534-Rikka.pdf 6/20/05 2:51 PM Page 75 Implementation: Indicators that have been developed are now used to monitor implementation progress by activity. Using the information flow chart and the reporting formats agreed upon, feed- back is circulated to support program management and stakeholders. The central data processing and information consolidation point is the M&E subunit of the project coordinating/implementation unit--usually under the implementing Ministry. The M&E focal points in the respective implement- ing agencies must provide routine reports on progress (inputs, outputs, outcomes) to the M&E sub- unit within the project management unit. Consolidated reports and analyzed data complemented with information on progress from periodic assessments, case studies, and activity appraisals, are put at the disposal of project management for dissemination and action as appropriate. Periodic updates of baseline studies are necessary to monitor and capture changes in selected areas resulting from proj- ect interventions and other factors. Supervision: Supervision/implementation support missions must include an M&E resource person or a core team member with the ability to review: (a) M&E activities; (b) the functionality and ade- quacy of M&E systems; and (c) ensure that project indicators are being monitored adequately by M&E focal points at central and decentralized levels. Indicators and their respective targets should be reviewed to ensure that planned project activities are on track in terms of planned levels of out- put realizations, achievement of outcomes, planned timeframes, and use of allocated resources. Specific roles of M&E during the supervision missions include: · Systems adjustment: M&E systems once designed and operational, should not be an end in itself; it should have some scope for adjustment in terms of (a) new issues that might emerge during implementation, and (b) lessons on the system operation and the functioning that may be incorporated into the design. Supervision missions will provide the scope for such adjustments; · Capacity adequacy: Missions will review, assess, and analyze capacity adequacy for M&E support team and propose training needs and additional skills as necessary; · Response capacity: Missions will review and assess the response capacity of high-level client/partner officials on M&E implementation process and propose ways of enhancing apparent deficiencies; · Sensitivity to targeted areas: Mission will pay particular attention to progress achieved in gender/ethnicity/poverty sensitive indicators. In particular, the extent to which the project has succeeded in integrating these target areas into project implementation; and · Guidance to Bank Task Team: on any salient M&E issue that might emerge from either client, wider stakeholder community, or members of the Bank core team. 7. Implementation and completion: The implementation completion report is prepared to evaluate the performance of both Bank and the borrower. Each party is required to prepare and submit an Implementation Completion Report (ICR). Institutionalization of M&E actions within implementing Ministries as well as harmonizing the M&E systems of various projects within sector Ministries has specific implications for project completion and post-project activities. M&E must not end at proj- ect completion--mainstreaming M&E within implementing Ministries establishes M&E as an ongo- ing function of government policy units and technology programs. Lessons must continuously be drawn from evaluation efforts throughout project implementation. A key indicator for project sus- tainability and continuity or mainstreaming project activities with regular functions of the Ministries, 75 3534-Rikka.pdf 6/20/05 2:51 PM Page 76 is the extent to which the project M&E system has been established and used at project completion and for post-project activities. Box 5.1 in chapter 5 provides specific highlights for M&E during ICR preparation and write-up. 8. Evaluation: The Bank's independent OED audits and evaluates the project. Analysis feeds into future project design. OED should pay particular attention to the ratings that have relevance to M&E implementation and outcomes. Figure A2.1. Diagram of M&E Steps in the World Bank Project Cycle. 8 Project Evaluation 7. 1. Independent OED audits Implementation And CAS level And evaluates project Completion identification ICR evaluates performance Bank prepares lending Of both Bank and Borrower and advisory services M&E FOR AR & E 6. 2. Implementation by borrower In the Bank's Project identified in line Supervision by Bank with CAS and PRSP due regard to efficiency Project Cycle PCN prepared & effectiveness 5. 3. Negotiations and Project Preparation Board Approval 4. Bank provides policy Loan/credit proceeds Appraisal and Project advice are agreed upon Pad and draft legal along with financial asst documents prepared by Bank Source: Adapted from http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTLAC/0,,menuPK:257867~pagePK:64089870~piPK:6408 9874~theSitePK:257804,00.html 76 3534-Rikka.pdf 6/20/05 2:51 PM Page 77 APPENDIX 3. A BRIEF DESCRIPTION OF THE "EXTENDED" LOGICAL FRAMEWORK What is log frame? · Log frame is a program management technique used by the World Bank (since 1997) and other development organizations to manage the complete project cycle from design, imple- mentation, monitoring, and evaluation. · The log frame is a "cause and effect" model of project interventions to create desired impacts for beneficiaries. Why use it? · It is used to develop the overall design of the project, to improve project implementation monitoring, and to strengthen period project evaluation. It is a tool that has the power to communicate the essential elements of a project clearly and succinctly throughout the proj- ect cycle. · Developing the log frame helps the project team and the client to precisely define the dif- ferent components and how they are linked and contribute to the Project Development Objective (PDO). Components of the log frame: · In the Bank's log frame, the multi-step process by which investments lead to impacts is compressed into a four-step format under a 16-box matrix as presented in Table A3.1 below (excluding the rows on project component statements). · The matrix presents the cause-effect-logic between inputs (financial, human, and material resources), activities (tasks personnel undertake to transform inputs to outputs), outputs (products and services produced), Development Objective (effects of outputs on target groups), and Country Assistance Strategy goal (long-term, widespread improvement in society--impact). · In addition, consideration is given to the critical assumptions--the external risk and enabling factors outside the direct control of the project. · The basic logic is that when the critical assumptions hold, the project interventions through inputs and activities generate component outputs. The outputs, in turn, are expected to result in desired impacts (project objective level outcomes and impacts) for beneficiaries. To track the achievement of each objective, a set of performance indicators will be set for each objective level. · Information regarding the use of log frame in program design can be found in detail in: The Log frame Handbook. A Logical Framework Approach to Project Cycle Management. http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf 77 3534-Rikka.pdf 6/20/05 2:51 PM Page 78 Why "extended" logical framework? · Although it is mandatory to develop the results framework and the arrangements for results monitoring, the project team may also develop full log frame that introduces an added level of detail for component inputs, outputs and outcomes. · The "extended log frame" approach may help the project team and the client to precisely define the different components and how they are linked and contribute to the PDO. · The extended log frame can be a useful tool in facilitating discussions and development of the project design with the national team. · It may also assist and complement project management, and may support the development of a sustainable M&E system beyond the lifetime of the project as it often better fits the requirements for M&E in the client countries. · Table A3.1 below provides an example of the "Extended" Logical Framework. Table A3.1. The "Extended" Logical Framework Monitoring & Narrative Summary Performance Indicators Evaluation Plan Critical Assumptions CAS Goal Sector goal level indicator The program evaluation system. CAS Goal to Super Goal Project Development Outcome indicator People, events, processes, sources PDO to CAS Goal Objective (PDO) of data for organizing the project evaluation system. Project Component Intermediate Outcome People, events, processes, sources Project Component Statement of data for organizing the project to PDO M&E system. Outputs Output indicator People, events, processes, sources Output to Project of data--supervision and Component monitoring system for project implementation. Project Component Inputs (budget for each People, events, processes, sources Component Activity Activities component/activity) of data and monitoring system to Output for project design. Source: Authors. Adapted from The Log frame Handbook. A Logical Framework Approach to Project Cycle Management. 78 3534-Rikka.pdf 6/20/05 2:51 PM Page 79 APPENDIX 4. LIST OF INDICATORS AND DATA SOURCES FOR ARE PROJECTS A. DIFFERENT INDICATOR LEVELS Budget and input level: Tracking budgets and input use is not only an important element of sound project planning and monitoring, but also crucial element in the evaluation of project suc- cess. During the planning process budgets and input requirements are established as part of pri- ority setting. These cost projections (budgets) must be linked to expected outcomes and are an essential element for evaluation of any program, as benefits are assessed in relation to program cost. Accurate and timely budget and expenditure data are important to the efficiency and effec- tiveness of a program evaluation. Expenditure data are commonly used and helpful in tracking implementation. The required information for the preparation of the log frame or results frame- work can be easily extracted from the cost tables. If used alone, however, it is an inappropriate and insufficient indicator of project implementation progress and performance. Based on the pro- jected and actual budgets two important evaluation methods must be highlighted here: ex ante and ex post economic analysis. Ex ante economic analysis relating project costs and benefits is a standard means of evaluating whether the proposed investments are financially and economi- cally justifiable. Ex-post economic analysis uses similar methodology to evaluate investment impacts (see chapter 6). Output level: Outputs are direct products of project activity and within the control of the project. Project output level M&E focuses on relatively simple quantitative measures of productivity and physical completion rather than on the qualitative assessment of the goods produced. Data on out- puts should come from the management information system (see Chapter 3). Bank-supported agricultural R&E projects typically focus on two objectives: (a) institutional devel- opment to strengthen the institutions and technology system necessary to develop and disseminate improved technology and management in the agricultural sector; and (b) productivity change due to technological innovation introduced through the technology system The two objectives are related and mutually supportive, but require different indicators for outputs and outcomes. An illustrative list of key indicators, with particular focus on CRGS and contracting extension services, is included in section B. Outcome level: Outcomes can be defined as the intermediate effects of outputs on clients but are not under full control of the project. Although the measurement of outputs is mainly based on simply quantifiable indicators, the measurement of outcomes adds a quality dimension which is more diffi- cult to quantify. The newly introduced results framework requires the definition of outcomes and out- come indicators not only at the PDO level, but also at the component levels. Component level: Outcomes and outcome indicators specify the expected benefit for specific target group from component implementation. In AIS projects, the target group of components are very often researchers and extension agents or institutions whose performance are expected to be improved to better serve their respective clients, agricultural producers. Hence, combined outcomes at the component level contribute to the desired outcome at the PDO level. At the PDO level, outcome indicators reflect the initial results or intermediate impacts of the overall project. 79 3534-Rikka.pdf 6/20/05 2:51 PM Page 80 While formulating the outcome statements of the different components it is important to note that the various outcomes are not independent of each other but instead reinforce each other to con- tribute to the overall objective of the project. In many recent Bank-supported ARE projects, the project components focus on institutional development outcomes, and the PDO focuses on productivity outcomes. However, it is critical to recognize that a PDO can also focus on institutional development outcomes directly. Institutional development outcomes vary considerably depending on the technology system needs and the pro- posed project design. These may reflect changes in the institutions themselves, institutional pro- ductivity, or institutional relations with clients. Project outcomes cannot always be reported on a routine basis by a project management information system. For this reason, special studies are usually necessary to measure outcomes. Chapter 3 and section C below provide more details on data sources. Sector goal indicators: Sector goal indicators are not required any longer for Bank-supported proj- ect M&E when a results framework is being developed. However, selection of sector goal indicators may still be useful (a) as project documents have to describe how the project contributes to higher level objectives and (b) when the project team and in particular the counterparts in client countries decide to implement a permanent M&E system with a long-term perspective based on the log frame approach. Sector goal indicators measure change in the broad development goal to which the project con- tributes. Changes in these indicators are indirectly attributable to the project, and usually can- not be measured during the life of a Bank-financed project. Although demonstrating direct project impact on a goal level indicator may not be within the "manageable interests" of a proj- ect, it is important the project provides a plausible hypothesis of how project activities will lead to change at the level of the sector goal. This often requires other sources of research and analy- sis to build a plausible link between project activities and the social goal. Ex post impact analy- sis are important to demonstrate past success and guide future investments. The trick with impact assessment is that of comparing the "with project" situation with the counterfactual "without project" scenario. Sector-level impact indicators are similar for research and extension projects. Despite the difficul- ties with attribution and detecting change, R&E projects should link activities to impacts on indica- tors of social goals as a basis of justifying investment. Models may be useful in relating R&E programs to changes in social goals and in estimating impacts that are not otherwise readily meas- urable (Wood and Pardey 1997). Methodologies for use of such models are still under development, and more work may be needed to develop practical models that bridge the gap between measurable outcomes (adoption and effects of adoption) and impacts on social indicators (Maredia, Byerlee, and Anderson 2000). Development and verification of such models are important tools for assessing var- ious influences on agricultural production processes, and may be appropriate for financing under R&E projects. Such modeling is likely to be more appropriate and useful as a research activity with- in a country research program, and less practical for estimating impacts for a Bank-financed project M&E system. 80 3534-Rikka.pdf 6/20/05 2:51 PM Page 81 B. INDICATOR LIST The following is a menu of possible goal, outcome, and output indicators for agricultural research and extension projects. The list includes social, environmental, and institutional-capacity-building indicators. It is important to note that the list is not exhaustive but provides suggestions. · Goal level indicators are not mandatory any more for World Bank-supported projects. However, these indicators may be of interest if the Bank team chooses to use the log frame approach as well as for the counterparts in the client countries who would like to imple- ment an M&E system like the project. · Due to cost and time considerations it is recommended that only a limited number of rele- vant indicators should be selected to meet the needs of specific projects. · The level (especially for PDO outcomes vs. component outcomes; and outputs vs. out- comes) at which a specific indicator is relevant may vary depending on the specific project or program objectives. 1. Goal level indicators (typically not monitored by projects) All ARE Projects Goal Indicator Measurement Source · Rural population below (national) poverty line % number GDB/CDB · Rural population with incomes < $1/< $2 per day % number GDB/CDB · Daily per capita dietary energy supply Calories/capita GDB/CDB · Malnutrition prevalence (rural and urban) % number GDB/CDB · Agricultural GDP value, % growth / year GDB/CDB · Agricultural GDP / agricultural worker value, % growth / year GDB/CDB · Agricultural GDP / rural worker value, % growth / year GDB/CDB · Agricultural exports value, % growth / year GDB/CDB · Agricultural employment (full-time; part-time) % or number GDB/CDB · Freshwater withdrawal for agriculture Water use / agricultural GDP; GDB/CDB total; % of total available · Deforestation % per year; GDB/CDB % change in rate over time · Agricultural land loss/total arable area % change in rate over time/ GDB/CDB % per year · Land use pattern % change in rate over time GDB/CDB Source: Authors. Adapted from Alex and Byerlee (2000). 81 3534-Rikka.pdf 6/20/05 2:51 PM Page 82 2. Outcome indicators (Level: Component and Project Development Objective) Extension Projects--Institutional Development Outcome Indicator Measurement Source · Co-financing of extension service by farmers (disaggregated by % of total cost PDB farm size, gender, age, farm type, income) · Co-financing of extension service by providers (disaggregated by % of total cost PDB type of service providers, e.g., private enterprise, NGO, producer organization, public agency) · Agencies providing extension services (disaggregated by type of Number, % of total number PDB service provider, e.g., private enterprise, NGO, producer of subprojects organization, public agency) · Regions covered by (decentralized) extension system (or covered Number of projects in each PDB by extension projects) region; % of total · Women, youth, and minority groups reached by extension Number, % of total PDB / SS services or participating in extension subprojects · Priority areas captured (as defined at national or regional level) Number of extension activities/ PDB through extension system / extension projects subprojects per priority area · Quality/skills of extension providers (natural resource Number of certified PDB / SS management, participatory technology development, enterprise extension agents development & marketing, organization skills) · Environmental topics captured through extension system Number of extension PDB activities/area · Motivation expressed by the extension staff % change PDB / SS · Perceived fit (by extension staff) between supply and demand % change PDB / SS for services (priorities/geographical area/target groups covered) · Intensity of extension coverage Families/agent or area/agent PDB / SS · Extension cost per farmer, per hectare, or per unit value Cost per unit PDB / SS of production · Efficiency of extension activity (demonstrations, training, field days) Number/agent, Cost / activity PDB / SS · Gender balance of extension staff % women PDB / SS · Ethnic balance or minority language capability of extension staff % minority or language capable PDB / SS · Non-salary operational costs as a percentage of total extension % of total budget CDB / PDB budget (national and project level) · Service provider awareness of competitive fund for extension Number of application per call PDB service provision and capacity to apply successfully (disaggregated and number or % accepted by type of service provider) · Farmer evaluation of extension providers Ratings SS · Farmer willingness to share extension costs Number or % change, % total SS · Farmer requests for extension services (disaggregated by gender, Number, % of total PDB / SS minority groups) 82 3534-Rikka.pdf 6/20/05 2:51 PM Page 83 Extension Projects--Institutional Development--continued Outcome Indicator Measurement Source · Farmer awareness of extension activities (disaggregated by % of farmer SS gender, minority groups) · Farmer, private sector, and civil society representation on extension management committees (by sex, ethnicity, farm size, wealth) Number, % SS / PDB · Functioning M&E system with active participation of executing institutions Number of impact studies; PDB / ER / number of progress reports; SS general assessment of system · Efficiency of fund administration unit Overhead costs (total and PDB % of total project costs); Average number of weeks or month from call for proposals and signing contract Extension Projects--Technology Dissemination Outcome Indicator Measurement Source · Adoption of new practices or technologies (disaggregated by Total number of adopters PDB/SS participants and non-participants); including studies of rationales (including spillovers); for adoption / non-adoption by gender, minority group, age % of targeted farmers and all farmers · Change in income, welfare, costs, yield, productivity, as a result % change from base SS of new technologies or practices (disaggregated by gender, minority groups, age) · Change in farm/household labor needs (disaggregated by gender, % change from base SS minority groups, age) · Change in relative income from on-farm and off-farm activities % change from base SS (disaggregated by gender, minority groups, age as appropriate) · Adoption of environmentally-sound technologies Area (ha); number of farmer; PDB/SS reduced use of agrochemicals (in liter or kg); reduced erosion (soil loss in kg per area) · Change in farmers organizations (autonomy, leadership, funding, % change, Number SS membership, participation, planning, activities) · Change in farmers/organizations/communities' ability to % change from base, Number SS market products · Changes in use of natural resources (water, forest, land) Rate of use; area SS · Yield gap between farmers' yields and on-farm trials % difference SS · Yield gap between national yields and neighboring countries % difference CDB continued on next page 83 3534-Rikka.pdf 6/20/05 2:51 PM Page 84 Extension Projects--Technology Dissemination Outcome Indicator Measurement Source · Time-series or case studies of change in farmer knowledge, % change from base SS skills, attitudes, or understanding of technologies and practices · Change in land area used for sustainable, integrated production Area (ha) PDB/SS · Farms/farmers participating in environmental farm plan programs Area (ha); number of PDB/SS farms/farmers Research Projects--Institutional Development Outcome Indicator Measurement Source · Institutions engaged in execution of research projects (in Number, % of total PDB / SS CGS subprojects); disaggregated by type (e.g. public universities, private universities, private enterprises, national research organization, producer organization) · Co-financing of research projects (CGS) by executing research Total or % of project costs PDB / SS institutions; disaggregated by type (e.g. public universities, private universities, private enterprises, national research organization, producer organization) · Awareness of competitive research fund and capacity to Number of application per call PDB apply successfully (disaggregated by type of research institute) and number or % accepted · Extent of inter-institutional, regional, and international Number of projects; PDB collaboration (CGS) % of budget · Identified research priority areas covered (CGS) Number of projects per PDB/CDB priority area; % of total budget; scientist time allocated · Disciplinary, product and regional balance of research projects Number of projects per priority PDB/CDB (CGS) area; % of total budget; scientist time allocated · Balance of strategic, applied, and adaptive research (CGS) Number of projects per category PDB/CDB · The degree to which the objectives and activities reflect % change PDB/SS stakeholder priorities · Graduates from education programs / projects (CGS) Number of MSc or BSc PDB · Competitive funding share of total national research funding Total; % of total budget PDB/CDB · Stakeholder participation in governance, priority setting, and Assessment of participation SS /PDB execution (e.g., stakeholder representation on boards) · Farmer representation on research management committees Number, % PDB/SS (by sex, ethnicity, farm size, wealth, age) · Research scientists by level of qualification (by sex; by priority area) Number and qualification CDB/SS (MSc and PhD) 84 3534-Rikka.pdf 6/20/05 2:51 PM Page 85 Research Projects--Institutional Development--continued Outcome Indicator Measurement Source · Research scientists by technical specialty--economics, social Number CDB/SS sciences, natural resource management, agronomy, business development (by sex; by priority area) · Recurrent costs or personnel costs relative to total research % of total budget CDB/PDB budget (CGS) · Peer reviewed publications, research reports, conference paper, Total number; number per PDB/SS patents (by priority area) year per researcher · Ratio of professional to support staff and trained technician Ratio PDB/SS (by sex) · Staff time devoted to research (CGS) % of time SS/PDB · Research-extension linkages (CGS) % of staff time devoted to SS/PDB extension activities; number of joint activities (workshops, field days, and field trials) · Research programs applying participatory technology Number or % PDB/SS development methodology/farmer participation in research activities · Annual research staff turnover % of all staff CDB/SS · Functioning M&E system with active participation of executing Number of impact studies; PDB/ER/ institutions number of progress reports; SS general assessment of system · Efficiency of fund administration unit Overhead costs (total and PDB % of total project costs); Average number of weeks or month from call for proposals and signing contract Research Projects--Technology Development Outcome Indicator Measurement Source · Adoption of new technologies and management recommendations Total number of adopters PDB/SS (disaggregated by participants and non-participants); including (including spillovers); % of studies of rationales for adoption / non-adoption by gender, targeted farmers and all farmers minority group, youth · Change in income, welfare, costs, yield, productivity, as a result % change from base SS of new technologies or recommendations (disaggregated by gender, minority groups, youth) · Change in farm/household labor needs (disaggregated by gender, % change from base SS minority groups, age) · Change in relative income from on-farm and off-farm activities % change from base SS (disaggregated by gender, minority groups, age as appropriate) continued 85 3534-Rikka.pdf 6/20/05 2:51 PM Page 86 Research Projects--Technology Development--continued Outcome Indicator Measurement Source · Client satisfaction with new technologies or management Rating (% of positive response) SS recommendations · Adoption of environmentally-sound technologies Area (ha); number of farmer; PDB/SS reduced use of agrochemicals (in liter or kg); reduced erosion (soil loss in kg per area) · Changes in use of natural resources (e.g., water, forest, land) Rate of use; area SS · Research programs applying participatory technology Number or % PDB/SS development methodology · Yield gap between farmers' yields and on-farm trials % difference SS · Yield gap between national yields and neighboring countries % difference CDB Source: Authors. Adapted from Alex and Byerlee (2000). 3. Project output indicators Extension Projects--Institutional Development Output Indicator Measurement Source · Regional/decentralized offices established Number PDB · Extension provider staffing (disaggregated by region; by sex) Number and qualifications PDB · Extension staff assigned in the field %; number of agents living in communities they serve · Extension service strategies and plan completed Number PDB (disaggregated by regions) · Extension agents training support (e.g., induction courses, and Number of staff trained; PDB in-service training) training days per year · Extension materials preparation unit established (by region) Number PDB · Extension agents trained on NRM issues Number of staff trained PDB · ICT support capacity (by region) Number of offices/agents PDB with computer (internet) access; number of computer/telephones per agent · Transport capacity (by region) Number of vehicles/bicycle PDB per agent · Extension management committees established Number · Competitive fund management unit established Assessed adequacy PDB/ER · M&E system (including management information system) established Assessed adequacy PDB/ER 86 3534-Rikka.pdf 6/20/05 2:51 PM Page 87 Extension Projects--Technology Dissemination Output Indicator Measurement Source · Farmer training courses conducted (per region) Number PDB · Farmer trained, contacted, visited (by sex, per region) Number PDB · Frequency of face-to-face meeting with farmer (by region, by sex) Average number per month; PDB/SS average time spent · Demonstration established (on-station and on-farm per region) Number PDB · Farmer contests conducted Number PDB · Number of producers organizations established Number PDB · Farmer group meetings held (per region) Number; number of participants PDB · Information material produced and disseminated (per region) Number of brochures and PDB pamphlets · Extension service projects (CGS) Number PDB · Environmental farm program trainings conducted Number PDB · Farmer-led trainings conducted Number PDB Research Projects--Institutional Development Output Indicator Measurement Source · Research projects (CGS) Number PDB · CGS promotion workshops/seminars Number PDB · Graduate courses established (PhD, MSc, BSc, short courses) Number PDB · Curricula revised or developed Number PDB · Costs of graduate programs Cost per graduate PDB · Proposal review committee established (CGS) Assessed adequacy PDB/ER · Competitive fund management unit established Assessed adequacy PDB/ER · Research priority setting mechanisms/committee implemented Assessed adequacy PDB/ER · Training support for researcher in priority areas Number of workshops, seminars PDB · ICT support capacity (e.g., computer/internet access; access to Number of researcher with PDB electronic journals) computer/internet access/ electronic journals; number of computers per researcher · Journals/publications acquired Number PDB · Laboratories equipped Assessed number and qualityPDB · Research-extension linkages established Number of full-time SMS; PDB number of joint seminars/ workshops continued 87 3534-Rikka.pdf 6/20/05 2:51 PM Page 88 Research projects--Institutional development--continued Output Indicator Measurement Source · Regulatory and legal frameworks for research and technology Assessed adequacy PDB/ER importation established · Researchers trained in participatory technology development Number PDB · Natural resource management researchers hired Number PDB Research Projects--Technology Development Output Indicator Measurement Source · New varieties developed Number PDB · Extension recommendations published Number PDB · Other research products completed; research tools / Number PDB methods developed · On-farm research projects (CGS) Number PDB · Papers published Number PDB · Environmentally sound technologies developed Number PDB Source: Authors. Adapted from Alex and Byerlee (2000). 4. Input indicators--All ARE projects Input Indicator Measurement Source · Civil works completed Number and value PDB · Goods (e.g., equipment and vehicles) procured Value PDB · Technical assistance provided Person-weeks and value PDB · Unit costs of inputs Cost per unit PDB Source: Authors. Adapted from Alex and Byerlee (2000). Abbreviations: GDB = Global Database; CDB = Country Database; PDB = Project Database; SS = Special Studies; and ER = Expert Review. 88 3534-Rikka.pdf 6/20/05 2:51 PM Page 89 C. SOURCES FOR BASELINE DATA AND NDICATORS I Agricultural Innovation Systems project M&E systems must balance data needs between what is ideal and also costly, and what is practical, and will generally need to draw on data from various sources. These sources will be described briefly in the following paragraphs.32 Management information system Project input-output data are generally available from a project management information system (MIS), the project's internal system for collection, analysis, and dissemination of project information. National and international level databases As it is generally not cost-effective for a project to measure national change in goal-level indicators, information needs at this level rely on national and international databases. These databases may pro- vide data for indicators such as rural population below poverty line, daily per capita dietary energy supply, agricultural GDP, and public expenditures on agricultural research as percentage of the agri- cultural GDP. Quality of national data may be suspect to poor sampling procedures or intentional dis- tortions, but it is readily available and comprehensive, represents considerable investment in data collection, and is official, government-sanctioned data. Changes in national-level indicators are usu- ally gradual and not observable over a typical five-year Bank-financed project. It may be advisable to dampen the fluctuations because agricultural statistics are subject to wide year-to-year fluctuations due to weather and commodity price changes, tracking three-year averages and statistical trends rather than annual data. For example, relevant data may be collected by the Ministry of Finance, Ministry of Agriculture, or Ministry of Science. These data can be supplemented by international databases.33 Special studies Bridging the gap between indicators from routine reporting on project activities and from national data sets is a challenge for ARE projects, and reflects the difficulty inherent in linking project per- formance and eventual impacts. Special studies are a practical tool for collecting information on agricultural systems, program impacts, and especially for assessing and measuring outcomes. In gen- eral, special studies should provide for disaggregating by gender, ethnicity, or other key characteris- tics of client populations. In addition to surveys or case studies to assess outcomes, special studies might be needed to track specific planned and unplanned impacts or program operational issues (e.g., technology access by women or minority groups, and changes in land tenure and other social conditions in the area). Beneficiary and stakeholder participation in M&E is critical to ensure that R&E projects are responding to genuine needs of intended clients. This can be accomplished by using survey tech- niques or participatory evaluation (see chapter 4). Special studies can usually be built into the regular work plan of ARE institutions, as extension agen- cies must track changes in farmer attitudes, knowledge, and technology adoption; research programs need to know results of new technologies; and training institutions need to know how graduates per- form. In some cases, it may be best for monitoring to be done by an independent institution to avoid 89 3534-Rikka.pdf 6/20/05 2:51 PM Page 90 overburdening the implementation institution, or to avoid bias in monitoring change within the insti- tution. The alternative is to finance special studies through a project office independent of the imple- mentation agency. No matter how special studies are carried out, there is a danger of them being neglected during project start-up, as studies are of relatively high cost, management intensive, and easily put off until data are needed for a Mid-Term Review or other critical project evaluation. The major types of special studies used in ARE programs include: a) Sample surveys. Formal sample surveys are useful to provide comprehensive and statistically valid data on program outcomes and impacts. However, formal sample surveys also have disadvantages in that many institutions lack capacity to conduct them, wide fluctuations in agricultural production make it difficult to establish statistically significant trends in production data over short time peri- ods, and survey data collection costs are high. Large-scale national surveys are probably best avoid- ed, unless they are linked to a regular program of national agricultural statistics. More focused regional or localized surveys may be more manageable, and are useful in characterizing agricultural conditions and changes in a particular area.34 Grosh and Glewwe (2000) provide detailed advice on how to design multiple-topic household surveys, that include defining the objectives of the survey, identifying the data needed, and drafting questionnaires.35 b) Rapid rural appraisal. Rapid rural appraisals are a compromise between formal surveys (which generate statistically valid, quantitative data but are costly and time consuming) and informal field visits (which provide anecdotal information) (Kumar 1993). They are low-cost, quick, open-ended, and flexible, but are not statistically valid and may be open to charges of bias. Five core methods of rapid rural appraisal approaches are: key informant interviews, focus group interviews, community interviews, structured direct observation, and informal surveys. c) Special topic studies. Special topic studies are often useful to analyze technology program impacts and develop an analytical base for routine M&E data collection. Ex ante economic analysis and eco- nomic models of the impact of technological innovation are common types of special topic studies. Others might focus on specific impacts on women, land tenure, food prices, living standards, or nutrition. d) Diagnostic case studies. Diagnostic case studies may provide a holistic, in-depth descriptive assessment of a program or local situation. These may combine information from various sources and can be useful to: (a) analyze potential social, technical, and financial factors affecting uptake of new agricultural technologies; (b) assess potential and economic benefits from technological inno- vation; (c) assess differential impacts of programs on different groups; and (d) identify key issues for effective extension program operations. Agricultural Innovation Projects should frequently build rou- tine field case studies into M&E systems. In fact, it may be difficult to envision an efficient technol- ogy system without a functional system for such studies to assess how well a project is doing and how well technologies are being accepted. A key challenge is that of relating "with-project" changes to the counterfactual "without-project" scenario. e) External peer reviews. Peer reviews are used in highly technical program areas, most commonly in research. External peer reviews are useful to obtain technical experts' assessment of research activities from three perspectives: (a) a prospective review (to assess implications and relevance of research); (b) an in-process review (to assess efficiency of research and performance against targets for outputs), and (c) a retrospective review (to assess scientific or technical quality of work complet- 90 3534-Rikka.pdf 6/20/05 2:51 PM Page 91 ed). They are also useful in developing linkages between scientists and institutions working in relat- ed fields. As an example, an external review was conducted for the Agricultural Research and Extension Project (Adaptable Program Loan) in Peru to evaluate the progress of the first project phase, with the specific task of verifying if triggers for the specific phase had been reached (the trig- gers or measures of project implementation progress must be met prior to any phase will be passed for approval and execution of the following phase can begin). 91 3534-Rikka.pdf 6/20/05 2:51 PM Page 92 APPENDIX 5. PROJECT AND MONITORING & EVALUATION COSTS Project Costs: Bank project costs are usually prepared using Costab 32 a PC-based project costing tool. Costab 32 is the project costing component of a software series to improve efficiency and effec- tiveness of Project Processing activities. Costab 32, upgraded in 2001 by ADB, and is designed to operate under Windows 2000 as well as earlier 32-bit versions of Windows. Costab 32 also incorpo- rates several new features which include: · Automatic calculation of financial charges during implementation; · Summary table format for both ADB and World Bank; · Option to compute economic costs in either domestic economic prices or border prices; and · Other operational parameters. Costab 32 helps task team leaders, project economists, and financial analysts organize and analyze project costing data in the course of project preparation and appraisal calculates physical and price contingencies, taxes and foreign exchange displays data in detailed cost tables coverts financial costs to economic costs for use in economic analysis. Costs should be broken down by component and by type of input. They should be expressed in local currency and foreign exchange, specifying the source of financing and, where applicable, co-financing arrangements. See table A5.1 for an example. It is important to ensure that all activities shown in a detail table are incorporated in the preparation of M&E tables such as a standard Log Frame or the results framework arrangements matrix. Table A5.1. A Project Component Table from Costab 32. Unit Cost 05/06 06/07 07/08 08/09 09/10 Total I. Investment Costs A. Review of national acts, policies and regulations Review of existing Land mgt. guidelines 329,787.234 134,553.2 104,951.5 109,149.5 ­ ­ 348,654.2 Awareness raising and IEC 329,787.234 134,553.2 104,951.5 109,149.5 ­ ­ 348,654.2 Field surveys and validation 329,787.234 168,191.5 174,919.1 ­ ­ ­ 343,110.6 SLM policy regulatory capacity building 329,787.234 168,191.5 174,919.1 ­ ­ ­ 343,110.6 Subtotal review of national acts, policies and regulations 605,489.4 559,741.3 218,299.1 ­ ­ 1,383,529.7 B. Ssatellite imagery of 3 Dzongkhags Satellite image of Radhi 1,000,000 ­ 1,060,800.0 ­ ­ ­ 1,060,800.0 Satellite image of Nangkhor 1,000,000 510,000.0 530,400.0 ­ ­ ­ 1,040,400.0 Satellite image of Phuentsholing 1,000,000 510,000.0 530,400.0 ­ ­ ­ 1,040,400.0 Subtotal satellite imagery of 3 Dzongkhags 1,020,000.0 2,121,600.0 ­ ­ ­ 3,141,600.0 1,625,489.4 2,681,341.3 218,299.1 ­ ­ 4,525,129.7 Source: Ghazali Raheem, Consultant, World Bank. 92 3534-Rikka.pdf 6/20/05 2:51 PM Page 93 Below is a typical M&E budget plan for an IDA/GEF environmental project, including a breakdown of key expense categories. Marine and Coastal Environment Management Program United Republic of Tanzania Indicative M&E Budget Unit Price PY1 PY2 PY3 PY4 PY5 Core Staffing and Administration Quantity Cost Quantity Cost Quantity Cost Quantity Cost Quantity Cost Staffing M&E Coordinator PCU 1 $ 1,172 1 $ 1,172 1 $ 1,172 1 $ 1,172 1 $ 1,172 MIS Analyst PCU 1 $ 956 1 $ 956 1 $ 956 1 $ 956 1 $ 956 Statistician/Analyst PMU 1 $ 109 1 $ 109 1 $ 109 1 $ 109 1 $ 109 Office equipment Computers $1,500 3 $ 4,500 0 $ -- 0 $ -- 0 $ -- 0 $ -- Printers $ 250 3 $ 50 0 $ -- 0 $ -- 0 $ -- 0 $ -- UPS $ 100 3 $ 300 0 $ -- 0 $ -- 0 $ -- 0 $ -- Furniture $1,000 2 $ 2,000 0 $ -- 0 $ -- 0 $ -- 0 $ -- MIS Applications Applications $8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 Training Intro to Computing Computing applications Statistical analysis Report preparation Data base and MIS Network principles Beneficiary training $ 3,4001 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 Field visits $1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 Workshops $3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 Assessments, Studies, and Surveys Baseline surveys $4,800 1 $ 4,800 0 $ -- 0 $ -- 0 $ -- 0 $ -- Annual M&E Review and Reports $3,600 1 $ 3,600 0 $ -- 0 $ -- 0 $ -- 0 $ -- M&E workshop $2,200 1 $ 2,200 0 $ 2,200 0 $ 2,200 0 $ 2,200 0 $ 2,200 Safeguards report $4,500 1 $4,500 0 $ -- 0 $ -- 0 $ -- 0 $ -- Social and Environ Impact Assessments $5,500 1 $ 5,500 0 $ -- 0 $ -- 0 $ -- 0 $ continued 93 3534-Rikka.pdf 6/20/05 2:51 PM Page 94 Marine and Coastal Environment Management Program--continued Unit Price PY1 PY2 PY3 PY4 PY5 Core Staffing and Administration Quantity Cost Quantity Cost Quantity Cost Quantity Cost Quantity Cost Mid-term review $1,200 1 $ 1,200 0 $ -- 0 $ -- 0 $ -- 0 $ -- Annual audit $3,200 1 $ 3,200 0 $ -- 0 $ -- 0 $ -- 0 $ -- Operations and Maintenance Supplies and Maintenance $3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 Installation $4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 Total $58,729 $28,379 $28,379 $28,379 $28,379 Source: Ghazali Raheem, Consultant, World Bank. 94 3534-Rikka.pdf 6/20/05 2:51 PM Page 95 APPENDIX 6. ECONOMIC EVALUATION OF COMPETITIVE RESEARCH GRANTS: THE CASE OF PROMSA, ECUADOR OBJECTIVES, EXPECTED RESULTS, AND MAIN CHARACTERISTICS OF PROMSA From the time leading up to the implementation of a CGS in 1999 Ecuador's national research sys- tem was dominated by a centralized public research institute, Instituto Nacional Autonomo de Investigaciones Agropecuaria (INIAP). Because public agricultural research funds were dedicated almost exclusively to supporting INIAP, financing was scarce for other potential suppliers of agri- cultural research such as private and public universities, NGOs, and private firms. The number and level of training of agricultural scientists in Ecuador was very low as well. In 1998 the total number of agricultural scientists holding a Ph.D. degree was just 16, and only 71 held an M.Sc. INIAP link- ages with domestic and international research entities were weak, limiting expertise and technology spill-ins. In 1998, INIAP had only 6 national, and 7 international research agreements with total funding of less than $200,000. The original objective of World Bank-support for agricultural research in Ecuador was to strengthen the national capacity to do agricultural research, with the aim of increasing agricultural productivi- ty and output by (a) introducing a competitive research grant program; (b) developing research and education exchange partnerships with international science institutions; and (c) strengthening the national agricultural research institute, INIAP. Ecuador's Competitive Grants System provides grants for individual research projects, and for research and educational exchange partnerships with international institutions. The former grant type was implemented as a mechanism for (a) increasing the efficiency of fund allocation for agricultur- al research; (b) assuring the participation of scientists in agricultural research from diverse institu- tions; and (c) strengthening linkages between public and private sector entities. The latter grant type was implemented to finance the formation of research and education partnerships between research agencies, universities, key NGOs, and international scientific institutions to facilitate the importa- tion of applicable technology, strengthen the research capacity of Ecuadorian research institutions, and strengthen graduate education programs in Ecuadorian universities. By 2004 the National Modernization Program for Agricultural Services (PROMSA) had financed 108 individual research projects carried out by national research entities, and 14 international alliances for research and educational programs. All programs follow the same competitive granting procedure. Individual research projects supported by the fund were not to last more than three years or cost more than $100,000, and were to have co-financing from other sources of at least 25%. The maximum size of international alliances was set at $500,000. A Project Implementation Unit (PIU) was created at project start-up and an international private firm was contracted to set up a monitor- ing and evaluation system (M&E). In addition to monitoring the progress of funded projects, the PIU is responsible for preparing annual budgets and work plans, making payments from the loan account for project expenditures, and preparing financial and progress reports. 95 3534-Rikka.pdf 6/20/05 2:51 PM Page 96 Monitoring a competitive grant system The management system that has been implemented in Ecuador is sophisticated and has contributed substantially to the success of the competitive fund mechanism in Ecuador. The PIU prepares an annual operating plan and quarterly reports on implementation of the proposed work plan. PIU field officers visit each project at least twice each year to observe first-hand processes being employed in administering the competitive fund. The computerized Monitoring and Evaluation system is state of the art. The project database provides ready access to data on implementation of projects, and over- all performance of the project portfolio. The database is relatively simple to maintain and use to gen- erate information for reports. Good practice: Adequate resources must be assigned to M&E activities. Resources should be ade- quate to staff an M&E unit, establish a database, and provide consistent and timely feedback to subproject scientists. A system of objectives and milestones (for inputs, activities, outputs) with projected dates for mon- itoring project implementation has been established and forms the basic elements for a log frame approach. The M&E system is relatively simple to understand and manage. It is primarily based on quarterly progress report by the project researchers. Data on all projects and milestones are record- ed in the M&E database. PIU prepares an annual evaluation of subprojects based on (a) annual reports by researchers, and (b) formal and informal reports by grupos de referencia (GR) (locally run individual project advisory boards) for each project, (c) bi-annual visits by PIU specialists to each project. Good practice: Formal project milestones must be established and monitored. A GR has been formed to oversee the implementation of each subproject. GRs are an innovative approach to decentralized M&E in which beneficiaries and researchers collaborate. Each GR is com- prised of 4­8 stakeholders (other researchers, extension staff, and agribusiness). The exact group com- position and function are flexible, but ideally the GR is expected to participate in project design, planning, and implementation. The groups meet several times a year to consult on project issues. The PIU has also solicited feedback from project leaders through a survey of their opinions about the CGS process. Funded researchers report that the PIU M&E reporting system is a substantial improvement over the lengthy reporting often required by their own institutions or other funding agencies. Good practice:Advisory groups should be organized. These groups can provide valuable feedback into the M&E process. Good practice: Reporting requirements should be designed to collect a minimum data set in an efficient manner. The needs for M&E information must be balanced with a respect for the burden that it places on busy scientists. The PROMSA M&E system employs a system of "alert" signals or flags that indicate if a project has become a "problem project." There are four ways a project can be flagged as a problem: (a) quarter- ly or annual reports that are overdue by more than a month, (b) completion of less than 80 percent of the quarterly milestones, (c) a rating by PIU field personnel of less than 2.5 on a 4-point scale, or (d) general problems in quality of implementation. While in alert status, a project is not eligible for new disbursements from the Fund. If the project does not clear its alert status by correcting deficien- cies or rescheduling milestones within a specified time frame, the project can be terminated. 96 3534-Rikka.pdf 6/20/05 2:51 PM Page 97 Good practice: An alert system should be implemented to enforce communication between researchers and CGS administrators. The criteria for triggering an alert should be clearly under- stood and uniformly applied. Evaluation of a competitive grant system Most funded projects are not expected to have results diffuse to end users until five or more years after the start of research. This makes it impossible to conduct a realistic economic evaluation of the CGS. Therefore, evaluation must focus on success in meeting overall CGS objectives, in creating necessary research institutional infrastructure, in measuring progress in implementing the proposed subproject research, and in generating research outputs such as publications and technologies. This evaluation relies on indicators that can be extracted from the PIU database (see the list below). There are a large number of potential indicators of CGS success, some of which are more important than others. Not all of those listed in the appendix are necessary for each evaluation. Indicators should allow evalua- tion of the degree of success in institutional, administrative, and scientific aspects of the CGS. Good practice: When the CGS funds research projects of longer gestation economic analysis will not be meaningful until the project has been active for several years. During this period indicators that allow evaluation of institutional, administrative and scientific progress should be monitored. List of potential M&D indicators 1. Research Project Indicators (both at project type and portfolio level) · Number of research project applications · Number of research projects approved · Number of research projects established · Number of research projects by priority area · Number of projects involving basic, strategic, applied, adaptive research · Number of projects financed in small-scale agriculture · Number of technical and scientific publications (differentiate according to quality such as peer-reviewed articles, pamphlets, or manuals for farmers) · Number of workshops · Number of international alliances established · Labor-capital ratio (e.g., person months--value of equipment) · Recurrent costs or personnel costs relative to research budgets · Cost per researcher · Number of technologies transferred to farmers fields · Rates of technology adoption by farmers 2. Education Project Indicators (both at project type and portfolio level) · Number of short-term course, MSc. Program, Ph.D. program application · Number of short-term courses, MSc. Programs, Ph.D. programs approved 97 3534-Rikka.pdf 6/20/05 2:51 PM Page 98 · Number of short-term courses, MSc. Programs, Ph.D. programs established · Number of short-term courses, MSc. Programs, Ph.D. programs by priority area · Number of MSc. degrees, Ph.D. degrees awarded · Number of fellowships awarded · Number of MSc. degrees, Ph.D. degrees awarded by international institutions · Number of MSc. degrees, Ph.D. degrees awarded by national institutions · Number of international alliances established · Length of time required to complete degree programs or obtain certificates · Cost of training programs per trainee · Training program costs recovered from payments by users · Employment rate of graduates 3. Overall Institutional Indicators (both at project type and portfolio level) · Total number of project applications, projects approved, projects established · Number of grant applications by INIAP, universities, private sector, others, joint projects (define type of partnership) · Number of grants awarded by INIAP, universities, private sector, others, joint projects (define type of partnership) · Number of projects established by INIAP, universities, private sector, others, joint projects (define type of partnership) · Amount of resources (%) captured from INIAP, universities, private sector, others, joint projects (define distribution of partnerships) · Ratio of project/counterpart funding · Amount of resources from government of Ecuador · Ratio of professional to support staff and trained technicians · Resource allocation in relation to established priorities · Competitive funding as share of total research funding 98 3534-Rikka.pdf 6/20/05 2:51 PM Page 99 NOTES Chapter 1. Challenges for M&E Systems in Agricultural Research and Extension Projects 1. The Note is a follow-up to earlier Bank publications in M&E (Monitoring and Evaluation for AKIS Projects by Alex and Byerlee, 2000, Ex-Ante Economic Analysis in AKIS Projects by Horstkotte-Wesseler et al., 2000, Beneficiary Assessment for Agricultural Extension by Salmen, 2000, and Ten Steps to Results-based Monitoring and Evaluation by Kusek and Rist, 2004) by accounting for recent changes in the design of World Bank-supported research and extension (R&E) projects and the trend toward results-based M&E. 2. The driving forces behind change in ARE systems stem from deficiencies experienced in tradi- tional systems: poor fiscal sustainability, difficulties with attribution of impact, low political com- mitment, insufficient local content, weak accountability/ transparency, and weak linkages between research and extension. Anderson, J. and G. Feder, 2004. Agricultural extension: good intentions and hard realities. The World Bank Research Observer, vol. 19, no 1. and Rajalahti, R., J. Woelcke, and E. Pehu, 2005. "Development of Research Systems to Support the Changing Agricultural Sector." Proceedings of a Research Workshop, World Bank, Washington, DC. Chapter 2. The World Bank Results Framework Requirements 3. World Bank, 1997. "Proxy indicators." The Log Frame Handbook: A logical framework approach to project cycle management. World Bank, Washington, DC. 4. A World Bank Toolkit: Integrating Gender Dimension into Monitoring & Evaluation of Rural Development Projects; and a Thematic Brief: Agricultural Knowledge and Information Systems; Agricultural Research, Extension, and Education, 2001 provide further examples on gender- disaggregated indicators. 5. Alternatively, the task team could present the component outcomes as combined outcomes at the PDO level. This was common practice in the past. The advantage may be that it is in some cases eas- ier just to define the combined effects of the components at the PDO level. Chapter 3. Data Collection, Reporting and Dissemination Requirements 6. Baseline and targets based on weighted average of farmer income in the different sub-projects from information provided in the proposals. 7. Association deemed successful if over 60% of the farmers are satisfied with the services provid- ed by the Association. 8. Guidelines for Management Information Systems in Social Funds. 9. Relevant Bank publications in this context include: Murphy and Marchant (1988) discussion of issues in planning M&E systems for extension agencies; Casely and Kumar (1987) discussion of issues for agricultural sector programs in general; World Bank (2003b)--the most recent Bank pub- lication on M&E. in agricultural operations; and Kusek and Rist (2004)--the most recent discussion of M&E system for development programs in general--with particular focus on results-based M&E systems. 99 3534-Rikka.pdf 6/20/05 2:51 PM Page 100 10. The thematic areas may include variety development; soil and water conservation; integrated pest management; agro-forestry; marketing; and livestock production in different agro-ecological zones. 11. Poor sampling or inadequate mix of information may introduce significant bias which might result in a disconnect between the objectives and true outcomes of the project. 12. Examples include the one by the World Bank (http://www.worldbank.org/data), FAO (http://www.fao.org/waicent/portal/statistics_en.asp) and IFPRI ASTI at http://www.asti.cgiar.org/. 13. This section written by Ghazali Raheem, Consultant, World Bank. Chapter 6. Economic Evaluation of Competitive Grants Schemes 14. It is usual in the empirical literature on project evaluation to classify projects' impacts, either potential or realized, in three main categories: economic, institutional, and environmental. This chap- ter deals with impacts within the first category--those that can be measured in monetary terms. 15. Details of practical procedures of incremental cost benefits analysis can be found in: CIMMYT (1988). From Agronomic Data to Farmer Recommendations: An Economics Training Manual. Revised Edition. CIMMYT: Mexico. 16. CIMMYT (1998) provides an excellent discussion about how to estimate the incremental net benefit between technological alternatives for annual crops. Extension to more than one period for perennial crops or technologies with longer beneficial effects is straightforward. 17. A partial budget compares estimated revenues and costs for a typical producer with and without the new technology. 18. It is recommended to transfer this simple table into EXCEL which would allow for a convenient calculation of NPV and IRR. 19. Methods for estimating the diffusion and the estimation of incremental net benefits have been explained in section 6.4. 20. For more details see section 6.1: economic impact evaluation of a CGS; and section 6.5: estima- tion of IRR. 21. Expert Opinion is often used in conjunction with simulation analysis. In this case the experts are asked for their estimation of three values of the parameters of interest: its most likely value, the max- imum, and the minimum. 22. The specific assignment of factors to the different categories of investment costs clearly depends on the nature of the project and no general recommendation can be made. However, as an example, the following factors have been assigned to the different categories of investment costs for the Modernizing Agricultural Knowledge and Information Systems Project (Romania): civil works- factor: 0.1; equipment-factor: 1; vehicles-factor: 1; technical assistance-factor: 0; training-factor: 0; competitive grant: 1. 23. A commonly used indicator for the level of public financial support of agricultural research is public expenditure for agricultural research as percentage of the agricultural GDP (AgGDP). Figures 100 3534-Rikka.pdf 6/20/05 2:51 PM Page 101 for public research expenditures can be obtained from the Ministry of Agriculture. Another recom- mendable source is the Agricultural Science and Technology Indicators (ASTI) provided by the CGIAR (http://www.asti.cgiar.org). Appendix 1. Narratives of New Approaches and Instruments in ARE Systems 24. Anderson, J. and Feder, G. 2004. "Agricultural extension: Good intentions and hard realities." The World Bank Research Observer, vol. 19, no 1. and Rajalahti, R., J. Woelcke, and E. Pehu 2005. Development of Research Systems to Support the Changing Agricultural Sector. Proceedings of a Research Workshop, Washington, DC: World Bank. 25. Farmer-to-farmer methodology is one type of participatory research and extension methodolo- gy. Here, it includes Farmer Field Schools, CIAL, use of volunteer contact farmers, and producer organizations. 26. Mass media and new ICT applications can be integrated into the strategic extension programs. Media applications include both "new and old" communication devices and methods, such as posters, booklets, radio (and enter-edutainment), television, drama, internet, and mobile phones. 27. World Bank (2003a). "Extension and rural development--Converging views on institutional approaches?" Workshop Summary. Agriculture and Rural Development Department, World Bank, Washington DC. 28. World Bank (2004b). Investments in Agricultural Science and Technology. Agriculture Investment Sourcebook. Module 2. World Bank, Washington, DC. 29. Veldhuizen, L., A. Waters-Bayer, and H. Zeeuw 1997. Developing technology with farmers. A trainers guide for participatory learning. London: Zed Books Ltd. 30. Braun, A., G. Thiele, and M. Fernandez 2000. "Farmer Field Schools and Local Agricultural Research Committees: Complementary platforms for integrated decision-making in sustainable agri- culture." AgREN Network Paper No. 105. Appendix 2. Implementation of a Monitoring and Evaluation Plan in the Bank's Project Cycle: Putting It All Together 31. This chapter written by Remileku Rakey, Consultant, World Bank. Appendix 4. List of Indicators and Data Sources for ARE Projects 32. The following section is taken mainly from Alex and Byerlee (2000). 33. Examples include (e.g., the one by the World Bank http://www.worldbank.org/data) and FAO (http://www.fao.org/waicent/portal/statistics_en.asp). 34. Important data sources include the CGIAR centers which regularly conduct relevant surveys in developing countries (e.g., IFPRI in Uganda). 101 3534-Rikka.pdf 6/20/05 2:51 PM Page 102 35. Grosh, M. and P. Glewwe (2000). Designing household survey questionnaires for developing countries: lessons from 15 years of the Living Standards Measurement Study Volumes 1­3. World Bank, Washington, DC. On agriculture, volume 2 lays out the most important agricultural policy issues in developing countries, discusses the date needs and measurement concerns, and introduces three versions of a draft agriculture module. Volume 3 presents draft questionnaire modules on sev- eral topics. On agriculture, the issues covered include land holdings, farm capital inventory, disposi- tion of crops, input purchases and sources, access and use of agricultural extension services, farm capital inventory and transactions, crop output, household member labor inputs, hired labor and non- labor inputs, input prices, expenditures and sources, and livestock. 102