58040 v2 World Bank Japan Social Development Fund Strategic Evaluation of the Japan Social Development Fund (JSDF) Final Report APPENDICES 17th March 2007 Prepared by JSDF Draft Final Report Appendices, December 2006 Appendices A. Terms of Reference ........................................................................................2 B. List of Grants in JSDF Portfolio..................................................................5 C. Methodology.................................................................................................12 C.1. Evaluation Framework .......................................................................................12 C.2. Document Review ...............................................................................................18 C.3. Interview Topic Guide ........................................................................................23 C.4. Field Visit ..............................................................................................................30 C.5. Web Based Survey ...............................................................................................59 D. Results ............................................................................................................60 D.1. Document Review ...............................................................................................60 D.2. Field Visit ..............................................................................................................72 D.2.1. PEP Scores....................................................................................................72 D.2.2. Assessment Frameworks (for each site-visit project).............................73 D.3. Web Based Survey .............................................................................................102 1 JSDF Draft Final Report Appendices, December 2006 A. Terms of Reference STRATEGIC EVALUATION OF THE JSDF Objectives The objectives of the strategic evaluation are: · To review the progress and development impact to date of the JSDF; · To assess the achievements of the JSDF as a whole, and of selected grants, in the light of the Fund's objectives; · To make recommendations on ways in which the Fund might better achieve its objectives in the future. Approach In performing the evaluation, Consultants will focus on the "regular" JSDF grants and on the seed fund grants which are intended to facilitate the process of preparing "regular" grants. The special programs for Afghanistan and the post-tsunami recovery grants will thus be excluded. The approach adopted by the Consultants will respond to the following questions: · To what extent are JSDF-supported activities achieving the Fund's objectives? · Are there ways in which the Fund's program management, procedures or policies might be changed to enable its objectives to be achieved more cost- effectively? · What other funding sources are used for similar activities and to what extent is the JSDF supporting development outcomes that are not being achieved by other initiatives? · Is the seed fund contributing to effective participative project preparation? How can it do this even better? Is project preparation more participative in the presence of a seed fund grant? · Have JSDF projects been effective in working with both local and international (including Japanese) NGOs? · How well are JSDF grants managed within the Bank, and what are or should be the roles of country teams and networks? · Within the different kinds of activities that JSDF has supported, are some more effective in meeting JSDF objectives than others? If so, what are the factors that are contributing to this? · What has been the value-added of JSDF projects to World Bank operations? Can the complementarities between JSDF projects and World Bank projects be improved? · Are there specific lessons learned by the grant activities, in terms of technical design, the most appropriate kinds of project, reaching target beneficiaries, and governance or management arrangements, that JSDF has funded which should be more widely disseminated? 2 JSDF Draft Final Report Appendices, December 2006 Scope of Services and Enquiry Sources The Consultants will secure all information needed to support their findings, include, but not limited to, the following: · A desk review of the background materials on the JSDF, including Agreements between the Government of Japan and the World Bank, Annual JSDF Reports, JSDF operating procedures, details of approved grants, and available progress reports and Implementation Completion Memoranda (ICMs) for completed grants; · Interviews with key JSDF stakeholders including JSDF Steering Committee members, JSDF Unit staff, Task managers, Trust Fund Coordinators, Japanese NGOs and staff of the Japanese Ministries of Finance and Foreign Affairs; · Assessment of stakeholders' and country team members' comments on JSDF activities (not necessarily though field visits); · Site visits to selected JSDF project locations, to carry out impact evaluations. These site visits should aim to identify the development impact of the JSDF interventions, and the nature and scale of the impact on the intended beneficiaries. · A final presentation of evaluation conclusions to the JSDF Steering Committee and to representatives of the donor, which will include a Workshop Presentation to be organized by the Bank. Time Schedule The evaluation will require twenty-eight (28) weeks to complete, including the submission of a final report, which period will encompass two key Tasks: Task I Desk review and Interviews in Washington This phase of the study will involve preparing an overview of the JSDF program and a preliminary assessment of its achievements, based on written documentation and interviews with World Bank and Japanese stakeholders. During this phase, the Consultants will also review the overall JSDF portfolio, and categorize the different projects financed, as a basis for selection of a representative sample of completed or substantially complete projects for detailed site review in Task II. The Consultants will propose the projects to review to the World Bank at the end of Task I. To the extent possible projects should be sampled on a stratified random basis ­ which might best be thematic as well as possibly based on location. Task II Site reviews of sampled projects, to determine impact and likelihood of achieving their objectives Task II shall only commence when the Bank Task Manager provides the Consultants written acceptance authorization to proceed with Task II. The written authorization will include the approved list of projects to be reviewed. It is expected that Task II would commence within two months of completion of Task I (to allow for the logistics in mobilizing to Task II locations, including alerting Bank offices and concerned stakeholders). The Consultants would be expected to meet with a variety of local stakeholders and to undertake site visits and field 3 JSDF Draft Final Report Appendices, December 2006 surveys in order to obtain rigorous data to evaluate progress in achieving project objectives and to assess the expected impact on the intended beneficiaries. After completion of the fieldwork the Consultants would report to the World Bank on the results and conclusions of their overall study. Reports The evaluation team will submit: · A brief initial inception report within one week of starting work; · An interim report towards the end of Task I, including proposals for the projects to be studied in detail during Task II ; · A draft final report for discussion with the Steering Committee and other stakeholders, at the conclusion of Task II. The draft final report will include an analysis of JSDF activities to date, including identification of the main disciplinary areas which projects have addressed; · Following draft final report submission, ITAD will make a presentation of its findings and recommendations in a workshop in Washington. The workshop will serve as forum for eliciting comments from attending stakeholders; and · A final report reflecting comments made on the draft final report. These reports will be in "full document" form (not "PowerPoint presentations"). They will be provided in electronic format and in ten hard copies. Services and Facilities to be Provided by the Bank The Trust Fund Operations (TFO) Department in the World Bank will have management responsibility for oversight and supervision of the evaluation, and has appointed a Task Manager, who will handle day-to-day implementation of the assignment on behalf of the Bank, as well as serve as lead representative in the Bank's relationship with the Consultants. A Review Group will be made available to guide and support the consultant team, and will meet with the team on several occasions, virtually or physically as appropriate. The Review Group will include representatives of TFO, the Bank's Social Development Department (SDV) and the JSDF Steering Committee. In addition, a representative of the Office of the Executive Director for Japan has been invited to join the Review Group. The Task Manager will provide the "secretariat function" for this Review Group. The task manager and the JSDF Unit in TFO will provide access to all relevant materials to the Consultants. TFO will provide office space to the consultant team during Task I and as needed. While the Review Group will provide specific comments in a consolidated manner on the Inception, Interim and Draft Final Reports produced by the Consultants, the exact dates of any meetings requested by the Consultants cannot be agreed at the outset of the work due to the difficulty in coordinating with Group members. 4 JSDF Draft Final Report Appendices, December 2006 B. List of Grants in JSDF Portfolio (Grants in the evaluation sample shaded) Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp CHILD DEVELOPMENT IN TF024743 DISADVANTAGED LCLS Bulgaria ECA 50,000 1,050,000 1,100,000 98% COMMUNITIES PILOT PROJECT FOR POVERTY REDUCTION IN TF026559 SIX COMMUNES IN LCLS Vietnam EAP 1,395,000 1,395,000 88% NORTHERN MOUTAIN PROVINCES OF VIETNAM IMPROVING ACCESS TO TF026562 HEALTH CARE BY THE LCLS China EAP 400,000 100% 400,000 RURAL POOR SQUATTER SETTLEMENTS TF026563 PILOT ASSISTANCE LCLS Indonesia EAP 114,400 2,190,400 2,304,800 94% PROJECT TF026564 WIDOWS AND POVERTY LCLS Indonesia EAP 85,040 2,112,000 2,197,040 71% REINTEGRATION OF TF026566 STREET CHILDREN IN CIVIL LCLS Mongolia EAP 443,000 90% 443,000 SOCIETY DEVELOPMENT OUT OF TF026567 SCHOOL (OSY) CHILDREN LCLS Philippines EAP 1,000,000 1,000,000 95% AND YOUTH PROJECT SPECIAL ZONE ON PEACE TF026568 AND DEVELOPMENT LCLS Philippines EAP 1,472,883 1,472,883 100% (SZOPAD) SOCIAL FUND EMPOWERING RURAL TF026592 LCLS Pakistan SAR 21,154 515,769 536,923 85% WOMEN IN PUNJAB STRENGTHENING URBAN TF026593 COMMUNITIES IN LCLS Albania ECA 191,500 191,500 69% INFORMAL SETTLEMENT CAPACITY BUILDING GRANT FOR THE COMMUNITY TF026597 LCLS Nicaragua LAC 765,500 765,500 99% DEVELOPMENT INNOVATION FUND TF026598 CHILD PROTECTION FUND LCLS Benin AFR 18,000 740,000 758,000 95% ENHANCING NGO TF026599 PARTICIPATION IN HIV/AIDS LCLS Cambodia EAP 412,000 412,000 95% PREVENTION AND CARE PROTECTION OF POOR TF026602 AND VULNERABLE GROUPS LCLS Samoa EAP 338,300 338,300 95% DURING HEALTH REFORM VILLAGE SELF-HELP TF026603 LCLS Sri Lanka SAR 750,000 750,000 99% LEARNING INITIATIVE PILOT SOCIAL PROTECTION TF026606 LCLS Tajikistan ECA 575,000 575,000 100% REFORM RECONCILIATION TF026774 THROUGH GENERAL LCLS Moldova ECA 800,000 800,000 100% SECONDARY EDUCATION BUILDING SOCIAL CAPITAL TF026777 IN SMALL DISADVANTAGED LCLS Bulgaria ECA 370,000 370,000 97% COMMUNITIES 5 JSDF Draft Final Report Appendices, December 2006 Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp INTEGRATION OF INTERNAL REFUGEES INTO REGULAR TF026778 LCLS Colombia LAC 75,000 2,750,075 2,825,075 76% SOCIAL SAFETY NET AND LABOR MARKETS BASIC EDUCATION IN TF026779 LCLS Cambodia EAP 45,500 1,575,000 1,620,500 99% RECONCILIATION AREAS EMPOWERING THE POOR TF026781 WOMEN IN THE WESTERN LCLS China EAP 950,000 950,000 100% REGION IMPROVING MICROFINANCE ACCESS FOR BARANGAY- TF026782 AND TOWN-BASED LCLS Philippines EAP 808,460 808,460 50% MICROENTERPRISE DEVELOPMENT UPSCALING URBAN POOR TF026783 COMMUNITY RENEWAL ACTV Philippines EAP 77,000 1,887,500 1,964,500 98% SCHEME MARGINAL FISHING TF026799 COMMUNITY ACTV Indonesia EAP 65,000 1,545,000 1,610,000 94% DEVELOPMENT PILOT LEGAL AID FOR POOR MEN TF026801 LCLS Jordan 191,000 191,000 97% WOMEN A LEGAL AID SERVICE TO TF026802 ACTV Sri Lanka SAR 207,684 207,684 100% POOR WOMEN PROJECT PROMOTING GENDER Yemen, MEN TF026803 EQUALITY IN THE REPUBLIC LCLS 383,435 383,435 98% Republic of A OF YEMEN VILLAGE INVESTMENT FOR TF026886 LCLS Lao PDR EAP 1,146,705 1,146,705 100% THE POOR PROJECT GRANT FOR REINFORCING TF050111 HIV/AIDS COMMUNITY ACTV CAR AFR 30,000 600,000 630,000 100% STRAGEGY IN THE CAR. ASSISTING THE URBAN TF050520 POOR IN LIAONING LCLS China EAP 36,000 1,207,000 1,243,000 99% PROVINCE NORTHERN UGANDA TF050522 YOUTH REHABILITATION ACTV Uganda AFR 50,000 1,611,000 1,661,000 100% FUND INTEGRATING "AT RISK" MEN TF050947 CHILDREN INTO LCLS Jordan 994,860 994,860 98% A MAINSTREAM SOCIETY STRATEGIC ALLIANCE FOR GRASSROOTS CAPACITY TF050948 BUILDING IN EARLY ACTV Guinea AFR 23,190 463,785 486,975 93% CHILDHOOD DEVELOPMENT LAW AND JUSTICE FOR THE TF051227 ACTV Ecuador LAC 1,780,000 1,780,000 100% POOR PILOTING REFORMS IN THE TF051331 ACTV Armenia ECA 961,000 961,000 85% CHILDCARE SYSTEM EMPOWERMENT OF LOCAL Lao People's TF051332 COMMUNITIES IN REMOTE ACTV Democratic EAP 20,150 717,270 737,420 100% UPLAND WATERSHED Republic COMMUNITY BASED URBAN TF051373 ACTV Armenia ECA 55,000 1,915,000 1,970,000 88% WATER SUPPLY 6 JSDF Draft Final Report Appendices, December 2006 Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp MANAGEMENT PROJECT COMMUNITY-LEVEL TF051374 VULNERABILITY ACTV Nicaragua LAC 1,489,300 1,489,300 94% REDUCTION PROGRAM EMPOWERING THE POOR: A PILOT IT PROGRAM FOR TF051375 ACTV India SAR 956,400 956,400 94% RURAL AREAS OF PUNE DISTRICT PILOT COMMUNITY DRIVEN TF051709 ACTV Moldova ECA 36,000 979,500 1,015,500 97% DEVELOPMENT PROJECT PUBLIC EDUCATION FOR TF052059 DISADVANTAGED LCLS Cambodia EAP 1,827,424 1,827,424 100% CHILDREN PILOT FOR COMMUNITY Kyrgyz TF052269 DRIVEN VILLAGE LCLS ECA 42,800 1,149,500 1,192,300 100% Republic INVESTMENTS PREVENTION AND TF052892 LCLS China EAP 90,000 2,000,000 2,090,000 92% CONTROL OF SARS SMALL COTTON FARMERS TF026776 ACTV Chad AFR 837,780 837,780 64% ASSOCIATIONS IMPROVE RURAL INCOMES & REDUCE RURAL POVERTY BY SUPPORTING Papua New TF050523 INTEGRATED SUSTAINABLE ACTV EAP 468,300 468,300 47% Guinea DEVELOPMENT OF COMMUNITIES AFFECTED BY OR CAPACITY BUILDING AND BASIC SERVICE TF050943 PROVISIONING IN ACTV Cambodia EAP 23,000 773,900 796,900 56% CAMBODIAN WATER USER GROUPS DEVELOPING COMMUNITY CAPACITIES FOR PRO- POOR BUDGETING AND TF050944 ACTV Philippines EAP 718,269 718,269 69% LOCAL GOVERNMENT ACCOUNTABILITY FOR POVERTY REDUCTION SINDH VULNERABLE TF050945 ACTV Pakistan SAR 40,000 910,000 950,000 3% GROUPS FUND IMPROVING LIVELIHOOD TF050946 SECURITY IN KOLKHETI ACTV Georgia ECA 1,379,875 1,379,875 65% LOWLAND REDUCING POVERTY IN HIGH MOUNTAIN TF051206 ACTV Tajikistan ECA 1,586,400 1,586,400 54% ENVIRONMENTS AROUND LAKE SAREZ. COMMUNITY-BASED URBAN TF051228 WATER SUPPLY ACTV Tajikistan ECA 93,500 2,539,000 2,632,500 54% MANAGEMENT PROJECT COMMUNITY-BASED TF051330 ACTV Georgia ECA 477,400 477,400 56% HEALTH INSURANCE 7 JSDF Draft Final Report Appendices, December 2006 Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp GIVING VOICE TO THE POOR IN INFLUENCING, TF051333 PLANNING AND ACTV Peru LAC 758,500 758,500 58% EVALUATING SERVICE DELIVERY IN PERU. STRATEGIC ALLIANCE FOR GRASSROOTS CAPACITY TF051548 BUILDING IN EARLY ACTV Mali AFR 23,189 463,785 486,974 59% CHILDHOOD DEVELOPMENT COMMUNITY-MANAGED AGRARIAN REFORM AND TF051601 ACTV Philippines EAP 2,000,000 2,000,000 32% POVERTY REDUCTION PROJECT East Asia TF051602 ASEAN SCHOOL NET. ACTV EAP 40,000 1,960,000 2,000,000 16% and Pacific EDUCATION AND EMPLOYMENT TF051701 ACTV Indonesia EAP 40,000 1,066,950 1,106,950 49% OPPORTUNITIES FOR UNDEREDUCATED YOUTH COMMUNITY MOBILIZATION TF051703 FOR SUSTAINABLE ACTV Mongolia EAP 1,300,790 1,300,790 63% LIVELIHOODS CONFLICT RESOLUTION TF051704 FOR NATURAL RESOURCE ACTV India SAR 9,825 196,500 206,325 48% MANAGEMENT IN INDIA ENHANCING PARTICIPATION IN EDUCATION THROUGH TF051706 ACTV Pakistan SAR 706,500 706,500 15% INNOVATIVE SCHEMES FOR THE EXCLUDED AND VULNERABLE. EMPOWERING POOREST OF POOR WOMEN&YOUNG GIRLS-FOCUSSING ON TF051707 ACTV Sri Lanka SAR 94,695 1,894,305 1,989,000 39% MOST AFFECTED AREAS OF HAMBANTOTA (SOUTH) & NORTH & EAST LOCAL GOVERNANCE AND Russian TF051789 CIVIC ENGAGEMENT IN ACTV ECA 1,252,700 1,252,700 45% Federation RURAL RUSSIA BUILDING CAPACITY FOR CIVIC ENGAGEMENT AND COMMUNITY-DRIVEN Sao Tome TF052058 ACTV AFR 649,450 649,450 54% DEVELOPMENT FOR and Principe IMPROVED SOCIAL SERVICES PREVENTION AND TF052836 ACTV Philippines EAP 200,000 200,000 22% CONTROL OF SARS TF052849 SEWA CAPACITY BUILDING ACTV India SAR 18,000 1,191,636 1,209,636 20% LIMA RAPID TF052877 ACTV Peru LAC 34,000 1,145,000 1,179,000 24% EMPOWERMENT FUND COMMUNITY DISASTER TF052879 MANAGEMENT IN THE ACTV Honduras LAC 1,984,500 1,984,500 70% BARRIOS OF TEGUCIGALPA PILOT PROJECTS ON TF052880 ACTV India SAR 18,000 595,140 613,140 25% PANCHAYAT 8 JSDF Draft Final Report Appendices, December 2006 Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp DECENTRALIZATION COMMUNITY ACTION FOR Congo, TF052882 REINTEGRATION AND ACTV AFR 34,400 708,400 742,800 31% Republic of RECOVERY EMPOWERING THE RURAL POOR THROUGH TF052884 ACTV Sri Lanka SAR 41,900 914,660 956,560 9% COMMUNITY INFORMATION AND LEARNING CENTER Egypt, Arab MEN TF052886 PREVENTING CHILD LABOR ACTV 75,000 1,877,487 1,952,487 23% Republic of A PASTORAL COMMUNITY TF052888 ACTV Ethiopia AFR 82,340 1,877,660 1,960,000 10% SUPPORT GRANT DEVELOPING CENTRAL AMERICANS SMALL Central TF052937 FARMERS LINKS TO ACTV LAC 20,000 900,000 920,000 28% America SPECIALITY COFFEE MARKET EARLY CHILDHOOD CARE TF052939 ACTV Vietnam EAP 1,910,800 1,910,800 15% AND DEVELOPMENT STRENGTHENING THE RESOURCES ASSESSMENT TF053107 ACTV Cameroon EAP 30,000 550,180 580,180 30% BASE OF RURAL COMMUNITIES PREVENTION AND TF053273 ACTV Vietnam EAP 500,000 500,000 44% CONTROL OF SARS COMMUNITY BASED COASTAL RESOURCE TF053440 MANAGEMENT AND ACTV Tanzania AFR 67,500 1,818,173 1,885,673 24% SUSTAINABLE LIVELIHOODS TF053442 SUPPORT FOR FEMALE & HEADED HOUSEHOLDS IN ACTV Indonesia EAP 40,000 1,656,700 1,696,700 35% TF054209 CONFLICT AREAS II PARTICIPATORY RURAL TF053444 ACTV Pakistan SAR 1,138,350 1,138,350 26% SANITATION IN SOUTHERN JHARKHAND PARTICIPATORY FOREST TF053445 ACTV India SAR 26,400 432,000 458,400 34% MANAGEMENT - CAPACITY BUILDING WOMEN'S INITIATIVE FOR SEX EDUCATION AND TF053811 ACTV Nigeria AFR 1,499,929 1,499,929 10% ECONOMIC EMPOWERMENT (WISE) CAPACITY BUILDING OF TF053812 NGOS FOR HIV/AIDS/STDS ACTV China EAP 60,000 661,400 721,400 13% PREVENTION IMPROVING EDUCATIONAL QUALITY THROUGH TF053814 ACTV Indonesia EAP 27,059 991,553 1,018,612 10% ENHANCED COMMUNITY PARTICIPATION UPPER SEMARANG LAND CONSERVATION AND TF053816 ACTV Indonesia EAP 65,000 1,299,030 1,364,030 23% POVERTY ALLEVIATION PROJECT 9 JSDF Draft Final Report Appendices, December 2006 Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp SUSTAINABLE RURAL Lao People's TF053818 MOBILITY AND ACTV Democratic EAP 26,000 909,200 935,200 13% PARTICIPATION Republic COMMUNITY-LED INFRASTRUCTURE TF053820 DEVELOPMENT FOR THE ACTV Mongolia EAP 95,000 1,892,100 1,987,100 12% URBAN POOR IN ULAANBAATAR DEV.& TESTING OF INNOVATIVE APPROACHES FOR MAINSTREAMING TF053822 ACTV Philippines EAP 30,480 1,148,280 1,178,760 15% INDIGENOUS PEOPLES(IPS)IN SELECTED AGRARIAN REFORM COMM. CAPACITY BUILDING FOR TF053826 COMMUNITY FOREST ACTV India SAR 36,600 1,915,900 1,952,500 15% MANAGEMENT EDUCATION IMPROVEMENT TF053828 FOR INDIGENOUS ACTV Colombia LAC 900,000 900,000 10% POPULATION ACCESS TO LEGAL SERVICES FOR TF054211 ACTV Cape Verde AFR 45,300 909,160 954,460 11% VULNERABLE GROUPS INCLUDING WOMEN ENERGY,GENDER AND POVERTY ALLEVIATION: TF054213 ACTV Ethiopia AFR 96,000 1,860,650 1,956,650 0% THE ETHIOPIAN WOMEN FUELWOOD CARRIERS PUBLIC-PRIVATE PARTNERSHIP FOR HAND TF054215 ACTV Senegal AFR 726,156 726,156 10% WASHING WITH SOAP (PPPHW) SOCIAL DEVELOPMENT TF054216 FUND AGENCY (AFDS) - ACTV Senegal AFR 32,800 1,967,200 2,000,000 31% CASAMANCE PROGRAM EMPOWERING POOR AND VULNERABLE GROUPS TO TF054218 PARTICIPATE IN THE ACTV Philippines EAP 2,000,000 2,000,000 17% NATIONAL ANTI-POVERTY PROGRAM SMALLHOLDER RECOVERY ASSISTANCE FROM AVIAN TF054219 INFLUENZE AND VIGILANCE ACTV Vietnam EAP 75,000 1,783,000 1,858,000 18% AGANST FURTHER OUTBREAKS CAPACITY BUILDING FOR TF054221 MANAGEMENT OF CHILD ACTV Bulgaria ECA 543,000 543,000 23% WELFARE SERVICES YOUTH SOCIO-ECONOMIC EMPOWERMENT THROUGH INCLUSIVE BUSINESS TF054222 ACTV Moldova ECA 41,525 1,727,140 1,768,665 58% DEVELOPMENT & INNOVATIVE SOCIAL SERVICE DELIVERY COMMUNITY-BASED URBAN TF054224 WATER SUPPLY ACTV Uzbekistan ECA 40,000 1,694,419 1,734,419 1% MANAGEMENT PROJECT 10 JSDF Draft Final Report Appendices, December 2006 Fund Bank % Grant Total Fund Name Status Recipient Reg. Inc Total Amount Amount Amount Disp GRASSROOTS CAPACITY- BUILDING IN SOCIAL TF054226 AUDITING FOR ENHANCED ACTV Ecuador LAC 20,000 411,850 431,850 3% SERVICE DELIVERY TO THE POOR CAPACITY BUILDING FOR BEHAVIORAL CHANGE: TF054228 ACTV Peru LAC 42,000 849,000 891,000 47% REDUCING DIARRHEA IN POOR CHILDREN COMMUNITY WATER Yemen, MEN TF054230 ACTV 50,000 1,078,200 1,128,200 10% MANAGEMENT PROJECT Republic of A FOSTERING SOCIAL TF054361 CAPITAL OF THE POOR AT ACTV Paraguay LAC 866,590 866,590 9% THE COMMUNITY LEVEL PARTICIPATORY TF054580 BUDGETING AND ACTV Indonesia EAP 1,999,988 1,999,988 19% EXPENDITURE TRACKING LEGAL AID SERVICES FOR TF054741 ACTV Mali AFR 45,200 902,940 948,140 0% THE POOR LEGAL AID SERVICES FOR TF054743 ACTV Mauritania AFR 43,300 863,520 906,820 0% THE POOR CAPACITY DEVELOPMENT TF054745 TO STRENGTHEN SOCIAL ACTV Sierra Leone AFR 99,790 1,888,700 1,988,490 19% CAPITAL INTRODUCING CDD APPROACHES IN CHINA'S TF054747 ACTV China EAP 45,000 1,929,600 1,974,600 10% RURAL POVERTY REDUCTION PROGRAM EMPOWERING FARMERS TF054751 FOR PARTICIPATORY ACTV Vietnam EAP 50,000 1,650,000 1,700,000 0% IRRIGATION MANAGEMENT INNOVATIVE APPROACHES TO COMMUNITY-BASED TF054753 ACTV Vietnam EAP 40,000 1,460,000 1,500,000 0% DISASTER RISK MANAGEMENT BUILDING CIVIL SOCIETY MEN TF054755 CAPACITY FOR EFFECTIVE ACTV Turkey 1,575,500 1,575,500 34% A LOCAL SERVICE DELIVERY CAPACITY BUILDING OF TF054756 ACTV Ukraine ECA 72,000 1,711,500 1,783,500 11% POOREST COMMUNITIES MAINSTREAMING TF054868 ACTV Cambodia EAP 65,000 1,800,291 1,865,291 1% INCLUSIVE EDUCATION ORISSA FUND FOR TF055552 ACTV India SAR 66,060 1,304,472 1,370,532 0% DEVELOPMENT INITIATIVES YOUTH DEVELOPMENT AND MEN TF055800 ACTV Turkey 60,000 1,872,000 1,932,000 26% SOCIAL INCLUSION A INTEGRATING REPRODUCTIVE HEALTH TF055811 WITH THE ACTV Guinea AFR 36,000 510,800 546,800 0% ONCHOCERCIASIS PROGRAM EDUCATION FOR VERY TF055900 ACTV Indonesia EAP 45,000 1,330,000 1,375,000 0% POOR CHILDREN 11 JSDF Draft Final Report Appendices, December 2006 C. Methodology C.1. Evaluation Framework Q. The main questions presented in Sub-Q A detailed elaboration of the evaluation questions Stakeholders & Sources of information No. the TOR number Relevance (and design) criteria 1. To what extent are JSDF-supported 1.1 To what extent are JSDF's objectives clear and measurable? Task I: Literature Review, HQ Interviews and activities achieving the Fund's Questionnaire Survey: objectives? 1.2 To what extent are the JSDF projects in line with the National poverty - Interviews with JSDF and TFO staff reduction strategy in the country? - Interviews with Bank Team Leaders 1.3 Does the JSDF (projects) have a clear and consistent [define terms!] - Documentary analysis of approved grants to analyse intervention logic that addresses identifiable problems of the poorest the clarity of the causal path and contribution to the and most vulnerable, and is this articulated? Fund's objectives - Findings from web-based survey and field data 1.4 What features of the project can be described as innovative? collection 1.5 Have indicators been stated and to what extent are they measurable? Task II: Field Visits: 1.6 Are there adequate arrangements for monitoring indicators? Who does - Interviews with Implementing NGOs it, what is done and when? - Interviews with non implementing NGOs (especially 1.7 To what extent have implementing partners been identified on the basis Japanese) of their grassroots understanding and training skills? - Quantitative and qualitative assessment using participatory methods with project beneficiaries 1.8 To what extent were the partner country stakeholders (including target beneficiaries) involved in the project identification and design process? Literature Review, HQ Interviews and Questionnaire 2.a Is the seed fund contributing to 2.1 To what extent is project preparation more participative in the presence Survey: effective participative project of a seed fund grant? - Interviews with Bank Team Leaders preparation? How can it do this even - Interviews with implementing NGOs better? - Documentary analysis of approved grants to analyse 12 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in Sub-Q A detailed elaboration of the evaluation questions Stakeholders & Sources of information No. the TOR number 2.b Is project preparation more 2.2 How does participation in JSDF project preparation compare with level and scope of participation with/without seed funds participative in the presence of a projects without seed funds, and those without any trust funds? - Findings from questionnaire survey of large sample of seed fund grant? task managers, project implementation managers, Japanese NGOs, other country-based stakeholders Field Visits: - Quantitative and qualitative participatory assessments with project beneficiaries Complimentarity 3.a What other funding sources are used 3.1 How well do JSDF activities achieve their social development outcomes Literature Review, HQ Interviews and Questionnaire for similar activities? vis-à-vis similar activities funded by other sources? Survey: - Interviews with JSDF and TFO staff Analyse outcomes from JSDF ratings - Interviews with Bank Team Leaders (including staff Is there data for analysis of similar ratings from other trust funds? managing comparable funds) Is there any data e.g. From OED of performance from an independent - Implementing NGOs source? - Documentary analysis of approved grants to analyse 3.b To what extent is the JSDF 3.2 To what extent have other [investment?] projects focused on the needs level and scope of participation with/without seed funds supporting development outcomes of the poorest and most vulnerable? - Document review of small sample of grants from other that are not being achieved by other trust funds. Independent data from another source e.g. initiatives? IEG 4. Can the complementarity between 4.1 Is there complementarity between JSDF and World Bank projects? ( JSDF projects and World Bank Coordination, cooperation and/or enhancing effectiveness of other Analyse outcomes from JSDF ratings. Is there data for projects be improved? operations?) analysis of similar ratings from other trust funds? 4.2 How could or should JSDF projects attempt to achieve greater complementarity with World Bank projects? Field Visits: - Interviews with project stakeholders (especially 4.3 To what extent are effective mechanisms in place to ensure greater municipal authorities and NGOs) complementarity with other projects? 4.4 What examples are there of projects being scaled up? 13 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in Sub-Q A detailed elaboration of the evaluation questions Stakeholders & Sources of information No. the TOR number Value Added 5 What has been the value-added of 5.1 Have JSDF projects reflected or enhanced World Bank best practice for Literature Review, HQ Interviews and Questionnaire JSDF projects to World Bank building capacity among civil society, local communities and NGOs? Survey: operations? - Interviews with Bank SDV and CDD departments (and PREM?) for social development and country priorities - Interviews with Bank IEG for recent capacity building 5.2 Have JSDF projects reflected or enhanced best practice and policy for evaluations the Bank's social development operations and country priorities (as reflected in CASs, PRSs - Interviews with Bank Team Leaders - Document review of findings related to JSDF in ICRs and other products from IEG (ex-OED) - Document review of ICMs Field Visits: - Findings from quantitative and qualitative participatory assessments with local stakeholders and beneficiaries Efficiency (Management) criteria 6.a How are JSDF grants managed within 6.1 Are management systems (including the application and approval Literature Review, HQ Interviews and Questionnaire the Bank, process) and decision-making structures designed efficiently? What is Survey: the project cycle? 6.b What are or should be the roles of 6.2 What are the views of country teams and networks regarding their - Interviews with JSDF and TFO staff country teams and networks? respective roles? - Interviews with Japanese Foreign Ministry and NGOs (if representation in Washington) 6.3 What is the organisational and decision-making framework? - Interviews with Bank Team Leaders 6.4 At what level are the key funding and management decisions made? - Review of the grant application-approval-reporting procedure - Findings from web based survey of large sample of task managers, project implementation managers, and Japanese NGOs. Field Visits: - Analysis of management and implementation arrangements in-country 14 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in Sub-Q A detailed elaboration of the evaluation questions Stakeholders & Sources of information No. the TOR number 7 Are there ways in which the Fund's 7.1 What are the resources required and relative costs (including Literature Review, HQ Interviews and Questionnaire program management, procedures or transaction costs) of JSDF staff, national and international consultants, Survey: policies might be changed to enable its and NGO implementers compared to alternatives? [Develop process - Interviews with JSDF and TFO staff objectives to be achieved more cost- chart which allows comparison across the project cycle, and then - Interviews (possibly V-C) with Japanese Foreign effectively? compare with other trust funds]. Develop cost structure of fund. Ministry and NGOs (if representation in Washington) 7.2 Are outputs achieved within the expected cost and time? Are there - Interviews with Bank Team Leaders (including staff major cost- or time-overruns or budget revisions? managing comparable funds and those who have 7.3 What alternative models of management have been considered / managed several JSDF projects) implemented? - Review of the grant application-approval-reporting procedure - Findings from web-based survey and field data collection 8 Have JSDF projects been effective in 8.1 To what extent have indicators of working relations with NGOs been working with both local and articulated and measured? international (incl. Japanese) NGOs Field Visits: 8.2 What are the views of JSDF staff and NGOs regarding their working - Analysis of management and implementation collaboration and relationship? arrangements in-country Monitoring and Evaluation 9 9.1 What is the monitoring system and how successfully do monitoring Literature Review, HQ Interviews and Questionnaire systems enable JSDF to collect relevant data on activities, outputs, Survey: including GRM and ICM forms outcomes and impact? - Interviews with JSDF , TFO IEG and e-Trust Fund staff 9.2 What type of evaluation is carried out and what have been the results? - Interviews with Japanese Foreign Ministry and NGOs - Interviews with Bank Team Leaders (including staff 9.3 What reporting and feed-back structures are in place and how are the managing comparable funds and those who have outputs of monitoring and evaluation procedures fed back into decision- managed several JSDF projects) making processes and management procedures - Review of the grant application-approval-reporting 9.4 How are the results of innovative practices disseminated and give procedure examples? - Findings from web-based survey of large sample of task managers, project implementation managers, Japanese NGOs. Field Visits: - Analysis of monitoring and evaluation arrangements in-country 15 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in Sub-Q A detailed elaboration of the evaluation questions Stakeholders & Sources of information No. the TOR number Effectiveness criteria (outputs and outcomes) 10 Within the different kinds of activities 10.1 (Note: What activities have been supported, and how far can these be Literature Review, HQ Interviews and Questionnaire that JSDF has supported, are some categorised for comparison?) Survey: more effective in meeting JSDF - Interviews with JSDF and TFO staff objectives than others? If so, what 10.2 To what extent have objectives, outcomes and targets been achieved? (List and enumerate) - Interviews with Japanese Foreign Ministry and NGOs are the factors that are contributing - Interviews with Bank Team Leaders (including staff to this? 10.3 To what extent can effectiveness in meeting objectives be attributable to managing comparable funds and those who have project design, management, monitoring and implementation? managed several JSDF projects) 10.4 To what extent can effectiveness be attributable to external factors, - Findings from web based survey of large sample of including social, economic and political context, and the environment for task managers, project implementation managers, aid and donor-government relations? (Japanese) NGOs. Field Visits: - Analysis of outputs, outcomes in-country Impact (including sustainability) criteria 11 {The development impact of the 11.1 (What has been the nature and scale of the development impact on the Literature Review, HQ Interviews and Questionnaire JSDF interventions, and the nature poorest and most vulnerable groups?) Survey: and scale of the impact on the - Interviews with JSDF and TFO staff intended beneficiaries} 11.2 To what extent have capacity building projects built capacity, - Interviews (possibly V-C) with Japanese Foreign participation and empowerment among civil society, local communities Ministry and NGOs (if representation in Washington) and NGOs? - Interviews with Bank Team Leaders (including staff 11.3 Are the poorest and most vulnerable satisfied with JSDF's managing comparable funds and those who have contributions? managed several JSDF projects) - Findings from web of large sample of task managers, 11.4 How far have JSDF interventions been able to avoid elite capture of project implementation managers, (Japanese) NGOs benefits by richer groups and individuals? 11.5 To what extent and how have benefits been sustained among the Field Visits: poorest and most vulnerable? Analysis of impact and sustainability in-country 11.6 To what extent have interventions been innovative, rapid, demonstrable and replicable? 16 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in Sub-Q A detailed elaboration of the evaluation questions Stakeholders & Sources of information No. the TOR number 11.7 Where the Japanese satisfied with the level of visibility (for Japan) on this project. How was this manifested? Are stakeholders aware the project is Japan grant funded? Which ones? Lessons Learnt 12 Are there specific lessons learned by - Interviews with JSDF and TFO staff the grant activities, in terms of: - Interviews (possibly V-C) with Japanese Foreign Ministry and NGOs § technical design - Interviews with Bank Team Leaders (including staff § the most appropriate kinds of managing comparable funds and those who have project managed several JSDF projects) § reaching target beneficiaries - Findings from web-based survey of large sample of § governance or management task managers, project implementation managers, arrangements (Japanese) NGOs that JSDF has funded which should be Field Visits: more widely disseminated? - Evidence of lessons learnt in-country and application on other projects 17 JSDF Draft Final Report Appendices, December 2006 C.2. Document Review The review of 43 JSDF projects was informed by documents produced for internal use by the Bank at the various stages of the grant programme cycle, namely the Grant Application Form (AF), Grant Reporting and Monitoring Summary (GRM) and the Implementation Completion Memorandum (ICM). ICM's are only produced at project completion for those grants with a value greater than $1 million. For this review, only 14 of the 43 projects being evaluated had an ICM. Document Review Proforma The proforma (as given in this Appendix below) was designed to follow the key information statements in the 3 grant documents reviewed, while also taking account of the Evaluation Framework. Using the proforma, the documents were reviewed by the evaluation team at three levels of detail. § The first included sections where the evaluator captured the key features of the content of a particular section in the documents by either summarising sections or inserting the World Bank rankings provided § The second level involved the evaluator rating certain aspects of the project recorded in the document based on details given in the text § Third level: After reviewing all documents available on a grant, the reviewer made a short summary assessment for each project according to the four pillars of JSDF ­ participation, innovation, rapid results and sustainability. For the second level of the document review (described above) the Evaluation team devised a rating system that could be differentiated from the Bank ratings. The numbers 1, 2, 3 and 4 were used to represent High, Substantial, Modest and Low respectively, where `High' describes situations where planned expectations have been exceeded, Substantial where objectives have been are met, Modest in situations that have fallen short of targets and Low when few, if any, targets have been met. 18 Appendix II: Document Review Proforma TF NUMBER & NAME: Grant Type Documents available to Review A GRANT PROPOSAL [Insert Text: Justification of rating] A1 To what extent are the grant's objectives clearly articulated? [Insert Text: Justification of rating] A2 To what extent are the grant's objectives relevant? Response given for; Are Japanese NGOs participating in this A3 JSDF activity? Response given for; Have Japanese aid agencies been A4 consulted in the development of the JSDF application? [Insert Text: Justification of rating] A5 To what extent are the grant's components clearly articulated? [Insert Text: Justification of rating] How clear and consistent is the intervention logic? A6 (activity>outputs>outcomes>impact) [Insert Value: USD] A7 Is there a specific M&E Component? [Insert Text: Summary] A8 Summarise the section on Genesis [Insert Text: Summary] A9 Summarise the innovative features of the projects. [Insert Text: Summary] A10 Summarise the mechanism for long term sustainability [Insert Text: Justification of rating] To what extent has the design involved the participation of local A11 stakeholders? [Insert Text: Summary] A12 Summarise how NGO/CBO partners were/will be selected? [Insert Text: Evaluator Comment on A13 Is the Grant linked to a specific WB financed operation possible complimentarity] [Insert Text: Summary] Summarise how the Grant is expected to add value to the WB A14 Financed Operation [Insert Text: Summary] A15 Summarise the rationale for Grant Funding vs Bank Lending [Insert Text: Justification of rating] A16 How robust was the identification of risks ] [Insert Text: Justification of rating] A17 Summarise the justification for incremental Bank Costs [Insert Text: Justification of rating] A18 How clearly are monitoring arrangements articulated? [Insert A19 How many KPIs are listed Number] [Insert Text: Justification of rating] A20 How clearly defined are the KP indicators? Amounts given in Grant Categories of Expenditure & Amounts A21 (US$) [Insert A22 Services Number] [Insert A23 Training Number] [Insert A24 Goods Number] [Insert A25 Civil Works Number] [Insert A26 Sub-Grants Number] [Insert A27 Incremental Costs Number] [Insert [Specify] A28 Other Number] A29 Total 0 B GRM B1 Date of GRM From [Day/Month/Year] [Date] B2 Date of GRM To [Day/Month/Year] [Date] B3 WB rating given for Main GDO Rating B4 WB rating given for Main Implementation Progress [Insert Text: Evaluators Comment on text] B5 WB rating given for Progress towards achieving Grant Objectives [Insert Text: Evaluators Comment on text] B6 WB rating given for Overall Implementation progress [Insert Text: Summary] B7 Summarise comments on Component Performance Rating [Insert Text: Summary] B8 Summarise comments on Component Output [Insert B9 How many Grant Outcome Indicators are reported on Number] [Insert Text: Justification of rating] B10 How clearly defined are the Grant Outcome indicators? ] [Insert Text: Evaluators Comment] How similar are the Grant Outcome indicators to the KPIs B11 identified in the Application form [Insert Text: Evaluators Comment on text] B12 WB rating for Project Management [Insert Text: Evaluators Comment on text] B13 WB rating for Financial Management [Insert Text: Evaluators Comment on text] B14 WB rating for Counterpart Funding [Insert Text: Evaluators Comment on text] B15 WB rating for Procurement [Insert Text: Evaluators Comment on text] B16 WB rating for M&E [Insert Text: Summary] B17 Summarise succesful outcomes of participatory activities highlighted [Insert Text: Summary] B18 Summarise strenghths/weaknesses of NGO involved [Insert Text: Summary] B19 Summarise Lessons learnt so far C ICM C1 Report date of ICM [Day/Month/Year] [Date] [Insert Text: Summary] C2 Summarise Changes to grant objectives [Insert Text: Evaluators Comment on text] C3 WB rating given for Achievementof Grant Objectives [Insert Text: Evaluators Comment on text] C4 WB rating given for acheivement of Outputs [Insert Text: Evaluators Comment on text] WB rating given for Achievment of Outcomes - Achievment of C5 development Results [Insert Text: Evaluators Comment on text] C6 WB rating given for Achievment of Outcomes - Relevance [Insert Text: Evaluators Comment on text] C7 WB rating given for Achievment of Outcomes - Efficacy [Insert Text: Evaluators Comment on text] C8 WB rating given for Achievment of Outcomes - Efficiency [Insert Text: Evaluators Comment on text] C9 WB rating given for Impact - Capacity Building Impact ] [Insert Text: Evaluators Comment on text] C10 WB rating given for Impact - Sustainability [Insert Text: Evaluators Comment on text] C11 WB rating given for Impact - Followup activities/investment [Insert Text: Evaluators Comment on text] C12 WB rating given on Performance - Bank [Insert Text: Evaluators Comment on text] C13 WB rating given on Performance - Recipient [Insert Text: Summary] C14 Summarise lessons learnt/ Recommendations for the Bank [Insert Text: Summary] C15 Summarise lessons learnt/ Recommendations for the Recipient [Insert Text: Summary] C16 Summarise lessons learnt/ Recommendations for the Donor [Insert Text: Summary] Summarise lessons learnt/ Recommendations for the C17 Development Community D EVALUATORS SUMMARY [Insert Text: Justification of rating] D1 Rate the innovativeness of the project [Insert Text: Justification of rating] Rate how the project responded to the needs of the poorest and D2 most vulnerable [Insert Text: Justification of rating] D3 Rate how rapid and demonstable the benefits of the project were [Insert Text: Justification of rating] D4 Rate the sustainability of the project [Insert Text: Justification of rating] D5 Rate the impact of capacity building activities [Insert Text: Justification of rating] Rate the participation & empowerment of civil society/ local D6 communities/ NGOs [Insert Text: Evaluators Comment] D7 Any other Comment ] JSDF Draft Final Report Appendices, December 2006 C.3. Interview Topic Guide This evaluation tool was designed specifically for the interviews held in Washington and, as with the other tools used is derived from the Evaluation Framework. It provided the facility to direct particular questions or groups of questions to specific interviewees depending on their role and knowledge of the Fund. Its format enabled the information gathered to be written up, compared and analysed for each key question across the range of interviews held. Interview Topic Guide Interviewee/s Interviewers Date Location Interviewee background Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... Relevance (and design) criteria 1. To what extent are JSDF-supported 1.1 To what extent are JSDF's objectives clear and activities achieving the Fund's measurable? objectives? 1.2 To what extent are the JSDF projects in line with the National poverty reduction strategy in the country? 1.3 Does the JSDF (projects) have a clear and consistent [define terms!] intervention logic that addresses identifiable problems of the poorest and most vulnerable, and is this articulated? 1.4 What features of the project can be described as innovative? 1.5 Have indicators been stated and to what extent are they measurable? 23 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... 1.6 Are there adequate arrangements for monitoring indicators? Who does it, what is done and when? 1.7 To what extent have [NGO?] implementing partners been identified on the basis of their grassroots understanding and training skills? 1.8 To what extent were the partner country stakeholders (including target beneficiaries) involved in the project identification and design process? 2.a Is the seed fund contributing to 2.1 To what extent is project preparation more effective participative project participative in the presence of a seed fund preparation? How can it do this even grant? better? 2.b Is project preparation more 2.2 How does participation in JSDF project participative in the presence of a preparation compare with projects without seed seed fund grant? funds, and those without any trust funds? [project preparation includes identifying indicators] Complementarity 3.a What other funding sources are used 3.1 How well do JSDF activities achieve their social for similar activities? development outcomes vis-à-vis similar activities funded by other sources? Analyse outcomes from JSDF ratings Is there data for analysis of similar ratings from other trust funds? Is there any data e.g. From OED of performance from an independent source? 3.b To what extent is the JSDF 3.2 To what extent have other [investment?] projects TTL supporting development outcomes focused on the needs of the poorest and most that are not being achieved by other vulnerable? [do table comparing fund objectives] initiatives? 4. Can the complementarity between 4.1 Is there complementarity between JSDF and TTL JSDF projects and World Bank World Bank projects? ( coordination, cooperation projects be improved? and/or enhancing effectiveness of other operations?) 24 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... 4.2 How could or should JSDF projects attempt to achieve greater complementarity with World Bank projects? 4.3 To what extent are effective mechanisms in place to ensure greater complementarity with other projects? 4.4 What examples are there of projects being scaled TTL up? Value Added 5 What has been the value-added of 5.1 Have JSDF projects reflected or enhanced World TTL JSDF projects to World Bank Bank best practice for building capacity among operations? civil society, local communities and NGOs? 5.2 Have JSDF projects reflected or enhanced best practice and policy for the Bank's social development operations and country priorities (as reflected in CASs, PRSs)? Efficiency (Management) Criteria 6.a How are JSDF grants managed within 6.1 Are management systems (including the TTL the Bank, application and approval process) and decision- making structures designed efficiently? 6.b What are or should be the roles of 6.2 What are the views of country teams and TTL country teams and networks? networks regarding their respective roles? 6.3 What is the organisational and decision-making framework? 6.4 At what level are the key funding and management decisions made? 25 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... 7 Are there ways in which the Fund's 7.1 What are the resources required and relative program management, procedures or costs (including transaction costs) of JSDF staff, policies might be changed to enable its national and international consultants, and NGO objectives to be achieved more cost- implementers compared to alternatives? effectively? [develop process chart which allows comparison across the project cycle, and then compare with other trust funds] Develop cost structure of fund. 7.2 Are outputs achieved within the expected cost and time? Are there major cost- or time-overruns or budget revisions? 7.3 What alternative models of management have been considered / implemented? Monitoring & Evaluation 8 8.1 What is the monitoring system and how TTL successfully do monitoring systems enable JSDF to collect relevant data on activities, outputs, outcomes and impact? [NB: this is also an effectiveness issue] 8.2 What type of evaluation is carried out and what TTL have been the results? 8.3 What reporting and feed-back structures are in place and how are the outputs of monitoring and evaluation procedures fed back into decision- making processes and management procedures? 8.4 How are the results of innovative practices TTL disseminated and give examples? 26 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... Effectiveness Criteria (results and outcomes) 9 Within the different kinds of 9.1 (Note: What activities have been supported, and TTL activities that JSDF has supported, how far can these be categorised for are some more effective in meeting comparison?) JSDF objectives than others? If so, what are the factors that are contributing to this? 9.2 To what extent have objectives, outcomes and TTL targets been achieved? (List and enumerate) 9.3 To what extent can effectiveness in meeting objectives be attributable to project design, management, monitoring and implementation? 9.4 To what extent can effectiveness be attributable to external factors, including social, economic and political context, and the environment for aid and donor-government relations? Have JSDF projects been effective in 9.5 To what extent have indicators of working working with both local and relations with NGOs been articulated and international (incl. Japanese) NGOs? measured? 9.6 What are the views of JSDF staff and NGOs regarding their working collaboration and relationship? Impact (including sustainability) Criteria 10 {The development impact of the 10.1 (What has been the nature and scale of the TTL JSDF interventions, and the nature development impact on the poorest and most and scale of the impact on the vulnerable groups?) intended beneficiaries} 27 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... 10.2 To what extent have capacity building projects TTL built capacity, participation and empowerment among civil society, local communities and NGOs? 10.3 Are the poorest and most vulnerable satisfied TTL with JSDF's contributions? 10.4 How far have JSDF interventions been able to TTL avoid elite capture of benefits by richer groups and individuals? 10.5 To what extent and how have benefits been TTL sustained among the poorest and most vulnerable? 10.6 To what extent have interventions been TTL innovative, rapid, demonstrable and replicable? 10.7 Were the Japanese satisfied with the level of visibility (for Japan) on this project? How was this manifested? Are stakeholders aware the project is Japan grant funded? Which ones? 28 JSDF Draft Final Report Appendices, December 2006 Q. The main questions presented in the Sub-Q A detailed elaboration of the evaluation questions Q. to No. TOR number ... Lessons learnt 1 Are there specific lessons learned by 11.1 TTL the grant activities, in terms of: § technical design § the most appropriate kinds of project § reaching target beneficiaries § governance or management arrangements that JSDF has funded which should be more widely disseminated? 29 JSDF Draft Final Report Appendices, December 2006 C.4 Field Visit Guide to Field Visit Methodology This version of the methodology for the JSDF fieldwork programme has been subject to a number of revisions in light of comments received during the pre-fieldwork workshop held on 15th August 2006 at ITAD, with further updates being made since then, including a section on the Project Visit Report and a note on the use of the scores in the PEP (added during the course of the first visit to Lao). Notes on the experience of the early field visits will be circulated as additional material for use by the field teams. Purpose of this paper This paper supports the preparation for country visits, and specifically the development of the methodology, as part of the JSDF evaluation. This paper is a draft for discussion only. It presents key steps and questions that have been developed on the basis of the overall evaluation framework. The approach will have to be further adapted to the context of the specific countries and projects that will be visited. Examples from one of the sampled JSDF projects1 are used to illustrate issues being raised for discussion. Objective of the country visits In the terms of reference (TOR) the aim of the country visits is given as being: "In order to obtain rigorous data to evaluate progress in achieving project objectives and to assess the expected impact on the intended beneficiaries" Based on the outcome of the project selection process, eight countries will now be visited (with 10 projects in total). These are listed below. List of Selected Projects Fund Fund Name Status Recipient Amount ($) Evaluator2 TF026598 CHILD PROTECTION FUND LCLS Benin 758,000.00 Ann Condy TF050522 NORTHERN UGANDA YOUTH REHABILITATION FUND ACTV Uganda 1,661,000.00 Corrie Bell TF026799 MARGINAL FISHING COMMUNITY DEVELOPMENT PILOT ACTV Indonesia 1,610,000.00 Ann Condy & Corrie Bell TF026563 SQUATTER SETTLEMENTS PILOT ASSISTANCE PROJECT LCLS Indonesia 2,304,800.00 TF026886 VILLAGE INVESTMENT FOR THE POOR PROJECT LCLS Lao PDR 1,146,704.95 Mike Felton 1 "Pilot project for poverty reduction in six communes in Northern Mountain provinces of Vietnam". 2 An in-country (national) evaluator will accompany the named evaluators for each of the country visits. 30 JSDF Draft Final Report Appendices, December 2006 PILOT PROJECT FOR POVERTY REDUCTION IN SIX Johanna TF026559 COMMUNES IN NORTHERN MOUTAIN PROVINCES OF LCLS Vietnam 1,395,000.00 Pennarz VIETNAM TF051331 PILOTING REFORMS IN THE CHILDCARE SYSTEM ACTV Armenia 961,000.00 Mike Felton COMMUNITY BASED URBAN WATER SUPPLY TF051373 ACTV Armenia 1,970,000.00 MANAGEMENT PROJECT TF051227 LAW AND JUSTICE FOR THE POOR ACTV Ecuador 1,780,000.00 Muriel Visser EMPOWERING THE POOR: A PILOT IT PROGRAM FOR TF051375 ACTV India 956,400.00 Chris Barnett RURAL AREAS OF PUNE DISTRICT Outline of methodology The complexity of this aspect of the evaluation reinforces the need to have a systematic and rigorous approach to the field visits. The starting point is the Evaluation Framework, which provides a guide to those questions that need to be addressed at field level. The next stage has been to begin the process of developing the methodology and a field evaluation toolkit. An outline for the methodology is provided in the table below (14 days total). This is a guide only, and while useful in presenting the likely steps, it should be adapted according to the country and project context. Indicative visit schedule Main steps Staff involved Time Pre-visit August Recruitment and briefing of local Team Leader and consultant partners; alert Country Bank coordinating staff office, government and project agencies; refine/agree specific approach. Prepare and agree visit itinerary. Document review of project documents Evaluators 1 day (Grant Proposal, Grant Monitoring and Reporting summaries, Implementation Completion Memorandum (where available); review of additional documents, as available, and country strategies. Draw up and prepare stakeholder Evaluators 0.25 days analysis based on available information Country information August - September Refine stakeholder analysis and update / Evaluators with local 0.25 days confirm interview arrangements consultants 31 JSDF Draft Final Report Appendices, December 2006 Main steps Staff involved Time Interviews with World Bank Country Evaluators, local 1 day Office staff, project management staff of consultants, key implementation partner departments. stakeholders Groups discussions with representatives Evaluators, local 0.5 days of key stakeholders (NGO, consultants, consultants, key other development and co-funding stakeholders partners) Brief review of project fit with national Evaluators, local 0.25 days policies, PRSP and other strategies and consultants country programs, as applicable. Field work: August - September Review information on project Evaluators, local 0.25 days communities, select communities for field consultants visit; logistical arrangements. Field visits to selected communities to Evaluators, local 5 days meet and interview project staff at consultants different levels, local officials and target beneficiaries; inspect physical works to cross check information from evaluation reports; visit selected households and other documentation. Feedback findings to key stakeholders; Evaluators, local 0.5 days debriefing with World Bank country office consultants Analysis and report-writing 3-5 days The main toolkit for the country visits are detailed in Section IV of this paper. This covers the following techniques: · Literature Review · Stakeholder Analysis · Key Informant (stakeholder) Interviews · Fieldwork in target communities There are three main deliverables from each country visit (see Annex 3, for full details of contractual requirements for all consultants). In short, the main deliverables are: · Aid Memoire: At the end of the country visit, the 2-person team will prepare an aide memoire of 1 ­ 2 pages. Likely audience and purpose: Internal team circulation, to share findings and improve methodology. For more details see Section III (a). · Project Evaluation Pro forma (or PEP): This is the main evaluation tool/deliverable. It sets out the evaluation questions and sub-questions, including prompts for semi- structured interviews. It is not a questionnaire in the conventional sense, but serves as both an interview guide and a report format with key sections and questions that have to 32 JSDF Draft Final Report Appendices, December 2006 be answered by the evaluator. Likely audience and purpose: Core evaluation team, used internally for the synthesis of projects and writing the Final Report. For more details see Section III (b). · Project Visit Report (PVR): A 5-page report summarising key findings and recommendations. Likely audience and purpose: JSDF/World Bank, to become part of the appendix to the Final Report. For more details see Section III (c). Consultant's deliverables As outlined above, there are three main deliverables. Details are provided in the sections that follow. If there is any uncertainty or lack of clarity, then clarification should be sought from the Team Leader (Mike Felton). The Aid Memoire The drafting of the aide memoire should not be an arduous task, but instead should focus on drawing together findings and the discussion between the two evaluators. It should ideally be drafted before the end of the country visit. The aide memoire should be 1-2 pages long, and should: (i) Identify initial findings and hypotheses (ii) Draft a first set of lessons and recommendations before the international consultant departs (iii) Provide comments on methodology (including recommendations to consultants making subsequent field trips) The Project Evaluation Pro forma (PEP) The Project Evaluation Pro forma (PEP) sets out the evaluation questions and sub-questions relevant to the field visit, together with further questions and prompts for the evaluators to follow. The PEP is not a questionnaire in the conventional sense, but serves as both an interview guide and a report format with key sections and questions that have to be answered by the evaluator. Thus, the sections/questions correspond with the Evaluation Framework, ensuring that collectively the country visit interviews and country document reviews address the key questions in the TOR. The format also provides a consistent structure for the individual project report. Taking account of the fact that the focus of each interview with different stakeholders will depend on the type of project and other issues, it is however, at the discretion of the evaluator to decide which questions should be asked and of whom. In this way the tool helps ensure both the systematic gathering of evidence, while also maintaining an element of flexibility. Flexibility is considered especially important, as country contexts and programmes will vary considerably in size and priorities ­ with some questions and lines of enquiry taking a higher priority than others. Although the PEP will be standardised across all projects, each country and project will require a substantial amount of pre-planning to fit a common set of questions into local circumstances. Where 33 JSDF Draft Final Report Appendices, December 2006 necessary, additional sub-questions (and prompts) can be added by the evaluators for this purpose. A feature of the PEP is the use of ratings, which are widely used in the World Bank. The evaluators believe ratings are a useful and practical way to assess progress against the key criteria, and while the results will need to be treated with caution, they will enable inter-project comparisons to be made. A one-sentence justification for each rating should also be provided by the evaluator. The current version of the PEP can be found following this discussion paper, Annex 1. Project Visit Report (PVR) The Project Visit Report (PVR) will be 5-page report produced after the country visit. The report will follow the classic "findings, conclusion and recommendations" approach and a standard layout / table of contents will be provided along with further information on content and style. The Project Visit Report will form part of an appendix to the Study Final Report and is to be provided within 12 days of return to UK / home country. See Annex 2, for outline. The country visit toolkit The PEP is not only a reporting structure (and deliverable) but also provides a basic structure for questions for the country visits. Some of these questions may need to be elaborated further for a particular context, e.g. when using the stakeholder analysis and visiting the target communities. In summary the PEP is the overriding structure (what to ask), while the section that follows provides various tools that can be used to gather the evidence. Wherever relevant, this section makes reference to the PEP questions/sections. Literature Review A number of reports on JSDF funded projects have already been made available to the core evaluation team. Under Task 1, a systematic document review was carried out to assess some of the key questions in the evaluation framework, especially those in relation to project design (e.g. intervention logic, indicators, etc). The findings of the documents review will provide the background information for the country visits. At the same time, they will help the evaluators to further specify the focus of country visits. In preparation for the country visits, the evaluators would need to review the available project documentation, which typically includes the funding proposal, status reports and the Implementation Completion Memorandum (ICM). These reports contain concise information on project approaches, implementation and disbursement schedules together with a broad assessment of achievements by the TTL. Additional project documents, where available, will also be reviewed both before and during country visits. These documents do not, however, necessarily capture wider aspects of social impacts, stakeholder participation and other important lessons that should be documented by this evaluation. It will therefore important for the evaluator to identify and assess additional documents and resources before the field visit. 34 JSDF Draft Final Report Appendices, December 2006 Review strategy and approach of JSDF Project In a first step, the evaluators need to review the strategy and approach of the JSDF project, especially how the project is expected to contribute to the JSDF objectives, which are: · To support innovative programs which directly respond to the needs of the poorest and most vulnerable groups of society; · To provide rapid and demonstrable benefits, which can be sustained, to the poorest and most vulnerable groups; · To build capacity, participation and empowerment of civil society, local communities and NGOs. Projects vary in their approach on how they intend to create benefits for the poorest and most vulnerable groups. Some JSDF projects are supporting grassroots initiatives to address the most urgent needs of target beneficiaries; others are working in a more strategic way in building capacities among key stakeholders at different levels. Types of JSDF projects: ¨ Projects with a stated objective of providing direct and visible investments into target communities. ¨ Projects with a stated objective of strengthening capacities of NGOs providing services or advocacy for poor and vulnerable groups. ¨ Projects with a stated objectives of strengthening the impact and effectiveness of existing initiatives, programs or politicise. The following is an example for a project that has a clear objective of facilitating wider change among stakeholders: In the case of the Northern Mountain Poverty Reduction Project, the World Bank Country Office has conducted a number of "learning exercises" together with a comprehensive impact assessment for the (WB-funded) main project which provide a good overview of the innovative features of this project and a more in-depth analysis of participation in this project. In addition, other donors have supported regular process reviews (conducted by CARE) with a particular focus on how poor communities have participated in all aspects of the project. The strategic nature and focus of JSDF project is of central importance for the fund and therefore needs to be assessed in further detail. Therefore, the evaluators will review the key policy documents that define the national strategy for poverty reduction (PRS) and the extent to which poor and vulnerable groups are meant to participate in the process. Relevant questions from the PEP To what extent are the JSDF projects in line with the National Poverty Reduction Strategy (PRS) in the country? [Q1.2] - Briefly describe the national policy context (relevant to this project)... 35 JSDF Draft Final Report Appendices, December 2006 - What are the key features of the PRS? Where is it consistent? Where is it inconsistent? - How does the JSDF project contribute (or not) to National PRS targets? How significant is this contribution? Additional questions in relation to representation and participation of the poor that should be asked: · What is the scope and role attributed to civil society organisations, NGOs or CBOs? · To what extent did the poor contribute to the National PRS and other relevant policies for poverty reduction? · To what extent are approaches for strengthened participation of poor and vulnerable groups supported and/or institutionalised? · To what extent are the objectives of the JSDF supported by national policies? Strategic orientation of JSDF projects in Vietnam Poverty Reduction: Vietnam has a strong commitment to poverty reduction, as expressed in the Comprehensive Growth and Poverty Reduction Strategy (CGPRS), the Vietnamese PRSP. The CGPRS emphasises the need to address persistent poverty among ethnic minorities which are the poorest and most vulnerable groups in Vietnam. The project supported by JSDF must be seen as a part of the concerted effort, supported by international donors, to identify innovative approaches for more effective poverty reduction among those groups. Civil society organisations: Political and institutional reform has opened a sizable window of opportunity for the development of civil society organisations. Laws have been put in places which enable registration of non-government organisations and community-based organisations which are increasingly taking over roles on servicing the poor and providing advocacy on important issues that affect poor ethnic minorities, e.g. land issues. Most of those organisations still depend on international funding, and in that respect projects like the JSDF fulfil an important function for strengthening NGO capacities and influence. Participation of the poorest and most vulnerable groups: The drive towards strengthened community participation is an important aspect of reform policies in Vietnam. The Grassroots Democracy Decree requires community consultation and disclosure of information in projects that directly affect communities, such as infrastructure. Practical ways for enabling good-quality, sustained participation of poor communities are still being explored, and this is where support from projects, such as JSDF, becomes crucial. Stakeholder Analysis It is expected that most projects have previously undertaken some kind of stakeholder analysis. This information, where available, can be used as a starting point, but a more in-depth analysis should be done on how roles have actually been worked out in the course of project implementation and how perceptions and views have been shaped by the lessons learnt within the project. 36 JSDF Draft Final Report Appendices, December 2006 The first step is identification of project stakeholders and their specific roles in JSDF include; who contributes; who benefits; who is excluded? The following matrix is recommended to structure the discussion and findings of the stakeholder analysis. Evaluators should focus especially on the role of the stakeholder in the project (3rd column): Example: Stakeholder Analysis NMPRP Vietnam Type Stakeholder Role in Interest in Support Capacities Project Project given to built project: through the (narrative) (narrative) ú Highly project: supportive ú Significant ú Supportive ú Average ú Less ú Less supportive significant (rated after (rated after discussion discussion with with stakeholders) stakeholders) World Bank WB TTL WB SD staff ... Government MPI PMO Provinces Districts Communes ..... NGO CARE ..... Consultants VICA .... Community Leaders Women Very poor households Ethnic minorities .... A key question that needs to be asked both at national and local levels: To what extent has the JSDF design been influenced by the priorities and interests of different stakeholders? 37 JSDF Draft Final Report Appendices, December 2006 Relevant questions from the PEP To what extent were the partner country stakeholders (including target beneficiaries) involved in the project identification and design process? - Which stakeholders were involved? At what stage? What did they contribute? - To what extent were the needs of the poorest and most vulnerable incorporated in the project design? And how? - Were there instances of the project design being altered due to the involvement of stakeholders (& target beneficiaries)? Key informant (stakeholder) interviews A key part of the country visit will be to interview stakeholders (government, donors, NGO/CBOs, etc) using semi-structured interviews. The PEP provides a prompt of useful questions, but it will be up to the evaluator to develop their own checklist for each interview, and decide which questions to ask of whom. The following gives an illustration of areas to cover. Stakeholder perceptions of JSDF projects A major aspect of stakeholder discussions will be to get their views and perceptions on the JSDF project: How are the opportunities provided by the project viewed by different stakeholders? How do they view the relevance and specific "value-added" of the JSDF project? Relevant questions from the PEP What features of the project can be described as innovative, and why? [Q1.4] - What is meant by "innovative" ­ i.e. it has not been tried in this region / country before? It is a new approach altogether? - To what extent have interventions been innovative, rapid, demonstrable and replicable? Are there examples of scaling-up? [Q11.6] How well do JSDF activities achieve their social development outcomes vis-à-vis similar activities funded by other sources? [Q3.1] - What other funding sources are used for similar activities? [Q3a] - To what extent do they focus on the needs of the poorest and most vulnerable? [Q3.2] - To what extent is the JSDF supporting development outcomes that are not being achieved by other initiatives? [Q3b] What has been the value-added of JSDF projects to World Bank operations? [Q5] - What examples are there of JSDF projects having enhanced World Bank best practice/policy for: (a) Building capacity among civil society, local communities and NGOs? [Q5.1] Social development operations and country priorities (as reflected in CAS & PRS)? [Q5.2] 38 JSDF Draft Final Report Appendices, December 2006 Assessing the capacities of implementing agencies JSDF projects are mostly designed to be implemented by civil society organisations (CSOs), non- government organisations (NGOs) or community based organisations (CBOs). In practice, however, the definitions and understanding of these concepts can vary significantly across different countries. In some countries NGOs in the strict sense are hardly existent, but similar functions are performed by other types of organisations: Types of institutions that have implemented JSDF projects at grass-roots levels> ¨ Groups that are located in government bodies with staff on government or project pay-rolls (e.g. project offices) ¨ Groups of individuals that provide paid services to target beneficiaries (e.g. consultants) ¨ Non-Government Organisations that are working on a non-profit base, officially registered and funded from a variety of sources (e.g. local charities, associations) ¨ Community-Based Organisations or other interest groups that are organised by target beneficiaries to provide grass-roots level services (e.g. farmers' organisations, local community organisations) ¨ others During country visits, the evaluators will get a better understanding how civil society organisations have been engaged. Relevant questions from the PEP Have JSDF projects been effective in working with both local and international (incl. Japanese) NGOs? [Q8] - What evidence is there of working relationships with NGOs? What indicators are used to measure such relationships? [Q8.] - What are the views of NGOs regarding their working collaboration and relationship with JSDF activities? [Q8.2] Evaluators will assess the capacities and mandate of those organisations for providing services and support to the poorest and most vulnerable people. 39 JSDF Draft Final Report Appendices, December 2006 Relevant questions from the PEP To what extent have implementing partners been identified on the basis of their grassroots understanding and training skills? [Q1.7] - How were they identified (by whom, and why)? - What is their background and relevant experience? - To what extent do they provide services to poor and vulnerable people as part of their routine work? - What is their mandate for working with poor and vulnerable people? - Do they have staff with a background on local communities or speaking local languages (where applicable)? - How and how often did they interact with target communities in the course of project implementation? To what extent has JSDF activities resulted in greater capacity of the target audience and implementing partners? [Q11.2] - What kind of training has been funded by the JSDF and who has benefited? - What did they learn during the course of project implementation? - The extent to which they strengthened their skills for facilitating participatory processes at grassroots levels, including skills for communication, conflict mediation and motivation of the poorest and most vulnerable groups who are often excluded. A note of caution: It is largely up to the evaluator to decide which stakeholders to meet, and what questions from the PEP to ask in each case. In most contexts, stakeholder interviews should aim to last around 45 minutes up to one hour. The PEP (or the evaluator's own checklist, as derived from the PEP) should thus be used to ensure that meetings stay `on track' as well as to cover a sufficient breadth of questions (although evaluators should always be open to following up relevant and interesting lines of enquiry). Wherever possible and within the constraints on time and resources, key findings should be triangulated with other stakeholder interviews and other sources of information. This is important to avoid an over reliance on the perceptions of just a handful of stakeholders, as well as accusations of methodological bias. Fieldwork in target communities It is anticipated that evaluators will visit a few project sites or communities. Therefore the careful selection of sites is important, and should be done in discussion with the in-country evaluators, implementing partners and other stakeholders. As only a few site visits will be visited, it is important that every attempt is made to fairly capture both the successes and failures of the project ­ and avoid `development tourism' where possible. In summary, various constraints mean that the communities and project sites will be purposively selected, but in each case, the evaluator should be able to justify the selection, ensure reasonable representation (of communities, and within communities), and recognise the likely implications (bias) for the findings. 40 JSDF Draft Final Report Appendices, December 2006 Selecting communities for field work JSDF projects aim at targeting the poorest and most vulnerable groups; this includes a wide range of social groups, depending on the local context. All project documents emphasize the focus on the poorest and most vulnerable groups. Field visits would further look into how these groups have been identified and how service providers facilitate outreach to these groups. Types of target beneficiaries (as they identified in project documents) ú Urban residents and water user associations (Yerevan) ú Poor rural and urban indigenous and Agro-Ecuadorian women and children (Ecuador) ú Poor rural households (India) ú Vulnerable youth (Uganda) ú Children at risk of being trafficked (Benin) ú Marginal fishing communities (Indonesia) ú Squatter communities (Indonesia) ú Poor rural households (Lao) ú Vulnerable children (Armenia) ú Poor ethnic minorities (Vietnam) A common characteristic of all the communities and groups targeted by JSDF projects is that most of them are marginal and difficult to identify and reach. In the case of Vietnam, geographical location is a main factor for determining the status of poverty; most of the impoverished communities living in remote mountain regions are ethnic minorities. It is important that chronically poor people who are often living in marginal locations within better-off provinces and communes are properly identified and targeted. Checklist for selection of field sites ú Identify the poorest provinces and communes and assess the relative status of provinces and communes selected for JSDF. ú Review and assess additional information on the situation of project communes, such as geographical location, access to transportation and settlements of ethnic groups. ú Select 2 out of the 6 JSDF communes, representing different types in terms of poverty and ethnicity; ensure that at least one of the two communes is remote and more difficult to access. ú Within each commune, cover at least 2 natural villages; one should be remote and more difficult to access. Within each community, the evaluators will meet local officials, community leaders and representatives of the poorest and most vulnerable groups that have been identified by the project. Discussions would take place through small meetings and interviews with individual households to examine how the project has changed their asset structure and particularly their access to, use of and satisfaction with services provided by of the project. We will particularly explore the how the 41 JSDF Draft Final Report Appendices, December 2006 project has helped build the social capital of the poorest and most vulnerable groups, and how they have been empowered through the project. Checklist for community visits: ú Briefly visit project staff in province and district to introduce purpose of visit; more detailed discussions where project staff has been directly involved in implementation of JSDF project. ú Interviews with community leaders to understand their perceptions and views on the project ú Identification of different groups within the community according to poverty status, gender, ethnicity and location. ú Visit project facilities (if any) and walk to remote parts of the communities (natural villages) ú Interviews with selected households and/or focus group discussion. Key topics for community visit While this section is not an exhaustive list of topics and questions, it highlights the importance during the community visits to focus on the key issues of impact and sustainability. Reference should also be made to the PEP, which covers in more detail questions on impact, empowerment and capacity building. (a) Assessing direct impacts on target beneficiaries Time for fieldwork will be limited, but evaluators should try to cover broad questions to assess the impact of JSDF projects on the poorest and most vulnerable people. Use of PRA tools is recommended to gain a quick overview of basic poverty issues and to cover the wider ground of JSDF impacts together with project beneficiaries ­ but in some contexts the use of in-depth interviews and focus group discussion may be most appropriate. For more information, see section below on fieldwork methods. Discussions during field visits should be developed from the main questions in the PEP; although the discussions at community level should be more open and explorative to gain a better understanding of changes and impacts from the perspective of the poor and most vulnerable. 42 JSDF Draft Final Report Appendices, December 2006 Relevant questions from the PEP What is the nature and scale of the impact (positive & negative) of JSDF interventions? [Q11] - Who are the "poorest and most vulnerable" people that have been targeted by the JSDF project? - The extent to which the project has helped to build assets of the poorest and most vulnerable people? - Physical capital (housing, drinking water, health and education facilities. etc.) - Natural capital and resources - Human capital (skills) - Financial capital (income, savings) - Social capital - The extent to which this resulted in visible improvements of livelihoods? - Who has benefited from the JSDF-supported activities most? Who has not benefited? - The extent to which the poorest and most vulnerable people have benefited. - The extent to which JSDF interventions been able to avoid elite capture of benefits by richer groups and individuals? [Q11.4] - How do the poorest and most vulnerable view the JSDF's project? [Q11.3] - The extent to which they are satisfied with the services provided by the project - The most important changes that have been brought about by the project - What could have been improved? - The extent to which the project has helped to build capacities among the poorest and most vulnerable people: - The extent to which they have been consulted during project preparation - The extent to they have participated in project implementation - The extent to which they have been empowered in the process - The extent to which they participated in monitoring and evaluating the project (b) Sustainability and other strategic impacts of the project Discussions with stakeholders in the field (as well as in the capital) should also explore perspectives of sustainability and further strategic impacts. Many JSDF projects are meant to perform a "pilot" function and it is expected that most of the methods and approaches trialled within a JSDF-funded perspective will contribute to wider process of change. Relevant questions of the PEP Are the benefits likely to continue for the foreseeable future? - To what extent and how have benefits been sustained among the poorest and most vulnerable? Is this likely to continue? [Q11.5] - The extent to which the JSDF project has helped to change perceptions on poverty and the poor among local officials and government staff; the extent to which it has changed views on participation. - Is there evidence of JSDF supported activities being taken up by others? If so, in what ways? Have these been scaled-up to a larger target audience? Methods for field visits There is a vast range of possible methods that could be used for the community visits. This section gives guidance on an overall approach, BUT it is strongly recommended that methods are selected 43 JSDF Draft Final Report Appendices, December 2006 and adapted to be culturally sensitive. Discussions on this matter should therefore be held with the in-country evaluator, and where appropriate, the project implementers. In general terms, there are two main techniques that are recommended: (i) In-depth interviews with key informants (local officials, community leaders, and selected beneficiaries); and, (ii) Focus group discussions (using visual or PRA tools, where appropriate). There are of course other methods that can be considered, and these are listed at the end. (a) In-depth interviews with key informants In most cases, it will be necessary to undertake in-depth interviews with key informants, such as local officials, community leaders and beneficiaries. Firstly, it will be evaluators should attempt unpick the `official' views of local leaders from those of the wider community, and especially the poorest and most vulnerable. This is a difficult task, but attempts should be made to ask about who benefits (and who doesn't); what problems they face and how they might do it differently next time; and, how might they better reach the poorest and most vulnerable (e.g. female-headed households, the elderly, orphans, refugees, HIV/AIDS suffers, etc). Key questions might include: · Who benefits? · Who does not benefit? And, why? · What would you do differently next time? And secondly, it will be important to understand the views of direct beneficiaries about the services being provided. For example, in Ecuador the JSDF project is providing legal services while in Armenia it is water supply. In such cases, questions should be asked about the access, use and satisfaction with the services being provided ­ and how they compare before/after, and the alternatives (e.g. government provision). For instance, access issues might include questions about who is (and who is not) able to obtain services, issues of cost and distance (including means of transport, such as to towns and local markets). (b) Focus group discussions (using visual tools) Focus group discussions are an important (and sometimes more efficient) way to discover the views of particular socio-economic groups and beneficiaries. For example, separate focus groups may be held with men and women, or with older women and younger women, teachers and pupils, and so on. The use of visual tools in such contexts can also help to stimulate and provide a focus for the discussion. Getting the group to draw a picture (say using marker pens on flipchart paper) is a useful tool to illustrate change in the community over the past five years. A series of questions can then be asked, and this can used to stimulate in-depth discussions about the impact of the project. Key questions might include: · What has changed? · How did these changes come about? And, who made them happen? 44 JSDF Draft Final Report Appendices, December 2006 · What did not change? And, why? · How important are these changes? (e.g. "Top Three") · What would you do differently next time? A similar discussion could be undertaken around a timeline of key events in the life of the project: What are the major events? What has changed? Etc.... (c) Other participatory methods (to consider) There is a range of other participatory methods to consider and adapt, in discussion with the in- country evaluator. A list of possible methods is given below: Methods for participatory project evaluation Mapping § How many households have participated in the project activities § How many people have used certain new services § Reductions or increases in disease patterns § The percentage of girls enrolled in school Diagramming § Venn diagram to illustrate which projects have been most useful § Venn diagram to illustrate new collaborations between organizations as a result of the project § Flow diagrams to show the progress of the project implementation § Flow diagrams to show how the project has affected the community in other ways Ranking/Scoring § Using matrix scoring to compare successful projects with unsuccessful projects according to various criteria (e.g. number of people who contributed labour, amount of external resources obtained, number of times the community met to work on the project) Semi Structured Interview § Discussions with community members about their perceptions of the project's success § Discussions about problems with project implementation § Discussions about the direct and indirect effects of the project on the lives of community members Pie Charts § Illustration of estimated incomes or expenses of households § Illustration of the increases or decreases in the percentage of people engaging in certain kinds of behaviours 45 JSDF Draft Final Report Appendices, December 2006 Project Evaluation Pro forma (PEP) Brief instructions: Complete right-hand column with a write-up of findings (and evidence) from the literature review, stakeholder interviews, and project site visits. Complete the "Project Synopsis" section before your country visit, and the "Summary of Findings" section at the end. Use succinct paragraphs and bullet points, and include references to your notes of individual meetings as well as documented sources. Then rate each key question, numbered from 1 to 19 on a four-point scale from `high' to `low'. Insert a one- sentence justification for each score given, in the space provided. You should aim for an indicative page length of 15-20 pages, providing more depth for the priority questions as highlighted (blue for stakeholder interviews; yellow for project site visits). PROJECT SYNOPSIS Country: Lead Evaluator: Number: TF........ National Expert: Name of Project: National Expert: Start/ End dates of visit: Briefly describe the historical background to this project / grant ... Instructions: - Timeline of significant events (what? when? by whom? why?) Complete before country visit - How can the project be characterised (strands/themes/groupings)? Briefly describe the overall design of the project... Instructions: 1) What are the objectives of the project? Complete before country visit 2) What is the intervention logic of the project? What problem(s) is the project attempting to address, why and how? 3) What are the objectives of each of the project components? 4) - Who are the project implementers, partners, and target audience(s)? 46 JSDF Draft Final Report Appendices, December 2006 RELEVANCE (1) Alignment to national strategies: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent are the JSDF project's objectives consistent (aligned) with the country's development strategy (esp. the PRS)? [Q1.2] Justification for rating (one sentence max.): 1) Briefly describe the national policy context (relevant to this Interview: project)... 2) What are the key features of the PRS? To what extent is it National stakeholders (donors, government, other) owned and implemented as the development strategy of the country? Project implementers (TTL, NGOs/government) 3) What other key national strategies are relevant to the work of JSDF in the country? 4) How does the JSDF project contribute (or not) to National PRS targets? How significant is this contribution? Are there missed opportunities? (2) Alignment to JSDF objectives: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent are the project's objectives consistent (in line) with the objectives and priorities of the Fund? [adapted Q1] Justification for rating (one sentence max.): 1) To which objectives (strands/themes/groupings) of the Fund Interview: does the project contribute? Project implementers (TTL, NGOs/government) 2) - How important is this project to the Fund achieving its overall objectives? Why? 47 JSDF Draft Final Report Appendices, December 2006 QUALITY OF DESIGN (3) Intervention logic: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent does the JSDF project have a clear & consistent intervention logic that addresses identifiable problems of the Justification for rating (one sentence max.): poorest and most vulnerable? [Q1.3] 1) What are the objectives of the project (list)? Instructions: 2) Is the intervention logic clearly articulated? Is it outcome- Summarise findings from Document Review Pro forma focused? 3) Have indicators been stated? To what extent are they (+ Add insights from interviews) measurable and SMART? [Q1.5] 4) - In the design, are there adequate arrangements for monitoring? Are roles and responsibilities clearly set out? [Q1.6] (4) Innovation: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent is the project innovative? [Q1.4] Justification for rating (one sentence max.): 1) To what extent have interventions been INNOVATIVE, Interview: demonstrable and replicable? National stakeholders (donors, government, other) 2) Give examples of innovative practices on this project 3) - In the context of what is meant by "innovative", has not Project implementers (TTL, NGOs/government) been tried in this region/country before? Is it a new approach altogether? 48 JSDF Draft Final Report Appendices, December 2006 PREPARATION PROCESS (5) Implementing partners: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent have implementing partners been appropriately identified (i.e. on the basis of their grassroots understanding and Justification for rating (one sentence max.): training skills)? [Q1.7] 1) How were they identified (by whom, and why)? What criteria and process was used to identify partners? 2) What are their background, reputation and relevant experience? 3) To what extent do they provide services to poor and vulnerable people as part of their routine work? 4) Do they have staff with a background on community Interview: development or speaking local languages (where applicable)? Project implementers (TTL, NGOs/government) 5) Do they have gender specialists / experts. What is the gender balance in the organisation 6) - How and how often did they interact with target communities in the course of project implementation? At the start, during implementation. What will happen to the support services after project completion? (6) Inclusiveness & targeting: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent were the partner country stakeholders (including target beneficiaries) involved in the project identification and design Justification for rating (one sentence max.): process? 1) Which stakeholders were involved? At what stage? What did they contribute? Interview: 2) - To what extent were the needs of the poorest and most vulnerable incorporated in the project design? And how? National stakeholders (donors, government, other) 3) Were there instances of the project design being altered due to the involvement of stakeholders (& target beneficiaries)? Project implementers (TTL, NGOs/government) 4) - To what extent are approaches for strengthened Beneficiaries participation of poor and vulnerable groups supported and/or institutionalised? 49 JSDF Draft Final Report Appendices, December 2006 (7) Use of Seed Fund: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent is the seed fund contributing to effective and a more participative project preparation? [Q2a] Justification for rating (one sentence max.): 1) How can this be better achieved? Are there lessons to be learnt? Interview: 2) - How does participation in JSDF project preparation compare with projects without seed funds, and those without Project implementers (TTL, NGOs/government) any trust funds? [Q2.2] COMPLEMENTARITY & VALUE ADDED (8) Complementarity: 1 ­ High 2 - Substantial 3 - Modest 4 - Low (a) How well do JSDF activities achieve their social development outcomes vis-à-vis similar activities funded by other sources (not Justification for rating (one sentence max.): World Bank)? [Q3.1] 1) What other funding sources are used for similar activities? [Q3a]. 2) To what extent do they focus on the needs of the poorest Interview: and most vulnerable? [Q3.2] National stakeholders (donors, government, other) 3) - To what extent is the JSDF supporting development outcomes that are not being achieved by other initiatives? Project implementers (TTL, NGOs/government) [Q3b], i.e. what other similar projects are operating in the region? (b) To what extent is complementarity between the JSDF project 1 ­ High 2 - Substantial 3 - Modest 4 - Low and the main IDA / World Bank projects being achieved? [Q4.1] Justification for rating (one sentence max.): 1) What mechanisms are in place to ensure greater complementarity 2) How can the complementarity between JSDF projects and Interview: World Bank projects be improved? [Q4, Q4.2] 3) - What examples are there of projects being scaled-up? Project implementers (TTL, NGOs/government) [Q4.4] 4) 50 JSDF Draft Final Report Appendices, December 2006 (9) Value Added: 1 ­ High 2 - Substantial 3 - Modest 4 - Low Assess the value-added of JSDF projects to World Bank operations? [Q5] Justification for rating (one sentence max.): - What examples are there of JSDF projects having enhanced World Bank best practice/policy for: (b) Building capacity among civil society, local communities and Interview: NGOs? [Q5.1] Project implementers (TTL, NGOs/government) (c) Social development operations and country priorities (as reflected in CAS & PRS)? [Q5.2] EFFICIENCY (10) Internal management arrangements: 1 ­ High 2 - Substantial 3 - Modest 4 - Low How efficiently are JSDF grants managed within the Bank? [Q6a] Justification for rating (one sentence max.): 1) What is the project cycle (application ­ approval ­ implementation)? 2) From the country perspective, do the management systems work efficiently? Interview: 3) Has the implementing partner the resources to manage the achievement of the project's objectives (staff numbers, Project implementers (TTL, NGOs/government) skills, per diems, vehicles) 4) - What are the roles of the country teams (and other networks)? How effective are they? What could be improved? [Q6b, Q6.2] 5) Are outputs achieved within the expected cost and time? Are there major cost- or time-overruns or budget revisions? [Q7.2] 6) - Have alternative models of management have been Interview: considered /implemented? Explain... [Q7.3] Project implementers (TTL, NGOs/government) 51 JSDF Draft Final Report Appendices, December 2006 (11) Monitoring and Evaluation: 1 ­ High 2 - Substantial 3 - Modest 4 - Low How effective is the monitoring system at country level? Justification for rating (one sentence max.): 1) Describe the main features of the M&E system... What type of evaluation is carried out? 2) How effectively does it enable the project to collect relevant data on activities, outputs, outcomes and impact? [Q9.1] 3) What evaluation resources are available (staff, computers etc) 4) What is the standard of reporting and quality of information Instructions: 5) To what extent does the monitoring system provide Summarise findings from Document Review Proforma evidence that JSDF-supported activities are achieving the Fund's objectives? [Q1] (+ Add insights from interviews) 6) How are the outputs of M&E procedures fed back into decision-making processes and management procedures? [Q9.3] 7) - How effectively have the results of innovative practices been - disseminated? To whom? Provide examples... [Q9.4] EFFECTIVENESS (12) Achievement of outputs and outcomes: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent has the JSDF project achieved its stated objectives (outcomes)? Justification for rating (one sentence max.): 1) To what extent have objectives, outcomes and targets been achieved? What is the quality of this data/information? Summarise findings from Document Review Proforma [Q10.2] 2) What are the factors contributing to success? What are the factors contributing to failure? [Q10] Interview: 3) To what extent are these factors attributable to the project National stakeholders (donors, government, other) (design, management, monitoring, implementation)? [Q10.3] Project implementers (TTL, NGOs/government) 4) - To what extent is effectiveness be attributable to external factors? (i.e. social, economic and political context; donor- Beneficiaries government relations) [Q10.4] 52 JSDF Draft Final Report Appendices, December 2006 (13) Partnerships & collaboration: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent has the JSDF project effectively worked with both local and international (incl. Japanese) NGOs? [Q8] Justification for rating (one sentence max.): - What evidence is there of working relationships with NGOs? What Interview: indicators are used to measure such relationships? [Q8.1] National stakeholders (donors, government, other) - What are the views of NGOs regarding their working collaboration and relationship with JSDF activities? [Q8.2] Project implementers (TTL, NGOs/government) (14) Visibility: 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent do JSDF projects achieve visibility for Japan? Justification for rating (one sentence max.): 1) Are stakeholders aware that the project is Japan grant Interview: funded? If so, which ones? [Q11.7] National stakeholders (donors, government, other) 2) How did visibility manifest itself? 3) - Were the Japanese satisfied with the level of visibility (for Project implementers (TTL, NGOs/government) Japan) on this project? 53 JSDF Draft Final Report Appendices, December 2006 IMPACT & SUSTAINABILITY (15) Overall impact on the poorest: 1 ­ High 2 - Substantial 3 - Modest 4 - Low What is the extent (nature and scale) of the impact (positive & negative) of JSDF interventions? [Q11] Justification for rating (one sentence max.): 1) Who are the "poorest and most vulnerable" that have been targeted by the JSDF project? 2) How many beneficiaries were targeted? How many were actually reached? 3) What proportion of villages / communities in the town / district / region are being covered by the JSDF project? 4) What are the tangible improvements in people's lives? What have been the most important changes brought about by the project? Interview: (a) Physical capital (housing, drinking water, health Project implementers (TTL, NGOs/government) and education facilities. etc.) Beneficiaries (b) Natural capital and resources (c) Human capital (skills) (d) Financial capital (income, savings) (e) Social capital 5) - To what extent are the poorest and most vulnerable satisfied with the services provided by the project? How do they view the JSDF's project? [Q11.3] (16) Impact: Winners & Losers 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent has the JSDF interventions been able to avoid elite capture of benefits by richer groups and individuals? [Q11.4] Justification for rating (one sentence max.): 1) How does the project seek to reach the poor and vulnerable, and overcome political-economic constraints? Interview: 2) Who has benefited from the JSDF-supported activities most? Who has not benefited? Why? Project implementers (TTL, NGOs/government) 3) - Are there examples of the less poor being included in the Beneficiaries project and if so, what is the justification for this approach 54 JSDF Draft Final Report Appendices, December 2006 (17) Impact: Capacity building 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent has JSDF activities helped to build capacities among the poorest and most vulnerable people? [Q11.2] Justification for rating (one sentence max.): 1) - What kind of training has been funded by the JSDF project and who has benefited? Who has not benefited, and why? 2) What did they learn during the course of project implementation? 3) To what extent has these activities strengthened the capacities of the poorest and most vulnerable? Types of skills: Interview: (a) Practical skills (IT, farming practices, etc) Project implementers (TTL, NGOs/government) (b) Facilitating participatory processes at grassroots levels (e.g. Beneficiaries communication, conflict mediation, and facilitating the inclusion of those most often excluded) (18) Impact: Empowerment 1 ­ High 2 - Substantial 3 - Modest 4 - Low To what extent has the JSDF project intervention empowered its target audience? Justification for rating (one sentence max.): 1) To what extent have the stakeholders and / or beneficiaries been consulted during project preparation? 2) To what extent have they participated in project implementation? And, how have they been empowered (involved in decision-making, local contracting, supervision...) during the process? Interview: 3) - To what extent do beneficiaries participate in monitoring and evaluation of the project? Project implementers (TTL, NGOs/government) Beneficiaries 55 JSDF Draft Final Report Appendices, December 2006 (19) Sustained benefits (& scaling-up): 1 ­ High 2 - Substantial 3 - Modest 4 - Low Are the benefits likely to continue for the foreseeable future? Justification for rating (one sentence max.): 1) To what extent and how have benefits been sustained (financial, institutional) among the poorest and most vulnerable? Is this likely to continue? [Q11.5] 2) The extent to which the JSDF project has helped to change Interview: perceptions on poverty and the poor among local officials and government staff? And, the extent to which it has Project implementers (TTL, NGOs/government) changed views on community participation? Beneficiaries 3) - Is there evidence of JSDF supported activities being taken up by others? If so, in what ways? Have these been scaled-up to a larger target audience? SUMMARY OF FINDINGS What are the key findings in terms of: § Relevance (to the national policy context & needs of the target audience) Instructions: § Efficiency (of management arrangements & working with NGO partners) Complete at end of Country Visit § Effectiveness (and achievement of outputs and outcomes) § Impact (on the poorest & most vulnerable) Are there specific lessons learned by the grant activities, in terms of: [Q12] § Technical design? § The most appropriate kinds of project? Instructions: § Reaching target beneficiaries? Complete at end of Country Visit § Governance or management arrangements? - Should these be widely disseminated? How? 56 JSDF Draft Final Report Appendices, December 2006 What preliminary recommendations could be made to improving JSDF-supported activities? Instructions: Complete at end of Country Visit APPENDICES: List of Key Stakeholders interviewed List of documents Itinerary 57 JSDF Draft Final Report Appendices, December 2006 Key: [Q1.3] = The number in the square brackets refers the question number in the Evaluation Framework. Priority questions for interviews with stakeholders (implementing partners, donors, government, etc) Priority questions for fieldwork in communities (project sites) GUIDANCE ON INTERPRETATION OF RATING SCALES Guidance on interpretation of rating scales: PEP rating scale Percentages Description 1 - High 75-100% High quality or exceptional, few weaknesses 2 - Substantial 50-75% Good quality, with some weaknesses 3 - Modest 25-50% Okay quality, with several weaknesses 4 - Low 0-25% Low quality, with many weaknesses Additional notes: · `Modest' is used to denote, "relatively moderate", rather than "unassuming in the estimation of one's abilities" (Oxford English Dictionary). · The scale is subjective and relative, and largely based on the evaluator's own judgement and experience. A one-sentence justification should be included for each rating, as this will help with interpretation and checking consistency. · It is encouraged that evaluators share experiences on ratings, as this will help achieve a degree of normalisation (between different evaluators). A useful mechanism for this is the Aide Memoire. 58 JSDF Draft Final Report Appendices, December 2006 C.5. Web Based Survey To capture opinions across a broader spectrum of TTLs other than those who were interviewed face to face in the course of the Washington visits and project visits, a web based questionnaire was designed and made available to all TTLs who had managed a JSDF grant. The questionnaire was administered electronically, using a proprietary web-based data collection tool that he evaluators had used extensively and successfully in other evaluations. The approach enabled rapid processing and analysis and is relatively unobtrusive for the staff being sampled. Considering the TTLs work load and the desire for a high response rate, the evaluation team kept the survey short and to the point. The survey was open for three weeks during the month of September, to account for the fact that many TTLs spend time out of the office on field missions. The original list received from WB had 98 addresses for TTLs. 7 of these addresses did not work and 1 TTL was on maternity leave so total number of TTLs who received the survey was 90. Fifty-three TTLs responded, giving a high response rate of 59%. As the evaluation team hoped, the respondents cover a broad range of experience of JSDF; § Respondents represented experience covering the 7 years of JSDF, since it started in 2000. § Most respondents had managed 1 grant (50%) although 35% had managed 2 and a small number (3 TTLs) had experience of managing 4 JSDF grants1. § Most had managed a grant from beginning to end (56%) and 74% had experience of the application process. § 44% of respondents had experience of unsuccessfully applying for a grant. § 17% (9) had experience of a seed fund. The Survey enabled the evaluation to capture the insight and experience of the World Bank Task Team Leaders (TTLs) who have experience of applying for and/or managing JSDF grants across the entire portfolio of projects. Including TTLs who are managing grants which are less than 80% dispersed and therefore more recently approved would allow the evaluators to assess whether the type of projects being funded, and the approaches being used have shifted and what impacts any changes in policies and procedures have had, for instance on the establishment of Seed Funds and the recent simplification exercise. 1 If a question related to one project only, we asked respondents who had managed more than one grant, to consider the most recently completed project. 59 JSDF Draft Final Report Appendices, December 2006 D. Results D.1. Document Review Result Tables from analysis of portfolio of 43 projects Table A: Analysis of grants by World Bank Defined sectors/themes Full Portfolio List of 43 grants Total Total WB Sectors / Themes % Total $ % % Total $ % No No Multi-sector - Social Protection and Risk Mgmt. 14 12% 18,835,447 14% 9 21% 10,219,700 22% Urban Development & Infrastructure 4 3% 5,703,800 4% 4 9% 5,703,800 12% Rural Development 8 7% 10,513,028 8% 4 9% 4,178,048 9% Health and Other Social Services 7 6% 7,828,679 6% 2 5% 3,105,500 7% Agriculture Fishing and Forestry - Rural Development 1 1% 1,610,000 1% 2 5% 3,005,000 6% Public Sector Governance 1 1% 2,825,075 2% 1 2% 2,825,075 6% Law and justice 6 5% 6,033,632 4% 3 7% 2,178,684 5% Multi-sector - Rural and Urban Development, and Infrastructure 8 7% 8,039,805 6% 2 5% 2,148,700 5% Education 8 7% 8,191,802 6% 2 5% 2,107,475 5% Multi-sector - Social Development, Gender and Inclusion 17 15% 24,170,581 17% 2 5% 2,044,435 4% Education -Primary Education 1 1% 1,827,424 1% 1 2% 1,827,424 4% Human Development (Education) 1 1% 1,000,000 1% 1 2% 1,000,000 2% Human development/ Social protection 1 1% 961,000 1% 1 2% 961,000 2% Human Development/ Social Protection 1 1% 950,000 1% 1 2% 950,000 2% Education/ Social Development 1 1% 800,000 1% 1 2% 800,000 2% Social Development 2 2% 1,693,980 1% 1 2% 737,420 2% Health 3 3% 1,330,000 1% 1 2% 630,000 1% Multi-sector - Social Development, Gender and Inclusion/Education 1 1% 536,923 0% 1 2% 536,923 1% Urban Development/ Social Protection 1 1% 443,000 0% 1 2% 443,000 1% Human Development 1 1% 412,000 0% 1 2% 412,000 1% 60 JSDF Draft Final Report Appendices, December 2006 Full Portfolio List of 43 grants Total Total WB Sectors / Themes % Total $ % % Total $ % No No Health/Human Development 1 1% 400,000 0% 1 2% 400,000 1% Human Development (Health Services) 1 1% 338,300 0% 1 2% 338,300 1% Agricultural Fishing and Forestry 1 1% 1,128,200 1% Agriculture, Fishing and Forestry 6 5% 5,414,608 4% Agriculture, Fishing and Forestry - Social Development 1 1% 1,952,500 1% Agriculture, Fishing and Forestry (Rural Development) 3 3% 4,953,000 4% Education/ Human Development 1 1% 2,000,000 1% Financial and Private Sector Development 1 1% 1,989,000 1% Health and Other Social Services - Social Protection 1 1% 1,696,700 1% Health and other Social Services, Social Protection Risk Management 1 1% 543,000 0% Law and Justice and Public Administration 1 1% 954,460 1% Multi-sector (Human Development) 1 1% 758,500 1% Multi-sector - Rural and Urban Development, and 1 1% 742,800 1% Infrastructure/Education Multi-sector - Social Development, Gender and Inclusion/Rural 1 1% 1,209,636 1% Development Transportation - Rural Development, Infrastructure 1 1% 935,200 1% Water & Sanitation 6 5% 9,015,425 7% Water and Sanitation (Human Development) 1 1% 891,000 1% Total 116 138,628,505 43 46,552,484.30 61 JSDF Draft Final Report Appendices, December 2006 Table B: Condensing the list of Sectors Condensed List of Original List of WB Sectors/Themes Sectors/Themes 1 Education -Primary Education 2 Education 3 Education/ Human Development 1 Education 4 Education/ Social Development 5 Human Development (Education) 6 Health 7 Health and Other Social Services 8 Health and Other Social Services - Social Protection 2 Health 9 Health and other Social Services, Social Protection Risk Management 10 Health/Human Development 11 Human Development (Health Services) 12 Human Devel./ Social Protection 13 Human develop., social protection 3 Human Development 14 Human Development 15 Law and justice 4 Law and justice 16 Law and Justice and Public Administration 17 Multi-sector - Social Dev. Gender and Inclusion/Education 18 Multi-sector - Rural and Urban Development, and Infrastructure 19 Multi-sector - Social Dev. Gender and Inclusion 20 Multi-sector - Social Protection and Risk Mgmt. 5 Multi Sectoral 21 Multi-sector (Human Development) Multi-sector - Rural and Urban Development, and 22 Infrastructure/Education 23 Multi-sector - Social Dev. Gender and Inclusion/Rural Development 24 Agriculture, Fishing and Forestry 6 Natural Resources 25 Financial and Private Sector Development 7 Private Sector Public Sector 26 Public Sector Governance 8 Governance 27 Agricultural Fishing and Forestry 28 Agriculture Fishing and Forestry - Rural Development 29 Agriculture, Fishing and Forestry (Rural Development) 9 Rural Development 30 Rural Development 31 Transportation - Rural Dev. Infrastructure 32 Social Development 10 Social Development 33 Agriculture, Fishing and Forestry - Social Dev. 34 Urban Development/ Social Protection 11 Urban Development 35 Urban Development & Infrastructure 36 Water & Sanitation 12 Water & Sanitation 37 Water and Sanitation (Human Development) 62 JSDF Draft Final Report Appendices, December 2006 Table C: Analysis of Grants across condensed list of Sectors/Themes Full Portfolio List of 43 grants Total Total Condensed Themes % Total $ % % Total $ % No No Social Development 3 3% 3,646,480 3% 1 2% 737,420 2% Law & Justice 7 6% 6,988,092 5% 3 7% 2,178,684 5% Human Development 4 3% 3,081,500 2% 3 7% 2,323,000 5% Public Sector Governance 1 1% 2,825,075 2% 1 2% 2,825,075 6% Health 14 12% 12,136,679 9% 5 12% 4,473,800 10% Education 12 10% 13,819,226 10% 5 12% 5,734,899 12% Urban Development 5 4% 6,146,800 4% 5 12% 6,146,800 13% Rural Development 13 11% 18,011,228 13% 6 14% 7,183,048 15% Multi Sector 42 36% 53,535,192 39% 14 33% 14,949,758 32% Natural Resources 7 6% 6,542,808 5% Private Sector 1 1% 1,989,000 1% Water & Sanitation 7 6% 9,906,425 7% Total 116 138,628,505 43 46,552,484 63 JSDF Draft Final Report Appendices, December 2006 Table D: Analysis of grants by country Full Portfolio List of 43 grants Total Total Recipient Country % Total $ % % Total $ % No No Indonesia 9 8% 4,191,621 10% 3 7% 6,111,840 13% Philippines 9 8% 1,235,392 8% 4 9% 5,245,843 11% China 6 5% 7,148,000 5% 4 9% 4,683,000 10% Cambodia 5 4% 6,388,615 5% 3 7% 3,859,924 8% Armenia 2 2% 2,876,000 2% 2 5% 2,931,000 6% Colombia 2 2% 3,650,075 3% 1 2% 2,825,075 6% Nicaragua 2 2% 2,254,800 2% 2 5% 2,254,800 5% Lao PDR 3 3% 2,773,175 2% 2 5% 1,884,125 4% Moldova 3 3% 3,506,640 3% 2 5% 1,815,500 4% Ecuador 2 2% 2,191,850 2% 1 2% 1,780,000 4% Uganda 1 1% 1,611,000 1% 1 2% 1,661,000 4% Bulgaria 3 3% 1,963,000 1% 2 5% 1,470,000 3% Vietnam 6 5% 8,698,800 6% 1 2% 1,395,000 3% Kyrgyz Republic 1 1% 1,149,500 1% 1 2% 1,192,300 3% Jordan 2 2% 1,185,860 1% 2 5% 1,185,860 3% Sri Lanka 4 3% 3,766,649 3% 2 5% 957,684 2% India 7 6% 6,592,048 5% 1 2% 956,400 2% Benin 1 1% 740,000 1% 1 2% 758,000 2% CAR 1 1% 600,000 0% 1 2% 630,000 1% Tajikistan 3 3% 4,700,400 3% 1 2% 575,000 1% Pakistan 4 3% 3,270,619 2% 1 2% 536,923 1% Guinea 2 2% 974,585 1% 1 2% 486,975 1% Mongolia 3 3% 3,635,890 3% 1 2% 443,000 1% Yemen, Republic of 2 2% 1,461,635 1% 1 2% 383,435 1% Samoa 1 1% 338,300 0% 1 2% 338,300 1% Albania 1 1% 191,500 0% 1 2% 191,500 0% Cameroon 1 1% 550,180 0% Cape Verde 1 1% 909,160 1% Central America 1 1% 900,000 1% Chad 1 1% 837,780 1% Congo, Republic of 1 1% 708,400 1% East Asia and Pacific 1 1% 1,960,000 1% Egypt, Arab Republic 1 1% 1,877,487 1% of Ethiopia 2 2% 3,738,310 3% Georgia 2 2% 1,857,275 1% Honduras 1 1% 1,984,500 1% Mali 2 2% 1,366,725 1% Mauritania 1 1% 863,520 1% Nigeria 1 1% 1,499,929 1% Papua New Guinea 1 1% 468,300 0% Paraguay 1 1% 866,590 1% 64 JSDF Draft Final Report Appendices, December 2006 Full Portfolio List of 43 grants Total Total Recipient Country % Total $ % % Total $ % No No Peru 3 3% 2,752,500 2% Russian Federation 1 1% 1,252,700 1% Sao Tome and 1 1% 649,450 0% Principe Senegal 2 2% 2,693,356 2% Sierra Leone 1 1% 1,888,700 1% Tanzania 1 1% 1,818,173 1% Turkey 2 2% 3,447,500 3% Ukraine 1 1% 1,711,500 1% Uzbekistan 1 1% 1,694,419 1% Total 116 135,392,408 43 46,552,484 Table E: Analysis of grants across regions Full Portfolio List of 43 grants Total Total Region % Total $ % % Total $ % No No SAR 15 13% 13,629,316 10% 4 9% 2,451,007 5% MENA 7 6% 7,972,482 6% 3 7% 1,569,295 3% LAC 12 10% 14,600,315 11% 4 9% 6,859,875 15% ECA 18 16% 20,902,934 15% 9 21% 8,175,300 18% EAP 45 39% 57,388,273 42% 19 44% 23,961,032 51% AFR 19 16% 20,899,088 15% 4 9% 3,535,975 8% Total 116 135,392,408 43 46,552,484 Table F: Analysis of grants by Recipient and Implementing Agency Full Portfolio List of 43 grants Total Total Recipient Agency % Total $ % % Total $ % No No Government 78 67% 92,064,061 68% 31 72% 30,943,575 66% Joint (Govt & NGO) 0 0% - 0% 0 0% - 0% NGO 33 28% 39,344,912 29% 10 23% 13,135,474 28% (blank) 5 4% 3,983,435 3% 2 5% 2,473,435 5% Total 116 135,392,408 43 46,552,484 Implementing Agency Government 71 61% 84,320,286 62% 28 65% 28,621,255 61% Joint (Govt & NGO) 2 2% 3,771,450 3% 0 0% - 0% NGO 38 33% 43,317,237 32% 13 30% 15,457,794 33% (blank) 5 4% 3,983,435 3% 2 5% 2,473,435 5% Total 116 135,392,408 43 46,552,484 65 JSDF Draft Final Report Appendices, December 2006 Tables showing results from JSDF Documentary Review Table 1: Relevance ­ Evaluators Rating Articulation of Relevance of Objectives Objectives Rank Total % Total % 1 16 37% 27 63% 2 15 35% 11 26% 3 10 23% 4 9% 4 2 5% 1 2% Total 43 43 Table 2: Design ­ Evaluators Rating Articulation of Intervention Identification of Participation in components Logic Risks design Rank Total % Total % Total % Total % 1 23 53% 3 7% 6 14% 12 28% 2 12 28% 20 47% 13 30% 6 14% 3 7 16% 14 33% 15 35% 10 23% 4 1 2% 6 14% 7 16% 13 30% No Rate 3 7% 2 5% Total 43 43 43 43 Table 3.1: Efficiency ­ WB Ratings Achievement of Performance of Performance of Implementation Outputs outcomes WB recipient Rank Total % Total % Total % Total % Total % Highly Satisfactory 2 14% 3 21% 3 21% 5 12% 6 43% Moderately Satisfactory 0 0 0 0 0 0 1 2% 0 0% Satisfactory 7 50% 8 57% 5 36% 28 65% 6 43% Satisfactory & highly 0 0% 0 0% 1 7% 0 0% 0 0% satisfactory Un- 0 0% 0 0% 0 0% 5 12% 0 0% satisfactory Not Specified 3 21% 2 14% 0 0% 4 9% 2 14% No rate 2 14% 1 7% 1 7% 0 0% 0 0% Total 14 14 14 14 14 No ICM 29 66 JSDF Draft Final Report Appendices, December 2006 Table 3.2: Efficiency ­ WB Ratings Financial Counterpart Implementation Project Procurement management funding Progress performance performance performance performance Rank Total % Total % Total % Total % Total % Highly Satisfactory 5 12% 5 12% 4 9% 3 7% 5 12% Moderately Satisfactory 1 2% 1 2% 1 2% 1 2% 4 9% Satisfactory 27 63% 20 47% 34 79% 23 53% 33 77% Un- 4 9% 2 5% 2 5% 0 0% 1 2% satisfactory Not Specified 6 14% 15 35% 2 5% 16 37% 0 0% Total 43 43 43 43 43 Table 4.1: M&E ­ Component in Project Design Total % No 15 35% Yes 27 63% Not Specified 1 2% Total 43 Table 4.2: M&E ­ Articulation of M&E ­ Evaluators Rating Rank Total % 1 7 16% 2 13 30% 3 16 37% 4 6 14% No rate 1 2% Total 43 67 JSDF Draft Final Report Appendices, December 2006 Table 4.3: M&E ­ Articulation of KPIs ­ Evaluators Rating Articulation of Articulation of KPIs (GRM) KPIs (AF) Rank Total % Total % 1 1 2% 2 5% 2 12 28% 3 7% 3 10 23% 4 9% 4 11 26% 9 21% No rate 9 21% 25 58% Total 43 43 Table 4.5: M&E ­ WB Ratings Rank Total % Highly Satisfactory 2 5% Not Specified 13 30% Satisfactory 25 58% Un- satisfactory 3 7% Total 43 Table 5.1: Effectiveness ­ WB Ratings Achievement of Progress of Grant GDO Outcomes Objectives Rank Total % Total % Total % Highly Satisfactory 5 36% 9 21% 10 23% Moderately Satisfactory 0 0 1 2% 1 2% Satisfactory 4 29% 25 58% 24 56% Un - satisfactory 0 0 1 2% 3 7% Highly Un- satisfactory 0 0 0 0% 0 0% Not Specified 5 36% 7 16% 5 12% Total 14 43 43 No ICM 29 68 JSDF Draft Final Report Appendices, December 2006 Table 5.2: Effectiveness ­ Evaluators Rating Rating Effectiveness at participation/ targeting poorest/ Effectiveness at Innovation rapid benefits empowerment vulnerable Rank Total % Total % Total % Total % 1 16 37% 11 26% 14 33% 15 35% 2 14 33% 12 28% 12 28% 9 21% 3 8 19% 8 19% 15 35% 6 14% 4 5 12% 4 9% 2 5% 4 9% No rate 8 19% 9 21% Total 43 43 43 43 69 JSDF Draft Final Report Appendices, December 2006 Table 6.1: Impact ­ WB Rating Achievement of Impact of Achievement of Achievement of Impact of CB Outcomes - Capacity Impact Objectives Outcomes (IFM) relevance Building Rank Total % Total % Total % Total % Total % Total % Highly Satisfactory 5 36% 4 29% 3 21% 3 21% 0 0 3 21% Not Specified 2 14% 8 57% 7 50% 8 57% 14 100% 8 57% Satisfactory 7 50% 2 14% 4 29% 3 21% 0 0 3 21% Total 14 14 14 14 14 14 No ICM 29 70 JSDF Draft Final Report Appendices, December 2006 Table 6.2: Impact of Capacity Building ­ Evaluators Rating Rank Total % 1 15 35% 2 10 23% 3 7 16% 4 4 9% No rate 7 16% Total 43 Table 6.3: Sustainability ­ WB Rating WB rating for impact - sustainability Rank Total % Highly Satisfactory 3 21% Not Specified 9 64% Satisfactory 2 14% Total 14 Table 6.4: Sustainability ­ Evaluators rating Rate Total % 1 4 9% 2 16 37% 3 13 30% 4 5 12% No rate 5 12% Total 43 Table 7: Japanese Involvement Agency NGOs consultation Total % Total % No 27 63% 20 47% Yes 12 28% 18 42% Not Specified 4 9% 5 12% Total 43 43 71 D2. Field Visit D2.1 Scores from Project Evaluation Proformas NB - Where no score was given, the cell is left blank. Reasons for no score given in PEP narrative. Indonesia Squatter Indonesia Armenia Armenia Ecuador Vietnam Uganda High Substantial Modest Low Av Av Benin India Laos Country No. Min Max Fishing Water Youth No % No % No % No % Mean Mode Child Child Law VIP ICT Project Relevance Relevance 1 Alignment to national strategies 1 1 2 2 2 1 1 1 1 1 7 70% 3 30% 0 0% 0 0% 10 1.8182 1 1 2 2 Alignment to JSDF Objectives 1 1 1 2 1 1 1 2 1 1 8 80% 2 20% 0 0% 0 0% 10 1.2 1 1 2 Quality of design Quality of design 3 Intervention Logic 2 2 3 2 2 2 4 2 4 2 0 0% 7 70% 1 10% 2 20% 10 2.5 2 2 4 4 Innovation 1 1 2 3 2 2 1 2 1 1 5 50% 4 40% 1 10% 0 0% 10 1.6 1 1 3 Preparation Process Preparation Process 5 Implementing Partners 1 1 1 1 2 1 2 2 1 2 6 60% 4 40% 0 0% 0 0% 10 1.4 1 1 2 6 Inclusiveness & targeting 3 1 4 2 2 2 2 2 3 2 1 10% 6 60% 2 20% 1 10% 10 2.3 2 1 4 7 Use of Seed Fund 4 0 0% 0 0% 0 0% 1 100% 1 4 #N/A 4 4 Complementarity & Value Added Complementarity & Value Added 8a Complementarity 3 3 2 2 2 2 1 1 2 25% 4 50% 2 25% 0 0% 8 2 2 1 3 8b Complementarity 3 1 3 4 3 2 4 1 1 1 4 40% 1 10% 3 30% 2 20% 10 2.3 1 1 4 9 Value Added 2 2 3 1 4 3 2 3 2 1 11% 4 44% 3 33% 1 11% 9 2.4 2 1 4 Efficiency Efficiency 10 Internal management arrangements 3 1 2 2 2 4 2 4 2 1 11% 5 56% 1 11% 2 22% 9 2.4 2 1 4 11 M&E 3 1 1 2 3 2 4 3 3 3 2 20% 2 20% 5 50% 1 10% 10 2.5 3 1 4 Effectiveness Effectiveness 12 Achievement of outputs & outcomes 3 2 3 2 2 2 2 2 2 3 0 0% 7 70% 3 30% 0 0% 10 2.3 2 2 3 13 Partnerships & Collaberation 1 1 1 1 2 1 4 4 2 2 5 50% 3 30% 0 0% 2 20% 10 1.9 1 1 4 14 Visibility 1 1 4 3 4 4 4 2 3 2 2 20% 2 20% 2 20% 4 40% 10 2.8 4 1 4 Impact & Sustainability Impact & Sustainability 15 Overall impact on the poorest 1 1 2 2.5 2 2 2 2 2 3 2 20% 6 60% 1 10% 0 0% 10 2.0 2 1 3 16 Imact: Winners & Losers 1 1 3 3 2 2 3 3 3 3 2 20% 2 20% 6 60% 0 0% 10 2.4 3 1 3 17 Impact: Capacity Building 2 3 3 2 2 1 2 3 1 2 2 20% 5 50% 3 30% 0 0% 10 2.1 2 1 3 18 Impact: Empowerment 3 2 2 3 3 2 3 2 3 2 0 0% 5 50% 5 50% 0 0% 10 2.5 3 2 3 19 Sustained benefits (& scaling up) 2 4 3 3 3 3 3 1 2 2 1 10% 3 30% 5 50% 1 10% 10 2.6 3 1 4 ` Table of PEP Scores.8Dec JSDF Draft Final Report Appendices, December 2006 D.2.2. Assessment Frameworks (for each site-visit project) These targets taken from the Application Form (AF), but also derived from the Evaluator's understanding of what the grant/project planned to achieve (for example from TOR of contracted NGOs or other planning documents developed after the AF. Most outputs and outcomes listed come from AGeFIB's Project Reports) Piloting reforms in the Childcare System Armenia TF 051331 Objectives Targets / planned Actual Achievement Outcomes & impact Grant The proposed grant aims at Number of visitors to the two Yerevan / Ajapnyak: 100 children benefiting (70 at the Centre and 30 in Development improving child welfare in Armenia community Centres outreach). Objective through: (i) improving effectiveness of (GDO) social support for children at risk; (ii) Gyumri: 241 beneficiaries involved in the project; 278 home visits piloting family-based care for carried out; new centre to be limited to 70 children at any one time (as currently institutionalized children above); working with 5 children in orphanages of which 3 successfully deprived of parental care and return to their families disabled children; and (iii) fostering community participation in support of children at risk and their families. Number of families at risk See numbers of children above ­ the programme also covers their receiving counselling families Number of children placed in Not implemented due to lack of agreed government policy on fostering foster families Other outcomes (not specifically Government and public attitude to child care has undergone shift from planned or foreseen) previous soviet style mentality 73 JSDF Draft Final Report Appendices, December 2006 Components Outputs A Two Pilot Community-Based Social Two SWCC established and Yerevan / Ajapnyak: Operational (at full capacity) with full compliment of Work and Care Center (SWCC) , with operational staff; issues: no telephone; some works to complete; insufficient outreach facility to be fully operational participation by local government bodies; over-bureaucratisation; lack of in Yerevan (Ajapynak) and Gumri community involvement Gumri: Social work & child care being carried out at two temporary sites by dedicated NGOs; New Centre still under construction (although Grant closed) B New family counselling and Facilities set up and operating from the new SWCC Centre in Yerevan rehabilitation services for families at and from two temporary sites in Gumri (pending completion of new risk and for their children provided Centre). Rating C Pilot de-institutionalisation Not implemented due to lack of regulation; lack of agreed policy on how programme established and to proceed. In retrospect considered inappropriate to include in the institutionalised children placed in JSDF grant foster homes or guardian families D Implement a public relations Seminars, materials and Campaign and management and operational website operation of Armenian Children Website E Conduct audit and monitoring and Grant audit and evaluation and The Ministry of Labour and Social Affairs initiated various monitoring evaluation monitoring reports arrangements, but no concrete system was put in place. The state system of monitoring was undertaken at various levels. The NGOs involved produced their own reports on process and progress Components not included in design documents F Training and Capacity Building No targets specified Numerous training courses for staff of the SWCC centres and other stakeholders to provide skills in working with children carried out by SOS and other NGOs Two training manuals prepared and distributed (also used outside JSDF grant as standard work) 74 JSDF Draft Final Report Appendices, December 2006 Community Based Urban Water Supply Management Project Armenia 51373 Objectives Targets / planned Actual Achievement Outcomes & impact Grant The safety and reliability of the water Improvement in daily hours of Seven contracts signed between condominiums and Water Supply Development supply services to communities in water service Company (WSC). This less than planned, but it argued that the Objective Yerevan participating in the project foundations for improvement laid through build up of trust and (GDO) are improved understanding. Although only 7 contracts signed this not considered a project failure. Information received after completion of evaluation is that the Water Company has now changed its policy and will now enter into Increased collection of water fees more contracts with CBOs / revenue. Increased reliability from only a few hours a day to a more regular and No. of Condominiums with direct reliable service with 24 hour provision in some places; water loss with Water Utility company to reduced (on the average estimated that per building losses reduced manage the water in the from 317 cumecs to 303 cumecs (saving of 113 cumecs per building) respective apartment blocks per year Fee collection increased: From 14% pre-project to 66% at project end Deep rooted change in apartment owners (including poor) attitude and responsibility to the consumption and need for payment and maintenance of utility services. However, maintenance of common services not yet fully accepted. 75 JSDF Draft Final Report Appendices, December 2006 Community Based Urban Water Supply Management Project Armenia 51373 Objectives Targets / planned Actual Achievement Components Outputs A Efficient community based Set up an Advisory Centre to Advisory Centre (AC) set up, but only on temporary basis and was organisations / condominiums build capacity of CBOs and disbanded at the end of the project. While the AC performed efficiently, created and strengthened to manage provide on-going advice the absence of longer term sustainable arrangement is a serious internal water supply distribution weakness systems No. of CBOs with efficiency in decision-making, planning and 180 CBOs selected for and participated in the project: Change in budgeting..& aptitude to attitude among condominiums (CBOs) and users in accepting and negotiating with utility service understanding need to (a) take responsibility for the maintenance and providers (b) pay for the water supply service Management of CBOs improved, but they do not yet have the trust of the householders in their respective apartment blocks A range of TV and newspaper advertisements / articles and other printed media prepared and disseminated. However, no public meetings Public awareness campaign held. Awareness of water related issues increased from 23% at start of carried out project to 98% at end. B New water meters supplied and No. of apartment blocks water 1753 apartment blocks have undergone internal water pipe renovation installed and water pipes and risers supply system improved (Final Report ­ Urban Institute); illegal connections removed replaced or repaired. Partial replacement of pipes and fittings in apartments carried out Water meters in poor households 2645 water meters installed in poor / vulnerable households (Final report installed Urban Institute) 76 JSDF Draft Final Report Appendices, December 2006 Community Based Urban Water Supply Management Project Armenia 51373 Objectives Targets / planned Actual Achievement C Strengthen of capacity of newly No of CBOs trained Training courses: created CBOs to interact with utility By NGO on Management and Finance of Condominiums: 218 chairmen service providers by establishing an / managers / accountants Advisory Centre in Yerevan, and; By Institute of Business Development on: Management and Finance of Build capacity of Municipality, State Condominiums: 100 people Committee on Water Economy and No. of staff trained the Yerevan Water & Sewage Company By Audit company (new standards in accountancy): 40 persons D Conduct audit Grant audit Carried out E Incremental Bank Supervision inputs Standard Bank reports prepared Prepared Components not included in design documents F Monitoring and Evaluation Not a component in AF, but in Series of regular progress reports and informative Final Report by the TOR of NGO managing the Consulting Team of the Urban Institute Advisory Centre 77 JSDF Draft Final Report Appendices, December 2006 Child Protection Fund Benin 026598 Objectives Targets / planned Actual Achievement Outcomes & impact Grant To pilot methods for preventing There was no specific target Summary of Impact : The most striking impact has been in terms of Development further child labour migration and number of children in design human capital: raising awareness of the perils of child trafficking, Objective emigration and the rescue and documents. 21 sub-prefectures empowering communities to try to prevent the phenomenon, sensitising (GDO) rehabilitation of current victims of were targeted and 25 villages community leaders and families about the rights of children and exploitative child trafficking. within these, based on studies responsibilities of parents. Child trafficking has been reduced, but these highlighting high levels of outcomes cannot be attributed to the JSDF project alone JSDF has trafficking (comprising zones of contributed to concerted donor efforts to achieve this impact (it is a good departure, transit and destination) example of successful donor harmonisation). Specifically: Rate of child trafficking reduced: Examples include: It is believed it was reduced by 60% in one village (source, ASPAD); in another, the evaluators informed that before the project every family experienced trafficking and at project end it was 3 in 10 families (source, Relais Communautaire). Other anecdotal evidence collected to support these views. Awareness of the perils of child trafficking raised (Women's Group in Gnangba village, CRAP NGO in Bohicon, Committee for Prevention of Child Trafficking in Ituque village ) Empowerment of women has been improved (anecdotal evidence) The quality of child protection services remains quite weak (Evaluator assessment, following meetings at UNICEF and Min Family) 78 JSDF Draft Final Report Appendices, December 2006 Components Outputs Summary of Effectiveness: Most project outputs were achieved, and there were impressive outcomes in terms of reducing child trafficking (see above). Project performance was weakened slightly due to a focus away from IEC in favour of `bricks and mortar' activities. A Capacity Building for the executing All 11 NGOs (selected on The executing agency provided training to all 11 NGOs in management AGeFIB and NGOs (11) restricted tender basis) and IEC techniques. Focus on capacity building of NGOs. A number of NGOs no longer operational, but five still active in child protection (quite impressive result in Benin context) B Integrated Village Components implemented (see below) Sensitisation At risk areas (all 21 sub- Sensitisation generally, possibly the most successful outcome (and may prefectures noted above, with 3 be responsible for the most significant outcomes). 31 sessions took types of zones) place for the general public, and 295 for a target audience (including police and judges). 84 discussions held regarding children's rights and responsibilities. 11 local radio stations had 132 transmissions on the subject. Innovative use of theatre, now adopted by UNICEF. Several partnerships created. Re-educating Intermediaries No. of intermediaries / trafficking Anecdotal accounts (for instance, one NGO said a reformed trafficker professionals had spoken on the radio about the perils of child trafficking). But generally little done in this area due to difficulties in identifying traffickers (and role played by many complicit community members). Creation of alternative opportunities No. of families or children Micro-projects implemented: 34 relating to improved leisure activities for assisted with opportunities (work children (theatre, games, folklore, others); 22 to classrooms, study and leisure) spaces and libraries; 16 relating to equipment (sport, music, folklore, library); 27 parent teacher associations helped to acquire civil registration (ID) cards for children; several projects include support to local Committees against Trafficking. Sustainable reinsertion No targets planned. Micro-projects implemented: 17 relating to the reinsertion of children who have been trafficked (? Nature of projects); 4 relating to professional/vocational training 79 JSDF Draft Final Report Appendices, December 2006 C Monitoring and Evaluation Monthly reports by implementing Intense monthly progress reporting by project partners (21 reports national NGOs produced by each NGO), but no time to reflect and make use of information for management purposes. Other reporting took place (not all of this assessed as unavailable) Components Rehabilitation and Leisure Centres Not in original AP. Introduced 5 Rehabilitation Centres and 3 Leisure Centres were begun during the not included constructed two thirds of way through project, but half of them unfinished at project completion. One receiving in design implementation ongoing local government funding, but in all cases unclear which documents structure should run them and what standards for caring for children should be established. 80 JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact Grant To improve access to alternative None given in AF Summary of Impact: The projects financed under this agreement Development means of dispute resolution and achieved a large number of beneficiaries and have begun to make a Objective qualified legal representation for poor contribution to attitude change, skills development and access to (GDO) rural and urban communities, services. The projects mostly lacked, however, detailed and specific indigenous peoples, women and strategies for reaching the poorest. Promising processes which would children. have ensured continued impact on the poorest and most vulnerable were cut short because the project ended. Specifically (summary): § Reports of improved quality of the services provided by the various legal services (mediators and others). § Increases in the number of cases handled by mediators in some centres. § Increased awareness on issues of poverty, social justice and strategies for conflict management among young people who participated in the culture of peace project. § Improved provision of legal services to the poor in some areas (Quito, for example). § More positive attitude of lawyers towards the provision of services for the poor. § Greater visibility of issues of access to law and justice through research, publications, and sharing of results with other partners. 81 JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact Note: project documentation focused on numeric output results, rather than outcomes and impact information. Components Outputs Summary of Effectiveness: All project components achieved more (and in some cases substantially more) than had been foreseen, in spite of difficult country context. Various interesting outcomes were achieved in terms of changed attitudes and enhanced skills. However, the short duration of the projects was a major constraint to consolidating processes and to effectively implementing strategies for sustainability. In addition, the individual components were in effect implemented without linkages and partnerships being established A Indigenous Justice: Improving The percentage change in the Numeric targets exceeded; access to justice by indigenous number of community justice communities by increasing the services available to the religious This component was revised prior to project implementation and understanding of conflict resolution communities foresaw three concrete results: mechanisms within these communities and the formal state § Conduct a research study in support of the activities to be The percentage change in the implemented by the NGOs under other components of the justice system and through the number of public consultations on project construction of common values implementing legislation around human rights and gender § Support the national process for the construction of indigenous concerned with clarifying equity. dispute resolution mechanisms which respect the customs of indigenous and Ecuadorian laws each community. § Train communities in legal processes and options for access to formal and informal mechanisms for dispute resolution. 82 JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact The following results were achieved: § Research-training study conducted on indigenous justice conducted in 19 ethnical sites, against the 16 planned in 11 provinces of the country. § Mapping of indigenous justice practices of each of these communities available. § Communities involved trained in aspects of the formal legal service (constitution, human rights, etc.) and aware of the various mechanisms for conflict resolution. § Close linkages established between formal and informal/indigenous justice systems. § Dissemination of information about indigenous justice; including through innovative means such as radionovelas. Sources: Informe de Monitoreo y Evaluacion de las Actividades Previstas en el Convenio de Donacion Japonesa TDF 051227 (Dec 2005) and Proyecto Derecho y Justicia para Los Pobres (no date), Projusticia and Final Evaluation Report Indigenous Justice Component (2006), Esther Sanchez Botero. 83 JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact B Peace Culture: Design and The percentage change in the Numeric targets exceeded for all planned outcomes: implement training programs in number of disputes resolved by elementary schools in both rural and the pilot mediation programme urban marginalized areas focussing Planned Results obtained on the development of a culture of % change in number 100 100 The number of child and teacher peaceful conflict resolution. of conflicts solved mediators trained through the pilot programme Number of 2862 4174 beneficiaries Number of students 2776 3879 Number of teachers 86 373 trained Number of children 158 185 mediators trained Gender distribution of 50:50 55:45 beneficiaries (M:W) Number of conflicts 240 2264 resolved peacefully Agreements of 245 329 change obtained Number of parents 194 220 participating in the workshops N.B. Achievements most relevant to initial targets highlighted in table. Overall results: 84 § Recognition by the Ministry of Education of the Peace Education programmes offered in schools § Improved detection of child abuse and referral to supporting JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact C Alternative Dispute Resolution The percentage change in the Numeric targets exceeded Services: Support establishment and resolutions being submitted to § 30% increase in the number of centres of mediation available strengthening of mediation centres alternative dispute resolution (13 new centres opened in areas of difficult access). with particular emphasis on areas services § Creation of a national coordination mechanism for stakeholders that are not accessible because of involved in Alternative Conflict Solutions Mechanisms. physical distance and of a national The percentage change of people association of alternative dispute receiving alternative dispute resolution providers. Proyecto Derecho y Justicia para Los Pobres (no date), Projusticia and resolution services provided by Informe de Monitoreo y Evaluacion de las Actividades Previstas en el NGOs Convenio de Donacion Japonesa TDF 051227 (Dec 2005) D Public Defence Services: Support The percentage change in the Numeric targets exceeded; the Association of Law School Deans number of people represented by and several local bar associations in pro-bono defence lawyers Specific outputs of the project developing mechanisms to provide qualified legal representation for § Interinsitutional group formed to carry out activities under the The percentage change in the subcomponent indigent litigants with an emphasis on number of private practitioners women and children. § Report with recommendations on the public defenders bill produced providing legal assistance § National training program for training certified attorneys as diplomados in defensoria publica (public defence lawyers) set up involving 240 hours of training and granting 15 credits to lawyers in Quito and Guayaquil. The training is recognized by the National Education Council. Training of 75 lawyers was planned, 110 were finally trained. The training had a specific focus on criminal justice, family law and employment law in order to comply with the emphasis on women and children. § 839 women and 439 men served during the project period. 85 JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact § Information about public defence services disseminated in newspapers, printed brochures and radio in both cities. § Training of 200 social lawyers (defensores socials) trained in Quito and Guayaquil resulting in a 25 times increase over the number previously available. § Incentives established to attract pro bono lawyers by providing professional vision, access to training and support. Overall conclusion from the evaluation of this sub-component: "The Project appears to have accomplished much more than was technically or contractually required. External support for this effort was essential, but reflects Ecuadorian success." (Brummer, 2006, p. 7). Source: Bummer (2006). Evaluation of Public Defense Services Sub- Component. 86 JSDF Draft Final Report Appendices, December 2006 Law and Justice for the Poor Ecuador 051227 Objectives Targets / planned Actual Achievement Outcomes & impact E/F Audit; Monitoring and Evaluation: Audit report All implementing partners complied with agreed upon reporting Creation of inputs, outputs, indicators requirements and a substantial amount of documentation exists on and project management tools for the project achievements, however base-line data is was not available for Quarterly and Annual Reports entire project and each sub-grant. the project as a whole, and monitoring and evaluation focused heavily on quantitative results, with less attention/information on outcome, impact and qualitative changes in processes and results. It was not possible to obtain a complete overview of this component which, contrary to all other components, was not subject to a final evaluation. It was established, however, that various consultancies were carried out in the context of this component which resulted in revised suggestions for indicators for measuring project progress and in the collection of base line information for some of the project activities. This component also resulted in the design of a website for the project as a whole. 87 JSDF Draft Final Report Appendices, December 2006 Empowering the Poor: A Pilot IT Program for Rural Areas India 051375 Objectives Targets / planned Actual Achievement Outcomes & impact Grant E-Education: "To promote IT literacy IT training is mainstreamed in 50 Summary of Impact: Development among rural and disadvantaged rural schools § There has been some impact on the poorest and most Objective populations" vulnerable, especially through the IT buses and the Farm (GDO) Radio. The decision to select government (ZP) schools for the E-Commerce: "To employ information IT buses has helped reach some of the poorest schools, plus Marketing: technology to empower the rural poor the initially subsidised fees enabled access by all students from Improvement of marketing of th th by stimulating local commerce and 5 to 7 standards, including the marginalised. The increase in products and services from rural the fee rates compelled those in the Tribal Areas to dropout, so business, and improving their access areas to Pune and other urban coverage is now more limited. to essential information and services" centres; § The objective to provide basic skills and exposure to IT was Creation of 10 new business achieved, but beyond this there is no means take it further. groups; There is also limited opportunity for students to practice newly 50% increase in productivity of acquired skills, and rural work opportunities are more limited for business groups (expressed in IT. The model is ripe for scaling-up, but cannot be sustained sales) without external support. § Farm Radio is providing extension to farmers, through Cooperatives experiential learning, expert advice (crops, diseases, etc), and information on agricultural practices. Formal systems of Systems improvements in feedback were developed in the first year (under KVK) but no cooperatives constituted primarily formal impact assessment has been undertaken for this of poor; component (to assess the listener base, the uptake of extension Increase in productivity messages, etc). (expressed in sales of § The radio has received informal, positive feedback. The radio participating cooperatives station appears to have been successful in bringing the radio in- house, and developing an up-and-coming team. 88 JSDF Draft Final Report Appendices, December 2006 Empowering the Poor: A Pilot IT Program for Rural Areas India 051375 Objectives Targets / planned Actual Achievement § A sustainable business model remains elusive, though changes in broadcasting regulations for community radio may provide (limited) adverting and sponsorship opportunities. There are th plans to adopt the model nationwide through the 11 National Plan. § Other aspects of the project did not reach full implementation, and so impact has consequently been limited. The Content development only reached the development stage (by design) and so did not reach the outcome level. § The MIS and Smart cards did not reach output stage and so were never utilised. The training of paraprofessionals reached the output level (250 people trained), but the link to the work opportunities and selection criteria meant that it did not lead to any specific impact. Components Outputs Summary of Effectiveness: Several components were highly successful in reaching their outcome targets (IT buses, Farm Radio), while others achieved high quality outputs that have yet to be adopted (Content development, Paraprofessionals), and some failed to produce satisfactory outputs at all (MIS & Smart cards, Web portal). A E-Education Sub · IT buses ­ buses installed Computer literacy and IT enabled IT buses (4-High): Exceeded output targets (6,351 students across 51 component with computers & trained education for 5000 rural school schools), with successful response from students and teachers (skills teachers to teach IT at rural children; learnt, confidence and attendance increased). schools. 89 JSDF Draft Final Report Appendices, December 2006 Empowering the Poor: A Pilot IT Program for Rural Areas India 051375 Objectives Targets / planned Actual Achievement Sub · Content development ­ use Education content (2-Modest): Outputs achieved to a high quality but component of multi-media to create IT training for 100 teachers; no mechanism to reach outcomes (i.e. adoption by teachers). While it is education content for all Action plan to promote noted that the objective of this component was for product development subject areas. sustainability only, the failure to plan for outcomes within the project limits the assessment of `sucess'. B E-Commerce Providing access to 5000 members of cooperatives to system; 1000 beneficiaries have improved access to outside markets; Creation of IT enabled system to aggregate demand for product and services from outside area; Diss-eminate information on agricultural and other related Research Sub · MIS & Smart cards ­ system Smart Cars used by members of MIS & Smart cards (1-Low): Outputs not fully completed, not pre- component to track producer records for cooperatives tested and the system is not being utilized. cooperative organisations Sub · Paraprofessionals ­ training Training for 500 IT Paraprofessionals (2-Modest): Outputs achieved (paraprofessionals component of unemployed and others to paraprofessionals; trained and satisfied with quality), but not tied to the utilisation of skills provide IT support in rural through work opportunities. Therefore outputs achieved, but outcomes areas limited. 90 JSDF Draft Final Report Appendices, December 2006 Empowering the Poor: A Pilot IT Program for Rural Areas India 051375 Objectives Targets / planned Actual Achievement · Web Portal ­ website to Web portal created dedicated to Web Portal (1-Low): Web Portal not fully developed; not even a facilitate new marketing marketing demonstration model showing pictures of sample SHG products. opportunities for Self-Help Conceptual framework remained weak and could not lead to planning Groups (SHGs) and implementation (development). Components not included in design documents · Farm Radio ­ mass media Not in plan Farm radio (4-High): Successful (reportedly being listened to), and has dissemination of agricultural had some policy influence (may be adopted by the National th advice and expertise Commission, under the 11 National Plan). · Capacity Building Good capacity building of IT teachers and school children (under E- education), and students and experts (under Farm Radio); Training and capacity building under MIS and Paraprofessionals not adequately linked to work opportunities and follow-up support. 91 JSDF Draft Final Report Appendices, December 2006 Marginal Fishing Community Development Project Indonesia 026799 Objectives Targets / planned Actual Achievement Outcomes & impact Grant To develop and support different Fishing productivity increased Summary of Impact: Benefits of the revolving funds are still being felt. Development participatory planning and (Nos.) However, some communities are struggling with repayments as a result Objective development management pilots in Income levels increased (Nos.) of disaster, seasonal conditions and some have defaulted on payments. (GDO) very poor coastal fishing communities Information on incomes indicates that while some communities have Maritime property rights reformed to raise fishing productivity and increased incomes, others have not benefited. Factors being income levels seasonality and increased costs (especially fuel). Project processes had improved relationships within the communities and prospects for more district funds. The most significant output for fishing families was the rotating fund. The fund may not have reached the poorest and most vulnerable in the community Components Outputs Summary of Effectiveness: The project was implemented over two years with contact with the communities took place over the last year making it very intense. Training was designed and delivered in this time, structures built (for the most part) and communities trained to receive and manage funds. Several manuals were produced for various participants in the project including a document which detailed lessons learnt A Coastal Community Grants (US$ No. of proposals 1,086,000) ­ finance community No. of grants / revolving funds proposals for economic infrastructure No. of fishing people receiving & fishing technology inputs. grants No. of communities benefiting No. of goods / equipment delivered 92 JSDF Draft Final Report Appendices, December 2006 Marginal Fishing Community Development Project Indonesia 026799 Objectives Targets / planned Actual Achievement B Appropriate technology & marketing No. of training sessions support (US$ 347,000) ­ identify No. fishing communities using technology support in each fishing better technology community. No. communities with stronger market linkage C Maritime Property Rights Initiative Strategy prepared on Maritime (US$112,000)- develop & implement law a strategy to life constraints o coastal Pilot project implemented and fishers. impact studied 93 JSDF Draft Final Report Appendices, December 2006 Squatter Settlements Pilot Assistance Project Indonesia 026563 Objectives Targets / planned Actual Achievement Outcomes & impact Grant The original objective was to provide NA Summary of impact: The re-focus of the project towards policy and Development assistance to squatter communities. programme solutions to land tenure has the potential to have a greeter Objective There was no `higher order' objective, impact on poverty alleviation than a series of small projects aimed at (GDO) but the components suggested a improving livelihoods and the environment. The greatest impact relates classic CDD-type project. The re- to physical capital, in the sense of `legal' outcomes, both on and off site focused objective (stated above) solutions to land tenure and (through facilitation in some sites) access to concentrated more on approaches to ID cards. However, the land tenure solutions have yet to be fully tested. developing sustainable policies and Secondly, significant social capital has been built through the programmes. institutionalisation of dialogue on land tenure issues (through the SCAPs, SSMs, and input to land use planning). Components Outputs Summary of Effectiveness: very effective in achieving outcomes (though not ones listed at start!), impressive that work has continued post- completion, but many policy challenges lie ahead to ensure approach understood, lessons documented and effective replication. A Programme facilitation NA From ICR: the project achieved outcomes far beyond those originally envisaged: all local governments in the project sites have formulated policies and medium term plans for dealing with squatters; national government has drafted a national strategy. The quality of the draft national strategy is good, in terms of diagnosis of different squatters and approach taken to find on and off site solutions. B community sub-grants (for activities No. of neighbourhoods These were used for a wide range of purposes; those used to improve to be decided by the communities the physical environment was considered on the whole more successful themselves); than the vocational training and micro-credit schemes [ICR]. Site visits confirm this 94 JSDF Draft Final Report Appendices, December 2006 C programme management assistance NA No outcomes expected D M&E/research. NA Judged to be marginally satisfactory: the independent evaluation and research on institutional readiness in the provision of land for squatters were carried out, and the quality quite good, but given many activities are ongoing, it is too soon to assess the approaches and the degree of suitability for replication [ICR]. 95 JSDF Draft Final Report Appendices, December 2006 Village Investment for the Poor Lao PDR 026886 Objectives Targets / planned Actual Achievement Outcomes & impact Grant To assist the recipient in improving None given in AF Summary of Impact: But based on the information gathered during the Development the welfare of poor rural households visit and the positive outlook for the sustainability of micro-projects as Objective and strengthening the capacity of village level as well as for the scaling out of the VIP approach that (GDO) rural communities (as in the Trust impact has been and will continue to be positive. A key to this is that Fund Status report and ICM) while support is needed from the Government service agencies (agricultural extension for example) at village level, beneficiaries were able to demonstrate an ability to manage the micro-projects with a considerable degree of independence. The Final Evaluation Report was very positive on increases in income level and food security, although, this view might be over-optimistic. As the loans and grants provided under VIP are relatively small, the benefits do not appear to be life changing but rather provide important additional benefits. Beneficiaries are not necessarily the very poor, as a requirement for a micro-loan is that the village must demonstrate development potential and the householder an ability to make use of the micro-loan facility. It was estimated (End of Project Report, February 2005) that of 125 villages in 4 provinces that were covered by the VIP, that almost 45,000 persons benefited in one form or another. 96 JSDF Draft Final Report Appendices, December 2006 Village Investment for the Poor Lao PDR 026886 Objectives Targets / planned Actual Achievement Components Outputs Summary of Effectiveness: The micro-finance programme has on the available information, been successful with ratings for individual villages in one of the four provinces rated as between good to excellent. The community development projects, which have been community managed, have been rated as excellent in 11 out of 17 villages (in Saravane Province) and good in the remainder. Village saving schemes while very popular appear either to have worked well in the majority of cases, although there have been villages where they have failed completely. B Workshops, Information No targets given Capacity building among beneficiaries on technical issues is very high ­ Dissemination and Monitoring and but only modest for community development approaches. Poor capacity Evaluation building in M&E together this gives a score of 2. A number of workshops conducted including: 5 PRA training over 7 days (18 ­ 26 participants) 3 Annual Planning over 5 days (18 ­ 27 particpants) 13 other training including M&E 13 training / demonstrations on activities C Programme Implementation and NA The management arrangements generally work well, but the grant is Coordination within an ADP structure ­ i.e. not fully part of a standard government arrangement. It has the majority of technical staff resources required, except the key one of community development skills. 97 JSDF Draft Final Report Appendices, December 2006 Northern Uganda Youth Rehabilitation Fund Uganda 050522 Objectives Targets / planned Actual Achievement Outcomes & impact Grant Interventions that target vulnerable NA Summary of impact: There was evidence of sustained benefits amongst Development youth in Northern Uganda' and `test beneficiaries contacted. Many were working in the area in which they Objective implementation arrangements, had been trained and reported improved livelihoods as a result of the (GDO) particularly how the project will training received under the project. However, with the overwhelming involve Civil Society Organisations response for training it is difficult to assess whether it is the `poorest and and Community based Organisations most vulnerable' benefited. Strong political involvement made targeting in implementation.' more complicated as it influenced selection procedures. Short implementation period did not help. A change in attitude towards what is possible amongst `redundant' youth was evident at all levels. There was a reported changes in behaviour of CBOs, VTIs and the implementing NGOs as a result of project. Strong belief that this model is a good example of how things should be done. Components Outputs Summary of Effectiveness: The project addressed a real need (overwhelming demand!) and delivered the training it set out to deliver. As higher level goals not explicit ­ and data not collected against implicit objectives at this level, difficult to access A to build the capacity of CBOs, PS and No, of communities / CSOs The capacity of CBOs /NGOs was increased, both those who were NGOs to respond to the unique trained; delivering the training and WV who were the main implementing agency. needs of the youth Guidelines prepared Institutions who carried out the training reported benefits from the training in financial management and psycho- social support, as well as the experience itself. World Vision gained experience in working with WB, GOU at both National and District level as well as learnt lessons that enriched their subsequent programmes. 98 JSDF Draft Final Report Appendices, December 2006 B To equip youth with specific skills and No. & categories of vulnerable The project delivered vocational skills training to 3,152 youth across 18 start up tool kits to enable them to youth and youth groups districts in a very short time frame. The beneficiary assessment earn incomes, provide services and supported conducted after the close of the projects, interviewed 350 beneficiaries improve their livelihoods No. of CSOs working with youth and found that 77% of them are practicing the vocational skill acquired. The project was successful in this regard No. and types of communities covered by IEC C To contribute towards community No. of individuals reached and No indicators were developed to measure success against this reconciliation and conflict No. of activities financed objective. Training in psycho-social support was given to trainers and management trainees and both groups said that the training was useful. There was also a reported change in attitude towards youth who had at one time been seen as `redundant' at all levels from national government, local government, NGOs and institutions involved in the programme, the communities and the beneficiaries themselves. This change in attitude where now people believed that vulnerable youth could be useful can be seen as an initial step towards community reconciliation. Components not included in design documents To enable the youth to participate in This objective has to some degree taken place. There are reported the NUSAF activities. NA instances where beneficiaries who did not get full tool kits have successfully applied to NUSAF for funding for tools. Reports of instances were those trained in building have been employed on construction NUSAF projects. There is a policy in place where NURYF beneficiaries are to be given priority (particularly in terms of applying for tools) but this is not necessarily executed at ground level. 99 JSDF Draft Final Report Appendices, December 2006 Northern Mountain Poverty Reduction Project Pilot Commune Program Vietnam 026559 Objectives Targets / planned (Note 1) Actual Achievement Outcomes & impact Grant Trialling and evaluating new project None given in AF Summary of Impact: Due to its pilot nature and strong emphasis on Development implementation approaches based on building capacities at institutional levels, the impact that has been Objective community participation achieved among target beneficiaries has been modest. However, it (GDO) must also be noted that impacts vary for different components and shortcomings in targeting and outreach to the poor that may have occurred during the pilot phase have been corrected during the subsequent implementation of NMPRP Components Outputs Summary of Effectiveness: There were significant adjustments to the (no targets provided in AF, but project budget; a number of activities have not been implemented under type of output expected given) the pilot project and deferred to the main project. Overall estimated that 80% of activities have been completed 2 communes in 3 Provinces participated A Community Development Methods, Revised community development NA plans and management plans; targeting guidelines; training in community development methods B Food Security and cash incomes No. Demonstrations conducted; Increase in number of crops per year; increased productivity (rice and generation number of farmers trained; corn) revised guidelines C Household water supply Designs completed; number of Water supply more convenient; improved safety; and regular supply people trained; number of 100 JSDF Draft Final Report Appendices, December 2006 Northern Mountain Poverty Reduction Project Pilot Commune Program Vietnam 026559 Objectives Targets / planned (Note 1) Actual Achievement household water supply systems built D Micro-irrigation No. micro-irrigation schemes completed; number of farmers with irrigated land under this component E Education and Health No. of teachers trained; No. of More children go to school and more safely; patients can stay in medical workers trained; No. of commune to for treatment health clinics built; No. of schools built F Community Grant No. of grants awarded and NA implemented; no. of villagers implementing grants No. of people trained G Participatory Monitoring and No. of communes trained in Monitoring was sufficient in general. All grant activities are well- Assessment participatory monitoring; documented at the commune, district and provincial level, and Assessment of impact completed; investments undertaken by the JSDF grant are clearly marked. participatory monitoring However, monitoring was weak in the beginning and mainly followed implemented in main project standard procedures, like routine reporting on progress. Training on communes monitoring was delayed and guideline only developed towards the end of the pilots. As a result, there was a delay in defining and implementing the monitoring system 101 JSDF Draft Final Report Appendices, December 2006 D.3. Web Based Survey 1. Years respondents were involved with the JSDF programme, either in applying for, or managing, grants. Year Response Response Percent Total 2000 9.3% 5 2001 22.2% 12 2002 31.5% 17 2003 50% 27 2004 61.1% 33 2005 70.4% 38 2006 63% 34 Total Respondents 54 2. Number of JSDF grants managed by respondents. No. Response Response Percent Total 1 50% 27 2 35.2% 19 3 9.3% 5 4 5.6% 3 Total Respondents 54 3. Stages of a JSDF grant managed. Response Response Stage Percent Total Application Process Only 20.4% 11 Project implementation from 55.6% 30 beginning to end Initial months/years of project 31.5% 17 implementation Final months/years of project 13% 7 implementation Project implementation (but not 9.3% 5 during start up or close down). Other (please specify) 9.3% 5 Total Respondents 54 102 JSDF Draft Final Report Appendices, December 2006 Other 1 From Seed Money to Implementation - But projects not completed yet 2 Application and implementation (around 1 year so far) 3 From application to ICM preparation. 4 proposal drafting and application process 5 Project implementation during the last year of initial implementation period 4. Respondents involved in the application process for a JSDF grant. Response Response Percent Total Yes 74.1% 40 No 25.9% 14 Total Respondents 54 5. Respondents involved in writing and submitting an application that was... Response Yes No Total ...successful 98% (39) 2% (1) 40 ...unsuccessful 44% (12) 56% (15) 27 Total Respondents 41 6. Respondents opinion on; the approach taken by the JSDF funded project was/is new to the country/region. Response Response Percent Total Strongly Agree 26.4% 14 Agree 39.6% 21 Disagree 20.8% 11 Strongly 5.7% 3 Disagree No Opinion 7.5% 4 Total Respondents 53 103 JSDF Draft Final Report Appendices, December 2006 7. Respondents opinion on the degree to which the following stakeholders participated in the design of the project Their Did not take Attended opinions Actively N/A part meetings influence involved design NGOs 14% (7) 4% (2) 26% (13) 50% (25) 6% (3) Japanese NGOs 58% (26) 4% (2) 7% (3) 7% (3) 24% (11) Provincial / Local 8% (4) 6% (3) 31% (15) 43% (21) 12% (6) Government National 4% (2) 12% (6) 29% (15) 50% (26) 6% (3) Government Beneficiaries 8% (4) 10% (5) 31% (16) 45% (23) 6% (3) Total Respondents 53 8. Respondents opinion on: JSDF should continue to encourage partnerships with NGOs on its projects. Response Response Percent Total Strongly Agree 42.3% 22 Agree 48.1% 25 Disagree 3.8% 2 Strongly 1.9% 1 Disagree No Opinion 3.8% 2 Total Respondents 52 9. Number of respondents who have been involved with a JSDF grant which received a Seed Fund Response Response Percent Total Yes 17% 9 No 83% 44 Total Respondents 53 104 JSDF Draft Final Report Appendices, December 2006 10. Respondents opinion on; Strongly Strongly Agree Disagree No Opinion Agree Disagree The seed fund increased the participation of 75% (6) 25% (2) 0% (0) 0% (0) 0% (0) beneficiaries in the design stage of the project The seed fund increased the level of participation by 38% (3) 38% (3) 0% (0) 0% (0) 25% (2) beneficiaries in the implementation of the project Total Respondents 9 11. Respondents opinion on; The JSDF programme supports development activities which are not funded by... Strongly Strongly Agree Disagree No Opinion Agree Disagree ... other initiatives within 23% (12) 52% (27) 21% (11) 2% (1) 2% (1) the Bank ... sources outside the 13% (6) 43% (20) 34% (16) 9% (4) 2% (1) Bank Total Respondents 53 105 JSDF Draft Final Report Appendices, December 2006 12. Respondents opinion on; Strongly Strongly Agree Disagree No Opinion Agree Disagree Compared to other initiatives in the Bank, the JSDF programme 21% (11) 54% (28) 19% (10) 0% (0) 6% (3) focuses more on the poorest and most vulnerable The JSDF programme is better at achieving social development outcomes 23% (12) 35% (18) 27% (14) 0% (0) 15% (8) compared to activities funded by other sources in the Bank The work carried out by the JSDF programme 44% (23) 52% (27) 2% (1) 0% (0) 2% (1) complements other work carried out by the World Bank Total Respondents 52 106 JSDF Draft Final Report Appendices, December 2006 13. Respondents opinion on; Strongly Strongly Agree Disagree No Opinion Agree Disagree JSDF programme objectives are clearly defined 13% (7) 73% (38) 10% (5) 0% (0) 4% (2) for TTLs applying for grants The process of revising the JDSF grant application form to incorporate comments from 0% (0) 38% (20) 31% (16) 8% (4) 23% (12) the TFO/Steering Committee improves the design of the JSDF project If the grant application process was decentralised regionally, the 19% (10) 38% (20) 13% (7) 4% (2) 25% (13) JSDF programme would be better at achieving its objectives Total Respondents 52 14. In line its objectives, the JSDF grant produced results that were; Strongly Strongly Agree Disagree No Opinion Agree Disagree ...rapid 10% (5) 49% (25) 25% (13) 10% (5) 6% (3) ...innovative 37% (19) 54% (28) 6% (3) 2% (1) 2% (1) ...demonstrable 29% (15) 67% (35) 2% (1) 2% (1) 0% (0) ...replicable 25% (13) 65% (34) 2% (1) 2% (1) 6% (3) Total Respondents 52 107 JSDF Draft Final Report Appendices, December 2006 15. The JSDF programme objectives (listed in the previous question) are still relevant. Response Response Percent Total Strongly Agree 21.6% 11 Agree 72.5% 37 Disagree 2% 1 Strongly 0% 0 Disagree No Opinion 3.9% 2 51 16. The JSDF funded project has built capacity among; Strongly Strongly Agree Disagree No Opinion Agree Disagree Beneficiaries 31% (16) 63% (32) 4% (2) 2% (1) 0% (0) NGOs 22% (11) 64% (32) 8% (4) 2% (1) 4% (2) Provincial / Local 14% (7) 55% (28) 20% (10) 4% (2) 8% (4) Government Government 4% (2) 43% (21) 31% (15) 6% (3) 16% (8) Total Respondents 51 17. Strongly Strongly Agree Disagree No Opinion Agree Disagree ... influenced other Bank 12% (6) 53% (27) 6% (3) 4% (2) 25% (13) Projects ... enhanced the Banks best practice for capacity building in civil 20% (10) 59% (30) 2% (1) 2% (1) 18% (9) society, beneficiary communities & NGOs ... enhanced the Banks policy for Social 10% (5) 35% (18) 10% (5) 2% (1) 43% (22) Development operations ... informed development approaches taken by 2% (1) 16% (8) 14% (7) 8% (4) 61% (31) Japanese Agencies (eg JICA) Total Respondents 51 108 JSDF Draft Final Report Appendices, December 2006 18. The results of the JSDF project influenced other activities being carried out in the recipient country by... Strongly Strongly Agree Disagree No Opinion Agree Disagree ... the government in 12% (6) 56% (28) 12% (6) 4% (2) 16% (8) the recipient country ... other 12% (6) 54% (27) 6% (3) 2% (1) 26% (13) organisations Total Respondents 51 19. There is no clear mechanism for disseminating lessons learnt from JSDF projects. Response Response Percent Total Strongly Agree 16.3% 8 Agree 51% 25 Disagree 18.4% 9 Strongly 0% 0 Disagree No Opinion 16.3% 8 Total Respondents 49 20. TTLs would benefit from training in the following areas of project management; Yes No No Opinion Project M&E 62% (31) 30% (15) 8% (4) Disseminating/Communicating 76% (39) 18% (9) 6% (3) Lessons Learnt Total Respondents 51 21. Implementing partners would benefit from training in the following areas of project management; Yes No No Opinion Project M&E 90% (46) 10% (5) 0% (0) Disseminating/Communicating 88% (45) 10% (5) 2% (1) Lessons Learnt Total Respondents 51 109 JSDF Draft Final Report Appendices, December 2006 Appendix E Table A: Condensing the list of Sectors Condensed List of Original List of WB Sectors/Themes Sectors/Themes 1 Education -Primary Education 2 Education 3 Education/ Human Development 1 Education 4 Education/ Social Development 5 Human Development (Education) 6 Health 7 Health and Other Social Services 8 Health and Other Social Services - Social Protection 2 Health 9 Health and other Social Services, Social Protection Risk Management 10 Health/Human Development 11 Human Development (Health Services) 12 Human Devel./ Social Protection 13 Human develop., social protection 3 Human Development 14 Human Development 15 Law and justice 4 Law and justice 16 Law and Justice and Public Administration 17 Multi-sector - Social Dev. Gender and Inclusion/Education 18 Multi-sector - Rural and Urban Development, and Infrastructure 19 Multi-sector - Social Dev. Gender and Inclusion 20 Multi-sector - Social Protection and Risk Mgmt. 5 Multi Sectoral 21 Multi-sector (Human Development) Multi-sector - Rural and Urban Development, and 22 Infrastructure/Education 23 Multi-sector - Social Dev. Gender and Inclusion/Rural Development 24 Agriculture, Fishing and Forestry 6 Natural Resources 25 Financial and Private Sector Development 7 Private Sector Public Sector 26 Public Sector Governance 8 Governance 27 Agricultural Fishing and Forestry 28 Agriculture Fishing and Forestry - Rural Development 29 Agriculture, Fishing and Forestry (Rural Development) 9 Rural Development 30 Rural Development 31 Transportation - Rural Dev. Infrastructure 32 Social Development 10 Social Development 33 Agriculture, Fishing and Forestry - Social Dev. 34 Urban Development/ Social Protection 11 Urban Development 35 Urban Development & Infrastructure 36 Water & Sanitation 12 Water & Sanitation 37 Water and Sanitation (Human Development) 110 JSDF Draft Final Report Appendices, December 2006 Table B: Table showing the eligible countries which have and have not received JSDF funding Total % No JSDF Grant Eligibl Receive Region Recipient Countries Received e d AFR Benin Angola Cambodia Burkina Faso Cape Verde Burundi Central African Republic Cape Verde Chad Comoros Congo, Republic of Cote d'Ivoire Ethiopia Eritrea Guinea Gambia Mali Ghana Mauritania Guinea Bissau Nigeria Kenya Sao Tome and Principe Lesotho Senegal Liberia Sierra Leone Madagascar Tanzania Malawi Uganda Mozambique Niger Republic of Congo Rwanda Somalia South Africa Sudan Swaziland Togo Zambia Zimbabwe Totals 16 27 43 37% EAP Cambodia Fiji China Kiribati East Asia and Pacific Marshall Islands Indonesia Micronesia Lao People's Democratic Republic Myanmar Mongolia Solomon Islands Papua New Guinea Thailand Philippines Tonga Samoa Vanatu Vietnam Totals 10 9 19 53% ECA Albania Azerbiajan Armenia Belarus Bulgaria Boznia & Herzegonia Georgia Kazkhstan Kyrgyz Republic Macedonia Moldova Romania Russian Federation Serbai & Montenegro Tajikistan Turkey Ukraine Turkmenistan 111 JSDF Draft Final Report Appendices, December 2006 Total % No JSDF Grant Eligibl Receive Region Recipient Countries Received e d Uzbekistan Totals 10 9 19 4% Central America Bolivia LAC Colombia Brazil Ecuador Dominican Republic Honduras El Salvador Nicaragua Guatemala Paraguay Guyana Peru Haiti Jamaica Surinam Totals 7 9 16 44% MENA Egypt, Arab Republic of Algeria Jordan Djibouti Turkey Iran Yemen, Republic of Iraq Morocco Syria Tunisia West Bank & Gaza Yemen Totals 3 8 11 27% SAR India Afghaniststan Pakistan Bangladesh Sri Lanka Bhutan Maldives Nepal Totals 3 5 8 38% 112