78524 Independent Evaluation of the Multi-Donor Trust Fund in Sudan FINAL EVALUATION REPORT 19 February 2013 This page is intentionally blank Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Independent Evaluation of the Multi-Donor Trust Fund in Sudan World Bank FINAL EVALUATION REPORT submitted by ICF GHK 19 February 2013 40252884 ICF GHK 2nd Floor, Clerkenwell House 67 Clerkenwell Road London EC1R 5BL T +44 (0)20 7611 1100 F +44 (0)20 3368 6960 www.ghkint.com Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Document Control Document Title World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Job number 40252884 Prepared by TE, KR, KT Checked by KT Date 19 February 2013 ICF GHK is the brand name of GHK Consulting Ltd and the other subsidiaries of GHK Holdings Ltd. In February 2012 GHK Holdings and its subsidiaries were acquired by ICF International. Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Abbreviations 5YP Five Year Plan BEP Basic Education Project BNEP Blue Nile Emergency Start-Up Project CDC Community Development Committee CDF Community Development Fund CPA Comprehensive Peace Agreement CPPR Country Portfolio Performance Review DFID (UK) Department for International Development DPA Darfur Peace Agreement ESPA East Sudan Peace Agreement FFAMC Fiscal and Financial Allocation and Monitoring Commission FMoH Federal Ministry of Health FNC National Forest Corporation GA Revitalization of the Sudan Gum Arabic Production and Marketing Project GA Grant Agreement GAPA Gum Arabic Producers Associations GoNU Government of National Unity GoS Government of Sudan ICC International Criminal Court IBRD International Bank for Reconstruction and Development IDA International Development Association IEG Independent Evaluation Group IFAD International Fund for Agricultural Development ILPM Livestock Production and Marketing Project I-PRSP Interim Poverty Reduction Strategy Paper IRND Inland River Navigation Department JAM Joint Assessment Mission Judiciary Capacity Building of the National Judiciary LIU Local Implementation Unit LTS Loan Tracking System MA Monitoring Agent MDG Millennium Development Goal MDTF Multi donor trust fund MDTF-N Multi Donor Trust Fund – National MDTF-SS Multi Donor Trust Fund for South Sudan MFI Microfinance institution MoFNE Ministry of Finance and National Economy MoGE Ministry of General Education Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report NETREP National Emergency Transport Rehabilitation Project NGO Non governmental organisation NHA National Highway Authority NJSC National Judiciary Service Commission NTMP National Transport Master Plan OC Oversight Committee PIU Project Implementation Unit PMS Pavement Management System PRSP Poverty Reduction Strategy Paper PSCAP Public Sector Reform, Decentralization, and Capacity Building Program Support Project SAAS Sudan Academy for Administrative Sciences SBD Standard Bidding Document SMDF Sudan Microfinance Project SPLM Sudan People’s Liberation Party TA Technical Assistance TAF Technical Assistance Facility ToR Terms of Reference TTI Teacher’s Training Institute TTL Task Team Leader TS Technical Secretariat UN United Nations UNDP United Nations Development Programme UNICEF United Nations Children’s Fund UNIDO United Nations Industrial Development Organisation Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Contents Executive summary ............................................................................................................ i Introduction ................................................................................................................................................i Findings .....................................................................................................................................................i Lessons learned ...................................................................................................................................... vi Recommendations ................................................................................................................................... ix 1 Introduction .........................................................................................................1 1.1 Context and operating environment ......................................................................................... 1 1.2 Objectives and scope of evaluation ......................................................................................... 2 1.3 The methodology of the evaluation .......................................................................................... 3 1.4 Limitations of the methodology ................................................................................................ 5 1.5 Structure of report .................................................................................................................... 6 2 The MDTF-N portfolio and the evaluation project sample .....................................7 2.1 Overview: The MDTF-N ........................................................................................................... 7 2.2 Project sample for this evaluation .......................................................................................... 11 2.3 Project summaries ................................................................................................................. 14 3 Findings .............................................................................................................. 32 3.1 Institutional and organizational relevance .............................................................................. 32 3.2 Cost effectiveness .................................................................................................................. 41 3.3 Development impact .............................................................................................................. 44 4 Lessons learned and recommendations for a way forward .................................. 55 4.1 Lessons learned ..................................................................................................................... 55 4.2 Recommendations for a way forward .................................................................................... 61 Annex 1 Documents consulted.............................................................................. 65 Annex 2 Respondents ........................................................................................... 68 Annex 3 Project site visits ..................................................................................... 74 Annex 4 Interview protocol .................................................................................. 75 Annex 5 Focus group discussion protocol .............................................................. 79 Annex 6 Project Outputs and Results .................................................................... 81 Annex 7 Terms of Reference ................................................................................. 89 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Executive summary Introduction The Sudan National Multi Donor Trust Fund The Sudan National Multi Donor Trust Fund (MDTF-N) was established in April 2005 following the Comprehensive Peace Agreement (CPA) and a donor-supported Joint Assessment Mission (JAM), which assessed the needs of Sudan for the interim period following the peace agreement. The strategy for the MDTF-N is focused on two central goals – the consolidation of peace and pro- poor growth. It is organized around five guiding principles: ■ Support priority national investments to consolidate peace; ■ Support state- and locality-level investment programs with high visibility in war-affected zones in the Northern States and the Three Areas; ■ Focus on making rural development pro-poor with support for micro/small-enterprises; ■ Make decentralization work in the context of reforming the public service and expanding access to basic services; ■ Lay the groundwork for good governance, in particular by opening up the private sector, unleashing the creativity of civil society, and possibly supporting direct interventions in opening up the media and establishing the rule of law. Funding for the MDTF-N has been from 10 donors (including the World Bank), to the value of US$249 million. Counterpart funding from the Government of Sudan (GoS) has brought the value of the portfolio to US$585 million. The portfolio comprises 15 projects, of which eight have closed. All projects are due to close by June 2013. The Independent Evaluation of the MDTF-N The objective of the assignment is to carry out an independent evaluation of the implementation of the Sudan MDTF-N in the period 2006-2012, focusing on lessons learned in terms of: ■ development impact, ■ cost effectiveness, ■ institutional and organizational relevance. The evaluation focused on a sample of seven projects – both closed and open – in five states across each of the key sectors of activity. The evaluation involved a five-week mission to Sudan during which the evaluation team conducted extensive consultations with a variety of stakeholders at the national, state and locality levels, including Government of Sudan officials, World Bank staff, donors, NGO observers, project implementation units, and beneficiaries. These were complemented by telephone and Skype interviews with members of the World Bank team located outside of Khartoum. Over the course of this consultation period, the team gathered relevant information from 39 stakeholder interviews, 29 focus group discussions, 18 project site visits which included discussions with 233 beneficiaries, and site observations. Findings The evaluation team identified two levels of findings. First, there are general findings around the activities and implementation of the MDTF-N as a financing mechanism and the activities surrounding its administration. At the second level, there are specific project-level findings based on the in-depth reviews of the seven projects in the evaluation sample. i Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Institutional and organizational relevance Alignment of priorities between the MDTF-N and the JAM, CPA, and the Government’s Five Year Plan (2006-11) Finding 1: At the design level, the MDTF-N was aligned with the development needs identified in the CPA, JAM, and Five Year Plan. However, in addressing some of the strategic priorities of the CPA – primarily through counterpart funding – the balance of interventions across the portfolio did not necessarily reflect the distribution of development needs identified by the JAM. At the design level, the MDTF-N took into consideration the objectives of these guiding documents in its programming orientation and priorities. It is the general consensus of stakeholders that the MDTF- N’s activities supported the development priorities identified in the JAM, CPA, and Five Year Plan, with a high majority of respondents stating that the MDTF-N was aligned with these documents. Counterpart funding was concentrated on projects which related more to the provisions of the CPA while the donor-funded portion of the MDTF-N related more to the development needs identified in the JAM. This has had consequences for the relative prioritization, in terms of allocation of resources, of the MDTF-N. Most notably, basic social services, with the largest financing requirement by far under the JAM at 61 per cent, was only the third most important cluster in terms of the MDTF-N grant, with 11 per cent of resources – and down in seventh place in terms of combined MDTF-N and GoS funding, with just 5 per cent. Relevance to post-conflict reconstruction in Sudan Finding 2: The MDTF-N was an appropriate aid financing mechanism for the post-conflict environment of Sudan and while success in implementation and impact varied by project, the Fund generally supported activities that responded to the needs of the country. Overwhelmingly, respondents agreed that the MDTF-N, as a financing instrument, was particularly relevant to the post-conflict context of Sudan. Taking into consideration the difficult and challenging operating environment, a multi-donor trust fund presented one of the most efficient and risk-adverse modalities. At the level of operations and project implementation, respondents felt that the MDTF-N addressed the needs of post-conflict Sudan. Some respondents pointed to the fact that in some of the states that were particularly affected by the long-running conflict, the MDTF-N supported development projects that were the first ever to be implemented in some communities. At the project level, particularly in the Three Areas, project selection, design and implementation might have benefited from an initial conflict impact assessment. Response to Sudan’s most-pressing needs Finding 3: The MDTF-N’s design and objectives addressed the most-pressing needs in Sudan; however, the extent to which it was able to deliver response and impact on the ground varied by and within project, as well as location. The majority of respondents agreed that the MDTF-N addressed Sudan’s most pressing challenges, but the extent to which it addressed the needs, and the impact it had, varied by project. A number of projects were designed to be pilot projects, but disappointed high expectations in the limited scale of coverage and impact. Other projects, while designed to have wide strategic impact at the national policy level, were found to have limited impact in implementation due to operational challenges. Support to the most critical institutions Finding 4: MDTF-N Technical Secretariat staff and personnel from the Monitoring Agent provided invaluable direct support, hands-on training, and capacity development to PIU staff, particularly in regulations and procedures related to project management, procurement and financial management. The MDTF-N provided support for implementation through direct capacity development and hands-on training for the project implementation unit (PIU) staff, which were drawn from the personnel of the corresponding ministries and government institutions. The MDTF-N’s primary implementing strategy through partnership with relevant Government of Sudan ministries was generally praised for its capacity development impact. ii Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report The evaluation team observed the MDTF-N Technical Secretariat and the Monitoring Agent working closely with the PIUs and the Ministry of Finance and National Economy (MoFNE) to ensure smooth implementation and to address programming challenges. This type of assistance was praised particularly by PIU members, who described the increase in their professional capacity, particularly related to procurement and financial management procedures. The MDTF-N’s architectural design Impact of rules and structure on intra-operability World Bank rules and structures had a clear negative impact at the start of the MDTF-N, when expectations for some evidence of quick impact were high. Project disbursements and expenditures were delayed, decision-making was slow and there was little evidence of actual or likely impact with which to satisfy government or donors. Government counterparts struggled to comply with World Bank process and procedural requirements, which was compounded by the early staffing problems of the Technical Secretariat. Over time, however, these early challenges delivered rewards in terms of the professional capacity that was built, through the work of both the Technical Secretariat and the Monitoring Agent, particularly in procurement and financial management. These rewards have been explicitly recognized by government stakeholders, helping to mend the credibility that had been dented by the Bank’s initial challenges. The key findings regarding impact of rules and structure are: ■ MDTF-N rules Finding 5: The World Bank procurement and financial management procedures and rules were an impediment for implementation at the beginning of the Fund’s activities, but as capacity improved through hands-on training and assistance from MDTF-N and Monitoring Agent staff, the negative impact subsided. ■ MDTF-N operational structure Finding 6: The fact that in-country staffing was a challenge for the World Bank in the MDTF-N’s early days had a direct and negative impact on the implementation of the Fund’s projects; however, the Bank eventually responded to the human resource and technical support needs and the current capacity of the Technical Secretariat is experienced and adequate for the current level of operations. ■ MDTF-N financing structure Finding 7: Projects dependent on counterpart funding reported delays in disbursement prior to early 2010 that negatively impacted project implementation. Role of the Monitoring Agent Finding 8: The Monitoring Agent plays important monitoring and capacity development roles, and the information and recommendations presented in the Quarterly Reports are integrated into project operations. The MDTF-N Monitoring Agent reviews paperwork related to procurement and finances, monitors projects through site visits, and reports on the status, progress and performance rating of MDTF-N projects on a quarterly basis. They also provide capacity development training and on-the-job assistance both to PIU staff and through other World Bank training activities. The evaluation found that the Monitoring Agent played a vital role in supporting the MDTF-N and PIU operations in the areas of international procurement and financial management regulations and processes. The majority of PIU staff reported positive experiences with the Monitoring Agent. Cost effectiveness Finding 9: The MDTF-N provided a cost-effective mechanism for all donors in aggregate in the early days of its existence. In particular, it has delivered the non-quantifiable benefit of facilitating development partner dialogue in a difficult context. When compared with bilateral donor efforts at the aggregate level, it is likely that the MDTF-N has been more efficient and promoted greater harmonization, particularly in the case of donors which might not have had the capacity or political mandate to implement a large portfolio of development projects. iii Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report The MDTF-N has delivered real but unquantifiable benefits in terms of the negotiating position of the World Bank, based on its politically neutral stance, at a time of difficult relations between the GoS and the international community. It would have been difficult for individual donors, particularly those who are party to the ICC, to continue to work as effectively as the Bank did during that period. Development impact The MDTF-N has delivered top-line development results to June 2012 in terms of: ■ Institutional development and capacity building 3,905 federal and state government officers trained. ■ Economic policy and management Engaged 31 microfinance institutions (MFIs) with reported clientele base of 210,800 business beneficiaries; introduced the Loan Tracking System (LTS) used by 11 local MFIs and one commercial bank; 200 tractors from the Agricultural Bank of Sudan to be distributed to Gum Arabic Production Associations (GAPAs) following the matching-grant procedures the Gum Arabic project put in place for the GAPAs; study of gum arabic completed which identified the gum arabic sector as a commodity well-placed for development of the sector in Sudan. ■ Productive sectors (rural and private sector development) The Livestock Project constructed six livestock markets which traded in 135,469 livestock heads, an increase from 62,837. Support and training to water associations across 28 water resource projects provided access to safe water to 156,547 beneficiaries in four states. 91 community animal workers were trained and deployed to pastoral communities. ■ Basic social services The CDF and BEP have enrolled 160,330 pupils in basic schools in five states; 11,380 teachers trained in English, Arabic, mathematics and core program; ongoing work to institutionalize English language in the curriculum; rehabilitation of 514 basic schools; 114 health facilities rehabilitated; 81 communities provided with basic medical equipment; increase in outpatient consultations from a baseline of 0.16 consultations per person per year (2008-09 estimate) to 0.27 (30 June 2012); increase in pregnant women attending at least one antenatal care consultation from baseline of 48 per cent (2008-09 estimate) to 72 per cent (30 June 2012); 113 village midwives graduating from training school in Q2 12, with an additional 215 who graduated in Q1 12; 407 water resource points developed. ■ Infrastructure 446km of railway line; 365km of road network. ■ Livelihoods and social protection Training of 400 fishermen; establishing an ice factory and fishery union in Blue Nile state; 49 community livestock restocking groups engaged, with increase in livestock offspring from 8,051 to 8,325 in one quarter; support to 26 GAPAs through matching grant sub-projects providing 14 water reservoirs, 12 gum stores, 16 tractors, four water stations and hafirs. The evaluation team found that while the outputs may have been delivered as reported, the actual development impact varied by project, often within different project components, and by location. Finding 10: There were two key positive results arising from the MDTF-N, namely i) the capacity development of GoS, PIU, WB, and MA staff and ii) the MDTF-N’s role as a forum for communication and interaction with the GoS. The hands-on training and development by Technical Secretariat and Monitoring Agent staff in procurement and financial management regulations and procedures employed by the World Bank provided invaluable capacity development for the GoS and PIU staff. Secondly, donor and observer group respondents repeatedly underlined the importance of the MDTF-N as a forum for engagement with the Sudanese government outside the sometimes difficult confines of bilateral relations. Finding 11: Unintended negative results occurred primarily at the project implementation level and were related to project design, the operating context, and the capacity to manage expectations. Negative results occurred primarily at the individual project implementation level. These negative consequences appear to be the result of insufficient analytical work prior to project design, the lack of baseline data, poor project design, challenges in implementation, and the scale of the need. iv Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Finding 12: The three main factors affecting the MDTF-N implementation and results delivery were: the limited number of quality Technical Secretariat staff in the first year, the low capacity of Government of Sudan staff, particularly related to the World Bank’s procurement and financial management procedures, and a volatile security situation. Three primary factors affected MDTF-N implementation and results delivery: ■ In the initial stages of the MDTF-N, there was a very limited number of quality staff at the Technical Secretariat. Additionally, some projects experience a high turnover of Task Team Leaders (TTLs). ■ The lack of capacity on the part of the Government of Sudan, particularly as it related to the World Bank’s procurement and financial procedures, severely delayed project implementation at the beginning of the Fund’s operations. ■ A volatile security context affected project implementation, particularly in the highly prioritized Three Areas of Abyei, Blue Nile, and South Kordofan, where conflict recurred. Security continues to affect operations and monitoring efforts in these areas. The MDTF-N and World Bank Safeguard and Gender Policies Finding 13: The MDTF-N did not include gender directly in its initial project design or monitoring indicators; however, this oversight was later addressed and projects are now reporting success in several areas related to gender inclusiveness and access to opportunity. Compliance with Safeguard and Gender and Development policies was required for all of the MDTF-N portfolio. However, most projects have invoked a special policy for emergency projects that allows environmental assessment and other studies to be carried out after implementation begins. Where safeguards have been applied, a recent review finds that the degree of compliance has been salutary – analysis of alternatives was not carried out and projects were screened according to checklists rather than the rigorous environmental impact assessments, environmental management plans and resettlement action plans prescribed under the Bank’s policies. The review also echoed the findings of this evaluation that public consultation during project preparation was minimal While this level of compliance ultimately had no adverse environmental or social effects, the review finds that the approach of the MDTF-N has been short-sighted, missing an opportunity to put environmental protection and social sustainability on a stronger footing in Sudan during the coming years. Similarly, the MDTF-N succeeded in delivering some positive results on gender despite very limited explicit acknowledgement of gender-related issues in the planning and design. The Basic Education Project and Community Development Fund have been particularly strong in delivering gender-related results. At the organizational level, however, progress in gender inclusiveness, particularly in the staffing of many PIUs, still leaves room for improvement. Role of stakeholders in MDTF-N success Finding 14: The three main stakeholders – the World Bank, the donor community, and the Government of Sudan – each played a separate but significant role in the success of the Fund. ■ World Bank With regards to its role as the fund administrator and Technical Secretariat, the World Bank fulfilled its role, employing a Monitoring Agent to support it in carrying out its fiduciary responsibilities. The Bank provided capacity development and training which were critical to implementation success given the low capacity in Sudan. ■ Donors The donor group participated actively in the Oversight Committee; however, the activities of the Sudan Consortium were less clear. The donors definitely pushed priorities and future directions for the Fund, with an emphasis on delivery of results. ■ Government of Sudan As the primary project implementer and major contributor to the Fund, the GoS played a significant role in its success. As capacity grew, the rate of project implementation increased and there is now a cadre of well-trained civil servants within the GoS as a result of their v Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report experience with the MDTF-N. The GoS also was active in the Oversight Committee, and identified projects and implementing agencies. Sustainability Finding 15: Long-term sustainability varies by project, within project, and by location; however, it is impacted by both negative and positive factors that can be mitigated or enhanced to increase the probability of a project continuing after the MDTF-N closes. Ahead of the scheduled closure of the MDTF-N, the Technical Secretariat undertook an assessment of project sustainability in 2011 that identified potential issues impacting sustainability and presented specific actions designed to strengthen the chances of on-going operations. Of the 13 projects evaluated, seven projects were rated Likely; the sustainability of five projects was rated Uncertain; and the sustainability of one project (Abyei) was considered Unlikely, stemming essentially from poor implementation performance, insecurity in the project area and political uncertainty. The findings of this evaluation generally align with the sustainability rating and key outstanding issues. At the project level, sustainability varies by project and often by and within states. Respondents to this evaluation identified several basic issues impacting sustainability, such as the buy-in and commitment to the project by the government at every level (federal, state and locality); constraints on budgets and resources, particularly at the state and locality level; and the fact that the critical mass needed for self- supporting and sustainable development at the community level has not yet been reached. Conversely, respondents also distinguished positive factors already supporting the sustainability of various projects, including the level of transfer of knowledge and skills to PIU staff within the GoS; the high degree of ownership of some projects like the CDF, especially by some state and locality governments; and the currently ongoing discussion on replication, expansion, and/or the integration of some projects into state/locality and federal budgets. The sustainability study included an outline exit strategy and action plan. A subsequent functionality assessment by the Monitoring Agent, focusing on infrastructure related issues and needs, identified areas for improvement in project activities prior to the closure of the Fund, detailing a required budget for each project. Lessons learned Institutional and organizational relevance Design Lesson 1: Ownership must be built into the design of both the Fund and its projects from the beginning. One of the lessons repeated by respondents was that ownership needed to be present at each level of the Fund’s activities from the design phase onward. At the Technical Secretariat level, this meant that the Government of Sudan needed to be strongly invested in the Fund’s activities and outcomes. At the project implementation level, the ministries supporting the PIUs with staff needed to understand the objectives and value the project added to their operations. Likewise in the states and localities, government buy-in, appreciation for the project’s goals and outcomes, and investment in its success was important for sustainability. Particularly for projects with implementation at the community level, close collaboration with communities is the foundation for success. Lesson 2: Development priorities, programming sectors and project-level activities must be determined by need, using a combined bottom-up/top-down approach. Evaluation respondents generally felt that the JAM and CPA did not provide clear priorities. Although the IOC-N, at its meeting in July 2005, identified indicative priorities, based on government priorities, for the 2005-07 period, it is not evident that this was subsequently formalized and/or clearly communicated to stakeholders. Programming should be needs-based, with a combined bottom- up/top-down approach incorporating all levels of stakeholder and beneficiary perspectives and voices. Lesson 3: Ensure that the Fund and its projects have clear, specific, and attainable objectives and targets. vi Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report One of the weaknesses in design pointed out by Country Portfolio Performance Review in 2010 was the fact that development objectives were broadly defined. Likewise, some projects had very unrealistic and sometimes unrelated goals, and suffered from limited analytical preparation, the absence of baseline studies and difficulty in gathering monitoring data. This impacted the ability to track progress and made it difficult to design and implement activities responsive to those broad objectives. Lesson 4: When designing projects, be aware of the appropriate time frame for the targeted activity and capacity context, and allow sufficient time for sustainable impact. Implementation of MDTF-N projects was frequently delayed, resulting in several extensions. In some cases, this was due to lack of implementing capacity; in others, outside factors delayed the project. Also, the objective of the project needed to be taken into consideration when establishing the project timeline. Projects that first require changing attitudes and procedures will need a longer implementation period than projects concentrated on physical infrastructure. Lesson 5: Design the Fund structure and individual projects to fit the operating context. When designing a fund and its projects, a thorough assessment and understanding of the context must be incorporated into both the structure of the fund’s operations and those of its project portfolio. This would include building in time for intensive capacity development for PIU staff prior to project implementation; extending the project cycle to account for anticipated delays in implementation; and tasking a Monitoring Agent to provide hands-on support to PIU staff throughout project implementation. Lesson 6: Incorporate environmental, social and gender issues into the design of the Fund and its projects from the beginning. Despite limited compliance with environment, social and gender policies, the MDTF-N has had no discernible negative environmental or social impact; and has in fact delivered some positive gender outcomes. It was a missed opportunity that these policies were not more robustly implemented from the outset. Lesson 7: Identify the appropriate managing unit for each project. It was the experience of some projects that the decision of where to situate the PIU seemed to be fraught and politically motivated, resulting in problems with implementation. To the extent possible, the responsible partner for the PIU should be identified on the basis of appropriateness rather than on political or personal motivations in order to enhance smooth delivery of project outputs. Lesson 8: Build from existing structures – do not reinvent the wheel. Project design and the selection of implementing partners works best when existing and functioning structures are relied upon. In this way, duplication is avoided, allocative efficiency of resources is ensured and national counterparts have the incentive to preserve existing structures that work and strive to improve them. Ownership is underpinned and sustainability strengthened. Lesson 9: Determine the capacity of counterparts and implementing partners during the design phase. Government capacity and the capacity of implementing partners needs to be assessed at the very start of an intervention, and appropriate mechanisms for capacity building and support must be determined in relation to the context and existing needs. Incorporating capacity development into program and project design increases the probability of success and ensures that lessons learned and successful aspects of the project will be incorporated into national structures and will continue to be capitalized upon following closure. Operations Lesson 10: Anticipate, monitor and manage expectations. One of the primary lessons is the need to manage expectations early. This applies to the Fund as an instrument and to individual projects. The mix of ‘quick impact’ with longer-term projects is a common approach to expectations management, as a means of generating buy-in and ownership early on through the delivery of short projects targeting ‘low-hanging’, highly visible benefits while the more institutional projects are in the vii Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report process of building momentum. The design of the MDTF-N did this only weakly. The ‘quick start’ projects were limited to those in the Three Areas, where the risk of disruption to projects was high. The experience of the MDTF-N has shown that traditional interventions in health, education and small community development projects, where implementation modalities are familiar and relatively straightforward, can yield tangible and visible results within a relatively short timeframe. Had the MDTF-N been front-loaded with more of these kinds of projects, early expectations might have been satisfied to a greater extent. Lesson 11: It is important to commit a sufficient number of qualified staff from the onset for best implementation. One of the main criticisms of the World Bank’s management of the MDTF-N in the early days was that it did not provide a sufficient number of qualified staff in-country from the beginning. There were understandable reasons for this, but in future, the Bank’s management should place a higher priority on providing a full complement of staff in a timely manner. Lesson 12: More could have been done to increase the visibility of the GoS role as a Fund supporter and project implementer. The Government of Sudan provided 57 per cent of the MDTF-N’s total funds, yet most beneficiaries met during the project site visits and even many of the PIU staff were not aware of the extent of the government’s involvement. The role and extent of the government’s financial support for the MDTF-N could have been strengthened. Lesson 13: Donors and other stakeholders could play a positive and active role in the project monitoring visits. Donors were very active in the Oversight Committee meetings and provided clear guidance on areas they supported; however, there was room for greater involvement in the monitoring missions and field visits. Regular donor participation in project site visits provides a first-person view of the development activities and impact which cannot be conveyed on paper. Implementation Lesson 14: Partnerships make a difference in implementation. Respondents repeatedly pointed to the positive impact that partnerships have on implementation. These partnerships include other MDTF-N projects, relevant government organizations at the state and locality level, international organizations and NGOs, and local civil society organizations, amongst others. And, particularly with community-led development implementation, a strong partnership with community members is key for success. Lesson 15: Intra-project collaboration should be based on similar project objectives and implementation strategies. Intra-project collaboration can be both positive and negative and should be undertaken when projects share similar programming objectives and implementation strategies. The partnership between CDF and the Gum Arabic and BEP projects illustrates this point well. Lesson 16: Projects can be well-designed and well-organized, but negatively impacted by the operating environment and other aspects outside of the PIU’s control, so flexibility is key. Flexibility by staff is key and a willingness to shift project focus and learn from experiences is vital. By all accounts, Bank management of the MDTF-N did demonstrate a relatively high degree of flexibility. Its response to its early staffing problems and major restructuring of a large portion of the portfolio, including reallocation of resources across projects, are indicators of a willingness to adjust to emerging realities in a rapidly changing operational context. Cost effectiveness Lesson 17: The cost effectiveness of a multi-donor trust fund relates primarily to its relevance to a given operational context; the MDTF-N has enabled donor engagement through a difficult period, and built national capacity to support future donor programming, although the relative cost of this is not clear. viii Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report A multi-donor trust fund is clearly a cost-effective alternative in a context in which it is nearly impossible for donors to operate bilaterally. The MDTF-N has helped to deliver softer benefits which have been somewhat catalytic – the provision of an entry point for donors, the development of project formulation and implementation capacity, and the facilitation of a platform for dialogue with government. Development impact Lesson 18: The capacity development role played by the World Bank and Monitoring Agent was significant, positive impact of the MDTF-N. A majority of the respondents repeatedly praised the capacity development role played by the World Bank and Monitoring Agent. Several stated that these sorts of activities and ‘hand-holding’ were critical in a context of extremely low capacity in Sudan. These activities made a real and lasting impact for the PIU staff. Lesson 19: Development impact is not immediate – it must be given time. The unrealistic expectations for quick development impact and progress did not take into account several key factors that make quick impact impossible. In a post-conflict context like Sudan, while some immediate results for projects in education or transportation could be seen quickly, other projects – particularly those that require changes in attitudes or practices – take much more time and involve greater effort and involvement to ensure successful implementation and long-term sustainability. Lesson 20: Policy reform is the foundation of sustainable change and can have wide-ranging positive impacts. Some of the MDTF-N projects lobbied for key policy reform and strategy formulation as the foundation for their efforts. In this way, efforts at the national level to revise policies can have dramatic impact at the local level and can be successfully coordinated with community-level implementation. Recommendations 1. It is recommended that the World Bank remain engaged in Sudan. Given the enormous development challenges facing Sudan, there is certainly the need for the World Bank to remain fully engaged in Sudan, if not through further funding to the MDTF-N, then through other funding mechanisms managed by the World Bank. Considering the time, financial resources, and human effort expended to arrive at the present point, it would be a massive waste not to capitalize on this experience to further Sudan’s development goals. 2. It is recommended that donors continue to support development activities throughout the whole of Sudan, not only in specific areas. The development needs are great throughout the whole of Sudan. There is room for continued engagement by donors. A trust fund offers a good option to donors at this time, as it allows them to support development in the country without a large footprint. 3. The MDTF-N Technical Secretariat should follow up the Sustainability and Functionality Reports with a clear action plan for sustainability. Regular discussions about sustainability should continue; however, more pressing is the need for the Technical Secretariat and each open project to outline a clear action plan for the remaining months, prioritizing actions that will strengthen sustainability. 4. Should further funding of the MDTF-N or a new fund be established, the objectives and programming areas should be streamlined, potentially following a sector-based approach. The current portfolio, which responded to the development sectors identified by the JAM, was wide-ranging in its scope. Any future programming should be more streamlined and targeted, aligned with the development priorities from the PSRP, as well as sector policies and strategies for the coming five years. 5. Clearly prioritize development activities from the outset. ix Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Future efforts need to be based upon clearly identified development priorities (see Recommendation 4 above). Where national development strategies fail to provide adequate guidance (as may be the case with the 2012-16 Five Year Development Plan), there is a need for a preliminary coordinated donor exercise to agree, in consultation with the beneficiary government, programming priorities. 6. Do not rush the design and implementation of a fund and its project portfolio. Any future fund should allow sufficient time for information-gathering, discussions in-country with a variety of stakeholders, and design. Likewise, a three-year project cycle is too short for a challenging context like Sudan, thus more time should be allowed for implementation and impact. 7. Conduct risk assessments and incorporate risk into the initial design of the project, and identify potential mitigating measures with close follow-up and flexibility to allow appropriate responses to changes throughout the project implementation cycle. Rather than avoiding risk, projects should be flexible and responsive to it. Conducting a risk assessment prior to the establishment of a fund, as well as risk assessments at the project level if projects are to be implemented in particularly risky areas, is essential. The findings and recommendations of these assessments should then be incorporated into the design and implementation at both the fund and project levels. 8. When operating in conflict-affected areas, conflict impact assessments should be included in project design and projects should be responsive to conflict and its causes in both design and implementation. Similarly, for projects implemented in conflict-affected areas, a conflict impact assessment should be conducted at the design stage of the project and mechanisms for response to conflict and its causes should be included in design and implementation. 9. Incorporate sustainability measures in the design phase of both the fund mechanism and individual projects. A fund and its projects should incorporate clear procedures to ensure sustainability from the design phase. Steps to strengthen long-term sustainability include: ensure government and beneficiary buy-in at all levels relevant to the fund and projects from the beginning; include the targeted beneficiaries in the project design process in order to accurately capture and address their perspectives and needs; locate the PIU within an appropriate government body with a unambiguous plan for the government to assume funding responsibility on a mutually agreed schedule; work within partnerships with other organizations for implementation, including the appropriate civil society groups, NGOs, and international organizations. x Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 1 Introduction The Sudan National Multi Donor Trust Fund (MDTF-N, or ‘the Fund’) was established in April 2005 following the Comprehensive Peace Agreement (CPA) which effectively ended 20 years of conflict in the country. The MDTF-N’s programming priorities are based on the Joint Assessment Mission (JAM), which assessed the needs of Sudan for the interim period following the peace agreement, as well as the Framework for Sustained Peace, Development, and Poverty Eradication that was jointly developed with Sudanese government authorities. The goals of the MDTF-N are centred on the consolidation of peace and pro-poor growth in an effort to reach the Millennium Development Goals (MDGs) and ensure a sustainable peace in the country, focusing many of the project activities on the Three Transitional Areas: Abyei, South Kordofan and Blue Nile. Working primarily through government counterparts, the MDTF-N has implemented a portfolio of 15 projects worth US$584.63 million1 since 2006 over a broad spectrum of sectors, including infrastructure, health, education, community development, agriculture, and public sector reform. With eight of the MDTF-N’s projects closed and seven remaining in the final stages of implementation, the MDTF-N is projected to close on 30 June 2013. In light of the MDTF-N’s closure, the Oversight Committee and staff of the Fund’s Technical Secretariat (TS) commissioned an independent evaluation to capture and preserve the institutional knowledge, experiences, capacity, and lessons learned by the various stakeholders involved with the operation of the MDTF-N since its inception. This report presents the findings of that evaluation, explores the lessons learned as conveyed by the numerous stakeholders involved in the Fund at national, state, and locality level, and offers recommendations for the remaining operations, as well as a road map for the post-MDTF-N environment. 1.1 Context and operating environment The MDTF-N was established in a challenging and unpredictable post-conflict environment in a country transitioning from decades of war. The Fund was designed as a financing mechanism to support Sudan as it tackled the challenges of underdevelopment and endemic poverty during the period while it emerged from conflict. 1.1.1 The impact of conflict Since its independence in 1956, Sudan has experienced two almost consecutive civil wars spanning over five decades with only a 10-year period of relative tranquillity between 1972 and 1983. Simultaneously, other armed conflicts based on ethnic, religious, economic and political elements have occurred throughout the country. One of the primary impacts of conflict on Sudan has been a decline in service provision and a significant lack of investment, leading to underdevelopment throughout much of the country. The second Sudanese civil war (1983 – 2005) ended with the signing of the Comprehensive Peace Agreement between the Government of Sudan (GoS) and the Sudan People’s Liberation Party (SPLM). The CPA established the framework for peace in Sudan in accordance with six agreements (protocols) and the Implementation Plan. The protocol on wealth sharing addressed Sudan’s needs for infrastructure, human resources, sustainable economic development and the capacity to meet human needs within a framework of transparent and accountable government. The six-year interim period since the signing of the CPA has been turbulent. Interstate conflict between north and south, as well as ongoing conflict in Darfur, have added to the existing obstacles vis-à-vis development efforts. In the international sphere, in July 2008, the International Criminal Court (ICC) issued an arrest warrant for Sudanese President Omar al-Bashir, for genocide, crimes against humanity, and war crimes in Darfur. This event fundamentally changed the MDTF-N’s 1 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 1 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report program context, creating an environment of uncertainty and speculation and monopolizing the attention of both the donor community and government partners. A 2009 review of the MDTF-N2 reported that the MDTF-N’s program environment had deteriorated significantly against the context of violent conflict in Darfur and the ICC decision. In January 2011, in accordance with the CPA, a referendum was held in southern Sudan to determine whether it should become independent. On 9 July 2011, the Republic of South Sudan became an independent state. South Sudan’s independence has triggered new violence between north and south, focused on the unresolved disputed areas – among them Abyei, South Kordofan and the Blue Nile. A significant portion of the MDTF-N portfolio has, therefore, been subject to renewed instability and insecurity in the final year of operation. 1.1.2 Current economic and development climate Despite international and national efforts to accelerate development through appropriate programming there is still ‘considerable inequality between the centre and the periphery’3. This has historically contributed to interstate conflict and been the result of conflict itself, as development agencies were unable to establish their presence in the country for years following Sudanese independence and during the first civil war, while the positive advances made in the brief 10-year period between the two civil wars were undermined following Sudan’s debt default. The World Bank withdrew from Sudan in the early 1990s. The result of these conditions has been a persistent lack of capacity, weak infrastructure, and limited provision of basic services, especially in periphery areas where underdevelopment is significant particularly when compared with Khartoum. The secession of South Sudan has also taken its toll on the Sudanese economy because three quarters of former Sudan’s production of oil came from that area4. In January 2012, South Sudan halted oil production following a disagreement with Khartoum over transit fees. This decision had grave consequences for the Sudanese economy, including inflation, currency fluctuations, and market stagnation, impacting severely on the most vulnerable. Additionally, recent austerity measures devised by the state with the support of international financial institutions have triggered demonstrations and produced further tension in border areas which have been historically disadvantaged. The recent deal between Sudan and South Sudan on resuming oil production has the potential to alleviate some of the pressure on the Sudanese economy and provide increased revenue for the country. 1.2 Objectives and scope of evaluation The objective of the assignment is to carry out an independent evaluation of the implementation of the Sudan MDTF-N in the period between 2006-2012, focusing on lessons learned in terms of: ■ development impact, ■ cost effectiveness, ■ institutional and organizational relevance. The evaluation has covered a sample of seven projects – both closed and open – in five states5 across each of the key sectors of activity, focusing particularly on these areas: ■ alignment of priorities between the JAM, CPA, the Government’s five-year plan (2006- 11), and the MDTF-N portfolio; ■ the architectural design of the MDTF-N, including the institutional set-up, management organization, and the use of the Monitoring Agent; 2 Scanteam (2007). Review of Post-Crisis Multi-Donor Trust Funds. Available at: http://www. norad.no%2Fen%2F_attachment%2F107611%2Fbinary%2F6064%3Fdownload%3Dtrue&ei=m6qKUKXKN- jL0QWSm4CgDQ&usg=AFQjCNGOamTuSyGWeupMLm1CWMzm-gXwfw&sig2=7SVB1uf7C_zeSfG_re_-HQ 3 World Bank (n.d.). Sudan Country Brief. Available at: http://go.worldbank.org/7TIDMM4L80. 4 World Bank (n.d.). Sudan Country Brief. Available at: http://go.worldbank.org/7TIDMM4L80. 5 A detailed explanation of the sampling process is provided in Section 2.2. 2 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report ■ the roles played by the Government of National Unity (GoNU) and donor partners in contributing to program success; ■ the results achieved by the MDTF-N; ■ cost effectiveness of MDTF-N; ■ sustainability of activities post-MDTF; and ■ lessons learned and perspectives for the post-MDTF phase, including options to consolidate and/or scale up successful projects after closure of the MDTF-N, and to initiate development programs that are aligned with new government priorities as set forth in the I-PRSP. 1.3 The methodology of the evaluation Throughout the evaluation, the team employed a methodological and technical approach that utilized a mixture of rigorous qualitative and quantitative methods consisting of four separate and inter-related research methods: ■ a document analysis and literature review; ■ semi-structured interviews with key stakeholders from the World Bank, the Sudanese government, implementing partners, donors, and members of civil society; ■ focus group discussions with project staff and beneficiaries during project site visits; and ■ observations during project site visits. These methods were selected to enable the collection of primary and secondary data that has sufficient depth and breadth to answer the evaluation questions. The methods also interlink, which permitted the evaluation team to triangulate the data in order to produce a verifiable body of evidence. This type of mixed-methods approach provided the team with clear and substantive information concerning the implementation, impact, and cost effectiveness of the MDTF-N and the sampled projects, allowing them to make targeted recommendations related to project sustainability as well as future programming strategies following the projected close of the Fund in June 2013. The team employed a participatory approach during the collection of the primary data in the field research phase, enabling program stakeholders to play an active role in shaping the design of the methodology6. Close consultation with relevant World Bank staff, government officials at the federal, state, and local levels, the Monitoring Agent, donors, project coordinators, implementing partner staff, and project beneficiaries was essential during the data collection process. Finally, the evaluation team sought their close collaboration in completing the design and refinement of the methodology and data collection tools. This inclusive, participatory approach continued throughout the evaluation through the provision for feedback and reporting in order to disseminate the findings and build consensus among stakeholders, while maintaining the independent nature of this evaluation. The activities of the evaluation were carried out in four phases as depicted in Figure 1.1. 6 See, for example, Barakat, S., Chard, M., Jacoby, T., Lume, W. (2002) ‘The Composite Approach: Research Design in the Context of War and Armed Conflict’, Third World Quarterly, Vol. 23, No. 5, pp. 991-1003; Connolly, D. (2010) ‘Participatory Research in Programme Evaluation: the Mid-term Evaluation of the National Solidarity Programme in Afghanistan’, in R. Bowd and A. Ozerdem (eds) Participatory Research Methodologies in Development and Post-Disaster/Conflict Reconstruction, London: Ashgate 3 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Figure 1.1 Structure of methodology PHASE I: INCEPTION/ PHASE II: CONSULTATION PHASE III: ANALYSIS & PHASE IV: FINAL PREPARATORY REPORT DRAFTING REPORTING Literature/document Field visits Analyse field research Review and feedback search Stakeholder interviews evidence Report finalisation Review project Beneficiary interviews) Data compilation Formal submission performance Follow-up interviews Prepare Draft Report documentation Identify/contact Stakeholders Develop data gathering tools for field visits Finalise methodology Inception report Draft report Final report Phase I: Inception/preparatory phase (22 July – 29 August 2012) The inception phase included the desk-based literature review of documents and other background materials provided by the World Bank and available electronically. Throughout this period, the evaluation team developed and verified the final methodology, including project sampling strategy, evaluation framework, and the interview and focus group protocols. Preliminary conversations were undertaken with identified stakeholders, including trust fund administrators and Government of Sudan project partners. These comprised brief discussions with the former chair of the Oversight Committee, the Director of the MDTF-N at the International Cooperation General Directorate of the Ministry of Finance and National Economy (MoFNE), the project coordinator of the Gum Arabic project, the current Blue Nile Emergency Project Coordinator and the former Coordinator of the Decentralized Health System Development Project. Emails were also exchanged with major bilateral donors in the preparation for interviews during Phase II. This phase culminated with the submission of the Inception Report. Phase II: Consultation (30 August – 5 October 2012) During this phase, the evaluation team conducted extensive consultations in Sudan with a variety of stakeholders at the national, state and locality levels, including Government of Sudan officials, World Bank staff, donors, NGO observers, project implementation units, and beneficiaries. They also conducted telephone and Skype interviews with members of the World Bank team located outside of Khartoum. Over the course of this consultation period, the team gathered relevant information from 39 stakeholder interviews, 29 focus group discussions, 18 project site visits which included discussions with 233 beneficiaries and site observations. Table 1.1 summarizes the activities conducted during this period. Table 1.1 Summary of field team consultations Project Sites visited Consultations TS staff, officials Beneficiaries with PIU staff and other stakeholders MDTF-N Technical Secretariat n/a n/a 46 n/a Basic Education Project (BEP) 3 6 3 23 Blue Nile Emergency Start-Up 2 6 20 18 Project (BNEP) Capacity Building of the 0 1 0 0 National Judiciary (Judiciary) 4 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Project Sites visited Consultations TS staff, officials Beneficiaries with PIU staff and other stakeholders Community Development 8 19 8 169 Fund (CDF) National Emergency Transport n/a 5 1 0 Rehabilitation (NETREP) Public Sector Reform, n/a 5 1 0 Decentralization, and Capacity Building Program Support Project (PSCAP) Revitalization of the Sudan 5 7 2 23 Gum Arabic Production and Marketing Project (GA) Phase III: Analysis and report drafting (6-31 October 2012) Following the evaluation team’s conclusion of the consultations in Sudan, they conducted an in-depth analysis of the data obtained during the field research period, in combination with the initial findings from the literature review and initial consultations in Phase I. The team also concluded telephone interviews with World Bank staff and in-person discussions with Government of Sudan officials that had not been conducted previously due to the respondents’ unavailability. The submission of the Draft Evaluation Report concluded this phase. Phase IV: Final reporting (1 November 2012 to close) In the final reporting phase, the evaluation team has prepared the Final Report, incorporating feedback and clarification received from the World Bank and other stakeholders. 1.4 Limitations of the methodology The evaluation team encountered five main challenges during the consultation phase in Sudan that impacted their ability to implement the methodology exactly as it was outlined in the Inception Report. Despite these, the team adapted to the changes in circumstances and took steps to ensure that they obtained data of sufficient depth and breadth. 1. The primary challenge that the team faced was their inability to travel to South Kordofan, where all seven of the projects in the evaluation sample had activities, due to ongoing security restrictions. In order to gather as much information as possible about the state’s current situation and MDTF-N project activities there, the evaluation team interviewed staff of Project Implementation Units (PIUs) and Local Implementation Units (LIUs) who had worked or are currently working there, community leaders, South Kordofan members of parliament, and state government officials. However, these efforts do not negate the fact that the team was unable to conduct project visits, site observations, or focus group discussions with beneficiaries, and interviews with other MDTF-N project staff and local stakeholders in South Kordofan. As a consequence, there is limited field data from the state. The information about activities in South Kordofan provided in the project summaries in Section 2 is limited to what is available in World Bank and project reports, as well as what the team was able to gather in Khartoum. 2. Additionally, the team encountered delays during the security clearance application process which required them to reduce the amount of time they were able to spend in Kassala. The team addressed this challenge by ensuring a full schedule of meetings and project visits while in Kassala, but some NETREP and Judiciary project sites were too remote to reach in the shortened timeframe. The team discussed the Judiciary and NETREP projects with stakeholders in Kassala in an effort to triangulate the information 5 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report received in Khartoum. While the delay in the security clearance process was unfortunate, it did provide the evaluation team with a better understanding of the operating environment in Sudan, particularly for international organizations and non-Sudanese staff. 3. Thirdly, the local consultants who were originally part of the evaluation team were unable to continue to participate in the evaluation – one for health reasons, the other due to prior professional commitments – reducing the team by half. As a result, the team was unable to conduct longer simultaneous field visits as proposed in the Inception Report; and reduced the number of days spent in Sennar and North Kordofan states. These two field visits were conducted by the Team Leader, assisted by a local support staff, rather than with a team of researchers, as originally planned. While there was only one project from the sample to evaluate in Sennar and sufficient information was gathered in the two-day field visit, it would have been optimal to have a second team member and an additional day or two during the North Kordofan visit, particularly since it was difficult to organize meetings in El Obeyed and some meetings had to be removed from the schedule. Despite the unforeseen obstacle of reduced staffing, the team did everything possible from their side to ensure that the maximum number of project visits, focus group discussions, and interviews were conducted in the time allotted. 4. The fourth challenge occurred when a supervisor at UNDP was unwilling to permit staff who had worked on the Judiciary project to discuss its implementation with the evaluation team, despite official and repeated requests. This is inconsistent with the stance of UNDP’s executive and MDTF management team, who were more than willing to share their general experiences and knowledge. As a result, the information obtained about the implementation of the Judiciary project is limited to interviews conducted with individuals not directly involved in its broader implementation, and information obtained from UNDP’s website. The lack of the perspective of UNDP staff directly involved in the project is unfortunate as UNDP continues to be active in the Judiciary sector and the organization’s experience could have provided important lessons around follow-on projects. 5. Finally, due to some respondents’ lack of availability, the team was unable to schedule meetings with several key Government of Sudan officials who are involved in the MDTF- N at various levels. The team made every possible attempt to meet with the key GoS stakeholders, but despite their best efforts, they were only able to interview six former and current senior GoS staff. Thus, the information gathered on the MDTF-N experience and lessons learned from the government’s point of view is considerably less than that of other stakeholders. At field level, however, there was close engagement with GoS staff involved at the operational level of projects. 1.5 Structure of report Following this introductory chapter, Section 2 provides a summary of the MDTF-N project portfolio, examining the seven projects selected for this evaluation in greater depth. Section 3 outlines the key findings along the three main areas of this evaluation, while Section 4 then explores the lessons learned with specific actionable recommendations for the current projects reviewed during the evaluation structured around the three areas of evaluation. The conclusion, Section 4.2, focuses on the post-MDTF-N environment, outlining possible strategies and approaches for future development efforts following the conclusion of the remaining projects on 30 June 2013. 6 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 2 The MDTF-N portfolio and the evaluation project sample This section, divided into three sub-sections, presents a general overview of the MDTF-N’s portfolio, explains the sampling strategy employed by this evaluation, and provides detailed summaries of the seven projects evaluated in-depth by the evaluation team during the consultation period. Section 2.1 outlines the goals and guiding principles of the Fund, the composition of its funding sources, and its performance to date as measured by disbursement and expenditures. Section 2.2 describes the sampling strategy and explains how the projects were selected for greater analysis, while Section 2.3 goes into detail about the activities, progress, and impact of the projects in the evaluation sample. 2.1 Overview: The MDTF-N The Sudan National Multi-Donor Trust Fund is a US$584.63 million7 portfolio of reconstruction and development projects implemented since 2006 by the World Bank. While many of the project activities supported by the Fund were concentrated in the conflict- affected areas of Sudan – particularly Abyei, Blue Nile, and South Kordofan – some of the portfolio’s projects addressed development at the national or strategic level. The Fund has been financed by nine bilateral donors, prominent among them the Netherlands, the UK and Norway, for a total of US$249 million8. The GoS has provided approximately US$335 million, 57 per cent of the total Fund budget. Originally due to close by December 2011, in October 2010 the OC agreed to an extension to December 2012. A further extension to December 2013 was requested and currently all projects are due to close by June 2013. 2.1.1 MDTF-N goals and principles The strategy for the MDTF-N is focused on two central goals – the consolidation of peace and pro-poor growth, both to help Sudan reach the Millennium Development Goals, and as a means to make peace sustainable. The strategy is organized around five guiding principles: ■ Support priority national investments to consolidate peace; ■ Support state- and locality-level investment programs with high visibility in war affected zones in the Northern States and the Three Areas; ■ Focus on making rural development pro-poor with support for micro/small-enterprises; ■ Make decentralization work in the context of reforming the public service and expanding access to basic services; ■ Lay the groundwork for good governance, in particular by opening up the private sector, unleashing the creativity of civil society, and possibly supporting direct interventions in opening up the media and establishing the rule of law. 2.1.2 MDTF-N donors Funding for the MDTF-N, as well as the MDTF in South Sudan (MDTF-SS), was pledged in two stages. At the first Oslo Donors’ Conference in April 2005, 10 donors (including the World Bank) pledged US$222.9 million and at a second conference in Oslo in May 2008, seven donors pledged US$29.4 million. At the end of October 2012, a total of US$249 million had been pledged to the MDTF-N9. The World Bank, Italy, and Greece pledged contributions 7 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 8 The total donor commitment was reduced from US$252 million in Q1 12 due to the cancellation of funds following the restructuring of the NETREP project in June 2012. Information obtained from the Monitoring nd Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 9 Ibid. 7 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report only at the first conference; and the Netherlands has recently withdrawn from the Fund, although it remains the largest bilateral donor to the MDTF-N. Figure 2.1 Bilateral donor contributions to the MDTF-N, cumulative to June 2012 5% 3% 6% Netherlands 7% 39% UK Norway Canada Sweden 17% Spain Other* 23% * Other: World Bank (Oslo I only), Italy (Oslo I only), Iceland, Greece (Oslo I only) Earmarking of donor funds for specific components, activities, or programs is not permitted; however, donors could indicate a preference for their contribution to be matched against a broad category of expenditure (e.g. JAM clusters). 2.1.3 MDTF-N project portfolio The MDTF-N portfolio comprises 15 projects of which eight have closed (Table 2.1). A number have been extended, up to three times, several with additional financing. Table 2.1 MDTF-N project portfolio Value MDTF-N Original Closing (US$m) component grant date (US$m) agreement ONGOING PROJECTS Basic Education Project 15.0 15.0 8-Jul-09 31-May-13 Revitalizing the Sudan Gum Arabic Production and Marketing Project 7.0 7.0 29-Jun-09 31-Dec-12 Livestock Production and Marketing Project 10.3 6.6 29-Aug-07 31-Dec-12 Sudan Microfinance Project 21.8 11.8 22-May-07 31-May-13 National Emergency Transport Rehabilitation Project 133.4 64.7 29-Oct-06 30-Jun-13 Decentralised Health System Development Project 15.2 12.0 29-Oct-06 31-Dec-12 Community Development Fund 95.3 52.8 16-Jan-06 30-Jun-13 CLOSED PROJECTS Abyei Start Up Emergency Project 6.0 6.0 29-Jun-09 30-Jun-11 Public Sector Reform, Decentralisation and Capacity Building Project 4.8 2.6 11-Sep-08 30-Jun-12 8 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Value MDTF-N Original Closing (US$m) component grant date (US$m) agreement Blue Nile Start Up/Emergency Project 18.1 10.1 3-Dec-07 30-Jun-12 South Kordofan Start Up Emergency Project 10.6 7.7 25-Jul-07 30-Jun-11 Sudan New Unified National Currency Project 145.2 25.2 22-May-07 30-Dec-08 Capacity Building of the Sudan Judiciary 9.6 4.6 23-Aug-06 30-Jun-09 Fifth Population Census Project 86.9 18.4 8-May-06 30-Jun-09 Technical Assistance Facility 5.6 4.9 16-Jan-06 31-Jan-10 TOTAL (US$m) 584.6 249.2 With the exception of three projects, the portfolio is recipient-executed by ministries of the GoS. The start-up/emergency projects in Blue Nile and South Kordofan included components implemented with the assistance of UNICEF and UNIDO; and the Capacity Building of the Sudan Judiciary project was implemented by UNDP. Rather than long-term development interventions, the MDTF-N’s activities were designed to be pilot projects that, alongside the analytical work undertaken by the Fund, aim to provide the government with credible evidence for future policy reform and development strategies. 2.1.4 Project disbursements The performance of the MDTF-N has been mixed. Independent reviews have highlighted significant problems with human resource capacity in the early stages which contributed to a very slow start and limited disbursements and expenditure. By the end of June 2012, however, disbursements rates across the portfolio were high, with an average for both the MDTF-N and the government components together of 95 per cent. For the MDTF-N component, disbursement rates ranged from 30 per cent, for the Abyei Start Up Emergency project10, to 100 per cent and averaged 86 per cent. For the government component, there was significant variation, with disbursement rates ranging from 5 per cent for the Capacity Building of the Sudan Judiciary project to 255 per cent for the state component of the South Kordofan Start Up Emergency Project, giving an average of 95 per cent. 10 The Abyei project was technically closed on 30 June 2011, due to the insecure situation on the ground. The OC meeting in May 2012 confirmed the project’s official closure. 9 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Table 2.2 MDTF-N disbursement rates, 30 June 2012 160 140 120 100 80 60 40 20 0 MDTF-N GoS Total Over the past year, disbursements have been on a declining trend with a significant drop from US$10.2 million in Q3 11 to US$5.6 million in Q4 11. In Q1 12, actual disbursements were US$4.4 million, just 35 per cent of forecast. They rose to US$8.1 million, with the portfolio achieving an overall disbursement rate of 70 per cent in Q2 12, but only one of nine projects met the planned disbursement level for the quarter and four projects had 11 disbursement rates of less than 50 per cent of their projected targets . This lower disbursement rate is set against a context of considerable uncertainty relating to restructuring of projects, as well as other extensions and reallocations of budget (including across projects). In addition, continued insecurity in Blue Nile and South Kordofan, low disbursement and absorption of counterpart funds (in the NETREP and CDF projects, for example), and the limited use of funds disbursement forecasting tools have all had a constraining effect on disbursement of funds. 2.1.5 Expenditure Expenditures to the end of June 2012 amounted to US$493 million, 84 per cent of commitments. Across seven ongoing projects, 79 per cent of commitments had been spent12. 11 NETREP slightly exceeded its disbursement target for Q2 12, while SMDF achieved only 47 per cent, BEP 46 nd per cent, ILPM 41 per cent, and PSCAP 14 per cent. Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 12 Excluding additional components funded by IFAD and DFID. 10 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Table 2.3 Expenditure as % of commitments for active portfolio, June 2012 160 140 120 100 80 60 40 20 0 MDTF-N GoS Total The Abyei Start Up Emergency project was closed with an unspent balance of US$4.3 million. US$2.2 million from this project was reallocated to the Livestock Production and Marketing and Decentralized Health System Development projects. The Public Sector Reform, Decentralization and Capacity Building project was operationally closed at the end of June 2012 with expenditures at 67 per cent of the total budget (91 per cent for the MDTF- N and 40 per cent for government) and 96 per cent of disbursements. Outstanding payments were due to the end of October 2012. The Blue Nile project also closed at the end of June 2012, with expenditure of 69 per cent of budget (across all components of the grant – 98 per cent for the MDTF-N and 34 per cent for government) and 65 per cent of disbursements (103 per cent for the MDTF-N; 27 per cent for government). The MA reports that attention was expected to focus on disbursements to ensure that the entire additional financing grant was fully liquidated by the disbursement deadline date on 30 October 2012; and that the project continued to make payments for implemented activities during the grace period ending 31 October 2012. 2.2 Project sample for this evaluation As outlined in the Terms of Reference (attached as Annex 7 to this report), the team selected a sample of the Fund’s 15 projects for in-depth evaluation and project site visits in order to explore further key issues identified during the desk review. The information gathered during the field visits and discussions with Technical Secretariat staff, project implementers, beneficiaries, and other stakeholders provided the basis upon which the evaluation team built the findings presented in Section 3 of this report, as well as the lessons learned and recommendations outlined in Section 4. 2.2.1 Selection of projects When selecting the projects to be examined in this evaluation, the evaluation team relied on a multi-layered cluster sampling strategy primarily based on three key elements: ■ Geographical scope ■ Project status ■ Monitoring Agent performance rating This purposive approach was driven by the need to balance several factors, including access, ensuring broad sectoral coverage, and MDTF-N goals, which would not have been afforded through random sampling. 11 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Geographical scope The project selection was guided by practical and logistical considerations that informed the itinerary for the field visits. Within the limits imposed upon the evaluation team by unforeseen delays in security clearance approval and restrictions on travel to South Kordofan, the project selection was structured to ensure optimum coverage of a large area within a limited period of time. Status and Monitoring Agent performance rating Secondly, projects were selected on the basis of their status – whether ongoing or closed – and their most recent overall performance rating conferred by the Monitoring Agent. Table 2.4 below organizes the 14 projects implemented in the four states by their status, and performance rating13. Table 2.4 Project status and performance matrix MA performance On-going Closed rating Satisfactory ■ Community Development Fund ■ Blue Nile Emergency Project ■ Decentralized Health System ■ Capacity Building of the National Development Judiciary ■ Livestock Production and ■ New National Currency Project Marketing Project ■ Technical Assistance Facility ■ National Emergency Transport Rehabilitation Project ■ Revitalizing the Sudan Gum Arabic Production and Marketing Project Moderately ■ Basic Education Project ■ Fifth Population Census Satisfactory ■ Sudan Microfinance Project ■ Public Sector Reform, Decentralization, and Capacity Building Program Support Project ■ South Kordofan Emergency Project The first stage of the process was the selection of one project from each of the following categories: ■ Ongoing/Satisfactory ■ Ongoing/Moderately Satisfactory ■ Closed/Satisfactory ■ Closed/Moderately Satisfactory The aim and approach of the data collection within each category varied and this difference was reflected in the data collection tools, which are attached in Annex 4 and Annex 5. Sectoral coverage In order to broaden sectoral coverage and thereby obtain data of sufficient breadth and depth for analysis, the evaluation team chose an additional three projects from Table 2.4 Project budget The size of the project budgets supported by the MDTF-N varies from US$4.8 million to US$145.2 million. To achieve a sampling truly reflective of this wide range of budgets, the evaluation team ensured that the project sample reflected of the variety of budget sizes within the MDTF-N portfolio. Based on the criteria outlined above, the evaluation team selected the following projects for in-depth review: 13 The MDTF-N portfolio contains 15 projects; however, for purposes of this table, the 15th project, Abyei Start Up Emergency Project, which closed on 30 June 2011, was excluded because it is outside the geographic scope of this evaluation. 12 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Table 2.5 Project sample for the evaluation MA Ongoing Closed performance Project Total Sector Project Total Sector rating budget budget (US$m) (US$m) Satisfactory Community 95.3 Livelihoods & Blue Nile Start 18.1 Cross-sector Development Social Up Emergency Fund Protection Project National 133.4 Infrastructure Capacity Building 9.6 Governance & Emergency of the National Rule of Law Transport Judiciary Rehabilitation Revitalizing the 7.0 Productive Sudan Gum Sectors Arabic Production and Marketing Project Moderately Basic Education 15.0 Basic Social Public Sector 4.8 Capacity Satisfactory Project Services Reform, Building & Decentralization, Institutional and Capacity Strengthening Building Program Support Project 13 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 2.3 Project summaries This section provides detailed summaries of the seven projects selected for further in-depth review. The table at the beginning of each project summary highlights basic information about the project, including its mode of implementation, budget size and composition, start and end dates, and locations of implementation. Each summary highlights the project’s objectives and progress, identifying key outputs and concluding with a brief summary and assessment of the project’s overall impact. Additionally, an output framework for these seven projects is included in Annex 6. 2.3.1 Basic Education Project (BEP) Project implementing unit Federal Ministry of Education State Ministries of Education in Blue Nile, North Kordofan, Red Sea, and South Kordofan States Community Development Fund Red Sea Engineering Office Budget US$15 million (all MDTF-N) Project start date 12 May 2009 Project end date 31 May 2013 Locations of implementation Blue Nile North Kordofan Red Sea South Kordofan Status Open Objectives and indicators The main objective of the Basic Education Project (BEP) is to increase access to improved basic education in four selected states: Blue Nile, North Kordofan, Red Sea and South Kordofan. The project’s activities focus on improving results in four key areas: ■ The number of students enrolled in primary education in selected schools. ■ The average number of days functional for the schools supported by BEP. ■ Improvement in teachers’ subject knowledge. ■ The number of direct project beneficiaries. The BEP has pursued its objective in several ways. It has rehabilitated and furnished schools, sometimes in partnership with MDTF-N’s Community Development Fund. Based upon priorities identified by the state governments, the project has supported the construction of a new Teacher Training Institute (TTI) in Red Sea state and the rehabilitation of one in North Kordofan. The BEP will also supply both TTIs with furniture, computer and laboratory equipment, and other necessities. The project also supported teacher training and is currently working with partners to upgrade the English language curriculum. Progress Progress has been measured against a baseline conducted in July 2009. As Table 2.6 shows, there has been an increase in the number students enrolled in the schools supported by the project, although this increase is entirely represented by girls with a small decrease in 14 the number of boys . With the goal of increasing the number of functioning school days for the participating schools, BEP has rehabilitated 25 schools of the targeted 31, providing an improved learning environment for these students and their teachers. The project has 14 Although there are several factors which impact the enrolment rates, this decrease in the number of male students has been attributed to poverty in several locations, with boys leaving school to work in order to help support their families. (Information obtained from interviews and focus group discussions with BEP project staff.) 14 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report supplied chairs and teachers’ desks and is currently working to provide water facilities to schools in North Kordofan. Through BEPs 21-day teacher training program conducted in four states, teachers’ subject knowledge and skills have improved in all categories and exceeded the target in core subjects and Arabic language. 15 Table 2.6 BEP outputs as of 30 June 2012 Project development objective level results Baseline Progress indicators (July 2009) (as of 30 June 2012) Indicator 1: Number of students enrolled in Total: 8,599 Total: 8,711 primary education in supported schools Male: 4,318 Male: 4,206 Female: 4,281 Female: 4,505 (Target: 9,028 or a 5% increase) Indicator 2: Average number of days A Schools: 175 days A Schools: 175 days 16 functional for the supported schools B Schools: 123.7 days B Schools: 123.7 days (Target:160.8 –a 30% increase of school days in Category B Schools) Indicator 3: Improvement of teacher’s Pre-training Post-training knowledge of core subjects Subject knowledge Subject knowledge (Target: 20% increase in teachers’ subject Core: 25.52 Core: 32.70 (28%) knowledge) Math: 26.13 Math: 31 (19%) English: 21.57 English: 23.42 (9%) Arabic 39.08 Arabic: 51.9 (33%) Teaching skills Teaching skills Core: 42.21 Core: 53.7 (30%) Math: 44.84 Math: 49.93 (11%) English: 45.43 English: 49.02 (8%) Arabic 27.86 Arabic 37.32 (34%) Indicator 4: Number of direct project Total students: 8,599 Total students; 8,711 beneficiaries, with specific gender targets Male students: 4,318 Male students: 4,206 Female students: 4,281 Female students: 4,505 (Target: 9,029 students/4,495 female students; 11,380 teachers/6,828 female Teachers: 0 Teachers: 11,380 teachers). Male teachers: 0 Male teachers: 3,555 Female teachers: 0 Female teachers: 7,825 Challenges Renewed conflict in Blue Nile and South Kordofan states is one of the primary challenges directly impacting the project’s capacity to deliver. For example, in South Kordofan construction had already begun on schools which had to be abandoned when the communities fled the fighting. Following a rigorous assessment and selection process, funds from South Kordofan were reallocated to North Kordofan, where TS and PIU staff, as well as contractors, had the access they did not in South Kordofan. there were also issues related to construction oversight and the lack and/or quality of engineer supervision for schools and one of the TTIs, in particular. The project has taken steps to address the ongoing issues by hiring a team engineer and implementing all new school rehabilitation projects through the PIU, rather than in partnership as had been the previous arrangement. The current challenge is to complete the construction/rehabilitation of the remaining schools before the close of the project. Additionally, the project has not been able to complete a planned curriculum review 15 Information obtained from the Monitoring Agent’s 2nd Quarterly Report (1 April to 30 June 2012), dated 10 October 2012 16 The project tracks the number of days the school is open on a monthly basis, but annual data will only be available by March 2013. Currently, the monthly data shows positive signs of an increase in the number of functioning days compared to the baseline data, but this will only be confirmed when the annual data is released. 15 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report for a number of reasons beyond the purview of the PIU; instead, an activity aimed to update the English language textbook is being undertaken in partnership with the British Council. Summary 1. BEP has made positive strides in increasing enrolment for girls, improving the learning environment, and improving teachers’ knowledge and skills while facing challenges due to conflict and the sector’s lack of capacity. 2. There are issues related to project administrative and lack of engineering oversight that have delayed implementation and resulted in lower-quality output in building construction. 17 Assessment of impact Project objective: 3.8 Project results: 3.5 2.3.2 Blue Nile State Start-Up Emergency Project (BNSSEP) Project implementing unit Blue Nile Ministry of Finance and Economy (MoFE) Blue Nile State Government UNICEF UNIDO Budget US$15.09 million ■ MDTF-N: US$7.07million (with an additional US$3 million provided by the MDTF-N) ■ GoS: US$8.02 million Project start date 24 January 2008 Project end date 30 June 2012 Locations of implementation Blue Nile State Status Closed Objective and indicators The Blue Nile State Start-Up Emergency Project (BNSSEP) had two primary aims: ■ Contribute to the supply of basic services and facilities to the conflict-affected residents in Blue Nile state; and ■ Help build the capacity of the government of Blue Nile state. Through eight separate cross-cutting components, BNSSEP sought to initiate the recovery of basic infrastructure and social services in this conflict-affected state. It also addressed capacity building needs in key state ministries with a view to ensuring future improvement in the decentralized delivery of services. The project’s initial budget was US$15.3 million to cover activities in education, WASH, demining, rural livelihoods, and health. When the project began in 2008, its original components included: ■ Construction and rehabilitation of improved water sources; ■ Maintenance and spot improvements of selected rural roads; ■ Purchase and distribution of primary school pupils’ and teachers’ kits, hand pumps and classroom equipment; ■ Demining suspected roads and other high priority areas; ■ Distribution of agricultural hand tool kits and animal traction farming equipment; ■ Capacity building for the Blue Nile state and locality governments; 17 Respondents to this evaluation were asked to assign a rating of the project’s impact. The scale was 1-5, with 1 being extremely negative impact and 5 being extremely positive impact. 16 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report ■ Marketing and distribution of long-lasting insecticidal nets; ■ Provision of technical assistance to the Project Coordination Unit (PCU). The project adjusted some of its objectives – the insecticidal nets component, for example – based on need and state government requirements. Additional financing of US$3 million from the MDTF-N supported a new livelihoods program targeting the fishing sector. Progress A baseline by which the project’s progress is measured was conducted in May 2008. Table 2.7 details the increases in each of the major results indicators for the original project components. Indeed, the project met or exceeded most of its end-of-project targets for these components, with only the rehabilitation of administrative buildings pending when that phase of the project closed on 30 June 2011. 18 Table 2.7 BNSSEP outputs from the original project components as of 30 June 2011 Project development objective level results Baseline Progress indicators (May 2008) (As of 30 June 2011) Indicator 1: People in rural areas provided Total: 0 Total: 171,000 with access to improved water sources General population: 0 General population: Pupils: 0 71,000 (Target: 171,000; general population: 71,000; Pupils: 100,000 pupils: 100,000) Indicator 2: Farmers trained and adopting 0 200 new technologies (Target:200) Indicator 3: 14 high priority or mined areas 0 54,000 and roads cleared or verified in about 33 communities, affecting a population of more than 54,000 (Target: 54,000 people with safe access to roads and other areas) Indicator 4: Capacity building in finance and 0 2,058 economy, local government, education and public planning for 720 government employees (Target: 1,720 ) Indicator 5: Improved disease control 0 350,000 through increased investigative samples undertaken by the disease control center (Target: 72,000 samples) Indicator 6: Number of direct project Total beneficiaries: 0 Total beneficiaries: beneficiaries, with specific gender targets Percentage female 942,270 beneficiaries: 0% Percentage female (Target:774,000, with 53% female beneficiaries: 52% beneficiaries) As of 30 June 2012, BNSSEP met or exceeded 13 of its 17 target indicators for the additional fishing sector component, including the training of women in fish processing; management and skills training for fishers union officers; the provision of nets and boats to fishers unions; studies, plans, and organizational restructuring to strengthen the fishing sector in Blue Nile; and strengthening the fishing sector infrastructure with the provision of an 18 Information obtained from the Monitoring Agent’s 2nd Quarterly Report (1 April to 30 June 2011), dated September 2011. 17 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 19 ice factory, fishers service center, and fish processing halls for women . An earlier livelihoods component focused on the training and provision of tools to farmers and training for 20 blacksmiths, as well as a beekeeping activity. With regards to education, BNSSEP provided training for 1,336 teachers, distributed school kits to 287,353 pupils, and supplied seating to 48,000 students. In the WASH sector, the project installed 54 new hand pumps and rehabilitated 50 more, constructed a new hafir, contributed to the provision of five new water yards, assisted with the procurement of a drilling rig and two trucks. As water supply is an important component of a fully functional school, the project also provided new hand pumps and latrines at 30 schools. BNSSEP also assisted with the establishment and training of community-based water management committees in order to strengthen sustainability at the local level. In the transportation sector, in addition to the demining and verification of roads, the project rehabilitated 107 roads and constructed 11 bridges. Although the project’s original objective in the health sector was the distribution of long- lasting mosquito nets, the Blue Nile state Ministry of Health requested that the project shift its focus to other activities, as the state already had adequate coverage. Instead, the state felt that the money would be better spent on establishing a Center for Disease Control (CDC) in Damazin which is now equipped and operational, demonstrating the project’s flexibility and responsiveness to the needs and priorities of the stakeholders it supported. Challenges BNSSEP was affected by the renewed conflict in Blue Nile but not to the same extent as other MDTF-N projects operating in the state because much of the project had been implemented by 2011. Regardless, the security situation contributed to delays in some project activities and the project still cannot have access to some schools and dormitories 20 which some respondents indicated there had been looting during the fighting . There were additional challenges at the beginning due to the capacity of implementing agencies, some of which did not have staff in the country at the beginning of the project. There was also a high-turnover of TTLs for the project, which contributed to the lack of continuity and appropriate level of support to the project. High-turnover of trained staff within the state government is also a challenge. Additionally, some of the activities, such as the training and tool provision to farmers, were ultimately not successful or responsive to the local context. There are remaining challenges surrounding the sustainability of the fishery component activities. The ice factory in Damazin, while physically completed, did not have a connection to the electrical supply, so was not functional as of mid-September 2012. The information gathered during the evaluation team’s visit to the Fishers Service Center raised questions regarding the level of involvement that the Fishers Union ultimately had in the Center going forward, as well as the overall sustainability of the ice factory and the market. Summary 1. There were challenges due to the capacity of implementing partners and the state government, as well high-turnover led to a challenging start; however, the project managed to meet nearly all of its objectives and exceeded its targets in quite a few of the key components. 2. There are lingering issues about the long-term sustainability of some of the outputs of the fisheries components. Assessment of impact 4.1 19 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 20 Information obtained during interview with project management staff, Khartoum, Sudan. 18 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 2.3.3 Capacity Building of the Sudan Judiciary Project implementing unit Sudan Judiciary UNDP Budget Original budget: US$10 million ■ MDTF-N: US$5 million ■ GoS: US$5 million Actual budget: US$5.25 million ■ MDTF-N: US$5 million ■ GoS: US$0.25 million Project start date 23 August 2006 Project end date 30 June 2009 Locations of implementation Khartoum Abyei Blue Nile Eastern Sudan South Kordofan Status Closed Objective and indicators The goal of this project was to building the capacity of the judiciary at both the national and state levels in order to: ■ Enhance the judiciary’s independence; ■ Build the knowledge base of its judges; ■ Empower the judiciary to effectively and fairly apply the law to deliver justice. The activities of the project were based around four key components: ■ The establishment of the National Judiciary Service Commission (NJSC); ■ Training of judges at the central and state level; ■ The establishment of the National Legal Training and Resources Center; ■ The rehabilitation of court facilities in Blue Nile, Kassala, Khartoum, and South Kordofan states. Progress A baseline survey was not conducted for this project, thus it was difficult to thoroughly assess and attribute impact. However, the project measured progress against planned progress indicators in each of the four components, elements of the final year of the project are outlined in Table 2.8. 21 Table 2.8 Judiciary outputs as of 30 June 2009 Project component Planned progress for year ending Dec 2009 Progress (as of 30 June 2009) Component 1: Support Activity 1: Study tour for NJSC members Completed. 6 NJSC to the National Judiciary to examine and assess the role of a members were sent on a 5- Service Commission judicial service commission in a modern day study tour to Malaysia. legal system 21 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2009), dated August 2009. 19 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Project component Planned progress for year ending Dec 2009 Progress (as of 30 June 2009) Activity 2: Group discussions with Not completed. The project selected international and local experts closed before this activity with the NJSC on the NJSC’s role was implemented Activity 3: Provision of video Completed. The procurement conferencing facilities for NSJC offices process was completed by and 8 corresponding locations June 2009, with the hand- over of equipment in July 2009 Component 2: Judicial Activity 4: Equipping of the Training Completed. The procurement training at the central Department including office facilities, process was completed by and state levels computer training facilities, training room, June 2009, with the hand- English training facilities, audio visual over of equipment in July equipment, furniture, and video 2009 conference facilities Activity 5: Provide support to the training Completed. Training for three of state-level judges groups of 20 judges – half of whom were state-level judges – was conducted between February and June 2009 Activity 6: Study tour for the head of the Completed. The head of the training department and the judge in training department and two charge of training to comparative NJSC judges were sent to jurisdictions to obtain knowledge on the Canada for an 8-day training management of judicial training visit departments Activity 7: Training in international law Completed. Training for three and women and children’s law for 30 groups of 10 judges – of judges at the Harun M. Hashim Law which 5 were women – was Center in Malaysia conducted in Malaysia between April and June 2009 Component 3: Activity 8: Support provided to the Completed. Provided 43,000 Establishment of a Technical Department, including the copies of the Sudan Law National Legal Training development of court libraries, court report (1900-2007) for the and Resource Center reports, the production of bench books, Technical Department, for administrative manuals, and other training distribution to 300 courts and and reference materials for the judiciary 900 judges and for court administration Component 4: Activity 9: Rehabilitation of selected Completed. Selected state Rehabilitation of court courthouses and judges’ houses in courts were rehabilitated. facilities in selected remote locations across four states states In addition to the selected activities planned for the project’s last year outlined above, the Judiciary project equipped and operationalized the National Judiciary Service Commission and worked with the NJSC. As part of this component, the project rehabilitated and furnished a conference hall and 13 offices. NJSC members participated in study tours in Germany, India, Malaysia and South Africa in order to learn more about how an independent judiciary functions. With regards to judicial training, the project assisted with the development of judicial training curriculum and materials and provided training to 400 central-level and 800 state-level judges on topics such as constitutional law, international human rights, modern legal analysis and application, and comparative international law. In addition to the provision of legal resources, the project supplied computers, furniture, projectors, and other equipment for judicial training facilities to the National Legal Training 20 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report and Resource Center. The project also successfully completed the planned construction of courthouses in the Blue Nile, Kassala, Khartoum, and South Kordofan states. Challenges The Judiciary project experienced a very slow start reportedly related to administrative and staffing capacity issues. By the final year of the project, these appeared to largely be resolved and the project began to successfully implement some of its planned activities. The Oversight Committee made the decision to close the project before its intended completion date and several activities to be undertaken by the project were not implemented and/or finalized. Resultantly, the project’s actual budget was nearly half of its original planned budget. The evaluation team was unable to conduct a full assessment of this project due to the lack of availability of project implementing staff; however, from the little information that was gathered, it appears that this project has much to offer in terms of lessons learned about project design and inter-organization/agency collaboration. There are also significant lessons about sustainability and scale-up, as it appears that UNDP has continued its engagement with the judiciary with similar activities through other funding following the closure of the MDTF-N project. Summary 1. Administrative and staffing capacity issues with the implementing partner at the beginning of the project negatively impacted the rate of implementation; however, these problems seemed to have been resolved and the project was performing well by the time it closed. 2. Due to the lack of access to and availability of key project staff, a clear assessment of this project's lessons learned could not be made; regardless, from the information obtained, it is clear that this project could provide valuable lessons, particularly around follow-on activities and scale-up. Assessment of impact Construction: 4 Support to the NJSC: 2 Training: 3 2.3.4 Community Development Fund (CDF) Project implementing unit Project Implementing Units (PIUs) comprised of national and state civil servants and non-government staff. Budget US$95.3 million ■ MDTF-N: US$52.8 million ■ GoS: US$42.5 million Project start date 26 April 2006 Project end date 30 June 2013 Locations of implementation Blue Nile Kassala North Kordofan South Kordofan Status Open Objective and indicators Through a community-led development process, the Community Development Fund (CDF) seeks to initiate recovery and development, particularly in those areas of Sudan that have 21 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report little service delivery and have been impacted by conflict. Its Project Development Objective is to ‘increase access to priority economic and social services and infrastructure in war- affected and underdeveloped areas of North Sudan, including the Three Areas’. Participating communities select and prioritize their development needs through an elected Community Development Committee (CDC) – a collaborative process that is intended to strengthen community participation and ownership, develop community capacity, and enhance long- term sustainability. While this Community Driven Development (CDD) model is not new to World Bank operations with similar programs implemented in places as diverse as Afghanistan, the Philippines, India, Nigeria and Honduras, the community-centred approach which allocates community groups the control of development decisions is innovative within the context of 22 Sudan, where development policy and initiatives were centralized, historically . While its primary focus may be poverty reduction and development, CDD programs like the CDF also 23 enhance people’s empowerment – the absence of which is an element of poverty . As witnessed by the evaluation team and attested to by CDF project staff and community members, the community’s empowerment is an on-going, transformative experience carrying the potential of long-term positive impacts that extends beyond the life of the project. Additionally, the successful design of community-based groups establishing priorities and overseeing development projects was mirrored in the other MDTF-N projects like Gum Arabic and Livestock. This local-level development activity was also intended to facilitate conflict resolution and strengthen the peace in communities impacted by conflict – key elements of the CPA and JAM. The CDF’s activities are aimed to enhance development in the participating communities measured by these indicators: ■ The number of girls and boys under 15 years old who are enrolled in school in the targeted communities; ■ The number of community members who have access to a basic development package, including health, nutrition, or population services; ■ The percentage of reported satisfaction with the delivery of basic services; ■ The number of project beneficiaries with gender targets. Progress The CDF’s support to community-selected development priorities includes a particular emphasis on improvement in access and quality of basic education, health and water supply and sanitation services. Through the community-led development process, 670 community development action plans have been drafted and approved, providing communities with a road map for future development activities. Since the project’s inception in 2006, it has supported 814 local-level subprojects that have reached over 2.5 million people. These subprojects include the provision of 137 water-supply systems, the rehabilitation of 484 schools and 96 health facilities, the construction or rehabilitation of 270 community infrastructure projects, and the installation of 97 village solar electricity systems. As part of the local capacity development objective of the project, over 69,100 people (40,040 male and 29,060 female) have participated in a variety of community-level workshops and training sessions on a number of topics, including development, management, and health. CDF has also trained 30 senior local government offices from 13 22 For an overview of the World Bank’s Community-Driven Development programs, please see Wong (2012). What have been the impacts of World Bank Community-Driven Development Programs: CDD Impact Evaluation Review and Operational & Research Implications. Washington, DC: The World Bank. Available at: http://www- wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2012/06/14/000386194_20120614062031/Ren dered/PDF/695410WP0SW0CD00Box370017B00PUBLIC0.pdf. 23 World Bank (n.d.). IDA at work: Community-Driven Development: Delivering the results people need. Washington, DC: The World Bank. 22 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report localities through its local government leadership enhancement initiative. The project has also supported the training of 235 health service providers. Progress towards the CDF’s overall project objective and indicators is measured against baseline data, outlined in Table 2.9. Development impact in the participating communities includes increased enrolment rates for both boys and girls; improved learning environments at schools, including water and sanitation facilities; reduced time and distance to fetch water; access to better quality water; and improved access to basic health services – and in some communities, improved quality of health care. 24 Table 2.9 CDF outputs as of 30 June 2012 Project development objective level results Baseline Progress indicators (2006) (as of 30 June 2012) Indicator 1: Number of girls and boys under Total: 80,800 Total: 151,619 the age of 15 enrolled in school in the Male: 64,640 Male: 89,308 supported communities Female: 16,160 Female: 62,311 (Target: Boys – 101,672; Girls – 82,899) Indicator 2: The number of community 212,200 783,406 members who have access to a basic development package, including health, nutrition, or population services (Target:316,800) Indicator 3: The percentage of reported 0% 75% satisfaction with the delivery of basic services (Target: 90%) Indicator 4: Number of direct project Total beneficiaries: 0 Total beneficiaries: beneficiaries, with specific gender targets % Male: 0 2,835,912 % Female: 0 % Male: 48 (Target: 2,621,794, with 51% female % Female: 52 beneficiaries) In addition to these indicators, the evaluation team observed important impacts that would be difficult to quantify related to a shift in community attitudes and capacity as a result of their participation in the project, as well as their acceptance of and trust in the CDF project staff. The community’s willingness to participate in the project and openness to development initiatives relied very heavily on the efforts of the CDF staff, particularly the social mobilizer. Project implementation, particularly in very closed and remote communities unaccustomed to outside involvement in their affairs, was dependent first upon building trust and meeting and managing the communities’ expectations. These elements, which took a considerable amount of time in some instances, were vital to successful implementation of the subprojects. As community members witnessed the consistency and dedication of the CDF staff, their willingness to participate fully in the process increased. The evaluation team heard repeatedly in the focus group discussion groups with CDF communities that CDF is one of the only dependable development partners. This trust in the CDF’s capacity to deliver on its promises – while not easily quantified – is a positive outcome which could be a cornerstone for future interventions. Challenges Much like the Basic Education Project, the CDF was negatively impacted by the resumption of violence. Although project activities continue in accessible parts of Blue Nile and South Kordofan, staff had to be relocated and some areas are still inaccessible. The status of the 24 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012 23 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report project’s physical assets and sub-projects are unknown in some cases. Additional challenges include the lack of capacity of contractors in remote areas, which has slowed implementation of subprojects. Access to remote areas can be difficult for CDF staff, particularly in the rainy season, which also impacts project implementation speed. Finally, establishing good relationships with communities also takes considerable time and effort and this can lead to unforeseen delays in project implementation. Summary 1. The CDF's community-led development design has had a positive impact on participating remote communities' access to education, water, health, and social services. 2. Positive unintended consequences, such as the excellent reputation and legitimacy of the CDF among participating communities is an important outcome that could be instrumental in future interventions or scale-up of this project. Assessment of impact 4.5 2.3.5 National Emergency Transport Rehabilitation Project (NETREP) Project implementing unit The Ministry of Transport through the Sudan Railway Corporation, the River Transport Corporation, and the National Highway Authority Budget US$133.4 million ■ MDTF-N: US$64.7 million ■ GoS: US$68.71 million Project start date 29 January 2007 Project end date 30 June 2013 Locations of implementation Blue Nile Kassala Red Sea South Kordofan Status Open Objective and indicators The main objective of the National Emergency Transport Rehabilitation Project (NETREP) is to ‘facilitate the country’s economic and social recovery through improved physical access to goods, markets, and administrative and social services’. To this end, the project focused its efforts on three main areas: ■ The removal of key transportation bottlenecks which hamper trade and recovery efforts; ■ Capacity building through the provision of equipment and technical assistance related to the operation, maintenance, and supervision of works; ■ Improving the capacity of the Ministry of Transport to plan and manage the transportation sector with the view to improved productivity. NETREP implemented projects that opened or enhanced key trade routes linking the North and the South with Port Sudan, as well as improving access for rural communities. By targeting the transport bottlenecks and opening access – particularly in communities in the Three Areas – the project sought to promote rapid economic integration and facilitate trade, ease the delivery of humanitarian aid, and enhance reconstruction efforts. The project’s successful implementation would be measured by these results indicators: ■ The share and number of rural population with access to all-season transportation infrastructure (roads and rail) in the areas targeted by the project; 24 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report ■ The share of roads in good condition from a total of all classified roads; ■ The size of the total classified road network; ■ A reduction in the travel time on selected roads; ■ Travel volume on selected roads; ■ Normal traffic operations on the key rail and waterway arteries. Progress Since its inception, NETREP has received a series of extensions and additional funds. Currently, the project is extended to June 2013, with an intention of allowing for completion of the road network currently under construction, the establishment of the Pavement Management System (PMS) within the National Highway Authority (NHA), and the continuation of support to the Ministry of Transport through coordination of policy reform implementation and technical assistance. Some baseline data had been compiled for several of the project indicators, but not all activities have reported information; thus, measurement of progress for all activities is incomplete, as indicated in Table 2.10. 25 Table 2.10 NETREP outputs as of 30 June 2012 Project development objective level results Baseline Progress indicators (as of 30 June 2012) Indicator 1: Share of rural population with 0 not provided access to an all-season road (Target: not provided) Indicator 2: Number of rural people with 0 not provided access to an all-season roads (Target: not provided) Indicator 3: Number of rural people with 0 not provided access to an all-season rail (Target: not provided) Indicator 4: Roads in good condition, as a 0 not provided share of the total classified network (Target: not provided) Indicator 4: Size of the total classified road 0 not provided network (Target: not provided) Indicator 5: Travel time (in hours) on selected transportation routes Babanousa – Wau Rail (Target: 15) N/A 20 Damzine - Kurmuk Road (Target: 1) 10 10 Hamsahkoreib – Gadamai Road (Target: 1) 10 1 Kadugli – Kauda Road (Target: 2) 15 15 Indicator 6: Number of direct project Total beneficiaries: 0 Total beneficiaries: beneficiaries, with specific gender targets Male: 0% 261,000 Female: 0% Male: 52% (Target: 301,500, 48% of which are female Female: 48% beneficiaries) 25 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 25 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report NETREP has supported the rehabilitation of a total of 446km of the Babanousa-Wau Rail Line. Completion of the rail line reduced the travel time between the two towns significantly; however, information obtained during the evaluation revealed that the railway is not currently functional and there are unresolved issues around the border areas that must be resolved for 26 services to resume . The project has also completed the construction of 112km of roads and currently about 95km of roads are in progress. Completion of the road between Hamsahkoreb and Gadamai has reportedly reduced the travel time from about 10 to two hours. The design of 995km of national and rural roads was completed with approximately 215km under procurement. Studies on feeder and community access roads in three states have been completed. The studies on private sector participation in rail transport, Port Sudan Trade Facilitation and Inland River Navigation Department (IRND) diagnostics, as well as the preparation of a National Transport Master Plan (NTMP) have delivered sound sector development policies and strategies, and implementation of their recommendations is in progress. Finally, an indirect positive outcome of the project was the provision of temporary employment for the local population in the targeted areas through the construction and maintenance of the infrastructure projects. Challenges The renewed conflict in South Kordofan and Blue Nile has had a significant negative affect on the implementation of road works and the continued operation of the newly rehabilitated railway between Babanousa and Wau. During the conflict, construction equipment and buildings were damaged or removed. Prior to the conflict, issues of quality and implementation with some road contractors, although addressed or outside the scope of the project’s control, impeded the progress of the road construction. While negotiations to resolve the remaining issues to restart work in the conflict-affected areas were ongoing at the time of the evaluation, it was unclear when the remaining segments of the Damazine- Kurmuk road in Blue Nile and the Kadugli-Kauda road in South Kordofan would be completed. Some components of the project were never completed – particularly activities related to the River Transport Corporation, which was privatized not long after the project became active. Current challenges to the successful implementation of the project by its completion date include the lack of counterpart funding, which was not provided for the previous two quarters, and continued delays in the implementation of some activities – the Damazine-Kurmuk road and the Pavement Management System in particular. The project’s monitoring and progress reporting system could also be improved in order to provide a more concise picture of the status and impact of the project’s activities. Summary 1. Renewed conflict has severely hampered the completion of project activities in the conflict-affected areas; however, for the components fully implemented in conflict-free areas, the results and impact on the targeted communities have been positive. 2. It is challenging to assess the exact impact of the project due to a lack of baseline data and unclear results indicators, as well as a consistent progress reporting system. Assessment of impact Railways: 4.5 Roads: 3.5 26 Information obtained during interviews with project implementing staff, Khartoum, Sudan. 26 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 2.3.6 Public Sector Reform, Decentralization, and Capacity Building Program Support Project (PSCAP) Project implementing unit Ministry of Labor, Public Service, and Human Resources Development Ministry of Finance and National Economy Budget US$4.77 million ■ MDTF-N: US$2.57 million ■ GoS: US$2.20 million Project start date 11 September 2008 Project end date 30 June 2012 Locations of implementation Sudan Status Closed Objective and indicators The objective of the Public Sector Reform, Decentralization and Capacity Building Support Project (PSCAP) was to improve capacity at federal, state and local levels to deliver decentralized services to the citizens of Sudan. This broad mandate was pursued through four main components: ■ Restructuring ministries and upgrading human resource management processes; ■ Improving capacity for fiscal decentralization; ■ Building the capacity of national public sector management training institutions; ■ Supporting the Project Implementation Unit (PIU) for the efficient implementation and oversight of project activities. Results indicators for the project activities include: ■ The development of a new human resources policy for the Ministry of Labor; ■ The number of targeted states that adopted the new human resource management database; ■ The development of a new fiscal formula in accordance with international best practices; ■ The number of targeted public sector training institutions that have undergone a structural and functional assessment; ■ The number of senior staff at the Fiscal and Financial Allocation and Monitoring Commission (FFAMC) trained in intergovernmental fiscal relations, with specific gender targets; ■ The number of Ministry of Finance staff at the federal and state level trained in topics such as procurement management, public financial management, project management, and monitoring & evaluation, with a specific gender target; ■ The number of project beneficiaries. Progress PSCAP had a wide-ranging objective, with four very different project components and only a very limited budget; resultantly, it functioned more as a pilot project. Activities undertaken under its four components include: the delivery of a computerized human resource records system and staff training in selected states; internal training for 243 Ministry of Finance staff at the national and state levels; capacity building activities related to M&E of fiscal allocation for 52 staff at both the national and state level; training for 33 senior FFAMC staff on intergovernmental fiscal relations; workshops on risk-based auditing and procurement management; training under the capacity building strategy for 1,847 state and local government officials; and the restructuring one of the public sector management training institutions. Arguably one of the key activities undertaken by the project was the development of a procurement manual and a standard bidding document. 27 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report PSCAP did not collect baseline data at the beginning of its activities; therefore, impact is hard to assess and attribute. However, Table 2.11 outlines the project’s progress as measured by end project targets. 27 Table 2.11 PSCAP outputs as of 30 June 2012 Project development objective level results Progress End project target indicators (as of 30 June 2012) (30 June 2012) Indicator 1: Development of a new human Yes Yes resource policy for the Ministry of Labor Indicator 2: The number of targeted states 2 5 that adopted the new human resource management database Indicator 3: Development of a new fiscal Yes Yes formula in accordance to international best practice Indicator 4: The number of targeted public 3 3 sector training institutions that have undergone structural and functional assessment Indicator 4: The number of FFAMC senior Total trained staff: 33 Total trained staff: 31 staff trained in intergovernmental fiscal % male: 70% % male: 60% relations, with specific gender targets % female: 30% % female: 40% Indicator 5: The number of direct project Direct beneficiaries: Direct beneficiaries: beneficiaries, with specific gender targets 2,800 4,200 % male: 60% % male: 60% % female: 40% % female: 40% Challenges The PSCAP project closed in June 2012 without implementing three activities: ■ Three workshops to develop the capacity of Sudan Academy for Administrative Sciences (SAAS); ■ Dissemination of the Procurement Act by the Ministry of Finance; ■ Computerization of state human resource records. There was a period of uncertainty surrounding the closure of the project, which had faced challenges throughout its operating period which affected implementation. Some of the challenges highlighted in previous project reports and by respondents to this evaluation included the high-turnover rate of TTLs and remote management; the wide-ranging project objective and unrelated main components; the very small budget compared to very large primary objective; the very short time frame in which substantive managerial and attitudinal changes were expected to occur; and the low level of capacity within the states and some of the consultants. Although the project has closed, the Ministry of Finance will provide a budget to the PIU to continue the fiscal decentralization component of the project. 27 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 28 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Summary 1. PSCAP suffered from a mismatch between its ambitious objective and small budget, as well as a short time frame in which to implement and see results or impact. The project was also challenged by turnover and remote management, as well as procurement-related issues for some of its activities. When it closed, the project had not met three of its seven targets and some of its project objectives were cancelled. 2. Despite these challenges, the project did deliver targeted training on key issues related to fiscal, procurement, and M&E management to key national and state staff; and laid the foundation for new policies and procedures in human resource management, fiscal decentralization, and procurement. Assessment of impact 3.5 2.3.7 Revitalizing the Sudan Gum Arabic Production and Marketing Project (GA) Project implementing unit The Ministry of Agriculture and Forestry’s National Forest Corporation (FNC) Budget US$10 million ■ MDTF-N: US$7 million ■ IFAD: US$3 million Project start date 30 September 2009 Project end date 31 December 2012 Locations of implementation Blue Nile North Kordofan Sennar South Kordofan While Nile Status Open Objective and indicators The Gum Arabic Project seeks to ‘increase the production and income of small scale gum producers in selected areas of the gum belt through improved performance of the production and marketing systems’. While the project has implemented numerous activities during the two phases since its inception, it measures progress by two main results indicators: ■ The percentage change in the income level from gum arabic production of beneficiary households following the project intervention; ■ The number of direct project beneficiaries, with specific gender targets. Progress During the first phase of the project, which ran from August 2009 to June 2011, the project undertook primarily preparatory activities. These included extensive studies of the gum arabic sector which guided the sector policy reform initiatives. Sector reforms, such as the abolishment of the gum arabic monopoly and the lowering of fees and taxes were key to the project’s objective of increasing the income of small-scale gum producers. It implemented a monitoring and evaluation system which included a baseline survey, and established a market information system. During this period, the project also conducted capacity building and training for the project implementing units and the gum arabic producers associations (GAPAs). These GAPAs, similar in design to the innovative community-based committees supported through the MDTF-N’s Community Development Fund, are cooperatives of producers who were the designated recipients of the subprojects which they chose. Support 29 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report to the GAPAs includes capacity development training and a matching grants program designed to support their infrastructure needs in order to increase production and efficiency. Throughout Phase II (July 2011 to June 2013), the project has monitored the sector reform process and provided support for the implementation of the recommendations arising from the analytical studies. It also continues to supply technical and financial support to GAPAs in each of the 11 localities participating in the project. Through the microfinance component of the project, 110 GAPAs in five states have been assisted, reaching approximately 11,346 beneficiaries, an estimated 25 per cent of them women. The matching grants component has approved and finalized 46 subprojects, including 14 water reservoirs, 12 gum stores, 16 tractors, and four water stations and hafirs. The civil works component of the project includes three training centers and seven tree nurseries. It has also conducted 22 training courses for 1,092 participants. The project has also served as a conduit for interventions with other organizations. For example, the GAPA Union and Agricultural Bank of Sudan partnered to distribute 200 tractors in seven states. Table 2.12 outlines the project’s progress to date. 28 Table 2.12 GA outputs as of 30 June 2012 Project development objective level results Baseline Progress indicators (2010) (as of 30 June 2012) Indicator 1: Percentage change in the Yes Yes income level from Gum Arabic production of beneficiary households following the project intervention (Target: 115%) Indicator 2: The number of direct project Direct beneficiaries: 0 Direct beneficiaries: beneficiaries, with specific gender targets % male: 0% 11,346 % female: 0% % male: 75% (Target: 11,475, with 20% female % female: 25% beneficiaries) Challenges One of the challenges facing the Gum Arabic project is the vast need – the number of GAPAs participating in the project is relatively small compared with the number of associations and the physical area in which gum arabic production is concentrated is quite large; therefore the scope of this project was that of a pilot with the potential for scale-up. While the project design was appropriate for the objective and context, the sector studies were strong guides for project activities, and the implementation plan was well-reasoned, observations by the evaluation team in a number of locations indicated that there were issues with and challenges to local-level implementation. In some locations visited, GAPAs appeared to be mini-chiefdoms, with the will of one member or a small group of members dominating the other members, resulting in their limited involvement and benefit. There was also an example of a nursery that was intended to produce acacia seedlings to provide to the GAPAs instead producing fruit trees and ornamental seedlings for profit after only one season supporting the GAPAs. Meanwhile the GAPAs still grow their trees from seeds, a longer and more tenuous process than planting seedlings. Instances like these raise questions about the consistency of project oversight and the extent to which the GAPAs and individual small-scale producers receive the maximum intended benefit from the project. 28 nd Information obtained from the Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012), dated 10 October 2012. 30 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Summary 1. The project design and the studies conducted during the first phase of implementation provided a strong foundation on which to build the subsequent project activities supporting the small- scale producers through selected GAPAs. Despite project success in the policy sector component, oversight and monitoring at the GAPA level could be strengthened to ensure that maximum benefit is reaching the producers. 2. Sector reforms, such as the removal of the gum arabic monopoly and the reduction of fees and taxes paid by producers was a catalyst for the increase of income and improvement of conditions for small-scale producers. This was an important benefit that impacted the entire sector – not only the producers and GAPAs targeted by the project. 3. Because of the small-scale of the GAPAs component, this project should be considered a pilot project which could be further supported for possible scale-up. Assessment of impact 4 31 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 3 Findings This section presents the findings from the desk review and five-week field visit to Sudan in which the evaluation team conducted 39 interviews, 29 focus group discussions, and 18 project site visits in Khartoum and five states. In total, team members met with 139 stakeholders and 233 beneficiaries over the course of this evaluation. Based upon the analysis of the information obtained in both the document review and the discussions conducted in Sudan with a broad range of stakeholders, the team extracted findings within the three main areas of focus for this evaluation: ■ development impact, ■ cost effectiveness, ■ institutional and organizational relevance. Following the evaluation structure outlined in the Inception Report, the evaluation team identified two levels of findings. First, there are general findings around the activities and implementation of the MDTF-N as a financing mechanism and the activities surrounding its administration, which is the primary feature of this section. At the second level, there are specific project-level findings based on the in-depth reviews of the seven projects in the evaluation sample. Section 3.1 explores the findings related to the institutional and organizational relevance of the MDTF-N, while Section 3.2 is devoted to a broad cost effectiveness analysis. In Section 3.3, findings around the development impact of the Fund, drawing on the review of the seven-project sample, are detailed. 3.1 Institutional and organizational relevance As outlined in the Inception Report, institutional and organizational relevance has several components. Foremost, relevance is measured by the alignment of the MDTF-N and its programming activities with the Joint Assessment Mission, the Comprehensive Peace Agreement and the Sudanese government’s Five Year Plan (2006-2011). Additionally, the Fund was assessed on its relevance as a mechanism for reconstruction in a post-conflict environment, the extent to which it addressed Sudan’s most pressing development needs, and its support to the most relevant institutions. The evaluation team also appraised the architectural design of the MDTF-N – the composition of its funding, its organizational structure, and the use of the Monitoring Agent. 3.1.1 Alignment of priorities between the MDTF-N and the JAM, CPA, and the Government’s Five Year Plan (2006-11) The establishment of trust funds – one for Sudan (MDTF-N), the other for what is now South Sudan (MDTF-SS) – as a primary source of aid and development financing was one of the recommendations of the CPA. When established, the MDTF-N’s objectives and programming activities were designed to align primarily with the needs and priorities identified in the JAM, CPA, and the Government’s Five Year Plan. The priorities of the documents which form the MDTF-N’s operating foundation can be summarized as follows: Comprehensive Peace Agreement, 2005 The CPA established the framework for peace in Sudan, according to six agreements (protocols) and an Implementation Plan. The protocol on wealth sharing addressed Sudan’s needs for development in the following areas: ■ infrastructure ■ human resources ■ sustainable economic development ■ capacity within a framework of transparent and accountable government 32 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report The two MDTFs recommended by the CPA were meant to support urgent recurrent and investment costs with a focus on priority areas of capacity building and institutional strengthening, and quick start/impact programs identified by the parties to the agreement. The protocol on power sharing called for the implementation of a census by the second year of the 2005-11 interim period. Joint Assessment Mission, 2005 The Joint Assessment Mission, carried out by the World Bank and the UN together with the Government of Sudan and the SPLM, was a year-long detailed assessment of recovery and development needs focusing on eight thematic clusters: ■ capacity building and institutional development ■ governance and rule of law ■ economic policy ■ productive sectors ■ basic social services ■ infrastructure ■ livelihoods and social protection ■ information and statistics The work on these clusters was complemented by work on several cross-cutting themes, including gender, HIV/AIDS, conflict prevention, and the environment. A further chapter focused specifically on issues relevant to the Three Areas. The strategic top priorities for the MDTF-N were: ■ National Government and the Northern States – Enabling consolidation of the peace, including implementing the range of measures outlined in the CPA, from the national level down to local peace building and reconciliation; – Improving governance, especially institutional capacity and public accountability; – Broad-based growth of income-earning opportunities, with a focus on smallholders, pastoralists and private sector development; and – Expanding access to basic services in the Northern states – in particular access to basic health care, primary education and safe water and sanitation to put progress on track for meeting the MDGs. ■ The Three Areas – Implementation of the Comprehensive Peace Agreement; – Establishment of key institutions; and – Return and resettlement of displaced persons. The Five Year Plan (2007-11) The Sudan Five Year Plan was the first strategic development plan since the signing of the CPA and aimed to provide a framework for focusing and coordinating the nation’s peace and development efforts over the 2007-11 period. It set a 25-year vision for the ‘Building of a United, Safe, Peaceful and Developed Sudanese Nation’. It focused on five priority areas: ■ Promote sustainable economic development by encouraging a competitive private sector, supporting key infrastructure and agriculture projects and building a knowledge- based economy; ■ Sustain peace and stability through continued implementation of the CPA, Darfur Peace Agreement (DPA) and East Sudan Peace Agreement (ESPA), whilst safeguarding national sovereignty and security, continuing to build consensus and reconciliation, and maintaining good relations with the international community based on mutual interests; ■ Reduce poverty and make progress towards achieving the Millennium Development Goals (MDGs) by expanding provision of basic services in health, education, water and sanitation, with a particular emphasis on quick impact projects for returnees and war affected groups; 33 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report ■ Strengthen public accountability, good governance and the rule of law; ■ Build capacity of public institutions and civil society at state and local levels and strengthen the social fabric of the Nation. The implementation plan involved the development of sector plans for the economic and private sector, the political sector, the defence and security sector, the social sector, the information and communications sector, and the traditional sector. Alignment of the MDTF-N portfolio Decisions on priority projects for the MDTF-N were taken by the Oversight Committee, chaired by the GoS and co-chaired by a donor, will the participation of all donors to the MDTF-N. The first stage of alignment with the objectives and needs identified in the three foundation documents above can be found in the MDTF-N’s design. Table 3.1 highlights the MDTF-N projects whose objectives correspond to each of the development goals from the CPA, JAM 29 and Five Year Plan . Table 3.1 Alignment of MDTF-N with the objectives of the JAM, CPA, and GoS Five Year Plan Development priority MDTF-N activity Consolidation of peace and ■ Abyei Start Up Emergency Project sustainable stability (JAM, 5YP) ■ Blue Nile Start Up Emergency Project ■ Fifth Population Census Project ■ South Kordofan Start Up Emergency Project ■ Sudan New Unified National Currency Project Expanded access to basic services, ■ Basic Education Project particularly health, education, water, ■ Blue Nile State Start Up Emergency Project sanitation (JAM, 5YP) ■ Community Development Fund ■ Decentralized Health System Development Project ■ South Kordofan State Start Up Emergency Project Improved governance, transparency, ■ Capacity Building of the Sudan Judiciary rule of law, and public accountability ■ Fifth Population Census Project (CPA, JAM, 5YP) Infrastructure (CPA, 5YP) ■ National Emergency Transport Rehabilitation Project Quick impact projects for war- ■ Abyei Start Up Emergency Project affected areas and groups (JAM, ■ Blue Nile Start Up Emergency Project 5YP) ■ South Kordofan Start Up Emergency Project Strengthening human resources and ■ Capacity Building of the Sudan Judiciary institutional capacity (CPA, JAM, ■ Public Sector Reform, Decentralization and Capacity 5YP) Building Project ■ Technical Assistance Facility Sustainable economic development ■ Livestock Production and Marketing Project and poverty reduction, including ■ Revitalizing the Sudan Gum Arabic Production and growth of income earning Marketing Project opportunities for smallholders, ■ Sudan Microfinance Project pastoralists, and private sector development (CPA, JAM, 5YP) The above analysis shows that, at the design level, the MDTF-N took into consideration the objectives of these guiding documents in its programming orientation and priorities. Likewise, it is the general consensus of stakeholders that the MDTF-N’s activities supported the development priorities identified in the JAM, CPA, and Five Year Plan, with a high majority of respondents stating that the MDTF-N was aligned with these documents. 29 Note that some projects addressed multiple development priorities and may appear more than once. 34 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Beyond the design level, there are issues surrounding actual implementation of MDTF-N projects and activities, and the degree to which they delivered the anticipated impact in relation to the identified development priorities. This was in part due to a lack of prioritization of needs in the foundation documents. For example, the JAM was considered to have established a good foundation for delivering on the peace dividends in terms of the needs identification and the joint processes and thus consensus and ownership that it was based on. Nonetheless, the Scanteam ‘Review of Post-Crisis Multi-Donor Trust Funds’ country study reports that informants were concerned that the JAM did not prioritize requirements, resulting in difficulty with MDTF program design. It notes, ‘The JAM process … did not provide direction on priorities and sequencing. In this regard, the report was not an operational document but rather an overview of needs. All things were given equal weight, which made a program response difficult.’ Respondents to this evaluation indicated the same – that due to the JAM’s lack of prioritization of development requirements, urgent needs carried the same weight as more long-term development goals. As a result, the balance of interventions across the MDTF-N portfolio did not necessarily reflect the distribution of needs in post-conflict Sudan. In terms of resource allocation across the MDTF-N portfolio, both in terms of the overall size of projects (including government contributions) and in terms of MDTF-N contributions, there is a degree of variation from the distribution of the resource requirement set out in the JAM framework. Table 3.2 shows which have been the most important sectors (in terms of percentage share of overall resource allocation) in the JAM framework (northern states and Three Areas), the overall MDTF-N portfolio and MDTF-N donor contributions. Table 3.2 Ranking of thematic priorities – JAM framework vs MDTF-N JAM themes (%) Overall portfolio (%) MDTF-N grant only (%) MDTF-N + GoS 1 Basic social services 60.9 Economic policy 24.8 Infrastructure 26.0 2 Infrastructure 11.9 Infrastructure 22.8 Livelihoods & social 21.2 protection 3 Livelihoods & social 9.2 Livelihoods & social 16.3 Basic social services 10.8 protection protection 4 Productive sectors 8.4 Information & 14.9 Productive sectors 10.2 statistics 5 Governance & rule of 3.8 Productive sectors 6.7 Economic policy 10.1 law 6 Capacity building & 2.9 Cross sector 5.9 Cross sector 9.5 institutional development 7 Economic policy 2.0 Basic social services 5.2 Information & 7.4 statistics 8 Information & 1.0 Capacity building & 1.8 Capacity building & 3.0 statistics institutional institutional development development 9 Governance & rule 1.6 Governance & rule of 1.8 of law law This reflects in part the emphasis of counterpart funding, which was concentrated on the Currency, Census and NETREP projects which related more to the provisions of the CPA than the development needs identified in the JAM. The Census project was a response to the CPA protocol on power sharing, while the Currency and NETREP projects reflected the unity considerations of the CPA. 35 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Most notably, however, basic social services, with the largest financing requirement by far under the JAM at 61 per cent, was only the third most important cluster in terms of the MDTF-N grant, with 11 per cent of resources – and down in seventh place in terms of combined MDTF-N and GoS funding, with just 5 per cent. Finding 1: At the design level, the MDTF-N was aligned with the development needs identified in the CPA, JAM, and Five Year Plan. However, in addressing some of the strategic priorities of the CPA – primarily through counterpart funding – the balance of interventions across the portfolio did not necessarily reflect the distribution of development needs identified by the JAM. Relevance to post-conflict reconstruction in Sudan The evaluation considered the MDTF-N’s relevance to post-conflict reconstruction in Sudan in two ways. First, the appropriateness of the MDTF-N as an aid financing mechanism in Sudan; secondly, its support to post-conflict reconstruction. MDTFs are one of the preferred instruments for post-conflict contexts. This aid modality offers several advantages that direct bilateral support and similar mechanisms do not. Primarily, MDTFs enable donors to manage risks in often volatile and unpredictable operating environments. They also reduce transaction costs and provide support to fragile states in a joined-up manner. Generally, MDTFs are designed to enhance aid effectiveness by providing adequate, flexible funding in a harmonized and coordinated manner through one delivery system which streamlines administration and management procedures, while strengthening local ownership. Overwhelmingly, respondents agreed that the MDTF-N, as a financing instrument, was particularly relevant to the post-conflict context of Sudan, with all of the respondents questioned on this point stating that it was the appropriate aid financing mechanism for the country. One stakeholder went so far to say that ‘[i]f it had not existed, we would have 30 invented it’ . Others said that it was the only mechanism that was suitable for the operating environment where capacity in international procurement and financial management procedures was extremely limited. Others pointed to the fact that the MDTF-N streamlined administrative procedures for both donors and the implementing units. Taking into consideration the difficult and challenging operating environment that the immediate post- conflict period presented donors and international organizations, a multi-donor trust fund presented one of the most efficient and risk-adverse modalities. The assessment of numerous stakeholders at the national and state levels confirms this analysis. When considering the suitability of the MDTF-N at the level of operations and project implementation, respondents agreed that the MDTF-N addressed the needs of post-conflict Sudan. Some respondents pointed to the fact that in some of the states that were particularly affected by the long-running conflict, the MDTF-N supported development projects that were the first ever to be implemented in some communities. During discussions with beneficiaries at project sites – particularly in communities where the CDF was implemented – on the whole beneficiaries seemed very pleased with MDTF-N’s project activities. This varied across projects, of course, with some projects exhibiting a higher degree of beneficiary satisfaction than others, but on average, stakeholders and beneficiaries agreed that the MDTF-N supported activities that responded to the needs and development challenges facing Sudan. 30 Respondent in a focus group discussion in Khartoum, 2 September 2012 36 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Impact Story 1: Bringing development to closed communities The Community Development Fund has been deeply involved with bringing development to several closed communities in Kassala state. Some of the tribes in this part of Sudan have long been wary of outside involvement in their communities and have experienced very little development as a result. Over a period of many months, CDF staff met multiple times with these communities, building trust and explaining the benefits the community members would gain from their involvement with the project. This investment in establishing a good relationship with community members eventually led to their participation in the CDF, which brought schools, training, and other services to these traditionally closed villages. During the Fund’s period of operation, the Three Areas – Blue Nile, South Kordofan, and Abyei – slipped back into conflict, which dramatically impacted the Fund’s ability to implement and monitor projects. Some ongoing projects in these areas were ultimately closed and damage was done to project structures and contractor equipment. To date, there are still areas in which project implementing and monitoring staff are unable to gain access to assess the status of the project. Despite the recurrence of violence in these areas, there is anecdotal evidence that communities in which MDTF-N projects were implemented were 31 less willing to support the conflict . While individual communities may have had little control over the return to violence in their states, the evaluation revealed examples of communities – the Hausa tribe in Blue Nile, border communities in the Gum Arabic zone, and CDF communities in the conflict-affected areas – that refused to join in the conflict or recalled their youth from rebel movements. While the MDTF-N was a suitable mechanism for the Sudanese context generally, at the local operating level, the evaluation team found that the MDTF-N did not capitalize on its positive features for operating in a post-conflict environment. In the field visits to the states and discussions at the locality level, it became evident that a conflict impact assessment had not been conducted prior to project design and implementation. This can be contributed to the fact that all stakeholders held high expectations for fast implementation during the early stages of the MDTF-N and time had not been taken for a thorough assessment of the conflict in the implementing areas. Although the JAM, from which the MDTF-N was derived, was prepared with a conflict analysis framework with the goal of ensuring that the activities proposed by the JAM did not trigger renewed conflict, it was broadly based and did not necessarily reach the community-level analysis that would have been useful. A project-level conflict analysis could have identified state or local flashpoints and issues that remained unresolved. An initial conflict impact assessment, particularly in the Three Transition Areas, would have better informed project selection, design, and implementation. Finding 2: The MDTF-N was an appropriate aid financing mechanism for the post-conflict environment of Sudan and while success in implementation and impact varied by project, the Fund generally supported activities that responded to the needs of the country. Response to Sudan’s most-pressing needs As outlined in the CPA, JAM, and Five Year Plan, the reconstruction needs of post-conflict Sudan were vast and the task that the MDTF-N faced when it was established in 2006 was enormous. Respondents during the interviews and focus group discussions identified the top development challenges currently facing Sudan as a lack of resources, particularly at the state and locality level and in some line ministries, and low capacity. The majority of respondents agreed that the MDTF-N addressed these challenges, but the extent to which it addressed those needs, and the impact they had, varied by project. The projects supported by the MDTF-N, like the Gum Arabic project for example, were designed to be pilot projects and, as such, could not be expected to have a scale impact. Similarly, while CDF beneficiaries reported a real and positive impact in their communities, there were only a limited number of communities involved in the project, so the impact was curbed in this way. 31 Discussions with Technical Secretariat and PIU staff 37 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Other projects, while designed to have wide strategic impact at the national policy level, were found to have limited impact in implementation due to operational challenges including poor government or implementer buy-in and support, delayed funding, poor institutional and human resource capacity, and conflict. Thus, while the objective may have addressed Sudan’s most-pressing needs, in practice, the impact the MDTF-N had varied dramatically by project. The impact also varied within project components and by location. Some projects, like the Basic Education Project and Gum Arabic, exhibited greater positive impact in some states than others. Finding 3: The MDTF-N’s design and objectives addressed the most-pressing needs in Sudan; however, the extent to which it was able to deliver response and impact on the ground varied by and within project, as well as location. Support to the most critical institutions Based upon the critical areas identified for development in the foundation documents, the MDTF-N provided support for implementation through direct capacity development and hands-on training for the project implementing staff, who were drawn from the personnel of the corresponding ministries and government institutions. For example, the BEP PIU was staffed by employees from the Ministry of Education. Similarly, the Gum Arabic PIU team came from the National Forestry Agency within the Ministry of Agriculture, while a multi- sector project such as CDF drew its personnel from a variety of primarily state government ministries. The MDTF-N’s primary implementing strategy through partnership with relevant Government of Sudan ministries was generally praised for its capacity development impact. Indeed, this approach echoes the broader World Bank policy of supporting country 32 ownership and capacity rather than short-term technical fixes . Impact Story 2: Streamlined procurement procedures One of the key outputs of the Public Sector Reform, Decentralization, and Capacity Building Program Support Project was the Standard Bidding Document (SBD) and subsequent procurement manual, which were formally accepted by Sudan’s Finance Ministry. The manual and SBD comprised Sudan’s first ever uniform procurement procedures and forms. Prior to this, each ministry had its separate procurement policies and documents. If the SBD and the procurement manual are adopted and implemented across all government ministries, it has a great potential to streamline the country’s procurement procedures. During the field consultation phase, the evaluation team observed the MDTF-N Technical Secretariat and the Monitoring Agent working closely with the PIUs and the MoFNE to ensure smooth implementation and to address programming challenges. Task Team Leaders (TTLs) for several ongoing projects were on mission in Sudan during the field consultation period. Procurement and financial specialists in the Technical Secretariat described the high level of capacity development support they provide to their government counterparts, both within the framework of the MDTF-N and through other Bank-supported activities, such as capacity building training for the Auditor General’s Chamber or hands-on training by Bank staff. This type of assistance was praised particularly by PIU members, who described the increase in their professional capacity, particularly related to procurement and financial management procedures. Both the evaluation team’s observations and the responses in interviews and focus groups indicate that the MDTF-N provided a high level of support and capacity development for the staff of the PIUs. Finding 4: MDTF-N Technical Secretariat staff and personnel from the Monitoring Agent provided invaluable direct support, hands-on training, and capacity development to PIU staff, particularly in regulations and procedures related to project management, procurement and financial management. 32 World Bank (2011). Capacity Development in Practice. Available at http://web.worldbank.org/WBSITE/ EXTERNAL/TOPICS/EXTCDRC/0,,contentMDK:20273839~menuPK:64169185~pagePK:64169212~piPK:641691 10~theSitePK:489952,00.html. 38 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 3.1.2 The MDTF-N’s architectural design As discussed in Finding 2, the MDTF-N is generally appropriate as an aid financing mechanism in the context of Sudan; however, this section examines more closely its architectural design to analyze how its rules and structures impacted the Fund’s operation and project implementation. Specifically, the evaluation explored the impact of the MDTF-N’s organizational structure, the level of flexibility in the design, structure, and regulations of the Fund, its response to priorities at each level of implementation and the Fund’s transparency as it relates to information sharing. Finally, the role, effectiveness and impact of the Monitoring Agent is analyzed. Impact of rules and structure on intra-operability This section examines the impact of the MDTF-N’s structure on operations and implementation in three areas: rules, operational structure and financial structure. MDTF-N rules As other evaluations have highlighted, the procurement and financial management regulations followed by the World Bank are often deemed difficult and time intensive. Many respondents, particularly PIU staff who follow these rules in their daily operations, stated that they are often time consuming and difficult to learn. Many stakeholders, including Technical Secretariat staff and donors, identified the lack of capacity and understanding of international procurement and financial management regulations and procedures as a major challenge in implementation. At the same time, PIU staff declared that their capacity and professional qualifications had been dramatically enhanced because they had been required to learn and follow World Bank procedures and, indeed, implementation rates improved markedly as PIU staff became familiar with Bank procedures. Several PIU and Technical Secretariat pointed to the high employability of staff who worked for PIUs and the Secretariat as a demonstration of the positive impact that working within the framework of the World Bank’s procedures can have. In this way, although the World Bank rules and procedures that governed the Fund’s operations negatively impacted operations and implementation at the start of the MDTF-N, the end result after a few years of training and hand-holding by the Bank and Monitoring Agent was the significantly increased capacity of core PIU staff and contractors who understood and were able to operate effectively within the procurement and financial management structures. Finding 5: The World Bank procurement and financial management procedures and rules were an impediment for implementation at the beginning of the Fund’s activities, but as capacity improved through hands-on training and assistance from MDTF-N and Monitoring Agent staff, the negative impact subsided. MDTF-N operational structure Previous reports and responses to interviews and focus group discussions support the conclusion that the lack of adequate World Bank human resources in Sudan negatively impacted the design and initial implementation of the MDTF-N’s projects. At the time that the Fund was established, the World Bank had not had a presence in Sudan for nearly 13 years due to arrears in the country’s debt payments, an ongoing issue that has implications for the post-MDTF-N environment. There was also a great deal of pressure from all stakeholders for the Bank to implement quickly with very few resources in place within the country to do so. This pressure and the lack of a World Bank office in Sudan meant that personnel needed to be provided from elsewhere, with some management staffing provided from Addis Ababa and TTLs designing projects from Washington, DC, sometimes with little or no personal knowledge of the country or operating context. All of these factors combined created a host of operating and implementation challenges that negatively impacted the MDTF-N’s ability to disburse funds and provide support to PIUs with limited capacity, resulting in delays in implementation. However, the Bank eventually responded to the staffing and resource allocation challenge, satisfactorily appointing a full 39 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report complement of personnel in-country. As one respondent explained, ‘We are happy with the 33 way things are managed now. We aren’t happy with how long it took to get here’ . Finding 6: The fact that in-country staffing was a challenge for the World Bank in the MDTF- N’s early days had a direct and negative impact on the implementation of the Fund’s projects; however, the Bank eventually responded to the human resource and technical support needs and the current capacity of the Technical Secretariat is experienced and adequate for the current level of operations. MDTF-N financing structure One of the unique characteristics of the MDTF-N is the high percentage of counterpart funding – 57 per cent of its US$584 million total budget is provided by the Government of 34 Sudan, with a majority of the counterpart funding (77 per cent) going to only a few projects . While the Currency project had, by all accounts and reports, received counterpart funding according to the agreed schedule, other projects reported implementation delays due to the lack of counterpart funding, including NETREP and the Census project. Due to the structure and requirements of the project design, projects which were to receive counterpart funding were required to wait until the funding was received before implementation could start. Some projects reported delays of over six months while they waited for counterpart funding, which created problems in procurement, particularly regarding payments for contractors. Some projects found ways to temporarily circumvent the requirement for counterpart funding, operating solely on MDTF-N funds until the counterpart funding was eventually received. These problems appear to have been resolved in early 2010, as new projects and additional financing for ongoing projects did not include counterpart funding. The PIU staff respondents to this evaluation do not report current delays in counterpart funding. However, in the last two quarterly reports from the Monitoring Agent, NETREP continues to report delays in counterpart funding. Counterpart funding is one way by which ownership and sustainability can be enhanced. It can increase a recipient government’s involvement in decision-making and strengthen the ties between the Technical Secretariat and its government counterpart. Conversely, it also carries with it the potential for lengthy delays in disbursement and implementation if not provided predictably and on-schedule, as was the case for several projects prior to 2010. Delays can result in the cancellation of program activities and the limitation of the scope of the impact. The delays in counterpart funding also placed an added burden on PIU and Technical Secretariat staff, who were forced to find ways to deal successfully with a difficult situation that was beyond their control. Finally, the impact on the beneficiaries who were required to wait for development projects should not be understated. If the negative consequences of delayed counterpart funding can be identified and adequately addressed at an early stage, counterpart funding can be a mutually beneficial financing option for an MDTF. In the case of the MDTF-N, however, it would have been difficult to mitigate delayed counterpart funding, as the GoS had consistently conveyed its intention to fulfil all of its counterpart obligations – and ultimately paid 57 per cent of the portfolio’s overall cost. In the case of the three start-up emergency projects, the early communication by the GoS that counterpart funding to these projects would stop allowed staff to fill the funding gap through contributions from other government entities, such as the Unity Support Fund and state governments, or to design projects without counterpart-funded components. Finding 7: Projects dependent on counterpart funding reported delays in disbursement prior to early 2010 that negatively impacted project implementation. 33 Respondent to a focus group discussion, Khartoum 34 The Government of Sudan provided US$120 million out of US$145.2 million for the Currency project, US$68.5 million out of US$86.9 million for the Census project, and US$68.5 million out of US$133.5 million for NETREP. These projects accounted for approximately 75 per cent of GoS contributions. 40 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report The role of the Monitoring Agent PwC, the MDTF-N Monitoring Agent, is charged with assisting the World Bank in the execution of its fiduciary role. This involves reviewing paperwork related to procurement and finances to ensure accuracy and smoother implementation, identifying potential fiduciary issues for the MDTF-N Technical Secretariat, monitoring projects through site visits, reporting the status, progress, and performance rating of MDTF-N projects on a quarterly basis, and preparing the MDTF-N quarterly reports. They also provide capacity development training and on-the-job assistance both to PIU staff and through other World Bank training activities, such as the training of the Auditor General’s Chamber staff, the training on procurement guidelines and procedures, and support for Bank TTLs through project updates on physical progress, financial and procurement issues, and monitoring & evaluation processes. In the context of Sudan, where there was little capacity with regard to international procurement and financial management regulations and processes, the evaluation found that the Monitoring Agent played a vital role in supporting the MDTF-N and PIU operations in these areas. The majority of PIU staff reported positive experiences with the Monitoring Agent – most stating that the area with which they provided the most support was with Withdrawal Applications, but some also pointing to the fact that they were available by phone to respond to questions and that they conducted regular visits to project sites. Some project TTLs pointed to the project site visits as an additional monitoring tool for the Technical Secretariat and PIUs. Finding 8: The Monitoring Agent plays important monitoring and capacity development roles, and the information and recommendations presented in the Quarterly Reports are integrated into project operations. 3.2 Cost effectiveness Cost effectiveness is a measure of how economically resources or inputs have been converted into results. Of particular importance is whether alternative approaches would have achieved the same results with fewer resources. As defined by the World Bank’s Independent Evaluation Group (IEG), cost effectiveness is ‘the extent to which the program has achieved or is expected to achieve its results at a lower cost compared with 35 alternatives’ . In order to assess the cost effectiveness of the MDTF-N, we sought to its cost versus results performance with that of both multilateral and bilateral comparators. The overarching question the cost effectiveness analysis sought to answer was: Could the same broad objectives been achieved with another alternative funding mechanism? This analysis proved particularly challenging, as the budget and costs of other mechanisms, including World Bank mechanisms, were not readily available. Lack of comparable data has meant that it has not been possible to conduct a meaningful or satisfactory comparison with other multilateral mechanisms in Sudan, or other mechanisms – Bank or non-Bank – in similar contexts. 3.2.1 Comparison with bilateral delivery channels in Sudan There are a number of assumptions regarding the cost benefits of multi-donor funding mechanisms compared with bilateral channels. These relate to: 1. Donor coordination and consolidation of programming, avoiding duplication and overlaps; 2. Reduced transaction costs as a consequence of reduced number of projects and single delivery channel; 35 World Bank (2007). Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards. Washington, DC: World Bank. Available at: http://siteresources.worldbank.org/EXTGLOREGPARPROG/Resources/sourcebook.pdf 41 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 3. Enhanced risk management, including progress and fiduciary monitoring by a monitoring agent; 4. Harmonisation of aid delivery procedures and consequent reduced burden on recipient; 5. Mobilisation of additional funds over what might have been contributed through bilateral channels only (and access for absent donors in fragile contexts); 6. Provision of a platform for development partner dialogue. Of these, benefits 1-4 are somewhat equivocal. The existence of the MDTF-N (or other MDTF) has not dissuaded donors to the MDTF-N from establishing a presence in Sudan, or from developing their own bilateral program of assistance to Sudan. In the case of these donors, the advantage of lowered transaction costs begins to diminish given that they need to establish their own administrative teams and procedures and that the work of administering an additional amount equivalent to their contribution to the MDTF-N, which is 36 relatively small , would only add to the overall workload at the margin. In the meantime, staff have the additional responsibility of guiding and contributing to decision-making and oversight of the MDTF-N. The presence of an MDTF in a country such as Sudan, therefore, does not in and of itself increase harmonisation (where donors also have a bilateral program) or reduce the burden on the recipient. Benefits 5 and 6, however, are additional and quantifiable. For countries such as Iceland and Greece, multi-donor vehicles remain the only opportunity for the channelling of donor assistance to Sudan. And this evaluation has found that the MDTF-N has functioned as an important platform for development partner dialogue in Khartoum – even for those who did not contribute financially to the Fund and participated only as observers. The scope for an MDTF to add value by these criteria depends on the number of similar pooling mechanisms available to bilateral donors. In the case of Sudan, funds have tended to focus investments on South Sudan, and the MDTF-N remains the only trust fund focused primarily on northern Sudan. (The UNDP-administered Sudan Common Humanitarian Fund has allocated 43 per cent to South Sudan.) These benefits therefore need to be borne in mind in the final analysis of cost effectiveness. Cost effectiveness in implementation: assessed against aggregate donor efforts The most thorough analysis would involve a compilation of the standard operating and project monitoring costs in Sudan for each donor implementing a portfolio the size of their donation to the MDTF-N. This information would then be compared to the MDTF-N’s administrative costs. Even without figures to demonstrate the differences between the 10 donors and a single MDTF, it is not difficult to hypothesize that the operating and administrative costs multiplied by 10 would be higher than one entity. This is particularly true in the case of Sudan, where operating costs are particularly high. By contributing to the MDTF-N and allowing the Fund to supervise and monitor development activities, the donors are reducing their costs and footprint, and more money is ultimately going into development spending. This is especially true in the case of recipient-executed MDTFs like the MDTF-N, where projects are implemented by government or national institutions, through PIUs staffed with detached government personnel (on leave of absence), plus externally recruited experts (primarily fiduciary staff). This provides a double benefit in terms of cost saving through the use of government staff while at the same time delivering a capacity and institution building benefit. When analyzing the cost effectiveness of bilateral donor support in Sudan, it is also important to consider the political environment and its impact on the interactions between the GoS and the international community. During the six years of the MDTF-N’s existence, there have been periods of difficult relations between donors and the Sudanese government. 36 UK net bilateral ODA to Sudan for the 2007-11 period, for instance, was £575.6 million, compared with a contribution of US$57.7 million to the MDTF-N. 42 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Many of the donors and observers who participated in the evaluation stated that the World Bank, with its mandate to refrain from political engagement, and the MDTF-N through the Oversight Committee meetings, provided mechanisms and a forum whereby donors could engage with the government, even during difficult times in their official bilateral relations. In this way, the MDTF-N presented a means for continuing communication that would not necessarily have been available to bilateral donors operating on their own. While this benefit may not be easily quantifiable, nevertheless, it is important when considering the outcome had the MDTF-N not existed. Taking into consideration the finding that the staff of the MDTF-N Technical Secretariat and the Monitoring Agent have played an important role in the capacity development of PIU/GoS staff with regards to procurement and project/financial management in particular, it is the conclusion of this evaluation that bilateral donors would not have been able to achieve this result through their own individual efforts. Had the MDTF-N not existed, project implementation would have been less harmonized, with the potential to be severely impacted by a decline in the relationship between the government and bilateral donors. The administrative costs would have been higher, as 10 donors attempted to administer, manage and monitor individual project portfolios. And it would have been difficult for donors to monitor to the high degree that is possible with the Monitoring Agent. Thus, while there may have been slow disbursement and implementation at the beginning of the MDTF-N due to lack of staffing and in-country resources on the part of the World Bank, and low capacity on the part of the PIUs, the current operation is much more streamlined and the long-term benefit of capacity building is now being seen. 3.2.2 Conclusion This analysis was difficult to conduct due to lack of data. Despite this challenge, this area is especially important for the overall cost effectiveness analysis, particularly as it relates to the question: Could the same broad objectives have been achieved with another alternative funding mechanism? The analysis has highlighted the following conclusions: ■ When compared with bilateral donor efforts at the aggregate level, it is likely that the MDTF-N has been more efficient and promoted greater harmonization, particularly in the case of donors which might not have had the capacity or political mandate to implement a large portfolio of development projects. ■ The MDTF-N has delivered real but unquantifiable benefits in terms of the negotiating position of the World Bank, based on its politically neutral stance, at a time of difficult relations between the GoS and the international community. It would have been difficult for individual donors to continue to work as effectively as the Bank did during that period. The analysis indicates that the MDTF-N was an appropriate response given the prevailing context in 2005, when few donors could find a way of engaging bilaterally in Sudan. It may continue to provide a delivery advantage over combined donor efforts given that not all of the donors have a large program, nor do they necessarily have significant implementation and influencing capacity. Over time, however, these advantages have become less important. Seven out of the nine bilateral donors to the MDTF-N have established a presence in Sudan and several are administering their own bilateral programs, of varying sizes. For donors without a Sudan program, or with only a few small projects, the MDTF-N’s aggregation of costs helps to reduce their footprint and delivers a cost effectiveness benefit. For more substantial donors, 37 however, this benefit declines. In fact, the DFID Country Programme Evaluation has commented that ‘with the proliferation of pooled funds that DFID has helped to encourage, it seems unlikely that any [efficiency, donor coordination or operational flexibility] benefits are currently being achieved to any significant extent, and there seems little reason for confidence that they will be in future’. In the case of DFID, disappointing performance of 37 DFID (2010). Country Programme Evaluation: Sudan 43 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report multilateral channels have, rather, proven a constraint: ‘with one of the strongest cadres of professional staff in the country, and with a capacity to move quickly and to adapt management systems to provide flexible support, DFID has been constrained to work mainly by seeking to improve the multilateral channels to which it has committed itself’. It should be noted that all multilateral channels supported by DFID were considered to have performed poorly, so it is unlikely to be the case that a different pooling mechanism (of those in existence at the time) would have been more effective than the MDTF-N. While the MDTF-N, therefore, has – especially in the early days – provided a cost-effective mechanism for engagement for all donors in aggregate it has not, necessarily, been a cost- effective choice for all donors individually. Finding 9: The MDTF-N provided a cost-effective mechanism for all donors in aggregate in the early days of its existence. In particular, it has delivered the non-quantifiable benefit of facilitating development partner dialogue in a difficult context. 3.3 Development impact This evaluation defines development impact as the intended and unintended changes, benefits, or other changes that occur in the lives of beneficiaries, perceived at the time of the evaluation, to which the MDTF-N projects have contributed. This definition incorporates the idea of ‘outcomes’, which the World Bank defines as ‘the uptake, adoption or use of project 38 outputs by the project beneficiaries’ . This evaluation examined several aspects of impact, including: ■ physical and financial assets ■ human assets ■ social capital ■ environment and resource base ■ institutions 39 ■ policies 3.3.1 Results achieved One of the key issues with results identified in this and earlier evaluations is the fact that there was not an overall results framework in place at the beginning of the Fund’s operations 40 identifying benchmarks by which progress could be measured . The frameworks that were in place did not clearly identify expected impacts, but instead, focused entirely on outputs. Without these benchmarks, it is nearly impossible to accurately attribute specific impact to the MDTF-N’s interventions. This issue was clearly outlined in the Country Portfolio Performance Review conducted in 2010. Efforts were then expended to ensure that outputs and outcomes could be measured effectively. 41 The Monitoring Agent’s latest quarterly report summarizes the results for the second 42 quarter of 2012 of the current MDTF-N portfolio of six active projects by implementing sector: 38 World Bank (n.d.). Results Terminology. Washington, DC: World Bank. Available at: http://siteresources.world bank.org/INTISPMA/Resources/383704-1184250322738/3986044-1250881992889/04_WorldBank_Results_ Terminology.pdf 39 IFAD (2003). A Methodological Framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation. Rome: IFAD. 40 Information provided by Technical Secretariat staff, Khartoum; World Bank (2010). Country Portfolio Performance Review (CPPR): Multi-Donor Trust Fund-National, Sudan. 41 nd PWC (2012). Monitoring Agent’s 2 Quarterly Report (1 April to 30 June 2012) National Multi Donor Trust Fund (MDTF-N) Administered by the World Bank. 42 Following the 30 June 2012 closure of the Blue Nile Start-Up Emergency Project (BNSSEP), the Public Sector Reform, Decentralization and Capacity Building Start-up Project (PSCAP), and the Abyei Start-Up Emergency Project, the six remaining active projects in the MDTF-N project portfolio are: Basic Education Project (BEP); Community Development Fund (CDF); Decentralized Health Systems Development Project (DHSDP); Improving 44 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report ■ Institutional development and capacity building 3,905 federal and state government officers trained. ■ Economic policy and management Engaged 31 microfinance institutions (MFIs) with reported clientele base of 210,800 business beneficiaries; introduced the Loan Tracking System (LTS) used by 11 local MFIs and one commercial bank; 200 tractors from the Agricultural Bank of Sudan to be distributed to Gum Arabic Production Associations (GAPAs) following the matching-grant procedures the Gum Arabic project put in place for the GAPAs; study of gum arabic completed which identified the gum arabic sector as a commodity well-placed for development of the sector in Sudan. ■ Productive sectors (rural and private sector development) The Livestock Project constructed six livestock markets which traded in 135,469 livestock heads, an increase from 62,837. Support and training to water associations across 28 water resource projects provided access to safe water to 156,547 beneficiaries in four states. 91 community animal workers were trained and deployed to pastoral communities. Impact Story 3: The whole school approach A fully functioning school requires more than classrooms, desks, and blackboards. Water points, toilets, and teacher housing are also important components to ensure that schools are completely operational. The Basic Education Project and the Community Development Fund – two MDTF-N projects active in the rehabilitation and construction of schools throughout Sudan – ensure that the schools have all of the facilities required to create a positive learning environment for both pupils and teachers. Given that many of the schools are located quite a distance from the closest town, teacher accommodation is often a necessity. As part of their whole school strategy, both projects ensure adequate dormitories are available for teachers as needed. The projects’ efforts positively impacted the lives of a group of female teachers in Deman, a village in Kassala state. Without the housing, they, and teachers like them in other villages in which the projects are active, would not have been able to travel daily to the schools due to the long commute. The dormitories allow them to remain in the village throughout the week and then return to their homes in town on the weekend. Through their inclusive approach, these projects ensure that schools and teachers have to provide the children in their communities with a better quality of education. ■ Basic social services The CDF and BEP have enrolled 160,330 pupils in basic schools in five states; 11,380 teachers trained in English, Arabic, mathematics and core program; ongoing work to institutionalize English language in the curriculum; rehabilitation of 514 basic schools; 114 health facilities rehabilitated; 81 communities provided with basic medical equipment; increase in outpatient consultations from a baseline of 0.16 consultations per person per year (2008-09 estimate) to 0.27 (30 June 2012); increase in pregnant women attending at least one antenatal care consultation from baseline of 48 per cent (2008-09 estimate) to 72 per cent (30 June 2012); 113 village midwives graduating from training school in Q2 12, with an additional 215 who graduated in Q1 12; 407 water resource points developed. ■ Infrastructure 446km of railway line; 365km of road network. ■ Livelihoods and social protection Training of 400 fishermen; establishing an ice factory and fishery union in Blue Nile state; 49 community livestock restocking groups engaged, with increase in livestock offspring from 8,051 to 8,325 in one quarter; support to 26 GAPAs through matching grant sub-projects providing 14 water reservoirs, 12 gum stores, 16 tractors, four water stations and hafirs. Livestock Production and Marketing (ILPM); Revitalizing the Sudan Gum Arabic Production and Marketing Project (GAPM); and the National Emergency Transport Rehabilitation Project (NETREP). 45 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report The evaluation team found that while the outputs may have been delivered as reported, the actual development impact varied by project, often within different project components, and by location, as further detailed below. Key areas of added value As the evaluation team observed and respondents reported, the implementation of the MDTF-N brought two very positive results over and above those anticipated by financial and procurement specialists given the Fund’s design. First, as discussed in a previous finding, the hands-on training and development by Technical Secretariat and Monitoring Agent staff in procurement and financial management regulations and procedures employed by the World Bank provided invaluable capacity development for the GoS and PIU staff. The hands- on training and interaction with Technical Secretariat staff was reported by respondents to this evaluation to be beyond the usual training package and was responsive to the 43 heightened need for capacity building in the Sudanese context . Project staff started to use computers, write reports, deal with structured and controlled procurement and work alongside international experts. Likewise, this additional training role built the capacity and professional qualifications of the TS and MA staff. Secondly, donor and observer group respondents repeatedly underlined the importance of the MDTF-N as a forum for engagement with the Sudanese government outside the sometimes difficult confines of bilateral relations. The MDTF-N was valuable in this respect to governments like Japan, who did not financially support the Fund but whose staff did attend Oversight Committee meetings, and the civil society and non-governmental organizations who were able to comment and provide feedback on OC meeting agendas and decisions. Finding 10: There were two key positive results arising from the MDTF-N, namely i) the capacity development of GoS, PIU, WB, and MA staff and ii) the MDTF-N’s role as a forum for communication and interaction with the GoS. Key negative results At the same time, there are also unintended negative results associated with the MDTF-N, which occurred primarily at the individual project implementation level. These negative consequences appear to be the result of insufficient analytical work prior to project design, the lack of baseline data, poor project design, challenges in implementation, and the scale of the need. Examples of some of the negative results that the evaluation team pinpointed during their project visits and discussions with respondents are: ■ According to changes in the legislation regulating the fishing sector, small-scale fishermen who are not part of the union will be banned from selling fish and will face fines if caught. While the aims of the MDTF-N project active in the Blue Nile fishing sector were to regulate and support this important livelihood sector, some of the poorest – those who are not members of the cooperatives or unions– will face fines if they are caught selling fish. ■ Under the BEP project, which supported decentralized decision-making, decisions made by states were not always conducive to delivering maximum benefit. Some states chose to expend a huge amount on one project, a Teacher Training Institute (TTI) which ultimately cost over US$1 million, while only rehabilitating three schools in the state. The result was a large building for which the state may not be able to cover the operating costs, raising questions of long-term sustainability; and fewer students benefiting at the school level, in that if more of the budgeted funds had been spent on rehabilitating and furnishing schools throughout the state more students would have benefitted from access to better quality education. Comparatively, another state divided the resources between rehabilitating an existing TTI at significantly less cost, while also rehabilitating schools throughout the state. The differences in the final impact on students and the education sector in each state are stark. While the objective – local ownership of decisions and projects – may be important, in this instance, it appears that questionable 43 Information obtained during interviews with PIU staff, Khartoum, Sudan 46 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report priorities resulted in high cost and low impact for the education sector in one state. With the benefit of hindsight, a stronger project design might have included a capacity analysis across the states to identify which states required support for analysis and planning of their component under the BEP. ■ Many of the MDTF-N projects, especially those that operate at the community level like CDF and Gum Arabic, are relatively small-scale projects that can only include a limited number of communities in their implementation. The MDTF-N aimed to pilot experiences to provide evidence to the government for potential replication. In the case of the Gum Arabic project, the evaluation team found that many of the communities who did not benefit from the project were angry at being excluded, as they felt that they should have been included in the project based on the selection criteria. When development needs are numerous, it is paramount to manage the expectations of communities, particularly when only a small number of communities within the area of implementation can participate in a project. This experience highlights the importance of communication at each stage of the project, particularly with representatives from nearby communities or organizations like unions within the sector who may not benefit from the project. Finding 11: Unintended negative results occurred primarily at the project implementation level and were related to project design, the operating context, and the capacity to manage expectations. Factors affecting implementation and results delivery Both earlier reports and respondents to this evaluation identified three primary factors affecting MDTF-N implementation and results delivery. ■ First, at the initial stages of the MDTF-N, there was a very limited number of quality staff at the Technical Secretariat and for some time the World Bank tried to address this through remote management, which slowed decision-making. Additionally, some projects experience a high turnover of TTLs, with one reporting seven different TTLs over a period of five years. M of the TTLs were located at the World Bank Headquarters in DC, travelling to Khartoum on short missions. Although the World Bank ultimately responded to in-country staffing needs, there was a negative impact on progress and the perception of the MDTF-N for some time. Since then, perceptions have improved in recognition of some very skilled and competent staff involved in the MDTF-N. Several respondents identified staff members whose consistent presence and institutional knowledge throughout each stage of MDTF-N implementation positively impacted implementation 44 and results delivery . ■ Secondly, the lack of capacity on the part of the Government of Sudan, particularly as it relates to the World Bank’s procurement and financial procedures, delayed project implementation at the beginning of the Fund’s operations. Intense capacity development and on-the-job training efforts since that time have addressed this issue and respondents estimated that there is now a cadre of at least 40 very highly trained procurement and financial management professionals within various ministries of the Sudanese 45 government because of these efforts by the MDTF-N . ■ Finally, the volatile security context dramatically affected project implementation, particularly in the highly prioritized Three Areas of Abyei, Blue Nile, and South Kordofan, where conflict recurred. Projects like BEP and NETREP were forced to abandon initiatives that were in the midst of implementation. The Abyei Emergency Start-Up Project implementation staff put the project on temporary hold to ascertain if they would be able to continue their activities at a later date, but were ultimately forced to close completely due to ongoing insecurity. Security continues to affect operations and monitoring efforts in these areas, although the area around Damazine in Blue Nile state was open to the evaluation team and the situation there seems to have stabilized. 44 Information provided by respondents during interviews and focus group discussions, Khartoum 45 Information provided by MDTF-N Technical Secretariat staff, Khartoum 47 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Finding 12: The three main factors affecting MDTF-N implementation and results delivery are the limited number of quality Technical Secretariat staff in the initial phase of implementation, the low capacity of Government of Sudan staff – particularly relating to the World Bank’s procurement and financial management procedures – and a volatile security situation. 3.3.2 The MDTF-N and World Bank Safeguard Policies One of the features of World Bank administered multi-donor trust funds is that they are not, in and of themselves, required to comply with the Bank’s Safeguard Policies, or its policy (4.20) on Gender and Development. Only some recipient executed trust funds – those that co-finance IDA/IBRD projects or that are larger than US$5 million – must follow standard Bank business processes for preparation, supervision, completion and risk management. Accordingly, compliance with Safeguard and Gender and Development policies was required for all of the MDTF-N portfolio given the minimum project size of US$5 million. A review of the 2005-08 period found that only some MDTF-N projects had completed safeguard and disclosure requirements. None of the projects had translated environment and social management frameworks in Arabic for local dissemination to communities and beneficiaries. And gender mainstreaming was weak. These concerns were reported in the 2010 CPPR and, although efforts were made to act on 46 the recommendations of the CPPR, a more recent review indicates that progress has been slow. The review found that Bank safeguard policies were applied for all MDTF-N projects, but most invoked a special policy for emergency projects that allows environmental assessment and other studies to be carried out after implementation begins. The review found that compliance was salutary – analysis of alternatives was not carried out and projects were screened according to checklists rather than the rigorous environmental impact assessments, environmental management plans and resettlement action plans prescribed under the Bank’s policies. The review also echoed the findings of this evaluation that public consultation during project preparation was minimal. Although civil society groups were part of the observation group to the OC, they reported that they had little say or substantive input in the decisions of the MDTF-N. Some projects, notably CDF, BEP and Livestock, were rather more consultative in the preparation of sub-projects. Critical documents were still not available in Arabic. Part of the reason behind the thin compliance is attributed to human resource capacity on the part of both the Bank and the GoS. The review reports that there was frequently no social safeguard specialist participating in supervision missions, even after 2009 when several Bank supervision missions requested that PIUs appoint a safeguards focal point responsible for safeguards implementation. While this level of compliance ultimately had no adverse environmental or social effects, the review finds that the approach of the MDTF-N has been short-sighted, missing an opportunity to put environmental protection and social sustainability on a stronger footing in Sudan during the coming years. Integration of gender and marginalized groups Similarly, the MDTF-N succeeded in delivering some positive results on gender despite very limited explicit acknowledgement of gender-related issues in the planning and design. It should be noted that Bank guidance on the operationalization of its gender policy focuses on work at the country and Country Assistance Strategy level to identify gender-related issues but, with no lending operations in Sudan, the Bank does not have a current country strategy. The 2008 Interim Strategy Note does mention addressing gender imbalances in recovery and development programs among its key themes, but this appears to be ex post based on the existing content of both Sudan MDTF portfolios rather than on a completed gender analysis. 46 Gross, D.R. & A. Kardash (2012) ‘Environmental and Social Safeguards Performance Review. Sudan National Multi Donor Trust Fund Portfolio’ 48 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Impact Story 4: Mainstreaming gender The MDTF-N strives to make its projects gender sensitive, conducting analytical work on how to mainstream gender in activities and developing projects and products that are gender-suitable. Several of the Fund’s projects have had a particularly high degree of impact on the lives of women. With the Community Development Fund, women have been incorporated into the community-level organization, decision making process, and project implementation from the beginning. During visits to communities in Kassala and Blue Nile State, the evaluation team found that the opportunities afforded to women through their involvement in the CDF had a positive impact on other parts of their lives as well – sometimes ensuring their greater involvement in other aspects of the community’s life. The Basic Education Project has addressed gender-specific needs at the design stage, ensuring that the school will be able to attract female students through proper fencing so girls will not be exposed, separate toilets, and building separate girls’ schools when needed. The Decentralized Health Systems Development Project has brought midwife training to women in more closed communities. This training provided women who normally would not have had access to adequate neo-natal and child-birth care a culturally sensitive avenue to improved medical care. As stated by one of the Technical Secretariat staff when discussing his project’s impact on 47 gender, ‘We hit the target by accident, not design’ . Indeed, from the results of this evaluation, it appeared that gender targets and indicators were not generally built into project design, nor were they part of the overall MDTF-N’s programming objectives. However, some projects, such as the health and CDF projects, conducted a Social & Gender Assessment during the project appraisal period and the findings were incorporated into project design. However, by the time the World Bank hired a gender specialist as part of its Technical 48 Secretariat staff, only the CDF project had collected baseline data on gender indicators . For an organization like the World Bank to overlook an important cross-cutting issue like gender in project design and MDTF-N operations is a grave oversight. It is an oversight that the World Bank eventually recognized and has since responded to and prioritized, hiring a gender specialist in 2009. The recommendations on gender that came from the CPPR in 2010 were taken on board and now projects are monitoring gender-related indicators. Some projects are performing particularly well on gender-inclusiveness and demonstrating community-sensitive gender programming. One of the best projects in this regard is the Basic Education Project. According to respondents, BEP incorporates gender-sensitivity into its programming at the design level, ensuring that schools provide separate toilets for girls, building separate schools for girls if required by the community, and building fences around schools so girls will not be 49 exposed . Likewise, the Livestock, Gum Arabic, and Blue Nile Emergency Project, through its work with the fisheries sector, have programming activities specifically targeting women’s livelihoods. And the health project provides midwives training schools, to increase the number of women with access to pre- and post-natal care in remote areas, while also providing context-appropriate livelihoods for women. The midwives training schools established in two Kassala state communities (Waggar and Hameshkoreib) that did not allow women to attend training outside of the community provided context-appropriate livelihoods opportunities for marginalized women. Impact Story 5: Livelihoods for women Part of the activities undertaken by Blue Nile State Start-Up Emergency Project targeted women from communities around Roseries Dam, helping them to develop livelihoods in the fishing sector. Through the project’s efforts, women like Suad from Kirma village received training in new fish drying and processing techniques. Suad, along with other residents from the Blue Nile state fishing communities travelled to Uganda where she learned proper hygienic practices, trained on new 47 Information obtained during interview with Technical Secretariat Staff, Khartoum 48 Information obtained during interview with Technical Secretariat Staff, Khartoum 49 It should be noted that CDF, which is also active in building schools, follows similar practices. 49 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report processing techniques and equipment, and received new uniforms and boots. Halima and Fatima, women from Suad’s village who were unable to attend the training in Uganda, attested to the importance of the new information that Suad shared with the other women in Kirma following her training. They stated that the new hygienic practices were already resulting in big changes in the general hygiene practices in the village. CDF is another project that is making significant impact on the lives of women. Through the project, women have opportunities for direct engagement in community decision-making through the Community Development Committees (CDCs), although the evaluation team found that their level of engagement varied by location. Some women were particularly active and engaged in the development of their community. In one community in North Kordofan, women sold their jewellery to purchase pipe, dug a trench, and laid the pipe 50 themselves in order to provide clean water to the community school . Conversely, during another project visit, the women were extremely reluctant to speak and it was clear from the manner in which the discussion proceeded that the men in the CDC were accustomed to 51 dominating the proceedings . Despite the ongoing challenges, clear progress on gender- inclusivity and awareness of gender issues has been made in some very closed 52 communities . At the organizational level, however, observations by the evaluation team demonstrated that progress in gender inclusiveness, particularly in the staffing of many PIUs, still leaves room for improvement. The majority of the focus group discussions with PIUs involved only male participants – because, with some exceptions, the PIU staff is mostly male. At the LIU level, there seemed to be a greater balance, particularly in the CDF, where many social mobilizers are women. Of the seven projects evaluated more closely, all PIU managers were male, with the exception of the BEP. With regards to the inclusion of marginalized groups, the MDTF-N seemed to target several groups directly, as demonstrated by the selection of the area for the construction of a road in Kassala, linking the Main Khartoum-Port Sudan Trunk Road to Hamashkouraib, which was in an area where a marginalized tribe is located. Similarly, the CDF, also in Kassala, has expended considerable effort, reaching out to more closed and marginalized tribes in that state, and has seen a great deal of progress opening the communities to the idea of 53 development and girls’ education . Likewise, all projects located in the Three Areas target marginalized groups and the Emergency Start Up projects focus exclusively on these groups. Finding 13: The MDTF-N did not include gender directly in its initial project design or monitoring indicators; however, this oversight was later addressed and projects are now reporting success in several areas related to gender inclusiveness and access to opportunity. 3.3.3 Role of stakeholders in MDTF-N success In order to appropriately assess the role of the various stakeholders in the MDTF-N’s 54 success, their intended roles, as outlined by the founding document , should be summarized: 50 Information provided by CDF project beneficiaries during a site visit in North Kordofan. 51 Observations during a project visit in Kassala State. 52 Information obtained during discussions with CDF LIU, Kassala and Blue Nile States and MDTF-N Technical Secretariat staff, Khartoum. 53 Information provided during discussions with CDF LIU staff, Kassala, and MDTF-N Technical Secretariat Staff, Khartoum. 54 World Bank (2005). Memorandum of the President of the International Bank for Reconstruction and Development to the Executive Directors on a Proposal for the World Bank to Administer Two Multidonor Trust Funds for Sudan. 50 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report ■ World Bank Administer the Technical Secretariat and work jointly with the government to receive and review proposals and undertake technical appraisals. As administrator, the Bank carried the sole fiduciary responsibility for the MDTF-N. Actual role With regards to its role as the fund administrator and Technical Secretariat, the World Bank fulfilled its role, employing a Monitoring Agent to support it in carrying out its fiduciary responsibilities. Despite a slow start with limited staffing and resources in Sudan, as well as the extremely limited capacity of the national counterparts, which resulted in delays and negatively impacted the rate of project implementation, the Bank quickly took on the responsibility of providing intensive capacity development to PIU staff in its procurement and financial management procedures, as well as the additional hand- 55 holding required to address the unique situation and low capacity in Sudan . In this regard, the World Bank can be viewed as contributing beyond the scope of the role laid out in the Memorandum. ■ Donors Have representation on the Oversight Committee and participate in the Sudan Consortium, which was to bring together on a biannual basis all donors and representatives of civil society and private sector to discuss past performance, and future directions/priorities with the goal of providing transparency and coordination. The Sudan Consortium was tasked with assessing progress against benchmarks and it also was to provide a forum for consultations on social and economic development and donor pledge renewal. Actual role The donor group participated actively in the Oversight Committee; however, the activities of the Sudan Consortium were less clear, particularly since the civil society observer group respondents indicated that the role of civil society was limited. The donors definitely pushed priorities and future directions for the Fund, which may not have necessarily coincided with the development objectives of the Sudanese government, with an emphasis on delivery of results. An example of this is the clear priority placed on the Three Areas by the Netherlands, the MDTF-N’s largest donor. When many projects were forced to close in these areas following the renewed violence, and with a change in priorities of the Dutch legislature, the Netherlands withdrew from the Fund. The development needs in Sudan are still great, but political realities in the Netherlands and Sudan changed within the six years the Fund has been in operation. Nevertheless, it is a good example of the decisive role that the donors could play in setting the Fund’s priorities and direction. Another role that the donors played was to ensure that the ‘value package’ relating to cross-cutting and safeguard issues (such as gender and environmental issues) was applied across the portfolio. ■ Government of Sudan As part of the Oversight Committee the GoS was to play a key role in submitting proposals consistent with national priorities and embodied in the budget. Only proposals endorsed following this procedure were eligible for MDTF-N funding, so the GoS was a gate-keeper in project selection. The GoS was also to be the primary implementer of projects with the goal of strengthening local and civil society capacities, effective and accountable governance, and budget transparency. Where appropriate, it should propose implementing agencies and consult on contract arrangements with the implementing agencies with the Administrator. Actual role As the primary project implementer and major contributor to the Fund, the GoS played a significant role in its success. Like the World Bank, there was an extremely slow start 55 Information provided during discussion with MDTF-N Technical Secretariat Staff, Khartoum and telephonic interviews. 51 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report which negatively impacted project implementation due to low capacity with World Bank procurement and financial management regulations and procedures. However, like the Bank, as capacity grew through the support provided by the Technical Secretariat and Monitoring Agent staff, the rate of project implementation increased. As identified earlier, there is now a cadre of well-trained civil servants within the GoS as a result of their experience with the MDTF-N. The GoS was also active in the Oversight Committee, and identified projects and implementing agencies. Finding 14: The three main stakeholders – the World Bank, the donor community, and the Government of Sudan – each played a separate but significant role in the success of the Fund. 3.3.4 Sustainability Ahead of the scheduled closure of the MDTF-N, the Technical Secretariat undertook an assessment of project sustainability in 2011 that identified potential issues impacting sustainability and presented specific actions designed to strengthen the chances of ongoing operations. The assessment measured sustainability by the following seven parameters: ■ technical resilience ■ financial resilience ■ community ownership ■ economic and social impacts ■ environmental impacts ■ ownership by government counterparts 56 ■ replicability of the project model Based on these indicators, the report assigned a sustainability rating to each of the remaining projects in the MDTF-N portfolio. The seven projects examined by the evaluation received the following ranking: 57 Table 3.3 Sustainability ratings of the seven projects included in this evaluation Project Sustainability rating Basic Education Project Likely Key outstanding issues: ■ Correct few infrastructure defects ■ Improve functionality of schools (water, fences, benches, etc.) ■ Ensure state governments provide for the operation and maintenance of TTIs Blue Nile Emergency Start- Likely Up Project Key outstanding issues: ■ Operating costs and detailed budget for the CDC Community Development Likely Fund Key outstanding issues: ■ Provision of water and electricity to schools and health centres ■ Provide separate drinking basins for animals and humans at water yards ■ Correct structural defects in some construction 56 World Bank (2011). MDTF-N: Assessment of Program Sustainability 57 All information drawn from World Bank (2011). MDTF-N: Assessment of Program Sustainability 52 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Project Sustainability rating Gum Arabic Uncertain Key outstanding issues: ■ Improve institutional framework with regards to monitoring ■ Ensure that reforms to the sector are fully implemented ■ The microfinance component should be incorporated into the national microfinance structure Judiciary Not rated Key outstanding issues: ■ Draw lessons from this project for further activities supporting the Judiciary sector NETREP Uncertain Key outstanding issues: ■ Completion of roads to technical standard ■ Inclusion of roads into the priority network managed by the National Highways Administration PSCAP Uncertain Key outstanding issues: ■ A follow-on project is needed, particularly around the implementation of new HR policies and fiscal procedures A subsequent functionality assessment by the Monitoring Agent, focusing primarily on requirements related to infrastructure, identified areas for improvement in project activities 58 prior to the closure of the Fund, detailing a required budget for each project . The findings of this evaluation generally align with the sustainability rating and key outstanding issues; however, the following areas were highlighted by the evaluation team for specific areas of concern with regards to efforts to enhance sustainability: ■ The ice factory and fish market supported under the Blue Nile Emergency Start-Up Project exhibit signs that there may be long-term sustainability issues around the disenfranchisement of key stakeholders and what appeared during a project visit to be the requisition of project resources by a state ministry with tangential connections to the 59 project’s intended main beneficiaries . ■ In the Gum Arabic project, the organizational structure of some community-based Gum Arabic Producer Associations, the primary beneficiaries of the project, suggests that some have been dominated by local leaders and they have little benefit for community members who are not closely connected with the local leadership or are otherwise marginalized. Additional evidence from project visits and focus groups with beneficiaries, as well as observations during site visits, indicates that the information available at the national PIU level does not necessarily reflect the reality on the ground. ■ There is evidence that, at this point, the long-term operability and sustainability of the TTIs is questionable and that concrete and realistic operating plans are required. In the Red Sea state, the evaluation observed design and construction faults resulting from low-quality engineering oversight and poor performance of consultants and contractors. It was unclear whether there was the technical capacity or planning to operate the TTI for long-term sustainability. The situation in North Kordofan was much better; however, the refurbished building did have some technical defects and it was handed over without 58 PWC (2012). Functionality Assessment Report : World Bank – National Multi Donor Trust Fund (MDTF-N) Administrator. 59 The evaluation team was providing with information indicating that this issue had been resolved and documented in writing by State authorities in the weeks following the site visit. 53 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report note or a plan to amend these defects. And while the Director of Training was planning a schedule of training activities, there was no money in the 2012 budget for them and it was unclear to the evaluation team whether funds would be allocated in the 2013 60 budget . At the project level, sustainability varies by project and often by and within states. Respondents to this evaluation identified several basic issues impacting sustainability, such as the buy-in and commitment to the project by the government at every level (federal, state, and locality); constraints on budgets and resources, particularly at the state and locality level; and the fact that the critical mass needed for self-supporting and sustainable development at 61 the community level has not yet been reached . Conversely, respondents also distinguished positive factors already supporting the sustainability of various projects, including the level of transfer of knowledge and skills to PIU staff within the GoS; the high degree of ownership of some projects like the CDF, especially by some state and locality governments; and the current discussion on replication, expansion, and/or the integration of some projects into state/locality and federal budgets. Finding 15: Long-term sustainability varies by project, within project, and by location; however, it is impacted by both negative and positive factors that can be mitigated or enhanced to increase the probability of a project continuing after the MDTF-N closes. 60 Information obtained during interviews at the TTIs in Red Sea and North Kordofan states and independent engineering inspections conducted during the site visits. 61 Information drawn from discussions with PIU and LIU staff, throughout Sudan, and MDTF-N Technical Secretariat Staff, Khartoum 54 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report 4 Lessons learned and recommendations for a way forward One of the central goals of this evaluation was the synthesis of the institutional knowledge and experiences of the various MDTF-N stakeholders into lessons learned and recommendations for a way forward following the expected closure of all MDTF-N projects by 30 June 2013. Section 4.1 presents key lessons learned, while recommendations for a way forward are summarized in Section 4.2 and are organized by recommendations for i) ongoing operations and ii) the post-MDTF-N environment. 4.1 Lessons learned The lessons learned presented here were drawn primarily from the respondents themselves. In each interview and focus group discussion, participants were asked to identify the main lessons they will take away from their involvement with the MDTF-N and to provide recommendations for future operations. The evaluation team has analyzed their responses, identifying recurring themes which are presented in the first part of this section. Additionally, the team has distilled other lessons learned through their field consultations, which are included as well. Like the findings of the report found in Section 3, the lessons have been organized into the three main focus areas of this evaluation: institutional and organization relevance, cost effectiveness and development impact. 4.1.1 Institutional and organizational relevance The majority of lessons learned identified by respondents related to the design, operations, and implementation of the MDTF-N and its projects. Due to the number of recurring themes related to institutional and organizational relevance that emerged from the field consultations, they have been organized under three main subheadings: ■ Design ■ Operations ■ Implementation Design Lesson 1: Ownership must be built into the design of both the Fund and its projects from the beginning. One of the lessons repeated by respondents was that ownership needed to be present at each level of the Fund’s activities from the design phase onward. At the Technical Secretariat level, this meant that the Government of Sudan – particularly the key ministry partner, the Ministry of Finance and National Economy – needed to be strongly invested in the Fund’s activities and outcome. At the project implementation level, the ministries (or in the case of a few projects, UN agencies) involved in implementation supporting the PIUs with staff needed to understand the objectives and value the project added to their operations. Likewise in the states and localities, government buy-in, appreciation for the project’s goals and outcomes, and investment in its success was important for sustainability. Particularly for projects with implementation at the community level, close collaboration with communities is the foundation for success, as seen with the CDF. On the more tangible indicators, the MDTF-N enjoyed a high degree of ownership, particularly in its early stages. The GoS committed substantially to the Fund, with its 57 per cent counterpart funding as well as the numbers of staff it was able to allocate to project implementation and its willingness to establish project implementation units within its ministries to manage implementation. On less tangible indicators, however – those which can identify more durable ownership – the MDTF-N struggled. This was in part due to the difficult geopolitical context, which undermined dialogue between donors and government. The early problems faced by the Bank in establishing fully functioning operations in Sudan also did some considerable damage to its credibility, resulting in a degree of disengagement by the GoS. This was reported by respondents who were aware of the decision by the 55 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Council of Ministers to decrease the level of active involvement due to the relatively small 62 size and perceived impact of the fund . Lesson 2: Development priorities, programming sectors and project-level activities must be determined by need, using a combined bottom-up/top-down approach. While the JAM identified the main sectors for development, it did not prioritize them, resulting in what was called a ‘shopping list’. Evaluation respondents generally felt that the JAM and CPA did not provide clear priorities. Although the IOC-N, at its meeting in July 2005, identified indicative priorities, based on government priorities, for the 2005-07 period, it is not evident that this was subsequently formalized and/or clearly communicated to stakeholders. Programming should be needs-based, with a combined bottom-up/top-down approach incorporating all levels of stakeholder and beneficiary perspectives and voices. National policy documents need to establish clearly ranked priorities to improve the targeting of development activities at the local community level. The Community-Driven Development (CDD) programs – a relatively new and innovative approach to development in Sudan – working in tandem with national policy reform and effective implementation outline a successful strategy. The efforts in the gum arabic sector provide a good model of this type of integrated bottom-up/top-down strategy, where policy and regulatory changes undertaken by the national government were combined with support to local small-level producers provided by the project. While only a pilot program in size, the Gum Arabic project’s work with communities offers a model for increased bottom-up accountability systems that can have positive impact across the sector. Impact Story 6: Increasing productivity for small-scale gum arabic producers Members of the female Gum Arabic Producers Association in Blue Nile State have seen their productivity and income increase as a result of the grants they received through the Revitalizing the Sudan Gum Arabic Production and Marketing project. Prior to their involvement with the project, they were borrowing money at very high interest rates from the merchants who sold their product. The grants from the Gum Arabic project ensured that they did not have to rely on the merchants for loans and could keep a higher percentage of their profits. Some GAPA members have been able to purchase animals and poultry with their increased profit, expanding and diversifying their livelihoods. Lesson 3: Ensure that the Fund and its projects have clear, specific, and attainable objectives and targets. One of the weaknesses in design pointed out by Country Portfolio Performance Review in 2010 was the fact that development objectives were broadly defined. The lack of baseline data exacerbated by the difficulty of obtaining data, as well as insufficient evidence obtained through analytical work due to the lack of time for analysis prior to implementation, contributed to this problem. Likewise, some projects had very unrealistic and sometimes unrelated goals. This impacted the ability to track progress and made it difficult to design and implement activities responsive to those broad objectives. An important parameter for the setting of Fund and project goals is the relative volume of funding it is able to mobilize. Although a number of respondents commented on the small size of the MDTF-N, relative to the GoS budget, the MDTF-N was not entirely insignificant, particularly when compared to the amount of government resource expended on development and services. The JAM estimated Sudan’s financing requirement at US$4.3 billion. Oslo I raised US$222.9 million in donor commitments to the MDTF-N, 5.1 per cent of the total requirement. Assuming 50 per cent counterpart funding, the MDTF-N covered in the region of 10 per cent of Sudan’s requirement, excluding Southern Sudan. This was an important indicator of what could have been expected of the MDTF-N alone – as more of a lever than a solution to Sudan’s development challenges. Development objectives 62 Information obtained during interviews with current and former Government of Sudan staff, Khartoum, Sudan 56 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report needed to be framed and articulated accordingly. This has, in part, contributed to the very high expectations of the MDTF-N discussed further below, which were difficult to meet. Lesson 4: When designing projects, be aware of the appropriate time frame for the targeted activity and capacity context, and allow sufficient time for sustainable impact. Time is a key success factor in project implementation. Underestimating the time required for effective implementation of a given activity can distort its projected outcome and consequently limit the overall project impact. Similarly, exaggerating the allotted time can result in a waste of resources and is likely to cause the community to lose faith in the development project. An activity’s temporal framework must be realistic and take into consideration the availability of resources and the capacity, or lack thereof, of those targeted to benefit in order to maximize impact and ensure sustainability. Time must also be allotted for the collection of baseline data during the project’s design phase. Baseline data is crucial in providing the evidence base for project activities and results indicators. In the case of the MDTF-N, project implementation was delayed, resulting in several extensions. In some cases, this was due to lack of implementing capacity; in others, outside factors delayed the project. Also, the objective of the project needed to be taken into consideration when establishing the project timeline. Projects that first require changing attitudes and procedures, as well as policy and strategies, or that depend upon institutional capacity to implement, require a longer implementation period than projects concentrated on physical infrastructure. Lesson 5: Design the Fund structure and individual projects to fit the operating context. Emerging from decades of conflict, Sudan faced enormous development challenges. Additionally, there was a great lack of capacity when it came to Bank protocol and procedures. The remote and difficult environment in the states is also demanding, 63 dramatically impacting operations in many states . When designing a fund and its projects, a thorough assessment and understanding of the context must be incorporated into both the structure of the fund’s operations and those of its project portfolio. This would include building in time for intensive capacity development for PIU staff prior to project implementation; extending the project cycle to account for anticipated delays in implementation as required; and tasking a Monitoring Agent to provide hands-on support to PIU staff throughout project implementation. The Technical Assistance Facility, for the value of the World Bank contribution to the MDTF- N, partly addressed this requirement in providing technical support to the GoS in the identification and preparation of projects. However, the TAF provided primarily policy-level support – it was not designed to deliver the kind of capacity building that the hand-holding activities of the Technical Secretariat and the Monitoring Agent eventually provided. An extension of the TAF could have supported MDTF-N projects through implementation; alternative initiatives might include a ‘bolt-on’ donor-funded TA project dedicated to supporting the MDTF-N, or the secondment of donor resources to implementing ministries to assist with implementation. (Although there were secondments in the early days, these were to the TS to fill critical staffing positions that the Bank had not been able to fill at early stages.) Lesson 6: Incorporate environmental, social and gender issues into the design of the Fund and its projects from the beginning. The MDTF-N did not originally include gender indicators in its programming and although subsequent reviews found that many of its projects were reaching women and girls, this was mostly through accident rather than design. Gender inclusiveness – a concept that does not merely single out women, but includes men in the discussion of women’s needs and how to address them – should have been integrated into the initial design of the Fund and its 63 Several of the PIU staff, and particularly LIU staff in the states indicated that the rainy season affected their ability to monitor and implement. This meant that advertisements for contractors needed to take into account the timing of the rainy season, to mitigate potential delays in construction. This was the case for the CDF and NETREP projects. 57 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report projects. Additionally, the MDTF-N’s projects should have more clearly addressed the inclusion and needs of other marginalized groups, such as the disabled. Other cross-cutting issues, such as the environment and social safeguards also needed to be built in from the start. Lesson 7: Identify the appropriate managing unit for each project. It was the experience of some projects that the decision of where to situate the PIU seemed to be fraught and politically motivated, resulting in problems with implementation. Examples include the Judiciary project and the PSCAP. To the extent possible, the responsible partner for the PIU should be identified on the basis of appropriateness rather than on political or personal motivations in order to enhance smooth delivery of project outputs. A transparent selection procedure can and should, in addition to relevance and qualification, also take into consideration the scope for institutional leverage. Where decision-making and resource management for local sub-projects remains in Khartoum or state capitals, this sometimes served as a disincentive for local engagement. Thus, from an operational point of view, at sub-project level greater decentralized autonomy might have eased implementation. However, the Basic Education Project offers a cautionary tale for decentralization, with arguably insufficient oversight and management at the federal level, leading to problems in strategic prioritization and management at the state level. A more balanced approach, with federal-level engagement on policy and strategic issues, centralized procurement procedures, and state and locality engagement and responsibility for operations, would be more appropriate. Lesson 8: Build from existing structures – do not reinvent the wheel. Project design and the selection of implementing partners works best when existing and functioning structures are relied upon. In this way, duplication is avoided, allocative efficiency of resources is ensured and national counterparts have the incentive to preserve existing structures that work and strive to improve them. Ownership is underpinned and sustainability strengthened. The MDTF-N has followed and continues to follow this approach. Lesson 9: Determine the capacity of counterparts and implementing partners during the design phase. An intervention cannot be deemed successful or sustainable unless it leaves behind solid structures which can be relied upon to maintain and build on its achievements and successes. In post-conflict contexts, such as Sudan, such structures are normally either lacking or are extremely weak. The capacity of government organizations and implementing partners needs to be assessed at the very start of an intervention, and appropriate mechanisms for capacity building and support must be determined in relation to the context and existing needs. Incorporating capacity development into program and project design increases the probability of success and ensures that lessons learned and successful aspects of the project will be incorporated into national structures and will continue to be capitalized upon following closure. Operations Lesson 10: Anticipate, monitor and manage expectations. One of the primary lessons that was repeated by many respondents is the need to manage expectations early. This applies to the Fund as an instrument and to individual projects. In the early days of the MDTF-N, there was much excitement surrounding the development opportunities and pressure to implement quickly, particularly in the conflict-affected areas of South Kordofan, Blue Nile, and Abyei. Several factors which impacted the slow rate of implementation were limited PIU capacity during the initial phases, the challenging environment in remote locations, and early staffing and resource shortages at the Technical Secretariat. With the slow implementation rate, high expectations turned to disappointment and severe criticism. The World Bank responded to much of the criticism, appointing a full complement of in-country staff and prioritizing capacity development for PIUs. The valuable 58 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report lesson for all stakeholders is that expectations need to be managed from the outset and open discussions around what can be delivered in a given timeframe must be held early. Along these lines, it is important to acknowledge the inherent tension between responding quickly to stabilize conflict-affected populations and longer-term development objectives. It is necessary to act quickly in an emergency operation, while building local capacity and both the stakeholders and project design must take this into consideration and respond with flexibility. The mix of ‘quick impact’ projects with longer-term projects is a common approach to expectations management, as a means of generating buy-in and ownership early on through the delivery of short projects targeting ‘low-hanging’, highly visible benefits while the more institutional projects are in the process of building momentum. The design of the MDTF-N did this only weakly. The ‘quick start’ projects were limited to those in the Three Areas, where the risk of disruption to projects was high; these were also the projects mostly implemented through UN agencies (presumably for reasons of access), limiting the scope for Bank oversight and direction. The experience of the MDTF-N has shown that traditional interventions in health, education and small community development projects, where implementation modalities are familiar and relatively straightforward, can yield tangible and visible results within a relatively short timeframe. Had the MDTF-N been front-loaded with more of these kinds of projects (the CDF was one of the first projects but the BEP was the last), early expectations might have been satisfied to a greater extent. Lesson 11: It is important to commit a sufficient number of qualified staff from the onset for best implementation. One of the main criticisms of the World Bank’s management of the MDTF-N in the early days was that it did not provide a sufficient number of qualified staff in-country from the beginning. There were logistical reasons for some of the staffing constraints. The World Bank had not had an office in Sudan for about 13 years before the Fund was established, so this proved to be a challenge initially. It was initially difficult to recruit qualified staff to work in Khartoum. With less than six months between the decision to establish the MDTF-N and the first Oversight Committee meeting, there was not enough time for an institution like the World Bank to ensure full capacity on the ground. The small staff that they could provide initially shared office space with UNDP for quite some time and did not have access to the World Bank servers during that period. Despite these challenges, as Administrator, the World Bank is ultimately responsible for the staffing and equipping of the Technical Secretariat. While it took some time for it to respond adequately to these needs, it did respond and currently most respondents report satisfaction with Technical Secretariat operations. Nonetheless, in future, the Bank’s management should place a high priority on providing a full complement of staff in a timely manner. Additionally, where possible, speakers of the native language – Arabic in the case of Sudan – should be selected for Technical Secretariat staff. Lesson 12: More could have been done to increase the visibility of the GoS role as a Fund supporter and project implementer. The Government of Sudan provided 57 per cent of the MDTF-N’s total funds, yet most beneficiaries met during the project site visits and even many of the PIU staff were not aware of the extent of the government’s involvement. Many beneficiaries considered the CDF project an NGO, rather than a program implemented with government funding with primarily government employees. While there was high visibility for the government during the opening ceremonies of projects, particularly with NETREP and some BEP projects, the role and extent of the government’s financial support for the MDTF-N could have been strengthened. Lesson 13: Donors and other stakeholders could play a positive and active role in the project monitoring visits. Donors were very active in the Oversight Committee meetings and provided clear guidance on areas they supported; however, there was room for greater involvement in the monitoring missions and field visits. Regular donor participation in project site visits provides a first- person view of the development activities and impact which cannot be conveyed on paper. 59 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report While donor representatives may have limited availability due to their professional obligations, when the opportunity is available, their regular involvement in field visits should be encouraged. Implementation Lesson 14: Partnerships make a difference in implementation. Respondents repeatedly pointed to the positive impact that partnerships have on implementation. These partnerships include other MDTF-N projects, relevant government organizations at the state and locality level, international organizations and NGOs, and local civil society organizations, amongst others. And, particularly with community-led development implementation – an innovative approach in the Sudanese context – a strong partnership with community members is key for success. In this respect, the social mobilizers are cornerstones of implementation. Lesson 15: Intra-project collaboration should be based on similar project objectives and implementation strategies. Intra-project collaboration can be both positive and negative and should be undertaken when projects share similar programming objectives and implementation strategies. The partnership between CDF and the Gum Arabic and BEP projects illustrates this point well. Because the CDF project was active in the education sector, building schools with communities, it would appear that the best approach would be for a partnership with the BEP. However, their implementation strategies were not similar, which ultimately led to the decision to move forward with new school construction separately. This does not mean that collaboration or cooperation between different projects is not possible or desirable – there are excellent examples of positive intra-project collaboration among the MDTF-N projects – it does mean that such partnerships should build on the strengths and objectives of the projects in question. For example, the CDF and Gum Arabic projects assumed similar community-led development approaches and their partnership was more sustainable. Lesson 16: Projects can be well-designed and well-organized, but negatively impacted by the operating environment and other aspects outside of the PIU’s control, so flexibility is key. Respondents provided examples of projects in which time and care had been taken in the design and implementation, but factors outside of their control negatively impacted the implementation, sometimes completely halting the project or severely limiting its outcome. In Kassala, the CDF had taken care to bring in several partner organizations to build, furnish, and provide meals at a community girl’s school. A seemingly good relationship with the community was established and the project was implemented. However, as a result of political manoeuvring behind the scenes in the community, the school never opened. Likewise, the Abyei Emergency Start-Up Project was, by all accounts, well-designed – taking into consideration the causes and responses to conflict, the needs of the community, and working closely with key community leaders to build trust to implement the project. The area relapsed into conflict, and although project staff waited quite some time to allow the conflict to resolve so activities could be renewed, they were ultimately forced to close. The context and the outcome in both of these cases was beyond the control of the project staff and provide good examples of the difficulties in operating in a conflict-affected and challenging environment. Thus, flexibility by staff is key and a willingness to shift project focus and learn from experiences is vital. By all accounts, Bank management of the MDTF-N did demonstrate a relatively high degree of flexibility. Its response to its early staffing problems and major restructuring of a large portion of the portfolio, including reallocation of resources across projects, are indicators of a willingness to adjust to emerging realities in a rapidly changing operational context. 4.1.2 Cost effectiveness A multi-donor trust fund is clearly a cost-effective alternative in a context in which it is nearly impossible for donors to operate bilaterally. This was the case in post-conflict Sudan, and threatens to be the case again currently. The MDTF-N’s cost effectiveness comprises not 60 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report only the delivery of project outputs where there might otherwise have been none, but the softer benefits which have, in the case of the MDTF-N, for a period of time, been somewhat catalytic – the provision of an entry point for donors (including in the Three Areas), the development of project formulation and implementation capacity, and the facilitation of a platform for dialogue with government. Compared with other donor interventions – to the extent that there is a comparison to be made – the MDTF-N has not necessarily been a cost-effective choice for all donors individually. However, donors have appreciated its value in providing a platform for donor engagement with the GoS under difficult circumstances, enabling some of them to stay engaged. Lesson 17: The cost effectiveness of a multi-donor trust fund relates primarily to its relevance to a given operational context; the MDTF-N has enabled donor engagement through a difficult period, and built national capacity to support future donor programming, although the relative cost of this is not clear. 4.1.3 Development impact Lesson 18: The capacity development role played by the World Bank and Monitoring Agent was a significant, positive impact of the MDTF-N. A majority of the respondents repeatedly praised the capacity development role played by the World Bank and Monitoring Agent. Several stated that these sorts of activities and ‘hand- holding’ exceeded that of the capacity development support normally provided by World Bank staff in their experience. The extensive capacity development and training activities was required for project implementation to proceed in the context of extremely low capacity in Sudan. These activities made a real and lasting impact for the PIU staff. At the same time, the professional capacity of the World Bank and Monitoring Agent staff was also enhanced. Lesson 19: Development impact is not immediate – it must be given time. The unrealistic expectations for quick development impact and progress did not take into account several key factors that make quick impact impossible. First, in general, long-term sustainable development impact often takes more than the typical three-year project cycle. In a post-conflict context like Sudan, while some immediate results for projects in education or transportation could be seen quickly, other projects – particularly those that require changes in attitudes or practices – take much more time and involve greater effort and involvement to ensure successful implementation and long-term sustainability. Additionally, in projects based on the Community-Driven Development (CDD) model, the full extent of the less tangible impact around the empowerment of communities and marginalized societal groups – particularly in contexts where communities are being included in the decision making process for the first time – may take time to become evident and measureable. Lesson 20: Policy reform is the foundation of sustainable change and can have wide-ranging positive impacts. Some of the MDTF-N projects lobbied for key policy reform and strategy formulation as the foundation for their efforts. For example, the Gum Arabic project worked towards the successful dissolution of the monopoly in that sector. As a result, opportunities increased for small producers and the technical study produced by the project highlights the income- generating potential this sector has for the national economy. In this way, efforts at the national level to revise policies can have dramatic impact at the local level and can be successfully coordinated with community-level implementation. 4.2 Recommendations for a way forward 1. It is recommended that the World Bank remain engaged in Sudan. Given the enormous development challenges facing Sudan, there is certainly the need for the World Bank to remain fully engaged in Sudan, if not through further funding to the MDTF- N, then through other funding mechanisms managed by the World Bank, as the unresolved issue of debt arrears precludes normal World Bank operations in the country after the MDTF- 61 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report N closes. Currently, a great deal of capacity exists within the Technical Secretariat. Considering the time, financial resources, and human effort expended to arrive at the present point, it would be a massive waste not to capitalize on this experience to further Sudan’s development goals as outlined in its recent Poverty Reduction Strategy Paper (PRSP). 2. It is recommended that donors continue to support development activities throughout the whole of Sudan, not only in specific areas. The donors to the MDTF-N provided US$249 million to the MDTF-N, providing the only funding support for a few of the projects where counterpart funding was not available. Although the MDTF-N is not as large as some funds, this is still a substantial amount of money and programming activities have targeted very underdeveloped and conflict-affected areas of Sudan. At this point, much attention is being paid to the area of Darfur in the build- up to the upcoming donor’s conference; however, the development needs are great throughout the whole of Sudan. There is room for continued engagement by donors. A trust fund offers a good option for donors at this time, as it allows them to support development in the country without a large footprint. 3. The MDTF-N Technical Secretariat should follow up the Sustainability and Functionality Reports with a clear action plan for sustainability. As the MDTF-N’s scheduled closure date approaches, much attention has been paid to ensuring and strengthening sustainability. The Sustainability Report published in 2011 and the Monitoring Agent’s follow-up Functionality Report of March 2012 provide concrete summaries of issues impacting sustainability by project and a budget for necessary activities prior to the closure of each current project. Regular discussions about sustainability should continue; however, more pressing is the need for the Technical Secretariat and each open project to outline a clear action plan for the remaining months, prioritizing actions that will strengthen sustainability. 4. Should further funding of the MDTF-N or a new fund be established, the objectives and programming areas should be streamlined, potentially following a sector-based approach. The current portfolio, which responded to the development sectors identified by the JAM, was wide-ranging in its scope. Any future programming should be more streamlined and targeted, aligned with the development priorities from the PSRP, as well as sector policies and strategies for the coming five years. In addition to the scale-up of some of the successful projects currently within the MDTF-N portfolio, some potential areas of focus that have potential for considerable impact are: ■ rural development; ■ capacity development, targeting not only the federal level, but the state and local level as well; ■ gender, particularly in regard to livelihoods; ■ basic education ■ health ■ NETREP Phase II ■ governance ■ private sector 5. Clearly prioritize development activities from the outset. The JAM did not prioritize development sectors, so provided limited guidance in programming MDTF-N activities. The balance of the portfolio was further skewed by the large volume and ‘lumpy’ nature of counterpart funding, with 77 per cent going to just three projects – the NETREP, Currency and Census projects – which did not necessarily align with Sudan’s identified development needs. Future efforts need to be based upon more clearly identified development priorities. Where national development strategies fail to provide adequate guidance (as may be the case with the 2012-16 Five Year Development Plan), 62 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report there is a need for a preliminary coordinated donor exercise to agree, in consultation with the beneficiary government, programming priorities. 6. Do not rush the design and implementation of a fund and its project portfolio. Problems with project design arose when various stakeholders pushed for fast implementation of the Fund and its projects. Resultantly, projects were designed quickly, with little appreciation for the context or lack of capacity. Any future fund should allow sufficient time for information-gathering – namely analytical work and baseline assessments, discussions in-country with a variety of stakeholders, and design. Likewise, a three-year project cycle is too short for a challenging context like Sudan, thus more time should be allowed for implementation and impact. 7. Conduct risk assessments and incorporate risk into the initial design of the project, and identify potential mitigating measures with close follow-up and flexibility to allow appropriate responses to changes throughout the project implementation cycle. Risk is inherent in fragile and conflict-affected states. Rather than avoiding risk, projects should be flexible and responsive to it. Conducting a risk assessment prior to the establishment of a fund, as well as risk assessments at the project level if projects are to be implemented in particularly risky areas, is essential. The findings and recommendations of these assessments should then be incorporated into the design and implementation at both the fund and project levels. 8. When operating in conflict-affected areas, conflict impact assessments should be included in project design and projects should be responsive to conflict and its causes in both design and implementation. Similarly, for projects implemented in conflict-affected areas, a conflict impact assessment should be conducted during and inform the design stage of the project. Mechanisms for response to conflict and its causes should be included in design and implementation. Some of these mechanisms could be staff well-trained in conflict analysis, management, and response; workshops with project beneficiaries around the causes and responses to conflict; amongst others. 9. Incorporate sustainability measures in the design phase of both the fund mechanism and individual projects. Sustainability must be a concern from the very inception of a fund and its projects. A fund and its projects should incorporate clear procedures to ensure sustainability from the design phase. Steps to strengthen long-term sustainability include: ensure government and beneficiary buy-in at all levels relevant to the fund and projects from the beginning; include the targeted beneficiaries in the project design process in order to accurately capture and address their perspectives and needs; locate the PIU within an appropriate government body with a unambiguous plan for the government to assume funding responsibility on a mutually agreed schedule; work within partnerships with other organizations for implementation, including the appropriate civil society groups, NGOs, and international organizations. 10. Setting a counterpart share needs to be realistic in a post-conflict context. The 50 per cent counterpart share in the MDTF, although agreed to by the GoS, was unprecedentedly high. This became evident when the GoS failed to meet some of its counterpart commitments, resulting in implementation delays. This matter was redressed through the MDTF-N’s Country Portfolio Performance Review exercise, although no resolution was reached for the three emergency projects for which the GoS had completely suspended its funding commitments – a significant factor in the project implementation delays. The government, as recipient of the grant, maintained that its share in an emergency project had been set at an unfairly high level, even though it had initially agreed to the counterpart arrangement. This experience highlights the importance of fully understanding the impact of financial constraints and economic fluctuations for post-conflict countries when designing funds. 11. Close interaction and engagement with local authorities is essential from the outset. 63 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report The Bank should broaden its engagement beyond its official counterpart, the Ministry of Finance and National Economy. The Bank’s engagement with the state governments, once projects face counterpart funding issues, proved successful and suggests a closer dialogue on a regular basis would be productive. Although state resources are dependent on federal transfers and are usually limited, local authorities place a much higher priority on recovery and development needs. Despite financial limitations, local authorities can be resourceful, as evidenced in some of the MDTF-N projects where states financed counterpart funding gaps and the O&M obligations for the majority of project activities. Project interventions are also more visibly felt by local authorities who are directly involved in the implementation, feel a strong sense of ownership and, as a result, can apply pressure at the federal level for additional resources. 64 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 1 Documents consulted Policy context (2005) Comprehensive Peace Agreement between The Government of The Republic of The Sudan and The South Sudan People’s Liberation Movement/Sudan People’s Liberation Army (2011) Sudan. Interim Poverty Reduction Strategy Paper Joint Assessment Mission Sudan (2005) Volume I: Framework for Sustained Peace, Development and Poverty Eradication Volume II: Cluster Costings and Matrices Volume III: Cluster Reports National Council for Strategic Planning, ‘The Five-Year Plan (2007-2011)’ World Bank (2008) ‘Interim Strategy Note for the Republic of the Sudan’ Design documents Sudan Multi Donor Trust Funds National and Southern Sudan (N&S). Final Project Proposals/Project Proposal Documents World Bank (2005) ‘Memorandum of the President of the International Bank for Reconstruction and Development to the Executive Directors on a Proposal for the World Bank to Administer Two Multidonor Trust Funds for Sudan’ World Bank Restructuring Papers for: (2010) South Kordofan State Emergency Project (2011) Basic Education Project (2011) Decentralized Health System Development Project (2011) National Emergency Transport Rehabilitation Project (2011) Sudan Gum Arabic Production and Marketing Project (2012) Decentralized Health System Development Project (2012) National Emergency Transport Rehabilitation Project (2012) Sudan Gum Arabic Production and Marketing Project (2012) Community Development Fund Project Annual Reports & Progress Reports Sudan National Multi-Donor Trust Fund (2006) ‘First Progress Report. Reporting Period July 1 - December 31, 2005’ (2007) ‘Second Progress Report. Reporting Period January 1 – December 31, 2006’ (2008) ‘Third Progress Report. Reporting Period January 1 – December 31, 2007’ (2009) ‘Fourth Progress Report. Reporting Period January 1 – December 31, 2008’ (2010) ‘Turning The Corner. 2009 Annual Report’ (2011) ‘Consolidating Gains. 2010 Annual Report’ World Bank (2007) Sourcebook for Evaluating Global and Regional Partnership Programs: Indicative Principles and Standards (2010) ‘Country Portfolio Performance Review (CPPR): Multi-Donor Trust Fund-National, Sudan’ 65 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report (2011) Implementation Status Reports (2009) Implementation Completion and Results Report. Fifth Population Census Project (2010) Implementation Completion and Results Report. Technical Assistance Facility (2012) Implementation Completion and Results Report. South Kordofan Start Up Emergency Project (2012) MDTF-N/World Bank Funded Studies CY2010-2011 Donor Pledges & Contributions as of 6-30-2012 MDTF-N Portfolio as of 6-30-2012 NS-Source & Use of Funds as of 6-30-2012 Results Terminology Independent reviews Development Initiatives (2006) ‘Review of Trust Fund Mechanisms for Transition Financing. Phase 2 Report. Scanteam/Norway (2007) ‘Review of Post-Crisis Multi-Donor Trust Funds. Final Report’ (2007) ‘Review, Sudan Multi-Donor Trust Funds. Phase One’ (2009) ‘Short Appraisal of the Sudan Multi-donor Trust Fund (National)’ (2010) ‘Flexibility in the Face of Fragility: Programmatic Multi-Donor Trust Funds in Fragile and Conflict-Affected Situations’ Rothenbuhler, H.M. (2008) ‘The Strategic and Financial Framework for Phase II of the Sudan MDTF-N’ PWC (2006-12) Monitoring Agent’s Quarterly Reports on the National Multi Donor Trust Fund (MDTF-N) Administered by The World Bank Other Barakat, S., Chard, M., Jacoby, T., Lume, W. (2002) ‘The Composite Approach: Research Design in the Context of War and Armed Conflict’, Third World Quarterly, Vol. 23, No. 5, pp. 991-1003 Barakat et al (2012) ‘What is the track record of multi donor trust funds in improving aid effectiveness? An assessment of the available evidence’ Connolly, D. (2010) ‘Participatory Research in Programme Evaluation: the Mid-term Evaluation of the National Solidarity Programme in Afghanistan’, in R. Bowd and A. Ozerdem (eds) Participatory Research Methodologies in Development and Post-Disaster/Conflict Reconstruction, London: Ashgate Downs, C. (2011) ‘Operational Effectiveness of the UN MDTF Mechanism’ Foster, M. et al (2010) ‘Country Programme Evaluation. Sudan’, UK Department for International Development GHK (2012) ‘Independent Review of the Iraq Trust Fund’ Gross, D.R. & A. Kardash (2012) ‘Environmental and Social Safeguards Performance Review. Sudan National Multi Donor Trust Fund Portfolio’ IFAD (2003). A Methodological Framework for Project Evaluation: Main Criteria and Key Questions for Project Evaluation Technical working group of the Federal Ministry of Health (2010) ‘Free Health Care For Under-Five Children And Pregnant Women In Northern Sudan: Progress so far And Recommendations for the Future’ OECD-DAC (2010) ‘Transition Financing: Building a Better Response’ PWC (2012) ‘Functionality Assessment Report’ 66 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report World Bank (2009) ‘Peace and Beyond. The Development Story of the Multi Donor Trust Fund - National (MDTF-N) in Sudan’ (2011) ‘National Sudan Multi-Donor Trust Fund (MDTF-N) Factsheet’ (2011) ‘Public Expenditure Tracking Survey (PETS) for Northern Sudan. Case Study of the Health Sector’ (2011) ‘MDTF-N: Assessment of Program Sustainability’ The World Bank Group (2011) ‘Modified Cash Basis Trust Funds Report on Internal Control over Financial Reporting & Combined Statements of Receipts, Disbursements and Fund Balance’ World Bank Independent Evaluation Group (2011) ‘Trust Fund Support for Development. An Evaluation of the World Bank’s Trust Fund Portfolio’ Geographic Focus of MDTF-N Projects 67 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 2 Respondents Name Position Location World Bank and MDTF-N Technical Secretariat Staff Mohamed Yehia Abdelkarim World Bank Financial Management World Bank, Khartoum Specialist Yousra Abdelrahaman Executive Assistant World Bank, Khartoum Hayidat Alyan Gender Specialist World Bank, Khartoum Mohmaed Elkheir Osman Beshir NETREP Co-TTL World Bank, Khartoum Rupert Bladon PSCAP TTL Telephonic Interview Moustapha Ould-Elbechir Sr. Procurement Specialist World Bank, Khartoum Aisha Elbereir Project Analyst - Acting Project World Bank, Khartoum Manager, Judiciary project Yousif Elfadil World Bank Post-Conflict Operations World Bank, Khartoum Officer Mohamed Osman Hussein Consultant, former TTL World Bank, Khartoum Aymen Musmar BEP Co-TTL World Bank, Khartoum Tesfamichael Nahusenay NETREP TTL Telephonic Interview Omer Nasir Former Engineer for Judiciary World Bank, Khartoum project, currently WB consultant for BEP Elizabeth Ninan BEP TTL Telephonic Interview Jack Pellekaan Former PSCAP TTL Telephonic Interview Isabel Soares Senior Operations Officer World Bank, Khartoum Allasane Sow World Bank Country Manager World Bank, Khartoum Endeshaw Tadesse CDF TTL World Bank, Khartoum Government of Sudan Officials Alsharif Mohamed Abbas National Congress Party & El Obeyed, North Kordofan Chairman of the CDF future Committee Mr. Abdalla Member of Parliament El Obeyed, North Kordofan Abbas Mohamed Abdelrahman Legal Advisor to the Governor Governor's Office, Dmazine, Blue Nile Abd Alsalm Yahia Abobaker Executive Manager of Financial BNEP Offices, Dmazine, Blue Minister's Office Nile Omer Ibrahim Ahmed Planning & Development BNEP Offices, Dmazine, Blue Administration Nile Mohamed Musa Ahmed Manager of Fish Administration BNEP Offices, Dmazine, Blue Nile Hazaifa Abd Alaziz Manager of Trade & Cooperation BNEP Offices, Dmazine, Blue General Administration Nile Abdalla Altom Alemam Ministry of Human Resources Legislative Assembly, Development Khartoum Mahgoub Abd Alazim Ali CDF 212/M BNEP Offices, Dmazine, Blue Nile 68 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Name Position Location Badr Aldeen Hamid Ali Manager of Trade & Quality Control BNEP Offices, Dmazine, Blue Administration Nile Mohamed Saeed Ali Planning & Development - Foreign BNEP Offices, Dmazine, Blue Funds Coordination Administration Nile Abbas Ibrahim Alkhalifa Financial & Administrational Affairs BNEP Offices, Dmazine, Blue Manager - Ministry of Social Care Nile Yasir Mohamed Daffa Allah Observer of Organizations & Foreign BNEP Offices, Dmazine, Blue Funds Administration Nile Mohamed Alemam Alnour Altadamun District / Alkarkar, Baloka Legislative Assembly, Unit Khartoum Jaafar Awadalla Sennar State DG of Forestry Sennar State Department of Forestry Offices Abdalla Ibrahim Babikir General Manager of Architectural BNEP Offices, Dmazine, Blue Planning & Public Facilities Nile Mahgoub Abd Alrahman Dhawlnour Manager of Organizations' BNEP Offices, Dmazine, Blue Coordination & Educational Projects Nile Administration Hind Fageeri CDF El Obeyed, North Kordofan Ms. Fatima Member of the Committee on El Obeyed, North Kordofan Women and Children Murghani Glood MoFNE MoFNE, Khartoum Murghani's Assistant MoFNE MoFNE, Khartoum Omer Hajjam MoFNE MoFNE, Khartoum Dr. Salih Ali Hamid General Manager of Agricultural, BNEP Offices, Dmazine, Blue Animals, Wealth, & Forests Nile Ahmed Ibrahim Hamid Social Studies Manager - Ministry of BNEP Offices, Dmazine, Blue Social Care Nile Omer Mohamed Ibrahim Administrational Manager BNEP Offices, Dmazine, Blue Nile Azhair Madeni Idris General Manager of Basic Education BNEP Offices, Dmazine, Blue Ministry Nile Hassan Jafr Head of IDB, EU funds, regional MoFNE, Khartoum institutions, acting director Mr. Khalid Member of Parliament El Obeyed, North Kordofan Mairy Jaimsey Koko South Kordofan, Hebian District Legislative Assembly, Khartoum Ms. Layla Chairman of the Social Affairs El Obeyed, North Kordofan Committee Mr. Mohammed Chairman of the Committee on El Obeyed, North Kordofan economy and investment Haider Abdalla Mohamed Financial & Administrative Affairs BNEP Offices, Dmazine, Blue Manager - Ministry of Finance Nile Eisam Aldin Ahmed Mohamed Executive Manager of Rosaires BNEP Offices, Dmazine, Blue Administration Nile Sidiq Abd Alrahman Mohamed Acting Manager of General BNEP Offices, Dmazine, Blue Administration of Fish Nile 69 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Name Position Location Sidiq Musa Executive Manager of Dmazine BNEP Offices, Dmazine, Blue Administration Nile Osheik Mohamed Oshiek Director of education Administration Basic School, Red Sea Botheyna Saad Rahma South Kordofan Kailek District - Legislative Assembly, Alfoula - Babanousa - Aldaib Khartoum Almairm Salikh Gen. Siddiq Ex-Chair OC and State Minister of PACT Consulting Offices, Finance Khartoum Mr. Suliman Member of Parliament El Obeyed, North Kordofan Dr. Abdalla Hamad Ahmed Ali Tengafi General Manager of Social Care & BNEP Offices, Dmazine, Blue Human Affairs Ministry Nile Donors Marco Bourguignoni Project Writer Italian Cooperation Office Carlo Cibó Development Cooperation Office Italian Cooperation Office Head Pedro de Figueiredo First Secretary Development Swedish Embassy, Khartoum Cooperation, Swedish Embassy Eskedal Trine Jøranli Head of Operations, Norwegian Swedish Embassy, Khartoum Embassy Guiseppe Masala Project Manager Italian Cooperation Office Ben Mellor Co-Chair of OC , DFID Swedish Embassy, Khartoum Yvonne Stassen Deputy Head of Mission/Head Swedish Embassy, Khartoum Development Cooperation, Dutch Embassy Observer Group Dr. Hajhamad NGO Observer Forum His offices, Khartoum Don McPhee Plan International, INGO Observer Skype Forum Kazuyo Mitsuhashi Aid Coordination Officer, JICA Embassy of Japan, Khartoum Patrik Olsson Swiss Development Cooperation Swiss Embassy, Khartoum Other Development Actors Sayed Aqa UNDP Country Manager UNDP, Khartoum Ali Alzadari UNDP Representative UNDP, Khartoum Pierre Muller Government of France French Embassy, Khartoum Georg Siep German Embassy German Embassy, Khartoum Assefaw Tewolde UNDP UNDP, Khartoum Laura Zampetti EC Delegation Social Services World Bank, Khartoum Project Officer Monitoring Agent Steven Kibira Manager, PWC PWC, Khartoum James Mwangi PWC PWC, Khartoum Nicodemus PWC PWC, Khartoum John Thiong'o Project Manager, PWC PWC, Khartoum 70 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Name Position Location Titus Waniala Monitoring & Evaluation Expert, PWC, Khartoum PWC Basic Education Implementing Staff Sayed Abdim PIM, Senior M&E, Ministry of BEP, Khartoum Education Amal Alkashif Programme Manager, Ministry of BEP, Khartoum Education Abdelbagi Daifalla Finance Officer, Ministry of BEP, Khartoum Education Roweida Ibrahim Unit Accountant, Ministry of BEP, Khartoum Education TTI Contractor El Obeyed, North Kordofan Blue Nile Emergency Start-Up Project (BNEP) Staff Aisha Hassan Ali Abbakr Hay Alzohor Society Member Dmazine, Blue Nile (BNEP) Rabih Hassan Abda Allah Secretary of Finance (Inner Village Dmazine, Blue Nile 'Algarya Aldakhla') Ali Mohamed Adam Almara Gadid Dmazine, Blue Nile Hussain Abd Alrahman Blue Nile Emergency Project, Dmazine, Blue Nile Project Engineer Mohamed Hassan Omer Abd Alwahid Fishers Union Society, Sub Dmazine, Blue Nile Secretary in village (Alaradieba), State's Union Executive Office Member Badreldin Operations Manager, Blue Nile BNEP Offices, Khartoum Emergency Start Up Program Elhussien Elkhazin Project Coordinator , Blue Nile BNEP Offices, Khartoum Emergency Start Up Program Khalifa Ali Mohamed Godos Blue Nile Emergency Project, Bout Dmazine, Blue Nile Unit Co-ordinator Salma Bashir Humaida Blue Nile Emergency Project, Dmazine, Blue Nile Procurement Officer Hussein Procurement Officer, Blue Nile BNEP Offices, Khartoum Emergency Start Up Program Ishag Eisa Mohamed Hay Alzohor Society Member Dmazine, Blue Nile (BNEP) Mohamed Musa Mohamed Executive Manager of Cooperative Dmazine, Blue Nile Union (BNEP) Alnazir Ahmed Osman Fishers Union, Sub-secretary in Dmazine, Blue Nile village (Karma) Women's Fisheries Group Emergency Project Dmazine, Blue Nile Capacity Building of the National Judiciary (Judiciary) Staff Omer Nasir Project Engineer Khartoum Community Development Fund (CDF) Staff Taha Musa Abdalla CDF - Kassala, Procurement Officer CDF, Kassala Abdulrahman CDF Seteet, Kassala 71 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Name Position Location Ahmed Tahir Adam Ahmed CDF - Kassala, Unit Accountant CDF, Kassala Nasr Ahmed CDF Procurement Manager CDF, Khartoum Mahjoub Abd Alazim CDF, LILI Manager Dmazine, Blue Nile Kamal Awu CDF Communications Officer CDF, Khartoum Abdullah Bushra CDF – Locality Manager South CDF, Khartoum Kordofan Hamiz CDF - South Kordofan (Kadulgli CDF, Khartoum Manager and State Coordinator) Salah Abdul Hassan CDF - South Kordofan CDF, Khartoum Ahmed Alser Ibrahim CDF Community Mobilization Officer Dmazine, Blue Nile Rashid Kamal CDF Finance Manager CDF, Khartoum Mohamed Osman Mohamed CDF - Kassala, Project Manager CDF, Kassala Hameshkorieb, Kassala Rural, Gash Kassala Mohamed Osman Mohamed CDF LIU Manager CDF, Kassala Omer Badawi Omer CDF, Executive Unit Engineer - Dmazine, Blue Nile Geisan Yassir Mohamed Omer CDF - Kassala, Unit Engineer CDF, Kassala Abd Alaziz Baraka Sakin CDF, LILI Manager Dmazine, Blue Nile Samia Suliman CDF, Community Mobilization Dmazine, Blue Nile Officer Hassan Tisane CDF CDF, Khartoum Abdel Gahdir Turakwi CDF Executive Manager CDF, Khartoum National Emergency Transport Rehabilitation (NETREP) Staff Abd Alrahman Eldaw Hamid Ministry of Transport, Sudan Ministry of Transportation, Railways Component Khartoum Modawi Programme Coordinator Ministry of Transportation, Khartoum Omer Ahmed Mohamed Ministry of Transport, NETREP's Ministry of Transportation, Financial Advisor Project Manager Khartoum Summaya Ahmed Mohamed Ministry of Transport, NETREP / Ministry of Transportation, Road Component Project Manager Khartoum Abd Alrahman Sharief Ministry of Transport, Sudan Ministry of Transportation, Railways Corporation Assistant Khartoum Project Manager - SRC Component Public Sector Reform, Decentralization, and Capacity Building Program Support Project (PSCAP) Staff Elham Abdallah PSCAP - M&E Officer PSCAP Office Saif Eldawla Beshir PSCAP Deputy Project Director PSCAP Office Abdullah Bushra PSCAP Project Director PSCAP Office Yousif Karrar PSCAP - Procurement Officer PSCAP Office Muhalab Musaad Mohamed PSCAP - Financial Management PSCAP Office Officer 72 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Name Position Location Revitalization of the Sudan Gum Arabic Production and Marketing Project (GA) Staff Alamin Abdalla Wadidrwda GAPA Department of Forestry, Um Rawaba, North Kordofan Mohammed Abdelbagi Ganeem GAPA Department of Forestry, Um Rawaba, North Kordofan Babiker Abdlmoula GAPA Finance Manager - Dahab Sennar State Department of Algaba, Aldali Locality Forestry Offices Mohammed Ahmed GAPA President - Dahab Algaba, Sennar State Department of Aldali Locality Forestry Offices Brima Bilal Samandia GAPA. Department of Forestry, Um Rawaba, North Kordofan Mohammed Elkhier Umjazeer GAPA Department of Forestry, Um Rawaba, North Kordofan Alfatih Hamid Gum Arabic, Bout Unit, Guidance Dmazine, Blue Nile Officer Hashim Hamsa Project Coordinator and M&E Gum Arabic Offices, Khartoum Manager, Department of Forestry Alfadil Ibrahim Sennar State Gum Arabic President Sennar State Department of Forestry Offices Ismael Omer Ismael Samandia GAPA Department of Forestry, Um Rawaba, North Kordofan Najmeldein Isamel GAPA Secretary - Dahab Algaba, Sennar State Department of Aldali Locality Forestry Offices Abdalla Mohammed Sennar State Gum Arabic Member Sennar State Department of Forestry Offices Ahmed Mohammed Amanalla GAPA Department of Forestry, Um Rawaba, North Kordofan Gum Arabic PIU Staff Project Coordination, M&E Officer, Department of Forestry, Gum Finance Officer, Procurement Officer Arabic Offices, Khartoum Gum Arabic Union 4 Gum Arabic Project Officers Ministry of Agriculture, 8 Members of the Gum Arabic Union Dmazine, Blue Nile Men's GAPAs Gum Arabic Ministry of Agriculture, Dmazine, Blue Nile Women's GAPAs 2 Gum Arabic Project Officers Ministry of Agriculture, 5 Members of All-Women GAPAs at Dmazine, Blue Nile Alharza 73 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 3 Project site visits Community State Type of Project Basic Education Project (BEP) Red Sea Teachers' Training Institute Dordeib Red Sea School North Kordofan Teachers' Training Institute Blue Nile Emergency Start-Up Project (BNEP) Dmazine Blue Nile CDC Dmazine Blue Nile Fishery and Ice Factory Community Development Fund (CDF) Abu Hashim Blue Nile Health Center School Alazaza Blue Nile Community Center Health Center School Algisim Blue Nile Health Center School Youth Center Haroun Blue Nile School Water Point Baryay Kassala School Deman Kassala School El Remaila Kassala School El Sawany North Kordofan School Shiekhan North Kordofan School Revitalization of the Sudan Gum Arabic Production and Marketing Project (GA) El Obeyed North Kordofan Gum Arabic Market Masudiya Sennar Nursery Sennar Town Sennar GA Guest House Training Hall Singa Town Sennar Project Office 74 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 4 Interview protocol Semi-Structured Interview Questions The following list, presented in the Inception Report, contains some of the relevant questions from which the evaluation team drew during semi-structured interviews, structured around the three areas of focus for this evaluation: ■ institutional and organizational relevance ■ development impact ■ cost effectiveness Interviewees were asked to reflect on lessons learned and recommendations from their experience with the MDTF-N. Table A4.1 Semi-structured interview questions Criteria Interview Question Stakeholder INSTITUTIONAL AND ORGNAIZATIONAL RELEVANCE 1A. Alignment of priorities 1A.1: Have the MDTF-N projects contributed to the ■ GoS between the JAM, CPA, the rebuilding of conflict-affected areas in Sudan? If so, (federal/state/local) Government’s five-year plan can you provide an example? ■ Bank staff (2006-11) and the MDTF-N ■ Bank conflict experts 1A.2: Have any of the projects had negative impact portfolio. ■ Donors on the rebuilding of conflict-affected areas in Sudan? ■ Implementing Can you provide an example? Partners 1A.3: What are the top three development challenges ■ Civil Society in Sudan? Organizations 1A.4: Do the MDTF-N projects respond to these challenges? To what extent? 1A.5: Has the security situation impacted project implementation? If so how? 1A.6: Have projects been able to continue in the most conflict-affected areas? 1A.7: How has the MDTF-N supported your institution? 1A.8: Do you feel that the level and type of support was relevant to your institution’s needs? Can you give an example? 1B. Architectural design 1B.1: How would you characterize your experience ■ GoS working with the MDTF-N? ■ Bank ■ Implementing partners 1B.2: Does the MDTF-N organizational structure all for the timely approval, funding, and implementation of projects? Can you give an example? 1B.3: Do the rules and organizational structure allow necessary adjustments to be made to projects in a timely manner? 1B.4: Are national/state/local priorities addressed in project design? 1B.5: Is information about project progress and implementation reported in a timely manner? 1B.6: What role does the Monitoring Agent serve? 1B.7: Is the information on project progress and MDTF-N operation from the Monitoring Agent’s 75 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Criteria Interview Question Stakeholder reports integrated into project operations? 1B.8: Was the MDTF-N design responsive to the needs of a variety of groups in the communities where its projects were implemented? If so, how? If not, why? 1B.9: Were environmental safeguards built into the design of the MDTF-N? If not, how were environmental issues addressed at the design and policy level? DEVELOPMENT IMPACT 2A. Results achieved by the 2A.1: What have been the results of the project? ■ GoS MDTF-N. ■ Bank 2A.2: Has the project achieved everything it set out to ■ Implementing accomplish? Partners 2A.3: Were there positive results from the project that ■ Civil Society were not planned when it was first implemented? Can Organizations you provide an example? ■ Beneficiaries 2A.4: Were there positive results from the project that were not planned when it was first implemented? Can you provide an example? 2A.5: Generally, what do you think the overall results of the whole MDTF-N have been? 2A.6: What have been the key factors that have impacted the implementation of the MDTF-N and its projects? Can you give examples? 2A.7: Were the projects and their activities implemented in a timely way? 2A.8: If not, can you explain why they were not? 2A.9: If yes, what were key factors that ensured timely implementation? 2A.10: Have the beneficiaries of the projects been satisfied with the projects? 2A.11: Have there been beneficiaries who have been dissatisfied? What were their reasons for this? 2A.12: To what extent has the capacity of the relevant federal/state/local government institutions been strengthened since 2005? 2A.13: Do you feel that the government at each level is committed to continuing to carry out efforts to build on the progress of the MDTF-N? 2A.14: How involved is the government at each level in the MDTF-N projects? 2A.15: Are there management mechanisms in place? 2A.16: Has the community been active in the implementation of the project? 2A.17: If projects are closed, are there ongoing activities that have been continued through other efforts? 2A.18: If projects are ongoing, are there plans to continue its activities following the end of the MDTF- 76 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Criteria Interview Question Stakeholder N funding? 2A.19: Have other funding sources been identified? 2A.20: Are the concerns of women and other socially marginalized groups taken into consideration for project development and integration? 2A.21: Are women and other marginalized groups actively participating in the project? If not, what is the reason? If yes, what has been the impact of their involvement on the project? 2A.22: Has the community experienced conflict over access to this project? If so, have they been resolved? How? If not, why? 2A.23: What sort of impact has this project had on the environment? (Negative, Neutral, Positive) 2A.24: If there was a negative environmental impact, has it been addressed? If so, how? If not, why? 2B. Roles played by the 2B.1: How would you characterize your Government of Sudan and organization’s support of the MDTF-N? donor partners in contributing to program 2B.2: What has been the level of your organization’s involvement in the operation of the MDTF-N and the success implementation of the projects? 2B.3: Is the funding from your organization provided on a regular and predictable basis? If not, what are the reasons? 2B.4: What assistance-in-kind or technical assistance did your organization provide in support of the MDTF- N and its projects? 2B.5: Did your organization publically support the MDTF-N? If not, what were its concerns? 2B.6: Generally, how would you rate the extent to which your organization contributed to MDTF-N success? 2C. Sustainability of activities 2C.1: While the MDTF has been in operation, what post-MDTF has been done to ensure that local ownership has been promoted? 2C.2: Has the GoS played an active role in decision- making and approval processes? 2C.3: Has the GoS played a role in implementation at the state/local level? If so, what? 2C.4: Has capacity building training been conducted in conjunction with fund and project activities for the relevant national/state/local government officials? 2C.5: How would you rate the level of ownership of fund activities and programs? 2C.6: What would you recommend to strengthen ownership? 2C.7: How would you rank the compatibility between the Bank planning process and the planning processes of the relevant GoS ministries? 77 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Criteria Interview Question Stakeholder 2C.8: Can you identify some of the challenges? 2C.9: What are some things that have worked well? 2C.10: Is there a plan in place to continue activities aimed at reaching the objectives of the JAM? COST EFFECTIVENESS 3.1: Have there been examples of where project costs exceeded expectations? If so, why? 3.2: How has the MDTF-N project costs measured against other MDTFs in Sudan? LESSONS LEARNED 4A. What have been the key 4A.1: What would you say are the three key lessons lessons emerging from the to take away from the experience of the MDTF-N? MDTF-N? 4A.2: What recommendations would you give to the MDTF-N or a similar program? 4A.3: Do you think that the MDTF-N should continue its work? If so, what should its main focus be with regards to programming? 4B. What would be the key 4B.1: What recommendations would you give to the recommendations for a new MDTF-N or a similar program? MDTF-N or similar program? 4B.2: Do you think that the MDTF-N should continue its work? If so, what should its main focus be with regards to programming? 78 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 5 Focus group discussion protocol A1.1 Agenda for focus group discussions with project beneficiaries These lists provide some of the relevant questions from which the evaluation team drew when drafting the agenda for focus group discussions. State: District: Community: Researcher: Date: Type of focus group: Number of participants: I. Introduction Welcome the respondents and describe the purpose of the focus group and the study overall. II. Discussion A. Nature of the local economy and development 12. What are the sources of income in your community? What are the opportunities in your village for agriculture, home-based enterprise, and other employment? 13. What are the challenges your community faces with regards to the economy? Follow up questions and prompts: – If people are facing greater hardship, how does this show? 14. What impact so far has this MDTF-N project had here on the local economy? Follow up questions and prompts: – Do you see a difference in your community’s economy after the project was implemented? – Is it a positive or negative difference? – Are there other projects in the community which also contribute to the community’s economy? – Is there anything that could strengthen the impact of the project on the economy? – What impact has this project had on the environment? (Negative, Neural, Positive) – If there was a negative impact, has it been addressed? If so, how? If not, why? B. Social cohesion 15. How is the community governed? Follow up questions and prompts: – Who are the community leaders? – Are there also other traditional leaders? – How did the community get involved with the MDTF-N project? 16. What is the impact of the MDTF-N project on the community? Follow up questions and prompts: – Has the MDTF-N changed the way the community interacts with each other? – Was the community consulted on the design of the project? – Was the community actively involved with the implementation of the project? 79 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report – What benefits will be/have been contributed to the community by the project? – What benefits do you expect the community to get from the project in the long term? – Does everyone in the community benefit from the project? – If the project doesn’t benefit everyone, which groups do not benefit? – Do you think the project helped bring the community together? III. Concluding comments Express thanks for taking the time to participate in the focus group. Explain the expected outcome/purpose of findings. Ask if there are any further questions or additional information the participants would like to add. 80 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 6 Project Outputs and Results In order to provide a clearer picture of the total progress to date of the seven projects in the evaluation project sample, this Annex compiles the latest output nd and progress information available in the 2 Quarterly Report published by the Monitoring Agent in October 2012. It also summarizes the progress compared with the results indicators and highlights the areas where progress has exceeded the target, as well as those in which the target has yet to be reached. Basic Education Program (BEP) Project development objective level results indicators Baseline Progress Summary of Progress (July 2009) (as of 30 June 2012) Indicator 1: Number of students enrolled in primary education in Total: 8,599 Total: 8,711 ■ Total increase of 112 students (1%); supported schools Male: 4,318 Male: 4,206 317 more students needed to be Female: 4,281 Female: 4,505 enrolled to reach the target of 9,028. (Target: 9,028 or a 5% increase) ■ Decrease of 112 male students. ■ Increase of 224 female students Indicator 2: Average number of days functional for the supported A Schools: 175 days A Schools: 175 days Unable to calculate due to lack of annual 64 schools B Schools: 123.7 days B Schools: 123.7 days data, which will be published March 2013. (Target:160.8 –a 30% increase of school days in Category B Schools) Indicator 3: Improvement of teacher’s knowledge of core Pre-training Post-training ■ Exceeded the target of 20% increase in subjects teachers’ subject knowledge in Core Subject knowledge Subject knowledge Subjects (28%) and Arabic (33%). (Target: 20% increase in teachers’ subject knowledge) Core: 25.52 Core: 32.70 ■ Teachers’ subject knowledge Math: 26.13 Math: 31 (19%) increased in both Math (19%) and English: 21.57 English: 23.42 (9%) English (9%), but failed to reach the Arabic 39.08 Arabic: 51.9 (33%) 20% target. Teaching skills Teaching skills ■ Similarly, the target of a 20% increase Core: 42.21 Core: 53.7 in teaching skills was exceeded in Core Math: 44.84 Math: 49.93 (11%) Subjects (30%) and Arabic (34%). English: 45.43 English: 49.02 (8%) ■ Teaching skills increased in Math Arabic 27.86 Arabic 37.32 (11%) and English (8%), but the 20% increase target was not met. Indicator 4: Number of direct project beneficiaries, with specific Total students: 8,599 Total students; 8,711 ■ 317 more students needed to be Male students: 4,318 Male students: 4,206 enrolled to reach the target of 9,028. 64 The project tracks the number of days the school is open on a monthly basis, but annual data will only be available by March 2013. Currently, the monthly data shows positive signs of an increase in the number of functioning days compared to the baseline data, but this will only be confirmed when the annual data is released. 81 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report gender targets Female students: 4,281 Female students: 4,505 ■ The target of benefitting 11,380 teachers has been met, with the (Target: 9,029 students/4,495 female students; 11,380 Teachers: 0 Teachers: 11,380 gender target of 6,828 female teachers teachers/6,828 female teachers). Male teachers: 0 Male teachers: 3,555 exceeded by 997. Female teachers: 0 Female teachers: 7,825 Summary 1. BEP has made positive strides in increasing enrolment for girls, improving the learning environment, and improving teachers’ knowledge and skills while facing challenges due to conflict and the sector’s lack of capacity. 2. There are issues related to project administrative and lack of engineering oversight that have delayed implementation and resulted in lower-quality output in building construction. 65 Assessment of Impact Project Objective: 3.8 Project Results: 3.5 Blue Nile State Start-Up Emergency Project (BNSSEP) Baseline Progress Summary of Progress Project development objective level results indicators (May 2008) (As of 30 June 2011) Indicator 1: People in rural areas provided with access to Total: 0 Total: 171,000 ■ The target of benefitting 171,000 improved water sources General population: 0 General population: 71,000 people, including 71,000 of the general Pupils: 0 Pupils: 100,000 population and 100,000 pupils has (Target: 171,000; general population: 71,000; pupils: 100,000) been met. Indicator 2: Farmers trained and adopting new technologies 0 200 ■ The target of training 200 farmers in new technologies has been met. (Target:200) Indicator 3: 14 high priority or mined areas and roads cleared or 0 54,000 ■ The target of 54,000 people with verified in about 33 communities, affecting a population of more access to roads and other areas has than 54,000 been met. (Target: 54,000 people with safe access to roads and other areas) Indicator 4: Capacity building in finance and economy, local 0 2,058 ■ The target of building the capacity of government, education and public planning for 1,720 government 1,720 government employees has employees been exceeded by 338. (Target: 1,720 ) 65 Respondents to this evaluation were asked to assign a rating of the project’s impact. The scale was 1-5, with 1 being extremely negative impact and 5 being extremely positive impact. 82 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Indicator 5: Improved disease control through increased 0 350,000 • The target of 72,000 samples taken by investigative samples undertaken by the disease control center the disease control center has been exceeded by 278,000. (Target: 72,000 samples) Indicator 6: Number of direct project beneficiaries, with specific Total beneficiaries: 0 Total beneficiaries: 942,270 • The target of a total of 774,000 project gender targets Percentage female Percentage female beneficiaries was exceeded by beneficiaries: 0% beneficiaries: 52% 168,270. (Target:774,000, with 53% female beneficiaries) • 52% of the total project beneficiaries were female, 1% lower than the target of 53%. Summary 1. There were challenges due to the capacity of implementing partners and the state government, as well high-turnover led to a challenging start; however, the project managed to meet nearly all of its objectives and exceeded its targets in quite a few of the key components. 2. There are lingering issues about the long-term sustainability of some of the outputs of the fisheries components. Assessment of Impact 4.1 Capacity Building of the Sudan Judiciary Project component Planned progress for year ending Dec Progress Summary of Progress 2009 (as of 30 June 2009) Component 1: Support to the National Judiciary Service Activity 1: Study tour for NJSC 6 NJSC members were sent ■ Activity completed. Commission members to examine and assess on a 5-day study tour to the role of a judicial service Malaysia. commission in a modern legal system. Activity 2: Group discussions with The project closed before ■ Activity not completed. selected international and local this activity was experts with the NJSC on the implemented NJSC’s role. Activity 3: Provision of video Procurement process ■ Activity completed. conferencing facilities for NSJC completed in June 2009, offices and 8 corresponding with the hand-over of locations. equipment in July 2009. Component 2: Judicial training at the central and state Activity 4: Equipping of the Training Procurement process ■ Activity completed. levels Department including office facilities, completed in June 2009, computer training facilities, training with the hand-over of room, English training facilities, equipment in July 2009. 83 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report audio visual equipment, furniture, and video conference facilities. Activity 5: Provide support to the Training for three groups of ■ Activity completed. training of state-level judges 20 judges – half of whom were state-level judges – was conducted between February and June 2009 Activity 6: Study tour for the head The head of the training ■ Activity completed. of the training department and the department and two NJSC judge in charge of training to judges were sent to Canada comparative jurisdictions to obtain for an 8-day training visit knowledge on the management of judicial training departments Activity 7: Training in international Training for three groups of ■ Activity completed. law and women and children’s law 10 judges – of which 5 were for 30 judges at the Harun M. women – was conducted in Hashim Law Center in Malaysia Malaysia between April and June 2009 Component 3: Establishment of a National Legal Activity 8: Support provided to the Provided 43,000 copies of ■ Activity completed. Training and Resource Center Technical Department, including the the Sudan Law Report development of court libraries, court (1900-2007) for the reports, the production of bench Technical Department, for books, administrative manuals, and distribution to 300 courts other training and reference and 900 judges materials for the judiciary and for court administration Component 4: Rehabilitation of court facilities in selected Activity 9: Rehabilitation of selected Selected state courts were ■ Activity completed. states courthouses and judges’ houses in rehabilitated. remote locations across four states. Summary 1. Administrative and staffing capacity issues with the implementing partner at the beginning of the project negatively impacted the rate of implementation; however, these problems seemed to have been resolved and the project was performing well by the time it closed. 2. Due to the lack of access to and availability of key project staff, a clear assessment of this project’s lessons learned could not be made; regardless, from the information obtained, it is clear that this project could provide valuable lessons, particularly around follow- on activities and scale-up. Assessment of Impact Construction: 4 Support to the NJSC: 2 84 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Training: 3 Community Development Fund (CDF) Project development objective level results indicators Baseline Progress Summary of Progress (2006) (as of 30 June 2012) Indicator 1: Number of girls and boys under the age of 15 Total: 80,800 Total: 151,619 ■ There was a 53% increase in school enrolled in school in the supported communities Male: 64,640 Male: 89,308 enrolment in supported communities Female: 16,160 Female: 62,311 from 2006 until June 2012. (Target: Boys – 101,672; Girls – 82,899; Total 184,571) ■ 32,952 more students under the age of 15 need to be enrolled in order to meet the target. ■ There was an increase of 24,668 male student enrolment since 2006; 12,364 additional male pupils needed to reach the target of 101,672. ■ There was an increase of 46,151 in female student enrolment since 2006; 20,588 additional female students needed to reach the target of 82,899. Indicator 2: The number of community members who have 212,200 783,406 ■ The target of 316,800 community access to a basic development package, including health, members with access to a basic nutrition, or population services development package was exceeded by 466,606. (Target:316,800) Indicator 3: The percentage of reported satisfaction with the 0% 75% ■ An increase of 15% in reported delivery of basic services satisfaction with service delivery is needed to reach the 90% target. (Target: 90%) Indicator 4: Number of direct project beneficiaries, with specific Total beneficiaries: 0 Total beneficiaries: ■ An increase of 15% in reported gender targets % Male: 0 2,835,912 satisfaction with service delivery is % Female: 0 % Male: 48 needed to reach the 90% target. (Target: 2,621,794, with 51% female beneficiaries) % Female: 52 Summary 1. The CDF’s community-led development design has had a positive impact on participating remote communities’ access to education, water, health, and social services. 2. Positive unintended consequences, such as the excellent reputation and legitimacy of the CDF among participating communities, are an important outcome that could be instrumental in future interventions or scale-up of this project. Assessment of Impact 4.5 85 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report National Emergency Transport Rehabilitation Project (NETREP) Project development objective level results indicators Baseline Progress Summary of Progress (as of 30 June 2012) Indicator 1: Share of rural population with access to an all- 0 Data not provided Unable to assess due to lack of data. season road (Target: not provided) Indicator 2: Number of rural people with access to an all-season 0 Data not provided Unable to assess due to lack of data. roads (Target: not provided) Indicator 3: Number of rural people with access to an all-season 0 Data not provided Unable to assess due to lack of data. rail (Target: not provided) Indicator 4: Roads in good condition, as a share of the total 0 Data not provided Unable to assess due to lack of data. classified network (Target: not provided) Indicator 4: Size of the total classified road network 0 Data not provided Unable to assess due to lack of data. (Target: not provided) Indicator 5: Travel time (in hours) on selected transportation ■ The travel time on the Babanousa – routes Wau Rail line is 20 hours, 5 hours short of the 15 hour target time. Babanousa – Wau Rail (Target: 15) N/A 20 ■ There has been no change in the travel Damazine - Kurmuk Road (Target: 1) 10 10 time on the Damazine – Kurmuk Road. Hamsahkoreib – Gadamai Road (Target: 1) 10 1 ■ The target travel time of 1 hour on the Kadugli – Kauda Road (Target: 2) 15 15 Hamsahkoreib – Gadamai Road has been met. ■ There has been no change in the travel time on the Kadugli – Kauda Road. Indicator 6: Number of direct project beneficiaries, with specific Total beneficiaries: 0 Total beneficiaries: 261,000 ■ 40,500 more people must benefit from gender targets Male: 0% Male: 52% the project in order to reach the target Female: 0% Female: 48% of 301,500 total beneficiaries. (Target: 301,500, 48% of which are female beneficiaries) ■ 48% of the beneficiaries are women, meeting the target. 86 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Summary 1. Renewed conflict has severely hampered the completion of project activities in the conflict-affected areas; however, for the components fully implemented in conflict-free areas, the results and impact on the targeted communities have been positive. 2. It is challenging to assess the exact impact of the project due to a lack of baseline data and unclear results indicators, as well as a consistent progress reporting system. Assessment of Impact Railways: 4.5 Roads: 3.5 Public Sector Reform , Decentralization and Capacity Building Support Project (PSCAP) Project development objective level results indicators Progress End project target Summary of Progress (as of 30 June 2012) (30 June 2012) Indicator 1: Development of a new human resource policy for the Yes Yes ■ The target has been met. Ministry of Labor Indicator 2: The number of targeted states that adopted the new 2 5 ■ The target has been exceeded by 3 human resource management database states. Indicator 3: Development of a new fiscal formula in accordance Yes Yes ■ The target has been met. to international best practice Indicator 4: The number of targeted public sector training 3 3 ■ The target has been met. institutions that have undergone structural and functional assessment Indicator 5: The number of FFAMC senior staff trained in Total trained staff: 33 Total trained staff: 31 ■ 31 staff were trained, missing the intergovernmental fiscal relations, with specific gender targets % male: 70% % male: 60% target of 33. % female: 30% % female: 40% ■ The gender target of 30% of staff trained was exceeded by 10%. Indicator 6: The number of direct project beneficiaries, with Direct beneficiaries: 2,800 Direct beneficiaries: 4,200 ■ The total direct beneficiaries target specific gender targets % male: 60% % male: 60% was exceeded by 1,400. % female: 40% % female: 40% ■ The gender target of 40% female beneficiaries was met. Summary 1. PSCAP suffered from a mismatch between its ambitious objective and small budget, as well as a short time frame in which to implement and see results or impact. The project was also challenged by turnover and remote management, as well as procurement-related issues for some of its activities. When it closed, the project had not met three of its seven targets and some of its project objectives were cancelled. 2. Despite these challenges, the project did deliver targeted training on key issues related to fiscal, procurement, and M&E management to key national and state staff; and laid the foundation for new policies and procedures in human resource management, fiscal decentralization, and procurement. 87 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Assessment of Impact 3.5 Revitalizing the Sudan Gum Arabic Production and Marketing (GAPM) Project development objective level results indicators Baseline Progress Summary of Progress (2010) (as of 30 June 2012) Indicator 1: Percentage change in the income level from Gum Yes Yes Exact change in the level of income not Arabic production of beneficiary households following the project provided—unable to assess. intervention (Target: 115%) Indicator 2: The number of direct project beneficiaries, with Direct beneficiaries: 0 Direct beneficiaries: 11,346 ■ The project must reach another 129 specific gender targets % male: 0% % male: 75% beneficiaries in order to reach the % female: 0% % female: 25% target of 11,475. (Target: 11,475, with 20% female beneficiaries) ■ The gender target of 20% female beneficiaries was exceeded by 5%. Summary 1. The project design and the studies conducted during the first phase of implementation provided a strong foundation on which to build the subsequent project activities supporting the small-scale producers through selected GAPAs. Despite project success in the policy sector component, oversight and monitoring at the GAPA level could be strengthened to ensure that maximum benefit is reaching the producers. 2. Sector reforms, such as the removal of the gum Arabic monopoly and the reduction of fees and taxes paid by producers was a catalyst for the increase of income and improvement of conditions for small-scale producers. This was an important benefit that impacted the entire sector – not only the producers and GAPAs targeted by the project. 3. Because of the small-scale of the GAPAs component, this project should be considered a pilot project which could be further supported for possible scale-up. Assessment of Impact 4 88 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report Annex 7 Terms of Reference World-Bank Administered National Multi Donor Trust Fund (MDTF-N) in Sudan Independent Evaluation Terms of Reference for an International Consultancy Firm I. Background 1. The Sudan National Multi Donor Trust Fund (MDTF-N) was set up in 2005 in the context of the Sudan’s Comprehensive Peace Agreement (CPA) and was informed by the recommendations of a Joint Government/UN/World Bank Assessment Mission (JAM). In line with the JAM’s objectives of consolidating peace and supporting recovery and development in war-affected areas, notably the Three Protocol Areas (South Kordofan, Blue Nile, and Abyei), the MDTF-N formulated a portfolio of 15 projects covering 12 out of the 15 Northern States. 2. The MDTF-N strategic approach follows six guiding principles: (i) high visibility impact of development activities at local and state levels ; (ii) improved income generating opportunities for smallholder farmers and pastoralists; (iii) national programs with impact on peace consolidation such as Emergency National Infrastructure; (iv) improved efficiency in the delivery of basic services through capacity building support for federal, state and local- level institutions along with reform of public procurement and financial management; (v) enhanced rule of law by improving capacity of judiciary institutions; and (vi) improved donor coordination and aid management through simplification of the external flow of fund. 3. MDTF-N funding has been provided by ten donors: the Netherlands, Norway, United Kingdom, Canada, Sweden, Spain, Greece, Iceland, Italy and the World Bank. At the Donors’ Conference on Sudan in Oslo, in April 2005, donors and the Government authorities requested the World Bank to administer the Trust Fund. Commitments amount to US$604 million, including US$252 million and US$352 million by the MDTF-N and the Government respectively. As of February 29, 2012, total of US$524 million (89% of total commitments) have been disbursed. Due to a slower than expected disbursement, the completion date of the MDTF-N has been extended from December 2011 to December 2012. A further extension to December 2013 has been requested due to the slow implementation of some projects in conflict areas. Most donors agreed to this latest extension while others are reaching a decision. II. Objective and Scope of the Evaluation 4. Over the period 2006-2012, the MDTF-N has been a key instrument for channeling development funding in Sudan including a sizeable Government counterpart contribution, while most development partners’ support to Sudan has been curtailed either by Sudan’s non-accrual status or political concerns. As the completion of the MDTF-N is nearing, donors and the Government have agreed to conduct an independent evaluation of the program focusing on lessons learned in terms of development impact, cost effectiveness, and institutional and organizational relevance. 5. The scope of the evaluation will cover: (i) alignment of priorities between the JAM, CPA, the Government’s five-year plan (2006-11), and the MDTF-N portfolio; 89 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report (ii) the architectural design of the MDTF-N, including the institutional set-up, management organization, and the use of the Monitoring Agent. (iii) the roles played by the Government and donor partners in contributing to program success; (iv) the results achieved by the MDTF-N; (v) cost effectiveness of MDTF-N; (vi) sustainability of activities post-MDTF; and (vii) lessons learned and perspectives for the post-MDTF phase, including options to consolidate and/or scale up successful projects after closure of the MDTF-N, and to initiate development programs that are aligned with new Government priorities as set forth in the I-PRSP. III. Organization of the Evaluation 6. Methodology/approach: The evaluation will use the following combination of methods: (i) A desk review of available documentation on country-level strategies and analytical work such as the JAM, CPA, the Government’s five-year plan (2006-11), the MDTF-N portfolio design and implementation, and independent assessment reports; (ii) Close consultations with: (a) the World Bank Task Team Leaders for the MDTF-N funded projects, (b) the Government officials, (c) project coordinators and other staff from projects’ implementing agencies, and state level authorities, where appropriate; (d) the Monitoring Agent; (e) donors, (f) beneficiaries; and (g) the World Bank staff working in the technical secretariat for the MDTF-N, including fiduciary and safeguard personnel; (iii) A field visit to a sample of project sites, to deepen understanding of key issues identified during the desk review, covering in particular the Three Protocol Areas. (iv) An assessment of all active projects to identify implementation issues and propose recommendations for actions to resolve them. 7. Duration of the evaluation and level of effort: the assignment which is expected to start by July, 2012 covers a period of 10 weeks including: (i) 2 weeks for review of available documentation and interviews of stakeholders, including the Government officials, donors, the monitoring agent, and the World Bank; and (ii) 4 weeks for field visit including discussions with beneficiaries and other stakeholders; (iii) 2 weeks for preparation of draft report; and (iv) 2 week for preparation of final report, reflecting comments received. The estimated level of effort is about five person months. Companies are requested to present, in their technical proposal, a level of effort chart showing their expected chronogram and time allocation for the development of each of the deliverables and any assumptions made regarding the level of effort for each of the deliverables and the scope of work to be performed. 8. Reporting: The consultant’s contract will be managed by the World Bank’s Task Team Leader (TTL). The draft report will be presented at an Oversight Committee meeting of the MDTF-N. The final report will be issued, after reflecting comments provided by the OC meeting. The TTL will inform the consultant of the date and time of the said OC meeting. 90 Independent Evaluation of the World Bank Administered Multi-Donor Trust Fund in Sudan Final Evaluation Report IV. Team Composition 9. It is expected that the consulting firm would ensure a team consisting of experts with experience in pooling mechanisms in post-conflict environments, and the evaluation of their performance. Experts with experience of evaluating Trust Funded projects in several sectors, including among others in Health, Education, Rural Development, and Infrastructure and with previous experience with the community development approaches, should be part of the team. Expertise in safeguards, particularly gender and environment, would be expected from the team. 10. Consultants should have a good understanding of aid procedures through previous work experience in the World Bank, European Commission, UN agencies and/or bilateral donor agencies. A familiarity with conflict and post-conflict environment is required. A familiarity with Sudan and its development challenges and relationships between Sudan and South Sudan would be a valuable asset. 11. The World Bank MDTF-N Technical Secretariat will provide documentation, including necessary background documents and reports. V. Deliverables 12. Deliverables with expected delivery times are listed below: • An inception report is expected to be prepared and submitted to the World Bank Task Team Leader 3 weeks following the project start. • A draft report is expected to be submitted in 2 weeks’ time following the completion of the field visits. • A final report, win an executive summary, covering main findings of the review and recommendations and reflecting comments provided by donors at the Oversight Committee meeting referred to in Paragraph 8, should be furnished to the Bank at the end of the assignment. The consultant is also expected to prepare and present a Power Point Presentation summarizing the objectives of the evaluation, the methodology and the main finding and recommendations at the OC meeting. VI. Reports and information to be consulted • All previous assessments/evaluations reports of the Sudan MDTFs. • World Bank (2005). Memorandum of the President of the IBRD to the Executive Directors on A Proposal For the World Bank to Administer Two Multi-donor Trust Funds for Sudan, Washington D.C., March 17, 2005. • Monitoring Agent quarterly reports. • World Bank Annual reports of the MDTF-N. • World Bank (2010). The Country Portfolio Performance Review: Sudan Multi- Donor Trust Fund –National, Khartoum, June 16, 2010. • World Bank (2011). The Sustainability Study for the Sudan Multi- Donor Trust Fund –National. • The MDTF-N Website can be reached at http://mdtfn.org/. • Other reports as shall be identified by the World Bank. 91