88366 World Bank Statistics for Results Facility – Catalytic Fund (SRF-CF) Evaluation Report of the Pilot Phase Brian T. Ngo Andrew J. Flatt February 2014 Management Response added May 2014 1 World Bank Statistics for Results Facility – Catalytic Fund (SRF-CF) Evaluation Report of the Pilot Phase Contents List of box and tables ...................................................................................................................... 4 Executive summary ............................................................................................................................. 5 List of acronyms and abbreviations .............................................................................................. 18 I. Introduction ............................................................................................................................... 21 i) Background and overview .................................................................................................... 21 ii) The pilot phase ...................................................................................................................... 22 iii) The evaluation: rationale, objectives, and intended outcomes.............................................. 23 II. Methodology and conduct of the evaluation ............................................................................. 25 i) Scope of the evaluation ......................................................................................................... 25 ii) Methodology and design of the evaluation ........................................................................... 25 iii) Conduct of the evaluation ..................................................................................................... 28 iv) Limitations to the evaluation ................................................................................................ 30 III. Main findings ........................................................................................................................ 32 i) Promoting a system-wide approach in statistics at the country level .................................... 32 ii) Increasing resources for implementing national statistical plans .......................................... 36 iii) Linking statistical systems to the needs of national monitoring frameworks ....................... 40 iv) Promoting dialogue between statistics users and statistical producers ................................ 43 v) Delivering more effective aid through coordination of development partners ..................... 47 vi) Impact of the in-country donor statistician ........................................................................... 49 vii) Broad issues related to the SRF Charter (realism of the expectations for the SRF, choice of countries, regional project considerations) ................................................................................... 52 viii) SRF governance arrangements and operational practices and procedures .......................... 58 ix) General questions on the impacts of the SRF-CF on national statistical systems ................ 66 x) Looking to the future ........................................................................................................... 74 IV. Conclusions and recommendations ....................................................................................... 76 i) Main conclusion and context ................................................................................................ 76 ii) Previous reviews of the SRF ................................................................................................. 77 iii) Limitations of the evaluation: A cautious interpretation is needed ....................................... 78 iv) Detailed conclusions and recommendations by SRF main objectives ................................... 78 v) Other conclusions and recommendations based on the survey ............................................... 82 vi) Insights and recommendations on three specific issues concerning the future of the SRF .... 86 ANNEX 1: Survey questionnaire for SRF-CF stakeholders ........................................................ 94 2 ANNEX 2: Tabulations from the survey of SRF-CF stakeholders ............................................ 103 ANNEX 3: Detailed reviews of Nigeria and Rwanda experiences with the SRF-CF ................ 144 ANNEX 4: The Results Framework and Monitoring of Progress ............................................. 157 ANNEX 5: Report on missions undertaken by the Evaluation Team ........................................ 178 ANNEX 6: List of persons met and contacted ........................................................................... 182 ANNEX 7: References ............................................................................................................... 186 ANNEX 8: Terms of Reference for the Evaluation of the SRF-CF Pilot Phase ........................ 192 Overview of the SRF-CF ............................................................................................................ 192 ANNEX 9: Management’s response to recommendations ........................................................ 198 3 List of box and tables Page Box 1 Status of Pilot Projects, as of October 2013 22 Table 1 Detailed Recommendations - A Summary 13 Table 2 Proposals on specific issues assuming the extension and expansion of the SRF approach - A summary 17 Table 3 Questionnaire response rates by country 28 Table 4 Questionnaire response rates by stakeholder group 29 Table 5 Success of the SRF-CF in promoting a system-wide 32 approach Table 6 Success of the SRF-CF in increasing resources for 36 implementing national statistical development strategies Table 7 Success of the SRF in linking improvements in the 40 statistical system to the needs of monitoring frameworks Table 8 Success of the SRF in promoting an improved dialogue 44 between users and producers of statistics Table 9 Success of the SRF in delivering more effective aid for 47 strengthening statistical systems Table 10 Expectations for the SRF 52 Table 11 Possible extension of the SRF to regional operations 54 Table 12 SRF grant approval and disbursement process 57 Table 13 Recipient-country execution of SRF projects 61 Table 14 Attention to SRF projects by World Bank line management 62 Table 15 The SRF concept and statistical capacity building 66 Table 16 Leadership of the statistical development process 69 Table 17 SRF project implementation and governmental reform measures 71 Table 18 SRF facilitation of the use of data to support evidence- based decision-making 72 Table 19 A two-step approach to country selection for SRF funding 87 Acknowledgements The Evaluation Team of Brian Ngo and Andrew Flatt would like to express its gratitude to all the persons met and contacted for their generous provision of time and assistance during this assignment. Special thanks go to World Bank staff in Washington, Nigeria and Rwanda, who assisted the Team in the planning and conduct of its missions and in the efficient supply of documentation and information. The advice and input of PARIS21 and DfID staff is also gratefully acknowledged. The Team benefited greatly from the interviews held with Task Team Leaders and in-country donor statisticians. It is also very grateful to all those stakeholders in the participating countries who responded to the survey questionnaire and provided many valuable insights into the SRF. The opinions expressed in this report are nevertheless those of the authors and do not necessarily reflect the views of the World Bank. 4 Executive summary Introduction 1. The World Bank has commissioned an independent evaluation of the Pilot Phase of the Statistics for Results Facility Catalytic Fund (SRF-CF) to assess experience, identify achievements to date, and recommend any changes in policies and practices that may be necessary in order to improve results. The SRF, which began its pilot phase in 2009 with support from UK/DfID and the Netherlands, is designed to support the capacity building of developing countries to improve the production, availability, and use of official statistics as a basis for evidence-based policy formulation and decision making. 2. The SRF was established on the broad principles of the Managing for Development Results process (MfDR) and of the Paris Declaration on Aid Effectiveness, and aims to promote a system-wide approach to strengthening national statistical systems. The SRF principles stress country ownership and leadership, dialogue across the national statistical system, and the alignment of donor support with country priorities. 3. The pilot phase of the SRF-CF was set up to provide financial and technical support to a limited number of countries that either had weak statistical capacity or had recently emerged from conflict. Five countries were initially invited by the World Bank to submit applications for assistance, with three more joining following a decision to expand the pilot phase. 4. The evaluation aims to assess the success of the pilot phase in achieving the SRF’s specific objectives of: i) promoting a system-wide approach in statistics; ii) increasing resources for implementing national statistical plans; iii) linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks; iv) promoting an improved national dialogue and partnership between statistics users and statistical producers; and v) delivering more efficient and effective assistance to statistics through better coordination and alignment to agreed national statistical plans. The evaluation also assesses the impact of the in-country donor statisticians in helping countries reach the SRF objectives, and the Facility’s governance arrangements and operational practices and procedures. Methodology and conduct of the evaluation 5. The evaluation commenced in mid-May 2013 and a member of the Evaluation Team undertook an inception mission to World Bank Headquarters towards the end of May. Telephone interviews were held with Task Team Leaders and in-country statisticians from late May to the end of July. The Evaluation Team conducted a field mission to Nigeria and Rwanda, coupled with a visit to the offices of PARIS21 and the OECD in Paris, in June 2013, and one member of the Team also visited DfID Headquarters in East Kilbride, United Kingdom, in July 2013. 6. The rather limited implementation experience -- in Afghanistan, Nigeria and Rwanda, and latterly in Lao PDR -- posed significant challenges in drawing conclusions from the assessment, which therefore focuses more on processes and outputs than on final outcomes related to evidence-based policy-making. To partly remedy this lack of information, the Evaluation Team attempted to make extensive use of a questionnaire covering the questions and issues to be addressed in the evaluation report. 7. The questions designed for the assessment covered the outputs and practices engendered by the SRF-CF approach, the inputs and contributions to the SRF objectives 5 made possible by that approach, and broad governance issues related to the SRF Charter and to arrangements and operational practices and procedures for the preparation and implementation of pilot projects. While mainly multiple-choice in design, the questionnaire also encouraged respondents to provide additional comments. Over 1000 comments were received, and contributed significantly to the understanding of the Evaluation Team of the issues involved. 8. The wording of the questionnaire attempted to cover the situation in countries where activities were well underway as well as those where implementation had yet to begin. The questionnaire was sent to 7 different groups of stakeholders involved in the SRF:  World Bank task team leaders (TTL);  Primary recipient agencies of the SRF grant, usually national statistical offices (NSO);  Other agencies within government involved in producing and/or using statistics (OGA);  Other national users of statistics such as the media, researchers, and academia (ONU);  In-country donor statisticians (IDS);  Local bilateral and multilateral development partners (LDP); and  National poverty reduction or MDG coordination units (PRC). 9. Of the seven categories of stakeholders, only the Task Team Leaders and in-country statisticians were readily identifiable. For the other categories, the process of drawing up the lists of potential respondents was not completed until the middle of August 2013. Analysis of the completed questionnaires continued into September 2013 and report writing proceeded concurrently. Given the lack of any responses from the PRC group, it was decided to exclude them from the tabulations. Both the length and the composition of the respondent lists varied considerably among the countries. 10. Significant problems were encountered in soliciting responses from questionnaire recipients. After a drawn-out process of recalls and reminders, a total of 110 responses were received by the Team, representing an overall response rate of 36 per cent. For the purposes of analysis, Afghanistan, Nigeria and Rwanda were termed ‘implementing countries’, and the remainder – DR Congo, Ethiopia, Ghana, Lao PDR and Senegal -- ‘pipeline countries’. 11. The Evaluation Team submitted its draft report by end-October 2013; comments and additional material requested by DECDG were incorporated in the final report in February 2014. In reaching its findings and preparing its conclusions and recommendations, the Evaluation Team has drawn not only on the quantitative analysis of the questionnaire responses but also on the numerous comments supplied by respondents, as well as the in- depth discussions with the Task Team Leaders and the in-country statisticians, the field visits to Nigeria and Rwanda, and the desk study of documentation. To the extent possible, the Team has attempted to verify the findings and recommendations of this assessment with key stakeholders. 12. Nevertheless, it was clear from the outset that drawing conclusions from the experience of a relatively small group of countries, a majority of which had not commenced implementation of their SRF projects, would present a significant challenge. In view of this and in the light of the response to the questionnaire, care needs to be exercised in interpreting the findings of the evaluation. In particular, the Evaluation Team draws attention to the widely differing response rates among the various stakeholder groups, and to the likelihood that different groups may have different ‘stakes’ in the outcome of the assessment. Comparisons between countries also need to be undertaken with care, because of the different mix of respondents from each country. 6 Main findings 13. Promoting a system-wide approach to strengthening national statistical systems. Over 70 per cent of respondents in the countries which have started project implementation believe that the SRF-CF is at least somewhat successful in promoting a system-wide approach to statistics. In the pipeline countries, where many respondents understandably noted that it was premature to draw conclusions, some nevertheless saw reasonable promise in the SRF’s potential in this regard. Contributory factors included support for the operationalization of national statistical development strategies, the general alignment of donors behind those strategies, and the promotion of dialogues between the NSO and development partners and among government institutions. 14. Three sub-questions were asked to obtain a more in-depth assessment of the role of the SRF approach in promoting a system-wide approach to statistical development in the pilot countries. They covered the degree of inclusiveness of the NSDS, the perception of support by local development partners of the system-wide approach, and challenges to the system-wide approach. Eighty-five per cent of all respondents, led by TTLs and NSOs, felt that the NSDS process was at least somewhat inclusive. 15. Eighty per cent of local development partners supported the development of a system- wide approach at least to some extent, with those in the implementing countries generally more positive than those from the pipeline countries. They did however identify two groups of problems in this regard, the first concerning limitations on the part of NSOs, difficulties of coordination among government agencies, and a lack of commitment from some government partners. A second set of difficulties related to development partners themselves and included a lack of statistical expertise, frequent turnover among staff, and weak harmonization and coordination among some development partners. 16. Despite some positive findings about the success of the SRF in promoting a system- wide approach, only 15 per cent of respondents overall indicated that such an approach would not pose any significant challenge in their countries. Reasons cited included the lack of a data culture and limited capacity of the national statistical system, while a shortfall of political will and commitment in some countries has contributed to weak collaboration among government agencies and to the failure of government to take a leading and sustained role in mobilizing funding for statistics. 17. Increasing resources for implementing national statistical plans. While acknowledging that levels of funding were difficult to measure exactly, 72 per cent of respondents in implementing countries stated that the SRF-CF had been at least somewhat successful in its objective of attracting supplementary resources for the implementation of country-owned statistical development strategies. Positive results from SRF projects, along with the secure nature of SRF funding, were instrumental in building confidence in both influential government agencies and among other development partners, opening up the actual or potential flow of resources. 18. However, attempts to obtain additional details on the effect of the SRF project on domestic and external resources revealed very wide variations among countries and a significant lack of knowledge among respondents about the resource situation. What emerges clearly from the responses is that only 10 per cent of respondents attributed increases in domestic resources directly to the SRF, with a similar percentage indicating that the increase would have occurred anyway. 7 19. As to external resources, there was a clear divergence of views between non-LDP respondents and the donors themselves about the catalytic effect of the SRF, with only 7 per cent of local development partners attributing increases in their resources to the Facility. Despite a degree of uncertainty, it appears that the direct effect of the SRF project on increases in both domestic and external resources for statistics in the pilot countries has so far been quite modest. There would also appear to be scope for the freer exchange of information so that a much clearer idea can be gained of the future availability of funding for the implementation of national statistical development strategies. 20. More than half of all respondents thought that SRF funds should be seen as possible core finance for implementing national statistical development strategies, and the figure rose to 60 per cent in the implementing countries. In an environment of inadequate domestic resources and limited and changing contributions by donors, the SRF was seen as central and sustained core funding for broader, more fundamental issues, thus helping to institutionalize a system-wide approach to statistics through the NSDS. A sizeable minority of respondents, however, expressed the opinion that core financing for the NSDS should be the responsibility of the government for purposes of ownership and sustainability. In some countries, such as Nigeria and Ghana, it was argued that SRF funding constitutes a very small proportion of total requirements for statistical development and thus cannot have the necessary impact to be viewed as core finance. Notwithstanding this line of argument, it is worth noting that SRF resources are generally much larger than what has been provided for statistical capacity building from other sources, either bilateral or multilateral. 21. Linking statistical systems to the needs of national monitoring frameworks . The most frequent answer overall to the question of linking improvements in the capacity and outputs of the statistical system with the requirements of national monitoring frameworks was that it was too early to judge the performance of the SRF against this criterion. But in the implementing countries, some 60 per cent of respondents felt that the SRF was being at least somewhat successful in fostering this linkage, and the figure rose to 75 per cent in Nigeria. In some instances the stimulus provided by the SRF process to collaborative effort between the statistical office and other stakeholders was seen as having a positive effect on national and sectoral monitoring and evaluation frameworks. 22. In the implementing countries, half of all respondents considered that the SRF had played a very significant role to date in enhancing the production, quality and timeliness of data, and that figure rose to nearly 80 per cent when those who felt that it had played some role were added in. There were however significant differences among the implementing countries, while in the pipeline countries the impact to date of SRF activities was understandably much smaller. 23. The effect of the SRF in enhancing the accessibility and usability of data was reported to be somewhat less than for their production, quality and timeliness; nevertheless 72 per cent of respondents in the implementing countries reported that the SRF had played at least some role to date. Enhancements reported included, in general terms, the encouragement of open data initiatives, the promotion of transparency, and awareness-raising about statistics. 24. Promoting an improved national dialogue between users and producers of statistics. In the implementing countries, a quarter of all respondents felt that the SRF was very successful in this regard, and a further 32 per cent rated it as somewhat successful. In the pipeline countries, over half the respondents either felt that it was too early to judge on this issue, or failed to answer the question. In so far as producers and users of statistics could be distinguished from among the stakeholders responding, it appeared that the two groups held quite similar views on the success of the SRF to date in promoting a dialogue between them. 8 25. Generally speaking, the responses to the questionnaire reflected limited knowledge of user satisfaction surveys, which are a requirement under the SRF and have so far been undertaken in four pilot countries. Of more concern is the fact that, in the countries where surveys had already been conducted, over three-quarters of respondents in what could be considered the main user groups (OGA and ONU), and over half of local development partners, either professed ignorance about the status of user satisfaction surveys in their respective countries or gave an inaccurate answer. Respondents who were aware of these surveys generally expressed satisfaction about their utility. 26. Various efforts under the SRF to enhance the capacity of data users were reported by many respondents in the implementing countries, and by a few in pipeline countries as preparatory activities got underway. However, over one-third of respondents were unsure of the situation in this regard, and again, the stakeholder groups most closely identified with users of statistics were the least well informed about the activities being undertaken for their purported benefit. The survey responses highlighted the need to maintain the momentum of user-producer dialogue, in order to enhance coordination and increase awareness among users. 27. Delivering more effective aid through coordination of development partners. The survey showed that in the implementing countries, 65 per cent of respondents felt that the SRF was at least somewhat successful in delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement. There were fairly substantial differences among the three implementing countries, with Nigeria the most positive in this regard. Respondents frequently made reference to the fact that the SRF project addressed the overall statistical system in each country, and provided a good platform for development partners to share information and experiences, become aware of what others were doing, and plan and channel their support to statistics. 28. Overall, 35 per cent of respondents in the implementing countries considered that the SRF had already had a positive impact on aligning support for statistical capacity building to the NSDSs. Fifty three per cent of respondents in implementing countries, and 81 per cent of those in pipeline countries, considered that the SRF had the potential to make a positive impact. Similar figures applied to the effect of the SRF on improved donor coordination. The number of development partners active in statistics in each country, and the ease or otherwise of gathering them together for meetings organized by the lead donor, were noted as practical factors affecting coordination of statistical capacity building. 29. The impact of the in-country donor statistician (IDS). At the time of this report, statisticians were working with the SRF projects in Afghanistan, DRC, Ghana and Nigeria. In all three countries where the statisticians have been working for some time, there was a general consensus that they have had a very significant positive impact on the successful implementation of, or preparation for, the respective projects and on helping to achieve the various objectives of the SRF. The SRF-Catalytic Fund guidelines on the contributions by the in-country statisticians are very broad and include guidance and advice to the statistical offices, coordination of donor activities, leadership of the development partner group, and advocacy of the importance of the SRF and its role in supporting the NSDS. In addition to the various objectives listed above, the statisticians also provide support for recipient country-executed projects. In the particular case of Nigeria, the IDS also plays the very important function of a ‘clearing house’ for all data used in World Bank projects, thus further complicating the statistician’s role. 30. A large majority of respondents indicated that they saw no actual or potential areas of friction or contention surrounding the role of the statisticians in relation to the statistical 9 office staff. There was widespread agreement that the fact that the in-country donor statistician represents a major donor had a positive impact on the incumbent’s effectiveness, especially where the statistician’s agency was also the lead donor in the national partnership group. 31. Emerging very clearly from the survey is a strong sentiment that because of a lack of capacity in statistics, the post of in-country donor statistician would be needed over a substantial period of time. Most respondents did see the need to have an exit strategy in place; it was feared that otherwise a vacuum would be created, and all the progress made under the project would be undone. 32. Broad issues related to the SRF Charter. Questions were asked about selected issues that relate to the Charter. The great majority of respondents considered the expectations for the SRF in terms of its impact and timing to be ‘very reasonable’ or ‘somewhat reasonable’, with however considerable variations among the implementing countries. Responses about the impact of the SRF covered the gamut of SRF objectives, but well over half the ‘most significant’ impacts related either to improvements in the statistical system in terms of data production, quality and usability, or to the promotion of a system-wide approach. 33. Some 38 per cent of respondents considered that the SRF should be extended to regional initiatives and operations, with another 37 per cent being undecided. South-south exchange of experiences among statistical offices and the sharing of best practices and international expertise were frequently mentioned. Many respondents felt that a regional focus in these initiatives was critical as countries learned from each other and were given the opportunity to share knowledge and demonstrate results. Despite the enthusiasm for regional projects, a minority of respondents were anxious that the SRF projects should first help to tackle more important national issues. 34. SRF governance arrangements and World Bank operational practices and procedures. The survey also sought information on how well the governance arrangements of the SRF-CF and the operational practices and procedures of the Facility were serving the SRF’s objectives and its various stakeholders. Some 30 per cent of all respondents answering the question stated that the SRF grant approval and disbursement processes had been beset by difficulties and delays, some of which had occurred for reasons internal to the pilot countries. Many respondents reported that the Bank’s administrative procedures had caused significant delays in SRF projects. In general, respondents pointed out that more awareness of and training in SRF principles, norms and procedures was needed on the part of both Bank staff and government officials in the SRF project team. 35. Nearly half of respondents in the implementing countries stated that their experience in implementing the SRF grant was very positive, and another 30 per cent had a somewhat positive impression. While negative experiences were in a small minority, they were nevertheless strongly expressed. Major problems faced in implementing the SRF projects included the inadequacy of counterpart government budgetary support, a lack of coordination among government agencies and among development partners, and a lack of capacity. 36. A mere 5 per cent of TTLs, NSOs and in-country statisticians reported their experience with using the SRF Results Framework to be very positive. A common complaint was that the generic framework was often not in line with, or was even at odds with, the country-level results frameworks. Criticisms made of the generic framework were that it contained too many indicators, was too heavy to maintain, required input from user surveys, and included indicators and targets that were too ambitious given the countries’ capacities to report. 10 37. General questions on the impacts of the SRF-CF on national statistical systems. Over half of all respondents, in both implementing and pipeline countries, felt that the concept of the SRF met statistical capacity building needs to a great extent When those who felt that the concept met such needs to some extent were added in, the overall figure among all respondents who answered the question rose to 90 per cent. There were however wide variations among countries; it appears from the Evaluation Team’s survey that the SRF concept may better meet the statistical capacity building needs of those countries which already have higher statistical capacity as measured by the SCI. Notwithstanding any differences between countries, however, the Evaluation Team felt that overall, the SRF was playing a special role in statistical capacity building in the pilot countries, due to the relative size of the grants and the concept and principles embodied in the SRF approach. 38. In the implementing countries, over half of respondents considered that the SRF had facilitated effective government ownership and leadership of the statistical development process to a great extent, and a further third thought that this was the case to some extent. In terms of potential, the response from the pipeline countries was similar. There were, however, wide variations among both groups of countries as well as among the various stakeholder groups. Some 65 per cent of respondents thought that, in the course of implementing the SRF projects, national authorities were, or would be, taking reform measures that would not otherwise have been embarked on. 39. On the key issue of the extent to which statistical data are used in policy processes, just over half of all respondents considered that the concept of the SRF has facilitated, or has the potential to facilitate, the greater use of statistical data and indicators to support evidence- based decision-making. As for other questions, the proportion of respondents taking this view varied greatly across countries, ranging from 80 per cent in Ghana and 72 per cent in Rwanda to 33 per cent in DRC and only 11 per cent in Afghanistan. Although differences in the mix of respondents from one country to another cannot be ignored, this range of figures again appears to reflect, to a substantial degree, the level of statistical capacity development in the respective countries. 40. In terms of forward-looking comments, respondents in all three implementing countries expressed the wish that SRF funding should be continued. In the pipeline countries where projects were under preparation, the SRF grants were eagerly awaited as a means of, among other things, financing key statistical activities and filling data gaps to monitor development plans. Respondents also felt that the SRF-CF should be expanded to other countries, drawing on lessons learned in the pilot phase. The notional resource allocation ceiling of $10 million was described as rather arbitrary, and respondents advocated more flexibility for larger countries and for those countries that were performing well. Conclusions and recommendations 41. Based on the generally positive feedback received, the Evaluation Team concludes that the SRF appears to be reasonably successful in meeting its objectives. It is, however, important to note the limitations of this evaluation, which draws conclusions from the experience about a program which is basically operational in only three countries. In view of this, care needs to be taken when considering this report in general and the conclusions and recommendations presented in this section. Most importantly, there is a clear general impression that for most of the success criteria utilized in the survey, the stakeholder groups which could be said to have a larger stake or vested interest in the SRF (TTL, NSO and IDS) were more positive than the remaining groups which largely comprise users of statistics. While this general conclusion is not necessarily unexpected, the Evaluation Team feels that in 11 the future, more conscious attention needs to be paid in the course of SRF implementation to involving these groups. 42. There are real achievements that can be attributed to the SRF in all three implementing countries. In general, the five SRF objectives are being met, but with somewhat differing rates of success. The Team therefore recommends that the SRF approach should be extended and expanded. Details of the recommendations per SRF objectives together with the assessments to provide the context are shown in Table 1. 12 Table 1 Detailed recommendations - A summary SRF Objectives Assessment Recommendations Promotion of a system-  Positive findings but the  One particular aspect that may need wide approach to survey results suggest that further consideration is the role of the in- statistics implementation of such an country donor statistician on coordination. approach would pose a  The World Bank should lead by example moderate to significant by ensuring that it has staff in the country challenge. office suitably qualified in statistics or M&E to serve on the National Partnership Group and should encourage other development partners to do likewise. Resource mobilization  The direct effect of the SRF  Promote a more systematic exchange of on increases in resources, both information on funding intentions within from domestic and external national partnership groups. sources, has so far been quite  Given the role of funding in ensuring modest. sustainability of the support for statistics generated by the SRF, it is desirable for all concerned parties to increase efforts to generate additional funding and to actively explore additional resource mobilization methods, for example through matched funding arrangements.  Promote the practice of producing regular NSDS implementation reports to help in monitoring the commitment of resources and their delivery. The TORs of the in- country donor statisticians should specifically mention the provision of support to national authorities in the preparation of NSDS implementation reports.  Aided by a conducive environment, the basket fund in Rwanda was proving effective for NSDS implementation, and the SRF should be used to encourage the use of basket funds elsewhere, taking country circumstances into consideration. Linking Statistical  In implementing countries,  Highlight the importance of the use of Systems to National the SRF has played a statistics in future SRF projects. Monitoring Frameworks significant role in enhancing  More research aimed at improving the the production, quality and quality and efficacy of user satisfaction timeliness of data. surveys. In the interim fully support the  Its impact on enhancing the proposal by the SRF Administration Unit to accessibility and usability of lighten the surveys and to be more selective data is somewhat less but still on key data users. significant.  Opportunities should be taken to critically . assess initiatives to measure the use of statistics in the policy process. Promoting Dialogue  A modest majority of  Implementing agencies need to mainstream Between Statistics Users respondents in implementing user satisfaction surveys in their work and Statistical Producers countries consider the SRF to programs, to inform other stakeholders and to be at least somewhat publicize the results of these surveys. successful in promoting this  Pay more attention to user-oriented dialogue. awareness raising activities through greater  But users of statistics publicity and outreach. generally professed ignorance  Encourage sharing among countries of 13 about the status of user various practices such as workshops, training satisfaction surveys in their of journalists and mobile education for user countries and were not well outreach. informed about capacity building activities designed for their benefit. Delivering More  A small majority of  The importance of sustaining these Effective Aid respondents felt that the SRF relationships needs to be borne in mind, was successful in delivering particularly given the frequent staff rotation. more efficient and effective  The SRF Administration Unit should be aid and technical assistance. kept up-to-date on staff changes by the  Progress in this area respective World Bank offices so as to brief depends to a great extent on new TTLs on the special features of the SRF personal relationships between approach. members of the local development partners. Other survey-based issues Assessment Recommendations In-country donor  The statisticians have been  Emphasize the coordinating role of the statisticians an almost unqualified success statisticians, both between producers and in facilitating the system-wide users of statistics and among development approach and helping partners. coordinate donor activities.  Fund the post of in-country donor  The fact that the statistician statistician for a substantial period of time. represented a major donor was  Consider more than one statistician in seen as having a positive larger SRF projects subject to funding impact on their effectiveness. availability.  Consider developing an exit strategy for the IDS in future SRF projects.  Explore different implementation models for the IDS. Issues related to the SRF  The SRF approach is  While retaining the country focus of the Charter certainly broad and flexible SRF approach, the SRF Council should enough to have a significant explore the possibility of extending the SRF impact in each of the diverse to regional initiatives including introducing a situations represented by the regional dimension to country projects. three implementing countries.  The Council should promote the sharing of  A small majority of experiences across countries. respondents considered that  Take steps for the Charter to be publicized, the expectations of the SRF in so that countries and institutions become terms of its impact and timing more knowledgeable about the SRF initiative were realistic. and the approach and principles underlying it.  A rich vein of suggestions This action could be taken in conjunction for regional activities was put with more visibility of the SRF on the World forward, featuring south-south Bank internal and external websites. exchange of experiences and the sharing of best practices and expertise. SRF governance  The majority of respondents  The actions already identified by the SRF arrangements and in the implementing countries Administration Unit to improve SRF operational practices and rated their experience in governance and operational issues, including procedures implementing the grant, in efforts to reduce administrative barriers for terms of reporting project implementation, should continue to be arrangements and support pursued. from Bank staff, as either very  Extend the support to implementing or somewhat positive. agencies in procurement, financial  Problems faced included management and overall project management inadequacy of counterpart to World Bank staff where necessary. 14 government budgetary  Endeavor to ensure greater continuity in support, a lack of post of TTLs and other staff assigned to SRF coordination, low levels of project implementation. capacity, and overly  Promote and pursue proactive lesson bureaucratic processes on the learning and experience sharing among the part of both the Bank and pilot countries. national administrations.  Modify SRF guidelines to TTLs to stress the point that the coverage of broader SRF issues, as noted in the generic SRF Results Framework, needs to be taken into account during project supervision, so as to be able to respond to SRF requirements. Other issues  Respondents in all countries  The SRF Council should re-examine the expressed the wish that SRF current notional resource allocation ceiling funding should continue, and consider upward flexibility in specified citing achievements to date circumstances and tasks yet to be addressed Insights and proposals on specific issues assuming the extension and expansion of the SRF approach 43. Based on the main conclusion of this evaluation that the SRF approach should be extended and expanded, the Evaluation Team, upon the request of the SRF Administration Unit, has undertaken additional work to provide insights and proposals on three important issues which have significant bearing on the future of the SRF approach. These issues are a) country selection, b) the results framework, and c) ways to enable greater flexibility in project implementation and to enhance greater ownership by countries and the World Bank operations units. 44. By comparison to the ‘recommendations’ presented above which are based on a clearly defined approach to data collection, the ‘proposals’ suggested below are based on more limited underpinnings including the review of existing documents on the SRF-CF and discussions and insights shared among the members of the Evaluation Team. But assuming that the SRF approach will be extended and expanded, it is hoped that the proposals summarised in Table 2 would, at least in part, contribute to a more methodical approach to country selection, to a clearer approach to the design and monitoring of results frameworks, to greater flexibility in project implementation and to greater ownership of the SRF projects. 15 Table 2 Proposals on specific issues assuming the extension and expansion of the SRF approach - A Summary Issues Assessment Proposals Country  While country selection principles are  Fragile states versus other IDA selection clear, the SRF Charter is not sufficiently countries. Based on the current specific to help the actual decision classification by the World Bank which process and pilot country selection was includes 36 fragile states out of 82 IDA- based on a compromise between eligible countries, it is suggested that analytical and political considerations. future SRF funding may roughly be  The SRF approach, as highlighted in allocated 1/3 to fragile states and 2/3 to the annual reports, revealed a continuing other countries. focus on difficulty, variety and fragility  Open competition versus ‘by invitation’. for the selection of countries. The As the generally positive reactions to the positive feedback of the SRF from all 8 SRF approach become better known, pilot countries including post-conflict which would normally be expected to countries appears to confirm the original enhance the attractiveness of the SRF-CF belief that the SRF approach does work as support for statistical capacity building, in all country circumstances although it is the opinion of the Evaluation Team more care and support should be that open competition should be the provided to fragile states. primary option for SRF funds allocation  All SRF pilot countries were ‘invited’ going forward (while acknowledging that to apply for SRF grants, allowing for there may still exist justification for greater subjectivity compared to an specific countries to be ‘invited’ to apply). ‘open competition’ approach among all  Country selection criteria. The IDA countries where defined criteria and Evaluation Team believes that a ‘two -step prerequisites play an important role. procedure’ would allow for a more practical approach to country selection for future SRF funding. The first-order criteria would consist of prerequisites while the second-order criteria would help to differentiate countries in terms of their eligibility for SRF funding. This two-step approach would apply to both regular IDA-eligible countries and to fragile states, with some minor adjustments for the latter. The SRF  Translating the generic results  A more systemic consideration of the generic results framework into specific indicators which two results frameworks. A useful way to framework and are measurable and which can be help bridge the gap between the two country monitored over time has proven to be results frameworks is to view the SRF applicability much more challenging than originally generic results framework as an effort to expected with data availability being the monitor a medium- to long-term, system- most critical constraint. While the focused effort to sustainably develop the approach adopted by the SRF statistical capacity of countries, while Administration Unit to delete several specific project M&E mechanisms indicators including most of those monitor time-bound, data and action- related to financial resources for focused operations. Much more thinking statistics may be justified on the basis of is required to translate the above into data unavailability, without information operational practices. on resources, there is the risk that the  Greater efforts to obtain information on sustainability of support provided by the budgetary commitments and support. SRF projects may not be assured. Based on the field mission to Nigeria and  There has been limited success by the Rwanda, the Evaluation Team believes project teams in blending the indicators that information on resources for statistics of the generic results framework with is available but does require special country priorities based on individual efforts to monitor. The Team proposes 16 country NSDSs. the re-introduction of some measure of  The results frameworks of the pilot budgetary resource monitoring, while projects have a strong focus on the acknowledging that various challenges production of data and their accessibility. must be overcome including access to  World Bank’s supervision mission information on resources provided by reports show a relatively narrow focus development partners. on specific project development  Going beyond the production of and objectives and intermediate results accessibility to statistics and highlighting indicators and limited efforts by user-oriented awareness raising activities. supervision teams to attempt to cover the It would be useful for future SRF projects broader indicators of the SRF generic to have specific provisions for the active results framework. support of other government agencies outside the NSO as well as users of all sorts and local development partners. This would include revising the content of the new indicator of the generic results framework -- the number of government staff trained under the SRF -- which should be supplemented by ‘total number of persons trained’ to reflect training of users and emphasize ‘type of training’ to distinguish technical training from user- oriented and other types of training. Greater  The nomination of experienced and  On World Bank procedures. More flexibility in committed TTLs and local support staff awareness of and training in SRF project was seen by respondents to the principles, norms and procedures was implementation questionnaire survey as key to successful needed on the part of both Bank staff and and to enhance project implementation. government officials in the SRF project greater  The questionnaire survey showed that team. ownership by ‘ownership’ is most pronounced among  Greater ownership by countries and the countries and NSOs and is much less evident among World Bank. Discussions on country the World the other government agencies and the ownership, which is at the core of the SRF Bank users or potential users of statistics. approach, need to highlight the importance operations  The Evaluation Team’s impressions of the SRF being ‘owned’ by a broad range units indicate a wide range of interest and of stakeholders and not exclusively by the involvement by World Bank country government statistics office. The teams in the SRF projects. promotion of more systematic dialogue  The review of country experiences between producers and users of statistics shows that there may well be a case for would go a long way in promoting linkage of SRF grants with IDA credits ownership for local stakeholders. Having where the IDA funds are meant to the degree of World Bank support to address high-level, non-technical statistical capacity building in the country concerns, while more technical topics are as a prerequisite for country selection focused on in the SRF project. Properly could help in improving the situation. combined, the two could in principle be  Linkages with matching IDA credits. very effective. More in-depth assessment of the experiences of Ghana and Nigeria where efforts have been made to combine the SRF with IDA funds -- in parallel for Ghana and sequentially in the case of Nigeria. 17 List of acronyms and abbreviations AfDB African Development Bank Afristat L’Observatoire Economique et Statistique d’Afrique Subsaharienne ASEAN Association of Southeast Asian Nations AU African Union BBSC Bulletin Board on Statistical Capacity of the World Bank BOP Balance of Payments CIDA Canadian International Development Agency CPI Consumer price index CSO Central Statistics Organization (Afghanistan) DECDG Development Data Group (of Development Economics, World Bank) DfID Department for International Development (UK) DG Director General DHS Demographic and Health Surveys DP Development partner (Rwanda context) DRC Democratic Republic of Congo EC European Commission ECOWAS Economic Community of West African States ESSPIN Education Sector Support Program in Nigeria FAO Food and Agriculture Organization of the United Nations GDP Gross domestic product GoR Government of Rwanda ICT Internet and communication technology IDA International Development Association (of the World Bank group) IDS In-country donor statistician IES Income and Expenditure Survey ILO International Labor Organization IMF International Monetary Fund IMIS Integrated Management Information System IPAR Institute of Policy Analysis and Research of Rwanda ISR Implementation Status and Results (World Bank) KIST Kigali Institute of Science and Technology (Rwanda) L2A Listening to Africa project Lao PDR Lao People's Democratic Republic 18 LDP Local development partner LSMS Living Standards Measurement Survey MAPS Marrakesh Action Plan for Statistics MDAs Ministries, Departments and Agencies MDG Millennium Development Goal M&E Monitoring and evaluation MfDR Managing for Development Results MICS Multiple Indicators Cluster Survey (UNICEF) NBS National Bureau of Statistics (Nigeria) NCCS National Consultative Committee on Statistics (Nigeria) NGO Non-governmental organization NISR National Institute of Statistics of Rwanda NPopC National Population Commission (Nigeria) NSBS Niger State Bureau of Statistics NSDS National Strategy for the Development of Statistics NSO National statistical office NSS National statistical system OECD Organisation for Economic Co-operation and Development OGA Other government agency ONU Other national user OPM Oxford Policy Management PARIS21 Partnership in Statistics for Development in the 21st Century (OECD- based) PDO Project Development Objective PPI Producer price index PRC National poverty reduction or MDG coordination unit PRESS Partner Report on Support to Statistics (PARIS21) SBS State Bureau of Statistics (Nigeria) SCCS State Consultative Committee on Statistics (Nigeria) SCI Statistical Capacity Indicator SC/NPG Steering Committee/National Partnership Group (Rwanda) SDDS Special Data Dissemination Standard (of the IMF) SPARC State Partnership for Accountability, Responsiveness and Capability (Nigeria) SRF Statistics for Results Facility SRF-CF Statistics for Results Facility Catalytic Fund 19 SSA State Statistical Agency (Nigeria) TORs Terms of reference TTL Task team leader UEMOA Union économique et monétaire ouest Africain (Economic and Monetary Union of West Africa) UN United Nations UNAIDS Joint United Nations Programme on HIV/AIDS UNDP United Nations Development Programme UNESCO United Nations Educational, Scientific and Cultural Organization UNFPA United Nations Population Fund UNICEF United Nations Children's Fund UNIDO United Nations Industrial Development Organization US AID United States Agency for International Development USS User satisfaction survey WB World Bank WHO World Health Organization 20 I. Introduction i) Background and overview 1. The World Bank is conducting an independent evaluation of the Pilot Phase of the Statistics for Results Facility Catalytic Fund (SRF-CF). The main purpose of the evaluation is to assess the experience of the programs in the pilot countries, to identify achievements to date, and to recommend any changes in policies and practices that may be necessary in order to improve results. 2. The Statistics for Results Facility (SRF), which began its pilot phase in 2009 with support from UK/DfID and the Netherlands, is a multi-donor initiative designed to provide finance for investment in statistical capacity building in the poorest countries, act as a catalyst in mobilizing additional resources, and improve the effectiveness of financial and technical assistance. In terms of outcome, the objective of the SRF and of the Catalytic Fund (SRF-CF) is to support the capacity building of developing countries to improve the production, availability, and use of official statistics as a basis for evidence-based policy formulation and decision making. In more practical terms, the SRF was designed to support the preparation and effective implementation of national frameworks for the development of statistics such as the National Strategy for the Development of Statistics or NSDSs. 3. Compared to previous efforts to help build statistical capacity in developing countries which were designed more narrowly, the SRF was established on the broader principles of Managing for Development Results process (MfDR) and of the Paris Declaration on Aid Effectiveness which, together, aim to: (i) ensure effective ownership and leadership by countries in developing their national statistical systems; (ii) promote the alignment of donor support using national framework for statistical development (such as NSDSs); (iii) harmonize donor practices; (iv) support developing country efforts to generate better statistics and use them to support evidence-based decision making; and (v) promote mutual accountability by monitoring the implementation and outcomes of NSDSs while recognizing that building sustainable statistical systems is a long-term effort. 4. Drawing on lessons learned from sector-wide approaches used successfully in sectors such as health, education and agriculture, the SRF promotes a system-wide approach to strengthening national statistical systems. In particular it stresses the provision of support to a country-owned and -led program to develop a national statistical system using country systems and procedures for implementation. A system-wide approach also helps to promote a stronger focus on strategic issues and coordination and to support a dialogue across the national statistical system involving line ministries and sub-national entities where applicable. At the same time, a system-wide approach provides the framework for aligning donor support with country priorities and harmonizing donor support for more effective capacity building. 5. The experience gathered from pilot countries prior to this evaluation exercise suggests that the system-wide approach is likely to be more successful when the following conditions are in place:  A well-prepared NSDS that takes into account the needs of the most important data-producing agencies and main data users;  Strong leadership of the statistical system by the government and a general consensus on priorities both in terms of objectives and actions;  Agreement on a medium- to long-term approach to statistical capacity building; 21  Due considerations to institutional reforms, human development issues, and technical challenges;  A partnership for statistical support to improve donor coordination and the identification of a lead donor; and  Recognition of the need to support and strengthen the dialogue between data providers and users. 6. For the above elements to be in place, specific capacity building will be needed both in participating countries and within the development partner local agencies. For participating countries, this includes: the political will and national leadership to improve statistics; effective financial management capacity; and adequate institutional and professional expertise. In many countries technical assistance may be needed in some or all of these areas. In addition, central statistical agencies will need to have the capacity to lead the reform process while, at the same time, being able to maintain their regular work programs. For local donor agencies, a genuine effort for technical and financial support coordination with a lead donor has proven to be critical for success. ii) The pilot phase 7. The pilot phase of the SRF-CF was set up to provide financial and technical support to a limited number of countries that either had weak statistical capacity or had recently emerged from conflict. Experience drawn from these countries was expected to help to identify elements of the SRF approach that work well and the necessary modifications needed to improve the impact of the SRF approach. An evaluation of the pilot countries and recommendations for modifications to the approach of the SRF are expected to provide the basis for extending the role of the Catalytic Fund in providing support for statistical capacity building to other countries. In the meantime, non-pilot countries will be encouraged to make investment in their statistical systems in line with the SRF principles. 8. The SRF-CF Council is responsible for determining the criteria by which countries were identified for the pilot phase to receive grants from the SRF-CF or technical assistance. The criteria included primarily the level of statistical capacity and the extent of the country's commitment to the SRF principles. In line with the SRF Charter, the Council identified five countries that were invited by the Bank to submit applications for assistance during the pilot phase, namely Afghanistan, the Democratic Republic of Congo (DRC), Ghana, Nigeria, and Rwanda -- all IDA-eligible countries. These five countries represented various levels of statistical development, diverse statistical systems and different problems and concerns. Two of the countries – Afghanistan and DRC – were included on the basis of their status as fragile, post- conflict states which posed particular problems relating to the development of statistical capacity building. 9. Following a decision in November 2010 to expand the pilot phase, the SRF-CF Council agreed to invite three more countries -- Lao PDR, Senegal and Ethiopia -- to apply to the SRF pilot phase. In light of new approaches and technologies that had emerged since the SRF inception, the pilot phase was also extended to fund testing the use of innovative approaches to data collection. In January 2013, the Council furthermore approved funding for the 'Listening to Africa' project (L2A), a project aimed at piloting t he use of cell phones to gather high frequency household level data in three African countries, Cameroon, Malawi, and Senegal. Given the very early stage of preparation, this 2- year project was not covered in this evaluation. 10. As a general approach, the steps involved in the application for SRF support include: 22  The requirement that the country either has a statistical strategy for statistical development in place or under development, with high-level political endorsement;  A formal request to the SRF-CF from the government seeking support for the implementation of the NSDS together with a broad proposal on the use of or plan to adopt a system-wide approach;  The identification of budgetary resource requirements for statistics and multi-year national resource frameworks, such as medium-term expenditure frameworks where they exist;  The initiation of a dialogue with local donor representatives;  An agreement to the SRF principles by all or at least a core set of local donors who have expressed interest in providing support for statistical capacity building, and the identification of a lead donor to assist in coordinating the dialogue with the government; and  The identification of existing initiatives in statistics and a clear understanding of how these support the implementation of the NSDS. 11. Experience learned on the original five countries has led to some effort to simplify the application and approval procedures for SRF projects. As of October 2013, four projects in Afghanistan, Lao PDR, Nigeria and Rwanda are under implementation. The Lao PDR project has just begun implementation. The Ghanaian project still awaits parliamentary approval and the signing of legal documents by the Government to become effective. The DRC project is in the negotiation stage while the remaining three projects (Ethiopia, L2A, and Senegal) are at different stages of preparation (see Box 1). Box 1: Status of pilot projects, as of October 2013 Afghanistan ($14 m) Implementation Effective since March 2011 Negotiation expected in December DRC ($11.8 m) Project finalization 2013 Ethiopia (10 m) Project preparation Project appraisal in June 2013 Subject to Parliamentary approval Ghana ($10 m) Awaiting effectiveness Lao PDR ($8 m) Implementation Effective since July 2013 Concept note approved June 2013 L2A ($1.8 m) Project preparation Nigeria ($10 m) Implementation Effective since June 2011 Rwanda ($10 m) Implementation Effective since June 2012 Concept review in September 2013 Senegal ($2.2 m) Project preparation iii) The evaluation: rationale, objectives, and intended outcomes 12. According to the SRF Charter, once the projects in the pilot countries are well underway, progress towards achieving results using the SRF approach would be evaluated 23 and assessed to learn what worked well and what elements would require modifications to increase its impact. With the evidence from the assessment, the SRF could then be more confidently expanded to include additional countries. The main purpose of the evaluation would be to assess the experience of the pilot projects, to identify the achievements and challenges and to recommend changes in policies and practices in order to improve results. 13. The terms of reference (TORs) for the evaluation of the SRF-CF pilot phase were discussed and agreed with donors in the spring of 2013, with the aim of assessing: a) How successfully the SRF-CF has achieved its objectives particularly in: i) Promoting a system-wide approach in statistics at country level; ii) Increasing resources for implementing country owned National Statistical Plans; iii) Linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks and promoting an improved national dialogue and partnership between statistics users and statistical producers; and iv) Delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement, through better coordination and alignment to agreed national statistical plans (such as NSDSs) and through better alignment to the Paris Declaration on Aid Effectiveness; b) The impact of the in-country donor statisticians in helping countries reach the SRF objectives; c) How the governance arrangements and operational practices and procedures are working; and d) To recommend what changes may be needed to enable the SRF-CF to reach its objectives (as amended). 24 II. Methodology and conduct of the evaluation i) Scope of the evaluation 14. The period covered by this evaluation is relatively short -- since the establishment of the SRF pilot phase in 2009 -- and only four of the nine pilot projects are under full implementation, with the Lao PDR project becoming effective while the evaluation was underway. Thus the assessment, as described under the objectives detailed above, focuses more on processes and outputs than on final outcomes, which would have required an investigation into the broader practice of evidence-based policy-making based on improved statistical information. Based on discussions the evaluators have had with eight of the nine teams in charge of the preparation and implementation of the country-based pilot projects, a list of questions was identified to address specific demands for information related to the broad areas of inquiry agreed by the SRF donors. Drawing on the findings of the field visits to Nigeria and Rwanda as well as discussions with PARIS21 staff, these questions were refined and other questions added to help further explore lessons from the pilot projects. The thrust of the questions is discussed in more detail below. ii) Methodology and design of the evaluation 15. The questions designed for the assessment of how successful the SRF has been in achieving its intended objectives can be logically grouped into three categories. The first group of questions relates to outputs and practices engendered by the SRF-CF approach, namely: (i) the promotion of a system-wide approach in statistics at the country level; (ii) an increase, actual or potential, in the resources in the pilot countries for implementing their statistical development plans or strategies; (iii) a linkage of improvements in the statistical system to the needs of national and sectoral monitoring frameworks; and (iv) the promotion of an improved national dialogue and partnership between statistics users and producers. The second group of questions refers more specifically to inputs and contributions to the above objectives made possible by the SRF approach, namely: (i) the delivery of more efficient and effective aid and technical assistance for strengthening statistical systems; and (ii) the impact of the in-country donor statisticians. The third set of questions covers broad governance issues related to the SRF Charter and to arrangements and operational practices and procedures for the preparation and implementation of pilot projects. These three sets of questions are summarized in the following paragraphs. 16. Group I: Outputs and practices engendered by the SRF-CF approach a) The role played by the SRF in helping to promote a system-wide approach in statistics at the country level, which covers issues such as: i) the degree of inclusiveness in the formulation and implementation of the national strategy for the development of statistics (NSDS); ii) the degree of acceptance by local development partners of a system-wide approach which is led and owned by the government; and iii) the extent of the challenge that the system-wide approach imposes on countries, especially those with weak human and institutional capacity. b) The role played by the SRF in helping to increase resources for implementing country-owned national statistical plans and strategies, which covers issues such as: i) the impact on government budgetary resources for statistics and on contributions by donors; ii) the extent to which SRF funds should be used as core finance for NSDS implementation; 25 and iii) more specifically for the World Bank, the relationship between the SRF Catalytic Fund and IDA funding. c) The role played (currently or potentially) by the SRF in linking statistical improvements to the monitoring of the MDGs and other country-specific needs and priorities (including sectoral projects), which covers issues such as: i) i) evidence that the SRF has led to improvement in the production, quality and timeliness of data; and ii) instances where the SRF has helped to enhance the accessibility and usability of data (through, for example, 'open data' and web-based initiatives). d) The role played by the SRF in helping to promote an improved national dialogue and partnership between users and producers of statistics which covers issues such as: i) evidence that the SRF has led to efforts to enhance the capacity of data users (e.g., through the training of journalists); and ii) the experience with user satisfaction surveys, which are a requirement under the SRF approach. 17. Group II: Inputs and contributions to the above made possible by the SRF approach a) The role played by the SRF in helping to deliver more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement through better alignment to the Paris Declaration on Aid Effectiveness, which covers issues such as: i) the extent to which the SRF has had any impact in better aligning support for statistical capacity building to the NSDSs; and ii) evidence that the SRF approach has led to improved donor coordination at the local level. b) The impact of the in-country donor statisticians in helping to reach the SRF objectives. 18. Group III: SRF governance arrangements and operational practices adopted by the World Bank a) Broad issues related to the SRF Charter such as: i) the realism of the expectations placed on the SRF both in terms of impact and timing; ii) the practice of inviting selected countries, including post-conflict countries with very weak statistical capacity, to apply for SRF grants (rather than having countries compete for SRF funds) and how this may have impacted the SRF approach; iii) the setting of the usual amount of SRF funding at US$10 million irrespective of country size or stage of statistical development and iv) the possible extension of the SRF, hitherto comprising almost exclusively country-specific projects, to regional initiatives and operations; b) The impact of current World Bank operational practices and procedures on the implementation of the SRF, which covers issues such as: i) the ease or otherwise of the SRF grant approval and disbursement process; ii) the degree of attention paid by Bank line management to SRF projects; iii) the possible advantage of making SRF grants part of a larger Bank project; and iv) experience with the SRF generic Results Framework. 19. The above issues and questions provided a common framework for the evaluation, and were addressed by the Evaluation Team through a combination of methodologies. These included in-depth discussions with Task Team Leaders and in-country statisticians, field visits to Nigeria and Rwanda, and the administration of questionnaires. In addition, desk 26 studies were undertaken of various World Bank reports and working documents from pilot countries. A review based on World Bank reports was conducted to i) categorize the characteristics of the SRF approach by item across countries and ii) provide an assessment of progress over time from the project preparation period to the most recent information obtained from the questionnaires. A list of the major source documents consulted appears in Annex 7. 20. At the start of the evaluation in May 2013, SRF projects were actually under implementation in only three countries (see Box 1); this rather limited implementation experience posed significant challenges in drawing conclusions from the assessment. To partly remedy this lack of information, the Evaluation Team attempted to make extensive use of a questionnaire covering the questions and issues detailed above, along with some general questions relating to the principles underlying the SRF. For countries in which the SRF projects are not yet under implementation, the Team hoped that the questionnaire would provide useful information on the readiness of pilot countries to adopt the SRF-CF approach to statistical capacity building. 21. The fact that the questionnaire had to address a range of situations with respect to implementation status and other factors, such as the presence or absence of an in-country donor statistician, had an impact on its design. Thus questions were often couched in terms of potential as well as actual effects in the countries of the SRF approach, and respondents were frequently given the option of signaling their uncertainty about the impact of the SRF project, or of indicating that it was too early to judge the effect of SRF-related activities – an entirely reasonable position to take in countries where the SRF project had yet to commence. The master questionnaire designed is shown in Annex 1 (in its English version; it was translated into French for DR Congo and Senegal). Given the likelihood that some respondents would not be very familiar with the Statistics for Results Facility, the questionnaire as administered to respondents was accompanied by a short introduction to the SRF – this also appears in Annex 1. 22. The Evaluation Team recognized that there were several different groups of stakeholders involved in the SRF and categorized them as follows:  World Bank task team leaders (TTL);  Primary recipient agencies of the SRF grant, usually national statistical offices (NSO);  Other agencies within government involved in producing and/or using statistics, including line ministries, central banks, ministries of finance etc. (OGA);  Other national users of statistics such as the media, researchers, academia, chambers of commerce and other civil society organizations (ONU);  In-country donor statisticians (IDS);  Local bilateral and multilateral development partners with an interest in supporting statistics (LDP); and  National poverty reduction or MDG coordination units (PRC). 23. Slightly different versions of the master questionnaire were formulated for administration to the various stakeholder groups identified above. The purpose was not only to reduce the response burden but also to attempt to restrict questions to those which each stakeholder group might reasonably be expected to have knowledge of and an opinion on; for example, specific questions such as the relationship of the SFR grant to other World Bank funding were dropped for some groups of stakeholders. It was also hoped that the questionnaire would, besides providing feedback on the relevance and effectiveness of the 27 SRF, also help to enlighten the evaluators on the dynamics of relationships between various users of data and the national statistical offices. 24. The questionnaires as administered were also made country-specific, citing where possible the title of the national statistical development strategy and the name of the national partnership group or other institutional mechanism in the field of statistics, as well as the name of the in-country donor statistician where one was present. While most of the questions were framed as multiple choices, the questionnaire also encouraged the provision of written comments and elaborations on the various themes addressed. iii) Conduct of the evaluation 25. The evaluation commenced in mid-May 2013 and a member of the Evaluation Team undertook an inception mission to World Bank Headquarters towards the end of May; an inception report was subsequently submitted to DECDG. Telephone interviews were held with Task Team Leaders and in-country statisticians from late May to the end of July, as opportunities arose (those contacted are listed in Annex 6). The Evaluation Team conducted a field mission to Nigeria and Rwanda, coupled with a visit to offices of PARIS21 and the OECD in Paris, from 12 to 27 June 2013 (see Annexes 3 and 5). One member of the Team also visited DfID Headquarters in East Kilbride, United Kingdom, on 24 July 2013. 26. After finalizing the master questionnaire, the Evaluation Team sought the assistance of the Task Team Leaders and in-country donor statisticians in drawing up lists of potential respondents to the questionnaire in each country. It is important to note that of the seven categories of stakeholders (see paragraph 22 above) only the Task Team Leaders and in- country statisticians were readily identifiable1. For the other categories, the Team requested that persons and agencies involved in the implementation or preparation of the SRF project, or those active in the NSDS or national institutional mechanisms in the field of statistics, be identified along with their email addresses. TTLs or in-country statisticians were also requested to designate a team assistant in the World Bank office, not directly involved with the SRF project, who could follow up with non-respondents and deal with any questionnaires that were returned in hard copy form. 27. The process of drawing up the lists of potential respondents took some time and was not completed until the middle of August 2013. Consequently the dispatch of the questionnaires became a drawn-out exercise covering some six weeks. Given the considerable difference in the size of the countries and the status of implementation of the respective SRF projects, it was to be expected that the length of the respondent lists would vary: from 23 in DR Congo to 74 in Nigeria, with a median of 40. These differences need to be borne in mind when comparing response rates. In addition, the composition of the lists also varied considerably: in two countries, there were very few other government agencies (OGA) on the list, while four countries provided hardly any or no stakeholders in the category of other national users (ONU). No doubt this reflects to some extent the stage of development of the official statistical community in the countries concerned, and the degree of inclusiveness of the process of developing the respective national strategies for the development of statistics. 28. Significant problems were encountered in soliciting responses from questionnaire recipients. This was due to a number of factors including invalid email addresses, the 1 In fact, during the period of the evaluation, there was some turnover of TTLs of the SRF projects. 28 absence of many recipients on summer leave,2 rapid turnover of personnel with consequent loss of institutional memory (particularly among local development partners), a reported lack of knowledge about the SRF and, in some countries, objections to the concept of evaluating a project which had not yet commenced.3 29. After a number of reminders from the Evaluation Team and active follow-up by the respective World Bank team assistants and by DECDG, and in some cases the TTLs and in- country statisticians, a total of 110 responses were received by the Team, representing an overall response rate of 36 per cent. In a few instances, the Team noted that individual replies had been provided on behalf of multiple recipients in the same agency, thus implying a slightly wider coverage of respondents than the figures would otherwise indicate. For the purposes of analysis, Afghanistan, Nigeria and Rwanda were termed ‘implementing countries, and the remainder ‘pipeline countries’. The responses by country and by stakeholder group are shown below: Table 3 Questionnaire response rates by country Questionnaires Response rate Dispatched Received (per cent) Afghanistan 43 19 44 Nigeria 70 20 29 Rwanda 38 18 47 All implementing countries 151 57 38 DR Congo 22 6 27 Ethiopia 38 13 34 Ghana 32 10 31 Lao PDR 33 14 42 Senegal 28 10 36 All pipeline countries 153 53 35 All SRF countries 304 110 36 Note: excludes questionnaires dispatched to invalid email addresses Source: SRF Evaluation Survey 2013 2 Other evaluations, including those of the Trust Fund for Statistical Capacity Building, have observed that it was difficult to contact World Bank and other agency staff during July and August. 3 When the opportunity presented itself, the Evaluation Team stressed that the assessment was focusing on the approach to the SRF rather than on the progress of the individual country projects; they pointed out that several of the questions related to the prerequisites or conditions that are supposed to be in place for the SRF approach to be successful, and that these should be relevant even in countries just embarking on the SRF. Nonetheless, there is no doubt that this factor dissuaded some questionnaire recipients from responding. 29 Table 4 Questionnaire response rates by stakeholder group Questionnaires Response rate Dispatched Received (per cent) TTL Task Team Leaders 13 12 92 NSO Primary recipient agencies 69 41 59 OGA Other government agencies 83 19 23 ONU Other national users 23 5 22 IDS In-country donor statisticians 4 4 100 LDP Local development partners 109 29 27 PRC Poverty reduction or MDG units 3 0 0 All stakeholder groups 304 110 36 Note: excludes questionnaires dispatched to invalid email addresses Source: SRF Evaluation Survey 2013 As noted earlier, the questionnaire encouraged recipients to provide additional comments on the particular topics covered by the survey as well as on any other aspect of the SRF. Over 1000 comments were received, and contributed significantly to the understanding of the Evaluation Team of the issues involved. 30. Analysis of the completed questionnaires continued into September 2013 and report writing proceeded concurrently. Given the lack of any responses from the PRC group, it was decided to exclude them from the tabulations. Following completion of the draft report by end-October 2013, comments were received from DECDG requesting further input from the Evaluation Team on selected issues. This material was incorporated into the Team’s final report in February 2014. One member of the Evaluation Team was scheduled to present the final report on the evaluation at a side event to the United Nations Statistical Commission in early March 2014. iv) Limitations to the evaluation 31. In reaching its findings and preparing its conclusions and recommendations, the Evaluation Team has drawn not only on the quantitative analysis of the questionnaire responses but also on the numerous comments supplied by respondents, as well as the in- depth discussions with the Task Team Leaders and the in-country statisticians, the field visits to Nigeria and Rwanda, and the desk study of documentation. To the extent possible, the Team has attempted to verify the findings and recommendations of this assessment with key stakeholders, and in particular with World Bank TTLs, in-country donor statisticians, the 30 heads of the primary recipient agencies and PARIS21 colleagues who have extensive familiarity with NSDSs and resource mobilization for support to statistics. 32. Nevertheless, it was clear from the outset of the evaluation that drawing conclusions from the experience of a relatively small group of countries, a majority of which had not commenced implementation of their SRF projects, would present a significant challenge. In view of this and in the light of the response to the questionnaire, detailed in Tables 3 and 4 above, care needs to be exercised in interpreting the findings of the evaluation. In particular, the Evaluation Team draws attention to the widely differing response rates among the various stakeholder groups, and to the likelihood that different groups may have different ‘stakes’ in the outcome of the assessment. Comparisons between countries also need to be undertaken with care, because of the different mix of respondents from each country. These points need to be borne in mind when considering this report. 31 III. Main findings i) Promoting a system-wide approach in statistics at the country level 33. The SRF promotes a comprehensive system-wide approach to strengthening national statistical systems, which is defined broadly as the country having a clear nationally owned strategy for statistics with arrangements for resources, broad consultation mechanisms that involve all stakeholders, mechanisms for dialogue, coordination, and monitoring and evaluation 34. Over 70 per cent of respondents in the countries which have started SRF implementation believe that the SRF-CF is at least somewhat successful in promoting a system-wide approach to statistics in the respective countries (see Table 5 below and Table II.5 in Annex 2). That figure rises to around 80 per cent when respondents who feel optimistic in this regard are added in. The equivalent figure for countries where implementation has yet to fully commence is 50 per cent, with 43 per cent of respondents in the pipeline countries feeling that it is too early to judge. In the implementing countries, the national statistical office respondents were more sanguine than others about the SRF’s success in promoting the system-wide approach. Task Team Leaders and in-country statisticians were noticeably more cautious than other respondents in coming to a conclusion on this issue. 35. Among the implementing countries, contributory factors to the perceived success of the SRF project in promoting the system-wide approach included SRF support for the operationalization of clear prioritized statistical development strategies (in Nigeria primarily at the state level), the alignment of donors (with some exceptions) behind those strategies, and the promotion by the SRF of dialogues between the national statistical office and development partners and among government institutions. SRF support has also helped foster technical and other coordination mechanisms and coordinated development initiatives, although it was noted that more needs to be done. 36. SRF projects have also helped strengthen a system-wide approach through capacity building of the statistical systems at national (and in Nigeria, state) level through the implementation of various activities including training and the availability and practical use of information technology. That support has to date been primarily through the national statistical office, although in Nigeria it has been successfully used to bring the National Population Commission and the National Bureau of Statistics into closer collaboration (see Annex 3). 37. In Rwanda some respondents noted that the late arrival of the SRF monies into the existing basket fund operation for statistics made it difficult to assign credit to the SRF for facilitating a system-wide approach, although the clear recognition in the upcoming NSDS2 of the need to strengthen the capacity of the major data-producing MDAs will provide that opportunity in the near future. Other reported factors which hampered the SRF’s potential for promoting the system-wide approach included delays in the twinning partner procurement process in Afghanistan, and mixed experience at the state level in Nigeria. 38. In the pipeline countries, where many respondents understandably noted that it was premature to draw conclusions, some nevertheless saw reasonable promise in the SRF’s potential to promote a system-wide approach to statistics. 32 Table 5 Success of the SRF-CF in promoting a system-wide approach 1E Overall, how successful do you Barely Too consider the SRF-CF to be in successful or Promises early promoting a system-wide approach to Very Somewhat not successful at to be to No Total statistics in [name of country]? successful successful all successful judge answer responses Number of responses Implementing countries 19 22 4 5 7 0 57 Pipeline countries 7 8 1 12 23 2 53 All SRF countries 26 30 5 17 30 2 110 Percentage distributions Implementing countries 33 39 7 9 12 0 Pipeline countries 13 15 2 23 43 4 All SRF countries 24 27 5 15 27 2 Stakeholder groups: TTL 25 17 0 0 58 0 12 NSO 32 22 2 22 22 0 41 OGA 26 32 11 16 16 0 19 ONU 20 60 0 20 0 0 5 IDS .. .. .. .. .. .. 4 LDP 10 31 7 14 31 7 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 33 a) Inclusiveness of the NSDS 39. The requirements for developing a system-wide approach to statistical development at the country level include a well-prepared statistical strategy in place or being developed, with high-level political endorsement. Although the Evaluation Team did not make any detailed study of the national statistical development strategies of the countries in which the SRF is operating or is scheduled to operate, it did seek to determine the degree of inclusiveness of the approach to formulating each NSDS. Inclusiveness in this context refers both to the participation of key stakeholders and the coverage of important sectors. 40. The SRF prerequisite of an inclusive NSDS was felt to be met at least to some extent by 85 per cent of all respondents, with over half (53 per cent) indicating that the process was very inclusive (see Table II.1 in Annex 2). Ethiopia and Afghanistan fell somewhat below that figure, while respondents from Senegal were unanimous that the NSDS had been formulated in a very inclusive manner, and added comments to that effect. Among stakeholder groups, three-quarters of TTLs and national statistical offices felt the process had been very inclusive; other groups were significantly less positive, with over a quarter of local development partners indicating uncertainty on the matter. b) Role of development partners 41. Some 63 per cent of respondents who were not development partners noted that development partners with an interest in supporting statistics very much agreed with the development of a system-wide approach (see Table II.2 in Annex 2). When those who somewhat agreed were added in, the figure rose to 90 per cent; indeed nobody opposed the notion. That being said, confidence in developing partners very much agreeing with the system-wide approach was considerably less than average in Ethiopia and Afghanistan. 42. When a similar question was posed to the development partners themselves, support for the development of a system-wide approach was less pronounced, but nevertheless fully 80 per cent of development partners subscribed to this view at least to some extent (see Table II.3 in Annex 2). Local development partners in the implementing countries were generally more positive than those from the pipeline countries, so it is possible that enthusiasm for the approach will increase among development partners as implementation gets under way. 43. Local development partners, in both implementing and pipeline countries, identified two groups of problems when asked about difficulties they might face in being involved in or supporting the development of a system-wide approach. One group of issues concerned limitations on the part of national institutions: limited technical and human resource capacities in national statistical offices, difficulties of coordination among government agencies, and a lack of commitment from some government partners; in one case a lack of consultation was cited as the reason for minimal involvement of a donor agency in the country’s system-wide approach. 44. A second set of difficulties, at least as large as the first, related to development partners themselves. Local donor agencies sometimes lacked in-house organizational capacity, as well as technical expertise in the field of statistics, which was not a focal area for some donors. Constant turnover among development partner staff (including in some instances Task Team Leaders of SRF projects) was also an issue, inhibiting the development of a common understanding and purpose in relations with the national statistical leadership. There was, in addition, an admitted lack of harmonization and coordination among some development partners, who were tied to their own statistical support agendas; in one case an 34 agency’s own mandate and financial arrangements meant that it could work only with the national statistical office rather than the broader statistical system. c) Challenges to the system-wide approach 45. Despite some positive findings about the success of the SRF in promoting a system- wide approach, the survey results did not suggest that implementation of such an approach would be a straightforward task. To the contrary, 35 per cent of respondents indicated that a system-wide approach would pose a significant challenge and another 45 per cent a moderate challenge. Only 15 per cent of respondents overall indicated that a system-wide approach would not pose any significant challenge in their countries (see Table II.4 in Annex 2). That proportion varied somewhat among both implementing and pipeline countries, with Rwanda and Ghana predicting the fewest problems and respondents in DR Congo, Lao PDR, Nigeria and Senegal citing the greatest challenges. Among stakeholder groups, Task Team Leaders were the most likely to rate the system-wide approach as a great challenge. 46. The lack of a data culture in society and limited capacity of the national statistical system, often of the national statistical office but almost inevitably of other government agencies, line ministries and provincial and local administrations, were again cited by respondents as a challenge to the establishment of a system-wide approach. In some countries, fragmented legislative structures for statistical work, highly decentralized statistical systems, and a lack of harmonized statistical procedures and processes also provided obstacles. The capacity of statistical offices in many countries was adversely affected by uncompetitive salaries and high turnover. 47. Even where assistance was being brought to bear, for example through twinning arrangements, it was felt that a lack of institutional, management and coordination capacity in the national statistical office may still limit the effective achievement of a system-wide approach. It was pointed out that the development of such an approach could be challenging not only as concerns implementation but also in terms of conceptualization, and of prioritization of user needs and management of user expectations. On a practical level, managing different procurement procedures in a multi-donor environment could also be problematic. 48. A shortfall of political will and commitment in some countries was also seen by respondents as hampering the development of a system-wide approach. Contributing factors were the weak collaboration of agencies within the statistical system, and the failure of government to take a leading and sustained role in mobilizing funding for statistics including infrastructure and ICT facilities; it was noted that the effective implementation of a system- wide approach demanded resource availability over the medium and long term. In Nigeria, challenges were also seen in the relative autonomy of the states and the slow pace of strategic planning in statistics in some of them. 49. In addition to the problems mentioned earlier by local development partners, the past (and sometimes present) practice of different development partners supporting different line ministries also created difficulties. Not infrequently, development partners lacked an overall vision, and pressure for short-term results still often favored one-off survey activities rather than system-wide development. The separate development of project-related data collection mechanisms and databases, project teams working in parallel to the civil service (and sometimes poaching its best talent), and the fact that much if not most funding for statistics came from donors not government, also militated against the system-wide approach. In a large country like Nigeria, where donors tended to focus their statistical activities in different sets of states, parallel systems could be created. 35 ii) Increasing resources for implementing national statistical plans 50. As the name of the SRF Catalytic Fund indicates, the initiative was designed in part to help mobilize additional resources from other sources for the implementation of country- owned national statistical plans and statistical development strategies. While acknowledging that levels of funding for strategy implementation, from both government and development partners, were difficult to measure exactly 4 , the Evaluation Team designed a number of questions to attempt to gauge the extent to which this SRF objective was being met in the pilot phase countries. 51. Half of all respondents considered that the SRF-CF had been either very successful or somewhat successful in increasing resources for implementing national statistical development strategies. However, that figure included pipeline countries where, unsurprisingly, as many as 47 per cent of respondents indicated that it was too early to judge (see Table 6 below and Table II.16 in Annex 2). When looked at from the more realistic perspective of implementing countries alone, 72 per cent stated that the SRF-CF had been at least somewhat successful in attracting supplementary resources; in Afghanistan, that figure rose to 90 per cent. On this issue, there were no particularly large differences of opinion among the various stakeholder groups. 52. In commenting on this topic, respondents cited some specific instances where additional resources had been found for implementing survey activities within the NSDS, as well as support for capacity building and coordination related to the NSDS. In several cases the increase in resources could be traced to particular activities funded or promoted by the SRF, including statistical advocacy leading to greater awareness of the important role played by statistics, and institutional improvements in the pilot states of Nigeria. Positive results from SRF projects, along with the secure nature of SRF funding, were stated to be instrumental in helping to build confidence in both influential government agencies and among other development partners, opening up the actual or potential flow of resources. However, as noted earlier, there was a delay in the arrival of the SRF funds in Rwanda, and some respondents felt that the SRF’s catalytic potential would be fully put to the test only during the course of the upcoming NSDS2. a) Effect on domestic resources 53. The questionnaire attempted to obtain details of the effect on domestic and external resources of the decision of each country to enroll in the SRF pilot phase. Regarding domestic resources for statistics, 39 per cent of all respondents indicated that funding had already increased. Interestingly, this figure was slightly higher for pipeline countries than for implementing countries (see Table II.6 in Annex 2), but it must be noted that there was a very wide variation among countries and a significant proportion of respondents (30 per cent overall) who were not sure of the domestic resource situation. In Rwanda, respondents specifically noted that the increase in domestic resources for the NSDS had preceded the arrival of the SRF funding. 4 The original indicators on increased resources have been dropped from the current version of the SRF Results Framework. 36 Table 6 Success of the SRF-CF in increasing resources for implementing national statistical development strategies 2I Overall, how successful do you Barely consider the SRF-CF to be in increasing successful Too resources for implementing the national or not Promises early statistical development strategy in [name Very Somewhat successful to be to No Total of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 22 19 2 8 6 0 57 Pipeline countries 9 5 0 12 25 2 53 All SRF countries 31 24 2 20 31 2 110 Percentage distributions Implementing countries 39 33 4 14 11 0 Pipeline countries 17 9 0 23 47 4 All SRF countries 28 22 2 18 28 2 Stakeholder groups: TTL 33 17 0 8 42 0 12 NSO 29 27 0 22 22 0 41 OGA 42 11 5 26 11 5 19 ONU 20 20 0 40 20 0 5 IDS .. .. .. .. .. .. 4 LDP 17 24 3 10 41 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 37 54. Nearly half of all respondents did not answer follow-up questions on domestic resources, presumably because of a lack of knowledge or a view that no increase was forthcoming or planned (see Tables II.7 and II.8 in Annex 2). Of those who did reply, Task Team Leaders, in-country statisticians and NSOs were far more likely to report significant increases than were other government agencies or development partners. What emerges clearly is that only 10 per cent of respondents attributed increases in domestic resources directly to the SRF, with a similar percentage indicating that the increase would have occurred anyway. b) Effect on development partner resources 55. When respondents other than development partners were asked about the performance of the latter, one third indicated that development partners had already increased resources for statistical work – a figure that was corroborated by the donors themselves (see Tables II.9 and A.12 in Annex 2). However, when it came to the future intentions of development partners, respondents – particularly those from national statistical offices and other government agencies – were far more optimistic about plans to increase resources for statistics than were the donors. As was the case for domestic resources, there was a significant ‘don’t know’ component, and follow-up questions on resources of development partners also drew a fairly high rate of non-response among the other stakeholder groups. 56. This notwithstanding, around 60 per cent of respondents who were not development partners indicated that donors had increased, or planned to increase, resources for statistics to at least a moderate degree. The equivalent figure among donor respondents was only 27 per cent, with the rest not replying to that question (see Tables II.10 and II.13 in Annex 2). In similar vein, 20 per cent of respondents in other stakeholder groups considered that development partner increases in resources were directly attributable to the SRF, with another 22 per cent citing indirect attribution; in stark contrast, only 2 donor respondents (7 per cent) attributed their increases to the SRF (see Tables II.11 and II.14 in Annex 2). Whatever the situation, it is clear that the direct effect of the SRF on increases in resources for statistics in the pilot countries, both from domestic and external sources, has so far been quite modest. 57. A further issue which has emerged clearly is a significant divergence of views about the plans of development partners with regard to future disbursements for statistics. With most donor countries facing budget constraints, their increased reluctance to discuss the future availability of funding is understandable. There would nevertheless seem to be scope for the freer exchange of information within national partnership groups and similar bodies, so that a much clearer idea can be gained of the future availability of funding for the implementation of national statistical development strategies. c) The SRF as possible core finance for NSDS implementation 58. The Evaluation Team’s questionnaire also elicited opinions on whether SFR funds should be seen as core finance for implementing the respective national statistical development strategies (see Table II.15 in Annex 2). More than half of all respondents replied in the affirmative, and the figure rose to 60 per cent in the implementing countries, although respondents in Nigeria were markedly less enthusiastic than those in Afghanistan and Rwanda. 59. In support of the majority view that SFR funding should constitute core finance for NSDS implementation, respondents stated that government resources for statistical work were often very inadequate; many if not all SRF countries did not possess sufficient domestic 38 resources to fully finance their NSDS. In addition, resources from other development partners were in many instances also limited, and in some cases donors were seen to have changing and inconsistent agendas; moreover they often financed particular surveys or training courses. In that environment the SRF was seen as central and sustained core funding for broader, more fundamental issues such as institutional development (especially in post- conflict countries), capacity building, ICT and statistical infrastructure, and the statistical capacity of line ministries, thus helping to institutionalize a system-wide approach to statistics through the NSDS. 60. Elsewhere, it was noted that the SRF resources were being utilized for the collection and dissemination of data in line with the NSDS and its priorities. In situations where the SRF was financing equipment, it was felt that recurrent funding to maintain it should also be included, as regular government funding was not always sufficient for this purpose. In at least one pipeline country, SRF funding for NSDS implementation was seen as helping move the process forward and contributing to future resource mobilization from other sources; in addition, since the NSDS would provide the overall statistical development framework for government and development partners, it was important that the SRF be closely associated with it. 61. Those respondents who commented that SRF funds should not be used as core finance for the NSDS were in a minority, although they were more likely to articulate their views than those who were in favor. Their objections were essentially on two grounds. First, in countries such as Nigeria and Ghana, and in Senegal where the allocation is very modest, SRF funding is seen to comprise only a very small proportion of total requirements for statistical development and NSDS implementation, and thus cannot have the necessary impact to be viewed as core finance. In this context, the Evaluation Team notes that while the SRF grants may be limited relative to the needs of NSDS implementation, they are generally much larger than what has been provided for statistical capacity building from other sources, either bilateral or multilateral. 62. Second, and an opinion expressed more frequently5, is that core financing for the NSDS should be the responsibility of the government for purposes of ownership and sustainability, given that all national development goals were anchored on evidence-based planning, and assuming that the SRF funding would not be provided indefinitely. In Nigeria and Ghana at least, it was felt that adequate internal resources could be mobilized for the proper funding of the statistical agencies, but the political will might not be present; the SRF- CF should remain in a catalytic role, as its title specifies, creating an appropriate environment for improved data production and dissemination. 63. While giving varied replies to the question, a number of respondents emphasized the importance of the SRF funding being catalytic in the short to medium term, in the sense of producing results in a selected number of key critical areas. That could help boost the profile of statistics and demonstrate why they are important, thus providing a platform for government and other donors to join and contribute their support over a longer timeframe. 64. Among the respondents who were undecided on the issue, the view was expressed that much depended on how well the SRF was managed and the extent to which the provisions of the Paris Declaration on Aid Effectiveness could be implemented. While an objective of the SRF was to stimulate the flow of resources from other sources, the danger of the government actually reducing funding for statistics when donor finance increased was mentioned by one respondent, although there was no evidence that that had happened in 5 Particularly among respondents in Ghana. 39 respect of the SRF. In addition, a respondent in one country expressed the view that, outside the SRF, budgetary resources for statistics were being decreased and were unlikely to increase in the medium term. 65. In commenting on this question, respondents from Rwanda made frequent reference to the existing basket fund for statistical work, to which the SRF funds were added in 2012 (see Annex 3 for more details). The Evaluation Team notes that the dynamic statistical leadership evident in Rwanda, along with a strong results focus within government, has provided an environment conducive to the establishment and smooth operation of the basket fund. It nevertheless was impressed by the efficacy of this vehicle, which appears to be unique in the field of statistics, and was encouraged by suggestions elsewhere that other development partners might use the SRF (or a window in it) to channel their own resources for statistical development. d) Optimal relationship between the SRF-CF and IDA funding 66. In answering this question, and while recognizing that the optimal relationship might be country-specific, respondents most commonly mentioned complementarity. Ideally, SRF and IDA programs would be prepared jointly; such coordination and linkage would avoid duplication of support and help ensure a wider, more coherent approach to statistical development. 6 In larger countries especially, SRF finance could be used as counterpart funding or to leverage larger-scale IDA lending, which was seen as critical support for the overall success of SRF-CF funded projects; IDA funding should build on SRF work and support the priority activities it identified. Where IDA terms were stringent, linking SRF and IDA finance could make investment in statistics (and particularly in statistical systems) a less unattractive proposition. It was felt important that IDA funding for statistics should be more strategic, possibly eschewing support for stand-alone surveys in favor of longer-term strengthening of statistical systems. iii) Linking statistical systems to the needs of national monitoring frameworks 67. One of the aims of the SRF Catalytic Fund is to explicitly link improvements in the capacity and outputs of the statistical system with the requirements of existing national monitoring frameworks, such as those for the Millennium Development Goals and poverty reduction strategies, as well as of sectoral frameworks in each country. 68. The most frequent answer overall to this question, as posed in the Evaluation Team’s survey, was that it was too early to judge the performance of the SRF against this criterion. But in the implementing countries, some 60 per cent of respondents felt that the SRF was being at least somewhat successful in fostering this linkage (see Table 7 below and Table II.20 in Annex 2), and the figure rose to 75 per cent in Nigeria. In these implementing countries, local development partners were more likely than other stakeholder groups to withhold judgment on this aspect of the SRF. 6 In the case of Ghana, however, the linkage of the two programs caused the SRF project to be delayed. One element of the IDA credit is to support implementation of institutional reforms including staff redundancy, for which parliamentary approval has yet to be granted. However, the main reason for the delay was the decision by the Ministry of Finance not to sign any more loans for the time being due to the worsening fiscal environment. 40 Table 7 Success of the SRF in linking improvements in the statistical system to the needs of monitoring frameworks 3D Overall, how successful do you consider the SRF to be in linking improvements in the Barely statistical system to the needs of successful Too national and sectoral monitoring or not Promises early frameworks in [name of Very Somewhat successful at to be to No Total country]? successful successful all successful judge answer responses Number of responses Implementing countries 19 16 3 8 10 1 57 Pipeline countries 6 5 1 11 24 6 53 All SRF countries 25 21 4 19 34 7 110 Percentage distributions Implementing countries 33 28 5 14 18 2 Pipeline countries 11 9 2 21 45 11 All SRF countries 23 19 4 17 31 6 Stakeholder groups: TTL 25 8 0 17 42 8 12 NSO 24 20 2 20 34 0 41 OGA 26 26 11 26 5 5 19 ONU 40 60 0 0 0 0 5 IDS .. .. .. .. .. .. 4 LDP 14 10 3 14 41 17 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 41 69. Some comments by respondents (in Afghanistan as well as in the pipeline countries) echoed the above sentiment that this question was premature, and others did not directly address the linkage between improvements in statistical systems and monitoring frameworks. However, several respondents did cite SRF-supported priority, large-scale surveys and other activities that were directly providing high quality data for monitoring frameworks on the MDGs and on poverty and other economic and social phenomena. Improvements to administrative information systems and the creation of a data bank were also mentioned as assisting monitoring systems. In other instances the stimulus provided by the SRF process to collaborative effort between the statistical office and other stakeholders, to NSDS implementation, and to the mainstreaming of the strategy into the national developmental agenda, was seen as having a positive effect on national and sectoral monitoring and evaluation frameworks. a) Internalization of the MDGs and poverty reduction targets 70. The Evaluation Team’s survey showed that just over half of all respondents considered that monitoring the achievement of the Millennium Development Goals and/or poverty reduction targets had been entirely or almost entirely incorporated into the respective national monitoring frameworks (see Table II.17 in Annex 2). But that figure varied dramatically between countries, from 94 per cent in Rwanda and 80 per cent in Ghana to 30 per cent in Nigeria and a mere 16 per cent in Afghanistan. Nevertheless, when those who felt that a degree of incorporation existed were added in, over four-fifths of respondents in both implementing and pipeline countries replied positively to this question and this held for all the stakeholder groups except other government agencies. b) Enhancing the production, quality and timeliness of data 71. The views of questionnaire respondents were sought on how the SRF had affected two broad facets of improved statistical output: the production, quality and timeliness of data, and their accessibility and usability. 72. In the implementing countries, half of all respondents considered that the SRF had played a very significant role to date in enhancing the production, quality and timeliness of data, and that figure rose to nearly 80 per cent when those who felt that it had played some role were added in (see Table II.18 in Annex 2). Again, though, there were significant differences among the implementing countries: the SRF was seen to play a very important role in Nigeria and Rwanda, but very much less so in Afghanistan where many activities were just starting. In the pipeline countries the impact to date of SRF activities was understandably much smaller, while among the stakeholder groups the statistical office respondents were markedly more likely to assign the SRF a very significant role. 73. Invited to give examples of enhancements in the production, quality and timeliness of data, respondents in the implementing countries referred in many cases to general factors facilitated by the SRF funding, including human and institutional capacity building in the national statistical offices, cooperation among data producers (including between federal and state levels in Nigeria), development of sectoral statistical strategies for line ministries, the provision of consultative mechanisms, and coordination of the national statistical system. More specific examples cited included the successful conduct of key statistical operations (especially in Rwanda, although it was pointed out that under the principles of the basket fund, the SRF and other donor contributions were individually untraceable). Also mentioned were improvements in the CPI, the operationalization of common templates for data production in Nigeria, publication and dissemination activities including website 42 enhancements, and the provision of equipment including vehicles, generators and computers which helped improve the timeliness of data. In the pipeline countries, comments were by and large restricted to expectations surrounding stakeholder awareness, staff training, equipment and technologies, as activities had yet to commence. c) Enhancing the accessibility and usability of data 74. The impact of the SRF in enhancing the accessibility and usability of data in the implementing countries was reported to be somewhat less than for the production, quality and timeliness of data; nevertheless 42 per cent of respondents in the implementing countries reported that the SRF had played a very significant role to date, and another 30 per cent indicated that it had played some role (see Table II.19 in Annex 2). Differences among the three countries were still present but less pronounced, as were the differences among stakeholder groups. For pipeline countries, as with the previous question, uncertainty about the SRF’s role in improving accessibility and usability was the most common response. 75. Enhancements reported by respondents in the implementing countries included, in general terms, the encouragement of open data initiatives, the promotion of transparency, and awareness raising about statistics (particularly among other government agencies and at sub- national levels). Very detailed information was provided about the improved timeliness and frequency of key economic statistics in Rwanda which, along with the successful execution of major statistical operations, provided users with a sounder appreciation of the status of different sectors of the economy and a better basis for evidence-based policy and decision making. 76. Specific SRF-supported activities among the implementing countries which have led to heightened data accessibility and usability included improvements to dissemination and publication of statistics (most notably through increased computerization and upgraded statistical office websites), the creation of data portals containing raw survey datasets, media training for statistical office staff, workshops for journalists, the production of brochures for users, and the development of a compendium of standards (in Nigeria). Respondents in the pipeline countries mentioned preparatory activities including awareness-raising among policy makers, the promotion of statistical data to users, and website re-design, which could be attributed at least in part to the SRF. iv) Promoting dialogue between statistics users and statistical producers 77. An important requirement of the system-wide approach espoused by the SRF is the promotion of an improved dialogue between actual and potential users of statistics and statistical producers, as part of a government-led mechanism for stronger coordination and partnership across the national statistical system. The Evaluation Team asked respondents how successful they thought the SRF was in promoting such a dialogue. 78. In the implementing countries, a quarter of all respondents felt that the SRF was very successful in this regard, and a further 32 per cent rated it as somewhat successful (see Table 8 below and Table II.24 in Annex 2). Another 23 per cent (and as many as 39 per cent in Rwanda) felt that it promised to be successful, perhaps indicating that the promotion of a dialogue between users and producers is a longer-term activity than some others. In the pipeline countries, over half the respondents either felt that it was too early to judge on this issue, or failed to answer the question. In so far as producers and users of statistics could be distinguished from among the stakeholders responding, it appeared that the two groups held quite similar views on the success of the SRF to date in promoting a dialogue between them; 43 however, the only respondents stating that the SRF was barely or not successful at all in this regard seemed to be users. 79. Commenting on the user-producer dialogue, respondents from implementing countries indicated that it had been furthered in many cases by workshops, seminars and consultative meetings organized under the SRF, which had been well attended by a wide range of stakeholders including high-level Government users. The development of survey instruments, the conduct of surveys and the dissemination of survey results under the sponsorship of the SRF-CF had also contributed to generating collaboration and dialogue between producers and users. In Rwanda, regular steering committee meetings and sector working groups involving all relevant stakeholders were proving successful and productive. 80. However, respondents also stressed that continuous attention needed to be paid to maintaining the momentum of user-producer dialogue, in order to enhance coordination and increase awareness among users. Improving the quality of data was seen as very important in building the confidence of users. Even where considerable progress had been made in data dissemination and in improving the accessibility and usability of statistics, greater publicity and outreach were needed to keep non-core data users, such as those in research and academic institutions and the media, abreast of developments. a) Functioning of existing institutional mechanisms 81. Overall, only some 27 per cent of all respondents felt that institutional mechanisms in the field of statistics in their respective countries functioned well (see Table II.21 in Annex 2).7 That figure would have been lower still had not Rwanda registered 67 per cent, and in fact in three pipeline countries 10 per cent or fewer respondents viewed such mechanisms as functioning satisfactorily. In all, some 41 per cent of respondents considered that institutional mechanisms were functioning only to some extent, and 22 per cent that they were functioning badly (if at all). Taking the responses to this question together with the Evaluation Team’s observations in Nigeria and Rwanda, it seems reasonably clear that in these two countries , the SRF has been having a positive impact on the workings of the respective national (and in Nigeria, state) mechanisms bringing together users, producers and other stakeholders in the statistical arena. 7 Where possible, a specific mechanism was mentioned in each country’s questionnaire. They included consultative committees, NSDS steering committees, national partnership groups, and the like. 44 Table 8 Success of the SRF in promoting an improved dialogue between users and producers of statistics 4D Overall, how successful do you consider the SRF to be in Barely promoting an improved national successful Too dialogue and partnership between or not Promises early users and producers of statistics in Very Somewhat successful to be to No Total [name of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 14 18 5 13 5 2 57 Pipeline countries 5 6 1 13 21 7 53 All SRF countries 19 24 6 26 26 9 110 Percentage distributions Implementing countries 25 32 9 23 9 4 Pipeline countries 9 11 2 25 40 13 All SRF countries 17 22 5 24 24 8 Stakeholder groups: TTL 33 17 0 17 25 8 12 NSO 17 20 0 34 24 5 41 OGA 16 21 16 26 16 5 19 ONU 20 40 0 20 0 20 5 IDS .. .. .. .. .. .. 4 LDP 10 24 10 14 28 14 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 45 b) User satisfaction surveys 82. Periodic user satisfaction surveys are a requirement under the SRF, and since their status in each country is known, the question on this topic aimed mainly to elicit the experience of respondents with such surveys and to gather information on action subsequently undertaken.8 However, the replies also served to illustrate the degree to which respondents were familiar with the situation in their own country. On this measure, respondents from Ghana, Rwanda and Senegal were the best informed (see Table II.22 in Annex 2). Altogether, over a third of respondents were not sure of the situation with regard to a user satisfaction survey, to which could be added a further 7 per cent who gave an incorrect answer. Of even more concern, in the countries where surveys had already been conducted, some 77 per cent of respondents in what could be considered the main user groups (OGA and ONU), and 55 per cent of local development partners, either professed ignorance about the status of user satisfaction surveys in their respective countries or gave an inaccurate answer9. It seems clear that almost all implementing agencies should make much greater efforts to inform other stakeholders of their plans for user satisfaction surveys, and to publicize the results of those surveys which have already been conducted. 83. Respondents’ experience with user satisfaction surveys naturally varied widely from one country to the next, and the Evaluation Team considers that it is not worthwhile to try to summarize the results of those surveys already undertaken (Annex 7 lists the relevant survey documentation). In general a degree of satisfaction was expressed about the utility of the surveys conducted, and in at least one case local development partners were involved in the data analysis and report writing. 84. According to respondents, the action most commonly taken as a result of user satisfaction surveys was awareness raising among government agencies, development partners, research institutions and others concerning the availability and use of statistical data and the key role of the statistical office itself; as seen from the above, such action is certainly needed. Publication of a compendium of statistical standards, adjustments in data collection instruments and improvements in data dissemination, including through CDs and websites, were also cited as resulting from the surveys. In Rwanda, the program implications of the survey findings are being incorporated into annual plans of action and the future NSDS2. c) Enhancing the capacity of data users 85. Some 44 per cent of respondents in the implementing countries considered that there had already been efforts under the SRF to enhance the capacity of data users; 12 per cent thought that such efforts had not yet been made, but were planned for the future (see Table II.23 in Annex 2). Over one-third of respondents stated that they were unsure of the situation with regard to efforts to enhance the capacity of data users; that was understandable in the pipeline countries, but strikingly it held true in two of the three implementing countries also. Again, the stakeholder groups most closely identified with users of statistics were the least 8 As of September 2013, user satisfaction surveys have been conducted in Afghanistan, Ghana, Nigeria (national and state levels) and Rwanda, and are planned in Ethiopia, Lao PDR and Senegal. 9 More understandably, those figures rose in the countries yet to undertake user satisfaction surveys: to 100 per cent of OGA and ONU respondents, and 73 per cent of local development partners. 46 well informed about the activities being undertaken for their purported benefit. Action to remedy this lack of awareness would seem to be clearly warranted.10 86. SRF-supported training in various forms was reported by many respondents in the implementing countries and by a few in pipeline countries as preparatory activities got underway. Users receiving training included central and state/provincial government officials, university students, the media, and civil society organizations. Examples of the type of training being imparted were seminars for journalists (in both the public and private sectors) on statistical reporting and statistical literacy, for sectoral data users on the interpretation of statistical findings for policy development, and for researchers on micro data access tools. Among the many other activities designed to enhance the capacity of users were stakeholder workshops, high level fora with ministerial participants, Bank-facilitated informal dialogue between the statistical office and other ministries, television and radio programs, and other advocacy and outreach efforts to explain the importance of statistics. These efforts notwithstanding, it was recognized by several respondents that more needed to be done to keep pace with users’ needs and expectations. v) Delivering more effective aid through coordination of development partners 87. The SRF-CF aims not only to provide an increase in the financial resources for statistical development, but also to promote and support efforts to improve the effectiveness of those resources, specifically by applying the principles of the Paris Declaration on Aid Effectiveness to statistical projects and programs. SRF objectives include the alignment of donor support for statistical capacity building with the country-led national statistical plan or development strategy, and the better coordination of development partner programs within that framework. 88. The Evaluation Team’s survey showed that in the implementing countries, 28 per cent of respondents felt that the SRF was very successful in delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement, and a further 37 per cent thought it was somewhat successful (see Table 9 below and Table II.27 in Annex 2). There were fairly substantial differences among the three implementing countries, probably reflecting to some extent the position in each country prior to the SRF intervention. Respondents in Nigeria were the most positive in this regard, while nearly half those in Afghanistan felt that the SRF was somewhat successful in delivering more effective assistance; in Rwanda, some 44 per cent of respondents were non-committal on the issue. 89. Predictably, nearly half of all respondents in the pipeline countries felt that it was too early to judge the success of the SRF in meeting this objective, although those in Lao PDR were much less inclined to select this option -- possibly because implementation was getting underway at the time of the evaluation. Among the stakeholder groups in both the implementing countries and overall, local development partners were only slightly less positive than national statistical office respondents in ascribing a degree of success to the SRF in the better delivery of aid and technical assistance. 10 It is of course possible that those statistics users who returned the questionnaire were more than usually unaware of the efforts being made under the SRF to assist them. The Evaluation Team considers this improbable; from various indications including correspondence with potential respondents who eventually failed to submit a questionnaire, the bias is likely to be in the other direction. 47 Table 9 Success of the SRF in delivering more effective aid for strengthening statistical systems 5C Overall, how successful do you consider the SRF to be in delivering more efficient and effective aid and technical assistance for Barely strengthening statistical successful Too systems and results or not Promises early measurement in [name of Very Somewhat successful to be to No Total country]? successful successful at all successful judge answer responses Number of responses Implementing countries 16 21 2 10 6 2 57 Pipeline countries 6 6 0 12 25 4 53 All SRF countries 22 27 2 22 31 6 110 Percentage distributions Implementing countries 28 37 4 18 11 4 Pipeline countries 11 11 0 23 47 8 All SRF countries 20 25 2 20 28 5 Stakeholder groups: TTL 33 8 0 8 42 8 12 NSO 20 24 0 27 27 2 41 OGA 32 16 5 32 11 5 19 ONU 0 60 0 0 20 20 5 IDS .. .. .. .. .. .. 4 LDP 10 31 3 14 34 7 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 48 90. In their comments on this question, respondents in the implementing countries frequently made reference to the fact that the SRF project addressed the overall statistical system in each country (in one instance providing the only external funding for the NSDS), and provided a good platform for development partners to share information and experiences, become aware of what others were doing, and plan and channel their support to statistics. Coordination of assistance had improved, although in at least one country there was further scope for improvement and a reduction in duplication; in addition, it was pointed out that the effectiveness and efficiency of aid disbursement could be difficult to measure. In Rwanda, as mentioned elsewhere, respondents considered that the SRF could make an impact by mobilizing and coordinating further support and aligning it towards the upcoming NSDS2. As expected, there were few comments from pipeline countries, but respondents in Ethiopia referred to the SRF’s promise in terms of improving technical expertise, user-producer relations, and awareness among data providers of the importance of statistics. a) Aligning support for statistical capacity building to the NSDS 91. Overall, 35 per cent of respondents in the implementing countries considered that the SRF had already had a positive impact on aligning support for the respective national statistical development strategy (see Table II.25 in Annex 2); Rwanda registered a lower figure of 22 per cent, probably reflecting the later arrival of the SRF funds compared to Afghanistan and Nigeria. 53 per cent of respondents in implementing countries, and 81 per cent of those in pipeline countries, considered that the SRF had the potential to make a positive impact; only in the Lao PDR was there an appreciable degree of uncertainty about the efficacy of the SRF in this regard. b) Better donor coordination of statistical capacity building 92. In general, responses to the question of whether the SRF approach was leading to better donor coordination closely mirrored those to the issue of aligning statistical capacity building to the NSDS. Thus, 37 per cent of respondents in implementing countries stated that the SRF approach had already led to better donor coordination of statistical capacity building, and a further 53 per cent thought that it had the potential to do so (see Table II.26 in Annex 2). In the pipeline countries, positive responses also approached this 90 per cent figure. A handful of development partners were the only respondents to record an opposing view, namely that the SRF had not improved or would not improve donor coordination; in this regard it was noted that contention could arise when most donor institutions followed their own monitoring and reporting mechanism rather than fully supporting a common nationwide approach. The Evaluation Team felt that the number of development partners active in statistics in each country, and the ease or otherwise of gathering them together for meetings organized by the lead donor, were also practical factors affecting coordination of statistical capacity building. 93. It was again pointed out that in Rwanda, the existence of the basket fund meant that the SRF could not claim full credit for the success being achieved, and it was noted that in at least one country, the improvement in coordination of development partners was mainly due to the efforts of the in-country donor statistician (for more details see the following section). vi) Impact of the in-country donor statistician 94. The SRF approach relies on government and partners working together to provide coordinated support to implement national statistical strategies through the creation or 49 strengthening of national partnership groups in statistics. The SRF Charter also calls for an in-country donor statistician to be identified to assist the partnership group in technical matters and to promote the use of a system-wide approach. At the time of this report, statisticians were working with the SRF projects in Afghanistan, DRC, Ghana and Nigeria, and the Evaluation Team posed a number of questions to respondents in these four countries to try to ascertain the impact of the in-country donor statisticians in helping to achieve the SRF’s objectives. a) Usefulness of the in-country donor statistician 95. In all three countries where the statisticians have been working for some time, there was consensus, with just one exception, that they have had a very significant positive impact on the successful implementation of, or preparation for, the respective projects – both in terms of the role of the statistician and their personal contributions, which some respondents described as committed and passionate in support of the SRF objectives. The Evaluation Team’s survey found that the statisticians have been providing excellent support and guidance to the national statistical offices concerned; very good relationships have been built up and the statisticians’ advice has been taken on board. In at least one instance the statistician had been instrumental in influencing important institutional change, and in one country it was reported that having an expatriate rather than an indigenous donor statistician was contributing to the project’s success. 96. The statisticians have also played an important, sometimes crucial, role in facilitating the system-wide approach, coordinating donor activities in statistical capacity building and M&E, proactively leading the respective development partner groups, and maintaining good relations with other donors (again with one reported exception). In the DRC, expectations are high that the in-country donor statistician will be able to make a very useful contribution, for example in terms of advocacy and dialogue with government and donors, in what is recognized as a challenging situation. b) Impact on the statistician’s effectiveness of representing a major donor 97. There was widespread agreement among respondents that the fact that the in-country donor statistician represents a major donor had a positive impact on the incumbent’s effectiveness, especially where the statistician’s agency was also the lead donor in the national partnership group. 11 It was felt that this connection gave the statistician more influence among donor colleagues, and helped to ensure the active participation of other stakeholders. Respondents also commented that the relationship underlined the importance of statistical development, and helped to ensure the availability of information and of resources for effective coordination, as well as for statistical capacity building itself. However, in more than one case it was pointed out that the in-country statistician was effective in their own right rather than because of the influence of the organization which they represented. c) Contributions of the in-country donor statisticians to SRF objectives 98. The Evaluation Team sought feedback on the contributions of the in-country statisticians to the achievement of various objectives of the SRF as outlined in previous survey questions, and on whether the balance of their efforts among those objectives was 11 The in-country statisticians in Afghanistan and Ghana are funded by DfID; the lead donors are DfID in Afghanistan and the World Bank in Ghana (where the DfID-funded statistician is based in the Bank office). In DRC and Nigeria the in-country statisticians are funded by the World Bank, the lead donor, and located in its respective country offices. 50 appropriate. Respondents described various contributions made by the in-country statisticians, as already noted above and including guidance and advice to the statistical offices, coordination of donor activities, leadership of the development partner group, advocacy of the importance of the SRF and its role in supporting the NSDS, and general implementation and preparation of the respective SRF projects including effective targeting and utilization of funds. In fact, some respondents saw the role of the in-country donor statistician as pivotal to the success of the project, and needed throughout the SRF project cycle. The post served as a good interface between the World Bank and the recipient agencies, providing technical support for the proper implementation of the project. In Nigeria, the IDS also played a very important clearing-house role for all data used in World Bank projects. In the DRC, it was suggested that the in-country donor statistician’s presence should be utilized to develop an effective and sustainable partnership with donors, with the NSO leading the dialogue. 99. Discussions with all the current IDSs (Afghanistan, DRC, Ghana and Nigeria) reveal that office arrangements can influence priority settings among the three main groups of tasks performed by the IDSs, namely project implementation, coordination and helping the government strategize its statistical capacity development. While most IDSs are located in the local office of the lead agency which also finances them, in one case the IDS is financed by one major donor but based at the local office of a different donor (see footnote 11). This unique arrangement has helped to lighten the demand on the IDS’s time for project-related activities and has allowed the IDS to allocate more time for coordination and strategizing tasks. 100. In general, respondents felt that the statisticians’ contributions to the various objectives was quite appropriate; while their presence was useful in monitoring project implementation and providing their agency with feedback on progress made, any apprehension that the statisticians were overly involved with statistical work specific to their own agency appeared to have little foundation. d) Role of the in-country donor statistician in relation to the NSO 101. Respondents in the four countries where the statisticians are in place were asked whether they perceived any actual or potential areas of friction or contention surrounding the role of the in-country donor statisticians in relation to the national statistical system and its staff. Although there were one or two dissenting voices, a large majority of those replying indicated that they saw no friction at all; the statistical office staff recognized that the statisticians contributed to teamwork, were technically capable and provided useful advice on a professional basis; in addition they maintained cordial relations with all or almost all stakeholders. In one country the need for direct contact of the in-country statistician with the heads of statistics-producing institutions was identified, so as to exchange ideas and help ensure sound data quality and production procedures. e) Possible need for an exit or disengagement strategy 102. The Evaluation Team assumed that the post of in-country donor statistician was not a permanent one, and asked respondents whether they saw the need for an exit or disengagement strategy to cover the eventual withdrawal of the post. 103. Of those respondents who answered this question more or less unambiguously, about four-fifths did see the need to have such a strategy in place; it was feared that otherwise a vacuum would be created, and all the progress made under the project would be undone. It was suggested that locally recruited government-funded staff, perhaps in the form of a small 51 secretariat, should take over the role in due course in order to maintain some continuity; that would be preferable to consultants, who might not be able to speak with authority and might not be so committed or dedicated to building statistical capacity. 104. Emerging very clearly from the responses to this question is a strong sentiment in all the countries that because of a lack of capacity in statistics, the post of in-country donor statistician would be needed over a substantial period of time, until sustainable systems for the production and use of statistics could be established (including at state level in Nigeria). Long-term commitment was needed both from donors, to fund the post for the duration of the project, and from the government to maintain it subsequently; ownership and strong leadership by government was seen as important for sustainability and capacity building. It was also suggested that the continued engagement of the in-country donor statistician was needed so long as the implementation of the NSDS remained heavily funded by donors. 105. The Evaluation Team notes that the very substantial differences between SRF country situations make it impossible to offer a definitive opinion on how long the IDS should stay in post. Nevertheless, and based in part on the observations of respondents, factors to be taken into account before deciding on termination of the IDS post could include the degree of ownership and leadership by government of the statistical development process; the track record and level of functioning of the national partnership group; the level of cooperation and coordination among local development partners; the capacity of the World Bank country office and other Bank departments to support SRF project implementation; and the extent to which the national statistical strategy is funded from external sources. Some of the elements mentioned above are closely related to indicators in the generic Results Framework, which further warrant their close monitoring as part of the SRF project implementation. vii) Broad issues related to the SRF Charter (realism of the expectations for the SRF, choice of countries, regional project considerations) 106. The Charter of the Statistics for Results Facility Catalytic Fund was drawn up in 2009 and covers all salient points surrounding the objectives, principles, approach, governance structure and administration of the SRF. The Evaluation Team sought the views of respondents in most stakeholder groups about selected issues that relate to the Charter. a) Expectations for the SRF 107. Only around 3 per cent of respondents overall considered the expectations for the SRF, in terms of its impact and timing, to be unrealistic (see Table 10 below and Table II.28 in Annex 2). However, whether the expectations were considered ‘very reasonable’ or ‘somewhat reasonable’ varied considerably from one country to the next. Among the implementing countries, respondents from Nigeria were more than twice as likely to consider the expectations for the SRF to be very reasonable than their counterparts from Rwanda and Afghanistan; in the latter country, views on the realism of the SRF in terms of timing were coloured by the fact that the project and particularly the twinning process had taken longer to get off the ground than expected. A similarly wide range of views was observed among the pipeline countries, although here over a third of respondents did not express a clear view. Among the stakeholder groups, the TTLs were the most likely to consider the expectations reasonable, and other government agencies the least. 52 b) Significant impacts of the SRF 108. Respondents in the implementing countries recalled answers to earlier questions in describing the most significant impacts of the SRF in their respective countries (see, for instance, section III i) above). When looked at in total, the responses illustrated the wide range of impacts that the SRF was having across the gamut of its objectives. However, the impacts reported were by no means equally spread across the five objectives covered in this report.12 Well over half the ‘most significant’ impacts related either to improvements in the statistical system in terms of data production, quality and usability, or to the promotion of a system-wide approach. Impacts citing increased resources ranked third, while those concerning coordination of development partners and enhanced user-producer dialogue were each cited in fewer than 10 per cent of cases. Moreover, there were large differences between the countries: over half the impacts mentioned by respondents from Afghanistan related to the system-wide approach, while 80 per cent of those from Rwanda focused on increased resources and improvements in data production, data quality and dissemination. Nigerian responses, which were the most numerous, were more equally spread across the five objectives, although increased resources were hardly mentioned. 12 Although there were as many as 70 comments on this question, caution is needed in reaching any firm conclusions; the respondents were self-selected and from different mixes of stakeholder groups, many mentioned more than one impact, and it was not always straightforward for the Evaluation Team to categorize the comments by SRF objective. 53 Table 10 Expectations for the SRF 7A. How reasonable or realistic do you I'm think the expectations for the SRF are in Very Somewhat not No Total terms of its impact and timing? reasonable reasonable Unrealistic sure answer responses Number of responses Implementing countries 21 26 2 3 1 53 Pipeline countries 18 13 1 19 1 52 All SRF countries 39 39 3 22 2 105 Percentage distributions Implementing countries 40 49 4 6 2 Pipeline countries 35 25 2 37 2 All SRF countries 37 37 3 21 2 Stakeholder groups: TTL 50 42 0 8 0 12 NSO 46 34 2 17 0 41 OGA 26 26 0 42 5 19 IDS .. .. .. .. .. 4 LDP 31 41 3 21 3 29 Notes: This question was not posed to the ONU group. Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 54 c) Impact of the choice of countries on the SRF approach 109. The SRF-CF Council is responsible for determining which countries can apply for grants from the SRF, and selection is based on a number of criteria including levels of statistical capacity and extent of commitment to the SRF principles. Two of the original five countries – Afghanistan and DRC – were included because they are fragile states emerging from conflict and present particular problems for statistical development, including having a weak data culture and little history of data use. Although the terms of reference for the evaluation touched on the question of country selection only as a background issue, the Evaluation Team did ask respondents in selected stakeholder groups – Task Team Leaders, in-country donor statisticians and local development partners – about the extent to which they thought the choice of countries invited to apply for SRF grants had affected its impact. 110. The survey responses reflected a lack of information on the country selection process -- nearly half of all respondents either indicated that they were unsure on this issue, or didn’t answer the question. That figure was lower for respondents in the implementing countries, where 35 per cent felt that the choice of countries had affected the impact of the SRF approach to a great extent, and another 22 per cent to some extent. Overall, however, it was felt that the results from the survey did not warrant tabulation. . d) Possible extension of the SRF to regional operations 111. With the exception of the Listening to Africa project13, which became effective only in June 2013 and which the Evaluation Team has not attempted to cover in this assessment, the SRF has to date comprised country-specific projects. The views of respondents were sought on whether the SRF should be extended to regional initiatives and operations. 112. All told, some 38 per cent of respondents answered this question in the affirmative, with another 37 per cent being undecided (see Table 11 below and Table II.29 in Annex 2). Respondents in implementing countries were slightly more enthusiastic about the prospect of regional operations than were those in pipeline countries. Among stakeholder groups, nearly three-fifths of local development partners were undecided on the issue or did not answer the question. Among all stakeholders, though, there was a minority of respondents – about 16 per cent overall –who felt that the SRF should not be extended beyond its current essentially national scope. 113. Respondents who thought that there were, or might be, advantages to the SRF launching regional projects provided a wide range of suggestions for topics which might benefit most from a regional approach. It would be fair to say that if the SRF Council decides to venture into further regional projects, or introduce a regional dimension to current and future country projects, there will be no shortage of ideas to pursue. The general theme of capacity building in other statistical offices was frequently mentioned, along with regional training programs, south-south exchange of experiences among statistical offices (including through study visits and temporary exchange of expert staff), and the sharing of best practices and international expertise in various areas including modern data collection tools and data dissemination. It was felt that a regional focus in these initiatives was critical as countries learned from each other and were given the opportunity to share knowledge and demonstrate 13 Listening to Africa (L2A) is a Region wide project that aims at piloting household surveys with mobile interviews in three African countries (Cameroon, Malawi and Senegal). The pilot will last two years and will serve to demonstrate the ability of the approach to collect high quality, timely welfare data that is comparable over time and across countries. 55 results. The Evaluation Team noted that within the current country focus of the SRF, there appeared to be substantial scope for productive activities of a regional nature, including as a minimum the sharing of experience between pilot countries and reporting on SRF progress to regional entities. 114. Running through several of the comments from African SRF countries was the strong suggestion that any regional SRF statistical capacity building projects should be undertaken in close collaboration with existing regional and sub-regional entities such as the African Union, the UN Economic Commission for Africa, Afristat, UEMOA and ECOWAS, as well as with the Africa regional region of the Bank itself through its initiative to build capacity within the African Union Commission. 115. Apart from participating in strategic partnerships for the provision of expertise, these organizations (and others such as ASEAN in the case of Lao PDR) were also seen as playing a role in another major theme of respondents’ comments, namely the harmonization of concepts and methods. There were numerous suggestions for surveys, seminars and other projects that would help standardize methodologies, classifications and data compilation and dissemination procedures, in order to promote cooperation and facilitate comparisons at regional and international level. Specific ideas for regional work included the adoption and implementation of new systems such as the 2008 System of National Accounts (SNA), the harmonization of poverty monitoring survey methodologies, improvement of the quality of trade statistics via cooperation between statistical agencies in the region, a common labor force enquiry, and the harmonization of systems of data collection and production for agriculture and livestock. Regional approaches to tackling MDG data requirements, delivering systematic administrative data, and improving the quality of economic statistics were also suggested. 116. Some respondents from African countries felt that SRF funding could be used to support regional initiatives such as the New Partnership for African Development (NEPAD), the African Peer Review Mechanism, and the African Charter on Statistics. There was also support for the formulation and financing of regional strategies for the development of statistics, and for the Reference Regional Strategic Framework for Statistical Capacity Building in sub-regions of Africa. Institutional strengthening of the existing regional centres of excellence in statistical training was also advocated. A proposal from the user perspective was the creation of a regional or sub-regional data warehouse containing archived socioeconomic data from different countries, with well-managed web access for researchers, academics and development experts. 117. Despite the enthusiasm of many respondents for regional projects and initiatives, caution was expressed by others who were undecided on this issue. They felt that care would be needed to avoid the creation of an additional layer of bureaucracy, and to select countries where the level of statistical capacity and pace of statistical development were not too dissimilar. 56 Table 11 Possible extension of the SRF to regional operations 7C. Do you consider that the SRF, hitherto comprising almost exclusively country-specific projects, should be extended to regional initiatives and No Total operations? Yes Perhaps No answer responses Number of responses Implementing countries 24 20.5 9.5 3 57 Pipeline countries 18 20 8 7 53 All SRF countries 42 40.5 17.5 10 110 Percentage distributions Implementing countries 42 36 17 5 Pipeline countries 34 38 15 13 All SRF countries 38 37 16 9 Stakeholder groups: TTL 42 42 17 0 12 NSO 44 33 13 10 41 OGA 47 26 11 16 19 ONU 40 20 20 20 5 IDS .. .. .. .. 4 LDP 24 52 17 7 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 57 118. The minority of respondents who considered that SRF funding should not be utilized for regional projects were generally anxious that the SRF projects should first be satisfactorily completed in the respective pilot countries, with a strengthening of the NSO as an institution, and improvements in data quality, sectoral statistics and coordination. Lessons from the national projects should be learned and documented before regional ventures were contemplated. Concern was also expressed about a lack of control and monitoring in regional projects, and in one instance it was considered that regional approaches were ineffective. In at least two countries it was felt strongly that there were already too many actors and conflicting interests in the field of statistics, with uncoordinated regional initiatives drawing time and capacity away from more important national issues. e) Other issues related to the SRF Charter 119. Comments on other Charter-related issues were limited in number, as many respondents were not familiar with it. In fact, one proposal was that the Charter should be disseminated, so that countries and institutions became more knowledgeable about the important SRF initiative. It was also suggested that the document be simplified and the administrative portion separated out. Although the view was expressed that the Charter needed no amendment, it was also pointed out that it currently addresses challenges to national statistical systems; thus additions might be needed if regional programs are to feature more prominently in the future. viii) SRF governance arrangements and operational practices and procedures 120. In accordance with its Terms of Reference, the Evaluation Team examined various aspects of the governance arrangements of the SRF-CF and sought information on how well the operational practices and procedures of the Facility were serving the SRF’s objectives and its various stakeholders. In the administration of the questionnaire, an effort was made to restrict the questions to those stakeholder groups involved with the various issues raised. a) SRF grant approval and disbursement process 121. As many 25 per cent of all respondents did not provide an answer to this question, which was addressed to the TTL, NSO and IDS groups (see Table 12 below and Table II.30 in Annex 2). Of those who did, only around a quarter in both implementing and pipeline countries indicated that the review, approval and disbursement processes surrounding the SRF grant were smooth and timely. Task Team Leaders were much more likely than NSO respondents to give a positive response on this issue. Some 30 per cent of all respondents answering the question stated that the processes had been beset by difficulties and delays; as might be expected, this figure was higher in the pipeline countries overall, with Lao PDR an exception. 122. Closer examination of the replies, especially the remarks of respondents in describing particular facilitating or delaying factors in the process, reveals a more nuanced picture than might be gained from the tabulated responses. The fact that the World Bank’s grant approval and disbursement process had been considered smooth and timely by some respondents did not, however, mean that problems had not been encountered even in these cases. In both Afghanistan and Nigeria problems and delays in disbursements have occurred for internal reasons, over which the SRF Administration and Bank country staff and project teams have been able to exercise only limited control. 58 Table 12 SRF grant approval and disbursement process 8A. How smooth and timely was, or has Smooth Beset by been to date, the SRF grant approval and and Somewhat difficulties No Total disbursement process? timely problematic and delays answer responses Number of responses Implementing countries 6 12 4 5 27 Pipeline countries 5 7 9 9 30 All SRF countries 11 19 13 14 57 Percentage distributions Implementing countries 22 44 15 19 Pipeline countries 17 23 30 30 All SRF countries 19 33 23 25 Stakeholder groups: TTL 42 25 17 17 12 NSO 15 34 24 27 41 IDS .. .. .. .. 4 Notes: This question was posed only to the above stakeholder groups. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 59 123. Moreover, about half the responses falling into the ‘somewhat problematic’ and ‘beset by difficulties and delays’ categories were accompanied by comments which indicated clearly that the problems being referred to resulted mainly if not solely for internal reasons, rather than as a result of the Bank’s processes and procedures. In Nigeria the channeling of SRF funds through the Federal Projects Financial Management Division rather than directly to the two implementing agencies has caused serious delays in requests for replenishment, thereby affecting effective implementation of the work plan, including at state level. In Ghana, as described elsewhere, the initial grant approval process was reported as smooth, but the linking of the SRF to an IDA credit has caused a delay, due mainly to the decision by the Ministry of Finance not to sign any more loans because of the worsening fiscal environment. In Afghanistan the failure of the first round of negotiations on a twinning partner for the CSO was a significant source of delay, although the Evaluation Team was unable to determine the main source of the problem. 124. Elsewhere, a lack of skills and commitment among the relevant government officials and an unfamiliarity with World Bank procedures were cited as contributing to difficulties in the SRF approval process – although to other respondents, the cause of the delays was unclear. 125. The above factors notwithstanding, many respondents reported that the Bank’s administrative procedures had caused significant delays in SRF projects. That was particularly the case with procurement procedures and the issuance of non-objection clearance by the Bank, which were frequently described as slow and overly bureaucratic; oversight functions could usefully be simplified. In Rwanda these ‘heavy’ procedures had contributed to the delay in the release of the SRF funds into the Basket Fund account, along with frequent changes in TTL and local Bank staff which caused a lack of continuity (see Annex 3, for more details). In Senegal also, the process of (not) designating the TTL was seen as a hurdle, with the time since project formulation already exceeding 18 months. The nomination of experienced and committed TTLs and support staff was seen by respondents as key to successful project implementation, and the Evaluation Team was not always convinced that the Bank had sufficient capacity at country level to fully support the SRF process, despite the best efforts of the in-country statisticians and others involved. In general, respondents pointed out that more awareness of and training in SRF principles, norms and procedures was needed on the part of both Bank staff and government officials in the SRF project team, in order for the requirements as well as the benefits of an SRF project to be fully understood. b) Experiences with implementing the SRF grant 126. Nearly half of respondents in the implementing countries stated that their experience in implementing the SRF grant, judged on criteria such as the ease of reporting arrangements and support from World Bank staff, was very positive, and another 30 per cent had a somewhat positive impression (see Table II.31 in Annex 2). Although numbers were relatively small (only the TTL, NSO and IDS groups were asked this question), respondents in Nigeria and Afghanistan appeared to have slightly more favorable views than those in Rwanda. Negative experiences were in a very small minority, but were nevertheless strongly expressed; insufficient support from the SRF Administration Unit was mentioned, along with frustration that Headquarters staff sometimes failed to understand the realities of the situation in the field. At the same time, the Evaluation Team understood that the Administration Unit did not always receive all the necessary documentation relating to field activities such as supervision missions. 60 127. For respondents in pipeline countries, grant administration is largely an academic issue, and thus it was no surprise that 70 per cent either did not answer the question or indicated that they had insufficient experience to judge. c) Problems faced in implementing the SRF projects 128. The Evaluation Team also attempted to learn about the main sorts of problems that Task Team Leaders, in-country donor statisticians and national statistical offices had been facing in implementing the respective SRF projects. Although the question concerned implementation, it evoked nearly as many responses from the pipeline countries as from those where implementation is well under way; only from Lao PDR were there no comments offered. 129. The inadequacy of counterpart government budgetary support for the SRF project was felt acutely in Nigeria; it was also mentioned by respondents in Afghanistan, but not in Rwanda. Among the pipeline countries, respondents in DRC and Ethiopia foresaw this issue as a major problem. Linked to this, a failure or at least delays by government in translating commitment into practical action in support of the SRF project was also noted by some respondents in Nigeria, as well as in DRC and Ghana – the two of the original five SRF pilot countries where implementation has yet to commence. As noted earlier in this report, the view was expressed from Nigeria that the Government lacks sufficient political will to provide the project with adequate counterpart resources. 130. Another major set of issues concerned a lack of coordination. Among government agencies this lack of coordination was identified as a problem in Afghanistan, Nigeria, DRC and (presumably potentially) Ethiopia; in Nigeria, with its Federal – state system, there were coordination problems both horizontally and vertically. A lack of coordination among development partners was cited as a problem in Nigeria, DRC and Ethiopia. In Rwanda, coordination issues seemed much less acute. 131. Respondents in all three implementing countries, but particularly in Afghanistan, reported a lack of capacity as a major problem being faced in implementing the SRF project. In Afghanistan this low level of capacity was evident at all levels of government including the national statistical office, and related not only to substantive technical matters but also financial management and procurement. In Nigeria it was mentioned that some government agencies had inadequate capacity to generate relevant data relating to their functions. 132. Also reported under this question were numerous comments echoing those made earlier on problematic administrative procedures, both on the part of government administrations and the World Bank. In one country, the long delay between its identification as a pilot and the (to date, non-) arrival of SRF funds was even seen as raising questions about the full commitment of the Bank to the program. It is clear that these bureaucratic processes are seen as one of the major challenges facing the effective and efficient implementation of SRF projects in the field. d) Recipient-country execution of SRF projects 133. An important SRF principle, embedded in the Charter, is that countries applying for SRF grants must exercise leadership over their statistical development; in line with this, the Charter specifies that grants shall normally be recipient-executed. Bearing in mind the choice of SRF pilot countries, some with limited institutional capacity and memory, the Evaluation Team sought the views of selected stakeholder groups (Task Team Leaders, national statistical offices, in-country statisticians and local development partners) on the extent to which recipient execution was proving, or was likely to prove, a challenge. 61 134. Overall, about 60 per cent of respondents indicated that recipient-country project execution posed at least somewhat of a challenge, and this was broadly the case in both implementing and pipeline countries (see Table 13 below and Table II.32 in Annex 2). However, as might be fully expected, responses varied greatly among countries; a large majority of respondents in DRC and Afghanistan, but also in Senegal and Lao PDR, stated that recipient execution involved at least some degree of challenge. Only in Rwanda were a majority of respondents confident that this was not the case. Among stakeholder groups, TTLs were much more ready than NSO respondents to consider recipient-country execution a challenge. e) Attention to SRF projects by World Bank line management 135. As a result of some of its early discussions with various stakeholders, and in line with its remit to examine SRF governance arrangements, the Evaluation Team asked selected respondents whether they thought line management in the World Bank paid adequate attention to SRF projects.14 In the implementing countries overall, nearly half did – but this average concealed scores ranging from some 70 per cent in Nigeria to only 30 per cent in Rwanda (see Table 14 below and Table II.33 in Annex 2). 136. The pipeline countries showed lesser variation but, understandably, some 31 per cent of respondents were unsure on this issue or didn’t answer the question. National statistical office respondents, who comprised nearly half the total, took a more positive view than other stakeholder groups, with none indicating that attention by Bank line management was inadequate; a minority of other respondents felt otherwise, but the overall level of dissatisfaction was still under 10 per cent. 14 In retrospect, it might have been prudent for the Evaluation Team to have defined ‘line management’ more closely, as some respondents may have interpreted it to mean any World Bank manager. 62 Table 13 Recipient-country execution of SRF projects 8D. How large a challenge do you consider that recipient-country Somewhat Not a I'm execution of the SRF project has been A great of a significant not No Total or will be in [name of country]? challenge challenge challenge sure answer responses Number of responses Implementing countries 14 11 13 5 1 44 Pipeline countries 10 16 8 4 4 42 All SRF countries 24 27 21 9 5 86 Percentage distributions Implementing countries 32 25 30 11 2 Pipeline countries 24 38 19 10 10 All SRF countries 28 31 24 10 6 Stakeholder groups: TTL 42 42 8 0 8 12 NSO 17 32 29 17 5 41 IDS .. .. .. .. .. 4 LDP 38 24 24 7 7 29 Notes: This question was not posed to the OGA and ONU groups. Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 63 Table 14 Attention to SRF projects by World Bank line management 8E. Do you consider that SRF projects receive adequate attention by line To some I'm not No Total management in the World Bank? Yes extent No sure answer responses Number of responses Implementing countries 21 13 4 5 1 44 Pipeline countries 16 10 3 10 3 42 All SRF countries 37 23 7 15 4 86 Percentage distributions Implementing countries 48 30 9 11 2 Pipeline countries 38 24 7 24 7 All SRF countries 43 27 8 17 5 Stakeholder groups: TTL 33 50 17 0 0 12 NSO 56 22 0 15 7 41 IDS .. .. .. .. .. 4 LDP 24 28 14 31 3 29 Notes: This question was not posed to the OGA and ONU groups. Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 64 f) SRF grants as stand-alone projects 137. The same set of stakeholder groups was also asked whether they saw any advantage in the SRF grant being part of a larger World Bank operation (for example an IDA credit), rather than a stand-alone project as is currently the case. There was no great enthusiasm for this idea, with the overall yes vote, at 20 per cent, being slightly outweighed by those opposing it, i.e., opting for the status quo (see Table II.34 in Annex 2). A small majority of respondents overall, though, were either undecided on the question or failed to answer it; quite possibly they felt that they didn’t know enough about the pros and cons of the issue. Of comments received, one positive response came with the proviso that the larger Bank operation should also be in the field of statistical capacity building, while another respondent supported the existing situation on the grounds that statistics was a specific sector for public policy. g) The SRF Generic Results Framework 138. Annex 4 to this report is devoted to a detailed description and critique of the SRF Results Framework. The Evaluation Team also sought information on the degree of familiarity of survey respondents with the Results Framework, and on their experience in utilizing it. Again, stakeholder groups which were assumed to be unlikely to be using the Framework were not asked the relevant questions. 139. Overall, about a quarter of respondents stated that they were very familiar with the Results Framework, and another 40 per cent claimed some familiarity with it (see Table II.35 in Annex 2). Those very familiar with the Framework were more numerous in the implementing countries (especially Rwanda) than in the pipeline countries, but overall the proportion of respondents, who said they were not at all familiar with the Framework, or did not answer the question, was the same across the two groups of countries. As would be expected, Task Team Leaders and in-country donor statisticians were much more familiar with the Results Framework than respondents from the national statistical offices or local development partners.15 140. When TTLs, NSOs and in-country statisticians were asked about using the SRF Results Framework, a mere 5 per cent (3 respondents) reported their experience to be very positive; double that number found the experience to be generally negative (see Table II.36 in Annex 2). More than half of all respondents, however, indicated that they didn’t have enough experience with the Framework to make a judgment, or they left the question unanswered. 141. A large majority of those making suggestions for improving the generic Results Framework were very familiar with it, and the remainder somewhat familiar. The view was expressed was that the approach of having a generic SRF Results Framework and a set of country-specific results frameworks was confusing and needed clarification and simplification. A common complaint was that the generic framework was often not in line with, or was even at odds with, the country-level results frameworks in some respects (as is clearly shown in Annex 4). Criticisms made of the generic framework were that it contained too many indicators, was too heavy to maintain, required input from user surveys, and included indicators and targets that were too ambitious given the countries’ capacities to report. 15 The Evaluation Team was surprised to discover that local development partners indicated a slightly higher degree of familiarity with the generic Results Framework than did national statistical office respondents. If this was indeed the case, it might in retrospect have been advisable to have asked the subsequent survey question of the LDP stakeholder group also. However, there is also the distinct possibility of some confusion on the part of respondents between the generic and country-specific results frameworks. 65 142. Respondents recognized that the SRF pilot countries were heterogeneous and that it would be difficult to find indicators that were relevant and measurable in all of them. It was regretted, nevertheless, that the initial generic results framework had not been designed in close enough consultation with Task Team Leaders and others who would have to utilize it, to ensure that it was more realizable at country level; that needed to be borne in mind in the future. Suggestions for improvement included tailoring the generic framework to country circumstances as much as possible, and refining the generic framework indicators so that they could be easily measured at country level; at the same time it was considered important to ensure that key SRF generic results framework indicators were included in country project results frameworks, in order to reduce additional reporting requirements. h) Further issues relating to SRF governance and operations 143. Given that the SRF was a pilot exercise, it was felt that there was much more scope for proactive lesson learning and experience sharing among the pilot countries and in-country donor statisticians than had been the case to date. In addition, there was an expectation that the SRF team and the associated TTLs would have sufficient knowledge of Bank operational procedures to enable them to provide strong and clear direction to the statisticians and the implementing agencies. ix) General questions on the impacts of the SRF-CF on national statistical systems 144. The Evaluation Team also sought the views of survey respondents on a number of general, higher-level issues which relate to the principles, underlying approach and development objective of the Statistics for Results Facility Catalytic Fund, as described in its Charter and associated documents. a) The SRF concept and statistical capacity building 145. The background to the creation of the SRF-CF is briefly summarized in the introductory section of this report. Established on the principles of the Managing for Development Results process and the Paris Declaration on Aid Effectiveness, the SRF has attempted to bring a broader approach to the building of statistical capacity than previous initiatives. The Evaluation Team was interested to learn of the extent to which the SRF concept was meeting (or had the potential to meet) the needs for statistical capacity building in the pilot countries. 146. Over half of all respondents, in both implementing and pipeline countries, felt that the concept of the SRF met statistical capacity building needs to a great extent (see Table 15 below and Table II.37 in Annex 2). These figures did however differ very markedly from one country to the next, from about four-fifths of respondents in Ghana and Rwanda to only around one third in Afghanistan and DRC. When those respondents who felt that the SRF concept met the needs for statistical capacity building to some extent were added in, the overall figure among all respondents who answered the question rose to 90 per cent. A majority of every stakeholder group responded at least somewhat positively to this question, although over a third of other government agency respondents were unsure about the issue or failed to answer. 66 Table 15 The SRF concept and statistical capacity building 9A. To what extent do you consider that the concept of the SRF meets the needs of To a To To an I'm statistical capacity building in [name of great some insignificant not No Total country]? extent extent extent sure answer responses Number of responses Implementing countries 30 21 1 3 2 57 Pipeline countries 30 14 1 6 2 53 All SRF countries 60 35 2 9 4 110 Percentage distributions Implementing countries 53 37 2 5 4 Pipeline countries 57 26 2 11 4 All SRF countries 55 32 2 8 4 Stakeholder groups: TTL 58 42 0 0 0 12 NSO 61 34 2 0 2 41 OGA 32 32 0 26 11 19 ONU 60 20 0 20 0 5 IDS .. .. .. .. .. 4 LDP 55 28 3 10 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 67 147. Setting aside any concerns about the limitations of the World Bank’s Statistical Capacity Indicator (see the Appendix to Annex 4), it appears from the Evaluation Team’s survey that the SRF concept may better meet the statistical capacity building needs of those countries which already have higher statistical capacity as measured by the SCI. As shown in Annex 4, Rwanda had the highest overall SCI in 2013 by some distance, while DRC and Afghanistan had easily the lowest. Notwithstanding any differences between countries, the Evaluation Team received the clear impression that, overall, the SRF was playing a special role in statistical capacity building in the pilot countries. This was partly as a result of the size of the SRF grants in relation to most other sources of funding for statistical development, as mentioned earlier in this report. But it was also acknowledged in the Team’s conversations with numerous stakeholders, as well as in the comments of survey respondents, that the concept behind the SRF and the principles embodied in the SRF projects appeared to represent a new and effective approach to building sustainable capacities in statistics. b) Major statistical capacity building needs of SRF countries 148. Although the Team assumed that statistical capacity building, broadly defined, was the raison d’être of every country’s SRF grant application, and indeed of its approval by the Bank, it was interested to find out more about the main motivations behind each government’s acceptance of the Bank’s invitation to apply for an SRF grant, and about the major needs to be addressed. This question was posed to stakeholders working in national statistical offices and other government agencies only. 149. Motivations reported by respondents could roughly be divided into those relating to higher-echelon objectives corresponding to the outcome level of the SRF Results Framework, or possibly even beyond it, and those relating to institution and capacity building within the national statistical system itself. 150. Within the first category, in which Nigerian and Rwandan responses were prominent, motivations cited for acceptance of the SRF grant included the need to improve the evidence basis of planning for economic development (including for countries in post-conflict situations); to enhance governmental policy formulation, decision-making and planning through reliable and timely data; and to provide the ability (for government and development partners) to monitor and evaluate program implementation and country performance against targets. More specific objectives related to improvement of education standards and the eradication of poverty. The need to enlighten policy-makers about the importance of statistical information in policy- and decision-making, and a more general scaling up of statistical literacy, were also mentioned. A number of respondents referred to the aspiration of improving public access to statistics, and to the role of statistics in helping promote accountability and transparency in governance. 151. In the more internally oriented set of responses relating to the national statistical system, there were many more or less general references to the need to build institutional, infrastructural, technical and human capacities within the agencies of the system, particularly (in some countries) the national statistical office. More specific means of strengthening the national statistical system, cited by respondents from several countries including Ghana, included enhancing coordination of statistical activities, building capacity at sectoral level, improving the mechanism for detailed data sharing among agencies, harmonizing statistical standards and procedures, and improving the institutional and regulatory framework. Some 68 responses focused on the role of SRF funding in the effective implementation of the respective national statistical development strategies. Other motivations included, in Nigeria, a desire for holistic reform of the entire statistical system, streamlining the operations of statistical producers across three tiers of government; in Senegal, the alignment of statistical production to demand for data rather than supply; and in Rwanda, the achievement of self- sufficiency in terms of statistical expertise, through capacity building of local statisticians and researchers. 152. Many respondents cited the need for improvement in the availability of data, including sectoral data and information at state level in Nigeria, as a reason for applying for SRF grants. Funding from the SRF was also destined for improvements in the quality, reliability and timeliness of statistical information, as well as for several country-specific requirements in various fields of statistics. Apart from the direct use of the SRF project funds, the catalytic role of the SRF in further resource mobilization at national level was also mentioned. c) Leadership of the statistical development process 153. A basic principle underlying the SRF approach is that governments should exercise ownership and leadership of their national statistical development, by preparing comprehensive and realistic strategies and subsequently taking the lead in coordinating donor and government support for the implementation of those plans. The Evaluation Team asked survey respondents whether the SRF had facilitated that process, or appeared to have the potential to do so. It needs to be pointed out that in most of the pilot countries (DRC is the only exception), the NSDS had been designed before the SRF project preparation process got underway; thus the questionnaire responses relate more to the implementation stage of the respective statistical development strategies. 154. In the implementing countries, over half of respondents considered that the SRF had facilitated effective government ownership and leadership of the statistical development process to a great extent, and a further third thought that this was the case to some extent (see Table 16 below and Table II.38 in Annex 2). In terms of potential, the response from the pipeline countries was similar, although here some 20 per cent of respondents were unsure on the issue or didn’t answer the question. Among both groups of countries, however, there were wide variations among countries, with respondents in Rwanda and Ghana responding the most positively and those in DR Congo and Afghanistan the least. Among the stakeholder groups, some 70 per cent of respondents from national statistical offices felt that the SRF was facilitating (or would facilitate) government leadership to a great extent – more than double the percentage from other government agencies, a third of whose respondents were uncertain about the matter. d) SRF project implementation and governmental reform measures 155. According to the SRF Charter, countries applying for grants from the Catalytic Fund must in principle be ready to carry out any administrative reforms that might be required for proper implementation of their national statistical development strategies. The questionnaire sought information as to whether reform measures that would not otherwise have been embarked on were in fact being undertaken, or would potentially be involved, in the course of implementing the respective SRF country projects. 69 Table 16 Leadership of the statistical development process 9C. To what extent do you consider that the SRF has facilitated, or has the potential to facilitate, effective ownership and leadership by the To a To To an I'm government of the development of the national great some insignificant not No Total statistical system of [name of country]? extent extent extent sure answer responses Number of responses Implementing countries 31 19 1 5 1 57 Pipeline countries 27 15 0 8 3 53 All SRF countries 58 34 1 13 4 110 Percentage distributions Implementing countries 54 33 2 9 2 Pipeline countries 51 28 0 15 6 All SRF countries 53 31 1 12 4 Stakeholder groups: TTL 58 33 0 0 8 12 NSO 71 22 2 5 0 41 OGA 32 26 0 32 11 19 ONU 60 20 0 20 0 5 IDS .. .. .. .. .. 4 LDP 41 41 0 14 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 70 156. Across all the SRF countries, some 30 per cent of respondents thought that national authorities were, or would be, taking such reform measures to a significant extent, and this figure rose to 65 per cent when those who agreed ‘to some extent’ were added in (see Table 17 below and Table II.39 in Annex 2). Fewer than 10 per cent of respondents overall, in fact, were reasonably confident that SRF implementation would not involve reform measures. There were wide variations among countries: over half of Nigerian respondents, but only around 10 per cent of those in Afghanistan and Rwanda, saw national authorities undertaking reforms to a significant extent. Unusually, the proportion of respondents who were unsure about the situation or didn’t answer the question was smaller in the pipeline countries than in those where the SRF was already under implementation. Among stakeholder groups, Task Team Leaders were much more likely than others to consider that significant reforms were being or would be undertaken. e) SRF facilitation of the use of data to support evidence-based decision- making 157. The extent to which statistical data are used in policy- and decision-making processes in the grant-recipient countries is one of the two indicators which attempt to measure Outcome as defined in the Generic Results Framework for the SRF Catalytic Fund – increased capacity to formulate policies and make decisions for development using better statistics. However, as shown in Annex 4 to this report, baseline information for this indicator from user satisfaction surveys is only currently available for Nigeria. The Evaluation Team therefore hoped to gain insight into this important question through the survey questionnaire. 158. Overall, just over half of all respondents considered that the concept of the SRF has facilitated, or has the potential to facilitate, the greater use of statistical data and indicators to support evidence-based decision-making to a great extent (see Table 18 below and Table II.40 in Annex 2). Barring some 10 per cent of respondents who were unsure on this issue or did not reply, the remainder supported this position to some extent. As in other areas, there were distinct differences among both implementing and pipeline countries: respondents considering that the SRF facilitated (or might facilitate) more use of data in policy- and decision-making to a great extent ranged from 80 per cent in Ghana and 72 per cent in Rwanda to 33 per cent in DRC and only 11 per cent in Afghanistan. Although differences in the mix of respondents from one country to another cannot be ignored, this range of figures again appears to reflect, to a substantial degree, the level of statistical capacity development in the respective countries. 159. As has been the case on several other issues, the responses to this question of Task Team Leaders, in-country donor statisticians and NSO respondents were more positive than those representing other groups of stakeholders including other government agencies and local development partners. Nevertheless, a large majority of all the stakeholder groups agreed that the SRF concept was, at least to some extent, a facilitating factor in the greater use of statistics in the development process. 71 Table 17 SRF project implementation and governmental reform measures 9D. Does implementation of the SRF project, currently or potentially, involve national Yes, to a To No, or I'm authorities taking reform measures that would significant some probably not No Total not otherwise have been undertaken? extent extent not sure answer responses Number of responses Implementing countries 15 20 5 16 1 57 Pipeline countries 18 18 3 9 5 53 All SRF countries 33 38 8 25 6 110 Percentage distributions Implementing countries 26 35 9 28 2 Pipeline countries 34 34 6 17 9 All SRF countries 30 35 7 23 5 Stakeholder groups: TTL 67 33 0 0 0 12 NSO 32 29 12 20 7 41 OGA 26 26 5 32 11 19 ONU 20 40 0 40 0 5 IDS .. .. .. .. .. 4 LDP 17 41 7 31 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 72 Table 18 SRF facilitation of the use of data to support evidence-based decision-making 9E. To what extent do you consider that the concept of the SRF has facilitated, or has the potential to facilitate, the greater use of statistical data and indicators to support To a To To an I'm evidence-based decision-making in [name great some insignificant not No Total of country]? extent extent extent sure answer responses Number of responses Implementing countries 28 25 0 3 1 57 Pipeline countries 30 16 0 5 2 53 All SRF countries 58 41 0 8 3 110 Percentage distributions Implementing countries 49 44 0 5 2 Pipeline countries 57 30 0 9 4 All SRF countries 53 37 0 7 3 Stakeholder groups: TTL 58 42 0 0 0 12 NSO 66 29 0 5 0 41 OGA 37 37 0 16 11 19 ONU 40 40 0 20 0 5 IDS .. .. .. .. .. 4 LDP 41 48 0 7 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 73 x) Looking to the future 160. In replying to the questionnaire, respondents provided a number of further comments on the SRF, many of which are forward-looking in nature. These comments are summarized here. 161. Respondents in all three implementing countries expressed the wish that SRF funding should be continued. Although there was a minority view that the SRF could live up to expectations only with committed and experienced management and project staff, most respondents from Afghanistan took the opportunity to thank the SRF project staff and team members for their assistance and underlined the importance of the project to statistical capacity building, in particular to respond to data requirements for proper monitoring and evaluation of the expected results and outcome of the national development plan. 162. Despite some concerns that a lack of political will may have affected project implementation, and differing views about the effectiveness of monitoring of the project, it was generally felt that the SRF activities in Nigeria were bringing tangible results and that the project had made a positive impact within a relatively short period of time. It was viewed as a catalytic intervention in Nigeria’s long-term efforts to develop a well-integrated statistical system that was in line with global best practice and that would facilitate the production and use of data in national decision making. The SRF was also seen as helping develop critical statistical infrastructure and encouraging the incorporation of statistics into a national M&E framework. Concern was however expressed about the present limited coverage of the SRF project in relation to the size of the country and to the possible scope of activity. Respondents requested that additional funds be provided in order to consolidate the gains being made in the pilot phase, to extend coverage from the pilot states to the whole country, and to support the conduct of socio-economic and other surveys at sub-national level to facilitate the availability of baseline data16. 163. In Rwanda, while there was some concern expressed about a lack of feedback to certain development partners, there was a strong feeling that the SRF project was helping the national statistical office improve its systems and deliver data in timely fashion in line with the implementation of the NSDS. SRF funding needed to be continued in order to finance surveys and enhance the links between data producers and users. 164. It was also felt that the SRF-CF should be expanded to other countries, drawing on lessons learned in the pilot phase. The view was further expressed that the SRF could be utilized to disseminate and share knowledge through regional programs and partnerships, and could fund R&D on modern statistical tools and technology to improve institutional information on national statistical offices in Africa, where the current dearth of data hampered organizational planning and institutional development. 165. In the pipeline countries where projects were under preparation, the SRF grants were eagerly awaited as a means of, among other things, financing key statistical activities, subscribing to the SDDS, strengthening statistical training facilities, undertaking user satisfaction surveys, and filling data gaps to monitor development plans. In DR Congo, it was considered important to avoid further delay in commencing the SRF project, which was seen as an 16 The Evaluation Team was informed that the World Bank is currently preparing a follow-up technical assistance project in support of statistics which is projected to be at least $100 million in size and be ready for implementation in mid-2014 i.e. shortly after the completion of the SRF project. 74 opportunity for government to identify major shortcomings in the statistical system and take the necessary remedial steps. Respondents from Ethiopia felt that the SRF held out great potential if implemented with strong leadership and full stakeholder participation, with the project serving as the core vehicle for coordinated donor support to statistical capacity building. In Lao PDR, where the recently commenced SRF project was the key donor vehicle supporting the modernization of the statistical system, it was foreseen that it would improve coordination of sectoral information and access to data, and make a significant contribution towards implementation of the national statistical development strategy. 166. A number of respondents in both categories of countries felt that the notional resource allocation ceiling of $10 million was rather arbitrary, even though it was acknowledged that this sum could leverage additional resources in line with one of the SRF-CF’s core objectives. It was proposed that there should in future be some flexibility in the allocation amount for larger countries, and as an incentive for those countries that were performing well. Where implementation was proceeding rapidly, additional resources should be granted towards the achievement of the project development objectives, while follow-on grants could be provided for scaling up pilot survey programs or other statistical activities relevant to overall national development goals. 75 IV. Conclusions and recommendations i) Main conclusion and context 167. Based mainly on the generally positive feedback from the questionnaires received, the in-depth discussions with the Task Team Leaders and the in-country statisticians and the field visits to Nigeria and Rwanda, the Evaluation Team’s main conclusion is that in general, the SRF appears to be reasonably successful in meeting its objectives. It is, however, important to note the limitations of this evaluation, which draws conclusions from the experience about a program which is basically operational in only three countries. In view of this, care needs to be taken when considering this report in general and the conclusions and recommendations presented in this section. Most importantly, there is a clear general impression that for most of the success criteria utilized in the survey, the stakeholder groups which could be said to have a larger stake or vested interest in the SRF (TTL, NSO and IDS) were more positive than the remaining groups which largely comprise users of statistics. 168. While noting the need for a cautious interpretation of the results from the small sample size, the Evaluation Team believes that the positive results gathered from the survey and the field visits justify the recommendation that the SRF approach should be extended and expanded. For the sake of clarity, it is useful to recall that the above conclusion refers to the specific objectives set for the SRF-Catalytic Fund which are: i) promoting a system-wide approach in statistics at country level; ii) increasing resources for implementing country- owned national statistical plans; iii) linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks and promoting an improved dialogue and partnership between users and statistical producers; and iv) delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement. 169. The assessment of the SRF by the Evaluation Team also involved a thorough review of the results frameworks, generic and country-specific. As shown in Annex 4, in spite of some adjustments since the generic results framework was first designed, the lack of information on the key objectives of the SRF approach, the short implementation period and the limited number of countries (3) where SRF project implementation has been effective, has made the results framework less than satisfactory for evaluation purposes. Efforts by the SRF Administration Unit to populate the generic results framework, as presented in the SRF Annual Reports, are expected to provide useful insights on the SRF system-wide approach and on user-producer dialogue as more information is fed in from the second round of user satisfaction surveys. It is worth mentioning that based on the Evaluation Team’s survey, the experience with user satisfaction surveys is patchy and additional efforts will be required to improve them -- including, as suggested by the SRF Administration Unit (see below), the need to lighten the surveys and focus on key statistical producers and users. The results frameworks are discussed further in section vi) below. 170. The report also evaluates M&E arrangements of country-specific SRF projects and the Bank teams’ supervision reports. On the country M&E arrangements which are based on the national statistical development plans, the evaluation reveals that key information on the impact of the SRF projects on mobilizing additional resources for statistics, on the improved dialogue between users and producers of statistics, as well as on the harmonization and 76 coordination of development partners’ support, is generally lacking. A review of the World Bank’s supervision mission reports shows a more limited focus on the country projects’ development objectives and intermediate indicators, with few insights into the wider objectives of the SRF approach as detailed above. On resources, which are an important aspect of the Catalytic Fund, the Bank's monitoring has been focused on how much project money has been disbursed rather than on the broader aspects such as the amount of domestic and external funding generated for statistical work. 171. Because of the difficulties with the generic results framework and the country M&E tools and arrangements developed and utilized so far, the Evaluation Team concludes that a) the assessment, as envisaged in the SRF Monitoring and Evaluation Arrangements, was particularly timely; and b) it was important that the evaluation utilized techniques such as interviews, field visits and the questionnaire survey among all stakeholder groups, which are different from the tools routinely available. ii) Previous reviews of the SRF 172. In drawing up its conclusions and recommendations, the Evaluation Team recognizes that the SRF Administration Unit included in its annual reports in 2011 and 2012 a chapter on ‘Lessons Learned and Next Steps’. The first key lesson focuses on delays which arise from a variety of causes. Under the heading ‘How can we accelerate progress?’ the SRF Annual Reports identify four measures namely: a) making a better assessment at the initial stages of the project of what is in place and what additional support may be needed to ensure a smooth preparation; b) revisiting country selection for future SRF operations; c) lightening procedures for the application and preparation of SRF projects; and d) supporting the national project executing agencies in the areas of procurement, financial management, and overall project management including M&E. 173. The second lesson refers to the challenges for national statistical authorities to harmonize and coordinate the support and actions by development partners due to weak capacity and the need to juggle competing demands by development partners. The Annual Reports recommend more transparency on development partners’ activities in statistics and their sources of funding such as sharing of their programs of activities and a yearly review of these programs to enhance the monitoring of delivery and evaluation of effectiveness. The SRF Annual Reports also highlight the important role of the in-country donor statistician in helping to facilitate dialogue among the government agencies and with development partners and other users and thus to strengthen the role of the national partnership group. 174. The third lesson highlighted in the Annual Reports refers to the generic Results Framework and the difficulties encountered in getting the required indicators. Besides the implementation of practical steps such as reducing the number of indicators (see Annex 4 for more details), the Annual Reports recommend a number of steps, namely: i) starting the discussion on the M&E indicators at the beginning of the preparation process; ii) simplifying the user satisfaction survey and making it more selective both in terms of key users of statistics and main statistical areas; and iii) other measures such as fostering the network of 77 heads of national statistical offices of SRF countries and communications among the World Bank project teams. 175. Based on the questionnaire survey and the field visits to Nigeria and Rwanda, the Evaluation Team fully endorses the lessons and proposals for corrective measures identified by the SRF Administration Unit. Some of these issues will be revisited later in this chapter with insights from the survey responses. iii) Limitations of the evaluation: A cautious interpretation is needed 176. It was clear from the outset of the evaluation that the limited operational experience with the SRF would pose challenges to assessing its performance. Care therefore needs to be exercised in interpreting the findings of the evaluation. In particular, the Evaluation Team draws attention to the widely differing response rates among the various stakeholder groups, and to the likelihood that different groups may have different ‘stakes’ in the outcome of the assessment. Comparisons between countries also need to be undertaken with care, because of the different mix of respondents from each country. These points need to be borne in mind when considering this report in general and the conclusions and recommendations presented in this section. 177. Judging by the experience with the questionnaire, the SRF is not as well-known even in the implementing countries as one might expect, especially in the OGA, ONU and LDP groups. These groups are also less likely to agree that the NSDS was inclusive. While this general conclusion is not necessarily unexpected, the Evaluation Team feels that in the future, more conscious attention needs to be paid in the course of SRF implementation to involving these groups, which are largely users, in various ways – through user satisfaction surveys, reinvigorated national partnership groups, properly functioning statistical advisory councils etc. iv) Detailed conclusions and recommendations by SRF main objectives 178. There are real achievements that can be attributed to the SRF in all three implementing countries. In general, the five SRF objectives are being met, but with somewhat differing rates of success. a) Promotion of a system-wide approach to statistics 179. Over 70 per cent of respondents in implementing countries believe that the SRF is at least somewhat successful in promoting a system-wide approach. Contributory factors to this success can be identified in all three implementing countries. That figure rises to around 80 per cent when respondents who feel optimistic in this regard are added in. The equivalent figure for countries where implementation has yet to fully commence is 50 per cent, with 43 per cent of respondents in the pipeline countries feeling that it is too early to judge. But while development partners are seen to be, and say they are, on board in agreeing with the development of a system- wide approach -- fully 80 per cent subscribed to this view at least to some extent – their actual support for such an approach was less pronounced. 180. Despite some positive findings about the success of the SRF in promoting a system-wide approach, the survey results did not suggest that implementation of such an approach would be a straightforward task. Thirty five per cent of respondents indicated that a system-wide approach would pose a significant challenge and another 45 per cent a moderate challenge. One group of 78 issues concerned the general lack of a data culture in several of the pilot countries and limited capacity of the national statistical system. A second set of difficulties, at least as large as the first, related to development partners themselves. Local donor agencies sometimes lacked technical expertise in the field of statistics, which was not a focal area for some donors. Not infrequently, development partners lacked an overall vision, and pressure for short-term results still often favored one-off survey activities rather than system-wide development. Recommendations  Given the complex issues behind the promotion of a system-wide approach to statistics, it is difficult to formulate specific recommendations in the SRF context to tackle them. One particular aspect that may need further consideration is the role of the in-country donor statistician on coordination.  The World Bank should lead by example by ensuring that it has staff in the country office suitably qualified in statistics or M&E to serve on the National Partnership Group, especially in countries where an in-country statistician is absent or provided by another development partner, and should encourage other development partners to do likewise. b) Resource mobilization 181. The evaluation questionnaire addresses three issues: i) whether domestic resources to statistics have been increased since the start of the SRF project; ii) whether development partners have increased their funding for statistics; and iii) whether the SRF resources should be seen as core financing for implementing NSDSs. In general, the resource mobilization objective of the SRF may not have attracted as much success as some of the other objectives. 182. Regarding domestic resources for statistics, 39 per cent of all respondents indicated that funding had already increased. Interestingly, this figure was slightly higher for pipeline countries than for implementing countries, but it must be noted that there was a very wide variation among countries. This positive impression, however, seems to unravel in the detail: only 10 per cent of non-LDP respondents attributed increases in domestic resources directly to the SRF, and only 7 per cent of local development partners attributed increases of donor resources to the Facility. Also, a significant proportion of respondents (30 per cent overall) were not sure of the domestic resource situation and nearly half did not answer follow-up questions on domestic resources, presumably because of a lack of knowledge. A similar situation prevailed in relation to the resources of development partners; in addition, government respondents were far more optimistic about increases in LDP resources than were the donors themselves. Despite these levels of uncertainty, it seems clear that the direct effect of the SRF on increases in resources for statistics in the pilot countries, both from domestic and external sources, has so far been quite modest. 183. In support of the majority view that SFR funding should constitute core finance for NSDS implementation, many respondents stated that government resources for statistical work were often very inadequate. In that environment the SRF was seen as central and sustained core funding for broader, more fundamental issues such as institutional development (especially in post-conflict countries), capacity building, ICT and statistical infrastructure, and the statistical capacity of line ministries. 79 Recommendations  Because of big divergences of views revealed by the survey about the funding plans of development partners, it is recommended that the SRF should promote a more systematic exchange of information on funding intentions within national partnership groups.  Given the role of funding in ensuring sustainability of the support for statistics generated by the SRF, it is desirable for all concerned parties to increase efforts to generate additional funding and to actively explore additional resource mobilization methods, for example through matched funding arrangements.  The practice of countries producing regular NSDS implementation reports is considered to be very helpful in monitoring the commitment of resources and their delivery, and should be promoted during SRF implementation. While responsibility for producing NSDS implementation reports lies firmly with countries, the TORs of the in-country donor statisticians should specifically mention the provision of support to national authorities in this regard.  Aided by a conducive environment, the basket fund in Rwanda was proving effective for NSDS implementation, and the SRF should be used to encourage the use of basket funds elsewhere, taking country circumstances into consideration. c) Linking statistical systems to national monitoring frameworks 184. Sixty one per cent of respondents in implementing countries considered that the SRF was at least somewhat successful in fostering this linkage. In these countries, half of all respondents considered that the SRF had played a very significant role to date in enhancing the production, quality and timeliness of data, and that figure rose to nearly 80 per cent when those who felt that it had played some role were added in. To some degree, this result is not surprising given that for the seven countries for which detailed M&E arrangements are available, improvements in data collection, analysis and quality control figure very prominently. 185. The impact of the SRF in enhancing the accessibility and usability of data in the implementing countries was reported to be somewhat less than for the production, quality and timeliness of data; nevertheless 42 per cent of respondents in the implementing countries reported that the SRF had played a very significant role to date, and another 30 per cent indicated that it had played some role. Recommendations  In spite of problems mentioned in the introductory part of this section, user satisfaction surveys are one of the rare tools available to measure user satisfaction with accessibility, usability and other attributes of data. The Evaluation Team recommends that more research be undertaken into improving the quality and efficacy of such surveys and in the interim fully supports the proposal by the SRF Administration Unit to lighten the user satisfaction surveys and to be more selective on key data users. 80  Opportunities should be taken to critically assess initiatives such as PARIS21’s attempt to design a scoring system to measure the use of statistics in the policy process17.  The importance of the use of statistics should be highlighted in future SRF projects by, for instance, encouraging user satisfaction surveys to monitor this concern and making it more explicit in project development objectives. d) Promoting dialogue between statistics users and producers 186. Some 57 per cent of respondents in implementing countries (producers and users alike) consider the SRF to be at least somewhat successful in promoting this dialogue, with a further 23 per cent (and as many as 39 per cent in Rwanda) optimistic in this regard. This relatively high figure perhaps indicates that the promotion of a dialogue between users and producers is a longer-term activity than some others. In the pipeline countries, over half the respondents either felt that it was too early to judge on this issue, or failed to answer the question. In so far as producers and users of statistics could be distinguished from among the stakeholders responding, it appeared that the two groups held quite similar views on the success of the SRF to date in promoting a dialogue between them; however, the only respondents stating that the SRF was barely or not successful at all in this regard seemed to be users. In the countries where user satisfaction surveys had already been conducted, some three-quarters of OGA and ONU respondents, and over half of LDPs, either professed ignorance about the status of such surveys in their respective countries or gave an inaccurate answer. Moreover, users of statistics were the least well informed about capacity building activities designed for their benefit. Recommendations  Implementing agencies need to make much greater efforts to mainstream user satisfaction surveys in their work programs, to inform other stakeholders of their plans for those surveys and to publicize the results of surveys which have already been conducted.  More attention should be paid to user-oriented awareness raising activities through greater publicity and outreach as well as to covering these activities in country dissemination strategies.  Sharing among countries of various practices such as workshops, training of journalists and mobile education for user outreach should be encouraged. e) Delivering more effective aid 187. In the implementing countries, 28 per cent of respondents felt that the SRF was very successful in delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement, and a further 37 per cent thought it was somewhat successful. In their comments on this question, respondents in the implementing countries frequently made reference to the fact that the SRF project addressed the overall statistical system and provided a good platform for development partners to share information 17 PARIS21 (2012). ‘A Scoring System to Measure the Use of Statistics in the Policy -Making Process. Summary Report’. The scoring system provides an analytical framework for the assessment of national policy documents, and allows for a quantitative ranking based on each country’s use of statistics, which falls into three categories: upstream policy use, downstream policy use, and statistical capacity development in 44 low- and middle-income countries. 81 and experiences, become aware of what others were doing, and plan and channel their support to statistics. 188. Predictably, nearly half of all respondents in the pipeline countries felt that it was too early to judge the success of the SRF in meeting this objective. Among the stakeholder groups in both the implementing countries and overall, local development partners were only slightly less positive than national statistical office respondents in ascribing a degree of success to the SRF in the better delivery of aid and technical assistance. It was interesting to note that a handful of development partners were the only respondents to record an opposing view to the general view that the SRF has led to better aid coordination. Recommendations  In spite of the relatively positive reaction from the survey on aid effectiveness, the information gathered from the field visits and more in-depth discussions with in-country donor statisticians show that progress in this area depends to a great extent on personal relationships between members of the local development partners. The importance of sustaining these relationships needs to be borne in mind in a situation with possibly frequent staff rotation. Moreover, the SRF Administration Unit should be kept up-to-date on staff changes by the respective World Bank offices so as to brief new TTLs on the special features of the SRF approach. v) Other conclusions and recommendations based on the survey a) In-country donor statisticians 189. Respondents in countries where there is an in-country statistician were virtually unanimous that the statisticians have had a very significant positive impact on the successful implementation of the respective projects. Within the very broad SRF-CF guidelines for the in-country statisticians, they have been playing an important, sometimes crucial, role in facilitating the system-wide approach and helping coordinate donor activities in statistical capacity building. The fact that the statistician represented a major donor was seen as having a positive impact on their effectiveness, and there was little or no perception of friction surrounding the role of the statistician vis-à-vis the national statistical system. The Evaluation Team noted that office arrangements for the statisticians could influence priority settings, but in general, respondents felt that the balance of the statisticians’ contributions to the various SRF objectives was appropriate. The Evaluation Team concludes that the in-country donor statisticians have been an almost unqualified success. Recommendations  SRF projects should continue to incorporate the role of in-country donor statistician. In larger projects, more than one post should be considered subject to funding availability.  When drafting terms of reference for future in-country donor statisticians, or revising existing ToRs, consideration should be given to emphasizing the coordinating role, both between producers and users of statistics and among development partners. Given the 82 general lack of statistical capacity in the pilot countries, it is recommended that the post of in-country donor statistician should be funded for a substantial period of time.  Strong consideration should be given in future SRF projects to formulating an exit strategy for the eventual withdrawal of the IDS post.  In addition, the SRF Administration Unit should explore and analyze the lessons from different implementation models for the IDS and help to rebalance the IDS’ time allocation priorities where needed. b) Issues related to the SRF Charter 190. Only a very small minority of respondents considered that the expectations of the SRF in terms of its impact and timing were unrealistic, but among the rest expectations varied considerably from one country to another and among stakeholder groups. The implementing countries also exhibited large differences when the SRF’s most significant impacts were identified. Overall, though, it was clear that improvements in terms of data production, quality and usability, and the promotion of a system-wide approach, were seen by respondents as the SRF’s most important contributions to date. The Evaluation Team concludes that the SRF approach is certainly broad and flexible enough to have a significant impact in each of the diverse situations represented by the three implementing countries. 191. The possible extension of the SRF to regional operations drew varied responses, but those respondents in favor outnumbered those against. A rich vein of suggestions was put forward, featuring south-south exchange of experiences and the sharing of best practices and expertise. Many respondents proposed that any regional SRF capacity building projects should be undertaken in close collaboration with existing regional or sub-regional organizations. 192. From respondents’ comments on other issues related to the SRF Charter, and indeed their lack of comments, the Evaluation Team gathered that most respondents in the pilot countries – not unexpectedly--were not at all familiar with it. Recommendations  While retaining the country focus of the SRF approach, the SRF Council should explore the possibility of extending the SRF to regional initiatives including the introduction of a regional dimension to country projects, drawing on the (anonymous) suggestions put forward by respondents and collaborating where possible with other appropriate organizations .  The Council should promote the sharing of experiences across countries.  It is recommended that steps be taken for the Charter to be publicized, so that countries and institutions become more knowledgeable about the SRF initiative and the approach and principles underlying it. This action could be taken in conjunction with raising the visibility of the SRF on the World Bank internal and external websites. c) SRF governance arrangements and operational practices and procedures 83 193. Many of the Evaluation Team’s conclusions in this area corroborate the findings already documented by the SRF Administration Unit and mentioned above. Thus, initiatives under way to simplify procedures for the approval and disbursement of SRF grants can be readily supported by the Team – even though its survey revealed that many of the problems laid at the Bank’s door were in fact mainly or solely related to internal country difficulties. Similarly the call by survey respondents for more awareness of and training in SRF principles and procedures has essentially already been identified in the SRF Annual Reports -- although the Team notes that such activities need to be directed not only at national project executing agencies but also at World Bank country staff. 194. As opposed to only 22 per cent of respondents in the implementing countries who described the SRF grant approval and disbursement process as smooth and timely, 78 per cent rated their experience in implementing the grant, in terms of reporting arrangements and support from Bank staff, as either very or somewhat positive. In fact the nomination of experienced and committed TTLs and support staff was identified by respondents as key to successful project implementation. 195. Problems faced in implementing the SRF projects, identified by selected stakeholders not only in implementing but also pipeline countries, varied from country to country but included inadequacy of counterpart government budgetary support, a lack of coordination (both across government and among development partners), low levels of capacity, and overly bureaucratic processes on the part of both the Bank and national administrations. The Evaluation Team notes that procedural changes for the SRF projects have been designed to acknowledge and address most of these problems. 196. The generic and country-specific results frameworks for the SRF have been discussed earlier in this section as well as in Annex 4, and are taken up again in section vi) below. The Evaluation Team’s survey confirmed the difficulties that Task Team Leaders, national statistical offices and in-country statisticians have been having in using the generic Results Framework, with those finding the experience generally negative outnumbering those who thought it very positive. The review of the World Bank’s supervision mission reports shows a relatively narrow focus on the project development objectives and intermediate results indicators. Recommendations  The actions already identified by the SRF Administration Unit to improve SRF governance and operational issues should continue to be pursued. Support to implementing agencies in the areas of procurement, financial management and overall project management including M&E should where necessary be extended to the Bank staff concerned.  World Bank management should endeavor to ensure greater continuity in post of TTLs and other staff assigned to SRF project implementation.  Modify SRF guidelines to TTLs to stress the point that the coverage of broader SRF issues, as noted in the generic SRF Results Framework, need to be taken into account during project supervision, so as to be able to respond to SRF requirements. 84  Given that the SRF is a pilot exercise, proactive lesson learning and experience sharing among the pilot countries, both for Bank TTLs as well as for national government staff involved in the project and in-country donor statisticians, should be further pursued.  The SRF Administration Unit should continue its practice of including in the Catalytic Fund Annual Reports a chapter on Lessons Learned and Next Steps, incorporating where appropriate the issues raised in this evaluation report. d) General questions on the impacts of the SRF-CF on national statistical systems 197. Over half of all respondents, in both implementing and pipeline countries, felt that the concept of the SRF met (or had the potential to meet) their statistical capacity building needs to a great extent. However, these figures did differ markedly from one country to the next, and there appeared to be some relationship between the scores and the existing level of statistical capacity as measured by the SCI. Within the overall rubric of statistical capacity building, the major motivations for applying for an SRF grant also varied from one country to the next, but could roughly be divided into those relating to higher-echelon objectives corresponding to the Outcome level of the SRF Results Framework, and those concerning institution and capacity building within the national statistical system itself. 198. In the implementing countries overall, more than half the respondents considered that the SRF was facilitating effective government ownership and leadership of the statistical development process to a great extent; in terms of potential, the response from the pipeline countries was similar. On this question there were substantial variations among countries and among stakeholder groups. The same applied to the issue of whether implementation of the SRF project involved national authorities taking reform measures that would not otherwise have occurred; overall nearly two-thirds of respondents felt that this was, or would be, the case at least to some extent. 199. The Evaluation Team had particular interest in the extent to which statistical data are used in policy- and decision-making in the grant-recipient countries, as this is one of two indicators attempting to measure Outcome as defined in the Generic Results Framework, and evidence from user satisfaction surveys to date is weak at best. Around 90 per cent of respondents indicated that the concept of the SRF facilitated, or had the potential to facilitate, the greater use of statistical data and indicators to support evidence-based decision-making at least to some extent, although again there were distinct differences among both implementing and pipeline countries. The experiences of Rwanda and of Niger State in Nigeria which the Evaluation Team visited show that the importance given to statistics by senior level officials played a key role in the promotion of the use of data for planning and policy decision-making. 200. Overall, the Evaluation Team concludes that the principles, underlying approach and development objective of the SRF-CF, as described in its Charter, is finding a strong degree of resonance and relevance in meeting the statistical capacity building needs and aspirations of all the pilot countries. By and large, responses to this group of questions appear to be more positive from countries where the level of statistical capacity, by the measures that are available, is 85 higher. However, given survey-related limitations and the complexity of factors likely to be at work, the Team considers it difficult to formulate any specific recommendations in this area. e) Looking to the future 201. Respondents in all three implementing countries expressed the wish that SRF funding should continue, citing both achievements to date and unfinished (or in some cases unstarted) tasks still to be addressed. In the pipeline countries, the SRF grants were eagerly awaited as a vehicle for meeting their respective statistical capacity needs. As noted at the beginning of this section, the Evaluation Team recommends that the SRF approach should be extended and expanded. 202. The Evaluation Team notes that the current notional resource allocation ceiling of $10 million is rather arbitrary, and sees merit in the suggestions from a number of survey respondents that it be revisited. Recommendation:  While acknowledging the leveraging objective of SRF grants, the Evaluation Team recommends that the SRF Council re-examine the current notional resource allocation ceiling and consider upward flexibility for, for example, larger countries, as an incentive for high-performing projects, and to scale up pilot programs. . vi) Insights and recommendations on three specific issues concerning the future of the SRF 203. The discussion on conclusions and recommendations presented up until this point in sections iii) to v) is derived primarily from the responses gathered from the questionnaire survey in the 8 SRF pilot countries, enriched by in-depth discussions the Evaluation Team had with TTLs of the pilot projects, in-country donor statisticians and various stakeholders in Nigeria and Rwanda. The focus of the conclusions and recommendations is structured along the terms of reference of the evaluation namely, how successfully the SRF approach has achieved its objectives covering various outputs and practices engendered by the SRF, the inputs and contributions made possible by the SRF approach, and SRF governance arrangements including operational practices by the World Bank. The evaluation exercise takes as given the choice of the pilot countries, existing practices concerning the design and monitoring of the results frameworks (both the SRF generic and the country-specific ones), as well as procedures defined by the SRF Charter and World Bank project management practices. 204. Based on the positive feedback from the questionnaires received, the in-depth discussions with key stakeholders and the field visits to Nigeria and Rwanda, the main conclusion of this evaluation, as highlighted at the beginning of this chapter, is that the SRF approach should be extended and expanded. In this perspective, the Evaluation Team, upon the request of the SRF Administration Unit, has undertaken additional work to provide insights and proposals on three important issues which have significant bearing on the future of the SRF approach. These issues are a) country selection, b) the results framework, and c) ways to enable greater flexibility in project implementation and to enhance greater ownership by countries and the World Bank operations units. 86 a) Country selection 205. For the pilot phase, the SRF-CF Council has been responsible for determining the criteria by which countries have been identified to receive grants from the SRF-CF. The criteria include country levels of statistical capacity and the extent of commitment to the SRF principles. Pilot countries represent a mixture of different levels of statistical development, different types of statistical systems, and different problems and concerns relating to the development of statistics including those of fragile states emerging from conflict. While the selection principles are clear, the SRF Charter is not sufficiently specific to help in the actual selection of countries. 206. Discussions with DfID staff who were closely involved in the selection of pilot countries revealed that the actual decisions on country selection were based on compromises between analytical considerations (such as the level of statistical capacity building and the capacity for MDG monitoring) and political considerations. There was a generally shared belief that the SRF approach would work across countries including those facing challenging circumstances. The selection of Afghanistan and the Democratic Republic of Congo (DRC) was based on this belief in spite of the fact that DRC does not have an NSDS; having an NSDS in place or under development was specifically mentioned as a requirement for countries applying for SRF grants. The importance of the World Bank’s capacity on the ground, the role of the World Bank country director/country coordinator, and the possibility of leveraging resources such as IDA funding were also mentioned as important considerations, although it is not clear how these considerations played a role in the final selection of countries for the pilot phase. 207. An exhaustive review of the SRF annual reports revealed repeated references to the themes of difficulty, variety and fragility for the selection of countries. The 2010 report states that because of the flexibility of the SRF approach, support should also be given to fragile states which do not have basic statistical capacity in place. The 2011 report states that all five initial pilot countries represent ‘potentially difficult circumstances’. The reports also showed continuing efforts to reflect on the issue of country selection. For instance, the 2011 report states that despite progress, modifications to the country selection criteria (and to internal administrative processes) would be needed to ‘unleash the Fund’s full [catalytic] effect’. The 2012 report suggested revisiting the country selection criteria to focus more directly on the pre- conditions. 208. Another issue which may affect country selection in the future revolves around two possible options for the SRF grant application process, namely i) an open competition among all IDA countries where defined criteria and prerequisites play an important role, and ii) an ‘invitation-only approach’ where countries are identified on the basis of technical and political considerations as has been the case under the pilot phase of the SRF-CF. Proposals 209. The recommendations for specific objectives of the SRF approach presented earlier in this chapter are based on a clearly defined approach to data collection designed for this evaluation, using the responses from the questionnaire survey, findings from the field mission and interviews with all project task team leaders and in-country statisticians. By contrast, the proposals suggested here for the selection of countries are solely based on the review of existing documents on the SRF-CF and discussions and insights shared among the members of the 87 Evaluation Team. Hence, by comparison to the ‘recommendations’ presented in the earlier sections, the ‘proposals’ suggested by the Evaluation Team on the issue of country selection (and for that matter on the two other issues presented in this section, the results framework and issues related to project implementation and ownership), are based on more limited underpinnings. But assuming that the SRF approach will be extended and expanded, it is hoped that the proposals listed below could, at least in part, contribute to a more methodical approach to country selection. 210. Fragile states versus other IDA countries. The positive feedback of the SRF from all 8 pilot countries including post-conflict countries appears to confirm the original belief that the SRF approach does work in all country circumstances although more care and support should be provided to fragile states. Based on the current classification of fragile states by the World Bank which includes 36 countries out of 82 IDA-eligible countries, it is suggested that future SRF funding may roughly be allocated 1/3 to fragile states and 2/3 to other countries. 211. Open competition versus ‘by invitation’. The relatively large size of SRF projects as compared to other funding for statistical capacity building, and the generally positive reactions to the SRF approach, would normally be expected to enhance the attractiveness of the SRF-CF as support for statistical capacity building. While there may still exist justification for specific countries to be ‘invited’ to apply for SRF grants as was the case under the pilot phase of the SRF, it is the opinion of the Evaluation Team that open competition should be the primary option for SRF funds allocation going forward. 212. Country selection criteria. The Evaluation Team has done a systematic review of relevant criteria or preconditions that could affect country selection, many of which are drawn from the experience of SRF pilot countries. These are summarized in the Appendix to this chapter. While the comprehensive review of all criteria is a useful exercise, the Evaluation Team believes that a two-step procedure would allow for a more practical approach to country selection for future SRF funding. 213. Assuming that open competition will be adopted as the general rule for SRF grant allocations in the future and given the relative success of the SRF pilot phase, candidate countries can expect their applications for SRF grants to be subjected to a higher degree of scrutiny. In fact, their willingness to apply for SRF funding is the first and foremost signal of country commitment. The two-step approach mentioned above would apply to both regular IDA- eligible countries and to fragile states, with some minor adjustments for the latter as shown in Table 19.For regular IDA countries, all six first-order criteria must at least receive adequate attention. The second-order criteria would help to fine-tune the selection process, allowing for flexibility in the choice of countries. The criteria were identified based on the general impressions of the Evaluation Team gathered from interviews during the field visits, written comments from the survey, along with a study of the criteria utilised to date. . Based on the assessment by the Evaluation Team, the degree of World Bank support to statistical capacity building plays an important role. 88 Table 19: A two-step approach to country selection for SRF funding Regular IDA countries First-order criteria Second-order criteria a. The degree of commitment to the SRF 1. Statistical capacity (such as the SCI but principles. preferably using three- or five-year b. The existence of an NSDS. averages instead of the latest number). c. The degree of high-level political 2. The quality of the NSDS with preference endorsement of statistical capacity given to implementation efforts. building. Preference for a results focus at 3. Strength of leadership of the national the highest levels of government. statistical system. d. The degree of World Bank support to 4. The existence of a senior in-country statistical capacity building. statistician from a lead donor agency to e. The potential of SRF funding in help with coordination and leveraging other resources, both domestic harmonisation efforts. and foreign, in support for statistics. 5. The degree of adherence to a medium f. The degree of coordination, through to long-term approach to statistical mechanisms or structures such as the capacity building, with preference National Partnership Group, between given to countries with increasing producers and users of statistics and funding for statistics. among the local development partners 6. The degree of commitment by the involved in statistical capacity building. World Bank team in country office. Fragile states First-order criteria Second-order criteria a. The existence of an NSDS or a well- 1. The degree of commitment to the SRF prepared statistical strategy being principles. developed where an NSDS would prove 2. The quality of the NSDS with preference very challenging to establish. given to implementation efforts. b. The degree of high-level political 3. Strength of leadership of the national endorsement of statistical capacity statistical system. building. c. The degree of World Bank support to statistical capacity building. d. The degree of coordination among the local development partners involved in support to statistics. 89 b) The SRF generic results framework and country applicability 214. The detailed review in Annex 4 of efforts to monitor progress in statistical capacity in the context of pilot country projects using the SRF approach highlighted the following challenges: i) Translating the generic results framework into specific indicators which are measurable and which can be monitored over time has proven to be much more challenging than originally expected. In this context the Evaluation Team acknowledges that devising indicators for a results framework in this field is no easy task, as has been documented elsewhere.18 The response by the SRF Administration Unit to the challenge of data unavailability has been to simplify the generic results framework by deleting indicators for which information is not available. As a result of this response, since the start of the SRF-CF, six of the original 22 indicators have been deleted. In particular, almost all indicators on commitments/disbursements of resources by the government and development partners have been removed, the only exception being disbursements under the SRF projects; ii) Even with a reduced set of indicators, information required to populate the generic results framework remains sparse and incomplete, making it difficult to measure and monitor key SRF principles, most importantly: a) the leveraging impact on resources of the SRF Catalytic Fund; and b) development partner support to the implementation of NSDSs; iii) In the context of specific country projects, there has been limited success by the project teams in blending the indicators of the generic results framework with country priorities based on individual country NSDSs. Updates of the SRF generic results framework using country information are produced by the SRF Administration Unit for the purpose of reporting and as an input to the SRF Annual Reports; and iv) The review of the World Bank’s supervision mission reports shows a relatively narrow focus on specific project development objectives and intermediate results indicators and limited efforts by supervision teams to attempt to cover the broader indicators of the SRF generic results framework. Proposals 215. Generic results framework versus country-specific frameworks. Efforts by the SRF Administration Unit to streamline the generic results framework to achieve better coherence with data limitations at country level have been somewhat useful in helping to bridge the gap between the two results frameworks. This has been done with data limitations as the major consideration. A useful way to help bridge the gap between the two results frameworks is to view the SRF generic results framework as an effort to monitor a medium- to long-term, system-focused effort to sustainably develop the statistical capacity of countries, while specific project M&E mechanisms monitor time-bound, data-focused operations. Given that countries are at different stages of statistical capacity development, it is normal to expect that some country projects will 18 The OPM report on the evaluation of the Paris Declaration (OPM 2009) mentions a number of attempts to apply statistical capacity indicators, and notes concerns that finding indicators to include broader features of the statistical system is problematic. In the SRF generic results framework, indicators of statistical capacity other than the SCI itself have been deleted. 90 have more limited development objectives -- such as the production and accessibility of data and human resource development which are the common development objectives of all the country projects during the pilot phase -- while in countries such as Rwanda, the M&E mechanisms of SRF projects should basically cover all the elements of the generic results framework. While much more thinking is required to translate the above into operational practices, it is hoped that this more dynamic vision of the SRF generic results framework will make it easier for project teams and their counterparts in countries to conceive project M&E mechanisms for future SRF projects. 216. Greater efforts to obtain information on budgetary commitments and support. While the approach taken by the SRF Administration Unit in deleting all indicators related to funding and disbursement for statistics, except for disbursement under the SRF projects, may be justified on the basis of data unavailability, the impression of the Evaluation Team, based on the field mission to Nigeria and Rwanda, was that information on resources for statistics is available but does require special efforts to monitor. Without information on resources -- as is presently the case in the (modified) generic results framework -- there is the risk that the sustainability of support provided by the SRF projects may not be assured. The Evaluation Team therefore proposes the re-introduction of some measure of budgetary resource monitoring, while acknowledging that various challenges must be overcome including access to information on resources provided by development partners. 217. Going beyond the production of and accessibility to statistics and highlighting user- oriented awareness raising activities. The results frameworks of the pilot projects have a strong focus on the production of data and their accessibility. While the Evaluation Team appreciates the challenges in assessing the actual use of statistics, it would be useful for the SRF projects to have provisions for the active support of properly functioning institutional mechanisms which fully involve other government agencies outside the NSO as well as users of all sorts and LDPs. This would include, among other actions, revising the content of the new indicator of the generic results framework -- the number of government staff trained under the SRF -- which should be supplemented by ‘total number of persons trained’ to reflect training of users including journalists, researchers etc. In the same vein, emphasis should be put on the latter part of the indicator ‘by type of training’; it would at least be helpful to distinguish technical training in statistics from other types of training, which could relate to promoting the use of data or to management and institutional matters or even to change management. c) Project implementation and project ownership 218. A close examination of the replies to the survey shows that many respondents reported that the Bank’s administrative procedures had caused significant delays in SRF projects. That was particularly the case with procurement procedures and the issuance of non-objection clearance by the Bank, which were frequently described as slow and overly bureaucratic; oversight functions could usefully be simplified. Proposals 219. On World Bank procedures. The nomination of experienced and committed TTLs and local support staff was seen by respondents to the questionnaire survey as key to successful 91 project implementation, and the Evaluation Team was not always convinced that the Bank had sufficient capacity at country level to fully support the SRF process. More awareness of and training in SRF principles, norms and procedures was needed on the part of both Bank staff and government officials in the SRF project team. 220. Greater ownership by countries and the World Bank. While country ownership is vitally important and is at the core of the SRF approach, discussions19 on this issue in the broader context of statistical capacity building have highlighted the need for the project or program to be ‘owned’ by a broad range of stakeholders and not exclusively by the government statistics office. The different statistical priorities of the many relevant stakeholders mean that country ownership is difficult to establish. In the short run, meeting the most pressing needs in terms of the statistics have shared the focus on the government statistics office. A longer perspective must take into account the needs and priorities of other government offices and many users. 221. The questionnaire survey for this evaluation shows clearly that ‘ownership’ is most pronounced among NSOs and is much less evident among the other government agencies and the users or potential users of statistics. More positive results on data quality and accessibility and more importantly on dialogue between producers and users of statistics would go a long way in promoting ownership for local stakeholders. The Evaluation Team’s impressions indicate a wide range of interest and involvement by World Bank country teams in the SRF projects. Having as a prerequisite for country selection the degree of World Bank support to statistical capacity building in the country, as discussed earlier, could help in improving the situation. 222. Linkages with matching IDA credits. The review of country experiences by the Evaluation Team shows that there may well be a case for such a linkage where the IDA funds are meant to address high-level, non-technical concerns such as the results focus of government and a reform-oriented institutional environment, which are described as among the higher pillars of statistical capacity in the OPM evaluation report mentioned earlier, and which are presented there as necessary for the full success of other more technical capacity building initiatives which form lower pillars of statistical capacity in the OPM hierarchical framework. These latter more technical pillars will tend to be those which are focused on in an SRF project. Properly combined, the two could in principle be very effective. The experience of Ghana which has tried combining the SRF with IDA funds, however, showed the risk of this approach, which has caused a serious delay in the effectiveness of the SRF project. In the case of Nigeria, the preparation of a larger IDA-funded operation for statistical support following the SRF project appears to show the promise of potentially beneficial linkages. 19 See for instance OPM (2009). Evaluation of the Implementation of the Paris Declaration: Thematic Study - Support to Statistical Capacity Building. Synthesis Report. 92 Appendix to Chapter V Overarching criteria  Current level of statistical capacity and development  Extent of commitment to SRF principles Other criteria  NSDS-related  Quality NSDS in place [or being prepared].  NSDS has clear implementation priorities.  NSDS has high-level political endorsement.  NSDS takes account of needs of main data producers and main data users.  NSDS has widespread country support.  NSDS has linkage to existing statistical initiatives.  Formal request to WB for NSDS implementation using a sector-wide approach.  Background conditions  Identification of national resource requirements.  Strong leadership of the national statistical system.  Agreement on medium to long-term approach to statistical capacity building.  Institutional reform, human development issues, problems of coordination and technical concerns all taken into account.  Partnership issues  National partnership for statistics in place or agreement on its establishment.  Recognition of need to support and strengthen user-producer dialogue.  Start of dialogue with LDPs.  Agreement on need for donor coordination and some mechanisms in place to support it.  Identification of lead donor.  Project support related  Potential to put IDS rapidly in place.  Timetable for project preparation takes account of ongoing statistical programs.  Existence of strong and committed Bank team in country office.  Firm commitment from World Bank to appoint a TTL. 93 ANNEX 1: Survey questionnaire for SRF-CF stakeholders Introduction 1. The World Bank is conducting an independent evaluation of the Pilot Phase of the Statistics for Results Facility Catalytic Fund (SRF-CF). The main purpose of the evaluation is to assess the experience of the programs in the pilot countries, to identify achievements to date, and to recommend any changes in policies and practices that may be necessary in order to improve results. An evaluation of the pilot countries and recommendations for modifications to the approach of the SRF would provide the basis for extending the role of the Catalytic Fund in providing support for statistical capacity building to other countries. MM. Brian Ngo and Andrew Flatt have been tasked to undertake this evaluation. As part of this exercise, they will be visiting Nigeria and Rwanda for more detailed assessments of the SRF-CF. 2. The pilot phase of the SRF-CF, which began in 2009, was set up to provide financial and technical support to a limited number of pilot countries that either were endowed with a weak level of statistical capacity or had recently emerged from conflict. Originally, five countries that were invited by the World Bank to submit applications for assistance during the pilot phase, namely Afghanistan, the Democratic Republic of Congo (DRC), Ghana, Nigeria, and Rwanda -- all IDA-eligible countries. Two of the countries – Afghanistan and DRC – were included on the basis of their status of fragile, post- conflict states which posed particular problems relating to the development of statistical capacity building. It was subsequently decided to expand the pilot phase to three more countries -- Lao PDR, Senegal and Ethiopia -- and to extend funding for testing the use of innovative approaches to data collection through 'Listening to Africa' project, a project aimed at piloting the use of cell phones to gather high frequency household level data in three African countries, Cameroon, Malawi, and Senegal. 3. Generally speaking, countries that apply for SRF support must have either met or made plans to develop a number of criteria most of which having been identified as important features of aid effectiveness as covered by the Paris Declaration on Aid Effectiveness. These criteria call for: a. The preparation or implementation of a statistical strategy for statistical development such as an NSDS; b. Statistical development uses a system-wide approach which supports a locally-owned and locally-led program in a coordinated way and moving towards the use of country systems and procedures for implementation; c. Budgetary resource allocations for statistics, preferably on a multi-year basis; d. A broad agreement to the SRF principles by all or at least a core set of local donors and the identification of a lead donor. 4. As of May 2013, three projects in Afghanistan, Nigeria and Rwanda are under implementation. The Ghanaian project has been awaiting approval by the government. The Lao PDR project became effective in July or two months after this evaluation began and is ready for implementation, DRC is about to be negotiated, while the remaining three projects are at different stages of preparation. Funding for most projects is in the range of about US$10 million. 5. The terms of reference (TORs) for the evaluation of the SRF-CF pilot phase were discussed and agreed with donors in the spring of 2013. According to the agreed TORs, the 94 evaluation will assess the experience of the pilot projects to identify their achievements and impact and to recommend changes in policies and practices to improve results. More specifically, the objectives of the evaluation are as follows: a) How successful the SRF-CF has achieved its objectives particularly in: i. Promoting a system-wide approach in statistics at country level; ii. Increasing resources for implementing country owned National Statistical Plans; iii. Linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks and promoting an improved national dialogue and partnership between statistics users and statistical producers; and iv. Delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement, through better coordination and alignment to agreed National Statistical Plans and through better alignment to the Paris Declaration on Aid Effectiveness; b) The impact of the in-country donor statisticians in helping countries reach the SRF objectives; c) How the governance arrangements and operational practices and procedures are working; and d) To recommend what changes may be needed to enable the SRF-CF to reach its objectives. 6. The evaluation questions will be answered through a combination of methodologies including field visits to two countries -- Nigeria and Rwanda --, desk reviews of various World Bank reports and working documents from pilot countries, and extensive use of questionnaires covering the questions and issues detailed above to a broad range of relevant stakeholders including:  World Bank task team leaders (TTLs)  Primary recipient agencies (National statistical offices)  Other recipient agencies within governments  Other recipients outside of government  Local donor agencies  In-country donor statisticians  National poverty reduction or National MDG coordination (and M&E) units  Other users (media, chambers of commerce). 95 Master Questionnaire Evaluation of the Pilot Phase of the Statistics for Results Facility Catalytic Fund (SRF-CF) The World Bank has commissioned an independent evaluation of the Pilot Phase of the Statistics for Results Facility Catalytic Fund (SRF-CF). The main purpose of the evaluation is to assess the experience of the programs in the pilot countries, to identify achievements to date, and to recommend any changes in policies and practices that may be necessary in order to improve results. The Evaluation Team (Brian Ngo and Andrew Flatt) would like to seek your opinion on the following aspects of the SRF-CF. The questionnaire is arranged according to major objectives of the SRF-CF and key issues surrounding the SRF approach. Although many of the questions are multiple choice, the Evaluation Team would very much welcome additional comments and insights on any of the issues covered in the questionnaire. Your response to the questionnaire will remain strictly confidential, and the results of the survey will only be used in an aggregated form. Please return the completed questionnaire to Andrew Flatt (andrewjflatt@gmail.com) by [date to be decided]. If you are unable to transmit the completed questionnaire by email, please send the hard copy to [address in the World Bank office, to be decided]. I. Promoting a system-wide approach to statistics at the country level (A system-wide approach is defined broadly as the country having a clear nationally owned strategy for statistics with arrangements for resources, broad consultation mechanisms that involve all stakeholders, mechanisms for dialogue, coordination, and monitoring and evaluation) 1A. how inclusive has the approach been to formulating the national strategy for the development of statistics (NSDS) in [name of country]? [ ] Very inclusive [ ] Somewhat inclusive [ ] Barely inclusive or not inclusive at all [ ] I’m not sure 1B. to what extent do you think that development partners with an interest in supporting statistics have agreed with the development of a system-wide approach in [name of country]? [ ] Very much agreed [ ] Somewhat agreed [ ] Have not agreed at all [ ] I’m not sure 1C. To what extent is your agency or yourself in your professional capacity, involved and able to support the development of a system-wide approach in [name of country]? [ ] To a very great extent [ ] To some extent [ ] To a minimal extent or not at all Please identify any difficulties that your agency might face in being involved in or supporting the development of a system-wide approach: 1D. How large a challenge do you consider the system-wide approach promoted by the SRF project to be for a country like [name of country]? [ ] A great challenge [ ] Somewhat of a challenge [ ] Not a significant challenge at all [ ] I’m not sure Please identify some of the challenges: 96 1E. Overall, how successful do you consider the SRF project to be in promoting a system-wide approach to statistics in [name of country]? [ ] Very successful [ ] Somewhat successful [ ] Barely successful or not successful at all [ ] Promises to be successful [ ] Too early to judge Please share with us any further comments on this topic: II. Increasing resources for implementing country-owned national statistical plans or strategies 2A. Have domestic resources for statistics been increased since [name of country] decided to become an SRF pilot country? Or are there definite plans for an increase? [ ] Domestic resources have already been increased [ ] No increase yet, but there are definite plans for an increase [ ] There has been no increase yet and there are no plans for an increase [ ] I’m not sure about the domestic resource situation 2B. If there is an actual or planned increase, please give the approximate magnitude of such an increase: [ ] significant [ ] moderate [ ] indicate percentage increase if available To what extent, in your view, can this increase be attributed to the existence of the SRF? [ ] Directly attributable [ ] Indirectly attributable [ ] The increase would have occurred even without the SRF [ ] I’m not sure 2C. Have development partners increased resources for statistics since [name of country] decided to become an SRF pilot country? Or are there definite plans for an increase? [ ] Development partners have already increased resources [ ] No increase yet, but development partners have definite plans to increase resources [ ] There has been no increase yet and there are no plans for an increase [ ] I’m not sure about the resource situation of development partners 2D. If there is an actual or planned increase, please give the approximate magnitude of such an increase: [ ] significant [ ] moderate [ ] indicate percentage increase if available To what extent, in your view, can this increase be attributed to the existence of the SRF? [ ] Directly attributable [ ] Indirectly attributable [ ] The increase would have occurred even without the SRF [ ] I’m not sure 2E. Has your agency increased resources for statistics since [name of country] decided to become an SRF pilot country? Or are there definite plans for an increase? [ ] Resources have already been increased [ ] No increase yet, but there are definite plans for an increase [ ] There has been no increase yet and there are no plans for an increase 2F. If there is an actual or planned increase, please give the approximate magnitude of such an increase: [ ] significant [ ] moderate [ ] indicate percentage increase if available To what extent, in your view, can this increase be attributed to the existence of the SRF? [ ] Directly attributable [ ] Indirectly attributable [ ] The increase would have occurred even without the SRF 2G. Do you think that funds from the SRF should be seen as core finance for implementing the national statistical plan or statistical development strategy (NSDS)? [ ] Yes [ ] Maybe [ ] No Please elaborate: 97 2H. What do you consider to be the optimal relationship between the SRF-CF and IDA funding? 2I. Overall, how successful do you consider the SRF-CF to be in increasing resources for implementing the national statistical plan or statistical development strategy in [name of country]? [ ] Very successful [ ] Somewhat successful [ ] Barely successful or not successful at all [ ] Promises to be successful [ ] Too early to judge Please share with us any further comments on this topic: III. Linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks 3A. To what extent do you consider that monitoring the achievement of the Millennium Development Goals and/or poverty reduction targets has been incorporated into the national monitoring framework of [name of country]? [ ] Entirely or almost entirely incorporated [ ] Somewhat incorporated [ ] Hardly incorporated or not incorporated at all [ ] I’m not sure 3B. What role (if any) has the SRF played to date in enhancing the production, quality and timeliness of data in [name of country]? [ ] A very significant role [ ] Some role [ ] Hardly any or no role to date [ ] I’m not sure Please give specific examples of any enhancements where possible: 3C. What role (if any) has the SRF played to date in enhancing the accessibility and usability of data in [name of country]? [ ] A very significant role [ ] Some role [ ] Hardly any or no role to date [ ] I’m not sure Please give specific examples of any enhancements where possible: 3D. Overall, how successful do you consider the SRF to be in linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks in [name of country]? [ ] Very successful [ ] Somewhat successful [ ] Barely successful or not successful at all [ ] Promises to be successful [ ] Too early to judge Please share with us any further comments on this topic: IV. Promoting an improved national dialogue and partnership between users and producers of statistics 4A. Do you consider that existing institutional mechanisms in the field of statistics in [name of country], such as [specify mechanism(s) if possible], function well? [ ] Yes [ ] To some extent [ ] No [ ] I’m not sure 4B. Has a user satisfaction survey been undertaken, or is one planned, in [name of country]? [ ] Yes, a survey has already been undertaken [ ] A survey is being planned [ ] There is no such survey planned [ ] Not sure what the situation is with regard to a user satisfaction survey Please describe any experience you have had with a user satisfaction survey, and any action that has been undertaken as a direct result of it: 98 4C. Have there been any efforts under the SRF to enhance the capacity of data users (such as the training of local journalists in the use of statistics) in [name of country], or are any such efforts planned? [ ] Yes, there have been efforts to enhance the capacity of data users [ ] No efforts yet, but some are planned [ ] There are no such efforts in place or planned [ ] Not sure what the situation is Please describe any efforts to enhance the capacity of data users in [name of country]: 4D. Overall, how successful do you consider the SRF to be in promoting an improved national dialogue and partnership between users and producers of statistics in [name of country]? [ ] Very successful [ ] Somewhat successful [ ] Barely successful or not successful at all [ ] Promises to be successful [ ] Too early to judge Please share with us any further comments on this topic: V. Delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement, through better coordination and alignment to agreed national statistical plans and through better alignment to the Paris Declaration on Aid Effectiveness 5A. Has the SRF had an impact, or do you think it has the potential to make an impact, on aligning support for statistical capacity building to the national statistical plan or NSDS in [name of country]? [ ] Has already had a positive impact on aligning support [ ] Has the potential to make an impact [ ] Has not had any positive impact and/or does not appear to have the potential to do so [ ] I’m not sure 5B. Has the SRF approach led, or do you think it has the potential to lead, to better donor coordination of statistical capacity building in [name of country]? [ ] Has already led to better donor coordination of statistical capacity building [ ] Has the potential to lead to better donor coordination [ ] Has not led to better donor coordination and/or does not appear to have the potential to do so [ ] I’m not sure 5C. Overall, how successful do you consider the SRF to be in delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement in [name of country]? [ ] Very successful [ ] Somewhat successful [ ] Barely successful or not successful at all [ ] Promises to be successful [ ] Too early to judge Please share with us any further comments on this topic: VI. Impact of the in-country donor statistician in helping to achieve SRF objectives (one of the particularities of the SRF approach relative to other capacity building efforts is the nomination of an in-country statistician financed with external resources to facilitate the system-wide approach, coordination etc.). [These questions will not be asked where no in-country donor statistician is present] 6A. What are your views on the usefulness of the in-country donor statistician ([name of statistician])? Does the fact that the statistician represents a major donor have an impact on the statistician’s effectiveness? 99 6B. What have been the contributions of the in-country donor statistician in helping to achieve the objectives of the SRF as set out in previous questions, such as the development of a statistical development strategy, donor coordination and engaging various stakeholders to promote the demand for statistics? Has the balance of the statistician’s contributions to these various objectives been appropriate, in your view? 6C. Are there any actual or potential areas of friction or contention surrounding the role of the in-country donor statistician in relation to the national statistical system and its staff? 6D. Assuming that the post of in-country donor statistician is not a permanent one, do you see the need for an exit or disengagement strategy to cover the eventual withdrawal of the post? 6E. Please share with us any further comments on this topic: VII. Broad issues related to the SRF Charter 7A. How reasonable or realistic do you think the expectations for the SRF are in terms of its impact and its timing? [ ] Very reasonable [ ] Somewhat reasonable [ ] Unrealistic [ ] I’m not sure What have been in your view the most significant impacts of the SRF in [name of country]? 7B. To what extent do you think the choice of countries invited to apply for the SRF, particularly post- conflict countries with weak statistical capacity, has affected the impact of the SRF approach? [ ] To a great extent [ ] To some extent [ ] To an insignificant extent [ ] I’m not sure 7C. Do you consider that the SRF, hitherto comprising almost exclusively country-specific projects, should be extended to regional initiatives and operations? [ ] Yes [ ] Perhaps [ ] No If yes or perhaps, which sort of projects might benefit most from a regional approach? 7D. Please share with us any further comments on issues related to the SRF Charter: VIII. SRF-CF governance arrangements and operational practices and procedures 8A. How smooth and timely was, or has been to date, the SRF grant approval and disbursement process? [ ] Smooth and timely [ ] Somewhat problematic [ ] Beset by difficulties and delays Please describe any particular facilitating or delaying factors in the process: 8B. What has been your experience in implementing the SRF grant, in terms of reporting arrangements, support from World Bank staff and so on? [ ] Very positive [ ] Somewhat positive [ ] Generally negative [ ] I don’t have enough experience to judge 8C. What are the main sorts of problems that you have been facing in implementing the SRF project? (For example, inadequacy of government budgetary support; lack of coordination among government agencies 100 and/or among development partners; failure to translate government commitment into practical action at the operational level; etc…) 8D. How large a challenge do you consider that recipient-country execution of the SRF project has been or will be in [name of country]? [ ] A great challenge [ ] Somewhat of a challenge [ ] Not a significant challenge [ ] I’m not sure 8E. Do you consider that SRF projects receive adequate attention by line management in the World Bank? [ ] Yes [ ] To some extent [ ] No [ ] I’m not sure 8F. Would there be an advantage to the SRF grant being part of a larger Bank operation rather than being a stand-alone project as is currently the case? [ ] Yes [ ] Perhaps [ ] No 8G. Are you familiar with the SRF standardized [generic] Results Framework? [ ] Very familiar [ ] Somewhat familiar [ ] Not familiar at all 8H. What has been your experience with the SRF standardized [generic] Results Framework? [ ] Very positive [ ] Somewhat positive [ ] Generally negative [ ] I don’t have enough experience to judge Please describe any suggestions you have for improving the Framework: 8I. Please share with us any further comments on the SRF governance arrangements and operational practices and procedures: IX. General questions 9A. To what extent do you consider that the concept of the SRF meets the needs for statistical capacity building in [name of country]? [ ] To a great extent [ ] To some extent [ ] To an insignificant extent [ ] I’m not sure 9B. Please describe the main motivation for your Government’s accepting the World Bank’s invitation to apply for a grant from the Statistics for Results Facility Catalytic Fund. What were, or are, the major needs that you wish to address? 9C. To what extent do you consider that the SRF has facilitated, or has the potential to facilitate, effective ownership and leadership by the government of the development of the national statistical system of [name of country]? [ ] To a great extent [ ] To some extent [ ] To an insignificant extent [ ] I’m not sure 9D. Does implementation of the SRF project, currently or potentially, involve national authorities taking reform measures that would not otherwise have been undertaken? [ ] Yes, to a significant extent [ ] To some extent [ ] No, or probably not [ ] I’m not sure 101 9E. To what extent do you consider that the SRF has facilitated, or has the potential to facilitate, the greater use of statistical data and indicators to support evidence-based decision making in [name of country]? [ ] To a great extent [ ] To some extent [ ] To an insignificant extent [ ] I’m not sure 10A. Please share with us any further comments on any aspect of the SRF: ** Thank you very much indeed for your cooperation and assistance in responding to this questionnaire. Please return it to Andrew Flatt (andrewjflatt@gmail.com) by [date to be decided]. If you are unable to transmit the completed questionnaire by email, please send the hard copy to [address in the World Bank office, to be decided]. 102 ANNEX 2: Tabulations from the survey of SRF-CF stakeholders This annex contains tabulations of the responses from the survey of SRF stakeholders undertaken from July to early September 2013. Descriptions of the questionnaire and its administration, along with response rates to the survey, are given in Section II of the report. Shorter versions of selected tables appear throughout Section III. The master questionnaire utilized is shown in Annex 1. The stakeholder group abbreviations used in the tabulations are as follows: TTL Task Team Leaders NSO Primary recipient agencies (usually the national statistical office) OGA Other government agencies ONU Other national users IDS In-country donor statisticians LDP Local development partners 103 Table II.1 Inclusiveness of the NSDS formulation approach 1A. How inclusive has the approach been to formulating the national strategy Barely for the development of statistics (NSDS) Somewhat inclusive or not No in [name of country]? Very inclusive inclusive inclusive at all I'm not sure answer Number of responses Implementing countries 26.5 21 4 5.5 0 Pipeline countries 32 14 1 6 0 All SRF countries 58.5 35 5 11.5 0 Percentage distributions % % % % % Afghanistan 37 42 16 5 0 Nigeria 55 35 5 5 0 Rwanda 47 33 0 19 0 Implementing countries 46 37 7 10 0 DR Congo 50 50 0 0 0 Ethiopia 23 46 0 31 0 Ghana 80 0 0 20 0 Lao PDR 57 36 7 0 0 Senegal 100 0 0 0 0 Pipeline countries 60 26 2 11 0 All SRF countries 53 32 5 10 0 Stakeholder groups: % % % % % TTL 75 25 0 0 0 NSO 74 22 2 1 0 OGA 42 42 11 5 0 ONU 20 80 0 0 0 IDS .. .. .. .. .. LDP 34 31 7 28 0 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 104 Table II.2 Perceived agreement of development partners with a system-wide approach 1B. To what extent do you think that development partners with an interest in supporting statistics have agreed with the development Very of a system-wide approach in much Somewhat I'm not No [name of country]? agreed agreed Have not agreed at all sure answer Number of responses Implementing countries 26 10 0 1 3 Pipeline countries 25 12 0 4 0 All SRF countries 51 22 0 5 3 Percentage distributions % % % % % Afghanistan 38 62 0 0 0 Nigeria 88 12 0 0 0 Rwanda 60 0 0 10 30 Implementing countries 65 25 0 3 8 DR Congo 50 33 0 17 0 Ethiopia 29 57 0 14 0 Ghana 67 33 0 0 0 Lao PDR 75 17 0 8 0 Senegal 71 14 0 14 0 Pipeline countries 61 29 0 10 0 All SRF countries 63 27 0 6 4 Stakeholder groups: % % % % % TTL 50 42 0 8 0 NSO 68 22 0 5 5 OGA 58 26 0 11 5 ONU 80 20 0 0 0 IDS .. .. .. .. .. Notes: This question was not posed to the LDP group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 105 Table II.3 Involvement and support of development partners in a system-wide approach 1C. To what extent is your agency or yourself in your professional capacity, involved and able to To a support the development of a very system-wide approach in [name of great To a minimal extent or country]? extent To some extent not at all No answer Number of responses Implementing countries 8 5 2 2 Pipeline countries 3 7 1 1 All SRF countries 11 12 3 3 Percentage distributions % % % % Afghanistan 33 17 17 33 Nigeria .. .. .. .. Rwanda 38 50 13 0 Implementing countries 47 29 12 12 DR Congo Ethiopia 17 67 17 0 Ghana .. .. .. .. Lao PDR .. .. .. .. Senegal 67 33 0 0 Pipeline countries 25 58 8 8 All SRF countries 38 41 10 10 Stakeholder groups: % % % % LDP 38 41 10 10 Notes: This question was posed to the LDP group only. Responses from some countries have been suppressed. Source: SRF Evaluation Survey 2013 106 Table II.4 Challenge of the system-wide approach 1D. How large a challenge do you consider the system-wide approach promoted by the I'm SRF to be for a country like [name of Somewhat of Not a significant not No country]? A great challenge a challenge challenge at all sure answer Number of responses Implementing countries 16 26 11 3 1 Pipeline countries 21 23 5 2 2 All SRF countries 37 49 16 5 3 Percentage distributions % % % % % Afghanistan 32 42 26 0 0 Nigeria 45 45 0 5 5 Rwanda 6 50 33 11 0 Implementing countries 28 46 19 5 2 DR Congo 100 0 0 0 0 Ethiopia 23 46 15 8 8 Ghana 20 50 30 0 0 Lao PDR 43 50 0 7 0 Senegal 40 50 0 0 10 Pipeline countries 40 43 9 4 4 All SRF countries 34 45 15 5 3 Stakeholder groups: % % % % % TTL 58 33 8 0 0 NSO 32 39 20 2 7 OGA 47 42 0 11 0 ONU 0 60 40 0 0 IDS .. .. .. .. .. LDP 21 55 17 7 0 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 107 Table II.5 Success of the SRF-CF in promoting a system-wide approach Barely 1E. Overall, how successful do you successful Too consider the SRF-CF to be in or not Promises early promoting a system-wide approach to Very Somewhat successful to be to No Total statistics in [name of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 19 22 4 5 7 0 57 Pipeline countries 7 8 1 12 23 2 53 All SRF countries 26 30 5 17 30 2 110 Percentage distributions % % % % % % Afghanistan 26 42 11 5 16 0 19 Nigeria 25 45 5 10 15 0 20 Rwanda 50 28 6 11 6 0 18 Implementing countries 33 39 7 9 12 0 DR Congo 0 17 0 17 67 0 6 Ethiopia 8 23 0 31 31 8 13 Ghana 0 10 0 40 50 0 10 Lao PDR 36 14 7 14 29 0 14 Senegal 10 10 0 10 60 10 10 Pipeline countries 13 15 2 23 43 4 All SRF countries 24 27 5 15 27 2 Stakeholder groups: % % % % % % TTL 25 17 0 0 58 0 12 NSO 32 22 2 22 22 0 41 OGA 26 32 11 16 16 0 19 ONU 20 60 0 20 0 0 5 IDS .. .. .. .. .. .. 4 LDP 10 31 7 14 31 7 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 108 Table II.6 Increase in domestic resources Domestic No increase I'm not 2A. Have domestic resources for resources yet, but There has been sure about statistics been increased since [name of have there are no increase yet the country] decided to become an SRF already definite and there are domestic pilot country? Or are there definite been plans for an no plans for an resource No Total plans for an increase? increased increase increase situation answer responses Number of responses Implementing countries 19 10 6 17 1 53 Pipeline countries 22 13 2 14 1 52 All SRF countries 41 23 8 31 2 105 Percentage distributions % % % % % Afghanistan 11 21 21 47 0 19 Nigeria 44 22 11 17 6 18 Rwanda 56 13 0 31 0 16 Implementing countries 36 19 11 32 2 DR Congo 50 33 0 17 0 6 Ethiopia 17 8 0 67 8 12 Ghana 30 30 20 20 0 10 Lao PDR 43 43 0 14 0 14 Senegal 80 10 0 10 0 10 Pipeline countries 42 25 4 27 2 All SRF countries 39 22 8 30 2 Stakeholder groups: % % % % % TTL 75 8 0 17 0 12 NSO 34 34 2 27 2 41 OGA 37 32 0 32 0 19 IDS .. .. .. .. .. 4 LDP 31 7 17 41 3 29 Notes: This question was not posed to the ONU group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 109 Table II.7 Magnitude of increase in domestic resources 2B1. If there is an actual or planned increase (in domestic resources), please give the approximate No Total magnitude of such an increase Significant Moderate answer responses Number of responses Implementing countries 13 15 25 53 Pipeline countries 18 12 22 52 All SRF countries 31 27 47 105 Percentage distributions % % % Afghanistan 11 47 42 19 Nigeria 33 22 44 18 Rwanda 31 13 56 16 Implementing countries 25 28 47 DR Congo 50 33 17 6 Ethiopia 17 8 75 12 Ghana 30 20 50 10 Lao PDR 29 36 36 14 Senegal 60 20 20 10 Pipeline countries 35 23 42 All SRF countries 30 26 45 Stakeholder groups: % % % TTL 67 17 17 12 NSO 41 27 32 41 OGA 11 42 47 19 IDS .. .. .. 4 LDP 7 21 72 29 Notes: This question was not posed to the ONU group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 110 Table II.8 Attribution of increase in domestic resources to the SRF The increase 2B2. To what extent, in your view, would have can this increase (in domestic occurred even I'm resources) be attributed to the Directly Indirectly without the not No Total existence of the SRF? attributable attributable SRF sure answer responses Number of responses Implementing countries 5 12 4 12 20 53 Pipeline countries 5.5 6.5 7 14 19 52 All SRF countries 10.5 18.5 11 26 39 105 Percentage distributions % % % % % Afghanistan 5 37 0 5 53 19 Nigeria 17 11 0 22 50 18 Rwanda 6 19 25 44 6 16 Implementing countries 9 23 8 23 38 DR Congo 25 25 0 33 17 6 Ethiopia 0 0 8 42 50 12 Ghana 20 10 0 20 50 10 Lao PDR 7 29 14 21 29 14 Senegal 10 0 40 20 30 10 Pipeline countries 11 13 13 27 37 All SRF countries 10 18 10 25 37 Stakeholder groups: % % % % % TTL 13 38 17 0 33 12 NSO 12 20 7 20 41 41 OGA 11 26 0 32 32 19 IDS .. .. .. .. .. 4 LDP 3 0 21 41 34 29 Notes: This question was not posed to the ONU group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 111 Table II.9 Increase in development partner resources (external perspective) No increase There has 2C. Have development partners yet, but been no I'm not sure increased resources for statistics Development development increase yet about the since [name of country] decided partners have partners have and there resource to become an SRF pilot already definite plans are no plans situation of country? Or are there definite increased to increase for an development No Total plans for an increase? resources resources increase partners answer responses Number of responses Implementing countries 15 10 0 10 5 40 Pipeline countries 12 13 5 11 0 41 All SRF countries 27 23 5 21 5 81 Percentage distributions % % % % % Afghanistan 38 31 0 23 8 13 Nigeria 35 35 0 24 6 17 Rwanda 40 0 0 30 30 10 Implementing countries 38 25 0 25 13 DR Congo 17 33 33 17 0 6 Ethiopia 29 29 14 29 0 7 Ghana 56 22 11 11 0 9 Lao PDR 8 50 0 42 0 12 Senegal 43 14 14 29 0 7 Pipeline countries 29 32 12 27 0 All SRF countries 33 28 6 26 6 Stakeholder groups: % % % % % TTL 50 25 25 0 0 12 NSO 41 32 0 17 10 41 OGA 11 32 5 47 5 19 ONU 0 20 0 80 0 5 IDS .. .. .. .. .. 4 Notes: This question was not posed to the LDP group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 112 Table II.10 Magnitude of increase in development partner resources (external perspective) 2D1. If there is an actual or planned increase (in development partner resources), please give the No Total approximate magnitude of such an increase Significant Moderate answer responses Number of responses Implementing countries 9 16 15 40 Pipeline countries 12 11 18 41 All SRF countries 21 27 33 81 Percentage distributions % % % Afghanistan 15 62 23 13 Nigeria 24 41 35 17 Rwanda 30 10 60 10 Implementing countries 23 40 38 DR Congo 17 33 50 6 Ethiopia 29 29 43 7 Ghana 44 22 33 9 Lao PDR 25 25 50 12 Senegal 29 29 43 7 Pipeline countries 29 27 44 All SRF countries 26 33 41 Stakeholder groups: % % % TTL 17 50 33 12 NSO 37 29 34 41 OGA 11 32 58 19 ONU 0 60 40 5 IDS .. .. .. 4 Notes: This question was not posed to the LDP group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 113 Table II.11 Attribution of increase in development partner resources to the SRF (external perspective) 2D2. To what extent, in your view, The increase can this increase (in development would have I'm partner resources) be attributed to the Directly Indirectly occurred even not No Total existence of the SRF? attributable attributable without the SRF sure answer responses Number of responses Implementing countries 11 12 0 6 11 40 Pipeline countries 5.5 5.5 2 11 17 41 All SRF countries 16.5 17.5 2 17 28 81 Percentage distributions % % % % % Afghanistan 23 54 0 8 15 13 Nigeria 35 18 0 18 29 17 Rwanda 20 20 0 20 40 10 Implementing countries 28 30 0 15 28 DR Congo 17 0 0 33 50 6 Ethiopia 14 14 0 29 43 7 Ghana 22 11 0 33 33 9 Lao PDR 8 25 0 17 50 12 Senegal 7 7 29 29 29 7 Pipeline countries 13 13 5 27 41 All SRF countries 20 22 2 21 35 Stakeholder groups: % % % % % TTL 13 46 8 8 25 12 NSO 20 22 2 17 39 41 OGA 21 11 0 32 37 19 ONU 40 0 0 60 0 5 IDS .. .. .. .. .. 4 Notes: This question was not posed to the LDP group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 114 Table II.12 Increase in development partner resources (internal perspective) 2E. Has your agency increased resources for statistics since [name of country] Resources No increase yet, There has been no decided to become an SRF pilot country? have already but there are increase yet and Or are there definite plans for an been definite plans for there are no plans No Total increase? increased an increase for an increase answer responses Number of responses Implementing countries 6 2 2 7 17 Pipeline countries 3 1 4 4 12 All SRF countries 9 3 6 11 29 Percentage distributions % % % % Afghanistan 17 0 17 67 6 Nigeria 67 33 0 0 3 Rwanda 38 13 13 38 8 Implementing countries 35 12 12 41 DR Congo 0 Ethiopia 0 17 33 50 6 Ghana .. .. .. .. 1 Lao PDR .. .. .. .. 2 Senegal 67 0 0 33 3 Pipeline countries 25 8 33 33 All SRF countries 31 10 21 38 Stakeholder groups: % % % % LDP 31 10 21 38 Notes: This question was posed to the LDP group only. Responses from some countries have been suppressed. Source: SRF Evaluation Survey 2013 115 Table II.13 Magnitude of increase in development partner resources (internal perspective) 2F1. If there is an actual or planned increase (in your agency's resources), please give the approximate No Total magnitude of such an increase Significant Moderate answer responses Number of responses Implementing countries 1 4 12 17 Pipeline countries 2 1 9 12 All SRF countries 3 5 21 29 Percentage distributions % % % Implementing countries 6 24 71 Pipeline countries 17 8 75 All SRF countries 10 17 72 Stakeholder groups: LDP 10 17 72 Notes: This question was posed to the LDP group only. The number of responses at country level is considered too small to be presented. Source: SRF Evaluation Survey 2013 116 Table II.14 Attribution of increase in development partner resources to the SRF (internal perspective) The increase 2F2. To what extent, in your view, can would have this increase (in your agency's resources) Directly Indirectly occurred even No Total be attributed to the existence of the SRF? attributable attributable without the SRF answer responses Number of responses Implementing countries 1 0 5 11 17 Pipeline countries 1 0 3 8 12 All SRF countries 2 0 8 19 29 Percentage distributions % % % % Implementing countries 6 0 29 65 Pipeline countries 8 0 25 67 All SRF countries 7 0 28 66 Stakeholder groups: LDP 7 0 28 66 Notes: This question was posed to the LDP group only. The number of responses at country level is considered too small to be presented. Source: SRF Evaluation Survey 2013 117 Table II.15 The SRF as possible core finance for implementing the NSDS 2G. Do you think that funds from the SRF should be seen as core finance for implementing the national No Total statistical development strategy? Yes Maybe No answer responses Number of responses Implementing countries 34 11 10 2 57 Pipeline countries 26 12 15 0 53 All SRF countries 60 23 25 2 110 Percentage distributions % % % % Afghanistan 79 21 0 0 19 Nigeria 35 20 40 5 20 Rwanda 67 17 11 6 18 Implementing countries 60 19 18 4 DR Congo 67 17 17 0 6 Ethiopia 46 31 23 0 13 Ghana 20 10 70 0 10 Lao PDR 71 21 7 0 14 Senegal 40 30 30 0 10 Pipeline countries 49 23 28 0 All SRF countries 55 21 23 2 Stakeholder groups: % % % % TTL 50 25 25 0 12 NSO 59 12 29 0 41 OGA 58 21 16 5 19 ONU 20 20 60 0 5 IDS .. .. .. .. 4 LDP 55 31 10 3 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 118 Table II.16 Success of the SRF-CF in increasing resources for implementing national statistical development strategies 2I Overall, how successful do you consider the SRF-CF to be in Barely increasing resources for successful Too implementing the national or not Promises early statistical development strategy in Very Somewhat successful to be to No Total [name of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 22 19 2 8 6 0 57 Pipeline countries 9 5 0 12 25 2 53 All SRF countries 31 24 2 20 31 2 110 Percentage distributions % % % % % % Afghanistan 32 58 0 11 0 0 19 Nigeria 35 30 5 25 5 0 20 Rwanda 50 11 6 6 28 0 18 Implementing countries 39 33 4 14 11 0 DR Congo 0 17 0 17 67 0 6 Ethiopia 23 8 0 23 46 0 13 Ghana 10 0 0 40 50 0 10 Lao PDR 29 21 0 21 29 0 14 Senegal 10 0 0 10 60 20 10 Pipeline countries 17 9 0 23 47 4 All SRF countries 28 22 2 18 28 2 Stakeholder groups: % % % % % % TTL 33 17 0 8 42 0 12 NSO 29 27 0 22 22 0 41 OGA 42 11 5 26 11 5 19 ONU 20 20 0 40 20 0 5 IDS .. .. .. .. .. .. 4 LDP 17 24 3 10 41 3 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 119 Table II.17 Incorporation of MDG and poverty reduction target monitoring into national monitoring frameworks 3A To what extent do you consider that monitoring the achievement of the Hardly Millennium Development Goals and/or Entirely or incorporated poverty reduction targets has been almost or not I'm incorporated into the national monitoring entirely Somewhat incorporated not No Total framework of [name of country]? incorporated incorporated at all sure answer responses Number of responses Implementing countries 26 21 5 4 1 57 Pipeline countries 30 14.5 3 5.5 0 53 All SRF countries 56 35.5 8 9.5 1 110 Percentage distributions % % % % % Afghanistan 16 63 16 5 0 19 Nigeria 30 40 10 15 5 20 Rwanda 94 6 0 0 0 18 Implementing countries 46 37 9 7 2 DR Congo 50 25 17 8 0 6 Ethiopia 54 38 0 8 0 13 Ghana 80 10 0 10 0 10 Lao PDR 43 36 7 14 0 14 Senegal 60 20 10 10 0 10 Pipeline countries 57 27 6 10 0 All SRF countries 51 32 7 9 1 Stakeholder groups: % % % % % TTL 50 29 17 4 0 12 NSO 59 32 0 7 2 41 OGA 32 32 16 21 0 19 ONU 40 60 0 0 0 5 IDS .. .. .. .. .. 4 LDP 59 24 10 7 0 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 120 Table II.18 SRF role in enhancing the production, quality and timeliness of data 3B What role (if any) has the SRF played to date in enhancing the A very Hardly any I'm production, quality and timeliness of significant Some or no role not No Total data in [name of country]? role role to date sure answer responses Number of responses Implementing countries 29 16 5.5 6.5 0 57 Pipeline countries 10 6 11 19 7 53 All SRF countries 39 22 16.5 25.5 7 110 Percentage distributions % % % % % Afghanistan 21 32 29 18 0 19 Nigeria 70 25 0 5 0 20 Rwanda 61 28 0 11 0 18 Implementing countries 51 28 10 11 0 DR Congo 0 0 50 33 17 6 Ethiopia 31 8 15 31 15 13 Ghana 20 10 40 20 10 10 Lao PDR 14 29 7 43 7 14 Senegal 20 0 10 50 20 10 Pipeline countries 19 11 21 36 13 All SRF countries 35 20 15 23 6 Stakeholder groups: % % % % % TTL 33 0 33 17 17 12 NSO 51 17 11 13 7 41 OGA 26 26 5 42 0 19 ONU 20 60 0 20 0 5 IDS .. .. .. .. .. 4 LDP 24 21 21 31 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 121 Table A.19 SRF role in enhancing the accessibility and usability of data Hardly 3C What role (if any) has the SRF played to date A very any or no I'm in enhancing the accessibility and usability of significant Some role to not No Total data in [name of country]? role role date sure answer responses Number of responses Implementing countries 24 17 4 8 4 57 Pipeline countries 10 7 10 18 8 53 All SRF countries 34 24 14 26 12 110 Percentage distributions % % % % % Afghanistan 21 42 16 11 11 19 Nigeria 60 20 0 15 5 20 Rwanda 44 28 6 17 6 18 Implementing countries 42 30 7 14 7 DR Congo 0 0 50 33 17 6 Ethiopia 31 15 15 31 8 13 Ghana 20 0 40 30 10 10 Lao PDR 14 36 0 36 14 14 Senegal 20 0 10 40 30 10 Pipeline countries 19 13 19 34 15 All SRF countries 31 22 13 24 11 Stakeholder groups: % % % % % TTL 33 8 25 8 25 12 NSO 46 12 10 22 10 41 OGA 21 37 0 37 5 19 ONU 40 40 0 20 0 5 IDS .. .. .. .. .. 4 LDP 14 28 21 28 10 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 122 Table II.20 Success of the SRF in linking improvements in the statistical system to the needs of monitoring frameworks 3D Overall, how successful do you consider the SRF to be in linking Barely improvements in the statistical successful Too system to the needs of national and or not Promises early sectoral monitoring frameworks in Very Somewhat successful to be to No Total [name of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 19 16 3 8 10 1 57 Pipeline countries 6 5 1 11 24 6 53 All SRF countries 25 21 4 19 34 7 110 Percentage distributions % % % % % % Afghanistan 21 26 5 11 37 0 19 Nigeria 35 40 5 15 5 0 20 Rwanda 44 17 6 17 11 6 18 Implementing countries 33 28 5 14 18 2 DR Congo 0 0 0 17 83 0 6 Ethiopia 23 8 0 23 23 23 13 Ghana 0 10 0 20 70 0 10 Lao PDR 14 21 7 29 21 7 14 Senegal 10 0 0 10 60 20 10 Pipeline countries 11 9 2 21 45 11 All SRF countries 23 19 4 17 31 6 Stakeholder groups: % % % % % % TTL 25 8 0 17 42 8 12 NSO 24 20 2 20 34 0 41 OGA 26 26 11 26 5 5 19 ONU 40 60 0 0 0 0 5 IDS .. .. .. .. .. .. 4 LDP 14 10 3 14 41 17 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 123 Table II.21 Functioning of existing institutional mechanisms 4A Do you consider that existing institutional mechanisms in the field of statistics in [name of To some I'm not No Total country].......function well? Yes extent No sure answer responses Number of responses Implementing countries 23 21 6 6 1 57 Pipeline countries 7 24 18 2 2 53 All SRF countries 30 45 24 8 3 110 Percentage distributions % % % % % Afghanistan 21 58 11 11 0 19 Nigeria 35 40 20 5 0 20 Rwanda 67 11 0 17 6 18 Implementing countries 40 37 11 11 2 DR Congo 0 33 67 0 0 6 Ethiopia 8 23 46 8 15 13 Ghana 10 40 40 10 0 10 Lao PDR 21 57 21 0 0 14 Senegal 20 70 10 0 0 10 Pipeline countries 13 45 34 4 4 All SRF countries 27 41 22 7 3 Stakeholder groups: % % % % % TTL 50 33 17 0 0 12 NSO 32 44 17 7 0 41 OGA 26 42 32 0 0 19 ONU 0 40 40 20 0 5 IDS .. .. .. .. .. 4 LDP 21 38 17 14 10 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 124 Table II.22 Status of user satisfaction surveys Yes, a There is Not sure what the 4B Has a user satisfaction survey survey has A survey no such situation is with been undertaken, or is one already been is being survey regard to a user No Total planned, in [name of country]? undertaken planned planned satisfaction survey answer responses Number of responses Implementing countries 28 5 3 20 1 57 Pipeline countries 12 14 7 19 1 53 All SRF countries 40 19 10 39 2 110 Percentage distributions % % % % % Afghanistan 42 26 0 32 0 19 Nigeria 35 0 15 50 0 20 Rwanda 72 0 0 22 6 18 Implementing countries 49 9 5 35 2 DR Congo 17 0 67 17 0 6 Ethiopia 8 8 15 62 8 13 Ghana 100 0 0 0 0 10 Lao PDR 0 43 7 50 0 14 Senegal 0 70 0 30 0 10 Pipeline countries 23 26 13 36 2 All SRF countries 36 17 9 35 2 Stakeholder groups: % % % % % TTL 33 50 8 8 0 12 NSO 54 22 12 12 0 41 OGA 11 0 11 79 0 19 ONU 20 0 0 80 0 5 IDS .. .. .. .. .. 4 LDP 28 14 3 48 7 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 125 Table II.23 Efforts to enhance the capacity of data users 4C Have there been any efforts under There the SRF to enhance the capacity of data Yes, there have No are no users (such as the training of local been efforts to efforts such Not sure journalists in the use of statistics) in enhance the yet, but efforts in what the [name of country], or are any such capacity of data some are place or situation No Total efforts planned? users planned planned is answer responses Number of responses Implementing countries 25 7 4 20 1 57 Pipeline countries 9 14 6 20 4 53 All SRF countries 34 21 10 40 5 110 Percentage distributions % % % % % Afghanistan 47 26 5 21 0 19 Nigeria 45 0 10 45 0 20 Rwanda 39 11 6 39 6 18 Implementing countries 44 12 7 35 2 DR Congo 17 17 50 17 0 6 Ethiopia 8 8 8 69 8 13 Ghana 40 40 0 0 20 10 Lao PDR 21 36 7 36 0 14 Senegal 0 30 10 50 10 10 Pipeline countries 17 26 11 38 8 All SRF countries 31 19 9 36 5 Stakeholder groups: % % % % % TTL 50 17 25 8 0 12 NSO 41 32 7 15 5 41 OGA 0 11 5 84 0 19 ONU 60 0 0 40 0 5 IDS .. .. .. .. .. 4 LDP 17 10 10 52 10 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 126 Table II.24 Success of the SRF in promoting an improved dialogue between users and producers of statistics 4D Overall, how successful do you consider the SRF to be in Barely promoting an improved national successful Too dialogue and partnership between or not Promises early users and producers of statistics in Very Somewhat successful to be to No Total [name of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 14 18 5 13 5 2 57 Pipeline countries 5 6 1 13 21 7 53 All SRF countries 19 24 6 26 26 9 110 Percentage distributions % % % % % % Afghanistan 11 53 16 5 16 0 19 Nigeria 35 25 5 25 5 5 20 Rwanda 28 17 6 39 6 6 18 Implementing countries 25 32 9 23 9 4 DR Congo 0 0 0 33 50 17 6 Ethiopia 8 0 0 31 38 23 13 Ghana 10 20 0 10 50 10 10 Lao PDR 14 21 7 36 21 0 14 Senegal 10 10 0 10 50 20 10 Pipeline countries 9 11 2 25 40 13 All SRF countries 17 22 5 24 24 8 Stakeholder groups: % % % % % % TTL 33 17 0 17 25 8 12 NSO 17 20 0 34 24 5 41 OGA 16 21 16 26 16 5 19 ONU 20 40 0 20 0 20 5 IDS .. .. .. .. .. .. 4 LDP 10 24 10 14 28 14 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 127 Table II.25 SRF impact on aligning support for statistical capacity building to the NSDS 5A Has the SRF had an impact, or do Has already Has not had any you think it has the potential to make had a Has the positive impact an impact, on aligning support for positive potential and/or does not statistical capacity building to the impact on to make appear to have I'm national statistical plan or NSDS in aligning an the potential to not No Total [name of country]? support impact do so sure answer responses Number of responses Implementing countries 20 30 2 4 1 57 Pipeline countries 5 43 0 4 1 53 All SRF countries 25 73 2 8 2 110 Percentage distributions % % % % % Afghanistan 42 47 5 5 0 19 Nigeria 40 50 0 5 5 20 Rwanda 22 61 6 11 0 18 Implementing countries 35 53 4 7 2 DR Congo 0 100 0 0 0 6 Ethiopia 8 92 0 0 0 13 Ghana 10 90 0 0 0 10 Lao PDR 14 57 0 29 0 14 Senegal 10 80 0 0 10 10 Pipeline countries 9 81 0 8 2 All SRF countries 23 66 2 7 2 Stakeholder groups: % % % % % TTL 33 67 0 0 0 12 NSO 27 68 0 5 0 41 OGA 11 74 0 16 0 19 ONU 20 60 0 20 0 5 IDS .. .. .. .. .. 4 LDP 17 62 7 7 7 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 128 Table II.26 SRF impact on donor coordination of statistical capacity building Has already Has not led to led to better better donor 5B Has the SRF aproach led, or do donor Has the coordination you think it has the potential to lead, coordination potential to and/or does not to better donor coordination of of statistical lead to appear to have I'm statistical capacity building in capacity better donor the potential to not No Total [name of country]? building coordination do so sure answer responses Number of responses Implementing countries 21 30 2 3 1 57 Pipeline countries 3.5 42.5 1 5 1 53 All SRF countries 24.5 72.5 3 8 2 110 Percentage distributions % % % % % Afghanistan 53 32 5 11 0 19 Nigeria 35 60 0 0 5 20 Rwanda 22 67 6 6 0 18 Implementing countries 37 53 4 5 2 DR Congo 0 100 0 0 0 6 Ethiopia 15 77 8 0 0 13 Ghana 10 80 0 10 0 10 Lao PDR 0 71 0 29 0 14 Senegal 5 85 0 0 10 10 Pipeline countries 7 80 2 9 2 All SRF countries 22 66 3 7 2 Stakeholder groups: % % % % % TTL 38 63 0 0 0 12 NSO 22 73 0 5 0 41 OGA 0 79 0 16 5 19 ONU 0 80 0 20 0 5 IDS .. .. .. .. .. 4 LDP 28 52 10 7 3 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 129 Table II.27 Success of the SRF in delivering more effective aid for strengthening statistical systems 5C Overall, how successful do you consider the SRF to be in delivering Barely more efficient and effective aid and successful Too technical assistance for strengthening or not Promises early statistical systems and results Very Somewhat successful to be to No Total measurement in [name of country]? successful successful at all successful judge answer responses Number of responses Implementing countries 16 21 2 10 6 2 57 Pipeline countries 6 6 0 12 25 4 53 All SRF countries 22 27 2 22 31 6 110 Percentage distributions % % % % % % Afghanistan 21 47 11 5 11 5 19 Nigeria 40 30 0 15 10 5 20 Rwanda 22 33 0 33 11 0 18 Implementing countries 28 37 4 18 11 4 DR Congo 0 0 0 33 67 0 6 Ethiopia 15 8 0 15 46 15 13 Ghana 10 10 0 20 60 0 10 Lao PDR 21 21 0 36 21 0 14 Senegal 0 10 0 10 60 20 10 Pipeline countries 11 11 0 23 47 8 All SRF countries 20 25 2 20 28 5 Stakeholder groups: % % % % % % TTL 33 8 0 8 42 8 12 NSO 20 24 0 27 27 2 41 OGA 32 16 5 32 11 5 19 ONU 0 60 0 0 20 20 5 IDS .. .. .. .. .. .. 4 LDP 10 31 3 14 34 7 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 130 Table II.28 Expectations for the SRF 7A. How reasonable or realistic do you think the expectations for the SRF are in Very Somewhat I'm not No Total terms of its impact and timing? reasonable reasonable Unrealistic sure answer responses Number of responses Implementing countries 21 26 2 3 1 53 Pipeline countries 18 13 1 19 1 52 All SRF countries 39 39 3 22 2 105 Percentage distributions % % % % % Afghanistan 21 63 5 11 0 19 Nigeria 67 22 6 6 0 18 Rwanda 31 63 0 0 6 16 Implementing countries 40 49 4 6 2 DR Congo 0 50 0 50 0 6 Ethiopia 33 17 0 42 8 12 Ghana 60 20 10 10 0 10 Lao PDR 21 29 0 50 0 14 Senegal 50 20 0 30 0 10 Pipeline countries 35 25 2 37 2 All SRF countries 37 37 3 21 2 Stakeholder groups: % % % % % TTL 50 42 0 8 0 12 NSO 46 34 2 17 0 41 OGA 26 26 0 42 5 19 IDS .. .. .. .. .. 4 LDP 31 41 3 21 3 29 Notes: This question was not posed to the ONU group. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 131 Table II.29 Possible extension of the SRF to regional operations 7C. Do you consider that the SRF, hitherto comprising almost exclusively country- specific projects, should be extended to No Total regional initiatives and operations? Yes Perhaps No answer responses Number of responses Implementing countries 24 20.5 9.5 3 57 Pipeline countries 18 20 8 7 53 All SRF countries 42 40.5 17.5 10 110 Percentage distributions % % % % Afghanistan 32 50 13 5 19 Nigeria 50 25 15 10 20 Rwanda 44 33 22 0 18 Implementing countries 42 36 17 5 DR Congo 17 33 33 17 6 Ethiopia 46 38 8 8 13 Ghana 40 10 20 30 10 Lao PDR 21 64 14 0 14 Senegal 40 30 10 20 10 Pipeline countries 34 38 15 13 All SRF countries 38 37 16 9 Stakeholder groups: % % % % TTL 42 42 17 0 12 NSO 44 33 13 10 41 OGA 47 26 11 16 19 ONU 40 20 20 20 5 IDS .. .. .. .. 4 LDP 24 52 17 7 29 Note: Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 132 Table II.30 SRF grant approval and disbursement process 8A. How smooth and timely was, or has Beset by been to date, the SRF grant approval and Smooth Somewhat difficulties No Total disbursement process? and timely problematic and delays answer responses Number of responses Implementing countries 6 12 4 5 27 Pipeline countries 5 7 9 9 30 All SRF countries 11 19 13 14 57 Percentage distributions % % % % Afghanistan 27 55 9 9 11 Nigeria 30 40 10 20 10 Rwanda 0 33 33 33 6 Implementing countries 22 44 15 19 DR Congo 0 0 50 50 6 Ethiopia 20 40 20 20 5 Ghana 11 22 33 33 9 Lao PDR 40 60 0 0 5 Senegal 20 0 40 40 5 Pipeline countries 17 23 30 30 All SRF countries 19 33 23 25 Stakeholder groups: % % % % TTL 42 25 17 17 12 NSO 15 34 24 27 41 IDS .. .. .. .. 4 Notes: This question was posed only to the above stakeholder groups. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 133 Table II.31 Experiences in implementing the SRF grant 8B. What has been your experience in I don't have implementing the SRF grant, in terms of enough reporting arrangements, support from Very Somewhat Generally experience to No Total World Bank staff and so on? positive positive negative judge answer responses Number of responses Implementing countries 13 8 1 3 2 27 Pipeline countries 2 6 1 17 4 30 All SRF countries 15 14 2 20 6 57 Percentage distributions % % % % % Afghanistan 55 36 0 9 0 11 Nigeria 60 20 0 20 0 10 Rwanda 17 33 17 0 33 6 Implementing countries 48 30 4 11 7 DR Congo 0 17 0 50 33 6 Ethiopia 0 40 0 60 0 5 Ghana 11 11 11 56 11 9 Lao PDR 20 20 0 60 0 5 Senegal 0 20 0 60 20 5 Pipeline countries 7 20 3 57 13 All SRF countries 26 25 4 35 11 Stakeholder groups: % % % % % TTL 25 33 0 25 17 12 NSO 24 24 2 39 10 41 IDS .. .. .. .. .. 4 Notes: This question was posed only to the above stakeholder groups. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 134 Table II.32 Recipient-country execution of SRF projects 8D. How large a challenge do you consider that recipient-country execution of the SRF Somewhat Not a I'm project has been or will be in [name of A great of a significant not No Total country]? challenge challenge challenge sure answer responses Number of responses Implementing countries 14 11 13 5 1 44 Pipeline countries 10 16 8 4 4 42 All SRF countries 24 27 21 9 5 86 Percentage distributions % % % % % Afghanistan 47 35 0 18 0 17 Nigeria 31 23 23 15 8 13 Rwanda 14 14 71 0 0 14 Implementing countries 32 25 30 11 2 DR Congo 67 33 0 0 0 6 Ethiopia 18 27 27 9 18 11 Ghana 10 30 40 10 10 10 Lao PDR 14 57 0 29 0 7 Senegal 25 50 13 0 13 8 Pipeline countries 24 38 19 10 10 All SRF countries 28 31 24 10 6 Stakeholder groups: % % % % % TTL 42 42 8 0 8 12 NSO 17 32 29 17 5 41 IDS .. .. .. .. .. 4 LDP 38 24 24 7 7 29 Notes: This question was not posed to the OGA and ONU groups. Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 135 Table II.33 Attention to SRF projects by World Bank line management 8E. Do you consider that SRF projects receive adequate attention by line management in the To some I'm not No Total World Bank? Yes extent No sure answer responses Number of responses Implementing countries 21 13 4 5 1 44 Pipeline countries 16 10 3 10 3 42 All SRF countries 37 23 7 15 4 86 Percentage distributions % % % % % Afghanistan 47 35 0 12 6 17 Nigeria 69 8 8 15 0 13 Rwanda 29 43 21 7 0 14 Implementing countries 48 30 9 11 2 DR Congo 50 33 0 17 0 6 Ethiopia 36 9 9 36 9 11 Ghana 40 10 20 20 10 10 Lao PDR 57 14 0 29 0 7 Senegal 13 63 0 13 13 8 Pipeline countries 38 24 7 24 7 All SRF countries 43 27 8 17 5 Stakeholder groups: % % % % % TTL 33 50 17 0 0 12 NSO 56 22 0 15 7 41 IDS .. .. .. .. .. 4 LDP 24 28 14 31 3 29 Notes: This question was not posed to the OGA and ONU groups. Responses from the IDS group have been suppressed Source: SRF Evaluation Survey 2013 136 Table II.34 SRF grants as stand-alone projects 8F. Would there be an advantage to the SRF grant being part of a larger Bank operation rather than being a stand- No Total alone project as is currently the case? Yes Perhaps No answer responses Number of responses Implementing countries 8 23 9 4 44 Pipeline countries 9 18 12 3 42 All SRF countries 17 41 21 7 86 Percentage distributions % % % % Afghanistan 12 65 18 6 17 Nigeria 15 38 23 23 13 Rwanda 29 50 21 0 14 Implementing countries 18 52 20 9 DR Congo 17 33 33 17 6 Ethiopia 36 36 18 9 11 Ghana 10 40 50 0 10 Lao PDR 0 100 0 0 7 Senegal 38 13 38 13 8 Pipeline countries 21 43 29 7 All SRF countries 20 48 24 8 Stakeholder groups: % % % % TTL 33 25 33 8 12 NSO 20 59 17 5 41 IDS .. .. .. .. 4 LDP 17 45 24 14 29 Notes: This question was not posed to the OGA and ONU groups. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 137 Table II.35 Familiarity with the SRF (standardized) generic Results Framework Not 8G. Are you familiar with the SRF Very Somewhat familiar at No Total standardized [generic] Results Framework? familiar familiar all answer responses Number of responses Implementing countries 14 15 13 2 44 Pipeline countries 7 20 13 2 42 All SRF countries 21 35 26 4 86 Percentage distributions % % % % Afghanistan 24 41 29 6 17 Nigeria 31 23 38 8 13 Rwanda 43 36 21 0 14 Implementing countries 32 34 30 5 DR Congo 17 33 33 17 6 Ethiopia 27 36 36 0 11 Ghana 20 30 50 0 10 Lao PDR 0 86 14 0 7 Senegal 13 63 13 13 8 Pipeline countries 17 48 31 5 All SRF countries 24 41 30 5 Stakeholder groups: % % % % TTL 58 42 0 0 12 NSO 15 39 41 5 41 IDS .. .. .. .. 4 LDP 17 45 31 7 29 Notes: This question was not posed to the OGA and ONU groups. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 138 Table II.36 Experience with the SRF (standardized) generic Results Framework 8H. What has been your experience I don't have with the SRF standardized [generic] Very Somewhat Generally enough experience No Total Results Framework? positive positive negative to judge answer responses Number of responses Implementing countries 2 11 2 9 3 27 Pipeline countries 1 7 4 16 2 30 All SRF countries 3 18 6 25 5 57 Percentage distributions % % % % % Afghanistan 0 36 9 55 0 11 Nigeria 10 50 10 20 10 10 Rwanda 17 33 0 17 33 6 Implementing countries 7 41 7 33 11 DR Congo 0 0 33 50 17 6 Ethiopia 20 20 0 60 0 5 Ghana 0 11 22 56 11 9 Lao PDR 0 40 0 60 0 5 Senegal 0 60 0 40 0 5 Pipeline countries 3 23 13 53 7 All SRF countries 5 32 11 44 9 Stakeholder groups: % % % % % TTL 0 58 33 8 0 12 NSO 7 24 0 56 12 41 IDS .. .. .. .. .. 4 Notes: This question was posed only to the above stakeholder groups. Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 139 Table II.37 The SRF concept and statistical capacity building 9A. To what extent do you consider that the concept To a To To an of the SRF meets the needs of statistical capacity great some insignificant I'm not No Total building in [name of country]? extent extent extent sure answer responses Number of responses Implementing countries 30 21 1 3 2 57 Pipeline countries 30 14 1 6 2 53 All SRF countries 60 35 2 9 4 110 Percentage distributions % % % % % Afghanistan 32 53 5 5 5 19 Nigeria 50 40 0 5 5 20 Rwanda 78 17 0 6 0 18 Implementing countries 53 37 2 5 4 DR Congo 33 67 0 0 0 6 Ethiopia 54 15 8 15 8 13 Ghana 80 20 0 0 0 10 Lao PDR 50 29 0 21 0 14 Senegal 60 20 0 10 10 10 Pipeline countries 57 26 2 11 4 All SRF countries 55 32 2 8 4 Stakeholder groups: % % % % % TTL 58 42 0 0 0 12 NSO 61 34 2 0 2 41 OGA 32 32 0 26 11 19 ONU 60 20 0 20 0 5 IDS .. .. .. .. .. 4 LDP 55 28 3 10 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 140 Table II.38 Leadership of the statistical development process 9C. To what extent do you consider that the SRF has facilitated, or has the potential to facilitate, effective ownership and leadership by the government of the To a To To an I'm development of the national statistical system of [name great some insignificant not No Total of country]? extent extent extent sure answer responses Number of responses Implementing countries 31 19 1 5 1 57 Pipeline countries 27 15 0 8 3 53 All SRF countries 58 34 1 13 4 110 Percentage distributions % % % % % Afghanistan 37 53 0 5 5 19 Nigeria 55 25 5 15 0 20 Rwanda 72 22 0 6 0 18 Implementing countries 54 33 2 9 2 DR Congo 17 50 0 17 17 6 Ethiopia 46 23 0 23 8 13 Ghana 70 30 0 0 0 10 Lao PDR 50 29 0 21 0 14 Senegal 60 20 0 10 10 10 Pipeline countries 51 28 0 15 6 All SRF countries 53 31 1 12 4 Stakeholder groups: % % % % % TTL 58 33 0 0 8 12 NSO 71 22 2 5 0 41 OGA 32 26 0 32 11 19 ONU 60 20 0 20 0 5 IDS .. .. .. .. .. 4 LDP 41 41 0 14 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 141 Table II.39 SRF project implementation and governmental reform measures 9D. Does implementation of the SRF project, currently or potentially, involve national authorities taking reform measures Yes, to a No, or that would not otherwise have been significant To some probably I'm not No Total undertaken? extent extent not sure answer responses Number of responses Implementing countries 15 20 5 16 1 57 Pipeline countries 18 18 3 9 5 53 All SRF countries 33 38 8 25 6 110 Percentage distributions % % % % % Afghanistan 11 58 11 16 5 19 Nigeria 55 20 0 25 0 20 Rwanda 11 28 17 44 0 18 Implementing countries 26 35 9 28 2 DR Congo 17 67 17 0 0 6 Ethiopia 31 38 0 15 15 13 Ghana 40 30 10 10 10 10 Lao PDR 36 29 7 29 0 14 Senegal 40 20 0 20 20 10 Pipeline countries 34 34 6 17 9 All SRF countries 30 35 7 23 5 Stakeholder groups: % % % % % TTL 67 33 0 0 0 12 NSO 32 29 12 20 7 41 OGA 26 26 5 32 11 19 ONU 20 40 0 40 0 5 IDS .. .. .. .. .. 4 LDP 17 41 7 31 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 142 Table II.40 SRF facilitation of the use of data to support evidence-based decision-making 9E. To what extent do you consider that the concept of the SRF has facilitated, or has the potential to facilitate, the greater use of statistical data and indicators to support To a To To an I'm evidence-based decision-making in [name of great some insignificant not No Total country]? extent extent extent sure answer responses Number of responses Implementing countries 28 25 0 3 1 57 Pipeline countries 30 16 0 5 2 53 All SRF countries 58 41 0 8 3 110 Percentage distributions % % % % % Afghanistan 11 84 0 0 5 19 Nigeria 65 30 0 5 0 20 Rwanda 72 17 0 11 0 18 Implementing countries 49 44 0 5 2 DR Congo 33 50 0 17 0 6 Ethiopia 62 23 0 8 8 13 Ghana 80 20 0 0 0 10 Lao PDR 50 36 0 14 0 14 Senegal 50 30 0 10 10 10 Pipeline countries 57 30 0 9 4 All SRF countries 53 37 0 7 3 Stakeholder groups: % % % % % TTL 58 42 0 0 0 12 NSO 66 29 0 5 0 41 OGA 37 37 0 16 11 19 ONU 40 40 0 20 0 5 IDS .. .. .. .. .. 4 LDP 41 48 0 7 3 29 Note: Responses from the IDS group have been suppressed. Source: SRF Evaluation Survey 2013 143 ANNEX 3: Detailed reviews of Nigeria and Rwanda experiences with the SRF-CF Introduction 1. As described in the Methodology section, owing to the short period covered by this evaluation and the fact that only three country projects were under implementation when it started, the assessment of the SRF’s objectives has relied on a number of different approaches including in-depth discussion with the teams in charge of the preparation and implementation of pilot projects as well as field visits to two of the three countries with relatively longer implementation experience. The visits provided useful information on how the key principles of the SRF approach work in a concrete country context, which was very helpful to the Evaluation Team in getting better insights into the responses and comments drawn from a larger population of respondents from the eight pilot countries. The field visits also provided the opportunity for the Evaluation Team to refine the questionnaire and the identification of responding groups for the survey. NIGERIA a) Background 2. The Nigeria Statistical System. Nigeria is a three-tiered federal state composed of 36 States and the Federal Capital Territory. Each state is made up of administrative units called Local Governments. By the provisions of the Nigeria Constitution, government at each level has the authority to produce its own statistics. In practice, most of the statistics are produced by the National Bureau of Statistics (NBS) which operates as the apex organization and coordinator of the National Statistics System (NSS), the State Statistical Agencies (SSAs) and line Ministries, Departments and Agencies (MDAs) both at the federal and state levels. The NBS and Federal MDAs produce national statistics, while the SSAs and state-level MDAs are in charge of statewide statistics. Other important producers of official statistics include the Central Bank of Nigeria (CBN), responsible for financial statistics, and the National Population Commission (NPopC) which has formal responsibility for population and demographic statistics, including carrying out population and housing censuses, and operates at all three administrative tiers of the government. Given the federal structure of governments in Nigeria, accurate population figures are important and politically sensitive since they are used to determine budget allocations for and transfers to the States. 3. The NBS was established by the Statistics Act of 2007 from the merger of the Federal Office of Statistics and the National Data Bank. The National Strategy for the Development of Statistics (NSDS) -which will be discussed in greater detail below- also promoted the conversion of SSAs into State Bureau of Statistics (SBSs) to play the role of counterparts of the NBS at the state level. In order to change the status of the agency, the State must have passed their Statistics Edicts20. Overall coordination of the NSS is provided by the National Consultative Committee on 20 The creation of the SBS is a key step in the reform of the state statistical systems. The former Director of the State Statistical Agency will become the Statistician General of the state and serves as a counterpart of the Federal Statistician General. To switch from an SSA to an SBS, all the states have to pass Statistical Edicts. As of the time of the Evaluation Team’s visit to Nigeria (end-June 2013), 21 of the 36 states have passed the Edicts and are on their way to creating SBSs. 144 Statistics (NCCS), a forum where overall coordination, development of statistical programs, and uniform standards and methodologies among the various agencies are discussed and adopted by all producers of official statistics in Nigeria, with a view to improving the quality, comparability and timeliness of statistical production. All policy issues regarding statistics are managed by the National Statistics Board which was established as the oversight body of the NSS and as the Steering Committee for statistical development in Nigeria with the NBS as its secretariat. 4. The National Strategy for the Development of Statistics (NSDS). In the course of rebuilding the NSS, the Government prepared a five-year statistical strategic plan for 2010/11- 2014/15 which was developed in part with a grant from the World Bank’s Trust Fund for Statistical Capacity Building and which aims at achieving the following strategic objectives: (i) creating the demand for the use of statistics at the state level through a focus on statistical communications; (ii) strengthening the national framework for data management and development through an emphasis on reforming the legal framework (especially at the state level); institutional restructuring of the MDAs; and streamlining the coordination arrangements; (iii) developing the necessary statistical framework and information technology (both hardware and software) with a focus on enhancing the efficiency and effectiveness in data collection, processing and analysis, and their dissemination; and (iv) promoting the development and use of sound and well-accepted statistical techniques for data production and management across the entire NSS. 5. The NSDS incorporates a generic State Statistical Master Plan (SSMP) for strengthening the Nigeria NSS at sub-national levels. The strategy also envisages a Sectoral Statistics Plan (SSP) targeting implementation of the NSDS at sectoral level. The latter is to be done in phases; the first phase includes 16 Federal MDAs chosen on the basis of the volume of their respective statistical activities and the relative significance of the content of the mandate of each of the MDAs in national survival and development. The NSDS is expected to be evaluated in the middle of next year. 6. The SRF Project. The US$ 10 million three-year project became effective in June 2011 and implementation is expected to be completed by February 2014. The project’s main components provide support to help to achieve the four strategic objectives of the NSDS as described above with a focus on NBS and NPopC at the federal level and six pilot states (Anambra, Bauchi, Edo, Kaduna, Niger and Ondo states), one per geopolitical zone of the Federation. 7. Development partners. Eighteen development agencies 21 are actively and directly involved in statistical development activities in Nigeria with the World Bank playing the role of lead donor. Their main support activities are summarized in Table III.1. 21 They are AfDB, CIDA, DfID, ECOWAS, ESSPIN-DfID (DfID funded Education Sector Support Programme in Nigeria), European Union, FAO, ILO, IMF, SPARC-DfID (DfID funded State Partnership for Accountability, Responsiveness and Capability aimed at supporting state-level governments in the use of public resources), UNAIDS, UNDP, UNFPA, UNICEF, UNIDO, USAID, WHO and the World Bank. 145 Table III.1: Main Activities in Statistics by Development Partners Agency Main Activities World Bank/DfID  Implementation of the NSDS both at the Federal level and in 6 pilot states. State Master Plans finalized and included in the NSDS.  National, MDAs and State Statistical Yearbooks. Templates have been distributed and are being populated by the various agencies.  Support to survey data collection and analysis (Core Welfare Indicators Survey; Nigeria Living Standards Survey (NLSS); Labor Force and Manpower Survey; National Agricultural Sample Census; National Agricultural Survey.  National M&E Framework.  Roll out of Automation of Death and birth registries through NPoPC. This will assure more accurate population projections and yield better demographic statistics for planning.  Digitalization of the Cartographic System with NPoPC, to get digital maps and improve Geographic Information Systems at NBS and NPoPC reporting.  Sectoral data collection and analysis in Health (Malaria Control Survey) and Education (Service Delivery Survey).  GDDS (General Data Dissemination System)  Improving access to administrative data  With OECD/PARIS21: Accelerated Data Program.  Data harmonization by sector just completed Water & Sanitation. UNDP  Nigeria Data Nervous System, through NBS, currently piloted in 17 States and will be rolled out to the remaining states through the NSDS.  Financing pilot program to set up of automatic death and birth registration centers, through NPoPC. A request is being finalized (which will be sent to the World Bank shortly), for Assistance in rolling out the Birth and Death Registry. UNICEF  Regular conduct of the Multiple Indicators Cluster Survey. Data for last survey 2008 being processed. UNFPA  Support to Census activities. Currently assisting the NPoPC in processing the 2006 Census data US Aid  Assist NPoPC in data collection and analysis of the Census and the Nigeria Demographic and Health Survey. Last survey conducted in 2008 with the data processing still on-going. AfDB  PPP (purchasing power parity) project.  Survey on Remittances: Financing a survey to study the amount and effect of remittances in the Nigeria economy. The study will be conducted with coordination from the World Bank. Source: IDA, Country Partnership Strategy Paper for the Federal Republic of Nigeria 2010-13. The information on activities shown in this table refers to the situation in 2009. b) Nigeria’s Experiences with the SRF Approach/Project 8. Promotion of a system-wide approach to statistics at the country level. The experiences of the two countries visited by the SRF evaluation mission reflect two startlingly different systems which have a significant impact on coordination. Leaving aside differences due to largely different geographical sizes and populations of the two countries, the top-down structure of government in Rwanda has proven to facilitate coordination and the design and implementation of a system-wide approach. In contrast, the three-tiered government structure in Nigeria has made it much more challenging to achieve similar results. 146 9. In spite of the above challenges, the design of the Nigeria SRF project which allocates funding equally between NBS and NPopC22 has been instrumental in incentivizing these two major Statistics agencies to work together. In addition to this significant achievement given that these two major statistics agencies were not effectively collaborating in the past, the SRF evaluation mission observed signs of greater ownership of the statistics development program at both the Federal and the Niger State levels. The difficulty of coordination is partly due to the Federal structure of governments in Nigeria. For instance it was reported that although the NBS is legally supposed to act as the overall coordinator on statistics matters for Nigeria as a whole, control of the NBS on state-level administrations and ministries remains confused in practice. 10. The evaluation mission also met with the SRF Technical Committee which is chaired by the Director of International Relations of the Federal Ministry of Finance and includes representatives of the Central Bank of Nigeria, the National Planning Commission, the Auditor General Office, the Accountant General Office, the Nigeria Statistical Association and the two executing agencies of the SRF project, NBS and NPopC. The interactions between the various members of the Technical Committee reveal a number of useful insights such as the complexities of coordination among various agencies in Nigeria where the authority of the lead agency, in this case the Federal Ministry of Finance, cannot be taken for granted. Based on the discussions during the meeting, the sharing of documents and other information on the SRF project among the members of the Committee may need to be improved although it is not clear whether this should be the role of the Ministry of Finance or of the World Bank. 11. The situation in Niger State reflects a strong ownership of statistics at the state level by all concerned including the Chairperson of the Board of the Niger State SBS (who was a former Director-General of NBS), the Planning Commissioner, senior staff at the state-level ministries of health and education as well as major statistics users including university researchers and local media representatives. More specifically, all the state-level MDAs have joined the State Consultative Committee on Statistics – SCCS, the state-level NCCS. 12. Increasing resources for implementing NSDS. As can be seen in Table III.2, allocations for investments have gone up sharply both for the NBS (from N290 million in 2011 to N 1 billion in 2013) and for the NPopC (from N290 million in 2011 to N 1.5 billion in 2013). Table III.2: Budget allocations for Statistics 2011-13, in billions of Naira 2011 (budgeted) 2012 (budgeted) 2013 (budgeted) Capital Recurrent Capital Recurrent Capital Recurrent expenditures expenditures expenditures expenditures expenditures expenditures National Planning 1.31 5.30 1.05 5.85 2.80 6.37 Commission NBS 0.29 2.42 0.36 3.37 1.00 3.96 NPopC 0.29 5.18 1.96 6.93 1.50 5.36 22 In June 2012, changes were made at the top management level of NPopC which have caused delays in the implementation of the SRF project on the components managed by this agency. The new management of NPopC has also developed very ambitious plans for the Commission which has rendered more difficult recent discussions between NPopC and other partners involved in statistics, including with the World Bank staff involved in the implementation of the SRF project. The recent resignation of the NPopC chairperson will likely lead to continued uncertainty in the near future. 147 13. The NBS is taking steps towards the creation of the Trust Fund for Statistics using a similar trust fund arrangement in education as a model23. Development partners are also in the early process of considering the creation of a basket fund for statistics. The World Bank is currently preparing a new technical assistance project – with a notional amount of at least $100 million for a 2-3 year project focusing on aligning existing methodologies to international standards and on helping to build Nigeria’s capacity to produce and disseminate statistics using the updated methodologies. This project is tentatively scheduled to become effective in June 2014, shortly after the end of the current SRF project. 14. In Niger State, the major developments in terms of resources have been the establishment of a separate budget for the SBS; funding for statistics used to be allocated through the Planning Commission. The dramatically increased budget for statistics has jumped from N1 million to N52 million in the most recent year. 15. Linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks. In spite of important progress, the impression of the SRF evaluation mission is that more work needs to be done on advocacy and on helping to raise awareness of the benefits of good data on the quality of economic policies. The challenges of how to promote usage of data for economic planning and how to allay the general skepticism of the public at large on economic and social data in Nigeria were a common theme. Improving the regularity of the publication of key data together with the organization of press conferences on the release of new information on the CPI and GDP have led to an improved coverage of economic data in the local media. The evaluation mission was also informed that earlier in the year, the MDG Office has commissioned a special study by the NBS to cover more MDG specific data. 16. In the Niger State Vision 3:2020 -- which aims to make of Niger State one of the top three states in Nigeria by the year 2020 --, statistics play a very critical role. Using the model of the Africa Peer Review Mechanism, senior political leaders of the State are considering setting up a state peer review mechanism to learn from best practices amongst Nigerian states in public finance, procurement practices, and statistics generation and dissemination. Senior officials met by the SRF evaluation mission also shared their plans for funding social and economic research in the State. The participation of the evaluation mission in a discussion between senior officials at the state-level ministries of health and education and the State Statistician General revealed the strong desire for the SBS to help with improving the quality of statistics in health and education required for policy making. The regular publication of key documents including the Niger State Statistical Yearbook, statistics on public finance, education, manpower and agriculture to name a few have led to the sharply rising demand for data. 17. Promoting an improved national dialogue and partnership between users and producers of statistics. The SRF has raised awareness and interest in statistics through a number of important measures including: (i) the publication of a release calendar for the NBS website; (ii) the dissemination of several publications; and (iii) the revamping of the NBS website which includes a powerful data portal with data mapping features. As a result of these measures, official statistics are now being mentioned in the media at an increasing rate; notably a number of editorials in the largest national newspapers covered poverty data, GDP, unemployment and 23 The latest information is that the idea of a Statistics Trust Fund will require more time for reflection and has been temporarily put on hold. 148 inflation. A user satisfaction survey was conducted at the federal level and in the 6 pilot states during the period of July to October 2011. The results of the survey, collected during workshops, have been analyzed and are available. NBS has organized journalist training with a focus on the interpretation of data, and a statistics user-producer seminar. Users of statistics have worked together to compile their needs which have been revised in December 2012 to be used as information to convey to the Federal level agencies for information and increased support. 18. The SRF project has also facilitated the conduct of 2 NCCS meetings. The NCCS is the largest forum on statistics in Nigeria. It meets twice a year; every producer of official statistics is invited to this meeting, and the way forward for the NSS is usually discussed at this forum. During the last meeting in December 2012, two main achievements were noted: (i) the adoption of the updated template for the State Statistical Yearbooks and (ii) a dialogue with all stakeholders on the way forward with re-basing the GDP (current base year is 1990) and GDP calculations for state and federal governments. The NSS is currently organizing itself to be in a position to compute state-level GDP, following the conclusion of the dialogue at the NCCS. 19. In Minna, the mission was able to get a first-hand exposure to the workings of statistical development at the State level, including a very useful joint meeting with the State Consultative Committee on Statistics (SCCS) chaired by the chairman of the board, various board members, university researchers and the local media to go over the internal dynamics on the development of statistics in Niger State, where rising demand for data has led not only to the regular publication of key economic and social indicators but also to a constant effort to improve the quality and coverage of data published by the SBS and state-level MDAs. Meetings with the permanent secretaries of the Planning Commission, the new officials of the ministries of health and education and the State Statistician General provided the evaluation mission with first-hand exposure to the close and constructive relationships on their respective roles in promoting both the reliance on and concerns for timely and relevant data for economic planning and its implementation. 20. Delivering more effective and efficient aid and technical assistance for strengthening statistical systems. The development partners have established the Nigeria Development Partners’ Statistics, Monitoring and Evaluation Group to maximize the impact of their support in improving production and use of M&E and statistics in Nigeria. The newly created forum provides for a unified dialogue with the Government and other Nigerian partners, for sharing information, implementation progress reports and lessons on support for statistics in Nigeria. The World Bank is the Chair and provides secretarial support with DfID as the backup agency. The Group’s priority work plan for the 2012-14 period focuses on the enhancement of the quality of data and their availability at the state level in four areas considered as key to national development namely the national accounts, health, education and agriculture. To this end, the Group will promote and support a better coordination among the main producers of official statistics at federal level and state levels, the survey program identified by NBS and NPopC, as well as the M&E component of the Transformation Agenda. 21. The development partners have devised in early 2013 a joint support strategy to the NBS Corporate Strategy Implementation Plan (CSIP) for 2013-201424, which covers three main areas: 24 This is the first time that the NBS has developed such a strategy and shared it with its development partners for information and eventual support. Given the limited resources coming from both the Government and development partners, further prioritization of the strategy document’s Action Plan is needed. 149 (i) how to better use administrative data and survey programs to generate better quality statistics, (ii) how to better align training programs with data production needs, and (iii) how to improve the ICT strategy for better data production and dissemination. In spite of these various efforts for more coordinated assistance in statistics, there remain concerns that support by development partners on statistics, and particularly to the MDAs, is not always in collaboration with the NBS, thereby rendering coordination of the NSS less than effective. 22. Impact of the in-country donor statistician (IDS) in helping to achieve the SRF objectives. The multiplicity of national stakeholders involved in the implementation of the SRF project both at the federal and state levels together with difficulties in travelling inside the country -- in particular security concerns in addition to long distances between Abuja and the six pilot states included in the SRF project -- have made the tasks of the IDS most challenging. Discussions with other development partners in Nigeria brought up the critical issue as to what should be the main role(s) of the IDS. The SRF-Catalytic Fund guidelines on terms of reference for the IDS are very broad and cover: (a) coordination of national agencies in charge of statistics; (b) coordination of development partners; (c) support for a national partnership to promote and strategize statistics development in the country including institutional reforms; and (d) provision of capacity development and technical assistance for the implementation of the NSDS. 23. In addition to the difficulties in allocating the ID S’s time to the various objectives listed above, the need for hands-on support for a recipient country-executed project and in the particular case of Nigeria where the IDS also plays the very important role of a ‘clearing house’ for all data used in World Bank projects -- the World Bank portfolio in Nigeria of approximately $5 billion is by far the largest in sub-Saharan Africa -- further complicated the role of the IDS. Given the very broad tasks performed by the current IDS in Nigeria, discussions by the evaluation mission with national stakeholders at both the Federal and State levels revealed a high appreciation of the IDS’s roles. 24. Discussions with all the current IDSs (Afghanistan, DRC, Ghana and Nigeria) reveal a broad range of priority settings in their current day-to-day activities covering the three main tasks which are project implementation, coordination and strategizing. In terms of office arrangements, most IDSs are located in the local office of the lead agency which also finances the IDSs – the World Bank for DRC and Nigeria and DfID for Afghanistan. In Ghana, the IDS is financed by DfID but based at the local World Bank office. This arrangement has helped to lighten the demand on the IDS’s time for regular office tasks such as project-related activities and to allow the Ghana IDS to focus her work much more on coordination and strategizing. Based on the assessment by the evaluation mission, the SRF guidelines may need to be reviewed. For large countries, and more particularly federal systems such as Nigeria and Ethiopia where states play an equally important role in the generation of data, it may be worth considering having more than one IDS. 25. Broad issues related to the SRF Charter. SRF-CF governance arrangements and operational practices and procedures. The mission also met with the World Bank's Country Director who shows a very strong support for statistics and has empowered the in-country donor statistician to play an active role in decision meetings on various WB projects in Nigeria with a focus on data. The Country Director for Nigeria highlighted the importance of statistics as a key sector in the country assistance strategy than that the SRF has helped raise the profile of statistics. She also mentioned that the development partners group in Statistics is performing best among the various sectors in Nigeria. 150 RWANDA a) Background 26. Compared to most other developing countries, the Government of Rwanda’s (GoR) management of the development process has a strong results focus. The government has a strong commitment to managing for results; all Government entities have results-based performance contracts, and increasingly evidence-based planning is becoming more widespread. 27. The National Institute of Statistics of Rwanda (NISR). At the center of Rwanda’s National Statistical System is the NISR which was established in 2006 as a semi-autonomous executive agency and coordinating agency of the NSS with the mandate to provide official statistics on social and economic living conditions and to conduct large-scale household-based and establishment-based surveys and censuses. Other important statistics producers are line ministries for administrative data, the Rwanda Revenue Authority for revenue statistics, the National Bank of Rwanda for balance of payments, banking and monetary statistics and the National Identification Agency for personal identification, driver’s licences and civil registration. NISR is also mandated to track and monitor progress of the Economic Development and Poverty Reduction Strategy (EDPRS, 2008-12) using key performance indicators and targets identified in the country’s Common Performance Assessment Framework (CPAF). 28. The National Strategy for the Development of Statistics (NSDS1, 2009-14). As was the case for most national statistical offices in developing countries, there remained several weaknesses in terms of coordination, collaboration and information sharing between NISR and other data producers and users of statistics. The development and implementation of the National Strategy for the Development of Statistics (NSDS1, 2009-14), by providing the overall direction and the framework for coordinating statistical activities, has led to significant improvement in the recent past. Developed through a consultative process involving the Government, NGOs, civil society, the private sector and development partners, NSDS1 covers five main components: (i) data development and management; (ii) information dissemination and services to users; (iii) coordination of NSS; (iv) capacity development (human and infrastructure); and (v) financing. The NISR is the main agency responsible for the overall NSDS1 program implementation and project executing agency. 29. Steering Committee/National Partnership Group. A joint Government/development partners (DPs) Steering Committee/National Partnership Group (SC/NPG) was established and provides the national oversight for the implementation of the NSDS, including the implementation of individual projects such as the SFR project. Chaired by the Permanent Secretary in the Ministry of Finance and Economic Planning, the SC/NPG is mandated to oversee, promote, coordinate, support, monitor and evaluate the implementation of the NSDS plan. The Steering Committee’s detailed schedule of meetings is presented in Table III.3. 151 Table III.3: Schedule of SC/NPG meetings and their coverage October  Review of overall performance against NSDS1 Annual Action Plan and Budget in the previous financial year25;  Discussion of independent annual review report;  Discussion of overall NSDS1 strategies and policies for next financial year;  Review of progress so far in the current financial year; and  Review of work plans and budgets for the third quarter of this financial year. January  Discussion of draft Annual Action Plan and Budget (including Procurement Plan) for next financial year (approval by March in time for GoR budget);  DPs to advise on indicative funding for next year's budget;  Review of progress so far in this financial year;  Review of work plans and budgets for the fourth quarter of the new financial year. April  Discussion of approved Annual Action Plan and Budget;  Discussion of status of DP funding and GoR funding for the next financial year;  Review of progress in current financial year; and  Review of work plans and budgets for the first quarter of the next financial year. June/July  Review of progress in final quarter of the last financial year; and  Review of work plans and budgets for the second quarter of the new financial year. 30. The role of development partners. Since the launch of NSDS1 in 2010, Rwanda’s development partners have aligned their support to the NSDS implementation. DfID and the European Commission, subsequently joined by the World Bank through the SRF project in 2012, have been actively involved in financing NSDS priorities through the NISR-managed NSDS1 ‘Basket Fund’ approach with pooled resources while other important partners --UNDP, UNICEF, UNFPA, Belgian Technical Cooperation, and AfDB -- provide their support through separate processes but under the overall priorities set out in the NSDS. Currently, the other DPs are involved in:  International Comparison Program Statistics (AfDB)  Civil Registration, DevInfo, DHS4 (UNICEF);  Population census, Civil Registration, IMIS Rwanda (UNFPA);  MDG monitoring and reporting (UNDP);and  Local Capacity Development in Administrative Data Management in Health, Education, Agriculture, and Population (Belgian Technical Cooperation). 31. The functioning of the Basket Fund. Among the eight SRF pilot countries, Rwanda is the only one which has set up pooled funding by a group of development partners under a Basket Fund. 32. Working Agreements. Once the development partners have informed the government of the level and timing of their participation in the "NSDS1 Basket Fund", their contributions will be specified in the bilateral arrangements between them and the GoR in which they will have agreed to:  establish an approved disbursement schedule in order to assist the GoR in planning for implementation of the NSDS1 25 The financial year in Rwanda runs from July to June. 152  make funding available according to approved disbursement schedules, on the basis of approved Annual Action Plans and Budgets;  support the GoR in the management, coordination and implementation of the NSDS1;  inform the GoR and other development partners of any change that may interfere with the implementation of the NSDS1, so that the partners can jointly take necessary actions;  lessen the administrative burden on NISR by deciding on harmonized reporting and auditing systems to the extent practicable; and  The DPs will not bear any responsibility and/or liability to any third party with regard to the implementation of the NSDS1. 33. Each participating DP's contribution is channeled to the National Bank account designated for the sole use of the "NSDS1 Basket Fund". NISR is responsible for maintaining a financial management system, including records and accounts, and preparing financial statements covering all DPs’ funds utilized. The oversight of the "NSDS1 Basket Fund" will be conducted by the NSDS1 Steering Committee/National Partnership Group which will be responsible for monitoring and directing the Fund's activities. It is the highest authority governing implementation of the NSDS1 Basket Fund. 34. Planning and Budgeting. GoR will produce an Annual Action (Work) Plan and Budget for the NSDS1, to be funded by the "NSDS1 Basket Fund". Once approved, the plan of work and budget will provide a basis on which the GoR and DPs pledge their levels of financial support to implement the NSDS1. DPs will endeavor to provide information on future disbursements in advance of the budget year. The GoR commits to ensuring that development partners' pledges are reflected in the Ministry of Finance’s annual budget estimates. Once appropriated, the NSDS1 budget will be allocated according to the activities set out in the Annual Action Plan to achieve specified targets. This process will be repeated each year to incorporate forward budgets and to make any necessary modifications to the plan of work and budget to incorporate intermediate outcomes and outputs to be achieved in future years. 35. Recent evaluations of the performance of NISR in managing the implementation of NSDS1. An independent institutional assessment 26 of NISR in 2010-11, financed by the European Commission and DfID and conducted in mid-2012, highlighted the very strong achievement by the latter with a limited budget. Overall, there is a very positive perception of NISR by all stakeholders. NISR is seen as making good progress. Sector ministries were uniformly appreciative of the work that NISR is doing to deliver social and economic statistics and in helping to improve the quality of administrative statistics and to drive the agenda for the need for evidence and good quality statistics to support this. And despite a qualified audit report, the financial management of the Basket Fund which supports the implementation of the NSDS is generally seen as good. 36. The evaluation, however, highlighted the high reliance of NISR on foreign consultancy support to help NISR deliver during the first two years of NSDS1. While acknowledging that this dependency is set to increase over the remaining two years of NSDS1 as it copes with the challenge of an increased demand for support to the rest of the government, NISR will need to 26 Roger Edmunds, ‘National Institute of Statistics, Rwanda Mid-term Review of the National Strategy for the Development of Statistics’, August 2012, unpublished. 153 develop the capacity and skills of its staff. More recently, a five-year capacity building program with a focus on staff skills training has recently been approved. Rwanda’s Experiences with the SRF Approach/Project 37. Promotion of a system-wide approach to statistics at the country level. As described in the background section, it was generally recognized by the people met by the Evaluation Team that ownership of the strategy for the development of statistics by the GoR is well anchored, which made it easy to bring DPs on board. And although Rwanda was already implementing the NSDS (2010-14) with significant progress as shown above before the SRF project became effective, the latter was recognised to have played a significant role in helping to give statistics a more powerful and relevant role in economic and social management in Rwanda both in terms of technical support and finance. In contrast to previous support for statistics which was scattered, by replicating the five main components27 of NSDS1, the SRF project has had a direct impact on the production of data, which was considered to be critical in raising the profile of statistics in Rwanda. 38. Discussions with other producers of data such as the National Identification Agency, the National Bank of Rwanda, and the Rwanda Development Board (in charge of the promotion of investments) all confirmed the strong coordination within the government of Rwanda on matters related to statistics and the critical role played by NISR, although the role of the SRF project was not clear. 39. Increasing resources for implementing NSDS1. What transpired from the discussions with the NISR was the beneficial impact of the SRF project in bringing significant additional resources and, through this process, in helping to attract other resources to statistics. In the words of the Director-General of NISR, ‘dialogue without money brings no respect’. Some of the local DPs who are not heavily involved in statistics have, however, raised the issue of sustainability in funding. From the minutes of the NSDS Steering Group meetings, the contribution by the GoR to statistics in the context of the implementation of NSDS1 remains practically stagnant between fiscal year 2013 (July 2012 to June 2013) and the following fiscal year (ending June 2014) at about RWF 3.2 billion. It was more difficult to evaluate contributions by DPs given that contributions to the Basket Fund may be used in different fiscal years. 40. Linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks. The mandate given to NISR on tracking progress of the Economic Development and Poverty Reduction Strategy and its active collaboration with line ministries reflect the importance of linking improved statistics to the need for monitoring progress in Rwanda. While the SRF project provides some focus on this issue, the attention has been much more on harmonising data collection tools among the various sector ministries, in particular local government, agriculture, health, education, economy and finance. This again highlights the bias of the SRF project design toward data development which as mentioned above, was much appreciated by the top management of NISR. 41. Promoting an improved national dialogue and partnership between users and producers of statistics. Meetings with the Institute of Policy Analysis and Research of Rwanda 27 Namely: (i) data development and management; (ii) information dissemination and services to users; (iii) coordination of NSS; (iv) capacity development (human and infrastructure); and (v) financing as discussed earlier in this annex. 154 (IPAR), the Kigali Institute of Science and Technology (KIST) and the Managing Director of New Times, a major local newspaper, again confirmed the strong relationships between producers and users of data in Rwanda -- although there remain challenges as to the timeliness of data in some areas, and weak capacity in comprehending statistics by the local media. There was unanimous recognition that the NISR website has significantly improved in the last year. Currently, in addition to economic and sectoral data, the website also allows for the online retrieval of micro data as well as a comprehensive literature on techniques and methods (metadata). For some of the research institutes, the fact that the NISR has to coordinate all data collection activities as well as to set high standards has helped to enhance the quality and reliability of statistics in the country. The NISR has close working relations with the National Council on Higher Education. In terms of needs, it was pointed out to the evaluation mission that the NISR needs to go beyond general social, economic, and administrative data into new areas such as employment, labour force, tourism, mining etc. 42. Discussion with the local media reflects some weaknesses in understanding key economic indicators, which point to the need for more training of data users. Also, domestic data users appear confused when faced with differences in numbers provided by international organizations such as the World Health Organization and by the Rwandan Ministry of Health. Similar discrepancies exist for enrolment rates between UNESCO and NISR, which may in part be accounted for by the use of different methodologies. Training of domestic users of data is rare and inadequate and based on the evaluation mission’s own assessment, will need to be given more emphasis. Another issue relates to the need for NISR and other statistics producers to provide commentary on the data so as to make them more accessible to non-experts. There were suggestions that the NISR should enhance its public relations role through, for instance, publication of more articles on data release on its website as well as through the national radio and television channels. According to some of the people interviewed by the Evaluation Team, the focus of NISR in recent years has been on the production of data. How to make statistics more understandable to the lay person must be given more attention. The SRF project, which has as one of its intermediate result indicators the implementation of the Statistical Information Dissemination Policy, may be helpful on this issue. 43. Delivering more effective and efficient aid and technical assistance for strengthening statistical systems. The close collaboration between GoR and the DPs under the leadership of the NSDS Steering Group/National Partnership Group, together with the establishment of the NSDS1 Basket Fund, are examples of best practices in terms of aid and technical assistance delivery. While the World Bank, through the SRF project, joined the Basket Fund as a latecomer and midway through the implementation of NSDS1, the important resources brought by the project have had a very significant impact on the production of statistics in Rwanda. While World Bank procurement procedures may have caused delays in resource availability at the earlier stage of the SRF project, flexibility in the design of the Basket Fund arrangements have made it possible to move funding by the various members of the Basket Fund around to cover funding gaps and to help avoid delays in the implementation of NSDS1 activities. 44. Impact of the in-country donor statistician in helping to achieve the SRF objectives. Rwanda does not currently have an in-country donor statistician financed under the SRF project. The evaluation mission was informed that the technical advisor to the DG of NISR has recently left the country and his replacement is being advertised. However, it was not clear to the mission that this person has played the role of the in-country statistician, particularly since the DG of 155 NISR mentioned that he does not need an in-country donor statistician. What are needed instead are specialized skills for highly technical challenges which are time-specific. It was also mentioned that the country may need assistance from somebody with global experience in helping to develop a long-term vision for statistics in Rwanda for NSDS2. 45. On the use of consultants, the former approach by UNDP to bring in West African consultants in statistics was not considered satisfactory by the NISR. Currently, faced with the challenge of producing credible data on a timely basis, the NISR relies rather heavily on Oxford Policy Management (OPM) consultants for a wide array of specialized tasks including helping to build data sets. Concerns were expressed by some of the people met by the Evaluation Team about the overreliance on OPM and the lack of hands-on training for local NISR staff, many of whom were recruited in the past two years and thus have little experience. The need to embed capacity building within the institution is critical; the experience of UNFPA with hands-on support for local enumerators on the population census has shown good results. 46. Broad issues related to the SRF Charter. On the option of regional SRF activities, efforts towards regional integration and consequently the need to harmonize statistics across countries may need a different approach to capacity building, but a consistent message was the need to consolidate progress at country level to allow countries to develop first before they can share experiences and vision. 47. SRF-CF governance arrangements and operational practices and procedures . Discussions the Evaluation Team had with NISR managers and staff and with local development partners highlighted a general frustration with the World Bank’s heavy procedures, particularly concerning procurement rules. It was generally believed that Rwanda’s procurement system is sufficiently solid28. In the specific case of the SRF project, $5.5 million for the preparation of the population census was not spent according to World Bank procedures, and expenditure commitments may have been incurred before the SRF project became effective, leading to significant delays in providing non-objection clearance for the retroactive replenishment of the Basket Fund. The lack of continuity both in the TTL and the local World Bank staff rendered the situation even more complicated. The flexibility granted by other partners in the Basket Fund has made it possible to continue with the census operation while awaiting clearance by the World Bank. 48. It was highlighted to the evaluation mission that the lack of clarity in supervision assignment and budget between DECDG and the Operations division was a factor in the lack of adequate project support by the World Bank. It was reported that in the early days of the SRF project, there was confusion as to how the supervision budget was managed, including a lack of clarity on the charge code for supervision needed to access budgetary resources. It was highlighted that the rapid turnover of the WB resident project officer was also a major stumbling block, although this now appears to have been stabilised. 28 Rwanda is currently a candidate for piloting the use of country systems in World Bank financed projects. The assessment is at an advanced stage and a decision will be taken in the near future on the timeline needed for Rwanda to become a pilot country. Until then, Government and DPs have agreed, in order to ensure a unified approach that the Bank Procurement and Consultant Guidelines will apply. It must be noted that the Rwanda systems and bidding documents are to a large extent aligned to the Bank procedures and bidding documents. The few gaps identified will be reflected as exceptions to the law and modifications in the national bidding documents. 156 ANNEX 4: The Results Framework and Monitoring of Progress a) The Results Framework - Conceptual Approach and Guidelines 1. The broad approach of the SRF Catalytic Fund is that financial and technical assistance resources are provided to countries to assist in the implementation of national strategies for the development of statistics (NSDS) or similar national priorities with a focus on a program-based approach and mechanisms for user feedback. More efficient and effective delivery of support designed using the principles of the Paris Declaration on Aid Effectiveness together with an increase in resources for statistics, which is a desirable key feature of the Catalytic Fund, will facilitate a sustained improvement in the production and use of official statistics and lead to an enhanced capacity in recipient countries to formulate and manage development policies and strategies. A more detailed presentation of the results chain, consisting of four inputs, four activities, three outputs at program level, three outputs at country level, and one outcome can be found in Table IV.1. Table IV.1: Results Chain for the SRF Catalytic Fund Inputs Activities Outputs Outputs Outcome (program level) (country level) 1) Increased 1) Finance from the 1) Implementation of 1) More efficient 1) Sustained capacity for policy SRF-CF, other good quality and effective delivery improvement in formulation and development national statistical of support to the capacity to produce decision-making partners, and development plan. national statistical and use official for development. government. system, and improved statistics. 2) Promote use of alignment with 2) Technical program-based 2) Improved principles of Paris assistance from the approach. response of Declaration on Aid SRF-CF statistical system to 3) Strengthen Effectiveness. Administration Unit national and mechanisms for user and from other 2) Adoption of sectoral needs. feedback and development program- based prioritization of 3) Improved partners. approach for statistical activities. dialogue and supporting statistical 3) Staff time of the partnership 4) Design efficient development in grant- lead donor, other between statistics and effective project recipient countries. development users and implementation partners, and 3) Increase in producers. arrangements, government resources for consistent with Paris officials. implementing Declaration on Aid national statistical 4) In-country donor Effectiveness. development plans. statistician. Source: Reproduced from 'Catalytic Fund - Monitoring and Evaluation Arrangements' by the SRF Catalytic Fund Administration Unit (March 2010). 2. It was furthermore proposed that specific approaches be used to monitor progress both at the levels of expected outputs -- particularly those referring to country-level outputs -- and outcomes. For country-level outputs, three specific indicators are proposed to monitor progress: i) the Statistical Capacity Indicator of the World Bank; ii) the publication of data quality 157 assessments in priority data areas (to be agreed in each country), using the Data Quality Assessment Framework of the IMF or similar assessment frameworks; and iii) an improved data dissemination policy that includes the publication of the calendar for the release of key statistics and of detailed guidelines for users on the access to survey metadata. The monitoring of outcomes will be based on feedback from users through user surveys. It is therefore recommended that all recipient countries will need to undertake user surveys -- first, during project inception to serve as a baseline, and subsequently at relevant junctures such as the preparation of the Mid-Term Review and at project completion. 3. Whereas the monitoring and evaluation (M&E) arrangements for the SRF Catalytic Fund were clearly spelled out as described above, efforts were made to allow for country leadership in the spirit of the Paris Declaration and for country-specificity in the implementation of the logical framework at the country level. The M&E guidelines for project preparation clearly specify that M&E arrangements for each country program supported by the Catalytic Fund should be based primarily on arrangements already agreed in NSDSs or similar documents. To link country- specific M&E frameworks with the Catalytic Fund 'generic' M&E arrangements, project teams and their counterparts in grant-recipient countries were advised, to the extent possible, to use the 'generic' framework in developing country-specific arrangements and to include most of the generic framework indicators in country-level M&E arrangements. 4. At the country level, implementing agencies will be responsible for measuring and monitoring the chosen indicators and for preparing semi-annual implementation progress reports in collaboration with development partners through the National Partnership Group and the lead donor. At the project level, where the supervising entity is the World Bank, progress reporting will consist of six-monthly assessments of implementation status and results (also known as ISRs) and Grant Monitoring Reports to track grant activities. b) The Generic Results Framework in Practice 5. As shown below, as the appendix to this Annex deals only with the SCI, the original results framework further expanded the results chain shown in Table IV.1 by identifying a list of 22 indicators which were designed to allow for objective measurements of inputs, activities, outputs at program and country levels and outcomes. For each of the 22 indicators, specific sources of information to be used for monitoring progress were specified with trend lines showing a baseline figure, a target figure at project completion, and two milestones during the project life. A full list of the indicators together with their sources of information is shown in Table IV.2, which also traces some subsequent changes to the indicators. 158 Table IV.2: Generic Results Framework - Coverage, Sources and Availability of Information Outcome, outputs, activities Indicators Sources of Availability of and inputs information information OUTCOME Increased capacity for 1/1. Proportion of user User satisfaction surveys a/. Baseline information is policy formulation and survey respondents satisfied available for four decision-making for that statistical outputs meet countries where user development their priority data needs. satisfaction surveys have been completed (Afghanistan, Ghana, Nigeria and Rwanda). 2/2. Extent to which User satisfaction surveys. Baseline information is statistical data is used in only available for policy and decision-making Nigeria and more processes. detailed analysis of other USSs is awaited. OUTPUTS (COUNTRY LEVEL) Sustained improvement 3/3. Statistical capacity World Bank statistical SCI (statistical capacity in capacity to produce building indicator. capacity database. indicator) scores are and use official statistics. available for 2013. 4. Additional indicators of Progress or implementation statistical capacity reports. established Country- b/. specific projects Improved response of 5/4. Percent increase in User satisfaction surveys. Available for four statistical system to user satisfaction of official countries. national and sectoral statistics. needs. Improved dialogue and 6/5. Proportion of users in Interviews with users by Using as a proxy, partnership between priority sectors who agree project supervision teams. regular annual statistics users and that consultation producer- user meetings producers. mechanisms with data (Lao PDR) or producers work well. stakeholder dialogue (Rwanda). OUTPUTS (PROGRAM LEVEL) More efficient and 7. Commitments Surveys of donor activities effective delivery of expenditures for statistical in statistics by the National support to the national activities by development Partnership Group. statistical system, and partners not participating in improved alignment with the National Partnership principles of Paris Group. Declaration on Aid 8/6. Proportion of support Surveys of donor activities Figures are available Effectiveness. to statistics by development in statistics organized by for most countries partners that is aligned with the National Partnership although they may be national priorities. Group or NSO. hard to interpret. Adoption of program- 9/7. Existence of National Minutes of National Available for all based approach for Partnership Group to Partnership Group countries. supporting statistical monitor the implementation meetings. 159 development in grant- and status of national recipient countries. statistical development plans. Increase in resources for 10. Levels of funding Minutes of National implementing national committed and spent by Partnership Group statistical development government and donors for meetings. plans. the implementation of NSDSs compared to budgeted amounts. 11. Levels of funding Government budget audit committed and spent by reports. government for the implementation of NSDSs compared to budgeted amounts. ACTIVITIES 1) Implementation of good 12/8. Percentage of Implementation progress Only available for quality national activities implemented reports of NSDSs. Rwanda. statistical development compared to agreed plan. implementation plan. 13/9. Implementation (WB) Implementation Available for countries progress rating (if World Status and Results under implementation. Bank is supervising entity). monitoring. 1) Processes to promote use 14/10. Percentage of Minutes of National Available for all of program-based development partners Partnership Group countries. approach. participating in the meetings. National Partnership Group relative to all donors involved in statistical development. 15/11. Publication of Progress reports of Only available for annual NSDS national authorities. Rwanda. implementation progress report. Design and use of 16/12. Pooled funding Reports of National Only available for efficient and effective arrangements used by Partnership Group. Rwanda on a national project implementation development partners. basis. Pooled funding arrangements, consistent arrangements may be with Paris Declaration available for specific on Aid Effectiveness. activities such as censuses. INPUTS 1) Finance from the SRF- 17. Ratio of firm Reports of National CF, other development commitments compared to Partnership Group. partners, and NSDS agreed by National government. Partnership Group. 18/13. Catalytic Fund Progress reports of Available for all project disbursement rates. supervising entity. countries under implementation. Technical assistance 19/14. Technical assistance Reports of National Measured by whether from the SRF-CF provided. Partnership Group. or not there are in- Administration Unit and country twinning from other development arrangements 160 partners. Staff time of the lead 15. Number of government Project supervision reports. Currently only available donor, other staff trained under the SRF for Nigeria development partners, grant by type of training. and government 20/16. Existence of lead Reports of National Lead donor officials. donor in National Partnership Group. arrangements exist for Partnership Group. most pilot countries. 21. Number of national Project supervision reports. staff devoted mainly to SRF project activities. In-country donor 22/17. Existence of in- Reports of National In-country donor statistician. country donor statistician. Partnership Group. statisticians are currently operational in Afghanistan, DRC, Ghana and Nigeria. a/ Notes: User satisfaction surveys have been conducted in four countries (Afghanistan, Ghana, Nigeria and Rwanda); are being planned for Ethiopia, Lao PDR and Senegal: and are not currently planned for DRC. b/ Deleted (original) indicators are shown with strikethrough. One new indicator to replace the deleted original one is shown in italics. 6. Whereas the generic results framework as summarized above and shown with full details in the M&E Arrangements Guidelines for the SRF Catalytic Fund provides a logical framework to monitor the outcome of the SRF approach as well as the intermediate steps such as inputs, activities, and outputs, translating those into specific indicators which are measurable and can be monitored over time has proven to be much more challenging. For lack of information, six of the original 22 indicators were subsequently deleted -- and one new indicator introduced. As a result of these changes, the number of indicators was reduced from the original 22 to 17. 7. In particular, efforts to measure and monitor the catalytic impact of the SRF approach in leveraging more resources for the implementation of national statistical development plans, through monitoring the levels of funding commitments and effective disbursements relative to budgeted amounts both by the governments and development partners, were discarded. Also, efforts to measure a sustained improvement in the capacity to produce and use official statistics by identifying additional indicators of statistical capacity building in addition to the World Bank Statistical Capacity Indicator (SCI) were subsequently dropped. 8. Even with a reduced set of indicators, the availability of information required to populate the generic results framework remains sparse and incomplete, making it difficult to measure and monitor key SRF principles, most importantly: a) the leveraging impact on resources of the SRF Catalytic Fund; and b) development partner support to the implementation of NSDSs. For indicators which rely on user satisfaction surveys as the main sources of data such as the improved dialogue and partnership between statistics users and producers and the use of statistics for planning, monitoring and evaluation and more specifically, improved response of the statistical system to national and sectoral needs, as shown in Table IV.2, information is available in four countries (Afghanistan, Ghana, Nigeria including for the six pilot states and Rwanda) for which user satisfaction surveys have already been carried out. Surveys are being planned for Ethiopia, Lao PDR and Senegal but as of the latest information, no user satisfaction surveys are currently being planned for DRC. With the exception of Rwanda, implementation progress reports of national statistical development plans are not available which makes it difficult if not impossible 161 to monitor progress on a broad range of indicators. Regular updates of the SRF Generic Results Framework using pilot country information are produced by DECDG and made available in the SRF Annual Reports. c) The Results Framework for the Pilot Countries 9. While project teams and countries were advised to incorporate the elements of the generic results framework in developing country-specific M&E arrangements, the latter should be based primarily on arrangements agreed in NSDSs or similar national documents. A review of the seven pilot SRF projects 29 shows limited success in blending the elements of the generic results framework and country-specific M&E priorities based on individual country NSDSs. A more careful examination of the country-specific results frameworks and monitoring reveals that for the five key SRF principles, only the one dealing with the promotion of an improved national dialogue and partnership between users and producers of the statistics has been identified as a result indicator in most pilot countries. The other key principles -- promoting a system-wide approach in statistics, increasing resources for implementing national statistical plans, linking improvements in statistics to the needs of national and sectoral monitoring frameworks, and delivering more efficient and effective technical assistance -- have received more limited attention as shown in Table IV.3. Table IV.3: Coverage of Key SRF Principles in Country-Specific M&E Arrangements Specific results indicators monitored in Key SRF Principles country project M&E arrangements Promoting a system-wide approach to statistics at the None of the pilot projects have identified elements of country level (including mechanisms for dialogue, the system-wide approach as a specific result indicator; coordination, and monitoring and evaluation). however, since all the SRF projects are aligned with the respective NSDS implementation plans, the need to promote and monitor the system-wide approach may not be seen as necessary. Increasing resources for implementing country-owned None with one exception: execution of annual work national statistical plans or strategies. plans including government externally financed budget (DRC). Linking improvements in the statistical system to the None with one exception: number of MDG indicators needs of national and sectoral monitoring frameworks. supported by the NSO and sourced as NSO data (Afghanistan). Promoting an improved national dialogue and User satisfaction, data dissemination policy, and partnership between users and producers of statistics. increased data availability through web portals are monitored by several countries. Only the Lao PDR project proposed to monitor data user-producer dialogue through annual conferences. Delivering more efficient and effective aid and None of the pilot projects have identified monitoring technical assistance for strengthening statistical better coordination and alignment of technical systems and results measurement, through better assistance as a result indicator. The Lao PDR project coordination and alignment to agreed national monitors an inter-agency coordination mechanism on statistical plans and through better alignment to the macroeconomic and poverty related statistics. Paris Declaration on Aid Effectiveness. Sources: Project appraisal documents for seven of the eight pilot countries. 29 Information on the Senegal project was not available at the time this report was finalized. 162 10. First and foremost, the individual SRF pilot projects were designed to support national capacity development in statistics. Giving the relatively larger size of the SRF pilot projects compared to more narrowly designed capacity building projects in statistics, the scope of the SRF pilot projects is generally broader covering: i) specific capacity development components such as staff training, human resource management, and IT infrastructure; ii) efforts to improve data collection, analysis and quality control; and iii) efforts to enhance data access and dissemination. A summary of the results project development objectives (PDOs) and intermediate results indicators can be found in Table IV.4. 163 Table IV.4: SRF Projects - M&E Indicators by Pilot Country Project Project Country Development Intermediate Results Indicators component Objectives Overall Ethiopia  Statistical information on microdata access institutional policy and dissemination framework in place. reforms Lao PDR A comprehensive  Inter-agency coordination mechanism (rules and (including macroeconomic framework regulations) on macroeconomic and poverty-related coordination) is developed and used for statistics in place and operational. policy making and planning. Nigeria  Annual progress reports presented to NSOa/ boards and uploaded to website.  Code of conduct based on NSO core values developed and disseminated.  Increase in the incidence of statistics quoted in the local media. Capacity Afgha- Improvement in statistical  Number of NSO and line ministry statistical staff development nistan capacity. who have completed specialized training program. (including human  IT infrastructure coverage. resource DRC  Technical training  Number of staff trained (of which women). development and sessions of which for  Improved offices. IT technology) women.  Graduate scholarships financed by the project of which for women. Ethiopia  Organizational strengthening and infrastructure development at NSO.  Number of staff trained in strategy planning, management and IT, and statistical subjects.  Number of branch offices equipped with modern tools, transport and work methods.  IT data and videoconferencing centers operational. Ghana Improvement in statistical  Number of full-time staff employed by NSO. capacity score disaggregated  Proportion of staff with professional by: methodology, sources of qualifications. data, periodicity and  MDA and regional statistics offices fully timeliness. equipped and operational. Lao PDR  Inter-agency coordination mechanism for statistics in place and operational.  Human resource management strategy in place.  IT plans including staff training developed and implemented.  Efforts to retain trained staff. Nigeria  Number of staff competent in core areas.  Availability and furnishing of state statistical agency offices. Rwanda  Training of NSO and other ministry staff in ICT and statistical subjects. 164 Project Project Country Development Intermediate Results Indicators component Objectives Data collection, Afgha- Number of MDG indicators  Number of data with proper metadata (data analysis and nistan supported by NSO and collection methodologies). quality control sourced as NSO data.  Number of provinces where price data are collected and included in the national estimates.  Percentage of source data developed for national income accounts. DRC  National surveys using  Population and enterprise censuses. the revised master sampling  Provisional master sample for surveys developed frame. and available in electronic format.  National accounts  Quality of census mapping. produced annually.  Availability of final census documentation.  Statistical products  Revision of enterprise sampling frame. available on the NSO website. Ethiopia Improved data quality and  Use of manual for standards in the collection, timeliness. compilation and verification of data.  Increased frequency of publication of key statistics.  Full implementation of surveys and other statistical tasks for key statistics.  Release of major statistical publications as per release calendar. Ghana Timely and robust statistics.  Proportion of planned censuses/surveys implemented.  Time lag between end of data collection and release of report.  Economic statistics produced in accordance with internationally accepted standards (CPI, PPI, Supply and Uses tables, and GDP).  Use of master sampling frame based on updated cartographic maps for household surveys. Lao PDR Improvement in  Internationally accepted standards, macroeconomic and poverty classifications and methodology adopted for related statistics including macroeconomic, poverty, social and demographic methodology, source data, statistics. periodicity and timeliness. Nigeria  Publications publicly  Release calendar established and functional. available according to  Various actions at six pilot states including release calendar. integration of micro-data into National Data  Internationally accepted Archive Centre (NADA); mapping of economic and common concepts adopted social statistics; availability of data on website; and used for selected increase in numbers of pilot registration centers; and thematic areas. increase in electronic birth/death data capture. Rwanda Adherence to standards on  Granting of visas for surveys, use of manuals for data quality (methods of balance of payments and government finance compilation). statistics.  Full and timely implementation of surveys and censuses.  Routine statistics developed/strengthened in five sectors. 165 Project Project Country Development Intermediate Results Indicators component Objectives  Improvement in economic statistics (in particular CPI and national income accounts). Data access and Afgha- Improvement in user  Percentage of users satisfied with statistical dissemination nistan satisfaction with services of products and services. the NSO.  Growth in number of hits on the NSO website.  Average number of months taken for surveys to be published. DRC Statistical products available  Data sets covered by the dissemination policy. on the NSO website. Ethiopia Improvement in data quality  Percentage of users satisfied with statistical access and in user products and services. satisfaction.  Modernization of open data portal.  Launch of mobile education unit for user outreach. Ghana Improvement in user  Communication and dissemination policy satisfaction with statistical adopted by NSO technical committee. products and services of the  Proportion of statistical products released in NSO. accordance with release calendar. Lao PDR Improvement in user  Percentage of users satisfied with statistical satisfaction with products and services. macroeconomic and poverty  Data user-producer dialogue established and statistics. functional (annual conferences).  Dissemination strategy developed and implemented.  Improved web portal with increased data availability. Nigeria Improvement in user  Percentage of users satisfied with statistical satisfaction. products and services. Rwanda  Timeliness and frequency  Design and storage capacity enhanced. of publication of key  Open Data Portal in place. statistics.  Implementation of the statistical information  Availability of data dissemination policy. through the Open Data  Level of user satisfaction. Portal.  Financial DRC  Execution of annual work plans including monitoring and government and externally-financed budget. audits a/ Note: The generic term NSO is used to designate the national statistical offices in the various countries. 11. For the three projects under implementation namely Afghanistan, Lao PDR, Nigeria and Rwanda, project implementation supervision is provided by the World Bank teams. One key document produced by supervision missions as a requirement is the Implementation Status and Results (ISR) which provides: i) implementation status overview of progress compared against Project Development Objective (PDO) indicators as well as against Intermediate Results Indicators which are summarized in the previous table; ii) disbursements expressed in dollar terms and as a share of total grants; and iii) ratings of progress towards achievement of PDOs, of overall implementation progress of intermediate results indicators, and of the overall project risk. 166 The ISR form may be supplemented by other documents such as aide-memoires which provide a more detailed description of project implementation, progress in achieving various indicators together with remedial actions where necessary. The ISR and other supporting documents focus on the project development objectives and intermediate results indicators but do not cover broader issues such as those highlighted in the SRF generic results framework. Table IV.5 below shows the overall ratings for the projects currently under implementation. Table IV.5: Overall Project Ratings Afghanistan Lao PDR Nigeria Rwanda (June 2013) (July 2013) (May 2013) (April 2013) Progress towards achievement of Moderately Moderately Satisfactory Satisfactory PDO satisfactory satisfactory Moderately Moderately Moderately Moderately Overall implementation progress satisfactory satisfactory satisfactory satisfactory Overall risk rating Substantial Substantial Low Moderate d) Assessment 12. The review of the SRF generic results framework and of the seven country-specific M&E arrangements -- the Senegal project M&E arrangements are not yet available -- highlighted the limited success so far by the project teams in their effort to blend the key elements of the generic results framework with country priorities based on individual country NSDSs. Six output indicators constitute the core of the generic results framework, namely: i) more efficient and effective delivery of support; ii) adoption of a program-based approach to statistics; iii) increased resources; iv) improved response to national and sectoral monitoring needs; v) improved dialogue between users and producers of statistics; and vi) sustained improvement in capacity to produce and use statistics. Since improvements in statistical capacity, and to a lesser extent data access, form the core of most developing country NSDSs, it is not surprising that data collection, analysis, quality control and dissemination and the capacity needed to achieve the above constitute the core of most SRF pilot project M&E arrangements. Although it may be premature to come to any firm conclusion on the impact of the SRF on the quantity and quality of data and access to official statistics, trends in the SCI (see Appendix) do show significantly larger improvements among the SRF pilot countries compared to the larger sample of developing countries. But we should not, without the benefit of a longer time frame, draw any definitive conclusion on this issue. 13. Based on the generally positive responses from the questionnaire survey and on the field visits, the Evaluation Team believes that the experiences in the pilot countries in areas such as assessing user satisfaction and improving data dissemination policies and practices have the potential to make it easier for the M&E arrangements in future SRF projects to better reflect the objectives of the SRF generic results framework, particularly those concerning a system-wide approach to statistics, user-producer dialogue, strong linkages to national and sector monitoring needs, and alignment of partner support to statistics with the NSDS. Information on resources, domestic and externally funded, while remaining a major challenge for lack of readily available data, is critical to ensuring the financial sustainability of the process of support to statistics. Table IV.6 proposes a short list of ‘high-level’ indicators which the Evaluation Team suggests 167 could be used as inputs in the design of results indicators for future SRF projects. These are based on key SRF principles which have served as the basis for this evaluation. In addition to the indicators, the table also highlights the sources and availability of information as well as recommended accompanying measures to facilitate the operationalization of the proposed indicators. 14. On resources for statistics, Rwanda’s basket-fund arrangements provide an example of good funding practices for statistics but may require organizational and human skills which are beyond the reach of most low-income countries. The positive response to the survey that SRF resources should be seen as core finance for the implementation of NSDSs may reflect a broader support of a system-wide approach to statistics through the NSDS. Lastly on the issue of more efficient and effective aid to statistics, the majority of survey respondents are generally optimistic about the impact of the SRF on helping to align support to statistics. National partnership groups in pilot countries also appeared to help promote better coordination and harmonization of donor support to statistics, and more sustained efforts will be needed. 168 Table IV.6: SRF – Proposed High-Level Indicators Key SRF Desired Proposed Sources of Availability of Recommended principles objectives indicators information information accompanying measures Promoting Encouraging the Existence of a Reports of NPG - Monitor role and a system- development of statistical supplemented by activities of NPG such as: wide statistical coordination interviews with frequency of meeting, approach to systems based on mechanism (e.g. key informants composition of NPG (is it statistics. country National and stakeholders. inclusive?), degree and ownership, Partnership level of active participation broader Group or NPG) by NPG members, NPG’s accountability, in place and role in reviewing NSDS more predictable operational in the implementation progress funding and national statistical reports. coordinated system. - Encourage the support by preparation of annual donors. NSDS implementation progress reports. - Promote stronger synergy between NSO and sector ministry statistical units. - Promote the role of the in-country donor statistician on coordination Increasing Ensuring - Trends in Reports of NPG Published - Promote a more financial financial donor funding supplemented by information systematic exchange of resources sustainability for for statistics interviews with may not be information on funding for long-term (yearly). key informants readily intentions within NPGS. statistical statistical - Proportion of and stakeholders. available and - Promote engagement systems. development. budget for Analysis of will require with initiatives such as national existing additional Paris21’s Country’s statistical system documentation work (help by Reports on the Support to covered by on support and the in-country Statistics (CRESS). domestic budgets. donor resources statistician). (yearly). Linking Focusing on the Percentage User satisfaction Results of user - Make the importance of improveme use of increase in user surveys (USS). satisfaction the use of data more nts in the information to satisfaction of surveys where explicit in projects’ statistical improve official statistics. available. development objectives. system to decision-making. - Need to revise the SRF the needs Guidelines on USS with of national the view to improving the and quality and efficacy of sectoral such surveys including monitoring making it lighter and more framework selective on key data users. s. - Availability of official websites on statistics. Promoting Encouraging the - Number of Reports of NPG Published - The number of events dialogue shift from a partnership and NSO information will need to be placed in between focus on events involving supplemented by may not be the context of the overall 169 Key SRF Desired Proposed Sources of Availability of Recommended principles objectives indicators information information accompanying measures producers availability of users and interviews with readily SRF project and all its and users data to greater producers of key informants available and other various activities; of use of data. and stakeholders. will require comparisons could be on a statistics. information. - Number of additional financial basis, or on a events or work (help by measure of person-days as programs the in-country possible sub- designed to donor indicators. Where increase the statistician). possible, post-event capacity to assessment of the analyze and use effectiveness of these statistics. events will need to be conducted. - Encourage efforts to identify key users of statistics, better understand their needs, and facilitate the use of statistics (examples such as: creating media relations unit; consultation with users on statistics to identify gaps). Delivering Promoting aid Proportion of Reports of NPG Published More information on more effectiveness support to supplemented by information related issues such as: i) effective through better statistics by interviews with may not be the predictability of aid. alignment and development key informants readily funding and ii) and joint harmonization. partners that is and stakeholders. available and donor missions on aligned to the will require statistics. country additional strategies. work (help by the in-country donor statistician). Capacity Ensuring Number of Records of Published Identification of training (skills) adequate human people trained training. information needs and HR development building. resource for broken down by may not be plans including for users of long-term categories: readily data. sustainability of government staff available but statistical v. other data should be development. users; technical relatively easy v. broader to collect. organizational. 170 15. The review of the World Bank’s supervision mission reports shows a relatively narrow focus on the project development objectives and intermediate results indicators. It would be useful for supervision teams to also cover the broader issues reflected in the SRF generic results framework. In practical terms, this may require some modifications in the SRF guidelines to TTLs to stress this point, so as to be able to respond to SRF requirements. More standardized reports on the broader SRF principles by the in-country donor statisticians would also be helpful. 171 Appendix to ANNEX 4: Note on the Statistical Capacity Indicator30 1. The Statistical Capacity Indicator (SCI) provides an overview of the statistical capacity of a country and is available for over 140 developing countries. It is based on a diagnostic framework developed with a view to assessing the capacity of national statistical systems using metadata information generally available for most countries, and monitoring progress in statistical capacity building over time. The framework has three dimensions: statistical methodology; source data; and periodicity and timeliness. For each dimension, a country is scored against specific criteria, using information available from the World Bank, IMF, UN, UNESCO, and WHO. This multi-dimensional approach is based on the notion that producing and disseminating reliable, relevant and timely statistics requires a certain level of capacity in all dimensions. Any imbalance would point to weaknesses in some aspects of the statistical process. 2. A composite score for each dimension and an overall score combining all three dimensions are derived for each country on a scale of 0 to 100. A score of 100 indicates that the country meets all the criteria. The first dimension, statistical methodology, measures a country’s ability to adhere to internationally recommended standards and methods. This aspect is captured by assessing guidelines and procedures used to compile macroeconomic statistics, and social data reporting and estimation practices. Countries are evaluated against a set of criteria such as use of an updated national accounts base year, use of the latest BOP manual, external debt reporting status, subscription to IMF’s Special Data Dissemination Standard, and enrolment data reporting to UNESCO. Known as the Practice sub-score in the overall SCI, this component is based on 10 indicators with each carrying the same weight of 10 yielding a maximum score of 100. Table IV.7: Statistical Methodology Max. Indicators 1 0 Weight score 1. National accounts base year Within last 10 years or annual chain Otherwise 1 10 2. Balance of payments manual in linking Balance of Payments Manual, the Otherwise 1 10 use fifth edition 3. External debt reporting status Actual or preliminary Otherwise 1 10 4. Consumer Price Index base year Within last 10 years or annual chain Otherwise 1 10 5. Industrial production index linking and available from IMF Produced Otherwise 1 10 6. Import/export prices Produced and available from IMF Otherwise 1 10 7. Government finance accounting Consolidated central government concept Otherwise 1 10 accounts 8. Enrolment reporting to UNESCO Annual or missed reporting only once in the last 4 years Otherwise 1 10 9. Vaccine reporting to WHO Nationally reported data on measles vaccine coverage consistent with Otherwise 1 10 WHO estimates 10. IMF’s Special Data Subscribed Otherwise 1 10 Dissemination Standard Maximum total score is 100 30 Individual countries’ scores can be viewed online from the Bulletin Board on Statistical Capacity: http://www.worldbank.org/data/bbsc 172 3. The second dimension, source data, reflects whether a country conducts data collection activities in line with internationally recommended periodicity, and whether data from administrative systems are available and reliable for statistical estimation purposes. Specifically, the criteria used are the periodicity of population and agricultural censuses, the periodicity of poverty and health related surveys, and completeness of vital registration system coverage. The Collection sub-score is based on only 5 indicators with each single source data indicator having a weight of 20, again yielding a maximum score of 100. Table IV.8: Source Data Max. Indicators 1 1/2 0 score Weight 1. Periodicity of population census <_10 years Otherwise 1 20 2. Periodicity of agricultural census <_10 years Otherwise 1 20 3. Periodicity of poverty related surveys (IES, LSMS, <_ 3 years <_ 5years Otherwise 1 20 etc.) 4. Periodicity of health related surveys (DHS, MICS, <_ 3 years <_ 5years Otherwise 1 20 Priority 5. survey, etc.) Completeness of vital registration system Complete Otherwise 1 20 Maximum total score is 100 4. The third dimension, periodicity and timeliness, looks at the availability and periodicity of key socioeconomic indicators, of which nine are MDG indicators. This dimension attempts to measure the extent to which data are made accessible to users through transformation of source data into timely statistical outputs. Criteria used include indicators on income poverty, child and maternal health, HIV/AIDS, primary completion, gender equality, access to water and GDP growth. Like the Collection sub-score, the Availability sub-score is based on 10 indicators with each carrying the same weight of 10 yielding a maximum score of 100. 173 Table IV.9: Periodicity and Timeliness Max. Indicators 1 2/3 1/2 1/3 0 Weight score 1. Periodicity of ≤ 3 years ≤ 5 years > 5 years Not income poverty available/ 1 10 indicator accessible 2. Periodicity of ≤ 3 years ≤ 5years > 5 years Not child available/ 1 10 malnutrition accessible indicator 3. Periodicity of National or Not child mortality international available/ 1 10 indicator estimates accessible 4. Periodicity of available Annual Not annual/ available/ accessible Immunization 1 10 indicator 5. HIV/AIDS National or Not available/ accessible indicator international estimates available for 1 10 at least one year out of the last 3 years 6. Periodicity of ≤ 3 years ≤ 5 years > 5 years Not maternal health available/ indicator accessible 1 10 7. Periodicity of Observed for Observed for at Observed for Not gender equality at least 5 out least 3 out of 5 1 out of 5 available/ in education of 5 latest latest years latest years accessible 1 10 indicator years 8. Primary Observed for Observed for at Observed for Not completion at least 5 out least 3 out of 5 1 out of 5 available/ 1 10 indicator of 5 latest latest years latest years accessible 9. Access to years Observed for Observed for 1 Not water indicator 2 out of 6 out of 6 latest available/ 1 10 latest years years accessible 10. Periodicity of Annual ≤ 1.5 years > 1.5 years Not GDP growth available/ 1 10 indicator accessible Maximum total score is 100 174 5. By the nature of how it is computed, the SCI can move substantially from one year to the next. As an illustration, while the SCI has not changed very much in the last 10 years for Ethiopia (Table IV.10 below), the index experienced a significant improvement in the 2008-10 period reaching 80 out of a maximum of 100; that ranked Ethiopia as top among the 8 pilot countries at that time. The SCI dropped precipitously in 2011-12 to go back to the level prevailing in 2004, on account of several developments including the fact that the 2000 base year for national accounts went beyond the 10 year mark in 2011, causing a 10 point drop in the Statistical Methodology (Practice) sub-score. Furthermore, the Government of Ethiopia went from consolidated accounts capturing all the central government’s fiscal activities -- which were introduced in 2006 -- to a more restrictive ‘central government accounts’ accounting definition, causing the Collection sub-score to decline from 80 in 2010 to 70 in 2011. In the case of Senegal, not having an agriculture census conducted in the last 10 years caused the Collection sub-score to decline by 20 points in 2011, since that sub-score relies on only 5 indicators. 175 Table IV.10: Statistical Capacity Indicator by Sub-Component of SRF Pilot Countries (1999-2013) 1999 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Afghanistan Overall score 17 24 31 32 34 46 47 52 51 53 53 Practice sub-score 0 0 20 20 20 40 30 40 40 50 50 Collection sub-score 0 20 20 20 20 30 40 40 40 40 40 Availability sub-score 50 53 53 57 63 67 70 77 73 70 70 DRC Overall score 39 32 41 38 38 38 34 36 43 43 50 Practice sub-score 40 40 50 40 40 30 20 20 30 30 40 Collection sub-score 20 10 20 20 20 30 30 30 30 30 30 Availability sub-score 57 47 53 53 53 53 53 57 70 70 80 Ethiopia Overall score 60 67 68 64 68 79 80 80 73 66 67 Practice sub-score 40 40 40 50 60 70 70 70 60 40 40 Collection sub-score 70 80 80 60 60 80 80 80 70 70 70 Availability sub-score 70 80 83 83 83 87 90 90 90 87 90 Ghana Overall score 50 51 53 56 56 59 59 66 62 59 62 Practice sub-score 30 30 30 30 30 40 40 50 50 50 60 Collection sub-score 40 40 40 50 50 50 50 50 40 40 40 Availability sub-score 80 83 90 87 87 87 87 97 97 87 87 Lao PDR Overall score 52 69 70 70 69 68 68 70 66 70 72 Practice sub-score 30 50 50 50 50 50 40 40 50 50 60 Collection sub-score 50 70 70 70 70 70 80 80 60 70 70 Availability sub-score 77 87 90 90 87 83 83 90 87 90 87 Nigeria Overall score 49 44 50 48 60 68 67 69 73 74 72 Practice sub-score 20 20 40 30 40 40 40 40 50 50 50 Collection sub-score 60 40 40 40 60 80 80 80 80 80 80 Availability sub-score 67 73 70 73 80 83 80 87 90 93 87 Rwanda Overall score 49 61 63 64 67 67 74 68 68 77 77 Practice sub-score 30 40 40 50 60 60 60 40 40 60 60 Collection sub-score 50 60 60 60 60 60 80 80 80 80 80 Availability sub-score 67 83 90 83 80 80 83 83 83 90 90 Senegal Overall score 68 79 77 73 78 72 70 73 67 63 71 Practice sub-score 60 60 60 60 70 60 50 60 60 50 50 Collection sub-score 60 80 80 70 70 70 70 70 50 50 70 Availability sub-score 83 97 90 90 93 87 90 90 90 90 93 Source: World Bank. 6. To minimize the impact of the year-on-year variations, using three-year averages may be a better way to look at progress over time, as shown in Table IV.11 below. Starting with the year 2004, which happens to coincide with the Marrakech Action Plan for Statistics (MAPS), most 176 SRF countries (or rather, most of the countries which became SRF countries) have indeed advanced in terms of statistical capacity, and have done so (much) more than the average country. With the exception of Lao PDR and Senegal, both of which started from surprisingly high figures and which have experienced stagnation and moderate decline, respectively, between 2004-06 and 2011-13, other SRF pilot countries have experienced rates of progress that are very much higher than the average rate of statistical capacity increase of IDA countries. While it is premature to draw any conclusion from this trend, the rapid increase of the SCI in Afghanistan and Nigeria is noteworthy. Table IV.11: Statistical Capacity Indicator, 3-year Moving Averages (2004-06 to 2011-13) Overall SCI 2004-2006 average 2011-2013 average Percentage increase Afghanistan 29.0 52.3 80.3 DRC 37.0 45.3 22.4 Ethiopia 65.3 68.7 5.2 Ghana 53.3 61.0 14.4 Lao PDR 69.7 69.3 -0.6 Nigeria 47.3 73.0 54.3 Rwanda 62.7 74.0 18.0 Senegal 76.3 67.0 -12.2 All BBSC 66.3 68.5 3.3 countries IDA countries 60.3 63.0 (2011-12 avg.) 4.5 177 ANNEX 5: Report on missions undertaken by the Evaluation Team 1. As part of the evaluation of the SRF-CF, Brian Ngo and Andrew Flatt travelled during the period of June 13-27 to: 1) meet with PARIS21 and OECD/Development Co-operation Department (DCD) staff on June 14 to discuss and obtain information on national development strategies for statistics in the 8 SRF pilot countries; and 2) visit Nigeria (June 16-21) and Rwanda (June 22-26) to experience first-hand the implementation of the SRF-CF projects and their impacts. In Nigeria, besides meetings with relevant stakeholders in Abuja, we also visited Minna, the capital city of Niger State which is one of the six pilot states covered by the SRF project. Arrangements were made by Alain Gaugris, Senior Statistician, in Nigeria and by Tom Bundervoet, Poverty Economist, in Rwanda. Alain and Tom attended most of the meetings and their help in facilitating the mission's visit is very much appreciated. During the country visits the mission focused its attention more on the impact of the SRF approach than on the implementation of the respective SRF projects. a) Meetings in Paris - Main Points 2. The mission met with Eric Bensel, Administrator and Samuel Blazyk, Program Coordinator in charge of Africa at PARIS21. The meeting discussed PARIS21's early support for the system-wide approach to statistical capacity building in order to facilitate the scaling up of government and donor support for statistics and using the country's national strategy for the development of statistics (NSDS) as the framework for coordination. The prerequisites for SRF - - dialogue with donors, formal request of support at a high political level, the use of the NSDS as a framework, and the in-country statistician -- are very much in line with the approach by PARIS21 towards statistical capacity building. But in spite of the very similar approaches towards statistics between PARIS21 and the SRF and the fact that the SRF Charter stipulates that the developing country co-chair of PARIS21 or a designated alternate is a member of the SRF Catalytic Fund Council, PARIS21 staff have been detached from the implementation of the SRF- CF country projects and appreciated the opportunity to discuss the objectives of and approaches to the evaluation of the SRF pilot phase with the mission. 3. The meeting reviewed the status of statistical capacity building and the NSDS in the 8 pilot countries. Generally speaking, with possibly the exception of the Democratic Republic of Congo, the NSDS has contributed to raising the profile of statistics in these countries. The meeting touched upon the selection of countries for the SRF pilot phase, including the targeting of some fragile states, without arriving at any conclusion. 4. The meeting then discussed the Partner Report on Support to Statistics (PRESS), which is conducted annually to estimate financial commitments to statistical development worldwide. PARIS21 colleagues reiterated that while the PRESS is a useful tool, it should not be taken as a pure accounting exercise (indeed, a small element of double-counting is present) but is intended primarily to serve as a tool to facilitate collaboration and co-ordination among developing countries and providers of development co-operation. PARIS21 staff explained the sources of information, of which the main one is the OECD’s Creditor Reporting System. The latest PRESS report (2012) shows that financial commitments to statistical development for the period 2010–12 amounted to roughly US$ 2.3 billion with Africa receiving a little less than h alf (46%) of total statistical support. The other important conclusions of the report are 1) that 178 support to statistics continues to be highly concentrated both in terms of recipients, with the top 15 countries accounting for almost 55% of total commitments, and in terms of donors, with four providers (including the Bank) furnishing 73% of total support; 2) that little relationship can be drawn from the volume of support and the recipient's statistical capacity; and 3) that fragile states receive considerable attention. PARIS21 staff also mentioned efforts to collect financial commitments at country level through the Country Report on Support to Statistics (CRESS) for which responses have been rather weak. The mission decided, as a result, to make some changes to the draft questionnaire to be administered to stakeholders. 5. The mission also met with Rudolphe Petras, Policy Analyst at the OECD Development Co-operation Directorate to discuss the aid management platform in some of the pilot countries and possible relevance to statistical development support. Two of the 8 SRF pilot phase countries (Lao PDR and Senegal) have issued aid reports while two others (Ethiopia and DRC) have begun to collect the information. Mr. Petras suggested that the mission contact the Washington-based Development Gateway for more detailed information on the pilot phase countries. b) Meetings in Nigeria 6. With Alain's efficient help, the mission was able to meet with most of the relevant stakeholders both in Abuja (for the federal level) and in Minna (for the State level). In Abuja, the mission held separate discussions with the project coordinators representing the National Bureau of Statistics (NBS) and the National Population Commission (NPopC) which jointly ensure the implementation of the SRF project, other representatives of these two institutions including the Statistician-General of the NBS, some of the local donors (DfID, UNAids, UKAid/ESSPIN on Education, and AfDB), and a representative of the local media. At the end of the visit to Abuja, the mission met with most of the members of the SRF Technical Committee led by the Ministry of Finance and including representatives of the National Planning Commission, the Central Bank of Nigeria, the Accountant General as well as of the National Bureau of Statistics and the National Population Commission. The representative of the Nigerian Statistical Association was absent. The mission also had a separate discussion with Ms. Esther Forgan of DfID on the role of the in-country donor statistician. 7. The mission also met with the World Bank's Country Director who shows a very strong support for statistics and has empowered Alain to play an active role in decision meetings on various WB projects in Nigeria with a focus on data. Ms. Marie-Françoise Marie-Nelly, Country Director for Nigeria, highlighted the importance of statistics as a key sector in the country assistance strategy and that the SRF has helped raise the profile of statistics. She also mentioned that the development partners group in Statistics is performing best among the various sectors in Nigeria. From her perspective, the challenges are 1) where do we start (including how best to position the SRF in order to achieve concrete results); 2) how to sequence statistical capacity building (mentioning that the Bank is preparing a larger IDA project to support statistics in Nigeria); and 3) what does it take to build a strong technical base at NBS. She also suggested that the mission brief her and other country directors/managers on the findings and recommendations before finalizing the evaluation report. The mission took note of the request but thinks that, in view of the short deadline for the completion of the report; there may not be sufficient time for the briefing as requested. The mission also met with the Lead Economist and briefed other WB colleagues on the evaluation mission at the weekly Bank staff meeting. 179 8. In Minna, the mission was able to get a first-hand exposure to the workings of statistical development at the State level. Following a thorough briefing by the State Statistician-General on recent developments in statistics in Niger State (new publications, a separate budget for the State Bureau of Statistics which was formerly included under Planning, and a huge increase in funding from Naira 1 to 52 million), the mission met with the State Consultative Committee on Statistics (SCCS) including its chairman, various board members, University researchers, and the local media to go over recent developments in statistics in Niger State. Subsequently, the mission met with the permanent secretaries of the Planning Commission and the Ministry of Education and witnessed first-hand the close and constructive relationships between the Statistician-General and senior managers at other State-level ministries on their respective roles in promoting both the reliance on and concerns for timely and relevant data for economic planning and its implementation. The mission was also shown numerous pieces of furniture and equipment that had been purchased with SRF project funds, including some cartographic equipment for the State NPopC that was literally still under wraps. 9. One positive effect of SRF funding was that the NBS and NPopC, reportedly previously at loggerheads, were now working harmoniously together on the project. The mission did however hear concerns that this cooperation could be threatened by recent changes in the leadership of NPopC. c) Meetings in Rwanda 10. The mission met separately with the Director-General of the National Institute of Statistics of Rwanda (NISR) and subsequently with several of the NISR senior staff on the impacts of the SRF project. What transpired from the focused discussions with the NISR was: 1) although Rwanda was already implementing the NSDS (2010-14), the SRF has played a significant role in helping to give statistics a more powerful and relevant role in economic and social management in Rwanda by providing significant additional resources and by helping to attract other resources to statistics; 2) differently from earlier World Bank projects that focused on capacity building, the SRF had a direct impact on the production of data which was considered to be critical in raising the profile of statistics in Rwanda; and 3) there was a sharp acceleration in the use of statistics in Rwanda although it is hard to attribute this development to the SRF. 11. The mission had detailed discussions on the workings of the 'basket fund' for Statistics which currently includes DfID, the European Commission, and the World Bank. The fact that the Bank joined the basket fund as a latecomer while implementation of the NSDS and particularly the population census was already well advanced meant that procurement procedures and clearance for funding by the Bank had caused problems at the beginning of the SRF project implementation. Delays in implementation were avoided thanks to the basket fund arrangements which make it possible to move funding by the various members of the basket fund to cover funding gaps. It was highlighted to the mission that the rapid turn-over of the WB resident project officer was also a major stumbling block. Subsequent discussions with the World Bank's Country Manager, Ms. Carolyn Turk, pointed to the need for more clarity, in the early phase of the Rwanda project implementation, in supervision assignment and resources. 12. The mission met with other producers of data such as the National Identification Agency (in charge of the issuance of IDs, driver's license and of civil registration), the National Bank of 180 Rwanda, and the Rwanda Development Board (in charge of the promotion of investments) and received confirmation of the strong coordination within the government of Rwanda on matters related to statistics (for example, agencies need clearance from the NISR before conducting a survey). Meetings with the Institute of Policy Analysis and Research of Rwanda (IPAR), the Kigali Institute of Science and Technology (KIST) and the Managing Director of New Times, a major local newspaper, again confirmed the strong relationships between producers and users of data in Rwanda although there remain challenges as to the timeliness of data in some areas and weak capacity in comprehending statistics by the local media. There was however, unanimous recognition that the NISR website has significantly improved in the last year. Currently, in addition to economic and sectoral data, the website also allows for the online retrieval of micro data as well as a comprehensive literature on techniques and methods (metadata). 13. The mission also met separately with representatives of some of the development partners involved in Statistics namely the European Union, UNFPA which represents the One-UN in Rwanda, and the Head of Development Cooperation at the Embassy of the Netherlands. Arrangements are being made to reach the DfID representative in East Kilbride later this week or early next week. The discussions revealed a number of useful insights. First, the EC/DfID had conducted an institutional assessment of NISR in 2010/11 which highlights the very strong achievement by the latter with a limited budget. 14. Second, in relation to the in-country donor statistician, the mission was informed that the technical advisor to the DG of NISR has recently left the country and his replacement is being advertised. However, it was not clear to the mission that this person has played the role of the in- country statistician particularly since the DG of NISR mentioned that he didn’t need or want an in-country donor statistician. On the use of consultants, the former approach by UNDP to bring in West African consultants in Statistics was not considered satisfactory by the NISR. Currently, faced with the challenge of producing credible data on a timely basis, the NISR relies rather heavily on Oxford Policy Management (OPM) consultants for a wide array of specialized tasks. Concerns were expressed by some of the people we met about the overreliance on OPM and the lack of hands-on training for local NISR staff, many of whom were recruited in the past 2 years and thus have little experience. 181 ANNEX 6: List of persons met and contacted * Contacted by telephone World Bank, Washington DC (May 21-24 2013) Development Economics Data Group (DECDG) Grant Cameron Manager Barbro Hexeberg Senior Economist, SRF Project Coordinator Ghislaine Delaine Senior Statistician, TTL for DR Congo John Ngwafon Senior Economist/Statistician, TTL for Ghana and Nigeria Morgan Brannon Consultant Annette Kinitz* Consultant Christelle Kouame Program Assistant Lisa M Burke Program Assistant Other World Bank staff contacted in Washington Waleed Malik Senior Public Sector Management Specialist, TTL for Ethiopia and Rwanda Chris Rockmore Economist, former TTL for DRC Congo World Bank and DfID staff contacted in the field (various dates) Afghanistan Aphichoke Kotikula* Economist, TTL Jill Fletcher* DFID Statistics Adviser, IDS (outgoing) Omar Joya* Economist, World Bank Country Office DR Congo Abdoullah Beidou* Statistician, DECDG (IDS) Ghana Lynne Henderson* Statistician, DECDG (IDS) Lao PDR Somneuk Davading* Economist, TTL Senegal Mamadou Ndione* Senior Economist, former TTL Djibril Ndoye* Poverty Economist, TTL PARIS21 Secretariat, Paris (June 14 2013) Eric Bensel Partnership and Studies Programme Coordinator Samuel Blazyk Programme Coordinator, NSDS in Africa 182 OECD, Paris (June 14 2013) Rudolphe Petras Policy Analyst, Development Co-operation Directorate Nigeria (June 17-21 2013) Abuja World Bank Marie-Françoise Marie-Nelly (Ms) Country Director John Litwack* Lead Economist Alain Gaugris Senior Statistician, DECDG (IDS and co-TTL) Kabiru Mohammed Team Assistant National Bureau of Statistics Yemi Kale* Statistician General Simon Harry SRF Co-Project Coordinator and Assistant Director, Corporate Planning and Technical Coordination Department Samanja Maudo Deputy Director Peter Mbamo Senior Statistician National Population Commission Alfa Mohammed SRF Co-Project Coordinator and Director, Demographic Statistics Babagana Wakil Project Manager/Deputy Director, Vital Statistics Abubakar Madaki Project Assistant Federal Ministry of Finance Ihechukwu Madubuike A.A. (Mrs) Secretary, Technical Committee for the Implementation of the SFR (SRF Technical Committee), Director, International Economic Relations Department, Federal Ministry of Finance Pius Ailoyafen Assistant Director (Member, SRF Technical Committee) Suleiman Kolo Principal Administrative Officer (Member, SRF Technical Committee) Central Bank of Nigeria Ibrahim Adamu Member, SRF Technical Committee Mainasara S. Member, SRF Technical Committee Office of Accountant General of the Federation Onojeta Solomon Deputy Director (Member, SRF Technical Committee) Local media National Mirror Tola Akinmutimi, Assistant Editor (Business) Development partners Esther Forgan Results and Statistical Adviser, DfID Kingsley Ogbonna Monitoring and Evaluation Coordinator South, ESSPIN-DfID Ifeanyi Ugwuoke Niger State Programme Manager, SPARC-DfID Masauso Nzima Senior Strategic Information Advisor, UNAIDS 183 Andoh Mensah Acting Chief Country Economist, African Development Bank Niger State: Minna Abubakar Aliyu Permanent Secretary, Niger State Planning Commission Alhaji Ahmadu Umaru Chairman, Board of Directors, Niger State Bureau of Statistics (NSBS) Dr. Audu Isan Statistics Unit of Polytechnic , NSBS Board Member Abdu Idris NSBS Board Member Usman Liman Statistician General, NSBS Abdul Gamyn Bells NSBS Abubakar Garba NSBS Hassan A. Mohd NSBS Ahmed Sam NSBS Naibi Adamu NSBS Mohammed Am NSBS Dr. Mohammed N. Mahmud Permanent Secretary, Niger State Ministry of Education Muhammed Idrisu Guregi Head of Department, Administration, Niger State Office, National Population Commission Muhammed Shehu Yisako Head of Department, Technical, NPoPC Nma Alhaji Shehu Head of Department, Public Administration, NPoPC Muhammad Aliyu Mahmood GIS Lab Operator, NPoPC Abu Mbodi Leadership (newspaper) Rwanda (June 24-26, 2013) World Bank Carolyn Turk Country Coordinator Tom Bundervoet Poverty Economist Yoichiro Ishihara Senior Economist Antoinette Kamanzi Procurement Assistant Sylvie Ingabire Team Assistant National Institute of Statistics of Rwanda Yusuf Murangwa Director General Odette Mbabazi Deputy Director General of Corporate Services Yahya Hassani Coordinator of the NSDS Coordination Team Dominique Habimana Director of Statistical Methods, Research and Publications Department Antoinette Habinshuti Planning Officer National Bank of Rwanda Willybrold Nizeyimana Manager of Monetary Statistics Jean Chrysostome Rwililiza Manager, Economic Statistics & MIS Database, Statistics Department National ID Agency Pascal Nyamulinda Director General 184 Rwanda Development Board Zephanie Niyonkuru Acting Head of Planning Division Other national users Institute for Policy Analysis and Research of Rwanda Pamela Abbott Director of Research Unit Dickson Malunda Senior Research Fellow Kigali Institute of Science and Technology Anselme Sano Director of Planning and Development Kevin Ganza Statistician The New Times Publications Collin Haba Managing Director Development partners EU Delegation Ruben Alba Aguilera Attaché, Economy & Governance Mugeni Kayitenkore Program Officer, Economy & Governance Netherlands Embassy Pieter Dorst Head of Development Cooperation, Netherlands Embassy Gaspard Ndagijimana Regional Adviser Water and Land, Netherlands Embassy UN System Bolaji Taiwo Chief Technical Adviser, UNFPA United Kingdom DfID, East Kilbride (July 24 2013) Frances Harper Team Leader, Global Statistics Partnership Department Kim Bradford-Smith* Senior Statistics Adviser, IDS for Afghanistan 185 ANNEX 7: References SRF-CF Administration Unit SRF-CF Administration Unit, Statistics for Results Facility-Catalytic Fund, Charter, July 2009. SRF-CF Administration Unit, Statistics for Results Facility-Catalytic Fund, Guidelines and Procedures for Country Implementing Agencies, January 2010. SRF-CF Administration Unit, Statistics for Results Facility-Catalytic Fund, Guidelines on Terms of Reference for Lead Donors and In-Country Donor Statisticians, February 2010. SRF-CF Administration Unit, Statistics for Results Facility-Catalytic Fund, Monitoring and Evaluation Arrangements, March 2010. SRF-CF Administration Unit, Statistics for Results Facility-Catalytic Fund, User Satisfaction Survey on the Quality and Use of Official Statistics for Informed Policy- and Decision- Making – Guidelines, June 2010. SRF-CF Administration Unit, Statistics for Results Facility-Catalytic Fund, Status Update as of August 2013. The World Bank, Note on the Statistical Capacity Indicator (no date). The World Bank, Statistics for Results Facility-Catalytic Fund, Annual Report, April 2010, SRF-CF Administration Unit, Washington DC. The World Bank, Statistics for Results Facility-Catalytic Fund, Annual Report, April 2011, SRF-CF Administration Unit, Washington DC. The World Bank, Statistics for Results Facility-Catalytic Fund, Annual Report, April 2012, SRF-CF Administration Unit, Washington DC. The World Bank, Statistics for Results Facility-Catalytic Fund, Annual Report, May 2013, SRF- CF Administration Unit, Washington DC. Afghanistan Central Statistics Organization, Application of the Central Statistics Organization of the Government of Islamic Republic of Afghanistan for the Statistics for Results Facility (SRF), February 2010. Central Statistics Organization, User Satisfaction Survey - Initial Results, October 2012. Central Statistics Organization, Capacity Development Plan (2011-2014), June 2011. Joint Inter-Agency Mission, Afghanistan: A Statistical Master Plan, July 2004. The World Bank, Project Appraisal Document On a Proposed Statistics For Results Facility Grant In The Amount Of US$ 14 Million To The Islamic Republic Of Afghanistan For Strengthening The National Statistical System Project, October2010. The World Bank, Implementation Status & Results (ISR). Afghanistan: Strengthening the National Statistical System (P121883), various dates. 186 The World Bank, Statistics for Results Facility of Central Statistics Organization of the Islamic Republic of Afghanistan: Summary Report, January 2013 Democratic Republic of Congo National Statistical Institute, Roadmap for the Elaboration of a National Strategy for the Development Statistics in the Democratic Republic of Congo (Feuille de route pour l’élaboration de la stratégie nationale de Développement de la statistique en République Démocratique du Congo), November 2005. National Statistical Institute, Proposed Grant Request for the Statistics for Results Catalytic Fund, August 2010. The World Bank, Project Appraisal Document on a Proposed Grant in the Amount of US$ 11.8 Million to The Democratic Republic of The Congo for the Catalytic Project to Strengthen the National Statistical Institute, August 2013. Ethiopia Central Statistical Agency, National Strategy for the Development of Statistics (2009-14). Central Statistical Agency, Application of the Central Statistical Agency for the Statistics for Results Facility, September 2012. The World Bank, Draft Project Appraisal Document on a Proposed Grant in the Amount of US$ 10 Million to the Federal Democratic Republic of Ethiopia for a Statistics for Results Project, August 2013. Ghana Ghana Statistical Service, Ghana Grant Application for the Statistics for Results Facility Catalytic Fund, December 2009. Ghana Statistical Service, the Satisfaction of Users of Official Statistics in Ghana, December 2012 Government of Ghana, Ghana Statistics Development Plan, 2009-2013, November 2008. The World Bank, Project Appraisal Document on a Proposed IDA Credit in the Amount of SDR 18.8 Million (US$ 30 Million Equivalent) and a Proposed Grant from the Statistics for Results Facility Catalytic Fund in the Amount of US$ 10 million to the Republic of Ghana for the Statistics Development Project, July 2011. The World Bank, Implementation Status & Results (ISR). Ghana: Ghana Statistics Development Program (P 118858), various dates. Lao PDR Lao Department of Statistics, Ministry of Planning and Investment, The Strategy for the Development of National Statistical System 2010-2020, May 2010 Lao People’s Democratic Republic, National Assembly, Statistics Law, June 2010 (unofficial translation). 187 Lao Statistics Bureau, Lao PDR Grant Application for the Statistics for Results Facility Catalytic Fund, June 2011. National Statistics Center, Strategies and Measures for the Official Statistical System Development of the Lao PDR 2006-2010, 2005. The World Bank, Project Appraisal Document On A Proposed Statistics For Results Facility Grant In The Amount Of US$ 8,000,000 To The Lao People’s Democratic Republic For Strengthening The National Statistical System Project, March 2013. The World Bank, Implementation Status & Results (ISR), Lao PDR: LAOSTAT – Strengthening the National Statistical System Project (P129825), July 2013 Nigeria Federal Republic of Nigeria, Statistical Master Plan for the Nigeria National Statistical System 2004/5-2008/9. Federal Republic of Nigeria, Strategy for National Development of Statistics, 2008/09 - 2012/13. International Development Association, Country Partnership Strategy for the Federal Republic of Nigeria (2010-13), July 2, 2009. National Bureau of Statistics, Nigeria - Grant Application for the Statistics for Results Facility Catalytic Fund (SRF-CF), December 2009. National Bureau of Statistics and the World Bank, Nigeria Statistical Development Project Statistics for Results Facility (SRF). Report of User Satisfaction Baseline Survey – National; States of Anambra, Bauchi, Edo, Niger, Ondo and Kaduna (November 2012). National Bureau of Statistics, Corporate Strategy Implementation Plan 2013-2014, 2012. Niger State, Vision 3:2020 (no date). Niger State Bureau of Statistics, Niger State Statistical Yearbook 2011 Edition. The World Bank, Nigeria Office, Nigeria Economic Report, May 2013. The World Bank, Nigeria Office, Minutes of Nigeria Development Partners’ Statistics, Monitoring and Evaluation Group Meeting, various dates. The World Bank, Nigeria Office, Terms of Reference of the Nigeria Development Partners’ Statistics, Monitoring and Evaluation Group, February 2012 The World Bank, Project Appraisal Document on a Proposed Grant from the Statistics for Results Facility Catalytic Fund in the Amount of US$ 10 million to The Federal Republic of Nigeria for a Statistical Development Strategy Implementation Project, April 2011. The World Bank, Aide Memoire: Nigeria Implementation Support Mission for Statistics for Results (SRF) Project (TF 099783), various dates. The World Bank, Nigeria Office, Note to the file - Round of visits to development partners to facilitate and strengthen effective donor coordination mechanisms in the field of statistical development, May 2013. 188 The World Bank, Implementation Status & Results (ISR), Nigeria: Nigeria Statistics Development Program (NSDP) (P119872), various dates. Rwanda Abbott, Pamela and John Rwirahira, Millennium Development Goals Progress Report, Rwanda Country Report 2010, Institute of Policy Analysis and Research of Rwanda. United Nations Development Programme 2010. Edmunds, Roger, Mid-Term Review of the National Strategy for the Development of Statistics, National Institute of Statistics of Rwanda, August 2012. Ministry of Finance and Economic Planning (MINECOFIN), Memorandum of Understanding For NSDS1 Basket Fund between the Government of Rwanda Represented by the Ministry Of Finance and Economic Planning and Development Partners Contributing to the Basket Fund to Support Implementation of the National Strategies for the Development of Statistics, 2009-2014, October 2010. Ministry of Finance and Economic Planning (MINECOFIN), Memorandum of Understanding For NSDS1 Basket Fund between the Government of Rwanda Represented by the Ministry Of Finance and Economic Planning and Development Partners Contributing to the Basket Fund to Support Implementation of the National Strategies for the Development of Statistics, 2009-2014 – Addendum for the World Bank, July 2012. National Institute of Statistics of Rwanda, NISR Strategic Plan 2007-2011, March 2007. National Institute of Statistics of Rwanda, National Strategy for the Development of Statistics 2009-2014. National Institute of Statistics of Rwanda, Integrated Annual Work Plan: July 2013-June 2014. National Institute of Statistics of Rwanda, NSDS Steering Committee/National Partnership Group for Statistics, Annual Report July 2011-June 2012, June 2012. National Institute of Statistics of Rwanda, NSDS Steering Committee/National Partnership Group for Statistics, Mid-Year Report July-December 2012, December 2012. National Institute of Statistics of Rwanda, NSDS Steering Committee/National Partnership Group for Statistics, Annual Report July 2012-June 2013, June 2013. National Institute of Statistics of Rwanda, Statistics for Results Facility Catalytic Fund, Rwanda’s Grant Application, January 2010. National Institute of Statistics of Rwanda, User Satisfaction Survey on the Quality and Use of Official Statistics for Informed Policy and Decision-Making in Rwanda, February 2013. National Institute of Statistics of Rwanda, Capacity Building Strategy for Rwanda National Statistical System 2013-2018, April 2013. NSDS Steering Committee, Minutes of NSDS Steering Committee Meetings, various dates. Office of the Auditor General of State Finances, Final Audit Reports for the Year Ended 30 June 2012, March 2013. 189 The Republic of Rwanda, Economic Development and Poverty Reduction Strategy 2008-2012, September 2007. The World Bank, Project Appraisal Document on a Proposed Grant in the Amount of US$ 10 Million to the Republic of Rwanda for the Statistics for Results Project, December 2011. The World Bank, Implementation Status & Results (ISR), Rwanda: Statistics for Results Facility (P124629), various dates. The World Bank, Aide Memoire: Rwanda – Statistics for Results Project (P124629), Implementation Support Mission February 2013 The World Bank, Rwanda Economic Update: Maintaining Momentum, May 2013 Senegal National Agency for Statistics and Demography, National Strategy for the Development of Statistics 2008-13 (Schéma Directeur de la Statistique du Sénégal - Stratégie Nationale de Développement de la Statistique, Rapport Principal), June 2007. National Agency for Statistics and Demography, Application for Statistics for Results Facility Catalytic Fund, April 2011. Other References African Development Bank, Situational Analysis of the Reliability of Economic Statistics in Africa: Special Focus on GDP Measurement, Statistical Capacity Building Division, Statistics Department, June 2013. Jerven, Morton, Poor Numbers. How We Are Misled by African Development Statistics and What to Do about It, Cornell University Press, Ithaca and London, 2013. Morrison, Thomas, Mid-Term Evaluation Report on the Enhanced Data Dissemination Initiative Project (covering the period April 1, 2010 to September 30, 2012), International Monetary Fund, March 2013. Oxford Policy Management, Evaluation of the Implementation of the Paris Declaration - Thematic Study of Support to Statistical Capacity Building, Synthesis Report, May 2009. Oxford Policy Management, Evaluation of the Implementation of the Paris Declaration - Thematic Study of Support to Statistical Capacity Building, Synthesis Report and Evidence Report, Part 2, May 2009. PARIS21, New NSDS Guidelines (draft for comments), fall 2013. PARIS21, Statistics for Transparency, Accountability and Results: A Busan Action Plan for Statistics, November 2011. PARIS21, PARIS21 Meeting on Bridging the User-Producer Divide: Statistical Capacity Development in a Post Busan World, February 2013, New York. PARIS21, Draft Record of the Proceedings, Annual Meetings, 9-11 April 2013, Paris, France. PARIS21 Secretariat, A Guide to Using a System-wide Approach to Implement National Strategies for the Development of Statistics, October 2007. 190 PARIS21, A Scoring System to Measure the Use of Statistics in the Policy-Making Process - Summary Report, September 2012. PARIS21 Secretariat, Partner Report on Support to Statistics – PRESS 2012 Round, Final Report, December 2012. PARIS21 Secretariat, NSDS Status in IDA and Lower Middle Income Countries - Progress Report as of November 2013. 191 ANNEX 8: Terms of Reference for the Evaluation of the SRF-CF Pilot Phase Overview of the SRF-CF Background 1. Improving the ability of countries, agencies, and institutions to manage for results is central to the international community’s aim to support country outcomes such as the Millennium Development Goals (MDGs). The Managing for Development Results (MfDR) process – often termed the “results agenda” – has emphasized the importance of better statistical data for highlighting issues, making policy choices, allocating resources, monitoring outcomes and evaluating impact. 2. The Statistics for Results Facility (SRF), which began its pilot phase in 2009, with support from UK/DFID and the Netherlands, is a multi-donor initiative designed to increase the level of investment in statistical systems in developing countries and to improve the effectiveness of financial and technical assistance. The objective of the SRF and of the Catalytic Fund (SRF-CF) is to support capacity building, and better policy formulation and decision making through a sustained improvement in the production, availability and use of official statistics in these countries. 3. The SRF is part of a long-term process to improve the effectiveness of development aid and a key part of (MfDR). The SRF builds on substantial progress that has been achieved since 2004, when delegates to the Second International Roundtable on MfDR set forth the Marrakech Action Plan for Statistics (MAPS). The MAPS plan identified six key actions to help develop and strengthen national statistical systems and international coordination. Recognizing that long- term and sustained improvements in national statistical systems could be achieved only if countries themselves took the lead in setting priorities and in managing change, MAPS focused on helping countries prepare National Strategies for the Development of Statistics (NSDS). 4. The SRF was specifically designed to support the implementation of NSDSs. However, recognizing that the challenge then (in 2010), was to move from the preparation of plans to their implementation, focus was immediately turned to preparation of plans for implementation. The SRF-CF was also designed to promote and support efforts to improve the effectiveness of aid, as well as providing an increase in the financial resources for investment in statistics, specifically by applying the principles of the Paris Declaration on Aid Effectiveness to statistical projects and programs. The SRF was expected to: a) Ensure that developing countries exercise effective ownership and leadership in developing their national statistical systems and that donors respect this leadership; b) Promote the alignment of donor support for statistics in the context of an effective strategic framework and work program for national statistical development (NSDSs or similar); c) Harmonize donor support through well-planned and well-coordinated projects and programs; 192 d) Support developing country efforts to manage for results not only by generating better statistical data and indicators but also by ensuring that these outputs are used to support evidence-based decision making; and e) Promote mutual accountability by monitoring the implementation and outcomes of NSDSs within a framework of mutual accountability, recognizing that developing a sustainable statistical system may take a long time. 5. As part of this process, Paris Declaration partners agreed on the indicators and targets that would be used to monitor the delivery of inputs, the production of outputs, the achievement of wider outcomes, the impact of better statistics and a more effective statistical system. Agreement was also made on the targets that were to be achieved and what action needed to be taken when implementation was not on track. 6. The SRF promotes a comprehensive system-wide approach to strengthening national statistical systems. This approach draws on lessons learned from the sector-wide approaches used successfully in sectors such as health, education and agriculture as well as a number of system-wide statistical capacity building operations undertaken by the Bank. In particular it stresses the provision of support to a locally owned and led program to develop the national statistical system in a comprehensive and coordinated way, using country systems and procedures for implementation. A system-wide approach also focuses on strategic issues and coordination, promotes dialogue across the national statistical system and the involvement of line ministries, allows more scope for country leadership and ownership, helps to harmonize approaches and align donor support with country policies and provides for the more effective scaling up of support while optimizing capacity building. The process of preparing a statistical strategy covering the whole national statistical system, or at least its most important parts, provides a framework within which the system-wide approach can deliver coordinated and harmonized financial and technical assistance. 7. The requirements for developing a system-wide approach at the country level for SRF funding include: a) A well-prepared statistical strategy (NSDS) in place or being developed, with high-level political endorsement; b) A formal request to the SRF-CF from the government seeking support for the implementation of the NSDS and proposing the use of a system-wide approach; c) Identification of resource requirements for statistics in government budgetary allocations and national resource frameworks, such as medium-term expenditure frameworks where they exist; d) Initiation of a dialogue with local donor representatives; e) Agreement to the approach by all donors (or at least a core set) with an interest in supporting statistics, where appropriate, identification of a lead donor to assist in coordinating the dialogue with government; and f) Identification of existing initiatives in statistics and a clear understanding of how these support the implementation of the NSDS. 193 8. For these elements to be in place, capacity will be needed both in participating countries and within the development partners. Essential components for participating countries include: leadership and the political will to improve statistics; effective financial management capacity; and sufficient institutional and professional expertise. In many countries technical assistance may be needed in some or all of these areas. Statistical agencies, especially the central organization, will have to have the capacity to lead the reform process while, at the same time, being able to maintain their regular work programs. The pilot phase 9. The pilot phase of the SRF-CF was set up to provide financial and technical support to a limited number of pilot countries that had a low level of statistical capacity or had recently emerged from conflict. The experience from these countries would serve to identify what elements of the program worked well, and what require modifications to improve impact. Once the findings from the pilot countries had been evaluated, modifications to the approach would be developed, where needed. Then, the Catalytic Fund would consider inviting other countries to apply to the SRF-CF. In the meantime, other countries would be encouraged to make investments in their statistical systems in line with the SRF principles. 10. The SRF-CF Council is responsible for determining the criteria by which countries were identified for the pilot phase – both to receive grants from the SRF-CF or technical assistance. The criteria included current levels of statistical capacity and the extent of the commitment to the SRF principles. In line with the SRF Charter, the Council identified five countries that were invited to submit applications for assistance during the pilot phase. These countries were: Afghanistan, the Democratic Republic of Congo (DRC), Ghana, Nigeria, and Rwanda -- all IDA eligible countries. Pilot countries represented a mixture of different levels of statistical development, different types of statistical systems and different problems and concerns. Two of the countries – Afghanistan and DRC – were included because they were fragile states emerging from conflict and presented particular problems relating to the development of statistics. The SRF-CF Council agreed to invite Lao PDR, Senegal and Ethiopia to apply later in the pilot phase as some initial lessons had been learned in improving the application process. In light of new approaches and technologies that had emerged since SRF inception, the pilot phase was also extended to fund testing the use of innovative approaches to data collection. The Evaluation 11. According to the SRF charter, once the programs of the pilot countries were well underway, progress towards achieving results would be evaluated and assessed to learn what worked well and what elements would require modifications to increase its impact. With the evidence from the assessment, the SRF could then be more confidently expanded to include additional countries. The main purpose of the evaluation would be to assess the experience of the pilots, to identify the achievements and to recommend changes in policies and practices in order to improve results. 12. It is on this basis that the SRF administration is seeking consultants to evaluate the impact of the SRF-CF pilots, by assessing what has worked well (and less well) vis-à-vis its objectives, particularly: 194 a) How successful the SRF-CF has achieved its objectives particularly in: i) Promoting a system-wide approach in statistics at country level; ii) Increasing resources for implementing country owned National Statistical Plans; iii) Linking improvements in the statistical system to the needs of national and sectoral monitoring frameworks and promoting an improved national dialogue and partnership between statistics users and statistical producers; iv) Delivering more efficient and effective aid and technical assistance for strengthening statistical systems and results measurement, through better coordination and alignment to agreed National Statistical Plans and through better alignment to the Paris Declaration on Aid Effectiveness; and b) The impact of the in-country donor statisticians in reaching the SRF objectives; c) How the governance arrangements and operational practices and procedures are working; and d) To recommend what changes may be needed to enable the SRF-CF to reach its objective (as amended). Management of the evaluation 13. The SRF administration at the World Bank will hire 2-3 consultants as outlined below. The consultants will report to Barbro Hexeberg, SRF Project Coordinator, Development Economics Data Group (DECDG), liaise with the SRF Administration Unit and provide regular updates of progress. The Administration Unit will be responsible for providing access to relevant project documentation. Timing 14. The assignment will be carried out between May and September 30, 2013. Work Program 15. It is expected that the work program for the team will be as set out below. Activities Date Initial review of documentation, consultations and preparation May, 2013 of the inception report Preparation of instruments to be used to collect information Mid-May, 2013 Consultations with implementing agencies, donors, TTL and in- Mid-May to June, country statisticians 2013 195 Activities Date Analysis of the information collected from field trips and Late June, 2013 questionnaires etc. Drafting of the final report Summer, 2013 Presentation of Final report September, 2013 Presentation of evaluation report to SRF Council TBD 16. It is anticipated that a variety of methods will need to be employed to collect information and get feedback from grant recipients, donors, government officials involved in implementation of the project and others. The team will be responsible for the design, implementation and analysis of questionnaires, reporting forms and other instruments. The consultants will be expected to make two visits to Washington DC and also to undertake field trips to a minimum of 2 of the recipient countries and projects. The countries will be selected in collaboration with the SRF administration and the program for field trips will be agreed with the World Bank and set out in the Inception Report. The World Bank will be responsible for making arrangements for the field trips. Outputs 17. The consultant(s) will be required to prepare an inception report within 10 days of the start of the assignment setting out how the tasks will be carried out, what information will be required and what kind of information will be prepared. The inception report should also contain an outline of the final report. 18. A draft final report will be prepared after 11 weeks of work. This will be reviewed by the SRF administration, at the World Bank, who will provide comments and feedback within 14 days. The draft report will also be sent to the recipients for comments and validation of the results and findings. A final report, which will have an executive summary, be based on the discussions with recipient countries and reflect the views of those consulted, will then be provided by the consultants within a further fifteen days of work. The lead consultant may be required to make a presentation on the final report to the SRF Council. Consultants 19. The World Bank will hire suitably qualified consultants with skills/experience in the following domains:  An experienced statistician, manager of statistical capacity building projects and programs, or someone deeply familiar with statistics and national statistical systems in developing countries;  Knowledge and experience in evaluation of statistical systems; 196  Knowledge and experience in project management specialist with a background in the financial management and procurement aspects of donor funded projects and programs. 20. It is anticipated that a total amount of 45-60 working days would be needed. One member of the team will be designated team leader. Those bidding for the evaluation will be required to set out how they propose to undertake the evaluation as well as submitting a financial proposal. 21. The contract will cover fees as well as mission costs. Point(s) of contacts 22. The World Bank contact points for administrative and financial matters linked to the execution of this evaluation are: Name: Grant J. Cameron Address: World Bank 1818 H Street, NW, Washington, DC 20433, USA Telephone: + 1-202 458 0144 Fax: + 1 202 522 3645 E-mail: gcameron1@worldbank.org Name: Barbro Hexeberg Address: World Bank 1818 H Street, NW, Washington, DC 20433, USA Telephone: + 1-202 473 3733 Fax: + 1 202 522 3645 E-mail: bhexeberg@worldbank.org 197 ANNEX 9: Management’s response to recommendations Table 1 Management’s response to detailed recommendations SRF Assessment Recommendations Management’s Objectives Responses Promotion of a  Positive findings but i) One particular aspect that i) Agree. Terms of references system-wide the survey results may need further consideration are to be strengthened on this approach to suggest that is the role of the in-country point. When SRF funds are statistics implementation of such donor statistician on utilized, this point will be an approach would pose coordination. included. a moderate to ii) The World Bank should lead ii) Agree that this could be a significant challenge. by example by ensuring that it criterion for country selection. has staff in the country office suitably qualified in statistics or M&E to serve on the National Partnership Group and should encourage other development partners to do likewise. Resource  The direct effect of i) Promote a more systematic i) Agree. Encourage the mobilization the SRF on increases in exchange of information on introduction of an annual resources, both from funding intentions within ‘Country Report on Support to domestic and external national partnership groups. Statistics’ by the National sources, has so far been Partnership Group, or a similar quite modest. report on domestic and foreign funding intentions. ii) Given the role of ii) Continued efforts to funding in ensuring increase resources to statistics sustainability of the support for should be encouraged within the statistics generated by the SRF, National Partnership Group, as it is desirable for all concerned well as domestically with the parties to increase efforts to government and Bank teams to generate additional funding and include statistical operation in to actively explore additional the Country Program Strategy. resource mobilization methods, This could be a possible for example through matched criterion for SRF funding. funding arrangements. iii) Promote the practice of iii) Agree, though it should producing regular NSDS be clear that the preparation of implementation reports to help NSDS/work program in monitoring the commitment implementation progress reports of resources and their delivery. is the responsibility of the The TORs of the in-country government. TORs will be donor statisticians should strengthened on this issue, as specifically mention the necessary. provision of support to national authorities in the preparation of NSDS implementation reports. 198 iv) Basket funds as in iv) Agree, but the Rwanda are proven to be applicability of basket funds is effective for NSDS difficult in many countries. implementation and the SRF Another option that can also be should be used to encourage explored is the use of Results them taking country Based approaches. circumstances into consideration. Linking Statistical  In implementing i) Highlight the importance of i) SRF guidelines will be Systems to countries, the SRF has the use of statistics in future modified to include this aspect National played a significant role SRF projects. in all projects. Monitoring in enhancing the Frameworks ii) More research aimed at ii) Agree. To make user production, quality and improving the quality and surveys more manageable, their timeliness of data. efficacy of user satisfaction scope and sample size should be  Its impact on surveys. In the interim fully reduced as necessary. Guidelines enhancing the support the proposal by the SRF are to be revised by the end of accessibility and Administration Unit to lighten June 2014. usability of data is the surveys and to be more somewhat less but still selective on key data users. significant. iii) Opportunities iii) This will be considered. . should be taken to The most likely source of critically assess initiatives information is the user to measure the use of satisfaction survey. statistics in the policy process. Promoting  A modest majority of i) Implementing agencies need to i) Agree. This should be strongly Dialogue Between respondents in mainstream user satisfaction encouraged, as part of Statistics Users implementing countries surveys in their work programs, NSDS/work program and Statistical consider the SRF to be to inform other stakeholders and implementation process. Producers at least somewhat to publicize the results of these successful in promoting surveys. this dialogue. ii) Pay more attention to ii) This could be part of  But users of statistics user-oriented awareness raising each government’s generally professed activities through greater dissemination policy. SRF ignorance about the publicity and outreach. projects could include a “user” status of user component for outreach satisfaction surveys in activities and training in use of their countries and were statistics. not well informed about iii) Encourage sharing iii) To be envisaged within capacity building among countries of various the project training program or activities designed for practices such as workshops, special events organized by the their benefit. training of journalists and SRF Administration Unit. mobile education for user outreach. Delivering More  A small majority of i) The importance of sustaining i) The strength of the national Effective Aid respondents felt that the these relationships needs to be partnership group is critical in SRF was successful in borne in mind, particularly given this area, as is the presence of an delivering more the frequent staff rotation. in-country donor statistician. efficient and effective aid and technical 199 assistance. ii) The SRF ii) Agree. An important  Progress in this area Administration Unit should be criterion for country selection is depends to a great kept up-to-date on staff changes the Bank’s commitment to the extent on personal by the respective World Bank SRF approach. The SRF relationships between offices so as to brief new TTLs Administration Unit will also members of the local on the special features of the pay more attention to development partners. SRF approach. accountability, and will review Bank operational procedures to ensure effective and timely execution of projects. Other survey- Management based issues Assessment Recommendations Responses In-country donor  The statisticians have i) Emphasize the coordinating i) The terms of references are to statisticians been an almost role of the statisticians, both be strengthened for statisticians. unqualified success in between producers and users of facilitating the system- statistics and among wide approach and development partners. helping coordinate ii) Fund the post of in- ii) Agree, depending on donor activities. country donor statistician for a the country situation and  The fact that the substantial period of time. financial availability. statistician represented a iii) Consider more than one iii) Agree, depending on major donor was seen as statistician in larger SRF the country situation and having a positive impact projects subject to funding financial availability. on their effectiveness. availability. Issues related to  The SRF approach is i) While retaining the country i) Move cautiously on this. The the SRF Charter certainly broad and focus of the SRF approach, the preference is to focus on flexible enough to have SRF Council should actively information sharing of SRF a significant impact in explore the extension of the SRF experience among pilot each of the diverse to further regional initiatives. countries, prospective SRF situations represented countries, and with regional by the three entities. implementing countries. ii) The Council should promote ii) Agreed, sharing of  A small majority of the sharing of experiences across experiences should be done at all respondents considered countries. levels. that the expectations of iii) Take steps for the iii) The Administration the SRF in terms of its Charter to be publicized, so that Unit will ensure that the Charter impact and timing were countries and institutions remains accessible and visible realistic. become more knowledgeable on the SRF/World Bank web-  A rich vein of about the SRF initiative and the site. suggestions for regional approach and principles activities was put underlying it. This action could forward, featuring be taken in conjunction with south-south exchange of more visibility of the SRF on the experiences and the World Bank internal and sharing of best practices external websites. and expertise. 200 SRF governance  The majority of i) The actions already identified i) Agree. Must be in line with arrangements and respondents in the by the SRF Administration Unit the Bank’s rules and procedures. operational implementing countries to improve SRF governance and practices and rated their experience in operational issues should procedures implementing the grant, continue to be pursued. in terms of reporting ii) Extend the support to ii) This support will be arrangements and implementing agencies in continued, depending on the support from Bank staff, procurement, financial time availability of country as either very or management and overall project Bank staff. somewhat positive. management to World Bank  Problems faced staff where necessary. included inadequacy of iii) Endeavor to ensure iii) Agree. This is an counterpart government greater continuity in post of ongoing issue, given the budgetary support, a TTLs and other staff assigned to required mobility of staff within lack of coordination, SRF project implementation. the Bank. Will implement low levels of capacity, improved hand-over procedures. and overly bureaucratic processes on the part of iv) Promote and pursue iv) Agree. Meetings will be both the Bank and proactive lesson learning and organized on a regular basis, as national experience sharing among the warranted. administrations. pilot countries. v) Modify SRF guidelines to v) Agree. Guidelines will be TTLs to stress the point that the modified. coverage of broader SRF issues, as noted in the generic SRF Results Framework needs to be taken into account during project supervision, so as to be able to respond to SRF requirements. Other issues  Respondents in all i) The SRF Council should re- i) Agree, but given the limited countries expressed the examine the current notional resources, there are tradeoffs wish that SRF funding country specific resource between greater country should continue, citing allocation ceiling and consider coverage and more in-depth achievements to date upward flexibility in specified support to specific countries. and tasks yet to be circumstances addressed 201 Table 2 Management’s response to the proposals on specific issues assuming the extension and expansion of the SRF Issues Assessment Proposals Management’s Responses i) This will be discussed Country  While country selection i) Fragile states versus other IDA with the SRF Council, selection principles are clear, the SRF countries. Based on the current but preference is to Charter is not sufficiently specific classification by the World Bank keep flexibility on to help the actual decision process which includes 36 fragile states out country selection. and pilot country selection was of 82 IDA-eligible countries, it is based on a compromise between suggested that future SRF funding analytical and political may roughly be allocated 1/3 to considerations. fragile states and 2/3 to other countries.  The SRF approach, as highlighted in the annual reports, ii) Open competition versus ii) Fully agree. revealed a continuing focus on ‘by invitation’. As the generally difficulty, variety and fragility for positive reactions to the SRF the selection of countries. The approach become better known, positive feedback of the SRF from which would normally be expected all 8 pilot countries including post- to enhance the attractiveness of the conflict countries appears to SRF-CF as support for statistical confirm the original belief that the capacity building, it is the opinion SRF approach does work in all of the Evaluation Team that open country circumstances although competition should be the primary more care and support should be option for SRF funds allocation provided to fragile states. going forward (while acknowledging that there may still  All SRF pilot countries were exist justification for specific ‘invited’ to apply for SRF grants, countries to be ‘invited’ to apply). allowing for greater subjectivity iii) Agree in compared to an ‘open competition’ iii) Country selection criteria. principle with the ‘two - approach among all IDA countries The Evaluation Team believes that a step procedure’. The where defined criteria and ‘two-step procedure’ would allow suggested criteria will prerequisites play an important for a more practical approach to be considered. role. country selection for future SRF funding. The first-order criteria would consist of prerequisites while the second-order criteria would help to differentiate countries in terms of their eligibility for SRF funding. This two-step approach would apply to both regular IDA-eligible countries and to fragile states, with some minor adjustments for the latter. 202 The SRF  Translating the generic results i) A more systemic consideration of i) Suggested selective generic results framework into specific indicators the two results frameworks. A generic Results framework which are measurable and which useful way to help bridge the gap Framework indicators and country can be monitored over time has between the two results frameworks will be considered. applicability proven to be much more is to view the SRF generic results challenging than originally framework as an effort to monitor a expected with data availability medium- to long-term, system- being the most critical constraint. focused effort to sustainably While the approach adopted by the develop the statistical capacity of SRF Administration Unit to delete countries, while specific project several indicators including most M&E mechanisms monitor time- of those related to financial bound, data and action-focused resources for statistics may be operations. Much more thinking is justified on the basis of data required to translate the above into unavailability, without information operational practices. on resources, there is the risk that the sustainability of support provided by the SRF projects may ii) Greater efforts to obtain ii) Agree. An indicator not be assured. information on budgetary on budgetary resources commitments and support. Based on will be re-introduced  There has been limited success the field mission to Nigeria and into the generic Results by the project teams in blending Rwanda, the Evaluation Team Framework. the indicators of the generic results believes that information on framework with country priorities resources for statistics is available based on individual country but does require special efforts to NSDSs. monitor. The Team proposes the re-  The results frameworks of the introduction of some measure of pilot projects have a strong focus budgetary resource monitoring, on the production of data and their while acknowledging that various accessibility. challenges must be overcome including access to information on  World Bank’s supervision resources provided by development mission reports show a relatively partners. narrow focus on specific project iii) Agree. An development objectives and iii) Going beyond the indicator on trainings intermediate results indicators and production of and accessibility to and activities to limited efforts by supervision statistics and highlighting user- promote the use of teams to attempt to cover the oriented awareness raising statistics will be broader indicators of the SRF activities. It would be useful for introduced into the generic results framework. future SRF projects to have specific generic Results provisions for the active support of Framework. other government agencies outside the NSO as well as users of all sorts and local development partners. This would include revising the content of the new indicator of the generic results framework -- the number of government staff trained under the SRF -- which should be supplemented by ‘total number of persons trained’ to reflect training of users and emphasize ‘type of training’ to distinguish technical training from user-oriented and other types of training. 203 Greater i) Present efforts will be  The nomination of experienced i) On World Bank procedures. More strengthened. flexibility in and committed TTLs and local awareness of and training in SRF project support staff was seen by principles, norms and procedures implementatio respondents to the questionnaire was needed on the part of both Bank n and to survey as key to successful project staff and government officials in the enhance implementation. SRF project team. greater ownership by  The questionnaire survey ii) Greater ownership by countries ii) Agree. countries and showed that ‘ownership’ is most and the World Bank. Discussions on the World pronounced among NSOs and is country ownership, which is at the Bank much less evident among the other core of the SRF approach, need to operations government agencies and the users highlight the importance of the SRF units or potential users of statistics. being ‘owned’ by a broad range of  The Evaluation Team’s stakeholders and not exclusively by impressions indicate a wide range the government statistics office. The of interest and involvement by promotion of more systematic World Bank country teams in the dialogue between producers and SRF projects. users of statistics would go a long way in promoting ownership for  The review of country local stakeholders. Having the experiences shows that there may degree of World Bank support to well be a case for linkage of SRF statistical capacity building in the grants with IDA credits where the country as a prerequisite for country IDA funds are meant to address selection could help in improving high-level, non-technical concerns, the situation. while more technical topics are focused on in the SRF iii) Linkages with matching iii) Agree. project. Properly combined, the IDA credits. More in-depth two could in principle be very assessment of the experiences of effective. Ghana and Nigeria where efforts have been made to combine the SRF with IDA funds -- in parallel for Ghana and sequentially in the case of Nigeria. 204