Results of the World Bank RAS Program in Romania, 2012-2015 Changing Institutions to Achieve Development Goals Romania and Hungary Country Office Country Management Unit for EU Countries Europe and Central Asia Region July 17, 2017 i Disclaimer This report is a product of the International Bank for Reconstruction and Development / the World Bank. The findings, interpretation, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. Copyright Statement The material in this publication is copyrighted. Copying and/or transmitting portions of this work without permission may be a violation of applicable laws. For permission to photocopy or reprint any part of this work, please send a request with the complete information to the World Bank Group Romania (Vasile Lascăr Street, No 31, Et 6, Sector 2, Bucharest, Romania) ii Acknowledgements The World Bank Romania Country Office commissioned this evaluation of the Romania RAS program for the 2012-2015 programming period to assess how effectively the World Bank is contributing to the achievement of development objectives in collaboration with the Government of Romania. The report was written by Dawn Roberts (independent consultant) and András Tamás Torkos under the guidance of Corina Mirabela Grigore and Suzy Yoon-Yildiz. Oana Maria Caraba provided invaluable assistance developing graphs to communicate the results. Elisabetta Capannelli, Country Manager for Romania and Hungary provided overall leadership for the study. Many people provided valuable inputs throughout the evaluation and reporting process. The authors gratefully acknowledge the contributions of client counterparts in the Government of Romania and other key stakeholders who agreed to be interviewed and provided candid reflections about the RAS implementation experience and outcomes during a mission in May 2016. Members of the Romania Portfolio Team helped to identify key documents and stakeholders and schedule interviews. World Bank task team leaders (TTLs) provided detailed survey responses and worked with the evaluation team as needed to fill information gaps. RAS results in the report reflect updates provided by teams through December 2016. Finally, this report has benefited greatly from an in-depth review process. Sara Okada and Izlem Yenice served as peer reviewers and Antony Thompson (Country Manager for Bulgaria) chaired a constructive review discussion to strengthen the findings and recommendations. The authors are also grateful for having had the opportunity to present the RAS evaluation framework to the Results Management and Evaluation Stream community during the RMES Together event in May 2017, which provided important feedback that allowed the finalization of this report. iii Table of Contents Acronyms and Abbreviations ....................................................................................................... Executive Summary ................................................................................................................... iii I. Introduction ...................................................................................................................... 1 Background .......................................................................................................................... 1 Methodology ........................................................................................................................ 2 Report Structure ................................................................................................................... 5 II. Results for the Operational Program for Human Resources Development ..................... 7 Overview .............................................................................................................................. 7 Ministry of Education and Scientific Research .................................................................... 8 Ministry of Labor, Family, Social Protection and Elderly ................................................. 10 III. Results for the Regional Operational Program .............................................................. 17 Overview ............................................................................................................................ 17 Integration of Urban Marginalized Groups ........................................................................ 18 Project Selection Models .................................................................................................... 19 Growth Poles ...................................................................................................................... 20 Communication and Collaboration between MA and IBs ................................................. 21 Reflections .......................................................................................................................... 22 IV. Results for the Technical Assistance Operational Program ........................................... 24 Overview ............................................................................................................................ 24 Chancellery of the Prime Minister ..................................................................................... 25 Ministry of Regional Development and Public Administration ......................................... 31 Ministry of Environment, Waters, and Forests .................................................................. 38 Ministry of European Funds ............................................................................................... 41 Ministry of Public Finance ................................................................................................. 44 National Agency for Cadastre and Land Registration........................................................ 45 Regional Development Agency South Muntenia ............................................................... 47 Tulcea County Council ....................................................................................................... 48 V. Results for the Operational Program for Administrative Capacity Development ......... 50 Overview ............................................................................................................................ 50 iv Ministry of Agriculture and Rural Development ............................................................... 51 Ministry of Education and Scientific Research .................................................................. 56 Ministry of Justice .............................................................................................................. 58 Ministry of Public Finance ................................................................................................. 59 Ministry of Transport and Infrastructure ............................................................................ 63 Ministry of Regional Development and Public Administration ......................................... 66 Romanian Competition Council ......................................................................................... 67 VI. Results for the Operational Program for Increased Economic Competitiveness ........... 70 Overview ............................................................................................................................ 70 Ministry of Information Society ......................................................................................... 70 VII. Formative Feedback from World Bank Task Team Leaders and Clients ...................... 72 Overview ............................................................................................................................ 72 TTLs’ RAS Implementation Experience ............................................................................ 72 Feedback from Clients........................................................................................................ 77 VIII. Conclusions .................................................................................................................... 81 Value of the Conceptual Approach .................................................................................... 81 Aggregate RAS Results ...................................................................................................... 81 Efficiency of Policy Instruments ........................................................................................ 82 Effectiveness of Organizational Arrangements ................................................................. 82 Strength of Stakeholder Ownership ................................................................................... 83 Lessons ............................................................................................................................... 84 Strengthening the Results Focus of RAS Engagements ..................................................... 85 References ................................................................................................................................... A-1 Annex 1. List of Agreements in the World Bank RAS Program, 2012-2015 ............................ A-2 Annex 2. List of Stakeholders Interviewed................................................................................ A-5 Annex 3. Measurable Characteristics for Three Types of Institutional Capacity Changes ....... A-7 Annex 4. Rating System for Assess Institutional Capacity Development……………………...A-8 Annex 5. Data Collection Instruments………………………………………………………….A-9 v Acronyms and Abbreviations ACIS Authority for the Coordination of Structural Instruments ACS Administrative and Client Support ADI Intercommunal Development Association AMTB Bucharest Metropolitan Transport Authority ANCPI National Agency for Cadastre and Land Registration APIA Agriculture Payments and Intervention Agency ARDRP Paying Agency for Rural Development and Fisheries CDRF Capacity Development Results Framework CLLD Community Led Local Development CoG Center of Government CPM Chancellery of the Prime Minister CVM Cooperation and Verification Mechanism DCPP Department for the Coordination of Policies and Programs of the Chancellery of the Prime Minister DU Delivery Unit EC European Commission ERDF European Regional Development Fund ESF European Social Fund ESL Early school leaving EU European Union GDPF General Directorate of County Public Finance GDTPD General Directorate of Treasury and Public Debt GoR Government of Romania IB Intermediate Body ICT Information and communication technology IDP Integrated Development Plan IFMS Integrated Financial Management System IMCS Internal Management and Control System IT Information Technology ITI Integrated Territorial Investment IUDS Integrated Urban Development Strategy KPI Key Performance Indicator LLL Lifelong Learning M&E Monitoring and evaluation MA Managing Authority MARD Ministry of Agriculture and Rural Development MEWF Ministry of the Environment, Waters, and Forests MIS Management information system MIS Ministry of Information Society MO Middle Office of the General Directorate of Treasury and Public Debt MoEF Ministry of European Funds MoESR Ministry of Education and Scientific Research MoJ Ministry of Justice MoLFSPE Ministry of Labor, Family, Social Protection, and the Elderly MoPF Ministry of Public Finance MOU Memorandum of Understanding MRDPA Ministry of Regional Development and Public Administration MTI Ministry of Transport and Infrastructure NACS National Agency of Civil Servants NAFA National Agency for Fiscal Administration NBR National Bank of Romania NCCC National Commission of Climate Change NEET Not in Education, Employment, or Training NGO Non-governmental organization NPCLB National Program for Cadastre and Land Book NSDAR National Strategy on the Digital Agenda for Romania OIPOSDRU Intermediary Body for Operational Program for Human Resources Development OPCS Operations Policy and Country Services OPTA Operational Program for Technical Assistance PIM Public Investment Management PNDL National Local Development Program PODCA Operational Program for Administrative Capacity Development POR Regional Operational Program POSCCE Operational Program for Increased Economic Competitiveness POSDRU Operational Program for Human Resources Development PPP Public Private Partnership RAS Reimbursable Advisory Services RCC Romanian Competition Council RIA Regulatory Impact Assessment i RM Resource management SOE State Owned Enterprise STC Short-term consultant TTL Task Team Leader WBG World Bank Group ii Executive Summary Background In October 2015, the World Bank Romania Country Office commissioned an evaluation of its reimbursable advisory services (RAS) program for the 2012-2015 period. The program included 39 RAS agreements conducted with line ministries, government agencies and local authorities for a total value of approximately $59 million. The conceptual approach for the evaluation was based on the Capacity Development Results Framework (CDRF) to explore how RAS outcomes extend beyond the knowledge and skills gained at an individual or group level to strengthen institutional capacity. According to the CDRF, analytic and advisory services are designed to contribute to one or more of three types of institutional capacity, including the strength of stakeholder ownership (the sociopolitical environment), the efficiency of policy instruments, and the effectiveness of organizational arrangements. There should be a program logic and evidence of intermediate outcomes that signal how a RAS is contributing to strengthening institutional capacity over the longer term. Progress ratings (box i) were assigned to indicate the extent to which each RAS Box i. Rating System for Assessing RAS Contribution helped to address capacity challenges that to Institutional Capacity Change were blocking the achievement of a 1. No outputs: Limited or no actions taken to address development objective. Findings from the identified capacity challenge. desk review of project documents, survey of task team leaders, and interviews with 31 2. Change initiated. Activities or products delivered to Government of Romania (GoR) counterparts address capacity challenge but no evidence of and other RAS stakeholders all helped to outcomes is available. inform which capacity challenges a RAS 3. Knowledge gained. Results of the activities/products was designed to address and the extent to indicate emerging changes in knowledge, skills, which RAS activities and outputs helped to and/or relationships of relevant stakeholders. achieve progress. 4. Knowledge used. Clear evidence shows new Overview of RAS Results knowledge, skills and/or relationships are being applied as an intermediate step toward institutional The evaluation documented some capacity change. progress achieved by 38 of the 39 RAS agreements in addressing a total of 98 5. Capacity change emerging. One or more changes institutional capacity challenges. For the have emerged related to the targeted institutional majority of these improvements (59), capacity challenge, but these changes are not formal stakeholders had already started using new or their effects cannot be measured. knowledge, skills, behaviors and/or 6. Capacity change in effect. Evidence confirms that relationships (figure i). The changes were the targeted institutional capacity has been enhanced. likely to translate to stronger institutional capacity over the longer term even without continued World Bank support. iii Figure i. Ratings Achieved on the Identified Capacity Challenges by 39 RAS 35 Number of constraints Number of RAS under each rating 30 30 26 25 19 20 19 19 15 16 14 10 13 5 3 2 1 1 0 Capacity change in Capacity change Knowledge used Knowledge gained Change initiated No outputs effect emerging Most impactful Least impactful Note: Ratings are based on a 6-point scale: No outputs (1); Change initiated (2); Knowledge gained (3); Knowledge used (4); Capacity change emerging (5); and Capacity change in effect (6). RAS agreements are often designed to address more than one institutional capacity challenge blocking the achievement of a development goal. Two-thirds of the projects (26) addressed multiple challenges, according to the project documents and TTL input. As shown in Figure ii, the portfolio collectively addressed 54 challenges related to the effectiveness of organizational arrangements (37 projects), 34 challenges related to the efficiency of policy instruments (26 projects), and 10 challenges related to the strength of stakeholder ownership (8 projects). Figure ii. Number and Share of Capacity Challenges, by Type of Institutional Capacity 10 Challenges addressed by 8 RAS Effectiveness of Organizational 34 Challenges Arrangements addressed by 26 54 Challenges RAS Addressed by 37 RAS Efficiency of Policy Instruments Strength of Stakeholder Ownership iv Although more than half of the targeted challenges were related to organizational performance, RAS engagements were most effective overall for supporting the elaboration of policies and strategies. Ratings of “knowledge used� or even better results were achieved for a larger share (79%) of the identified challenges related to inefficient policy instruments (laws, regulations, strategies, etc.) compared to those related to ineffective organizational arrangements (56%) or weak stakeholder ownership (40%). Overall, RAS engagements reflected comprehensive support for public sector reform in Romania. The program directly served 15 client organizations at the national, regional, and county levels (figure iii). Organizations that benefitted included multiple units within more than 20 ministries, agencies, commissions and other bodies of the GoR along with countless other institutions at the sub-national level. Examples of these include local municipalities and regional development agencies using new tools for spatial planning and urban development and school inspectorates in all 42 counties receiving new regulations and guidance for organization and functioning. Figure iii. Number of RAS and Level of Impact, by Client Institution -1 Knowledge used or 8 better 2 No evidence yet of 2 -2 knowledge use 3 -2 4 -3 14 -4 4 -6 3 -6 1 4 3 -3 6 -3 3 -2 4 -5 15 10 5 0 5 10 Most impactful Least impactful Note: Ratings are based on a 6-point scale: No outputs (1); Change initiated (2); Knowledge gained (3); Knowledge used (4); Capacity change emerging (5); and Capacity change in effect (6). Efficiency of Policy Instruments The collaboration between the World Bank and the Government of Romania was most effective for the elaboration of laws and strategies. Results assessed in terms of “more efficient policy instruments� included changes in legislation, regulations, strategies, standards or other formal incentives established to guide stakeholder actions more efficiently in pursuit of development goals. As shown in Figure iv, there were 34 identified capacity challenges related to policy instruments, and new knowledge acquired through RAS engagements was being used by stakeholders to address 26 of these. v Figure iv. Progress of RAS in Addressing Challenges Related to the Efficiency of Policy Instruments (n=34) 16 14 13 12 10 10 8 6 5 4 3 3 2 0 Capacity Capacity Knowledge Knowledge Change No outputs change in change used gained initiated effect emerging Most impactful Least impactful Note: Ratings are based on a 6-point scale: No outputs (1); Change initiated (2); Knowledge gained (3); Knowledge used (4); Capacity change emerging (5); and Capacity change in effect (6). These RAS results played a critical role for helping Romania to leverage EU funding for achieving development objectives. Bank teams helped to elaborate strategies to satisfy ex-ante conditionalities for accessing European funds for the 2014-2020 programming period related to lifelong learning, reducing early school leaving, increasing tertiary education attainment, supporting the elderly and active aging, and promoting social inclusion and poverty reduction. RAS outputs also provided the coordination mechanisms to harmonize State and EU-funded projects for integrated planning and development, informed the policy lines regarding integrated territorial development for the Romanian Partnership agreement 2014-2020, and contributed to the design of the Community Led Local Development and Sustainable Development priority axes for the new Regional Operational Program. Some of the most notable progress achieved by these engagements is described in Table i. Table i. Examples of RAS Contributions to More Efficient Policy Instruments Contribution of RAS for Addressing Identified Institutional RAS Results* Capacity Challenge Assistance to the Romanian Capacity change in effect—New merger regulation improved Competition Council— administrative ease, reflected by an average length of 1.8 months to improvements to legal and complete a merger case in 2014. This represents a 23% decrease in the regulatory framework amount of time required compared to 2013. Capacity change emerging—New Romanian competition law adopted in 2015 included changes to reduce the cost of doing business and improve administrative ease for the RCC. More time is needed to assess the effects of these changes. Harmonizing State and EU Capacity change in effect—The PNDL now has a multi-annual Funded Projects in Regions-- implementation timeline, multi-annual budgeting, allocation of funds by informed the redesign of the counties based on needs and prioritization, and criteria for types of National Local Development investment financed. These changes are in line with EU best practice. Program (PNDL) vi Contribution of RAS for Addressing Identified Institutional RAS Results* Capacity Challenge Social Inclusion and Poverty Capacity change emerging—New strategy and action plan satisfied ex- Reduction--provided the basis for ante conditionalities for accessing EU funds for the 2014-2020 national strategy adopted in 2015 programming period. RAS outputs were informed by an extensive public consultation process. Evidence of the contribution to development objectives will emerge with strategy implementation. Strengthening Debt Capacity change emerging—Romania’s Public Debt Management Strategy Management—inputs to for 2014-2016 was assessed by the EU and IMF as being aligned to Romania’s Public Debt international sound practice. More time is needed for evidence to emerge Management Strategy from the strategy’s implementation. Strategic Frameworks in the Capacity change emerging—National strategies adopted for lifelong Education Sector (three RAS)— learning; reducing early school leaving; and increasing tertiary education provided the basis for three new attainment, quality and efficiency satisfied thematic ex-ante national strategies for 2015-2020. conditionalities. Evidence of contributions to development objectives will emerge when strategies are implemented. * Selected examples are illustrative and do not reflect all results for an individual RAS or for the portfolio. Effectiveness of Organizational Arrangements RAS activities and outputs helped to strengthen the capacity of public sector organizations at the national, regional and local levels. As shown in Figure iv, notable progress (where new knowledge was being used) was documented in 29 instances, commonly contributing to an organization’s increased operational efficiency, improved financial management, increased adaptability, or better achievement of outcomes related to a development objective. Figure iv. Progress of RAS in Addressing Challenges Related to the Effectiveness of Organizational Arrangements (n=54) 20 19 18 16 14 12 12 12 10 10 8 6 4 2 1 0 Capacity Capacity Knowledge Knowledge Change No outputs change in change used gained initiated effect emerging Most impactful Least impactful Note: Ratings are based on a 6-point scale: No outputs (1); Change initiated (2); Knowledge gained (3); Knowledge used (4); Capacity change emerging (5); and Capacity change in effect (6). vii Most of the RAS results related to the effectiveness of organizational arrangements focused on refining or reinventing administrative structures, systems, and/or processes. For example, establishing criteria for prioritizing unfair competition cases will allow the Romanian Competition Council to operate more efficiently in processing complaints about unfair commercial practices. Local municipalities such as Alba Iulia are using new analytical approaches and tools for a more efficient planning process, including pursuing inter-municipal planning with nearby cities and setting criteria for project selection and prioritization. Assistance to the Ministry of Education and Scientific Research included producing new regulations for organization and functioning to clarify roles, functions and responsibilities across the pre-university school system. Some examples of notable successes related to improving the effectiveness of organizational arrangements are in Table ii. Table ii. Examples of RAS Contributions for More Effective Organizational Arrangements Effectiveness of RAS for Addressing Identified Institutional RAS Results* Capacity Challenge Strategic Planning for the Capacity change emerging-- Institutional Strategic Management Plan Agricultural Administration— (2014-2017) included programs, measures and budgets and a developed institutional strategic performance assessment framework. An ICT prototype was designed to management plan and ICT prototype track progress against targets. Feasibility studies on paying agencies and to support strategic planning the decentralization of the ARD administration provided information on public administration reform options reported to be highly valued by the client. Assistance to the Ministry of Capacity change emerging—Pilot showed promising results for freeing Education and Scientific Research management up from time-consuming administrative tasks, and client to Develop Administrative stakeholders confirmed that the exercise was valuable, with clear Capacity—designed pilot and documentation and templates to guide delegation and decision-making. operational procedures for the The extent to which the delegation model would be rolled out and delegation of authority implemented was not clear given the political transition. Strengthening Institutional Capacity change emerging—Officials at the MoPF and at the National Capacity in Fiscal Policy Agency for Fiscal Administration became more effective in their roles by Formulation—Provided new using reports on property taxation, oil and gas taxation, natural resources knowledge and skills for the Ministry (oil) taxation and VAT law to amend the tax policy framework in line of Public Finance (MoPF) and tax with international best practice. More time must elapse to assess the agency officials effects of these changes. Support for the National Strategy on Capacity change emerging—The mapping tool enables the Ministry of the Digital Agenda—Developed a Information Society (MIS) to design tailored solutions for broadband tool to map village typology to supply and demand depending on the community profile. More time is determine appropriate PPP model. needed to assess how effectively MIS is using the tool. Assistance to the Romanian Capacity change emerging— Criteria for prioritizing unfair competition Competition Council—Informed cases have been established, and the RCC will process complaints changes in the unfair competition law received from individuals or legal entities only to the extent that they to clarify the RCC’s role and provide evidence about the possible unfair commercial practice, the improve operational efficiency legitimate interest, and the risk of damages. * Selected examples are illustrative and do not reflect all results for an individual RAS or for the portfolio. ICT-based tools developed by RAS teams contributed to improved processes documented at the organizational level. Of the 14 instances where ICT solutions were designed to address institutional capacity constraints, half (7) contributed “knowledge used� or “capacity change emerging.� An ICT prototype for institutional strategic planning and performance assessment developed for the Ministry of Agriculture and Rural Development is expected to be adapted and used by other ministries to strengthen their strategic planning functions. An online monitoring and viii evaluation platform with a performance dashboard is being used by the Ministry of Information Society for evidence-based decision-making. Assistance to the Romanian Competition Council resulted in automation of the mergers process for improved operational efficiency. Training activities alone were generally not adequate for building technical capacity in public sector units. For example, the RAS for strengthening debt management included training on cash management and liability operations, with participants demonstrating increased knowledge as a short-term outcome. However, the RAS team observed that an internal reorganization was needed within the General Directorate of the Treasury and Public Debt in order for the staff to apply the new skills as intended. Other RAS training exercises were also expected to have a limited effect without additional follow-on support, such as for monitoring the implementation of new national strategies in the education sector, implementing a new methodology for managing public private partnerships in the transport sector, and applying promising new sectoral analysis tools for climate- related decision-making in the energy, transport, urban, and water sectors. Strength of Stakeholder Ownership RAS agreements implemented between 2012 and 2015 tended to focus on the supply side. GoR requests for technical assistance were typically for help developing laws and strategies or strengthening the administrative or technical capacity of an organization. However, these efforts had limited success in cases where there was inadequate stakeholder buy-in and commitment for participating in the process and implementing the recommendations, suggesting that more attention should be paid to the demand side during RAS design. Only eight RAS engagements included design elements to strengthen stakeholder ownership. These institutional capacity results on the demand side included increasing the commitment of leaders to the achievement of a development goal, improving the transparency of information, increasing stakeholders’ demand for accountability from service providers, and changing widespread practices. The progress for addressing ten challenges are shown in Figure v. Figure v. Prog 5 4 4 3 3 2 2 1 1 0 Capacity change Capacity change Knowledge used Knowledge Change initiated No outputs in effect emerging gained Most impactful Least impactful Note: Ratings are based on a used (4); Capacity change em ix The portfolio was generally less effective in contributing to these demand-side changes. The eight RAS agreements identified and targeted a total of ten capacity challenges related to the strength of stakeholder ownership, but only four of these changes contributed a rating of “knowledge used� or better progress. In all of the cases, the RAS design included components for contributing to multiple institutional capacity changes, without the enhancement of stakeholder ownership alone being a clear priority. The more effective engagements provided some guidance for what works best in building demand-side capacity. First, a continuous or iterative process rather than a one-off forum or event is needed to foster broad stakeholder participation and commitment to the achievement of a development objective. Second, new findings and concepts need broad dissemination through a variety of formats to gain traction in shifting public awareness and attitudes. Third, a change on the supply side—such as a new organizational mechanism or new legislation—can serve as an important lever for strengthening stakeholder ownership. For example, the RAS providing support for the establishment of a Delivery Unit included a mix of activities in the energy sector for increasing the transparency of information and helping to ensure that consumers demand accountability. Table iii provides examples of how RAS activities contributed to strengthening stakeholder ownership. Table iii. Examples of RAS Contributions for Strengthening Stakeholder Ownership Effectiveness of RAS for Addressing Identified Institutional RAS Results* Capacity Challenge Enhanced Spatial Planning Capacity change emerging: Concepts and spatial planning tools are being as a Precondition for Urban adopted by private consulting companies (e.g., SEARCH Corporation) and Development—developed think tanks (e.g., Institute for Public Policy in Romania). The reports are used new concepts and spatial as teaching materials in academia (e.g., Ion Mincu University Bucharest, planning tools to support a Babeș-Bolyai University Cluj-Napoca, West University in Timișoara). More paradigm shift planners will be using the spatial planning tools and the concepts are being integrated into new development strategies at the local, regional, and national levels. Support to the Establishment Capacity change emerging: Improvements are reflected by better scores in of a Delivery Unit—improved the governance index and the dissemination index for SOEs. The performance the transparency of State- indicators’ targets have not yet been achieved, but continued improvements are owned enterprises (SOEs) in expected to result from the resources and support provided by the RAS. the energy sector * Selected examples are illustrative and do not reflect all results for an individual RAS or for the portfolio. Lessons Progress related to developing more efficient policy instruments or stronger stakeholder ownership is not achieved in isolation. Without exception, the RAS engagements producing “knowledge used� or better outcomes for either of these areas also had at least one component focused on enhancing the effectiveness of organizational arrangements. Even when outputs designed to improve organizational arrangements did not result in stakeholders using the knowledge provided, the supplementary support seemed to act as an important lever for helping the client to develop and adopt new strategies and frameworks and to strengthen demand-side capacity. An adequate authorizing environment on the client side is critical for strong RAS results. The ten RAS agreements that did not achieve any instances of new knowledge use all suffered from limited commitment, motivation, and/or institutional authority on the part of the client leadership x for supporting the delivery of RAS activities and implementing recommendations. Some examples include the engagements focused on building capacity in the Ministry of Transport and Infrastructure for public private partnerships, on the enhancement of risk-based systems for the Operational Program for Human Resources Development, on support to the Ministry of Public Finance to strengthen public investment management, and on the analysis of capacity building activities in the public administration for the Managing Authority of the Operational Program for Administrative Capacity Development. Both high-level (political) and technical-level commitment is needed for RAS engagements to be effective. Efforts to establish a Delivery Unit in the Center of Government were challenged by the limited involvement of the Prime Minister at the time, Victor Ponta. For the assistance to the Romanian Competition Council, the combination of high-level and technical-level commitment yielded strong results for all identified institutional capacity challenges. In the Ministry of Public Finance, there was strong technical-level commitment but more political commitment was needed for implementing recommendations (i.e. reorganizing the Treasury, reforming public investment management, etc.). In addition to adequate commitment on the client side, other common success factors helped to facilitate the achievement of RAS outcomes. These characteristics, listed in Box ii, were frequently mentioned by both clients and Bank team members as key ingredients to help ensure the desired RAS results. Most or all of the success factors were present for the 17 RAS engagements that were found to have contributed to capacity changes that were emerging or already in effect. Box ii. Common Success Factors for RAS Implementation In general, other factors related to the timeline and  Frequent communication and close scope of a RAS did not appear to influence how collaboration between Bank and effective the advisory services were. Across the client teams diverse portfolio, there was no correlation between the progress ratings and the length of the engagement.  Local TTL or senior specialist to The ambition of a RAS in terms of the number of facilitate collaboration institutional capacity changes targeted also did not  Relevant international expertise to appear to matter as long as the common success bring in best practice examples factors were present.  Local expert(s) to ensure diagnosis and examples are appropriate for The turnover in public administration management current context and staff has challenged efforts to enhance the  High-level commitment by client to administrative or technical capacity of implementing recommendations organizations. For the climate change RAS, changes in the leadership and key staff of the Ministry of Environment, Waters and Forests have contributed to the low implementation capacity of the ministry. Frequent management changes at the National Agency for Cadastre and Land Registration limit its institutional capacity to carry out its mandate as defined in the real estate RAS. Similarly, changes in management at the Danube Delta Inter-Communal Development Association (ADI) and staff turnover at the end of the ADI ITI Danube Delta RAS are likely to constrain the use of RAS outputs. Finally, the World Bank’s standard approach to reporting RAS results does not provide useful information for understanding and tracking needed institutional changes. The Activity xi Completion Summary filed at the end of each RAS tends to include subjective ratings without any associated theory of change through which to assess the quality and sustainability of outcomes. In contrast, the CDRF-based approach to results measurement associates outcomes with an institutional change objective to report on not only what progress was achieved but also what progress is still needed. Next Steps Clients’ reflections on their RAS implementation experiences were very positive. Ideas for strengthening the collaboration focused largely on how to further improve the evolving partnership between the Bank and the GoR given the established trust and shared objectives. Recurring recommendations included:  Active negotiation and communication by both the Bank and the GoR at the start of the RAS;  An emphasis on making RAS outputs (reports, strategic frameworks, etc.) to be “useable products� for the client;  A more integrated approach to technical assistance to ensure that the client can apply the new knowledge and skills generated by RAS activities;  A local point-of-contact and a good grasp of the changing local context throughout the RAS implementation period; and  A plan from the start for engaging the right stakeholders. Defining outcomes in terms of incremental progress along an institutional capacity change pathway provides an immediate filter for relevance, effectiveness, and sustainability. Reviewing the results of advisory services through the lens of the CDRF provides a meaningful assessment of how the World Bank’s RAS Program is contributing to the achievement of development objectives. More work is needed to foster a productive monitoring and evaluation (M&E) culture within the Bank’s RAS Program. To strengthen the focus on results, the Romania Country Office could continue to refine this approach to build a common language related to institutional change, integrate M&E tools throughout the RAS cycle, educate stakeholders about the value of M&E, and create a system to disseminate and apply lessons from RAS implementation. xii I. Introduction Background In October 2015, the World Bank Romania Country Office commissioned an evaluation of its reimbursable advisory services (RAS) program for the 2012-2015 period. The program included 39 RAS agreements conducted with line ministries, government agencies and local authorities for a total value of approximately $59 million. The evaluation was designed to assess how effectively the World Bank has been contributing to the achievement of development objectives in collaboration with the Government of Romania (GoR) and to identify lessons for improvement. The RAS instrument is demand-driven, generated from client requests and then regulated by separate legal agreements between the Bank and the recipients of the services. The access to analytical and advisory skills provided by the RAS program and the options for financing, technical cooperation, and knowledge services allow the Bank to engage with client countries as a development partner to support complex structural reforms. The RAS portfolio is an integral part of the Bank’s country program and is governed by all of the standard Bank policies and quality assurance practices, including operational policies and safeguards. In Romania, the reimbursable advisory service model was first used for a set of “functional reviews� in 2010 and 2011 drawing on funding from the European Structural and Cohesion funds to provide an independent assessment of key challenges in 13 ministries and government agencies.1 This engagement was encouraged by the European Commission because of Romania’s weaknesses in the public administration revealed by the financial crisis and low absorption of European Union (EU) funds. The reviews diagnosed the structural bottlenecks to growth and provided operational recommendations for capacity building. These recommendations provided the foundation for action plans by the GoR for strengthening public administration capacity and help the Government to prepare for the 2014-2020 period. The successful completion of the functional reviews led to increased demand from the GoR for reimbursable services under the then-established RAS instrument. The memorandum of understanding (MOU) on “Partnership and Support in the Implementation of the EU Structural and Cohesion Funds and Modernization of Public Administration in Romania� signed in 2012 between the GoR and the World Bank identified complementary areas of focus and targeted support for the 2012-2015 period. The RAS program has continued to grow since the signing of the MOU to address the evolving needs of the Government. For example, World Bank RAS support has helped the GoR to satisfy ex-ante conditionalities in preparation for the next financing cycle.2 RAS agreements covered by this evaluation have been paid by the GoR using EU funds under five operational programs between 2007-2013 with separate managing authorities: 1 At that time, reimbursable advisory services were provided under the Fee-based Services instrument which was launched as part of the Bank’s modernization agenda to respond to individual client country needs beyond the traditional package of services. For more details, see Functional Reviews in Romania: Process and Results (World Bank 2015). 2 The European Structural and Cohesion Funds are now called the European Structural and Investment Funds (ESIF). 1  Operational Program for Human Resources Development (POSDRU),  Regional Operational Program (POR),  Operational Program for Technical Assistance (POAT),  Operational Program for Administrative Capacity Development (PODCA), and  Operational Program for Increased Economic Competitiveness (POSCCE). The list of the 39 RAS agreements by operational program is in Annex 1. The World Bank Romania RAS portfolio team systematically monitors program inputs, outputs, and invoicing but has limited evidence to date of the longer-term outcomes of the RAS engagements. Various challenges have limited the team’s ability to document RAS results. First, RAS agreements are typically negotiated and implemented on a tight timeframe to address an immediate client need, so the main emphasis has been on the successful delivery of the agreed upon outputs and the level of client satisfaction. The envisioned institutional changes supported by the analytic and advisory services often require more time than the duration of a RAS agreement. In addition, the RAS program has spanned a period of relatively frequent political transitions with a high level of turnover among personnel in the public sector. In many cases, longer-term outcomes depend on the ongoing commitment of leadership to the development process and on the career development of trained staff. During FY15, a first-phase evaluation of the World Bank’s RAS Program in Romania focused on four case studies to explore the outcomes, sustainability, accountability, and effectiveness of RAS engagements. The methodology tested through this case study process was refined to allow for this more comprehensive evaluation of the full RAS portfolio implemented in 2012-15. Methodology Building on the case study experience from FY15, the evaluation of the RAS program used a mixed method design to provide a systematic, qualitative assessment of how RAS engagements are contributing to the achievement of development objectives over time. Data sources included a desk review of project documents using a standard template, a survey of the RAS task team leaders (or their designated representatives) for reporting on RAS results and implementation experiences, and in-depth interviews with 31 GoR counterparts and other RAS stakeholders. The list of stakeholders interviewed is in Annex 2. The conceptual approach for the evaluation was based on the Capacity Development Results Framework (CDRF) to explore how RAS outcomes extend beyond the knowledge and skills gained at an individual or group level to strengthen institutional capacity over time.3 This approach was designed to complement existing monitoring and evaluation (M&E) practices used for analytic and advisory services and to address the following key challenges to understanding the results of the RAS program within Romania’s local development context: 3 Otoo, S., Agapitova, N., and Behrens, J. 2009. The Capacity Development Results Framework: A Strategic and Results-oriented Approach to Learning for Capacity Development. World Bank Institute. 2  Standard ratings for RAS effectiveness lack objectivity and comparability across engagements. In the Activity Completion Summary, task team leaders are required to rate the progress of the activity in achieving its development objective using a 10-point scale. However, the rating levels are not well-defined and the self-reporting process allows for a high degree of subjectivity.  Outcomes of project activities are assessed in isolation. As described in Table 1, common M&E approaches too often assess the individual outcomes of analytic and advisory services without tracking them within a longer-term theory of change. Progress in addressing an institutional capacity challenge can only be accurately assessed by looking at the local context and the logical sequence of results. Table 1. Shortcomings of Common M&E Approaches for Analytic and Advisory Activities RAS Activities Common M&E Approaches Issues and Outputs Participant reaction questionnaires; Use of new knowledge/skills depends on career Trainings tests of learning; trainer path of trained individuals and commitment of observations managers Technical reports/ Expert reviews for quality and Longer-term outcomes require that findings are diagnostics relevance used and/or recommendations are implemented Strategic Expert reviews; tracking of whether Evidence of institutional capacity changes will only Frameworks/ strategy/policy is adopted emerge after new strategy/policy is implemented policy advice Participant reaction questionnaires; Stakeholders inform outputs, but Stakeholder review of stakeholder inputs; focus recommendations/policies must be implemented for forums groups sustainable change “Capacity development� is defined broadly by the CDRF as “a locally-driven process of learning by leaders, coalitions, and other agents of change that brings about changes in sociopolitical, policy-related, and organizational factors to enhance local ownership for and the effectiveness and efficiency of efforts to achieve a development goal.�4 This extends beyond the more traditional focus that is limited to training outcomes and instead covers the range of results expected from the RAS portfolio. According to the CDRF, analytic and advisory services are designed to address one or more of three types of institutional capacity challenges:  Low priority attached to the development goal by key stakeholders (strength of stakeholder ownership): Local leaders lack commitment to the development goal; relevant stakeholders do not participate adequately in decision-making; and/or widespread changes in attitudes or behaviors are needed.  Deficiencies in the policy framework guiding pursuit of the development goal by different stakeholders (efficiency of policy instruments): Administrative rules, laws, regulations, standards, or other formal incentives are not efficiently guiding stakeholders’ actions to achieve a development goal.  Weak performance of the organization(s) charged with achievement of the goal (effectiveness of organizational arrangements): The rules of action, processes, personnel, 4 Otto, S., Agapitova, N, and Behrens, J. 2009. p. 3. 3 systems, or other resources within a specific organization are contributing to weak performance or formal organizational arrangements have not been established to serve a key function related to the development goal. As shown in Figure 1, the CDRF provides a theory of change for assessing progres toward institutional change objectives. Intermediate capacity and final (institutional capacity) outcome indicators can be flexibly applied across sectors to assess the effectiveness and relevance of each RAS. An intermediate capacity outcome is an improvement in the ability or disposition of agents (key stakeholders) to take actions. Common examples among RAS outcomes include raised awareness, enhanced knowledge or skills, improved consensus or teamwork, new or stronger coalitions, enhanced networks, and new implementation know-how attained through “learning by doing.� While a RAS is unlikely to fully address identified institutional capacity challenges within its limited timeframe, there should still be a program logic and evidence of intermediate outcomes that signal how a RAS is (or is not) contributing to strengthening institutional capacity over the longer term. The types of measurable characteristics that could be assessed within each area of institutional capacity area are described in Annex 3. Figure 1. Capacity Development Process Source: Guide to Evaluating Capacity Development Results: A Collection of Guidance Notes to Help Development Practitioners and Evaluators Assess Capacity Development Efforts (World Bank 2012) The completion of four case studies in FY15 confirmed the potential value of a CDRF-based approach for systematically assessing how the RAS program is helping to strengthen institutional capacity in Romania.5 The CDRF provided a lens through which to interpret and track 5 The four cases reflected a diversity of sectors and Managing Authorities within the GoR and included Assistance to the Romanian Competition Council, Enhanced Spatial Planning as a Precondition for Urban Development, 4 incremental progress that the RAS program is helping to achieve toward the development goals identified by the World Bank and the Government of Romania. However, the in-depth case study approach was not practical for detailing a comprehensive set of results stories across all 39 RAS agreements and the qualitative narratives did not easily support any aggregate assessment of the overall portfolio performance. For this evaluation, the method was adapted to apply a standard rating scale for marking the progress achieved by each RAS in addressing identified capacity challenges blocking the achievement of development objectives. Findings from the desk review of project documents, survey of TTLs, and interviews with RAS stakeholders all helped to inform which capacity challenges a RAS was designed to address and to provide evidence of incremental progress. Ratings according to the six levels of progress—from “No outputs� to “Capacity change in effect�—are presented in Box 1. More detailed definitions and examples of the types of evidence needed for each level are in Annex 4 and the data collection protocols are in Annex 5. Box 1. Rating System for Assessing RAS Contribution to Institutional Capacity Change 1. No outputs: Limited or no actions taken for addressing identified capacity challenge. 2. Change initiated. Activities and/or products delivered to address capacity challenge but no evidence of outcomes is available. 3. Knowledge gained. Early results of activities show changes in knowledge, skills, and/or relationships of relevant stakeholders. 4. Knowledge used. Clear evidence shows new knowledge, skills and/or relationships are being applied as an intermediate step toward institutional capacity change. 5. Capacity change emerging. One or more changes have emerged related to the targeted institutional capacity challenge, but these changes are not formal or their effects cannot be measured. 6. Capacity change in effect. Evidence confirms that the targeted institutional capacity has been enhanced. RAS engagements are implemented on a tight timeframe and are not generally be expected to achieve institutional capacity changes as direct outcomes. Nonetheless, it is important to understand the extent to which each RAS has contributed to intermediate outcomes and incremental progress for building this capacity over the longer term. Report Structure The following sections explore the results of the World Bank’s RAS program in Romania and provide a snapshot of their contributions for developing institutional capacity. As shown in Figure 2, RAS were grouped based on the source of EU Financing tapped into by the GoR: POSDRU, POR, POAT, PODCA, and POSCCE. After the presentation of results, a section on Preparing a Strategic Framework for Lifelong Learning, and Assistance to the Ministry of Public Finance for Strengthening Debt Management. 5 formative findings shares the feedback from RAS TTLs about how well the Bank’s systems, processes, and personnel are supporting the RAS program. Crosscutting themes and recommendations from clients are also presented to inform the design of future RAS engagements. The final section provides an aggregate summary of the Bank’s RAS results for the projects delivered during the 2012-2015 period and draws conclusions for strengthening this collaboration in the future. Figure 2. Number of Capacity Challenges by EU Operational Program (n=98) 45 40 7 35 30 14 11 25 20 15 2 22 10 20 6 5 2 1 7 1 3 2 0 POAT PODCA POR POSCCE POSDRU Effectiveness of Organizational Arrangements Efficiency of Policy Instruments Strength of Stakeholder Ownership 6 II. Results for the Operational Program for Human Resources Development Overview The Operational Program for Human Resources Development (POSDRU) defined the priorities and areas of intervention for implementing EU financial assistance through the European Social Fund (ESF) for the programming period 2007 to 2013. ESF interventions in Romania were designed to “assure investment in human capital, modernization of education and training systems, increasing the access to employment and strengthening the social inclusion for vulnerable groups.� 6 Seven of the RAS agreements included in this evaluation were paid for by the Government of Romania using funds under the Managing Authority for POSDRU as listed in Table 2. Clients included the Ministry of Education and Scientific Research (MoESR) and the Ministry of Labor, Family, Social Protection, and the Elderly (MoLFSPE).7 Table 2. POSDRU RAS Agreements Completed by the World Bank, 2012-2015 RAS Title Client Date Completion Duration Value Signed Date (months) (US$ millions) Reducing Early School Leaving MoESR 12-Jun-13 11-Oct-15 28 $2.01 Preparing a Strategic Framework for MoESR 17-Jul-13 16-Oct-15 27 $1.92 Increasing Tertiary Education Attainment, Quality and Efficiency Preparing a Strategic framework for MoESR 16-Sep-13 15-Jan-15 16 $1.11 Lifelong Learning Study on Diagnostics and Policy Advice MoLFSPE 17-Jun-13 16-Apr-14 10 $0.55 for Supporting Roma Integration in Romania Enhancement of Risk Based Systems of MoLFSPE 28-Jun-13 27-May-14 11 $0.20 the Sectoral Operational Program for Human Resources Development Preparation of a Draft National Strategy MoLFSPE 26-Feb-14 25-Feb-15 12 $1.28 Regarding Elderly and Active Aging Provision of Inputs for the Preparing of a MoLFSPE 26-Feb-14 11-Nov-15 21 $1.83 Draft National Strategy and Action Plan on Social Inclusion and Poverty Reduction A key focus of the Bank’s RAS program under POSDRU was supporting the GoR in developing national strategic policy frameworks to fulfill prerequisites for accessing ESIF funding for the 2014-2020 programming period. Five agreements helped to satisfy these ex-ante conditionalities and made important contributions to increasing “the efficiency of policy instruments�—referring 6 Ministry of Labour, Family and Equal Opportunities. 2007. Sectoral Operational Programme Human Resources Development 2007-2013. Government of Romania. p. 58 7 These are the current ministry titles at the time of this report. Some RAS documents refer to the Ministry of National Education and the Ministry of Labour, Family, and Equal Opportunities. 7 to the CDRF-based assessment of how efficiently strategic frameworks or other formal incentives guide stakeholders’ actions to achieve development objectives. RAS activities and outputs also made modest contributions to strengthening stakeholder ownership and increasing organizational effectiveness. Ministry of Education and Scientific Research The three RAS agreements paid for by the client using POSDRU funds in the education sector focused on lifelong learning (LLL); reducing early school leaving (ESL); and increasing tertiary education attainment, quality and efficiency. Each RAS focused in its own content area and engaged the relevant stakeholder groups, but they shared similar objectives (supporting the preparation of national strategic frameworks), overlapping implementation periods, and parallel design elements. The components for each RAS included developing a draft strategic framework, disseminating the framework, and strengthening the capacity to monitor LLL, ESL, or tertiary education programs. The RAS engagements achieved their objectives and the data compiled for this evaluation provided a consistent picture of high quality contributions by the Bank team. As illustrated by Box 2, in-depth interviews with 10 stakeholders across the education sector underscored the value of the RAS outputs and the achievement of collaborative results. Particular strengths for informing the strategic guidance that MoESR representatives highlighted included the RAS teams’ detailed analysis of the “not always flattering� current context and conditions at the start, the inclusion of best practice examples, and the convening and/or consultation of all relevant stakeholders. Box 2. Example Perspectives on the Support for Strategies in the Education Sector “Romania has an important partnership with the World Bank in the education sector. There is a growing group of local experts working with the Bank team and we know the capacity and skills to expect. We can collaborate well together.� “The World Bank support for the strategies has been hugely important for writing the operational programs for the next funding period. We could not do this strategic approach ourselves and we really benefited from the combined efforts happening at the right moment. As we used to say in Romania, ‘you don’t have spring with one flower’.� “The analysis of the context and reasons for early school leaving and the resources needed is very thorough and useful for the ministry to substantiate future decisions related to education policies.� “I always found the World Bank team to be very open and flexible. The strategy required several reviews and changes and they provided the needed support during all the stages.� Source: Interviews with education sector stakeholders Outcomes for each RAS extended beyond the changes in awareness, knowledge and skills of individuals to reflect important progress toward strengthening institutional capacity for the education sector. Data from the review of project documents, TTL survey, and stakeholder interviews identified key institutional capacity challenges that each RAS was designed to address. As described in Table 3, all three RAS engagements were very effective in improving 8 policy frameworks but the real evidence of these capacity changes will only be available once the strategies are under implementation. Each RAS also made modest contributions toward enhancing organizational effectiveness related to monitoring relevant programming financed with EU funds. In the summer of 2015, the GoR approved and adopted three national strategies for lifelong learning; reducing early school leaving, and increasing tertiary education attainment, quality and efficiency. Each national strategy was heavily based on the draft strategic framework and advice provided through the respective RAS engagement. Stakeholders noted that the new strategies were transparent and helped to clarify the roles and responsibilities across organizations. According to the CDRF-based assessment applied for this evaluation, each strategy is expected to guide stakeholder actions efficiently for achieving its development objective given the strategy’s consistency with international standards and legitimacy with local stakeholders.  Alignment with EU standards. The strategies satisfied the criteria for addressing ex-ante conditionalities related to ESIF financing during the 2014-2020 programming period.  Legitimacy and inclusivity. The development of each strategy was based on an extensive consultative process with relevant stakeholders and review of data sources including the following: o Lifelong learning. Public consultation exercises engaged more than 530 stakeholders representing the GoR, education and training providers, employers, and social partners. These consultations included face-to-face group discussions is six of eight development regions of Romania and an online survey to further explore and confirm key findings. In addition, the RAS team compiled relevant existing data by consulting a range of non-governmental organizations and public agencies such as the National Institute of Statistics, the National Authority for Qualifications, the National Agency for Workforce Development, and many others for a total of 75 organizations. o Early school leaving. The comprehensive consultation process also engaged more than 500 individuals at the central, regional, and local levels. Key stakeholder groups engaged during the strategy formulation phase included government representatives, students, parents of students, employers, and social partners. Face-to-face discussions were organized in Bucharest, Brasov, Buzau, and Teleorman counties and a public online survey was conducted. o Tertiary education. The consultation process engaged representatives from the MoESR, academic staff, students’ unions, employers, as well as higher education agencies (160 individuals). An online survey was conducted and face-to-face roundtables were organized in four regions of Romania. In each case, input from the public consultation process helped to inform and prioritize the measures included in the strategies. All three RAS engagements included components designed to build capacity for monitoring LLL, ESL, or tertiary education programs. RAS activities and outputs provided guidance for setting up monitoring arrangements and included the development of a monitoring information system and training on its use for 70 to 80 staff in the Intermediary Body for Operational Program for Human Resources Development (OIPOSDRU). Both OIPOSDRU and MoESR 9 stakeholders commented on the usefulness of the training for introducing “a monitoring culture� but emphasized the need for more preparation and support before the new monitoring arrangements and information systems could be implemented. Table 3. Results of Three RAS Engagements to Satisfy Prerequisites for EU funds: Lifelong Learning, Reducing Early School Leaving, and Tertiary Education Attainment Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments—a lack Capacity change emerging: RAS outputs were used as the basis for three of strategic frameworks for (1) new national strategies for 2015-2020. These were formally adopted by lifelong learning; (2) reducing early the GoR in June and July of 2015 to satisfy three thematic ex-ante school leaving; and (3) increasing conditionalities for accessing European funds for the 2014-2020 tertiary education attainment, quality, programming period. Each framework is consistent with other related and efficiency policies and EU standards and was informed by a comprehensive public consultation process. Evidence for how effectively this institutional capacity change is contributing to longer-term development objectives will emerge as each strategy is implemented. Limited organizational Change initiated: The training of OIPOSDRU staff and the effectiveness—low capacity of the development of new monitoring information systems were designed to MoESR and OIPSDRU to monitor promote the use of up-to-date data to monitor and evaluate programs and EU-financed projects related to develop evidence-based policy. The monitoring systems are not yet under lifelong learning, reducing early implementation and long-term effectiveness will depend on the career school leaving, and increasing development of the trained staff and the commitment of leadership to tertiary education attainment additional training. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Building the capacity of the MoESR and the OIPOSDRU was not the main Box 3. Example Reflection on the Need for objective of these RAS agreements and Continued Support the RAS activities and outputs did not “I can’t tell you about the results for sure. My notably increase the capacity for either worries stem from the fact that the analysis in the implementing or monitoring of the [ESL] strategy reveals the financial resources needed strategies. As indicated in Box 3, to implement all the measures. So this could be a stakeholders broadly reflected that success or not. The resources that the state should ongoing advisory services and resources allot for education should be significantly greater would be needed to continue to build on than what we have now. We need to attract and use the RAS results. New RAS agreements external funds and we need to focus and target these focused on supporting higher education resources in order to get the possible results.� quality assurance reform and building Source: Interview with MoESR stakeholder capacity for monitoring strategy implementation were expected to provide critical support. Ministry of Labor, Family, Social Protection and Elderly The World Bank collaborated with MoLFSPE to satisfy ex-ante conditionalities related to reducing poverty and promoting social inclusion and active aging. Other engagements focused on supporting Roma integration and reviewing the risk assessment and verification procedures 10 used for POSDRU. A focus group discussion with four MoLFSPE stakeholders supplemented and corroborated the findings from the desk review of project documents and the TLL survey. The RAS engagement to provide diagnostics and policy advice for supporting Roma integration in Romania was designed to help MoLFSPE to develop national policies and identify cost- effective programs in the key areas of poverty and social safety nets, employment, education, housing and habitat upgrading, health, anti-discrimination, and institutional mechanisms for effective local service delivery and the use of EU instruments. The assistance was designed to support Romania in meeting the targets of the Europe 2020 Strategy and the ones of the Romanian National Strategy for the Inclusion of the Romanian citizens belonging to Roma minority 2012-2020; As described in Table 4, the RAS was effective in contributing to more efficient policy instruments. The diagnostics focused on gaps, challenges, good practices and lessons learned to provide policy recommendations for supporting the social inclusion of Roma. The analysis described in detail how legal provisions for addressing Roma needs were spread across a range of policy instruments, including a total of 58 laws, decisions and ordinances that relate to local service delivery, many of which cover multiple layers of government. This complicated the development of a strategy that would be aligned with all these policy instruments. RAS outputs were widely disseminated and discussed with policymakers involved in developing and revising key national strategies and policies, including the National Roma Integration Strategy. Table 4. Results of the RAS Study on Diagnostics and Policy Advice for Supporting Roma Integration in Romania Identified institutional capacity challenge Contribution of RAS for Addressing Capacity Challenge Weak stakeholder ownership— Not all Change initiated: Stakeholder participation did shape the policy government stakeholders at the national and recommendations. However, it is not clear how much these subnational levels are supportive of the informed the revisions to the national strategy or whether social inclusion agenda for the Roma and government stakeholders have increased their commitment to the relevant stakeholders have had limited development objective. opportunities to participate in decision- making. Inefficient policy instruments— Mandates, Knowledge used: The RAS diagnostic study provided inputs for strategies, reporting requirements, and the National Roma Inclusion Strategy, which was successfully budget allocations for pertinent government updated by the government as part of ex-ante conditionality. bodies were not consistent, making it However, there is limited evidence of how much the strategy was difficult for responsible parties to directly informed by the RAS outputs. The RAS diagnostic study implement policies for supporting Roma also informed the Partnership Agreement with the EC and integration. operational programs for the 2014-2020 period. One notable example is related to housing, with the RAS recommendation accepted to make “informal settlement upgrading� eligible for funding under the ROP. Limited organizational effectiveness— Change initiated: Reports from the RAS team indicated that There is a lack of coordination among the some stakeholders gained a clearer sense of their organization’s institutions tasked with implementing the mission and roles related to Roma integration, including NRIS. Specifically, between sector representatives from National Agency for Roma; Ministry of ministries and the National Agency for European Funds; MoESR; Ministry of Health; MoLFSPE; and Roma. the Ministry of Regional Development. However, longer-term effects of this raised awareness is not yet clear. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. 11 Some notable contributions of the diagnostic study related to housing. A key recommendation was to make “informal settlement upgrading� eligible for funding under the new Regional Operational Program. Another recommendation was to view “Roma housing� as one part of the overall housing sector rather than a separate program, and to develop a National Housing Strategy that is all-encompassing. This recommendation was implemented a year later when the Bank was invited to help the government formulate a new National Housing Strategy under Component 4 of the Harmonizing State and EU Funds RAS. The diagnostic study presented new knowledge based on qualitative information collected from field visits, focus groups, and interviews. The RAS involved four rounds of consultative workshops to inform policymaking decisions and held a dissemination event with over 50 stakeholders form the government, civil society, and academia. This broad consultation informed the policy recommendations delivered by the RAS team, but the extent to which stakeholder participation helped to inform the national strategy and increase the commitment of leaders to be supportive of the Roma social inclusion agenda was not clear. Project documents highlighted the lack of coordination among the institutions tasked with implementing the national strategy as a key capacity constraint. Specifically, there was limited organizational effectiveness given the weak coordination between sector ministries and the National Agency for Roma. The diagnostic study provided knowledge to support evidence-based policymaking and for strengthening institutional arrangements. In addition, the RAS team reported that the study helped to raise awareness of ministry and agency personnel regarding the roles and responsibilities of their organizations related to Roma integration. The longer-term effects of this awareness is not yet clear. Managing authorities are responsible for managing and implementing operational programs in accordance with the principles of sound financial management. While MAs can delegate key tasks related to compliance and verification to Intermediate Bodies (IBs), they still have the responsibility to ensure these duties were properly carried out. European Social Fund auditors found deficiencies in the management and control systems for POSDRU funds during the 2007- 2013 programming period, and MoLFSPE sought World Bank assistance to address these deficiencies. In response to this request, a RAS agreement was designed to enhance the effectiveness of the risk-assessment and verification procedures of the Operational Program for Human Resources Development. The main components included the review and revision of the risk assessment methodology and verification procedures, manuals for effective implementation by the MA and IBs, and capacity building of the MA and IBs through training programs. As summarized in Table 5, not all of the RAS activities and outputs were delivered as planned, and no evidence was available to suggest that the manuals delivered by the RAS had contributed to organizational effectiveness. The MA for POSDRU now has the manuals available and could potentially adopt and use them in the future; however, the RAS team noted the following shortcomings:  Client feedback on the draft manuals was substantially delayed. 12  The RAS team lacked access to key people who could have facilitated piloting and walk- through tests.  Due to the points above, the proposed revised procedures were not tested or piloted using actual or simulated data. The feasibility and appropriateness of the risk assessment and verification procedures proposed could not be confirmed.  The three planned regional training programs were not delivered, given the delays noted above. Table 5. Results of the RAS for the Enhancement of Risk-Based Systems of POSDRU Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited organizational effectiveness— No outputs: The RAS produced a revised and streamlined risk The Managing Authority (MA) and assessment methodology, verification procedures and related Intermediate Bodies (IBs) administering manual. Although the draft manuals were delivered, the Bank team POSDRU need effective controls, did not receive timely feedback on the draft risk methodology and including a risk assessment methodology verification procedures nor access to key people. The feasibility and and procedures for verification of appropriateness of the manual could not be confirmed and the reimbursement requests planned regional training was not delivered. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The advisory services agreement on the Romania strategy for the elderly and active aging was designed to provide an analysis of the current situation, increase the technical capacity of the GoR, and assist in preparing the Draft National Strategy for Active Aging and the Elderly for 2014-2020. A final key objective was to help the raise public awareness around the issue of active aging and the elderly and build support for the Government's strategy. As presented in Table 6, the RAS was effective in supporting the development of the new strategy but only Change initiated in contributing to organizational capacity or changing widespread awareness and attitudes. The strategic framework produced as a RAS output informed the new national strategy adopted by the GoR in July 15, 2015. However, client stakeholders expressed dissatisfaction with the RAS outputs, noting that the draft was missing key content and revisions were required by the European Commission. For example, additional work was needed to add a detailed implementation plan with indicators and financial sources. 8 Nonetheless, the final version did satisfy ex-ante conditionality and has made a substantial contribution in terms of clarifying roles and responsibilities across organizations. 8 As described in the RAS progress report (November 1, 2014 - January 31, 2015), the Bank team arranged to deliver the action plan later to accelerate the delivery of the framework at the request of MoLFSPE. However, interviews with MoLFSPE stakeholders highlighted “a notable difference in quality� between the draft framework for active aging and the one delivered for social inclusion and poverty reduction. 13 Table 6. Results of the Elderly and Active Aging RAS Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments—a Knowledge used: The strategic framework informed the national strategy coherent strategy is needed to adopted by the GoR in August 2015 to satisfy a thematic ex-ante support active and healthy aging conditionality to access European funds for the 2014-2020 programming period. However, substantial additions were required in comments by the EC. The approved strategy is consistent with related policies and EU standards and helps to clarify roles and responsibilities across organizations. The strategy must be implemented before the effects can be assessed. Weak stakeholder ownership—there Change initiated: Inter-ministerial working groups and public is limited public awareness of the consultation workshops were designed to promote increased commitment issues related to active aging and and cooperation across agencies. However, there is limited evidence so leaders have not demonstrated far of sustained changes. adequate commitment to the development objective Limited organizational Change initiated: Steering group meetings and consultative workshops effectiveness—one RAS objective is informed the set-up of 2 new units within MoLFSPE for active aging to increase the technical capacity of advocacy and long-term care, but there is not yet evidence available of the GoR increased technical capacity. There is still only a small team dedicated to addressing issues related to active aging and the elderly. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. On the demand side, there has been weak stakeholder ownership related to active aging. Responsible government agencies have tended to view issues related to the elderly in a simplistic manner, limited to only a few aspects. Social participation levels are very low in Romania, with volunteering by the older population being neither encouraged nor valued, and instances of elder abuse being too frequent. As a starting point, the RAS established inter-ministerial working groups and public consultation workshops for promoting increased commitment and cooperation across agencies for addressing these issues. However, as noted by both the client and the Bank team, widespread changes in attitudes will not happen through a one-year project, and new initiatives and ongoing financial support will be needed to continue to keep raising awareness over time. While one objective of the RAS was to increase the technical capacity of the GoR, there is not yet evidence that the MoLFSPE has the capacity needed to implement the strategy. During the RAS, steering group meetings and consultative workshops informed the set-up of two new units within the ministry for active aging advocacy and long-term care. At the time of this evaluation, however, there was still only a small team within MoLFSPE dedicated to addressing issues related to active aging and the elderly. For the RAS designed to support the development of a Strategy on Social Inclusion and Poverty Reduction, evidence collected from both the Bank team and the client indicated that capacity constraints identified at the start included inefficient policy instruments and a lack of organizational capacity. First, a coordinated approach was needed in the provision of policies, programs, and interventions that are targeted to poor and vulnerable people and poor and marginalized areas. A range of sector policies, programs, and interventions were already in 14 existence in Romania aimed at reducing poverty and social exclusion, but the key missing ingredient was the coordination between these policies, programs, and interventions. Second, there was weak administrative capacity, with specific shortcomings including the lack of an integrated ICT system, the absence of evidence based policy decisions, and the need for the modernization of the payments system. RAS deliverables included a draft strategy for social inclusion and poverty reduction, a background document with the diagnosis and analysis underpinning the strategy, an action plan, and nine implementation plans for key policies (flagship initiatives) with high potential impact for reducing poverty and social exclusion. After a series of consultations and revisions, the draft strategy and action plan for 2015-2020 were adopted by the Government of Romania in May 2015. As noted in Box 4, client stakeholders were generally pleased with the work of the RAS team, both for the quality of technical inputs and for the process of convening, coordinating, and integrating a diversity of perspectives. Box 4. Stakeholder Views on the Bank’s Contribution to the Strategy for Social Inclusion and Poverty Reduction “The social inclusion draft was a huge and good document, highly appreciated by the European Commission…the [World Bank] team really worked with us on this process with a lot of ministries, 16 or 17, to synchronize meetings and bring in Romanian experts and in the end to make sure everything was in order without inconsistencies.� Source: Interview with MoLFSPE stakeholders The RAS contributed to more efficient “policy instruments,� referring to formal mechanisms or incentives that guide stakeholder actions across multiple ministries for achieving the longer-term development objective. As summarized in Table 7, the new national strategy and action plan addressed ex-ante conditionalities for attracting EU funds, and their implementation will be closely monitored by the EC. These documents set the strategic directions for the next six years in reducing poverty and promoting social inclusion. In addition, five implementation plans offer both a detailed implementation plan and an operational manual for key anti-poverty policies (for activating poor beneficiaries which are out of the labor market; for the consolidation of means- tested programs into a flagship anti-poverty program; for the development of integrated social services; for addressing inequalities in disadvantaged schools; and for reforming the specialized child care system). Table 7. Results of the Social Inclusion and Poverty Reduction RAS Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy Capacity change emerging: The draft strategy and action plan were the basis for instruments—a framework the new national strategy adopted by the GoR in May 2015 to satisfy ex-ante is needed to set the conditionalities for accessing European funds for the 2014-2020 programming strategic directions for the period. The framework was informed by a comprehensive public consultation next 6 years in reducing process and approved by the EC. Evidence for how effectively this institutional poverty and social change is contributing to longer-term development objectives will emerge as only inclusion after implementation begins. 15 Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Limited organizational Change initiated: Diagnostic documents detailed risks of weak administrative effectiveness—MoLFSPE capacity and of the four implementation plans include roadmaps for building has limited capacity to capacity related to policy formulation, M&E, MIS use and development, implement or monitor modernization of the payment system, and targeting systems. No action has yet measures to reduce poverty been taken by the MoLFSPE to implement changes. The client reported that the and social exclusion diagnosis was “extremely useful� for understanding the current context and challenges. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Almost all of the diagnostic documents emphasized weak administrative capacity as a key risk for the implementation of the measures adopted by the Government, and there was no evidence available at the time of this evaluation that the RAS had contributed to building this capacity. Four implementation plans offered a roadmap to gradually improve the administrative capacity in areas where there are important deficits: policy formulation, monitoring and evaluation, Management Information System (MIS) use and development, modernization of the payment system, and targeting systems. Implementing these plans will be a key mitigation measure to address this risk. The Government has now clear roadmaps on how to use their own resources, as well as EU funds, to improve the quality of the staff and other policy and implementation systems. It is too early to assess whether these resources will be used efficiently. 16 III. Results for the Regional Operational Program Overview The Regional Operational Program (POR) is financed through the European Regional Development Fund (ERDF) to support Romania’s sustainable and inclusive development. The overall program objective for the 2007-2013 period focused on “supporting the economic, social, territorially balanced, and sustainable development of Romanian regions, according to their specific needs and resources, focusing on urban growth poles, improving the business environment, and basic infrastructure.�9 As listed in Table 8, four of the RAS agreements included in this evaluation were paid by using funds accessed by the clients through the Managing Authority for the POR, in the Ministry for Regional Development and Public Administration (MRDPA). Table 8. ROP RAS Agreements Completed by the World Bank, 2012-2015 Value Date Completion Duration Project Number RAS Title Client (US$ Signed Date (months) millions) P143090 Elaboration of MRDPA 20-Nov-12 19-Apr-13 17 $0.58 integration strategies for poor areas and disadvantaged communities P143089 Technical assistance for MRDPA 20-Nov-12 19-Feb-14 15 $0.76 the identification of project selection models P132399 Upgrade of growth poles MRDPA 20-Nov-12 19-Nov-13 12 $0.88 for strategic planning and economic impact P143088 Assessment of the MRDPA 20-Nov-12 19-Dec-13 13 $0.49 communication and collaboration between MAs and Intermediate Bodies Available data sources provided extensive information about the activities, outputs, and results of all four of the POR RAS agreements. Progress ratings for how much each RAS contributed to changes in institutional capacity were based on the review of project documents, detailed input via the TTL survey, and in-depth interviews with nine stakeholders based in Bucharest (MRDPA and NGO think tank perspectives), Brasov, or Cluj-Napoca (local government and academia perspectives). 9 A detailed description of the program and priority axes is available at http://ec.europa.eu/regional_policy/en/atlas/programmes/2007-2013/romania/operational-programme-regional- operational-programme 17 Integration of Urban Marginalized Groups Advisory services for the elaboration of integration strategies for poor areas and disadvantaged communities focused specifically on identifying urban marginalized groups and including the needs of these groups (including Roma settlements) into urban planning. Deliverables produced under the RAS included an Integrated Intervention Tool for Urban Marginalized Communities, an Atlas of Urban Marginalized Communities, and six pilot studies of different types of urban areas. As summarized in Table 9, the RAS was effective in improving the “efficiency of policy instruments� in terms of how it contributed to formal incentives for guiding stakeholder actions toward the achievement of the development objective. The outputs provided a detailed implementation framework for the Community Led Local Development (CLLD) approach that included a typology of urban marginalized communities and their characteristics, mechanisms to identify the geographical location of different sub-types of the most marginalized urban communities in Romania, and a proposed institutional arrangement with a detailed time frame. As a result, the Regional Operational Program for the 2014-2020 programming period includes a dedicated CLLD Priority Axis. Evidence of the effectiveness of the CLLD axis for addressing marginalization will not be available until projects are implemented. The only risk noted by stakeholders that could affect the success of the new approach was that the CLLD mixes ERDF and ESF financing and thus requires close institutional collaboration between MRDPA and the Ministry of Labor, Family, Social Protection and the Elderly. Table 9. Results for the Integration of Urban Marginalized Groups Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy Capacity change emerging: The Regional Operational Program includes a instruments—Policies, dedicated CLLD Priority Axis for the 2014-2020 programming period. The new strategies, and other formal approach and guidelines have been heavily informed by the Integrated mechanisms do not address Intervention Tool, the Atlas, and the six conceptual pilots. The National Strategy challenges faced by urban and Action Plan on Social Inclusion and Poverty Reduction also draws on the marginalized communities Atlas’ methodology for identifying marginalized urban areas. These and other polices/strategies/plans are expected to effectively address marginalization when implemented. Limited organizational Knowledge used: Six pilots have simulated CLLD processes to demonstrate how effectiveness—MRDPA, municipalities/NGOs applying for EU funding of integrated interventions can municipalities, and NGOs address urban marginalization. The Atlas is viewed as highly relevant and useful lack tools and methods for by diverse stakeholders. New knowledge and implementation know-how from the effectively addressing tools and guidance can be expected to improve organizational performance in marginalization and related addressing issues related to marginalization over the longer term. challenges Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. 18 The RAS has effectively contributed to intermediate outcomes for increasing the organizational effectiveness of the GoR, municipalities, and non-governmental organizations (NGOs) over the longer term. The Integrated Intervention Tool serves as a practical handbook for implementing the CLLD and the six pilots simulated a CLLD process based on a diagnosis and needs assessment for six types of urban marginalized communities. Together, the guidance and detailed examples should help municipalities and NGOs to apply for funding to support integrated interventions addressing urban marginalization. Project Selection Models The RAS for the identification of project selection models for the regional development, public works, and tourism sectors produced recommendations to improve project selection models for the 2014-2020 programming period for increased impact. The Bank team provided outputs beyond what was stipulated in the Advisory Services Agreement, including detailed evaluation and selection grids for the proposed ROP axes. With the existing selection process, a major Box 5. Perspectives on the Need for problem in the strategic guidance for project Project Selection Models development was that neither the structure of the POR nor the regional development “This has been an important learning process. strategies reflected the specific priorities and Until now, we fund and forget projects. The needs for each region. The POR targeted the study of the Bank is important to consider general needs of all regions without allowing operational capacity and for the future, the the flexibility for regions to tailor their POR operational budget. We need to start taking into funding to address their specific priorities and account both the investment cost and the needs. Consequently, POR projects were not maintenance cost.� designed to contribute to the areas of highest “So we needed a new road and decided to put it “value for money� and greatest impact. This here. After the 5-year warranty, there is no first-in-first-out selection model did not money for maintenance, and we find that we include formal preliminary screening for have poorly planned.� projects. “A city could run on millions of Euros of investment without running any local budget Stakeholders interviewed emphasized the process. There was not the consideration of expected benefits of and the need for operational capacity, so in the last programming applying the new project selection guidance, period, investments were made on paper, but as illustrated in Box 5. Many of the longer- maybe no one could run them.� term outcomes of the RAS on Project Selection Models will be achieved only when Source: Local government stakeholders in the Bank’s recommendations are Brașov and Cluj-Napoca implemented during the new POR programming period. However, as presented in Table 10, the evidence reviewed for this evaluation indicated that the activities and outputs did help to develop awareness and implementation know-how that would likely serve as an important intermediate step for increasing the effectiveness of both MDRPA and the regional development agencies for addressing regional priorities. 19 Table 10. Results for the RAS on Project Selection Models Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Limited organizational Knowledge used: Key recommendations of this RAS have been applied in the effectiveness—Neither the programmatic document of the POR 2014-2020. Some highlights include an POR nor the regional increased focus on operations and maintenance expenses (planning for long- development strategies address term costs), the prioritization of infrastructure projects, and redefining the the specific priorities and needs evaluation and selection criteria for the POR 2014-2020 axes. While it of each region. Projects are remains to be seen how the models will work in practice, the changes are likely funded without adequate to enhance the effectiveness of the Managing Authority (under MDRPA) and consideration of their long-term the Regional Development Agencies for increasing both the absorption of EU costs and impact. funds and the impact of each project. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Growth Poles The World Bank also provided advisory services focused on the upgrade of growth poles for strategic planning and economic impact. Deliverables under this RAS included a growth poles policy report to guide the MA in improving the growth pole policy for the 2014-2020 programming period, a set of TRACE Energy Efficiency studies for Romania’s seven growth poles, and an analysis of the integrated development plans for the seven growth poles. The functional review of the 2008 growth poles policy highlighted several key shortcomings. For example, it lacked a clear rationale for how designated growth poles can contribute to the general development of Romania and it emphasized regional boundaries even though some urban centers could capitalize on cross-regional synergies. A metropolitan area is defined to have a 30- kilometer radius around a growth pole, but in practice workers are willing to commute up to one hour each way on a daily basis. A larger metropolitan area could allow for a stronger economy with a larger labor pool and better incentives for investments. In addition, the 2008 policy has inconsistencies with other laws related to inter-communal cooperation. At the time of the RAS completion, the Bank team expected that the RAS outcomes would contribute to important improvements in the growth poles policy. However, the political leadership within MRDPA considered the growth poles approach to be flawed and the recommendations were not implemented. As reflected in Box 6, this resulted in some frustration at the local level among municipal government stakeholders who viewed the analyses and recommendations to be useful. Overall, given the change in political leadership and the new approach adopted for the ROP 2014-2020, the Growth Poles RAS was a preliminary step for improving policy instruments over the longer term. This case highlighted the real challenges RAS engagements encounter in trying to achieve progress toward long-term development objectives within a changing political context. Recommendations, even when informed by analyses and evidence, might require demand-side changes in terms of increased commitment of the political leadership before they could be implemented. Despite their expressed disappointment, the stakeholders interviewed highlighted the importance of the RAS for achieving small incremental gains in terms of new 20 awareness and knowledge related to the potential roles and needs of growth poles for driving Romania’s economic development. Box 6. Reflections on Changes to the Growth Poles Policy “We worked with the Bank, thinking they had official support of the Ministry. Some growth poles, like us, started to develop a strategy for the next period based on the World Bank recommendations. Then the Ministry changed the approach and communication has not gone well. Most meetings are bilateral between the Bank and the ministry or the EC. We invested money and time and resources and now have to change gears.� “The Bank provided very important recommendations, but now the decision to give equal investment to all cities does not help very much. We have to think bigger for sustainable development. Transylvania is Cluj-centric, especially for hospitals and education. People come and they stay. There is a very high demand for affordable housing, for services, and for transportation.� “The Growth Pole Policy was not continued as envisaged. Still, all the studies and the analysis provided real support for us and helped to substantiate actions that we designed at the local level.� Source: Municipal government stakeholders in Brașov and Cluj-Napoca Table 11. Results for the Growth Poles RAS Identified institutional capacity challenge Contribution of RAS for Addressing Capacity Challenge Inefficient policy instruments—The 2008 Knowledge gained: The growth poles policy was not continued growth poles policy lacked understanding of as expected under the current political leadership, so there is not how the 7 cities are positioned to become an yet evidence of how the project has contributed to improving economic engine for the country, and there policy instruments. However, the analyses and recommendations are inconsistencies with other laws, reflect new knowledge that is currently being considered and/or particularly related to inter-communal applied at the local, regional, and national levels. cooperation Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Communication and Collaboration between MA and IBs The Bank also completed a RAS designed to enhance the communication and collaboration between the Managing Authority for the POR and the Intermediary Bodies (IBs) and to improve the methods for providing support to beneficiaries. The main activities included an analysis of how the relationship between the MA and the Regional Development Agencies (RDAs) could be improved, an assessment of beneficiary support instruments, and a knowledge exchange with Spanish counterparts on MA-IB collaboration models to learn from the experiences of other EU member states. For the 2007-2013 programming period, the MA under MRDPA was responsible for the POR’s design and implementation. The IBs, which consisted of eight regional development agencies (RDAs) and one central IB for tourism, fulfilled tasks related to programming and implementation as delegated by the MA. Beneficiaries included public, private, and non-profit organizations that submitted financing applications and received POR funding for implementing 21 projects. Other key stakeholders included the Regional Development Council, and at the national level, the Monitoring Committee of the POR, the Audit Authority, the Certification and Payment Authority, and the Ministry of European Funds. The institutional and legislative framework governing MA-IB collaboration had several points of tension as a result of limited decentralization. The MA was overburdened with verifications, approvals, and other procedures designed to validate operational decisions at the IB level. Also, the day-to-day implementation tasks drained vital resources from the MA, distracting its staff from more critical roles like vision-setting, communication with IBs and beneficiaries, coordination among regional design and implementation, programming, and monitoring and evaluation (M&E). As summarized in Table 12, the RAS activities helped to improve the effectiveness of the POR’s organizational arrangements. The RAS proposed a shift in paradigm, with beneficiaries of the POR being put at the center of POR activities. A number of key recommendations have already been adopted with the redesign of the POR 2014-2020. Some of these include changing the audit trails to enable a faster processing of funding applications, strengthening the Project Implementation Units at the level of beneficiaries, enabling direct financing of payment to the contractors to avoid delays and to reduce the number of steps for the financing of works, enabling design-and-build approaches to reduce the problems cause by poor technical designs, and establishing less burdensome audit and monitoring procedures of beneficiaries. Table 12. Results for the RAS to Improve MA-IB Collaboration Identified institutional capacity challenge Contribution of RAS for addressing capacity challenge Limited effectiveness of organizational Capacity change emerging: Key recommendations from this arrangements—The highly centralized ROP RAS have already been applied in the redesign of the POR 2014- overburdens the MA with administrative 2020 with a shift to put beneficiaries at the center of POR rather than strategic tasks and limits the activities. Evidence of the POR’s improved efficiency and opportunities for priority setting at the local effectiveness in serving beneficiaries is expected to emerge as the level program is implemented. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Reflections Interviews with MRDPA and local government stakeholders provided a positive account of collaborating with the World Bank RAS teams. Without exception, any stakeholder who had direct engagement with the RAS team found the TTL and/or team members to be responsive and flexible. While there were some instances where the need for more local experts was noted, the comments predominantly underscored the comparative advantage of the World Bank as a convener for bringing stakeholders together and as a knowledge broker for providing needed international expertise. 22 Despite the positive assessment of the World Bank’s performance across all of the POR RAS engagements for 2012-2015, there were recurring reflections among stakeholders about the relative weakness of the Bank for providing technical guidance or helping to build implementation capacity. As shown by the examples in Box 7, perceived strengths included conducting in-depth diagnostics and elaborating strategies, but some frustration was expressed that the Bank was not able to do more to ensure that the strategies could be implemented. Box 7. Perspectives on the Relative Strengths of the World Bank “We have really good communication with the Bank team. They are very responsive and try to put the right specialists at the right moment. It is a good collaboration.� “The World Bank is very good at strategies, policy development, policy elaboration, but they are not the right partner for the very technical documentation. We select them for the first level, elaboration of the framework, not for the technical design and details of the implementation.� “The World Bank has brilliant competence. Their analysis is very good…but in the part of giving strategic solutions they should improve a bit. It is not so well defined and clear how we can implement what they have produced.� “I would not say the Bank helps to build our capacity yet. There is a study, but how do we transfer the expertise from them to us? Maybe this is the problem with the current approach and the type of contract. The analysis they do is very sophisticated. Could they follow up and stay a bit near us to help process the data and make decisions? Perhaps there could be coaching to show us how to use the methods. That would be really building capacity.� Source: MRDPA stakeholders; local government stakeholders in Cluj-Napoca 23 IV. Results for the Technical Assistance Operational Program Overview The objective of the Operational Program for Technical Assistance (POAT) is “to ensure support for the coordination and to contribute to sound, effective, efficient, and transparent implementation and absorption of the Structural Instruments in Romania.�10 Specifically, the program addresses needs identified for the whole system of management and implementation for the European Regional Development Fund, European Social Fund, and Cohesion Fund. Other sectoral operational programs benefit from the POAT for implementing operations, and the POAT was designed to also give enhanced support for the coordination process and the financial control and audit system. As listed in Table 13, 11 of the RAS agreements included in this evaluation were under the Managing Authority for the POAT, the Directorate for Technical Assistance within the Ministry of Public Finance. The World Bank completed these for clients at the national, regional, and county levels, including the Chancellery of the Prime Minister; Ministry of Regional Development and Public Administration; Ministry of Environment, Water, and Forests; Ministry of European Funds; Ministry of Public Finance; National Agency of Cadastre and Land Registration; the Regional Development Agency of South Muntenia, and the Tulcea County Council. Table 13. POAT RAS Agreements Completed by the World Bank, 2012-2015 Duratio Value Project Date Completion RAS Title Client n (US$ Number Signed Date (months) million) Support to the Establishment CPM 31-Jan-14 04-Dec-15 22 $6.04 P147482 of a Delivery Unit Strengthening the Regulatory CPM 24-Apr-14 15-Nov-15 18 $1.97 P150017 Impact Assessment Framework in Romania Competitiveness MRDPA 31-Aug-12 16-Aug-13 11 $0.99 Enhancement and Smart P131858 Specialization Policies in the West Region Enhanced Spatial Planning as MRDPA 26-Sep-12 25-Jan-13 16 $0.74 P143087 a Precondition for Urban Development Danube Delta Integrated MRDPA 4-Sep-13 03-Nov-15 26 $3.06 P146633 Sustainable Development Strategy Harmonizing State and EU MRDPA 27-May-14 26-Nov-15 18 $3.38 P147062 funded Projects in Regions Climate Change and Low MEWF 23-Jul-13 30-Nov-15 28 $6.15 P145943 Carbon Green Growth Program 10 A detailed description of the program objectives and priority axes is available at http://www.fonduri- structurale.ro/Document_Files/asistentatehnica/00000031/62t63_PO_Asistenta_tehnica_final_EN.pdf 24 Duratio Value Project Date Completion RAS Title Client n (US$ Number Signed Date (months) million) Improving the national MoEF 27-Jun-12 26-Apr-13 10 $0.96 framework for preparing and P130919 implementing public investments projects Establishment of a MoEF 11-Jun-14 10-Jul-15 13 $0.65 Performance Appraisal P147746 System for Government Officials Managing EU Funds Strengthening Public MoPF 16-Sep-14 15-Dec-15 15 $1.63 P146782 Investment Management Real Estate; Basis for NACLR 3-Apr-13 11-Nov-15 31 $3.39 P145716 National and EU Policies Ploiești Growth Pole RDA 9-Jan-15 8-Dec-15 11 $0.18 P153331 South Muntenia Support for the Institutional Tulcea 23-Dec-14 22-Nov-15 11 $0.53 P153436 Capacity Development of County ADI ITI Danube Delta Council As in the other sections, the group of POAT RAS engagements was assessed in terms of how each one contributed to addressing identified institutional capacity challenges such as weak stakeholder ownership, inefficient (or missing) policy instruments, and limited effectiveness of organizational arrangements. A standard desk review of project documents, survey of TTLs, and interviews with selected clients informed the assessment of each RAS. The following description of key outcomes is not intended to be an exhaustive list of RAS results but rather focuses specifically on relevant, documented outcomes that are likely to contribute to and/or support sustained institutional capacity changes over time. Chancellery of the Prime Minister Two of the RAS agreements paid for by the client under the Managing Authority for POAT were conducted for the Chancellery of the Prime Minister (CPM) to strengthen the Center of Government (CoG). These included the RAS providing support for the establishment of a Delivery Unit (DU) and the one focused on strengthening the regulatory impact assessment (RIA) framework in Romania. Data sources for these cases included the review of project documents, additional inputs from the Bank team, and interviews with three client stakeholders affiliated with the CPM and the National Agency for Fiscal Administration (NAFA). To rate the effectiveness of the DU RAS for addressing identified institutional capacity challenges, the evaluation also drew heavily on the sustainability review conducted by the RAS team and delivered as the final output.11 11 World Bank. 2015. Romania Advisory Service on Providing Support to the Establishment of a Delivery Unit. Output 18: Sustainability review report containing an analysis of undertaken actions, lessons learned, and recommendations related to the DU to enable the CPM to take action. 25 The DU RAS aimed to achieve three objectives: to support the CoG in improving public sector outcomes by mobilizing a new institution (the DU) to challenge and support line ministries, to embed results-oriented practices within the line ministries, and to achieve selected policy outcomes in four priority areas. The DU RAS was one of the largest RAS agreements completed during the 2012-2015 period in terms of value ($6.04 million) and included 31 deliverables. While the RAS did not achieve its central objective of establishing a Delivery Unit, the advisory services contributed to incremental progress for enhancing institutional capacity in several areas, as described in Table 14. Furthermore, while the CPM has not yet systematically embedded the results-oriented practices within the CoG, the RAS contributed to important intermediate outcomes in terms of raising the awareness of CPM staff and enhancing their knowledge of approaches and tools, particularly related to establishing performance agreements, monitoring progress against targets, reporting performance data, and using SMS mechanisms for citizen feedback. Some of these practices are being integrated into ongoing programs, as reflected in Box 8. Use of Results-Oriented Practices from the Delivery Unit RAS “We started consulting taxpayers by means of text messages and this is how we were able to get information on what to improve. Taxpayers like giving this citizen feedback. We did have some ways to get feedback before but they were quite primitive—for example, formal questionnaires that people did not take seriously.� “With the citizen feedback and other models, now there are new programs that this government will put into practice. We are forming new working groups with four ministries for the anti-poverty effort, including labor, health, education, and regional development…the action plans follow the model of the Delivery Unit. We will sign agreements with the ministries involved that reflect priorities and milestones and then the reporting will also follow the same model.� Source: Interviews with NAFA and CPM stakeholders Box 8. The DU RAS also achieved some important progress in the four priority areas even though the results were not necessarily those planned for during the RAS design stage. RAS activities were instrumental in facilitating the development of Romania’s new national strategy and action plan for public procurement. They supported NAFA in launching the Private Virtual Space, an online taxpayer platform, and in mobilizing the cooperation across agencies to create a registration database for the young unemployed (Not in Education, Employment, or Training—NEET). Institutions had failed to establish this cooperation on their own over the past ten years. In the energy sector, the RAS team provided technical inputs for the national energy strategy, but its drafting was delayed. However, other RAS activities were effective in improving the transparency of State-owned enterprises (SOEs) and in strengthening mechanisms across agencies for protecting consumer rights. Key performance indicators show important improvements on the demand side as consumers have gained increased access to information and opportunities for demanding accountability. 26 27 Table 14. Results for the Delivery Unit RAS Identified institutional capacity Contribution of RAS for addressing capacity challenge challenge Limited effectiveness of Knowledge gained: The RAS did not establish a Delivery Unit (DU) organizational arrangements— The within the CoG, capable of operating without external support. Center of Government (CoG) had Intermediate steps toward increasing the effectiveness of the CoG for limited authority and effectiveness improving the delivery of public services included increased awareness for supporting results-oriented and motivation of CPM staff for using innovative results-oriented practices in the public sector and methods and a set of concrete lessons (new knowledge) to guide the delivering government priorities potential set-up of a DU in Romania in the future. Limited effectiveness of Change initiated: The RAS established Delivery Agreements with the organizational arrangements—Line implementing organizations with performance targets and regular ministries lacked capacity for using monitoring, yet the actual use of the performance dashboard was limited. results-oriented practices and basing Without the management-level review of performance, institutions did decisions on evidence not face consequences for not achieving targets. The DU exposed the institutions to results-oriented practices, but changes are unlikely to be sustained without further support. Weak stakeholder ownership—The Change initiated: RAS activities introduced CPM staff to the concept of political leadership was not a Delivery unit in the CoG which would use evidence-driven practices to adequately committed to establishing improve the delivery of public services. However, the level of the institutional framework for commitment and participation of relevant stakeholders in working groups improving policy implementation and routines was limited and mostly driven by external consultants. The and the delivery of Government Prime Minister, Victor Ponta, was not sufficiently involved. priorities; Line ministries resisted interference of the CoG in their sectors PRIORITY AREAS Inefficient policy instruments— Capacity change emerging: The RAS facilitated the development of the Romania’s public procurement strategy and action plan and provided technical inputs. The strategy was process was unclear and inefficient, finalized in July 2015 in line with EU requirements to satisfy ex-ante with a national Public Procurement conditionality. The RAS also helped to design the monitoring system for Strategy pending and overdue. The the new Public Procurement Agency. Evidence of improvements to the strategy was an ex-ante procurement system is expected to emerge as the strategy and monitoring conditionality for accessing EU system are implemented. funds Limited effectiveness of Capacity change emerging: The RAS supported NAFA in launching an organizational arrangements—the online taxpayer platform, accompanied by a citizen feedback campaign. Tax Administration (NAFA) has The DU also developed a risk assessment model for the VAT funds. inefficient systems for collecting Evidence shows mixed results so far, with improved efficiency of tax taxes, with low compliance among audits but limited progress in improving taxpayer compliance. NAFA is taxpayers. still developing systems for using performance data. Limited effectiveness of Knowledge gained: RAS activities focused on increasing the registration organizational arrangements— of young unemployed and providing public employment services to Mechanisms are needed to them. An important intermediate step was that the DU succeeded in coordinate public sector efforts for mobilizing the cooperation between several agencies to create the increasing youth employment in line registration database. However, most planned actions were not with the EU Youth Employment implemented, including pilot initiatives and establishing an Guarantee program Entrepreneurship Agency. Sustainable institutional capacity changes are 28 Identified institutional capacity Contribution of RAS for addressing capacity challenge challenge not likely without further support. Inefficient policy instruments— Change initiated: World Bank experts provided energy scenario Romania needs a new energy modelling for the demand side as an input for the energy strategy; strategy to guide the energy sector, however, the new strategy has not yet been elaborated and there is no clarify roles and responsibilities, and evidence of how the RAS inputs will be used. improve coordination with the EU Weak stakeholder ownership— Capacity change emerging: RAS actions and implementation support State-owned enterprises (SOEs) in from the DU resulted in notable improvements according to the the energy sector have limited transparency and governance index and the dissemination index for transparency SOEs. While the performance indicators’ targets have not yet been achieved, continued improvements by the SOEs is expected from the resources and support provided. Weak stakeholder ownership— Capacity change emerging: RAS inputs contributed to organizational Consumers do not have adequate mechanisms and legislative changes to protect consumer rights in the rights for demanding accountability energy sector. While not all relevant KPI targets have been achieved, the in the energy sector evidence shows progress and continued improvements are expected from the resources and support provided. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used , capacity change emerging, and capacity change in effect as defined in Annex 4. The RAS for strengthening the RIA framework in Romania was designed to create technical capacity within the Romanian administration for using regulatory impact assessment, streamline the RIA system, and raise awareness at the political level regarding the relevance and value of RIA. Although the GoR has not implemented the recommended reforms, the RAS did make notable contributions toward building institutional capacity, as summarized in Table 15. First, the RAS was effective in starting the process of strengthening the RIA framework so that it would be used consistently for evidence-based decision making. RAS activities included a diagnostic exercise that was used to inform a series of recommendations. The RAS reports and recommendations were published by the GoR to inform a debate. Second, the RAS contributed to some intermediate outcomes for strengthening the technical capacity to conduct regulatory impact assessments. Technical and legal staff and political leadership gained awareness of the need for an improved RIA framework. The Department for the Coordination of Policies and Programs (DCPP) of the Chancellery of the Prime Minister developed and started using a guide based on the RIA methodology for the quality control of substantiation notes. Three RIA pilot exercises were conducted to help government officials participating in working groups to prepare RIA reports. 29 Table 15. Results for the RIA RAS Identified institutional capacity Contribution of RAS for addressing capacity challenge challenge Weak stakeholder ownership— Knowledge gained: There is not yet adequate commitment by the Political leadership lacks adequate leadership for the GoR to strengthen the RIA framework; however, they commitment for using RIA to base did demonstrate increased awareness and motivation by publishing the decisions on evidence reports and recommendations to promote a dialogue. Inefficient policy instruments— Knowledge used: The GoR has tested the RIA framework through pilot Updated RIA framework and system exercises but has not yet been able to accept and implement the is needed for consistent evidence- recommendations. One expressed concern is that the recommendation to based decision making in the public separate the substantiation note from the normative act would require a sector legislative change. Limited organizational Knowledge used: RAS activities have helped to enhance knowledge and effectiveness—DCPP and pilot awareness in pilot working groups and in DCPP, which developed and ministries need enhanced technical started using a guide based on the RIA methodology for the quality capacity to systematically use RIA control of substantiation notes. However, additional interventions are methodology needed to establish consistent use of the RIA methodology. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Despite this progress toward enhancing institutional capacity in two areas (for more efficient policy instruments and more effective organizational arrangements), the capacity change process seems unlikely to continue without additional resources and interventions to establish the consistent use of the RIA methodology. The recommendations in their current form have not been accepted in the Romanian context, particularly given that the instructions to separate the substantiation note from the normative act would require a legislative change. Also, leaders still lack adequate commitment to systematically base decisions on evidence, despite their regular requests for data, as illustrated by Box 9. Neither of the RAS agreements financed under POAT to strengthen the Center of Government achieved their objectives. However, with the current context of weak administrative capacity and political transition, they achieved important incremental gains toward Box 9. The Need to Build Leadership sustainable changes in institutional Commitment for RIA capacity. New RAS agreements can “There is no demand from politicians here for RIA—that be expected to build further on is very sad but true. We had this project because of the selected outcomes. These include the European Commission. And yet, the paradox is what you RIA RAS II focused on impact hear in our government meetings: “How much does this studies, a RAS for support to the cost?� “Where is the data?� “What are the options?� Public Procurement Strategy, and two Every little step we can make to raise awareness is an others to establish a strategy unit and important outcome so we can work to stop regulations to strengthen the strategic planning passing through without having data.� and budgeting capacity of the Source: CPM Stakeholder Romanian administration. 30 Ministry of Regional Development and Public Administration RAS engagements paid for by the client under the POAT Managing Authority included four completed by the World Bank for MRDPA. Assessments for how effective each engagement was for contributing to local institutional capacity were based on data from the survey of TTLs, desk review of project documents, and in-depth interviews with 11 non-Bank stakeholders from MRDPA, local government, the private sector, and academia.12 The RAS for Competitive Enhancement Policies in the West Region was designed to develop an in-depth competitiveness assessment of services and goods sectors in the West Region, and identify policy measures and interventions to enhance the region’s growth potential. Components included an assessment of trade competitiveness of the West Region’s economy in a comparative perspective, case studies of selected sectors identified based on the findings of the first activity, and identification of policy options and types of interventions to be supported during the 2014-2020 programming period to improve the competitiveness of the West Region’s economy. Over the 2014-2020 period, the West Region has the opportunity to use its natural, human, and financial resources to ensure consistent and sustainable development and achieve a gradual convergence with Bucharest and the highly developed regions of Europe. This progress will require that the region takes advantage of the available assets and financing and finds synergies between the EU Funds and other financial sources in a strategic and integrated approach in order to face the tremendous development challenges ahead and reach the strategic EU 2020 targets. The project delivered analytical inputs which could be directly incorporated into a regional smart specialization strategy. Findings from this evaluation indicated that the RAS activities and outputs achieved intermediate outcomes toward strengthening both the formal incentives that guide stakeholder behavior (efficiency of policy instruments) and the organizational effectiveness of the Regional Development Association of West Romania for designing support services to increase the export competitiveness of the region. These changes are summarized in Table 16. Table 16. Results of the RAS for Competitiveness Enhancement Policies in the West Region Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy instruments— Knowledge used: RAS activities included an in-depth competitiveness Formal mechanisms—such as a assessment of services and goods sectors in the West Region to identify regional smart specialization policy measures and interventions to enhance the region’s growth potential. strategy—are needed to address The draft Smart Specialization Strategy (March 2015) takes into account the local needs and priorities at the RAS recommendations for improving the automotive, ICT, textile, and national level, and to guide construction industries. The strategy has not yet been approved nor decision makers at the county implemented but it provides the framework for the West Region to use its level and potential investors. human, natural, and financial resources to ensure a consistent and sustainable development path. Limited effectiveness of Knowledge used: RAS activities and outputs expanded the available 12 Five of these stakeholders were interviewed in 2015 as part of the pilot case study on the RAS for Enhanced Spatial Planning as a Precondition for Urban Development. 31 Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge organizational arrangements— knowledge of one of the fastest growing regions and can serve as a pilot for The regional development other assessments at the regional level. Analytical tools were shared with the agency (RDA) of West Romania RDA. The RDA used the RAS recommendations to design the draft Smart lacks skills and access to Specialization Strategy of March 2015 (available online) demonstrating new diagnostic information to awareness, knowledge, and skills. It is not yet clear whether these effectively enhance the West intermediate outcomes have increased the capacity of the RDA for designing Region’s growth potential. and monitoring policies to foster smart specialization over the longer term. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The Enhanced Spatial Planning RAS was designed to provide the GoR with actionable proposals to improve the efficiency and effectiveness of territorial and spatial development policies and investments. The two project components included an evaluation of the spatial planning system and an evaluation of the urban development system in Romania. RAS activities produced five reports that have been broadly disseminated, including Enhanced Spatial Planning as a Precondition for Sustainable Urban Development, Competitive Cities: Reshaping the Economic Geography of Romania; and three case studies for different territories (the Brăila County, the Cluj Napoca Metropolitan Area, and the City of Alba Iulia). As detailed in Table 17, the RAS helped to shift widespread development practices more towards focusing on urban areas to drive economic growth, to inform the national territorial development strategy and other key planning documents, and to improve the planning capacity and operational efficiency of local governments. The benefits related to the administrative capacity of the MRDPA appeared to be more limited, with no clear evidence that the reports and RAS activities would directly support any long-term changes. Table 17. Results of Enhanced Spatial Planning as a Precondition for Urban Development Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Weak stakeholder Capacity change emerging: Evidence is available of how the concepts and ownership—Widespread spatial planning tools are being adopted by private consulting companies (e.g., practices emphasize balanced SEARCH Corporation) and think tanks (e.g., Institute for Public Policy in development rather than Romania) and the reports are used as teaching materials in academia (e.g., Ion focusing on urban areas to Mincu University Bucharest, Babeș-Bolyai University Cluj-Napoca, West drive economic growth University in Timișoara). Over time, there is likely to be a paradigm shift—with more planners using the spatial planning tools (widespread changes in behavior) and with the concepts being integrated into new development strategies at the local, regional, and national levels (reflecting increased leadership commitment). Inefficient policy Capacity change emerging: RAS findings were incorporated directly into the instruments—The existing law Romanian Partnership Agreement for the 2014-2020 Programming Period and on spatial and urban planning the recommendations were used in the elaboration of Romania's first Territorial had no strategy or policy as a Development Strategy and for the revisions of the Spatial Planning Law (Law reference framework 350). Evidence will continue to emerge as to whether these changes lead to more efficient and effective territorial and spatial development and help to correct negative market side effects. Limited organizational Change initiated: The RAS provided knowledge products with relevant effectiveness—MRDPA has analyses and tools that were used by MRDPA for designing several key limited technical capacity planning documents and to revise policies on spatial and territorial planning. 32 Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge related to territorial and spatial However, there was no evidence that the reports and RAS activities alone would planning and there is poor have any long-term effect for enhancing the administrative capacity of the coordination between ministry or improve the coordination between the department responsible for departments projects funded from the national budget and the one that coordinates EU financed programs. Limited organizational Capacity change emerging: RAS activities and outputs have contributed to effectiveness—local improved planning capacity at sub-national levels. For example, the city of municipalities have inefficient Alba Iulia is applying new knowledge to promote inter-municipal planning with planning with overlapping nearby cities such as Sebes and Vintu and to set criteria for project selection and development plans and little prioritization. Changes such as these are likely to improve the operational project prioritization based on efficiency of the local public administration. budget Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Even after its completion, evidence continues to emerge for how the Enhanced Spatial Planning RAS is strengthening spatial and urban planning in Romania. The broad dissemination of the spatial planning concepts and RAS reports are helping to engender a paradigm shift and to shape strategic, programmatic and legislative documents at different levels. The reports have been used in the Romanian Partnership Agreement for the 2014-2020 Programming Period (see Box 10), the National Territorial Development Strategy, the Regional Operational Program 2014- 2020, regional development plans and local and county development strategies. By the end of the RAS, these outputs had already prompted a request for follow-up work with the city of Alba Box 10. Excerpt from Romanian Partnership Agreement for the 2014-2020 Programming Period The main policy lines regarding integrated territorial development are in accordance with the National Spatial Development Strategy and World Bank study Competitive Cities: Reshaping the Economic Geography in Romania and imply the following actions:  stimulating the competitiveness of the main urban agglomerations as development engines of the regions and supporting their access to global flows and markets (national/international level);  ensuring better connections of less developed areas to development engines and stimulating the cooperation of cities with their hinterland, in order to concentrate available resources and allow lagging areas to benefit as much as possible from the spill- over effects (regional level);  ensuring equitable access to services of general interest and supporting deprived communities (local level), in order to increase the quality of life in all cities and regions. Source: Romanian Partnership Agreement for the 2014-2020 Programming Period (p. 135, Ministry of European Funds, March 2014) Iulia. 33 The RAS for the Danube Delta Integrated Sustainable Development Strategy was designed to contribute to the planned and environmentally sensitive development of the Danube Delta region and to support the GoR in applying a new EU funding mechanism called the Integrated Territorial Investments (ITI) in the Danube Delta and Constanta metropolitan area. The scope included also establishing a methodological framework for the efficient implementation of sustainable urban development activities and producing a guide for promoting sustainable urban development activities. These activities and outputs were intended to support the GoR in developing a dedicated priority axis for sustainable urban development under the Regional Operational Program 2014-2020. Changes were made to the RAS design during implementation to accommodate this need. During RAS implementation, the World Bank enlisted a team of over 100 experts to deliver 18 outputs. These experts represented a broad range of specialties such as fisheries, social inclusion, administration, climate change, transport, regional development. As described in the Activity Completion Summary, implementation was difficult for this project given the complexity of the new ITI tool, the urgency to produce results, and the need to coordinate communication and requests with the direct client (MRDPA) and one of the main future beneficiaries (Tulcea County Council). Examples of the challenges for collaboration and the RAS team’s efforts to respond to client concerns are in Box 11. Box 11. Reflections on Efforts to Improve Collaboration for Achieving RAS Results “The World Bank has very good experts. But the ones for the fishery, we never met them. This is not normal. We as the beneficiaries paid a lot of people and did not meet them. There was a big clashing debate about the fishery and we needed to validate the recommendations of the World Bank. But we could not arbitrate well because we were not even part of the discussion. For the next RAS, we will negotiate and communicate more to have a stronger joint collaboration.� “The flexibility of the World Bank is very important. At the start, the expert team was not appropriate for the activities, and things didn’t go well at the beginning. But they changed the project manager and the experts. They were open to revising the deliverables... we were quite pleased with the results.� Source: Interviews with MRDPA stakeholders The Danube Delta RAS was designed to address key capacity challenges related to inefficient policy instruments—that is, the strategies, policies, or other formal incentives that guide stakeholder actions for achieving sustainable urban development. Notable challenges at the start included:  The framework regulation for urban planning in the Danube Delta Biosphere Reserve (Government Decision no. 1516/2008) had not been fully implemented locally because some communes did not have the necessary resources to revise the general urban plans and the local regulations for urban planning accordingly.  The institutional system for the administration of the Danube Delta is extremely complex, both vertically and horizontally, due to the special status of the area. Many of the institutions 34 at different levels of government have had overlapping responsibilities with regard to environmental and biodiversity issues.  Despite some decentralization measures, the central government remains vital for the administration of the Delta: five ministries have responsibilities in the area, but there is no central coordination mechanism for the Danube, and this cannot be compensated by the local cooperation between the various territorial structures.  There have been several efficiency and legitimacy gaps between the organizations of the central government and those of the county / local government, the latter being forced to implement a number of central policies although they are often not involved in their drafting and they have an extremely low level of administrative capacity. As summarized in Table 18, the RAS contributed to important progress in addressing these constraints. Most notably, the Danube Delta Integrated Development Strategy (pending GoR approval) provides the framework for how 1.1 billion Euro in EU funds will be spent between 2014 and 2023 in one of the most unique environmental areas in the World and one of the poorest areas in Romania. The Sustainable Urban Development Guide provides the framework for how 1.4 billion Euro in EU funds will be spent between 2014 and 2023 in the largest cities in Romania to drive economic growth. In addition, the MRDPA now has a fully functional ITI framework, which it should be able to implement independently in future programming periods. Table 18. Results for the Danube Delta Integrated Sustainable Development Strategy RAS Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy Capacity change emerging: Based on the diagnostics and elaboration of the instruments—A strategy delivered by the RAS, MRDPA submitted the Danube Delta Integrated comprehensive integrated Development Strategy for GoR approval. The draft underwent public strategy is needed to guide the consultations for local legitimacy and reflects consistency and coordination with sustainable development of EU and national strategies and planning documents. Approval was achieved the Danube Delta (Government Decision 602/2016) at the end of the Strategic Environmental Assessment Procedure; however, it is not yet clear that the Danube Delta Intercommunal Development Association and MRDPA have adequate capacity for effective implementation and monitoring of the strategy. Inefficient policy instruments Capacity change emerging: RAS activities and outputs, particularly the —As part of the Regional Sustainable Guide for Urban Development, have directly informed the design of Operational Program for the new Sustainable Development Axis for the POR. Furthermore, the MRDPA 2014-2020, the new axis for now has a fully functional Integrated Territorial Investment (ITI) framework, Sustainable Development which it should be able to implement independently in future programming needs a formal framework and periods. While it is too soon to assess the evidence of implementing these guidelines to shape changes, the expected results will be a more consistent, coordinated, and stakeholder behavior across integrated approach to development. organizations Limited organizational Knowledge gained: The Sustainable Urban Development Guide provided a effectiveness—Cities need number of useful tools such as investment planning with implementation tools for urban development timelines, a framework for prioritizing investment projects, territorial beyond the absorption of EU coordination mechanism, and measures for strengthening project funds implementation capacity. Several of the innovations proposed in the guide have been incorporated into the applicant guidelines for the Sustainable Development Axis of the POR 2014-2020. However, more time is needed before results of improved urban planning from the guide will be available for the municipal level. 35 Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Over the longer term, the innovative tools produced by the RAS can be expected to foster and support more efficient planning and development by municipal governments. One valuable contribution from the RAS is the specification of operational budgets for all beneficiary cities for the 2014-2023 programming period. By providing an estimate of the funds beneficiaries could allocate for capital investments, the budgets enable local authorities to identify integrated priority project lists with multiple funding sources rather than just being limited to the EU funds available through the POR. The Danube Delta Integrated Sustainable Development Strategy was subjected to a longer-than- expected review process, but there remained high-level commitment to finalizing and implementing the strategy. On April 18, 2016, Prime Minister Dacian Cioloș participated in a public debate organized by MRDPA, in collaboration with the Danube Delta ITI Intercommunity Development Association and Tulcea County Council. Other high-level participants included Deputy Prime Minister Vasile Dîncu, Minister of MRDPA, Aura Răducu, Minister of European Funds, Dan Costescu, Transport Minister, and other representatives of central and local administration responsible for implementing the strategy. The debate was expected to be a final step in the consultation process, and the strategy received government approval in August (Government Decision No. 602/2016). The MRDPA also requested World Bank support for a RAS agreement focused on harmonizing state and EU-funded projects in regions. Key aspects of this request included redesigning the National Local Development Program (PNDL) and helping to elaborate a national housing strategy to help coordinate and harmonize state-budget investments in the housing sector. Romania’s Partnership Agreement, negotiated with the European Commission, outlines how 36 billion Euro worth of grants will be spent between 2014 and 2023. However, this funding will only cover part of the infrastructure development needs in the country and significant sums are also allocated from the state budget, county budgets, and local budgets in addition to private investments and arrangements for public private partnerships. At the level of local authorities, development strategies and plans are often geared toward specific funding opportunities, without considering actual needs on the ground in an integrated manner (e.g., most cities produced integrated urban development plans, which often were only geared towards attracting EU funds). At the central level, as the World Bank 2010 and 2011 Functional Reviews showed, communication and coordination across different ministries is deficient, leading to disjointed interventions (e.g., paving the road, then digging to upgrade the water and sewage networks) and failing to leverage potential synergies (e.g., financing back-to-back roads or “linking� two EU- funded water projects with a state-budget-funded extension). All public investment efforts have to be properly coordinated and harmonized. The PNDL is the largest state-budget investment program and is largely focused on subnational investments that are also covered under the EU-funded sectoral operational programs. At the start of the RAS, the PNDL did not have any strategic document as its foundation nor did it have a clear implementation timeline or clear criteria for prioritization and monitoring. 36 In addition, the GoR needed to define a comprehensive strategy for Romania’s social infrastructure, particularly housing. While the housing law has been revised relatively frequently, it lacks a rational framework as its basis. As described by the RAS team, “Romania’s housing sector faces special policy, legal, financial, and institutional challenges� and “Romania’s housing is among the most crowded, dilapidated, expensive, and poorly located of any country in Europe.�13 RAS activities and outputs helped to address the identified institutional capacity challenges, comprising one of the most ambitious efforts to reform public expenditure frameworks in Romania after 1989. Notable results are presented in Table 19. Several of the recommendations made by the Bank team (e.g. multi-annual implementation timeline, multi-annual budgeting) have already been taken over by the Ministry of Regional Development in the re-design of the PNDL. These are in line with best-practices in the design and operation of EU operational programs. Table 19. Results for the RAS on Harmonizing State and EU Funded Projects in Regions Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments—the Capacity change in effect: RAS activities and outputs have provided a National Local Development blueprint for integrated planning and development with a coordinated Program (PNDL) does not have any approach and the recommended coordination mechanisms are being formal strategic document as its operationalized by MRDPA. The PNDL now has a multi-annual foundation. It is not coordinated implementation timeline, multi-annual budgeting, allocation of funds by and harmonized with EU counties based on needs and prioritization, and criteria for types of operational programs. investment financed. These changes are in line with EU best practice. Inefficient policy instruments—A Knowledge used: The outputs and recommendations delivered on the national housing strategy is needed housing component were used by MRDPA to elaborate the National to overcome policy, legal, financial Housing Strategy, which is now under consultation. and institutional challenges Limited effectiveness of Knowledge used: Workshops, conferences and products delivered by the organizational arrangements— RAS are contributing to improved planning and effectiveness for a regional development actors number of programs and organizations. EU-funded operational programs (national, regional, and local) lack are using some of the prioritization criteria and investment guides to guidelines and tools to coordinate improve outcomes, and several county councils (e.g. Brasov, Cluj, investments for improving Calarasi) are using the investment guides for developing multi-annual operational efficiency and investment programs. Metropolitan areas (e.g. Ploiești, Constanta, effectiveness through an integrated Craiova, Iasi) are using the guides for identifying key investment needs in approach to development county roads, communal roads, water and wastewater, and social infrastructure. Evidence of increased organizational effectiveness from these changes is expected to emerge over time. Note: Ratings are based on a six-point scale:No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Although the MRDPA leadership was initially reluctant to undertake the relatively bold recommendations made by the Bank team, a political shift created more openness at the ministry. Changes in knowledge, skills, and behaviors that are expected to support sustainable changes in organizational capacity include the following examples: 13 The RAS context is detailed in Harmonizing State and EU Funded Projects in Regions: Final Synthesis (November 26, 2015) 37  Investment Guides prepared by the team are now used not just by the PNDL team but have also been adopted by EU-funded operational programs. For example, the POR 2014-2020 uses some of the prioritization criteria for the evaluation and selection grids for county roads. Similarly, they use the operational budgets proposed for each locality in Romania as a tool for the prioritization of investments for county capitals, under the Sustainable Urban Development Axis of the POR.  The National Rural Development Program is using the Local Human Development Index developed by the RAS as a tool for their prioritization of investments at the rural level.  The Large Infrastructure Operational Programme 2014-2020 is using the Investment Guide for Water and Wastewater for their investments in the sector.  The Ministry of Transport wants to use the Investment Guide for County Roads as a blueprint for the prioritization of investments in the national road network.  Several county councils (e.g. Brasov, Cluj, Calarasi) are already using the investment guides prepared during the RAS for developing multi-annual investment programs.  The guides are also used by several metropolitan areas (e.g. Ploiești, Constanta, Craiova, Iasi) for identifying key investment needs in county roads, communal roads, water and wastewater, and social infrastructure. Both the Client and the stakeholders involved have recognized the value of the work produced by the Bank and have started to think about state-budget funded investment programs in a different way. Organizations that stand to benefit from these outcomes in terms of improved efficiency and effectiveness include MRDPA, the Ministry of Public Finance, Ministry of European Funds, county councils, local administrations, metropolitan area intercommunal development associations, the Romanian Association of Municipalities, the Romanian Association of Communes, regional water companies, think tanks, and private consultancy companies. Ministry of Environment, Waters, and Forests The Climate Change and Low Carbon Green Growth Program was the largest RAS in the World Bank’s portfolio, with a value of $6.15 million. The advisory services agreement was signed with the Ministry of Environment, Waters and Forests (MEWF) in July 2013.14 The program produced more than 30 reports, held over 30 events (workshops, review meetings, and conferences), and developed an array of dissemination materials. The World Bank team involved more than 50 international and local experts and four consulting firms covering 13 sectors. The purposes of the RAS were to assist the GoR in updating and operationalizing Romania’s national climate change strategies 2016-2030, developing the national action plan for 2016-2020, identifying and integrating climate-related actions into EU-funded Operational Programs 2014- 2020, building an analytical base for impact assessments and climate-related decision making, and enhancing climate-friendly practices and monitoring systems. This evaluation of the overall RAS portfolio did not have the scope or timeframe to explore all of the potential outcomes of 14 MEWF was previously the Ministry of Environment and Climate Change at the start of the RAS. 38 this program. However, the desk review of project documents, input from the TTL, and interviews with two Climate Change Directorate stakeholders helped to document how effectively the RAS contributed to building institutional capacity in key areas. As described in Table 20, the RAS contributed to key intermediate outcomes that could help to facilitate needed institutional capacity changes over the longer term. Despite the period of political transition and repeated turnover of MEWF personnel, RAS events and reports improved the motivation and ability of some GoR officials in line ministries and agencies for focusing on the climate change agenda. With the improved awareness and access to information, Romanian policymakers are now in a better position to articulate and argue for integrating climate change concerns into cross-sectoral public policies at the local, national and international level. Table 20. Results for the Climate Change RAS Identified institutional capacity Contribution of RAS for addressing capacity challenge challenge Weak stakeholder ownership— Knowledge gained: RAS activities increased the motivation of Romanian Political leaders have not officials in some ministries and agencies to focus on the climate change demonstrated adequate agenda. However, there was frequent change in the MEWF leadership and commitment for integrating key staff during the project and climate change remains a low priority issue climate change action into of many line ministries and local governments. Their involvement is development plans at the national partially due to the EU requirements and the availability of EU funds. In and local levels addition to knowledge sharing and awareness raising, they need to have continuous incentive to address climate change Inefficient policy instruments— Capacity change emerging: RAS activities and outputs supported the A comprehensive, operational, MEWF in developing a comprehensive strategy and action plan for 2016- updated national climate change 2030 that covers both adaptation and mitigation, is based on analyses, and strategy is needed with an action works across key sectors. The GoR approved and published the national plan to clarify roles across strategy in late September 2016 with deadlines and budget allocations across organizations and increase economic sectors. However, the limited capacity of the GoR will represent a compliance with regulatory challenge for implementing proposed actions. mechanisms Limited organizational Knowledge gained: The program successfully identified a set of climate effectiveness— Relevant actions by sector and assisted the GoR in integrating them into the 2014- ministries and agencies involved 2020 operational programs for EU funds, which have now been approved by in the climate change and green the EU. However, low implementation capacity remains due to the turnover growth agendas need capacity and of MEWF leadership and staff, lack of the right experience and skills in line skills to plan, implement, ministries, and limited coordination among relevant line ministries and other monitor, and evaluate climate stakeholders. change actions Limited organizational Knowledge used: Sectoral and macroeconomic models were developed and effectiveness—The GoR needs an applied in the strategy and action planning studies. The CGE model for analytical base for impact macroeconomic assessment was transferred to the National Commission of assessments and climate-related Prognosis. RAS outputs of other sectoral models (energy, transport, urban, decision-making and water) were transferred to MEWF and to other relevant agencies through training and workshops. Additional knowledge, skills, and experience is needed among ministry and agency staff to apply sectoral analysis tools and macroeconomic models for policy analysis and decision-making processes. Limited organizational Knowledge used: RAS activities supported MEWF in restructuring NCCC effectiveness— The National in the spring of 2015 by Government Decision to enhance its role and ensure Commission on Climate Change a broader representation of ministries or agencies with competencies in (NCCC) has not been an effective climate change areas. The NCCC was not yet operational at the end of the 39 body for policy coordination RAS and finally met for the first time in April 2016, so evidence is not yet since its inception, meeting available for whether it will be serve as the needed cross-sectoral and inter- inconsistently and operating ministerial platform for implementing the climate change agenda. under the sole responsibility of MEWF Note: Ratings are based on a six-point scale:No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The Climate Change RAS helped to develop new awareness, knowledge, skills, and resources that could increase the capacity of relevant agencies and ministries to plan, implement, and monitor climate change actions. The program conducted a set of rapid sectoral analyses in key sectors. The GoR was able to draw on RAS activities and outputs to integrate needed climate actions by sector into the 2014-2020 operational programs for EU funds. The RAS also worked with a range of institutions to develop an analytical base for making informed decisions, including the operationalization of the climate change strategies for 2016-2030, development of the 2016-2020 action plan for climate change, and identification of carbon financing opportunities. As illustrated by the stakeholder perspectives shared in Box 10, the RAS reports and events played an important role in highlighting the importance of climate change actions— particularly for the key sectors of water, agriculture and rural development, forestry, energy, transport, and urban development—and motivating management and staff to engage more in the activities and decision-making. Box 10. Example Reflections on Successes and A reformed National Commission for Challenges for the Climate Change RAS Climate Change is critical for the “In terms of gaining the cooperation of other successful coordination and ministries, there was very good progress. We implementation of climate change policy. succeeded in having some high level representatives The NCCC has not been effective since for the conferences, including key line ministries.� its inception, largely because it rarely met in a consistent manner and operated “The conferences really showed progress in terms of under the sole responsibility of MEWF to the presentation of line ministries and the updating mainly approve joint implementation or enrichment of information they shared there. projects. MEWF, with the help of the When a ministry got onboard, this was reflected in Climate Change RAS, restructured the the quality of the presentation and the discussions National Commission on Climate Change were more vibrant. This was so important to raise (NCCC) in the spring of 2015 to enhance this awareness of what does climate action mean.� its role and ensure a broader representation of ministries or agencies “The sectoral studies listed in the action plan—not with competencies in climate change all of them will be achieved due to some budgetary areas. At the point of RAS completion, restrictions and lack of human resources with the however, the NCCC was still not right experience. A practical problem is that the operational and a cross-sectoral and inter- budget is set annually not multi-annually, so it’s hard ministerial platform was still critically to link the multi-year [climate change] action plan to needed. The first meeting of the NCCC line institutions.� was scheduled for early April 2016, and Source: Climate Change Directorate, MEWF evidence is not yet available to gauge the stakeholders effectiveness of the new arrangements. 40 Overall, clear challenges remain for planning, implementing, monitoring, and evaluating coordinated multi-sectoral climate change actions. After a lengthy review process, the national strategy and action plan were approved and published by the GoR in late September 2016. There is still not a proven platform for cross-sectoral coordination, and the policies and measures for mitigation and adaptation proposed as RAS outputs require an integrated approach to ensure effective and efficient implementation, since measures within one sector potentially influence others. The implementation capacity of relevant ministries and agencies, including MEWF, remains a concern, with a continued need to increase staffing and enhance staff skills. Ministry of European Funds Two of the Bank’s RAS engagements under the Managing Authority for POAT were completed in sequence for the Ministry of European Funds (MoEF). The first was to provide technical assistance for improving the national framework for preparing and implementing public investments projects and the second was to help establish a performance appraisal system for GoR officials managing EU funds. At the start of the first RAS on the national framework for public investments, Romania was programmed to receive substantial funding from the European structural and cohesion funds. However, Romania’s absorption of EU funds was the lowest among EU 10 countries, with constraints across the entire project cycle. The Authority for the Coordination of Structural Instruments (ACIS) therefore requested support from the World Bank to identify concrete proposals for streamlining core procedures related to the public investment cycle. In response to this request, the objective of the RAS was to provide the GoR with actionable proposals for rationalizing the bureaucratic procedures that negatively impact the timeliness and quality of public investments and enhancing the administrative processes and guidance to improve the quality of public investment outcomes. As described in Table 21, the RAS was designed to address challenges related to the current regulatory framework and initiated the change process toward more efficient policy instruments. One design flaw was that the primary client lacked authority for addressing many of the problem areas. Also, existing cabinet and CoG processes were too weak to address topics requiring heavy inter-sectoral coordination. One assumption had been that the Prime Minister's Delivery Unit, having been established to help strengthen the CoG, would play an essential role in achieving the RAS objective. However, the recommendations of this RAS were not adopted into the DU agenda, and the DU has not yet proven to be effective over the longer term. The MoEF is better informed regarding the issues that affect the Public Investment Management Framework. The three-volume final report delivered by the RAS included matrices with recommendations, potential timing, impact, and institutional responsibilities that were expected to inform the Ministry’s action plan. The client also requested two follow-up RAS engagements to develop a performance appraisal system for government officials managing EU funds and for strengthening public investment management. 41 Table 21. RAS Results for the Romania Public Investment Framework Identified institutional capacity challenge Contribution of RAS for Addressing Capacity Challenge Inefficiency of policy instruments-- The current Change initiated: RAS recommendations were based on regulatory framework (Law 500/2002 on Public process mapping for key permitting procedures within the Finance and Law 68/2010 on Fiscal and Budgetary regulatory framework and a detailed analysis of the Responsibility) is fragmented and inconsistent and public finance law. However, MoEF (the primary client) does not guide the efficient use of public investment lacks direct institutional authority over many of the resources. The team also identified significant problem areas. Also, some high impact areas such as weaknesses in inter-sectoral and sectoral policy public procurement practices were considered by GoR to frameworks that should be guiding the identification be beyond the scope of the project. of public investment priorities. Limited effectiveness of organizational Knowledge gained: The RAS report included draft arrangements—the MoEF needs to strengthen the action plans with recommendations for improving the prioritization and management of public investments efficiency and effectiveness of public investment. The for improved efficiency and effectiveness MoEF plans to develop an action plan for GoR. Raised awareness about capacity needs has led to requests for two follow-up RAS engagements. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. To strengthen absorption and improve the effectiveness of officials managing EU funds, the GoR and the EC agreed upon a Priority Action Plan for strengthening government effectiveness which included the introduction of a performance appraisal system. The Romanian public administration has had elements of a performance management system at the national and regional levels, but a unified central system for all entities managing EU funds was needed for the new programming period. As a follow-up to the RAS for strengthening the public investment framework, the World Bank provided advisory services to the MoEF to strengthen the performance management system for GoR officials involved in the management of EU Structural and Cohesion funds. The RAS was designed to analyze the revised institutional structure of managing authorities for the new programming period with a focus on streamlining workflow and job functions for improving performance, conduct a job analysis, and develop concrete qualitative and quantitative indicators for the performance management of the identified job functions. The MoEF envisioned that the new performance management system would act as an incentive for civil servants responsible for managing EU funds by setting and regularly measuring goals for managing individual performance and providing periodic opportunities to test and objectively evaluating staff competence. As described in Table 22, the MoEF is implementing the new performance appraisal system developed as a result of this RAS to assess the impact on the performance of approximately 1,000 civil servants involved in managing EU funds. However, it is too soon to assess whether this new system has contributed to enhanced institutional capacity over the longer term related to improved effectiveness of organizational arrangements. Example perspectives on the need for more time and effort to sustain longer-term changes is in Box 11. 42 Table 22. RAS Results for the Establishment of a Performance Appraisal System for Government Officials Managing EU Funds Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited effectiveness of Knowledge used: The transfer of knowledge has been confirmed as the organizational arrangements— client piloted the new performance appraisal system and then rolled it out to Romania does not have a unified all targeted job functions for civil servants managing EU funds. The client system across organizations for participated in developing the system and took the lead in developing the assessing the performance of implementation guidelines. However, no evidence is available yet as to officials responsible for managing whether the system is helping to improve organizational performance and, EU funds. ultimately, increasing the effectiveness of the public sector for absorbing EU funds. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The RAS team’s collaboration with MoEF in this case was challenging at times and provided an interesting example how the World Bank and GoR can effectively work together. The RAS was planned to start following revisions to the institutional structure for managing EU funds under the 2014-2020 programming period. The overall timeframe was therefore very tight for delivering outputs on the timeline set by the European Commission. In addition, the GoR and World Bank teams started with different understandings of the task. The client described “strong debates� at the start and a “confrontation� with the international expert. Nonetheless, both parties viewed this RAS as a successful engagement at completion. The main explanatory factors included that the TTL was responsive and willing to discuss options, the client was strong and actively participated in decisions, and local experts were engaged to inform the process and serve Box 11. Views on the need for more time and support to improve the management of EU funds “The indicators developed for performance management were technically accurate, but the problem is that people are not usually working with such indicators. In my opinion further guidance and training is needed. Also, a lot of data means an administrative burden, so we have to think in practical terms what is useful.� “We were in a hurry to finish the system and we needed another six months to have discussions with all the managers and explain how to discuss system with an employee. We noticed fear among colleagues, with everyone afraid that if they didn’t get a rating they would lose the ‘top up’ to their salaries.� “The system motivates people to keep track record for more accountability day by day but that is hard to link so far to organizational performance. The traceable system is good though—at last you can prove somehow what was done for the objectives that you had whereas in the past you had only a personal judgment that anyone could contest.� Source: Interview with MoEF stakeholders as intermediaries. 43 Ministry of Public Finance Romania’s efforts to transfer investments from the state budget to EU funds have been challenged by inefficiencies in the preparation and management of public investments. In this context, the World Bank received a request from the MoPF to improve the effectiveness of public investment management, rationalizing the existing project portfolio, and strengthening the link between prioritized projects and resources from EU funds and the state budget. The PIM RAS reflected an ambitious effort by the World Bank to deliver 14 analytical reports within 15 months. Project activities focused primarily on diagnostic and analytical activities in order to develop a more effective and efficient PIM system. Therefore, the main targeted institutional capacity change targeted by the RAS was to increase the effectiveness of organizational arrangements as described in Table 23. In fact, the RAS outputs potentially inform improvements for a broad range of organizational functions within the MoPF— particularly for the Public Investment Evaluation Unit (PIEU)—but these are grouped together under one summary rating in terms of the available evidence for whether RAS activities have resulted in measurable changes in organizational capacity and performance related to public investment management.15 Overall, the review of project documents indicated that RAS activities initiated the change process for contributing to the improved effectiveness of the PIM system. The diagnostic exercises provided valuable information about the existing strengths and weaknesses of the organizational structure, technical expertise and staff capacity, information systems, and other aspects which informed detailed recommendations for a proposed strategy and action plan presented in the final report. However, it is not yet clear how or whether this information will be used by the client. In the RAS Activity Completion Summary, the team noted that “it is highly likely that further technical assistance would be needed in order to guide the GoR on the implementation of the suggested reforms.� To a lesser degree, the RAS was also expected to help build leadership commitment to the PIM reform. However, the RAS team encountered unexpected challenges during implementation and there was no evidence that the activities increased stakeholder ownership of the PIM reform. The PIM RAS was originally strongly supported by the Minister of Budget – a position which has since been abolished. Since then, the ownership and interest in PIM reforms has decreased and there was no high-level administrative or political figure who could act as champion and convene and unite PIM-stakeholders behind the necessary reforms to facilitate their implementation. Also, the legal agreement was based on the assumption that MoPF and pilot ministry officials would play a much more active role in the project implementation, while the Bank experts would play more of a facilitative role. In reality, the participation of government officials was rather limited. 15 No client interviews were conducted to explore the results of the PIM RAS, so the overall assessment is based on the review of project documents and does not reflect any progress achieved after the final report (November 2015). 44 Table 23. Results of the RAS to Strengthen Public Investment Management Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited effectiveness of Change initiated: The RAS final report provides input for a PIM reform organizational arrangements—the strategy and action plan that addresses the need for a more effective PIEU, GoR needs an effective and a stronger gatekeeper role for the coordination and preparation of new efficient PIM system with a clear public investments, adequate information systems to guide decision- link between prioritized projects making, improved capacity to plan PPP projects, and clear criteria and a and resources from EU funds and methodology for rationalizing the existing public investment portfolio. the state budget. However, it was not yet clear at the close of the RAS period whether the GoR would endorse the recommendations or be able to implement them. Weak stakeholder ownership— Change initiated: a “continuous stakeholder consultation� was included in There is weak political acceptance the design to help foster buy-in for the reform at different levels, validate of the PIM reform. Increasing the findings and recommendations, and facilitate agreement on the reform challenge function of the MoPF agenda. However, the RAS team encountered unexpected challenges could lead to potential issues from during implementation and there was no evidence that the activities other sector ministries that could increased stakeholder ownership of the PIM reform. stall the reform process. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. In reporting lessons, the RAS team noted that defining a RAS based on legal or political requirements for actions to be taken by the client may provide for a clearer mandate and more active client participation. While all elements of the PIM RAS were crucial to achieve reform outcomes, only few of them were based on existing and clear legal or political requirements. For example, there was no requirement to rationalize the portfolio or improve the quality control despite the importance of such reforms to solve the overarching PIM-challenges. For the time being, such reforms hinge on the political willingness to go ahead and create the necessary legal framework. RAS activities early on identified problems in the current legal and regulatory framework, such as changes needed to strengthen MoPF’s quality assurance and independent review functions, for supporting portfolio rationalization, and introducing multi-annual commitment appropriations for public investment projects. However, the RAS was not designed to change laws or policies (i.e. to achieve progress related to the “efficiency of policy instruments�) but rather to enhance the capacity of the MoPF by providing guidance and recommendations to do this in the future. Therefore, no progress ratings are assigned for this type of institutional capacity change. National Agency for Cadastre and Land Registration The World Bank provided technical assistance to the National Agency for Cadastre and Land Registration (ANCPI) to strengthen the agency’s capacity to improve its policy, regulatory and institutional framework. RAS activities and outputs reflected an ambitious effort to inform the development of a strategy for land registration in Romania, revise ANCPI’s legal and institutional framework, and develop a national program for improving the coverage, efficiency, service delivery, and responsiveness of real estate registration in Romania. While no client counterparts for this RAS were interviewed as part of this evaluation, the project documents provided detailed information about existing constraints and RAS results. This 45 information was used to assess the extent to which RAS activities and outputs effectively contributed to longer-term institutional capacity changes, as described in Table 24. Some progress was achieved for two types of institutional capacity. First, the draft National Strategy for Systematic and Sporadic First Registration and the draft legislation for the Integrated Cadastre and Land Book System both reflect improvements related to the “efficiency of policy instruments� given their potential to formally guide diverse stakeholder actions to contribute to the development objective. Second, preliminary outcomes produced by the RAS could help to strengthen the “effectiveness of organizational arrangements� for both ANCPI and for the National Program for Cadastre and Land Book (NPCLB). Table 24. Results of the Real Estate RAS (Basis for National and EU Policies) Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments—a Knowledge gained: The RAS produced a draft National Strategy for coherent national strategy for land Systematic and Sporadic First Registration defining the full scope and registration is needed to increase the requirements to complete the National Program of the Cadaster and Land coverage and efficiency of the Registration. The strategy was informed by a comparative study of real system, increase transparency in the estate registration systems in EU and other countries. It was finalized economy, and provide equal rights to and serves as a reference for ANCPI; however, the strategy has never all landowners been formally adopted by the GoR nor has ANCPI been assigned any responsibility to implement it. Inefficient policy instruments—A Knowledge used: RAS outputs contributed to the development of a draft new legal framework for cadastre legislation on Integrated Cadastre and Land Book System that is and land registration is needed appropriate for the Romanian context and aligned to EU good practice. aligned to EU good practice in terms The law is ready to be submitted to Parliament for approval. of the principles of corporate governance and service delivery Limited organizational Change initiated: The RAS conducted an institutional assessment of effectiveness—ANCPI suffers from ANCPI and developed a detailed action plan based on the review that political and institutional instability provides guidance and performance indicators for monitoring progress of and needs to reform its institutional actions in 7 thematic areas. The action plan was not adopted and will framework to develop adequate need updating. Frequent management changes in ANCPI have resulted capacity for carrying out its mandate in limited buy-in for implementing the plan in the future. Limited organizational Knowledge used: RAS activities supported the development of the Draft effectiveness—a coherent national National Program for Cadastre and Land Book, the analytical work to program is needed for cadastre and support the EU financing proposal on property registration (ROP 2014- land registration 2020), and standards for reporting and planning to support the implementation of the new national program. ANCPI accepted NCLB and the GoR approved using land registration revenues to finance systematic registration. More time will be needed for evidence to emerge related to the program’s implementation and its outcomes. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The ongoing political and institutional stability of ANCPI presented substantial challenges during RAS implementation. ANCPI has had frequent transitions, with 12 Directors General over 17 years. One major institutional change during the RAS was the transfer of ANCPI from the General Secretariat of the Government to the MRDPA. Also, while the fees ANCPI’s services generate could fund, as in many other countries, a program to establish a comprehensive cadaster and land registration system, the agency instead has been a source of profits for the central government. In February 2014, the GoR approved an emergency ordinance establishing 46 the agency as a partially fee-financed agency, which completely changed the agency and created new legal requirements for ANCPI to respond to in a tight timeframe to prepare the NPCLB. Sustained efforts will be needed to complete the policy, regulatory and institutional reform processes that were started under this RAS. ANCPI has therefore requested follow-up support from the World Bank under a new RAS. The objective of this technical assistance is to support ANCPI in implementing its investment programs and legal and institutional reforms, and in developing new electronic services using the improved digital records as the base. Regional Development Agency South Muntenia The Regional Operational Program for 2007-2013, discussed previously, included an analysis of the Integrated Development Plans prepared for the growth poles and ways that these could be improved. The Office of the Growth Pole Coordinator in Ploiești and the Regional Development Agency South Muntenia reviewed this RAS work and requested that the World Bank help to implement recommendations through a new RAS Agreement focused on the Ploiești Growth Pole. This metropolitan area includes the city of Ploiești and 13 neighboring localities. The main activities of the RAS included helping the Ploiești Growth Pole Coordinators Office to update the Ploiești Integrated Development Plan (IDP) for the 2014-2020 Programming Period and to offer recommendations for an improved institutional framework for the implementation of the 2014-2020 IDP. Building on the methodology the World Bank prepared for the MRDPA, the team proposed an integrated package of priority projects for the Ploiești Growth Pole, the first time this was done in Romania. All the major cities in Romania will have to prepare similar priority project lists, and the Ploiești Strategy is expected to serve as an example for them. As described in Table 25, the RAS helped to shape formal incentives to guide stakeholder actions across organizations through the design of an Integrated Urban Development Strategy (IUDS) and a proposed institutional framework. The IUDS also serves as an important resource and model for other cities and will potentially enhance their planning capacity for developing their own strategies. The RAS team managed to bring several stakeholders together (the Ploiești City Hall, the Prahova County Council, the Prahova Prefect, the local authorities that are part of the Growth Pole, public companies, the local university, the private sector, and the citizens) around the shared goal of determining the key projects for the Ploiești Growth Pole through 2023. The scope of the development strategy was then adjusted from a list of around 600 projects to a priority list of 22, given the implementation capacity of the local authorities. RAS outputs included recommendations for a new institutional framework that highlighted various concerns with the current system. However, it is not yet clear to what extent these recommendations will be implemented. Also, the local authorities in Ploiești faced a number of challenges during the implementation of the project (e.g. the mayor was arrested and subsequently the acting mayor was ousted by the Local Council). As such, much of the dialogue with one of the most important clients of this work was done at a technical level and increased leadership commitment is still needed. 47 Table 25. Results of the Ploiești Growth Pole RAS Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy Knowledge used: The RAS supported the development of an integrated urban instruments—an integrated development strategy with a proposed institutional framework and a priority development plan is needed project list through 2023. However, substantial barriers remain to to guide the efficient implementation—including inadequate buy-in from local authorities in Ploiești, implementation of projects limited implementation capacity among key stakeholders, and unexpected changes in the defined metropolitan to the ROP guidelines for financing in the new programming period. area Limited organizational Knowledge used: The IUDS for the Ploiești Growth Pole is viewed as one of the effectiveness—local best sub-national development strategies in Romania since 1989. It is being used authorities lack adequate as a model for several other areas, including the Brasov Growth Pole, the Iasi capacity for metropolitan Growth Pole, the Cluj Growth Pole, the Constanta Growth Pole, and the City of planning Targoviste. More time is needed for evidence to emerge as to how well this resource helps to improve integrated development planning in urban areas, particularly now that the previous Growth Pole approach has been discontinued under the ROP 2014-2020. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Tulcea County Council The focus of the World Bank’s RAS collaboration with the Tulcea County Council was to provide support for the institutional capacity development of the Danube Delta Inter-communal Development Association (ADI) to coordinate the Integrated Territorial Investments (ITI) instrument. The ITI is a new EC instrument intended to draw on resources from different EC- funded operational programs to finance priority investments based on a development strategy. In this case, the ITI is intended to be the main funding mechanism for implementing the Danube Delta Integrated Sustainable Development Strategy. RAS activities were designed to support the development of the human, procedural and logistic resources needed to support the preparation and implementation of the ITI. Specific outputs included a capacity building report, operational manual, and organizational report. The team also provided training with the aim of enhancing capacity related to communication and visibility, financial management, procurement, project selection, monitoring and evaluation, and general management. As summarized in Table 26, the RAS contributed to intermediate outcomes for building the organizational capacity of the ADI. While the planned knowledge products and initial training were delivered, there remain substantial challenges for implementing the ITI. These include:  Frequent changes in ADI management during the last couple of months of the RAS implementation period. Leadership commitment to using the RAS outputs under these conditions has been unclear.  A need to address organizational structure and protocol issues. An analysis of the Articles of Association of ADI ITI DD highlighted key issues and presented recommendations. These mainly concerned protocol issues that are important from a legal and administrative point of view. Also, the current structure has some overlapping responsibilities such as between the Beneficiary Liaison Department and the MA Liaison 48 Department. Recommendations to clarify roles and responsibilities were made, but these require commitment and actions by the client organization to implement.  A risk of turnover among trained ADI staff. Continued support and commitment is needed to retain the training participants and facilitate their application of new knowledge and skills. Table 26. Results for the Institutional Capacity Development of the ADI ITI Danube Delta RAS Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited organizational effectiveness— Knowledge gained: The RAS delivered an operational manual to The ADI is a newly-created organization provide instructions on systems and procedures and provide training that lacks the staff skills, operational for staff. However, changes in ADI management at the end of RAS systems, and processes to carry out its implementation and the risk of turnover among staff could limit the mandate. effects. There is not yet evidence that the manual is in use to operationalize the ITI. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Despite the risks, the RAS outputs do have the potential to continue supporting institutional capacity changes related to the use of the EC’s new integrated financing instrument. The manual and reports provide recommendations for developing new organizational capacity, systems, and processes. These could form a strong foundation for the future operations of ADI-ITI Danube Delta if there is renewed commitment to implementing them. 49 V. Results for the Operational Program for Administrative Capacity Development Overview The Operational Program for Administrative Capacity Development (PODCA) focuses on creating and strengthening the legislative and institutional framework necessary for improving service delivery to support the longer-term aims of economic growth, improved social conditions, and good governance. The priority axes of PODCA for 2007-2014 were designed to address horizontal management problems at all public administrative levels (central and local) and target improvements to the decentralization of service delivery in prioritized sectors (health, education, and social assistance). Targeted beneficiaries therefore include the central administration (ministries and their subordinated agencies) and the local administration (county authorities, municipalities, and communes.16 The World Bank completed 14 RAS agreements that were financed by the client under the Managing Authority for PODCA, as listed in Table 27. The clients for these engagements included the Ministry of Agriculture and Rural Development (MARD), the Ministry of Education and Scientific Research, the Ministry of Justice (MoJ), the Ministry of Public Finance (MoPF), the Ministry of Transport and Infrastructure (MTI), and the Romanian Competition Council (RCC). One RAS was also completed directly for the MA within MRDPA. Table 27. PODCA RAS Agreements Completed by the World Bank, 2012-2015 Duratio Project Date Completion Value RAS Title Client n number Signed Date (US$ millions) (months) P143673 Assistance to the MARD MARD 28-Sep-12 27-Jun-14 21 $1.23 for Strengthening the Agri-food Sector Strategy Formulation P143674 Strategic Planning for the MARD 28-Sep-12 27-Sep-14 27 $1.28 Agricultural Administration P143675 Implementation of an MARD 28-Sep-12 27-May-14 20 $0.63 Internal Management System at the MARD and its Subordinated Structures P143676 Developing an Integrated MARD 28-Sep-12 27-Sep-14 24 $0.37 Financial Management System at the MARD P143659 Development of MoESR 25-Oct-12 24-Sep-15 35 $1.97 Administrative Capacity of the MoESR P129957 Assistance to the Ministry MoJ 17-Feb-12 01-Mar-12 13 $0.75 of Justice to Assess the Performance of the Judicial System 16 Ministry of Interior and Administrative Reform. 2007. Operational Programme Administrative Capacity Development. Government of Romania. 50 Duratio Project Date Completion Value RAS Title Client n number Signed Date (US$ millions) (months) P133720 Assistance to the Ministry MoPF 1-Oct-12 31-May-14 20 $0.62 of Public Finance for Strengthening Debt Management P144505 Improvement of Human MoPF 13-Dec-12 12-Apr-14 16 $0.37 Resource Management Instruments to Strengthen the Institutional Capacity of MoPF P144557 Strengthening MoPF 23-May-13 22-Sep-14 16 $0.53 Institutional Capacity in Fiscal Policy Formulation P144566 A Better Solution for MoPF 23-May-13 22-Dec-14 19 $0.44 Providing Clarifications, Interpretations and Rulings on Tax Issues P130508 Strengthen Strategic MTI 19-Mar-12 08-Dec-14 32 $3.95 Planning in the Transport Sector P130510 Prepare and Manage MTI 6-Sep-12 11-Dec-15 39 $1.12 Public Private Partnerships P133582 Analysis of capacity MRDP 10-May-13 9-Jan-14 8 $0.37 building activities in the A Public Administration P131824 Assistance to the RCC 4-Jun-12 3-May-15 35 $3.28 Romanian Competition Council Ministry of Agriculture and Rural Development The modernization of the Romanian economy and progress toward higher living standards over the past two decades have largely bypassed the rural economy. Transforming agriculture is therefore central to Romania’s European integration and social cohesion objectives. In 2012, the Ministry of Agriculture and Rural Development embarked on a far-reaching reform agenda that covered both policy and public administration. The program consisted of four interrelated RAS agreements signed with the World Bank intended to strengthen agri-food sector strategy formulation, develop strategic planning capacity within the agricultural and rural administration, implement an internal management and control system, and develop an integrated financial management system in the sector administration. The RAS for supporting the agri-food sector strategy formulation was designed to address inefficiencies in the current strategic and operational management of public policy cycles by MARD. For example, public policies in agriculture and rural development were driven mainly by a sectoral perspective (e.g. cereals, milk, fruits and vegetables). Gaps in horizontal issues – such as land and structural reform, taxation in agriculture, value chain and market integration, 51 and professionalization of farmers – were systematically overlooked. Key RAS components included developing a medium/long-term vision for the ARD sector, producing policy papers and technical notes to improve the effectiveness of policies, and proposing a new integrated solutions architecture to improve data management for decision-making. As summarized in Table 28, the RAS was effective in supporting the development of a transparent sector-wide strategy that reflects broad stakeholder input and is consistent with related policy documents. The strategy coherently integrates EU and non-EU programs and services, has received the needed clearances from ministries in the field, and is now available on the MARD website. Implementation of the strategy will require sustained effort that starts with comprehensive institutional reform for the agricultural administration to modernize and digitize its systems and procedures, strengthen its human capacity to better meet current and future business needs, and effectively manage the risks faced by farmers. Therefore, RAS activities also developed a prototype for the ARD sector information database and a proposed integrated solutions architecture to support strategic analysis and reporting. Feedback from the project team and from the client indicated that the ARD Sector Information Database Prototype focused on the livestock sector generated important new knowledge and skills about how to process, analyze, and report on data to form better management and decision- making. However, RAS outputs did not include a fully-functioning database and resource constraints inhibited further design and implementation progress within MARD. In a new collaboration between MARD and the Institute of Statistics to establish a similar database, it is not year clear to what extent the prototype and lessons learned from the World Bank RAS will inform the new design. Table 28. RAS Results for Strengthening the Agri-food Sector Strategy Formulation Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy Knowledge used: MARD has drawn on RAS outputs to establish a long-term instruments—The lack of a sector vision and strategy (2020-2030) with objectives and targets. The long-term vision to guide documents are based on broad stakeholder consultations, reflect consistency with sector development and related policy documents, and reflect improved transparency in the sector. RAS public interventions has activities and outputs were valued highly by the client, but substantial work was slowed change and limited required by MARD to operationalize the vision provided by the RAS team and the effectiveness of policies develop a detailed strategy with outcome indicators. Limited organizational Knowledge used: RAS outputs included a prototype for the ARD sector effectiveness—MARD has information database and an integrated solutions architecture to consolidate data inadequate sector information into a single warehouse to address problems related to fragmented data management to inform management, inefficiencies in data collection and storage, and inaccuracies in policymaking. data reporting. Client reported that prototype and guidance have improved understanding of data management for decision-making within the MARD, but it is not clear that the Institute for Statistics will decide to use RAS outputs (including prototype) in its current collaboration with MARD. Note: Ratings are based on a six-point scale:No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The second RAS in the reform agenda for the MARD was focused on increasing the capacity for strategic planning within the agricultural administration. The main activities included supporting 52 the drafting of an institutional strategic management plan, analyzing and providing proposals for the restructuring of the agricultural administration, training for staff to increase awareness and knowledge related to institutional strategic planning, and developing an ICT prototype for strategic planning and performance assessment to serve as a feedback loop for policy formulation. As part of the larger reform program, the second RAS faced some inherent challenges given that the development of an institutional strategic plan would normally follow the development and adoption of a sector strategy. The sequencing of activities should be to start with the sector strategy to determine what actions and resources are needed at the institutional level to achieve the medium and short term objectives and to monitor implementation. Despite the fact that the sector strategy has only recently been formally disseminated by the MARD, the Strategic Planning RAS was effective in enhancing the organizational capacity of the agricultural administration for strategic planning, as summarized in Table 29. Table 29. Strategic Planning for the Agricultural Administration RAS Results Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Limited organizational Capacity change emerging: The RAS supported the drafting of an Institutional effectiveness—The ARD Strategic Management Plan (2014-2017) with programs, measures and budgets and administration lacks a performance assessment framework. Training sessions helped develop awareness adequate strategic planning and knowledge among MARD staff. An ICT prototype for strategic planning and capacity for effective performance assessment was designed to track progress against targets. Feasibility strategic management and studies on paying agencies and the decentralization of the ARD administration policy making. provided information on public administration reform options reported to be highly valued by the client. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The Strategic Planning RAS placed significant emphasis on involving client ministry staff closely and directly in the development and testing of the new strategic planning tools. Also, the RAS made ongoing efforts to train client ministry staff in the use of the new tools, as well as to share good international practice. Surveys and interviews conducted at RAS completion indicated notable improvements in the knowledge and awareness of the client ministry management and staff. Comments from the client—as shared in Box 12—highlighted the value added by the RAS team not only for MARD but also potentially for other ministries throughout the GoR working to strengthen their strategic planning functions. 53 Frequent political changes remain the main risk to the Box 12. Contributions of the RAS for sustainability of RAS results, as the new leadership Building Strategic Planning Capacity may not be aware of or discount some of the results achieved by its predecessors. Also, the lack of “This project provided the most financial resources to develop information systems that concrete results and contributed to developing our strategic planning would streamline both decision making and capacity. Two experts worked closely administrative processes in the MARD is an additional with our team to develop an risk. To build on the current momentum, such systems institutional strategic plan. Then the IT should be created while the knowledge and awareness prototype was considered to be best developed through the RAS are still fresh. Although practice by the Public Policy the client ministry is lacking the financial capacity to Directorate and the Chancellery of the take this work forward and to roll out a full-fledged Prime Minister. They took it and strategic planning system in the agricultural multiplied it for use in other ministries administration, this risk could be mitigated somewhat as well.� if a strong commitment for better strategic management processes was made at the central Source: Interview with MARD government level, and all line ministries were held to stakeholder higher standards of policy quality and accountability. The third RAS implemented by the World Bank as part of the MARD reform agenda was focused on the implementation of an internal management system for the client ministry and its subordinated structures. RAS activities included reviewing the implementation status of the internal management and control system (IMCS) for MARD and the Agriculture Payments and Intervention Agency (APIA), Paying Agency for Rural Development and Fisheries (APDRP), National Agency for Livestock Amelioration and Breeding, National Agency for Fisheries and Aquaculture, State Domain Agency, and four County Agricultural Directorates selected to ensure regional representation. Other RAS components included developing modern management tools necessary to ensure efficient internal management/control systems and support for operating them, strengthening the human resource capacities of the MARD to develop the internal management/control system through dedicated training, and assessing progress for the implementation of the management tools. The diagnostic review conducted at the start highlighted important shortcomings of the existing system to inform improvements, including:  Self-assessment reports, as stipulated in MoPF Order 946/2005, are not sufficient to assess the real level of IMCS implementation. The reports are based on personal declarations of the senior managers of each institution and are subject to interpretation and subjectivity.  The assessment of MARD, APIA, and APDRP revealed that IMCS implementation is mainly driven by the obligation to strictly comply with the legal provisions in place, without a real impact on the efficiency, effectiveness, and financial side of the operations. Nearly all of the institutions have instituted ad hoc internal working groups and internal structures for IMCS implementation.  There is a strong need for hands-on training for the IMCS coordinators. All staff in general and new staff in particular need to be included in an institutional awareness program on the rules of IMCS. 54 As described in Table 30, the RAS was effective in strengthening the IMCS for MARD and selected subordinated institutions. The RAS assisted the client in developing a new framework of policies and procedures while streamlining and aligning the standards and controls to the internal legal requirements and international practice. These changes were supported by a proposed IT platform with specifications for a document management system and dashboard reporting for MARD. The system was also designed to assist new staff members become effective in their roles and have a clear understanding of how the institution operates soon after their recruitment. Table 30. RAS Results for Implementation of an Internal Management System at the MARD and its Subordinated Structures Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited organizational Knowledge gained: RAS outputs provided a comprehensive framework effectiveness—the MARD and for an improved internal management system which the MARD leaders (at selected subordinated institutions the time) committed to implement. The RAS developed an understanding need standard processes, systems, of a sound framework of policies, procedures and controls that is neutral and controls to guide and monitor to management changes and provided a staff training plan that is the performance of organizational compliant with international and European best practice. The lack of funds functions for increased operational for the development and implementation of the proposed approach is a key efficiency. risk to reaching the expected results. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The wide participation of staff in meetings with the RAS team and their active contribution to the RAS activities enhanced their exposure to international practice. They gained knowledge through participating in the process and through training on the implementation of the new system (IMCS) and should be prepared to contribute to system development. Despite the intermediate outcomes achieved by the RAS, there remain some key risks for sustaining the institutional capacity change process for increasing operational efficiency at the MARD and its subordinated agencies. Although the client has produced and distributed a manual outlining standards for the IMCS, the MARD has been challenged to formulate data sheets for the standards and did not receive the expected support from the RAS team for this process. In addition, the lack of resources to support the development and implementation of the proposed approaches has delayed the implementation of the action plans, including the staff training plan. Increased leadership commitment to the principles and components of the IMCS is needed to reduce the risks and promote implementation. The fourth RAS implemented by the World Bank as part of the reform agenda for the MARD focused on the development of an integrated financial management system (IFMS). RAS activities included evaluating the performance of the existing financial management system of the MARD and its subordinate agencies, identifying how the systems could be integrated, proposing solutions through the adoption of modern financial management tools, and providing the basic functional specifications for the IFMS. RAS activities initiated the change process for contributing to improved financial management for the MARD, as detailed in Table 31. RAS outputs were designed to address identified weaknesses, including the lack of coordination and integration across financial management functions and across subordinated units and agencies at MARD and the need for automation to 55 standardize manual-intensive processes or ad hoc solutions. However, MARD has yet to start implementing the IFMS, with the first release limited to Accounting, Payroll and Human Resource Management. As such, other IFMS core components such as Budget Management, Cash Management, Procurement and Asset Management and Integrated Reporting are not being implemented. Further, additional plans for implementation remain unclear. Table 31. RAS Results for Developing an Integrated Financial Management System at the MARD Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited organizational Change initiated—RAS outputs included the Solution Architecture and effectiveness—Operational Migration Plan for an IFMS to improve the coordination and integration weaknesses of the MARD include across financial management functions and across subordinated units and the need for prioritization of financial agencies at the MARD and to address the need for automation. To date, allocations, a more efficient use of the lack of financial resources to implement the proposed system has led the budget, and system to track to there being no changes in practical terms in the activity of the intended expenditures on a regular basis. MARD departments. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The lack of financial resources to fully implement the system designed under this RAS has limited the effectiveness of the RAS for contributing to a sustainable institutional capacity change. A partial implementation of IFMS will not provide MARD with comprehensive, reliable and timeline financial information for planning, management and reporting. Also, the IT landscape will not be simplified through a piecemeal implementation of the IFMS, and financial accounting and management will continue to be challenged with multiple sources of information and inefficient, manual processes to transfer data from one system to another. Ministry of Education and Scientific Research As a result of the Functional Review conducted by the World Bank in 2010 and 2011 for the pre- university sector in Romania, the MoESR requested assistance in enhancing its administrative capacity and in developing the management and leadership skills of its managers and staff. Activities for the RAS to assist the MoESR in developing its administrative capacity focused mainly on reorganizing the MoESR and helping to improve its organizational culture and management practices. Information on the RAS results was collected through the TTL survey, in-depth interviews with six client stakeholders, and a review of project documents. As described in Table 32, the RAS provided integral support for strengthening the effectiveness of the Ministry. An innovative approach was used by the RAS team for analyzing organizational culture, management and organizational structure issues, some of which reflected very sensitive areas. Management and staff participated in extensive debates around the desired organizational culture and helped to design action plans on how to achieve needed changes. The team reported active engagement of key stakeholders throughout the process, with 85 percent of all Ministry staff participating in surveys and key outputs disseminated and debated in all 42 county school inspectorates in Romania. 56 Table 32. Results from the Assistance to the MoESR for the Development of Administrative Capacity Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Limited effectiveness of Capacity change emerging: The RAS produced new Regulations for organizational arrangements— Organization and Functioning for organizational units of the MoESR and all the pre-university school job holders in the school system at the management and expert levels. These system* does not have clearly ROFs were formally promoted as management tools by the Minister in defined roles, functions, and September 2015 and evidence regarding their effectiveness will emerge as responsibilities they are implemented. Limited effectiveness of Knowledge used: RAS activities included an analysis and report on the organizational arrangements— organizational culture in the schooling system examining the alignment of the MoESR needs clarity of customer, public, staff, and management orientations. The diagnosis provided mission. Differences in values important new knowledge and recommendations were incorporated by the and beliefs among school, MoESR as part of a broader training program in September 2015. ministry, and inspectorate personnel reflect the need for a stronger, more consistent organizational culture. Limited effectiveness of Capacity change emerging: The RAS prepared a pilot and operational organizational arrangements— procedures for the delegation of authority. Client stakeholders reported that the management of the MoESR the exercise was very valuable and that the documentation and templates to needs to delegate authority to guide delegation and decision-making worked well. However, given political improve operational efficiency transitions, it is not clear the extent to which the delegation model will be rolled out and implemented over the longer term. * In this case, the targeted organization is not just the Ministry but also the entire pre-university schooling system, including the county school inspectorates in all 42 counties in Romania Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. RAS activities demonstrated the possibility of a much more efficient allocation of resources and responsibilities within the MoESR and across the pre-university schooling system. A pilot on the delegation of authority showed promising results for freeing management up from time- consuming administrative tasks. Similarly, new internal functioning regulations and job descriptions were promoted as management tools aimed at clarifying the roles and relevance of different directorates for the mission and vision of the Ministry. MoESR stakeholders recognized both the value and the ambitious scope of the change processes that had been started, as shown by the examples in Box 13. Given ongoing political transitions, the longer-term results of this RAS engagement remain unclear. However, inputs from the Bank team and the MoESR stakeholders reflected a determination to continue implementation of the planned changes in the organizational structure and culture for a more effective and efficient pre-university schooling system. 57 Box 13. Reflections on the Ambitious Changes Started for the MoESR “This work is revolutionary. Activities for each Directorate are broken down into objectives and functions and tasks. Every General Directorate will come up with managerial plans that highlight concrete issues based on functionality. I have been here for 15 years now and this is the first time we had this kind of change.� “The delegation project should be rolled out in the ministry because it allows you to unblock the managers. Managers should focus on preparing policies, not so much solving administrative issues. The General Directorate with the highest number of petitions has 5,000 per year. A huge part of the time goes to this activity. So the delegation should continue but it is up to the DG and needs approval of the management of the ministry. I don’t know of any plans to continue it.� “Looking at the organizational structure and culture was such an interesting discovery process. For example, the consultant went to the high school to see who there is responsible for learning outcomes? No one!� Source: Interview with MoESR stakeholders Ministry of Justice After Romania’s accession to the EU in January 2007, a Cooperation and Verification Mechanism (CVM) was established to help Romania address shortcomings in areas related to judicial reform and corruption and to monitor changes in these areas through annual progress reports. Starting with the 2010 Progress Report, the EC recommended that Romania launch an independent performance review of its judicial system to inform needed improvements. The Ministry of Justice therefore requested assistance from the World Bank to assess performance of the judicial system. The main objective of the RAS for the MoJ was to analyze the functioning of institutions of the judicial system in Romania and to provide analytical and advisory input to enable the Romanian authorities to formulate an action program for improving its performance. The longer-term aim was that the recommendations could lead to the end of the CVM and to Schengen accession. Key RAS activities included:  Assessment of the organizational structure and functioning of the judicial system;  Analysis of the use of human and financial resources allocated to the judicial system;  Assessment of the functionality of the ICT environment and architecture of the judicial system; and  Provision of a systematic framework to identify and mitigate risks affecting the performance of the judicial system. In addition, the RAS team organized workshops with relevant institutions individually and jointly to discuss findings and build consensus on the recommendations. As described in Table 33, the RAS provided the basis for an action plan, an important intermediate outcome for strengthening the performance of the judicial system. The detailed analysis and subsequent recommendations addressed challenges related to the overall use of the system’s resources, quality in terms of the uniform application of the law, efficiency in dealing 58 with priority cases, and access to justice. The data and analysis have informed the GoR’s policy dialogue with the EC on justice in general and on the CVM in particular. Table 33. RAS Results for the Assistance to the Ministry of Justice to Assess the Performance of the Judicial System Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Limited effectiveness of Knowledge gained: RAS activities included an in-depth analysis and organizational arrangements—the functional review of the institutions in the judicial system and their use of judicial system suffers from the lack human, financial, and ICT resources. Recommendations were based on of strategic management and broad consensus among stakeholders, including the EC. The client plans planning across a fragmented to draw on the recommendations to implement an action plan for reform system that limits its effectiveness but there is no evidence yet that changes to organizational performance and operational efficiency have been achieved. Note: Ratings are based on a six-point scale:No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Political volatility and transitions were acknowledged by the RAS team to be the greatest risk for building on the RAS results. At RAS completion, the MoJ started the process for developing an action plan based on the recommendations. Ministry of Public Finance Four of the World Bank’s RAS engagements under PODCA were conducted for the Ministry of Public Finance. Stakeholder interviews were not conducted in 2016 for this set of projects, so findings are based on the review of project documents, inputs from TTLs, and the pilot case study conducted for the RAS on Strengthening Debt Management during 2015. RAS engagements for the MoPF followed on a functional review of the public finance sector conducted by the World Bank in Romania in 2010. The recommendations from the functional review were approved and included in the MoPF’s action plan. As part of the implementation of this action plan, the MoPF requested assistance from the Bank to modernize the budget execution and treasury functions, and the Advisory Services Agreement for Strengthening Debt Management in the Ministry of Public Finance was signed in October 2012. The Strengthening Debt Management RAS was designed to help bring debt and cash management practices in line with that of other countries in the European Union by filling in the gaps in policy guidelines for debt and cash management and by improving the execution capacity of the Romanian Treasury in the MoPF. The World Bank team worked in close collaboration with client counterparts in the General Directorate of the Treasury and Public Debt (GDTPD) to complete a range of diagnostic exercises and training for the staff in the Middle Office (MO) and Front Office of the GDTPD. The effectiveness of the RAS in contributing to multiple sustainable institutional capacity changes is described in Table 34. Perhaps the most notable achievement of the RAS team was the early and ongoing progress in enhancing the capacity among MO staff for developing and implementing a medium-term public debt management strategy, which culminated in the approval of Romania’s Public Debt Management Strategy for 2014-2016. 59 Table 34. RAS Results for Strengthening Debt Management Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments— Capacity change emerging: RAS activities and outputs supported the Romania lacks a debt development of Romania’s Public Debt Management Strategy for 2013 - management strategy formulated 2015 and then for 2014-2016. The second one was confirmed by the EU on the basis of clear cost-risk and IMF as being aligned to international sound practice defined by the tradeoffs. Public Debt Management Guidelines produced jointly by the World Bank and IMF. More time is needed for evidence to emerge from the strategy’s implementation. Limited effectiveness of Knowledge used: As a result of RAS training, the Middle Office staff were organizational arrangements— able to demonstrate new knowledge and skills in aggregating portfolio data, The Middle Office of the GDTPD finding information for the forward interest and exchange rates in has limited capacity to design and Bloomberg, and calculating the cost-risk trade-offs. The MO staff drafted implement a debt management the debt management strategy. However, skills for the analysis of the strategy market conditions for both the local and foreign currency securities need to be further developed, requiring continued participation of the Front Office in the preparation of the debt management strategy. Limited effectiveness of Knowledge gained: The RAS provided policy guidelines and a review of organizational arrangements— the legal framework for cash management transactions, an infrastructure The Middle Office and Front roadmap, and training on cash management and liability operations. Staff Office have limited capacity to demonstrated increased knowledge and stills related to forecasting tools, carry out their respective roles for instruments for active cash management, cash flow smoothing, and related the planning and execution of areas. However, the RAS team observed that an internal reorganization is cash management and liability needed for staff to apply the new knowledge and skills as intended. High operations turnover is also a threat to the sustainability of these outcomes. Limited effectiveness of Change initiated: The RAS provided diagnostics and recommendations for organizational arrangements— streamlining the debt servicing process; however, there was no evidence there are major operational available for this review that the recommendations had been implemented. inefficiencies in debt servicing due to multiple units within and outside of the MoPF participating in the processes, some with their own information systems that do not interface with each other Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Outcomes for the Strengthening Debt Management RAS are emerging or were achieved despite notable challenges including leadership transitions within the GTDPD and persistent staff turnover. Increased political support will be needed to implement the needed cash and liability management policies that might involve changes in the role of the National Bank of Romania and within units of the MoPF. Constraints such as these to further progress were identified by the RAS team but not intended to be resolved during the limited RAS implementation period. The second RAS designed to help implement the MoPF’s action plan for improving the management of public finances was focused on improving human resource management for the MoPF and subordinated bodies. RAS activities included an institutional analysis of human resource management, assessment of the training needs of the MoPF and subordinated agencies, development of a medium-term HR strategy and action plan for the MoPF and subordinated agencies, and the training of civil servants at these institutions. 60 Diagnostic activities at the start of the RAS highlighted various challenges to effective human resource management. First, MoPF policies are largely governed by those set by the National Agency of Civil Servants (NACS), limiting the flexibility to adapt to changing needs. Also, the HR Department’s role is mainly administrative and not strategic, and there has been no restructuring or review of current HR deployment. There has been no recruitment for several years and the recruitment process lacks transparency. Finally, there is also an inadequate performance appraisal system and a lack of staff development programs. As described in Table 35, the RAS was effective in producing key intermediate outcomes that are likely to lead to enhanced organizational capacity related to human resource management over the longer term. However, no evidence was available yet that the new HR strategy has translated into any measurable changes in HR management practices. Going forward, the RAS team recommended additional support to facilitate the successful implementation of the HR strategy including designing an executive development program with a special emphasis on HR topics such as group dynamics, coaching techniques, change/strategic management, the use of a balanced score card, performance management, operational/process management, and total quality management for all the MoPF managers. Table 35. Results for the RAS on the Improvement of Human Resource Management Instruments to Strengthen the Institutional Capacity of the MoPF Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Limited organizational Knowledge used: RAS activities included the development of an HR strategy effectiveness—More strategic and training on modern HR management approaches for managers. The management of human resources training evaluation indicated that participants acquired new knowledge and is needed at the MoPF and skills and the medium-term HR strategy was approved by MoPF order subordinated agencies. 564/2014. Evidence of improvements from the strategy is not yet available. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The World Bank conducted a third RAS for the MoPF financed under POAT to strengthen the capacity of the Ministry of Public Finance in fiscal and non-tax policy formulation. The RAS team reviewed Romanian tax legislation on specific issues to compare them with best practice in other EU member states, assessed current training needs to develop a training plan, and conducted one-week training sessions for groups of MoPF and tax agency officials. As described in Table 36, the RAS activities transferred knowledge to officials at the MoPF and the National Agency for Fiscal Administration, which is enabling these organizations to formulate tax policy more effectively. The RAS team noted that reports in the areas of property taxation, oil and gas taxation, natural resources (oil) taxation, and VAT law were used to make relevant amendments to the existing tax policy framework to align it better with international best practice. 61 Table 36. RAS Results for Strengthening Institutional Capacity in Fiscal Policy Formulation Identified institutional capacity challenge Contribution of RAS for Addressing Capacity Challenge Limited effectiveness of organizational Capacity change emerging: The RAS reports and training arrangements—The MoPF and NAFA have sessions provided a knowledge base for MoPF and NAFA limited capacity to formulate policies related to officials. The training evaluations show high ratings from direct taxation, excise duties, VAT, fiscal participants (9 on a 10-point scale on average) and the procedure and fiscal sanctions, and non-tax recommendations produced by the RAS are now being used revenues. to frame tax policy. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The RAS was strongly driven by client demand, with the technical assistance for each tax policy topic specifically identified and requested by the MoPF. Both MoPF and NAFA have demonstrated increased capacity that is likely to be sustained. Political changes might have an impact on some of the tax policy changes (i.e., labor taxation), but will not on others (i.e., VAT, which is bound by EU regulations). In the future, the World Bank expects to continue the tax policy dialogue with the GoR and to offer Bank services to work with the MoPF as needed to bring about specific tax reforms. The World Bank also provided advisory services to develop a unified and transparent set of procedures for issuing clarifications, interpretations and rulings on tax issues based on requests from taxpayers, NAFA or the General Directorates of County Public Finance (GDPFs). The RAS team conducted diagnostics and comparative analyses to draft a new system procedure for facilitating institutional communication and provided training sessions for the staff in the selected directorates of the MoPF and NAFA. As described by the project documents, RAS activities were designed in response to clear institutional capacity challenges. The services and communication with taxpayers provided by the MoPF, NAFA, and the GDPFs were not fully clarified with a large number of different normative acts providing a fragmented regulatory and procedural framework. This situation undermined the consistency and quality of information provided to taxpayers. Furthermore, there were no integrated procedures within the MoPF, ANAF and GDFPs to facilitate institutional communication in relation to issuing clarifications, interpretations and rulings on tax issues. The system is highly decentralized, which makes it difficult to operate, both logistically and from a quality control perspective. This RAS was an interesting case given that it is likely to contribute to enhancements in all three general areas of institutional capacity over the longer term. By design, the RAS directly contributed to improving the formal system procedures (more efficient policy instruments) and building capacity within the MoPF and NAFA to implement the new procedures (increased effectiveness of organizational arrangements). Incremental progress for these two institutional change processes, based on available evidence, is described in Table 37. However, it should be noted that both of these changes are intended to lead to improvements in the broader socio- political environment (strength of stakeholder ownership) over the longer term. As aptly described in the Activity Completion Summary, the main criteria for establishing a process for issuing resolutions is that “the delineation of power is transparent, logical and well-defined, so 62 that there is no confusion either among the taxpayers or the government about which Tax Authority has to issue resolutions for specified subject matters.� Table 37. RAS Results for A Better Solution for Providing Clarifications, Interpretations and Rulings on Tax Issues Identified institutional capacity challenge Contribution of RAS for Addressing Capacity Challenge Inefficient policy instruments-- a fragmented Knowledge gained: RAS outputs included a comparative study regulatory and procedural framework limits comprising the legislative and procedural framework in at least 5 institutional communication and EU member states regarding clarifications, interpretations, and coordination and undermines the quality of rulings related to fiscal issues and a proposed system procedure information provided to taxpayers drafted along with technical and functional specifications for IT support. The procedure is expected to be approved and implemented by the MoPF. Limited effectiveness of organizational Knowledge gained: The RAS team developed a training plan arrangements—MoPF has limited capacity and provided intensive training for 250 civil servants of the to implement a new system procedure for MoPF and NAFA in the field of system procedure regarding issuing clarifications and interpretations on providing clarifications, interpretations, and rulings related to tax issues to improve the uniform fiscal issues. Participants rated the training very positively (9 on implementation of the tax administration a 10-point scale on average), but evidence is not yet available for throughout the country. how they are applying new knowledge and skills. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. While it is clear that the RAS has produced intermediate outcomes related to knowledge, skills, and implementation know-how (i.e. the draft system procedure), there was not yet evidence available for this review as to how these changes are being applied in the MoPF and NAFA. Some of the recommendations are likely to be implemented through other World Bank support; for example, the Revenue Administration Modernization Project (RAMP) will be able to provide the resources needed to set up the IT systems for responding to taxpayer queries and resolving issues. Ministry of Transport and Infrastructure The functional review of the public administration commissioned by Romania in 2010 and conducted by the World Bank included Transport as one of the 12 sectors covered. As a follow- up engagement to this review, the Ministry of Transport and Infrastructure requested technical assistance for strengthening strategic planning and transport sector governance. RAS objectives also included improving the MTI’s capacity for evaluating road and railway infrastructure cost recovery, determining transport sector funding requirements consistent with EC directives and principles, and ensuring the cost-effectiveness of road contracts. In addition to this direct focus on enhancing the ministry’s capacity, RAS activities were designed to support the setup of an urban transport coordination authority. Overall, the advisory services were designed to provide the MTI with the tools and knowledge to implement an ambitious reform agenda. The RAS spanned a broad range of issues and produced 24 deliverables. The approach of the Bank team was to facilitate a participatory process, providing international best practice examples but leaving the relevant stakeholders to decide what would work best in the Romanian context. The series of reforms therefore reflect an ongoing process and will take time to complete. 63 The evaluation of the World Bank RAS portfolio did not include any interviews with stakeholders in the transport sector. Instead, the assessment of this RAS is based on the review of project documents and focuses on general areas of institutional capacity to which RAS outcomes appear to have contributed. As described in Table 38, the RAS made effective contributions towards establishing a coherent strategic plan for the MTI and building the capacity of the ministry for prioritizing road and railway infrastructure investments and monitoring the effectiveness of road construction costs. The RAS also provided support for reforming the governance of State Owned Enterprises (SOEs) in the transport sector to move them from being national liabilities to national assets. To a lesser extent, the RAS also contributed to the institutional changes needed to establish a fully integrated and sustainable urban transport system in Bucharest. High quality outputs were delivered related to reviewing the current legal framework for the Bucharest Metropolitan Transport Authority (AMTB) and prescribing transitional arrangements, drafting the AMTB’s first strategic plan, and other guidance to provide a roadmap for transferring AMTB from MTI to the metropolitan level. However, the implementation of key recommendations was delayed given the volatile political environment, and high-level commitment will be needed to advance these changes further. Table 38. RAS Results for Strategic Planning in the Transport Sector Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge Inefficient policy instruments— Knowledge used: RAS activities provided training to 293 staff and senior the lack of a consistent strategic management of the MTI and subordinated agencies on developing and vision and clear priorities has led implementing a medium-term strategic plan. The team drafted an MTI to high uncertainty in the sector, strategic plan and reviewed the existing strategic planning procedures to poor policy performance, produce the Strategic Planning Guide. Elements of the strategic plan are inefficient resource allocation, starting to be reflected in operational documents, such as the administrative and unsustainable fiscal costs. plans of the railway companies and the draft road and rail performance contracts. The new General Transport Master Plan has been finalized and linked to the Master Plan. However, implementation of the Strategic Plan cannot start until it has been formally adopted by Ministerial Order. Limited effectiveness of Knowledge gained: RAS activities and outputs included an analysis of the organizational arrangements— existing road and rail infrastructure charges; benchmarking of these charges Poor investment prioritization by with 6 EU countries; road and railway infrastructure provision costing the MTI has led to an uneven models; and training on funding requirements, financing mechanisms, and quality of spending and cost recovery options. Evidence was not yet available that the new unsustainable investments knowledge and skills are being applied or that MTI is using the costing models. Limited effectiveness of Knowledge gained: The RAS team reviewed contractual arrangements with organizational arrangements— CFR SA (Railway Company) and RNCMNR SA (Roads Company), MTI has no system to monitor recommended contractual amendments, and drafted multi-annual the effectiveness of road performance-based contracts. Outputs also included an assessment of final construction costs to support road construction costs compared to estimated costs, a benchmarking of road informed decision-making and construction costs, an analysis of the potential impact of reforms, and a accountability for results. monitoring methodology for MTI. Evidence as to the extent that recommended changes have been implemented by the MTI was not available. Limited effectiveness of Knowledge gained: Outputs included reviewing the legal framework and organizational arrangements— providing recommendations for transitional arrangements for the Bucharest the MTI and transport delivery Metropolitan Transport Authority (AMTB), drafting the AMTB’s first 64 Identified institutional Contribution of RAS for Addressing Capacity Challenge capacity challenge agencies lack adequate capacity strategic plan with technical advice on operational procedures and staffing, for the coordination and strategic and providing training and public consultations on urban transport for MTI planning of urban transport in staff and urban transport stakeholders. Following RAS recommendations, the Bucharest metropolitan area. there is now a roadmap and support at the technical level for the transfer for AMTB from MTI to the metropolitan level; however, there is not yet sufficient political support, so the recommendations have not been implemented. Limited effectiveness of Knowledge used: The RAS has provided core recommendations and organizational arrangements— knowledge needed to reform SOEs, with a key recommendation being the weak corporate governance of appointment of professionally managed boards of directors who could transport SOEs has led to long- manage the companies in accordance with business plans. A key output was term mismanagement and the Romania Corporate Ownership Guide prepared for the MTI. Uneven serious indebtedness. progress has been achieved so far by SOEs, but the MTI has signaled continued commitment to this reform process through the recent establishment of a High Powered Commission on SOEs performance chaired by the Prime Minister. Note: Ratings are based on a six-point scale:No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. To undertake the much-needed improvements in the transport infrastructure, the GoR anticipated that a number of projects should involve private sector participation in the form of long-term financing or operating commitments. The World Bank’s second RAS for the MTI therefore focused on increasing the capacity of the relevant government agencies responsible for the management, preparation and implementation of public private partnerships (PPP) in the transport sector, with a specific focus on the road sector. The RAS components included an institutional and legal review of the PPP framework, support to the identification, preparation and bidding of a pilot transport PPP project, and capacity building on specific topics covered by the activity. By design, the RAS targeted two important institutional capacity challenges related to an inconsistent legal framework and inadequate capacity within the MTI and subordinated agencies for implementing PPP projects. As described in Table 39, the RAS team was not able to achieve the expected progress in these areas due to changing circumstances during the implementation period. A key clarification regarding the results of the PPP RAS is that part of the RAS was premised on the implementation of a PPP program that did not happen. This meant that the counterpart could not advance as much as was planned on the development of a PPP project. The RAS team responded to this situation by changing, with the client’s approval, the scope of work to keep the spirit of the RAS (and the development objectives) while adapting it to the reality of the practical developments. 65 Table 39. RAS Results for MTI Capacity Building on Public Private Partnerships Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments— Knowledge gained: The RAS team reviewed the legal framework and made Restrictions and inconsistencies recommendations to help the GoR clarify the legal framework for PPPs and in the PPP legal framework limit make it more attractive to investors in projects in Romania. Challenges the efficient implementation of a evolved during the project with the GoR decision to repeal the PPP law, and PPP program in the transport the team adjusted the recommendations; however, it is not clear whether or sector when the recommendations will be implemented. Limited effectiveness of Knowledge gained: RAS activities and outputs included a technical organizational arrangements— guidance note on PPP methodology and training seminars for high-level The MTI needs a methodology officials from the MTI and MoPF. Participants reported gaining knowledge for developing and managing PPP and skills in training evaluations. However, the methodology was not projects applied and tested as planned in a pilot project because the client could not mobilize and select a project. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Ministry of Regional Development and Public Administration The 2007-2013 Operational Program on Administrative Capacity Development was designed as an instrument to support public administration reform in Romania, but it did not have the impact that many hoped it would. The Managing Authority for PODCA therefore requested World Bank assistance to help increase the effectiveness of administrative capacity programs so that central and local administrations can deliver better public services. The RAS focused on the analysis of capacity building activities in the public administration to inform the design of PODCA for the 2014-2020 programming period. The RAS provided diagnostic services across two components focused first on capacity building activities to strengthen the civil service and factors that influence their effectiveness and second on PODCA's Role in supporting the capacity of local governments and decentralization. The team actively consulted and shared findings with key stakeholders throughout this process including the MA for PODCA (the client), the General Secretariat of the Government, the Chancellery of the Prime Minister, representatives of the National Agency for Civil Service, the MoESR, Ministry of Health, the MoLFSPE and local government associations. The review also included the analysis of the central NACS-managed civil servants database, documentation of past projects, and a survey of 68 institutions from all administrative levels. By design, intermediate outcomes achieved by the RAS were intended to contribute to the increased “efficiency of policy instruments� by strengthening PODCA as an instrument for guiding stakeholder actions related to public administration reform. However, as summarized in Table 40, there was not yet evidence that the RAS contributed to this longer-term capacity change. The outputs do not appear to have been used to inform PODCA design for the 2014- 2020 period. Furthermore, the team identified substantial implementation challenges related to policy development processes and human resource management practices that will need to be addressed for PODCA to achieve its agenda for public administration reform. 66 Table 40. RAS Results for the Analysis of Capacity Building Activities in the Public Administration Contribution of RAS for Addressing Capacity Identified institutional capacity challenge Challenge Inefficient policy instruments-- PODCA 2007- Change initiated: The RAS provided a detailed analysis 2013 lacked the guidelines and consistency needed and assessment of results of the activities supported by to produce the expected outcomes. Some PODCA and identified recommendations for the next challenges included the lack of a coherent program cycle. However, there was no evidence that the decentralization strategy and the misalignment of client has used these outputs or that the project has spending restrictions and regulations with the contributed to improvements in the design of PODCA or PODCA agenda. in plans for its implementation. Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. Romanian Competition Council The Romanian Competition Council (RCC) is charged with guaranteeing market competition, enforcing compliance with competition law, ensuring that state aid meets rigorous EU criteria without creating market distortions, and promoting a broader understanding of competition rules and benefits. The 2010 Functional Review of the RCC highlighted the need for a whole-of- government approach to mainstream competition principles in Romania’s administrative agencies. Based on the findings and recommendations of this review, the RCC and GoR endorsed a Reform Action Plan, and the RCC requested the World Bank to provide advisory services for increasing the effectiveness of competition policy and supporting the implementation of the action plan. The objective of the RCC RAS was to increase the effectiveness of competition policies, including the enforcement of competition law, integration of competition principles in sectoral policies, and enhancing the capacity of the RCC to efficiently perform its mission. RAS components included a review of the legal and regulatory framework governing market competition, advocacy activities to promote competition in public and governmental bodies, implementation of a new business architecture in the RCC, and human resources capacity building. The RAS produced more than 42 technical reports, 12 training sessions, and 12 advocacy events. Overall, the technical assistance focused strategically on positioning and enabling the RCC to serve as a competent and effective change agent to promote the achievement of development goals related to domestic market efficiency, economic growth, and European market integration. As summarized in Table 41, the RAS effectively contributed to numerous institutional capacity change processes, including revisions to the legal and regulatory framework and improvements in the systems and operating procedures of the RCC. Given the scope of this broad review, the areas of institutional capacity targeted by the RCC RAS that are described here should be considered illustrative rather than exhaustive. Project documents provided the main source of information and did not necessarily reflect recent developments. No RCC stakeholder interviews were conducted as part of this evaluation in 2016, but the RAS was one of the four RAS case studies conducted in 2015 for which selected stories of knowledge results were explored. 67 Table 41. RAS Results for the Assistance to the Romanian Competition Council Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge Inefficient policy instruments— Capacity change emerging: RAS recommendations informed a new the Romanian Competition Law competition law which was prepared and adopted in 2015. Changes raises the cost of doing business include eliminating the 40 percent threshold for the presumption of with unclear provisions and dominance; limiting the parties’ right to challenge access to file and administrative steps, an undue confidentiality before the Courts; and creating the Independent Procedural burden on the private sector, and Officer role specialized in the disposition of access to file, confidentiality the inability of the RCC to and other procedural matters separate from the enforcement team within the prioritize cases RCC. Inefficient policy instruments— Capacity change emerging: RAS activities included advocacy events and the legal framework for unfair consultations on unfair competition law and recommendations to amend the competition is inconsistent, does legal and regulatory framework. Resulting modifications to Romania’s not clarify roles across agencies, Unfair Competition Law eliminate overlapping mandates among the RCC and imposes high costs on and other agencies, define unfair competition, and clarify the role of the businesses RCC. Enforcing the law on unfair competition will require lower administrative costs given the establishment of de minimis and opportunity tests, in which the RCC will move to sanction actions only if they present a certain degree of social threat that affects either the public interest or market structure. Limited effectiveness of Capacity change emerging: Changes in the Unfair Competition Law organizational arrangements— described above will allow the RCC to use resources more efficiently. the RCC has limited operational Criteria for prioritizing unfair competition cases have been established, and efficiency to enforce unfair the RCC will process complaints received from individuals or legal entities competition law given the lack of only to the extent that they provide evidence about the possible unfair case prioritization and overlapping commercial practice, the legitimate interest, and the risk of damages. competencies with other agencies Inefficient policy instruments— Knowledge used: RAS activities and outputs included advocacy and the State Aid legal and regulatory consultations with State Aid stakeholders, recommendations for amending framework is not aligned with EU the legal and regulatory framework, and a guide for assessing the impact of law and does not clarify the RCC’s granted State Aid. The enactment of Emergency Ordinance 77/2014 on role in the state aid area vis-à-vis State Aid national procedures clarifies aspects related to State aid granting authorities and courts. notification, obligations of State aid granting authorities, aid recovery, the role of national courts, and de mimimis aid. Time will need to pass before the effects of implementation can be assessed. Limited effectiveness of Knowledge used: RAS outputs included the Target State Business organizational arrangements— Architecture, Target State Technology Architecture, Target State Solution the RCC lacks a coherent and Architecture and Migration Plan, and Enterprise Architecture Method comprehensive business Guide. The RCC has started to apply recommended changes (Release 1.0 architecture, leading to the of the implementation roadmap), including automation of the mergers misalignment of staff roles with processes, improving the capabilities of IT forensics and upgrading the IT institutional objectives, a lack of infrastructure. The RCC is also applying key concepts related to human internal and external transparency, resource development. The ongoing implementation of the roadmap is and reduced effectiveness. expected to enhance administrative capacity of the RCC and provide evidence of increased operational effectiveness and efficiency over time. Inefficient policy instruments— Capacity change in effect: Following RAS recommendations A new merger regulations and procedures merger regulation was adopted on September 30, 2014, allowing for the impose unnecessary administrative more efficient use of fast-track procedures. This has resulted in an average burden on businesses associated length of 1.8 months to complete a merger case in 2014, representing a 23 with merger review and high percent decrease in the amount of time required compared to 2013 . administrative costs for the RCC Limited effectiveness of Knowledge used: the RAS produced a Guide for Economic Analysis of organizational arrangements— Mergers to serve as a useful tool for RCC’s Economic Analysis Group, for the RCC has performed limited improved effectiveness and operational efficiency in carrying out its 68 Identified institutional capacity Contribution of RAS for Addressing Capacity Challenge challenge economic analysis of mergers. functions. The guide provides a reference framework for using internal Unclear market impact of methodologies for economic analysis in merger cases; however, evidence anticompetitive mergers is was not yet available for this review of how the guide is being used and the associated with higher resulting benefits. prices/supply limitations especially for consumers Inefficient policy instruments— Capacity change in effect: RAS recommendations informed new leniency leniency guidelines do not except guidelines, which were approved by the RCC Board in 2014. The benefits leniency applicants from potential of the new procedure were reaped through the first leniency application in criminal prosecution. Insufficient 2015. incentives for whistleblowers to disclose cartels lead to limited cost savings for consumers associated with cartel detection Note: Ratings are based on a six-point scale: No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. The list of identified capacity challenges included here should be considered illustrative rather than exhaustive given the broad scope of this RAS and the level of information available via the desk review of project documents. The outputs of the RCC RAS will continue to contribute to institutional capacity changes and time must elapse before the effects of legal and regulatory changes can be fully assessed. The implementation of the new business architecture for the RCC will evolve with the new releases to maintain alignment with ongoing reforms. The RCC and the World Bank expect to continue this collaboration through a follow-on RAS using funds under the 2014-2020 programming period. This ongoing support will further help to position the RCC as a center of excellence within Romania’s public administration and as an agent of change for achieving domestic market efficiency, economic growth, and European market integration. 69 VI. Results for the Operational Program for Increased Economic Competitiveness Overview The objective of the Operational Program for Increased Economic Competitiveness (POSCCE) is to increase Romanian companies’ productivity, in compliance with the principle of sustainable development, and reduce disparities compared to the average productivity of the EU.17 The program was developed based on an analysis of the current status of entrepreneurship and innovation in Romania with special emphasis on the small and medium-sized enterprises sector; on resources for the research, development, and innovation sphere; on the ICT sector; and on energy efficiency and environmental protection issues in the energy and industry sectors. As listed in Table 42, only one of the RAS agreements included in this evaluation was under the Managing Authority for POSCCE in the Ministry of Public Finance. The World Bank’s engagement was specifically to support the priority axis focused on ICT for the private and public sectors. Table 42. POSCCE RAS Agreements Completed by the World Bank 2012-2015 Duratio Project Date Completion Value RAS Title Client n Number Signed Date (US$ millions) (months) P152542 Implementation Ministry of 2-Apr-15 1-Dec-15 8 $0.91 Support for the Information National Strategy on Society Digital Agenda in Romania Ministry of Information Society The National Strategy on Digital Agenda for Romania (NSDAR) is designed to foster both economic growth and economic competitiveness. Some progress has been achieved in implementing the strategy, such as the development of the National Plan on Next Generation Access and Network Infrastructure (NGA, NGN) and the Smart Specialization Strategy and Policy. As part of its efforts to further operationalize the NSDAR, the Romanian Ministry of Information Society (MIS) requested World Bank assistance. The overall development objective of this RAS engagement was to help the GoR in leveraging ICT to facilitate integration into the European digital single market and align Romania’s ICT development with other countries in the region. Technical assistance provided through this RAS was designed to assist the MIS in the implementation of the NSDAR. Specifically, the RAS team provided support related to broadband infrastructure development and the real-time monitoring and evaluation of the NSDAR. Evidence of the results of RAS activities and outputs were collected and analyzed through a desk review of project documents and the survey of TTLs. No stakeholders were interviewed in the ICT sector as part of the World Bank RAS evaluation. 17 Ministry of Economy and Finance. 2007. Sectoral Operational Programme “Increase of Economic Competitiveness� 2007-2013. Government of Romania. 70 As described in Table 43, the RAS effectively helped to address at least four identified institutional capacity constraints. Notable achievements included building stronger leadership commitment to the implementation of the NSDAR, informing potential changes to improve the legal and regulatory framework for broadband services, helping MIS develop the capacity to propose tailored solutions for broadband supply and demand at the community level, and developing a real-time, online M&E system. RAS activities engaged and helped to address capacity issues for the MIS, ANCOM (telecom regulatory authority), and most of the 17 other public entities involved in the implementation of the Romania Digital Agenda (ICT Strategy). Table 43. Results for Implementation Support for the National Strategy on the Digital Agenda in Romania Identified institutional capacity challenge Contribution of RAS for Addressing Capacity Challenge Weak stakeholder ownership—the MIS Knowledge gained: The RAS team engaged with all relevant GoR lacks authority over other ICT stakeholders entitites from the start to build ownership. The team conducted (17 public entities) and they have weak strategic “restricted� public consultations with targeted ICT ownership of and commitment to the stakeholders to gain input to and buy-in for RAS deliverables. For National Strategy for the Digital Agenda of the M&E component, the team used an iterative process of Romania (NSDAR). consultations with 26 institutions identified as having roles in the implementation of NSDAR. Continued efforts will be needed to build commitment for implementation. Inefficient policy instruments—revisions to Knowledge gained: RAS activities included a detailed analysis of the legal and regulatory framework for the legal, regulatory, and institutional framework to provide broadband services are needed to increase recommendations to MIS. Limited evidence was available for this consistency, reduce costs, clarify roles for review related changes to the legal framework resulting from this coordination, and include PPP legislation. exercise; however, the GoR has implemented some of the associated recommendations, such as the development of a mapping tool (described below). Limited effectiveness of organizational Capacity change emerging: The RAS developed a mapping tool arrangements—MIS could increase its that mapped village typology to determine the most appropriate effectiveness and operational efficiency if it PPP model. The accuracy and level of detail provided the GoR developed the capacity to propose tailored with extensive knowledge on the supply and demand for solutions for broadband supply and demand Broadband across the country. Evidence was not yet available for depending on the community profile this review of the extent to which this information is leading to tailored solutions. Limited effectiveness of organizational Capacity change emerging: RAS outputs and activities included arrangements—the MIS and other ICT an M&E manual to develop a comprehensive M&E system and stakeholder agencies have weak M&E training to support implementation. MIS has implemented the capacity for monitoring the implementation recommendations for an M&E framework and online M&E of the Digital Agenda platform (dashboard). Evidence will continue to emerge as to how effectively M&E data is being used for decision-making. Note: Ratings are based on a six-point scale:No outputs, change initiated, knowledge gained, knowledge used, capacity change emerging, and capacity change in effect as defined in Annex 4. At RAS completion, the team was aware of the remaining risks to sustainability regarding the implementation of the NSDAR given the weak institutional capacity within MIS and the broad range of stakeholders involved without roles and responsibilities being clearly defined. The GoR has expressed an interest in a potential follow-up RAS to conduct a functional review of the MIS to strengthen its capacity related to strategic planning, human resources, work flow processes, and ICT. 71 VII. Formative Feedback from World Bank Task Team Leaders and Clients Overview The data collection for this evaluation captured advice from RAS task team leaders and clients about how to continue and strengthen effective collaboration for future RAS engagements. The survey of TTLs focused primarily on documenting RAS results but included a section about the TTL’s RAS implementation experience.18 Similarly, in-depth interviews with clients and other stakeholders were structured to identify evidence of results but also included questions to explore the quality of collaboration between the World Bank and the GoR. Findings presented in this section are therefore based on survey responses from 20 TTLs, interviews with 19 stakeholders representing the GoR, and a review of lessons presented by Bank teams in each RAS Activity Completion Summary. TTLs’ RAS Implementation Experience The RAS TTLs were asked to reflect on their general RAS experience in Romania and indicate which factors during implementation challenged the World Bank’s ability to achieve targeted development objectives. As shown in Figure 3, more than half of the respondents (55 percent) indicated problems related to the cumbersome WB administrative processes, the high turnover and/or weak capacity in public administration, and the compressed timeframe of the RAS engagements. 80 60 55 55 55 40 40 40 Percentage of Respondents 35 25 20 0 Cumbersome Turnover/weak Compressed Cumbersome Lack of buy-in Limited cross- Difficulties WBG capacity in timeframe government from client sectoral amending RAS administrative public admin processes coordination agreement processes Figure 3. Most Common Challenges Reported by TTLs (n=20) 18 TTLs with more than one RAS were asked to complete a separate results module for each RAS but only completed the implementation experience module once. Therefore, there were 20 TTL responses providing formative feedback based on the experience of implementing 40 RAS agreements. 72 The least common challenges indicated by TTLs included the limited availability of international experts, the lack of local experts, and the inadequate translation of outputs into Romania. Each of these factors was selected by only two respondents (10 percent). TTLs had the option to add comments to describe other challenges, which included differences with EU staff in Brussels on the main challenges to include in the scope of the RAS and underestimated costs. One TTL noted that “many Bank staff find it difficult to deal with a client that has clear demands and expectations.� Respondents also indicated which factors during RAS implementation helped the Bank’s ability to achieve targeted development objectives as shown in Figure 4. The most common facilitating factors included frequent communication with the client (75 percent) and having continuity in the WBG team, meaning that the same personnel worked with the clients on multiple RAS engagements (65 percent). More than half of respondents also identified the availability of relevant international expertise as a helpful factor supporting successful RAS implementation. 80 75 65 60 55 45 45 45 40 40 40 Percentage of Respondents 20 0 Frequent Continuity of Availability of Ability of the Effective Adequate local In-country High-level communication WBG team relevant World Bank to coordination expertise to leadership of support on client with client international convene diverse across RAS support RAS team side expertise stakeholders components implementation Figure 4. Common Facilitating Factors Reported by TTLs (n=20) TTLs generally indicated that their RAS engagements were relevant, high quality, and effective for achieving development objectives. As presented in Figure 5, most respondents provided positive ratings for how well the current structure and processes helped to ensure the RAS program’s relevance (77 percent) and high quality and effectiveness (79 percent). Nearly two thirds (64 percent) provided positive ratings for efficiency, in terms of helping to ensure that the RAS achieved targeted results at the lowest cost. 73 74 Figure 5. TTL Ratings for How Well the Current Governance Structure, Processes and Systems Help to Ensure the Following Characteristics for the World Bank’s RAS Program in Romania Relevance of RAS to WBG's CPS in Romania (n=17) 59 18 High quality and effectiveness (n=19) 68 11 Efficiency (n=19) 42 21 Coordination of RAS implementation with other relevant WBG operations (n=19) 28 17 Sustainability of RAS results (n=18) 33 11 0 20 40 60 80 100 Percentage of Respondents Very good Excellent Note: Ratings are on a 5-point scale from poor (1) to excellent (5). The order of systems and processes are listed according to their overall mean scores. Fewer than half of respondents were confident that the current structure and processes helped to support the sustainability of RAS results (44 percent). Concerns about sustainability were presented in TTLs’ comments and were commonly expressed by clients, given that RAS agreements often help support longer term reforms that require follow-up. Although fewer than half (45 percent) of the respondents indicated that the current structure and processes were very good (28 percent) or excellent (17 percent) for helping to ensure the coordination of RAS implementation with other relevant WBG operations, TTLs noted in various comments that the leveraging of RAS findings has been instrumental for preparing new Bank operations. In fact, as presented in Figure 6, the RAS was the most highly rated instrument in the Bank’s portfolio for effectively addressing development objectives in Romania.19 19 Ratings were on a 5-point scale: Poor, fair, average, very good, excellent. Respondents also had the option to select “Don’t know/not relevant� in considering the use of the instrument for their sectors. 75 Figure 6. General Ratings of World Bank Instruments for Effectively Addressing Development Objectives in TTL’s Sectors in Romania Reimbursable Advisory Services (RAS) (n=20) 60 35 Investment Project Financing (n=17) 47 18 Development Policy Financing (n=14) 43 14 Non-reimbursable economic and sector work (ESW) (n=16) 31 25 Non-reimbursable technical assistance (TA) (n=15) 27 27 0 20 40 60 80 100 Percentage of Respondents Very good Excellent All except one TTL (95 percent) found the RAS to be a very good (60 percent) or excellent (35%) instrument for effectively addressing development objectives. This share was sharply above the proportion of positive ratings assigned for investment project financing (65 percent), development policy financing (57 percent), economic and sector work (56 percent) and technical assistance (54 percent). Comments throughout the survey mentioned the unique value of the RAS and the importance of having the client pay for the services to demonstrate commitment. Responses to the TTL survey reflect a relatively small group of stakeholders (20 TTLs), but the ratings provide useful guidance for areas to explore for strengthening RAS results. For example, the World Bank country management could consider whether there are any opportunities to streamline administrative processes or help to facilitate the processing of documents and approvals given the compressed timeline of RAS engagements. As explored in Box 14, TTLs’ comments and ratings indicated a need in particular to explore potential improvements in the ways that units in World Bank headquarters support effective RAS implementation. Box 14. Example Reflection on the Need for Better Support from World Bank HQ “The most hiccups we had with the RAS implementation is the slow response from staff in HQ. Everything seems to move slower there and we consistently have things falling through the cracks (e.g. STC contracts that are not processed, Consultancy companies that are not being paid, documents that are not processed).� Source: TTL Survey 76 Feedback from Clients The design for this evaluation did not include a survey of clients to minimize their response burden and to allow for a more nuanced qualitative data collection to understand RAS results. Instead, guidance from clients on the current strengths and areas for potential improvements in RAS implementation was gained through the Client Feedback Survey of FY14 Advisory Services and Analytical Activities and customized semi-structured interviews focused on a subset of the Romania RAS engagements. World Bank Operations Policy and Country Services (OPCS) conducted the Client Feedback Survey and provided the subset of the responses in the FY14 data linked specifically to the Romania RAS program. This subset included 22 respondents reporting on 13 RAS agreements. Although the TTL Survey and the Client Feedback Survey do not share the same questions or rating scales, a general comparison of the response patterns by topic shows a convergence of attitudes and experience across these two groups. As shown in Table 44, both sets of stakeholders generally viewed the RAS Program to be effective, relevant, and of high quality. However, they were much less positive about issues related to the sustainability of RAS results. Fewer than half of TTLs (44 percent) indicated that current systems were adequate for helping to ensure sustainability, and less than a quarter (22 percent) of clients reported that theKnowledge used via the RAS was actually used and “led to change.� Table 44. Comparison of Ratings from Clients and TTLs for Similar Topics General Topic % Positive Ratings in FY14 Client Survey for % Positive Ratings in TTL Survey Romania RAS Program respondents (n=20) (n=22, for 13 RAS) Effectiveness 77% --activity effective in achieving its 79%--current systems help to ensure objectives high quality and effectiveness 77%--Effective engagement Relevance 68%--“Relevance to your needs� 77%--current systems help to ensure relevance to the World Bank’s CPS for Romania Quality 73%--Technical quality 79%--current systems help to ensure high quality and effectiveness Sustainability of 22%--knowledge from RAS was used and “led to 44%--current systems help to ensure results change� sustainability of RAS results 36%--a strength is “implementing follow-up activities� 45%-- a strength is “providing long-term support/partnership� Communication 77%--a strength is “communicating regularly and 75%--a success factor is “frequent clearly� communication with client� 73%--a strength is “being responsive to your needs� Convening 64%--a strength is “bringing different groups 47%--a success factor is “ability of the together to enable learning and collaboration World Bank to convene diverse stakeholders� Note: “Percentage of positive ratings� includes all ratings that are “effective� or higher on the client survey (8 or above on 10-point scale) and “very good� or higher on the TTL survey (4 or above on 5 -point scale). 77 Both surveys explored strengths and weaknesses of the RAS experience. For clients, this section focused on what the World Bank did well versus poorly during RAS delivery. For TTLs, the focus was on what factors challenged or helped to facilitate RAS implementation. While the closed-ended response options were quite different, there were some areas of commonality. For example, both clients and TTLs indicated that the success of the RAS program is helped by frequent clear communication and by the Bank’s ability to bring together diverse stakeholders. As part of the evaluation of World Bank RAS results for the 2012-2015 period, interviews were conducted in March and April of 2016 with 19 clients to explore the results of 20 RAS agreements.20 These semi-structured discussions identified evidence of RAS outcomes (presented in previous sections) but also elicited formative feedback about how effectively the World Bank was collaborating with the GoR and what changes might help to strengthen this collaboration. Together, the reflections on strengths and areas for improvement highlighted recurring themes across sectors. These inform the following crosscutting lessons for future RAS engagement.  Active negotiation and communication by both the Bank and the GoR at the start of the RAS engagement will promote smoother implementation and greater effectiveness. GoR officials from MoEF, MARD, MRDPA, and MoLFSPE in particular emphasized that strong engagement by both parties is necessary to identify and achieve the right objectives. Key ingredients identified for this success include that both the GoR and the Bank teams must be comfortable in describing any expectations and concerns and that the Terms of Reference and Inception Report should provide concrete details regarding processes and deliverables. Example reflections are in Box 15. Box 15. Perspectives on the Need for Strong Communication at the Start of the RAS “If there is a lack of interest from the beneficiary and an assertive TTL, then the results will not be good.� “Technical assistance must involve thinking about the details. The ToR should explain clearly what you want. When things are left vague, you might discover later that the RAS team had a different understanding of the task than what we had.� “There might be a misunderstanding. When we say we need a certain deliverable on a certain date, two or three months from now, that means that is the day we need to use it. So then we are confused and disappointed if we receive the report at 9 or 10 in the evening on that date and it has not yet even been translated.� Source: Interviews with GoR stakeholders  RAS teams should consider whether RAS outputs (reports, strategic frameworks, etc.) are “useable products� for the client. Nearly all stakeholders interviewed expressed satisfaction with the quality of World Bank’s diagnostic work and reporting. However, 20 A total of 31 non-Bank stakeholders were interviewed, and 19 of these were current or recent GoR personnel who offered client perspectives. The remaining 12 represented local government, academia, and a think tank. 78 they also made clear suggestions for how the RAS products could trigger and facilitate action rather than “just ending up on a shelf.� Their recommendations included: o Including local examples if possible to supplement international best practice examples to show how the concept or approach would work in the local context o Providing specific implementation instructions to articulate how the audience is expected to use the information, with concrete next steps o Summarizing dense narrative text in easy-to-read tables, with objectives, indicators, and suggested actions as appropriate o Placing a higher priority on the translation of outputs into Romanian, with at least one content expert engaged in the process to ensure that the language is technically accurate  A more integrated approach to technical assistance will help to ensure that the client can apply the new knowledge and skills generated by RAS activities. Closely related to the lessons above, client stakeholders broadly reflected that the new generation of RAS engagements could include more comprehensive support by the World Bank to help ensure that the RAS outputs are used and recommendations are implemented over time. MoESR and MRDPA officials among others described the value they would gain by having a RAS include coaching or mentoring, as shown by the example in Box 16. Another common suggestion was that substantial “academic� reports should be accompanied by presentations and follow-up sessions to help facilitate their understanding and use.  Strong collaboration by the Bank team requires frequent communication and a good grasp of the changing local context throughout the RAS implementation period. One oft- cited reason why GoR officials Box 16. Example Reflection for Strengthening valued their RAS engagements Future Collaboration with the World Bank was because they found the teams to be “The RAS was the proper intensity and scope. responsive and adaptable. Key Coaching would not have been good because people success factors described by needed to get used to the external expertise. But stakeholders from CPM, MARD, now we have reached the critical part of the change MRDPA, MoESR, and MEWF and the coaching idea is haunting me. An expert included having either a local TTL could stay here in my office and show me how to or a local senior specialist on the address problems. We believe the future team who could meet with the collaboration with the World Bank should include team on short notice if needed. this.� Stakeholders also emphasized the Source: Interview with MoESR stakeholder importance of engaging relevant local experts as part of the team and allowing the client to participate in identifying the best matches for address project needs. Examples of these common themes voiced by clients are in Box 17.  Engaging the right stakeholders is critical, so the plan for doing this should be formed at the start of the RAS. As with the Client Feedback Survey and the TTL Survey, the 79 interviews highlighted the value offered by the World Bank as a convening power. Most GoR stakeholders mentioned the efforts invested by RAS teams to both identify and include relevant stakeholder groups in awareness raising and priority setting activities. This was viewed as critical to the success of the RAS and linked to the other lessons. The negotiation and communication during the design stage should include clear decisions about which groups to include and how to engage them, and a participatory approach should be supported as appropriate throughout the RAS period. Box 17. Reflections on the Need for Local Support and Local Expertise “We had good communication with the TTL and a chance to discuss everything hundreds of times. It is important in each project to have a good local team with people here on the ground that know the realities. They can respond to the most urgent issues.� “I don’t want to minimize the international experts—certainly important—but local experts really made the difference.� “I feel the need for more local expertise. They understand the context of the institutions. It’s about local context but also about culture: it’s very difficult for international experts to be giving the advice and more comfortable and familiar to rely on the local people. So the reform really needs to have both.� “We need really concrete inputs from the World Bank that work here in Romania. So for example, the project models need to be adapted to our legislation. We cannot just have examples from other countries. We must have the national context and therefore we need experts from Romania. Then we can understand how to use the information.� Source: Interviews with GoR stakeholders Overall, the clients’ reflections on their RAS implementation experiences were very positive. The teams were responsive and generally met or exceeded the expectations for providing high quality technical expertise. Ideas for strengthening the collaboration largely focused on how to further improve the evolving partnership between the Bank and the GoR given the established trust and shared objectives. 80 VIII. Conclusions Value of the Conceptual Approach Reviewing the results of advisory services through the lens of the Capacity Development Results Framework provides a meaningful assessment of how the World Bank’s RAS Program is contributing to the achievement of development objectives. Rather than counting piecemeal outputs and outcomes (e.g., number of officials trained, adoption of a strategy, etc.), the evaluation assessed how targeted institutional capacity challenges were being addressed. The method can be applied systematically across sectors. Defining outcomes in terms of incremental progress along an institutional capacity change pathway provides an immediate filter for relevance, effectiveness, and sustainability. Outcomes lack meaning if they are not tied to a development objective. An intervention is unlikely to have any long-term effect if it has supported only preliminary changes in knowledge, skills, and behavior without helping to ensure that these changes can and will be applied in the desired context. Therefore, any RAS that did not achieve at least one rating of “knowledge used� or better to show progress in addressing an identified capacity challenge is unlikely to have sustainable outcomes without some form of follow-up support to the client. Given the broad scope of this evaluation, some assessments of individual RAS agreements might be superficial or outdated. The level of detail included in project documents, the responsiveness of TTLs, and the engagement of clients for interviews all varied widely across the portfolio. This dynamic led to more detailed and up-to-date assessments of institutional capacity changes for some engagements and summary reviews for others. Nonetheless, the systematic assessment of each RAS allows for a comprehensive snapshot of how the World Bank’s Romania RAS Program from 2012 to 2015 helped to improve the strength of stakeholder ownership (on the demand side), and the efficiency of policy instruments and eff ectiveness of organizational arrangements (both on the supply side). Aggregate RAS Results The evaluation documented some progress achieved by 38 of the 39 RAS agreements implemented from 2012 to 2015 in addressing a total of 98 institutional capacity constraints. For the majority of these improvements, stakeholders had already started using changes in knowledge, skills, behaviors and/or relationships, and the changes were likely to translate to stronger institutional capacity over the longer term without continued World Bank support. Overall, RAS engagements reflected comprehensive support for public sector reform in Romania. The program directly served 15 client organizations at the national, regional, and country levels. Organizations that benefitted included multiple units within more than 20 ministries, agencies, commissions and other bodies of the GoR along with countless other institutions at the sub-national level. Examples of these include local municipalities and regional development agencies using new tools for spatial planning and urban development and school inspectorates in all 42 counties receiving new regulations and guidance for organization and functioning. 81 RAS agreements were often designed to address more than one institutional capacity challenge blocking the achievement of a development goal. Two-thirds of the projects (26) addressed multiple challenges, according to the project documents and TTL input. The portfolio collectively addressed 54 challenges related to the effectiveness of organizational arrangements (37 projects), 34 challenges related to the efficiency of policy instruments (26 projects), and 10 challenges related to the strength of stakeholder ownership (8 projects). Most of the RAS engagements (11 of 13) that targeted just one institutional capacity change focused on improving the effectiveness of organizational arrangements. This streamlined approach proved to be effective for strengthening organizations in relation to specific development objectives: nearly two thirds (7 of 11) of these engagements resulted in new knowledge being used, compared with only half (23 of 46) of the efforts in projects targeting more effective organizational arrangements among other capacity changes. Efficiency of Policy Instruments The collaboration between the World Bank and the Government of Romania was most effective for the elaboration of laws and strategies. Results assessed in terms of “more efficient policy instruments� included changes in legislation, regulations, strategies, standards or other formal incentives established to guide stakeholder actions more efficiently in pursuit of development goals. Of 34 identified capacity challenges related to policy instruments, RAS engagements contributed to ratings of “knowledge used� or better results for 26 of these. RAS results played a critical role for helping Romania to leverage EU funding for achieving development objectives. Bank teams helped to elaborate strategies to satisfy ex-ante conditionalities for accessing European funds for the 2014-2020 programming period related to lifelong learning, reducing early school leaving, increasing tertiary education attainment, supporting the elderly and active aging, and promoting social inclusion and poverty reduction. RAS outputs also provided the coordination mechanisms to harmonize State and EU-funded projects for integrated planning and development, informed the policy lines regarding integrated territorial development for the Romanian Partnership agreement 2014-2020, and contributed to the design of the Community Led Local Development and Sustainable Development priority axes for the new Regional Operational Program. Effectiveness of Organizational Arrangements RAS activities and outputs helped to strengthen the capacity of public sector organizations at the national, regional and local levels. Evidence of effective improvements (“knowledge used� or better results) was documented in 29 instances, commonly indicating an organization’s increased operational efficiency, improved financial management, increased adaptability, or better achievement of outcomes related to a development objective. Most of the RAS results related to the effectiveness of organizational arrangements focused on refining or reinventing administrative structures, systems, and/or processes. For example, establishing criteria for prioritizing unfair competition cases will allow the Romanian Competition Council to operate more efficiently in processing complaints about unfair commercial practices. Local municipalities such as Alba Iulia are using new analytical approaches and tools for a more efficient planning process, including pursuing inter-municipal 82 planning with nearby cities and setting criteria for project selection and prioritization. Assistance to the Ministry of Education and Scientific Research included producing new regulations for organization and functioning to clarify roles, functions and responsibilities across the pre- university school system. ICT-based tools developed by RAS teams contributed to improved processes documented at the organizational level. Of the 14 instances where ICT solutions were designed to address institutional capacity constraints, half (7) were rated as “knowledge used� or “capacity change emerging.� An ICT prototype for institutional strategic planning and performance assessment developed for the Ministry of Agriculture and Rural Development is expected to be adapted and used by other ministries to strengthen their strategic planning functions. An online monitoring and evaluation platform with a performance dashboard is being used by the Ministry of Information Society for evidence-based decision-making. Assistance to the Romanian Competition Council resulted in automation of the mergers process for improved operational efficiency. Training activities alone were generally not adequate for building technical capacity in public sector units. For example, the RAS for strengthening debt management included training on cash management and liability operations, with participants demonstrating increased knowledge as a short-term outcome. However, the RAS team observed that an internal reorganization was needed within the General Directorate of the Treasury and Public Debt in order for the staff to apply the new skills as intended. Other RAS training exercises were also expected to have a limited effect without additional follow-on support, such as for monitoring the implementation of new national strategies in the education sector, implementing a new methodology for managing public private partnerships in the transport sector, and applying promising new sectoral analysis tools for climate-related decision-making in the energy, transport, urban, and water sectors. Strength of Stakeholder Ownership RAS agreements tend to focus on the supply side. GoR requests for technical assistance were typically for help developing laws and strategies or strengthening the administrative or technical capacity of an organization. However, these efforts had limited success in cases where there was inadequate stakeholder buy-in and commitment for participating in the process and implementing the recommendations, suggesting that more attention should be paid to the demand side during RAS design. Only eight RAS engagements included design elements to strengthen stakeholder ownership. These institutional capacity results on the demand side included increasing the commitment of leaders to the achievement of a development goal, improving the transparency of information, increasing stakeholders’ demand for accountability from service providers, and changing widespread practices. The portfolio was generally less effective in contributing to these demand-side changes. The eight RAS agreements identified and targeted a total of ten capacity constraints related to the strength of stakeholder ownership, but only four of these changes reflected Knowledge gaineds or better results. In all of the cases, the RAS design included components for contributing to 83 multiple institutional capacity changes, without the enhancement of stakeholder ownership alone being a clear priority. The more effective engagements provided some guidance for what works best in building demand-side capacity. First, a continuous or iterative process rather than a one-off forum or event is needed to foster broad stakeholder participation and commitment to the achievement of a development objective. Second, new findings and concepts need broad dissemination through a variety of formats to gain traction in shifting public awareness and attitudes. Third, a change on the supply side—such as a new organizational mechanism or new legislation—can serve as an important lever for strengthening stakeholder ownership. For example, the RAS providing support for the establishment of a Delivery Unit included a mix of activities in the energy sector for increasing the transparency of information and helping to ensure that consumers demand accountability. Lessons Effective changes related to more efficient policy instruments and stronger stakeholder ownership are not achieved in isolation. Without exception, the RAS engagements that resulted in knowledge being used or better results had at least one component focused on enhancing the effectiveness of organizational arrangements. Even when targeted changes in organizational arrangements did not lead to knowledge use in that area, the supplementary support seemed to act as an important lever for helping the client to develop and adopt new strategies and frameworks and to strengthen demand-side capacity. An adequate authorizing environment on the client side is critical for strong RAS results. The ten RAS agreements that did not result in “knowledge used� or better ratings all suffered from limited commitment, motivation, and/or institutional authority on the part of the client leadership for supporting the delivery of RAS activities and implementing recommendations. Some examples include the engagements focused on building capacity in the Ministry of Transport and Infrastructure for public private partnerships, on the enhancement of risk-based systems for the Operational Program for Human Resources Development, on support to the Ministry of Public Finance to strengthen public investment management, and on the analysis of capacity building activities in the public administration for the Managing Authority of the Operational Program for Administrative Capacity Development. Both high-level (political) and technical-level commitment is needed for RAS engagements to be effective. Efforts to establish a Delivery Unit in the Center of Government were challenged by the limited involvement of the Prime Minister, Victor Ponta. For the assistance to the Romanian Competition Council, the combination of high-level and technical-level commitment yielded strong results for all identified institutional capacity constraints. In the Ministry of Public Finance, there was strong technical-level commitment but more political commitment was needed for implementing recommendations (i.e. reorganizing the Treasury, reforming public investment management, etc.). In addition to adequate commitment on the client side, other common success factors helped to facilitate the achievement of RAS outcomes. These characteristics were frequently mentioned by both clients and Bank team members as key ingredients to help ensure the desired RAS results. 84 Most or all of the success factors were present for the 16 RAS engagements with evidence that a capacity change was emerging or in effect. In general, other factors related to the timeline and scope of a RAS did not appear to influence how effective the advisory services were. Across the diverse portfolio, there was no correlation between the progress ratings and the length of the engagement. The ambition of a RAS in terms of the number of institutional capacity changes targeted also did not appear to matter as long as the common success factors were present. The turnover in public administration management and staff has challenged efforts to enhance the administrative or technical capacity of organizations. For the climate change RAS, changes in the leadership and key staff of the Ministry of Environment, Waters and Forests have contributed to the low implementation capacity of the ministry. Frequent management changes at the National Agency for Cadastre and Land Registration limit its institutional capacity to carry out its mandate as defined in the real estate RAS. Similarly, changes in management at the Danube Delta Inter-Communal Development Association (ADI) and staff turnover at the end of the ADI ITI Danube Delta RAS are likely to constrain the use of RAS outputs. Finally, the World Bank’s standard approach to reporting RAS results does not provide useful information for understanding and tracking needed institutional changes. The Activity Completion Summary filed at the end of each RAS tends to include subjective ratings without any associated theory of change through which to assess the quality and sustainability of outcomes. In contrast, the CDRF-based approach to results measurement associates outcomes with an institutional change objective to report on not only what progress was achieved but also what progress is still needed. Strengthening the Results Focus of RAS Engagements The traditional measurement of outcomes has limited value in the context of frequent political transitions and high turnover among public administration staff. Defining the pathways or program logic for achieving institutional capacity changes extends beyond the knowledge gained at a workshop or the strategy adopted by the GoR to identify what should happen as a result. Thus, while Bank teams and their clients do discuss and document outcomes, they often miss opportunities to situate these outcomes within a larger change process, link them to longer-term targets, and plan accordingly. There are also rarely opportunities for continuing to document and learn from RAS results after RAS completion. More work is needed to foster a productive M&E culture within the Bank’s RAS Program. The methods and findings of this evaluation indicate that the following steps would strengthen the focus on results. 1. Continue to refine the CDRF approach to build a common language related to institutional capacity changes. Both the desk review process and the survey of TTLs highlighted remaining challenges with the CDRF’s terminology, where the types of institutional changes and their accompanying characteristics are not necessarily commonly understood. Continued peer review and stakeholder engagement in the development of M&E guidance would help ensure the value and consistency of its use across RAS teams. 85 2. Integrate tools and guidance throughout the RAS cycle. Current Bank templates (e.g., concept note, activity completion summary) prompt teams to think about and identify outcomes, but they fall short of tracing a theory of change for how activities and outputs will contribute to local capacity over the longer term. Simple guidance and checklists to apply during the design stage can help teams to identify up front what institutional changes are targeted by RAS activities and outputs and to negotiate changes in the RAS design if warranted. Any outputs that are not expected to contribute to institutional changes over the longer term should be removed if possible. 3. Educate stakeholders about the value of M&E. Defining and tracking intermediate capacity outcomes as part of larger institutional change processes will allow Bank teams and clients to better plan, monitor, strengthen, and communicate results. When RAS results are presented in terms of outputs, such as a new analytical report or a new strategic framework, the benefits of M&E are lost. Discussions of RAS design and presentations of RAS results should include examples whenever possible to build awareness of how M&E can provide more value than burden. 4. Create a system to disseminate and apply lessons from RAS implementation. Common themes identified during stakeholder interviews for this evaluation and for the case studies conducted in 2015 provide guidance and tips on how to strengthen the collaboration between Bank teams and the GoR. Similar themes are reported in the lessons section of the Activity Completion Summary for each RAS. However, there is no formal mechanism for sharing these lessons when new RAS agreements are being designed. At minimum, there could be a simple checklist of points to consider during the design phase. A central repository of lessons reported by TTLs would also be a useful knowledge resource. Overall, the findings of this evaluation reflect strong collaborative results and a productive working relationship between the World Bank and the Government of Romania. Incremental changes to define and strengthen the results focus will help to further elucidate how RAS outcomes fit within the Bank’s development agenda and to prioritize areas for additional support when needed. 86 References Note: All available RAS project documents were reviewed through a standard process for this evaluation. This list contains only the additional publications and resources that were cited in the report. Ministry of Economy and Finance. 2007. Sectoral Operational Programme “Increase of Economic Competitiveness� 2007-2013. Government of Romania. Ministry of Economy and Finance. 2007. Operational Programme Technical Assistance 2007- 2013. Government of Romania. Ministry of European Funds. 2014. Romanian Partnership Agreement for the 2014-2020 Programming Period. Government of Romania. Ministry of Interior and Administrative Reform. 2007. Operational Programme Administrative Capacity Development. Government of Romania. Ministry of Labour, Family and Equal Opportunities. 2007. Sectoral Operational Programme Human Resources Development 2007-2013. Government of Romania. Otoo, S., Agapitova, N., and Behrens, J. 2009. The Capacity Development Results Framework: A Strategic and Results-oriented Approach to Learning for Capacity Development. World Bank Institute. World Bank. 2012. Guide to Evaluating Capacity Development Results: A Collection of Guidance Notes to Help Development Practitioners and Evaluators Assess Capacity Development Efforts. Washington, DC. World Bank. 2015. Functional Reviews in Romania: Process and Results. Europe and Central Asia Region. A-1 Annex 1. List of Agreements in the World Bank RAS Program, 2012-2015 Note: RAS agreements are listed by the date signed. Operational Project Date Completion Duration Value RAS Title Client Program Number Signed Date (months) (US$ million) PODCA P129957 Assistance to the Ministry of Justice to Assess the MoJ 17-Feb-12 1-Mar-13 13 $0.75 Performance of the Judicial System PODCA P130508 Strengthen Strategic Planning in the Transport MTI 19-Mar-12 8-Dec-14 33 $3.95 Sector PODCA P131824 Assistance to the Romanian Competition Council RCC 4-Jun-12 3-May-15 35 $3.28 POAT Improving the national framework for preparing MoEF 27-Jun-12 27-Apr-13 10 $0.96 P130919 and implementing public investments projects POAT Competitiveness Enhancement and Smart MRDPA 31-Aug-12 16-Aug-13 11 $0.99 P131858 Specialization Policies in the West Region PODCA P130510 Prepare and Manage Public Private Partnerships MTI 6-Sep-12 11-Dec-15 39 $1.12 POAT Enhanced Spatial Planning as a Precondition for MRDPA 26-Sep-12 25-Jan-14 16 $0.74 P143087 Urban Development PODCA P143673 Assistance to the MARD for Strengthening the MARD 28-Sep-12 27-Jun-14 21 $1.23 Agri-food Sector Strategy Formulation PODCA P143674 Strategic Planning for the Agricultural MARD 28-Sep-12 27-Sep-14 24 $1.28 Administration PODCA P143675 Implementation of an Internal Management MARD 28-Sep-12 27-May-14 20 $0.63 System at the MARD and its Subordinated Structures PODCA P143676 Developing an Integrated Financial Management MARD 28-Sep-12 27-Sep-14 24 $0.37 System at the MARD PODCA P133720 Assistance to the Ministry of Public Finance for MoPF 1-Oct-12 31-May-14 20 $0.62 Strengthening Debt Management PODCA P143659 Development of Administrative Capacity of the MoESR 25-Oct-12 24-Sep-15 35 $1.97 MoESR A-2 Operational Project Date Completion Duration Value RAS Title Client Program Number Signed Date (months) (US$ million) ROP P143090 Elaboration of integration strategies for poor areas MRDPA 20-Nov-12 19-Apr-14 17 $0.58 and disadvantaged communities ROP P143089 Technical assistance for the identification of MRDPA 20-Nov-12 19-Feb-14 15 $0.76 project selection models ROP P132399 Upgrade of growth poles for strategic planning and MRDPA 20-Nov-12 19-Nov-13 12 $0.88 economic impact ROP P143088 Assessment of the communication and MRDPA 20-Nov-12 19-Dec-13 13 $0.49 collaboration between MAs and Intermediate Bodies PODCA P144505 Improvement of Human Resource Management MoPF 13-Dec-12 12-Apr-14 16 $0.37 Instruments to Strengthen the Institutional Capacity of MoPF POAT P145716 Real Estate; Basis for National and EU Policies NACLR 3-Apr-13 30-Nov-15 31 $3.39 PODCA P133582 Analysis of capacity building activities in the MRDPA 10-May-13 9-Jan-14 8 $0.37 Public Administration PODCA P144557 Strengthening Institutional Capacity in Fiscal MoPF 23-May-13 22-Sep-14 16 $0.53 Policy Formulation PODCA P144566 A Better Solution for Providing Clarifications, MoPF 23-May-13 22-Dec-14 19 $0.44 Interpretations and Rulings on Tax Issues POSDRU P145841 Reducing Early School Leaving MoESR 12-Jun-13 11-Oct-15 28 $2.01 POSDRU P145035 Study on Diagnostics and Policy Advice for MoLFSPE 17-Jun-13 16-Apr-14 10 $0.55 Supporting Roma Integration in Romania POSDRU P133830 Enhancement of Risk Based Systems of the MoLFSPE 28-Jun-13 27-May-14 11 $0.20 Sectoral Operational Program for Human Resources Development POSDRU P146187 Preparing a Strategic Framework for Increasing MoESR 17-Jul-13 16-Oct-15 27 $1.92 Tertiary Education Attainment, Quality and Efficiency A-3 Operational Project Date Completion Duration Value RAS Title Client Program Number Signed Date (months) (US$ million) POAT Climate Change and Low Carbon Green Growth MEWF 23-Jul-13 30-Nov-15 28 $6.15 P145943 Program POAT Danube Delta Integrated Sustainable Development MRDPA 4-Sep-13 3-Nov-15 26 $3.06 P146633 Strategy POSDRU P146632 Preparing a Strategic framework for Lifelong MoESR 16-Sep-13 15-Jan-15 16 $1.11 Learning POAT Support to the Establishment of a Delivery Unit CPM 31-Jan-14 4-Dec-15 22 $6.04 P147482 POSDRU P147650 Preparation of a Draft National Strategy Regarding MoLFSPE 26-Feb-14 25-Feb-15 12 $1.28 Elderly and Active Aging POSDRU P147269 Provision of Inputs for the Preparing of a Draft MoLFSPE 26-Feb-14 30-Nov-15 21 $1.83 National Strategy and Action Plan on Social Inclusion and Poverty Reduction POAT Strengthening the Regulatory Impact Assessment CPM 24-Apr-14 15-Nov-15 18 $1.97 P150017 Framework in Romania POAT Harmonizing State and EU funded Projects in MRDPA 27-May-14 26-Nov-15 18 $3.38 P147062 Regions POAT Establishment of a Performance Appraisal System MoEF 11-Jun-14 10-Jul-15 13 $0.65 for Government Officials Managing EU Funds P147746 POAT P146782 Strengthening Public Investment Management MoPF 16-Sep-14 15-Dec-15 15 $1.63 POAT Support for the Institutional Capacity Tulcea 23-Dec-14 22-Nov-15 11 $0.53 P153436 Development of ADI ITI Danube Delta County Council POAT Ploiești Growth Pole RDA South 9-Jan-15 8-Dec-15 11 $0.18 P153331 Muntenia POSCCE P152542 Implementation Support for the National Strategy MIS 2-Apr-15 1-Dec-15 8 $0.91 on Digital Agenda in Romania A-4 Annex 2. List of Stakeholders Interviewed Name Title Organization Tania Irimia Director, Directorate for Social Ministry of Education and Scientific Inclusion and Educational Research Partnerships Gabriel Leahu Secretary General Ministry of Education and Scientific Research Ella Stefan Director Intermediate Body Human Resources Development Sectoral Operational Programme (OISPODRU) Mihnea Costoiu Rector/President University Politehnica of Bucharest (Former Minister of Higher Education and Research) Remus Pricopie Rector The National University of Political (Former Minister of Education) Studies and Public Administration (SNSPA) Liliana Preoteasa Director, Unit for the Management Ministry of Education and Scientific of Externally Financed Projects Research (Former Under-Secretary of State for Pre-University Education) Valentin Popescu Director- Unit for Strategies and Ministry of Education and Scientific Public Policies Research Teofil Gherca Head of Unit, Directorate for Ministry of Regional Development and Policies and Strategies Public Administration Sorin Maxim Secretary of State Ministry of Regional Development and (Former Director of the Regional Public Administration Development Agency West) Ionut Trinca Head of Office, Managing Ministry of Regional Development and Authority for the Regional Public Administration Operational Programme Adrian Purcaru Brasov Growth Pole Coordinator Brasov Metropolitan Area Sorin Ganea Program Director Brasov Metropolitan Area Catelin Transport specialist Brasov Metropolitan Area Ariana Oana Inspector General Brasov School Inspectorate Bucur Adrian Chirca Director Cluj Metropolitan Area Ovidiu Campean Director of Strategy EU Funds Cluj Metropolitan Area A-5 Name Title Organization Adi Raulea Director Cluj Metropolitan City Hall Coraian Zoltan General Manager Intercommunity Development Association Cluj Metropolitan Area Calin Hintea Dean—College of Political, Babes Bolyai University Administrative and Communication Sciences Stefana Varvari Professor, Faculty of Economics Babes-Bolyai University and Business Administration Dana E. Bako Head of Economics Department, Babes-Bolyai University Faculty of Economics and Business Administration Elena Tudose Director Institute for Public Policy Elena Dobre Director of Social Services Ministry of Labor, Family, and Social Protection Andrada Trusca Counselor Ministry of Labor, Family, and Social Protection Mihaela Toader Director General Ministry of European Funds Mirela Director for Bucharest Tax Offices National Agency for Fiscal Calugareanu Administration Anca Lupu Counselor Chancellery of the Prime Minister Dana Gafitianu Public Manager Ministry of Agriculture and Rural Development Narcis Jeler Unit Chief, Project Implementation Climate Change Directorate, Ministry of Unit Environment, Waters, and Forests Gabriela Popescu Member, Project Implementation Climate Change Directorate, Ministry of Unit Environment, Waters, and Forests Elena Constantin Chancellery of the Prime Minister A-6 Annex 3. Measurable Characteristics for Three Types of Institutional Capacity Changes STRENGTH OF STAKEHOLDER EFFICIENCY OF POLICY INSTRUMENTS-- EFFECTIVENESS OF ORGANIZATIONAL OWNERSHIP-- the priority that Administrative rules, laws, regulations, ARRANGEMENTS—the systems, rules of government, civil society, and the standards, and other formal incentives that a action, processes, personnel, and other private sector give to a society uses to guide stakeholder actions beyond resources stakeholders bring together development goal. Measurable any one organization to achieve development within an organization (or within a group characteristics include: goals. Measurable characteristics include: of organizations functioning as a system) to achieve a development goal.  Commitment of political  Clarity in defining rights and responsibilities Measurable characteristics include: and social leaders of stakeholders  Clarity of mission  Compatibility of social  Consistency norms and values  Achievement of outcomes  Legitimacy  Stakeholder participation in  Operational efficiency  Incentives for compliance setting priorities  Financial viability and probity  Ease of administration  Demand for accountability  Communications and stakeholder  Risk for negative externalities relations  Transparency of information to stakeholders  Flexibility in addressing varying situations  Adaptability in anticipating and  Resistance to corruption responding to change A-7 Annex 4. Rating System for Assessing Institutional Capacity Development Each RAS is designed to address one or more institutional capacity constraints that impede the achievement of a development goal. Progress for addressing each targeted institutional capacity constraint can be rated by selecting the highest level along the continuum below that is supported by the current evidence. This rating system was used to measure changes in three areas of institutional capacity: the strength of stakeholder ownership, the efficiency of policy instruments, and the effectiveness of organizational arrangements. Ratings Explanation Institutional capacity challenge is identified in project documents as a constraint to the achievement 1. No outputs: Limited or no actions were taken for of the development objective; however, no activities or products have been delivered to address this addressing identified capacity challenge. challenge. Activities and/or products were designed to address the institutional capacity challenge, but only the 2. Change initiated: Activities and/or products were outputs have been measured to date. delivered to address capacity challenge but there is no evidence yet of any resulting changes. M&E data related to the activities/products indicates promising changes, but the evidence of these 3. Knowledge gained: Early results of the outcomes is not yet conclusive. For example, preliminary results might include positive satisfaction activities/products indicate emerging changes in ratings of participants, informal observations from trainers, reports from participants on planned knowledge, skills and/or relationships of relevant actions, new lists of contacts, etc. stakeholders. M&E data that confirms knowledge use could include products developed by participants that 4. Knowledge used: New knowledge, skills, and/or demonstrate new implementation know-how, concepts in knowledge products applied by local relationships are being applied as intermediate step practitioner(s), new MOUs or activity agreements that formalize collaboration, meeting minutes or toward institutional capacity change. correspondence that reflects new network, etc. Evidence of preliminary institutional capacity changes could include informal comments from 5. Capacity change emerging: Stakeholders have leaders or plans for a new citizen feedback mechanism (to strengthen stakeholder ownership); taken promising actions to address targeted adoption of a new strategy meeting quality standards (to improve the efficiency of policy institutional capacity challenge, but the changes are instruments); and plans to implement a new already-defined organizational structure, performance not yet formal or their effects cannot yet be management system, etc. (to increase the effectiveness of organizational arrangements for measured. contributing to a development objective). Some examples of evidence include formal announcements and/or budget allocations by leaders to 6. Capacity change in effect: Evidence confirms that reflect increased commitment, outcomes from strategy implementation or data showing reduced the targeted institutional capacity has improved. administrative costs to demonstrate an improved policy, and organizational data showing improved operational efficiency, the achievement of outcomes aligned to the organization’s mission, better communication with stakeholders, etc. A-8 Annex 5: Data Collection Instruments TTL Survey Questionnaire Note: This survey was administered online via SurveyMonkey. TTLs (or designated representatives) were asked to complete one questionnaire per each RAS they managed to allow for the systematic analysis of results and implementation experiences across the RAS portfolio. RESPONDENT INFORMATION You have reached this page through a unique URL linked to a specific RAS in the World Bank Romania RAS portfolio. You may leave this questionnaire and return later to continue entering data; however, please click NEXT at the bottom of the page to save any data you have entered. You will be able to review and revise responses throughout the questionnaire until you click the final SUBMIT button at the end. 1. Your name: ______________________________________________________ 2. RAS title:________________________________________________________ The development objective and outputs of this RAS have already been recorded. The purpose of this questionnaire is to explore the RAS outcomes and implementation experiences. Please reflect on your experiences with this specific RAS to respond to the following questions. DEMAND-SIDE CHALLENGES AND OUTCOMES 3. What challenges related to the demand side did you face during RAS implementation in working to achieve the development objective? (check all that apply) � One or more key leaders lacked commitment � The development objective was not to achieving the RAS development objective aligned with widespread behaviors (social norms and values) of local stakeholders � Stakeholders did not have adequate � Information related to the development opportunities to participate in decision making objective was not easily accessible to stakeholders � There were not adequate mechanisms for � NONE--There were no notable stakeholders to rate or influence the quality of constraints related to the demand side public service delivery [Skip to next section] � Other (specify:____________________) 4. You indicated that the [Q2] RAS faced demand-side challenges during implementation. Please indicate whether any of the changes below occurred as a result of RAS activities and/or knowledge products: (check all that apply or select ‘None’ if these types of changes have not yet been achieved) � One or more key leaders increased their � Widespread beliefs or behaviors have commitment to achieving the development shifted to be more supportive of the objective development objective � Stakeholders increased their participation in � Information related to the development setting priorities related to the development objective has become more transparent objective A-9 � Stakeholders have more opportunities to � NONE—There is no evidence yet that demand accountability from public service demand side changes have been providers achieved Comments—Use this space to list any evidence for the changes noted above and/or to describe any other demand-side result that does not seem to fit within these categories: 5. Has the [Q2] RAS helped to change the knowledge or behavior of individuals or groups in the following ways to help achieve or support longer-term demand-side changes? (check all that apply) � Raised awareness—improving the � Enhanced knowledge and skills understanding, attitude, confidence, or motivation of key stakeholders or groups � Improved consensus and teamwork—as � Strengthened coalitions—reflected by a shown by better communication, common agenda for action and coordination, and/or contributions increased commitment to act among stakeholders � Enhanced networks—fostered common � Increased implementation know- interest, created process for collaboration, or how—stakeholders formulated or provided incentives for participation among implemented new action plan, policy, diverse stakeholders strategy, etc. � NONE—the RAS did not contribute to notable changes in the knowledge or behavior of individuals or groups to promote demand-side changes Comments—Use this space to list any evidence for the changes noted above and/or to describe any other intermediate outcomes that reflect key steps towards demand-side changes: POLICY-RELATED CHALLENGES AND OUTCOMES 6. What challenges related to “policy instruments� did you face during RAS implementation in working to achieve the development objective? Note: “Policy instruments� could include administrative rules, laws, standards, or other formal incentives used to guide stakeholder actions. (check all that apply) � Roles, rights and responsibilities across � No strategy existed to satisfy ex ante organizations were not clearly defined related conditionality to the development objective � One or more policies or strategies to support � Existing policies or other regulatory the development objective were in conflict mechanisms related to the development with other existing regulatory mechanisms objective lacked transparency � There was not enough compliance by � Strategies or other policy instruments stakeholders with regulatory mechanisms were too difficult or costly to administer � Existing policy instrument(s) created � The policies or other regulatory unintended negative effects mechanisms do not adequately limit opportunities for corruption A-10 � Existing policy instrument(s) lacked � NONE--There were no notable flexibility and did not accommodate changes constraints related to policy in the social and/or political environment instruments � Other (specify:_______________________) 7. You indicated that this RAS faced challenges related to “policy instruments� during implementation. Please indicate whether any of the changes below have occurred as a result of RAS activities and/or knowledge products: (check all that apply or select ‘None’ if these types of changes have not yet been achieved) � Roles and responsibilities for organizations � New strategy has been approved by have been clarified related to development government to satisfy ex-ante objective conditionality � Revisions to policy instrument have made it � Policy/strategy has become more more consistent with other related documents transparent � Policy/strategy has been improved to reduce � Revised policy or other regulatory unintended negative effects mechanism has minimized opportunities for corruption � Revised policy/strategy is more adaptable to � New or revised policy/strategy is easier changing social or political conditions or less costly to administer � New policy/strategy has improved compliance � NONE—There is no evidence yet that among stakeholders needed policy-related changes have been achieved Comments—Use this space to list any evidence for the changes noted above and/or to describe any other policy-related result that does not seem to fit within these categories: 8. Has the RAS helped to change the knowledge or behavior of individuals or groups in the following ways to help achieve or support needed changes to policy instruments? (check all that apply) � Raised awareness—improving the � Enhanced knowledge and skills understanding, attitude, confidence, or motivation of key stakeholders or groups � Improved consensus and teamwork—as � Strengthened coalitions—reflected by a shown by better communication, common agenda for action and coordination, and/or contributions increased commitment to act among stakeholders � Enhanced networks—fostered common � Increased implementation know- interest, created process for collaboration, or how—stakeholders formulated or provided incentives for participation among implemented new action plan, policy, diverse stakeholders strategy, etc. � NONE—the RAS did not contribute to notable changes in the knowledge or behavior of individuals or groups to promote changes in policy instruments Comments—Use this space to list any evidence for the changes noted above and/or to describe any other intermediate outcomes that reflect key steps towards needed changes for policy instruments: A-11 ORGANIZATIONAL CHALLENGES AND OUTCOMES 9. Did the limited capacity of one or more organizations challenge the achievement of the RAS development objective? � No � Yes 10. Did the RAS activities and/or knowledge products help to strengthen the capacity of any organization(s)? � No—Skip to Next Section � Yes—list specific organization by names or type:__________________________________ ____ 11. Please indicate which of the changes below have occurred for any organization as a result (at least in part) of [Q2] RAS activities and/or knowledge products: (check all that apply or select ‘None’ if these types of changes have not yet been achieved) � Organization has a clearer mission or � New organizational structure has been vision related to roles and responsibilities established to align roles of units and for achieving the development objective personnel with organization’s mission � Organization has evidence of increased � Organization has improved its effectiveness (producing more or better operational efficiency outcomes) � Organization has improved tracking and � Organization has improved financial reporting on annual income and sustainability expenditures � Organization has improved � Organization has improved its communications with its stakeholders adaptability and uses up-to-date information to adjust operations as needed) � NONE—There is no evidence yet that changes to organizational performance have been achieved Comments—Use this space to list any evidence for the changes noted above and/or to describe any result related to organizational capacity that does not seem to fit within these categories: 12. Has the RAS helped to change the knowledge or behavior of individuals or groups in the following ways to help achieve or support needed changes to organizational capacity? (check all that apply) � Raised awareness—improving the � Enhanced knowledge and skills understanding, attitude, confidence, or motivation of key stakeholders or groups � Improved consensus and teamwork—as � Strengthened coalitions—reflected by a shown by better communication, common agenda for action and coordination, and/or contributions increased commitment to act among stakeholders � Enhanced networks—fostered common � Increased implementation know- interest, created process for collaboration, or how—stakeholders formulated or provided incentives for participation among implemented new action plan, policy, diverse stakeholders strategy, etc. A-12 � NONE—the RAS did not contribute to notable changes in the knowledge or behavior of individuals or groups to improve organizational capacity Comments—Use this space to list any evidence for the changes noted above and/or to describe any other intermediate outcomes that reflect key steps towards strengthening organizational capacity: SUSTAINABILITY 13. What are the most notable results of this RAS that you expect to be sustained over the longer term? Be specific as possible in describing how RAS activities or outputs facilitated this change. 14. What are the main risks to the sustainability of the RAS results? IMPLEMENTATION EXPERIENCE Note: TTLs with more than one RAS would be asked to complete this section only once (i.e. it asks for general perceptions of quality and formative feedback and SurveyMonkey will be programmed to skip these items if the respondent has already answered them previously) 15. Based on your general RAS experience in Romania, what factors during implementation have challenged the Bank’s ability to achieve targeted development objectives? (check all that apply) � Compressed timeframe/ implementation � Lack of engagement or buy-in among period senior management (client) � Turnover/weak capacity among public � Limited cross-sectoral coordination administration staff � Inadequate translation of outputs into � Lack of local experts Romanian � Limited availability of international � Difficulties amending RAS agreement to experts address changing needs � Cumbersome World Bank administrative � Cumbersome Government processes processes � Other (specify: __________________________________________________________) 16. What factors during implementation have helped the Bank’s ability to achieve targeted development objectives? (check all that apply) � High-level support on client side � Continuity of WBG team (same personnel worked with client on previous task(s)) � Adequate local expertise to support � Availability of relevant international implementation expertise � Effective coordination across RAS � In-country leadership of RAS team A-13 components � Frequent communication with client � Ability of the World Bank to convene diverse stakeholders � Other (specify: __________________________________________________________ ) 17. How would you rate the quality of the following types of World Bank instruments for effectively addressing development objectives for your sector in Romania? Don’t Very Excellent Average Fair Poor know/ Not good relevant Reimbursable Advisory Services � � � � � � (RAS) Non-reimbursable economic and � � � � � � sector work (ESW) Non-reimbursable technical � � � � � � assistance (TA) Investment Project Financing � � � � � � Development Policy Financing � � � � � � Comments (optional)—Use this space if desired to explain ratings above. 18. If you have experience using a RAS and another type of World Bank instrument to address similar development objectives, please briefly describe this experience and note which instrument was more effective in that case. 19. Please indicate how well the following RAS support systems, processes, and Bank personnel are working to support the quality and delivery of the RAS: Don’t Very Excellent Average Fair Poor know/ No good experience a. Local RM � � � � � � b. Headquarters RM � � � � � � c. Global practice � � � � � � d. ACS support � � � � � � e. Quality review meetings � � � � � � f. Country management � � � � � � g. Program leaders � � � � � � h. Engagement of peer � � � � � � reviewers i. Flow of documents between � � � � � � the Managing Authority, Beneficiary, and the Bank as Advisory Services Provider A-14 j. Engagement of Country � � � � � � Senior Counsel during RAS agreement stage k. RAS processing through the � � � � � � Operations Portal Comments (optional)—Use this space if desired to explain ratings above. 20. In general, how well does the current RAS governance structure, processes and systems at the World Bank help to ensure each of the following characteristics for Romania’s RAS Program? Don’t Very Excellent Average Fair Poor know/ No good experience a) Relevance of each RAS to � � � � � � the World Bank’s Country Partnership Strategy for Romania b) Sustainability of RAS results � � � � � � c) High quality and � � � � � � effectiveness d) Efficiency (achieving � � � � � � desired high quality results at the lowest cost) e) Coordination of RAS � � � � � � implementation with other relevant Bank operations Comments (optional)—Use this space if desired to explain ratings above. 21. Describe any changes in support, requirements, or supervision that are needed to improve the quality of RAS design and implementation. Be as specific as possible. A-15 Interview Protocol for RAS Results Consultations Non-Bank Stakeholders—Master Template Both the introduction and interview questions were customized based on the RAS sector, role of the stakeholder(s) being interviewed, and findings from the desk review and TTL survey. However, each interview followed the generic framework below. Overview of RAS Engagement 1. Could you describe briefly your role for [RAS Name] in terms of what your responsibilities were during the design and implementation stages? 2. What engagement did you have directly with the World Bank team during the design and implementation of the [RAS]? a. How frequently did you communicate with the Bank team? b. What method(s) did you use to communicate with the Bank team? (e.g., in- person meetings, telephone discussions, email exchanges, etc.) 3. From your perspective, what were the most important results of these advisory services? Results I would like to explore in a bit more detail some of the key activities and results of these advisory services. [select questions from the following list and customize based on RAS] Knowledge Products 4. How have the [reports] been used? Who has used them? 5. Are these [reports] likely to support long-term changes in the sector? a. If so, please describe the changes that you expect. [Probe as needed for how the knowledge or behavior of key stakeholders is changing or might change as a result of these knowledge products] b. Is there any evidence yet that these changes are taking place? Learning Events and Multi-Stakeholder Engagements 6. Did the participants gain new awareness or knowledge from attending this [training, forum, etc.]? 7. What evidence have you seen of these changes since the [training]? 8. Were there new relationships or arrangements established during this [forum, meeting]? If so, could you describe these? 9. Looking ahead, are the outcomes of the [training, forum] expected to lead to longer-term changes in the sector? A-16 a. If yes, please describe the changes that you expect. Is there any evidence yet that these changes are being achieved? b. If no, please describe the current challenges that could hinder the achievement of the longer-term outcomes. 10. From your perspective now, were the right participants engaged for this [training, forum, meeting]? If not, what changes would have made the activities more effective for addressing Romania’s development objectives? 11. What support or resources would help to ensure that the Government of Romania can continue to build on the [learning outcomes] achieved by these advisory services? Strategy/Policy Development 12. To what extent, did [agency] receive the support needed for developing a [strategy, policy] for addressing an existing development challenge? 13. Were the right stakeholders engaged during these advisory services for an effective process? If not, what improvements could have been made? 14. How has the [agency] used the draft [policy, strategy] that was developed through these advisory services? What is the current status of the [policy/strategy]? 15. How satisfied are you with the new strategy/policy for addressing Romania’s development challenges related to [sector area]? [If not satisfied] What changes would improve the [strategy/policy]? 16. Does the government have adequate capacity to implement the new [strategy/policy] effectively? What concerns do you have about the implementation of the [strategy/policy]? 17. What support or resources for [agency] would help to ensure the effective implementation of the [policy/strategy]? Reflections on the Planning and Implementation Experience 18. Now that you can reflect on this RAS experience, was this the best design for this advisory service agreement to meet the objectives? If not, what changes would have made it more effective? [probe as needed on scope, sequencing, and timeframe] 19. From your perspective, did the implementation of these advisory services go as planned? If not, why not? 20. In general, how responsive did you find the Bank representatives were to your communication and requests? A-17 21. How adequate was the project coordination for supporting the achievement of the RAS objectives? Did this coordination foster the engagement of the right stakeholders? 22. Did the World Bank provide the right mix of expertise for these advisory services to be effective? a. If yes, what expertise was particularly valuable during this process? b. If no, what expertise was missing or inadequate during this process? 23. Are there any specific lessons you gained from this experience that will influence how you work with the World Bank in the future? [e.g., design considerations, personnel engagement, timing, etc.] 24. Are there any other reflections about your collaboration with the World Bank that you would like to share? A-18 A-19