Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) Report Number : ICRR0021645 1. Project Data Country Practice Area(Lead) Serbia Macroeconomics, Trade and Investment Programmatic DPL Planned Operations: 2 Approved Operations: 2 Operation ID Operation Name P127408 YF First SOE Reform DPL L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-84910 31-Mar-2016 101,460,585.20 Bank Approval Date Closing Date (Actual) 24-Mar-2015 31-Mar-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 100,000,000.00 0.00 Actual 101,460,585.20 0.00 Prepared by Reviewed by ICR Review Coordinator Group Felix Oppong Clay Wescott Malathi S. Jayawickrama IEGEC (Unit 1) PHPROJECTDATATBL Country Practice Area(Lead) Serbia Finance, Competitiveness and Innovation Operation ID Operation Name P149750 YF Second Programmatic SOE Reform DPL ( P149750 ) Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-84910,IBRD-86540 31-Dec-2017 96,944,712.50 Bank Approval Date Closing Date (Actual) 25-Oct-2016 31-Dec-2017 IBRD/IDA (USD) Co-financing (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 100,000,000.00 0.00 Actual 96,944,712.50 0.00 2. Program Objectives and Policy Areas a. Objectives According to the program documents (PD) of the operations, the program development objectives (PDOs) are to (i)reduce state participation and level of direct and indirect support to the real sector, (ii) enhance state owned enterprise (SOE) performance, governance and accountability, and (iii) mitigate the short term social and labor impacts of the SOEs’ restructuring and disposition plans (DPL1 program document, 2015:4; DPL2 program document, 2016: 4).” Given that the PDOs did not change during the implementation of the programmatic series, this ICR Review will use this formulation of the objectives for the validation. b. Pillars/Policy Areas The program had three pillars: 1. To accelerate the restructuring and divestiture program for companies in the Privatization Agency portfolio and selected state-owned enterprises operating in the commercial sector. The pillar aimed to reduce state participation in non- performing State-Owned Enterprises (SOEs) that had been included in the portfolio of a Privatization Agency. It was expected that the achievement of this objective would directly impact the fiscal accounts through a reduction in annual subsidies and soft loans given to SOEs, non-accumulation of tax and social contribution arrears and a reduction in the arrears of companies in Privatization Agency Portfolio to public utilities. Actions envisaged to achieve this objective were: (i) enact and implement a new Privatization Law and amend the Bankruptcy Law to facilitate the disposal and restructuring of companies; (ii) assist government to design and implement methods for privatizing the companies; and (iii) assist companies in the Privatization Agency portfolio to manage potential environmental risks that were slowing the privatization process. Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) 2. . To Strengthen governance regulatory, institutional framework, monitoring and transparency arrangements for improved performance and accountability of state-owned enterprises. The pillar aimed to improve the performance, governance and accountability of public utilities by amending the legal and regulatory framework in line with international standards and practices. Improved corporate governance of the affected SOEs was expected to reduce their reliance on the state for funding, attract private investment, and support broader public expenditure management reforms. The actions undertaken to achieve the objectives focused on adopting a Law on Public Enterprises to address fragmentation of the monitoring of public enterprises, clarification of corporate governance and performance monitoring provisions, and removal of ambiguities in the selection process of General Managers and Board Members of public enterprises. 3. To mitigate the social and labor market impact of the reform program. The pillar aimed to: (i) to ensure adequate financial protection of redundant workers of companies from the Privatization Agency portfolio; and (ii) to facilitate the transition into employment and provide a temporary safety net for vulnerable redundant workers. Through this the pillar, the Bank aimed to support the enactment of a decree to detail the options and amounts available for social programs and to facilitate the utilization of resources under the special budget-financed, and transition funds. It also aimed to support efforts to link redundant workers with services offered under the National Employment Service. c. Comments on Program Cost, Financing, and Dates The first operation (DPL1) was approved on 3/24/2015 and closed on 3/31/2016; the second operation (DPL2) was approved on 10/25/2016 and closed on 12/31/2017. All dates were as planned. Both loans were single tranche disbursement, denominated in dollars and on IBRD terms. The total amount of US$198.4 million was disbursed, of which US$101.5 million was for DPL1 and US$96.9 million for DPL2. There was no co-financing or borrower contribution. 3. Relevance of Objectives & Design a. Relevance of Objectives At appraisal, the objectives of the series were relevant and were directly derived from the government strategy. The Fiscal Strategy for 2015-17 indicated the need to reduce government allocations in the national budget to public and state enterprises by: (i) reducing subsidies to SOEs, (ii) strictly limiting guarantees for new loans, and (iii) strengthening of accountability and transparency in the operation of the enterprises (Fiscal Strategy, 2015:102-103). The strategy emphasized the importance of disposing of all enterprises within the Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) portfolio of the Privatization Agency through privatization or by bankruptcy. Also, the government planned to assist other enterprises outside the Privatization Agency portfolio to find strategic partners and to restructure those in the utility and infrastructure sectors (such as energy, railway, transport and roads). These objectives were described in the February 2015 Memorandum of Economic and Financial Policies with the IMF as a clear priority of the government to improve SOEs operational viability and limit fiscal risks to the budget. The DPL program was aligned to the FY2012-15 and FY2016-20 Country Partnership Frameworks of the World Bank Group (WBG) in Serbia. The privatization objective of government was articulated in focus area 1 of the Country Partnership Framework (FY2012-15) which was related to economic governance and role of the state. One of the objectives of this focus area (titled “productive SOE assets transferred to private ownership”) discussed the need for the state to reduce its involvement in commercial SOEs that posed a significant drain to scarce fiscal resources in Serbia. It noted the importance of accounting for the impact of privatization on affected SOE workers and the need to mitigate these effects through development policy lending. SOE reform was also identified as one of the top priorities which could impact on the WBG twin goals by the 2015 Systematic Country Diagnostic of the Bank (SCD, 2015:104). At completion, SOE reforms continued to be relevant to the government. It is explicitly indicated in the government’s Fiscal Strategy for 2018 and 2019 that “the operation of SOEs is a source of significant fiscal risk to both revenue and expenditure sides of the budget.” Some SOEs are still facing numerous problems in revenue collection, settlement of liabilities, and profitability. It notes that the major fiscal risks relate to government guarantees of SOE Loans. The sector remains relevant because it employs a significant segment of the Serbian economy (nearly 79,000 people). The 2018/19 Fiscal Strategy of government notes efforts by the World Bank, International Monetary Fund and European Bank for Reconstruction and Development and European Bank for Reconstruction and Development to reduce the fiscal risks from large SOEs (Fiscal Strategy, 2017:60). Rating Substantial b. Relevance of Design The policy actions pursued by the DPL series were consistent with the objectives and could be expected to contribute to reducing state participation in SOEs, strengthen the regulatory institutional framework and mitigate the adverse impact of privatization on workers and the environment. In DPL2 (PD, pp. 12-13, 34), Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) four triggers became prior actions with no substantial change. Two triggers were combined to become one. One trigger on establishing audit committees was revised because the restructuring of 4 companies in the railroad, electricity and energy sectors delayed establishing audit committees in those cases. The trigger on enhancing labor market incentives didn't become a prior action because it couldn't be completed within the time frame. Three targets concerning reducing subsidies to SOEs were strengthened. The program document of DPL1 (program document 2015: 11-12) indicated that guarantees to Srbijagas and the utility companies were sources of significant domestic risks. The institutional depth of the reforms supporting the objectives was mixed across the three pillars. While some prior actions in the DPL1 had follow up actions, there were others in DPL2 that required additional actions to ensure the achievement of results. For instance, the second DPL had prior actions that required the borrower to issue public announcement of bids for companies, select companies for bankruptcy and initiate process of bankruptcy. The institutional depth of these prior actions was not adequate by themselves to change incentives and behavior. Nonetheless, the World Bank supported other reforms that added value to the achievement of the objectives such as the enactment of a new Privatization Law, amendment to the bankruptcy law, and the adoption of a Law on Public Enterprises. The choice of a programmatic series to implement structural SOE reforms was appropriate, given the large number of SOEs involved in the reforms and complexities associated with dealing with a sector that employed about 15 percent of the formal Serbian workforce. The design was aided by effective dialogue and engagement with the government and other stakeholders. The use of analytical products, including rapid technical notes for policy dialogue strengthened the design of the series. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To reduce state participation and level of direct and indirect support to the real sector. Rationale To achieve the above objective, the series focused on reducing the direct and indirect support to companies in the Privatization Agency Portfolio, minimizing the disposal and treatment of waste generated by companies through the enactment of the Privatization Law, amending the Bankruptcy Law, assessing the Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) volume of environmental risks posed by selected companies, and initiating a process to implement the laws. The intention was to restructure problematic SOEs, and thereby reducing the annual direct subsidies and soft loans channeled to SOEs, while minimizing the accumulation of arrears from new taxes, public utilities and social contributions. Outputs and Outcomes Evidence provided by the ICR indicates the enactment of a new Privatization Law and an amendment of the Bankruptcy Law eliminated the moratorium prohibiting debt enforcement actions against entities being restructured, and those affected by the bankruptcy provisions. It notes (ICR 2019:12) that as of 2011 Serbia’s universe of SOEs comprised the following three groups: (i) enterprises in the Privatization Agency’s portfolio (approximately 640 companies); (ii) large SOEs (approximately 50 companies); and (iii) municipal enterprises (approximately 645 companies). Evidence provided by IMF 2019:13 (Article IV consultation report and Second Review under the Policy Coordination Instrument) states that by March 2019, more than 310 companies had entered bankruptcy, and 54 had been privatized since 2014. Of these, the World Bank ISR of March 2016 confirmed that (at appraisal of the second operation in the series) 89 companies had been resolved, mostly through bankruptcies by 2016. IMF 2019 also noted that about 34,800 employees from 336 companies have received severance payments between 2014 and 2019. Nonetheless, Serbia does not keep a single comprehensive list of all SOEs for all purposes. While the ICR reports the existence of about 1,200 SOEs in Serbia as of 2014, the IMF reports about half of the number over the same period. Serbia keeps multiple SOE lists for various reasons (e.g., statistical, employment, former Privatization Agency portfolio, etc.). There are also two categories of SOEs: (i) regular SOEs, which are governed in line with the general Company Law; and (ii) public enterprises (PEs), which operate in industries that have been defined as general interest in the Law on Public Enterprises (IMF, 2019:25). The World Bank Group cannot be sure of results when the baseline (number of SOEs) is not consistently reportedly from one year to the another. The challenges addressed are likely to re-occur after the operations close because of inaccurate data reporting. The series nearly achieved its target (strengthened between DPL1 and DPL2) to reduce the state’s annual direct subsidies and soft loans to companies in the Privatization Agency portfolio to less than €5 million by 2017 from a baseline of €85 million in 2010-2012. The ICR (2019:17) reports that the State’s annual direct subsidies and soft loans were reduced to €20 million in 2017 and €8 million in 2018. It also noted that three petrochemical SOEs (Azotara (fertilizers), MSK (industrial acids), and Petrohemija (basic plastics)) relaunched their production in 2017 and requested government assistance to buy the gas. Contrary to the objectives of the series, the government supported the purchase of gas for these SOEs resulting their inability to meet the target. It is noted that the decision to support the three petrochemical SOEs was politically sensitive. Also, the DPL2 (program document, 2016: 34) noted that the series targeted to reduce new tax obligations and social contributions arrears to €5 million in 2017, from a baseline of €190 million (2010-2012). The ICR reports that the target was nearly met. The government was able to reduce the arrears significantly from an average (2010-2012) of €70 million to €10 million by 2017, noting that 43 out of 121 SOEs from the Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) Privatization Agency portfolio increased their arrears (by €10 million), 35 companies reduced by €15 million, and 43 companies kept the same level (ICR, 2019: 28-29). In addition, the largest SOEs accumulated about 0.3 percent of GDP in tax and social contribution arrears during 2016-2018, and arrears of the 10 largest SOEs to the state electricity company amount to 0.4 percent of GDP (IMF, 2019:58). The series (DPL2) used an indicator measuring potential environmental damages, that could be an issue in privatization. The indicator was defined as: “disposal and treatment of legacy hazardous waste generated by companies in the Privatization Agency portfolio”, to reach a cumulative target of 1,500 tons between 2015-2017 for 180 companies in the portfolio, up from a baseline of zero. The ICR reports evidence that 1,071 tons were disposed and treated (permanently or contracted for) for one company (ICR,2019: 8-9). While the government verbally informed the Bank that 2,300 tons was disposed and treated for all 12 companies, no documentation on this for 11 of the 12 companies could be found. It is noted the fiscal costs of the disposal and treatment of legacy hazardous waste continued to weigh on different levels of government rather than the SOEs themselves. Municipalities continued to provide large subsidies to SOEs, mostly related to local public enterprises as of 2019, including some that were turned over to municipalities after attempting to privatize them (IMF, 2019:57; ICR, 2019:16). It can be concluded that the target may have been achieved, but that the evidence on this was not conclusive. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To enhance performance, governance and accountability of State-owned enterprises Rationale To improve the performance, governance and accountability of state-owned enterprises, the series took a particular focus on public utilities, through amendments of the legal and regulatory framework in line with international standards and practices. It also aimed at correcting the poor operational and financial performance in the SOE sector driven by weak monitoring and corporate governance arrangements, resulting from political influence in the decision-making and operations of SOEs. It was expected that the strengthening of the Law on Public Enterprises will catalyze improvements in the governance of SOEs. Outputs Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) The series supported the adoption of a new Law on Public Enterprises and the appointment of 15 supervisory boards to strengthen the regulatory framework for monitoring and ensuring the accountability and transparency of SOEs in Serbia. In addition, the series supported the operationalization of the Public Oversight Board for auditing and helped establish the publication of quarterly business monitoring for public enterprises. Outcomes The results were measured by two indicators. The first required all public enterprises to prepare and publish their audited financial statements as required by the Law on Public Enterprises and Law on Accounting and Auditing for all public enterprises (for which the founder is the Republic of Serbia). The ICR reported that this legal provision was followed by all public enterprises that were required to. Random checks using a link provided by the ICR to the online repository of the financial statement validates the claim that some financial statements are posted online. IMF (2019:53) notes that “Financial statements are prepared according to international accounting standards, are audited, and published, but are not reviewed thereafter. It can be verified that some information is publicly available, and the annual financial statements can be accessed through the Serbian Business Registry Agency. However, Serbia does not produce an aggregate SOE report on the entire sector.” Thus, it can be concluded that the series contributed to promoting accountable governance of SOEs. The second indicator for this objective proposed to reduce direct subsidies to large SOEs for recurrent expenditures to less than € 200 million for 2015 and € 150 million for 2017. The ICR reports that this target was partially achieved, given that direct subsidies reached € 170 million at the end of 2017 from a baseline of € 250 million, encompassing SOEs in the portfolio of the Privatization Agency and other large SOEs. It further notes that the SOEs in the portfolio of the Privatization Agency were resolved and did not require any direct or indirect support (ICR 2019:30). The budgetary support to large SOEs accounted for the inability of the government to meet the target. IMF (2019:51) recognizes that steps had been taken by the government to reduce losses and lower fiscal risks of SOEs but the contingent liabilities stemming from unfinished reforms are likely to arise. The ICR reported that annual guarantees for liquidity purposes to SOEs were completely eliminated by 2017 from a baseline of € 265 million in 2014. However, IMF (2019:31) reports gross new issuance of guarantees to SOEs of RSD 84 billion (2016), RSD 54 billion (2017) and RSD 8 billion (2018). The evidence shows that guarantees to SOEs were reduced but not completely eliminated as reported by the ICR. The evidence provided indicates that the series made substantial progress in achieving this objective. Rating Substantial PHREVDELTBL Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) PHEFFICACYTBL Objective 3 Objective To mitigate the short term social and labor impacts of the SOEs restructuring and disposition plans. Rationale To mitigate the short-term labor and social impacts of resolution of the companies from the Privatization Agency portfolio by ensuring that there is adequate financial protection of redundant workers of companies from the PA portfolio, the series facilitated the transition into employment of redundant workers and provided a temporary safety net. Output The series supported the creation of a special budget financed fund through the 2015 budget law to provide financial assistance to redundant workers in lieu of severance pay. It also supported through the Ministry of Labor, Employment, Veteran, and Social Affairs and its National Employment Service, an improved design and targeting of public works that prioritized applications from persons who are deemed redundant, unskilled, or hard-to-employ. In addition, it supported government to limit the payment of remuneration of RSD 15,000 per person per month under a casual employment contract. Outcome Three results indicators were used to monitor the achievement of the above objective. First, the series aimed at increasing the number of redundant workers receiving compensation from the Transition Fund from about 5,700 (in 2014) to at least 25,000 by 2016. The ICR notes that about 27,981 people benefited from the Transition Fund, of which 8,951 ( about 32 percent) were women. IMF Article IV published in December 2015 (IMF 2015: 62) acknowledged the allocation of budgetary funds into the Transition Fund to pay severance redundant workers. It is noted however, that the payments from the Transition Fund is not a long- term solution to the redundancy challenges of workers but a solution that could complement other measures in the labor market. To that extent, the objective of the series to mitigate short term social impact was achieved. The second indicator aimed to register at least 30 percent of workers made redundant from public enterprises during 2016 with National Employment Service. The DPL2 (program document 2016: 26) notes that the National Employment Service designed measures to support affected workers, such as, through individual action plans, and outreach to potential new employers and registering job vacancies. The ICR reports that 85 percent (4,758 out of 5,587) were registered, of which 1,787 were women. It notes that the target was exceeded because the National Employment Service visited every company that laid off more than 10 workers and consulted local employment councils for support. The third indicator aimed to increase number of participants in public work from 2,882 in 2013 (1,187 female and 1,695 male) to at least 6,000 in 2015 and to at least 7,000 in 2016. The ICR (2019:32) reports that the targets were exceeded for both 2015 (10,853, out of which 4,767 were women) and 2016 (7,357 of which 3,319 were women). Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) The evidence indicates that substantial progress was made in the achievement of this objective, and the series contributed to this achievement. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Outcome The objectives of the series were substantially relevant to the World Bank’s Strategy for Serbia and the government priorities during the operations that aimed to reduce subsidies to SOEs, limit guarantees for new loans, and strengthen of accountability and transparency in the operation of SOEs. Relevance of design is rated substantial, since the operations were based on analytical products and provided technical notes for the implementation of the reforms. Two of the objectives are rated Substantial for making significant progress toward strengthening SOE governance and mitigating the social and labor impact of the SOE reform program. One objective is rated modest because the targets to reduce state participation and level of direct and indirect support to the real sector were not fully met, although they were nearly met. a. Outcome Rating Satisfactory 6. Rationale for Risk to Development Outcome Rating Some of the reforms support by the series may be undermined if the government does not take stock and publish the comprehensive list of SOEs in the country and consistently publish the list of SOEs that benefit from the government budget either directly or indirectly. There are risks associated with putting in place supervisory boards and audit committees without ensuring the preparation of proper reporting guidelines, comprehensive assessment of SOEs financial data and implementation of uniform KPIs. There is also the risk that vested interests will secure new SOE subsidies despite the laws and policy opposing them. It is also noted that the alleviating impact of the measures on workers is only short term, indicating the possibility of longer-term risks if other structural measures are not taken. Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) a. Risk to Development Outcome Rating Modest 7. Assessment of Bank Performance a. Quality-at-Entry The series benefited from a consultative process that influenced the production of a 2015 Systematic Country Diagnostics and the Country Partnership Framework (FY2016 -20). The ICR also reports that the frameworks of the DPLs were developed jointly with the authorities and in consultation with other stakeholders and the results indicators were directly linked to the reform pillars and data was to be collected by the Ministry of Economy, line ministries and other relevant agencies (ICR, 2019:24). In addition, the prior actions were supported by analytical products such as (i) the Poverty and Social Impact Assessment; and (ii) the regulatory and implementation framework for addressing environmental liabilities in privatization in Republic of Serbia (World Bank, June 2013; and OECD Guidelines on the Corporate Governance of State-Owned Enterprises). The series was supported by a limited number of World Bank prepared knowledge products. The program documents and ICR referenced a policy note prepared on the topic “ OECD guidelines on the corporate Governance of State- owned Enterprises”, and a Poverty and Social Impact Assessment that underscored the relevance of supporting redundant workers. The World Bank based many of the prior actions on external papers such as Goldberg and Nellis (2007), Methods and Institutions – How do They Matter? Lessons from Privatization and Restructuring in the Post-Socialist Transition, and a 2013 analysis of the legal and regulatory framework produced by Milieu, Law and Policy Consulting. The macroeconomic framework during design and implementation was adequate to support the series (as discussed in the DPL1 program document, 2015: 7-8; DPL2 program document, 2016:7-12). The Executive Board of the IMF approved a three-year Stand-by Arrangement for Serbia on February 2015 which focused on restoring the health of public finances, financial sector stability and resilience, and job creation. The DPL1 program document (2015:12); and the ICR (2019:21) noted that Bank staff collaborated with the IMF and reinforced each other’s efforts on the restructuring and privatization of public enterprises. This is confirmed by IMF (2015:2) that stresses the importance of government keeping the momentum of implementing the SOE structural reforms due to the expected impact on reducing fiscal risks. In addition, the IMF cites World Bank consultations in relation with financial restructuring plan of Srbijagas noted by DPL 2 program document (IMF, 2015:12; DPL 2 program document, 2016:17-19). In August 2016, the IMF’s combined fourth and fifth reviews of Serbia’s economic performance under the Stand-by Arrangement indicated that the fiscal had over- performed due to stronger revenues and tighter controls on current expenditures. Finally, the choice of a programmatic series for SOE reforms was appropriate in sustaining the impact of the policy actions pursued under that series. Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR, the Bank team had strong and near continuous dialogue with the authorities, in addition to three to five missions per year to assess the progress of implementation of the reforms (ICR 2019:37). It also notes provision of technical notes to the government during the implementation of the series. A March 2016 ISR filed by the Bank team detailed progress the authorities had made in implementing the prior actions for the second operation, progress in achieving the results indicators of the first operation and challenges associated with the SOE prior actions for the second operations. This evidence and other ISRs validate the Bank’s active supervision of the reforms. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 8. Assessment of Borrower Performance a. Government Performance The ICR (2019:38) notes that the government developed a strategy to address the key priorities in the program because the reforms were highly political and required broad consensus within government. At the same time, the government worked with multiple development partners (World Bank and IMF) on different aspects of SOE reforms. The UNDP and Austrian Development Cooperation provided support to redundant workers of SOEs (2007), ahead of the series and reported the need for longer term reforms (UNDP 2007- Severance to job project- Support to re-employment). The reforms supported by the Bank were second generation reforms and yet the government was committed to finding lasting solutions to challenges arising from the fiscal costs of supporting SOEs, SOE governance and redeployment of redundant workers. Government Performance Rating Satisfactory b. Implementing Agency Performance Not Applicable Implementing Agency Performance Rating Not Rated Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) Overall Borrower Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Of the seven indicators in the results framework, only one (disposal and treatment of legacy hazardous waste generated by companies in the Privatization Agency portfolio) was not directly linked to the objectives of the series. However, it is noted that disposal of legacy hazardous waste could facilitate privatization of the SOEs. The results framework had appropriate baselines and targets required to monitor the achievement of results but the ICR reports of difficulties in collecting data to ascertain some of the results. One aspect that could be improved is that the monitoring framework measures uses different samples of SOEs for different indicators It could have been clearer why a particular sample was used for a particular indicator. b. M&E Implementation The second operation of the series amended the results indicators without taking into account the impact of the decision of the government to call early elections. The targets for the indicator A1 changed from cutting annual direct subsidies and loans from €10 million to €5 million. Also, the target for new tax and social contribution arrears was reduced from €20 million to €5 million; and the target for new arrears to public utilities was reduced from €20 million to €5 million. Finally, the target for the result indicator on annual guarantees for liquidity purposes was brought to zero from €100 million. As acknowledged by the ICR (2019:16), the changes were ambitious but were informed by the progress the government had made in the implementation of the reforms supported by the first operation. In addition, tracking of some of the results indicators proved difficult after the series closed, though the results framework was designed to align with regular data collected by the government on SOEs. As noted earlier the multiple reporting of SOE data, changes in definition of indicators, and some indicators not being tracked anymore was a concern. c. M&E Utilization The M&E framework was used to make changes to the reforms supported by the series, especially during the design of the second operation. M&E Quality Rating Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) Substantial 10. Other Issues a. Environmental and Social Effects The DPL2 program document (2016:27) notes that the reforms supported by the series has no significant adverse effects on the environment, forests and natural resources. The ICR confirms that the series strengthened the capacities of the government to effectively address the social, labor, and environmental impact of the reforms in the program (ICR 2019:13). For instance, the series supported the improvement of environmental management to properly dispose hazardous waste generated by companies in the PA portfolio. It also supported the management of redundant workers. b. Fiduciary Compliance There were no fiduciary issues reported in the ICR report. c. Unintended impacts (Positive or Negative) No unintended impacts reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Modest Modest --- Outcome Bank Performance Satisfactory Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Modest --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 12. Lessons The ICR draws a number of lessons, of which two are of particular interest and restated as follows: Structural SOE reforms require a long period of engagement and implementation before achieving expected outcomes. In the case of Serbia, the ICR reports that it took five years from the first identification to completion in 2016 and yet some of the impact of the new laws passed are yet to be realized. Maintaining the policy dialogue and the provision of critical support over the medium term is essential for realizing the full impact of the series. Secondly, when a DPL series has not completely addressed challenges related to structural SOEs reforms, continuing the series with other programmatic operations with follow-up actions or technical assistance may be appropriate. Serbia has a relatively large SOE sector with large fiscal costs that could potentially affect real sector growth. In this case, the series tackled short term reforms without sufficient time and attention to achieving longer term results. The management of the fiscal costs of SOEs still requires support. After implementation of the series, Serbia’s SOE governance still shows significant gaps against OECD best practices, with room to improve SOE governance frameworks across all dimensions (IMF, 2019:58). There is room for more transparency, assessment, and reporting to improve the governance and performance of SOEs, and the Bank is continuing to support SOE reforms with a follow-on DPL series. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well written overall but failed to provide sources of the information used to populate some of the indicators in the results framework. In addition, there were many hypothetical statements that were not accompanied by evidence. The relevance of objectives was discussed using a recent government strategy prepared after the project had closed without reference to relevance at the appraisal stage. The Task Team Leader did not respond to IEG's request for a meeting to discuss the ICR. a. Quality of ICR Rating Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review YF First SOE Reform DPL(P127408) Modest Page 16 of 16