Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) Report Number: ICRR0021614 1. Project Data Project ID Project Name P103974 CCT Country Practice Area(Lead) North Macedonia Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-77350 28-Feb-2014 21,507,935.00 Bank Approval Date Closing Date (Actual) 16-Jun-2009 31-Aug-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 25,000,000.00 0.00 Revised Commitment 22,345,355.32 0.00 Actual 21,507,935.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Robert Mark Lacey Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (p. 5), the objectives were "to strengthen the effectiveness and efficiency of the Borrower’s social safety net through: (a) the introduction of conditional cash transfers (CCT); and (b) improvements in the administration, oversight, monitoring and evaluation of social assistance transfers." Three outcome targets were revised at a May 2013 restructuring. As they were revised upward, this review does not undertake a split evaluation. Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 23-May-2013 c. Will a split evaluation be undertaken? No d. Components The project had three components: 1. Enhancing the Human Capital Links to Cash Assistance (appraisal: Euro 16.15 million; revised at 2013 restructuring: Euro 12 million; actual: Euro 8.67 million). This component was to finance: (i) the implementation of a CCT program for poor families with children in secondary education, paid as a "top-up" for existing Social Financial Assistance (SFA) recipients, conditioned on an 85% school attendance rate, and paid directly to beneficiaries on a quarterly basis; and (ii) the identification, development, and implementation of possible CCT extensions, including diagnostics on defining target groups, incentives, outreach mechanisms, and other program parameters. The means-tested SFA program, a "benefit of last resort" similar to guaranteed minimum income schemes in many European Union countries, had been demonstrated as well targeted to the poor. 2. Strengthening Safety Net Administration (appraisal: Euro 1.97 million; revised at 2013 restructuring: Euro 5.46 million; actual: Euro 7.11 million). This component was to build on and extend activities being supported under the then ongoing Social Protection Implementation Loan (SPIL) Project (2004-2011, US$ 9.45 million), which focused on strengthening the efficiency of benefits administration and oversight, improving service delivery, and building capacity for policy development, monitoring, and evaluation. The component's planned activities included further improvements in the cash benefits management information system (CBMIS); assessments of targeting, eligibility, and registry systems for various cash benefit programs; development and application of oversight and control mechanisms to monitor implementation of the CCTs and other social benefits; a functional analysis of Social Welfare Centers (SWCs); and capacity building for program management. 3. Project Management, Monitoring, and Impact Evaluation (appraisal: Euro 1.13 million; revised at 2013 restructuring Euro 1.79 million; actual: Euro 1.42 million). This component was to finance activities to support project administration and implementation; development and establishment of a monitoring and impact evaluation system; and development and implementation of a public awareness campaign. At a 2013 restructuring, an amendment to the first component allowed for financing of the CCT-subsidized youth employment program beyond a pilot phase, as it became clear that this would the main supported Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) CCT "extension"; and a new subcomponent was added to the second component to finance minor civil works for refurbishment of eleven existing SWC offices. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: Total costs were estimated at Euro 20.3 million (US$ 25 million). Euro 17.25 million (US$ 21.5 million) was actually spent. The project spent considerably more than planned on the second component, and about half of what was planned on the first component. Financing: The project was to be financed by a Specific Investment Loan in the amount of Euro 19.3 million (US$ 25 million). Euro 17.25 million (US$ 21.5 million) was disbursed, and Euro 1.6 million (US$ 2.9 million) was cancelled. Borrower contribution: A Borrower contribution of Euro 1.0 million was planned. The ICR did not state whether this contribution was actually made. The Borrower's comments stated that the government contributed about Euro 27,000 (ICR, p. 60). Dates: The project was approved on June 16, 2009 and became effective on October 28, 2009. It underwent five restructurings:  January 2011: The financing percentage of the loan categories was changed to 100% Bank financing, and disbursement conditions were amended to allow for the financing of the first year of CCT program implementation.  May 2013: Some changes were made to the language and targets of three of the five key performance indicators, as well as to some intermediate indicators. Targets were revised upward, and edits to indicator formulation were for clarity and higher precision rather than substance. Elements of the first component were expanded, and a new subcomponent was added to the second component. The closing date was extended by 22 months, from February 2014 to December 2015.  February 2015: Funds were reallocated between disbursement categories.  July 2015: The closing date was extended by two years to December 2017, to complete activities related to cash benefits administration and further policy dialogue.  September 2017: The target value for number of direct project beneficiaries was revised upward, some funds were cancelled, and the closing date was extended a final time to August 2018. The mid-term review was held in September of 2012. The project closed four years later than its originally scheduled closing date, on August 31, 2018. 3. Relevance of Objectives Rationale The project's objectives were straightforward, and measurable, and their attainment was within existing capacity. The project was highly responsive to country conditions and government strategy at the time of Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) appraisal. At that time, it appeared likely that poverty and demand for social assistance benefits were likely to remain high, particularly as a consequence of the 2008 global financial crisis. Although coverage and targeting of the SFA to the poor were strong, the social safety net faced challenges related to fragmentation and administration. A four-pronged government strategy for reform, initiated in 2007, focused on legislative action to harmonize benefits, improvements in the cash benefit information system, structural reform of SWCs, and introduction of CCTs. This project built on activities initiated under the predecessor SPIL project to support this strategy. This project also aimed to correct important demand-side gaps in secondary school enrollment, attendance, and completion, acknowledging access and quality of education as one of the government's top five stated priorities at the time (PAD, p. 4). The objectives were also highly relevant to Bank strategy at appraisal and at closing. The Country Partnership Strategy (CPS) approved by the Bank in March 2007 and covering FY 2007-2010 established three strategic priorities: enhancing economic growth and job creation, improving governance, and improving public service delivery. The project supported these priorities by linking elements of the human development agenda (education) into a comprehensive effort to tackle persistent and inter-generational poverty. The CPS at closing (FY 2015-2018) contained a pillar on "skills and inclusion," focusing on both employability and more efficient public services; CCT was specifically listed as one of the tools for achieving the CPS outcome "social protection system more efficient and compatible with employment." Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen the effectiveness of the Borrower’s social safety net Rationale The PAD did not define "effectiveness," but the ICR reasonably inferred from the indicators and activities assigned to the objective that effectiveness was intended to mean improved coverage of the CCT and impact on secondary school enrollment and attendance rates for children in beneficiary households. The theory of change for this objective held that support for the CCT program for poor families with children in secondary education, as well as the identification, development, and implementation of possible CCT extensions, would enhance participation and benefits associated with the country's social safety net. Outputs Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) The project supported implementation of the CCT program. 355 Ministry of Labor and Social Protection (MLSP) and SWC staff were trained, far exceeding the target of 30. The CCT Implementation Unit was staffed, and one staff member in each SWC was assigned to the CCT program, meeting the target. A public awareness campaign was developed and implemented. Beneficiary awareness was reported to have increased from 58.37% in 2011 to 75.9% in 2016, not quite reaching the target of 80% (ICR, p. 37), though the ICR did not specify how this indicator was measured. A Subsidized Employment Program (SEP) was implemented as the main "CCT extension," providing incentives (wage subsidies and tax incentives) for companies to hire persons from socially vulnerable categories, such as young SFA beneficiaries, youth without parental assistance, and ex-beneficiaries of the secondary education CCT program (the ICR did not explain how these incentives were monitored or controlled). This program reached 315 young underprivileged persons. An impact evaluation showed a "notable increase" in employment of participants during the program period compared to a non-participant control group, with employment levels remaining relatively high even after the SEP was completed (ICR, p. 18). The ICR (p. 28) also noted challenges with employability of these beneficiaries, however, ranging from psychological (fear of losing benefits if employed) to practical (lack of skills). Outcomes There were 20,173 project beneficiaries, exceeding the original target of 18,000 and the revised target of 19,500. Of these, 48.1% were female, exceeding the original target of 45% and the revised target of 47%. The percentage of SFA beneficiary children ages 15-18 who were registered with CCT reached 85.91% by project closure, exceeding the original target of 40% and the revised target of 65%. The school attendance rate of children ages 15-18 in CCT beneficiary households increased from 85% in 2015 to 93.9% in 2018, exceeding the original/revised target of holding steady at 85%. This indicator replaced the original indicator, the school enrollment rate of children ages 15-18 in CCT beneficiary households, which increased from 54.9% in 2010 to 75.9% in 2018, exceeding the original target of 43% and the revised target of 60%. An impact evaluation found an increase of 10 percentage points in secondary school enrollment attributable to the CCT, with the effect explained by lower dropout rates among older youth (ICR, p. 18). Rating Substantial OBJECTIVE 2 Objective Strengthen the efficiency of the Borrower’s social safety net Rationale The PAD did not define "efficiency," but the ICR reasonably inferred from the indicators and activities assigned to the objective that efficiency was intended to mean improved targeting and strengthened administration of safety net programs. The theory of change for this objective held that reviewing and Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) strengthening business processes in the MLSP and SWCs, development of software and implementation of the new cash benefit information system, identification of options for decentralization and local planning of social services, development of standards for delivery of those services, and establishment of the MLSP Policy Unit would lead to improved registry, eligibility, and targeting systems, as well as enhanced oversight and control for CCTs and other SFA benefits. Outputs The CBMIS was introduced and linked with administrative registries in the Employment Agency, Cadaster Agency, and Pension Fund. Assessments of targeting, eligibility, and registry systems for cash benefit programs was carried out. Oversight and control mechanisms were developed and applied to monitor implementation of CCTs and other SFA benefits. As a result, registration procedures have been simplified, administrative costs have been reduced for benefit claimants, and errors have been reduced. By project closure, all cash benefits programs had been integrated from previously fragmented and only partially automated systems to administration through the MIS, exceeding the target of 90%. The processing time for SFA applications was reduced from 69 days in 2010 to 11 days in 2018, far exceeding the target of a reduction to 30 days. All education conditionalities (secondary school enrollment and attendance) are now monitored across all quarters of the school year in the CBMIS, exceeding the target of 90%. However, according to the ICR (p. 34), the percentage of transfer payments that were on time remained stagnant, at 50% in 2012 and 50.1% in 2018, not achieving the target of 80%; the ICR did not explain this seemingly inconsistent result. The project team later explained that some schools did not enter data into the proper CBMIS modules in a timely manner, necessitating late/make-up payments to those whose information was not provided by established deadlines, but that these delays were minor, as all payments were consistently made by the end of each school year. Outcomes Overall safety net targeting accuracy (percentage of safety net transfers going to the poorest quintile) increased from 60.8% in 2014 to 77.7% in 2018, exceeding the original target of 50% and the revised target of 65%. This result places North Macedonia among the best performers in the Europe and Central Asia Region. The ICR further stated (p. 16) that leakage incidence to the richest quintile is low (1.4%) in all of the country's social assistance programs except disability benefits, but no further information was provided. Introduction of the CBMIS has reduced room for abuse of the system, resulting in a decline in the number of social assistance recipients and thereby savings of about Euro 290,000 monthly, or about Euro 6 million since the system was established (ICR, p. 17). Rating Substantial OVERALL EFF TBL OLD Rationale Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) Targets for key outcome indicators, which were reasonable and adequate measures of achievement of both objectives, were reached or exceeded. The project's interventions were logically and plausibly connected to observed outcomes, which were highly unlikely to have been achieved in the project's absence. Overall Efficacy Rating Substantial 5. Efficiency The PAD conducted a cost-benefit analysis using CCT program costs (with an estimated 10% for administrative costs), finding that CCT was expected to increase secondary school enrollment by 10 percentage points per beneficiary; that each additional year in secondary school would produce a roughly 4% increase in consumption; and that benefit-cost ratios would be 0.7, 1.1, and 1.83, based on alternative growth scenarios of 3%, 5%, and 7% respectively. The benefits therefore outweighed the costs in all but the lowest-growth scenarios. The ICR (Annex 4) updated these calculations, observing that CCT transfers corresponded to only a small share of average household expenditures (5-6%) and consumption (3-4%). Consequently, it found that the CCT was unlikely to have had a significant effect on household expenditure or on poverty. This unexpected result arises from households having reported levels of monthly expenditure 5-8 times larger than household income, suggesting significant underreporting of income. The project faced implementation challenges that had significant impact on efficiency. After two years of slow initial implementation, the project was extended for a total of 4.5 years, in essence doubling its life span. Early delays had to do with the predecessor SPIL project having not finalized the CBMIS, which was originally a disbursement condition for this project; although the Bank and the government reacted promptly to this limitation by amending the legal agreement, the adjustment nevertheless took time. As implementation progressed, the disbursement rate was constantly low due to a lower-than-expected number of CCT secondary education beneficiaries. Program coverage was high, but the number of SFA beneficiaries in general had decreased by half (from about 50,000 to 25,000) because of improved targeting of the benefit administration system (after the introduction of the CBMIS) and reduced inclusion errors. Finally, a volatile political environment during the project's last few years, resulting in frequent changes in ministers (five in total), caused support for the project to vary; early implementation was adversely affected by insufficient allocations in the state budget for the project (ICR, p. 24), a situation that was later corrected through amendments to the project's legal agreement that stabilized the budget planning process. Efficiency Rating Modest Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives were highly relevant to country conditions, government strategy, and Bank strategy at appraisal and at closing. The objectives to improve effectiveness and efficiency of the social safety net were both substantially achieved. Efficiency, however, was modest, given multiple project delays and extensions as well as a volatile political environment. These ratings indicate moderate shortcomings in the project's preparation and implementation, resulting in an Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Despite political volatility, the Bank's dialogue with all political structures in the social protection and inclusion areas has remained strong. A follow-on operation, the Social Services Improvement Project, was approved in September 2018 (US$ 33.4 million, 2018-2024) to build on the achievements of the CCT to introduce second-generation policy reforms in social and child protection. A new Law on Social Protection has been prepared and was awaiting parliamentary approval at the time of the ICR; the project team later confirmed that this law has been passed. The CBMIS remains fully operational, allowing for regular collection and analysis of administrative data related to all cash transfers and the socioeconomic situation of beneficiary households. However, poverty and equity remain significant concerns. An increasing share of social assistance resources is being directed toward categorical (non-means-tested) programs, though spending on means-tested programs has not declined, as overall family and child protection spending is increasing (ICR, p. 22). Additionally, the political environment remains complex, raising the possibility of sudden changes in course with regard to social protection policy. Persistent technical capacity constraints may slow implementation of further reform steps (ICR, p. 28). 8. Assessment of Bank Performance Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) a. Quality-at-Entry The project's objectives and planned activities were realistic and well specified, based on existing capacity, the policy environment, and country context. Neither the PAD nor the ICR reported on other external partners involved in the sector. The project built on lessons learned from previous operations in the country, including the SPIL project, as well as other Bank-financed CCTs: the essential role of government commitment and appropriate project preparation, the benefits of mainstreaming CCT within the ministry responsible for other cash benefit programs, the importance of oversight and controls mechanisms for reducing fraud and other errors, and the need for a strong monitoring and evaluation system for continuously improving program operations and for program sustainability (PAD, p. 11). Risk assessment (PAD, pp. 15-16) was thorough and candid. Financial management risk (potential for fraud, errors, and leakages of social cash benefits to the non-poor) was rated as high, and both sector-level risks (fragmentation, changes in government, political pressures for "quick-fix" reforms) and technical design risks (inappropriate criteria, inadequate targeting or compliance monitoring) were rated as substantial. Mitigation measures, including public awareness campaigns, a strong project results framework and monitoring, and attention to legal frameworks, were detailed and appropriate. M&E design was sound (see Section 9a), as were arrangements for fiduciary compliance (see Section 10b). Implementation arrangements were well established, though the CBMIS -- a prerequisite for this project that was to have been completed under the previous SPIL program -- was not ready on time due to software contracting challenges, producing significant delays due to the project's disbursement condition that withdrawals could not be made for payment of the CCT grants without an operational CBMIS. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision During a period of sustained political instability in the country (through which the Bank in general experienced significant delays in its portfolio), the project team remained engaged and kept the CCT "relatively stable," thereby moving project activities forward, albeit with some delay (ICR, p. 24). The team was responsive to everyday implementation challenges, especially during the initial period of delayed implementation (ICR, p. 27), responding with solutions such as introducing amendments to the Loan Agreement and modifying disbursement conditions to remove impediments to implementation. Appropriate adjustments were made to outcome and intermediate outcome indicators and targets. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD (p. 14) stated that the project was to support the government in implementing an M&E system for the project and the CCT program, with a focus on results. The results framework was well detailed (PAD, pp. 40-45), with appropriate and measurable outcome and intermediate outcome indicators clearly reflecting the project's implicit theory of change. Many baseline values were not set in the PAD, but were to be established in the project's first year. Monitoring was to draw from administrative data from the management information system developed under the SPIL project (CBMIS), annual Household Budget Surveys, CCT impact evaluations supported by the project, and operations and performance audits. The main features of the impact evaluation design were outlined in the PAD. The Policy Unit in the MLSP was to be the focal point for data collection and analysis. b. M&E Implementation The CBMIS, once operational, collected "a wealth of easily accessible and important" administrative data (ICR, p. 25). Data were systematically collected and analyzed by the government and the Bank. Two impact evaluations were conducted on the CCT program (on its overall impact, and on the impact of different payment and recipient modalities on education outcomes and household consumption), and two on the SEP. c. M&E Utilization According to the ICR (p. 25), project data were regularly used for decision making, adjustment (restructuring), and planning purposes. Indicators were redefined and adjusted as needed, based on M&E data and analysis. Based on analysis of the SEP pilot, the project was restructured to continue and expand it beyond the pilot phase. Key pieces of national legislation were informed by findings from project-financed analytical reports (ICR, p. 28). M&E Quality Rating High 10. Other Issues a. Safeguards The project was rated Environmental Assessment Category C. No safeguard policies were triggered. At the second restructuring, when some minor refurbishment of offices was added to the second component, the environmental category was reviewed and reconfirmed as Category C. Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) b. Fiduciary Compliance Financial management reviews and audits found arrangements for planning and budgeting, accounting, internal controls, flow of funds, financial reporting, and external audit to be adequate. The CCT program has relatively complex arrangements for channeling funds to students and employers, but eligibility and procedures were well detailed in operational manuals. No significant problems were reported during the project's lifetime, and financial management was "steadily" rated as Satisfactory (ICR, pp. 26, 27). Performance, operational, and financial audits were delivered on time. Procurement was consistently rated Satisfactory, and was carried out in accordance with applicable Bank guidelines. No major issues were reported with either procurement or contract implementation (ICR, p. 27). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory The CBMIS -- a prerequisite for this project that was to have been completed under the previous SPIL program -- was not ready on time, producing Bank Performance Satisfactory Moderately Satisfactory significant delays due to the project's disbursement condition that withdrawals could not be made for payment of the CCT grants without an operational CBMIS. The M&E system had no notable shortcomings and was more than sufficient to assess Quality of M&E Substantial High achievement of objectives. Its findings were disseminated and effectively used for decision Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) making, adjustment, and planning purposes. Quality of ICR --- Substantial 12. Lessons The ICR (pp. 28-30) presented several insightful lessons, including the following (with some adaptation of language): Management information system investments must be fully in place prior to beginning implementation of programs that rely on them. In this case, the cash benefit MIS was not complete at project start, delaying launch of the entire conditional cash transfer program. Government budgetary commitments cannot be considered fully certain in a volatile political environment. In this case, annual allocations from the state budget were not always in line with project financing agreements and needs, requiring later adaptation of project financing arrangements. Careful assessment of the budget process during project appraisal, as well as flexibility during implementation, is required to adapt to changing circumstances outside the project's control. The poorest and most vulnerable social program beneficiaries are frequently difficult to employ. They may be concerned that they will lose benefits if they work, and they may lack skills attractive to local labor markets. In this case, an individual approach, considering the unique situation and challenges of each beneficiary family, was effective. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, concise, and evidence-based. It followed established guidelines. Its analysis was well aligned to the project's objectives. The Borrower's ICR presented some information and analysis -- on project financing, dates, CCT and SEP beneficiaries, outputs related to development and structure of the CBMIS, and findings from the impact evaluations (especially on payment modalities) -- more precisely and thoroughly than the main ICR. The main ICR's lessons, which were comprehensive and insightful, drew heavily from the Borrower's ICR. a. Quality of ICR Rating Substantial Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CCT (P103974) Page 13 of 13