Document of The World Bank Report No: ICR2880 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-78480 IDA-39210 TF-51331 TF-97400) ON A CREDIT IN THE AMOUNT OF SDR3.6 MILLION (US$5.15 MILLION EQUIVALENT) AND A LOAN FOR ADDITIONAL FINANCING IN THE AMOUNT OF US$5.0 MILLION TO THE REPUBLIC OF ARMENIA FOR A SOCIAL PROTECTION ADMINISTRATION PROJECT November 27, 2013 Human Development Sector Unit South Caucasus Country Department Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective 31st May 2013) Currency Unit = Dram (AMD) 1000.00 = US$ 2.39 US$ 1.00 = AMD 417 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS AF Additional Financing CAS Country Assistance Strategy CNIF Corporate Network for Information Flows CPOVRD Center for Professional Orientation and Vocational Rehabilitation of Disabled CPS Country Partnership Strategy DCA Development Credit Agreement DPO Development Policy Operation FBP Family Benefit Program FFPMC Foreign Financing Projects Management Center of the Ministry of Finance and Economy FM Financial Management FSU Former Soviet Union GOA Government of Armenia GDP Gross Domestic Product ICT Information Communications Technology IDA International Development Association ILO International Labor Organization ISPC Integrated Social Protection Centers (name for ISSC’s in SPAP2) ISSC Integrated Social Services Center (SPAP and SPAP AF) IT Information Technology M&E Monitoring and Evaluation MOFE Ministry of Finance and Economy MIS Management Information System MLSI Ministry of Labor and Social Issues MTEF Medium Term Expenditure Framework MTR Mid-Term Review NGO Non-Governmental Organization PAD Project Appraisal Document, 2004 PCU Project Coordination Unit (in MLSI) PIU Project Implementation Unit (FFPMC, in MOFE) PDO Project Development Objective POMC Professional Orientation Methodological Center PP Project Paper (Additional Financing), 2010 PRSP Poverty Reduction Strategy Paper RA Republic of Armenia SAC Structural Adjustment Credit SDR Special Drawing Rights SES State Employment Service ii SESA State Employment Service Agency SIL Specific Investment Loan SPAP Social Protection Administration Project (the Project) SPAP2 Planned follow-on operation of SPAP SRC State Revenue Committee SSIF Social Security Insurance Fund SSSS State Social Security Service TOR Terms of Reference UNICEF United Nations Children’s Fund USAID U. S. Agency for International Development US$ m Million US Dollar YPOC Youth Professional Orientation Center Vice President: Laura Tuck Country Director: Henri Kerali Sector Manager: Andrew Mason Project Team Leader: Ivan Drabek ICR Team Leader: Ulrich Hoerning and Matteo Morgandi iii REPUBLIC OF ARMENIA Social Protection Administration Project Contents A. Basic Information ................................................................................................................... v B. Key Dates ............................................................................................................................... v C. Ratings Summary ................................................................................................................... v D. Sector and Theme Codes ....................................................................................................... vi E. Bank Staff .............................................................................................................................. vi F. Results Framework Analysis ................................................................................................ vii G. Ratings of Project Performance in ISRs.............................................................................. xiv H. Restructuring (if any) .......................................................................................................... xiv I. Disbursement Profile ........................................................................................................... xiv 1. Project Context, Development Objectives and Design ........................................................... 1 2. Key Factors Affecting Implementation and Outcomes ........................................................... 8 3. Assessment of Outcomes ...................................................................................................... 14 4. Assessment of Risk to Development Outcome ..................................................................... 26 5. Assessment of Bank and Borrower Performance.................................................................. 26 6. Lessons Learned.................................................................................................................... 29 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ....................... 30 Annex 1. Project Costs and Financing ...................................................................................... 32 Annex 2. Outputs by Component.............................................................................................. 34 Annex 3. Economic and Financial Analysis ............................................................................. 44 Annex 4. Bank Lending and Implementation Support/Supervision Processes ......................... 45 Annex 5. Beneficiary Survey Results ....................................................................................... 46 Annex 6. Stakeholder Workshop Report and Results ............................................................... 47 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ................................. 48 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ................................... 63 Annex 9. List of Supporting Documents and Interviews .......................................................... 64 Annex 10. Results Framework for the SPAP and AF ............................................................... 65 MAP IBRD 33364 iv A. Basic Information Social Protection Country: Armenia Project Name: Administration Project IBRD-78480,IDA-39210,TF- Project ID: P087620 L/C/TF Number(s): 51331,TF-97400 ICR Date: 08/01/2013 ICR Type: Core ICR Lending Instrument: SIL Borrower: REPUBLIC OF ARMENIA Original Total USD 10.25M USD 5.15M Disbursed Amount: Commitment: Revised Amount: USD 10.15M Environmental Category: C Implementing Agencies: Republic of Armenia / Ministry of Labor and Social Issues (MLSI) and Ministry of Finance - Foreign Financial Projects Management Center (FFPMC) Co-financiers and Other External Partners: The borrower co-financed the project with USD 1.25 M. No external co-financiers in this project. USAID and UNICEF as important external partners B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 01/15/2004 Effectiveness: 12/28/2004 12/28/2004 Appraisal: 04/15/2004 Restructuring (AF): 02/25/2010 Approval: 06/10/2004 Mid-term Review: 06/27/2012 12/28/2012 Closing: 12/31/2007 05/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Negligible Bank Performance: Moderately Satisfactory Borrower Performance: Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Satisfactory Overall Bank Overall Borrower Moderately Satisfactory Satisfactory Performance: Performance: v C.3 Quality at Entry and Implementation Performance Indicators Implementation Performance Indicators QAG Assessments (if any) Rating Potential Problem Project at No Quality at Entry (QEA): Satisfactory any time (Yes/No): Problem Project at any time Yes Quality of Supervision (QSA): None (Yes/No): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Compulsory pension and unemployment insurance 30 30 Other social services 70 70 Theme Code (as % of total Bank financing) Improving labor markets 40 40 Other public sector governance 20 20 Social risk mitigation 40 40 E. Bank Staff At Approval Positions At ICR (Phase 1 / Phase 2 (AF)) Vice President: Laura Tuck Shigeo Katsu / Philippe Le Houerou Country Director: Henry Kerali Donna Dowsett-Coirolo / Asad Alam Hermann A. von Gersdorff / Sector Manager: Andrew Mason Kathy A. Lindert Project Team Leader: Ivan Drabek Elena Zotova / Lire Ersado Ulrich Hoerning and ICR Team Leader: n/a Matteo Morgandi ICR Primary Author: Suzana Abbott n/a vi F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document, Page1 The development objective is improved performance (efficiency, effectiveness, quality) by the labor and public employment, pension and social assistance agencies in providing services to the population. 1 Revised Project Development Objectives (as approved by original approving authority) 2 The development objective is to improve performance (efficiency, effectiveness, quality) by public employment, pension and social assistance agencies in providing services to the population. 1 Page 5 of the PAD gives a slightly different version of the PDO: “The development objective is improved performance (efficiency, effectiveness, quality) by the public employment, pension and social assistance agencies in providing services to the population.” In the IDA DCA, yet another wording appears (p. 13): “the objectives of the Project are to improve the Borrower's capacity to provide a more efficient and sustainable system of labor and public employment, pension and social assistance to enhance social protection to poor and vulnerable population groups” 2 AF PP (p. 4) and AF IBRD Loan Agreement (p. 4) provide a consistent wording of the PDO vii (a) PDO Indicator(s) Original Target Actual Value Values (from Formally Revised Achieved at Indicator Baseline Value approval Target Values Completion or documents) 3 Target Years The share of eligible pensioners 50.4% n/a 100% 100% that receive their pensions timely 4 (12/2004) The share of eligible pensioners 50.4% n/a 100% 98.2% that receive their pensions fully (12/2004) Proportion of eligible social 85.5% n/a 100% 93.2% assistance beneficiaries who (12/2004) receive their benefit timely Proportion of eligible social 50.4% n/a 85% 95.3% assistance beneficiaries who (12/2004) receive their benefit fully Overall Placement Rate for 7% 15-20% n/a 16% Jobseekers 5 (2005) (2012) The share of disabled placed in a 5.5% n/a 25% 16.2% job (% of total registered in (12/2005) Gyumri SESA regional office) No. of youth who received 7,314 n/a 12,000 16,098 consultation in Yerevan SESA (12/2005) offices The share of youth placed in a job 15% n/a n/a 23% (% of total registered job seekers (2007-2008) (2012-2013) in the Yerevan SESA offices) 6 No. of youth who received 1,549 n/a 4,000 3,047 consultation in Youth Professional (12/2008) Orientation Center Pensioners satisfied with the 18.9% 83% 95.8% pension system (% of total) (12/2005) Social assistance beneficiaries 21.8% 86% 97.4% satisfied with social benefit (12/2005) system (% of total) Share of job seekers satisfied with 76.9% 90% 81.7% activities of State Employment (12/2005) Agency (% of total) 3 No target values were set at the time of the initial project (Phase 1) for the PDO level indicators, target values were only set in the AF Project Paper 4 The first PDO indicator “evidence of eligible beneficiaries receiving accurate and timely pensions and social assistance” (Supplementary Letter Two to the Development Credit Agreement) was composite. For clarity it has been broken down into four indicators, since each result was tracked separately. 5 This outcome indicator has not been set as part of the PDO or Component level indicators in the PAD or the AF PP. It was given as an example (in fact the only example) in the “Economic and Financial Analysis” of the PAD (Annex 9, p. 44) for the type of measurable improvements the project would achieve. It is for this reason, that the indicator is included on this level. 6 This indicator was not included in the Project’s Results Framework, but has been included to show improvements in job placement for youth. viii Original Target Actual Value Values (from Formally Revised Achieved at Indicator Baseline Value approval Target Values Completion or documents) 3 Target Years Improved contributions collection 71.6% 86% 77.8% in the pensions system, as a ration (12/2006) (06/2012) of covered wage bill over gross wage bill (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years A national social information system is in place, which allows for coordinated targeting Indicator 1-1: and delivery of social services Value 100% (quantitative None None Cancelled (AF PP) or qualitative) Date set / 2005 2010 2013 achieved Cancelled, but achieved outside of SPAP. Comments (incl. % Plans for a social protection sector information system (CNIF - Corporate Network of achievement) Information Flows) were abandoned during the project. Nevertheless, all local offices of MLSI and its agencies are connected to Intranet / Internet. Indicator 1-2: Appropriate information is transmitted into the system Value 100% (quantitative None None Partially Completed (AF PP) or qualitative) Date set / 2005 2010 2013 achieved Comments Partially Completed. Information exchange is assured in current system via central (incl. % structure, but development of a unified “Social Data Bank” still outstanding. achievement) Indicator 1-3: Agency programs are adapted to national information Value 100% (quantitative None None Partially Completed (AF PP) or qualitative) Date set / 2005 2010 2013 achieved Comments Partially Completed. Information exchange is assured in current system via central (incl. % structure, but development of a unified “Social Data Bank” still outstanding. achievement) Indicator 1-4: Introduction of a joint package of social services and standards Value 100% (quantitative -- / -- None Completed (AF PP) or qualitative) Date set / 2005 2010 2013 ix Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years achieved Completed. ISSC functional model defined for pilot. “Social worker” qualification specified. ISSC model to be decided on in SPAP2. Comments (incl. % Indicator introduced in the AF PP. Last reported as “Development of structural-functional achievement) model of joint system of social services integrated provision (SSICJS) and joint package of social services standards” (ISR 19, May 2013) Indicator 2-1: Information on labor supply and demand available on continuous basis Modernization of system Value completed, 120 Completed via (quantitative None None specialists “Gorts” system or qualitative) trained (AF PP) Date set / 2005 2010 2013 achieved Completed. In AF PP, wording of indicator changed to include: “Active labor market Comments services selectively offered, and based on the labor market information systems and results (incl. % monitoring; including the targeting of specific measures at the disabled and young achievement) workers”. Indicator 2-2: ALMPs selectively offered and results monitored Value (quantitative None None --/-- Partially achieved or qualitative) Date set / 2005 2010 2013 achieved Partially achieved. Limited ALMPs are offered to jobseekers, but no comprehensive Comments results-monitoring. (incl. % achievement) Indicator dropped. In AF PP, indicator merged with above indicator Indicator 2-3: Systematic record-keeping on system participants and flow of funds Value (quantitative None None --/-- Partially achieved or qualitative) Date set / 2005 2010 2013 achieved Partially achieved. SESA Gorts IT system allows monitoring of registered jobseekers and Comments most benefit and ALMP payment flows (incl. % achievement) Indicator dropped. In AF PP, indicator merged with above indicator Indicator 2-4: Increase participation of youth and rural jobseekers in state employment service Value (quantitative --/-- None 35% Not tracked or qualitative) Date set / 2005 2010 2013 achieved Comments Indicator introduced in the AF PP, only tracked for youths in Yerevan (incl. % x Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years achievement) Indicator 3-1: Improved collections in the pension system Value (quantitative None None 100% 86% or qualitative) Date set / 2005 2010 May 2013 achieved Nearly achieved. Comments (incl. % In AF PP, wording of indicator changed to include: “and comprehensive record-keeping of achievement) system participants (central register) is established and fully operational” Indicator 3-2: Central register operational Value (quantitative None None 100% Completed or qualitative) Date set / 2005 2010 2013 achieved Comments Completed (incl. % achievement) In AF PP, indicator merged with above indicator. Indicator 3-3: Business processes and institutional procedures are operational Value (quantitative None None 100% Completed or qualitative) Date set / 2005 2010 2013 achieved Completed Comments (incl. % In AF PP, wording of indicator changed to “Business processes and institutional achievement) procedures are functioning as intended” Indicator 3-4: Participants are aware of their rights and responsibilities 26.6% Rights Value 26.4% Responsibilities 34% Rights (quantitative None n/a (AF PP) 36% Responsibilities or qualitative) Date set / 2010 2010 May 2013 achieved Partially completed Comments (incl. % In AF PP, wording of indicator changed to “System participants are aware of their rights achievement) and responsibilities” Indicator 3-5: Actuarial forecasting applied in knowledge-based planning Value (quantitative None None n/a not tracked or qualitative) Date set / 2005 2010 2013 achieved xi Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator not tracked. But evidence-based capability of pension administration enhanced Comments via parallel WB SPAP and USAID projects. (incl. % achievement) In AF PP, wording of indicator changed to ”Policy analysis is used for policy formulation and evaluation functions, and better management of resources” Development and installation of software for introduction for personified Reporting and Indicator 3-6: Record-Keeping and its smooth operation at the State Revenue Committee Value Completed (quantitative --/-- None 100% (not in SPAP though) or qualitative) Date set / 2005 2010 2013 achieved Comments PRK operational, but not implemented under WB SPAP. (incl. % achievement) Indicator introduced in the AF PP. Indicator 3-7: Backlog of unprocessed personified data eliminated Value (quantitative --/-- None 0% 0% or qualitative) Date set / 2005 2010 2013 achieved Fully achieved. Comments (incl. % MLSI approved work of contractor in March 2013 (Final M&E Report). achievement) Indicator introduced in AF PP. Indicator 4-1: Targeting mechanism applied by all social welfare offices Value (quantitative None Yes Yes Yes or qualitative) Date set / 2005 2010 2013 achieved Achieved. A uniform targeting mechanism for FBP is applied nation-wide. Comments (incl. % In AF PP, wording of indicator changed to “New approaches to targeting are being applied achievement) by all social welfare offices” Indicator 4-2: MIS used to determine eligibility Value (quantitative None None Yes Yes or qualitative) Date set / 2005 2010 2013 achieved Achieved. Eligibility determination in FBP is MIS-based. Comments (incl. % In AF PP, wording of indicator changed to “The management information system is being achievement) used to determine eligibility” Indicator 4-3: Monitoring and evaluation results used in knowledge-based policy formulation xii Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Value (quantitative None Yes --/-- Not tracked or qualitative) Date set / 2005 2010 2013 achieved Comments Indicator not tracked. FBP policy formulation based on household survey results and other (incl. % empirical evidence. In AF PP, wording of indicator changed to “M&E results are being achievement) used in policy formulation” Indicator 4-4: Participants are aware of rights and responsibilities Value 21.5% Rights 46.9% Rights (quantitative 15.5% Responsibilities --/-- n/a 45.03% Respon- or qualitative) (2005) sibilities Date set / 2005 2010 May 2013 achieved Comments (incl. % Partly Achieved. Indicator dropped in AF PP but continued monitoring in M&E system. achievement) Improvement of Family Benefit Program coverage of the poor through enhanced Indicator 4-4: awareness campaign, particularly among the very poor Value (quantitative 50% 75% --/-- n/a or qualitative) Date set / 2010 2013 achieved Comments Indicator introduced in AF PP but not tracked through M&E system. No public awareness (incl. % campaign conducted. Only “Rights and Responsibilities” tracked as an indicator (see achievement) above) xiii G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 06/28/2004 Satisfactory Satisfactory 0.00 2 12/22/2004 Satisfactory Satisfactory 0.00 3 03/18/2005 Satisfactory Satisfactory 0.30 4 04/11/2006 Satisfactory Satisfactory 0.44 5 07/27/2006 Moderately Satisfactory Moderately Satisfactory 0.56 6 03/21/2007 Moderately Satisfactory Unsatisfactory 1.07 7 06/27/2007 Moderately Satisfactory Moderately Satisfactory 2.55 8 09/27/2007 Satisfactory Satisfactory 2.96 9 04/04/2008 Satisfactory Satisfactory 3.61 10 06/26/2008 Satisfactory Satisfactory 3.91 11 11/19/2008 Satisfactory Satisfactory 4.87 12 06/18/2009 Satisfactory Satisfactory 5.10 13 10/08/2009 Satisfactory Satisfactory 5.10 14 06/15/2010 Satisfactory Satisfactory 5.53 15 01/04/2011 Satisfactory Satisfactory 5.73 16 10/26/2011 Satisfactory Satisfactory 5.97 17 05/30/2012 Satisfactory Moderately Satisfactory 6.39 18 01/01/2013 Satisfactory Moderately Satisfactory 7.31 19 05/29/2013 Satisfactory Satisfactory 10.00 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Restructuring Disbursed at Reason for Restructuring & Key Approved PDO Date(s) Restructuring in Changes Made Change DO IP USD millions 02/25/2010 S S 5.43 Additional Financing I. Disbursement Profile xiv 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. At the time of Appraisal of the Social Protection Administration Project (SPAP, the Project) in 2004, Armenia had witnessed several years of fast and increasing macroeconomic growth. In 2003, the GDP growth rate reached 13.9 percent compared with 12.9 percent in 2002. Successful stabilization policies, since the mid-1990s, coupled with significant advances in structural policies aimed consistently at growth promotion, job creation and public management of resources had taken place. Nevertheless, Government’s reforms and resulting economic growth had not led to significant social achievements. About 48 percent of the population remained below the poverty line, and poverty remained a key development issue in Armenia. Economic risk remained a major feature of life, indicating the need for a strong formal social protection system to help households manage risk. 2. Although relatively well targeted, Armenia’s social protection system existing at the time of Appraisal was modest. Its main components were a labor pension and a last resort social assistance program (the Family Benefit Program, FBP). Benefits were low, but nonetheless an important source of family income, particularly for low-income families. Annual budget spending on the benefits averaged 4.8 percent of GDP. 3. The Government of Armenia (GOA, the Government) had prepared, together with civil society partners, and approved a Poverty Reduction Strategy Paper (PRSP) in 2003. The PRSP was a comprehensive program that recognized the priority of ensuring sustainable growth, increased employment, productivity and incomes as a means of promoting poverty reduction. At the same time, the PRSP highlighted the need for putting in place an efficient social protection system that could provide meaningful protection to the poor and most vulnerable groups. 4. Faced with severely constrained pension revenues and fiscal resources that limited the scope to increase benefit levels, the GOA was pursuing a strategy aimed at increasing the efficiency in the use of program resources that would then allow it to increase benefit levels and ensure that the most needy individuals and households were supported. Better results with poverty reduction were also constrained by weaknesses in social protection administration, including: (i) lack of coherence and coordination in the performance of tasks; (ii) lack of a comprehensive approach in business planning and control; (iii) administrative procedures and rules that were not systematically scrutinized, streamlined and rationalized; (iv) no clear distinction between management processes and operational processes; (v) operations offices were not managed by coherent and integral management concepts; ad hoc management was more regular than exceptional; (vi) modern principles of human resource management and development were not fully applied; (vii) lack of training of staff members in information communications and technology (ICT) applications; (viii) no clear unified approach for addressing the deficiencies in the operations of local offices; (xi) lack of management information systems; (x) cost accounting did not comply with International Accounting Standards (IAS); and (xi) budget preparation and implementation procedures were inefficient. 5. The GOA was planning to carry out a range of actions aimed at increasing efficiency, effectiveness and sustainability of its social protection programs. For social assistance, priorities included: (i) use of the family benefit program to reduce poverty among the most vulnerable people through substantial enhancement of targeting towards extremely poor households and by 1 ensuring that their consumption levels were not below the food poverty line; and (ii) providing high quality targeted services to categories of population in need of special protection, such as handicapped persons, orphans, refugees and displaced people in view of substantially meeting demands for such services. For social insurance, priorities included: (i) reducing the number poor pensioners and increasing pension rates to bring them above the general poverty line, rating pensions on the basis of pension investments and period of employment; and (ii) comprehensive reform of the mandatory social insurance system, including enhancing its efficiency, substantial review of principles of appointing pensions and ensuring financial stability and sustainability of the system. For employment policies, the priority was to replace the inefficient policies of unemployment insurance and transition to active employment policies. 6. The Rationale for World Bank involvement with the Project was strong. Modernization of social protection administration was an important priority for Armenia, and the World Bank was uniquely placed to support the GOA in its efforts with IDA resources. The implementation of Armenia’s social reform program (including the PRSP) required strong administrative capacities both at the central and local government levels. The World Bank had extensive international experience in supporting social protection reforms, coupled with a strong understanding of the social protection system in Armenia that was developed through an extensive dialogue with the authorities on poverty and social protection. The GOA counted on significant donor support for the implementation of social protection reforms, but that support often focused on individual institutions, without taking a systemic approach to social protection administration. The Project was designed to add value to then ongoing donor activities by supporting the development of the cross institution management systems and by addressing management information system development needs in key social protections institutions such as the Ministry of Labor and Social Issues (MLSI), the State Social Insurance Fund (SSIF), the State Employment Services Agency (SESA), as well as their regional bodies. The development of a nation-wide social protection information system was expected to act as a catalyst for further reforms aimed at improvements of social services delivery. The Project was expected to serve as an umbrella facility for other donor support. 7. The SPAP was fully aligned with the Government’s PRSP 7, and with the World Bank’s 2005-2008 Country Assistance Strategy for Armenia. 8 . Specifically, the Project’s objectives align with two of the PRSP’s five themes defined as priority areas of action, including: (i) enhancing human development and improving social safety nets; and (ii) improving core public sector functions. The PRSP recognized social insurance and social assistance as key elements of the poverty reduction framework. Targeted cash family poverty benefits—one of the then most successful poverty alleviation programs in the region—were a major pillar of Armenia’s safety net. The PRSP envisaged several measures aimed at further improving the administration of those benefits, including better monitoring and evaluation (M&E), more streamlined appeal procedures, and a more active role for territorial centers in providing social assistance that reached out to the poorest. For social insurance, the PRSP focused on completing the introduction of individual contribution records in the pension system, better collection of social insurance contributions and an increase in pensions. In the area of employment, the PRSP focused on the increase of employment as the most socially acceptable and economically efficient 7 Armenia PRSP, Document Number 27133-AR, October 20, 2003, Human Development / Social Protection System priorities outlined in para. 105, p. xxii 8 FY05-08 Country Assistance Strategy for the Republic of Armenia, Document Number 28991-AM, discussed by the World Bank’s Board of Directors in June 2004 2 means of poverty reduction, as well as on increased efficiency of employment programs and services. The World Bank’s CAS, based on the PRSP, highlighted the following objectives in the area of social protection: (i) reduce poverty primarily through private sector growth; (ii) reduce inequalities of consumption and access to services; and (iii) improve human capital by ensuring that Armenians have skills and health to take advantage of employment opportunities. The project aimed to address one of the “principal problems” addressed in the CAS (“improvement of the administration of social protection”) 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 8. The SPAP’s Development Objective was improved performance (efficiency, effectiveness, quality) by the labor and public employment, pension and social assistance agencies in providing services to the population.9 9. The Key Indicators identified to measure progress towards achieving project outcomes were the following: 10 • Evidence of eligible beneficiaries receiving accurate and timely pensions and social assistance payments; • Evidence of improved job placement, especially among the disabled and young workers; • Evidence of improved user satisfaction regarding service delivery related to pensions, employment services and social assistance 10. Output indicators for the main project components were also defined in the Project Appraisal Document (PAD). These are presented in Annex 10. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 11. The Additional Financing (AF) approved on February 25, 2010 (Section 1.7) did not modify the SPAP’s PDO fundamentally. However, the PDO defined in the Development Credit Agreement was revised under the AF, as well as in the Loan Agreement for the AF, to make it consistent with the wording in the original PAD approved by the World Bank’s Board: “The development objective is to improve performance (efficiency, effectiveness, quality) by public employment, pension and social assistance agencies in providing services to the population.” 9 This was the PDO as described in the PAD (p. 4). The Credit Agreement (p. 13) has a slightly revised wording: “The objectives of the Project are to improve the Borrower’s capacity to provide a more efficient and sustainable system of labor and public employment, pension and social assistance to enhance social protection to poor and vulnerable population groups. The wording in the Credit Agreement was revised under the AF to make both consistent (Section 1.3). 10 An additional indicator (“Improved collections in the pension system”) was included in one annex of the PAD documents (Annex 3 of the 2004 PAD), but neither the Development Credit Agreement, nor Section 2 or the Table (Results Framework and Monitoring) in Annex 3 of the Project Appraisal Document include this as a Key Outcome Indicator. This indicator is, however, included as an Intermediate Outcome under Component 3, which is its most appropriate location given the overall project context. 3 1.4 Main Beneficiaries, 12. Three groups of beneficiaries can be discerned for the project activities of SPAP. On the first level, the projects activities aimed to benefit the: (i) institutions of the social protection sector in Armenia. Across the components of the Project, the technical capacity equipment and physical infrastructure of MLSI, SESA and SSSS has been strengthened. In addition, the creation of new institutions has been supported, such as the Integrated Social Service Centers. Secondly, the project benefited the (ii) experts and front-line staff in the social protection agencies through professional development (e.g. social worker professional standards), study tours and capacity building. Lastly, the Project benefited the (iii) beneficiaries of the social protection programs, through improved and more accurate payment (a key outcome indicator of the results framework), better accessibility and reduced hassle-cost through integrated service centers (not measured in results framework) and better services in the respective specialized agencies. 1.5 Original Components (as approved) 11 13. The SPAP comprised five components, which would finance small works (refurbishment and minor repair) and provision of equipment, materials, technical assistance and training, as follows: Component 1: General Support to the Social Protection Sector (US$ 4.26m Total Project Cost, including contingencies and US$ 1.76m IDA Financing) 14. This component would help develop a comprehensive management information system (MIS) for the creation, acquisition, storage, maintenance and exchange of digital information required for the operation of an effective social protection system. The component included four sub-components: (i) development of a communications network for transmission of information between and within the different social protection programs and training specialists on the maintenance of the network; (ii) the establishment of a related technology platform; (iii) review of existing MLSI programs for adaptation to the new network structure; and (iv) upgrading the existing programs operated by MLSI offices and related agencies, integrating them into the national information network, and training staff on their use. Component 2: Improving Labor and Employment (US$ 1.59m Total Project Cost, including contingencies and US$ 1.24m IDA Financing) 15. This component would help to increase efficiency and effectiveness of the SESA in labor market intermediation and support labor reform in the country. The component included the following four sub-components: (i) development of a MIS that would form part of the national information network (Component 1); (ii) introduction of labor market monitoring and piloting a series of programs including new programs to improve the employment processes in selected local employment services; (iii) establishment of a training, advisory and job rehabilitation center for disabled unemployed; and (iv) establishment of a professional orientation center for youth on a pilot basis. It would also provide assistance to MLSI for the implementation of the Labor Code, once adopted. 11 All financial information as given in Annex 1 4 Component 3: Modernization of Pensions System Administration (US$ 2.46m Total Project Cost, including contingencies and US$ 1.20 IDA Financing) 16. This component aimed to create a modern pension administration. It included the following sub-components: (i) operational improvements in key business processes— contributions collections and revenue/expenditure monitoring, audit/inspection, pension management and customer services; and (ii) strengthening of the institutional capacity of SSIF, including general and financial management and actuarial forecasting. Improvements in these areas were expected to be underpinned by (iii) installation of a central register-related system based on a client data-base of individual records (a central register) containing pertinent information on all insurance system clients. Component 4: Improvement of Social Assistance (US$ 0.94m Total Project Cost, including contingencies and US$ 0.67m IDA Financing) 17. This component would help to improve poverty targeting by supporting community- focused household surveys and developing in-depth monitoring and evaluation of the effectiveness of poverty targeting. The Project would support improvement of poverty targeting mechanisms through: (i) conducting community surveys; (ii) developing capacity for in-depth monitoring and evaluation of the efficiency of targeting; (iii) upgrading equipment used to target family poverty benefits; and (iv) providing training to social assistance staff. Component 5: Project Management (US$ 0.82m Total Project Cost, including contingencies and US$ 0.50m IDA Financing) 18. This component would ensure professional implementation of the Project by using appropriate mechanisms of project management, and proper tools of project monitoring and evaluation. 19. A detailed description of the original Project’s components is provided in Annex 2. 1.6 Revised Components 20. The Project’s original components were not revised, although several activities were incorporated and/or eliminated under the components under the AF, and the sources of funding of some activities were modified (Section 1.7). Subsequently, during the MTR in June and July 2012, the MLSI informed the World Bank that it intended to cancel three activities under Component 2, namely: (i) refurbishment, equipping and training of staff of a regional office of “Youth Professional Orientation Center” (activities for the Yerevan office had been completed); (ii) further modernization of the information system of the SESA; and (iii) improvement of the information system of the State Labor Inspectorate. The first activity was cancelled due to institutional policy changes amid the new ISSC concept, in accordance with which this Center would be transformed into the methodical center for youth professional orientation (Section 3.2). The latter two were put on hold, and therefore eliminated from the Project, until the development of new strategies for the employment services and labor inspection had been completed. These and other changes did not lead to a restricting of the project because the overall PDO remained relevant. 5 1.7 Other significant changes 21. On October 23, 2007, Schedule 1 to the Credit Agreement was amended and the Credit’s original December 31, 2007 Closing Date was extended by 18 months to June 30, 2009. The amendment to Schedule 1merged categories of disbursements (except for Category 4 “Training and Study Tours”) into one disbursement category, and reallocated remaining funds into the new, single category. The amendment was expected to provide an opportunity for faster implementation of the remaining project activities, as well as to effectively utilize project savings gained because of US dollar depreciation and to partially cover the shortage of funds for civil works financing that arose because of the dramatic increase of prices for rehabilitation works and construction materials. The Closing Date extension would help to achieve fully the PDO, especially as it related to the establishment of the MIS for Armenia’s social protection sector, which was still in process of design completion and equipment procurement. Also, it would allow the World Bank to continue providing advice and technical assistance to the GOA on the future role and functions of the SSIF, for which discussions were ongoing at the time and a decision by the GOA was expected in late 2007. As such, the Closing Date extension was seen as important to the success of Armenia’s plans for pension reform. 22. On December 16, 2008, the Closing Date was again extended, this time for six months, until December 31, 2009, to provide continuing support to the MIS for social protection and to the MLSI, also for developing and improving its information system. 23. A third, five-month Closing Date extension of the Credit Agreement, until May 31, 2010, was approved on October 1, 2009. This extension was approved to ensure continuity and successful completion of then ongoing activities while at the same time allow sufficient time for the GOA and World Bank teams to prepare an Additional Financing (AF) that would scale up the Project. 24. A World Bank Loan for AF in the amount of US$5.0 million was approved on February 25, 2010. 12 The AF extended the May 31, 2010 Closing Date by three years to May 31, 2013. It aimed to help the GOA enhance the sustainability, impact and development effectiveness of the original, then ongoing, Project, and continue with institutional capacity building for efficient and effective social service delivery for the population. The AF was to scale-up activities under the Project in support of its original development objective. The GOA’s Second PRSP (also known as the Sustainable Development Program), approved in 2008, reaffirmed its commitment to poverty reduction, and the need for an efficient social protection system that would protect Armenia’s poor and vulnerable populations. The reforms under the Project had led to significant achievements, but gaps in improving the efficiency and effectiveness of the social protection administration remained. The global economic crisis had presented additional challenges to the GOA, especially in terms of mitigating the consequent economic and social impacts, and laying the foundations for an efficient and effective post-crisis social protection system assumed increased importance. The fiscal constraints resulting from the crisis added impetus for reforming the pension system, enhancing the efficacy of social assistance programs, integrating social services and strengthening systems for monitoring and evaluation. 25. Although the PDO remained the same, some of the approaches and institutions involved in implementing the Project under the AF were modified. Since approval of the SPAP, the GOA 12 Project Paper on a Proposed Additional Financing in the amount of US$5.0 million to the Republic of Armenia for the Social Protection Administration Project, Report No. 51936-AM, dated January 25, 2010. 6 had moved towards a unified revenue collection system for pension contributions and income taxes. The new State Revenue Committee (SRC) was charged with managing this unified revenue collection system. A more efficient, unified and user-friendly administration of revenue collection and reporting for personal income taxes, pensions and other social insurance contributions was expected to improve tax compliance and state revenue. The AF would support the SRC by improving its personified reporting and record-keeping system under Component 3. 26. With respect to implementation arrangements, at the beginning of 2008 the GOA transformed its SSIF into the State Social Security Services (SSSS) under the auspices of the MLSI. Although the main functions of the agency were essentially unchanged (i.e., effective implementation of the state social protection programs), social insurance contributions had become a revenue source of the central budget. Consequently, all social protection programs were now financed from the Government budget, a shift that was expected to result in greater consistency in social protection policy formulation and implementation. The changes in the governance structure of social protection were expected to help improve efficiency and quality of service delivery through a more efficient organization of work, strengthened information base and improved accountability and transparency in service delivery. The transformation of all local social protection offices into Integrated Social Services Centers (ISSCs) or “one-stop-shops”, supported by a corporate network of information flows for the social protection sector, was central to this new model. 27. The AF incorporated new activities intended to deepen the Project’s sustainability and development impact. Several outstanding activities that were considered essential for further improving and maintaining the efficiency and effectiveness of the social service delivery system were included under the Project, including those directed towards: (i) rehabilitating the remaining regional social service offices; (ii) ensuring the sustainability and widespread use of the network of information flows; (iii) introducing personified reporting and record-keeping for pensions, other social insurance and unified income taxes; (iv) addressing the institutional and reform bottlenecks for rural employment; (v) improving the coverage and administration of the Family Benefit Program; and (vi) integrating social service delivery. New output indicators were added to its Results Framework to monitor progress with these activities (Annex 10 highlights the new indicators). 28. Then, in late 2011, the GOA requested that several of the activities described above incorporated under Components 2, 3 and 4 of the Project be financed under the State Budget instead of Loan financing (Section 2.2). In a Level 2 restructuring, the corresponding funds were reallocated towards the integration of social services at the local level through ISSCs in the crosscutting Component 1. Although this reallocation was fully consistent with the PDO, the Project’s Results Framework was not restructured to eliminate output indicators that were no longer supported by activities financed. On April 6, 2012, the World Bank approved the requested reallocation of loan proceeds, reallocating US$545,000 from Category 2 (Goods, Consultants’ Services and Training for Component 3 of the Project) to Category 1 (Goods, Works, Consultants’ Services, Incremental Operating Costs and Training for Components 1, 2, 4 and 5 of the Project). 29. On June 24, 2013, the amount of SDR 2,509.59 remaining undisbursed in the Credit account was cancelled effective May 31, 2013. 7 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 30. Project preparation was expeditious and responsive to the GOA’s request for assistance to support its social reform program (including the PRSP). A SIL was selected as the most appropriate instrument in support of institutional strengthening and related investments. Although this was the first social protection project supported by the World Bank in Armenia, the preparation team incorporated lessons from similar projects in other countries in its design. According to the PAD, stakeholders (the GOA through the MLSI, the SESA and the SSIF) were full partners in developing the Project, and beneficiary consultations were taken into account in its design, based on: (i) findings of then recent poverty assessments; (ii) periodic social assessments of local governments and communities carried out by the Armenia Social Investment Fund; (iii) rapid assessments of pensioners and families with children; and (iv) NGO and donor forums. 31. Preparation (of both the Project and the AF) identified four main risks: not effectively applying the MIS, shortages in counterpart funding, weak government commitment, and ineffective coordination between the Project Coordination Unit (PCU) in MLSI and the Foreign Financing Projects Management Center (FFPMC) of the Ministry of Finance and Economy (MOFE). 32. Preparation of the AF took into consideration experience with implementation of the Project, new macroeconomic realities that the GOA faced and several reforms that it had put in place since the Project’s approval that impacted implementation and other arrangements. In spite of this, and with the benefit of hindsight, several aspects of the Project’s design impacted its implementation. These included: complexity and lack of flexibility, and the formulation of its Results Framework. 33. The Project was undoubtedly complex and ambitious. With multiple agencies implementing separate and inter-related components, including development of information systems and improvements to monitoring and evaluation (which are notoriously complex), its initial three and one-half year planned implementation was overly ambitious. The use of the FFPMC to support project administration by these agencies was positive, but could not substitute for actual implementation of the Project by each of the responsible agencies in view of the fact that this was fundamentally an institution strengthening project. The fact that the Project was implemented during a period where the GOA was continuously implementing reforms to its system of social protection, that on occasion impacted its implementation arrangements directly, only compounded its complexity. Perhaps a greater propensity to adjust the results framework of the Project and its components might have mitigated the impact of this constantly evolving environment, which had implications for the Project’s Results Framework, both as originally approved and amended in AF’s Project Paper. 34. The definition of the Project’s Results Framework, both at Appraisal and during processing of the AF presented several issues. PDO indicators defined in the PAD and maintained for the AF measured user satisfaction and share of pension and social assistance beneficiaries receiving payments timely and fully, which were not dependent only on the components’ activities. Several baselines were not provided in the PAD--they were only provided after the first year of implementation. When the AF was approved, the PDOs as defined in the PAD, as measured by the Key Outcome Indicators, had largely been achieved; the percentage of satisfied pensioners and beneficiaries had jumped from around 20% to 80%, in part 8 due to an increase in the size of payment (not related to service delivery). Other PDO indicators originally introduced in the PAD and retained in the PP for the AF were problematic (Section 2.3). 2.2 Implementation 35. The Project’s implementation was impacted by several factors that in large part resulted in the processing of the AF and then the Level 2 restructuring described in Sections 1.6 and 1.7. These factors included the framing of the Project within a comprehensive program of World Bank assistance and the GOA’s continued commitment to reform of its social protection sector, the global financial and economic crisis, the resulting changes to the Project and its implementation arrangements. 36. The Project was the World Bank’s first in support of Armenia’s social protection sector, but it built upon an extensive dialogue on poverty and social protection that had begun in the preparation of the country’s PRSPs in the early 2000s, developed over the years, and was implemented in a program of World Bank assistance that emerged over time. On a broader level, through a series of four PRSCs starting in 2004 13 , the World Bank supported policy and institutional reforms for implementation of the GOA’s PRSP that included, inter alia, improvements in the coverage and targeting of the FBP and the development and implementation of foundations for a new multi-pillar pension system. Starting in 2009, a programmatic series of three Development Policy Operations (DPOs) supported the GOA’s reform program in addressing vulnerability by protecting the poor in face of the global crisis (see below), and strengthening the effectiveness and efficiency of social nets. 14 Specifically, the DPOs aimed to contribute to improved social outcomes in the face of the economic slowdown by maintaining social safety net spending and (further) improving the targeting and coverage of the FBP program. However, the Implementation Completion and Results Report for the DPO series reports that the outcome targets were not met although some progress was made. 15 Analytical work also supported the dialogue with the GOA on social protection. This included: poverty and social assessments, economic reports and programmatic public expenditure reviews that focused, inter alia, on the efficiency of social protection programs. Finally, the World Bank also assisted in mobilizing grant funding in specific areas that complement activities supported by the Project. These include a US$430,000 IDF Grant for Pension Public Awareness and Literacy in Armenia: Building MLSI Capacity, to create capacity (i.e., pension internet portal, pension telephone hotline, equipped pension reform trainers) to inform, educate, and familiarize broad public with pension reform process in Armenia, signed by World Bank on January 15, 2013, and a US$2.7 million JSDF Grant for Strengthening the Livelihoods and Voice of Vulnerable and Disabled Persons in Armenia, approved in early 2013. 13 Poverty Reduction Support Credit (P078673) approved on November 18, 2004; Second Poverty Reduction Support Credit (P093459) approved on January 19, 2006; Third Poverty Reduction Support Credit (P093460) approved on March 8, 2007, and Fourth Poverty Reduction Support Credit (P101486) approved on November 27, 2007. 14 Armenia First Development Policy Operation, (P115626), approved July 2, 2009; Armenia Second Development Policy Operation, (P116451), approved January 11, 2011; and Armenia Third Development Policy Operation, (P122195), approved February 14, 2012. 15 Implementation Completion and Results Report on Credits to the Republic of Armenia for Development Policy Operations I-III, Report No. ICR0002593, dated June 5, 2013. The indicator “Social protection and pensions as a share of total spending” was not met; the indicator “Coverage of the poor by the FB program” was not met, but progress made; and the indicator “Leakage of FB resources to the non-poor” also was not met, but some progress made. 9 37. Armenia was one of the countries hardest hit by the 2008-09 global economic crisis. Its economy contracted by over 14 percent in 2009, exposing weaknesses in the policy and institutional spheres, and disrupting the earlier momentum in poverty reduction that had resulted in a decline from 56 percent to 23.5 percent in poverty incidence between 1999 and 2008. For the first time since the late 1990s, poverty incidence increased in 2009 by an estimated three percentage points 16. The GOA responded aiming to mitigate the adverse impact of the crisis on the poor and vulnerable, with measures that included protection of public spending on social safety net programs, re-prioritization of government expenditures towards programs that protected or created jobs in the short term, and improvements in the targeting of the FBP, although as mentioned above, these were not entirely successful. 38. The crisis introduced greater urgency for defining and securing approval of reforms aimed at further strengthening its social protection programs, and increasing the efficiency of government expenditures for those programs. The reforms designed and approved since that time include: a comprehensive pension reform (including modified institutional framework for revenue collections, etc.), a new strategy and approaches for employment, and the adoption of a new model of integrated social service provision that envisages the functional integration of four existing agencies (responsible for pensions, social assistance, employment and disability certification) under one roof in all marzes (“provinces”) across the country. While not all of these reforms were directly supported by the Project, the fact that they often impacted the Project’s institutional framework, or the GOA’s priorities for external funding, on occasion led to implementation delays (pension reform) and the need for a modification of implementation arrangements (e.g., the inclusion of State Revenue Committee (SRC) in the AF) as described in section 1.7. 39. In part precipitated by the above, the Project’s implementation witnessed several revisions to the activities financed by the World Bank credit and loan. The AF already included several new or revised activities and implementation arrangements (Section 1.7). Later, less than one year after approval of the AF in October 2011, the Prime Minister initiated a review of all World Bank financed technical assistance activities with the aim of rationalizing the many donor- funded TA activities involving international consultants. TA activities that were to be financed under loans required specific approval from the Prime Minister’s office. Further, the timing of the GOA’s constantly evolving reform agenda precipitated the need to move ahead expeditiously with activities in support of some reforms (such as pension modernization under Component 3) with more agility than would have been possible following World Bank requirements, while it put on hold the implementation of activities under other Components (such as those focused on labor and employment under Component 2) as new strategies and approaches that were then under development, and that would impact the need for project activities, took longer than expected to materialize. In view of the above, a large portion of the TA activities under the social assistance, pensions and labor components was eliminated from the Project to be financed under the State Budget. At that stage, funds were reallocated towards integration of social services (encompassing social assistance, pensions and employment services) at the local level through expansion of ISSCs in a pilot phase under the cross cutting component. This activity was considered fully consistent with the PDO. However, the Project was not restructured to amend the Results Framework, and activities financed by the World Bank became more oriented towards provision of infrastructure, that involved a need for closer supervision through rolling action 16 World Bank, Armenia, The 2008-09 Global Economic Crisis, Policy Responses, and Household Coping Strategies, Report No. 55011-AM dated June 2010. 10 plans that were prepared and agreed to keep implementation on schedule during the Project’s final years. 40. Throughout, there was very strong coordination of donor assistance in response to GOA priorities and funding availability, not only with the GOA in the driver’s seat, but among the donors themselves as well. This emerges as a very strong point of the Project and goes well beyond what usually could be described as “donor coordination” but could be classified as an integrated partnership network of donors with GOA/MLSI at the center. This view was strongly endorsed by the Minister of Labor during the ICR interview. In no instance did the ICR team encounter any indication of turf-wars, duplication of activities etc. between the main partners World Bank, USAID and UNICEF. On the contrary, throughout the Project, the main protagonists in the named agencies continuously updated and aligned their work programs, decided on appropriate division of labor and executed accordingly in partnership with MLSI. 41. The review of TA activities in 2011 described above is an example of the leadership in coordination by the GOA. At the time, the GOA required a review of all loan-financed TA activities, leading to a partial halt of TA in some of the World Bank-financed projects. While this caused initial inconvenience for project execution, it emerged as a constructive checkpoint to avoid duplication of TA activities between line agencies. The collaboration of UNICEF and USAID on the evaluation of the ISSC pilots and the proposed continued financing of the PCU in MLSI by USAID (during the transition period until the proposed World Bank financing of SPAP2 is approved) are other examples of coordination among donors. Additional instances of donor coordination and financing of GOA social protection programs and the Project are described in Section 3.2. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 42. The Project Appraisal Document (PAD) specified that monitoring and evaluation of outcomes/results would be based on data generated by qualitative surveys to be carried out by MLSI, SSIF (later SSSS) and SESA on the impact of project interventions on beneficiary populations, and that staff of these agencies at headquarters and in local offices would be trained in the application of monitoring and evaluation techniques and the adaptation of policy and programs strategies to their findings. A consulting company on behalf of the Project Implementation Unit in MLSI produced comprehensive annual M&E Reports. This company also conducted the beneficiary assessments used to monitor progress towards the Project’s Key PDO Indicators. 43. Monitoring of the Project’s outcomes and outputs presented several challenges. The outcome indicators as originally defined were composite indicators (e.g., “eligible beneficiaries receiving accurate and timely pensions and social assistance payments” 17), which were then split and tracked separately in the AF. Baselines and targets for each of the Key Indicators (Section 1.2, Annex 10) were not specified in the PAD; baselines were introduced in the first year of implementation. Further, it is not clear that progress towards the outcomes, as defined by the indicators, would be attributable solely to activities supported by the Project. 17 PAD, Annex 3 11 44. The Project Paper (PP) for the Additional Financing maintained the outcome indicators defined in the PAD, but included additional output indicators under each of the components to reflect progress of activities directly attributable to the additional financing. 45. The revisions in activities financed under the Project throughout 2011 (Section 2.2) presented a further challenge to monitoring of the Project’s outcomes and outputs. With the removal of several activities from the Project, World Bank financing was reallocated towards the integration of social services at the local level under Component 1, an activity that was fully consistent with the SPAP’s PDO. However, the Project’s Results Framework was not amended to reflect the new reality. As a result, several Intermediate Outcome Indicators were no longer supported by the Project’s activities and were no longer monitored. Several new Intermediate Outcome Indicators began to be monitored, yet the World Bank’s team opted to not restructure the Project’s Results Framework in view of the limited time remaining until closing, since apparently the team felt that this would take time and resources away from the critical project implementation activities remaining to be completed. 46. The first outcome indicator states that it is tracking the share of "eligible" pension or social assistance beneficiaries receiving their benefits "timely and fully." Since the PDO aims to improve efficiency, effectiveness and quality of agencies in providing services, this was interpreted by the team and counterparts as an indicator to track improvements in service delivery to those pension or social assistance beneficiaries actually receiving benefits. However, the indicator could be interpreted as aspiring to capture both the improvement in the agencies' performance in delivering services for actual beneficiaries and the reduction in the gap between eligible and actual beneficiaries - i.e. improving coverage of the pension and social assistance programs. A few activities under the social assistance component could have contributed in a minor way to increasing coverage of the Family Benefit Program (reducing application documentation burden, increasing cross-checks to reduce inclusion errors, and communications campaigns), but the Government of Armenia has been clear that it does not aim to give Family Benefits to all poor (as defined by the World Bank) but rather only to the most vulnerable among them. Hence, both the team and counterparts were using the outcome indicators to track service delivery as a contribution towards the PDO and the project is being assessed using this interpretation in this ICR. Furthermore, tracking an outcome related to 'eligible' beneficiaries would have been nearly impossible, because the identification of the eligible who are not covered is constrained by data availability. In fact, the eligibility criteria for social assistance in Armenia is complex (see p.23 below), as it depends on multiple factors beyond income, which the national household survey does not monitor. Such an interpretation of the indicators was not intended under this project. 47. There were other issues with respect to the selection of indicators and targets, once these were established. Progress towards outcome indicators was tracked with a very small sample of beneficiary surveys (quantitative, with a sample size of about 400), while tracking of other indicators was based on official documents with no reference to beneficiaries. The values of several indicators jumped after the second year, and continued to climb, even though there had been few, if any, completed project activities to support continued increases (in user satisfaction). This may, in part, reflect factors exogenous to the Project that impacted improvements in some indicators. For example, the percentage of satisfied pensioners and beneficiaries jumped from around 20 percent in 2005 to 97 percent in 2011, most likely due to an increase in the size of the payment as opposed to a result of improved service delivery. 12 48. At the time the AF was approved, the Project’s expected outcomes had largely been met. While some target values were increased a bit, no new outcome indicators were added. As a result, it was not clear how, for example, user satisfaction with social services, already at almost 100 percent, could be increased by additional project activities. Other indicators presented similar issues, for example, the intermediate outcome “number of complaints that receive a positive resolution” that was already 100 percent at baseline and not supported by any activities financed under the Project. In other cases, the indicators served to monitor pure outputs (like number of people trained), in spite of the availability of a monitoring system capable of tracking outcomes (young people placed in jobs). 2.4 Safeguard and Fiduciary Compliance 49. Fiduciary (Procurement and Financial Management) functions were implemented by the Foreign Financing Projects Management Center (FFPMC) of the Ministry of Finance (MOF), in coordination with the Project Coordination Unit (PCU) in the MLSI. Created in 1993 and charged with similar functions for numerous projects funded by IDA Credits and a variety of trust funds, the FFPMC had solid experience in managing the fiduciary aspects of World Bank-financed projects in accordance with the World Bank’s procurement guidelines and financial management requirements. 50. The World Bank’s implementation support teams carried out procurement post reviews periodically and findings were generally positive. Recommendations were mostly of a procedural nature (e.g., use of Standard Bidding Documents for Non-Consulting Services for the procurement of the internet portal for the SSSS instead of the Standard Bidding Documents for the supply and installation of information systems, the fact that procurement issues on post review contracts need not be sent to the World Bank for review and no objection, etc.). Delays in procurement were mostly the result of broader delays in adopting key decisions (e.g., adoption of the ISSC model, transfer of ownership of buildings for ISSCs, etc.) or the result of changing priorities for World Bank financing that resulted in numerous adjustments to the Project’s procurement plans. 51. The Project’s financial management arrangements and requirements (including flow of funds, staffing, accounting, planning and budgeting, reporting and auditing) were also reviewed periodically through implementation support, and found to be highly satisfactory throughout. Sound internal control procedures were in place throughout implementation, and audit reports were received in a timely manner, with unqualified auditors’ opinions on the Project’s financial statements. Towards the end of implementation FFPMC shifted effectively to the World Bank’s E-disbursement system of disbursements. 52. Although the Project included minor civil works for the refurbishment and rehabilitation of buildings, no significant environmental impacts were expected; as a result, the Project and the AF was classified as an environmental Category C and none of the World Bank’s Safeguard Policies were triggered, owing to the mainly refurbishment nature of the work in existing buildings. 2.5 Post-completion Operation/Next Phase 53. The GOA remains fully committed to the Project’s PDOs and to sustain the reforms and institutional capacity that it supported, both with and without World Bank financing. The GOA has requested a follow-up project that would continue supporting the reforms initiated under the 13 SPAP. A follow-up project has been included in the World Bank’s CPS, and a Concept Review was held on September 12, 2013. The objectives of the proposed project would be to: (i) improve efficiency of social protection service delivery for social protection beneficiaries and (ii) strengthen analytical and monitoring and evaluation functions of the agencies delivering social protection benefits and services. Based on the experience under the SPAP, the follow-on project would roll out the model of integrated delivery of pensions, social assistance, employment and disability certification through functional integration within the Integrated Social Service Centers (ISSC), now called Integrated Social Protection Centers (ISPC) from 19 ISPCs piloted so far to an additional 37 ISPCs, effectively providing this service model across the country. The proposed project would also provide continuing support for: (i) implementation of the new employment strategy, including the building of nationwide career orientation service, strengthening SESA’s capacity, piloting of one or more active labor market policies, and evaluating the impact of new employment programs. It would also provide continuing assistance to SSSS in improving its core business processes. Specifically, this assistance would focus on strengthening SSSS’ technical capacity through completing its electronic document management system, intranet and internet services, building analytical, forecasting, and statistics capacity supporting the MLSI policymaking activities, introduction of the risk-based supervision of insured individuals’ statuses and providing information to the insured individuals on covered service periods on a 24/7 basis. Finally, the proposed project would support the strengthening of MLSI’s analytical capacity and monitoring and evaluation, and developing an integrated system of monitoring and evaluation of social protection benefits and services. 54. Further support to FBP, especially insofar as identifying possible reform options aimed at expanding the coverage of the poor and further improving targeting efficiency, would be provided under the World Bank’s planned Armenia Development Policy Operation (DPO) and the ongoing Social Inclusion and Labor Knowledge for Reforms, Operations and Advancing Dialogue Technical Assistance. During the design phase of the DPO, it was expected that the number of beneficiary households would reach 104,000 by May, 2015, and reflecting the GOA’s commitment, this target has already been met. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 55. The Project’s objective was relevant when it was approved, and its relevance increased throughout implementation. The GOA was committed to the SPAP’s PDOs when it was approved, and increased its commitment to these objectives in the aftermath of the global financial crisis that, in view of the severe contraction in the economy, resulted in a jump in poverty incidence (from 28 percent in 2008 to 36 percent in 2011). This commitment, and the priority the GOA assigns to the Project’s objectives, is reflected in the “Armenia Sustainable Development Strategy 2025”. 18 In that document, the overarching objectives are defined as creating jobs, enhancing human capital, improving social protection and modernizing the public administration and governance. These are all, directly or indirectly, objectives supported by the Project’s PDO. 56. The Project is also consistent with the World Bank’s draft Country Partnership Strategy (CPS) for FY14-17 that focuses on the integrated social protection service reform while also 18 Republic of Armenia, Sustainable Development Program, October 2008, Government Decree No. N 1207-N 14 supporting the GOA’s pension, employment and social assistance reform agenda. Within the draft CPS’ second cluster—improving efficiency and equity in social services—the draft CPS supports selected elements of the GOA’s programs aimed at increasing efficiency and equity, inter alia, in the area of social protection. 57. The SPAP’s design fit within the context of a program of assistance and was relevant at Appraisal and approval, but began to lose relevance during implementation in view of the fact that the GOA pursued an ambitious reform agenda, especially in the area of pensions, and opted to finance several key activities outside the scope of the Project. This was the result of exogenous factors, especially the global financial crisis, and the GOA’s commitment to mitigating its effects on the population. The GOA’s resolve to move ahead with reforms to its system of social protection increased, and it sought to carry out some of the reforms and activities originally envisaged under the Project through alternative means. Further, several of the initial activities financed under Component 3 became somewhat irrelevant, although during the ICR mission the MLSI highlighted that the experience gained by the staff in the process improved their capacity to address later the greater challenges they were to face with implementing the pension reform. Similarly, one could argue that the redefining of the Youth Professional Orientation Center’s mandate as a methodological center affected the relevance of financing the initial operations and programs. Nevertheless, the new Professional Orientation Methodological Center (POMC) has refurbished offices, and through experience gained initially as an orientation center is in a stronger position to jump-start its new mandate. 58. The Project increased its relevance during implementation, especially through changes introduced in the AF and the refocusing of project activities in 2011 that ensured its continued relevance, albeit not formally reflected in a project restructuring. The AF and the project modifications (Section 1.7) were timely, and presented appropriate responses towards maintaining the Project’s continued relevance in the context of an agenda that was moving forward rapidly to further the Project’s PDO. 3.2 Achievement of Project Development Objectives Rating: Moderately Satisfactory 59. The Achievement of Project Development Objectives, as measured by the defined Key Outcome Indicators (Table 1), is considered to be Moderately Satisfactory. It is obvious that the Social Protection sector in Armenia is undergoing a major transformation process, a process that began in 2004 supported by the Project, and took on added impetus with the economic crisis and the GOA’s resolve to accelerate this process to protect an increasing number of the population that came to depend on the system to escape poverty. 60. The Project had an important role in this process, in part as an investment vehicle that touched on almost every aspect of social protection, and in part as a catalytic program to organize some of the reforms of the sector in a more integrated fashion. Together with assistance provided by other donors, the Project helped address the GOA’s investment priorities in what proved to be an ever changing agenda. Directly the Project contributed to achieving improved performance (efficiency, effectiveness, quality) by the public employment, pension and social assistance agencies in providing services to the population. For instance, the establishment of management information systems for both social assistance and SESA were important contributions to the 15 modernization of SP administration and service delivery, even if they fell short of the envisaged “national social information system”. The individual reforms pursued by the Government cannot be attributed entirely to the Project, also because GOA decided to finance on its own budgetary resources several of the Project’s original activities. Further, several of the activities supported by the Project were carried out in the final years of implementation, and their impact on the achievement of the objectives will likely only materialize in the coming years. Yet, some of those activities, specifically piloting a model for ISPCs that will now be rolled out throughout the country, have had a major impact on redefining the model of SP service delivery in Armenia, in a manner consistent with the Project’s objectives (i.e., improved performance in delivering services to the population in terms of efficiency, effectiveness and quality). In any event, the Project did have an important role, together with other World Bank-financed operations including the DPO series and analytical work, in providing broad support to the GOA’s ambitious social protection agenda, and to the overall positive results that it has achieved thus far. 61. While the indicators in the Project’s Results Framework did not prove sufficient to track the Project’s impact (Section 2.3) the collected intermediate indicators and the M&E system put in place in MLSI allow to reconstruct actual progress in most areas. These results were achieved through activities carried out under the Project’s five components, financed both with the original IDA credit and the AF. TABLE 1: STATUS OF ACHIEVEMENT OF KEY OUTCOME INDICATORS 19 Outcome Indicator Baseline Actual Target (5/31/2013) Evidence of eligible beneficiaries receiving accurate and timely pensions and social assistance payments; --Efficiency (timely), Effectiveness (accurate) The share of eligible pensioners that receive their 50.4% 100% 100% pensions timely (12/2004) 19 These indicators have been broken down for reporting purposes, e.g., the indicator “share of pensioners that received their pensions timely and fully” has been subdivided into two indicators, one that measures “timely” and the other “fully”. 16 Outcome Indicator Baseline Target Actual (5/31/2013) The share of eligible pensioners that receive their 50.4% 100% 98.2% pensions fully (12/2004) Proportion of eligible social assistance 85.5% 100% 93.2% beneficiaries who receive their benefit timely (12/2004) Proportion of eligible social assistance 50.4% 85% 95.3% beneficiaries who receive their benefit fully (12/2004) Evidence of improved job placement, especially among the disabled and young workers; Overall Placement Rate for Jobseekers 20 7% 15-20% 16% (2005) (2012) The share of disabled placed in a job (% of total 5.5% 25% 16.2% registered in Gyumri SESA regional office) (12/2005) No. of youth who received consultation in Yerevan 7,314 12,000 16,098 SESA offices (12/2005) The share of youth placed in a job (% of total 15% 23% registered job seekers in the Yerevan SESA (2007-2008) (2012-2013) offices) 21 No. of youth who received consultation in YPOC 1,549 4,000 3,047 (12/2008) Evidence of improved user satisfaction regarding service delivery related to pensions, employment services and social assistance-- Quality Pensioners satisfied with the pension system (% of 18.9% 83% 95.8% total) (12/2005) Social assistance beneficiaries satisfied with social 21.8% 86% 97.4% benefit system (% of total) (12/2005) Share of job seekers satisfied with activities of 76.9% 90% 81.7% State Employment Agency (% of total) (12/2005) Improved contributions collection in the pensions 71.6% 86% 77.8% system, as a ratio of covered wage bill over gross (12/2006) (06/2012) wage bill 20 This outcome indicator has not been set as part of the PDO or Component level indicators in the PAD or the AF PP. It was given as an example (in fact the only example) in the “Economic and Financial Analysis” of the PAD (Annex 9, p. 44) for the type of measurable improvements the Project would achieve. It is for this reason, that the indicator is included on this level. 21 This indicator was not included in the Project’s Results Framework, but has been included to show improvements in job placement for youth. 17 General Support to the Social Protection Sector Intermediate Result: Targeting and delivery of social protection services coordinated 62. The Project contributed to improved coordination in the targeting and delivery of social protection services, although it did so in a manner that differed from that envisaged when it was approved. In fact, in part through the AF, the Project’s impact on this intermediate result actually went beyond what had been expected. Although actual outcomes resulting from increased coordination are yet to materialize, several outputs supported by the Project are expected to have a positive impact on these in the future in terms of improved effectiveness, efficiency and quality of service delivery. These outputs include activities aimed at improving the availability, accessibility and compatibility of information, improving working conditions, and designing and introducing a model of integrated social services delivery at the local level. 63. With respect to improved information flows, the Project financed the development of social protections corporate information and communication network, including provision of computer, network and related equipment, complete automation of business processes within the social protection system, and integrated within an MIS linking central and regional agencies of social protection sector, the majority under WEB technologies, and training of users of systems to operate software and information systems. The original proposal for establishing a CNIF (Corporate Network of Information Flows), based on broad band system of interconnectivity, became a bottleneck, in part because of evolving technology that eventually resulted in dropping this activity from the Project and moving towards the present system of web-based technology. 64. The Project also financed improved working conditions of central and regional social protection offices through refurbishment of offices and procurement of office furniture (34 regional offices and one central office of social services). 65. More importantly, however, for purposes of eventually achieving a coordinated system of targeting and social services delivery, the GOA in 2012 issued a Decree on the Approval of the Operational-Structural Model of Comprehensive System of Integrated Social Service Provision, setting out a clear concept of a model of integrated social service provision and a proposed implementation plan, with a clear division between piloting and roll-out phases. Financing for the design and initial piloting of this new service delivery model was provided under the AF. Specifically, this included: design (including preparation of alternatives and legislative backing) and introduction of integrated social services delivery model involving pensions, social assistance, employment and disability certification by specialists under one roof, as Integrated Social Services Centers (ISSCs), or “one-stop-shops”, and piloting this first under four sites, and subsequently in another 15 sites; adjusting the MIS in these centers to establish linkages with MLSI and other databases; preparing design-cost estimates of the refurbishment of 19 pilot centers and actual refurbishment (following a somewhat lengthy delay to transfer ownership of the spaces); provision of furniture and IT equipment to pilot centers as well as software, a new information system, that integrates multiple information systems needed by the ISSCs, including beneficiary intake and follow-up, and queuing machines that allow monitoring of services provided, analysis and calculation of numbers of visitors, and time spent on provision of services. 66. The experience with these 19 pilots is being evaluated in the context of SPAP2 preparation. The expectation is that this integration of services under one roof will allow the delivery of effective and coordinated social services that meet beneficiaries’ needs in an efficient manner. The first ISSC has been in operation for only seven months; it is still too early to evaluate its experience. However, preliminary findings suggest the following points: 18 • The need for a functional model that ensures consistent management control for the realization of efficiencies and improved beneficiaries services • The commitment by the GOA to finance operating costs of the ISSCs • The balance between front and back office functions needs to be monitored carefully and calibrated, aiming for a leaner intake function and integrated specialist delivery. • Careful monitoring of the management model, with specialist staff reporting formally to their centralized units, and only indirectly to ISSC Directors will need to be monitored closely and adjusted to ensure the full benefits of service integration. • Challenges towards full integration apply especially to SESA staffs that see their beneficiaries and roles as more complex than those of their colleagues. • It will be important to provide support for system for monitoring, work flow and intermediate processing to support ISSC operations, especially since ISSC Directors are starting to produce these analyses on a manual basis. • The databases for Social Assistance and SESA remain separate, and the extraction of data for policy analysis cumbersome. These will need to be reconfigured to support an integrated case-management centered business process, once this is fully spelled out 67. As provided in the Decree, the GOA plans to establish a total of 56 ISSCs (now called ISPCs) (including the 19 under the pilot phase), based on the model defined and piloted under the Project. About 181 regional bodies and committees will be unified under one roof at the community level, will have unified reception areas, similar information systems and operate according to unified methodology. The new ISSC model is expected to improve coordination of social protection service delivery by following a “case” approach, with the complete assessment of beneficiaries’ social needs and the development and delivery of a unified program for satisfying those needs. Improving Labor and Employment Intermediate Result: Employment services provide diversified job search assistance 68. The Project made an important contribution to SESA’s ability to deliver employment service that provides diversified job search assistance. This was achieved this through a mix of activities aimed at both strengthening SESA’s ability to deliver effective and quality employment services in an efficient manner and the initial implementation and piloting of several active employment projects. The national unemployment rate in Armenia has remained stubbornly high between 2009 and 2012 (17.4 percent in 2012), and the majority of the unemployed are new labor market entrants, or persons without previous work experience, mostly youth. Facilitating labor market entry, particularly by easing the school-to-work transition through active programs, is a relevant intervention to tackle unemployment. In this difficult labor market environment, the improvement of the placement rate of jobseekers by SESA from 7% in 2005 to 16% of 2012 can be interpreted as supportive evidence of the increased effectiveness of the institution. The rate of satisfaction of SESA users also improved slightly, from 77% in 2005 to 82% at closure. 69. The GOA’s development of a new labor and employment strategy starting in 2011 and leading to 2013 resulted in the elimination from the Project of several activities that had been envisaged under the AF. A proposed new Employment Law has been approved by the Government and is awaiting parliamentary consideration. The new strategy, reflected in the draft law, promotes greater emphasis on active as opposed to passive labor market policies, and provides SESA with greater flexibility in determining those policies in response to emerging labor market realities. Therefore, the promising experience with employment activities under the 19 Project provides relevant input to the further development of active labor market policies to be supported under a follow-up project. 70. The development of a new, real-time, information management system for State Employment Services Agency (SESA) has strengthened its ability to carry out its mandated functions, while improving the quality, efficiency and effectiveness of its services for beneficiaries. SESA’s new “Gorts” internet-based system provides a comprehensive search engine that allows the exchange of information on job vacancies (including for remunerated public works) and on job seekers (including information on education, experience, etc.), on a regional basis. The system allows the users to update data on persons receiving unemployment benefits. The system also tracks participation in active labor market programs and placement rates, as well as refused job offers, thus supporting at all stages the implementation of the employment legislation. SESA’s central and 51 regional employment centers are connected to the system in real-time, and data is input by staff in regional offices. In all, about 180 staff of SESA has received training on use of the system. 71. Initial data provided by SESA with respect to outcomes is promising. The number of youth job seekers placed in a job as a share of registered job seekers in the Yerevan SESA offices increased from 15 percent in 2007-2008 to 23 percent in 2012-2013. Support for further improvements to the “Gorts” system will be provided under the proposed SPAP2 project, especially in terms of providing linkages with other social protection systems. A system for monitoring and evaluation of the effectiveness of projects implemented in the employment sector is being developed as part of unified system of monitoring and evaluation of social services in the sector (see below). Finally, SESA has invested in training of staff and managers in 51 regional centers through the delivery of 16 seminars using a train the trainers’ model; SESA now has 23 qualified trainers on its staff to replicate this training on an ongoing basis. 72. Various new, pilot programs were carried out successfully, and while the effectiveness of these in terms of outcomes has not been evaluated, SESA has institutionalized these models as a basis for future activities. These include the organization of 37 job fairs in 15 cities across Armenia, with the participation of about 1,500 employers and 71.5 thousand job seekers from 2007-2013. Also, following the training of SESA staff and refurbishment of premises, three pilot job clubs were established, providing consultancy to about 950 visitors, 115 of which were employed and 123 were involved in professional training programs. These include the organization of 3 job markets in Yerevan, Abovian, Goris, with the participation of about 7,000 thousand job seekers and 120 employers. USAID has provided funding to expand the number of job clubs from 3 to 12 adjacent to the employment centers and all these clubs continue to operate so far. Beginning from 2009 this project acquired the status of a state program, and within its framework during 2009-2012, 25 job fairs in various cities across Armenia were organized with the participation of about 52.7 thousand job seekers and more than 1,000 employers. 73. A Center for Professional Orientation and Vocational Rehabilitation of Disabled (CPOVRD) in Gyumri, was established under the Project, including the development of programs preparing disabled to enter the job market, refurbishment of the center and training of staff. Since it began operations in 2009, the center’s 16 employees have been provided professional training, advisory services and job rehabilitation (either at the center or at their homes) to about 3,000 disabled. In 2013, 16.3 percent of the registered disabled registered in the Gyumri SESA Office were placed in a job, up from 5.5 percent in 2005. The operating costs of the CPOVRD are now funded by the state budget. 20 74. As envisaged, the Project supported the establishment of an experimental Youth Professional Orientation Center (YPOC) in Yerevan, through financing of rehabilitation of the center at the selected location, development of guidelines for its operation and training of staff. It also provided for the development of a web-based electronic testing system on professional orientation that would help youth determine best career aspirations; and films explaining the requirements and responsibilities of ten different professions. The Center began operations providing group and individual consultations to youth, although the effectiveness of their operations has not been monitored. In December 2012, the Government issued a decree reforming the YPOC into the Professional Orientation Methodical Center (POMC). The POMC, now funded by the GOA budget, is still in the stage of developing its methodology and procedures, but it will no longer work directly with youth. Instead, in an attempt to increase effectiveness and efficiency by multiplying the impact of its operations, it will work with SESA regional staff in ISSCs, schools, etc. to distribute and disseminate the professional orientation materials that it prepares. By the time the ICR team visited the center its operation as an orientation center had stopped, and it was not possible to assess the soundness of the Center’s approach to orientation or its impact. 75. Finally, SESA has invested in training of staff and managers in 51 regional centers through the delivery of 16 seminars using a train the trainers’ model; SESA now has 23 qualified trainers who were selected from its staff and trained to replicate this training on an ongoing basis. Modernization of Pensions System Administration Intermediate Result: SSIF applies better processes and procedures to administer the public pension system 76. The Project supported the implementation of several activities aimed towards improving SSIF’s (later SSSS, see below) processes and procedures. These included the development of proposals on adoption of necessary legal acts for improvement of mandatory social insurance system, on improvement of financial management systems, on improvement of services provided to beneficiaries, introduction of information systems for electronic archiving and procurement of information technology equipment, etc., but many activities were dropped from Project (and funded mostly with state budget) and the impact of most of the activities that were financed became minor inputs in the face of the major pension reform enacted by the GOA in 2010. Perhaps the most relevant input in terms of increasing the efficiency of the SSSS was the development and introduction of the inner information portal, introduced in May 2013. This portal, that operates in real-time, regulates information flows, presents data in any requested format, assures effective cooperation between different agencies and employees and increases employees’ involvement in the decision-making process. Attribution of progress towards the Project’s respective outcome indicators to activities financed by the Project alone is not always straightforward. For instance, the improvement in pensioners’ satisfaction with the pension system (to 95 percent by the end of the project) could be a reflection of the increase in the amount of pension payments in 2008. On the other hand, improvement in the outcome indicator related to timely and complete payment of pensions could be more directly related to multiple interventions under the project to improve the administration of the pension benefits. 21 Figure 1: PDO Indicator for Pension Component 77. With or without World Bank financing, the GOA has made significant progress in reform and rationalizing its public pension system, and the Project—but one component in an integrated program of World Bank assistance—provided a vehicle, among others, through which the World Bank provided technical and advisory support to the GOA on pension reform. Specifically, the GOA’s reform of the public pension system was enacted through a set of laws focusing on state pensions (pay-as-you-go pillar), fully funded pensions (the second pillar), unified income taxes, personified recordkeeping, and investment funds. With this reform, SSIF was transformed from an independent pay-as-you-go pension agency with its own contribution collection, individual recordkeeping, and pension payment capacity to the State Social Security Service (SSSS), as a department of the MLSI. With social insurance contributions amalgamated into the personal income tax, revenue collection transferred to the SRC, and personified monthly tax report introduced and administered by the SRC, the SSSS will redirect capacity to strengthening its core business processes and client relations. Important aspects of this reform had been defined at the time the AF was approved, and selected inputs, e.g., the system for personified reporting and recordkeeping, had been included for financing by the World Bank under the AF. Nevertheless, in view of the ambitious timetable for introducing the new system (January 2014), the GOA opted to finance even those inputs with state budget in view of the perceived possible delays that could result if it were to contract the works following the World Bank’s procurement procedures. Social Assistance 78. The GOA took several actions to improve the administration and the targeting of its social assistance benefits. The outcome indicators tracked the former aspect, and show a steep improvement in the timeliness and accuracy of payments among the beneficiaries of the Family Benefits Program (reaching 93% by closure). Several of the activities also contributed to achieving higher targeting performance (see figure 2 below) which can be defined as the percentage of the target population (such as the percentage of the target population within the total beneficiaries. This was achieved through two complementary channels: by improving the ease of application, and by increasing the capacity to screen for ineligible households. Easing application was achieved through the development and 100 percent use of the electronic MIS for all applicants by the end of the project (as shown in result framework). The reduction of inclusion error was achieved through the enhancement of cross-checks and the integration of various government databases for the purpose of income verification, which led to the disqualification of many ineligible households. The efforts to increase targeting accuracy led to a reduction in total 22 beneficiaries, including of poor families (from 43 percent in 2008 to 36 percent in 2010), because the strict eligibility criteria was not relaxed to allow more households into the program. However broadening of coverage was not the objective of the project, as the PDO aimed at improving the performance (efficiency, effectiveness, quality) by the three agencies in services delivery. Figure 2: Targeting performance and beneficiary numbers for social protection programs 22 2004 2006 2008 2011 2013 % of beneficiaries in poorest quintile (targeting) 46% 55% 57% 55% N.A. Total households covered by the program 134,000 130,406 121,000 92,000 106,000 Source: GoA and World Bank Staff calculations in various Policy Notes (2006, 2012) based on Income and Living Conditions Surveys 2004, 2006, 2008, 2011 79. Finally, an input towards better targeting of social assistance is the capacity to monitor and evaluate the benefit system, and the Project made an important start in putting this capacity in place, as described below. Starting with a baseline household survey starting in December 2005, the Project financed several consultancies aimed at evaluating the efficiency of targeted social programs, recommending improvements to the FBP monitoring system, procurement of IT equipment, modernizing the MIS to allow for exchange of other MLSI databases for the purpose of eligibility determination, and training of staff to utilize the new system. In 2007, and later in 2009, the GOA enacted decrees aimed at improving the FBP’s targeting mechanism, by modifying the formula for determining family vulnerability and benefit amounts, leading to improved targeting performance. Eligibility is determined through a family vulnerability score, calculated in accordance with a formula that takes into account the social group of each household member, the number of household members incapable to work the domicile, dwelling conditions, average monthly income of the household, etc. 80. The Project also supported the development of the Institute of Social Worker, a virtual mechanism, as opposed to a “structure” per se. For this, the Project supported the analysis of different categories of social work in social protection sector, identification of problems faced by social workers, professional compliance, skills required, contractual regulations, and the preparation of professional job-descriptions of social work and criteria for training. Training programs were delivered to 350 social work specialists in 19 of MLSI’s regional centers. 81. Although a full M&E system could not be completed under the Project given time constraints, MLSI has made important strides in developing a tested methodology for a structural- functional model for monitoring and evaluation of social services, and trained specialists that will use it in its Monitoring Department. This activity was implemented in three phases. In Phase I, the existing capacity for M&E was analyzed, together with the legislative requirements. For Phase II, a functional model of M&E developed for all MLSI programs, together with a proposal for phasing in the methodology across 84 of the MLSI’s programs. Finally, in Phase III, training was provided for the methodology to be tested and applied to the following five programs: (i) “Employment Pensions”; (ii) “Services with Cash Support to Pensioners, Honorary Pensions and Payments under Other Programs” (relating to employment pensions); (iii) “Organization of Training for Unemployed Job Seekers”; (iv) “Child Care and Protection Services in Boarding 22 Beneficiaries expressed in terms of households; LRSA (Last resort social assistance) is the Family Benefit Program 23 Schools”; (v) “Provision of Prosthesis/Orthopedic Appliances and Rehabilitation Technical Means to Disabled and their Repair. 82. The GOA now plans, with budgetary resources to roll out the M&E of all budget financed programs, by applying the revised methodology and taking into account the lessons it learned in the five pilots. MLSI has identified challenges to be addressed in the further expansion, including: development of monitoring indicators; the need of methodology of research on collection of qualitative indicators; introduction and further adjustments of MIS; and dissemination/public awareness of M&E results. The RoA Minister of Labor and Social Issues issued an order endorsing the concept paper on introduction of the M&E system for programs implemented in the area of social protection, furthermore permanent M&E responsibilities within MLSI were established ensuring the continuity and expansion of M&E of the programs implemented in the social sector. 3.3 Efficiency 83. From the outset of the Project, the economic efficiency and benefit estimates werevery rudimentary. Neither the PAD, nor the Project Paper (PP) for the AF gave any concrete targets that were then related to the results indicators. Assessment of efficiency was further complicated by the fact that the focus of the Project changed over time. Some efficiency projections given in the PAD and related to improved social insurance contribution collection have become obsolete in light of the outright abolishment of social insurance contributions in favor of a unified tax and contribution collection system. 84. While assessment of efficiency on the basis of the project documents is nearly impossible, a set of top-down estimates can be established for the efficiency of the individual components. The ICR team was able to gain an impression of the institutional quality and professional capacity of the social protection agencies and their staff. When faced with technical questions or request for performance data in Yerevan or in field offices, staff of the ISSCs or the SES was able to provide these data and rely on systems to generate them. 85. The efficiency effects of the Project are expected to unfold over a wide area of administrative units and agencies, building on improved IT systems, office equipment and training of staff. With 2.2 percent in administrative costs as a share of total program costs, the FBP was already the most efficient last resort social assistance program in the Europe and Central Asia Region in 2006, according to a regional study currently being finalized.23 Information on costs in later periods is not available, and it is not clear whether the improvement of the administration system had an impact on cost-effectiveness; however, the integration of databases reduced the burden of documentation for application to individual citizens. 86. Another key efficiency dimension of the Project lies in the set-up of the ISSCs, Integrated Social Service Centers. At this point in time, it will be too early to assess any efficiency effects of the ISSCs, as their management and control model still needs to be finally decided and firmed up. Similar experiences in the region and in the OECD context have yielded substantial improvements in staff efficiency (either via reduced staffing numbers for the same amount of 23 Tesluic et al (forthcoming): Last Resort Income Support for the Poorest: A Review of Experience in Eastern Europe and Central Asia. 24 cases or via increased citizen face-time for case workers) in one-stop-shop operations. In an example from Ukraine, processing time for applications dropped from 4.5 hours in 2005 (prior to the introduction of integrated centers) to 1 hour by 2012. 24 In case Armenia opts for an effective management control model of its ISSCs and ensures spatial co-location of staff in all localities, similar efficiency gains could be expected. 87. On the citizen-side, improved efficiency arises from a unified collection of tax and social insurance contributions, easing filing burdens on workers and employers. For users of ISSCs, time efficiencies arise from the single-location approach and single application form for different services. The ISSCs chosen and inspected during the ICR missions were all located in central and accessible locations (municipality, close to bus routes, etc.), enabling citizens to engage with ISSC staff easily. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 88. The Project’s Overall Outcome Rating is Moderately Satisfactory. The relevance of the Project’s objective is unquestionable. Its design, both in terms of institutional arrangements and activities financed lost relevance somewhat during the project period, but throughout implementation support and flexibility ensured that its design at closing continues to be particularly relevant to the GOA’s social protection agenda at present and in the future. Despite problems with the Results Framework, the accomplishment of the Project’s objective of improving performance in social protection service delivery was largely met, and the full extent of the direct impact of project activities on this improved performance on beneficiaries will require further time to materialize and measure, along with continued efforts to roll out the new service delivery model. The GOA remains committed to sustain these efforts (Section 4), and the World Bank is planning to provide further support for this (Section 2.5). Actual measures of the Project’s efficiency are difficult to measure at this stage, but the new model of social assistance delivery is definitely expected to impact efficiency in the social protection sector in the foreseeable future. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 89. Overall the project deign did not include gender-specific components and targets, which is reflected in the absence of gender disaggregated indicators. An exception was the satisfaction surveys conducted among beneficiaries of SESA, published in the M&E report 2011-2012 25. This survey showed that 76percent, of female beneficiaries were satisfied or mostly satisfied with the services received, compared to 15percent among male beneficiaries. 24 ICR2672. Ukraine “Social Assistance System Modernization Project”, p. 23 25 Monitoring and Evaluation Interim Stage Report, 2011-2012: page 51 25 (b) Institutional Change/Strengthening 90. The Project was essentially an institutional development and strengthening project and its impact on longer-term capacity and institutional development is mentioned throughout Section 3.2. (c) Other Unintended Outcomes and Impacts (positive or negative) 91. n/a 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 92. A beneficiary survey element was included in the general household survey of the National Statistical Office. Results are given in Annex 5. 4. Assessment of Risk to Development Outcome Rating: Negligible 93. The Risk to Development Outcome is rated Negligible. The GOA’s commitment to continuously improving its system of social protection is unquestionable. The Minister of MLSI confirmed to that World Bank that the Ministry would finance any parts of contracts under the Project that are not completed by the Closing Date with their internal budget. The activities carried out under the Project, and those that the GOA opted to finance with budgetary funds have been implemented by Government agencies that have counted on institutional strengthening, and those ministries and agencies have internalized those activities. The GOA is financing the operating costs of project-supported agencies on a routine basis, and will continue to do so. A cross-sectoral steering committee is in place to oversee the implementation of integrated social protection reform. Finally, the GOA continues to seek, and given its strong commitment and record, should continue to count on continued external funding, including from the World Bank, to support its social assistance reform priorities (Section 2.5). 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 94. Bank Performance in Ensuring Quality at Entry is rated Moderately Satisfactory. (This section addresses Quality at Entry of the Project, as appraised in 2004; the processing and approval of the AF is described in Section 5.1(b) below). This was the World Bank’s first engagement in support of Armenia’s social protection sector. The Project was timely and responsive to the GOA’s request for assistance, and was grounded in strong analytical work. At the same time, being a first engagement, the Project was unquestionably ambitious, both in terms of the number of components, stretching across all of the domains of an effective social protection system, except health, and in terms of the number of activities under each of those domains. The project size was relatively low (US$5 million): this amount could be justified given that this was the first engagement of the Bank in the sector, and that the policy objectives at 26 start were less ambitious than those that later emerged. Yet, the Project’s Results Framework presented important challenges, as described in Section 2.3 and were it not for the ample evidence of the Project’s pivotal role in support of the GOA’s many and continuously evolving reforms, it would be difficult to evaluate the achievement of its objectives through the chosen indicators. (b) Quality of Supervision Rating: Moderately Satisfactory 95. The Quality of Supervision is rated Moderately Satisfactory. Over its nine-year implementation period, the Project was supervised by four Task Managers and their teams supported by Country Office staff. A strong local presence provided continuity in the dialogue with the GOA, as well as strong Headquarters-Country Office communication and coordination, including with the several World Bank task teams responsible for other social protection sectors tasks. 96. During the ICR mission, the GOA commented on the World Bank’s receptiveness to requests for and strong analytical capacity of advice provided by its high-level specialists in response to their evolving priorities, and highlighted their strong collaboration and especially the GOA’s ability to count upon an open and constructive dialogue in areas of priority to them. Especially important to the GOA was the World Bank’s adaptability to the changing environment and sectoral priorities, including the processing of the AF, where the World Bank teams first attempted to understand the impact of changes in Government policies and strategies on the Project, then arrange to provide best-practice knowledge and advice in the area, and finally adjust the Project to reflect the GOA’s changed priorities. 97. The World Bank’s supervision was implementation-focused, especially towards the end of implementation, when the World Bank’s team worked on an almost daily basis with PIU and FFPMC, with detailed, weekly reports to monitor progress and ensure completion of the ISSC pilots, thereby ensuring almost complete utilization of AF funds. The World Bank’s team also worked closely with the GOA, starting with the processing of the AF and later during a somewhat delayed but nevertheless comprehensive Mid-Term Review, to adjust the Project, in coordination with donors, in response to the Government’s evolving funding priorities in the sector. Some activities were withdrawn from the Project to ensure completion as per the Government’s mandated timetable (e.g., developing the system of personified record keeping), since following the World Bank’s procurement procedures would have required much longer lead times and thereby put in jeopardy completion by the required date. 98. Yet, the World Bank’s decision not to restructure formally the Project’s Results Framework and indicators following the Mid-Term Review, when it became clear that the GOA’s priorities for World Bank financing had evolved, was probably not the best decision, particularly at the AF stage, when the PDO level indicators were confirmed in spite of the fact that most had been achieved, Further, the large increase in the amount of civil works for rehabilitation of the ISSCs suggests a revised environmental classification may have been warranted, but there is no documentation of any discussion of this topic, under a joined effort of the client and the World Bank to ensure tangible project achievements by the projected closing date. Nevertheless, the supervision team was faced with the decision of either responding to the GOA’s changing priority (e.g., piloting the ISSC concept) and providing intensive support for making it work during the slightly more than one year of implementation remaining, or of devoting what would have been a considerable part of this remaining period towards negotiating and seeking external and internal approval for a project restructuring that might then not have had sufficient time to produce results. 27 (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 99. Overall Bank Performance is rated Moderately Satisfactory, based on the ratings for Quality at Entry and Supervision. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 100. The Borrower Performance was Satisfactory. For purposes of this Section of the ICR, Government is taken to be the broader GOA, including the MOFE and MLSI, at the highest levels. This Project, together with the many adjustments it witnessed, definitely had strong Government ownership and was adjusted during implementation to adapt to its ambitious social protection reform agenda. The GOA’s commitment to the SPAP’s objective of achieve improved performance (efficiency, effectiveness, quality) by the public employment, pension and social assistance agencies in providing services to the population was unwavering throughout, and even intensified in light of the economic crisis that affected the country starting in 2009, despite constrained finances. Throughout the implementation period, Armenia had three Prime Ministers (of the same party) and six Ministers of Labor and Social Issues (of several parties, often different from that of the Prime Minister), yet commitment to the Project and its objectives was maintained throughout. 101. The GOA continuously sought to optimize the use of external assistance to meet its evolving priorities, by reviewing and making compatible its internal, budget and external funding to its needs, although in retrospect, one could question whether, at the time the AF was approved, the GOA did not envisage the then imminent policy changes and financing priorities for those changes, which could have avoided use of project resources on a few activities that later became irrelevant. The decision to carry out within the Government many of the most knowledge intensive components of the Project demonstrated strong ownership over policy-making, but it was difficult for the team to assess whether the opportunity to incorporate global knowledge into strategic policy planning was not sacrificed in the process. In any event, the GOA continues to have an ambitious social protection agenda, and has been very engaged in designing a follow-up project. For this it is actively seeking to evaluate activities under the Project to extract lessons relevant to the “next step”, especially with respect to the experience with pilot ISSCs. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 102. The implementing Agencies’ Performance is rated Satisfactory. As a result of the Project’s broad scope across the domains of the social protection sector, its activities were implemented by several implementation agencies, including the PIU in MLSI, the SESA, and the SSIF (later SSSS), and lastly by ISSCs piloted during the final year of implementation. In general, these implementation agencies demonstrated strong commitment to the Project and its objectives, as well as to the GOA’s evolving priorities and ensuring that the activities financed by the Project were consistent with those priorities. Implementing agencies assigned highly capable staff to the Project’s implementation that sought advice when needed. Although there were some changes in the staff of these agencies throughout implementation, as might be expected, their impact on the Project’s implementation were negligible. FFPMC provided instrumental support 28 to the carrying out of the Project’s fiduciary aspects and compliance with the World Bank’s fiduciary requirements, and collaborated closely with the PIU in MLSI to ensure the responsiveness to project priorities. The effective coordination between FFPMC and the PIU, strengthened throughout implementation, was especially important to the strong implementation performance during the Project’s last year or so of implementation. (c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory 103. The Overall Borrower Performance is rated Satisfactory, based on the ratings for Borrower and Implementing Agency Performance. 6. Lessons Learned 104. Client capacity-building for M&E should have priority over use of external contractors. The Project aimed to strengthen monitoring and evaluation of social protection programs, in general, and through the design and implementation of information systems needed for this purpose. As this capacity was being developed by the client, the task of routine project monitoring was outsourced to an external contractor, based mostly on information provided by the PIU. In retrospect, little, if any, value-added was provided by this external contractor, other than compiling and formatting data for the beneficiary surveys. When even a minimal capacity for monitoring is in place, the World Bank should opt instead for financing the training and employment of specialized staff within the client agencies to fulfill project monitoring functions, thereby increasing capacity, as opposed to “contracting out” those functions. 105. Management model and staffing arrangements of the ISSCs need to be geared towards efficiencies and effectiveness. The current management and staffing arrangements of ISSCs are centered on a “coordinator” model in which staff of the four specialized services (pensions, social assistance, labor/employment, and disability) operates in a matrix relationship to both their functional units and to the “coordinator” of the ISSC. Coordinators lack formal management control over the total number of staff in the ISSC and have difficulties raising efficiencies and improving service quality for citizens. Generally, management control for coordinators appears to be limited to the front-office/reception part of the ISSC and to any staff they might have managed as part of their legacy function within one of the four domain services. It is important to continuously review these types of operational issues and have the government take decisions towards a stronger mandate of the ISSC management. For the model to improve effectiveness and efficiency, the ISSCs need to provide more than simply the sum of their four functional parts. 106. An investment loan can be a powerful vehicle for sector policy dialogue. It is unquestionable that the GOA made steady progress in practically all of the areas of social protection during the Project’s implementation period. In part because of the Government’s strong commitment to putting in place an efficient, effective and high quality social protection system, and in part because of the strong program of technical assistance, the lending and the technical advisory services provided both the World Bank and other donors showed to be complementary and mutually reinforcing. In retrospect, two strong lessons derive from the World Bank’s experience with the Project (and the broader sector dialogue that accompanied its implementation). First, even if an activity/reform (e.g., pension, FBP) is not financed directly by 29 the World Bank, involvement through a project can provide a vehicle, together with others, through which advice and be provided and supported if need be. Second, this involvement is as important for the reforms and activities that can be supported, as for those that through implementation support and other forms of advice, clients decide against. 107. Mid-Term Reviews (MTR) should not be delayed just because projects have not reached the expected “mid-point” of their implementation. In fact, especially when projects are not implementing as scheduled, the MTR can provide an opportune venue for identifying issues impeding implementation, redefining priorities and reexamining expected results to put a project back on schedule. All MTRs should as a matter of routine reexamine the project’s original Results Framework, both in terms of its quality at entry as well as its continued relevance, including that of feasibility of data collection to show progress towards results. 108. When disbursement plans are shifted, the project needs to be restructured. Task teams should be reminded of the need to continuously reexamine projects’ results frameworks, and adjust them as soon as problems with those frameworks are identified, either due to poor initial design or changes in project activities and focus. It would have been relatively easy to process a restructuring of the Project’s Key Indicators, especially in cases such as that of this project, when policy performance is good, the clients’ reforms dynamic, and the Project’s Development Objective(s) remains unchanged. An early MTR could have reduced the pressure on the task team to tradeoff between last-year disbursement performance and formal restructuring. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 109. The team discussed with the borrower’s project implementation team the content of the ICR during a dedicated mission. Minister Asatryan submitted written comments to the draft ICR on October 2013. The main points raised are summarized below and the author’s response is in brackets. i. Suggested minor changes to the tables A and B of Annex 1, consisting of an update of the final expenses at the time of project closing, based on the accounts of the RoA Ministry of Finance Foreign Financing Projects Management Center. [Annex 1 currently reflects the changes proposed by the borrower]. ii. Table A. Basic Information, “Co-financiers and Other External Partners” should specify that the RoA government was involved as a co-financier and show the amount of its co- financing. [Included] iii. Suggested adding additional detail on sustainability of some activities (training, job market fairs, job clubs) conducted in Component 2 – Improving Labor and Employment - [Included] iv. Suggested adding narrative describing the adoption of the M&E system, developed as part of the project, into the MOLSI regular monitoring process of programs. [Included in revised version] 30 v. The RoA Ministry of Labor and Social Issues highly appreciated the effective performance of the World Bank team under the program and propose to assess it as “Satisfactory” rather than “Moderately Satisfactory”. [Authors explained to the borrower the rationale for the ratings but agreed to raise this point during the Decision Meeting and finalize the ratings following the meeting] (b) Co-financiers 110. No co-financiers collaborated in the project. The cooperation partners of the World Bank (mainly USAID and UNICEF) lauded the excellent collaboration with the World Bank. (c) Other partners and stakeholders 111. No other non-commercial and non-government partners were involved in the implementation of the project. 31 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Estimate Actual (USD M) (USD M) Ph. 1 Percentage of Components Ph. 1 Ph. 2 (AF) Appraisal Ph. 2 (AF) Sum Sum Ph.1+2 Ph.1+2 1.76 3.01 171% 1. General Support to the Social 2.50 3.82 152% Protection Sector 4.26 6.83 160% 1.24 1.01 81% 2. Labor and Employment 0.35 0.03 9% 1.59 1.04 65% 1.20 0.33 28% 3. Modernization of Pension System 1.26 0.82 65% 2.46 1.15 47% 0.67 0.37 55% 4. Improvement of Social Assistance 0.27 0.32 115% System 0.94 0.69 72% 0.50 1.15 230% 5. Project Management, Monitoring 0.32 0.64 197% and Evaluation 0.82 1.79 217% 5.37 Total Baseline Cost 4.70 10.07 0.20 Physical Contingencies --/-- 0.00 0.20 0.14 Price Contingencies --/-- 0.00 0.14 --/-- Co-financing from borrower (Ph2) 1.10 1.10 5.71 5.87 103% Total Project Costs 6.10 5.63 119% 11.81 11.50 110% Note: these estimates include both IBRD/IDA financing and borrower co-financing. Note that for phase 1, estimates included co-financing from the borrower by component. The AF PP did not specify borrower co-financing by component, so it is reported as a separate line. Source: Appraised Estimates: PAD (May 2004), PP AF (January 2010). Actual estimates: Borrower Completion Report (July 2013) and Borrower comments to the ICR (November 2013) 32 (b) Financing Source of Funds Appraisal Estimate Actual (USD M) Percentage of (USD M) Appraisal Borrower Ph. 1 0.56 0.34 60% Ph 2. 1.10 0.90 81% Ph 1+2 1.66 1.25 75% International Development Association (IDA) - Phase 1 5.15 5.53 107% International Bank for Reconstruction and Development (IBRD) - Phase 2 5.00 4.73 94% (AF) IBRD + IDA 10.15 10.25 101% Ph. 1 5.71 5.87 103% Total Ph 2. 6.10 5.63 91% (Borrower+ IDA+IBRD) Ph 1+2 11.81 11.50 97% Sources: Appraised Estimates: PAD (May 2004), PP AF (January 2010). Actual estimates: Borrower Completion Report (July 2013) and Borrower comments to the ICR (November 2013) 33 Annex 2. Outputs by Component Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF Component 1: General Support to the Social Protection Sector – This component will address critical weaknesses in the MIS for the social protection sector. It will assist in the development of an effective MIS for the creation, acquisition, storage, maintenance and exchange of digital information required to operate an effective social protection system, based on individual social security cards and numbers. Sub-component 1.1: Communications network for information flows (CNIF) – This sub-component would help establish a communications network for the transmission of information within and between the different social protection programs of MLSI A pre-design study of the Completed communications network Drafting of TORs to conduct a tender Completed for the design and establishment of a communications network Delivery, installation and testing of the Completed The Project financed computers, technology equipment network and related IT equipment delivered to MLSI, SSSS, the “Nork” Information Center, and to MLSI’s regional offices Setting up communications services at Cancelled under the Project, The Project financed only the following, regional and local offices; technical but completed with before this activity was removed from maintenance of the network Government funding the Project: • Consultations on IT and business processes • Consultants’ services to develop technical specifications for modernizing technical system • Assistance in developing the CNIF system (from 9/2007 to 6/2008 only) • Technical Support, connectivity, staff training and retraining was to be provided under the Public Sector Modernization Project. However, the GOA opted to support the development of the private sector in telecoms, especially in internet connectivity through increasing coverage and assuring public-private 34 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF partnership. Hence, further development of CNIF was not required. Training of specialists responsible for See comment immediately maintenance of the corporate network above Finalize the development of the social protections corporate information and communication network, complete automation of business processes within the social protection system and integrate them into an MIS Complete physical and/or Initiated with setup of ISSCs virtual transformation of all local social protection offices into a “one-stop-shop” Integrate administratively Initiated with setup of ISSCs territorial centers for social services into one administrative structure and make them a part of the “one- stop-shop” organization of service delivery Sub-component 1.2: Technology Platform -- This sub-component would help select the optimal platform for introducing cost effective and efficient information sub-systems that reflect the requirement and operating environment of the MLSI Analysis of existing information sub- Completed The analysis of all existing systems of the MLSI, SESA, SSIF and subsystems of MLSI was appropriate local agencies and the completed; modernization elaboration of proposals for their (or providing the basis for modernization, taking into account the modernization) was needs for future integration of new completed. However, due programs to policy and legal changes the results were not utilized. The GOA financed the development of a new “epension” system in accordance with the reforms to the pensions 35 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF system. Analysis and modernization of the GORTS system is complete, and the system is functional. The FBP data base was partially modernized. Selection of the platform for the Completed implementation of the information system Delivery and installation of the Completed equipment and development and introduction of system software Sub-component 1.3: Procedures -- This sub-component would adapt current MLSI programs to the new network structure Analysis of current Completed programs/information Elaboration of proposals and Completed development of TORs for the modernization of existing programs and their integration in a new national information network Sub-component 1.4: Software Development -- This sub-component would modernize existing programs and integrate them into the new national information network Upgrading of programs that are Completed currently operated by MLSI offices and related agencies, according to TORs prepared under the sub-component 1.3 Training of specialists in the Completed development processes Training of staff in new technologies Completed Component 2: Improving Labor and Employment – This component would increase efficiency and effectiveness of the state employment services (SESA) in labor market intermediation and to support labor reform Sub-component 2.1: Management Information System – This sub-component would improve business practices in the SESA by introducing an MIS as a part of the national information network. 36 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF Refurbishment of SESA offices to Completed; air-conditioners accommodate the MIS at central and procured for central SLI office local office levels Installation of MIS-related hardware IT equipment for central and and voice and data communications at regional offices procured central and local levels MIS-related software development and New “Gorts” MIS of implementation at the center and in the employment sector introduced in local offices SESA; it is operated by 51 regional SES offices Training of personnel in use of the MIS 181 employees trained on new system at central and local employment “Gorts” MIS office levels Sub-component 2.2: Improvements in employment services provision – This sub-component would pilot new programs in selected local employment offices. A pilot survey on a national basis of Completed employers as a first stage in determining labor market trends and a survey of disabled individuals in Gyumri. Piloting job clubs, and vacancy and job Job Fairs carried out in Yerevan, fairs in three regions (including Abovyan and Goris with preparation of guidelines for the participation of over 7,000 job- activity, training of personnel, seekers and 120 employers; publication and dissemination of Pilot job clubs in ERCs in information materials) Erebuni, Abovyan and Goris established. Introduction of improved employer Consultancy on improving contact services, job search assistance employment services provision programs, job search skill training carried out and SESA staff and programs (including preparation of the managers received training in 16 guidelines for the activity, training of seminars personnel, publication and dissemination of information materials) Initiatives to provide Dropped employment to rural population and unemployed without record of service Establishment of a Regional Dropped in view of change to 37 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF center for professional Methodological Center orientation of unemployed Strengthening SESA to Dropped enhance its capacity to train and re-train job seekers Study of labor market Dropped conditions and earnings structure Sub-component 2.3: Job Rehabilitation Center for Disabled Unemployed -- This sub-component would establish Professional Training, Advisory Services and Job Rehabilitation center for the disabled unemployed in Gyumri. Refurbishment and equipping of a Completed building for the Center Introduction of relevant programs for Completed The Gyumri center is in the reconditioning of the disabled for operation since 2010 with the job market Government budget financing Training of personnel Completed Sub-component 2.4 Professional orientation services to unemployed youth – This sub-component would support the establishment of an experimental Youth Professional Orientation Center in Yerevan. Rehabilitation and equipping of the Completed facility Development of guidelines for the Completed services to be provided by the Center Training of personnel Completed Component 3: Modernization of Pensions System Administration – This component aimed at creating a modern pension administration Sub-component 3.1: Business Processes – This sub-component would provide TA, equipment and training to strengthen critical business processes related to benefit administration. Contributions collection and Completed The TA was completed, the revenue/expenditure monitoring would approaches developed, be strengthened through procedures for however, due to the reconciliation of payments, refining of abolishment of SSIF in reporting mechanisms, facilitating of December 2007 under the filing procedures GOA pension reform, they were never introduced Audit functions complementary to and Completed The TA was completed, the 38 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF supportive of the routine collection approaches developed, process, would be improved however, due to the abolishment of SSIF in December 2007 under the GOA pension reform, they were never introduced Efficiency improvements would be Completed The TA was completed, the undertaken to the process of pension approaches developed, calculation and post-adjustment, and however, due to the delivery of pensions abolishment of SSIF in December 2007 under the GOA pension reform, they were never introduced Client services would be improved by Completed The TA was completed, the measures aimed at lowering costs to approaches developed, clients of participating in the system however, due to the and maintaining public confidence in abolishment of SSIF in the system December 2007 under the GOA pension reform, they were never introduced Foreign and local experts would work Completed The TA was completed, the with SSIF to develop practical approaches developed, procedures, guidelines, handbooks and however, due to the other materials so that the processes abolishment of SSIF in can be implemented in a predictable, December 2007 under the monitorable and reasonably uniform GOA pension reform, they manner were never introduced Sub-component 3.2: Institutional Development – This sub-component would support the strengthening of internal institutional processes and introduce new functions that are needed for the effective administration of the pension system. The Project would support the creation of institutional capacity in SSIF to conduct actuarial forecasting, program evaluation and policy development, public information and education about SSIF programs, and human resource development, including staff development. Strengthening of the general Completed-recommendations management function in the SSIF, provided including revision of processes, procedures and rules Development of a performance Completed improvement plan for local SSIF offices aimed at enhancing (i) quality of service delivery, (ii) efficiency of 39 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF administrative processes, and (iii) satisfaction of system participants (contributors, beneficiaries and staff members) Strengthening of asset management, Completed With GOA funding, the involving systems to ensure integrity in proposal was developed cash management, and standardized further; since 2011 financial management procedures, pensioners can received including introduction of international their pensions through accounting standards banks using debit cards. Enhancing capacity to conduct short Completed and long-term projections and simulations of program revenues and expenditures under different assumptions Development of a public affairs Completed This unit functions well function as a source of ongoing contact within SSSS and outreach to the media, and direct information to fund contributors and beneficiaries Enhancing human resources Completed management, focusing on recruitment and placement, compensations and benefits, and training and career development Development of a capacity for Completed providing policy makers with analysis of the implications of programs and program changes Improving the quality of client services in SSSS and disability pension certification system Training staff in MLSI, SSS and SRC engaged in new processes Sub-component 3.3: Creation of a central register – This sub-component would help develop a central register (an integrated information system, IIS) that would underpin the operational improvements in key business processes and expanded institutional capacity for the administration of the pension system. 40 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF Minor refurbishment of SSIF offices to Completed-equipment purchased allow the installation of an MIS for the and installed, and office space processing of information for the renovated central register Installation of central register related Completed-equipment purchased hardware and communications at and installed, and office space central and local levels renovated Software development for the critical The Unified Income Tax business processes and institutional and Individual Mandatory improvements to be supported by the Funded Pension IIS Contributions Personified Record Keeping System was developed/is being developed by the GOA with budget financing. The Unified Income Tax Personified Record Keeping Module is functioning; the Individual Pension Contribution Module is under development and testing and should start functioning in Jan. 2014 as the mandatory funded pillar of the pension system becomes effective as of that date. The introduction of the IIS at the center See immediately above and in local offices (processes for the identification and registration of system participants, contributions collections, maintenance of earnings and contribution histories, establishment of entitlements, payment of pensions, maintenance of payment records, post adjustments Training in use of the IIS system at See above central and local office levels 41 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF Strengthening of the IT department See above (building in-house capacity to operate and maintain the IIS Development and installation See above of software and provision of IT equipment for personified reporting (registry) and record-keeping at the SRF Development/improvement See above of the MLSI’s software and provision of IT equipment for pension determination and pay-as-you-go (PAYG) database management Component 4: Improvement of Social Assistance – This component would support hardware update and introduction of electronic data exchange in the administration of FPB and other social assistance programs, and would help improve the poverty targeting mechanism. Two rounds of a community survey Household survey carried out in 2006; Development of capacity for in-depth Several consultancies carried out monitoring and evaluation of the to evaluate efficiency of targeted effectiveness of poverty targeting social programs and recommend improvements to monitoring systems of the FBP Modernization of hardware (equipment Completed upgrade) used for targeted family poverty benefits Associated training of social assistance Seminar for MLSI staff and staff social service regional office employees on FBP monitoring and evaluation conducted; 3 study visits of specialists of central and regional social assistance departments Establishment of a social Completed worker institute for training Community-focused Completed household surveys Carrying out a public Dropped 42 Activities proposed in PAD Changes as proposed in PP Other Modifications Implemented under Project Comment for AF awareness campaign Developing a simplified Completed The integration of public system for applications and databases reduced eligibility certification documentation burden of application Component 5: Project Management – This component would ensure professional implementation of the Project by using appropriate mechanisms of project management, and proper tools of monitoring and evaluation. Support agencies in procurement and Completed; office equipment financial management according to and stationary procured; project IDA requirements audit financed yearly; Project Coordinator in place Completed Short term IT/social protection Completed consultants in instances where the implementing agencies need support Hiring of a Monitoring and Evaluation Annual monitoring carried out Specialist throughout project Design cost estimates for refurbishment of 35 offices of SESA, SLI, SSSS and SSRA financed 43 Annex 3: Economic and Financial Analysis -- please refer to results and efficiency section -- 44 Annex 4: Bank Lending and Implementation Support/Supervision Processes Task Team members Names Title Unit Lending Anna Goodman Program Assistant ECSHD Susanna Hayrapetyan Senior Health Specialist ECSH1 Plamen Stoyanov Kirov Senior Procurement Specialist LCSPT Arvo Kuddo Senior Labor Economist HDNSP Anahit Poghosyan Office Manager ECSHD Aleksandra Posarac Country Sector Coordinator EASHS Hjalte S. A. Sederlof Consultant IEGPS Seyoum Solomon Consultant TWITR Arman Vatyan Sr. Financial Management Specialist ECSO3 Elena Zotova Sr. Technical Spec. ECSHD Supervision Zoran Anusic Senior Economist ECSH3 Alexander Astvatsatryan Senior Procurement Specialist ECSO2 Natalia Barinyan Temporary ECCAR Olena Fadyeyeva Senior Country Officer SACIA Susanna Hayrapetyan Senior Health Specialist ECSH1 Aleksan Hovhannisyan Senior Social Protection Consultant ECSH3 Arvo Kuddo Senior Labor Economist HDNSP Nicole L. La Borde Office Manager MNSHD Carmen F. Laurente Senior Program Assistant ECSHD Karina Mostipan Senior Procurement Specialist ECSO2 Gabriel Francis Program Assistant ECSHD Satik S. Nairian Program Assistant ECCAR Aleksandra Posarac Country Sector Coordinator EASHS Arman Vatyan Sr. Financial Management Specialist ECSO3 Anna L Wielogorska Senior Procurement Specialist EASR1 Yingwei Wu Senior Procurement Specialist LCSPT Implementation Completion Report Suzana Abbott Primary Author ECSH3 Gabriel Francis Program Assistant ECSH3 Susanna Karapetyan Social Protection Consultant ECSH3 Ulrich Hoerning ICR Co-Team Leader ECSH3 Matteo Morgandi ICR Co-Team Leader ECSH3 45 Annex 5: Beneficiary Survey Results Results from the 2011 Income and Living Conditions Survey, module on Family Benefits Program Was it easy to collect the Did you pay for documents needed documentation necessary for to apply? registration? (% FBP applicants) (% FBP applicants) 60 100 40 50 20 0 0 Yes, easy Not so easy Very difficult Yes No Refuse Were you satisfied with the work of your regional social inspector? 80 (% FBP applicants) 60 40 20 0 yes to some no cannot extent answer Do you think that the FBP system What % of families receiving FBP is justified? (% FBP applicants) are in your opinion poor? 80 50 % FBP 40 60 % FBP applicants 30 applicants 20 40 10 20 0 almost more half less a few of do not 0 all than than them know yes no not sure do not half half know Source: Armenia ILCS 2011. N=3998 out of 4,089 (FBP applicants), N= 31.024 out of 31.024 (Population) 46 Annex 6: Stakeholder Workshop Report and Results n/a 47 Annex 7: Summary of Borrower's ICR and/or Comments on Draft ICR Note: The Borrower Completion Report was received in July 2013 and provided a comprehensive overview of the project activities and their actual execution.The Borrower Report has been edited by the ICR Authors, by omitting information on the project description and objectives that are already reported elsewhere in the ICR report. The full report is available upon request to the authors. Borrower Completion Report 1. OUTPUTS OF THE FIRST PHASE OF THE PROJECT PER COMPONENTS Component 1: General Support to the Social Protection Sector (Actual amount is USD 3 011 819.32, which makes 51.3 % of total disbursed amount)  For the creation of CNIF preliminarily provided was appropriate consultancy for technological platform for provision of IT, and also implemented were the works for preparing of ToRs on provision of consultancy for IT business processes, and for modernization of IS. The activity “Consulting services for the development of CNIF” (network design) was fulfilled within the period of September, 2007-June, 2008. The other works (technical provision, connection of channels, training and re-training of employees), envisaged for the creation of CNIF were planned to be implemented together with RoA Ministry of Territory Management in the scope of “Modernization of State Sector” project, as the funds envisaged under SPAP were not sufficient for the fulfilment of works of such a volume. However, as a result of a new policy, adopted by the RoA Government, it was decided to develop the private sector of telecommunication, particularly of internet provision thus enlarging the coverage and ensuring the cooperation of the State and private sector in this issue. As a result the funds for 2 projects, envisaged for CNIF were re- disbursed in 2008. It should be noted, that within 2008-2013 Armenia recorded a great progress from the viewpoint of internet coverage and capacity building, and as of the closing date of SPAP almost all organizations of SP sector were provided with internet or /and intranet and the overwhelming majority of information systems are operated under WEB technologies.  In the scope of SPAP voluminous procurements were made for the provision of computer, network and related equipment for the offices of central and regional agencies of SP sector. Particularly, under Component 1 procurements were made for MLSI, SSSS,”Nork” information-analytical centre, MLSNI, regional agencies/ committee of SP sector (55 SSRAs/SSRDs, 51 SSSS RDs, 51 EMCs/ERCs, 25 MSECs), and delivered were servers, computers, monitors, notebooks, UPSs, printers, multifunctional devices, cameras. Created are also local computer networks for the offices of mentioned regional agencies. Mentioned deliveries went on also in the scope of SPAP AF assuring a great progress for the quick development of IT substructure of SP sector. At the same time, in the scope of the Component, designed and created was the local phone-network of MLSI, which is successfully operated ensuring efficient and cheap service.  In the scope of works for improvement and modernization of information systems a task was set to ensure combined and automated use and exchange of data between operating ISs of SP sector, and also the input and use of data in a real regime. However, for the immediate operation of created new tools the channels, which would connect regional offices with central offices, were missing. Later, associated with problems arisen in the course of reforms, the approaches of using modernized systems were changed. Particularly, in the scope of 48 pension reforms it was decided to develop and introduce a new state pension electronic system, which was implemented in the course of 2011-2013, and currently the new “Electronic Pension” system is successfully operated by the SSSS. As a result of introduction of a new system of identification cards in Armenia, the MLSI stopped the operation of “Social Security Card” IS. In RoA, in the scope of ispection reforms it was required to develop and introduce a new IS ensuring inspections based on risk assessment, and the operation of developed and modernized system became inexpedient. The introduction of CSS unified system under AF resulted in new tasks of integration and combined use of data, which were mainly implemented through the development and introduction of a new information system of ISS. As a result, ensured was the comined and online use of data from 18 ISs. So, it should be noted that the information systems, modernized in the scope of SPAP 1st phase, are used partially, however the task of assuring of integrated informational environment and automated data exchange is mainly implemented as a result of works supported by SPAP AF and RoA State Budget.  One of the most important activities for the development of ICT subsystem of SP sector was the training and re-training of users of ICT subsystems , in the scope of which almost all employees of 55 SSRA/SSRDs, 51 SMC/SRCs, 51 SSSS RDs, and also of MLSI were trained in order to be able to operate softwares and currently operating information systems. Mentioned trainings granted the opportunity to use the existing and newly created ICT resources more efficiently thus increasing the quality of general management and service provision. At the same time trainings were conducted for the employees of SP departments of Local Self-Governing bodies. The following activities were fulfilled for the improvement of working conditions of central and regional offices of SP sector: 1. Refurbishment of offices of regional agencies of SP sector (SSRAs/SSRDs, EMCs/ERCs, SSSS RDs) and SLI central office, 2. Procurement of office furniture for offices of regional agencies of SP sector (SSRAs/SSRDs, EMCs/ERCs, SSSS RDs).  During the first phase of the Project in general 34 regional officies and one central office of SS providing agencies of Yerevan and marzes of Aragatsotn, Gegharqunik, Vayots Dzor were refurbished, including: i. 15 regional centers of SESA, ii. 12 regional departments of SSSS, iii. 7 offices of SSRA/SSRD, iv. SLI central office. As a result of refurbishments the working conditions are essentially bettered, and also SSRAs offices were centralized under one roof as much as possible. Mentioned policy assured an important basis during SPAP AF for CSSS introduction from the viewpoint of providing SSs at unified receptions. 3. One of the most important activities of improving of working conditions of SS providing regional offices was the procurement of office furniture for refurbished 6 SSRAs/SSRDs, 15 EMCs/ERCs, 10 SSSS RDs. Many of mentioned offices were still using the furniture from Soviet Union period. A small quantity of furniture was procured also for MLSI. 49 Component 2: Labor and Employment (Actual amount is USD 1 012 335, 29, which makes 17,24 % of total disbursed amount) As a result of implementation the following outputs and developments are available:  The internet system “Gorts”, which regulates Employment sector, is operated by 51 regional centers of SESA thus assuring the input and use of data in a real regime; the exchange of data on vacancies and job-seekers between the offices; efficient supervision over implemented works; conduction of statistical analysis; and fulfillment of other functions of SESA;  There are highly-qualified specialists, who operate the information system “Gorts”;  The regional and central offices of SLI operate in good working conditions. The technical means, required for the organization of works, are available;  The organization of job-fairs is considered one of the most important activities of Employment sector supported by RoA State Budget. Within the period of 2007-2013, 37 job-fairs were organized in 15 cities of the republic, per 5 times in each, including in Yerevan, Gyumri and Vanadzor. About 1500 employers and 71.5 thousand job-seekers participated in those job-fairs.  In 2009-2010 mentioned 3 job clubs provided consultancy to about 950 visitors, 115 of which were employed and 123 were involved in professional training programs. At the end of the project in 17 regional centres of SES operated job clubs. All means acquired for the operation of job clubs are used according to the objective. At the beginning of the project trainings were organized for 55 employees of SESA on the operation of mentioned new employment technologies. In the premises, refurbished in the scope of CSSS introduction, the conditions required for the operation of job clubs are available and will be operated in parallel to the complete introduction of the system.  The CPOVRD of Gyumri, established under the RoA Government Decree of April 15, 2010, successfully operates being financed out of RoA State Budget proceeds. As of Project closing date, 16 people are employed by the Center, 11 of which work according to their specialty. Within 3.5 years of its operation the CPOVRD fulfilled professional work with 3 thousand people, the 847 of which within the period of July 1, 2012-July 1, 2013 (with 733 people they worked at home and with 114 at the center). Since the establishment, as a result of CPOVRD consultancy 43 disabled have been employed for the first time.  By 2013 the YPOC successfully operated being supported out of State Budget proceeds. Under the Decree N1549-Ն, dated December 13, 2012, the center was reformed into the “Professional Orientation Methodical Center” SNCO principally changing its functions. Currently 17 employees are employed at the center, 11 of which are specialized employees. The capacity and means acquired in the scope of SPAP are being completely used.  The SESA currently has 361 employees, 338 of which are highly-qualified civil servants. 43 people work at the Agency, 35 of which are civil servants, who fulfil professional work. According to the annual State Employment Program, the Agency fulfils 13 state projects. Besides, the SESA cooperates also with a number of international organizations and is currently implementing different projects with 5 such organizations. In the final phase of SPAP a new stage of reforms launched in the state employment regulation sector, the mission of which is to assure a new progress in the sector through the introduction of new flexible and active employment projects. The activities implemented within 2005-2013 in the scope of SPAP are important basis for the implementation of mentioned reforms.  For the perfection and improvement of Labor Legislation developed were conceptual approaches on the amendments to RoA Labor Legislation, submitted were a number of legislative drafts and expertise conclusions regarding a number of legislative drafts developed by the MLSI. Mentioned proposals were mainly applied by the MLSI. 50 Component 3: Modernization of Pension System (Actual amount is USD 326 540,11, which makes 5,56 % of total disbursed amount) In the course of the Project the outputs of this activity were used partially conditioned with legal and SSSS status changes, and also with launching of new pension reforms. Particularly, the majority of legislative and financial management documents and proposals were not applied as the RoA State Social Insurance Fund was reformed into the RoA State Social Security Service (the Statute and staff structure were approved under the RoA Government Decree N1531-Ն, dated December 27, 2007) operating in the management sector of RoA Ministry of Labor and Social Issues under the RoA President Order N305-Ն, dated December 24, 2007. The information system of electronic archive was not operated as SSSS regional department were not provided with channels, and the electronic archive had been developed as a WEB operated system. Besides, within 2011-2013, a new electronic pension IS was introduced in the SSSS, which efficiently fulfils the electronic archiving of documents. The proposals regulating archiving processes were not applied as unified archives for 4 SSs operating under one roof, were established in CSSRCs in the scope of CSSS introduction and consequently required were quite different approaches for archiving of documents. In addition, the personified reporting system is currently being operated by the SRC and this has significantly cut down the volume of SSSS paper documentation. It should be noted that some of outputs of implemented activities, for instance the proposals on improving the ITs, some proposals on inner regulation, training activities and printed materials have had their impact and have been used for a certain period of time. The voluminous and fundamental pension reforms, launched since 2010, resulted in a new situation and new solutions, which were partially implemented in the scope of SPAP AF. At the expense of initial Project’s savings ICT equipment were procured for SSSS, which are being operated according to the objective. Some activities of this Component were removed from the Procurement Plan associated with the process of reforms and new situations. Particularly, it was envisaged to develop a new IS for personified recording, however the function of operating mentioned system was planned to transfer to the SRC, which then was realized in the scope of pension reforms and the PRS was developed and operated out of RoA State Budget proceeds. Component 4: Improvement of Social Assistance System (Actual amount is USD 366 533.95, which makes the 6.24% of total disbursed amount)  In March-October 2006 “Economic Development and Research Center” NGO conducted household survey to a) evaluate social services provided to RoA public and define priority of the required services, b) evaluate needs of households, c) develop alternative ways of social assistance, d) adjustment of poverty evaluation formula of families. As a result following tree analytical reports were developed: 1. “Overall picture and classification of communities in Armenia”. 2. “Needy families and social services in Armenia”. 3. “Improvement of social assistance targeting in Armenia”, which includes policy recommendations for organizing family allowance system and improvement of poverty evaluation formula.  In 2007-2008 local and international consultants conducted works for evaluation of efficiency of targeted social programs and improvement of monitoring system (“Family Benefit” program (FBP)). 51 • International consultant Miroslav Beblav provided consulting services for improvement of FBP monitoring and evaluation efficiency. Consultant submitted 3 reports, which reflected: 1. International leading experience in the field of social systems programs implementation, monitoring and evaluation efficiency assessment based on evaluation of family neediness system. 2. Analysis of FBP monitoring and evaluation system and recommendations for system improvement. 3. Outputs of the seminar conducted for the presentation of FBP monitoring and evaluation system improvement results for corresponding departments of MLSI staff and social service regional offices employees. • In May 2007-June 2008, the consortium of local consultants “Avag Solutions” LLC and “Lisam” LLC provided consulting services for improvement of FBP monitoring and evaluation system. As a result 1. A report on “Analysis of current system of FBP” was submitted, 2. A methodical manual, named “Organization of FBP monitoring and evaluation activities” was developed for the professionals involved in monitoring and evaluation of social assistance programs, 3. Training was conducted for the staff involved in monitoring and evaluation. Based on the proposals made as a result of above mentioned surveys, on December 27, 2007 and January 15, 2009 RoA Government adopted N 1530-Ն and N 39-Ն Decrees aimed at clarification of targeting mechanisms of social assistance services. Under the mentioned decrees the points of some social groups and the income formula were changed. The outputs of mentioned consultancies were mainly used in the course of SPAP. As a result the FBP is considered one of the programs of social network in European and Central Asian territory (ECA), which have the best targeting. Component 5: Project Management, Monitoring and Evaluation (Actual amount is USD 1 154 172.06 which makes the 19.66 % of total disbursed amount) It should be noted that the capacity of all major social protection agencies (SESA, SSSS, Ministry of Labor and Social Issues (MLSI)) has been substantially improved under the Project. Social assistance benefits and pensions are paid on timely manner; targeting of social assistance benefits and services has also improved. Overall satisfaction of beneficiaries with the social service and benefit provision has increased. Recommendations to improve monitoring and evaluation of the Family Benefit Program and its targeting efficiency has been implemented; the Program has been judged as one of the best targeted social safety net programs in the ECA region. These achievements have led to better performance by the public employment, pension and social assistance agencies in providing services to the population. 52 2. THE OUTPUTS OF THE SECOND PHASE OF THE PROJECT Component 1: General Support to the Social Protection Sector (Actual amount is USD 3 808 743.6, which makes the 68 % of total disbursed amount). The following activities were implemented in the scope of Component 1 of Additional Financing: Working-out of social services joint package and structural-functional model of joint ystem of complex provision of social services (SSICJS) On the 22nd of July, 2011 a contract was signed with “Harmonic Society” NGO, who had submitted a bid together with the “International Fund for Social Services” of Switzerland. The works fulfilled within the period of July, 2011- September, 2012 and submitted 3 reports allowed the MLSI to make appropriate decisions and take measures for the possible directions and models of introduction of Complex (Integrated) Social Services System in Armenia and for the steps and required means to reach the latters. Various possible models for integration were submitted as a result of which 52 CSSRCs with 56 unified receptions will be established in Armenia. About 181 SS providing regional bodies and committees (55 SSRA/SSRDs, 51 EMC/ERCs, 51 SSRDs, 23 MSECs) will be unified under one roof at community level, will have unified receptions, unified informational environment and will operate according to unified methodology. Particularly, important is the institute for the conduction of a social case in the scope of which a complete assessment of social needs of beneficiaries will be carried out and a unified program for satisfying those needs will be developed and operated. One of the most important problems of CSSS is the formation of social network and the centralization of possible resources for solving of various problems that the vulnerable groups of population face. The following Decrees were adopted for the introduction of SSICJS: 1. On the 3rd of June, 2010 RoA Government adopted protocol N21 “On the approval of concept of organizing the process of provision of integrated social services”. 2. On the 23rd of December, 2010 RoA Government adopted protocol N 50 “On the approval of conceptual approaches of structural-functional model of joint system of integrated social services’ provision”. 3. On the 24th of March, 2011 RoA Government adopted protocol N11” On the approval of Plan-Schedule of activities of introduction of joint system of integrated social services’ provision”. 4. On the 26th of July, 2012 the RoA Government adopted a Decree (N 952 - Ն) “On approving the program of the introduction of integrated social services system in the Republic of Armenia”. The latter also stipulated the pilot project of social services integrated provision, which was implemented in Ararat marz. 5. In order to ensure the effectiveness of integrated social services pilot project, a special working group responsible for implementation of the pilot project was created by the order of the Minister of Labor and Social Issues (N 82, A / 1 - September 18, 2012), as well as executives were appointed to be responsible for 4 Complex Social Services Regional Centers in RoA Ararat marz. 6. On the 22nd of Mach, 2013 according to the N 44- A/1 order of the RoA Minister of LSI on “Designing of Complex Social Services Regional Centers (CSSRC) and appointment of coordinators for the CSSRCs” other 15 CSSRSs were designed and coordinators were appointed. The 19 centers to be refurbished and furnished in the scope of SPAP Additional Financing by May 31, 2013 were simultaneously being involved in the introduction process of Complex Social Services System. 53 The introduction of CSSS is one of the priority issues for the RoA Government and in the scope of SPAP AF actually 19 CSSRC were established and a new methodology and tools were operated for the provision of CSS. The mentioned phase of SPAP, perhaps, is the initial phase of CSSS introduction in the course of which the new system still need a period to be corrected and perfected. The deadline of the pilot project was extended by the RoA Government for October, 2013 and based on the outputs of the latter the RoA Government will adopt a final Decree on the final CSSS model in Armenia. Preparation of design-cost estimate documents of regional (integrated) services’ offices of social protection field and implementation of technical and author's supervision In August 2011 contracts were signed with 2 companies for the fulfilment of works. The works on preparing of design-cost estimate documents and also interior design projects were fulfilled by the latters in 25 CSSRC (Ararat, Masis, Vedi, Artashat, Armavir, Baghramyan, Tshambarak, Noyemberyan, Charentsavan, Nairi, Maralik, Ashotsq, Avan, Norq-Marash, Nubarashen, Ajapnyak, Davitashen, Erebuni, Vardenis, Talin, Aparan, Tsaghkahovit, Ashtarak, Vanadzor and Akhuryan). Only in 19 of the mentioned 25 designed centers, it was possible to implement refurbishment works, because it was decided to postpone the refurbishment works of the other six centers (CSSRCs of Vanadzor, Akhuryan, Talin, Aparan, Tsagkahovit and Ashtarak) due to the lack of time and other resources. It is notable that during the refurbishment of those 19 centers, the client, the Contractor, as well as the designing and supervising organizations raised several issues, which have led to changes in the projects and budget estimates. Those were mainly connected to the ambiguities in the model of complex social services (CSS); to the design and construction of large-scale projects in a short period of time; to the depreciation of buildings and the resultant numerous problems arising in the process of construction. Within one year (June 1, 2012- May 31, 2013), over 25 Centers were designed and 19 were refurbished, which though had to be implemented within three years. The design and construction works were accepted and launched by the request of WB only after the Decrees of community councils and the Government were adopted on area allocations, as well as after the Decree of the RoA Government on the introduction of the complex social services system (except for the three pilot centers of Ararat Marz – Ararat, Masis and Vedi, tenders for which started earlier). As of May 31, 2013, the design works of all 25 centers, as well as the technical and author’s supervision of refurbishment works in 19 centers were completed. It should be noted that one of the most important achievements of the Project is the making of decision on the right to unlimited and free of charge use of areas of CSSRCs. Before the implementation of reforms, different agencies of SP system occupied the offices, in general, under temporary renting contracts, which sometimes resulted in various obstacles. The right to unlimited and free use of areas assures stable basis for the reforms of CSSS introduction and for the long-term impact of implemented contributions. 54 Implementation of refurbishment of regional (integrated) services’ offices of social protection field In 2010-2012, a research on possible locations and conditions of regional bodies and centers of SP sector was conducted. Proposals on the problems of organization of works and on suitable centers that are subject to refurbishment were presented; as well as visits to the appropriate centers were carried out. At the same time, with the support of the Ministry of Territorial Administration and cooperation of local authorities, works on bringing the centers under one roof and positioning of integrated receptions had being implemented. 17 contracts had been concluded with different companies, who implemented the refurbishment works of 19 complex social services regional centers (CSSRC). The mentioned centers were located in Ararat, Masis, Vedi, Artashat, Armavir, Baghramyan, Tchambarak, Vardenis, Noyemberyan, Charentsavan, Nairi, Maralik and Ashocq, as well as in Avan, Norq Marash, Nubarashen, Ajapnyak, Davitashen and Erebuni administrative districts of Yerevan. As of May 31, 2013, all the refurbishment works of 19 complex social services centers were completed. Provision of information and telecommunication systems for SSICJS The following works were fulfilled for the implementation of envisaged activities: Procurement of SSICJS network technical equipment The specifications of servers required for the operation of centralized informational system were worked out. However, the RoA Government provided two powerful servers to the MLSI, as a result of which this activity was removed from the procurement plan and the procurement amount was reallocated. The procured servers currently ensure the smooth operation of ICT systems of SP sector. Procurement of computer and other technical equipment for regional (integrated) services’ offices of social protection sector Preliminary estimations for technical equipment required for ISS unified receptions were prepared. In the framework of development of the model “Harmonik Society” NGO implemented inventorization of technical conditions of currently functioning regional centers providing SSs (SSRBs, EMCs/ERCs, SSSS RC, MSECs). Based on that, it was possible to calculate the cost of and the quantity of the required technical equipment for Complex Social Services Regional centers (CSSRC) to be established. In order to obtain digital queuing machines and electronic information terminals, a contract was signed with “Lime Tech” LLC in March 27, 2013. As of May 31, 2013 all the equipment required for 25 Centers were delivered by the latter. In order to obtain Household Appliances, TV sets and Conditioners, 3 contracts were signed with “Aray” LLC in February 6, 2013. Contract terms were prolonged till May 20, 2013, as the installation of the latter was directly related to the completion of refurbishment works. During the mentioned period, the delivery and installation of the equipment of 19 CSSRCs were made. 55 In order to obtain computer and related technical equipment, contracts were signed with “Ino-Technology LLC (January 31, 2013) and “Fine” LLC (February 1, 2013). Significant part of computers for Masis, Vedi and Ararat centers were delivered in March 26, 2013; while the other part of the equipment was delivered in the period of April-May, 2013. The delivery of computer equipment for the other 22 centers was delivered in few stages and completed by “Fine” LLC in May. As of December 19, 2012, “Microring” company had already delivered the scanners and multifunctional devices required for SSSS. As a result all technical conditions required for the operation of CSSRCs are assured in order to fulfil the works more efficiently and to increase the quality of provided SSs through up- to-date ICT means. The queuing machines allow not only regulating the service provided to citizens, but also carrying out some analysis and calculations regarding to the number of visitors, provided services, the time spent on the provision of services, etc. And the electronic terminals allow visitors to solve their problems through self-service. SSCPJS software assuring activities Contracts were signed with “Harmony” IT and Education Development Fund in January 31. As of May 15, 2013, the Company had completed all works. As a result a new IS, which integrates over 18 different information systems, is operating in CSSRCs. Through the latter implemented are also the unified acceptance and addressing of applications, recording of data required for the conduction of the social case, formation of social passport. Mentioned information system assures stable basis for the operation within integrated informational environment according to unified methodology in CSSRCs and this is a key prerequisite for the provision of complex SSs. Procurement of furniture for CSSRCs The furnishing works for all refurbished 19 Centers were carried out within December, 2012-May, 2013 under signed 7 delivery contracts. As a result, all furniture conditions are assured in CSSRCs for organization of reception of citizens at unified receptions, delivering of high-quality services, and also for working, rest and eating of employees. *** The two activities envisaged under mentioned component aimed at raising public awareness on SSCPJS and training of specialists were removed from the procurement plan based on the assignment of the Prime Minister of RoA. The awareness campaign is envisaged to be implemented in the scope of upcoming projects in parallel to the introduction of CSSS. The training of specialists of established CSSRCs were conducted under the Component 4 of the Project. Component 2: Labor and Employment (Actual amount is USD 28 839.39 which makes 0,51 % of total disbursed amount) The fundamental changes envisaged in Labor and Employment sector lead to resigning of the main part of activities envisaged under SPAP AF. The funds were reallocated to the introduction of CSS system. This was implemented according to the RoA Prime-Minister's assignment. Particularly, 1. The contracts signed for the implementation of activities “Study of RoA labor market, including the assessment of possibilities of youth employment and integration of disabled to employment” and “Situational study of labor and salary sphere” were terminated. 56 The following activities were removed from the Procurement Plan: 2. “SESA (of MLSI of RoA) capacity building for organization of and control over short-term professional trainings for job-seekers and unemployed”, 3. “Training of agricultural and market-oriented professions to job-seekers with rural inhabitation”, 4. “Consultancy and organization of practical training for unemployed having profession but no record of professional work”, 5. “Training and re-training of cadres of labor and employment sector”, 6. “Refurbishment of one regional representation office of "Youth Professional Orientation Center" SNCO”, 7. “Procurement of furniture and computer equipment for one regional representation office of Youth Orientation Center”, 8. “Training of staff of one regional representation office of Youth Orientation Center”. Last 3 activities were removed from the procurement plan, because, according to RoA Government Decree N 1549-Ն, dated December 13, 2012, on “Renaming the “Youth Professional Orientation Center” SNCO” and making amendments to the RoA Government Decree N 1915-Ն, dated December 14, 2006” the “Youth Professional Orientation Center” SNCO was renamed as “Methodological Center for Youth Orientation” SNCO, which should not have regional representation offices. 9. “Development of system for monitoring and evaluation (MES) of effectiveness of the projects implemented in the employment sector”, 10. “Development and introduction of electronic system for monitoring and evaluation of effectiveness of the projects implemented in the employment sector”. These 2 activities were removed from the PP and the fund was reallocated between the “Development of a system of monitoring and evaluation of efficiency of social services” and “Development and introduction of electronic system of monitoring and evaluation of efficiency of social services” activities. This change is due to the new approach of the MLSI of the RoA and the adoption of new conceptual documents in the field of social services, according to which a unified system of social services Monitoring and Evaluation of Effectiveness (MES) should be created in Social Protection sector. 11. “Modernization of “Gorts” internet system fulfilling the functions of the RoA Employment sector” 12. “Development and installation of a new information system of State Labor Inspectorate of MLSI of RoA”. These 2 activities were removed from the procurement plan due to the development of new strategies and approaches in Employment sector and in the course of inspection reforms. In the current stage of reforms, when not all problems and approaches are clarified yet, it is inexpedient to implement the procurement of the creation or modernization of IT tools. 13. “Printing of information materials for Labor and Employment sector”. This activity was removed from the procurement plan as active reforms were taking place in Labor and Employment sector, which though have not been completed yet, and the elucidation of the latter would be a waste of time. In the scope of Component 2 of SPAP AF actually implemented are 2 activities: 1. “Development and introduction of electronic testing system for Youth Professional Orientation and Professional Career”, 2. “Preparation of 10 films about specialties for the youth”. 57 The outputs of mentioned 2 procurements are efficiently used by the “Youth Professional Orientation Center” SNCO (currently “Methodological Center for Youth Orientation” SNCO). It should be noted that the introduction of CSSS has had a great impact on the capacity of Employment sector, as currently the needs of job-seekers are completely assessed and a complex social program is developed and implemented for satisfying the latters. It should also be recorded that the problem of employment of population still remains one of priority issues for Armenia. Component 3: Modernization of Pension Sector (Actual amount is USD 824 150,02, which makes 14.71 % of total disbursed amount) Under this component the AF supported the Government’s efforts to improve the administration of the pension system, (in particular as it relates to reporting and record-keeping for unified revenue collection) (the function of the SRC) and state pensions administration (function of the SSSS)). The following activities were initially included in the scope of AF activities: (i) development and installation of software and provision of IT equipment for personified reporting (registry) and record-keeping system at the SRC; (ii) development/improvement of the MLSI’s software and provision of IT equipment for pension determination and 'pay-as-you-go (PAYG)' ( payment of taxes during getting the salary) database management; (iii) increasing the quality of customer services in SSSS; and (iv) training the staff of MLSI, SSSS and SRC engaged in new processes. The development and installation of software of the inner portal of SSSS, the procurement of hardware, related digital equipment and furniture, training and study tours were supported under SPAP AF. A number of activities under this Component were also removed from the Procurement Plan for various reasons, and new activities were added to the same Component. Particularly, the following activities were removed from the Procurement Plan: Development and installation of software packages (including the ones, ensuring data exchange between Personalized Reporting (registry) System and other informational systems) assuring the introduction and smooth operation of Personalized Reporting (Registry) System The National Assembly of RoA approved the Pension Reform Package in December 2010. By the help of the clients - MLSI and State Revenue Committee, in support with USAID/PALM - before and after the approval of Reform Package, works were carried out towards the development of the appropriate ToR. Since this activity was planned to be implemented through international tender, which presupposed a quite long procedure and since the system would have to be introduced by the January, 2013, RoA Government made a decision according to which the project had to be implemented by the means of the State Budget in order to make the entire implementation process more effective and fast. Corresponding resources were allocated from the State Budget for the implementation of current activity and the fund envisaged under SPAP AF was reallocated. On the 24th of September, 2011 a contract was signed with “IU-network” company which was selected through a tender to develop a ToR for the implementation of “Development and installation of software packages (including the ones, ensuring data exchange between Personalized Reporting (registry) System and other informational systems) assuring the introduction and smooth operation of Personalized Reporting (Registry) System” activity in the State Revenue Committee. The Company had already completed the implementation of the works and the system is already operating. 58 Procurement of computer and related digital equipment, as well as licensed software packages, assuring the operation of the database of the Personalized Reporting System and Pension Security System. Corresponding resources were allocated from the RoA State Budget for the implementation of this activity. As a result, the activity was removed from the procurement plan, and the fund was reallocated. Introduction of Database of Pension Security The draft of appropriate ToR was developed, which was discussed by the Working Group. However, resources were allocated from the State Budget for the implementation of this activity and the fund envisaged under SPAP AF was reallocated. “Electronic pension” system was developed and introduced and is being efficiently operated by SSSS ensuring the implementation of functions of State Pension System. In the scope of Component 3 of SPAP AF the following procurements were made: Completion of reforms of distributive component of pension system Within January-May, 2013 as a result of 3 reports submitted by “Ameria” company, the international practice, the main steps and directions for reforms, and also draft legal amendments were presented for the upcoming reforms of distributive component of pension system. Procurement of computer and related digital equipment, as well as licensed software packages, ensuring the operation of the unified database of Pension Security System In October 31, 2012 a contract was signed with “Fine” LLC, which had won the tender. As of January 25, 2013, all computers, related digital equipment, as well as the licensed software were delivered by “Fine” LLC. As a result the SSSS is completely provided with necessary technical means in order to efficiently operate the newly introduced “Electronic Pension” information system. Implementation of Training Programs within the frameworks of Pension Reforms Study visits of corresponding employees of SSSS and MLSI were regularly carried out, including to Moscow, Georgia, Lithuania, Brazil, and so on for the exchange of experience and for conference participation. Large-scale training activities were fulfilled in 2013 as a result of which about 290 employees of SSRDs were trained for the operation of “Electronic Pension” IS. As a result, the system is efficiently operating. Completion of current Personalized Reporting System database and its involvement in the new database of pension security system A contract was signed with the winner company “Nathan Ass. Armenia” in September 28, 2012. The final report was submitted by the Company in March 28, 2013. As a result of provided services, the problems existing in the IS were analysed, proposals were developed to correct the latters and more than 100 000 cases were inputted in the IS. Mentioned services allow inputting into the newly introduced «Electronic Pension» IS more accurate and complete data. Development of structural-functional model of Subsystem assuring the automatic checking of mandatory social security (insurance) payments made within the period of 1992-2004 It was necessary to develop a structural–functional model for the subsystem in order to assure the centralized and automatic input into the State Security database and also the use of the 59 data on insurers and social payments made by them within the period of 1992-2004, which existed in SSRDs. For the implementation of this activity a contract was concluded with freelance consultant Harutyun Kharatyan in August 10, 2012. The service was fully provided and completed in January 18, 2013. 3 reports were submitted, as a result of which the appropriate ToR was made for creation of required Information Subsystem. Development and installation of Subsystem assuring the automatic checking of mandatory social security (insurance) payments made within the period of 1992-2004 In order to make automatic checking of mandatory social security payments made by insurers and insured people in the scope of unified database of State Pension system and in order to exempt beneficiaries from submitting appropriate documents a contract was concluded with “DevelopWay” CJSC in March 21, 2013 for the development of corresponding IS subsystem. Works were fully completed on May 8. As a result, in the scope of newly developed “Electronic Pension” system operated was the new subsystem assuring the automatic checking of mandatory social security (insurance) payments made within the period of 1992-2004. Introduction of inner information portal of State Social Security Service In order to increase the efficiency of State Security system operation, to increase the working efficiency and transparency of SSSS employees, to regulate the information flows regarding SSSS operation or arisen in the course of it, to present the data in an available way, to assure an efficient cooperation between different agencies of the system and employees (to encourage team work), to increase the involvement of employees in decision-making process, to use up-to-date training methods for them, the appropriate inner portal was introduced in SSSS within the period of December, 2012 and May, 2013. It is used according to the objective granting the opportunity to use the existing inner resources in a real regime. Procurement of furniture for SSSS For the procurement of furniture for SSSS and its regional offices, a contract was signed with the winner of the tender - “Gladzor” LLC on April 11, which supplied the furniture for SSSS and its regional offices within 45 days. As a result, in the SSSS the conference hall was furnished, a working part for the acceptance of applications was created, and the SSSS and its regional departments' employees were provided with office tables and chairs. Component 4: Improvement of Social Assistance System (Actual amount is USD 314 541,95 which makes 5,62 % of total disbursed amount) Refurbishment of training premises of “Nork” Information-Analytical Center for ensuring the preparing and training of cadres of required number A contract was signed with “Khapartshin” LLC for the implementation of this activity on May 25, 2012. The refurbishment works were completed in August 29, 2012. As a result, 2 training rooms are available in «Nork» center, which serve for the training of the specialists of the sector. 60 Assuring of technical conditions for preparing and training the required number of cadres (MLSI capacity building) In 2012, contracts were signed with «Ino-Technology» LLC, «Zorashen» LLC and «Consell» companies. As a result, computer equipment, furniture and air-conditioners were procured and supplied. Currently, there are furnished and comfortable training rooms in «Nork» center and National Institute of Labor and Social Research, which serve for the training and re- training of specialists of the sector. Preparation of professional job-descriptions of social work for the introduction of the institute of "Social Worker", development of unified criteria for training and re- training of specialists of the Sector According to the contract signed on August 1, 2012, the Consulting Company implemented the works envisaged under the ToR. 3 reports were submitted on the outputs of implemented works. As a result, studied were the peculiarities of implementing of social work in SP sector, existing problems, the professional compliance and skills of specialists, contractual regulations and other issues. Proposals were submitted on the professional job-descriptions of social work and on the unified criteria for training and re-training of cadres, the outputs of which were used for preparing of staff lists, for legal regulation, clarification and regulation of the scope of professional work of social workers, and also for preparing of training and re-training curriculums. Training and retraining of employees for introducing the institute of “Social Worker” On February 4, 2013, a contract was signed with “Harmonic Society” NGO for implementation of the training and retraining, based on which 350 specialists of established 19 CSSRCs were trained. As a result, the specialists are able to operate the CSSRCs according to the new methodology and new tools. This assures an important basis for the efficiency of CSSS reforms. Development of Structural- Functional model of monitoring and evaluation system of social services In January, 2012 according to the order of RoA Minister of Labor and Social Issues N10A/1 a concept on “Introduction of a system for monitoring and evaluation of effectiveness of the projects implemented in the Social Protection sector” was approved. The plan-schedule of events for the development of Monitoring and Evaluation system was approved by the Minister of Labor and Social Issues of RoA. “Business Consult” CJSC was selected during the tender and a contract on development and testing of Structural- Functional model of monitoring and evaluation system of social services, provided by the institutions of RoA MLSI, was signed in December 24, 2012. The Consultant submitted 3 reports on the implementation of works. As a result, based on the international practice, a developed and tested methodology for MEE of projects of SP sector is available, and the specialists, who will use it, are trained. The latters, in general, are the specialists of the Monitoring Department of MLSI, who will ensure the operation of mentioned methodology. Computer equipment for the needs of Ministry of Labor and Social Issues and Medical-Social Expertise Agency of the RoA 61 A contract was signed with the winner of the tender, “Lans” LLC, on May 7, 2013 for 25 days period. The company has delivered all the corresponding computer equipment, which are used according to the objective thus increasing working efficiency of specialists of MLSI and MSEA. *** In the scope of this component of SPAP AF as well initially envisaged some activities were not implemented due to various reasons and were removed from the procurement plan. Particularly, “Development and introduction of electronic subsystem of monitoring and evaluation of efficiency of social services” activity was removed from the procurement plan as it had to be implemented after implementation of the “Development of a system of monitoring and evaluation of efficiency of social services” activity the deadline of which corresponded to the deadline of the Additional Financing of SPAP. Printing of information material, included the printing aimed at awareness of beneficiaries of the sector” activity was removed from the procurement plan, as it was desirable to complete the on-going reforms, then to implement the propaganda of information. Otherwise, the dissemination of printed materials would be done untimely. “Implementation of reforms in Family Benefit policy” activity was removed from the procurement plan according to the assignment of Prime Minister of RoA. “Use of RoA MLSI Financial Management Information Systems for determining beneficiaries’ eligibility” activity was removed from the procurement plan according to the assignment of Prime Minister of RoA. Component 5: Project Management, Monitoring and Evaluation (Actual amount is USD 624 741,20, which makes 11,15 % of total disbursed amount) In the scope of this component of ensured was the professional implementation of the AF using the existing mechanisms of project management. In the scope of the component implemented were: 1. Fulfilment of consulting services for Project management, including consulting services by the project coordinator, expert on social protection, monitoring and evaluation issues, IT specialist, engineer, translator and administrative assistant (a PCU was established by mentioned consultants). Because of the difficulties of the project implementation, there was a need to include one more engineer and translator. In October, 2012, contracts were signed with the consultants that had won the tender for the period till May 31, 2013. 2 procurements of technical means were made for the implementation of the Project. 1. Procurement of computer equipment for PCU. 2. Procurement of two vehicles for PIU and PCU. 3. Annual and final Monitoring and Evaluation of Project for the period of 2010-2013. 4. Project audit for the period of 2010-2013. All the activities fulfilled in the scope of the Component served for the efficient implementation of the Project. 62 Annex 8: Comments of Co-financiers and Other Partners/Stakeholders n/a 63 Annex 9: List of Supporting Documents and Interviews • Documents o SPAP Monitoring & Evaluation Report, published by contracted consulting firm, Yerevan 2013 o Operational reports and data from SESA “Gorts” IT System (on request from SESA management) o Final Project Report by Ministry of Labor and social Issues o Project Implementation Plan o Aide Memoires, Back-to-Office Reports, and Implementation Status Reports o Project Appraisal Document for Armenia: Social Protection Administration Project (SPAP) dated May 10, 2004 o Project Paper for Armenia: Additional Financing to the Social Protection Administration Project. • Interviews o ICR mission June 2013: Mr. Jean-Michel Happi (ECCAR, Country Manager), Ms. Sona Harutyunyan (RA MLSI, Deputy Minister), Ms. Susanna Hayrapetyan (ECSH1, TTL 2007-2008), Mr. Ara Petrossyan (RA MLSI, Deputy Minister), Mr. Artak Sargsyan (RA MLSI, Head of SPAP PIU) o ICR mission September 2013: Mr. Vanik Abasyan (“NORK” Information Center), Mr. Arthur Aivazov (UNICEF Armenia), Mr. Artem Asatryan (RA MLSI, Minister), Ms. Gohar Aslanyan (YPOC Yerevan), Mr. Edgar Avetyan (RA FFPMC), Mr. Jean-Michel Happi (ECCAR, Country Manager), Mr. Artak Magasaryan (Head of SESA), Ms. Astghik Minasyan (RA MLSI, Head of Social Assistance Department), Mr. Hayk Muradyan (RA MLSI, Head of M&E Department), Mr. Artak Sargsyan (RA MLSI, Head of SPAP PIU), Mr. Smbat Sayiyan (RA SSSS, Deputy Chairman), Mr. Armen Shahnazaryan (RA FFMPC), Mr. Tachat Stepanyan (USAID Armenia), as well as the General Manager and Project Coordinator of the consulting firm contracted for M&E o Implementation team at WB HQ / in between missions: Mr. Ivan Drabek (ECSH3, TTL 2011-2013), Aleks Havhannisyan ( ECSH3 Senior Consultant), Mr. Lire Ersado (AFTP2, TTL 2008-2011), Ms. Benedicta T. Oliveros-Miranda (ECSO2, Procurement Analyst), Mr. Garik Sergeyan (ECSO3, ETC, Financial Management), • Field Visits o ISSCs at Masis Marz, Ararat Municipality, Avan and Nor Nork Sub- Municipalities of Yerevan o Professional Rehabilitation Center Gyumri o SESA Offices in Yerevan and Gyumri o Youth Professional Orientation Center Yerevan 64 Annex 10: Results Framework for the SPAP and AF Baseline Baseline Target for Actual Comment (at (at Year 3 (at inception Approval Approval of of SPAP) of AF) AF in PP) Outcome Indicators Evidence of: Eligible beneficiaries receiving timely pensions 50.4% 82% 100% 100% For clarity the indicator is reported as split payments between “timely” and “fully” since both are tracked separately. Eligible beneficiaries receiving fully pensions 50.4% 82% 100% 98.2% We report here the indicator as expressed payments in the “arrangement for results monitoring” section of the AF PP. In the LA and other parts of the AF the indicator were “Eligible beneficiaries receiving accurate and timely pensions”. Eligible beneficiaries receiving timely social 50.4% 86.5% 100% 93.2% For clarity the indicator is reported as split assistance benefits between “timely” and “fully” since both Eligible beneficiaries receiving fully social 50.4% 86.5% 100% 95.3% were tracked separately. We report here assistance benefits the indicator as expressed in the “arrangement for results monitoring” section of the AF PP. In other parts of the paper the indicator was “Eligible beneficiaries receiving accurate and timely pensions”. Better job placement, especially of disabled and 7% -- / -- n/a 16% Overall job-placement rate. youth Indicator dropped in AF PP, but recorded for ICR accountability purposes Improved user satisfaction None 65% 90% 95.8% In PP, wording of indicator changed to: pensioners “Improved user satisfaction regarding satisfied; service delivery related to pensions 97.4% social payments, employment services, and assistance social assistance” beneficiaries satisfied; 81.7% job seekers satisfied 65 Baseline Baseline Target for Actual Comment (at (at Year 3 (at inception Approval Approval of of SPAP) of AF) AF in PP) Results Indicators for Each Component Component 1: General Support to the Social Protection System (1-1) National social information system is in place None None 100% Cancelled In PP, wording of indicator changed to: “A national social information system is in place, which allows for coordinated targeting and delivery of social services” (1-2) Appropriate information is transmitted in None Partial 100% Partially Partially Completed: Information system Completed exchange is assured in current system via (1-3) Agency programs are adapted to national None None 100% Partially central structure, but development of a information network Completed unified “Social Data Bank” still outstanding. (1-4) Introduction of a joint package of social - None 100% Completed ISSC functional model defined for pilot. services and standards “Social worker” qualification specified. ISSC model to be decided on in SPAP2. Indicator introduced in the AF PP. Last reported as “Development of structural- functional model of joint system of social services integrated provision (SSICJS) and joint package of social services standards” (ISR 19, May 2013) Component 2: Labor Market and Employment (2-1) Information on labor supply and demand None System Modernizati Completed In PP, wording of indicator changed to available on continuous basis tested, on of system via “Gorts” include: “Active labor market services personnel completed; system selectively offered, and based on the labor trained 120 more market information systems and results specialists monitoring; including the targeting of trained specific measures at the disabled and young workers” (2-2) ALMPs selectively offered and results None -- / -- In PP, indicator merged with above monitored indicator (2-3) Systematic record keeping on system None -- / -- In PP, indicator merged with above participants and flow of funds indicator (2-4) Increase participation of youth and rural job- - None 35% not tracked Indicator introduced in the AF PP, only seekers in state employment services tracked for youths in Yerevan Component 3: Support to the Pension 66 Baseline Baseline Target for Actual Comment (at (at Year 3 (at inception Approval Approval of of SPAP) of AF) AF in PP) Administration (3-1) Improved collections in the pension system None Partial 100% 86% In PP, wording of indicator changed to (May 2013, ISR include: “and comprehensive record- 19) keeping of system participants (central register) is established and fully operational” (3-2) Central register operational None Partial 100% Completed In PP, indicator merged with above indicator (3-3) Business processes and institutional None Partial 100% Completed In PP, wording of indicator changed to: procedures are operational “Business processes and institutional procedures are functioning as intended” (3-4) Participants are aware of their rights and None 26.6% 34% In PP, wording of indicator changed to: responsibilities aware of Rights “System participants are aware of their rights; rights and responsibilities” 26.4% 36% aware of Responsibilities responsibi (May 2013, ISR lities 19) (3-5) Actuarial forecasting applied in knowledge- None Not tracked In PP, wording of indicator changed to: based planning “Policy analysis is used for policy formulation and evaluation functions, and better management of resources (3-6) Development and installation of software for - None 100% Completed (not PRK operational, but not implemented introduction of personified reporting and record- in SPAP) under SPAP. keeping and its smooth operation at the State Indicator introduced in the AF PP Revenue Committee (3-7) Backlog of unprocessed personified data - None 0% Completed MLSI approved work of contractor in eliminated March 2013 (Final M&E Report) Indicator introduced in the AF PP Component 4: Improvement of Social Assistance (4-1) Targeting mechanism applied by all social None Yes Yes Yes In PP, wording of indicator changed to: welfare offices “New approaches to targeting are being applied by all social welfare offices” (4-2) MIS used to determine eligibility None Partial Yes Yes In PP, wording of indicator changed to: “The management information system is being used to determine eligibility” 67 Baseline Baseline Target for Actual Comment (at (at Year 3 (at inception Approval Approval of of SPAP) of AF) AF in PP) (4-3) Monitoring and evaluation results used in None Yes Yes Not tracked In PP, wording of indicator changed to: knowledge-based policy formulation “Monitoring and evaluation results are being used in policy formulation” (4-4) Participants aware of rights and 21.5% 34.0% 46.9% Indicator dropped in PP but continuously responsibilities Rights Rights Rights included in M&E reports 15.5% 37.1% 45.03% Respon- Respon- Responsibilities sibilities sibilities (2012/2013) (2005) (2010/11) (4-5) Improvement of Family Benefit Program 50% 75% n/a Indicator introduced in the AF PP but not coverage of the poor through enhanced awareness tracked in M&E Report. Only “Rights and campaign, particularly among the very poor Responsibilities” of beneficiaries tracked as indicator Component 5: Project Management, Monitoring and Evaluation (5-1) Project implementation according to schedule PIP PIP Yes Project terminated on schedule. Borrower Report received. (5-2) Fiduciary obligations fulfilled FMS FMS Yes FM and Procurement raised no objections 68 43°E To T’bilisi To T’bilisi 46°E 47°E 44°E GEOR GI A G E OR GI A To Borjomi L e s sT Tashir ashir er C Alaverdi To Gäncä auc Ku asu ra Mingechevir 41°N 41°N Stepanavan s M To Gäncä Reservoir ou LORRI nt SHIRAK ai Ijevan n s Gyumri Vanadzor Vanadzor AV U S H T AV To Kars Dilijan Artik Artik Artsvashen Artsvashen Karmir Karmir Aragats Sevan To pa Gäncä Ar (4090 m) ARMENIA Hrazdan A R A G AT S O T N Hraz d a n KO Gavar Lake Ashtarak T UR KEY Sevan TA GEGHARK’UNIK’ YK YEREVAN YEREVAN Vardenis Vardenis ’ Aras AV I R A R M AV YEREVAN YEREVAN Armavir Armavir Martuni Martuni nge ni s Ra 40°N 40°N r de Artashat Artashat Va A R A R AT This map was produced by the Map Design Unit of The World Bank. VAY O T S ’ VAY Ar Ararat as The boundaries, colors, denominations and any other information shown on this map do not imply, on the part of The World Bank Jermuk Jermuk Group, any judgment on the legal status of any territory, or any DZOR endorsement or acceptance of such boundaries. Yegegnadzor Yegegnadzor 43°E 44°E Arpa AZ E RB AIJ AN Vaik Vaik To Naxçivan Angekhakot ARM E NI A To Goris Qubadli Vo o tan r Z To Naxçivan A To ra a SELECTED CITIES AND TOWNS Qubadli s ng PROVINCE (MARZ) CAPITALS A ZERBA IJA N SYUNIK’ ez I SLA MIC REPUBLIC Kapan ur NATIONAL CAPITAL RIVERS O F IRA N Ra ng MAIN ROADS 0 10 20 30 40 50 Kilometers s e RAILROADS Ara 39°N SEPTEMBER 2004 IBRD 33364 PROVINCE (MARZ) BOUNDARIES 0 10 20 30 Miles To Füzili INTERNATIONAL BOUNDARIES Megri 45°E To Ordubad 47°E