Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) Report Number: ICRR0021703 1. Project Data Project ID Project Name P146015 Nonformal Appr. Training Edu Jobs in Afg Country Practice Area(Lead) Afghanistan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-16354 30-Dec-2018 5,529,045.80 Bank Approval Date Closing Date (Actual) 24-Mar-2014 31-Oct-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 15,000,000.00 15,000,000.00 Revised Commitment 5,529,045.80 5,529,045.80 Actual 5,529,045.80 5,529,045.80 Prepared by Reviewed by ICR Review Coordinator Group Anthony Martin Tyrrell Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project objective as stated in the Financing Agreement (p. 4) of April 11, 2014 was to increase the potential for employment and higher earnings of targeted young Afghan women and men in rural and semi- urban areas through non-formal skills training. Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) The project will be assessed against the two separate parts of the objective as follows: (i) to increase the potential for employment of targeted young Afghan women and men in rural and semi-urban areas through non-formal skills training; and(ii) to increase the potential for higher earnings of targeted young Afghan women and men in rural and semi-urban areas through non-formal skills training. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1: Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees (Planned US$7.7 million; Actual US$1.05 million). The component was designed to induce non-formal training providers (TPs) to improve the quality of their training and the placement of their graduates. Initially, an incentive of US$100 was to be granted to non-formal TPs for each graduate who was certified or found a job within six months of graduation or started a business, but the incentives were modified in the Project Implementation Manual, with US$50 being allocated upon certification (passing the National Occupational Skills Standards (NOSS) test) and US$50 upon employment or self-employment. TPs were to be selected from across the country, with some preference given to those who offered courses exclusively for women or had a high percentage of women in their regular training programs. Teachers were to be trained at the Technical Teachers’ Training Institute (TTTI) (which was to be established under the Bank-financed Afghanistan Second Skills Development Project (ASDP II, P132742, US$55 million, 2013-2021)) and contracted to provide tailored short-term courses at some of the non-formal TPs. Courses were to be aligned with the NOSS test, which was to be used for graduates’ certification. Component 2: Project Management, Capacity Building of Ministry of Labor, Social Affairs and Martyred and Disabled (MOLSAMD), and Monitoring and Evaluation (M&E) (Planned US$3.6 million; Actual US$ 2.04 million). This component was designed to build MOLSAMD capacities in procurement, financial management, Labor Market Information Systems (LMIS), and monitoring. Capacities were also to be built at the National Skills Development Program (NSDP, within MOLSAMD), a national program responsible for skills training with multiple projects (including ASDP II). Its overall management capacity was to be built (including in the areas of channeling financial incentives, fiduciary, and safeguards), as were its capacities in M&E, including the sustained monitoring of project indicators, the conduct of third-party validation exercises, and the design and implementation of randomized experiments for evaluating impact. The component was also to cover project management costs including salaries, incremental operating costs, and expenses for M&E. Component 3: Piloting Entrepreneurship among Unskilled and Illiterate Young Afghans (Planned US$ 3.7million; Actual US$2.43 million). There were three sub-components: Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015)  Sub-Component 3.1 (Encouraging Rural Employment) was designed to provide customized training in hard and/or soft skills as well as functional literacy to members of Savings Groups (SGs) that had been set up in five provinces as part of the Bank-financed Afghanistan Rural Enterprise Development Project (AREDP, P110407, US$30 million, 2010-2018). The training was to be customized based on the findings of a baseline survey. The possibility was also held out that interested SG members would receive guidance on establishing small businesses and links to finance programs. The TPs (providing the customized training) were to receive US$100 for each trainee who was certified or found a job within six months of graduation or started a business.  Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) was designed to provide grants of US$500 on a competitive basis to 1,890 individuals of three provinces - Kabul, Nangarhar and Balkh - to help them start their own business, and to 630 individuals to expand an existing business. The grants were to be targeted at females and males aged 18-35 years with an entrepreneurial mindset who were mostly illiterate and unemployed/underemployed. The applicants were to develop business plans with assistance from MOLSAMD. The project was to provide successful candidates with counselling and mentoring support during implementation of their plans, as needed.  Sub-Component 3.3 (Feasibility Study on Apprenticeship Program) was to carry out a feasibility study to assess whether there was a need for short-term practical apprenticeship training and how such training might be piloted using various options. If the study found that piloting was viable, then the most promising option was to be finalized and implemented in selected provinces. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Estimated cost at appraisal was US$18 million. Actual total cost was US$5.53 million. of which US$5.53 million was disbursed at project close. Financing: Planned support for this Investment Project Financing was with funding of US$15 million from the Afghanistan Reconstruction Trust Fund (TF-16354), of which US$5.53 million was disbursed at project close, and a Borrower contribution of US$3 million. Borrower Contribution: A Borrower contribution of US$3 million was planned, but no direct contribution to the project was made. The ICR (p.19) noted that the government had committed to provide US$3 million to cover salaries of technical teacher trainees and governmental support staff. As it happened, about US$1.5 million was allocated to the National Skill Development Program during the project period, but no funding was used for salaries related to the project or for support staff. The lack of direct recipient financing resulted Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) in a disproportionate level of expenditure on project operating costs. The ICR suggested that this shortfall overstretched the project team and led to implementation delays. Dates: The project was approved March 24, 2014 and became effective April 11, 2014. The Mid-Term Review (MTR) was completed November 24, 2016. The project closed on October 31, 2018, two months before planned closing December 30, 2018. Following the MTR, the government and the Bank agreed to restructure the project to address poor project implementation and unsatisfactory progress towards the project objectives. Had that restructuring occurred, the PDO and PDO indicators (including targets) would have changed, and other operational changes would have been introduced. However, extended discussion between the government and the Bank ensued regarding the relative merits of restructuring as per the MTR recommendations versus an early closing of the project. During this post-MTR discussion period (a year and a half), progress was made on Component 2 (Project Management, Capacity Building of MOLSAMD, and M&E) and Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), but little was achieved on the other components. Finally, in July 2018, the Ministry of Finance formally requested the Bank to close the project early and cancel unspent funds. This was effected through a restructuring paper that enabled an amendment to the Grant Agreement and the closure of the project two months earlier than originally planned. 3. Relevance of Objectives Rationale At closing (as at preparation), the project objective remains highly relevant. Since 2005, the National Priority Programs of the Government of the Islamic Republic of Afghanistan were the main instruments emphasizing government priorities. The Programs had consistently identified evidence to suggest that neither recent graduates nor the labor force as a whole were fully meeting the skill demands of the economy. The Afghanistan National Development Strategy (2009 - 2013) stated that all Afghans would have equal access to quality education to enable them to develop their knowledge and skills and thereby maximize their potential. MOLSAMD’s Strategic Plan (2013-2015) stressed the need to increase accessibility and quality of market-relevant skills development opportunities to close the gap between labor supply and the market. The project directly linked literacy and skills development by designing interventions that sought to improve access to skills for illiterate and unskilled young Afghan men and women. The project was designed to respond to significant unemployment and underemployment among young Afghan men and women - about 400,000 young people entered the labor market each year without basic skills to avail of job opportunities (both existing and future) - and the project was informed by equity concerns for the sizeable segment of the population without skills training opportunities. The project was also motivated by the need to strengthen both NSDP and MOLSAMD so that they would have the capacity to facilitate the generation of a pool of semi-skilled and literate young Afghans (PAD, p.10). The PAD (p.7) stated that the various pilots proposed under the project were designed to foster linkages between non-formal training providers and employers, with unskilled trainees at the center of the project design. The project was also to dovetail with existing Bank-supported projects such as ASDP II and Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) AREDP, combining to contribute to economic growth and substantially improve the livelihoods of rural and semi-urban Afghans, allowing many to escape poverty. At closing, the project objective continued to be relevant with reference to the 2017-2020 Afghanistan National Peace and Development Framework and to the Bank Group’s Country Partnership Framework (CPF, FY17-20), both of which have a focus on improved employment and earnings. The project objective directly supports Pillar 3 (Social Inclusion), Objective 3.1 of the CPF that seeks to improve human development, including through improved access to skills training and other measures and to enhance the quality and employability of graduates from skills training institutes. The project objective also supports the World Bank Group’s gender strategy in Afghanistan that seeks to empower women economically in key markets, enhancing their roles as producers in the rural economy, and increasing opportunities for employment and entrepreneurial activities. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase the potential for employment of targeted young Afghan women and men in rural and semi-urban areas through non-formal skills training Rationale The theory of change governing both objectives presented a fairly simple proposition. Improvement in the capacity of NSDP/MOLSAMD (Component 2) would support the potential for increased employment and higher earnings among adequately and relevantly skilled young women and men in areas targeted by the project (Component 1), and young people would also be in a position to create their own wealth through entrepreneurship support (Component 3). Outputs Soon after effectiveness, elections were held, the results of which were delayed. Further delays ensued in forming a new government and in appointing ministers. The new Minister for MOLSAMD eventually took up her post about a year after project effectiveness and then had to familiarize herself with the project and its objectives. All of this delayed decision making, including decisions regarding the procurement of Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) large technical assistance packages that were vital to implementation. The situation was compounded when the new President decreed a freeze on hiring, in response to which MOLSAMD used NSDP staff for non- project tasks. Partly as a result, there were relatively few activities implemented during the first year of the project, and no progress on any of the intermediate indicators. Following the MTR there were further delays, as the government and Bank failed to come to agreement about whether or not to close the project early. Ultimately, the envisaged vocational training interventions never materialized and, as such, achievement under this objective was effectively limited to in-service training for officials and disbursement of business development grants with associated training and development.  Number of teachers from non-formal training from TTTI or any other accredited teacher training center. From a baseline of '0' a target of 50 was set, against which there was no achievement.  Number of non-formal training providers that adopt NOSS for improvement of their teaching quality. From a baseline of '0' a target of 70 was set, against which there was no achievement.  Number of MOLAMD and NSDP staff trained under the project. From a baseline of '0' a target of 100 was set, against which 150 officials received training.  An LMIS established and functional. At a literal level, this was achieved. An elaborate software-based system was established, but it was not put into use. Outcomes The PDO indicator was the share of trainees who find employment within six months of graduation, with respect to control groups (to be increased from a baseline of 0 percent to 15 percent). There were no trainees under Component 1 and, as such, no achievement. The ICR noted that, as a result of the business development grants, training, and mentoring, the 2,482 Sub-Component 3.2 beneficiaries (of whom 30 percent were female) achieved an employment rate of 88 percent, a statistically significant 13 percentage points higher than the control group. The gains for women were even more impressive, with a statistically significant 33 percentage points higher than the control group. These achievements are notable (although we also note that recipients of the business development grants do not equate to the trainees envisaged under the indicator). A second PDO indicator was the number of beneficiaries and share of female beneficiaries (to be increased from zero to 44,500; female share to be increased from 20 percent to 30 percent). The number of beneficiaries increased from zero at baseline to 2,482, of whom 30 percent were female. All beneficiaries were recipients of the business grant under Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), as noted above. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) A third PDO indicator was the amount of incentive(s) disbursed to TPs (to be increased from zero to US$ 4 million). The actual amount of incentive(s) disbursed to TPs remained at the baseline value of zero because no TPs/trainees were approved to receive incentives under the project. Rating Modest OBJECTIVE 2 Objective Increase the potential for higher earnings of targeted young Afghan women and men in rural and semi-urban areas through non-formal skills training Rationale Outputs Number of new micro and small-scale businesses established and running by project close. Against a baseline of '0' a target of 70 was set. Ultimately, actual achievement reported was 1,871. Outcomes The PDO indicator was the increase in trainees' earnings with respect to control groups, to be increased from a baseline of 0 percent to 15 percent. There were no trainees under Component 1 and, as such, no achievement. However, the ICR noted a 17 percent increase in income for recipients of the business development grants. Rating Modest OVERALL EFF TBL OLD Rationale Most of the intended interventions were not implemented due to delays associated, inter alia, with political change and lack of decision making regarding the project on the part of the client and the Bank. That said, it is notable that the impact evaluation for Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) found that grant recipients (of whom 30 percent were female) established or expanded businesses that increased economic Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) activity in response to increased human capacity and skills sets and business services, increased employment among the poor and vulnerable, and increased gender equality. Overall Efficacy Rating Primary reason Modest Low achievement 5. Efficiency There are no efficiency projections provided in the PAD. The ICR calculated net present value (NPV) and internal rate of return (IRR) for Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth, or the business development grant scheme), as this was the only aspect for which measurable outcomes were available. The following assumptions applied. First, the benefit was the incremental income earned by beneficiaries as a result of the grants, which was determined by: (a) calculating the difference in monthly net income between the treatment and control group individuals at baseline (AFN 630 greater for treatment group) and at endline (AFN 1,500 greater), and then (b) subtracting the former from the latter (AFN 870). This difference showed that, in relation to the control group, individuals in the treatment group had increased their monthly net income by AFN 870, or US$ 11.60, on average. The incremental income generated by the grants at the time of the impact evaluation was assumed to be constant (in real terms) for a period of 15 years, in the base case. The costs used to calculate the NPV/IRR included all project expenditures for Sub-Component 3.2 and a portion of Component 2 (Project Management, Capacity Building of MOLSAMD, and M&E) which, combined, made up 48 percent of all project disbursements by closing. The NPV was found to be US$1.7 million and the IRR 12.6 percent under the base case, ranging from 8.4 percent to 16.4 percent using a sensitivity analysis. The ICR noted that stringent low case assumptions indicated that the positive result was robust and that, given the pilot character of the Sub-Component 3.2 activities, these returns were in line with sectoral expectations. The ICR also noted that Sub-Component 3.2 activities entailed relatively high learning and overhead costs that would be proportionately reduced if the grant scheme were to be scaled up, giving an IRR of about 26 percent (base case). However, the ICR also added that, if all project expenditures were set against the returns of Sub- Component 3.2 (the only measurable outcomes available), then the project would have a negligible, below- discount IRR (1.7 percent) and negative NPV. From an operational efficiency perspective, the substantial implementation delays and failure to successfully complete Component 1 and Sub-Component 3.1 are notable. Operational efficiency was impacted by early delays associated with political change - elections being held shortly following effectiveness, delayed confirmation of election results, delays in appointing ministers, and delays associated with the new Minister for MOLSAMD getting up to speed – that made it difficult to get decisions approved, including for the procurement of large TA packages that were vital to implementation, as well as to make progress on integrating NSDP into the ministry (ICR, p.16). The ICR noted that these delays were associated with an overall funds utilization rate of 37 percent, as well as substantial turnover in project personnel. Component 2 activities were completed within the overall project schedule, though with some initial delays. Sub-Component 3.2 experienced initial delays associated with the recruitment of a grant management firm, though the project team took advantage of the Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) delay to strengthen the design of the scheme through the operation of a small pilot that laid the ground for the sub-component's eventual effectiveness. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 48.00 ICR Estimate  12.60  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objectives is rated High, given alignment with country context and strategy and with the Bank’s country strategies over time. Efficacy is rated Modest, given low achievement of objectives and associated targets. Efficiency is rated Negligible, reflecting poor overall rates of return and significant delays in implementation. The overall outcome rating is therefore Unsatisfactory, consistent with major shortcomings in the project's preparation and implementation. a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome Notwithstanding the difficulties caused for the NSDP (e.g., more complicated decision-making) when it was integrated into MOLSAMD as part of the project, the move has arguably strengthened overall capacity and sustainability given strengthened operating procedures, skills transfer by NSDP staff, and training. The project also strengthened training programs, which may benefit future trainees (ICR, p. 13). On the other hand, the ICR noted various potential risks to development outcome associated with the broader socio- economic context in Afghanistan. Any deterioration of the economy may hinder the beneficiaries’ capacities to maintain employment and generate income through their businesses. Business-related income and employment gains for female beneficiaries who started up or expanded an existing business could also be threatened by any move towards social/religious norms that restrict women’s mobility. Finally, government ownership of the replication of the Business Grant Scheme for technical/vocational education and Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) training might waver post-project, or MOLSAMD might not have the necessary capacity to successfully sustain the scheme. Budget cutbacks and/or further fiscal tightening may also threaten sustainability. 8. Assessment of Bank Performance a. Quality-at-Entry Project design and intervention logic was largely sound, covering demand and supply side aspects, including capacity building and incentives, to ensure the availability of suitably skilled young people capable of taking advantage of labor market opportunities. The project was designed to address a major development constraint. It was well integrated into the government’s sectoral and national strategies and was also designed with reference to the experience of other Bank-supported projects. Proposed Bank inputs and processes were adequate, and included regular supervision missions and reporting using a fully qualified team. Risk assessment and mitigation were reasonably sound and accurate, noting feasibility challenges given the security situation, though the political challenges that eventually diminished achieved outcomes were not anticipated and mitigated successfully. The ICR noted that the project was, however, somewhat unrealistic, i.e. the proposal to reward training providers for the achievement of successful trainees (certified, found employment, or started a business) assumed there would be sufficient students enrolled in trades with good employment prospects, which, in turn, relied upon the students’ fees being paid either by themselves or other development partners. The ICR also noted that the project did not formally identify priority trades and, as a result, some training providers offered courses that would not (according to MOLSAMD) lead to employment. Finally, implementation arrangements set out in the PAD changed at negotiations when it was agreed that the NSDP would be formally integrated into the MOLSAMD. This complicated implementation for NSDP in terms of having to cope with increased bureaucracy and a more complicated decision making process. As noted in the ICR, implementation arrangements arguably should have been re-appraised prior to completing negotiations. Also, more capacity building under Component 2 should have been incorporated, with clear targets pertaining to expected results. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank conducted five missions that were supplemented through Kabul-based staff conducting technical and fiduciary supervision, as well as regular, ongoing communications. There were two task team leaders (TTLs) from the preparation stage to closure, and one co-TTL based in Kabul who acted throughout the entire project cycle. The Bank provided various supports to the client throughout, for example, through the provision of detailed guidance on the work of the Inspection Agent (Component 1), the design of the first Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) stage of Sub-Component 3.2, and the ensuring of quality sampling for the impact evaluation. Fiduciary aspects were generally well supervised. On the other hand, there were significant deficiencies in relation to safeguards. The safeguards team did not approve the Safeguards and Environmental Assessment (SEA, see Section 10a) or follow up the non- receipt of a safety manual. The safeguards team had no input to the Grievance Redress Mechanism when it was finally established in mid-2017, nor did it have input to the MTR. No records of client reports on safeguards compliance were gathered. The Bank followed-up post-MTR with a mission seeking to clarify with the government how best to implement the MTR recommendations. This consultation turned into a protracted period of vacillation that the ICR described as "unproductive if not paralyzing" (p. 25) noting, in particular, a lack of agreement on the way forward between the country office and the Global Practice. This disagreement resulted in the absence of a unified corporate position in dealings with the client that fed indecision on the future of the project. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Broadly speaking, the results framework indicators captured much of the intended outputs and outcomes. The design of impact evaluations (for, respectively, Sub-Components 3.1 and 3.2) were detailed in the PAD. Special aspects of the monitoring design were identified, including third-party monitoring (to assess quality of TP selection and transparency of beneficiary selection processes), an independent assessment of trainee competencies (to monitor the quality of training), and special purpose reviews (to monitor the cash incentives transactions). On the other hand, the PDO indicators did not capture the potential contribution of Sub-Component 3.2 to increased employment and income, which was unfortunate given that this sub-component was the most successfully implemented of all components and contributed to positive outcomes in terms of both employment and income. b. M&E Implementation The planned Management Information System (MIS) located within the M&E unit of MOLSAMD’s General Planning Directorate was never operationalized in support of the project due, in part, to the fact that the Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) main activities for which it was required were not implemented. Annex 14 (p. 62) of the ICR elaborated that the failure to operationalize the project MIS was also due to the fact that the individual hired to design and operate it was not able to transfer his capacities to the two persons working at the M&E unit at MOLSAMD, leaving only a User’s Manual for those taking over. In addition, there were conflicting views about where best to host the server for the MIS. As it became clear that the main activities for which the MIS was required (under Component 1 and Sub-Component 3.1) were not going to be implemented, the impetus to resolve these operational issues diminished. Other implemented activities were monitored through monitoring databases and mechanisms that were developed by NSDP staff and technical assistance working on Component 1, Component 2, and Sub- Component 3.2. The M&E Unit of the Planning Directorate also monitored Sub-Component 3.2 activities, and an evaluation of Sub-Component 3.2 was successfully conducted using a randomized selection mechanism. More generally, the project supported MOLSAMD’s monitoring functions as per design. In particular, an LMIS was developed with project support, fed internally by the MOLSAMD using registration of work permits for Afghan nationals and for foreigners. Ultimately the LMIS will also be fed by internal data pertaining to Employment Service Centers (only one, in Kabul, is currently operational) and by data externally from five other ministries, using (for example) data on graduates and the labor market. The LMIS also incorporates household and enterprise survey data. In this regard, the project supported the Human Resource Demand and Supply Survey (2016), which provided data for decision- making as noted below. c. M&E Utilization The ICR reported that, while the project MIS was not operationalized, other monitoring and evaluation mechanisms were used that enabled activities to be monitored and decisions taken, including: monitoring of Component 1 that led MOLSAMD to reconsider selection criteria for TPs; Sub-Component 3.2 being closely monitored for quality, leading to improvements and timeliness during the second stage of implementation; the conducting of a well designed and implemented impact evaluation for Sub- Component 3.2; and the LMIS being used to register work permits, and results from the labor demand and supply survey being used by MOLSAMD to decide on priority areas for training and by the Ministry of the Economy to design new private sector development programs M&E Quality Rating Modest 10. Other Issues a. Safeguards The PAD (p. 30) anticipated that no social safeguards policies would be triggered by the project and noted that an Social Assessment would be undertaken to: (i) establish a baseline for the project; (ii) develop a social inclusion strategy; and (iii) develop a strategy for informal vocational training compatible with/strengthening existing traditional apprenticeship systems across the relevant sectors. Annex 9 of the Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) PAD presented draft terms of reference for the Social Assessment. The PAD (p. 31) also envisaged low environmental risk but noted that viable planning would be crucial to address the health and safety issues associated with training, noting that the aim was to ensure that skills training would include practical knowledge on occupational health and safety aspects that could be applied in the workplace. In that regard, the ministry would need to hire a Safeguard Focal Point, and it would be necessary to provide extensive training. One of the potential identified risks related to hygiene issues of the working environment, e.g. ventilation, proper lighting, access to safe drinking water, and sanitation. The PAD noted that the project should consider these factors when contracting local trainers or should allocate adequate budget to improve working conditions and surroundings. Environmental Assessment (OP/BP 4.01) was triggered at assessment, and the project was rated Environmental Assessment category B. As a result, the project developed an Environmental and Social Management Framework to lay out specific requirements, processes, and responsibilities for ensuring that project activities would avoid negative impacts on the environment and people. An international firm worked with a local company to undertake an assessment, and a draft SEA report was produced and was reviewed by the Bank team who reverted with comments to address specific shortcomings of the report. However, the report was never shared again for the Bank’s review and, as such, was never officially endorsed by the Bank’s safeguards team, although it was approved by the project TTL in the context of making payments due to the firm. The ICR noted that the project was also required to prepare a safety manual, although no evidence was found of its completion, as it was not shared with the World Bank for review. The ICR (p. 23) also noted that institutional arrangements for a Grievance Redress Mechanism were established under the project, mainly pertaining to Sub-Component 3.2; however, the Bank Safeguards Team was not provided with evidence as to the mechanism's functionality, operation, and effectiveness. Finally, the Bank’s safeguards team did not take part in MTR meetings held during November 2016. The government reported that a dedicated Environment Specialist and a Social & Gender Specialist were hired to oversee implementation of relevant safeguards compliance. Contracts for relevant staff were found to be valid until June 2018, but the ICR team found no records of reports that have been shared with the Bank by the client on the monitoring and supervision of safeguards compliance. b. Fiduciary Compliance The ICR (p. 23) stated that the project maintained orderly and timely financial management (FM) arrangements throughout most of the implementation period, except for the internal audit report which was not conducted regularly or shared with the Bank on a timely basis, although all were ultimately available for review. No ineligible expenditures were recorded, and all the fixed assets purchased from grant proceeds were to be transferred to MOLSAMD before the end of December 2018. However, the FM performance rating was downgraded from Satisfactory to Moderately Satisfactory in December 2014 and remained as such until project close. The downgrade was due to low budget credibility and weak fixed assets management, noting, however, that there were no FM issues that undermined project performance. Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) On the other hand, the ICR noted significant shortcomings in procurement that affected project performance. Non-fiduciary factors contributed to procurement difficulties, i.e., the procurement function was hampered by the non-execution of reviews (e.g., Special Purpose Review and the Feasibility Study) on which the procurement process relied. The ICR noted that the security situation meant that some international firms were reluctant to bid on Afghan contracts. Other procurement shortcomings included the plan not being regularly updated, and the procurement of luxury goods. Procurement was rated Moderately Unsatisfactory in Implementation Status Reports between August 2016 and project close. c. Unintended impacts (Positive or Negative) Beneficiaries under the business grant scheme were required to open a bank account and present government-issued identification (ID). Many of the beneficiaries were previously unbanked and, as such, became finally included, and this conferred banking benefits beyond receiving the incentive grant itself. Many of the beneficiaries – including a large proportion of the women – did not have the required ID to open a bank account but managed to obtain one in order to receive the grant. The ICR (p. 14) noted that, aside from accessing the grant, having the ID (Tazkira) also allows beneficiaries to vote and is a form of civil inclusion. The ICR also noted that the Italian Agency for Development Cooperation (IADC) funded a smaller-scale project using the same design as the Bank-supported project but in different geographic zones. d. Other n/a 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory Bank Performance Unsatisfactory Unsatisfactory Quality of M&E Modest Modest Quality of ICR --- High 12. Lessons The ICR presented a significant number of lessons and recommendations. Some referred to the particular context and recommended, for example, the prioritization of ongoing skills development, and continued capacity building within MOLSAMD to enhance its project management capabilities Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) as well as the continuation of the business grants scheme. The ICR also noted that concerns of all stakeholder ministries should be explicitly addressed during the preparation process, to ensure there is broad government ownership at the outset and to avoid down-the-line implementation delays. The value of cooperation between development partners within an overall strategic context was highlighted in the ICR; had the restructuring recommended at the MTR been implemented, greater coordination of effort among the partners operating in the technical and vocational educational and training sector – the Bank, the German Agency for Technical Cooperation, and IADC – would have been possible. The ICR also noted that the uncertainty around project direction, particularly post MTR, hampered staff morale and did not alleviate implementation bottlenecks. As such, the ICR recommended that project partners should take advantage of World Bank guidelines to change course when it becomes clear that some project design aspects need to be modified and that, within the Bank itself, there needs to be a more clearly consistent position articulated – between the Global Practice and Country Management Unit (CMU) – in interactions with the client. Finally, the ICR noted that the results framework failed to include indicators that adequately captured the PDOs and suggested that greater care is needed during the concept, preparation, and appraisal stages to ensuring greater accuracy in that regard. IEG emphasizes the ICR's lesson regarding the need for the articulation of a more coherent corporate perspective in dealing with the client. In this case, the ICR reported a lack of agreement between the Global Practice and CMU that resulted in mixed messages to the client and likely contributed to the long, post-MTR indecision regarding whether or not the project should be closed. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well written and consistent in its internal logic. The ICR was candid about the project’s shortcomings and presented significant detail in that regard with reference, for example, to the results framework, safeguards, procurement, and Bank performance. The ICR also provided adequate evidence to assess achievement where such was evident, and provided additional information on related contextual developments, including changes in the political economy that impacted early implementation plans. Finally, the ICR presented a significant number of valuable and well-articulated lessons, some of which are particular to the context of this particular project and others that are more generally relevant. a. Quality of ICR Rating High Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nonformal Appr. Training Edu Jobs in Afg (P146015) Page 16 of 16