Document of The World Bank FOR OFFICIALUSEONLY Report No: ICR00004452 IMPLEMENTATION COMPLETION AND RESULTS REPORT CREDIT (IDA-43190) ON A CREDIT IN THE AMOUNT OF SDR 185.1 MILLION (US$ 280 MILLION EQUIVALENT) TO THE REPUBLIC OF INDIA FOR THE INDIA: VOCATIONAL TRAINING IMPROVEMENT PROJECT MARCH 26, 2019 Education Global Practice South Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective September 30, 2018) Currency Unit = Indian Rupee (INR) INR 72.5 = US$1 US$ 1.395 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Hartwig Schafer Country Director: Junaid Kamal Ahmad Senior Global Practice Director: Jaime Saavedra Practice Manager: Mario Cristian Aedo Inostroza Task Team Leader(s): Elizabeth Ninan Dulvy ICR Main Contributor: Juan Prawda ABBREVIATIONS AND ACRONYMS AHI Apex Hi-tech Institute at Bangalore AMs Advance modules ATIs Advanced Training Institutes BBBT Broad based basic training CAG Comptroller and Auditor General of India CFI Centrally funded institutes COE Center of Excellence CPF Country Partnership Framework CSTARI Central Staff Training and Research Institute at Kolkata CTS Craftsmen Training Scheme EAP Equity Assurance Plan (EAP), EDUSAT Education satellite EMF Environmental Management Framework GOI Government of India IIM Indian Institute of Management IMCs Institutional management committees IMPs Instructional media packages INR Indian rupee IRR Internal rate of return ISR Implementation Supervision and Results Report ITC Industrial Training Center ITI Industrial Training Institute ITOTs Institutes of Training of Trainers ITWs Instructor training wings IUFR Interim Un-audited Financial Report M&E Monitoring and evaluation MIS Management information system MOLE Ministry of Labor and Employment MSDE Ministry of Skill Development and Entrepreneurship MOU Memorandum of Understanding NCVT National Council for Vocational Training NIMI MOLE’s National Instructional Media Institute at Chennai NPIU National Project Implementation Unit NPV Net present value NSQF National Skills Qualification Framework NVQF National Vocational Qualification Framework NVTI National Vocational Training Institutes PAD Project Appraisal Document PDO Project development objectives PRF Progress Report Format PWD Public Works Department RF Results Framework SCVT State Council for Vocational Training SM Specialized module SPIUs State Project Implementation Units STRIVE Strengthening for Industrial Value Enhancement Project TCPCs Training Counseling and Placement Cells TTL Task team leader TVET Technical vocational education and training UT Union Territories VET Vocational education and training VTIP Vocational Training Improvement Project WB World Bank TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .....................................10 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDOs ............................................................................................................13 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................14 C. EFFICIENCY ...........................................................................................................................17 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................18 E. OTHER OUTCOMES AND IMPACTS .........................................................................................18 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 19 A. KEY FACTORS DURING PREPARATION ...................................................................................19 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................21 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 24 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................24 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................26 C. BANK PERFORMANCE ...........................................................................................................28 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................30 V. LESSONS AND RECOMMENDATIONS ............................................................................. 30 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 32 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 46 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 47 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 49 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 51 The World Bank India: Vocational Training Improvement Project (P099047) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P099047 India: Vocational Training Improvement Project Country Financing Instrument India Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Republic of India Ministry of Skill Development and Entrepreneurship Project Development Objective (PDO) Original PDO The Project development objective is to improve the employment outcomes of graduates from the vocational training system, by making the design and delivery of training more demand responsive. Page 1 of 52 The World Bank India: Vocational Training Improvement Project (P099047) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 280,000,000 252,218,754 251,740,708 IDA-43190 Total 280,000,000 252,218,754 251,740,708 Non-World Bank Financing 0 0 0 Borrower/Recipient 79,000,000 0 79,000,000 Total 79,000,000 0 79,000,000 Total Project Cost 359,000,000 252,218,754 330,740,708 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 05-Jun-2007 17-Dec-2007 30-Jun-2011 31-Dec-2012 30-Sep-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 05-Nov-2014 206.49 Change in Loan Closing Date(s) 17-Sep-2015 232.11 Change in Implementing Agency Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories 21-Sep-2016 241.73 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Unsatisfactory Moderately Unsatisfactory Modest Page 2 of 52 The World Bank India: Vocational Training Improvement Project (P099047) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 07-Dec-2007 Satisfactory Satisfactory 0 02 06-Jun-2008 Moderately Satisfactory Moderately Satisfactory 40.00 03 05-Dec-2008 Moderately Satisfactory Moderately Satisfactory 59.75 04 01-Jun-2009 Moderately Satisfactory Moderately Satisfactory 68.81 05 28-Nov-2009 Moderately Satisfactory Moderately Satisfactory 86.49 06 28-May-2010 Moderately Satisfactory Moderately Satisfactory 95.17 07 28-Dec-2010 Moderately Satisfactory Moderately Satisfactory 117.27 08 26-Jun-2011 Moderately Satisfactory Moderately Satisfactory 147.15 09 25-Jan-2012 Moderately Satisfactory Moderately Satisfactory 151.25 10 05-Aug-2012 Moderately Satisfactory Satisfactory 151.27 11 13-Mar-2013 Moderately Satisfactory Satisfactory 151.27 12 15-Oct-2013 Moderately Satisfactory Satisfactory 189.75 13 01-Feb-2014 Moderately Satisfactory Satisfactory 201.24 14 28-Oct-2014 Moderately Satisfactory Satisfactory 206.49 15 01-May-2015 Moderately Satisfactory Satisfactory 227.09 16 01-Sep-2015 Moderately Satisfactory Moderately Satisfactory 232.11 17 26-Apr-2016 Moderately Satisfactory Moderately Satisfactory 240.54 18 07-Dec-2016 Moderately Satisfactory Moderately Satisfactory 242.74 19 26-Jun-2017 Moderately Satisfactory Moderately Satisfactory 247.62 20 25-Dec-2017 Moderately Satisfactory Moderately Unsatisfactory 247.75 Moderately 21 14-Jun-2018 Moderately Unsatisfactory 248.33 Unsatisfactory Page 3 of 52 The World Bank India: Vocational Training Improvement Project (P099047) SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 10 Central Government (Central Agencies) 7 Sub-National Government 3 Education 90 Workforce Development and Vocational Education 90 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 100 Jobs 100 Human Development and Gender 102 Education 68 Access to Education 17 Science and Technology 17 Teachers 17 Standards, Curriculum and Textbooks 17 Labor Market Policy and Programs 34 Labor Market Institutions 17 Active Labor Market Programs 17 Page 4 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ADM STAFF Role At Approval At ICR Regional Vice President: Praful C. Patel Hartwig Schafer Country Director: Fayez S. Omar Junaid Kamal Ahmad Senior Global Practice Director: Julian F. Schweitzer Jaime Saavedra Chanduvi Practice Manager: Michelle Riboud Mario Cristian Aedo Inostroza Task Team Leader(s): Nalin Jena, Hong W. Tan Elizabeth Ninan Dulvy ICR Contributing Author: Juan Prawda Witenberg Page 5 of 52 The World Bank India: Vocational Training Improvement Project (P099047) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of appraisal, India, then the 12th largest economy in the world as measured by the GDP 2006, recorded a fast-growing economy with a rising demand for a skilled workforce. One of the key suppliers of this workforce was the vocational education and training (VET) system through the Craftsmen Training Scheme (CTS) under the auspices of the Ministry of Labor and Employment (MoLE) and the National Council for Vocational Training (NCVT) at the national level, and the State Departments dealing with vocational training and the State Council for Vocational Training (SCVT) at the State level. 2. The CTS, at that time with 741,000 student places in 5,114 institutions, operated through two types of institutions: (a) Industrial Training Institutes (ITIs), financed and managed by State Departments dealing with VET providing places for about 400,000 students in 1,896 institutes; and (b) Industrial Training Centers (ITCs), owned, financed and managed by private organizations or non-governmental organizations and providing places for about 341,000 students in 3,218 centers. 3. The Vocational Training Improvement Project (VTIP) is an offshoot of the Government of India’s (GOI) Finance Minister’s Budget Speech in 2004-5 announcing the GOI’s proposal to up-grade 500 ITIs to improve the quality of training to world standard and to keep pace with the technological demands of industry and the expanding universe of knowledge. Subsequent to the GOI’s request for the project, the World Bank (WB) carried out a sector study highlighting the following key issues affecting the skills development systems in India: (i) weak capacity and fragmented governance of the skills development system; (ii) absence of industry involvement; (ii) absence of autonomy and weak accountability of training institutions; (iii) poor quality of training perpetuated by outdated training infrastructure, obsolete curricula, lack of trained teachers and other quality inputs; and (iv) poor labor market outcomes because of various factors, including weak industry-institute linkage and lack of placement support to the students. 4. In principle, VTIP encapsulated a paradigm of change that was incremental including the following 5 elements: (i) reform in management at national, state and institution levels by introducing industry/employers participation; (ii) reform at the institution level through providing autonomy, enhancing accountability, and building leadership, and establishment of strong industry-institute linkage; (iii) improving quality of training through up-gradation of infrastructure, enhancing teacher training, introducing shop floor training in the industry as part of the course; (iv) strengthening the resource institutions for the vocational training system through up-grading the central institutions responsible for curricula development, textbook and teaching and training material development, trainers training; and (v) introducing a strong monitoring and evaluation mechanism and culture in the training system. 5. The following were the key issues addressed by the VTIP: a) Low employability of ITI graduates one year upon graduation estimated at about 30% as employers felt that these graduates lacked the appropriate sector specific and soft skills; b) Lack of information about the availability and effectiveness of training programs; c) Limited participation of employers in defining training policies and developing trade courses; and d) Lack of incentives to improve performance at the ITIs. Page 6 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Rationale for Bank involvement. 6. The WB, based on the lessons learned from five previous technical education and vocational training investments in India dating back to 1989 1 and its extensive international related experience and analytical work done, was appropriately placed to: (i) support institutional development and provide technical assistance; (ii) facilitate formulation of urgently needed policy reforms; and (iii) build the foundation for changing vocational training with the involvement of industry in the management of ITIs to gradually transform a supply-driven system into one driven by the skill demands from employers. Theory of Change (Results Chain) 7. Table 1 below depicts the VTIP Theory of Change as included in the original PAD linking the interventions included in each one of the sub-components (inputs) to their expected output, outcomes and long-term impact. An assessment of this result chain concerning its strengths and gaps, especially with respect to its assumed attribution/connectivity of all its links is provided in the M&E section of the ICR. Table 1. VTIP Theory of Change (Results Chain) Components and Inputs Outputs Outcomes Long-term impact sub-components C1: Improving Quality of Vocational Training Refurbishing and equipping ITIs’ Proportion of a State/UTs workshops and laboratories allocation that has been Percent of pass-outs from expended project ITIs that exit from the Revising curricula of existing CTS system with a NCVT trades and developing for new certificate as compared to a trades baseline Filling teacher vacancies Proportion of filling vacancies Appointing new instructors that are filled Acquisition of learning materials Sub 1.1: Supporting the adoption of the Strengthening of center of excellence (COE) Percent pass-outs from project ITIs model ITIs who find employment Establishing institutional Percent of ITIs with active within one year of finishing management committees participation and leadership in training as compared to a (IMCs) and training, counseling IMCs as measured by baseline and placement cells (TCPCs) attendance at committee meetings and subjectively through field visits by SPIU Real monthly earnings of Contributing to India’s Establishing 10 instructor Number of ITWs employed pass-outs from human capital Sub 1.2: training wings (ITWs) in selected established/upgraded to project ITIs measured one year development for Strengthening ITIs. provide entry-level training after finishing training, as sustaining growth and 1 Vocational Training Project (Cr. 2008/Ln 3045 for US$280 million), approved in FY89 and completed in December 31, 1998, after a two- year extension; US$146 million were cancelled at the end of the project. Electronics Industry Development Project (Ln 3093 for US$8 million), approved in FY89 and completed in December 31, 1997, after a two- year extension; US$5 million were cancelled. The Swiss Development Cooperation co-financed the project for US$14.3 million. Technician Education I Project (Cr. 2130/Ln. 3195 for US$260 million), approved in FY90 and completed on June 6, 1998 after an extension of three months; US$43.6 million were cancelled. Technician Education II Project (Cr. 22230 for US$307 million), approved in FY92 and completed on time in October 31, 1999; US$51.3 million were cancelled. Technician Education III Project III (Cr. 3413-IN for US$64.9 million), approved in September 2000 and completed on June 2006. Page 7 of 52 The World Bank India: Vocational Training Improvement Project (P099047) instructor training Number of new and current compared to a baseline poverty reduction and Instructor training instructors given basic and to learning through refresher or specialized evaluating the instructor courses implementation of Number and type of grants innovative public- provided to well performing private partnerships in Sub 1.3: Incentive Setting up a State/Union State/UTs and the distribution the delivery of fund Territories (UT) incentive fund of these resources to project/ vocational training non-project ITIs. C2: Promoting systemic reform and innovations Contracting studies Number of studies Sub 2.1: Promotion commissioned by the NPIU of reforms Carrying study tours Provide short term fellowships Setting up the innovation fund Number of innovation proposals financed by the innovation fund Sub 2.2: Innovation Developing web-based training fund and new delivery for informal training Strengthening CSTARI for the Number of trade curricula development of trade curricula develop by CSTARI with private with private sector input sector input Strengthening AHI for updating Number of trades upgraded by the curricula and learning AHI with new technology Sub 2.3: resources curricula and instructor training Strengthening introduced capacity to develop Norms, standards and curriculum and assessment procedures for COE learning resources instructors developed by AHI Strengthening NIMI for Number of IMPs and self- development of institutional learning programs developed media packages (IMPs) and self- and published by NIMI learning programs C3: Project management, monitoring and evaluation Carrying out tracer studies of ITI ITIs baseline and graduates’ Sub 3.2: Monitoring graduates. tracer study completed and evaluation Developing a web-based MIS. Web-based MIS fully functional. Carry out impact evaluation Impact evaluation studies completed Project Development Objectives (PDOs) 8. The PDO were stated in the VTIP’s Financing Agreement as “to improve the employment outcomes of graduates from the Recipient’s vocational training system, by making the design and delivery of training more demand responsive”. Key Expected Outcomes and Outcome Indicators 9. As displayed by the VTIP’s Results Framework (RF) included as Annex 1, the achievement towards the PDO were to be measured by the following three outcome indicators: (a) Outcome indicator 1: Percent of pass-outs from project ITIs that exit from the CTS system with a NCVT certificate as compared to a baseline; (b) Outcome indicator 2: Percent pass-outs from project ITIs who find employment within one year of finishing training as compared to a baseline; and (c) Outcome indicator 3: Real monthly earnings of employed pass-outs from project ITIs measured one year after finishing training, as compared to a baseline. Page 8 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Components 10. The original Project design included the following 3 components: (a) improving quality of vocational training, (b) promoting systemic reforms and innovations, and (c) project management, monitoring and evaluation. 11. Component 1: Improving Quality of Vocational Training [Total: US$301 million, IDA: US$228 million] aimed at: (a) improving quality and relevance of training imparted in 400 eligible ITIs selected competitively from eligible States/Union Territories (UTs); (b) upgrading training of ITI instructors; and (c) providing incentive funds to States to reward good performance in project implementation. This component was to be implemented along the following three sub-components: (i) Strengthening of ITIs (sub-component 1.1); (ii) Strengthening instructor training (sub-component 1.2); and (iii) Establishing an Incentive Fund (sub-component 1.3). 12. Sub-component 1.1: The strengthening of ITIs was to support and finance: (a) refurbishing the teaching-training infrastructure; (b) upgrading the training facilities; (c) revising the training curricula with the participation of industry; (d) modernizing and strengthening the existing facilities including workshops for the purpose of enabling Project ITIs to impart training on the revised curricula; (e) launching new trade courses to meet the relevant industrial demands and discontinuing trade courses not demanded by industry; (f) establishing centers of excellence (COEs) in about 300 Project ITIs in participating States; (g) establishing facilities for management information system; (h) establishing facilities including workshops for providing new trade courses and for providing training in advance modules through COEs; (i) increasing utilization of learning resources and other information available through media; (j) filling instructor vacancies and appointing additional instructors; and (k) establishing Training, Counseling and Placement Cells (TCPCs). 13. Sub-component 1.2:The strengthening instructor training was to support and finance: (a) establishing 10 instructor training wings (ITWs) in selected Project ITIs; (b) improving training facilities in 11 Centrally-Funded Institutions (CFIs) and conducting training programs for would-be instructors; (c) starting refresher training courses for instructors who have been in the system for more than 5 years in 20 selected COEs; and (d) organizing specialized training programs for instructors in public organizations outside the purview of MOLE and in private organizations, as necessary. 14. The strengthening instructor training was also to support and finance the strengthening capacity for managing the activities listed under Sub-component 1.2 above through, for example: (i) developing and maintaining a database of instructors which provided, inter-alia, full details of work experience and training history of each instructor; (ii) undertaking an analysis for identifying the training needs of instructors; (iii) developing curricula for basic, refresher and advanced training programs for instructors; (iv) reviewing the existing basic-level training programs, analyzing the needs of instructors and recommending changes to the training programs to NCVT on the basis of such review and analysis; (v) setting norms and standards for new training programs; and (vi) carrying out periodic inspections for ensuring adherence to norms and standards for instructor training. 15. Sub-component 1.3: The Incentive Fund was to provide Incentive Grants to the participating States for the purpose of carrying out Incentive Sub-projects for, inter-alia, strengthening vocational training in such states, conducting short-term and medium-term training programs in emerging areas requiring specialized skills, strengthening training facilities in non-Project ITIs, and translating instructional packages into regional languages as required. 16. Component 2: Promoting Systemic Reforms and Innovations [Total: US$30 million, IDA: US$29 million] aimed at: (a) promoting reforms (sub-component 2.1); (b) setting an innovation fund (sub-component 2.2); and (c) strengthening the capacity for development of Curricula and Resource Materials (sub-component 2.3). Page 9 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 17. Sub-component 2.1:Promoting reforms through supporting studies and international and national consultations to: (a) set up a National Vocational Qualification Framework (NVQF) 2; (b) design a policy framework for registering private training providers and the courses/programs offered by them that meet NVQF set standards; (c) design models for training of the informal sector workforce; and (d) assess the feasibility of establishing a training fund for financing both formal training courses in publicly and privately-financed training institutions, and training programs conducted by employers for newly hired and existing workers. This component was also to finance study tours and training programs for the purpose of, inter alia, enhancing the capacity of policy makers and select key staff of the Project Executing Agencies. 18. Sub-component 2.2: The Innovations Fund aimed at providing Innovation Grants for Innovations Sub-projects for carrying out pilot activities for, inter alia, improving the quality of vocational training provided by ITIs, developing web- based training programs for delivering vocational training through distance-learning models (like the Education Satellite (EDUSAT)), developing a system for comparing the training standards with the international standards, developing public- private partnerships and developing new delivery systems for informal training. 19. Sub-component 2.3: Strengthening the capacity for development of Curricula and Resource Materials aimed at supporting the institutional capacity buildup of: (i) The Apex Hi-tech Institute at Bangalore (AHI) for developing curricula and resource materials in emerging technologies, benchmarking vocational training against international standards and developing curricula for the 3 types of instructor training programs described under sub-component 1.2; (ii) The Central Staff Training and Research Institute at Kolkata (CSTARI) for developing procedures for evaluating COE students’ achievement and instructor training; and (iii) The MOLE’s National Instructional Media Institute (NIMI) at Chennai for developing, printing, publishing and disseminating instructional media packages (IMPS). 20. Component 3: Project Management, Monitoring and Evaluation [Total: US$28 million, IDA: US$23 million] aimed at financing Project Management (sub-component 3.1) and monitoring and evaluation (sub-component 3.2.). B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 21. The PDOs remained unchanged during the four approved restructurings of VTIP as further described below. Revised PDO Indicators 22. The outcome target for PDO indicator 1 pertaining to females i.e. percentage of pass-outs from project ITIs that exit from the CTS system with a NCVT certificate was changed during the restructuring in 2015. The target for “females” was revised downwards from 89 percent to 80 percent based on implementation experience. In addition, the metric to measure the following two outcome indicators was redefined in 2016, as further detailed in the M&E and Bank Performance Sections – outcome indicator 1 (Percent pass-out rate from project ITIs that exit from the CTS system, with 2 When established at the end of 2013 it became the National Skills Qualification Framework (NSQF). Page 10 of 52 The World Bank India: Vocational Training Improvement Project (P099047) a NCVT certification) and outcome indicator 2 (Percent pass-out from project ITIs who find employment within one year of finishing training, as compared to a baseline). Revised Components 23. The VTIP experienced four level-two restructurings. Minor changes to component 1 were made during the first restructuring approved in December 2012. First, the 10 Instructor Training Wings (ITWs) intended to provide basic level training were replaced by the establishment of 5 Institutes of Training of Trainers (ITOTs) as full-fledged institutions with separate infrastructure and human resources with the mandate to offer basic and advanced instructors training to 3,200 trainers. Second, the refresher training courses for instructors in 20 selected Centers of Excellence to be financed under sub-component 1.2 were now absorbed by the upgraded Advance Training Institutes (ATIs), thus this sub-component was dropped at the restructuring. 24. Some reallocation of the Credit’s financial resources was made during the first (December 2012) and third (September 2015) restructurings as detailed in Table 2 below. Table 2. Reallocation of the Credit’s financial resources At Board approval First restructuring Second Third restructuring Fourth restructuring At the closing of in 5/6/2007 December 2012 restructuring September 2015 September 2016 VTIP September 30, (as per ISR 11) November 2014 (as 2018 per ISR 15) Goods, services, training workshops, studies, study tours and operating costs US$77,350,000 US$35,191,093 US$35,191,093 until December 31, 2009 Goods, services, training workshops, studies, study tours and operating costs US$85,950,000 US$139,108,907 US$105,060,636.99 from January 1, 2010 Goods, works, services, training, workshops and studies for the US$21,800,000 US$10,800,000 No change US$8,607,542.15 No change No change Incentive and Innovation Funds until September 30, 2015 Goods, works, US$36,240,727.86 services, training, (Clubbed all workshops and undisbursed studies for the proceeds) Incentive and Innovation Funds from October 1, 2015 Total US$185,100,000 US$185,100,000 US$185,100,000 Page 11 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Other Changes 25. Common to all the four restructurings was a request by the GOI/MOLE and then by GOI/Ministry of Skill Development and Entrepreneurship (MSDE), for an extension of the closing date: (i) from December 31, 2012 to November 30, 2014 (approved by the WB on December 19, 2012); (ii) from November 30, 2014 to September 30, 2015 (approved by the WB on November 5, 2014); (iii) from September 30, 2015 to September 30, 2016 (approved by the WB on September 17, 2015); and (iv) from September 30, 2016 to September 30, 2018 (approved by the WB on 22 September 2016). 26. Table 3 displays the changes made to the VTIP’s results framework in the four restructurings as compared to the original one included in the Project Appraisal Document (PAD). During the third restructuring approved by the WB in September 2015, four new intermediate indicators were introduced into the RF as indicated below. Table 3. Changes in the RF during each restructuring Indicator Sub- At Board approval First restructuring Second Third restructuring Fourth component in 5/6/2007 December 2012 restructuring September 2015 restructuring November 2014 September 2016 ITWs replaced by Number of ITWs established number of Institute of or upgraded to provide entry 1.2 End-of-project Training of Trainers level instructor training target set at 10 (ITOTs) becoming fully functional set with an end-of-project target of 5 Number of grants provided to well performing States/UTs, End-of-project End-of-project target and the distribution of these 1.3 target set at 10 decreased to 6 resources to project and non- project it is Number of innovation End-of-project End-of-project target proposals financed by 2.2 target set at 10 reduced from 10 to 6 innovations fund Number of project ITIs that New intermediate have signed at least one MoU 1.1 indicator. End-of-project with a local industry partner target set at 50 Project ITIs. Number of ITI instructors New intermediate trained using the distance 1.2 indicator. End-of-project education network target set at 20,000 Number of new basic and New intermediate refresher modules developed indicator. End-of-project and ready for delivery using 2.3 target set at 25 the distance education network Enabling e governance by New intermediate automating admissions, indicator. examinations processes and End-of-project target set certification for all trainees 3.2 as “MIS to support across all courses offered complete e governance under NCVT purview capabilities” Page 12 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Rationale for Changes and Their Implication on the Original Theory of Change 27. All the four extensions were requested to ensure full implementation of project activities, to consolidate the gains of the reform and to ensure sustainability. The first extension was mainly requested to compensate for the implementation delays at the beginning of the project and to undertake some changes in component 1 as explained before. The second restructuring was also requested to: (i) address implementation delays caused by the 2014 General and some State Elections; (ii) take advantage of the substantial savings generated by the changes in the exchange rate 3;(iii) complete the large scale end-term tracer study of 2012 graduates contracted in August 2014 and some of the innovative activities already initiated; and (iv) enable the states to fully utilize performance based incentive grants given to them at the end of FY2013 and at the beginning of FY2014. 28. The third restructuring: (i) addressed a change of implementing agency decided by the newly elected Central Government transferring the overall VTIP implementation and coordination responsibility from the MOLE to the newly established MSDE; (ii) clubbed all undisbursed funds in the original three existing expenditure categories into a new fourth one to support all ongoing activities; and (iii) attempted to provide technical assistance (to be hired by the MSDE) to guide the MSDE, the 14 centrally funded institutions (CFI) 4, the state project implementation units (SPIUs), the national project implementation unit (NPIU) and ITIs on their various ongoing activities to scale them up and/or accelerate completion. 29. The fourth and final restructuring was requested by the GoI to: (i) allow for the MSDE/NPIU to complete ongoing activities including improving the quality of training in ATIs and ITIs, Center of Excellence (COE) migration back to the Craftsmen Training Scheme (CTS) and expansion and improvement of the existing Management Information System (MIS); (ii) enable the States to fully utilize the performance-based incentives grants awarded to them by VTIP; and (iii) better bridge the conceptualization and preparation of the then proposed Strengthening for Industrial Value Enhancement (STRIVE, P156867) operation to be approved in March 2, 2017. 30. The original Theory of Change was not modified during the four restructurings mentioned above. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 31. Relevance of the VTIP PDOs at closing stage is rated high and its objectives are fully consistent with GOI strategies and WB current Country Partnership Framework (CPF). At the time of the appraisal, VTIP was well aligned with the India Country Assistance Strategy (Report No. 29374, September 2004), which aimed at providing significant support for improving the performance and reach of India’s vocational training system, ensuring that the trainees coming out of the system were able to compete in an increasingly globalized economy. In addition, VTIP was designed to contribute to: (i) India’s long-term objective of human capital development for sustaining economic growth and poverty reduction; and (ii) learning through evaluating the implementation of innovative public-private partnerships in the delivery of vocational training. 32. At the closing stage, VTIP is also aligned to Focus Area 2 – Enhancing Competitiveness and Enabling Job Creation 3 At the time of Board approval, the exchange rate stood at about 40 INR for one US dollar. By the time of the second restructuring in November 2014 the exchange rate was about 63 INR for one US dollar (57.5 percent depreciation). 4 AHI, CSTARI, NIMI and 11 ATIs. Page 13 of 52 The World Bank India: Vocational Training Improvement Project (P099047) - under the current World Bank Group’s India Country Partnership Framework (CPF) for FY18-22 (Report 126667-IN; July 25, 2018), through mainly: (i) Objective 2.4: Increase access to quality, market-relevant skills development; and (ii) Objective 2.5: Enabling access to more quality jobs for women. 33. VTIP is also aligned to the GOI new National Policy for Skills Development and Entrepreneurship (2015) seeking to: (i) address the above-mentioned mismatches through making quality skills training aspirational for both youth and employers, with a focus on occupational standards and developing outcome-based, market-relevant approaches; (ii) strengthened national and state delivery mechanisms supported by a robust Management Information System for demand and supply projections; and (iii) ensure a consistent application of the National Skills Qualification Framework (NSQF). B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 34. Achievement of the PDOs (efficacy of the VTIP) is rated modest. The rating is based on the assessment of: (i) the achievements of each of the three project development objective indicators; (ii) the reasonable causal relationships between some of the operation’s activities (outputs of each component) and the PDOs as intended in the results chain (table 1); and (iii) other non-project factors that may have affected the observed outcomes. 35. Outcome indicator 1 – percent pass-out rate from project ITIs that exit from the CTS system, with a NCVT certification – increased significantly from the baseline (10.3% for all students, 16.2% for male students and 2.1% for female students), but fell slightly short of its revised end-of-project targets (by 7.8% for all, 2.1% for male students and 5.3% for females students). 36. These increases can be attributed to several interventions related to strengthening of ITIs as per the theory of change in Table 1, such as: a) Refurbishing and equipping a little over 400 ITIs’ workshops and laboratories with new training equipment for both the CTS as well as the COE training models under sub-component 1.1. Out of the 411 ITIs, 311 were COEs offering courses in 21 industry sectors. The conceptual characteristics of the COE training paradigm are described further down in Section III-A of the ICR report. b) Decreasing teacher vacancies so that at the closing of VTIP about 70.9% of all teaching sanctioned positions in the VTIP’s ITIs were filled, falling slightly short of the 80% set as the end-of-project target; c) Acquisition and/or development of training materials under sub-component 2.3. Accordingly, CSTARI, the NIMI and the AHI upgraded their capacity for curricula development/revision, development of textbooks, and teaching and training aids for the COE’s broad based basic training and advanced modules. NIMI revised their training materials twice during the VTIP cycle: (i) in 2014 when the CTS training migrated from a one year to a six-month training course syllabus; and (ii) in 2017 to ensure alignment with the recently enacted NSQF. Some of the training materials published by NIMI have been translated to 13 vernacular languages spoken in India; d) Offering training in newly demanded/more popular CTS trades and phasing out less popular CTS trades 5; and 5 Some examples of newly introduced CTS trades in the last five years in the project’s ITIs are: (i) mechanic auto electrical and electronics/ Page 14 of 52 The World Bank India: Vocational Training Improvement Project (P099047) e) Providing entry-level instructor and refresher training as well as leadership training under sub-component 1.2. Accordingly, eleven ATIs 6 were upgraded to offer advance module instructors training, refresher training and short-term craftsmen basic training to upgrade the skills of industrial workers and unemployed technical graduates. In the period 2007-2015, nearly 20,000 instructors from project and non-project ITIs (100% of the end-of-project target) were trained on a variety of training courses including 6-10 weeks advanced module training and refresher courses. ITOTs at Haryana (Rohtak), Madhya Pradesh (Bhopal), Odisha (Talcher) and Karnataka (Davangere) were constructed and operationalized. Principals of close to 1,100 ITIs underwent one- week long training in leadership and management at the Indian Institute of Management (IIM), Ahmedabad, IIM Lucknow, IIFT Delhi, IIPA Delhi and MDI Gurgaon. This was a first of its kind training in the ITI system. 37. Outcome indicator 2 – Percent pass-out from project ITIs who find employment within one year of finishing training, as compared to a baseline. Though recent data is not available, data reported in March 2018 7 for 2012 graduates indicate that this percentage exceeded its end-of-project targets for all students by 3%, with male students exceeding the target by 4.8%, but the target for female students falling short by 14.2%. Against the baseline, the employment situation of 2012 graduates 8 were 64.1% better for all students, 63.2% better for males and 120% better for females. Despite these accomplishments, there is no information on the current situation (i.e. employment outcomes for 2016 graduates). 38. Outcome indicator 3 – Real monthly earnings (INR) of employed pass-out from project ITIs measured one year after completing training as compared to the baseline. Though recent data is not available also for this outcome indicator, data reported for 2012 graduates indicate that this percentage exceeded its end-of-project targets by 41.3% for all students, 38% for male students and 79.4% for female students. Against the baseline, the salary remuneration situation of 2012 graduates was 76.6% better for all students, 72.5% better for males and 124% better for females. The lack of recent data does not allow ascertaining if this situation held at the closing of the VTIP. 39. The increases observed for outcome indicators 2 and 3 above, can be attributed to several VTIP interventions such as: a) Improving the quality and relevance of the training delivered through more qualified trainers, more relevant training packages and curricula, more appropriate training materials, equipment and physical facilities under sub-components 1.1 and 1.2 and strengthened apex institutions under sub-component 2.3 detailed above; and b) The various interventions implemented by the VTIP contributing in the direction of having a more demand-driven training provision and stronger linkages amongst the training provider and the labor market like, inter-alia: (i) the establishment of Industrial Management Committees (IMC) in project ITIs; (ii) the entering of MoUs between body repair/ painting; (ii) welder fabrication and fitting/ structural/pipe; (iii) computer aided embroidery and designing; and (iv) agro- processing/ food/ beverage/ milk products. Examples of CTS trades deleted in the project’s ITIs in the last five years are: (a) stenographer; (ii) hair and skin care; and (iii) cane willow and bamboo work. 6 Renamed under STRIVE as National Vocational Training Institutes (NVTIs) of which there are 11 the country. 7 This tracking study was contracted in August 2014 despite that the closing extension approved by the WB in December 2012 included an agreement by the GOI to conduct this tracking in the period 2013-2014. This study tracked 12,000 2012 ITI graduates from a sample of 200 project ITIs, 200 non-project ITIs and 100 private ITIs in about 20 states. A new tracking study on 2016 ITI graduates was requested by the WB in October 2017 to update the RF, but MSDE did not follow through at the closing of the VTIP. 8 Unknown if in a trade related to their training or not. However, as per the 2012 tracer study, 90.4% of those finding employment was in the private sector and only 9.6 % in the public sector. Page 15 of 52 The World Bank India: Vocational Training Improvement Project (P099047) VTIP ITIs and local industry partners; and (iii) the administering of the ITI institutional survey as an instrument to assess the degree of active participation of the employers in the IMCs 40. However, it is the view of this ICR report that the significant growth of the Indian economy experienced in the period 2009-2013 averaging 7.17%/year 9 was a key player in contributing to the employment of VTIP ITI’s graduates and their increased remuneration. 41. The ICR team attempted to gather other sources of data on employment outcomes of graduates from vocational training programs but there is little updated data that would be relevant for this purpose. Employment information for short-term skill development programs, the majority of which are run by the National Skill Development Corporation (NSDC), are largely available on a regular basis since financing under these programs is partially tied to employment rates. However, long-term skill development programs of ITIs do not have a regular mechanism to measure labor-market insertion rates, or labor market churning over time after initial placements. 42. Estimating employment rates from ITIs in such a scenario is tough. The closest estimates could have been drawn from “education level specific usual status worker population ratios” reported in National Sample Survey Reports of the Ministry of Statistics and Program Implementation (MOSPI). However, the last report available publicly is from 68th round of the survey conducted during July 2011 - June 2012 which is before the period we are interested in, i.e 2016. A more recent publication by the Ministry of Labor and Employment entitled ‘Annual Employment-Unemployment Surveys’ which captures household-level data, estimated the Labor Force Participation Rate (LFPR) in India to be 50.3 per cent in 2015-16. This data, however, is not disaggregated by education/training levels nor by year of graduation, and is therefore, not relevant for this analysis. Therefore, in the absence of a recent tracer survey, estimating current employment outcomes of ITI graduates is challenging. Justification of Overall Efficacy Rating 43. Overall efficacy of VTIP is rated modest because: (i) the outcome indicator 1 fell slightly short of its end-of- project targets in the context of increased pass-out in project’s ITIs in the period 2006 – 2016; (ii) there was no updated information provided by MSDE/NPIU to assess the achievements of the outcome indicators 2 (employability) and 3 (earnings) and thus no way to assess if this situation still holds for the 2016 graduates; and (iii) the employability and earning gains for 2012 graduates could be due to substantial economic growth in India during the period 2009-2013 rather than to the VTIP interventions. 44. In addition, as shown in Annex 1 to the ICR, 12 out of 17 intermediate indicators achieved or exceeded their original (or revised) end-of-project targets. No end-of project evaluation was conducted by an independent entity at the close of the VTIP. In particular, intermediate output indicators associated to VTIP activities attempting to make the training delivery more demand driven (for example, active participation of private sector in IMCs and MoUs with local industry partners) fell short of their original or revised end-of-project target. Accordingly, the ICR concludes that the VTIP intermediate targets were partially achieved at the time this operation was closed in September 30, 2018. 9 The period 2009-2013 was selected because, as explained further down in the ICR report, the implementation of key VTIP activities likely impacting these two outcome indicators started around mid-2009 and the tracer study refers to the employment situation of 2012 graduates one year after graduation (2013). The impressive India GDP annual growth was: 8.59% for 2009-10, 8.91% for 2010-11, 6.69% for 2011-12 and 4.47% for 2012-13, averaging 7.17% in the 2009-2013 period. Page 16 of 52 The World Bank India: Vocational Training Improvement Project (P099047) C. EFFICIENCY Assessment of Efficiency and Rating 45. The efficiency of the project is rated modest. A cost-benefit analysis conducted at the time of the ICR shows a modest economic rate of return for the project. Considering the stream of costs and benefits generated, the project’s overall net present value (NPV) of net cash flows from the project is estimated to be US$78 million based on a discount rate of 7%, which is comparable with the discount rate used at the appraisal, and the economic internal rate of return (IRR) is 9%. It can be said that the project has yielded moderate positive economic returns. 46. A tracer survey of 2012 ITI graduates was carried out after 2014 and used for the end of project targets of PDO indicator 2 and 3 in 2018. The survey tracked 12,000 graduates from a sample of 500 ITIs (200 supported under VTIP, 200 public ITIs not supported under VTIP and 100 private ITIs not supported under VTIP), and concluded that 61 percent of graduates of COE courses in project ITIs were employed one year after graduation. In contrast, 67 percent of graduates of CTS courses in project ITIs were employed one year after graduation compared to 69 percent of graduates from public ITIs that were not supported by the project. The average nominal monthly wage of male graduates of VTIP supported ITIs of 8908 INR was similar to wages of male graduates of ITIs not supported under VTIP of 8873 INR and wages of females from ITIs supported by VTIP and other ITIs not supported by VTIP was also comparable. No further survey was carried out after 2014 to measure the outcomes of interventions introduced through the last four restructurings hence the outdated 2014 results were used to measure the success of the project in 2018. 47. The above findings of the latest tracer study confirm that the impressive economic growth of the Indian economy averaging 7.17%/year in the period 2009-2013 was a key contributor to the employability and increased earning of project and non-project ITI graduates. 48. Besides a few minor indicators that were added during the third restructuring in 2015, most of the PDO targets set at the beginning of the project were not increased and not all original targets were achieved by the end of the project. It is not possible to conclude that the end results were achieved or not given the outdated data on employment outcomes of ITI graduates. In addition, the project would have achieved more favorable results and higher return if adequate outreach and supervision were given to support the COE model to reach its intended outcomes, which includes better industrial linkage and market relevant training courses, before it was prematurely cancelled in 2014 and converted to CTS. Page 17 of 52 The World Bank India: Vocational Training Improvement Project (P099047) D. JUSTIFICATION OF OVERALL OUTCOME RATING 49. The overall outcome rating of the project is Moderately Unsatisfactory. The outcome target for PDO indicator 1 pertaining to females i.e. percentage of pass-outs from project ITIs that exit from the CTS system with a NCVT certificate was changed during the restructuring in 2015. The target for “females” was revised downwards from 89 percent to 80 percent based on implementation experience. The overall outcome rating is derived after applying the split rating method. Table 4. Overall Outcome Rating Without restructuring (2015) With restructuring Relevance H H Efficacy M M Efficiency M M 1 Outcome MU MU Numerical value of outcome 2 rating 3 3 3 Disbursement 232.11 19.63 4 Share of disbursement 0.92 0.08 Weighted value of the 5 Outcome rating (2X4) 2.77 0.23 6 outcome rating 3.00 Final Outcome rating Moderately Unsatisfactory E. OTHER OUTCOMES AND IMPACTS Gender 50. As reflected in Annex 1, 2012 female graduates from project ITIs increased their employability one year after graduating by 120% and their real monthly earning in INR by 124% as compared to the 2007 baseline. Their pass-out rate from project ITIs exiting from CTS system with a NCVT certificate also increased by 2.1% as compared to the 2007 baseline. Additionally, 2017 data relating to trainee enrollment suggests that trades such as computer operator, sewing technology, dress making, and cosmetology are some of the popular trades preferred by female trainees and constitute 68% of all female trainees in 1-year trades. It is also encouraging to note that non-traditional trades for women, such as Electrician, Fitter, Electronic Mechanics, Draughtsman (Civil) and Wireman constitute 85% of females enrolled in two-year trade’s training. Page 18 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Institutional Strengthening 51. VTIP made a significant contribution to infrastructure development in 400 ITIs with new training equipment, refurbished workshops and laboratories and with more teaching vacancies filled. CSTARI, NIMI and AHI upgraded their capacity for curricula development/revision especially in new trades (with the participation of employers), development of textbooks, and teaching and training aids and for becoming a nodal managing institution for the management of instructors training networks, respectively. Training materials have been translated into 13 major vernacular languages spoken in India. The MIS and e-governance systems developed under VTIP enabled the MSDE to carry out of on-line admissions, examinations processes and certification for all trainees of ITIs across courses offered under NCVT purview as well as apprenticeship registration for individuals and firms, placements, trainer registration module and other records details of ITIs. Mobilizing Private Sector Financing 52. There is no private financial data available, though the private sector participated in the: (i) development of training curricula; (ii) in the delivery of the six-month apprenticeship/specialized modules phase of the COE model; and (iii) in the IMCs established in the project ITIs. Poverty Reduction and Shared Prosperity 53. About 74 percent of ITIs’ 2017 total enrollment corresponded to vulnerable groups (29% schedule caste/tribes and 45% backward castes). It is a known and documented fact that increased youth employment, especially of females and of poor students is positively correlated with poverty reduction. As mentioned above, as compared to a 2007 baseline, more graduates, especially females and vulnerable groups, graduated and were certified from project ITIs. Their employability as well as their remuneration also increased. Thus, it can be asserted that these graduates have made a positive contribution to poverty reduction in India. Other Unintended Outcomes and Impacts 54. Not applicable. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 55. Strengths. The ICR highlights the following seven strengths of the VTIP design. First, VTIP provided an appropriate vehicle to elevate the vocational training discussion to the next level of policy-related decision-making and actions by the GOI under the NSQF 10 and other related international development agencies-financed interventions, like, inter-alia, the ongoing STRIVE. In addition, VTIP aimed for a gradual alignment of the vocational training system with the NSQF. 10 As mentioned elsewhere in the ICR, developed outside the VTIP. At the time of the preparation cycle it was labeled National Vocational Qualification Framework (NVQF), which later evolved to the NSQF. Page 19 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 56. Second, VTIP aimed at financing significant capacity building of the participating ITIs through infrastructure and provision of training equipment and supplies, filling of vacancies, training of instructors and principals (badly needed) and provision of training/learning materials. 57. Third, VTIP supported a more relevant training paradigm – the COE model – that was being piloted in 100 ITIs prior to the design and approval of VTIP. This model was a paradigm shift from the traditional mono-trade single entry and exit vocational training program (characteristic of the CTS) to a multi-skilling, multi-entry and exit program. It was also a shift from an entirely institution-based training to a blend of institution-cum-enterprise based training. The COE consisted of a curricular structure designed at the national level in partnership with relevant industries in 21 economic sectors with the following characteristics: (i) the curricular structure comprised 6 modules of broad-based basic training (BBBT) of 12 months total duration (2 months per module), followed by 3 advance modules (AMs) totaling 6 months and finalizing with apprenticeship/specialized modules (SM) of 6 months (the training substance changed by sector). Two very important features of this model was its recognition that: (i) nowadays, workers need to be multi-skilled given their rotation on the job or between employers; and (ii) trainees might not be able to pursue a 24 month training, characteristic of many CTS, thus allowing for multiple exits and re-entry into the training scheme. 58. Fourth, VTIP attempted to link the supply side (training) with the demand side (employers) through the IMCs arrangement displaying a wide range of performance effectiveness in their mainly advisory role. The COEs chosen were done in consultation with local industry/IMC. 59. Fifth, VTIP attempted to connect training graduates with employers (especially from the private sector), both at the start of their specialized phase under the COE for a six-month apprenticeship or at the completion of their training for a more permanent employment, through the training, counseling and placement cells (TCPCs) established in each participating ITI with a dedicated staff. Sixth, VTIP attempted through the envisaged MIS and distance-learning network to digitalize the training process from registration to graduation. Seventh, VTIP, aimed for a gradual connectivity buildup with the technical higher education, like for example, with the concept of incubation centers. 60. Preparation gaps. The ICR also identifies three preparation gaps that would hinder the physical and financial implementation progress in the first three years after Credit effectiveness. First, the preparation process running from identification in August 2006 to appraisal in March 2007 did not appropriately identify likely implementation readiness conditions gaps and as a result did not propose/agree on suitable mitigation measures. Consequently, physical and financial implementation progress at the start of the project was significantly delayed, especially, for example, with respect to the establishment of the NPIU and SPIUs and the commencement of civil works required by project ITIs. Civil works was implemented by the States/UT’s respective Public Works Department (PWD) and delays in construction led to delays in the establishment of advanced module workshops, affecting the quality of COE training. 61. Second, the COE training model being piloted by the MOLE in some ITIs 11 was not appropriately assessed prior to the start of the project. Accordingly, two key implementation issues were not detected and addressed in a timely manner prior to the expansion from a pilot of 100 ITIs to 300 training institutes. The first implementation issue concerned the sequence of the BBBT training. This sequence was, by and large, determined, not by the trade requirements, but by the allocation of one sixth of the entry cohort to a module as a function of the availability of space and equipment in the given ITI. While some trainees were given the right BBBT sequence, others were not, implying that at a certain point of their basic training, they lacked the required prior knowledge and skills demanded by that module. This would be, in the opinion of some ITI’s 11 Prior to preparation, the COE model was being piloted by MOLE in 100 ITIs from 23 states covering training in 18 industrial sectors. Page 20 of 52 The World Bank India: Vocational Training Improvement Project (P099047) principals interviewed by the ICR team, one cause, among others, for student dropping out from the training. According to the Price Water House Coopers COE Report of 2016, the average dropout rate in the BBBT phase was 27 percent. 62. The second implementation issue of the COE paradigm not detected during the preparation process refers to the SM phase of its pedagogical architecture. Assumptions were made at the preparation stage that the participating ITIs were going to link to interested and suitable industry partners in their vicinity where, through appropriate outreach, students would be placed and that these students would not be “hijacked” by the labor market while carrying out their six-month apprenticeship. As explained below, these assumptions did not hold especially in COEs situated in non-industrial areas. 63. Third, lack of social marketing strategy to brand the CoE. The VTIP preparation team (both from the GoI as well as from the WB) did not anticipate the risk of non-recognition of COE courses by public and private employers hindering the employability of COE graduates because: (i) the rules of recruitment of public enterprises, followed also by some private employers, did not include the qualifications of COE; (ii) there was not an integrated certificate for the 3 phases of the COE (BBBT, AM and SM) provided by the NCVT; (iii) no equivalence with traditional and most popular CTS training was available; and (iv) lack of a continuous and intensive marketing and sensitization campaign to brand the COE model. This led, as further explained below in the report to migrate most (not all) of the COEs to the CTS model by 2014 12. This issue could have been resolved by aligning the 3 NCVT COE certificates to the NSQF that was enacted at the end of 2013. B. KEY FACTORS DURING IMPLEMENTATION 64. The VTIP, a centrally sponsored scheme implemented by the MOLE, approved in June 2007 and made effective in December 2007, was a continuation of WB commitment to technical vocational education and training (TVET) in India 13. VTIP was implemented through a NPIU, SPIUs in 33 States/UTs 14 and 14 CFIs. Performance among the States and UTs varied with the following five States having been the recipients of about 60 percent of the VTIP proceeds, not necessarily fully utilized by them: Punjab, Maharashtra, Odisha, Gujarat and Bihar. While Maharashtra, Gujarat, Madhya Pradesh, Goa, Orissa, Punjab, Karnataka, Kerala, West Bengal, and Haryana performed well on key activities, other States did not perform well such as Assam, Himachal Pradesh, Andhra Pradesh and Tamil Nadu. 65. Annual flooding, normally occurring during the monsoon season from June to September, like the one in Tamil Nadu 2015, Gujarat 2017 and Kerala 2018, affected the functioning of some project ITIs. The national general election carried out in April-May 2014 and the various State elections in the period 2007-2018 also affected project implementation. Factors subject to government and/or implementing entities control Enabling factors 66. As mentioned previously, the filling of teaching vacancies, the provision of instructor and ITI principal training and the institutional capacity installed by the VTIP in apex institutions to support quality aspects were some key enabling factors. In addition, there were others detailed below. 12 At the time of drafting this ICR, only 18 ITIs out of the total 300 VTIP COE universe continued offering the COE-based training. 13 In the period 1989-1999 WB committed a total of US$855 million to TVET in India of which US$534 million were disbursed. 14 At the time of drafting the ICR in 2018 the number had increased to 36. Page 21 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 67. Efforts to link the ITIs with employers. IMCs were established in the participating ITIs under sub-component 1.1, meeting at least every quarter and expected to act as purchase advisory committees and to advise the ITIs on placement, arrangement of student visits to industry, guest lecturers from industry, and to facilitate the signing of memorandums of understanding (MOUs) between ITIs and big industries. Some IMCs, especially those chaired by lead employers in their sector, like for example the Lonavale ITI in Maharashtra (training in the hospitality sector and chaired by a representative of the Taj group), ensured effective linkages, while IMCs of ITIs located in areas void of industry presence displayed a weaker or non-existent linkage. In addition, all project ITIs established or designated a training counseling and placement cells (TCPCs) of which, the best, maintained up-to-date trainee databases, shared information from industries on trainees, kept track of industry needs for employees with certain skills, arranged for job fairs and prepared trainees for job interviews. 68. Digitalizing the ITIs business processes. The NCVT-MIS portal developed under the VTIP’s sub-component 3.2 digitized most of the business operations related to training delivery at more than 11,000 ITIs and helped trainees obtain their examination hall tickets, mark-sheets, and certificates online. After the first phase launch, NCVT-MIS portal contained details of about 1 million trainees admitted in 2014 available online, 0.16 million e-certificate were issued to past trainees, and an online certificate validation system is available for 2 million certified trainees who were given paper based certificates in earlier years. Hindering factors 69. The ICR also detected the following implementation hindering factors attributed to government and/or implementing entities’ control some of which were satisfactorily addressed as explained below a) Initial implementation delays. The VTIP implementation was initially significantly delayed due to: (i) lack of shared understanding among all key stakeholders on the reforms introduced by the project; (ii) postponements in startup at the level of the ITIs that as per the original project design had about three years of implementation support and full five year implementation period to equip them to obtain NCVT affiliation and accreditation; and (iii) initial delays in funds flow from the state finance departments to the line departments and capacity constraints slowed down the pace of implementation. These delays were mostly addressed from 2010 onwards. b) Training vacancies. Other implementation hurdles reported in various ISRs included: (i) timely sanctioning of new instructor positions; (ii) filling up of vacant and new instructors positions; (iii) delays in the revision process concerning state recruitment rules for AM instructors; (iv) delays on deputing instructors for training; (v) filling up of ITI principals vacancies; and (vii) the strengthening of the SPIUs. At the time of Credit closing about 71% of teaching vacancies were filled falling short with respect to the end-of-project target of 80%. c) Low system capacity. Despite efforts made under the VTIP and the Public-Private Partner Scheme implemented by the MOLE and the states, the planning, implementation and monitoring capacity of states, particularly implementation of the COE training paradigm, remained low. Capacity in terms of number of staff and required professional competence of CFIs such as CSTARI, NIMI, AHI, ATIs and ITIs remained deficient to meet the changing needs in the sector and industries. The TCPCs needed to be strengthened as well as the industry-institute linkages. IMCs needed capacity building. d) Weak VTIP central governance and monitoring. As reported for the first time in the ISR #5 of November 2009, there was an urgent need to hire an implementation support agency to supplement NPIU's capacity in procurement, financial management, institutional development and quality of training, research and documentation, and monitoring and evaluation. Page 22 of 52 The World Bank India: Vocational Training Improvement Project (P099047) This issue was addressed from 2013 onwards by the hiring of a consulting firm – PWC -which provided significant management support to the NPIU and another firm - WIPRO – that designed the MIS. This support arrangement set the model for STRIVE. e) Fractures in the COE model not appropriately addressed. By the end of 2012, about 90% COEs did not have the NCVT affiliation, resulting in lack of recognition of COE courses by NCVT, affecting the sustainability of the program. Despite the establishment in 2013 of Sector Mentor Councils in 25 industry sectors (with significant industry participation) to revise the COE as well as trade curricula, as of mid 2014, most of the participating ITIs started discontinuing the COE courses as per a government decision taken in June 2014 due to challenges including the lack of recognition of these courses in Recruitment Rules of public sector enterprises and low employability. The MSDE constituted a COE Review Committee to finalize the restructuring of COE’s BBBT/AM courses to CTS trades/MES modules and provided operational guidelines for States that decided to undertake the same. One consequence of the dismantling of the COE model is that, as per the preliminary findings of the 2018 Asset Audit corroborated in the two ICR field missions (March and October 2018), about 22% of the training equipment acquired under VTIP, especially the one for the advanced modules, remains sub-utilized or not utilized at all. f) In addition, the SM phase of the COE training model was not very successfully implemented except in a few ITIs, generally located in urban-industrial areas in a few states. Reasons for poor implementation included: (i) lack of interest from industry partners to carry out the apprenticeship training; (ii) lack of suitable industries close to the ITI; (iii) poor outreach on the part of ITIs; and (iv) trainees preferring to enter the job market due to delayed placement in SM and for other reasons. Apprenticeship training did not allow for curricula to be adapted according to industry needs. Nor did apprenticeship cover all trades, in which SMs was offered. Moreover, curriculum, monitoring, assessment and certification of SM were also poorly organized. g) Mixed IMC performance. The functioning of the COEs was to be guided by the IMCs, set up as a mandatory requirement in each project ITI, acting as bodies with delegated academic, managerial, administrative and financial functions and constituted by about 11 members strong, up to 5 of which came from the private sector or industry, including the committees’ chair. IMCs had the power to suggest modifications to different existing training courses, add new trades and abolish trades that were redundant or irrelevant, retain and use revenues from the sale of goods and services and engage contract faculty. Albeit IMCs were an important part of the management structure of the ITIs, not all of them were successful in fully carrying out their envisaged roles, strengthening their linkages with the employment sector (like, for example, through MoUs with Toyota and the Taj Group) and contributing in placement related initiatives. h) Sub-components showing poor performance. At the closing of the VTIP Credit the following activities were not undertaken or failed to achieve their intended target: (i) the Space based Distance Learning Program; (ii) the Upgrading of Advanced Training Institutes; (iii) the Operationalization of Incubation Centres (5 were established but did not attract students and lacked sufficient financial funds to cover the honorarium of trainers); and (iv) Communication strategy design and implementation support. At the time of drafting this ICR, distance training remains inactive and it will not be supported by STRIVE. Factors subject to World Bank control 70. It is the ICR view that in 2014 the WB could have, together with the MSDE, explored other options to address the implementation issues of the COE instead of agreeing on its cancelation and reverting to the CTS training (some of these possible solutions are described further below in the ICR document). Page 23 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 71. The ICR is also of the view that the WB team should have requested some feasibility studies (under sub-component 3.2) prior to supporting the distance training modality carried out by NIMI under sub-component 2.3 and the incubation centers under sub-component 2.2, in order to factor in international best practices and lessons learned. As it happened these two activities failed to achieve their objectives and they are not being supported by MSDE after the closing of VTIP. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 72. Results Framework (RF). The RF’s design, though complex having 3 outcome indicators and 17 intermediate indicators (Annex A), is considered appropriate. All outcome indicators – percent pass-outs, employment within one year of completing training and real monthly earnings - were strongly connected to the PDOs. 73. The RF included 17 intermediate indicators (IO) aimed at assessing the contribution of various VTIP interventions to improve the quality of the training and/or make the training delivery more demand driven. The RF could have been further simplified by eliminating or rephrasing about 3 IOs, which, the way they were stated and measured, added limited information to gauge their contribution to the attainment of the PDOs. For example, the number of ITIs instructors trained using the distance education network is irrelevant unless its impact on training delivery is assessed. 74. Tracer studies. Three tracer studies were conducted during the VTIP cycle. The first one, conducted by the WB in 2006 provided the baseline of the 3 outcome indicators. The second and third were carried out by the MOLE/MSDE on a sample of 2010 and 2012 ITI graduates from project, non-project and private ITIs. 75. MIS. VTIP financed the development of a web-based MIS portal (www.ncvtmis.gov.in) launched in December 2014 and currently covering about 1.5 million student in more that 11,000 public and private institutes including ITIs and all apprentices in India. The original scope of the MIS was to be a monitoring tool for VTIP to gather and analyze data of project ITIs. However, during its design from 2009-2014 the scope morphed to one where the ITIs’ and apprenticeships business processes were digitized. The MOLE and then the MSDE significantly expanded the physical and functional scope of the system and additional indicators were included. Accordingly, ITIs were trained to collect and the States to aggregate information on a range of indicators, like inter-alia, enrollment, drop out, retention and transition of students and pass out rates of trainees. The ICR recognizes the complexity of the MIS conceptualization, development and implementation covering both public and private ITIs, and thus commends its achievement. 76. Other modalities of VTIP monitoring. VTIP also supported other four modalities of monitoring, including: (a) the progress report forms (PRFs) through which ITIs provide comprehensive information on all project components and sub- components 15; (b) field visits; (c) formative and summative policy and project level studies; and (d) environmental and asset audits. 15Every six months, the NPIU collected information on all key performance indicators, intermediate outcome indicators, institutional reforms, instructor numbers and their training and a host of other data from each project ITI and CFI using the PRFs; the institutional level data is analyzed intensively by the Bank team based on which state-level report cards and the national report cards are prepared. These report cards are used in review discussions and ITI level supervision during field-visits. Page 24 of 52 The World Bank India: Vocational Training Improvement Project (P099047) M&E Implementation 77. Results Framework. The outcome target for PDO indicator 1 pertaining to females i.e. percentage of pass-outs from project ITIs that exit from the CTS system with a NCVT certificate was changed during the restructuring in 2015. The target for “females” was revised downwards from 89 percent to 80 percent based on implementation experience. In addition, as reflected in the ISR#19, in December 2016 the WB implementation support team recommended to revise the methodology for the collection of information related to two outcome indicator - percent of project ITIs' pass-outs who find employment within one year after graduation and percent of pass-outs from project ITIs that exit from the CTS system with a NCVT certificate. The first outcome indicator was incorrectly reported since the inception of VTIP, as those values took into account percentage of ITI graduates who found employment at any point of time after graduation not within one year after graduating. The measurement of the second outcome indicator was corrected from the percentage of trainees assessed who had passed the CTS exam and received an NCVT certificate to the percentage of trainees enrolled who exited the CTS system with an NCVT certificate. The above-mentioned WB team also recommended a change on the instrument to collect information about the IMCs as the reporting done up to that point resulting in a finding of 100% IMCs being active was not reliable, and in fact, never verified. Indeed, once this instrument was applied, the percent of project ITIs having an active IMC decreased to 56.5%. 78. MIS. Due to its complexity, it took almost 8 years to have a functional MIS after Credit effectiveness. The MIS is a public portal with four sub-portals on ITIs, Apprenticeships, Centrally Funded Institutes and Placement. The first two modules are functional and provide data at student level in more than 11,000 public and private institutions. The ICR recognizes this to be one the most important achievements of VTIP. While the last two modules have been already designed, at the time of drafting the ICR, they were not yet being implemented. 79. Other studies including tracer studies. In the particular case of the third tracer study agreed at the approval of the first closing extension in December 2012, there was a significant delay in getting it started (August 2014) and delivering the final results (March 2018). The WB requested in October 2017 the carrying out of a fourth tracer study on a sample of 2016 graduates, but this study was never contracted by MSDE. 80. Other studies were undertaken during the VTIP implementation cycle, especially at the first mid-term in 2010, with the objective of assessing achievements and implementation gaps. These studies included: (a) two civil works, environment and asset audits; (b) a management review of the project encompassing the national, state and ITI level project related management; and (c) a COE-related assessment (October 2016). The studies have identified critical gaps for which some corrective measures have been incorporated in STRIVE. M&E Utilization 81. Utilization of the VTIP M&E at the time of drafting the ICR, especially for decision-making processes, has been limited given its database potential, but showing signs of gradual utilization increase. On the positive side, the MIS currently has online details of 9.65 Lakh trainees admitted since August 2014 session, and has been used to issue 1.6 lakh e certificates to past trainees cutting the time taken in issuing certificates manually. The portal also has an online certificate validation system for all new trainees as well as 20 lakh certified trainees who were given paper based certificates in earlier years. 82. The utilization of the MIS is currently customized for MSDE’s requirements. The next step to be financed under STRIVE is to customize the database to fit the information and decision-making needs of: (i) the trainees; (ii) the trainers; (iii) the ITIs; Page 25 of 52 The World Bank India: Vocational Training Improvement Project (P099047) and (iv) the State training authorities. The MIS needs to roll-out implementation of the remaining modules of CFIs and placement and ensure that synergies between the modules. 83. Despite that States were monitoring the performance of ITIs regularly (mostly on VTIP input/procedural and administrative-related aspects) and frequent meetings were convened by the MoLE/MSDE in Delhi with the SPIUs of poor performing institutions, the ICR is of the view that the NPIU did not conduct regular on-site monitoring to the field on more substantive and outcome-oriented topics. In particular, the ICR highlights that no assessment was carried out about the relevance and effectiveness of the basic-level entry, advance module and refresher training provided to trainers including the distance education modality to gauge if: (i) the training content and delivery of the training was appropriate; and (ii) the trained trainers were delivering the training to its students appropriately. 84. Also, it appears that data flowing upwards through the MIS is not currently used for decision-making (remains static) and does not traverse in the opposite direction (top down) for state and institutional stakeholders to also take pertinent decisions. This is understandable given the data system was recently established and data formats are in the process of being developed for various types of users of the data. The STRIVE project will continue to address this data utilization issue. Justification of Overall Rating of Quality of M&E The ICR rates the overall quality of M&E as Modest. The project accomplished an important milestone with the development of a nation-wide MIS portal covering both public and private ITIs and developing a portal that is used for planning/decision- making purposes, and as an operational tool for enabling e governance by automating admissions, examinations processes and certification for all trainees across all courses offered under NCVT purview. Nevertheless, there were some moderate shortcomings in the M&E system design and substantial shortcoming in the M&E implementation and utilization. The final tracer study was not contracted by MSDE and as a result, there is no recent data available to verify the achievement of project objectives. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Financial management 85. Financial management performance of VTIP was consistently rated moderately satisfactory throughout the implementation cycle except during the period 2008-09 where it was downgraded to moderately unsatisfactory for the reasons flagged during various joint review missions (described below), which were eventually satisfactorily addressed. 86. There were initial delays in submitting acceptable Interim Un-audited Financial Report (IUFR) and Funds release due to delays (i) in the preparation of IUFRs by some states; (ii) in submission of IUFR for reimbursement; and (iii) by some State Treasuries (outside VTIP control) in releasing funds to the line department. The delays in submission could have originated either at the implementing entity level and/or at the compilation stage at NPIU. By mid-2009, these issues were satisfactorily addressed. 87. In every fiscal year, there were significant delays by some states and CFIs in furnishing their audit report on time. As the audit in the State had to be done by the Comptroller and Auditor General of India (CAG), which is an autonomous body, some states showed their helplessness in mobilizing the CAG to conduct the audit in a timely fashion. In some cases, the delays were also caused by the implementing entities not having all the required supporting documentation Page 26 of 52 The World Bank India: Vocational Training Improvement Project (P099047) for the audit ready. To address this issue, the WB agreed to extend the original submission date for audit reports from September 30 (6 months after the end of the fiscal year in March 31) to December 31 (9 months). 88. Review of audit reports in some States showed the following key observations: (a) records not produced for civil works; (b) procurement in excess of required quantity; (c) non-utilization of funds and issue of incorrect utilization certificates; (d) excess expenditure incurred against central share; (e) non-utilization of funds; (f) Ineligible expenses; (g) difference between reported and audited expenditure; and (h) physical verification of fixed assets and stocks not conducted. As a consequence, some State expenditures where disallowed resulting in a lengthy process to settle these audit disallowances 16 including the submission by the corresponding State of the necessary supporting documentation and a request to the NPIU to recertify the given expenditures so the claims could be then submitted to the WB for reimbursement. 89. By October 2017, about one year before the closing of the VTIP, the WB support team flagged the large amount of undisbursed funds and requested the NPIU to budget eligible expenditures that could be implemented before the closing date. At the time the VTIP closed there was about US$27.58 million of undisbursed funds, some of which might have to be used to reimburse for recertification currently in progress. Procurement 90. The procurement performance of the VTIP was rated moderately satisfactory throughout the implementation cycle except during the period 2009-10 where it was downgraded to moderately unsatisfactory for the reasons flagged during various joint review missions (described below), which were eventually satisfactorily addressed. 91. VTIP-related procurement was largely handled at the ITI level following a shopping procedure (contracts up to 900,000 INRs) and at the SPIU level following National Competitive Bidding (NCB) and shopping procedures. A Procurement Manual no objected by the WB and the carrying out every year of a procurement workshop delivered by the NPIU/WB procurement team supported and guided the VTIP procurement process. All concerned persons handling procurement across all 49 implementing agencies under VTIP underwent procurement training on WB procurement procedures. For example, a Fiduciary Workshop at the National Level, with participation from all VTIP SPIUs was organized in September 2013 in which the New Procurement Business Model applicable to the Project and findings of last year's Procurement Post Review and Independent Procurement Review were shared and discussed with the participants. Nevertheless, contracts needing prior review by the WB (above 100 million INRs) took a long-time in being issued because after receiving the no-objection of the WB the contracting process had to be reviewed and endorsed by the Expenditure Finance Committee at the MOLE and subsequently, the MSDE. 92. The WB conducted post procurement reviews on every alternate year since inception in 2008 on a 10 percent sample of contracts without prior review. The WB also contracted an independent procurement review after the first restructuring. No mis-procurement was declared during the entire VTIP cycle. 16 The recertification process on questionable expenditures flagged by the audit report needs to be clarified by the implementing entity at the request of the NPIU. The implementing entity then requests the CAG to review again the corresponding expenditure based on the additional supporting evidence furnished. Due to excess workload of the CAG Office, recertification of questionable expenditure is usually done when the audit cycle of the next year starts. Consequently, NPIU/WB is provided information on the recertified amounts only after one year from the date of submission of the audit report for any given financial year. Page 27 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 93. In mid-2015 the WB support team requested another VTIP asset audit. This study was contracted by the NPIU with significant delay in the summer 2018 and the findings indicate that 90% of newly constructed civil infrastructures were found to be suitable for intended use in current and future use; only 52% of assets procured in the sampled ITIs were in functional condition whereas 22% of the assets were not being utilized; 19% of the procured assets are non- functional due to lack of maintenance and continuous use of the asset; 51% of the procured assets are not complying with national or international relevant quality and safety standard; 78% of the assets under physical review were found to be complying with NCVT requirements; out of total purchased assets, 68% are still being dedicated to the trades; and only 7% of the assets are insured. Environmental safeguards 94. The implementation rating of Environmental Management Framework (EMF) and Equity Assurance Plan (EAP), the two safeguards instruments developed for the VTIP, remained satisfactory throughout the implementation cycle. The EMF review done by the WB task team looked at aspects like health, safety provisions, water supply, gender- oriented toilets, drainage, building maintenance, power supply, waste water disposal, solid waste disposal, overall cleanliness, provision of barrier free access for physically challenged, storage, handling of toxic & hazardous material, spill management, first aid for emergency response arrangement, solar energy, rain water harvesting and plantation. The quality and efficacy of EMF implementation steadily improved since the project execution began and has been recognized as a desirable/good practice for improving the teaching and learning environment at the ITIs. Many states have scaled-up the EMF implementation to cover all the public ITIs in the state, moving beyond those covered under VTIP. Gujarat, Goa, Maharashtra and Punjab displayed good environmental practices. Social safeguards 95. VTIP had a positive impact on gender and a likely impact on poverty reduction due to the participation of vulnerable groups in the project ITIs as 60 out of the 400 project ITIs were located in districts with a concentration of minority groups. In most states, the monthly stipend provided to schedule caste and schedule tribe students increased and most ITIs report timely disbursement of stipends to students. Grievance Redress Mechanism 96. VTIP installed a Grievance Redress Mechanism for procurement related grievances. Additionally, the MSDE receives project related grievances from the GoI/Centralized Public Grievance Redress and Monitoring System, which is accessible to project-affected people. C. BANK PERFORMANCE Quality at Entry 97. Quality at entry of VTIP had strengths and weaknesses. On the positive side, the ICR identified the seven preparation strengths fully described before, placing the innovative quality-oriented labor market driven VTIP operation way ahead of its time. Page 28 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 98. On the downside: (i) the preparation process did not appropriately address serious implementation readiness conditions involving 33 (currently 36) States/UT and 14 CFIs resulting in a significant implementation delay of about 4 years after Credit effectiveness; (ii) the COE model was not adequately assessed and consequently the VTIP operational design did not anticipate implementation issues leading to the cancelation of the COE model by 2014; and (iii) preparation did not adequately consider a social marketing campaign to brand the COE model. Quality of Supervision 99. Three task team leaders (TTLs) carried out the supervision of the VTIP, the first two located in Delhi 17. The VTIP files record 14 joint review missions implying that during the entire implementation cycle lasting 129 months (almost 11 years), the WB team formally supervised this operation on the average every 9 months. No supervision was carried out from April 2016 to October 2017 (17 months), a time period coinciding with the transferring of TTL responsibilities. However, there have been frequent interactions with the MOLE/MSDE outside the regular support missions. 100. The following are some key supervision strengths of the WB team identified by the ICR. First, the WB team took the initiative to request a correction in the methodology for measuring two outcome indicators (employment and pass out rates) as well as introduced a new information source/instrument to assess the effectiveness of the IMCs. Second, as of late 2017 there was more realism in the ratings downgrading the PDOs and implementation progress to moderately unsatisfactory. Project management was downgraded to unsatisfactory. (ISRs #20 and 21). The downgrading was generated because the MSDE/NPIU did not follow up on requirements agreed with the WB related to: (i) the delivery of preliminary findings of the tracer study of 2012 graduates; (ii) the undertaking of a new tracer study for 2016 graduates; (iii) the carrying out of the second asset audit; (iv) documentation on the utilization of equipment following the conversion from the COE model back to the CTS; (v) completion of the benchmarking of ITIs; and (vi) the upgrading of ATIs. Third, the WB fiduciary team provided hands-on support to address financial management and procurement initial implementation issues. 101. The following are some key areas where the WB supervision team could have performed better. Some of the ratings provided by the TTLs in the ISRs are unrealistic and inconsistent with the story line included in the report and further detailed in the corresponding Aide-Memoire, like for example: (i) the MS rating for implementation progress in an operation displaying significant delays in getting all the pieces together during the first 4 years; or (ii) the satisfactory rating to M&E in a context where it took five years to design and implement the first two modules of the MIS (2009-2014) and almost 4 years to deliver results on the tracking of 2012 graduate (study contracted in 2014 with results delivered by March 2018). 102. However, the most significant WB supervision shortcoming was not being more persuasive with the MOLE/MSDE in 2014 about considering and reviewing other options, like addressing the COE serious implementation issues rather than endorsing the cancelation of the COE model. As it turned out, two key implementation issues had obvious mitigation measures at that time: (i) a rearrangement of the six BBBT modules to address the sequencing issue, by converting the six two month modules into two basic modules of six month duration each with a continuous assessment at the end of each semester 18; and (ii) the triple NCTV COE certification by aligning each certificate to the recently enacted NSQF at the 17 First TTL from preparation to last quarter of 2014; second TTL from the last quarter of 2014 to the third quarter of 2017; third TTL from the 4th quarter 2017 to the closing in September 30, 2018 and currently also TTL for STRIVE. 18 This mitigation measure was proposed to the NPIU by CSTARI but went nowhere until the cancelation of the COE model in 2014. Page 29 of 52 The World Bank India: Vocational Training Improvement Project (P099047) end of 2013 and thus equating these certificates to the NCTV CTS certificate widely accepted by public and private employers. Justification of Overall Rating of Bank Performance 103. The ICR rates the overall WB performance as moderately unsatisfactory because the shortcomings detected in the preparation and supervision stages overweigh the strengths as detailed above and in previous sections of this report. D. RISK TO DEVELOPMENT OUTCOME 104. In the view of the ICR, the major risk to sustain the VTIP’s Development Outcome is the cancelation by MSDE in 2014 of the COE model and reverting to the CTS traditional mono single entry and exit training, a training paradigm that is not suitably aligned to the requirements of the modern labor market. 105. On the positive side, STRIVE approved in March 2017 aims to improve access to quality and market-driven vocational training provided in ITIs and apprenticeships. Investment through STRIVE were designed to leverage previous reforms introduced through VTIP by focusing on the critical policy and program interventions for systemic reforms and expanding focus to include apprenticeship. V. LESSONS AND RECOMMENDATIONS 106. Pilot models need to be assessed before scaling them up. The COE model that was piloted by the MOLE in 100 ITIs during the preparation of VTIP and that underpinned the VTIP’s design was never properly assessed before scaling it up to reach 311 ITIs. As a consequence, the COE implementation flaws fully described before in the ICR, were not anticipated during the lending process nor addressed during the implementation cycle leading to its cancelation by MOLE/MSDE in 2014. This lesson can now be extended to the required assessment of the pilot apprenticeship intervention financed under STRIVE before scaling it up. 107. Consistent, structured and periodic on-site monitoring, especially of outcome-related topics, is an essential part of good management practices. As mentioned before in the M&E section of the ICR, there was insufficient on-site monitoring of project interventions by the NPIU to assess the relevance and effectiveness of the various quality-related intervention financed by VTIP, and if required, to fine tune their design and implementation arrangements. In a large- scale operation of national scope like VTIP, the NPIU did not have the capacity to undertake such required on-site quality/outcome monitoring. Agreement was reached with the MOLE/NPIU before the first restructuring to outsource this activity to a qualified third party. This outsourcing never took place during the VTIP cycle 19, but the lesson has been learned and incorporated into the implementation arrangements of STRIVE through the retaining of a management consulting firm. It is now of utmost importance that the STRIVE management consulting firm undertakes this required continuous quality/outcome on-site monitoring on the various STRIVE financed interventions. 108. Marketing a new brand is essential to increase awareness and modify consumption patterns. Even though VTIP included an approved item of expenditure allocated for communication, no social marketing campaign to brand the COE 19 As mentioned before, the two consulting firms contracted under VTIP where to support the NPIU on managerial tasks and for the development of the MIS, but not to carry out outcome-related on-site monitoring. Page 30 of 52 The World Bank India: Vocational Training Improvement Project (P099047) was ever carried out during the implementation cycle. As a result, some ITIs found it difficult to fill in their entry quotas, as a significant percentage of their freshman student intake preferred the known CTS training model to a new and unknown COE. Likewise, with employers that -as per existing recruiting rules- preferred CTS graduates with one (one or two year) NCVT certificate over COE graduates with 3 NCTV certificates. This lesson can now be applied to STRIVE to change the mind setting of all the stakeholder that STRIVE is not an investment aimed at only procuring required training equipment or repairing/constructing training workshop but to improve access to quality and market-driven vocational training provided in ITIs and apprenticeships for which procurement and rehabilitation of spaces might be needed. 109. Empowering the IMC. One lesson learned from VTIP (that has been already incorporated in STRIVE) is to significantly empower these IMCs, migrating them from their current role to a more institutional managerial and decision- making body under the public private partnership model, where the IMC, chaired by the employers, is vested with decision making powers and the attribution to manage a significant amount of funds. 110. Empowering the States to take a more leading role in the day-to-day monitoring and coordinating of the various ITI and apprenticeship related interventions under STRIVE. VTIP exposed that carrying out these required monitoring and coordinating activities in a geographical vast and culturally diverse country like India from Delhi become untimely and ineffective. It is the view of the ICR that the national skill training authorities need to limit their function to: (i) skills development policy and strategy setting; (ii) periodic overview of the progress towards achieving skill development related medium-term targets and if needed, fine-tuning of strategies based on accomplishments on the ground; (iii) approval of yearly or multi-annual working plans and proposed operational budgets; and (iv) allocation of targeted financial and technical resources to achieve stated skill development targets. 111. Results data must be collected on a regular basis through the project period. VTIP did not collect outcome level data on employment outcomes in a consistent way through the project and as a result, it was challenging to assess whether the interventions under the project effective. The STRIVE operation has disbursement linked indicators related to the completion of tracer surveys of ITI graduates at different points in the operation. 112. Graduate tracer studies need to incorporate data of minority groups. As 60 of the 400 project ITIs are located in districts with a concentration of minority groups the next tracer study, likely to be implemented and financed by STRIVE to address its information requirements, needs to adopt purposive sampling to include these type ITIs to find out how well students from these groups do in the labor market. 113. Decrease the scope of the design by limiting the number of interventions and intensify their thoroughness. VTIP included too many interventions, some which were unsuccessfully implemented, like the incubator centers, or had an insignificant impact, like the promotion of reforms under sub-component 2.1 and the innovation fund under sub- component 2.2. . Page 31 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent of pass-outs from Text All: 61.0% Male: All: 73.0% All: 73% Male: 73% All: 67.31%*, Male: project ITIs that exit from the 61.5% Female: 74.2% Male: 73.0% Female: 80% 71.45%, Female: CTS system with a NCVT 75.78% certificate, as compared to Female: 89.0% the baseline-disaggregated by gender. 31-Jan-2007 31-Dec-2012 28-Sep-2018 30-Sep-2018 Comments (achievements against targets): * The latest data corresponds to students entering two-year trades in 2014 and graduating in 2016. End-of-project target for 2016 graduates fell short by 7.8% for all, 2.1% for male students and 5.3% for female students. As compared to the baseline, the increases are: (i) 10.3 % for all students; (ii) 16.2% for male students; and (iii) 2.1% for female students. Page 32 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent of project ITIs' pass- Text All: 32.0% Male: All: 50.0% All: 50% Male: 52% All: 52.5%***, Male: outs who find employment 33.4% Female: 18.7% Male: 52.0% Female: 48% 54.5%, Female: 41.3% within one year of finishing training, as compared to the Female: 48.0% baseline. 31-Jan-2007 31-Dec-2012 28-Sep-2018 30-Sep-2018 Comments (achievements against targets): ** The latest data corresponds to 2010 graduates. *** The latest data corresponds to 2012 graduates. With respect to 2010 graduates, as per reporting at the end of 2012, end-of project targets: (i) were exceeded by 20% for all students and by 23.1% for male students; and (ii) felt short by 20.8% for female students. With respect to 2012 graduates, as per reporting in March 2018, end-of project targets: (i) were exceeded by 3% for all students and by 4.8% for male students; and (ii) fell short by 14.2% for female students. Against the baseline, the employment situation of 2012 graduates were 64% better for all students, 63.2% better for males and 120% better for females. Despite these accomplishments, there is no information on the current situation (i.e. employment outcomes for 2016 graduates). Accordingly, the ICR cannot assess if the VTIP achieved this objective at the closing stage. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Real monthly earnings (INR) Text All: INR 2421 All: INR 3026 All: INR 3026, Male: All: INR 4275***, of employed pass-outs from Male: INR 2474 Male: INR 3093 INR 3093, Female: INR Male: INR 4268, project ITIs measured one 2451 Female INR: 4397 year after completing Female:INR 1961 Female: INR 2451 training, as compared to the 31-Jan-2007 31-Dec-2012 28-Sep-2018 30-Sep-2018 baseline. Page 33 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Comments (achievements against targets): ** The latest data corresponds to 2010 graduates. *** The latest data corresponds to 2012 graduates. With respect to 2010 graduates, as per reporting at the end of 2012, end-of project targets: (i) were exceeded by 17.4% for all students, 16.1% for male students and 18.7% for female students. With respect to 2012 graduates, as per reporting in March 2018, end-of project targets: (i) were exceeded by 41.3% for all students, 38% for male students and 79.4% for female students. Against the baseline, the salary remuneration situation of 2012 graduates were 76.6% better for all students, 72.5% better for males and 124% better for females. Despite these accomplishments, there is no information on the current situation (i.e. remuneration outcomes for 2016 graduates). Accordingly, the ICR cannot assess if the VTIP achieved this objective. A.2 Intermediate Results Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Instructor Text 0 4 ITOT to be 4 ITOT to be 4 established Training Wings (ITWs) established established. established or upgraded to provide entry level instructor 31-Jan-2008 28-Sep-2018 28-Sep-2018 30-Sep-2018 training. Comments (achievements against targets): The original intermediate indicator referred to the ITWs was modified during the first restructuring and replaced for the construction and operationalization of 10 ITOs. Then in the third restructuring the end-of-project target was reduced to 5 ITOTs, and then reduced one more time to 4 during the fourth restructuring. VTIP achieved 100% of the revised end-of-project target. Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 34 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Target Completion Number of Project ITIs that Percentage 30.00 50.00 50.00 42.00 have signed at least 1 MoU with a local industry partner 30-Aug-2016 28-Sep-2018 28-Sep-2018 30-Sep-2018 Comments (achievements against targets): This is a new intermediate indicator included at the third restructuring in September 2015. The achievement of 42% fell short of the end-of-project target set at 50% of ITIs. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of new basic and Number 10.00 25.00 25.00 72.00 refresher modules developed and ready for delivery using 30-Aug-2016 28-Sep-2018 28-Sep-2018 30-Sep-2018 the distance education network Comments (achievements against targets): New basic and refresher modules developed but not yet delivered using the distance education network against a baseline of 10 VTIP exceeded the end-of-project target by almost threefold. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of ITI instructors Number 15.00 20.00 20.00 20.04 Page 35 of 52 The World Bank India: Vocational Training Improvement Project (P099047) trained using the distance (Thousand) education network 30-Aug-2016 28-Sep-2018 28-Sep-2018 30-Sep-2018 Comments (achievements against targets): This is a new intermediate indicator included at the fourth restructuring. VTIP achieved its end-of- project target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion E-governance enabled by Text No Yes Yes Yes automating examination processes and certification 30-Aug-2016 28-Sep-2018 28-Sep-2018 30-Sep-2018 for all trainees across all courses offered under NCVT purview Comments (achievements against targets): Indicator achieved Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of new and current Text 0 15000 instructors 15000 instructors 19404 instructors given entry-level trained trained Page 36 of 52 The World Bank India: Vocational Training Improvement Project (P099047) or refresher/specialized 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 instructors courses annually. Comments (achievements against targets): The original end-of-project target of 5,000 was revised in the second restructuring and in increased to 15,000. VTIP surpassed the revised target by 29.4%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Web-based Management Text No MIS exists Fully functional MIS Fully functional MIS Fully function MIS but Information System (MIS) only for the following developed and implemented. modules: ITIs, Trainees, Basic Training Providers, Apprenticeship – Establishments & Trainees, Grading of ITIs, Centrally Funded Institutions, and the Public Module 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 Comments (achievements against targets): Indicator achieved and in the view of the ICR, one of the most important achievements of the VTIP. Page 37 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of policy studies Text 0 5 5 5 commissioned by National Commissioned by the Project Implementation Unit NPIU: (NPIU) 1. Performance Evaluation of Industrial Training Institutes (2011); 2. Study of TCPCs; 3. Training Needs Analysis for Principals; 4. Automobile Sector Study; 5. ITI Grading Study (2014). Commissioned at the MTR: 1. Management review of VTIP (2011); 2. Review of Civil Works, Environmental Management, and Equipment Verification (2012) 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 Page 38 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Comments (achievements against targets): Indicator achieved Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of innovations Number 0.00 6.00 6.00 5.00 proposal financed by innovation fund 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 Comments (achievements against targets): The end-of-project target was reduced during the second restructuring from 10 to 6. Despite having achieved 83% of the end-of-project target, the incubation centers had difficulties in attracting student and displayed many implementation issues (explained in the Implementation /section of the ICR). Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Project evaluation Number 0.00 4.00 4.00 4.00 undertaken at mid-term and end-term by independent 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 local/international consultant firms Comments (achievements against targets): Indicator achieved for 2012 targets but not for the end-term (September 2018) as no tracer study was ever conducted for 2016 graduates. Page 39 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent of project ITIs having Percentage 0.00 100.00 100.00 56.50 active private sector participation in IMCs 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 measured by their attendance at IMC meetings and through field visits by SPIU staff Comments (achievements against targets): The 100% achievement on this indicator reported up to and including ISR#16 (September 2015) was not verified. Acknowledging that the 100% was likely an unrealistic number, a Progress Report Format (PRF) was circulated in the first quarter of 2016 to the States asking when the meetings were held, how many members were present, and whether the Chairman (representing the employers) attended the meeting. The information recorded from the ISR#19 (June 2017) onwards of 56.5% reflects the findings of the PRF. The achievement fell short of the end-of-project target of 100%. The target, in this case, was perhaps unrealistic. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion The proportion of relevant Percentage 0.00 80.00 80.00 70.86 instructor vacancies that are filled. 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 Page 40 of 52 The World Bank India: Vocational Training Improvement Project (P099047) Comments (achievements against targets): The achievement fell short by 11.4% of the end-of-project target. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of grants provided Amount(USD) 0.00 6.00 6.00 14.00 to well-performing States/UTs and the 31-Jan-2007 28-Sep-2018 28-Sep-2018 30-Sep-2018 distribution of these resources to project/non- project ITIs Comments (achievements against targets): VTIP surpassed the end-of-project target by 133.3% Page 41 of 52 The World Bank India: Vocational Training Improvement Project (P099047) h B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1:Improve the employment outcomes of graduates from the vocational training system. 1. Percent pass-out from project ITIs that exit from CTS system with a NCVT certificate as compared to a baseline. 2. Percent pass-out from project ITIs who find employment within Outcome Indicators one year of finishing training, as compared to a baseline 3. Real monthly earnings (INR) of employed pass-out from project ITIs measured one year after completing training as compared to the baseline 1.The proportion of relevant instructor vacancies that are filled 2.Number of ITWs established or upgraded to provide entry-level instructor training. Later modified by ITOT established 3.Number of new and current instructor given entry-level or refresher/specialized instructor courses annually Intermediate Results Indicators 4. Number of ITIs instructors trained using the distance education network 5. Number of new basic and refresher modules developed and ready for delivery using distance education network 6. Tracer study of ITIs pass-out 1.Increased number of graduates from VTIP ITIs. 2. Increased number of graduates from VTIP ITIs employed and earning more. Key Outputs by Component 3. More teacher vacancies filled in VTIP ITIs (linked to the achievement of the Objective/Outcome 1) 4. New ITOTs fully functional 5. Entry-level or refresher/specialized training delivered to VTIP ITI trainers Page 42 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 6. ITIs’ instructors trained using the distance education 7. Tracer survey administered and pass-out findings analyzed and utilized to further strengthen the VTIP ITIs. Objective/Outcome 2:Making the design and delivery of training more demand responsive Outcome Indicators 1.None 1.Percent of project ITIs having active private sector participation in IMCs measured by their attendance at IMC meetings through field visits by SPIU staff Intermediate Results Indicators 2.Numberof ITIs that have signed at least one MoU with local industry partner against a baseline 3.Number of innovations proposals financed by the innovation fund 4. ITI institutional survey 1. More IMCs acting as per their envisaged role 2. More MoUs signed between a VTIP ITI and a local employer Key Outputs by Component 3. Incubator fully functional and with demand (linked to the achievement of the Objective/Outcome 2) 4. ITI institutional survey administered, findings analyzed and utilized to further strengthen the IMCs in VTIP ITIs. Page 43 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Elizabeth Ninan Dulvy Task Team Leader(s) Balagopal Senapati Procurement Specialist(s) Neha Gupta Financial Management Specialist Ritu Sharma Team Member Neha Pravash Kumar Mishra Environmental Safeguards Specialist Qi Shen Young Team Member Gitanjali Chaturvedi Social Safeguards Specialist Kunal Datt Team Member Alejandro Welch Team Member B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY06 26.235 157,874.49 FY07 69.849 296,745.74 FY08 0 8.99 Total 96.08 454,629.22 Supervision/ICR Page 44 of 52 The World Bank India: Vocational Training Improvement Project (P099047) FY07 0 0.00 FY08 41.685 229,495.41 FY09 54.297 191,269.06 FY10 49.604 228,497.28 FY11 52.981 415,671.80 FY12 61.696 357,279.04 FY13 66.372 199,801.13 FY14 54.612 221,078.09 FY15 60.009 207,300.59 FY16 16.685 94,040.21 FY17 2.975 53,469.62 FY18 12.545 135,590.48 FY19 9.638 114,080.65 Total 483.10 2,447,573.36 Page 45 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Improving Quality of 228 221 97% Vocational Training Promoting Systemic 29 9 32% Reforms and Innovations Project Management, 23 11 46% Monitoring and Evaluation Total 280 241 86% Exchange rate used 56.88 Page 46 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ANNEX 4. EFFICIENCY ANALYSIS This Annex summarized the efficiency analysis for VTIP by presenting both the economic and financial analysis of the project: Economic Analysis: 1. The economic analysis of the 11 year VTIP assesses the benefits and costs associated with the project using realized data at the end of the project. It uses the approach of comparing net benefits in two scenarios, one with the project and one without (the counterfactual). The benefits came from the changes in the pass-out rate and employability of graduates, and earning increase over the period as a result of the project. Although there are different types of benefits from providing an individual with formal training, the cost-benefit analysis focuses on economic benefits. The economic costs include project costs, public and private costs, and opportunity costs for the training. 2. Economic Benefits: the benefits include the addition benefits due to increase in pass-out rates and improvement in employability of graduates, and life-time earning increase as a result of the project. The following data and assumptions were used to calculate the economic benefits: a) Enrolments at ITI are obtained from the NCVT-MIS 2014 through 2017 data and other years were estimates. The total numbers of graduates over the project period, from 2008 to 2018, with the last cohort of students enrolled in 2016 and graduated in 2018, are estimated as 3.1M at public ITIs, including 0.6M at project ITIs, and 5.1M at private ITIs; b) The percentage of graduates who found employment within one year of finishing training were obtained from a Tracer Study carried out in 2014 by Mott McDonald and published in March 2018. The study tracked 12,000 ITI graduates of 2012 academic year from a sample of 200 project ITIs, 200 non-project ITIs and 100 private ITIs in about 20 states. This was the last tracer study financed by the project, therefore data for 2013-2018 graduates is not available for the analysis and the 2013 and forward estimate is based on the assumption that the improvement will continue at same pace as they were in 2008 through 2012. c) The real monthly earnings (INR) of employed graduates from project and non-project ITIs measured one year after completing training was also from the tracer study mentioned above. The calculation is segregated by gender. 3. Economic Cost: there are four elements: project cost, public investments, private investments and opportunity costs to households. a) Project cost: The disbursement of $252M of VTIP funding from 2008 to 2018. b) Public Investments: The cost includes the Government budget going to the TVET sector. The TVET budget is calculated as 0.02% of GDP, based on the 2013 UNESCO India TVET sector study. c) Private investments: Private investments are the direct costs borne by households to enroll in an ITI. According to the 2017 National Sample Survey Office (NSSO) by the Ministry of Statistics and Program Implementation of the Government of India, the private cost of attending formal training (ITI) was INR 27,676/year. Page 47 of 52 The World Bank India: Vocational Training Improvement Project (P099047) d) Opportunity cost to households: it is assumed that trainees currently in training would have worked for wages had they not gone for the training. The opportunity cost of ITI for a year is assumed to be 50% of wage rate for unskilled labor. 4. Economic Rate of Return and NPV: the internal rate of return is calculated as 9% based on the 2008 to 2024 time period, among which, 2008 through 2018 were the project period and the 2019 to 2024 are the period that will see continuous benefits from the project. Net Present Value (NPV), net of inflation, was calculated as US$ 78 million based on the discount rate of 7%, which is consistent with discount rate used at the project appraisal. Financial Analysis: 5. The share of the VTIP disbursement to the total TVET expenditure during the project period was around 5.6%, ranging from 1% to 23%, reflecting that the project has had a substantial impact on the central TVET education sector. Table: VTIP % of TVET and total education expenditures (US$, millions) 11-year Description 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 average 37,834 43,839 56,714 70,000 70,684 71,385 78,398 80,830 87,437 99,993 106,674 73,071 Total Education Budget TVET Budget 258 280 360 390 391 392 416 432 459 522 557 405 VTIP Disbursement 60 27 31 34 0 48 15 21 8 5 4 23 Share of VTIP in the 23.2% 9.6% 8.6% 8.7% 0.0% 12.2% 3.6% 4.8% 1.8% 1.0% 0.7% 5.6% TVET budget (%) Share of VTIP in the 0.2% 0.1% 0.1% 0.0% 0.0% 0.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.03% education budget (%) Page 48 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS 1. Para 37(d) and related Footnote no. 5 on P16 - ‘Some examples of newly introduced CTS trades in the last five years in the project’s ITIs are: (i) mechanic auto electrical and electronics/ body repair/ painting; (ii) welder fabrication and fitting/ structural/pipe; (iii) computer aided embroidery and designing; and (iv) agro-processing/ food/ beverage/ milk products”. This may be considered as a supportive observation on conversion of COE infrastructure to CTS trades which were introduced as an exercise on COE-CTS mapping carried out by Sector Mentor Committee (SMCs) in 2014 for COE sectors viz. Automobile, Fabrication among others. 2. Para 49: Overall efficacy of VTIP is rated as modest whereas PDOs and intermediate indicators (12 out of 17 intermediate indicators achieved or exceeded their original (revised) end-of-project targets – Para 43) have been mostly achieved. The ICR has also listed out VTIP interventions (mentioned in para 37 and 40) that have led to improved PDOs as compared to baseline. The interventions under VTIP that have made positive impact on PDOs till 2014 (Year of end term tracer study) would continue to bring improvement in the system. As per ICR, since latest status on PDO 2 & 3 can’t be ascertained, the efficacy has been rated as Modest. 3. As per Para 45 - Efficiency of the project has been rated as ‘modest’ when the project has yielded moderate positive economic returns. When ‘Relevance’ of the Project has been rated as ‘High’, Efficacy rated as ‘Modest’ and Efficiency rated as ‘Modest’, how the overall rating is ‘Moderately Unsatisfactory’. The overall outcome rating awarded to VTIP (as reproduced below from ICR) not agreeable and may be re-examined. We are of the opinion that the project outcome deserves better rating. Table 4 (Para 49) Overall Outcome Rating Relevance Efficacy Efficiency Overall Rating High Modest Modest Moderately Unsatisfactory 4. Para 70 footnote 14 and Para 119: ‘At the time of drafting the ICR in 2018 the number had increased to 36’.: Correction to no. of States/UTs participated under VTIP is 34 States/UTs 5. ‘Para 126. The ICR rates the overall WB performance as moderately unsatisfactory because the shortcomings detected in the preparation and supervision stages overweigh the strengths as detailed above and in previous sections of this report.’ As already stated that the market acceptability of the COE model was the main impediment in the success of the COE model. Due to this reason, COE courses were converted to the DGT’s flagship training scheme, i.e. - Craftsmen Training Scheme (CTS). The ICR recognises that MIS is one of the most important achievement of VTIP. In respect of tracer study of a big sample size of 12000 trainees across the nation is a time consuming study which as also recognized by various TTLS of the project. Moreover during that period, the DGT vertical from MOLE shifted to a newly created Ministry, i.e. – MSDE. The ‘moderately unsatisfactory’ rating on WB performance on VTIP, despite the challenges overcome and strengths listed, is not agreeable and may be re-examined. We are of the opinion that the project outcome deserves better rating. Page 49 of 52 The World Bank India: Vocational Training Improvement Project (P099047) 6. Footnote 7 on P 16: There has been repetitive reference in the ICR to the tracer study not taken up in October 2017 ex. Footnote 7 on P 16 - ‘A new tracking study on 2016 ITI graduates was requested by the WB in October 2017 to update the RF, but MSDE did not follow through at the closing of the VTIP.’ This may be supported by further elaboration i.e - A ‘Quick Tracer study’ was proposed by the World Bank in October 2017, keeping in view the available time before close of project in September 2018. However, with available time, the ‘Quick Tracer study’ could not be taken up by NPIU due to not having mutual agreement with the World Bank in hiring of agency on ‘Single source basis’. 7. Point 68 ‘Third, lack of social marketing strategy to brand the CoE’: The VTIP preparation team (both from the GoI as well as from the WB) did not anticipate the risk of non-recognition/ non- acceptance of COE courses by public and private employers hindering the employability of COE graduates. Furthermore, as mentioned in the last line of this paragraph, despite NSQF alignment, recognition and acceptability would have still been an issue as the relevance of COE was not well accepted by market . 8. Point 112: Since the World Bank team has not reviewed the 2018 Asset Audit, it is requested to review and re-word this section accordingly. 9. Footnote no. 6 on P 16: Correction - renamed as National Skill Training Institutes (NSTIs) 10. Lastly, it seems that NCVT is incorrectly spelt as NCTV inadvertently, in a few instances. This may kindly be corrected. 11. The ICR mentions that: ‘The project would have achieved more favorable results and higher return if adequate outreach and supervision were given to support the COE model to reach its intended outcomes’. We are of the opinion that market acceptability of the COE model was the main impediment in the success of the COE model. Due to this reason, COE courses were converted to the DGT’s flagship training scheme, i.e. - Craftsmen Training Scheme (CTS). 12. Point 65: Incorrect years are mentioned in this point. The preparation process ran from identification in August 2006 to appraisal in March 2007. Page 50 of 52 The World Bank India: Vocational Training Improvement Project (P099047) ANNEX 6. SUPPORTING DOCUMENTS Bank preparation documents for the original IDA Credit World Bank. India Country Assistance Strategy Report 29374-IN, September 2004. World Bank. India Country Partnership Framework for FY2013-2017 Report 76176-IN, April 15, 2015 World Bank. India Country Partnership Framework for FY2018-2022 Report 126667-IN, July 25, 2018. World Bank. Aide-Memoires from the identification/preparation mission carried out in July-August 2006 up to the appraisal mission carried out March 2-15, 2007. World Bank. Project Appraisal Document (PAD) for the Republic of India: Vocational Training Improvement Project (VTIP) Report No. 39697-IN, May 9, 2007. World Bank. Signed Minutes of Negotiations, April 16-18, 2007. World Bank (IDA). Financing Agreement between India and the International Development Association for the Vocational Training Improvement Project Credit 4319-IN, April 18, 2007. Bank preparation documents for the four restructurings World Bank, Restructuring paper on a project restructuring of the India: Vocational Training Improvement Project, Credit Number 4319-IN approved June 5, 2007, Report No. 74373-IN, December 19, 2012. World Bank, Restructuring paper on a proposed project restructuring of the India: Vocational Training Improvement Project, Credit Number 4319-IN IN approved June 5, 2007, Report No. RES 16135, November 4, 2014. World Bank, Restructuring paper on a proposed project restructuring of the India: Vocational Training Improvement Project, Credit Number 4319-IN IN approved June 5, 2007, Report No. RES 20312, September 11, 2015. World Bank, Restructuring paper on a proposed project restructuring of the India: Vocational Training Improvement Project approved on June 5, 2007, Report No. RES 24827, 16 September 2016 Bank and Borrower project implementation documents Aide-Memoires of all the implementation support missions/Joint Review missions from September 2007 to July 2018. Aide-Memoire of the VTIP ICR Field mission October 22-30, 2018. World Bank. Implementation Status and Results Reports (ISRs) archived in the World Bank India VTIP electronic files Page 51 of 52 The World Bank India: Vocational Training Improvement Project (P099047) from December 7, 2007 (ISR 1) to June 14, 2018 (ISR 21). Other project implementation-related documents CENPAP. Tracer Study of ITI Graduates (2010) in India. Final Report. May 2012 Mott MacDonald.Tracer Study of ITI Graduates (2012). Final Report. January 2018. IRCLASS Systems Solutions Pvt Ltd. Asset Review under VTIP. Final Report, October 2018 PWC. Diagnostic Report of Current VTIP Initiatives. July 2016. PWC. Report on Learning from Centers of Excellence (COE) in ITIs. October 2016. Government of India National Policy for Skills Development and Entrepreneurship (2015) Page 52 of 52