Document of The World Bank Report No: ICR 2316 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-43280 TF-56736) ON A CREDIT IN THE AMOUNT OF SDR 39.0 MILLION (US$ 59.4 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR A SECOND HIGHER EDUCATION PROJECT December 31, 2012 Human Development Sector Unit East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 15, 2012) Currency Unit = Vietnamese Dong (VND) VND 10,000 = US$0.35 US$1.00 = VND 20,840 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CPS Country Partnership Strategy DETA Department of Educational Testing and Accreditation DHE Higher Education Department DO Development Objective DPF Department of Planning and Financing DPL Development Policy Lending GDAA Department of Assessment and Accreditation GoV Government of Vietnam FM Financial Management HEMPIS Higher Education Management and Policy Information System HEP I First Higher Education Project HEP II Second Higher Education Project HERA Higher Education Reform Agenda HERSP Higher Education Reform Support Program ICR Implementation Completion Report IDA International Development association KPI Key Performance Indicator M&E Monitoring and Evaluation MoET Ministry of Education and Training PAD Project Appraisal Document PDO Project Development Objective Development Objectives ct PMU Project Management Unit PRM Project Result Framework QA Quality Assurance QEA Quality at Entry QSA Quality of Supervision SIL Specific Investment Loans TMU TRIG Monitoring Unit TRIG Teaching and Research Innovation Grant Vice President: Pamela Cox Country Director: Victoria Kwakwa Sector Manager: Luis Benveniste Project Team Leader: Thanh Thi Mai ICR Writer: Cecilia Zanetta SOCIALIST REPUBLIC OF VIETNAM SECOND HIGHER EDUCATION PROJECT CONTENTS DATA SHEET A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 2 3. Assessment of Outcomes ............................................................................................ 9 4. Assessment of Risk to Development Outcome..........................................................22 5. Assessment of Bank and Borrower Performance ......................................................23 6. Lessons Learned ........................................................................................................26 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ...........28 ANNEXES Annex 1. Project Costs and Financing ...........................................................................29 Annex 2. Outputs by Component ..................................................................................30 Annex 3. Economic and Financial Analysis ..................................................................40 Annex 4. Bank Lending and Implementation Support/Supervision Processes .............42 Annex 5. Beneficiary Survey Results ............................................................................44 Annex 6. Stakeholder Workshop Report and Results................................................... 47 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 49 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 52 Annex 9. List of Supporting Documents ...................................................................... 53 MAP A. Basic Information Socialist Republic of Second Higher Education Country: Project Name: Vietnam Project (HEP II) L/C/TF Project ID: P079665 IDA – 43280; TF-056736 Number(s): ICR Date: 10/22/2012 ICR Type: Core ICR Specific Investment Socialist Republic of Lending Instrument: Borrower: Loan Vietnam Original Total Disbursed XDR 39.00 million XDR 36.9 million Commitment: Amount: Environmental Category: C Borrower: Socialist Republic of Vietnam Implementing Agency: Ministry of Education and Training and 22 participating universities Co-financiers and Other External Partners: Japan (PHRD) – Co-financing B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/11/2003 Effectiveness: 11/01/2007 03/26/2008 Appraisal: 03/05/2007 Restructuring(s): Mid-term Approval: 06/20/2007 08/16/2010 08/16/2010 Review: Closing: 06/30/2012 06/30/2012 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Low to Moderate Bank Performance: Satisfactory Borrower Performance Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Satisfactory PMU – Highly Quality of Implementing Satisfactory Highly Satisfactory Supervision: Agency / Agencies: TRIG Universities - Satisfactory Overall Bank Overall Borrower Satisfactory Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Quality at Entry Project at any time No None (QEA): (Yes/No): Quality of Problem Project at any No Supervision None time (Yes/No): (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Tertiary education 100% 91% Central Government Administration 0% 9% Theme Code (Primary/Secondary) Education for the knowledge economy 67% 67% Administrative and civil service reform 33% 33% E. Bank Staff Positions At ICR At Approval Vice President: Pamela Cox James W. Adams Country Director: Victoria Kwakwa Klaus Rohland Sector Manager: Luis Benveniste Christopher J. Thomas Project Team Leader: Mai Thi Thanh Jeffrey Waite ICR Team Leader: Mai Thi Thanh ICR Primary Authors: Maria Cecilia Zanetta F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document – PAD) To increase the quality of teaching and research in universities, in ways to improve the employability of graduates and the relevance of research. Revised Project Development Objectives (as approved by original approving authority) The PDO was not modified during the life of the loan. (a) PDO Indicators Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Proportion of TRIG-recipient universities having undergone self- Indicator 1 evaluation and external evaluation at least one under the national pilot accreditation system Value quantitative or 3 n.a. 20 Qualitative) Date achieved 6/30/2007 6/30/2012 6/30/2012 Achieved: The twenty-two TRIG-recipient universities have undergone a self-evaluation process consistent with the guidelines Comments under the national pilot accreditation system. Of these self-evaluations, (incl. % twenty have been subject to an external evaluation and have been achievement) deemed to achieve accreditation standards. Actual accreditation has not been granted, however, since new legislation calls for independent accreditation agencies, which are not still operational in Vietnam. Number of articles published by academic staff in the previous Indicator 2 academic year in international refereed academic journals in TRIG- recipient universities. A total of 508 A total of 207 Greater than the articles, incl. 472 articles, incl. 214 in baseline in all Value in science, 30 in sciences, 20 in three sub-fields quantitative or social sciences, social sciences, and (later specified as Qualitative) and 6 in the arts none in the arts and a total of 250 in and humanities in humanities in 2006i ISRs) 2011 Date achieved 6/30/2007 6/30/2012 6/30/2012 Achieved: The number of publications by TRIG universities in peer- reviewed academic journals more than doubled between 2006 and 2011, particularly in the sciences. While this increase cannot be fully Comments attributed to the TRIG grants, they have had a substantial impact. (incl. % During the life of the project, TRIG funding has resulted, directly o achievement) indirectly, in a total of 2,037 and 876 national and international publications, respectively, including academic journals, conference proceedings, and theses and dissertations.1 1 These include publications resulting from TRIG-funded projects, Ph.D. dissertations and Master’s theses from TRIG-funded students, publications resulting from research that utilized TRIG-funded equipment, and articles that were presented at professional meetings using TRIG funding. (b) Intermediate Output Indicators Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Component 1 – Capacity Building for Policy Development Analytic work and presentation of policy options for the development Indicator 1 of higher education governance issues produced. Several important Analytical work policy studies Value to be phased into have been (quantitative None DPL at time of produced, incl. the or Qualitative) adoption of Higher Education policy option Master Plan Date achieved 6/30/2007 6/30/2012 6/30/2012 Achieved: Several pieces of analytical work have been completed focusing on various aspects of higher education governance. Among these studies is the Higher Education Master Plan, which provides an overview of the strengths and weaknesses of Vietnam’s higher education system and its governance mechanisms. It has been used as a Comments key input by the MoET in the drafting of their recommendations to the (incl. % Party on higher education reform as well as by the Bank’s team in the achievement) definition of the policy actions included under the DPL operations. Several of its recommendations are reflected in the newly enacted Higher Education Act (June 2012) as well as several regulations issued by the Prime Minister and the MoET. Other studies included an overview of private universities, as well as an assessment of the impact of the adoption of a credit-based system. Analytic work and presentation of policy options for the development Indicator 2 of higher education finance mechanisms produced. Several important policy studies have been Analytical work produced, incl. Value to be phased into the determination quantitative or None DPL at time of of unitary costs as Qualitative) adoption of well as a study policy option focusing on public financing mechanisms for higher education. Date achieved 6/30/2007 6/30/2012 6/30/2012 Comments Achieved: The analytical work financed under this sub-component has (incl. % helped MoET departments in the development of policies regarding achievement) public funding for higher education in general as well as determining the allocation of funds to individual colleges and universities based on unitary costs. It also served to improve financial reporting of higher education institutions. Findings and recommendations from the two studies were an important input of the decree regulating on tuition fees for the period 2011-2015 (issued on May 2010). Analytic work and presentation of policy options for the establishment Indicator 3 of accreditation standards, quality assurance mechanisms and admission systems produced. Substantial knowledge building has been Analytical work generated, Value to be phased into focusing on quantitative or None DPL at time of quality assurance Qualitative) adoption of mechanisms for policy option higher education. Date achieved 6/30/2007 6/30/2012 6/30/2012 Achieved: Substantial knowledge building was conducted under the HEP II focusing on the development of a quality assurance system for Vietnam’s higher education. The activities carried out under operation also served to strengthen technical capacity among those responsible for internal quality assurance within individual colleges and Comments universities, helping them implement quality assurance systems and (incl. % gradually build a quality culture. They also served to strengthen the achievement) technical capacity at the MoET’s ability to provide substantive inputs in the drafting of quality-assurance related regulations, including the Comprehensive Framework on Higher Education Quality Assurance and Accreditation, which was adopted in Sept. 2010. Analytical work on admissions was not carried out, as MoET issued new admission regulations. Component 2 – Capacity Building for University Teaching and Research Number and frequency of participation of representatives from TRIG- Indicator 4 recipient universities in training courses Value quantitative or N.A. 22,775 Qualitative) Date achieved 6/30/2007 6/30/2012 6/30/2012 Achieved: A total of 22,775 faculty and staff have attended short training courses. The focus of the training varied across universities, as Comments it was determined by the goals individual TRIG sub-projects. It (incl. % generally focused on research and teaching methods, utilization and achievement) maintenance of technical equipment, curricula development, certification standards, and management capacity. In addition, TRIG universities also received training on sub-project design and implementation, including procurement and M&E. Indicator 5 Positive impact of TRIG scheme as evaluated by independent firm A TRIG Impact Evaluation Study was conducted by TRIG-evaluation Value a panel of report submitted, quantitative or None university validated and Qualitative) researchers and disseminated disseminated among TRIG universities. Date achieved 6/30/2007 6/30/2012 6/30/2012 Achieved: While this evaluation was originally programmed as an ongoing activity to be conducted by an international consultant firm, the contract was eventually cancelled due to the firm’s poor performance. A panel of university experts was subsequently Comments commissioned an independent evaluation of the TRIG scheme. The (incl. % study, which reflected results from more than 500 personal interviews achievement) and 5,500 surveys distributed among administrators, faculty and students of TRIG universities indicates that the TRIG scheme was highly effective, resulting in a significant impact in relation to the project’s PDO. G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 07/26/2007 Satisfactory Satisfactory 0.00 2 10/09/2007 Satisfactory Satisfactory 0.00 3 07/22/2008 Satisfactory Satisfactory 6.38 4 03/20/2009 Satisfactory Satisfactory 8.56 5 10/19/2009 Satisfactory Satisfactory 18.80 6 10/18/2010 Satisfactory Moderately Satisfactory 28.86 7 05/01/2011 Satisfactory Satisfactory 35.32 8 10/31/2011 Satisfactory Satisfactory 40.48 H. Restructuring (if any) Board ISR Ratings at Amount Reason for Restructuring Approved Restructuring Disbursed at Restructuring & Date(s) PDO Restructuring Key Changes DO IP Change (in US$ million) Made N.A. I. Disbursement Graph 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal Having recognized the key importance of higher education and research for the country’s economic and social development, the Government of Vietnam (GoV) has made it its priority to expand both quality and access to higher education. The government’s commitment toward higher education was already clearly reflected in the 2006-2010 Socio-Economic Development Plan 2006-2010, which aimed to increase enrollment in universities and colleges by 10 percent annually and to gradually approach the more advanced education standards of the region and the world. Presently, the GoV's priorities in higher education are set out in its Higher Education Reform Agenda (HERA) which reflects a forward-looking strategy for the development of higher education and research. The World Bank has been a long-standing partner supporting the GoV’s higher education reform efforts, including those geared toward developing a quality higher education system that is innovative and responsive to the demands for skills. Approved in 1998, the First Higher Education Project – HEP I (P004828; US$83.3 million) was the first Bank operation to focus specifically on higher education. Based on the HEP I successful experience, the GoV requested the Bank to expand its support to higher education reform efforts. This operation, the Second Higher Education Project - HEP II, built upon its predecessor, providing further support for capacity building among participating universities while complementing a series of development policy operations aimed at strengthening the higher education system as a whole. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) Project Development Objectives (PDOs) and Key Performance Indicators (KPIs) According to the Project Appraisal Document (PAD) PDO 1 – To increase the quality of teaching and research in universities, in ways that improve the employability of graduates and the relevance of research. The KPIs associated to the PDO were:  The proportion of participating universities having undergone self-evaluation and external evaluation at least once under national pilot accreditation system; and  Number of articles published by academic staff in the previous year in internationally refereed scientific journals in TRIG-recipient universities. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification The PDO as well as KPIs remained unchanged during the operation’s lifetime. 1 1.4 Main Beneficiaries Primary Target Groups The project directly benefited:  The Ministry of Education and Training (MoET) by strengthening its policy making capacity; and  Administrators, research and teaching staff, and students from the 22 participating universities. Overall, participating universities have an annual enrollment of roughly 100,000 students in 2011. 1.5 Original Components 1 - Capacity Building for Policy Development. (US4.8 million) This component aimed to develop policy and build capacity in policy development in the key policy areas:  The organization and governance of the higher education system;  The public supply- and demand-side funding of higher education; and  Quality of students at entry and quality of services provided by higher education institutions. 2 - Capacity Building for University Teaching and Research (US$61.4 million ) This component aimed to strengthen teaching and research in selected universities, by:  Building management capacity in universities; and  Financing Teaching and Research Innovation Grants (TRIGs). 3 - Project Management, Monitoring and Evaluation (US$4.3 million) This component aimed to support effective project implementation, including project coordination and monitoring and evaluation. 1.6 Revised Components N.A. 1.7 Other significant changes N.A. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry (QAE) QAE – Satisfactory This ICR rates the project’s QAE as satisfactory. The project was fully aligned with the GoV’s higher education reform strategy and constituted an important element of the Bank’s support strategy for Vietnam’s higher education system. The preparation team fully capitalized on the learning acquired under the preceding operation (HEP I), and incorporated several modifications that contributed to the project’s timely execution and effectiveness. In retrospect, the Project Result Framework could have been further 2 strengthened by including additional KPIs to address the multiple dimensions of the PDO. The specific design strengths and weaknesses can be summarized as follows: Assessment of Quality at Entry – QAE Strengths  Strong learning from preceding operation: Based on the experience under HEP I, the implementation of the TRIG subcomponent was improved as follows: i) the selection process of participating universities took place during preparation as opposed to after effectiveness, which avoided having a slow project start-up; ii) participating universities were required to complete highly detailed standardized templates as part of their application package for procurement, training, etc., which provided the foundations for a sound monitoring and evaluation (M&E) system; and iii) substantively, participating universities were induced to focus more on “soft investments� including staff development and research activities rather than just in the purchase of equipment.  Balanced targeting criteria: The project’s targeting effectively balanced efficiency and equity criteria, focusing on two distinctive groups of universities. The so-called Group 1 included 17 of the most established universities in the country, while Group 2 included five universities located in disadvantageous areas, such as remote mountainous regions of the country and the Mekong delta. Moreover, among the most established universities (i.e., Group 1), the project focused only on those that had demonstrated ability to perform under the HEP I, thus, maximizing the project’s likelihood of success and potential impact.  Effective capacity building approach: The utilization of a demand-driven approach to determine the specific interventions to be supported under the TRIG subcomponent served to ensure the alignment between the project’s support and the needs of participating universities. In addition, requirements for TRIG proposals, which required their full consistency with the universities’ current strategic plan, as well as the selection process by independent panels ensured the quality of the proposals and provided applicants with access to expert technical support and advice, which was particularly important for Group 2 universities.  Effective institutional arrangements: The project’s institutional arrangements supported stakeholder autonomy and accountability. Within Component 1, responsibility for policy development activities was assigned to the corresponding MoET's core business units, thus contributing to their institutional strengthening and the quality of the outputs. Within Component 2, universities were responsible not just for determining the TRIG-supported activities, but for the corresponding procurement processes as well. This constituted an innovation with respect to the preceding operation, under which the MoET had required that all procurement be done at the central level. The HEP II decentralized approach promoted the universities’ autonomy and enhanced transparency and accountability. The decision also denoted the MoET’s ability to capitalize on learning from the HEP I experience.  Robust M&E system: The Bank and Government team correctly identified the need to design and develop a sound M&E system as a key project input and developed the tools needed to implement it (e.g., TRIG templates). Although it proved difficult to implement, the project adopted an innovative approach toward M&E, calling for a 3 consultant firm to conduct an independent evaluation of the TRIG grants concomitantly with implementation.  Multi-pronged Bank support strategy: The operation benefited from the Bank’s multipronged support strategy for Vietnam’s higher education, which included two specific investment loans – SILs (i.e., the HEP I and II), a series of Development Policy Lending operations - DPLs (i.e., Vietnam’s Higher Education Reform Support Program - HERSP) to support the government’s reform efforts over five-year time span, and substantial sector work.2 The two SIL operations provided much needed capacity building within the MoET and among participating universities, effectively complementing the policy-oriented DPL operations. The SILs contribution in terms of capacity building is particularly important in countries such as Vietnam, where capacity constraints can seriously hindered the effective implementation of higher level policy reforms. Weaknesses  Weak Project Result Framework (PRF): The PRF could have been further strengthened by: i) establishing a more direct relationship between the project’s activities and the PDO; and ii) including additional KPIs to capture more fully the multiple dimensions embedded in the PDO. Specifically, according to the PDO, the project aimed to increase the quality of both teaching and research in universities, in ways that improve the employability of graduates and the relevance of research. The importance given to employability and relevance of research by including under the PDO seems to have been overemphasized as the unemployment rate of higher education graduates at the time of the project design was less than 3%. Increasing the quality of teaching and research through encouraging university-industry links, which are critical to ensure the alignment of labor skills supply and demand as well as to ensure the relevance of research.3 The KPIs, therefore, were not fully aligned with the PDO. While one of the two key outcome indicators focused on research quality (i.e., number of articles published in academic journals), the second one focused on the proportion of participating universities undergoing self-evaluations processes under the pilot national accreditation program. While this had been a central area of focus of the HEP I, it was not directly supported under the HEP II and, more importantly, did not capture the project’s impact on quality of teaching. 2.2 Implementation The project benefited not only from a solid design but also from a particularly stable implementation environment, strong ownership on the part of the MoET, and very strong teams on both the Borrower’s and the Bank’s sides. These factors resulted in the HEP II 2 World Bank (2012), Putting Higher Education to Work: Skills and Research for Growth in East Asia, East Asia and Pacific Regional Report, Washington, D.C.; World Bank (2008), Vietnam: Higher Education and Skills for Growth, Human Development Department, East Asia and Pacific Region, Washington, D.C. 3 University-industry links were encouraged indirectly, by including representatives from industry and/or business in the independent TRIG evaluation panels and in the project’s Advisory Committee. These links could have been more directly encouraged, for example, by requiring universities to partner with industry and/or business as part of the TRIG eligibility criteria. 4 being the first project in Vietnam’s education portfolio to fully disburse within the original timeframe. The most notable implementation difficulty was in the development of the Higher Education Management and Policy Information System (HEMPIS), which was eventually cancelled. The factors affecting implementation include: Factors Affecting Project Implementation Factors outside the control of the Government and the implementing agency  There were no negative external factors affecting implementation. The GoV managed the 2008 financial crisis effectively and the flow of counterpart funds took place as anticipated, even during the height of the crisis.  Solid Bank team with strong field presence: The project benefited from a particularly strong team on the part of the Bank, with long-term engagement in Vietnam’s education sector, deep first-hand sector knowledge, and a strong field presence. During supervision, the Bank team also exhibited a highly proactive project management, anticipating implementation bottlenecks and addressing them effectively. Factors Generally Subject to Government Control  Stable support at the national level: The project received consistent support on the part of the government. The MoET has a clear strategic vision with respect to both the importance and the role of higher education in the country’s economic and social development and the needed reforms. These, in turn, are reflected in the reform strategy (HERA). The project, being an integral part of such strategy, received full support throughout implementation.  Failed implementation of HEMPIS: The development of the (HEMPIS proved to be more difficult than anticipated and was eventually cancelled. Among the factors that contributed to its failed implementation was the lack of a clear definition regarding its overall objective. Specifically, there was no agreement on whether it was aimed at compiling and reporting statistical data or, alternatively, it was going to serve as a more ambitious management tool. Likewise, there was no agreement on whether the focus was on enhancing transparency on the use of resources at the university level or, alternatively, the emphasis was to be placed on results. The lack of a clear definition was partly related to the large number and diverse nature of the MoET departments involved in the process—e.g., higher education, planning and finance, science and technology, facilities, international training and testing and accreditation—without any one entity assuming a strong leadership role. Moreover, it is important to point out that the roughly four hundred higher education institutions that were expected to feed data to HEMPIS, most of which already have their own information management systems in place, were not involved in the process. Factors Generally Subject to Implementing Agency Control  Solid counterparts. The PMU was well staffed with a team of top-notch professional that had solid administrative experience and commanded respect among the participating universities and the MoET business units. Likewise, 17 of the 22 participating universities not only had participated in the preceding operation but were also the best performers.  Active project management: The project was actively managed by both the PMU and 5 Bank, which anticipated and effectively corrected implementation weaknesses and bottlenecks (see Section 5 – Bank and Borrower Performance for a more detailed discussion. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization Monitoring and Evaluation (M&E) Monitoring and Evaluation (M&E) Design The development of a sound M&E was an important priority at the design stage and included several innovative features, including:  Indicators: The MoET was to monitor four types of indicators, including: i) Government-strategy indicators: These indicators, which are directly linked to the objectives outlined in the GoV’s strategy HERA, were to be monitored for TRIG universities on an annual basis to assess the overall progress of GoV’s higher education reform strategy. They reflected the main TRIG activities, including training of faculty members at the Master’s and Ph.D. levels and employability of graduates. It would have been a valuable addition to the project’s M&E framework to also monitor HERA indicators for non-TRIG universities as well to be able to compare HERA performance for TRIG and non-TRIG universities. ii) Project-level outcome indicators: These indicators were intended to measure the project’s impact on quality of teaching and research. However, as discussed earlier, the outcome indicator aimed at measuring teaching quality did so only tangentially, as it focused on the proportion of TRIG universities undergoing self-evaluations. iii) Project-level output indicators: These output indicators corresponded to key activities planned under the various components. iv) University-level indicators: As part of their TRIG proposals, participating universities were required to include a battery of indicators for each set of activities, including baselines and specific targets, according to detailed templates.  Internal audit: In addition to the usual external independent auditors, the MoET appointed an Internal Auditor to identify weaknesses early on in the process and build capacity as needed. The internal auditor function was also to be replicated at the university level, for which substantial training was provided.  Independent TRIG evaluation: It was anticipated to contract a consultant firm to conduct a TRIG evaluation study to determine the effectiveness of the various interventions supported under the TRIG component. Data was to be collected and analyzed throughout project implementation to obtain adequate information for data analysis. Monitoring and Evaluation (M&E) Implementation The project’s M&E system was partially implemented as compared to the original design. Specifically:  Indicators: i) Government-strategy indicators: HERA indicators were monitored as anticipated for the TRIG universities. ii) Project-level outcome indicators: The outcome indicator on self-evaluations was monitored regularly. Alternatively, the outcome indicator focusing on the number of papers published in academic journals was technically too complex for the 6 PMU to monitor routinely.4 Instead, the PMU relied on the number of publications reported by universities as being funded by TRIGs. However, it soon became apparent that there were no clear criteria determining the reporting of publications in terms the type of publication or their relationship to TRIG funding. Efforts were made to fine tune the criteria. iii) Project-level output indicators: These output indicators were closely monitored by the PMU and highly instrumental in facilitating implementation monitoring and the Bank’s supervision. iv) University-level indicators: During implementation, universities collected and reported sub-project level indicators periodically. These indicators constituted an indispensable tool for the PMU to monitor project implementation at the university level and ultimately ensure the full utilization of funds.  Internal audit: The internal audit function did not function as expected. An internal audit coordinator has been appointed at the national level and technical assistance, including internal audit methodology and training, was provided to participating universities by a consulting firm. This was designed to support participating universities to follow a report-based disbursement mechanism. However, the actual disbursement method remained traditional (transaction-based) hence universities focused their efforts in managing their accounts/receipts/invoices rather than building financial management system with functioning internal audit. Therefore internal auditing was not emphasized or closely monitored by the PMU, which led to the downgrading of the project’s Financial Management rating in 2011.  Independent TRIG evaluation: A consultant firm as planned was hired during project implementation, but it failed to collect and analyze the data during implementation as originally anticipated. Its contract was eventually cancelled as a result of substantial delays in the submission of deliverables. As a substitute, a panel of researchers was contracted toward the end of the project to conduct an independent evaluation of the impact of TRIG grants. As opposed to the originally envisioned study, which was to systematically assess the impact of the TRIG interventions by comparing “before,� “during� and “after,� this evaluation is based mainly on the level of satisfaction reported by administrators, teaching and research staff, and students from participating universities “after� the project. Thus, its usefulness in assessing the impact of TRIG interventions is more limited. Monitoring and Evaluation (M&E) Utilization The M&E, in particular the monitoring of project level output indicators was effectively utilized to monitor implementation progress, including the early detection of implementation weaknesses, bottlenecks and delays. It was a key instrument in ensuring the timely and successful implementation of the project. The PMU used the framework to benchmark TRIG universities’ performance with regards to: (i) number of contracts signed; (ii) disbursement; (iii) number of research project launched; (iv) number of labs installed and put into use; (v) number of curricula revised/updated etc. The PMU had very strong ownership of its M&E role, which is consistent with the responsibilities at the 4 The operationalization of this indicator was rather technical and required access to specific electronic databases. 7 ministry level. While all participating universities fulfilled their reporting responsibilities, they did so with different levels of promptness and thoroughness. For a selected group of universities, the utilization of the project’s M&E system went beyond being a project requirement and became an important self-management tool. The project-level outcome indicators and the independent TRIG evaluation, however, were not fully effective in capturing the project’s impact for the reasons discussed above. 2.4 Safeguard and Fiduciary Compliance Safeguard and Fiduciary Compliance Environmental Safeguards The project was classified as being Category ‘C’ according to the Bank's Safeguards Policies, indicating that it had minimal or no adverse social or environmental impacts. Financial Management – FM Overall, the PMU had adequate financial management systems, software, and trained staff. In addition to standard FM procedures, disbursement applications by universities had to obtain the clearance from the local State Treasury office, which, although it added an extra layer of bureaucracy, also helped ensure the legitimacy of disbursements at the university level. However, the project’s Financial Management rating was downgraded on August 2011 because audited financial statements, auditors’ report and Interim Financial Reports were not always submitted on time and, as discussed earlier, the internal audit function was not carried out as envisioned. Procurement Despite some delays and failed bidding processes, the PMU and participating universities performed their procurement functions satisfactorily. This is evidenced by the fact that all IDA funds were disbursed by the project’s original closing date. This is not a minor achievement, considering the project’s complexity of the project in terms of procurement, which it involved 750 contracts, of which more than 100 were International Competitive Bidding (ICB) processes. Several factors contributed to the project’s satisfactory procurement performance, including:  Detailed procurement plans were required as part of the TRIG proposal prepared by universities before the beginning of the project ;  Universities being responsible for their own procurement procedures;  Intense training on procurement provided to universities, with over 250 of people being trained;  For the procurement of sophisticated technical equipment, universities were required to identify at least three potential suppliers to avoid having overly narrow technical specifications. Low procurement thresholds, however, resulted in roughly 60 to 70 percent of all contracts requiring prior review, resulting in a high volume of work for all parties, particularly the Bank and PMU, as well as implementation delays for the universities. The thresholds have since been raised. Compliance with Covenants All covenants stipulated under the Loan Agreement were met. 8 2.5 Post-completion Operation/Next Phase Next Phase: A potential follow-up operation While the universities have stated a strong demand for a follow-up operation (see Annex 6 – Stakeholders Workshop), the MoET is still to define the areas in which it will seek the Bank’s support in the medium term. From a substantive perspective, there is a natural progression in terms of the focus of a third operation. Specifically, a third operation could combine the demand-based based approach of the HEP I and II operations with key policy priorities to be defined by the MoET. Some of the potential areas to particularly emphasize under a potential HEP III operation include:  University-industry links: A follow-up operation could explicitly focus on strengthening linkages between universities and the private sector (i.e., industry and business) assigning higher priority to teaching and research initiatives that are conceived in partnership with private sector stakeholders.  Creation of regional higher education networks: Such networks would facilitate the “cascading� of the institutional capacity built under the two preceding operations. Specifically, HEP I & II universities could play an active role supporting the development of less advanced colleges and universities in their region, providing technical assistance to strengthen teaching quality, forging collaboration in teaching and research, and developing mechanisms for resource sharing and vertical learning pathways.  Macro-economic impact: A third operation could aim to maximize economic spillovers by giving higher priorities to proposals from universities and/or research institutes that focus on teaching and research activities with strong macro-linkages or that are directly aligned with national priority programs (e.g., energy, environment, agriculture).  Local economic development: The inclusion of Group 2 universities yielded significant economic impact in the local communities as a result of research activities related to extension services. This type of impact could be actively encouraged under a potential follow-on operation.  Other higher education priorities: A third operation could be designed to support specific MoET priorities, such as, for example, strengthening the incipient quality assurance framework by supporting the adoption of international standards by participating universities or the establishment of independent accreditation entities. . 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Relevance: High The HEP II objectives were highly consistent with those of the GoV as well as those of the Bank. Specifically, the operation is fully aligned with the Bank’s 2007-2011 (CPS),5 which, in turn provided support to the GoV’s “Five-Year Socio-Economic Development 5 Report No. 38236-VN, dated January 3, 2007. 9 Plan 2006-2010.� Under CPS Pillar I, which aimed to assist the GoV in its transition to a market economy, the Bank’s program focused on the development a higher education system that produces graduates better suited to the new economy’s needs, measured by changes in the proportion of students finding employment within six months of graduation by field of study. The project contributed to this objective by improving the quality of higher education in ways that enable central government and universities to produce graduates with the knowledge and skills that meet labor market demands. Moreover, the HEP II operation provided direct support to the GoV Higher Education Reform Agenda (HERA) and the achievement of some of its targets, such as increasing the proportion of faculty members with doctoral and Master’s degrees. 3.2 Achievement of Project Development Objectives The achievement of the project’s PDOs is deemed Satisfactory, as the HEP II operation has made a positive contribution toward enhancing the quality of teaching and research both at the policy level and among participating universities. At the policy level, the HEP II operation has contributed to the continuous development of Vietnam’s higher education system by financing key pieces of analytical work that have helped enhance the ongoing policy dialogue, providing evidence-based inputs and expanding the range of potential policy options. These studies have been used to define the reforms to be implemented over the mid-term, including those supported under the Higher Education DPL series. As reflected in the Policy Matrixes of these operations, the HEP II policy-related outputs are important building blocks within the higher education reform strategy being supported by the Bank. In addition, many of the studies financed under this operation have been key inputs in newly adopted higher education legislation and regulations. Other studies have had a direct impact on Vietnam’s higher education system, such as the impact study on the implementation of the credit-based system, which significantly accelerated the adoption of the credit-based system among other universities (See Section 3.5.b – Institutional Change, for a detailed discussion of the HEP II’s impact at the policy level). The HEP II operation has also succeeded in enhancing the quality of teaching and research among participating universities. It has done so by providing Teaching and Research Innovation Grants to support sub-projects identified by each university to improve the quality of teaching and research within the framework of the universities’ strategic plan. While TRIG sub-projects varied greatly among universities in terms of focus and individual activities, they all placed a strong emphasis on the development of human resources, including training of faculty members at the Master’s and Ph.D. levels as well as the development specific skills needed in the context of the specific proposal (e.g., teaching and research methods, curricular development, equipment utilization and maintenance, etc.). TRIG also included funding for individual research projects (these were financed with counterpart funds on the part the universities) as well as the purchase of equipment, including laboratory equipment, teaching and communication technology, and library resources. As consistently reported in the self-evaluation studies prepared by each university, as well as by the administrators, faculty and students interviewed under an independent TRIG evaluation study, TRIG sub-projects have resulted in better 10 prepared administrators, teachers and researchers with adequate tools at their disposal, which, in turn, have positively impacted the quality of teaching and research. The specific achievements for each of the dimensions included in the PDOs can summarized as follows: PDO 1 – To increase the quality of teaching and research in universities, in ways that improve the employability of graduates and the relevance of research. To increase quality of teaching - Achieved KPI 1 - Proportion of TRIG-recipient universities having undergone self-evaluation and external evaluation at least one under the national pilot accreditation system – Achieved  Twenty out of 22 TRIG-recipient universities have undergone a self-evaluation process consistent with the guidelines under the national pilot accreditation system. Out of these self-evaluations, twenty have been subject to an external evaluation and have been deemed to achieve accreditation standards. Actual accreditation has not been granted, since new legislation calls for independent accreditation agencies, which are not yet operational in Vietnam.  The achievement of this indicator indicates that these institutions meet the interim standards for self accreditation defined by the MoET in 2005. These standards focus on quality of overall management, faculty, curriculum, student services and libraries, all of which are important determinants of teaching quality. TRIG Evaluation Study – Results from the TRIG Evaluation study conducted by a panel of independent researchers present a highly positive evaluation of the impact of the TRIG interventions on key factors affecting the quality of teaching and, consequently, on teaching effectiveness itself. Specifically:  Administrators and faculty rate the impact of TRIG interventions on management effectiveness as 3.7 and 3.4, respectively, in a scale of 1 to 5.  Administrators and staff rate the impact of TRIG interventions on program and course development and restructuring as 3.7 and 3.5, respectively, in a scale of 1 to 5. Much of the curricular development was done as part of the transition to the newly adopted credit-base system. In some cases, program and course restructuring was done with input from employers.  More than half of the administrators being interviewed as part of the study (i.e., 54 percent) rate the improvements on teaching effectiveness resulting from TRIG interventions to be 4 or higher in a scale of 1 to 5.6 Some of the specific improvements being reported include: enhanced communication, including more teacher-student interaction as well as communication being more open and friendly; and added emphasis on self-learning and creative problem solving; development of teamwork skills and other important soft skills.  Administrators and staff rate the impact of TRIG interventions on the development of lab-related teaching materials as 3.7 and 3.5, respectively, in a scale of 1 to 5. In most 6 No results were reported for faculty responses. 11 cases, these activities complemented the expansion and/or upgrading of labs as well as the establishment of new ones.  Faculty report high levels of utilization of TRIG-financed equipment and materials (i.e., 3.4 in a scale of 1 to 5) for their teaching activities as well as satisfaction with the current level of supply relative to their demand (i.e., 3.2 in a scale of 1 to 5). Both satisfaction and usage are particularly high for library materials. HERA indicators – HERA indicators show that inputs affecting quality of teaching have improved substantially between 2007 and 2012 among TRIG universities. Likewise, HERA indicators focusing on student performance also exhibit a sustained improvement, suggesting an underlying improvement in quality of teaching during the same period. Specifically:  Input HERA targets and indicators affecting the quality of teaching exhibited a sustained improvement for both Groups 1 and 2 among TRIG universities between 2007 and 2012; including: i) the quality of the teaching staff as denoted by their level of education (i.e., percentage of faculty holding Master’s and doctoral degrees); ii) the student/teacher ratio; and iii) the percentage of teaching staff receiving training in teaching methods. The only exception is for the percentage of teaching staff with doctoral degrees in more disadvantageous universities (i.e., Group 2), which decreased slightly during the same period as the enrolment expanded in the more disadvantaged areas where group 2 TRIG universities locate.  Outcome HERA indicators related to quality of teaching also show improvement for both Groups 1 and 2 among TRIG universities between 2007 and 2012; including: i) average scores in standarized tests for graduating seniors; ii) the average number of years of university enrollement among graduating seniors; and iii) the proportion of graduating seniors continuing their studies at foreign institutions. While there are multiple factors that affect student performance, including student quality; the quality of teaching is undoubtedly one of the main factors.  While the lack of data on non-TRIG universities does not allow to isolate the effect of the project, it is reasonable to infere that the project has had a positive effect on both the inputs affecting quality of teaching and quality of teaching itself, as refelcted in enhanced student performance between 2007 and 2012. 12 Table 1. HERA Indicators Related to Quality of Teaching among TRIG universities, 2007-2012 Group 1 Group 2 HERA Indicators TRIG-Universities TRIG-Universities 2007 2010 2012 2007 2010 2012 Inputs affecting quality of teaching Proportion of university teaching staff with Master’s level degrees to increase to 40 percent by 2010, 44.4 45.5 50.0 34.5 45.2 38.8 and to 60 percent by 2020 Proportion of university teaching staff with doctoral level degrees increased to 25 percent by 2010 and 22.7 25.8 24.6 3.4 4.4 3.2 to 35 percent by 2020. Ratio of university students to teaching staff 22.6 20.7 18.2 19.6 21.4 16.6 reduced to 20:1 by 2020. Proportion of teachers who received training in 44.4 65.4 75.9 68.0 78.6 77.4 teaching methods durign the previous year. Outcomes related to teaching quality Average score in standarized student tests for 4.4 6.7 6.3 5.0 7.0 5.8 graduating university students for the previous year. Average number of years of enrollment for graduating university students for the previous 4.6 4.5 3.6 3.6 3.4 3.2 year. Proportion of students from the last graduating class that pursue further studies in foreign 1.7 2.2 2.4 0.1 0.2 0.4 institutions upon six months of graduation. Source: PMU (October, 2012). To increase quality of research - Achieved KPI 2 - Number of articles published by academic staff in the previous academic year in international refereed academic journals in TRIG-recipient universities. – Achieved  The number of publications by TRIG universities in peer-reviewed academic journals more than doubled between 2006 and 2011, particularly in the sciences. Specifically, a total of 508 articles by faculty in participating universities were published in peer- reviewed journals in 2011, compared to only 207 in 2006. The bulk of the publications continued to be in the sciences, which amounted to 472 publications in 2011 compared to 207 in 2006. While they continued to represent a small fraction of the total, the number of publications in the social sciences, as well as in the arts and humanities also increased between the two periods, from 20 and none in 2006 to 30 and six in 2011, respectively.7  While the university-wide increase in the number of peer-reviewed academic 7 A bibliographical search was conducted for both 2006 and 2011 using the following databases: i) the Science Citation Index Expanded; ii) the Social Sciences Citation Index; and iii) the Arts Humanities Citation Index to identify the number of publications associated with the 22 TRIG universities in peer- reviewed academic journals. The number of articles identified for 2006 was slightly lower than the one shown on the PAD (i.e., 207 versus 230), most probably due to differences in translation of the name of the universities. The data reported above was, thus, modified to reflect the same operational definition for both time periods—i.e., 2006 and 2011. 13 publications cannot be fully attributed to TRIG grants, they have certinly had a substantial impact. During the life of the project, TRIG funding has resulted, directly o indirectly, in a total of 2,037 and 876 national and international publications, respectively, including academic journals, conference proceedings, and theses and dissertations. 8 TRIG Evaluation Study – Results from the independent TRIG Evaluation study report highly positive impacts on research skills and availability of lab equipment, which, in turn, can be expected to enhance the quality of research itself. Specifically:  Teaching and research faculty members report enhanced ability to perform the various tasks involved in conducting research as a result of training or research funding received under TRIG initiatives. Specifically, in a scale of 1 to 5, they rank the impact of TRIG initiatives as 3.48 in terms of enhancing their substantive knowledge on the research topic; 3.53 in terms of their ability to identify and select of a research topic; 3.44 in terms of their ability to write a research proposal; 3.61 in terms of their ability to collect data; 3.53 in terms of their ability to analyze and interpret data; and 3.52 in terms of their ability to write a research paper.  Faculty report high levels of satisfaction with the current level of supply of lab equipment relative to their demand (i.e., 3.8 in a scale of 1 to 5). HERA indicators – The one HERA indicator focusing on research shows an improvement in the quantity of publications. Specifically:  The number of articles published in international academic journals per faculty member has increased substantially between 2007 and 2012 for both Groups 1 and 2. Table 2. HERA Indicators Related to Research among TRIG universities, 2007-2012 Group 1 Group 2 HERA Indicators TRIG-Universities TRIG-Universities 2007 2010 2012 2007 2010 2012 Outcomes related to research Number of articles per faculty member published in international journals during the previous year (x 6.2 6.6 10.9 0.4 4.2 1.5 100) Source: PMU (October, 2012). Indirect impacts – Employability and relevance of research Employability – Undetermined  As discussed above, the project’s impact on employability is expected to be 8 These include publications resulting from TRIG-funded projects, Ph.D. dissertations and Master’s theses from TRIG-funded students, publications resulting from research that utilized TRIG-funded equipment, and articles that were presented at professional meetings using TRIG funding. 14 indirect—i.e., higher employability as a result of better teaching and research. Given that the project did not include any specific activites to better align the supply and demand of skills, no direct impact can be expected. Moreoever, given information imperfections, labor markets do not react immediately, for which the impact of the project on employability is expected to be negligible at this time but to increase over time.  Data from HERA shows that the proportion of graduates being employed within six months of graduation increased for TRIG universities in Group 1, from an average of 68.4 percent in 2007 to 74.2 percent in 2012. Curiously, HERA data show that employability decreased among graduates from TRIG universities in Group 2—i.e., those in disadvantageous or remote locations—from an average 69 percent in 2007 to 51.5 percent in 2012.  Data from VHLSSs and Labour force surveys in 2010 showed the the overall high employement rate of higher education graduates (more than 95%) and the continued rise in returns to higher education relative to other levels of educational attainment. Figure 1. Estimated return to education on earnings, relative to no education 80% 77% 70% 65% 60% 49% 50% 47% 50% 54% 40% 46% 29% 30% 23% 18% 32% 20% 26% 10% 25% 10% 2% 0% 7% 7% 7% Under Primary Secondary Elementary High Secondary Trade College University primary vocational school vocational school 2007 2010 Relevance of research – Likely to have been achieved  The same considerations apply to relevance of research. Given that the project did not include any specific activities or incentives to better align the supply and demand of research, the project’s impact on enhancing relevance of reseach is expected to be indirect—i.e., more relevant research as a result of having better trained researchers with international expossure, as well as more sophisticated equipment that makes possible to expand the range of research activities.  There is evidence that indicates that TRIG universities might be conducting more relevant research, as shown by the new research and service contracts with government agencies and industries reported by TRIG universities in their self- evaluation reports. 15 Figure 2. Percentage of revenue from science and technology activities in TRIG universities 14.0 11.8 12.0 9.9 Percentage 10.0 8.0 8.8 5.2 6.0 6.4 4.0 2.0 0.0 1.7 1 2 3 Group  1 Group  2  HERA data shows that revenue from science and technology activities has increased for TRIG universities in both groups, suggesting that these activites respond to the need of government agencies and private companies willing to fund it. Specifically, it increased from 5.2 percent to 11.8 percent between 2007 and 2012 for Group 1— equivalent to a 230 percent increase. Likewise, it increased from 1.7 percent to 8.8 percent between 2007 and 2012 for Group 2—equivalent to a 520 percent increase. While both groups of universities are still from meeting HERA targets (i.e., 15 and 25 percent by 2010 and 2020, respectively), the trend is clearly positive.  Although it is not possible to attribute the revenue increase to the project given the lack of data on non-TRIG universities, the data are consistent with the fact that participating universities have reported additional research contracts and user fees among the project outcomes at the university level. 3.3 Efficiency Efficiency Economic Efficiency The economic analysis performed at appraisal continues to be relevant in view of the project’s actual achievements, which are in line with those anticipated. Specifically the economic analysis took into consideration three types of benefits: i) productivity gains of graduates from participating universities; ii) efficiency gains at the university level and ii) rates of return from research and development investments under the project. Under a set of conservative assumptions, the economic analysis yielded an Internal Economic Rate of Return (IERR) of 34 percent, indicating the project’s strong economic efficiency (see Annex 3). Financial Rate of Return The project’s financial rate of return was not estimated at appraisal. However, there are three observations that indicate the universities will have the ability to finance operating expenditures of TRIG investments beyond the life of the project. Specifically:  Universities complied with 10 percent co-financing requirements as anticipated. 16  In several cases, universities have secured research and service contracts with government organizations and private firms as a result of new equipment financed under TRIGs that will provide them with recurrent additional revenues.  HERA indicators indicate a gradual but continuous increase in universities own- source revenues, independently of the project. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory The HEP II operation is rated satisfactory, as it has attained its objective of enhancing the quality of teaching and research among participating universities. The operation was highly relevant at appraisal and continues to be so at closing, as it reflected the GoV’s medium term reform strategy for higher education. The operation was also an important component of the Bank’s support strategy for Vietnam, which focuses on higher education as a way to contribute to the country’s economic competitiveness under Pillar 1 of CPS 2007-2011 and CSP 2012-2016.9 In this context, the HEP II operation effectively complemented the ongoing series of development policy operations by supporting solid and relevant analytical work as well as much needed capacity building at the MoET and university levels. 3.5 Overarching Themes, Other Outcomes and Impacts Overarching Themes, Other Outcomes and Impacts a) Poverty impacts, gender aspects, and social development  The inclusion of universities located in less advanced regions is important not only from an equity perspective but also in terms of the impact of their research activities on local economic development. (see discussion below).  Results from the surveys and interviews conducted under the TRIG Independent Evaluation Study indicate that there was no discrimination among women or minorities for teaching and research opportunities supported under the project. (b) Institutional change and strengthening As described above, the project has supported important analytical work to complement ongoing higher education reforms. Given the sequencing of this operation and the DPL series, the policy outputs of the HEP II have served as key inputs into the HERSP series, providing an evidence-driven diagnostic of the current situation as well as recommendations and policy options that served to better design the DPL policy actions. In addition, given the ample dissemination of the various pieces of analytical work among the various line ministries, there is now a more systematic and unified view on the importance of higher education policy development. This is important, since higher education institutions are under the orbit of different line ministries, which often results in a fragmented view of the critical issues regarding higher education. The specific impacts in terms of institutional strengthening include: 9 Report No. 38236-VN, dated 3 January 2007 and Report No. 65200-VN, dated 7 November 2011, respectively. 17 Governance of the higher education system The operation provided financial support for analytical work focusing on organization and governance aspects of Vietnam’s higher education system, including the development of a Higher Education Master Plan. The process of developing the Higher Education Master Plan, the actual end-product and its dissemination have served to bring increasing awareness over the main weaknesses of existing organizational and governance arrangements of Vietnam’s Higher Education System, including the fractured nature of the system, and the lack of academic, managerial, and financial autonomy of higher education institutions. The Master Plan’s diagnosis and recommendations were important inputs in the Higher Education Act (approved in June 2012), which represents an important step toward strengthening autonomy and improving efficiency, quality, responsiveness and accountability of higher education and research institutions. The Higher Education Act reflects many of the recommendations of the Master Plan, including the adoption of a multi-tiered system, the definition of functions to be carried out by key bodies within colleges and universities differentiated by tier, and the definition of the management structure at the university level. The Master Plan has also served as an input in the MoET’s proposal for the drastic and comprehensive reform of Vietnam’s Higher Education, which has been submitted to the Communist Party of Vietnam to be considered for adoption. Finally, the recommendations of the Master Plan are being reflected in several regulations issued by the Prime Minister or MoET regarding the overall organization and governance structure of Vietnam’s higher education system. In turn, the approval of Higher Education Act and the various regulations have served to meet key policy conditions under the World Bank’s HERSP. Financing of the higher education system The HEP II operation provided financial support for studies focusing on: i) higher education unitary costs; and ii) public funding mechanisms. This analytical work has helped MoET departments in the development of policies regarding public funding for higher education in general as well as determining the allocation of funds to individual higher education institutions based on unitary costs. It also helps the MoET in the drafting of recommendations to improve the financial reporting of higher education institutions. Findings and recommendations from the two studies were an important input of the decree regulating on tuition fees for the period 2011-2015 (issued on May 2010). Wider expansion of credit-based system The MoET has been encouraging higher education institutions to shift from the current “learning unit� system—a hybrid between the classical education and credit systems—to a pure credit-based system for some time now. Although more flexible than the classical education system, the learning unit system is still quite rigid. Students who enter a program in the same class stay together through the rest of their education; most courses are still compulsory, leaving little room for elective coursework. Additionally, no standard system for calculating learning units across institutions exists yet; thus, the transfer rate is still quite low. While the MoET has been promoting the adoption of a credit-based system, the transition had been very slow. The HEP II was instrumental in accelerating the adoption of credit-based system across the entire system. Specifically, it supported a study on the impact of the adoption of the 18 credit-based system and disseminated its findings widely among higher education institutions in 2010. The study’s positive findings and their dissemination contributed to the rapid adoption of the credit-based system among the higher education system, from only 60 institutions with 486,386 students in 2009 to 109 institutions with 875,188 students in 2011. The study was also an important input in the revision and expansion of regulation on credit-based tertiary training. 19 Quality assurance for higher education Vietnam’s accreditation system is still being developed. In 2002, MoET took an important step in addressing quality in higher education by establishing the Office of Accreditation within MoET’s Department of Higher Education. Subsequently, in 2003, the office was consolidated with the Office of Assessment to form the Department of Assessment and Accreditation (GDAA), which was responsible for developing provisional regulations on accreditation for higher education institutions. In 2005, the MoET issued interim quality standards for higher education institutions and began implementing an institutional accreditation process based on self-evaluation. The HEP II has contributed to the further development of the accreditation system as follows:  It promoted the further insertion of the self-accreditation model by including the self- and external evaluation of participating universities as one of the project’s two KPIs.  The activities carried out under operation also served to strengthen technical capacity among those responsible for internal quality assurance within individual colleges and universities, helping them implement quality assurance systems.  It provided support to two TRIG universities on their efforts to obtain certification under international standards. One of these universities (i.e., Ho Chi Min City Science and Technology) has made substantial progress towards meeting the ABET certification standards for its computer science and computer engineering programs.10 It has already undergone a mock review in 2010 and is scheduled to undergo the actual certification review in mid-2013.  The HEP II activities also served to strengthen the technical capacity at the MoET’s level, in particular GDAA’s ability to provide substantive inputs in the drafting of quality-assurance related regulation, including the Comprehensive Framework on Higher Education Quality Assurance and Accreditation, which was adopted in Sept. 2010. “Hands-on� capacity building By assigning responsibility for the coordinating the preparation of policy-related outputs to the corresponding MoET’s departmental units, the operation has contributed to strengthening their capacity and boost their ownership of the reforms. This, in turn, will facilitate the implementation of the higher education policy agenda after the project’s completion. The TRIG scheme also helped increase autonomy among participating universities. By providing them with discretionary resources and making them responsible for their administration, the HEP II had a significant impact on the institutional capacity of participating universities, strengthening not only research and teaching quality but also their planning, management, procurement, financial administration, and M&E functions, ultimately contributing to their overall autonomy. c) Other unintended outcomes and impacts  Expanded collaboration, both within and across universities in Vietnam and with universities overseas, including : 10 ABET is a nonprofit, non-governmental organization that accredits college and university programs in the disciplines of applied science, computing, engineering, and engineering technology. 20 i) New cooperation agreements across domestic higher education institutions, including shared access to TRIG-financed labs and equipment; ii) Stronger links with foreign institutions, as between the University of Arizona and HCMC University of Science and Technology (Group 1 university), and between institutions in Canada, Australia, Singapore and China and Travinh University (Group 2 university). iii) Hosting of graduate students from foreign universities; iv) Hosting of international conferences and workshops; and v) Expanded personal and professional connections among faculty members.  Generated additional sources of income for universities, such as research contracts with industry and government agencies, as illustrated by: i) Hanoi’s University of Science and Technology’s contracts between the School of Biotechnology and Food Technology and several aquaculture and other food companies; ii) Training contract between INTEL and HCMC University of Science and Technology.  Generated economic impacts at the local level through agricultural extension that had a substantial impact on the living conditions of local population. Local level economic impacts were particularly noteworthy among Group 2 universities, including: i) Food safety – Certification of disease-free zone for a large production area, which allow farmers to sell their products at a higher price; ii) Crop production – Watermelon production methods with water saving production modes by incorporating water tank, irrigation system and timers, which is consistent with climate change adaptation and has important impacts for producers in terms of labor and savings in the use of fertilizers.  TRIG research projects have had important value in solving problems of public interest. Some examples are: i) Food production  New tomato varieties have been recognized (i.e., HT 42 and HT 144);  Virus that affected rice cultivation was identified and isolated; recommendations were made to the Ministry of Agriculture and Rural Development that permitted its elimination by 2010,  A rapid diagnostic kit and vaccine was developed for the prevention of foot-and- mouth disease.  The PRRS virus in pigs was identified and isolated in 2010; recommendations were made to the Ministry of Agriculture and Rural Development to control the epidemic effectively. ii) Health  Although not a new research topic, findings from research project on the treatment of cranial nerve tumors and arterial-venous malformations of the brain have contributed to improve treatment and public health care. 21 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops A workshop was conducted to assess the implementation experience of the TRIG Program from Sept. 28-29, 2012 in Phan Thiết, Vietnam. Representatives from the World Bank, the MoET as well as the twenty two universities that participated in the program attended the workshop. The main conclusions are summarized in Annex 6. 4. Assessment of Risk to Development Outcome Factors Affecting Sustainability of the Project’s Achievements Rating: Low to Moderate Overall, the risk that the progress made under the HEP II operation might not be sustained is considered to be low to moderate. The risk that universities might not be able to finance the anticipated recurrent costs appears to be low.As part of their original TRIG proposals, universities were required to address the sustainability of the investments supported by the TRIG grant, identifying recurrent costs and the corresponding sources of funding after the operation’s completion. In this regard, it is important to consider that, in Vietnam, universities have relatively high levels of autonomy in term of their spending. In addition to the state budget, they generate—and are encouraged to—their own-source resources through student fees, research contracts, and other fee-for-service activities. They are also experienced in mobilizing alternative sources of financed, including public funding, international assistance, and funding from foreign universities, mainly in terms of graduate scholarships. Data from HERA indicators suggest that universities are becoming increasingly autonomous, with own-source funding as a proportion of total funding increasing substantially from 2007 to 2012, from 5.2 percent to 11.8 percent for Group 1 and from 1.7 percent to 8.8 percent between 2007 and 2012 for Group 2. The experience from the HEP I operation indicates that the universities are savvy in the way they have continued to manage the investments under the preceding operation, ensuring their adequate maintenance and operational inputs. This was corroborated by the actions taken by some universities during the HEP II implementation, such as monitoring the use of electronic databases acquired under TRIG to be able to make evidence-based decisions on which subscriptions to discontinue after the project’s completion. The only two considerations regarding potential risks to development outcomes are as follows:  The training of Ph.D. students is incompatible with the duration of an investment operation, as it usually takes longer to complete a doctoral program, which includes the writing of the dissertation. Of the 56 Ph.D. students financed under the operation, only ten were able to complete their studies during the life of the operation. This presents a problem for the remaining students in terms of the availability of financial resources. While both universities and the PMU are exploring alternative sources of funding to ensure the successful completion of their studies, this should be taken into account for future operations.  The TRIG Impact Evaluation Study identified two potential problems regarding the sustainability of equipment financed under TRIG. First, although only in very few 22 cases, the inputs needed to conduct experiments (such as chemicals) are very costly, which might pose a problem if external funding is not secured under research contracts as envisioned. Second, the physical facilities in which the TRIG-financed equipment has been located do not always meet the needed conditions in terms of temperature, level of humidity, ventilation, etc., which might hinder their adequate maintenance. 5. Assessment of Bank and Borrower Performance Bank’s Performance a. Bank’s Performance in Ensuring Quality at Entry Rating: Satisfactory As underscored in Section 2.1, the project benefited from its strong design, which capitalized on the lessons learned from the preceding operation. The project’s scope and focus reflected the Borrower’s and Bank’s mature, long term vision of Vietnam’s higher education and its key role in the country’s economic development. As a result, it was highly relevant from a policy perspective, fully aligned with the GoV’s own higher education reform agenda and an important component of the Bank’s CPS, which supported higher education reforms as a way to enhance the country’s competitiveness. Finally, the demand-driven nature of the project design fostered autonomy and institutional strengthening among participating universities and ensured that the technical assistance provided under the project fully responded to the needs and aspirations of participating universities. b. Quality of Supervision Rating: Highly Satisfactory The fact that the project is the first one in Vietnam’s education portfolio to close in a timely fashion having disbursed 100 percent of the credit proceeds while fully attaining its PDO is certainly an impressive accomplishment. Moreover, it attests to a highly effective management and supervision of the project by the Bank’s supervision team. Project supervision was in-depth and permanent, benefiting from a strong field presence, as well as the continuity and first-hand knowledge of Vietnam’s education sector of key team members. Close communication between the Borrower and the Bank, including regular supervision missions with field trips to participating universities (e.g.,12 universities were visited by the Bank team during the April 2011 supervision mission) helped keep track of the progress being made and identify the challenges being faced at various stages of project implementation. The Bank team effectively used the implementation ratings to signal to the Borrower those areas that needed improvement. An in-depth MTR was conducted on schedule. The Bank supervision team was highly proactive throughout the life of the project, working closely with the government in key aspects of project implementation and effectively helping to anticipate and address all potential bottlenecks. Specifically:  In the preparation of policy-related outputs, the team succeeded in convincing the government to have an effective mix of international and local resources that both 23 capitalized on their relative competitive advantages and helped expedite procurement processes. For example, large sector studies that called for qualified international consultancy services were paired with studies produced by local consultants that focused on providing an overview of the current situation. These studies proved to be critical inputs into the larger sector studies and, at the same time, helped the project to deliver tangible outputs early on in the implementation process.  The team also exhibited a highly proactive approach to help the government to manage the project funds. Specifically, surplus funds that resulted from procurement savings, activities that were cancelled and more favorable exchange rates were timely reassigned to Sub-component 2.1, increasing the funds available under TRIG grants. Eligibility criteria for these additional funds were consistent with the spirit of the project, and were allocated only among those universities with a demonstrated ability to implement.  Finally, when the international consultant firm that had been contracted to perform the independent evaluation of the project failed to deliver, the Bank team suggested the PMU alternative ways to conduct independent evaluations of the various project components by relying on independent panels of experts from recognized local universities. c. Justification of Rating for Overall Bank’s Performance Rating: Satisfactory Overall, the Bank’s performance is considered satisfactory, as it effectively prepared, appraised, and supported the implementation of the HEP II operation, ensuring its relevance and attainment of objectives in an efficient and timely manner. Borrower’s Performance a. Government’s Performance Rating: Satisfactory Top authorities from the MoET and participating administrative units provided their full support to the operation, including ensuring the stable flow of counterpart funds even during difficult macro-economic periods resulting from the 2008 international financial crisis. Their support is not surprising, given that the HSP II operation was fully aligned with the government’s own Higher Education Reform Agenda (HERA). It is also worth noting the enhanced responsibility given to universities in terms of procurement under this operation, which denotes the commitment of MoET authorities to strengthening universities’ autonomy and their ability to learn from the experience of the preceding operation. b. Implementing Agencies’ Performance Rating: Satisfactory National PMU: Its performance is deemed highly satisfactory. Given the project’s logistical complexity, sound project management was a necessary condition for its successful implementation. Throughout the project’s lifetime, the PMU exhibited strong ownership of the project, a clear strategic vision, and the ability to provide strong 24 technical and institutional support to participating universities. As discussed in the Bank’s performance section, the project benefited from a highly proactive management approach on the part of both the Bank and the PMU. Implementation bottlenecks, such as procurement delays and undisbursed funds were detected early and addressed effectively to ensure the project’s smooth implementation. The PMU also exhibited a strong capacity for M&E functions, which contributed toward improving project management throughout the life of the project. The PMU also succeeded in coordinating procurement activities from participating universities and provided support as needed. As mentioned earlier, this is not a minor accomplishment given the large number of contracts and ICB processes conducted under the operation. While the internal auditing function, however, was not implemented as originally envisioned, the project’s FM was adequate and weaknesses flagged by the Bank were effectively addresses. The high level of professionalism of the PMU team, their previous management experience in academic and research institutions, as well as their stability throughout the life of the operation were important factors contributing to the sound project management. It is also worth noting the strong counterpart role played by the representatives of the MoET administrative units (i.e., the Department of Planning and Finance; and the General Department of Education Testing and Accreditation) in the preparation of policy analytical work under Component 1. They closely supervised the work of consultant firms, both local and international, to ensure their overall quality and relevance for Vietnam’s context. At the same time, they provided consultants and international experts the freedom to provide candid assessments and recommendations. The sound quality of the analytical work supported under the HSP II operation is a direct result of their strong performance and ownership. University-level TRIG Monitoring Units (TMUs): Their performance is considered satisfactory. While there were differences in their relative level of performance, participating universities met the participation requirements satisfactorily. They all prepared TRIG proposals according to the specifications, including detailed information on expected outputs, costs, and implementation schedule. They all established a TMU with qualified staff as required and maintained throughout the life of the project. They also complied with reporting requirements, albeit with different levels of timeliness. The overall implementation of TRIG programs was satisfactory—with actual vis-à-vis anticipated disbursements ranging from 40 to 150 percent, with an average of 94.1 percent. c. Justification of Rating for Overall Borrower’s Performance Rating: Satisfactory Overall, the Borrower’s performance is deemed satisfactory for the reasons stated above. 25 6. Lessons Learned Lessons Learned from the Implementation of HEP II Lessons of wide general application Valuable lessons can be extracted from the implementation experience of the HEP II, which could be useful in the design of future operations focusing on higher education in Vietnam and elsewhere. The main lessons can be summarized as follows:  The key ingredients for project success always go beyond the sphere of project itself. The HEP II benefited from a clearly defined medium term higher education reform strategy on the part of the GoV, which has the support of all key stakeholders. Moreover, the Bank’s support strategy at both the country level and for the higher education sub-sector is in full alignment with the GoV agenda. Sustained political support and a relatively stable implementation environment were also key factors in the HEP II success.  A proper combination of lending instruments is crucial to consolidate complex reform processes. The Bank’s support for Vietnam’s higher education reform efforts has adopted a multi-pronged strategy. It combines both SIL and DLP operations that have been conceived as a whole and fully complement each other. The SIL operations were designed to generate analytical inputs needed for the formulation of the overall reform strategy. In addition, they were aimed at building much needed technical and institutional capacity at the central government and university levels. In turn, the DPL series has provided substantial financial incentives to move forward more challenging and politically costly policy reforms.  Sequential operations offer great learning potential. As illustrated by the experience of the HEP II, follow-on operations can effectively build upon the implementation experience of preceding operations to enhance both the achievement of objectives as well as their implementation. The HEP II design benefited from strong learning on the part of the Bank and the MoET, as well as the implementation capacity built by participating universities under the preceding operation.  Active project management is a key ingredient for successful project implementation. With the support of a robust M&E system, the PMU and the Bank were able to identify early on implementation weaknesses and potential bottlenecks. These were addressed promptly and decisively, making all the needed adjustments in line with the spirit of the project. This is illustrated, for example, by the performance-based reallocation of unused project funds among participating universities.  M&E systems have to be conceived as a critical element of an operation’s design. Although it was not fully implemented as originally anticipated, the development of a robust M&E system was a key feature of the HEP II design and an important factor determining its successful implementation. The M&E was particularly effective in tracking implementation progress at the university level—a complex task given the diverse nature of TRIG sub-projects, the decentralized implementation of the project, 26 and the large number of implementing entities (i.e., twenty two universities with a wide range of institutional capacity). The information requirements of the M&E system were taken into consideration at the design stage, and informed, among other aspects, the format and content of TRIG proposals.  Strong field presence, continuity and a sector-wide view on the part of the Bank teams give operations a substantial implementation advantage. The HEP II Bank team had several advantages, including being based on the field and being responsible for the entire higher education portfolio in Vietnam, including SIL operations (i.e., the HEP II and the preceding operation) and the DPL series.  It is important to ensure the full alignment between the PDO and the actions supported under the operation, as well as the alignment between the PDO and the Key Performance Indicators. In the case of the HEP II, the PDO encompassed multiple dimensions, which were not all fully captured by the Project Result Framework. In particular, enhancing the quality of teaching was not captured by its corresponding KPI (i.e., universities conducting quality self-assessments). At the same time, the PDO included some dimensions (i.e., enhancing the relevance of research and employability of graduates), that were only indirectly supported by specific actions or incentives under the operation  A project’s design can, in itself, reinforce its underlying strategic objectives. In the case of the HEP II, the demand-driven nature of the TRIG sub-projects was fully consistent with the emphasis on enhancing universities’ autonomy under the GoV’s higher education reform strategy. In addition, it ensured the alignment between the project’s support and the recipients’ self-determined needs. Moreover, by requiring that TRIG proposals be consistent with the universities’ strategic plans, the project avoided fragmenting the project’s impact, which is one of the risks of the demand- driven approach.  Balancing efficiency and equity targeting criteria can have multiple advantages. TRIG universities included both “top� and “disadvantageous� universities to balance efficiency and equity criteria. An unforeseen advantage of including “disadvantageous� universities was their strong economic development impact at the community level as a result of the agricultural extension nature of their research activities. It is also worth noting that a “disadvantageous� university had the highest level of actual disbursements relative to its initial allocation of funds, underscoring its performance potential as well as the effective support they received from the PMU. Project-specific lessons  The project’s lifetime was incompatible with some of its expected outcomes. The training of Ph.D. students, which was one of the project’s intermediate outcomes, is incompatible with the project’s duration, as it usually takes longer to complete a doctoral program, which includes the writing of the dissertation. The same problem arises in relation to one of the project’s outcome indicators—i.e., the publication of research papers in peer-reviewed academic journals, which is also a lengthy process.  Support for staff development requires an integrated, medium-term perspective. While the HEP II supported doctoral and Master’s level training of faculty members, not all universities were well prepared to fully capitalize on it. The universities that 27 benefitted the most were the ones that had already in place a long-term staff development strategy that had identified skill gaps as well as a pool of potential candidates. Some universities imposed additional eligibility criteria for their candidates, including adequate language skills and proven ability to perform by having published previously.  Ensuring the sustainability of equipment investments is always a challenge. While universities were required to ensure the adequate maintenance and operation of the equipment acquired under the operation, this is always challenging, particularly in the case of highly specialized technical equipment that requires expensive inputs as well as a controlled physical environment.  TRIG subprojects offered great potential for economies of scale and cross- fertilization. TRIG subprojects, which were being simultaneously implemented by multiple universities, offer a great potential for economies of scale and integration. As was the case in the HEP II, there was the opportunity to develop management tools at the national level that could be used by individual universities to support the program’s implementation, thus capitalizing on potential economies of scale and easing the institutional burdening on individual universities. In addition, this type of projects can serve as incubators, fostering the generation of innovative solutions for common problems. The ultimate impact, however, depends on the ability of higher levels of government—in this case the MoET—to recognize the value of an innovation and its potential for replication, and to ensure its dissemination among other participants. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies: The HEP2 has contributed to a new development stage of many leading universities in Vietnam. It has also offered universities the opportunity to improve their accountability, skills of project implementation, management and monitoring (see Annex 7 for a more detailed evaluation on the part of the Borrower). (b) Co-financiers The operation received US$4.8 million co-financing from Japan in the form of a Bank- administered Trust Fund, of which US$2.6 million were utilized. The Government of Japan was not involved in the project’s implementation; thus, their comments have not been requested. (c) Other partners and stakeholders N.A. 28 Annex 1. Project Costs and Financing (a) Project Cost by Component (in US$ Million equivalent) Percentage Appraisal Actual/Latest of Components Estimate (US$ Estimate Appraisal millions) (US$ millions) (%) 1 Capacity building for policy 4.5 2.3 51 development 2 Capacity building for university 61.3 58.2 95 teaching and research 3 Project management, monitoring 4.0 2.9 73 and evaluation Total Baseline Cost 69.7 Physical Contingencies 0.4 Price Contingencies 0.4 Total Project Costs 70.5 63.4 90 Total Financing Required 70.5 Note: As of October 2012. (b) Financing Appraisal Actual/Latest % of Estimate Estimate Appraisal Source of Funds (US$ (US$ millions) millions) Counterpart funds 6.3 5.45 86.5 SR Vietnam 0.8 0.95 118.6 TRIG Universities 5.5 4.5 81.3 IDA (World Bank) 59.4 55.5 93.4 Japan ((PHRD) 4.8 2.58 53.8 Total 70.5 63.5 90 29 Annex 2. Outputs by Component Component 1 - Capacity Building for Policy Development (US$4.8 million) Achievement: Satisfactory This component provided financial support for analytical work needed as inputs in the ongoing higher education reform efforts being carried out by the GoV and with World Bank support under the IBRD’s HERSP DPL series. The specific focus of the analytical work was on governance, financing and quality assurance aspects of Vietnam’s higher education system, each of which constituted an individual sub-component. To ensure sustainability of the capacity building activities, each of the three sub- components was implemented by the MoET department responsible for the relevant policy area: Higher Education Department (DHE), Department of Planning and Financing (DPF) and Department of Educational Testing and Accreditation (DETA). Coordination across these sub-components was achieved through a Senior Management Group, which brought together on a regular basis the heads of each of the three implementing units, the Project Director and the TRIG Manager. Each of the various activities involved a small counterpart teams constituted of MoET and university staff with external support from a consultant firm. This mix of local and international resources helped ensure strong ownership on the part of domestic stakeholders, an understanding of the country’s conditions, access to international good practice, and the transfer of skills from international and national experts. In each case, the teams' analytic work was combined with the design and implementation of practical applications through planning retreats, review workshops, policy seminars. Sub-component 1.1 – Higher education governance Description: This sub-component was implemented by MoET's Dept. of Higher Education. It focused on organizational and governance aspects of the higher education system at central higher education institution levels. Technical assistance provided under this sub-component included policy advice and capacity building focusing on: (a) developing policy options and recommendations on the organization and governance of the higher education sector; (b) developing policy options and recommendations on a regulatory frameworks for the integration of advanced research and teaching and for the promotion of private sector investment; and (c) building a new HEMPIS using a set of standard key indicators to enable decision-makers to monitor the performance of the higher education sector. Achievement of Intermediate Results  Organization and governance Achieved: A Higher Education Master Plan was prepared with input from 50 colleges and universities, six research institutions, and ten employers. It provides an overview of the higher education system in Vietnam, including the main weaknesses, and offers recommendations. An independent review panel constituted by representatives of higher education institutions concluded that the Master Plan presents an accurate depiction of Vietnam‘s higher education system and a series of key recommendation for policy reform that, if implemented, would drastically transform and modernize it.  Regulatory frameworks for the integration of research and teaching and private sector investment. 30 Achieved: Specific recommendations are included in the Higher Education Master Plan.  Higher education management and policy information system (HEMPIS) Not Achieved: The development of HEMPIS was eventually cancelled, as there was no shared vision within the MoET’s various departments of the specific role to be played by the new information system, which caused substantial delays in the preparation of the corresponding Terms of Reference. A more limited information system is being developed by the MoET. Overall impact: Even though HEMPIS was not implemented as anticipated, the analytical work produced under this sub-component constitutes a substantial contribution to the ongoing policy dialogue on higher education reform and has been utilized as an input in newly issued regulations and legislation that strengthen the overall governance framework of Vietnam’ higher education system. In particular, the process of developing the Higher Education Master Plan, the actual end-product and its dissemination have served to bring awareness over the main weaknesses of existing organizational and governance arrangements of Vietnam’s Higher Education System, including the fractured nature of the system, and the generalized lack of academic, managerial, and financial autonomy of higher education institutions. The diagnosis and recommendations were important inputs in the Higher Education Act (approved in June 2012), which represents an important step toward strengthening autonomy and improving efficiency, quality, responsiveness and accountability of higher education and research institutions. The Higher Education Act reflects many of the recommendations of the Master Plan, including the adoption of a multi-tiered system, the definition of functions to be carried out by key bodies within HEIs differentiated by tier, and the definition of the management structure at the university level. The analytical work carried out under this sub-component has also served to develop the proposal for drastic and comprehensive reform of Vietnam Higher Education, which was submitted to the Communist Party of Vietnam to be considered for adoption. Finally, it has served as inputs in several regulations issued by the Prime Minister or MoET regarding the overall organization and governance structure of Vietnam’s higher education system. The Higher Education Act and the various regulations have served to meet key policy conditions under the Higher Education Development Policy Program. Sub-component 1.2 – Higher education financing Description: This sub-component was implemented by MoET's Department of Planning and Finance. It was aimed at reviewing the public funding of higher education institutions, building on the recommendations concerning public financing set out in the joint Government-Bank Public Expenditure Review (April 2005). Technical assistance provided under this sub-component included policy advice and capacity building focusing on: (a) developing policy options and recommendations on the allocation of public expenditure across public and non-public HEIs; and (b) developing policy options and recommendations on student tuition fees and financial assistance for students from disadvantaged backgrounds, such as fee waivers. Achievement of Intermediate Results  Public funding for higher education Achieved: A study of the cost of higher education per student was carried out for various groups of disciplines within Vietnam’s higher education system based on 31 survey data from 150 universities and personal interviews with representatives of 24 public universities. The study also determined a reasonable level of expenditures per student and offered recommendations for future tuition policies. An independent review panel constituted by representatives of higher education institutions concluded that the policy recommendations are appropriate and relevant for Vietnam.  Student tuition fees and financial assistance Achieved: A study focusing on higher education financing was carried out, including an overview of current financial mechanisms as well as reporting requirements on the part of individual universities and colleges. The study also offered recommendations to improve reporting formats. Overall impact: The analytical work financed under this sub-component has helped MoET departments in the development of policies regarding public funding for higher education in general as well as determining the allocation of funds to individual higher education institutions based on unitary costs. It also provided recommendations to improve the financial reporting of higher education institutions. Findings and recommendations from the two studies were an important input of the decree regulating on tuition fees for the period 2011-2015 (issued on May 2010). Sub-component 1.3 – Higher education quality assurance Description: This sub-component was implemented by the MoET’s Department of Education Testing and Accreditation (DETA). It supported further strengthening of two dimensions of quality: i) quality of students at entry; and ii) quality of services provided by higher education institutions. Technical assistance provided under this sub-component included policy advice and capacity building focusing on: (a) developing policy options and recommendations on a set of minimum standards against which public and non- public HEIs can be accredited and on the establishment of an independent accreditation agency to reinforce these standards by auditing the internal quality assurance procedures in higher education institutions; and (b) developing policy options and recommendations on the higher education admissions system. Specific Intermediate Results  Accreditation and quality culture Achieved: There was substantial knowledge building on quality assurance supported under this sub-component, including workshops with the participation of international consultants focusing on the development of internal quality assurance systems at the institution level and the development of a quality assurance and accreditation system for the higher education as a whole. Tools and substantial technical assistance were provided to universities and colleges to support the development of an internal quality culture, including instruments to capture feedback from students, graduates and employers. In line with 2010 regulation calling for the establishment of independent accreditation agencies, a report was prepared, presenting a roadmap for the development and establishment of accreditation agencies and an accreditation system as a whole, which was deemed to be systematic, comprehensive, scientific and valuable study work by a panel of independent reviewers.  Admissions systems Cancelled: This activity was cancelled as the MoET issued new regulations regarding admissions. Overall impact: The activities carried out under this component served to strengthen 32 technical capacity among those responsible for internal quality assurance at individual colleges and universities, helping them better implement quality assurance systems gradually build a quality culture. They also served to strengthen the technical capacity at the MoET’s level, in particular DETA’s ability to provide substantive inputs in the drafting of quality-assurance related regulation (i.e., establishment of an independent accreditation agency and its functions, tasks and rights; accreditation cycle of universities, colleges and professional secondary schools; and accreditation standards). Component 2 – Capacity Building for University Teaching and Research (US$ 61.4 million) Achievement: Satisfactory This component aimed to strengthen teaching and research in twenty two participating universities, by: (i) building management capacity in universities, including the carrying out of regular surveys as a management monitoring tool; and (ii) financing Teaching and Research Innovation Grant (TRIG) sub-projects in two groups of universities: Group 1 concentrating on “major� universities as designated by the GoV, and Group 2 guaranteeing a small allocation for “equity� universities located in disadvantaged regions. Sub-component 2.1 – Management of TRIG capacity Description: This sub-component was implemented by the PMU and provided training and capacity building at the university level. Specifically, it provided training and capacity building on management skills, including procurement and financial management to universities implementing the TRIG sub-projects. It also provided support to the autonomous universities that were to emerge as a result of the ongoing policy reforms in terms of leadership and management skills as well as institutional quality assurance plans, including carrying out of regular surveys as a management monitoring tool, such as graduate tracer surveys. Capacity building under this program took the form of training workshops and seminars and included other forms of technical assistance such as mentoring or twinning arrangements and study tours for selected staff. Specific Intermediate Results  TRIG sub-project management Achieved: Substantial capacity was built among participating universities through a series of workshops and training seminars at various stages of implementation. These activities included: training and workshop on the design of TRIG sub-projects; implementation support, including reporting, financial management and procurement, including the development of TRIG Management Software.  University leadership and management Achieved: A series of capacity building activities were carried out to strengthen universities’ ability to design, implementation and management of HEP II.  Institutional quality assurance (QA) Achieved: Developed instruments to systematically collect feedback from students (e.g., teacher and course evaluations), graduates and employers on the quality of training; conducted pilots surveys to test the various instruments; disseminated among all colleges and universities. Overall impact: The activities carried out under this component served to strengthen implementation capacity among participating universities, as illustrated by the successful and timely project implementation. In addition, activities under this sub-component serve 33 to enhance leadership and QA capacity among the entire universe of higher education institutions. Sub-component 2.2 – Implementation of TRIGs Description: This sub-component was implemented by the twenty two participating universities, with a total enrollment of almost 50,000 students. It was aimed at building and strengthening capacity of universities in research, and teaching, by establishing closer links between research, teaching, industry and the community. Grants were used for human resources and professional development of university academic staff, improvement of teaching and research infrastructure, support for research projects and for national and international cooperation in teaching, research and industry. TRIGs were allocated using a peer-review process based on transparent and objective criteria among two groups of universities. Group 1, which concentrated the bulk of TRIG funds, focused on “major� universities as designated by GoV. Group 2, which represented a small allocation of funds, focused on “equity� universities located in disadvantaged regions. Thus, TRIG funding varied among individual universities based on: i) the type of universities; and ii) their specific proposal. Group 1 universities could apply for a maximum of US$5 million of TRIG funding, while the maximum for Group 2 universities was a maximum of US$2 million. Universities had to contribute at least ten percent in counterpart funds, but they could contribute more. Actual TRIG funds ranged from US$750,000 to US$4 million for individual universities. Each participating university identified the set of activities to be implemented under their TRIG grants. These activities were consistent with the strategic plan of each university and represented their own priorities. As a whole, universities gave priority to infrastructure (i.e., labs, teaching materials and equipment, library resources, etc.) under their grants, which represented 54 percent of total TRIG investments. However, “soft� investments in staff development and capacity building (i.e., short and long-term training) represented a significant share of TRIG investments, accounting for 33 percent of the total. Research activities, which were funded under this sub-component with counterpart funds by participating universities, represented seven percent of total TRIG investments (see Table 3 and Figure 3). Table 3 and Figure 3. Initial TRIG allocation by investment type Initial allocation of TRIG Areas of TRIG financial resources investments US$ millions % of total Training Training 19.60 32.7% Research Research 4.15 6.9% Infrastructure Infrastructure 32.32 53.9% Other Other 3.94 6.6% Total 60.00 100.0% Source: PMU (Oct., 2012) At the university level, grants exhibited a wide range of variation in terms of focus and type of investment (see Figure 4). For example, infrastructure investments ranged from only 1 percent of all TRIG investments in the case of Ho Chi Min City College of Education to 89 percent in Hanoi Medical University. Correspondingly, these two 34 institutions also exhibited the largest and smallest investment in training and staff development activities, representing 72 percent and 3.8 percent of their overall TRIG investments, respectively. There was also substantial variation in research investments, ranging from 34 percent in Hanoi Agricultural University to none in several universities. As mentioned earlier, this variation is consistent with the demand-driven nature of TRIG grants, which was a key determinant in ensuring the alignment of TRIG funding with the self-determined needs of participating universities. Figure 4. Variation in the university share of TRIG investment by type of investment Training Research Infrastructure 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80% Average: 32.7% Median: 23.4% Average: 6.9% Median:3.2% Average: 53.9% Median: 63.7% Source: PMU (Oct., 2012) Overall TRIG achievement is deemed satisfactory, although the level of performance varies across universities. As a whole, universities were able to implement 94.1 percent of the resources that were initially allocated to them. Some universities were more successful than others in terms of implementation and, as a result of two rounds of reallocation of funds, were able to implement more than their initial allocations—up to 150 percent in the case of Dong Thap University—one of the best performing universities that happened to be in Group 2. The University of Economics in HCMC was at the other end of the spectrum, having disbursed only 40 percent of their initial allocation. Overall, only two out of the twenty-two universities (i.e., less than 10 percent) disbursed less than 50 percent of their initial TRIG allocations and only six of them (i.e., 27 percent) disbursed less than 90 percent. Difference between actual disbursements and the initial allocation were the result of activities that had to be cancelled as a result of disbursement delays as well as other factors, such as estimation of the original budgets and savings from a weak dollar. Overall, the large majority of participating universities (i.e., 90 percent) is expected to have achieved or exceeded the objectives specified in their individual proposals. 35 Figure 5. Variation in university share of disbursements relative to initial TRIG allocation 0% 25% 50% 75% 100% 125% 150% 175% Average: 94.1% Median: 105% Source: PMU (Oct., 2012) Specific Intermediate Results  Faculty development Achieved: Overseas Ph.D. and Master’s training of teaching and research staff is roughly in line with the original target. Doctoral studies in foreign institutions have been funded under TRIG for a total of 56 faculty members. In addition, Master’s studies have been funded for 149 faculty members, with the large majority attending foreign institutions. Table 4. TRIG staff development outputs TRIG Staff Foreign institutions National institutions Total Development Activities Planned Actual Planned Actual Planned Actual PhDs 57 56 0 0 57 56 * Master's 147 143 4 6 151 149 ** Short training courses 1,051 1,368 20381 21407 21,432 22,775 Study tours 319 429 151 270 470 699 Workshops and seminars 184 207 8967 14321 9,151 14,528 Total 1,758 2,203 29,503 36,004 31,261 38,207 * Out of 56 PhD students, ten have already obtained their degrees an ** Out of 149 Master's students, 111 have already completed their degrees and returned to Vietnam. Source: PMU (Oct., 2012)  Capacity building Achieved: A total of 22,775 faculty and staff have attended short training courses, slightly exceeding the original target. Targets for the number of participants in study tours as well as workshops and seminars were also exceeded by roughly 50 and 60 36 percent, respectively. As mentioned earlier, the focus of capacity building activities and corresponding training plan was included under the original TRIG proposal and provided support to the overall objectives. In this way, depending on the focus of the TRIG activities being implemented by participating universities, capacity building activities focused on research and teaching methods, utilization and maintenance of technical equipment, curricula development, certification standards, management capacity, etc.  Curricula development Achieved: Substantial progress was made in terms of curricula development, with 467 programs and 1,173 courses being newly developed or substantially revised. The adoption of credit-based system was one of the main factors underlying curricula development activities. In addition, some courses are now offered in English. In addition, several TRIG sub-projects supported the development of teaching materials, including lab protocols and worksheets—often in conjunction with newly purchased equipment—as well as the expansion of existing test banks. Table 5. TRIG curricula development outputs TRIG Training-related Newly developed Restructuring Outputs Actual Planned Actual Planned Program curricula 238 165 229 177 Course syllabus 174 67 999 594 Experiment protocols 246 71 144 15 Experiment worksheets 373 200 38 4284 Source: PMU (Oct., 2012)  Infrastructure upgrading Achieved: TRIG funds supported the upgrading and provision of equipment for laboratories, computer networks, multi-media teaching facilities and library resources. The specific outputs include: 111 state-of-the art laboratories, six computer networks, 29 multi-media teaching classroom and lecture halls, as well as five new libraries. As in the case of all TRIG investments, infrastructure outputs were key components of the corresponding sub-project and were complemented with the training needed to ensure their full utilization. Table 6. TRIG infrastructure outputs TRIG-financed infrastructure Planned Actual Networks 8 6 Books and periodicals (i.e., hard and electronic copies) 44,953 38,755 Subscriptions to electronic databases 63 69 Labs New 59 54 Upgraded 26 57 * Multimedia classrooms and lecture halls 26 29 * This number increased substantially as a result of additional equipment financed under reallocated funds. 37 Source: PMU (Oct., 2012) Overall impact: The impact of TRIG sub-projects varied by university based on the specific focus of their individual proposals. Results from the TRIG Impact Assessment Study as well as Self Evaluation Reports prepared by each university report positive impacts in terms of: long-term development of faculty members in research and teaching; external accreditation; building institutional systems to support teaching, learning and research at the university level; linking research to teaching and learning; improving the content, instruction delivery, and assessment methods in undergraduate and postgraduate courses; developing linkages between universities, research institutions, the productive sector, and social and community organizations in Vietnam; supporting the internationalization of research and teaching in universities in the form of twinning, collaborative research, and sponsorship to achieve international recognition in teaching and research standards, and quality through international cooperation programs. Component 3 – Project Management, Monitoring and Evaluation (US$4.3 million) Achievement: Satisfactory This component aimed to support and to ensure a smooth and effective implementation of the project. This component covered both project coordination (including planning, procurement, financial management and communications) as well as project monitoring and evaluation. This Component also provided support to the MoET’s Department of Higher Education in the design and preparation of the Higher Education Reform Support (HERSP) DPL series. Sub-component 3.1 – Administration and Management Description: This sub-component provided support to the PMU in the areas of information flows, coordination, administration, financial management and procurement. Sub-component level of achievement: Satisfactory Specific Intermediate Results  Project management Achieved: Project management activities were carried out effectively, including independent audit, financial management. Internal auditing functions were not fully implemented as originally anticipated. Overall impact: The activities carried out under this component served to ensure the HEP II successful and timely implementation. Sub-component 3.2 – Monitoring and Evaluation Description: This sub-component was managed by the M&E Coordinator to monitor HEP2 implementation progress and the achievement of its development objective. In addition, the M&E Coordinator, with the support o f the Project Director, supervised the TRIG evaluation study and other technical assistance offered in the areas of M&E Sub-component level of achievement: Satisfactory Specific Intermediate Results  Project monitoring Achieved: Ongoing project monitoring activities were carried out effectively, with participating universities proving regular TRIG sub-projects progress reports and the M&E coordinator overseeing the overall project implementation. 38  Independent project evaluation Partially achieved: The international consultant firm selected to produce the Independent Project and TRIG Evaluation Study failed to deliver as anticipated. While the contract was signed in September, 2009 with an agreed timeline for deliverables, there were substantial delays in the submission of deliverables, which eventually led to the termination of the contract. To ameliorate the negative impact, the PMU commissioned a series of independent evaluation studies by panels of university experts focusing on sub-components 1.1, 1.2, and 1.3 as well as TRIG sub- projects implemented by participating universities. Overall impact: The activities carried out under this component served to ensure the close monitoring of implementation progress. The PMU and universities maintained very good data on the project activities and results, which were regularly presented in the Project M&E Progress Reports. In turn, these data were critical to effectively manage as well as supervise project implementation. Project evaluation, particularly with respect to TRIG sub-projects, as the independent concurrent M&E function that was originally envisioned was not implemented due to the poor performance of the international consultant firm. However, the set of independent ex-post evaluation studies commissioned by the PMU helped partially fill the void. 39 Annex 3. Economic and Financial Analysis Economic Analysis: The economic analysis performed at appraisal continues to be relevant in view of the project’s actual achievements, which are in line with those anticipated. The economic analysis took into consideration three types of benefits: i) productivity gains of graduates from participating universities; ii) efficiency gains at the university level and iii) rates of return from research and development investments under the project. The economic analysis was based on a set of conservative assumptions, including a ten-year benefit horizon for TRIG universities in terms of enhanced quality of teaching, a 30 year benefit horizon for graduates in terms of additional income, and a five-year benefit horizon for R&D activities (see Table 7). Table 7. Benefits – Assumed Parameter Values for Based Scenario Source: HEP II Project Appraisal Document (May, 2007) 40 The economic analysis yielded an Internal Economic Rate of Return (IERR) of 34 percent, indicating the project’s strong economic efficiency. The results of sensitivity analysis depicting less favorable assumptions still yield robust IERRs, ranging from 23 to 28 percent (see Table 8). Table 8. Summary of Sensitivity Analysis Source: HEP II Project Appraisal Document (May, 2007) Financial Analysis: The project’s financial rate of return was not estimated at appraisal. However, there are three observations that indicate the universities will have the ability to finance operating expenditures of TRIG investments beyond the life of the project. Specifically:  Universities complied with 10 percent co-financing requirements as anticipated.  In several cases, universities have secured research and service contracts with government organizations and private firms as a result of new equipment financed under TRIGs that will provide them with recurrent additional revenues.  HERA indicators denote a gradual but continuous increase in universities own-source revenues, independently of the project. 41  Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team Members Responsibility/ Names Title Unit Specialty Lending ALBERT-LOTH Agnes Senior Finance Officer LOAG2 ARISTORENAS Rosario Program Assistant EASHD BALL Carol Senior Operations Officer EASHD DUONG Quoc Vinh Environment Specialist EASES Lead Human Development Di GROPELLO Emanuela AFTEW Economist KIJIMA Rie M&E Specialist MNSHD MAI Thi Thanh Senior Education Speciatlist EASED Consulta MOOCK Peter Education Economist nt NESMITH Katherine Junior Professional Associate EASHD NGUYEN Hoai Linh Operations Analyst EASHD NGUYEN Hoi Chan Senior Counsel LEGEA NGUYEN Minh Nguyet Program Assistant EACVF NGUYEN Thi Phuong Environment Specialist EASRE Trang NGUYEN Thuy Anh Operations Analyst EASHD NGUYEN Van Linh Program Assistant EACVF Senior Social Development PHAM Thi Mong Hoa EASSO Specialist PHAN Thi Xuan Quynh Financial Management Analyst EAPCO THOMSON Jennifer Senior Financial Specialist EAPCO TRAN Thi Hai Chau Finance Analyst LOAG1 TRAN Quang Thong Financial Management Specialist EAPCO TRAN Thi Hang Hiet Procurement Specialist EAPCO WAITE Jeffrey Team Leader EASHD WATKINS Alfred Lead Science and Technology HNDED Specialist WEBER Luc Consulta Higher Education Specialist nt Higher Education Grants Consulta WOODHALL Maureen Specialist nt Supervision/ICR Thanh Thi Mai Senior Education Specialist, TTL EASHE Sylvester Kofi Awanyo Lead Procurement Specialist EAPPR Carol Chen Ball Consultant EASHD 42 Rie Kijima Consultant EASHD Hoi-Chan Nguyen Consultant OPCIL Anh Thuy Nguyen Operations Officer EASHH Nguyet Minh Nguyen Program Assistant EACVF Quynh Xuan Thi Phan Financial Officer GEF Jennifer K. Thomson Financial Management Specialist SARFM Hiet Thi Hong Tran Senior Procurement Specialist EAPPR Anh Lan Vu Temporary EACVF Binh Thanh Vu Senior Education Specialist EASHE Jeffrey Waite Lead Education Specialist MNSHE Consulta Cecilia Zanetta ICR Specialist nt (b) Staff Time and Cost Staff Time and Cost (Bank budget only) Stage of Project Cycle USD Thousands No. of staff weeks (incl. travel and consultant costs) Lending FY03 45.08 FY04 64.57 FY05 49.62 FY06 125.27 FY07 142.35 FY08 0.30 Total: 427.19 Supervision/ICR FY03 0.00 FY04 0.00 FY05 0.00 FY06 0.00 FY07 0.00 FY08 52.88 FY09 45.51 FY10 34.85 FY11 57.58 FY12 116.54 FY13 41.85 Total: 349.21 43 Annex 5. Beneficiary Survey Results An evaluation of the TRIG Program was commissioned to a panel of independent university researchers. The results from this evaluation reflect the findings from a total of 536 personal interviews with management, teaching and research staff and students of 17 TRIG–recipient universities (i.e., 188 managers, 185 staff, and 163 students), as well as responses from 5,502 surveys that were distributed among 750 managers, 1,585 staff and 3,167 students of the 22 TRIG universities. Results from this evaluation indicate that the TRIG program had substantial positive impacts in terms on enhancing management, teaching and research skills of faculty and staff, supporting curricular development, and providing universities with needed equipment and teaching materials. The specific results of the evaluation can be summarized as follows: TRIG Inputs TRIG Outcomes Staff development  TRIG funding focused on enhancing  Administrators and faculty rate the impact the structural quality of faculty of TRIG interventions on management members by financing their graduate effectiveness as 3.7 and 3.4, respectively, education at the Master’s and Ph.D. in a scale of 1 to 5. levels.  More than half of the administrators being  It also supported specific skill interviewed as part of the study (i.e., 54 development through short term percent) rate the improvements on training courses and workshops as teaching effectiveness resulting from well as study tours. The focus of TRIG interventions to be 4 or higher in a these programs was defined by scale of 1 to 5.11 participating universities and  Some of the specific improvements included, among others, reported in terms of teaching effectiveness management skills, teaching and include: i) enhanced communication, research methods, and training on including more teacher-student interaction the use of technical equipment. as well as communication being more open and friendly; and added emphasis on self-learning and creative problem solving; development of teamwork skills and other important soft skills.  Faculty members report enhanced ability to perform the various tasks involved in conducting research as a result of training or research funding received under TRIG initiatives. Specifically, in a scale of 1 to 5, they rank the impact of TRIG initiatives 11 No results were reported for faculty responses. 44 as 3.48 in terms of enhancing their substantive knowledge on the research topic; 3.53 in terms of their ability to identify and select of a research topic; 3.44 in terms of their ability to write a research proposal; 3.61 in terms of their ability to collect data; 3.53 in terms of their ability to analyze and interpret data; and 3.52 in terms of their ability to write a research paper. Curricula and program development  TRIG funding supported course and  Administrators and staff rate the impact of program restructuring, much of what TRIG interventions on program and was done as part of the transition to course development and restructuring as the newly adopted credit-base 3.7 and 3.5, respectively, in a scale of 1 to system. 5.  It also supported the development of  Much of the curricular development was new courses and programs, some of done as part of the transition to the newly them with input from employers adopted credit-base system.  In some cases, program and course restructuring was done with input from employers. Teaching materials  TRIG funding supported the  Administrators and staff rate the impact of development of lab protocols and lab TRIG interventions on the development of worksheets to support the use of the lab-related teaching materials as 3.7 and new labs established under the 3.5, respectively, in a scale of 1 to 5. In project most cases, these activities complemented  It also supported the development of the expansion and/or upgrading of labs as standardized tests, including the well as the establishment of new ones. improvement and expansion of test banks. Library resources  TRIG funding supported the  Faculty report particularly high levels of acquisition of library materials, satisfaction and utilization of library including books and electronic resources. databases.  It also supported the development of modern library management systems, including electronic catalogs. Lab equipment  TRIG funding supported the  Faculty report high levels of satisfaction 45 purchase of technical equipment to with the current level of supply of lab expand or upgrade existing labs or to equipment relative to their demand (i.e., establish new ones. 3.8 in a scale of 1 to 5).  It also supported the acquisition of  Faculty report high levels of utilization of state-of-the-art teaching technology, TRIG-financed equipment and materials including “smart� classrooms and (i.e., 3.4 in a scale of 1 to 5) for their lecture halls. teaching activities as well as satisfaction  It also supported the modernization with the current level of supply relative to of communication networks and IT their demand (i.e., 3.2 in a scale of 1 to 5). infrastructure. 46 Annex 6. Stakeholder Workshop Report and Results A workshop was conducted to assess the implementation experience of the Teaching and Research Innovation Grants (TRIG) Program took place on Sept. 28-29, 2012 in Phan Thiết, Vietnam. Representatives from the World Bank, the Ministry of Education and Training (MoET) as well as the twenty two universities that participated in the program attended the workshop. The Project Management Unit prepared a summary of the main conclusions arising from the discussions, as follows: TRIG Evaluation Workshop – Main Conclusions Project organization and management model  The HEP II project management model, which includes a PMU at the MoET level and TMUs at the university level is considered to be effective.  The HEP II management model has helped promote university autonomy and accountability as well as management capacity.  The decision to staff the TMU with part-time or full-time personnel depends on the specific characteristics of each university. Both approaches are valid.  Long-term staff development, including PhD. and Master’s training, should not be funded using project funds because of incompatible implementation times. Infrastructure investment strategies  Focalized investment in laboratory equipments helps promote the competitive strength of a university in specific research areas, while widespread investments across multiple departments help maximize access to new equipment.  Ultimately, the specific investment strategy depends on the specific situation of each university. Both approaches are valid. Publications  There were no clear criteria defined for reporting TRIG-related publications. Thus, the data is not consistent across universities.  It is suggested that, in the future, eligible articles should explicitly acknowledge TRIG funding. Program innovation and accreditation:  Quality improvement and assurance is a prerequisite for quality human resources training.  Accreditation is a measure for quality assurance so there should be a clear distinction between university accreditation (implemented by MOET) and program accreditation (implemented by universities).  The development of an accreditation system requires long-term implementation policies. International cooperation in training and research  International cooperation contributes greatly to training and research. However, there is little control over the type of training, which is largely defined by international partners.  Universities should make an effort to develop quality training and research programs, to build a strong reputation for Vietnam’s universities over the long term. 47  When assessing the impact of training and research activities, it is necessary to make a clear distinction between TRIG outcomes and the results coming from international cooperation activities. Recommendations:  The twenty two participating universities expressed strong interest in a potential follow-on operation—the HEP III.  It was also suggested to continue to foster the existing relationship between the TRIG universities. 48 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR I. Component 1: Enhance capacity in policy development Outcomes: Following important activities have been conducted under Component 1: Higher education governance, Higher education (HE) Master Plan, recommendations of consultants for Higher education Law, policy in HE quality assurance and finance. Assessment: Component 1 has been implemented in difficult situation and slow speed due to complex procedures in consultant selection; however, almost all tasks have been fulfilled. Departments (Department of Higher Education, Planning and Finance, General Department of Education Testing and Accreditation) in-charge-of sub-components have closely cooperated with Project Management Unit (PMU) of the Second Higher Education Project during implementation process. Outputs: • Governance: Many outcomes have been used during policy development process: HE Master Plan is considered a reference to develop HE Act; Survey on HE status; credit-based system. • Finance: Public financing allocation; unit cost per HE student; support processing of disclosure information; survey on HE finance. • Quality assurance: support development of quality accreditation; external evaluation of 20 universities; appraisal of internal assessment of 40 universities; develop quality accreditation culture; develop feedback instruments to evaluate quality of curriculum; survey on students’ employment. II. Component 2: Improve training and research capacity for 22 Universities 2.1. Improve leadership competence of Universities: • Organized workshops delivered by international consultants on project development, management and implementation on improving training and research capacity in Higher Education sector with participation of leaders in nearly 40 public and private universities. • Organized leadership teams of 22 TRIG Universities for study tours and learn lessons on governance, quality assurance in countries with developed education status. 2.2. Enhance teaching and research capacity for 22 Universities Outcomes: Outcomes of Component 2.2 include: invested laboratories, doctoral and master training, research themes, articles, innovation programs… All these outcomes are managed by universities benefited from TRIG Grant. 49 Assessment: All universities participating in the Project are actively conducting the Project’s activities, included: Teaching innovation • Newly develop and innovate curriculum, subjects, experiment test. • Many programs and curricula have been accredited. Staff training • HEP2 contributes to enhance training capacity for universities thanks to adequate investment in high-level human resources: long-term training with diplomas granted (doctors, masters) and international cooperation in short-term training, study tours, exchanges and learn lessons in training. • Via international cooperation programs, teachers in universities have established relationships in training and research, obtained scholarships. Research capacity improvement • Many research themes have been conducted in all participating universities and many works have been published in international and local journals (particularly prestigious international journals listed in SCI, SCIE, SSCI and ISI.) • Many works have been applied in reality and transferred. • Many lecturers and students have improved their knowledge in research thanks to TRIG’s research themes. Facility and equipment • Facility and equipment purchase in almost all participating universities have been basically completed. Laboratories, network, multi-media classrooms and libraries have been invested synchronously, contributing to improve quality of teaching and research. Universities in Group 2 are invested significantly to reduce the Universities’ difficulties in facilities and laboratories. • Equipments installed by HEP2 to Universities have been effectively used, improving quality of training and research. This contributes to: innovate contents and quality of curriculum, quality of research outcomes, innovate teaching and research methodology, contribute to heighten training and research status of universities locally and internationally, create chances for universities in training and research cooperation with overseas universities. III. Component 3: Project Management, Monitoring and Evaluation Component 3: Project Management – Monitoring - Conducted well, assuring quality of work - Fulfilled monitoring tasks, supported Project Implementation in 22 universities and Component 1, 2.1 and 3. - Fulfilled tasks of independent evaluation of outcomes in Component 1, 2, 3 according to forms set by World Bank and Ministry of Planning and Investment 50 General assessment • HEP2 granted with ODA source by World Bank has contributed to a new development stage of many leading universities in Vietnam • HEP2’s self-autonomy mechanism has helped universities to have chances of improving their accountability, skills of project implementation, management and monitoring • Regular and effective guidelines of MOET Leaders • Close cooperation between HEP2 and MOET Departments to fully enhance capacity in policy development on HE Governance, Finance, Quality Assurance and support to complete Higher Education Development Policy Program (HEDPP) with its three single tranches • PMU’s supervision and consultants for activities conducted in 22 universities have been implemented tightly and regularly, facilitating HEP2 to complete in schedule with set quality and targets • World Bank’s close supervision also contributes to the Project Implementation to fully follow regulations and avoid mistakes during procurement and disbursement process. 51 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders N.A. 52 Annex 9. List of Supporting Documents World Bank (2012), Putting Higher Education to Work: Skills and Research for Growth in East Asia, East Asia and Pacific Regional Report, Washington, D.C. World Bank (2010), Vietnam: Higher Education Development Policy Program – Second Operation, Program Document, Report No. 55039-VN, 29 October 2010, Washington, D.C. World Bank (2009), Vietnam: Higher Education Development Policy Program – First Operation, Program Document, Report No. 47492-VN, 23 June 2009, Washington, D.C. World Bank (2008-2012), Documents in Project’s Electronic File, including Credit Agreement, Aide Memoirs, Back-to-Office Reports, Project Status Reports, Project Financial Assessments, and Project Procurement Assessments; Washington, D.C. World Bank (2008), Vietnam: Higher Education and Skills for Growth, Human Development Department, East Asia and Pacific Region, Washington, D.C. World Bank (2008), Vietnam’s First Higher Education Project, Implementation Completion and Results Report, Report No: ICR 561, 21 February 2008, Washington, D.C. World Bank (2007), Vietnam’s Second Higher Education Project, Project Appraisal Report, Report No: 38399 - VN, 16 May 2007, Washington, D.C. Meetings Conducted during ICR Mission Ministry of Education and Training Nguyen Duc Thuan, Director HEP II Nguyen Van Kuu, Coordinator Component 1.1, Dept. of Higher Education Nguyen Van Ang, Coordinator Component 1.2, Deputy Director of Department of Planning and Finance Pham Xuan Khanh, Coordinator Component 1.3, Deputy of Department of Education Testing and Accreditation Phung Quoc Tuan, Coordinator Component 2 (TRIG) Pham Thi Thanh Hai, M&E Coordinator Le Thi Kim Anh, Procurement Coordinator Thai Thi Mai Huong, Administrative Coordinator TRIG Universities Lê Việt Dũng, Can Tho University Plus feedback from nine presenters at the TRIG Evaluation Workshop TRIG Independent Evaluation Team Ms. Nguyen Thi Hai, Vietnam’s National Institute of Education Sciences Mr. Ngo Vin Trung, Vietnam’s National Institute of Education Sciences World Bank Mai Thi Tang, TTL Suhas Panderkar, Sr. Education Specialist 53 102° 104° 106° 108° 110° C H I N A 4 Hà Giang 5 ` Cao Bang C H I N A Lào Cai Phong Tho 1 3 9 ` Can Bac 22° 8 22° Tuyên Lang , 7 Quang Son Yên Bái 13 10 Thai Nguyen 2 Tay Bac U. Tay Thuân Châu Viêt Trì Thai Nguyen U. 12 Co To Vinh Yen Vinh Thuc Diên Biên Phú Són La 11 Bác Giang Cai Chien Bác 14 15 6 16 Ninh Cai Bau HANOI 17 19 Ha Long Co To Haí Hòa 18 Dúóng Hai Phong Cat Ba Bình Hung Yen 20 21 Hà Nam 23 22 Thái Bình Hanoi National U. Hanoi U. of Agriculture Nam Dinh Hanoi U. of Education Ninh Bình Trade Hanoi U. of Foreign Trade 25 Hanoi U. of Medicine 20° 24 20° 26 Technology Hanoi U. of Technology Transport Hanoi U. of Transpor ranspo r t & Communications Thanh Hóa National Economics U. LAO PEOPLE'S Gulf DEMOCRATIC REPUBLIC 27 Me Nghi Son of Hainan I. To n k i n (China) Vinh U. Vinh ˜ Hà Tinh 28 18° 18° M ek on g Dông Hói 29 Con Co THAILAND Dông Hà 30 Huê Huê U. 31 Danang U. 32 16° 16° ˘ Dà Nang VIETNAM Cham Tam Ky 33 Ly Son Re Quàng Ngai 34 35 102° Kon Tum 37 VIETNAM 14° HIGHER EDUCATION 14° Play Cu Quy Nhon 36 Xanh MAJOR PUBLIC UNIVERSITIES UNIVERSITIES IN DISADVANTAGED REGIONS 38 Tuy Hòa PROVINCE CAPITALS CAMBODIA 39 Buôn Ma Thuôt NATIONAL CAPITAL Tay Nguyen U. Tay PROVINCE BOUNDARIES Gom INTERNATIONAL 41 40 Nha Trang BOUNDARIES Tre Gia Nghia 12° Dà Lat 12° PROVINCES: 42 1 Lai Chau 33 Quang Nam Dông 43 44 2 Dien Bien 34 Quang Ngai Xoái Mek Phan Rang- 3 Lao Cai 35 Kon Tum o Tháp Chàm 4 Ha Giang 36 Gia Lai 45 ng 5 Cao Bang 37 Binh Dinh 47 Tây Ninh 46 6 Son La 38 Phu Yen Thu Dãu 7 Yen Bai Dak Lak Môt 48 39 Biên Hòa Phan Thiêt 8 Tuyen Quang 40 Dak Nong 49 9 Bac Kan 41 Khanh Hoa HCMC National U. 10 Lang Son 42 Binh Phuoc 53 51 Ho Chi Minh City HCMC U. of Economics Dong Thap 11 Phu Tho 43 Lam Dong Educ. U. 50 HCMC U. of Education 55 Tan An HCMC Pharmacy Pha r macy U. of Medicine & Phar 12 Vinh Phuc 44 Ninh Thuan Cao Lanh 52 Long Xuyên My˜ Tho 13 Thai Nguyen 45 Tay Ninh ˜ Tàu Vung An Giang U. ˜ Long Vinh , 14 Bac Giang 46 Binh Duong Bên Tre Phu 15 Quang Ninh 47 Dong Nai Quoc 59 56 54 Nghe Rach Giá Cân 10° 16 10° Ha Noi 48 Binh Thuan 10° Cân Tho U. Hai Tre Tho 17 Bac Ninh 49 Ho Chi Minh City 58 Tra Vinh U. Tra 60 Trá Vinh 18 Hung Yen 50 Ba Ria-Vung Tau Rai Vi Thanh 57 MEKONG DELTA 19 Hai Duong 51 Long An Nam Du 20 Hai Phong Gulf 52 Tien Giang 61 Sóc Trang 21 Hoa Binh 53 Dong Thap 0 25 50 75 100 125 150 Kilometers 22 Ha Nam 54 Ben Tre 62 Tho Chu Bac Liêu 23 Thai Binh 55 An Giang Cà Mau 24 Ninh Binh of 56 Vinh Long 0 25 50 75 100 Miles 25 Nam Dinh 57 Tra Vinh 63 26 Thanh Hoa 58 Kien Giang 27 Nghe An Can Tho Con Dau 28 Ha Tinh Thailand 59 60 Hau Giang This map was produced by the Map Design Unit of The World Bank. IBRD 35262R 29 Quang Binh Soc Trang The boundaries, colors, denominations and any other information 61 shown on this map do not imply, on the part of The World Bank 30 Quang Tri 62 Bac Lieu Khoai Group, any judgment on the legal status of any territory, or any MAY 2013 31 Thua Thien Hue 63 Ca Mau endorsement or acceptance of such boundaries. 32 Da Nang 102° 104° 106° 108°