Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) Report Number : ICRR0021217 1. Project Data Project ID Project Name P106605 KH-HE Quality and Capacity Improve Proj Country Practice Area(Lead) Cambodia Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-47960,IDA-H6070 31-Dec-2015 23,000,000.00 Bank Approval Date Closing Date (Actual) 05-Aug-2010 30-Sep-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 23,000,000.00 0.00 Revised Commitment 22,572,623.31 0.00 Actual 22,849,855.64 0.00 Prepared by Reviewed by ICR Review Coordinator Group Eduardo Fernandez Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as stated in the Project Appraisal Document (PAD) was to (a) improve the quality of teaching, management, and research in project-supported entities; and (b) pilot the targeting of disadvantaged students for enhanced retention” (PAD, p. 4). The Financing Agreement (p. 5) provided an identical formulation. Although the ICR conducted a split rating, this ICR Review does not do so, because the changes in the results framework did not represent a lowering in the project’s scope or ambition that would warrant a split Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) rating. To the contrary, the changes introduced represented an improvement in the results framework through the introduction of an additional outcome indicator to better measure the achievement of the PDO (with no change in outcome target). For this validation, the efficacy discussion separates part (a) of the PDO into three distinct objectives: (i) improved quality of teaching; (ii) improved quality of management; and (iii) improved quality of research. The ICR discussed these elements as a single objective and measured achievement through the following single outcome indicator: “Percentage of Higher Education Institutions supported under Components 1 and 2 showing an increased aggregate score against a modified subset of Accreditation Committee of Cambodia (ACC) accreditation criteria." The ACC accreditation criteria used to measure the indicator included the following standards: (i) Governance, Planning, and Evaluation (Standard 1); (ii) Academic Program (Standard 2); (iii) Academic and Administrative Staff (Standard 3); (iv) Financial Management (Standard 4); and (v) Dissemination of Information (Standard 5). This validation breaks down the aggregate score and links the following standards to the different objectives: Standards 2 and 3 are linked to “quality of teaching," and Standards 1, 4, and 5 are linked to “quality of management." The ACC criteria used by the ICR were not relevant for measuring improved quality of research; for this reason, this validation relies on other information to measure the achievement of the quality objective. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component 1: Strengthening the Capacity of the Higher Education System (appraisal cost including contingencies US$7.46 million; actual cost US$10.36 million). This component was to finance capacity building activities aimed at improving the overall development, management, and governance of the Higher Education (HE) sector. More specifically, the component supported staff development in the Department of Higher Education (DHE), Department of Scientific Research (DSR), and the Accreditation Committee of Cambodia (ACC) with the purpose of improving their capacity to offer guidance and support to Higher Education Institutions (HEIs). In addition, the component also strengthened the capacity of individual HEIs through capacity building activities and the financing of postgraduate fellowships for existing faculty seeking master’s degrees in national priority areas. The fellowships were to target individuals with potential for becoming policymakers working in the government sectors related to the subject of their fellowships. The main requisite to be eligible for these postgraduate fellowships included belonging to an HEI that had obtained either full or provisional accreditation from the ACC. Component 2: Provision of Competitive Development and Innovation Grants (appraisal cost including contingencies US$4.58 million; actual cost US$3.66 million). This component was to finance Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) research grants with the aim of improving the research and innovation outputs of the HEIs. Specifically, the component was to finance grants under two windows. The first window was to be directed to improving the quality of teaching and learning, particularly with respect to academic disciplines prioritized in the draft Vision and Strategy of Higher Education in Cambodia. The second window was to enhance the capacity of HEIs to seek innovative solutions to address national/local development issues. Prioritized research areas were to include agricultural research and extension; food science and related technology; sciences including medical, health, and nutritional sciences and pharmacy; engineering and technology; social sciences; and law. Grant financed amounts were to fall into two categories: (a) Type A Grants: up to US$50,000 which were to be completed in 12 months; and (b) Type B Grants: US$50,001- US$200,000 which were to be completed in 24 months. To be eligible for participation in the competitive process of fund allocation for research, HEIs had to commit funding 5% of their proposals via cash or in-kind contributions. The main eligibility requisites for HEIs applying to these grants were: (i) having minimum financial management capacity to manage the grant; (ii) having obtained either full or provisional accreditation from the ACC; and (iii) having participated in the project’s capacity building activities under Component 1. Component 3: Provision of Special Priority Scholarships (SPS) to Disadvantaged Students (appraisal cost including contingencies US$5.85 million; actual cost US$4.24 million). This component was to finance a pilot scholarship program to increase the retention of disadvantaged students in HE through the financing of 1,000 SPS based on pro-poor targeting. Specifically, the component was to finance: (a) semi-annual guidance and awareness-raising activities related to the program in order to increase knowledge about the program, as well as to promote retention for those already enrolled in the program; (b) tuition and stipends to beneficiary students; (c) monitoring and evaluation activities at the project unit and participating HEIs; and (d) a rigorous impact evaluation. Tuition support to HEIs on behalf of beneficiary students was to range from US$250 to US$700, and stipends were to be US$70 per month for students in Phnom Penh and US$50 per month for students in the provinces. The pilot scholarship program was to be implemented by the DHE within the existing framework of the Ministry of Education, Youth and Sports (MoEYS) national scholarship scheme, which had provided only tuition waivers and suffered from low retention rates. To be eligible for these scholarships, applicants needed to meet certain socio-economic criteria measured through proxy means testing methods that were to generate a welfare score for every applicant. The DHE Scholarship Team was responsible for the Proxy-Means Testing Application Process, and the scoring formula was to be based on analysis of data from the most recent Cambodia Socio-Economic Survey. Component 4: Project Management and Monitoring and Evaluation (appraisal cost including contingencies US$5.11 million; actual cost US$4.59 million). This component was to finance activities under two subcomponents: (1) a project management subcomponent that financed activities to (a) support whole project management and administration; (b) develop HE long-term planning; (c) develop an HEI governance and financial management (FM) policy paper; (d) support project FM and build FM capacity of the Ministry and HEIs as prescribed in the MoEYS Public Financial Management Action Plan; and (e) support project procurement operations and build procurement capacity of the MoEYS and HEIs; and (2) a monitoring and evaluation subcomponent that financed activities in the MoEYS, DHE, ACC, and HEIs to systematically collect, collate, analyze, and report on the human and information resources needed to steer the HE sub-sector. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost The total project cost was US$22.85 million (ICR, p. 6) as compared to an original commitment of US$23 million (PAD, p. 51) that was later revised down to US$22.57 million, as there were US$427,791 cancelled. The appraisal estimate included physical and price contingencies, estimated at 4% of project costs. The project did not include a borrower contribution. Financing The project was to be financed through an International Development Association (IDA) credit and grant worth $11.5 million each. Both were fully disbursed. Total disbursements were US$22.85 million. The difference between the disbursed amount (US$22.85 million) and the revised amount (US$22.57 million) is explained by exchange rate fluctuations between US$ and SDR (Source: World Bank Team). Dates The project was approved on August 5, 2010 and became effective on December 30, 2010. The project closed on September 30, 2017. The original closing date was December 31, 2015. The project had three restructurings (March 2011, October 2014, and December 2015). • The first restructuring (March 2011, level 2) modified disbursement categories to accommodate a new government policy that impacted on the merit-based performance incentive scheme that had originally been envisaged in project design (Report No: 59553-KH). • The second restructuring (October 2014, level 2) introduced two modifications to the PDO outcome indicators. First, the restructuring introduced the following PDO outcome indicator: “40% of HEIs supported under Component 1 and 2 showing an increased aggregate score against a modified subset of ACC accreditation criteria.” The ICR noted (p.15) that the introduction of this indicator was made following the midterm review (MTR) because the ACC had been able to establish standards that could provide a more in-depth measure of quality related to teaching, management, and research. Second, the restructuring modified PDO outcome indicator #2 to “an average annual retention rate of 95% of project supported scholarship recipients” rather than just the annual retention rate as was stated in the original indicator. This newly introduced indicator was to measure the average retention rate over the entire five- year project period. Finally, the second restructuring also reallocated IDA credit funds from Category 5 “Unallocated” and Category 6 “Priority Operating Costs” to Category 1 “‘Goods, Works, Consultant Services, Training and Incremental Operating Costs." The restructuring also reallocated IDA grant funds from Category 3 “Scholarships” to Category 2 “Fellowships." This latter reallocation financed additional fellowships as well as additional capacity-building activities in responding to higher unit costs of postgraduate programs (all fellowship programs were organized with universities in Australia and New Zealand) as well as higher demands for training opportunities from HEIs. Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) • The third restructuring (December 2015, level 2) introduced a PDO indicator that had been mistakenly removed in the second restructuring (2014). The reintroduced indicator was “50% of sub-projects approved under the first round of proposals completed by the closing date and 70% of sub-projects approved under the second round completed by the closing date." Finally, this restructuring added an activity as per a Government request. More specifically, the Government requested the project to work with the Royal University of Phnom Penh to create a model HEI that exemplified good practice in governance and financial and human resource management. The Government also requested the project to provide support for science, technology, engineering, and mathematics (STEM)-related disciplines through the inclusion of an additional 15 fellowships specifically focused on STEM subjects. Because of these new activities, the restructuring extended the closing date by 21 months from December 31, 2015 to September 30, 2017. 3. Relevance of Objectives Rationale Relevance of objectives is rated High. The project development objectives were relevant to country priorities. After years of remarkable economic growth and rapid increase in higher education enrollment; Cambodia was still one of the poorest countries in Southeast Asia. Against this backdrop, the government began to focus on the need to spur growth by providing more relevant and better skills needed to move the country up the value chain and produce more sophisticated knowledge-intensive products. Moreover, the government started to become increasingly concerned that the rapid increase in HE enrollments had not been accompanied by improvements in educational or institutional quality. The project provided funding to strengthen the quality of the HE sector and, thus, supply more relevant skills. At the time of approval, the project development objectives fit into the government’s Education Sector Plan 2006-2010, which focused on (i) ensuring equitable access to education; (ii) increasing quality and efficiency of education services; and (iii) institutional development and capacity building for decentralization. Likewise, the project development objectives were aligned with the World Bank’s Country Assistance Strategy (FY05 -FY11) that focused on strengthening the higher education institutional framework to achieve improvements in quality assurance and accountability. At the time of project closing (September 2017), the project development objectives remained relevant to the country priorities as laid out in fourth pillar of the government's Industrial Development Policy 2015-2025 (i.e. human resource and skills development). With respect to World Bank strategies at project closing, the project development objective was aligned with the focus area of “Enhanced Connectivity and Improved Competitiveness” included in the World Bank’s Country Engagement Note (2016-2017). Rating High Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Improve the quality of teaching in project-supported entities Rationale The project’s theory of change was based on the premise that improvements in the quality of Cambodia’s higher education system could be achieved through: (i) faculty advanced degree training; (ii) establishment of a research culture through increased opportunities for conducing innovative research; and (iii) training of higher education staff involved in managerial and accreditation responsibilities. Most of these project- supported activities would have been expected to have synergistic impacts on teaching, research, and management. Moreover, the project also aimed to pilot the idea that enhanced retention of disadvantaged students could be achieved through scholarships that included stipends on top of tuition waivers. Overall, the results chain was clear and convincing, since it can be plausibly argued that the project activities would achieve the stated objectives: (a) to improve the quality of teaching, management, and research in project- supported entities; and (b) to pilot the targeting of disadvantaged students for enhanced retention. Outputs The following outputs supported achievement of the teaching, management, and research objectives: a) 7,395 staff from the MoEYS and HEIs received training in professional competencies related to higher education quality, management, assessment, and accreditation. Target: 1,500 (exceeded). b) 79 staff from MoEYS and HEIs received Master or Doctoral degrees in academic specialties. Target: 54 (exceeded). c) 579 academic staff from participating higher education institutions completed training in design and research methodologies. Target: 180 (exceeded). The formulation of this indicator included in the results framework suggested that the indicator was also supposed to report on the percentage of academic staff from participating HEIs that completed training in design and research methodologies as a percentage of total staff in these institutions. However, the ICR did not report on this percentage and instead provided only the absolute number (579 academic staff). d) During the life of the project, there were two rounds of funding available for research grants. For the first round (2012), 73% of the eligible higher education institutions submitted at least one proposal (22 out of 30) to the Research Grant Management Committee. Target: 70% (exceeded). For the second round (2013), 74% of eligible HEIs submitted at least one proposal (26 out of 35) to the Research Grant Management Committee. Target: 70% (exceeded for each round). e) 72% of the research project proposals submitted to the Research Grant Management Committee were awarded a grant after review and evaluation in accordance with the project processing timeline. Target: 47% (exceeded). The project funded 45 competitively selected research proposals, and all of them were completed during the project period. Target: 100% (achieved). f) The project supported the MoEYS in the preparation of a Higher Education Governance and Finance Policy Paper and a Higher Education Vision and Strategy 2030. Target: Achieved. Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) g) The project also supported the development of a Higher Education Management Information System (HEMIS) that was piloted in 6 higher education institutions. Target: piloting HEMIS in 4 higher education institutions (exceeded). h) Additional outputs achieved but not tracked in the results framework included: (i) Draft Higher Education Action Plan 2018–2022; (ii) Draft Regulation on HE Governance and Finance; (iii) Functional Review of the ACC; and (iv) Functional Review of the DHE and DSR. Outcomes a) 40% of the participating HEIs (n=36) improved their ACC Standard 2 (Academic Program). Target: 40% (achieved). b) 66% of the participating HEIs (n=36) improved their ACC Standard 3 (Academic and Administrative Staff). Target 40% (exceeded). In terms of improved quality of teaching, the project achieved or exceeded its targets, and achievement is therefore rated High. Rating High PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Improve the quality of management in project-supported entities Rationale Outputs relevant to this objective were as listed under Objective 1. Outcomes a) 56% of the participating HEIs (n=36) improved their ACC Standard 1 (Governance, Planning, and Evaluation). Target 40% (exceeded). b) 36% of the participating HEIs (n=36) improved their ACC Standard 4 (Financial Management). Target 40% (mostly achieved). c) 36% of the participating HEIs (n=36) improved their ACC Standard 5 (Dissemination of Information). Target 40% (mostly achieved). The objective of improved quality of management is considered partly achieved, as the project achieved only one out of three outcome targets; achievement is therefore rated Substantial. Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Improve the quality of research in project-supported entities Rationale Outputs related to this objective were as listed under Objective 1. Outcomes The project measured improved quality of research through an output-oriented indicator: “percentage of sub- projects approved and completed by closing date under the first and second round of proposals.” At project closing, all the approved projects had been completed by the closing date. Target: 70% (exceeded). However, this indicator did not directly measure improved quality of research. Annex 9 (ICR, p. 67) included additional information that is relevant for measuring quality: a) 40 research subprojects produced final reports on their research to be used for further research internally. b) 2 research subprojects published articles in domestic peer-reviewed documents to be used in-house or with other higher education institutions. c) 3 research subprojects published articles in international peer-reviewed journals. The project met targets for research support, final reports were generated for most (40 of 45) projects, and results for five projects were published in peer-reviewed outlets. Achievement of this objective is therefore rated Substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective Pilot the targeting of disadvantaged students for enhanced retention Rationale Outputs Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) Component 3 financed a pilot scholarship program to increase the retention of disadvantaged students in HE through the financing of 1,000 SPS based on pro-poor targeting and educational criteria. Target: 1,000 (achieved). Additional outputs included: (i) a manual on implementation of the SPS; (ii) a draft policy on scholarships, subsidies, and student loans; (iii) data collection to inform impact evaluation of the SPS program; (iv) a booklet to showcase SPS student success stories; (v) a University Guidebook to promote interest in university study among Grade 12 students; and (vi) a video showcasing examples of SPS student achievement that has been published on the World Bank website. Outcomes a) Enrollment rates: The enrollment rate of SPS students (tuition plus stipend) was 17 percentage points higher than students with tuition waiver only (Impact Evaluation, IE, p. 17). b) Retention rates: The annual pass rate of scholarship recipients at each grade was above 90% for the following categories: female, rural, and poor. Target: 80% (exceeded). The cumulative dropout rates of scholarship recipients were below 19% in 2013, 2014, 2015, 2016, and 2017 for female, rural, and poor recipients. Target: 19% (exceeded). The IE (p. 17) found that the SPS helped enrolled students to continue their studies into the second year at a rate 13.7 percentage points higher than for students with tuition waiver only. The impact of SPS on retention became higher with each additional year of enrollment. By the fourth year, the difference in retention rate between the SPS students and students with tuition waiver only was almost 34 percentage points (IE, p. 18) c) Completion rates: The completion rates of scholarship recipients exceeded 80% for female, poor, and rural students. Target: 67% (exceeded). The IE defined completion as applicable to those students who were promoted to the next academic year for four years consecutively without having to repeat any year, and found that the impact of SPS scholarships on on-time completion was positive and significant since SPS students were 33.9 percentage points more likely to meet the proposed completion definition than students who received a tuition waiver only. Overall, the results from the pilot indicated that, by providing additional financial assistance on top of tuition waivers, students from poor households were more likely to enroll in higher education and remain in the system until completion. Achievement of this objective is therefore rated high. Rating High PHREVDELTBL PHOVRLEFFRATTBL Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) Rationale Efficacy for the teaching objective is rated high, as targets were exceeded. Efficacy for the management objective is rated substantial, as financial management and information dissemination targets were mostly achieved. Efficacy for the research objective is rated substantial despite some measurement shortcomings, and the lack of a full-fledged counterfactual discussion of results, as the funded projects reached completion and several passed the scrutiny of peer review. With respect to the scholarship pilot for enhanced retention objective, efficacy is rated high as outcome targets were met or exceed. In conclusion, overall efficacy is rated substantial. Overall Efficacy Rating Substantial 5. Efficiency The project conducted an ex-ante and ex-post cost benefit analysis. Appraisal At appraisal, the project conducted an ex-ante cost benefit analysis with three scenarios (low, base, high) that covered Components 1, 2, and 3 (approximately 80% coverage). The analysis estimated the economic cost of the project at US$23 million over five years. The analysis identified two main economic benefits: First, the analysis assumed that some students will obtain a higher education degree because of the project. For this group, economic benefits were measured as the wage premium of the students who obtained a higher education degree because of the project. The wage premium between higher education graduates and secondary education graduates was calculated using available data from 2004, which was adjusted by a 5% inflation rate from 2010 onwards to identify the stream of benefits until 2040. The analysis assumed that the SPS would lead to the graduation of 250 students who would have otherwise not graduated without the SPS under Component 3. Moreover, the analysis assumed that the project would award 300 research grants under Component 2, and that each grant would contribute to the graduation of 5 graduates who would otherwise not have graduated. This latter assumption was inconsistent with the statement elsewhere in the PAD (p. 43) that the project would finance 150 grants. Overall, the analysis estimated that the project support would produce 1750 graduates who would have not have graduated in the absence of the project. Second, the analysis assumed that, thanks to the project support, students enrolled in higher education would graduate from a system that has better quality (for example, improved curricula, better lecturers, etc.). The analysis also placed a wage premium for these graduates equivalent to 1% (low scenario), 3% (base scenario), and 5% (high scenario). The analysis assumed that all 2014 graduates would benefit from these quality improvements. The PAD was not clear on how the number of graduates was calculated but seemed to suggest that it was calculated using enrollment rates. Using only 2014 graduates likely underestimated the stream of benefits, since it can be reasonably assumed that the quality improvements would be sustained beyond 2014 and thus would benefit subsequent graduate cohorts. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) The analysis found that, in the base scenario, benefits added up to nearly US$40 million (wage premium equal to 3% and IRR 22.9%), whereas in the low and high scenarios benefits added up to US$20.8 million (wage premium equal to 1% and IRR 14.3%) and US$58.9 million (wage premium equal to 5% and IRR 29.6%). The discount rate used for the estimations was 10%. In summary, the ex-ante cost benefit analysis indicated that the project's benefits would outweigh its cost. Closing At closing, the project conducted an ex-post cost benefit analysis with three scenarios (low, base, high) that covered Components 1, 2, and 3 (approximately 80% coverage). The analysis quantified two economic benefits: The ex-post analysis assumed that 10 cohorts of 44,659 students would be enrolled in a higher education system that has better quality and that, thus, these graduates will benefit from higher earning streams throughout their working lives. For these students, the analysis estimated a wage premium equivalent to 0 years of tertiary education (low scenario); 0.2 years of tertiary education (base scenario); and 0.4 years of tertiary education (high scenario). This estimation of benefits was problematic, as it assumed that any exposure to the higher education system would lead to a wage premium (Source: World Bank Team interview). In other words, the assumption implied that even dropouts will command a wage premium. A more realistic approach may have been to use graduation rates to estimate the stream of benefits for the 10 cohorts of students. The ex-post analysis assumed that 1,277 graduates would not have graduated in the absence of the project. The project provided fellowships to 72 higher education professors to pursue postgraduate degrees and thus improve the quality of their teaching under Component 1. The analysis assumed that this quality improvement would led to the generation of 10 additional graduates per professor. Moreover, under Component 2, 45 research grants were awarded. The analysis assumed that these grants would lead to the generation of 5 additional graduates each. Finally, under Component 3, the project provided special priority scholarships to 1,000 poor students. As per results from the IE showing that the dropout rates for this group declined from 48% to 15.5% because of the scholarships, the analysis assumed that the scholarships led to the graduation of 332 disadvantaged students who would not otherwise have graduated. As in the ex-ante analysis, the wage premium between higher education graduates and secondary education graduates was calculated using available data from 2004, adjusted by a 5% inflation rate from 2010 onwards to identify the stream of benefits until 2040. The ex-post cost-benefit analysis identified two costs. First, the analysis estimated the economic cost of the project at US$14.9 million over a five-year horizon. Second, the analysis included the additional public and private resources incurred for the 1,277 individuals who would not have received a degree in the absence of the project. The analysis estimated the public and private resources per student at US$ 1000 and US$769 per year. This cost was estimated at US$7.9 million. The overall project cost was estimated at US$22.85 million. The results of the analysis showed that, in the base scenario, the IRR was 22%, and for the low and high scenarios was 1.6% and 29.6% respectively. The discount rate used for the estimations was 5% instead of Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) the 10% used at appraisal. This lower discount rate overestimated future cash flows. The World Bank Team explained that the 5% percent discount rate chosen was based on Standard and Poor’s non-rated municipal bond index. In October 2014, this index showed that the average maturity of the unrated bond market sat at 19 years, with an average yield at 5.95%. Overall, the results of the analysis were based on some assumptions that are likely to overestimate benefits. Nonetheless these assumptions, the analysis suggests that the IRR would still have been positive even when adjusting for these factors. Implementation efficiency Implementation efficiency had important shortcomings, including multiple delays and challenges with the fellowship program, accreditation guidelines and standards, research projects, payment of scholarship stipends, and recruitment of trainers (ICR, pp. 28-29). These implementation issues motivated that the project had to request a 21 months extension in its closing date from December 2015 to September 2017. The main rationale for the 21-month extension, as noted in the Management restructuring paper, was to complete all original activities at a satisfactory level. Although the World Bank team worked effectively and proactively to address many of these challenges, the issues experienced nevertheless diminished the efficiency of use of project resources. Efficiency is therefore rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 80.00 Appraisal  22.90 Not Applicable 80.00 ICR Estimate  22.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The PDO were highly relevant to country conditions and to government and Bank strategy at appraisal and closing. Overall efficacy is rated Substantial, with two objectives highly achieved and two substantially achieved. Efficiency is rated modest due to implementation inefficiencies that led to a 21-month extension in project completion. The overall outcome rating is therefore moderately satisfactory. Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome This validation identifies three main risks to development outcomes: • Government commitment. The government sees higher education as one of its top priorities, as indicated in the government’s Industrial Development Policy (2015-2025). This indicates that government commitment to the sector is not likely to waver in the future. For instance, the ICR (p. 29) noted that the MoEYS has incorporated the successful aspects of the SPS program into its scholarship program. Overall, this risk is considered low. • Sustainability of institutional capacity gains. The capacity of the higher education sector improved throughout the project, and this strengthened capacity is unlikely to be reversed. To the contrary, capacity will be further augmented, since the World Bank is currently preparing, as per government request, another project that will continue support to the HE sector. Moreover, this project has also been instrumental in creating long- term professional and institutional relationships between Cambodian HEIs and wider international academic networks. These relationships are not likely to disappear. Finally, the project also supported the development of the HEMIS whose capacity is likely to grow in the future. This risk is therefore also considered low. • Migration of higher education graduates. A recent Asian Development Bank report noted that brain drain is a source of ongoing concern for members of the Association of Southeast Asian Nations (ASEAN). Brain drain, or the emigration of highly educated individuals, has been recognized as a significant obstacle to economic and social development. Specifically, the number of tertiary-educated ASEAN emigrants in Organization for Economic Co-operation and Development (OECD) countries increased from 1.7 million in 2000 to 2.8 million in 2010–11. The motivation to emigrate to OECD countries remains strong: a powerful base of jobs and support networks makes it likely that skilled emigration from ASEAN to OECD will continue. This risk is considered substantial. (Source: Asian Development Bank. Firing up regional brain networks: The promise of brain circulation in the ASEAN economic community. Mandaluyong City, Philippines: Asian Development Bank, 2017). 8. Assessment of Bank Performance a. Quality-at-Entry The World Bank developed a simple yet adequate evidence-based technical design for the project, reflecting learning from other World Bank higher education projects and globally recognized approaches for developing effective higher education interventions. The PDO formulation was precise and relevant to country priorities as reflected in the government’s Education Sector Plan 2006-2010. The PDO was also Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) relevant to World Bank’s Country Assistance Strategy (FY05 -FY11). However, the quality at entry was not without shortcomings. First, the initial monitoring and evaluation arrangements lacked an adequate indicator to measure improved “quality of teaching, management, and research." This shortcoming was addressed later during implementation (although an indicator directly to measure research quality remained absent). Second, although the risk assessment was mostly adequate, it failed to identify migration of higher education graduates as a risk to development outcome. Considering these shortcomings, quality at entry is rated moderately satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The World Bank team conducted 10 supervision missions which included a MTR in July 2013. The team systematically supervised the project through aides-memoire, back-to-office reports, and implementation and supervision reports. These documents kept management informed of progress and provided a solid basis for the elaboration of the ICR. Whenever implementation bottlenecks arose, the World Bank team worked with the government to find solutions. Several examples illustrate this proactive problem-solving approach of the World Bank team. First, the awarding of fellowships began slowly due to the need to make appropriate overseas university connections. To address these issue, the project hired an international consultant to assist the DHE in making the needed connections. Second, the implementation of research grants was delayed due to the limited experience of the universities and project managers with procurement and financial management processes. To address these bottlenecks, the World Bank team, through its procurement and financial management specialists, provided support and training on how to procure materials as well as how to monitor and report expenditures. Third, the World Bank team adequately adapted the project activities to respond to government requests. For instance, the government requested the project to work with the Royal University of Phnom Penh to create a model HEI that exemplified good practice in governance and financial and human resource management. The World Bank team responded by including activities to accommodate this request in the 2015 restructuring. Finally, at the 2014 restructuring, the World Bank team adapted the monitoring and evaluation design to better document development outcomes. Overall, the quality of supervision is rated highly satisfactory. Quality of Supervision Rating Highly Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) The PDO clearly articulated the outcomes sought by the project with respect to higher education: (i) improve the quality of teaching, management, and research in project supported entities; and (ii) pilot the targeting of disadvantaged students for enhanced retention. The M&E design included an adequate outcome indicator for measuring the impact of the project on targeting disadvantaged students for enhanced retention. The M&E design also included an IE to compare the retention rates of special priority scholarship recipients (tuition waivers plus living stipends) vis-à-vis a control group of traditional scholarship recipients (only tuition waivers). However, several shortcomings can be noted in the M&E design with respect to measurement of improved quality of teaching, management, and research. First, with respect to measuring the quality of teaching and management, the original M&E design lacked an indicator to measure these dimensions of the PDO. To remedy this shortcoming, the team introduced an additional outcome indicator at restructuring in 2014 to capture the quality of teaching and management. The indicator was built using ACC accreditation standards that measure performance of higher education institutions along the following dimensions: (i) Governance, Planning, and Evaluation; (ii) Academic Program; (iii) Academic and Administrative Staff; (iv) Financial Management; and (v) Dissemination of Information. Second, with respect to the quality of research, the M&E design included an output oriented measure (i.e. percentage of sub-projects approved and completed by closing date under the first and second round of proposals) that was not an adequate measure of “quality of research." Better measures would have been, for instance, the “number of publications in peer reviewed journals” or any of the typically used measures in the field of bibliometric analysis. Finally, the M&E design lacked a counterfactual for measuring the impact of the project on the quality of teaching, management, and research in project-supported entities. This lack of counterfactual diminishes the credibility of the reported results with respect to these outcomes. b. M&E Implementation Overall, the project successfully implemented the M&E system and systematically collected data for the indicators included in the results framework throughout the life of the project. The project also successfully implemented activities that have led to a strengthened monitoring and evaluation capacity for the entire higher education sub-sector. For instance, the project successfully designed and piloted the HEMIS in 6 higher education institutions. Finally, the project carried out the IE to assess the impact of the special priority scholarship program as planned. c. M&E Utilization The ICR reported that the DHE analyzed and reported on the results frameworks indicator achievement on a semiannual basis, maintained databases for all the components monthly, and provided cumulative data on all the capacity building activities. At the sector level, the data generated by the project and through the HEMIS was used to inform the Government’s Education Sector Plan 2014-2018 and Higher Education Road Map indicators. The data was also used by the DHE to develop results-based planning and a monitoring and evaluation framework for the HE sub-sector based on the policy areas of the Education Sector Plan (access, equity, and institutions). Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was rated Category “C” as per OP/BP 4.01, and thus it did not require an environmental assessment. The project did not trigger any additional safeguard policies, and therefore it did not require any further assessments. Implementation Status Reports (ISRs) elaborated by management indicated that safeguards were monitored and were consistently rated satisfactory throughout project implementation. b. Fiduciary Compliance Financial management Financial Management ratings were satisfactory throughout implementation until ISR Sequence 11 (March 2016) when they were downgraded to moderately satisfactory until the project closed. The downgrade was driven by shortcomings in the financial management internal controls of HEIs. To address this issue, the World Bank revised the project financial management manual to strengthen the internal controls. The ICR reported that: (i) the DHE / MoEYS complied with the financial covenants of the Financing Agreements; (ii) all quarterly unaudited interim financial reports were submitted to the World Bank on time; (iii) the required audits were submitted to the World Bank on time in accordance with stipulations of the Financial Agreements; and (iv) the auditors’ opinion was clean. Overall, compliance with financial management procedures was satisfactory. Procurement By and large, procurement performance was rated moderately satisfactory throughout implementation except for ISRs Sequences 3 to 5 where it was rated satisfactory. Two factors explain the MS ratings. First, the project design assigned procurement duties to the DHE staff who lacked experience with World Bank procedures. In hindsight, this responsibility would have been perhaps better placed in the MoEYS procurement unit that already had experience with World Bank procedures. Second, the HEIs receiving support had no experience with World Bank procedures. In practice, this meant that research project directors in HEIs required extensive training before being able to procure goods and services for their research projects. The combination of these two factors, the ICR noted, explains the slow disbursement rates at the beginning of implementation. To tackle this bottleneck, the World Bank delivered procurement trainings and support by Bank procurement staff that managed to improve implementation pace and disbursements. Overall, fiduciary compliance and performance was satisfactory. Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) c. Unintended impacts (Positive or Negative) The ICR (p. 25-27) reported several unintended positive impacts. First, the project supported the completion of the Vision 2030 and the Higher Education road map, both of which have been fully adopted by the MoEYS as guides for policy reform and guidance in the higher education sector. Specifically, the road map provided a detailed framework for implementing the higher education policy that was envisioned in the Vision 2030. Moreover, the MoEYS is using the two documents to conduct a legislative review of their current higher education institutions' policies, particularly as they relate to financial and human resource management, to develop legislation on increased university autonomy. Second, the ICR also reported that the research activities supported by the project have led to the establishment of the Cambodia Society for Research and Development and the Cambodia Citation Index. These two initiatives will contribute to sustain the culture of research support and collaboration among HEIs. Finally, the project also led to the establishment of a higher education working group whereby key officials from relevant line ministries and development partners meet regularly to discuss and analyze the sub-sector. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Efficiency is rated modest due Moderately to implementation Outcome Satisfactory Satisfactory inefficiencies that led to a 21- month project extension. Quality at entry is rated Moderately moderately satisfactory due Bank Performance Satisfactory Satisfactory primarily to shortcomings in the initial results framework. Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR lists six lessons (p. 34). Important among these are the following: 1. Peer partnerships and networks are effective in improving the quality of professional performance. The project activities have led to the creation of the Cambodia Society for Research and Development. This research community, through the practice of peer review, is expected to contribute to promoting rigor and relevance in the performance of higher education professionals. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) 2. Forging long-term relations with higher education institutions overseas can reduce training and capacity development costs as well as offer sustainable ways of providing consistent capacity development opportunities. The project led to increased professional connections and long-term partnership agreements with universities in Australia that include, among other things, grants to fund Cambodian students. 3. Research is essential if Cambodia is to achieve success in engaging with global knowledge networks and training its own postgraduate research students. Engaging with global knowledge networks requires exchanging research products, and students are better trained through the hands-on experience that research projects provide. IEG concurs with these lessons and adds another one: 4. Migration of higher education graduates can pose a risk to the development and sustainability of higher education systems in lower lower-middle-income economies such as Cambodia. A recent Asian Development Bank report noted that brain drain is a source of ongoing concern for members of the Association of Southeast Asian Nations. For this reason, strategies that aim to develop higher education systems in such countries should include mitigation measures that include: (i) approaches for minimizing migration outflows, and (ii) approaches for using the skills and experience of emigrant academics and high-skilled workers. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided a thorough overview of the project with a concisely and well written narrative structured around the project development objective that supported the ratings. In addition, the ICR included a clear and informative figure describing the project’s results chain (ICR, p. 12) and offered a detailed description of implementation challenges, how they were addressed, and the two restructurings (2014 and 2015) that modified the results framework. Overall, the quality of the evidence and of the analysis was solid, though the ICR could have done a more thorough job explaining the biases implicit in the assumptions that were factored into the efficiency analysis. Moreover, the ICR was not thorough in explaining how the ACC accreditation criteria were aggregated for measuring the outcome indicator introduced at the 2014 restructuring. Information on how the ACC accreditation criteria were aggregated was provided during the interview with the World Bank team. Notwithstanding these shortcomings, the quality of the ICR is rated substantial. a. Quality of ICR Rating Substantial Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KH-HE Quality and Capacity Improve Proj (P106605) Page 19 of 19