VIETNAM Education Projects: School Readiness and Escuela Nueva Report No. 135418 MAY 7, 2019 © 2019 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2019. Vietnam—Education accuracy of the data included in this work. The Any queries on rights and licenses, including Projects: School Readiness and Escuela Nueva. boundaries, colors, denominations, and other subsidiary rights, should be addressed to Independent Evaluation Group, Project information shown on any map in this work do World Bank Publications, The World Bank Performance Assessment Report 135418. not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC Washington, DC: World Bank. World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. Report No: 135418 PROJECT PERFORMANCE ASSESSMENT REPORT VIETNAM VIETNAM SCHOOL READINESS PROJECT (IDA CREDIT NO. 52070) GLOBAL PARTNERSHIP FOR EDUCATION—VIETNAM ESCUELA NUEVA PROJECT (LOAN NO TF-13048) May 7, 2019 Human Development and Economic Management Independent Evaluation Group Currency Equivalents (annual averages) Currency Unit = Vietnamese Dong (VDN) 2010 $1.00 VDN 18,612.92 2011 $1.00 VDN 20,509.75 2012 $1.00 VDN 20,828.00 2013 $1.00 VDN 20,933.42 2014 $1.00 VDN 21,148.00 2015 $1.00 VDN 21,697.57 2016 $1.00 VDN 21,935.00 2017 $1.00 VDN 22,370.09 2018 $1.00 VDN 23,195.00 Abbreviations DLI disbursement-linked indicator ECE early childhood education EDI Early Development Index ICR Implementation Completion and Results Report IEG Independent Evaluation Group M&E monitoring and evaluation MOET Ministry of Education and Training PMU Project Management Unit PPAR Project Performance Assessment Report SRPP School Readiness and Promotion Project VNEN Vietnam Escuela Nueva All dollar amounts are U.S. dollars unless otherwise indicated. Fiscal Year Government: 1 January – 31 December Director-General, Independent Evaluation Ms. Alison Evans Acting Director, Human Development and Economic Management Ms. Sophie Sirtaine Manager, Corporate and Human Development Ms. Emanuela Di Gropello Task Manager Ms. Susan A. Caceres ii Contents Preface .................................................................................................................................................................. v Summary ............................................................................................................................................................. vi 1. Background and Context ..........................................................................................................................1 Education in Vietnam ................................................................................................................................. 1 2. Design and Preparation: Strengths and Weaknesses ....................................................................4 Overall Design and Preparation ..............................................................................................................5 Teacher Training Design .......................................................................................................................... 10 Preparing for Scaling Up of Teacher Training ...................................................................................11 Monitoring and Evaluation Design ...................................................................................................... 12 3. Implementation and Supervision: Strengths and Weaknesses ............................................... 13 Overall Implementation and Supervision .......................................................................................... 13 Implementation of Teacher Training ................................................................................................... 17 Implementation of Scaling Up ............................................................................................................... 18 Implementation of Monitoring and Evaluation............................................................................... 20 4. Results ........................................................................................................................................................... 22 SRPP Results ............................................................................................................................................... 23 VNEN Results.............................................................................................................................................. 25 5. Lessons.......................................................................................................................................................... 26 Bibliography..................................................................................................................................................... 33 Box Box 2.1. Characteristics of Effective In-Service Teacher Training.......................................................11 Table Table 1.1. Vietnam’s Gross Enrollment by Level and Year.....................................................................3 Figures Figure 2.1. School Readiness and Promotion Project Theory of Change ........................................6 Figure 2.2. Vietnam Escuela Nueva Theory of Change ......................................................................... 7 iii Appendixes Appendix A. Vietnam School Readiness and Promotion Project Ratings ................................ 37 Appendix B. Vietnam Escuela Nueva Project Ratings ...................................................................... 52 Appendix C. Basic Project Information .................................................................................................. 68 Appendix D. List of Persons Interviewed .............................................................................................. 71 This report was prepared by Susan A. Caceres and Anthony M. Tyrrell, who assessed the projects in January 2019. The report was peer reviewed by Denise Vaillancourt and panel reviewed by Sian Williams. Aline Dukuze provided administrative support. Preface This Project Performance Assessment Report (PPAR) report for the Vietnam School Readiness and Promotion Project (SRPP; P117393) is part of a pilot initiative implemented by the Independent Evaluation Group (IEG) to improve the relevance of the instrument. Several PPARs are experimenting with various report structures this year. This report follows the structure of the project cycle examining the strengths, weaknesses, results, and lessons during preparation and implementation. The aim of the pilot is to make the PPAR more relevant and accessible to the principle audience of the report, World Bank operational staff. Accountability elements are retained in the report, since ratings and their justifications are contained in appendixes A and B. This report reviews the experience and achievements of World Bank support to education through the Vietnam SRPP and Global Partnership for Education—Vietnam Escuela Nueva Project (P120867). SRPP was approved on February 28, 2013, became effective on July 26, 2013, and closed on June 30, 2017. The Vietnam Escuela Nueva Project was approved on July 31, 2012, became effective on January 9, 2013, and closed on May 31, 2016. Both projects ended without an extension. These projects were selected for field-based assessment because they focused on improving education quality and the provision of in-service training. The assessment aims to verify whether the operations achieved their intended outcomes. The report provides evidence and data not available at project closure for a more complete picture of the outcomes and factors that influenced them. The information gained from this field-based study will also be an input into an IEG evaluation of the World Bank’s support for preservice and in-service teacher training. This report presents findings based on a review of the project appraisal documents, the Implementation Completion and Results Reports, Implementation Completion and Results Report Reviews, aide-mémoire, World Bank reports, and other relevant materials. An IEG mission visited Hanoi, Dak Lak, Lào Cai, and Lam Sum from January 7 to 18, 2019, to interview Ministry of Education and Training officials, provincial and district officials, principals, teachers, and parents (see appendix D for a list of persons interviewed). IEG gratefully acknowledges the logistical assistance and support of the Ministry of Education and Training and Ms. Huyen Thi Thanh Le in the World Bank Hanoi office. Dr. Hai Pham provided invaluable assistance before, during, and after the mission. Following standard IEG procedures, a copy of the draft PPAR was shared with relevant government officials for review and feedback. No comments were received. v Summary Background and Context: Education in Vietnam The government and people of Vietnam place a high value on education. The government’s Socio-Economic Development Strategy 2010–20 and the Socio-Economic Development Plan 2016–20 emphasize the importance of investment in human capital to develop people’s skills in support of a knowledge-based economy. Since 2008, the government has been spending about 20 percent of its annual budget on education (GPE 2019). Parents are also required to invest in education, paying for school lunches, textbooks, or school fees, with reductions and exemptions for poverty or disadvantage. Investment in education has resulted in enhanced infrastructure and increased enrollment. The number of students enrolled in primary school increased from about 7.5 million in 2014 to 8 million in 2017. The number of ethnic minority primary students has remained proportionate to the overall number of primary students (1.3 million in 2014 and 1.4 million in 2017). But ethnic minority children are less likely to attend upper secondary or tertiary schools. Near–gender parity is present at primary, lower, and upper secondary education (47 percent female to 53 percent male enrollments). Several initiatives have aimed to improve the quality of education since the early 2000s. Efforts have been made to further professionalize teachers through standards and awards for excellence and the introduction of required self-learning for teachers through in-service opportunities. The Ministry of Education and Training (MOET) is universalizing full-day programming at all levels of education. The Fundamental School Quality Levels set minimum quality standards for primary schools that include school self-assessment and validation. MOET (with World Bank support) is currently reforming the general education curriculum, moving toward a competency-based approach (World Bank 2016a). Positive outcomes are associated with this intensive investment in and attention to education, but several challenges have persisted. A national assessment of fifth graders showed improvement in math and reading scores between 2001 and 2007 (World Bank 2011a). Vietnam’s average Programme for International Student Assessment score in math, science, and reading is above the average for Organisation for Economic Co- operation and Development countries, though with a slight decline in average math and reading scores in 2015 (OECD 2018). Yet the preservice preparation of teachers remained weak, pointing to the importance of investing in in-service training and more effective pedagogical approaches; and improving learning outcomes for disadvantaged children (including ethnic minority children) remained a challenge. vi Role of the World Bank in Supporting Education in Vietnam It is largely in response to these challenges that the World Bank supported the Vietnam Escuela Nueva (VNEN) and School Readiness and Promotion Project (SRPP). VNEN sought to introduce a new pedagogical approach (at scale but not across the entire system) that would ultimately replace the traditional approach of teaching (rote learning) and introduce a more participative and collaborative approach to classroom learning. The hoped-for consequence of this new approach was that graduates would emerge as critical, independent thinkers who had strong communication skills and could work collaboratively in a modern, knowledge-based economy. SRPP sought to address quality issues through the introduction of school self-assessments in the rapidly growing preschool sector, and to enhance the training of preschool teachers to introducing a more child-centered and play-based approach. Both projects focused on disadvantaged groups at preschool and primary levels. The objectives, performance indicators, and targets of the two projects remained unchanged throughout their life cycles. The objective of SRPP (supported with an International Development Association credit of $95.19 million) was “to raise school readiness for five-year old children, in particular for those most vulnerable to not succeeding in a school environment, through supporting selected elements of Vietnam’s Early Childhood Education (ECE) program” (World Bank 2013a, 4). The development objective of VNEN (supported by a loan of $84.30 million from the Global Partnership for Education) was “to introduce and use new teaching and learning practices in the classroom targeting the most disadvantaged groups of primary students” (World Bank 2012, 3). The World Bank acted as trusted adviser, knowledge broker, and provider of funding in supporting government plans to reform the education sector in Vietnam. The World Bank has had a long engagement in the education sector in Vietnam (for example, before VNEN and SRPP the World Bank supported the School Education Quality Assurance project, and it continues to support the Renovation of General Education Program and the Teacher Training Education Program). During the sustained support in the sector, which largely focused on basic education, the World Bank has been an important partner and trusted adviser to the government in support of reform efforts. The World Bank was involved in ongoing policy discussion with counterparts and facilitated the introduction of an international and external dimension to education reform in Vietnam. For example, the World Bank team was in discussion with key actors in MOET about early childhood development policy that supported the government’s updated Early Childhood Education Strategy (2016–25) that expanded the focus of early childhood development to include children from birth to five years old. The World Bank arranged study tours for key MOET personnel under both projects and supported vii international influence throughout the life cycle of both projects (for example, international experts were present at central and local levels to support the introduction of VNEN). Client counterparts were uniformly positive about World Bank support for education reform. Client representatives interviewed by the Independent Evaluation Group (IEG) emphasized that the use of development aid for education projects was not just about the funding but also about bringing in World Bank knowledge and experience as well as international experts and advisers who could help realize the education strategy. Client representatives cited close working relationships with World Bank staff and the importance of those staff in helping to design and steer system innovations. Preparation: Strengths and Weaknesses Preparation and design for both projects benefited from significant investment in lead time and effort that was supported by ongoing policy discussions between the World Bank and MOET. For SRPP, preparation spanned more than four years from project initiation to approval by the World Bank’s Board. This time was well used and laid a solid foundation for developing the disbursement-linked indicators (DLIs) and the system approach for early childhood development that simultaneously promoted access and quality. Policy dialogue in preparation for SRPP was supported by multiple pieces of analytical work, technical assistance, and an early childhood development policy conference. Lead time for VNEN included a pilot under a preceding operation (the Primary Education for Disadvantaged Children Project), a regional workshop sharing best practices in rural education, and field trips to Colombia for key MOET officials, as well as technical assistance from international experts. Policy dialogue facilitated the government’s broader reform program to improve education quality through a new emphasis on critical thinking, problem solving, and communication skills. The design drew on prior lessons and international experience that benefited implementation and results. VNEN (based on a proven innovation from Colombia) was inherently a product of international experience and recognized good practice. The project design also incorporated lessons from the VNEN pilot, including the need for a “comprehensive package” to ensure effectiveness and “dedicated knowledge management” to produce and disseminate knowledge about the new pedagogical methods (World Bank 2012, 7). The design of SRPP built on lessons and experience from the Primary Education for Disadvantaged Children Project about the application of school accreditation to preschools. Both operations contained aspects of student assessment and both pursued child-centered learning as well as teacher-facilitated (rather than teacher-directed) learning. Simplified (but robust) financial management included in the design of both projects facilitated smooth implementation and progress. Both projects used retroactive viii financing. For VNEN, this mitigated some of the implementation risks associated with the short three-year implementation period. For SRPP, retroactive reimbursement of eligible expenditures was provided based on completion and verification of DLIs. VNEN also used a modified financial management arrangement—grant funds sent from MOET to a provincial (commercial) bank account and then directly to school bank accounts—that eased the flow of funding to keep the project moving. These design features contributed to successful implementation. The comprehensive package designed and delivered under VNEN (including training, training materials, learning guides, and funding for learning materials) had several strengths. First, the package included incentives. Schools had to apply and compete to be accepted into the program, so the program was already engaging with a motivated group. Successful schools would also receive funding for materials to support learning, and teachers, once trained, were encouraged to shoot videos that could be hosted as learning tools on the VNEN website. VNEN also adopted a modified cascade approach to training that removed layers between the original training and the recipients of training (the majority of local teachers in targeted schools). This served to enhance fidelity and quality, particularly as face-to-face training benefited from follow-up support through access to international experts. The project also involved group-level engagement between teachers in individual schools as well as cluster meetings between schools. The inclusion of school managers in the training was positive and ensured deeper understanding and support for VNEN. The introduction of the VNEN innovation into the reform process predated curriculum reform, which may not have been ideal. The sequencing of VNEN was discussed extensively; MOET decided that it was appropriate to push for pedagogical changes in the absence of curriculum reform. In hindsight, key stakeholders noted that the type of pedagogical change envisaged under VNEN would ideally have been sequenced subsequent to reframing the overall objective of the education system and the consequent revision of the curriculum, which would then open the way to the production of appropriate learning material and other supports. As it transpired, VNEN had to use the existing curriculum and text even though it made major changes in how teachers would instruct. The project also produced textbooks based on the existing outdated curriculum. Limited follow-up support and a lack of incentives for local actors were design weaknesses in SRPP. The provision of complementary materials was not envisaged and follow-up after the training was limited. As noted earlier, some incentives applied under VNEN were absent under SRPP, where there were no incentives for preschool teachers, school managers, or districts and provinces (who were asked to do more training and school accreditation with the existing budget transfer and incentives). In pursuit of systemwide coverage under SRPP, interventions—school self-assessments, accreditation, ix and training—were uniformly implemented without compensatory support based on the differentiated need or capacity at the local level. Implementation: Strengths and Weaknesses For the most part, project implementation ran smoothly. Both projects rapidly disbursed, were delivered within the planned period with no overruns or extensions, and fully disbursed at project close. The solid implementation of VNEN and SRPP is the result of a committed, well-working system as well as a strong working relationship between the World Bank and MOET, backed by strong design features. Implementation through the government education administrative structure supported by a Project Management Unit (PMU) located in MOET and personnel that were line staff at MOET was a particular strength. The staffing of PMU with MOET officials was realized for the first time under VNEN and subsequently applied under SRPP. The symbiotic relationship between MOET and PMUs meant that challenges were efficiently and effectively handled through immediate problem solving (all relevant personnel were working within their “own” system), and effective communication between the World Bank and the MOET PMU. Finding the balance between achieving breadth and depth in scaling up proved challenging. SRPP included a systemwide scaling up focused on breadth of coverage, which suited the transactional aspects of the project such as reimbursement for lunch subsidies for children of disadvantaged families. Other aspects of the project, such as teacher training and school assessment, were less well supported by the emphasis on breadth at the expense of depth. For example, SRPP training did not consistently offer the same opportunities across provinces for peer engagement through, for example, learning circles or school cluster exchanges. The VNEN project, in contrast, focused on depth, but at significant scale. Overall, a quality program was delivered to participating schools. However, efforts to expand the model beyond project-based schools (to voluntary schools) was problematic, since these schools did not get the same types or levels of support. The systemwide delivery of training under SRPP (93 percent of all preschool teachers trained by project close) was efficiently implemented, but certain elements that support quality were absent. There was no differentiation in training with reference to the age of preschool students. The training likely discussed developmental differences. This lack of significant differentiation is striking given the developmental differences in learning among very young children as well as the type of learning activities that teachers would need to use to engage students across a critical age span from a developmental perspective. x There was tension between the purported targeting and the approach adopted by the two projects. SRPP purported to target low-capacity preschools to deliver quality services for the most vulnerable children (“those whose cognitive, socio-emotional and physical development is most lagging” [World Bank 2013b, 6]), but at the same time focused for the most part on systemwide interventions. The project did not succeed in putting in place targeted supports for school self-assessment, accreditation, and training, despite the varying provincial capacity. As a result, the rate of accreditation was not achieved equally across provinces, and disadvantaged provinces had lower rates. In addition, no specific activity supported the language development of ethnic minority children, a group with high vulnerability. VNEN purported to target the most disadvantaged groups, particularly ethnic minority children. In practice, however, VNEN targeted (although not exclusively) disadvantaged provinces that had the highest concentration of ethnic minority children. Schools were invited to apply to create a demonstration effect to encourage further uptake of the VNEN approach. Thus, priority was given to schools in the most disadvantaged provinces rather than the most disadvantaged students or most disadvantaged schools. Communication challenges arose for both projects. Significant effort was put into communication efforts to support the introduction of VNEN (with local authorities, communities and parents, school administrations, and teachers). The project also used the media and a well-developed website to promote the innovation. Despite these efforts, the project encountered early resistance to change among some parents and teachers. This was associated with distrust of the new model and its possible impact on tradition, working styles, or student performance. Scaling up encountered potentially more serious issues associated with the less well-resourced expansion efforts (beyond the confines of project targets). More effective communication and consultation might have picked up the negativity associated with VNEN expansion to voluntary schools before it became public. Ultimately the negative press affected the potential longevity of the model. SRPP, for its part, involved the rollout of a mandatory national program— there was no obligation or need to “sell” the intervention. A communication strategy was established to raise awareness. Stakeholders reported a few meetings between teachers and parents at school, but this does not constitute meaningful parental engagement, which was particularly needed for parents of vulnerable children. Monitoring and evaluation efforts could have adopted a wider perspective on education reform. Project monitoring (particularly under VNEN) largely focused on output measures. The SRPP results framework did include two outcome measures: increased enrollment and decreased vulnerability in children. Additional sources of evidence were generated, such as classroom observations and impact evaluation (VNEN) and three rounds of the Early Development Index survey (SRPP); however, the monitoring and evaluation arrangements lacked a stretch element. As designed, results frameworks would not help MOET assess the effectiveness of its policies or reform efforts: the results xi framework for SRPP would not provide evidence related to the effectiveness of school self-assessment, lunch subsidies, or training, nor did the framework specify disaggregated indicators consistent with the focus on vulnerability. The impact evaluation carried out under VNEN was useful but limited (possibly associated with early implementation delays). Given the longevity and depth of World Bank support for education in Vietnam, a more programmatic approach to assessing outcome and impact (that embraced VNEN, SRPP, and preceding and succeeding projects) would have been strategically beneficial for MOET. Project Results and Ratings The joint contribution of the SRPP and VNEN projects to education reform in Vietnam is likely greater than the sum of their respective parts. Although impressive and efficiently realized, the measured results of both projects tell only part of the story. The projects were part of a national reform strategy as well as a program of World Bank support for education reform. Unfortunately, the monitoring and evaluation system that accompanied this sustained effort operated on a project-by-project basis—largely with output indicators—and without a longitudinal, programmatic assessment. Had such a system been in operation, it might have generated further objective evidence to substantiate attribution of World Bank efforts. In 2016 the government of Vietnam extended the time allowed for ongoing payment of the lunch subsidy that was supported under SRPP. The subsidies were expected to address some of the enrollment barriers for disadvantaged children, incentivizing the parents of these children to send them to preschools. The provision of lunch eliminated the need for parents of disadvantaged children to go to the school several times a day, freeing up time for the parents to work. A healthy meal also improved children’s nutrition (World Bank 2013b). Nearly all preschool managers (99 percent) and preschool teachers (93 percent) completed the priority modules. Teachers reported that the content was practical and the training was interactive. They experienced a positive response from children, which motivated them to apply the training. During the IEG mission, teachers still indicated a need for more hands-on support and for resources such as lesson plans. By January 2019, 96 percent of preschools had completed a self-assessment, and all 63 provinces (target of 50 provinces) achieved universal preschool for five-year-old children. By project close, 41 percent of preschools had achieved level 1 accreditation (against a target of 40 percent), although this was not systematically achieved in disadvantaged provinces. The Early Childhood Education Strategy (2016–25) has established a goal for 80 percent of preschools to attain level 1 accreditation. The number of children in kindergarten increased from 3.7 million in 2014 to 4.6 million in 2017. The overall rate of growth and the rate for ethnic minority children were similar xii (respectively 17.7 percent and 16.7 percent). The rate of enrollment (part and full day) for children five years old increased during the operation from 96 percent in 2012 to 99 percent in 2017, and the percentage of children five years old participating in a full- day program increased from 73 percent in 2012 to 88 percent in 2017. Analysis of the Early Development Index survey (conducted in 2012, 2014, and 2016) showed a reduction in the share of five-year-old children who are vulnerable. The analysis revealed improvements in children’s readiness although almost a third of children (29 percent) were vulnerable or at risk of vulnerability (lowest two groups) in the final survey compared with 50 percent at baseline. The rate of vulnerability among ethnic minority children decreased from 45 percent in 2012 to 22 percent in 2016 compared with a reduction in the rate of vulnerability among Vietnamese-speaking children (nonethnic minority) from 19 percent in 2012 to 9 percent in 2016. The overall rate of vulnerable five-year-old children decreased from 24 percent to 12 percent, which exceeded the low target (22 percent). Target values for all three project development objective indicators under VNEN were exceeded. For the first indicator, number of students participating in VNEN, 446,781 students in grades two to five participated against a target of 400,000 students. Approximately 79 percent (352,956) of these attended 1,143 schools (about 80 percent of the 1,447 schools supported under the project) in 20 priority 1 (most disadvantaged) provinces. The target value of 25 for the second project development objective indicator—number of VNEN learning guides, teacher guides, and Teacher Training Institute guides developed—was exceeded by about 472 percent (143 guides), noting a very low initial target. The target value of 30,000 for the third indicator—number of primary education teachers and education administrators completing VNEN training— was exceeded by approximately 76 percent (52,792). This last result is attributed to the modified cascade training, which allowed more teachers to be trained at school level. The target values for the four VNEN intermediate indicators were met or exceeded, and the project also sparked activity outside of the project boundaries. This was in part attributable to the highly participative and collegiate engagement of relevant stakeholders including teachers, administrators, and technical experts. The qualitative study undertaken as part of the project included classroom observations and the production of teacher videos on best practice teaching that continue to be available. A full suite of materials to support the use of the approach was available for use for grades one to six. In addition, 452,255 nonproject students experienced the VNEN approach in schools that voluntarily adopted the approach. This level of success is indicative of the profile attained by the project in the primary education system (more than 200 articles were published in local newspapers; they were written by teachers, parents, community leaders, and school administrators on the benefits of the VNEN model). xiii A key finding from the impact evaluation carried out under the VNEN project was that the program had a positive effect on the socioemotional skills of children enrolled in program-supported schools. The evaluation found that students in the VNEN program did better than their counterparts in traditional schools and that VNEN students did particularly well at the lower end of the distribution where there is a greater concentration of disadvantaged groups. The evaluation also found that VNEN students perform as well as or better compared with students under the traditional, frontal/didactic regime for Vietnamese language and mathematics, noting that students from VNEN schools already showed a higher mean score at the baseline of the study. IEG ratings for the VNEN and SRPP are elaborated in appendixes A and B. Table S.1 summarizes the ratings. Table S.1. Ratings for Vietnam Education Projects: School Readiness and Escuela Nueva Indicator VNEN SRPP Outcome Satisfactory Satisfactory Bank performance Satisfactory Satisfactory Borrower performance Satisfactory n.a. Monitoring and evaluation Substantial Substantial Risk to development outcome Modest Low Note: SRPP = School Readiness Promotion Project; VNEN = Vietnam Escuela Nueva. Lessons IEG has identified the following lessons from its evaluation of the two operations: • In addition to lending, the World Bank can add value through the transmission of knowledge from experiences and lessons that help shape reforms. This included World Bank economic and sector work, study visits, the introduction of international experts, and the introduction of lessons gleaned from an experienced team that built on the long-standing relationship of mutual trust between MOET and the World Bank. The World Bank’s knowledge capability was a useful complement to the implementation capacity of the education system in Vietnam. For SRPP, the World Bank advocated and helped support the rollout of a systemwide approach through preschool accreditation to improve quality. Since closure, the government has moved to universalize enrollment beyond kindergarten and has further refined the school self- assessment process. VNEN introduced a new pedagogic philosophy into the primary education system. Many teachers have expanded their pedagogical skills and the model laid part of the foundation for the current curriculum reform and associated teacher training. xiv • When significant pedagogical changes are required of teachers, incentives, support, and long-term commitment are needed (probably more than education systems realize). The changes envisioned in both of these operations were neither simple nor easy. Studies show the importance of follow-up support to facilitate the application of in-service training. Within the education system, existing mechanisms provide follow-up support to teachers such as school visits, principal feedback, and peer-to-peer learning at school cluster meetings. Some of these mechanisms were applied under both projects, but VNEN employed them more extensively. For example, during the school year, two to four technical support team meetings and training sessions were conducted for VNEN schools. These meetings and sessions involved classroom observations, interviews with teachers and school management, review of logbooks, and exchange of experiences with principals and teachers. E-learning was used under SRPP for follow-up knowledge. Although it may serve to bridge at least some of the existing gaps in follow-up support, e-learning is unlikely to represent a panacea. Constraints in the implementation of SRPP e-learning modules suggest the need to address these issues. This emphasizes the need for long-term commitment and to leverage existing mechanisms more fully to bring about the kind of quality teaching envisioned. • When scaling up or adopting a systemwide approach, it is important to understand and design this approach in accordance with the decentralized context and challenges faced at the various levels of administration. SRPP required provincial and district officials to provide follow-up support after face- to-face cascade training and to provide technical knowledge to the preschool accreditation and school self-assessment process. Follow-up support to teachers in SRPP and VNEN was implemented more effectively in some provinces than in others. An analysis was conducted (by SRPP) to understand how the decentralized systems operated and to detect capacity constraints. Yet design and implementation did not differentiate resources or technical support to provinces and districts to address differences in capacity. As a result, disadvantaged provinces had much lower percentages of accredited schools. The DLIs set an appropriate incentive for the central level but not for provincial and district levels. • Targeting disadvantaged areas does not translate into targeted efforts for specific vulnerable groups. The self-selection process under VNEN was used to encourage a demonstration effect in locations where ethnic minority and other disadvantaged populations were present. The school selection process under VNEN favored the best schools (for demonstration effect) in locations where ethnic minority and other disadvantaged populations were present but not necessarily schools in which such children were concentrated. The targeting strategy was xv implemented to generate the momentum needed to support the further adoption of the model in all provinces (this resulted in greater voluntary adoption in the wealthiest provinces). Although this approach is sensible, it may have missed opportunities to target even more ethnic minority children. SRPP provided lunch subsidies to disadvantaged families, but all other aspects of the project were applied systemwide. A social assessment detected multiple enrollment barriers for ethnic minorities, such as distance from school or parental knowledge, that were not addressed. The baseline survey substantiated the higher level of vulnerability among ethnic minority children (compared with those who speak Vietnamese). Considering the critical juncture for language development during the early years, targeted action was needed to support learning in the first language and introduction to Vietnamese. • When scaling up, the importance of consultation and communication cannot be underestimated. The literature suggests that scaling up programs requires those promoting the innovation to convince stakeholders of the effectiveness of the intervention. Both of these operations had broad political support. Despite the attention to communicating with stakeholders, particularly during the early stages of VNEN implementation, the scaling up of the model proved challenging. The communication challenge was associated with substantial changes for teachers and divergence from what parents traditionally experienced in school and, more particularly, the expansion of the model to nonproject voluntary schools that did not benefit from the same concentration of support as their project-based counterpart schools. Education is a heavily debated topic in Vietnamese society, suggesting the critical role for sustained and sophisticated communication regarding planned change. Sophie Sirtaine Acting Director Human Development and Economic Management Independent Evaluation Group xvi 1. Background and Context 1.1 Poverty has decreased rapidly in Vietnam over the past two decades; what remains is concentrated among those living in rural areas and ethnic minorities. Poverty declined from 58 percent in 1993 to about 6 percent in 2016.1 Yet ethnic minorities account for 65 percent of people in the poorest welfare decile, an increase from 53 percent in 2006 (World Bank 2012). The northern midlands and mountainous and rural areas have the highest levels of poverty and lowest per capita income.2 1.2 The level of vulnerability among ethnic groups is heterogeneous, since the culture, living conditions, and location differ for each group. Vietnam has 54 ethnic groups. The Kinh, who make up 84 percent of the population (77 million people), are the most populous ethnic group, with 53 ethnic groups making up the remaining 16 percent (15 million people; Mekong Delta Research Institute 2018. Six ethnic groups have populations of 1 million or more: Tay, Thai, Munong, Khmer, Mong, and Nung. The size of each of the remaining ethnic groups ranges from 446 (O Du) to 800,000 (Dao, Hoa) people. The highest levels of poverty are found among the La Hu (86 percent among 11,140 people), Mang (80 percent among 4,364 people), and Lo (75 percent poverty among 4,314 people; Mekong Delta Research Institute 2018). Thus, understanding and tailoring government programming to the differing situations and needs of ethnic groups may be needed. 1.3 The government of Vietnam and the Vietnamese people place a high value on education. The government’s Socio-Economic Development Strategy 2010–20 and the Socio-Economic Development Plan (2016–20) stress the need for human capital investment to increase productivity and capacity.3, 4 The goal of the Socio-Economic Development Plan is to develop people’s skills to support a knowledge-based economy. The government spent 18 percent of its national budget on education in 2013,5 which is higher than the average for high-income countries.6 Parents also invest in education, paying for school lunches, textbooks, and school fees (with subsidies, reductions, and exemptions for poverty or disadvantage). Education in Vietnam 1.4 The public education system has four levels. It consists of: (i) early childhood education (for children 3–5 years old7 in nurseries and kindergartens); (ii) general education, including primary education (grades one to five), lower secondary education (grades six to nine), and upper secondary education (grades 10 to 12); (iii) vocational education, including general education and higher education; and (iv) higher education. Primary education is compulsory, and kindergarten and lower secondary are universal. 1 Private institutions have a limited presence in Vietnam, except for higher education, but public institutions still dominate at this level. 1.5 Education is predominantly managed by the central Ministry of Education and Training (MOET), while provincial and districts implement. MOET creates education decrees that the provinces and districts implement. The central level develops standards, curriculum, and textbooks, and forecasts the human resources needed within the education system. Provinces and districts oversee and make implementation decisions related to preschool, primary, and secondary schools. Provincial capacity varies, with particular challenges in disadvantaged provinces where many of the poor reside (World Bank 2012). 1.6 Enrollments at preprimary, primary, and lower secondary levels of education have risen sharply between 1995 and 2017 (see table 1.1). For example, the number of students enrolled in primary school increased from about 7.5 million in 2014 to 8 million in 2017.8 The number of ethnic minority primary students has remained proportionate to the number of primary students (1.3 million in 2014 and 1.4 million in 2017), at 17 percent of total enrollment. Enrollment has also grown considerably at the upper secondary and tertiary levels. Near–gender parity is present across general education (47 percent female and 53 percent male) levels. Students are required to take an examination to graduate and to move into upper secondary education. Because the exam acts as a pathway, parents consider what is taught at the lower secondary level to be particularly important, as outlined in articles related to VNEN.9 Ethnic minority children are less likely to attend upper secondary schools (World Bank 2012; Gouleta 2009),10 as well as tertiary education. This is likely due to the limited spaces available within the system and academic selectivity to enter upper secondary. As a result, only a few people of ethnic minority background become teachers (Gouleta 2009). Some systems employ bilingual teachers to address the language needs of children who are not proficient in the instructional language. 2 Table 1.1. Vietnam’s Gross Enrollment by Level and Year (percent) Level 1995 2002 2007 2017 Preprimary 33 41 67 95 Primary 113 102 99 108 Lower secondary 76 86 87 99 Upper secondary 32 63 n.a. 73 Tertiary 3 9 18 28 Source: United Nations Educational, Scientific, and Cultural Organization; McAleavy, Ha, and Fitzpatrick 2018 (upper secondary data, 2002). n.a. = not available. 1.7 Several policies have been implemented to improve the quality of general education since the early 2000s. The country has made efforts to professionalize teachers through the introduction of standards and awards for excellence (with a small financial bonus) and has implemented in-service training programs. Teachers are also required to engage in a certain amount of self-learning each year. The Fundamental School Quality Levels set minimum quality standards for primary schools that include a process of school self-assessment and validation. MOET (with the support of the Russia Education Aid for Development Trust Fund) has implemented standardized assessments and has made improvements to the infrastructure of schools. Moreover, MOET is universalizing full-day programming at all levels of education and (with World Bank support) is reforming the general education curriculum. The new curriculum will be competency based and its rollout will involve a significant in-service training effort as well as an expansion in the number and range of textbooks, replacing the existing single-textbook approach (World Bank 2016a). 1.8 The preservice preparation of teachers has suffered from persistent weaknesses. At the preschool level, teachers can currently practice without completing college, although a college-level qualification will be required of new entrants as part of the reform effort, as is the case for general education. The preservice training for preschool teachers was reported as overly theoretical, lacking an emphasis on a play- based approach. In relation to primary school teachers, the preservice training is also described as too theoretical, focused on content knowledge and a didactic approach. MOET has invested in in-service training over the years to upgrade the knowledge, skills, and pedagogical practices of its teachers. This suggests misalignment between the content of preservice and in-service training. Coordination is also lacking among preservice actors (World Bank 2016a). MOET (with World Bank support) is developing a strategy to realign teacher education to the current curriculum reform of general education (World Bank 2016a). 3 1.9 Student achievement has improved in the past decade. A national assessment of fifth graders showed improvement in math and reading scores between 2001 and 2007 (World Bank 2011a). Vietnam’s average Programme for International Student Assessment score in math, science, and reading is above the average for Organisation for Economic Co-operation and Development countries; however, its average math and reading scores declined slightly in 2015 (OECD 2018).11 Analysis has pointed to potential explanatory factors such as parental contributions to education and considerable consistency across government schools (McAleavy, Ha, and Fitzpatrick 2018). 1.10 Yet improving learning outcomes for disadvantaged children (including ethnic minority children) has remained a challenge for the education system. Education for All in Vietnam (World Bank 2011a) showed that the disadvantaged children who were most at risk for learning outcomes were ethnic minorities and children from poor households. The Young Lives Survey (2011–12) continued to show lower performance of ethnic minority students, although there was a narrowing in the gap in math (27 points) and reading (56 points) scores between Kinh (majority) students and ethnic minority students. The scores for ethnic minorities rose from 434 to 497 (an increase of 63 points) in math; the scores rose in Vietnamese language also rose from 420 to 483 (an increase of 63 points; McAleavy, Ha, and Fitzpatrick 2018). However, the language of instruction remains an obstacle for ethnic minority children. There have been pilots with dual language with mother-tongue instruction in three provinces (Lào Cai, Gia Lai, and Trà Vinh) in preschools or primary schools with successful outcomes (UNICEF 2015). The placement of bilingual teaching assistants in classrooms is another strategy used to support ethnic minority students.12 However, these pilots have not been mainstreamed into the system. 2. Design and Preparation: Strengths and Weaknesses 2.1 The World Bank approved two projects in the education sector: the School Readiness and Promotion Project (SRPP) in 2013 and the Vietnam Escuela Nueva Project (VNEN) in 2012 to support the government’s efforts to improve the quality of education. 2.2 Objectives remained consistent across the life of both operations. The development objective of SRPP was “to raise school readiness for five-year-old children, in particular for those most vulnerable to not succeeding in a school environment, through supporting selected elements of Vietnam’s Early Childhood Education (ECE) program” (World Bank 2013a, 4; World Bank 2013b, ii). The development objective of VNEN was “to introduce and use new teaching and learning practices in the classroom 4 targeting the most disadvantaged groups of primary students” (World Bank 2012, 3). The objectives and key performance indicators and targets remained unchanged across these two operations. 2.3 SRPP had two components. The first component reimbursed the government for lunch subsidies for disadvantaged children between the ages of three and five years old and for salaries, including social and health insurance contributions. The project was to focus its resources and attention on preschools with low capacity to deliver quality services for the most vulnerable children in line with the new ECE curriculum. The second component financed technical assistance resources and operating costs to strengthen planning and reporting systems for ECE; prepare a new professional development program; develop ECE policy and regulation, including the development and piloting in selected locations of a new community-based Early Childhood Development Center for children from birth to age 3; verify performance audits of disbursement-linked indicator (DLI) reporting; and manage, monitor, and evaluate the project. 2.4 VNEN had four components. The first component developed pedagogical material for third through fifth grade, since materials for first and second grade had already been developed. The second component created training for teachers and school managers. The third component financed school grants to support VNEN implementation and school lunch subsidies for satellite campuses in disadvantaged areas. The fourth component focused on management at the central and provincial levels to roll out VNEN. It also financed videos of classrooms, school census data, and an impact evaluation. Overall Design and Preparation 2.5 The Independent Evaluation Group (IEG) developed a theory of change for SRPP (figure 2.1Error! Reference source not found.) that reflects realized inputs and outputs and likely outcomes and impact. This logical chain requires several assumptions to have an effect on teaching practices, which are required for longer-term outcomes such as improved student attainment and learning. Some of the critical assumptions include the following: (i) lunch subsidies were the primary obstacle to enrollment; (ii) targeted interventions were provided to support dual-language acquisition for young ethnic minority children; (iii) effective training and follow-up support were consistently employed; (iv) parents supported the development of their children; and (v) self- assessment, validation, and accreditation processes emphasize instructional practices and learning activities more than infrastructure. 5 Figure 2.1. School Readiness and Promotion Project Theory of Change Source: Independent Evaluation Group (adapted from the project appraisal document and the Implementation Completion and Results Report). Note: BOET = Bureau of Education and Training; ECE = early childhood education; DOET = Departments of Education and Training; EM =ethnic minority; MOET = Ministry of Education and Training. 2.6 IEG developed a theory of change for VNEN (figure 2.2) that reflects realized inputs and outputs and likely outcomes and impact. This logical chain also requires several assumptions to have an effect on teaching practices and subsequent outcomes and impact. Some of the critical assumptions include the following: (i) the VNEN model is not resisted; (ii) the VNEN model is an ideal vector to promote the educational reform aspirations; and (iii) continued government commitment is vital. 6 Figure 2.2. Vietnam Escuela Nueva Theory of Change Source: Independent Evaluation Group (adapted from the project appraisal document and the Implementation Completion and Results Report). Note: TTI = Teacher Training Institute; VNEN = Vietnam Escuela Nueva. 2.7 Both operational designs stemmed from ongoing policy dialogue between the World Bank and MOET that enhanced the eventual reform. The two parties had long-standing mutual trust that gave them the confidence to tackle complex reforms. For example, in the past decade the World Bank has supported School Education Quality Assurance, Renovations of General Education, and Enhancing Teacher Education.13 In the case of VNEN, the World Bank established a regional workshop sharing best practices in rural education and arranged a study tour to the Escuela Nueva Foundation in Colombia for senior MOET officials to learn firsthand about the program. Policy dialogue facilitated the government’s broader reform program (Resolution 29–NQ/TW of Party Plenum XI 2013) to improve education quality and move the education system from its focus on rote learning to an emphasis on critical thinking, problem solving, and communication skills. In the case of SRPP, policy dialogue resulted from multiple analytical works, technical assistance, and an early childhood development policy conference that promoted a system approach to ECE that simultaneously focused on 7 quality and access. In both operations, the aims of the World Bank and MOET aligned. Through policy dialogue, the World Bank added technical knowledge that improved what was ultimately designed. 2.8 Preparation spanned a number of years. World Bank management provided the space for task team leaders to take an extended period for preparation (particularly for SRPP). Preparation for SRPP spanned more than four years from project initiation (June 2009) to the internal decision review meeting (October 2012) and eventual approval by the Board (October 2013). This time was well used and laid a solid foundation for developing the DLIs and system approach for ECE. The VNEN also benefited from substantial lead time involving field trips to Colombia, a pilot during a previous operation (Primary Education for Disadvantaged Children Project), and technical assistance from international experts. The generous lead time afforded both projects likely contributed to the overall smooth implementation (see below) in both cases. 2.9 Project design drew on prior lessons and international experience. Child- centered learning and teacher-facilitated (rather than teacher-directed) learning were supported by both operations. VNEN used lessons from the pilot, which included the need for a “comprehensive package” (special-purpose didactic materials for students, training for teachers, and continued school-level support of trained teachers) to ensure effectiveness and “dedicated knowledge management” (that is, the need to produce and disseminate knowledge about the new pedagogical methods). SRPP learned lessons from Primary Education for Disadvantaged Children about the application of school accreditation to the preschool level. International experience also pointed to the need to regularly assess outcomes and use a curriculum that focuses on child development. Both operations contained aspects of student assessment. 2.10 However, the design did not draw on some important evidence, so some inadequately emphasized aspects created issues with sustainability or reduced potential impact. One lesson noted in the appraisal document and quality enhancement review for SRPP but not adequately emphasized in the operational design of the project was about engaging with parents. Parental engagement is linked to immediate improvements in children’s development and later outcomes, such as earnings and achievement.14 Stakeholders reported occasional meetings between teachers and parents at school through a communication strategy developed by the project. The baseline Early Development Index found more vulnerability among ethnic minority than Vietnamese-speaking children (World Bank 2017a). Engaging ethnic minority children’s parents would have had a significant effect on improving school readiness. The early years are a critical time for language development and building Vietnamese language skills in ethnic minority children (so that they can become prepared for primary school). 8 The DLIs did not support this aspect.15 Additional interventions for schools to support the language development of ethnic minority children were absent from both operations. In addition, there was an issue with how the VNEN program fit within the education system and broader reform efforts. For example, VNEN had to use the existing curriculum and text even though it made dramatic changes to the way teachers would instruct. There was considerable discussion on this topic with MOET and it was decided to push for pedagogical changes in the absence of curriculum reform. As affirmed (in hindsight) by key stakeholders, VNEN training may have been misaligned in terms of logical flow and sequencing relative to the broader reform effort. 2.11 The design of the SRPP school self-assessment lacked quality indicators consistent with international best practice. The self-assessment system was built on the Quality Standards for Preschool (and aligned with the previously developed ECE curriculum),16 with 29 indicators for assessment by school managers. Many of the indicators related to the condition of school infrastructure (which, according to stakeholder interviews, was the main challenge associated with progression through the assessment levels for schools) as well as the qualifications of teachers and school managers. An important indicator in the assessment was the regular monitoring and evaluation (M&E) of children. However, the self-assessment did not examine quality- enhancing aspects such as curriculum (what is taught to advance children’s development), the relationship with children (how adults foster children’s self- regulation and responsibility), or teaching approach (how teachers use various instructional approaches).17 These aspects may have been beyond the capacity of school managers and district and provincial staff to evaluate, since respondents reported the desire to keep indicators simple. However, a simple instrument (such as the Likert scale) could have addressed this issue. 2.12 The project designs appropriately used simplified financial management for both operations. In the case of VNEN, the effect of this design feature was to mitigate some of the implementation risks associated with the three-year implementation period. The operation also included a modified financial management arrangement so that grant funds would be sent from MOET to a provincial (commercial) bank account and then directly to school bank accounts. Schools were responsible for expenditure reporting. Project design also included retroactive financing—to mitigate implementation risks associated with the three-year implementation—which allowed Project Management Unit (PMU) to begin implementation activities before effectiveness. SRPP provided on- budget financing through DLIs for four interventions: (i) lunch subsidies for disadvantaged children, (ii) training, (iii) teachers’ salaries, and (iv) school accreditation/self-assessment. Retroactive reimbursement of eligible expenditures was 9 provided based on completion and verification of DLIs. All of these design features were appropriate and contributed to the successful implementation of both projects. Teacher Training Design 2.13 Teacher training was an important aspect of both operations, with variations in the design of face-to-face training. For cost reasons, a cascade model was implemented in both operations. A concern with cascade training is the lack of consistency in the content, as training is delivered from the experts to trainees who become trainers in subsequent phases (Orr et al. 2013; Karalis 2016). Factors that mitigate weaknesses with the cascade approach include ensuring that training is based on teachers’ needs, selecting experts and the first phase of trainers carefully, monitoring each phase of training, and providing comprehensive educational materials (Karalis 2016). 2.14 Both operations used a cascade approach for teacher training, with some differences in design. Both operations modified the cascade approach and reduced the layers of training. VNEN used central, provincial, and practicing VNEN classroom teachers as master trainers for both provincial- and school-level training.18 VNEN trainers conducted courses at the provincial level two weeks after trainees received training materials that allowed them to come prepared for the training. In the case of SRPP, expert trainers trained the provincial delegates in Hanoi—core teachers or representatives from the directors of Departments of Education and Training—who then were responsible for delivering the training in the districts and schools. 2.15 VNEN training more actively engaged teachers than did preschool training. VNEN training put teachers in the role of students so that they could experience the model’s methodology and thereby learn how to become facilitators and advisers of children’s learning rather than just transmitters of facts. Research shows the importance of active methods of training reflective of how adults learn (box 2.1). The SRPP training was described as practical and helpful; it involved discussion and some video examples. Its design lacked the level of modeling and experiential learning contained in VNEN workshops, however. Given the gaps in teachers’ knowledge, from an overly theoretical preservice training, the training would not be adequate to produce the types of pedagogical (behavioral) changes envisioned. Both operations also provided training to local and regional school administrators, a key part of ensuring coherence with the feedback provided by school managers. 10 Box 2.1. Characteristics of Effective In-Service Teacher Training Research converges on the following features of in-service training, which are associated with changes in teachers’ instructional practices and knowledge (and in some cases also improve student learning): (i) active learning of training participants (models the practice); (ii) content- focused; (iii) enabling environment provided (by school and education system); (iv) coherence with curriculum; (v) providing opportunities for teacher collaboration and collegiality; (vi) longer duration; and (vii) follow-up support through coaching and feedback to promote reflection. For example, participants in one math training program made short oral presentations to explain how they solved problems, received feedback on their solutions, engaged in discussions, and planned lessons that they would teach during the follow-up coaching visits (Garret et al. 2010). This 50-hour training program positively affected teachers’ instructional practice (such as the activities used to elicit student thinking). Training should be designed with attention to teachers’ needs and be reflective of context, school, and capability of teachers (for example, a teacher who is unskilled or unprepared; a teacher who teaches in a mechanical manner; a teacher who has a routine repertoire of skills; or a teacher who is able to adeptly select and alter the repertoire of skills). Sources: Akyeampong et al. 2013; Carrillo, Van den Brink, and Groot 2016; Darling-Hammond 2005; Darling- Hammond et al. 2009; Darling-Hammond et al. 2017; Darling-Hammond and Richardson 2009; Ganimian and Murnane 2016; Garret et al. 2001; Garret et al. 2010; Jacob, Hill, and Corey 2017; Luneta 2012; Popova et al. 2018; and Villegas-Reimers 2003. 2.16 Neither operation differentiated training based on grade level or the age of students, which is difficult to justify for SRPP. Teachers of the various grade (grades three to five under VNEN) or age levels (three, four, and five years old under SRPP) received the same training, with discussion about developmental differences. Similarly, training did not differ based on teacher capacity or competence. Considering the newness of the VNEN model and practices, all teachers were likely at a similar starting point. For preschool teachers, the lack of substantial differentiation in content is striking given the large developmental differences in learning between children aged three and five years old, as well as the type of learning activities that teachers would need to engage students within that developmentally critical age span. It is also striking considering the gaps in preschool teachers’ knowledge from an overly theoretical preservice training, according to stakeholders. Preparing for Scaling Up of Teacher Training 2.17 Each operation planned differently for the scaling up of teacher training. The scaling up of the Escuela Nueva model went from a pilot (during a previous operation implemented in 2010–11)19 to implementation in all 63 provinces but with an emphasis 11 in 20 provinces with the most disadvantaged populations. This approach served two purposes: (i) it allowed for the selection of schools with ethnic minority and disadvantaged children (priority 1 provinces) and (ii) it allowed for the selection of schools that would also provide a demonstration effect (priority 2 and 3 provinces). It was hoped that the second group would facilitate a broader system spread and voluntary adoption of the new approach by neighboring schools. The planning process understood a need for focused scale to develop a critical mass of change agents. In contrast to VNEN, SRPP was designed to provide nationwide coverage to preschool teachers, which led to trade-offs with the depth of training delivered. 2.18 Incentives were used differently to motivate the scaling-up process and to encourage teachers to apply the training. For example, the decision by individual schools to enter into the competitive VNEN selection process (schools had to apply and qualify for the program) was indicative of their motivation but also of the incentives offered: status (being part of a vanguard program), professional development opportunities, and additional support and resources. Incentives are often needed to change behavior (World Bank 2015), which in this case was a significant pedagogical shift. As will be discussed in the implementation section, VNEN also used other incentives. Incentives were absent in the planning process for preschool teachers, school managers, and districts and provinces in SRPP, however. For example, DLIs provided an incentive for MOET to ensure delivery and to be reimbursed based on key indicators (percentage of teachers trained, self-assessment, and accredited), but not provinces, districts, and schools. For example, provinces and districts were asked to do more training and school accreditation using their existing budgets. The planning process identified varying provincial capacity but failed to fully understand how the decentralized system operated, identify challenges, or put relevant supports in place for school self-assessments, accreditation, and training. The project provided face-to-face training on the accreditation process in disadvantaged districts but not targeted technical or financial assistance. Training provided existing intrinsic incentives to teachers such as recognition and professional pride. It reinforced the role of core teachers as instructional leaders, although this role took additional time. Monitoring and Evaluation Design 2.19 M&E shared similar characteristics. The results frameworks were largely composed of simple, numerical output indicators, although SRPP contained two outcome measures within its results framework (enrollment rate and decreased vulnerability in children). However, additional sources of evidence complemented the indicators. For VNEN, basic data were supplemented by classroom observations and an impact evaluation. For SRPP, three rounds of the Early Development Index (EDI) survey 12 were implemented, one each in 2012, 2014, and 2016. However, targets seemed to have been set based on what was known to be achievable, suggesting low aspirations. While this reflected some realism, a stretch element was missing in both operations. For example, the targeted number of VNEN learning guides, teacher guides, and Teacher Training Institute guides was ultimately exceeded by 472 percent.20 In relation to DLIs in SRPP, there were some “quick wins” that helped reimburse the government for work previously done to develop quality standards and professional development or produce a baseline enrollment report; this likely built confidence in the new instrument. Other DLIs were meant to disburse over the course of the operation, but by early 2016 approximately $85 million (85 percent) had already been disbursed. 2.20 The results frameworks were not designed to help MOET assess the effectiveness of its policies or reform efforts. The results framework for SRPP would not provide MOET with evidence in relation to school self-assessment, lunch subsidies, or training. The results framework did not specify disaggregated indicators consistent with the focus on vulnerability, even though MOET’s reporting systems regularly collect disaggregated data by province, gender, ethnic minority, and, recently, disability. An impact evaluation was included in the initial design of VNEN. 3. Implementation and Supervision: Strengths and Weaknesses Overall Implementation and Supervision 3.1 The implementation of both projects ran smoothly overall. Both projects rapidly disbursed, were delivered within planned periods with no overruns or extensions and were fully disbursed at project close. The solid implementation of VNEN and SRPP was the result of committed staff and a well-working system. Implementation was also facilitated by a strong working relationship between the World Bank and MOET, backed by a history of working together in support of education reform. 3.2 Smooth implementation was facilitated for both projects by the use of project management and other PMU staff who were officers of MOET. The staffing of PMU with MOET officials was realized for the first time under the VNEN project, an arrangement that also applied for SRPP. MOET management and staff occupied dual roles, absorbing project management duties into their existing roles. Some personnel were involved in that capacity for both projects (financial management and procurement). The symbiotic relationship between MOET and the PMUs meant that when challenges arose they were dealt with effectively. The integrated relationship also facilitated more immediate problem solving (all relevant personnel were working within their “own” system). The 13 efficacy of the integrated approach was evident, for example, in the successful operation of DLIs under SRPP. This modality had not previously been part of MOET’s engagement with the World Bank. To set it up required significant interaction between MOET and other authorities (such as the Ministry of Finance), and managing it required significant efforts on the part of PMU and provincial authorities. 3.3 Financial management did not progress smoothly in the early stages of either project, but issues were resolved.21 Project effectiveness for VNEN was delayed for six months due to delays in the signing of the grant agreement as well as the time taken to confirm the designated account agreements with the commercial bank (into which the Global Partnership for Education funds would be transferred) and the memoranda of understanding regarding the transfer of funds between the PMU and provincial PMUs. The intended reimbursement of salaries for preschool teachers was dropped from SRPP because the amounts in question could not be isolated from other salary-related items in budget codes. World Bank personnel interviewed as part of the IEG mission were of the view that the DLI approach was unwieldy in the context of supporting SRPP.22 However, once these early problems were addressed, the system worked well. 23 3.4 Key client informants were uniformly positive about the World Bank’s role in both projects. Client representatives emphasized that the use of development aid for education projects was not just about the funding but also about bringing in World Bank knowledge and experience as well as international experts and advisers who could contribute to the realization of the education reform strategy. Client representatives cited a close working relationship with World Bank staff and the key role of those staff in helping to steer system innovations. Valuable exposure to international influence continued throughout the life cycle of both projects. For example, under SRPP, international experts accompanied supervision missions and provided technical advice related to pre-primary education, in-service training, and preschool accreditation. Based on a Colombian innovation, VNEN involved the injection of new ideas into the education system in Vietnam on an ongoing basis with the support of international experts who worked at all levels of the system as part of the project rollout. Overall, the type and level of input received directly from World Bank staff and through international experts introduced by the World Bank were appreciated. 3.5 Aide-mémoire and back-to-office reports for the VNEN project provide a close narrative of the evolution of the project and the manner in which various issues were addressed. For example, the aide-mémoire for the Second Joint Implementation Support Mission (October 23–November 22, 2013)—in conjunction with the United Nations Educational, Scientific, and Cultural Organization—acknowledges significant progress in developing learning guides and teaching resources but recommends 14 preparation of a set of materials to support parents and communities. This document suggests the need to ensure flexibility in school level application of new materials. The document also notes the launch of the external VNEN website on October 30, 2013, and recommends that high priority be given to ensure stakeholders are aware of the website and that they encourage teachers and school managers to use the online content. World Bank advice and recommendations demonstrate an understanding of the need to involve and inform stakeholders at all levels as well as an awareness of the importance of providing follow-up to face-to-face training.24 3.6 Reporting was candid in relation to challenges the project faced. There were six World Bank review and implementation support missions for the VNEN project— three of which were jointly undertaken with United Nations Educational, Scientific, and Cultural Organization—including a midterm review. The first Implementation and Status Results Report (April 2013) stated that “the project has suffered on consequence of the long delay between start of the project activities on July 1, 2012, and the first disbursement of $13 million on February 5, 2013 (a period of 7 months)” (World Bank 2013c, 1). The World Bank team consistently reported on financial management and procurement matters and, as attested to by interviews conducted for this assessment, provided valuable advice and support to ensure smooth implementation of what was an ambitious scaling-up effort that was attempted in a short period of time (three years). 3.7 Under SRPP, the World Bank team provided active implementation support. As they did for VNEN, supervision reports and aide-mémoire provided clear and candid assessments. The documentation shows that the World Bank team was focused on monitoring the development impact, not just the implementation of activities. For example, the team encouraged MOET to plan for additional targeted actions for disadvantaged and underperforming provinces in relation to school accreditation and teacher training beyond the project. The team also continued to dialogue with key actors in MOET about early childhood development policy. These discussions supported the government’s updated Early Childhood Education Strategy (2016–25). This strategy expanded its focus to include children from birth to five years old. The World Bank team encouraged the government to present findings from the EDI interim reports at an early childhood development conference and engage other donors who support early childhood development. 3.8 The political economy was generally positive to the reforms pursued by the VNEN and SRPP projects, and this positivity underpinned and assisted project implementation. The favorable political economy supported the close alignment between the objectives of the VNEN and SRPP and the overall thrust of policy and reform efforts in the education sector. 15 3.9 VNEN targeted the most disadvantaged groups—specifically, ethnic minority children—by emphasizing disadvantaged provinces. The project appraisal document states that “a priority of the operation is helping ethnic minority children develop their Vietnamese language skills, so they can benefit from VNEN model” (World Bank 2012, 15). In practice, VNEN targeted (although not exclusively) disadvantaged provinces. Schools were invited to apply to participate in the project with the deliberate intent of creating a demonstration effect that would encourage further uptake of the VNEN approach. Hence, priority was given to schools in the most disadvantaged provinces rather than the most disadvantaged students or the most disadvantaged schools. This is not to say, however, that ethnic minority children did not benefit from the project, since a high concentration of them resided in targeted provinces.25 3.10 The program was also introduced, at a lesser scale, to priority 2 and 3 provinces in the expectation that it would be adopted by other schools nationwide.26 This approach was taken to generate the best possible momentum to support the VNEN innovation and to ensure that it had been implemented, at least to some extent, in all provinces. The decision to introduce VNEN outside of disadvantaged areas was a modification specific to Vietnam—the original Escuela Nueva in Colombia operated only in disadvantaged areas. One consequence of this decision was that the program was delivered in cities where there were a greater number of students in classes, making it more difficult for teachers to meaningfully implement the model. The decision to spread the program across all provinces with a view to engendering a demonstration effect appears to have resulted in greater levels of voluntary adoption in the wealthiest provinces—almost half of the 452,255 students reached through the voluntary schools attended schools in wealthier regions. 3.11 SRPP targeted the most vulnerable students, defined as “those whose cognitive, socio-emotional and physical development is most lagging” but was applied systemwide, with some resulting tension (World Bank 2013b). There was tension between the project’s dual foci—systemwide coverage and low-capacity preschools—to deliver quality services for the most vulnerable children. The planning process identified varying provincial capacity but failed to identify challenges and put relevant supports in place for school self-assessments, accreditation, and training. In addition, no specific activity supported the language development of ethnic minority children. In contrast, relevant funding provided by the World Bank (lunch subsidy) was directly associated with children from disadvantaged groups and, based on mission feedback, did serve to enhance the enrollment of disadvantaged children in preschool. 3.12 Four ECE centers for children from birth to age 3 were developed; however, based on stakeholder feedback, the centers have had varied levels of effectiveness. The 16 four centers—Bac Giang, Da Nang, Dak Lak, and Lào Cai—were intended to address ECE shortcomings in disadvantaged areas, providing opportunities for development and more choices for parents with early childhood care and education. These came into being late in the project cycle. The IEG mission was informed that there was limited demand for the services in two centers visited during the mission (Dak Lak and Lo Cai). The centers were put in place without an assessment to identify needs of children and parents. Implementation of Teacher Training 3.13 Follow-up support after the initial training was strongly emphasized by the VNEN teacher project. Follow-up to the initial training is needed because it takes time for teachers to work with colleagues or mentors to develop and refine their practice (Darling-Hammond 2005). VNEN had multiple modes of follow-up support, including study groups, school visits, and learning circles. Complementary resources help teachers implement the training content (Popova, et al. 2018). Lessons, guides, and videos were provided with VNEN, but not SRPP. It may be that SRPP did not provide complementary materials because the new ECE curriculum had been rolled out previously. VNEN developed 143 types of learning guides—including student learning guides, teacher guides, and Teacher Training Institute guides—for various subjects to support delivery of the program. VNEN project schools also received grants to facilitate the application of the new approach in classrooms. These grants were managed by the school based on an agreed upon list such as school equipment, Vietnamese language summer classes, training activities, teaching assistants, classroom furniture, and teaching and learning aids. VNEN also developed a website in 2013 to post and share lessons, videos, and examples. Postproject, MOET absorbed the site to become a functional part of its portal. The VNEN Impact Evaluation states that, whereas initial training was delivered faithfully and was “conducted in the form of a VNEN classroom, complete with groups and group leaders and warm-up games” (131), the observed high level of heterogeneity in implementation is likely associated with the low incidence in the ongoing implementation of follow-up cluster training.27 3.14 SRPP training emphasized breath and coverage. The core idea behind SRPP was to reach all ECE teachers with training, which it achieved.28 The emphasis was on getting the training implemented with more limited attention to depth and consistency. The World Bank made recommendations to improve the follow-up support. During supervision missions, teachers provided examples of how they applied training in their classrooms, but documents, stakeholder reports, and observations during the IEG mission suggested that preschool teachers needed far more hands-on support. This would be expected given the complexity of the pedagogical tasks (including the ability to develop children’s perspectives, praise and comfort, and develop targeted 17 lesson/activity plans to promote communication skills, social and emotional learning, and executive function; OECD 2012). 3.15 Advanced e-learning modules were developed under SRPP to provide follow-up information after the initial training, but their effectiveness was limited by various challenges. At the end of the operation, MOET decided to launch the e-learning format to reduce costs. Distance formats can improve undertrained teachers’ knowledge when there is reliable access to practical self-study material (Orr et al. 2013). Ten advanced modules were developed and implemented. Administrative officials at each level could track at any time how many teachers used or completed modules. This format was intended to create a forum for teachers to exchange views. Various challenges limited the effectiveness of this modality, including limited broadband access in remote areas, limited information technology skills of teachers, the absence of computer equipment in schools, a lack of dedicated time for teachers to complete the module, and a lack of practical hands-on application or support when questions arose. There were also issues with the platform. For example, completed modules could not be accessed later and content in the modules could not be subsequently updated. 3.16 Departments of Education and Training officials also provided school-based follow-up support for preschool teachers, but this was not uniformly applied. Essentially, teachers were required to engage in self-learning. Occasional peer learning and teacher meetings are a feature in the education system facilitated by district officials (preschool teachers engage in discussion with other teachers). In some instances, teachers visited other schools to observe lessons. The World Bank team made suggestions for follow-up support through feedback and mentors. This has resources implications (or may require incentives) for core teachers or district ECE staff to provide on-the-job support and needs to be supported with existing provincial resources, both human and financial. Implementation of Scaling Up 3.17 Notwithstanding the favorable climate for project implementation, the VNEN model encountered some resistance. First, the program marked a significant departure from the traditional approach to basic education in Vietnam, which is centered on a single curriculum presented through a single textbook for each subject. VNEN produced student workbooks and guides (for teachers and students) to facilitate a new form of engagement between teachers and pupils in the classroom. It promoted a child-centered approach to pedagogy that required teachers to fundamentally alter their teaching practice (moving from a didactic, “chalk-and-talk” model to a more facilitative, child-led approach). The new approach also introduced new ideas (critical analysis, self- learning) and modes of organization (student government) in the classroom. Whereas 18 support from MOET for this level of innovation was robust, support at the provincial level was not uniform (although there were many enthusiastic adopters) because of the devolved structure that applies within the system in Vietnam.29 The new approach encountered some resistance from both teachers and parents who were not convinced of its merits and who feared for the educational performance (with implications for exams and for upper secondary education) of the children exposed to VNEN, according to English-language media articles. 3.18 The expansion of the VNEN model to schools outside of the project (voluntary schools at primary and lower secondary level) ultimately diluted the overall effort. Voluntary schools generated some dissatisfaction that led to negative media coverage. Issues noted included unequal capability and inadequate infrastructure to implement with large classes. The latter is notable given that the original Escuela Nueva in Colombia focused on disadvantaged rural areas in that country before being introduced into urban areas, whereas the Vietnam model was deliberately targeted more broadly from the outset with the hope of broader adoption. The VNEN pilot under Primary Education for Disadvantaged Children had generated interest from nonparticipating schools—the pilot schools had a certain status and the pilot VNEN had generated considerable energy that attracted attention, and some of these schools voluntarily adopted the relevant pedagogical approach without being part of the pilot. Hence, MOET anticipated voluntary adoption when the larger-scale project was being rolled out. In fact, the placement of at least one VNEN school in every province was intended to encourage voluntary uptake by other schools.30 However, the project fell victim to its own success in that regard; in fact, ultimately there were more voluntary adopters than “project schools” (see the Results section). Although teachers from nonproject schools that voluntarily adopted the VNEN approach could avail themselves of training and could benefit from interaction with project-supported schools, neither the schools nor the teachers received the same support as project schools (for example, grants for equipment and advice from international experts). Parents of children at schools that opted into the practice also had to purchase textbooks, which were free for project schools. A number of lower secondary schools (1,214) were encouraged and opted to adopt the pedagogical approach. This proved to be even more problematic given a lack of “fit”—the Escuela Nueva model was designed for and typically applied to primary schools—and associated disgruntlement among teachers, students, and parents. Key informants suggested that this imbalance most likely contributed to some of the negative publicity that attached itself to the VNEN model toward the end of the project cycle. 3.19 Consultation with stakeholders, particularly those within the VNEN project, was extensive, which is an important condition for scaling up. The IEG mission 19 established that, typically, provincial authorities received guidance from the MOET PMU. Provincial officials supplemented this documentation with guidance at the district and school levels. On that basis, individual schools decided whether they would apply to become part of the VNEN project. At the start of the school year, successful schools met with parents to inform them of the new approach and its implications. IEG was informed that school managers needed to have a deep understanding of the model to support implementation and that ongoing support from the People’s Committee and the Departments of Education and Training was important in that regard. Participating schools also supported each other—for example, seven participating schools in the city of Buon Ma Thuot in the Dak Lak region regularly interacted (as a cluster) with each other. The VNEN communication effort also involved extensive outreach. For example, by June 2014, 100 articles had been written about the program in local newspapers, rising to 210 articles published by December 2014. Ultimately, all articles written about the program were collated in a book published by the project. The VNEN external website was launched on October 30, 2013, and was visited 3,646,844 times by project close. Because SRPP involved a mandatory rollout of a national-level program, communication arrangements were less elaborate than those in place for the VNEN— there was no need to “sell” what was being done. 3.20 The quality of system-generated feedback was less than optimal for both 31 projects. For example, VNEN received negative press that was associated, at least in part, with the lack of supports for voluntary adopters, which included inadequate communication with parents of students attending these schools. More effective communication and consultation might have identified that negativity before it became public and helped address the issues it raised. Ultimately, the negative press affected the potential longevity of the model and the influence of the approach in the overall renovation-of-education effort. In the case of SRPP, engagement with parents was limited. Stakeholders reported a few meetings between teachers and parents at school. The lack of active parental involvement potentially limited the value of the changes being made in the classroom, and the lack of a communication channel with parents deprived the system of potentially valuable feedback regarding child development that could have been used to improve ongoing education provision (including aspects of teacher training). Implementation of Monitoring and Evaluation 3.21 The data collection system for the education sector in Vietnam is extensive and robust and was used to underpin targeting and subsequent monitoring for both projects.32 For example, VNEN project preparation and implementation was facilitated by the availability of high-quality census data for primary schools (the District Fundamental School Quality Level Audit), a series that dated back to 2004/05. School 20 principals completed the census and their entries were verified by the district and provincial education officials based on a seven-step process. The survey developed a Fundamental School Quality Level Input Index consisting of 47 indicators that measured five main dimensions of school quality (including school organization and management, teaching staff, and infrastructure). M&E for the project would require modification of the District Fundamental School Quality Level Audit database33 to include variables specific to VNEN such as seating arrangements, availability of learning guides, and teacher training. It also contained data on the number of ethnic minority children needing support in the Vietnamese language and the number of ethnic minority children who have parity in learning levels with Kinh (the majority ethnic group) children. The Education Quality Management System was also used to produce education statistics for the whole system, including data on learning and teaching facilities, as well as statistics relevant to the VNEN model.34 In the case of the SRPP, a number of respondents said that meeting the data reporting requirements under SRPP could be onerous, although reporting was strong. 3.22 Project monitoring was well managed for both projects, although the use of output-type indicators required relatively simple collection techniques. Under SRPP, data were collected regularly and shared with the World Bank. Monitoring of SRPP DLIs was undertaken by the Vietnam Institute of Educational Sciences, which conducted verification exercise for 10 provinces several times between 2013 and 2016. These audits found only minor inconsistencies and errors in reporting. Implementation of the EDI was supervised by the MOET PMU and implemented by the Departments of Education and Training. Three rounds of the EDI were conducted (one each in 2012, 2014, and 2016) and issues arising with the second round were corrected following World Bank recommendations to the PMU regarding enhanced collection of data.35 Analysis of the EDI instrument provided policymakers with evidence of continued vulnerabilities and improvements in school readiness. Data have also been used by MOET and the government to help justify continued investment in young children. 3.23 Finally, an impact evaluation was undertaken under the VNEN project. By comparing results from randomly chosen VNEN schools and traditional schools (the counterfactual), the evaluation intended to measure the extent to which students acquired both cognitive and noncognitive skills. The evaluation was intended to span the project life cycle; however, it did not start on time due to funding issues. Ultimately, the evaluation started when it was agreed that project funds would be used to support the evaluation process and funds secured from Dubai Cares (the first such funding for Vietnam) would finance the analysis of the impact evaluation. By that stage, the VNEN program had already been in operation for a year. Consequently, it was not possible to undertake the planned random assignment of the program; instead, a propensity score– 21 matching method was used to determine the randomized sample of treatment and control group schools to be followed for two years. Data collection for the quantitative survey was done by field investigators who were trained to use handheld electronic devices to record responses. The impact evaluation conducted survey rounds over three school years (2013/14, 2014/15, and 2015/16), with three comprehensive sets of data collected on five stakeholder groups, consisting of students, parents, teachers, principals, and schools. Tests were administered for mathematics and for the Vietnamese language. 4. Results 4.1 The joint contribution of the SRPP and VNEN projects to education reform in Vietnam is likely greater than the sum of their respective parts. Although impressive and efficiently realized, the measured results of both projects tell only part of the story. The projects were part of a national reform strategy as well as a program of World Bank support for education reform.36 Unfortunately, the M&E system that accompanied this sustained effort operated on a project-by-project basis—largely with output indicators— and without a longitudinal, programmatic assessment. The results of SRPP and VNEN verify that objectives and targets were met and that some immediate outcomes have been realized, such as pedagogical changes and classrooms becoming more child- centered, consistent with the theory of change (see figures 2.1 and 2.2). 4.2 The SRPP and VNEN projects were used to spearhead reform. For VNEN, the concept underpinning the reform (the need for changes in pedagogy to meet the demands of the rapidly changing job context) was broadly accepted. Stakeholders interviewed for the IEG mission were clear in their assessment of the positive contribution of these projects to introducing new concepts regarding the purpose and practice of education and to changing mindsets in support of a durable shift in education philosophy, thereby influencing ongoing reform. However, the particular model adopted to implement change (Escuela Nueva) met resistance that limited the extent of the project’s potential to influence change. There was nothing contentious in the measures introduced by SRPP. Support provided for lunch subsidies, teacher training, and preschool assessment were part of the progressive development efforts associated with a burgeoning ECE sector. 4.3 A joint outcome attributed to both projects by key stakeholders is the establishment of a permanent PMU in MOET. A MOET PMU staffed by MOET officials was first established under the VNEN project and a similar model was applied in support of SRPP. The level of alignment achieved between the administration and project management functions helped get things done and provided the opportunity to 22 realize sustained capacity that is contributing to the rollout of ongoing World Bank– supported projects in the education sector. SRPP Results 4.4 Subsidies for lunch were expected to address some of the enrollment barriers for disadvantaged children. In eliminating a cost—lunch was provided at half the price (Vietnamese dong 120,000, about $5 per child per month)—the subsidy incentivized parents of disadvantaged children to send their children to the preschools.37 The provision of lunch also eliminated the need for parents of disadvantaged children to make four round-trips to the school each day, freeing them up to work. A healthy meal also improved children’s nutrition (World Bank 2013b). The IEG mission met with a group of parents in each of the preschools visited during the mission and found that they had confidence in the safety of the food handling by the preschool and the nutritional content of the lunch. 4.5 In 2016 the government of Vietnam extended the time allowed for payment of the lunch subsidy. This permitted ongoing, continued support for lunch subsidies beyond the project close and indicates that the government believes that the subsidy is working, a view confirmed by stakeholders at central and local levels during the IEG mission. 4.6 Nearly all preschool managers (99 percent) and preschool teachers (93 percent) completed the priority modules. Teachers reported that the content was practical and that training was interactive. They experienced a positive response from children, which motivated them to apply the training. Teachers gave examples of how the training was put into practice. During the IEG mission, teachers still indicated a need for more hands-on support and for resources such as lesson plans. They expressed a need to practice with children as they learn the content and then receive support as they apply it in the classroom. This would be expected given the substantive change needed (and lack of preservice training on these aspects) for teachers to understand and implement play-based and child-facilitated learning. 4.7 The IEG mission was informed that 96 percent of schools had completed a self-assessment (at project close the level of completion was 90 percent). Forty-one percent of preschools had received level 1 accreditation by the end of the project. The target for DLIs was 40 percent. This rate was not achieved equally across provinces, and disadvantaged provinces had much lower percentages of accredited schools. This suggests that provinces may have focused on those readily meeting quality standards, rather than bringing up low-capacity schools. The Early Childhood Education Strategy (2016–25) has established a goal of 80 percent of schools attaining level 1 accreditation. 23 Going forward, disadvantaged districts and provinces will require more support and resources to increase the number of accredited schools. 4.8 Over the project life cycle, all 63 provinces achieved universal preschool for five-year-old children, which exceeded the target (50 provinces).38 More than 843,000 children enrolled in kindergarten between 2014 and 2017 (table A.2). The number of children in kindergarten increased from 3.7 million in 2014 to 4.6 million in 2017. The rate of growth was similar for ethnic minority children (16.7 percent) to the overall increase (17.7 percent). The rate of enrollment (part and full day) for five-year-old children increased during the operation from 96 percent in 2012 to 99 percent in 2017. The percentage of children aged five participating in a full-day program increased from 73 in 2012 to 88 percent in 2017. The increase in enrollment has come about despite limited demand-side interventions, suggesting that these may not be necessary. Parents reported to the IEG mission that they felt it was their responsibility to send their child to school and had strong confidence in the value of education from both academic and socialization perspectives. The full-day programming (from 7 a.m. to 4 p.m.) was convenient for parents and facilitated their labor market participation, according to respondents. 4.9 Preschool enrollment has also increased for children aged three and four years by more than 46,000. The number of enrolled children aged three and four years increased from 661,877 in 2014 to 707,990 in 2017. The rate of growth for girls and ethnic minorities is similar to the overall rate. 4.10 Analysis of the EDI revealed improvements in children’s readiness, although almost a third of children remained off track. Twenty-nine percent of children were at risk of vulnerability or were vulnerable (lowest two groups) in the final survey compared with 50 percent at baseline. The groups most likely to be vulnerable were children from an ethnic minority background, poor households, or households with mothers having only a primary education (or lower). Girls were found to be less likely to be vulnerable in the analysis of each round of the survey results. The Central Highlands region had the highest percentage of vulnerable children in each round. The rate of vulnerability among ethnic minority children decreased from 45 percent in 2012 to 22 percent in 2016. The rate of vulnerability among Vietnamese-speaking children (nonethnic minority) decreased from 19 percent in 2012 to 9 percent in 2016. The overall rate of vulnerable five-year-old children decreased from 24 percent to 12 percent, which exceeded the low target (22 percent). 24 VNEN Results 4.11 Target values for all three project development objective indicators were met or exceeded. The three indicators were (i) the number of students participating in VNEN; (ii) the number of VNEN learning guides, teacher guides, and Teacher Training Institute guides developed; and (ii) the number of primary education teachers and education administrators completing VNEN training. The scaling up reached 446,781 students, of which 352,956 (approximately 79 percent) were in 20 priority 1 provinces. These students attended 1,447 schools, of which 1,143 (approximately 80 percent) were in 20 priority 1 provinces. The target value for the second project development objective indicator was exceeded by 472 percent.39 The Implementation Completion and Results Report attributes this gain to strong MOET capacity and technical assistance in VNEN material development. However, while both MOET capacity and technical assistance were strong relative to the baseline value achieved during the pilot, the original target looks particularly low, and this probably better explains the level of performance realized. The target value for the third project development objective indicator was exceeded by 76 percent; this gain was attributed to the modified cascade training, which allowed more teachers to be trained at school level. 4.12 The target values for the four intermediate indicators were also met or exceeded. The intermediate outcome indicators were (i) the number of workshops for material development, (ii) the number of schools receiving a full complement of VNEN materials, (iii) the number of schools receiving campus grants, and (iv) a qualitative study of classroom observations comparing VNEN and non-VNEN classrooms. The Implementation Completion and Results Report attributes the holding of additional workshops to a highly participative and collegiate engagement of all relevant stakeholders, including teachers, administrators, and technical experts. A high level of collegiality among stakeholders remained evident in the field visits to VNEN schools. The qualitative study included classroom observations and the production of videos on best practice teaching that are still available. A full suite of materials to support the use of the approach was available for use for grades one to five. 4.13 VNEN also helped deliver results outside of the project boundaries. In December 2013, given healthy progress within the project as well as the ambition of MOET champions of the approach, MOET indicated that it would expand the pilot to grade six and introduce VNEN to 1,214 lower secondary schools. The project supported the development of sixth grade materials while all of the training and provision of materials were supported through government resources. A total of 94,456 grade six lower secondary school students experienced the VNEN approach. 25 4.14 The hoped-for demonstration effect was realized, as evidenced by substantial voluntary adoption of the VNEN approach. A total of 452,255 nonproject students (of whom almost 50 percent were in priority 3 provinces, which are more advantaged provinces with far fewer project schools) experienced the VNEN approach in schools that voluntarily adopted the approach. This level of success is indicative of the profile attained by the project in the primary education system. 4.15 The key findings from the impact evaluation were that the program had a positive effect on the socioemotional skills of children enrolled in program-supported schools and that students performed as well or better than in traditional schools.40 This finding was based on parent feedback regarding the behavioral development of their children over the three-year development of the VNEN program. The evaluation not only found that students in the program did better than their counterparts in traditional schools, it also found that VNEN students did particularly well at the lower end of the distribution, where there is a greater concentration of disadvantaged groups. The impact evaluation also found that VNEN students perform as well as or better than students under the traditional, frontal/didactic regime for Vietnamese language and mathematics. Students from VNEN schools already showed a higher mean score at the baseline of the study. Regression analysis shows a statistically significant effect of the VNEN program, with some specifications showing effect sizes of about one-fifth of a standard deviation, approximately 15 points in Vietnamese and 18 points in mathematics. 5. Lessons 5.1 IEG identifies the following lessons from its evaluation of the two operations: • In addition to lending, the World Bank can add value through the transmission of knowledge from experiences and lessons that help shape reforms. This included World Bank economic and sector work, study visits, the introduction of international experts, and the introduction of lessons gleaned from an experienced team that built on the long-standing relationship of mutual trust between MOET and the World Bank. The World Bank’s knowledge capability was a useful complement to the implementation capacity of the education system in Vietnam. For SRPP, the World Bank advocated and helped support the rollout of a systemwide approach through preschool accreditation to improve quality. Since closure, the government has moved to universalize enrollment beyond kindergarten and has further refined the school self- assessment process. VNEN introduced a new pedagogic philosophy into the primary education system. Many teachers have expanded their pedagogical 26 skills and the model laid part of the foundation for the current curriculum reform and associated teacher training. • When significant pedagogical changes are required of teachers, incentives, support, and long-term commitment are needed (probably more than education systems realize). The changes envisioned in both of these operations were neither simple nor easy. Studies show the importance of follow-up support to facilitate the application of in-service training. Within the education system, existing mechanisms provide follow-up support to teachers such as school visits, principal feedback, and peer-to-peer learning at school cluster meetings. Some of these mechanisms were applied under both projects, but VNEN employed them more extensively. For example, during the school year, two to four technical support team meetings and training sessions were conducted for VNEN schools. These meetings and sessions involved classroom observations, interviews with teachers and school management, review of logbooks, and exchange of experiences with principals and teachers. E-learning was used under SRPP for follow-up knowledge. Although it may serve to bridge at least some of the existing gaps in follow-up support, e-learning is unlikely to represent a panacea. Constraints in the implementation of SRPP e-learning modules suggest the need to address these issues. This emphasizes the need for long-term commitment and to leverage existing mechanisms more fully to bring about the kind of quality teaching envisioned. • When scaling up or adopting a systemwide approach, it is important to understand and design this approach in accordance with the decentralized context and challenges faced at the various levels of administration. SRPP required provincial and district officials to provide follow-up support after face- to-face cascade training and to provide technical knowledge to the preschool accreditation and school self-assessment process. Follow-up support to teachers in SRPP and VNEN was implemented more effectively in some provinces than in others. An analysis was conducted (by SRPP) to understand how the decentralized systems operated and to detect capacity constraints. Yet design and implementation did not differentiate resources or technical support to provinces and districts to address differences in capacity. As a result, disadvantaged provinces had much lower percentages of accredited schools. The DLIs set an appropriate incentive for the central level but not for provincial and district levels. • Targeting disadvantaged areas does not translate into targeted efforts for specific vulnerable groups. The self-selection process under VNEN was used to encourage a demonstration effect in locations where ethnic minority and other 27 disadvantaged populations were present. The school selection process under VNEN favored the best schools (for demonstration effect) in locations where ethnic minority and other disadvantaged populations were present but not necessarily schools in which such children were concentrated. The targeting strategy was implemented to generate the momentum needed to support the further adoption of the model in all provinces (this resulted in greater voluntary adoption in the wealthiest provinces). Although this approach is sensible, it may have missed opportunities to target even more ethnic minority children. SRPP provided lunch subsidies to disadvantaged families, but all other aspects of the project were applied systemwide. A social assessment detected multiple enrollment barriers for ethnic minorities, such as distance from school or parental knowledge, that were not addressed. The baseline survey substantiated the higher level of vulnerability among ethnic minority children (compared with those who speak Vietnamese). Considering the critical juncture for language development during the early years, targeted action was needed to support learning in the first language and introduction to Vietnamese. • When scaling up, the importance of consultation and communication cannot be underestimated. The literature suggests that scaling up programs requires those promoting the innovation to convince stakeholders of the effectiveness of the intervention. Both of these operations had broad political support. Despite the attention to communicating with stakeholders, particularly during the early stages of VNEN implementation, the scaling up of the model proved challenging. The communication challenge was associated with substantial changes for teachers and divergence from what parents traditionally experienced in school and, more particularly, the expansion of the model to nonproject voluntary schools that did not benefit from the same concentration of support as their project-based counterpart schools. Education is a heavily debated topic in Vietnamese society, suggesting the critical role for sustained and sophisticated communication regarding planned change. 1 See https://www.gso.gov.vn/default.aspx?tabid=723. 2 See https://www.gso.gov.vn/default.aspx?tabid=723. 3See http://chinhphu.vn/portal/page/portal/chinhphu/noidungchienluocphattrienkinhtexahoi?docid=1 254&substract=&strutsAction=ViewDetailAction.do. 28 4 See http://congbao.chinhphu.vn/tai-ve-van-ban-so-142-2016-qh13-19527-13966?format=pdf. 5 See http://datatopics.worldbank.org/education/country/vietnam. 6 Organisation for Economic Co-operation and Development countries average 12.9 percent. 7 Kindergarten is typically attended by children who are five or six years old. 8 See http://www.moet.gov.vn/thong-ke/Pages/thong-ke-giao-duc-tieu-hoc.aspx?ItemID=5392. 9See http://english.vietnamnet.vn/fms/education/161126/many-parents-reject-new-teaching- methodology.html. 10The proportion of ethnic minority students in primary and lower secondary schools is 17 and 16 percent, respectively, while in upper secondary it is 12 percent. See http://www.moet.gov.vn/thong-ke/Pages/thong-ke-giao-duc-tieu-hoc.aspx?ItemID=5392. 11 See https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf. 12The Japanese Social Development Fund financed this effort, which was implemented by Save the Children and the World Bank. The pilot also developed learning materials in other languages. 13 It has also financed six operations on higher education. 14 IEG 2015 Later Impacts of ECD Interventions: A Systematic Review. World Bank; Washington DC provides a synthesis of evidence. 15With the devolved system, provinces need to use existing resources to develop appropriate supports for ethnic minority students. 16Aspects included organization and school management; qualification and training of managers and teachers; physical facilities, equipment, utensils, and toys; relationship between school, family, and society; and results of nourishment, care, and education. 17 See https://www.naeyc.org/accreditation The most skilled and motivated of the teachers trained during the pilot and practicing since 18 were included in the group of trainers. 19The Primary Education for Disadvantaged Children Project in 48 grade two classrooms in 24 schools across 12 districts in six provinces with large ethnic minority populations. The pilot demonstrated that the model could be readily adapted to the Vietnamese context. It also provided a “head” start for the project proper as materials were developed (for grades one and two) that could be used as part of the project rollout. 20Materials were developed for grades three to six. Grade one and two materials had been developed under Primary Education for Disadvantaged Children or the Vietnam Escuela Nueva (VNEN) pilot. 21 The Project Management Unit would maintain a designated account at a commercial bank into which Global Partnership for Education funds would be deposited before being transferred from 29 that account to intermediate designated accounts held by the provincial Project Management Units and on to the schools’ bank accounts. 22The system in Vietnam works on a devolved basis. The Ministry of Education and Training (MOET) has policy responsibility at the central level. Provinces must allocate funding from the Ministry of Finance to education. For the School Readiness and Promotion Project (SRPP), it was necessary to collect provincial data to substantiate coverage and associated spending to satisfy the disbursement-linked indicator approach. 23During the delay, the already-established Project Management Unit worked on other aspects of the project (such as opening the bidding process for the printing and production of grades one, two, and three teaching and learning materials and revising the project implementation manual and having it printed and distributed to all stakeholders) such that there was a grant disbursement of $19.92 million for fiscal year 2013, about 71 percent more than the original estimate for that fiscal year. 24The emphasis was on continual training and support to ensure understanding and sustainability of the VNEN model. For example, during the school year, two to four technical support team meetings and training sessions were conducted for VNEN schools; these meetings and sessions involved classroom observations, interviews with teachers and school management, a review of logbooks, and an exchange of experiences with principals and teachers. Teachers from neighboring schools involved in the project (and sometimes those voluntarily engaged) would also exchange experiences at cluster-level meetings held every two months. 25The distribution of schools was determined according to priority categories as follows: (i) the percentage of students classified as belonging to poor families; (ii) the percentage of students belonging to ethnic minority groups; (iii) distance of the school from the district center; and (iv) the percentage of students who perform as average or poor in Vietnamese student achievement measures. Having identified schools with at least one tag from the priority categories, 20 of 63 provinces were identified as having a high concentration of schools with the above categories and were classified as priority 1 provinces (having high numbers of ethnic minorities and poor districts); 21 were identified as priority 2 provinces with moderate numbers of disadvantaged students; and 22 were identified as priority 3 provinces. Eighty to 90 percent of the 1,447 schools supported under VNEN would be located in the poor provinces, since selection was further defined by the fact that (i) schools volunteered to participate in VNEN; (ii) schools participated in full-day schooling (9 to 10 sessions per week in grades two and three); (iii) schools had less than six class sections per grade; (iv) school leadership had a demonstrated record; (v) students achieved adequate proficiency in Vietnamese literacy; (vi) schools had less than five satellite schools; and (vii) schools had not already benefited from the School Education Quality Assurance Program. 26The project appraisal document notes that the project sought to indirectly benefit a larger number of children by helping to bring about a systemwide transformation through pedagogical innovation (World Bank 2012, 4). MOET committed to making available all relevant training modules, didactic materials, and knowledge regarding the VNEN protocol for all interested schools and noted that provinces can use their own financing to deploy these materials and extend the number of schools applying the VNEN model. To facilitate this, and in addition to the 30 20 priority provinces, MOET would support a small sample of VNEN demonstration schools in other provinces. 27 The training included the following: (i) initial training on teaching VNEN model to grades two to five, (ii) training for the hands-on method modules for central-level trainers and teachers and (iii) training on the VNEN student evaluation method. 28Nearly all preschool managers (99 percent) and preschool teachers (93 percent) completed the priority modules. 29The impact evaluation also comments on varying levels of commitment to implementing the program in its entirety at the provincial level. 30In this regard the project was highly successful, with 2,341 schools voluntarily adopting the approach in 2014–15 and 2,730 doing so in 2015–16. This compares with 1,447 project schools. Le (2018) is critical of the over-emphasis on increasing breadth rather than depth during the scaling up, arguing that the scaling up of educational innovations needs to be understood as “a political process of negotiating traditional power relations, notions of authority, and the implicit beliefs regarding the nature of teaching and learning” (223)—that is, it should be understood not in terms of numbers but in transforming existing educational beliefs and practices. 31 Vietnam—Country Partnership Framework for FY18–22 cites the need for effective feedback loops. 32As one interviewee noted, it is possible at the central level to identify the exact number of children in preschool at the village level throughout the country. 33Before VNEN project effectiveness (January 2013), MOET replaced the District Fundamental School Quality Level Audit approach, a paper-based system, with the online Education Quality Management System. This meant a short delay in data production but, ultimately, data for the more efficient Education Quality Management System was collected three times a year. 34For example, the frequency of VNEN training and teacher meetings, the number of schools voluntarily implementing the VNEN model, and the number of children exposed to the new approach. 35A consultant with strong research experience oversaw the final survey. There was improved training for teachers who completed the survey. An international consultant monitored the quality of data collection and analysis. Variables were added to the final round of the survey to identify factors about the school, teacher, or school managers that may mitigate children’s vulnerabilities. However, analysis with the new variables did not control for student characteristics. 36 Other donors provided significant support to the education sector. For example, the Asian Development Bank provided sustained support to the lower and higher secondary levels. 37Eligibility for the lunch subsidy (50 percent of the fee) was based on government criteria, which consisted of residents in frontier, mountainous areas or designated disadvantaged districts; registered poor families; orphans; or disabled children with economic difficulties. 31 38MOET defines universal preschool for five-year-old children based on 95 percent enrollment, with 85 percent in a full-day program; attendance of 90 percent; and underweight malnutrition of less than 10 percent of children. 39Materials were developed for grades three to six. Materials for grades one and two had already been developed under the Primary Education for Disadvantaged Children Program or the VNEN pilot. 40The impact evaluation followed a cohort of students from grades three to five and measured the extent to which students acquired cognitive and noncognitive skills by comparing results from randomly chosen sets of VNEN schools and traditional schools. 32 Bibliography Akyeampong, K., J. Pryor, J. Westbrook, and K. Lussier. 2013. “Improving Teaching and Learning of Basic Maths and Reading in Africa: Does Teacher Preparation Count?” International Journal of Education Development 33: 272–282. Belfield, Clive R., Milagros Nores, Steve Barnett, and Lawrence Schweinhart. 2006. “The High/Scope Perry Preschool Program: Cost-Benefit Analysis Using Data from the Age-40 Followup.” Journal of Human Resources 41 (1): 162–190. http://www.jstor.org/stable/40057261. Carrillo, Camilo, Henriëtte Maassen van den Brink, and Wim Groot. 2016. “Professional Development Programs and their Effects on Student Achievement: A Systematic Review of the Evidence.” Tier Working Paper Series 16/03, Top Institute for Evidence Based Education Research, Minderbroedersberg, the Netherlands. http://www.tierweb.nl/home/assets/files/UM/Working%20papers/TIER%20WP%2016- 03.pdf. Darling-Hammond, L. 2000. “Teacher Quality and Student Achievement: A Review of State Policy Evidence.” Education Policy Analysis Archives 8 (1). https://epaa.asu.edu/ojs/article/view/392. Darling-Hammond, L. 2005. “Teaching as a Profession: Lessons in Teacher Preparation and Professional Development.” Phi Delta Kappan 87 (3): 237–240. https://doi.org/10.1177/003172170508700318. Darling-Hammond, L., M. E. Hyler, and M. Gardner. 2017. Effective Teacher Professional Development. Palo Alto, CA: Learning Policy Institute. Darling-Hammond, L., R. C. Wei, A. Andree, N. Richardson, and S. Orphanos. 2009. Professional Learning in the Learning Profession. Washington, DC: National Staff Development Council.Ganimian, A. J., and R. J. Murnane. 2016. “Improving Education in Developing Countries Lessons from Rigorous Impact Evaluations.” Review of Educational Research 86 (3): 719–755. Garet, M., A. Porter, L. Desimone, B. Birman, and K. Yoon. 2001. “What Makes Professional Development Effective? Results From a National Sample of Teachers.” American Educational Research Journal 38 (4): 915–945. Garet, M. S., A. J. Wayne, F. Stancavage, J. Taylor, K. Walters, M. Song, and F. Doolittle. 2010. Middle School Mathematics Professional Development Impact Study: Findings After the First Year of Implementation (NCEE 2010-4009). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 33 Gertler, Paul J., Rodrigo Pinto, Arianna Zanolini, Christel Vermeerch, Susan Walker, Susan M. Chang, and Sally Grantham-McGregor. 2013. “Labor Market Returns to Early Childhood Stimulation: A 20-Year Follow-Up to the Jamaica Study.” Policy Research Working Paper 6529, World Bank, Washington, DC. Global Partnership for Education. 2019. “Vietnam.” GPE. https://www.globalpartnership.org/country/vietnam. Gouleta, Eirini. (2009). Indigenous Bilingual Education in Vietnam Initiatives and Lessons America’s Schools Can Learn From. NABE News. Heckman, James J. 2008. “The Case for Investing in Disadvantaged Young Children. Big Ideas for Children: Investing in Our Future.” https://heckmanequation.org/www/assets/2017/01/Heckman20Investing20in20Young20C hildren.pdf. Heckman, James J., Rodrigo Pinto, and Peter Save lyev. 2013. “Understanding the Mechanisms Through Which an Influential Early Childhood Program Boosted Adult Outcomes.” American Economic Review 103 (6): 2052–2086. Jacob, R., H. Hill, and D. Corey. 2017. “The Impact of a Professional Development Program o n Teachers’ Mathematical Knowledge for Teaching, Instruction, and Student Achievement.” Journal of Research on Educational Effectiveness 10 (2): 379–407. Karalis, T. 2016. “Cascade Approach to Training: Theoretical Issues and Practical Applications in Non-Formal Education.” Journal of Education & Social Policy 3 (2): 104. Luneta, K. 2012. “Designing Continuous Professional Development Programmes for Teachers: A Literature Review.” Africa Education Review 9 (2): 360–379. Martinez, Sebastian, Sophie Naudeau, and Vitor Pereira. 2012. The Promise of Preschool in Africa: A Randomized Impact Evaluation of Early Childhood Development in Rural Mozambique. Washington, DC: World Bank. McAleavy, T., T. Ha, and R. Fitzpatrick. 2018. Promising Practice: Government Schools in Vietnam. Reading, U.K.: Education Development Trust. https://www.educationdevelopmenttrust.com/our-research-and- insights/research/promising-practice-government-schools-in-vietnam. Mekong Delta Research Institute (MDRI) 2018. 54 Ethnic Groups: Why the Difference?. Hanoi, Vietnam: Author. OECD (Organisation for Economic Co-operation and Development). 2012. “Encouraging Quality in Early Childhood Education and Care. Research brief: Qualifications, Education and Professional Development Matter.” http://www.oecd.org/education/school/49322232.pdf. 34 ———. 2018. PISA Results in Focus (PISA 2015). Paris: OECD. https://www.oecd.org/pisa/pisa- 2015-results-in-focus.pdf. Orr, D., J. Westbrook, J. Pryor, N. Durrani, J. Sebba, and C. Adu-Yeboah. 2013. What Are the Impacts and Cost-effectiveness of Strategies to Improve Performance of Untrained and Under- trained Teachers in the Classroom in Developing Countries? London: EPPICentre, Social Science Research Centre, Institute of Education, University of London. Parandekar, S. D., F. Yamauchi, A. B. Ragatz, E.K. Sedmik, and A. Sawamoto. 2017. Enhancing School Quality in Vietnam through Participative and Collaborative Learning: Vietnam Escuela Nueva Impact Evaluation Study. Washington, DC: World Bank. Patrinos, H. A., and Psacharopoulos, G. 2010. “Returns to Education in Developing Countries.” In International Encyclopedia of Education, edited by P. Peterson, E. Baker, and B. McGaw, 2:305–312. Oxford: Elsevier. Popova, Anna, David Evans, Mary E. Breeding, and Violeta Arancibia. 2018. “Teacher Professional Development around the World: The Gap between Evidence and Practice. ” Policy Research Working Paper WPS 8572, World Bank, Washington, DC. https://doi.org/10.1596/1813-9450-8572. Schweinhart, Lawrence. 2007. “Outcomes of the High Scope Perry Pre-School Study and Michigan School Readiness Program.” In Early Childhood Development: From Measurement to Action, edited by M. Young and L. Richardson, 87–103. Washington, DC: World Bank. UNICEF (United Nations Children's Fund). 2015. Evaluation of Mother-Tongue Based Bilingual Education in Viet Nam 2006–2014. UNICEF: New York. Villegas-Reimers, E. 2003. Teacher Professional Development: An International Review of Literature. Paris: International Institute for Educational Planning. Walker, Susan, Susan Chang, Amika Wright, Clive Osmond, and Sally Grantham-McGregor. 2013. “Do Early Childhood Experiences Affect Development in the Next Generation? The Jamaica Intergenerational Study.” Paper presented at the Inter -American Development Bank conference, “Early Childhood Development in the Caribbean: Identifying Cost Effective Interventions,” Kingston, Jamaica, November 14. Westbrook, J., N. Durrani, R. Brown, D. Orr, J. Pryor, J. Boddy, and F. Salvi. 2013. “Pedagogy, Curriculum, Teaching Practices and Teacher Education in Developing Countries. ” Education Rigorous Literature Review, U.K. Department for International Development, London, U.K. World Bank. 2011a. Vietnam—High Quality Education for All By 2020. Washington, DC: World Bank. ___________.2011b. Vietnam—Country Partnership Strategy for FY12–FY16 (English). Washington, DC: World Bank. 35 http://documents.worldbank.org/curated/en/849061468133768943/Vietnam-Country- partnership-strategy-for-the-period-FY12-FY16. ———. 2012. “Vietnam—Global Partnership for Education—Vietnam Escuela Nueva Project.” Project Appraisal Document 72285-VN, World Bank, Washington, DC. ———. 2013a. Vietnam—Credit Number C5207-VN Financing Agreement. Washington, DC: World Bank. ———. 2013b. “Vietnam—School Readiness Promotion Project.” Project Appraisal Document 65942-VN, World Bank, Washington, DC. ———. 2013c. “Vietnam— Vietnam: Global Partnership for Education–Vietnam Escuela Nueva Project.” Implementation Status and Results Report ISR9427, World Bank, Washington, DC. ———. 2015a. Vietnam—Renovation of General Education Project. Washington, DC: World Bank. https://hubs.worldbank.org/docs/imagebank/pages/docprofile.aspx?nodeid=24632400. ———. 2015b. World Bank Support to Early Childhood Development. Independent Evaluation Group. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/812191468187794860/World-Bank-support- to-early-childhood-development-an-independent-evaluation. ———. 2016a. Vietnam—Enhancing Teacher Education Program Project. Washington, DC: World Bank. https://hubs.worldbank.org/docs/imagebank/pages/docprofile.aspx?nodeid=26485045. ———. 2016b. “Vietnam—Global Partnership for Education—Vietnam Escuela Nueva Project.” Implementation Completion and Results Report ICR3864, World Bank, Washington, DC. ———. 2017a. Vietnam—School Readiness Promotion Project Implementation Completion and Results Report.World Bank, Washington, DC. ———. 2017b. Vietnam—Country Partnership Framework for FY18–FY22. Washington, DC: World Bank. ——— Young, Mary Eming, and Linda M. Richardson, eds. 2007. Early Child Development from Measurement to Action: A Priority for Growth and Equity . Washington, DC: World Bank. 36 Appendix A. Vietnam School Readiness and Promotion Project Ratings Table A.1. Vietnam School Readiness and Promotion Project Ratings Indicator ICR* ICR Review* PPAR Outcome Satisfactory Satisfactory Satisfactory Bank performance Satisfactory Satisfactory Satisfactory Quality of M&E High High Substantial Risk to development outcome Low Low Low * The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible World Bank Global Practice. The ICR Review is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. Note: PPAR = Project Performance Assessment Report. Relevance of the Objectives According to the financing agreement and the project appraisal document for the Vietnam School Readiness and Promotion Project (P117393), the objective of the project was “to raise school readiness for five-year-old children, in particular for those most vulnerable to not succeeding in a school environment, through supporting selected elements of Vietnam’s Early Childhood Education (ECE) program” (World Bank 2013a, 4; World Bank 2013b, ii). The objective, performance indicators, and targets remained unchanged during the operation. The objectives were aligned with priorities contained in multiple strategic documents developed by the government. The government’s Socio-Economic Development Strategy 2010–20 and the Socio-Economic Development Plan 2016–20 emphasize human capital investments to increase productivity and capacity. The objectives were also highly relevant to the Development Strategy (2011–20) of the Ministry of Education and Training (MOET) and Article of Education Law (No. 44/2009/QH12 and2018/QH14). The recently amended Education Law (2018/QH14) and Early Childhood and Education’s Decree (Prime Minister Decision 239/2010) focused attention on universalizing enrollment for children age five and, later, age four. Consistent with the development objective, the aim of early childhood education was “Holistic physical, emotional, intellectual, and aesthetic development; forming the initial elements of children’s personality; and preparing children to enter grade 1” (2018/QH14). International evidence demonstrates the importance of early childhood development, since it has a substantial effect on later outcomes (Walker et al. 2013, Schweinhart 2007, and Gertler et al. 2013). Learning gaps were evident between Kinh (the largest ethnic minority) and non-Kinh primary school students (World Bank 2011a), suggesting the 37 need for more attention to younger children and improving children’s readiness for school. The objectives remained consistent with the pillars in the Country Partnership Strategies of 2012 and 2018, although there has been a shift away from support for preprimary education in the World Bank’s strategy. The priorities advanced in the 2012 strategy (current during preparation and implementation of this project) emphasized improving opportunities for people (World Bank 2011a). Both strategies recognized the importance of reaching marginalized groups for growth to be equitable, since poverty is concentrated among ethnic minority communities (World Bank 2017b). Moreover, Vietnam 2035, a joint government–World Bank plan for development, promotes equality and inclusion among marginalized groups to create a harmonious middle-class society. The project development objective (improving readiness for school) is a long-term means to improve educational outcomes and opportunities for the disadvantaged and vulnerable. While the current Country Partnership Framework (FY18–22) shifts the support from preprimary to tertiary education, the framework focuses on malnutrition and stunting. This multisector support for young children to address stunting and malnutrition, which are particularly high among ethnic minority children, is a long-term avenue to the desired outcomes. Thus, the objectives have continued to be highly relevant for the World Bank Group and government since preparation and beyond closure. The relevance of the objectives is rated high. Efficacy The project provided on-budget financing through disbursement-linked indicators (DLIs) for four interventions: lunch subsidies for disadvantaged children, training, teachers’ salaries, and school accreditation and self-assessment. Retroactive reimbursement of eligible expenditures was provided based on completion and verification of the DLIs. The reimbursement of preschool teacher salaries was discontinued due to financial reporting issues (see the Financial Management section). The operation also developed four pilot ECE centers for young children (birth to age 3) and provided technical assistance. Lunch Subsidies for Disadvantaged Children Subsidies for lunch were expected to address some of the enrollment barriers for disadvantaged children. A lunch (at half price) eliminated the need for parents to go to the school four times each day, which was a barrier for working parents. A healthy meal also improved children’s nutrition since poverty was a barrier (World Bank 2013b). The 38 Independent Evaluation Group (IEG) mission was told by preschool children’s parents that they were confident about the safety of the food handling by the preschool staff and the nutritional content of the lunch. Eligibility for the lunch subsidy (50 percent of the fee) was based on the government’s criteria, which included residents in frontier, mountainous areas or designated disadvantaged districts; registered poor families; orphans; or disabled children with economic difficulties. The project reimbursed MOET for the monthly amount it paid (Vietnamese dong 120,000.)1 The government extended the period of the decree, which permitted continued support for lunch subsidies to this date. Professional Development for Preschool Managers and Preschool Teachers Training built on previously developed resources and existing practices of MOET. The training program was created before project launch. International experts developed the training content, which aligned with the ECE curriculum. Modules were developed for both preschool managers and teachers. The modules included both content and pedagogical knowledge. Topics included child-centered teaching, partnerships with parents and community, nutrition and preschool lunch, disadvantaged and ethnic minorities, cognitive development, emotional and social development, and language development. The content was not differentiated by student age, despite children’s different development levels and the need for different learning activities and objectives. Developmental differences were discussed in the training. MOET conducted an assessment of teachers’ needs. A cascade model was implemented for cost reasons. Expert trainers trained the provincial delegates in Hanoi—core teachers or representatives from the Department of Education and Training—who then were responsible for delivering training in the districts and schools. Complementary resources (such as examples or lesson ideas) were not provided to preschool teachers. MOET evaluated the training and gathered feedback based on stakeholder reports. Nearly all preschool managers (99 percent) and preschool teachers (93 percent) completed the priority modules. Teachers reported that the content was practical and that training was interactive. They experienced a positive response from children, which motived them to apply the training. Teachers gave examples of how the training was put into practice. During the IEG mission, they still indicated a need for more hands-on support and for resources such as lesson plans, and expressed a need to practice with children as they learn the content and then receive support as they apply it in the classroom. This would be expected given the substantive change needed (and lack of preservice training on these aspects) for teachers to understand and implement play- based and child-facilitated learning. 39 E-learning modules were used to provide follow-up support. Ten advanced e-learning modules were developed at the end of the operation (as a delivery method to reduce training costs). Administrative officials at each level could track how many teachers used or completed modules. This format was intended to create a forum for teachers to exchange views. Based on reports, challenges limited the effectiveness of this modality. Challenges with e-learning included limited broadband access in remote areas, limited information technology skills of teachers, limited computer equipment available in schools, limited time, and a lack of practical, hands-on application or support. There were also issues with the platform. Once a module was completed, a user could not access it later because the modules were closed and could not be subsequently updated. MOET has not had the resources to improve the platform, and the e-learning has now been integrated into the MOET annual program. DOET officials provided school-based follow-up support, but this was not uniformly applied. Teachers are required to engage in self-learning. Peer learning and teacher meetings are a feature in the education system facilitated by district officials in which preschool teachers engage in discussion with teachers in the same school or nearby schools. Some teachers visited other schools to observe lessons. The World Bank team made suggestions to improve the impact of the teacher training with follow-up support through feedback and mentors. To implement this, MOET and DOET would need to use the existing budget. Thus, one consequence of the DLIs is that they incentivized broad coverage rather than follow-up support, which is critical to changing teacher’s knowledge and practices (Darling-Hammond 2005, Garret et al. 2001; Darling- Hammond et al. 2017; Popova et al. 2018). School Accreditation and Self-Assessment A preschool accreditation system was established to improve quality. This system was built from previous World Bank experience with primary education in Vietnam. National Quality Standards for Preschool Accreditation were developed and approved in February 2011. These standards were aligned with international practices to support child-centered learning. The standards and the recently developed ECE curriculum promoted play-based learning and guided children’s learning to improve cognitive, language, physical, and socioemotional development. The reporting arrangements for self-assessments were not fully developed during preparation and were untested (World Bank 2013b). In addition, the preschool accreditation and standards systems had to be merged. School self-assessment and accreditation were financed by DLIs. Thus, it is important to understand what was incentivized and not. The DLIs included the adoption of regulations for preschool quality and self-assessments and external accreditation for 40 preschools ($5 million); the percentage of preschools undertaking a self-assessment ($0.17 million for each percentage point increase); and the percentage of schools reaching level 1 accreditation ($0.375 million to $15 million for each percentage point increase). The self-assessment was built on the National Quality Standards for Preschool Accreditation (and aligned with the previously developed ECE curriculum). Aspects included organization and school management; qualification and training of managers and teachers; physical facilities, equipment, utensils, and toys; the relationship between school, family, and society; and the results of nourishment, care, and education. These areas were monitored using 29 indicators for school managers to assess the level of completion. Many of the indicators relate to conditions of school infrastructure, as well as the qualifications of teachers and school managers. The limitations of school infrastructure was a major challenge for moving up to the next level, according to stakeholder interviews. An important indicator within the assessment was the regular monitoring and evaluation of children. Although there were indicators related to the appropriate development of children, the basis for evaluation was not evident. Some aspects that occur in international best practices were not specified in the standards and indicators for Vietnam. The self-assessment did not examine curriculum (what is taught to advances children’s development), the relationship with children (how adults foster children’s self-regulation and responsibility), or teaching approach (how teachers use various instructional approaches).2 These aspects are critical to promoting quality, but they may have been beyond the capacity of school managers and district and provincial staff to evaluate. Respondents reported the desire to keep indicators simple. MOET enhanced the process of school self-assessment and accreditation. Software was developed to track the number of schools and districts that completed self-assessments. There were three levels. Level 1 was considered minimum quality and ranking was determined by the percentage of the indicators attained. Attainment of 85 percent of indicators was required for level 3, 75 percent of level 2, and 65 percent for level 1. District and provincial officials encountered challenges in implementing the independent monitoring of self-assessment completion. The project provided face-to-face training on the accreditation process in disadvantaged provinces but did not provided targeted technical or financial assistance. Schools assessed themselves annually and external monitoring occurred every five years. This process was meant to be technical support, rather than an inspection. District and provincial officials reviewed the self-assessments. An evaluator was required to visit the school to conduct the independent assessment and identify weaknesses. Some provinces did not allocate adequate funding for the 41 accreditation process. Some provinces also lacked the capacity to carry out a large number of accreditation visits annually. Schools needed to mobilize resources to improve the quality. Thus, there was no incentive or additional resources for provinces and districts to implement this new responsibility. Achievements were realized from the school self-assessments. By the end of the project, 90 percent of the schools had completed the self-assessment. The IEG mission was told that 96 percent of the schools had completed a self-assessment. As table A.1 shows, the physical conditions of preschool classrooms have substantially improved. About 97 percent of preschool classrooms are of acceptable (“so-so”) or good quality; only 3 percent are “not good.” Forty-one percent of preschools had level 1 accreditation by the end of the project. The target for DLIs was 40 percent. However, this average was not reached in all provinces, particularly disadvantaged ones. The goal is that 80 percent of schools will attain level 1 accreditation, as noted in the Early Childhood Education Strategy (2016–25). In the future, disadvantaged provinces will need more resources to increase the number of schools attaining level 1 accreditation. Table A.1. Rating of Conditions of Preschool Classrooms, 2014–17 (number of classrooms) Condition 2014 2015 2016 2017 Total (P+K) 160,335 166,243 161,790 170,995 Good 103,820 108,953 111,516 124,420 So-so 45,469 46,940 41,621 41,227 Not good 11,046 10,350 8,653 5,348 Source: Ministry of Education and Training. Note: P+K = preschool and kindergarten. ECE Center for Children (Birth to Age 3) There was a delay in the implementation of the ECE centers. After nearly two years of implementation (2015), there was no plan or implementation progress to develop the community-based centers. As a result, the World Bank encouraged cancellation of the funding from the ECE centers ($80,000) and suggested that it be applied to school accreditation, professional development, or developing the capacity of provinces, districts, and schools. In the end, MOET decided to implement four pilot centers in Bac Giang, Da Nang, Dak Lak, and Lào Cai. Despite the late start, MOET quickly developed operational guidance and trained staff, and construction or renovation was undertaken. IEG found issues with the implementation of the centers it visited. Demand for the services was poor in Dak Lak and Lào Cai, where assessments had not been done to identify the needs of children and parents. The centers did not have adequately qualified personnel. The buildings that IEG visited are now used for other purposes. 42 Outcomes All 63 provinces achieved universal preschool (for children aged five). MOET defines universal preschool for five-year-old children as 95 percent enrollment with 85 percent in a full-day program; attendance of 90 percent; and underweight and malnutrition present among less than 10 percent of children. More than 843,000 children enrolled in kindergarten between 2014 and 2017 (table A.2). The number of children in kindergarten increased from 3.7 million in 2014 to 4.6 million in 2017. The rate of growth was similar for ethnic minority children (16.7 percent) as the overall increase (17.7 percent). The rate of enrollment (part and full day) for five-year- old children increased during the operation from 96 percent in 2012 to 99 percent in 2016. Five-year-old children participating in a full-day program increased from 73 percent in 2012 to 88 percent in 2016. Enrollment has increased despite limited demand-side interventions, suggesting that they may not be needed in this context. Parents reported to the IEG mission that they felt it was their responsibility to send their child to school and that they had strong confidence in the school and the value of education. Preschool enrollment has increased by more than 46,000 (table A.2). The number of enrolled children aged three and four years increased from 661,877 in 2014 to 707,990 in 2017. The rate of growth is similar among girls and ethnic minorities. 43 Table A.2. Kindergarten and Preschool Enrollment, 2014–18 School Year Enrollment 2014–15 2015–16 2016–17 2017–18 Total P+K 4,416,852 4,627,316 5,085,635 5,306,536 Preschool Total 661,877 648,795 676,059 707,990 Girls 302,159 291,283 308,063 325,516 Ethnic 90,923 94,882 100,955 108,218 minorities Disability — — 1,417 1,586 Kindergarten Total 3,754,975 3,978,521 4,409,576 4,598,546 Girls 1,757,976 1,845,356 2,059,059 2,157,502 Ethnic 688,555 731,112 733,868 806,325 minorities Disability — — 7,826 6,946 Source: Ministry of Education and Training. Note: — = not available; P+K = preschool and kindergarten. The Early Development Index (EDI) examines five development domains: physical health and well-being, social competence, emotional maturity, language and critical thinking skills, and communication and general knowledge. The survey was completed by kindergarten teachers who answered questions about children’s ability to meet age - appropriate development achievements. The Offord Centre for Child Studies at McMaster University recommends following the broader group of “off track” (at risk and vulnerable) children or those scoring at the 25th percentile and below, as this group scored lower than expected for their age and is more likely to fall behind in primary school. Three rounds of the EDI survey were implemented, one each in 2012, 2014, and 2016. The initial sample was based on 8,400 children from 100 districts. Analysis of the final survey consisted of 7,564 children in 936 kindergarten classes of 400 preschools in 100 districts within 54 provinces. Findings from the initial and final survey are presented in the next paragraph. Although improvements in children’s readiness were found, almost a third of children remain off track. Twenty-nine percent of children were vulnerable or at risk of vulnerability (lowest two groups) in the final survey compared with 50 percent at baseline. This represents a decrease of 21 percent in at-risk or vulnerable children from the initial survey, which exceeds the low target of a decrease by 2 percent. However, nearly one-third of children may have difficulties in primary school due to their lack of readiness. Across surveys, the groups most likely to be vulnerable were ethnic 44 minorities, poor households, and mothers with primary education or lower. In each round, girls were less likely to be vulnerable. The Central Highlands region had the highest percentage of vulnerable children in each round. The rate of vulnerability among ethnic minority children was 45 percent in 2012 and decreased to 22 percent in 2016. The rate of vulnerability among Vietnamese-speaking children (nonethnic minority) decreased from 19 percent in 2012 to 9 percent in 2016. Results of the final EDI survey were analyzed to investigate factors contributing to decreased vulnerability. New elements were added to the final survey to look at school accreditation and teacher and managers and their participation in training modules. The title “preschool teacher,” training, and school conditions represented in the accreditation (such as teacher-student ratio, materials/toys) affect the readiness of children entering primary school. These findings need to be interpreted cautiously, since comparisons did not control for student characteristics. Efficacy is rated substantial. Efficiency Investments in ECE and development are cost-effective. One argument for investing in children from birth to age five is that gains in development lost at this critical period cannot be recouped (Heckman 2008). High-quality programming for ECE and development improves children’s cognitive, language, physical, and socio-emotional development and increase school readiness (Martinez, Naudeau, and Pereira 2012). Another reason these services are cost-effective is that they have an effect on medium and longer-term outcomes, such as learning in school (Belfield et al. 2006 and Heckman, Pinto, and Savelyev 2013); enhanced employment prospects, income, and labor productivity; and reduced criminal behavior. For children from poor families, these interventions can help break intergenerational poverty (Walker et al. 2013; Schweinhart 2007; and Gertler et al. 2013). The economic justification for the operation lay in the high returns to the individual and society from ECE. The economic rate of return was estimated to range from 14.3 percent to 16.9 percent with a 5 percent discount rate (World Bank 2013b). The updated analysis calculated an internal rate of return ranging from 12 percent to 23 percent (World Bank 2017a). The investment was estimated to yield $28 in benefit for every dollar spent from improved cognitive, socioemotional, physical, and language development and improved school progression, which would positively affect outcomes such as future earnings (World Bank 2013b). The project contributed to reducing the number of vulnerable students (by as much as half), as measured by the Early Development Index (EDI; World 45 Bank 2017a). However, about a third of children are vulnerable or at risk for vulnerability. The economic analysis was based on appropriate assumptions. Lunch subsidies for poor children were also expected to improve nutrition and promote learning, as well as remove an enrollment barrier. The kindergarten program was expected to improve school readiness and primary school learning, due to better teaching and management and more children participating in a full-day program. Educational quality was monitored through school self-assessments and validated at district and provincial levels. In this process, an independent evaluation is conducted every five years. At closure, the impact of the teacher training was estimated to range from 0.09 to 0.67 standard deviations increase in student test scores. This was calculated based on effect sizes established in prior meta-analyses (World Bank 2017a). Overall, the initial and updated estimate, as well as international evidence, suggest that this operation’s investment in ECE provided value for money. Other aspects also demonstrated project efficiency. This project provided on-budget financing based on DLIs. It disbursed rapidly—by early 2016 approximately $85 million had disbursed; the low targets could also be a factor for the fast disbursements. Since PMU staff have remained with the Early Childhood Department in MOET, capacities built during the operation have remained. The project implemented well, attained all DLIs, and ended without an extension. A minor weakness in project efficiency is the ECE pilot centers, which were a small portion of the loan ($800,000). The World Bank recommended cancellation of the pilot centers since this activity was very delayed, and a medium-term financial commitment would be difficult. The World Bank team believed that the time and attention could be better used to support the core activities of the project and better support provinces and district. However, it left the final decision to MOET on whether to proceed to create and implement the pilot centers. Ultimately, there were delays on the part of the government in approving procurement packages for the ECE centers, which added further delay to the activity and reduced implementation time. IEG’s mission identified issues with the continued operation and sustainability of these centers (see the Risk to Development Outcome section). Considering all of these aspects, the efficiency of the project is rated substantial. Outcome The overall outcome rating of the project is satisfactory. The relevance of objectives is rated high. The objectives were aligned with multiple government strategies and were 46 consistent with the Bank Group’s country strategies and partnership framework (World Bank 2012; 2018). Achievement of the objective was rated substantial. The outcome is attributable to the government’s broader program and policies, of which the World Bank financed a portion. Efficiency is rated substantial. This outcome rating is consistent with minor shortcomings in preparation, design, and implementation. Bank Performance Overall Bank performance is rated satisfactory. Quality at entry is rated satisfactory. Preparation was thorough and occurred over an extended period. Preparation spanned more than four years from project initiation (June 2009) to the internal decision review meeting (October 2012) and eventual approval by the Board (October 2013). This was the result of World Bank management who understood the rationale and benefits to develop a different approach to the support, which required an extended period. The analytical work and technical assistance laid a solid foundation for policy dialogue, project design, and the development of DLIs. During the extended preparation period, analytical work focused on early childhood development was completed. A conference on early childhood development policy was held, and policy was discussed with the government. The World Bank promoted the National Preschool Quality Standards as well as the government’s simultaneous focus on quality and access for preschool. Resources from the Russia Education Aid for Development Trust Fund were used in 2012 to implement the baseline assessment using the EDI. The analytical work clearly identified disadvantaged groups and barriers to enrollment. The Vietnam National Institute of Education Statistics conducted a social assessment that identified barriers to enrollment for ethnic minority children. The operation was built on lessons learned from the World Bank’s broader support in the education sector. The implementation arrangements were selected to build more sustainability within the government and facilitate faster implementation by eliminating parallel processing. It built on learning from the Primary Education for Disadvantaged Children Project to develop a school accreditation for the preschool level. It also sought to leverage findings from international experience such as regularly assessing outcomes and using a curriculum focused on child development. However, one lesson noted in the appraisal document and quality enhancement review built from international experience that was not adequately emphasized in the operational design was maximizing the involvement of parents in children’s learning. This would have had particular importance for children from disadvantaged areas or ethnic minorities. 47 Fiduciary, social, and environmental aspects were well prepared. This included proper assessments of policies, systems, and capacities to meet World Bank standards. The social assessment examined enrollment constraints of ethnic minority children and girls. Financial risks were appropriately identified and mitigated. For example, World Bank financial assessments found that financial management posed a limited risk to the project because the MOET had adequate arrangements to ensure the financing would be used for its intended purposes. Measures were established to address potential weak financial reporting capacity and internal controls of the spending units. Among the minor shortcomings of the project were the measures for strengthening technical capacities for the provincial and district levels, which may not have been adequate. Uneven provincial capacity was expected to be addressed by a new reporting system and technical assistance funded through the project. Provincial capacity clearly was heterogenous, yet the approach and DLIs may not have provided adequate incentives to differentiate support to provinces and districts. Considering the newness of the DLIs in Vietnam, there were some quick wins to build confidence in the instrument and disburse easily. Quality of supervision is rated satisfactory. The World Bank team provided active implementation support. International experts accompanied the supervision missions, providing technical advice related to preprimary education, in-service training, and preschool accreditation. Supervision reports and aide-mémoire provide clear and candid assessments and demonstrate that the World Bank team focused on monitoring the development impact, not just implementation of activities. For example, the World Bank encouraged MOET to plan for additional targeted actions for disadvantaged and underperforming provinces in relation to school accreditation and teacher training at the end of the operation and beyond the project. The World Bank continued discussions with key actors in MOET about early childhood development policy. These discussions supported the government’s updated Early Childhood Education Strategy (2016–25). This strategy expanded its focus to include children from birth to age five. The World Bank arranged study tours for staff from MOET to learn what other countries do to support the development of children (birth to age five). The World Bank team encouraged the government to present findings from the EDI interim reports at an early childhood development conference and engage other donors who support early childhood development. The World Bank focused its attention on monitoring and evaluation data and monitoring of DLIs. Solutions to remedy the issues with the second round of the EDI survey were provided (see the Monitoring and Evaluation section). Since most of the 48 resources were tied to DLIs, there was a limited procurement role, which ran smoothly. However, there were delay related to the ECE centers (see the Efficiency section). Although implementation generally was smooth, the World Bank team proposed actions to resolve problems when issues were detected during supervision missions. For example, when district accounting failed to detect errors in the financial accounting at preschools, the World Bank flagged the issue and proposed solutions to provide improved guidance to DOET, the Bureaus of Education and Training, and preschools. Unqualified audits resulted from incorrect budget codes used by preschools and ineligible budget expenses such as nonteacher salaries. The district offices did not detect these accounting errors. Because of this recurring issue, the World Bank recommended that MOET rely solely on the reimbursement of lunch subsidies in its reporting of eligible expenditures. This was recommended because the amount provided by MOET was more than sufficient and the budget code issue would have required a regulation on the part of the Ministry of Finance. The final independent audit of the off-budget support concluded that the project financial reports provide a true and fair account. Risk to Development Outcome The risk to development outcome is rated low. A major factor is the continued government commitment to ECE. It is evidenced by a revised ECE strategy and additional decrees in preprimary education that are implemented by provinces and districts. The government has sustained its focus on universalizing enrollment of children age five (kindergarten) and is moving to universalize enrollment children age four. Other factors also mitigate the risk to maintaining activities supported during this operation and the development outcomes. All of the activities continue to be funded as part of the MOET and provincial budget, such as salaries for preprimary teachers and lunch subsidies for disadvantaged children. E-learning modules and teacher meetings continue; teachers are required to have a program of self-learning. School self- assessments have continued and schools are expected to complete the assessments and meet the standards to attain level 1. A threat to the development outcome is adequate planning for increasing enrollment for children aged three and four years (consistent with the ECE strategy) in relation to space and financial resources. Existing preschools may not be able to accommodate the additional children, given existing classrooms and land. Limited space was a constraint shared by preschool managers during IEG’s mission. Additional efforts are needed for disadvantaged children and districts, which may require more resources. MOET has issued guidance to increase the required level of education to college for preprimary 49 teachers. This will have budget implications, since it will increase the pay for these teachers. The four pilot ECE centers have low sustainability. Considering these factors, the risk to the development outcome is rated low. Monitoring and Evaluation M&E design. The appraisal document provided a detailed plan for M&E. It defined the indicators and specified baseline data. The results framework included both monitoring indicators and DLIs. Performance indicators included measures of outcomes and outputs. However, the indicators would not help MOET assess the effectiveness of its ECE policies or interventions in relation to school self-assessment, lunch subsidies, or training. The results framework did not specify disaggregated indicators, consistent with the focus on vulnerability. However, the reporting systems of MOET regularly collects disaggregated data by province, gender, ethnic minority, and, recently, disability. DLIs included a mix of outputs and outcomes. Some DLIs reimbursed the government for work previously done to develop quality standards and professional development or produce a baseline enrollment report. Others were meant to disburse over the course of the operation. The target for increases in enrollment was low. The DLIs related to accreditation and professional development tracked the number that completed training and self-assessment. M&E implementation. Data were collected regularly and shared with the World Bank. Monitoring of DLIs was done by the Vietnam Institute of Educational Sciences, which conducted verification in 10 provinces several times between 2013 and 2016. These audits found minor inconsistencies and errors in reporting. Some respondents noted that the scope of monitoring was large compared with the allocated resources. Implementation of the EDI was supervised by MOET and PMU and implemented by the DOETs. Three rounds of the EDI were conducted, one each in 2012, 2014, and 2016. There were issues with the second round of the EDI. The World Bank made several recommendations to PMU to ensure improvement in the third round of data collection. A consultant with strong research experience oversaw the final survey. There was improved training for teachers who completed the survey. An international consultant monitored the quality of data collection and analysis. Variables were added to the final round of the survey to identify factors about the school, teacher, or school managers that may mitigate children’s vulnerabilities. However, analysis with the new variables did not control for student characteristics. 50 M&E use. MOET reviewed the indicators. In addition, analysis of the EDI instrument provided policy maker evidence of continued vulnerabilities and improvements in school readiness. Data has also been used with MOET and the government to help justify its continued investment in young children. The quality of the M&E is rated substantial. References Gertler, Paul J., Rodrigo Pinto, Arianna Zanolini, Christel Vermeerch, Susan Walker, Susan M. Chang, and Sally Grantham-McGregor. 2013. “Labor Market Returns to Early Childhood Stimulation: A 20-Year Follow-Up to the Jamaica Study.” Policy Research Working Paper 6529, World Bank, Washington, DC. Martinez, Sebastian, Sophie Naudeau, and Vitor Pereira. 2012. The Promise of Preschool in Africa: A Randomized Impact Evaluation of Early Childhood Development in Rural Mozambique. Washington, DC: World Bank. Schweinhart, Lawrence. 2007. “Outcomes of the High Scope Perry Pre -School Study and Michigan School Readiness Program.” In Early Childhood Development: From Measurement to Action, edited by M. Young and L. Richardson, 87–103. Washington, DC: World Bank. Walker, Susan, Susan Chang, Amika Wright, Clive Osmond, and Sally Grantham-McGregor. 2013. “Do Early Childhood Experiences Affect Development in the Next Generation? The Jamaica Intergenerational Study.” Paper presented at the Inter -American Development Bank conference, “Early Childhood Development in the Caribbean: Identifying Cost Effective Interventions,” Kingston, Jamaica, November 14. World Bank. 2011. Vietnam—Country Partnership Strategy for FY12–FY16 (English). Washington, DC: World Bank. http://documents.worldbank.org/curated/en/849061468133768943/Vietnam-Country- partnership-strategy-for-the-period-FY12-FY16. ———. 2013a. Vietnam—Credit Number C5207-VN Financing Agreement. Washington, DC: World Bank. ———. 2013b. “Vietnam—School Readiness Promotion Project.” Project Appraisal Document 65942-VN, World Bank, Washington, DC. ———. 2017. Vietnam—Country Partnership Framework for FY18–22. Washington, DC: World Bank Group. 1 Approximately $5. 2 Seehttps://www.naeyc.org/accreditation . 51 Appendix B. Vietnam Escuela Nueva Project Ratings Table B.1. Global Partnership for Education—Vietnam Escuela Nueva Project (P120867) Indicator ICR* ICR Review* PPAR Outcome Satisfactory Satisfactory Satisfactory Risk to development outcome Modest Modest Modest Bank performance Satisfactory Satisfactory Satisfactory Borrower performance Satisfactory Satisfactory Satisfactory * The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible World Bank Global Practice. The ICR Review is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. Note: PPAR = Project Performance Assessment Report. The project development objective (PDO) for the Global Partnership for Education—Vietnam Escuela Nueva Project (P120867) as stated in both the grant agreement and the project appraisal document was “to introduce and use new teaching and learning practices in the classroom targeting the most disadvantaged groups of primary students” (World Bank 2012, 3). One level 2 restructuring in 2015 did not involve any changes to the PDO. The restructuring was undertaken for two reasons: first, to move funding for school equipment from category 2 (school level) to category 1– (central level–) type expenditure in recognition of the fact that school equipment is procured at the central level (funding with which to procure school equipment had been erroneously placed under category 2 (local level); and second, to ensure that eligible expenditures for “equipment” to schools could be financed out of category 1, the description of category 1 was revised to include expenditures covered under part c of the project. A total of $10,868,100 was duly reallocated from category 2 to category 1 expenditure (World Bank 2012, 5). Relevance of the Objectives The Vietnam Escuela Nueva (VNEN) project was introduced into an already evolving education reform, with a particular emphasis on pedagogical renovation. The overall objectives of the Education Development Strategic Plan (2003–20) were to renovate the education system, move closer to regional and international quality standards, and ensure social equity and lifelong learning opportunities. The Strategic Plan also focused on nine strategic solutions aimed at renovating education management, expanding the education institution network, developing human resources, and strengthening investment and training. As part of its national education plans, Vietnam also had an Education for All Action Plan for 2003–15, which was updated in 2012. Resolution 29–NQ/TW of Party Plenum XI (2013) oriented the education system away from a focus on content and toward a focus on competency and quality. The Resolution requires that general education focus on developing intelligence, physical well-being, forming proper 52 citizenship qualities and competencies, promoting talents, and providing career orientation for students. In response to Resolution 29, MOET proposed the “Reform of General Education Curriculum and Textbooks after 2015,” which would result in a new general curriculum and new or substantially revised subject-specific curricula, textbooks, teacher guides, and other pedagogical materials. Vietnam’s commitment to education reform yielded impressive results and helped identify ongoing challenges. Vietnam has high primary school completion rates, strong gender parity, low student-teacher ratios, and a low out-of-school rate. Despite these achievements, Vietnam remains challenged on access (particularly in remote areas, and for ethnic minorities) and quality. Part of the challenge facing the education system concerned an excessive focus on content learning at the expense of inculcating competencies and soft skills such as critical analysis, problem solving, and communication skills. The VNEN project addressed factors such as pedagogical practices, school resources, school management that could support higher and more equitable educational performance as identified in Vietnam: High Quality Education for All By 2020 (World Bank 2011a). Regarding pedagogical practices, the study notes that teacher feedback was found to be the strongest predictor of student achievement differences in remote schools, and that homework and viewing pictures and maps were found to lower achievement inequality among students. These findings emphasize the benefits of interactive, student-centered pedagogical approaches over more traditional didactic and frontal approaches. Research and analysis conducted for the study also made it clear that the overall effect of full-day schooling in Vietnam is a combination of several elements such as better physical and human resources and more time to learn. The VNEN project also supported the elaboration of full-day schooling, particularly in remote areas and for children from ethnic minority group backgrounds. In transitioning to full-day schooling, the study recommended the introduction of more effective pedagogical practices that would, given additional time, involve more practical and interactive teaching, more interaction with parents, and more principal-teacher interaction. The study also suggested that other beneficial factors, such as homework and working in groups, could be accommodated under full-day schooling and new pedagogical approaches, and emphasized the importance of building in cultural sensitivity to teaching practices for ethnic minorities. The VNEN project was and continues to be well aligned with Bank Group country strategies. The project was well aligned with the third pillar (Outcome 3.2: Improved Basic Public Service Delivery and Access) of the Bank Group’s Country Partnership Strategy for Vietnam 2012–16 (World Bank 2011b). The project remains relevant to the Country Partnership Framework for Vietnam 2018–22, as part of which the World Bank continues to provide support “to improve the general education system to better equip students entering the higher education through reforming teaching methodologies, curriculum development and increased hours of instruction and also targeting Ethnic Minority children” (World Bank 2017b, 77).1 53 Finally, the World Bank has provided sustained support to the primary education sector over time. World Bank support to primary education in Vietnam predates and persists after the VNEN project. Hence, VNEN can be seen as one component in a strategically focused medium- term engagement in partnership with the Vietnamese government in support of progress and reform in basic education. Ongoing projects that draw on aspects of the VNEN innovation include the Vietnam Enhancing Teacher Education Project (P150060) and the Vietnam Renovation of General Education Project (P150058), The relevance of objectives is rated substantial. Relevance of the Design Project design was largely robust, facilitating the implementation of an ambitious and extensive program of pedagogical change, at scale, over a three-year period. Project design incorporated lessons learned from previous projects that sought to introduce active learning and child- centered pedagogies. The project appraisal document lists projects supported by the World Bank and other donors in this regard including, for example, the Primary Teacher Development Project (2002–07) that was jointly supported by the International Development Association and the United Kingdom Department for International Development (World Bank 2012, 16). Lessons learned from past projects that were applied in the design of the VNEN project included the following: • Focused scale—the need to focus on pedagogy and to implement at scale sufficient to develop a critical mass of change agents • Comprehensive package—the need to provide an integrated package of supports (for example, special-purpose didactic materials for students, training for teachers, and continued school-level support of trained teachers) to ensure effectiveness • Dedicated knowledge management—the need to produce and disseminate knowledge about the new pedagogical methods • Global partnership—the need to absorb inputs beyond financial including, for example, from international experts, the Colombia experience, and the larger Global Partnership for Education (GPE) partnership Project design was informed by a successful pilot of the Escuela Nueva model that was implemented in 2010–11. The pilot was carried out under the Primary Education for Disadvantaged Children Project in 48 grade three classrooms in 24 schools across 12 districts in six provinces with large ethnic minority populations. The pilot demonstrated that the model could be readily adapted to Vietnam. The pilot also provided a “head start” for the project proper as materials were developed for grades one and two that were later used in the project rollout. The pilot identified an appetite for change and for the new approach. For example, some classrooms and schools not directly supported by the pilot adopted the approach, something that occurred again during the scaling up of the project under VNEN. The pilot also demonstrated the 54 need for careful and dedicated attention supported by clear communication and the desire among trained teachers for ongoing support (for example, a support group including other teachers; World Bank 2012, 26). The project was supported by a clear statement of objectives, citing a desire to introduce and use new teaching and learning processes. Project design took into consideration comments from the peer reviewers during the concept and appraisal review meetings, including advice to simplify the PDO to focus on tangible and demonstrable objectives; hence the focus on the introduction and use of the new teaching materials (World Bank 2016b, 15). Project objectives were supported by components and activities (for example, training) that focused on pedagogical reform and the provision of relevant supports to ensure effectiveness (for example, school and classroom learning materials). Through this process, project design took heed of lessons learned to maintain a focus on pedagogical change. The simplification of the PDO brought focus and realism to the overall effort, especially given the short implementation period. The statement of project objectives was framed at the level of intermediary outcomes reflecting the three-year implementation period. The realism of this approach notwithstanding, the causal chain that underpinned project design could have included a stretch element to embrace change in educational outcomes, especially given the decision to carry out an impact evaluation (which was also introduced in response to peer review of project design). Ultimately, the results framework included output-oriented indicators that could be measured within the three-year implementation period. Thus, the project was simply and conservatively designed to counterbalance its more experimental intent and the desire to produce demonstration effects that could both deepen and accelerate the strategic renovation of basic education in Vietnam. Although backed by intent within MOET for nationwide reach, the design was realistic within the project’s parameters (financial resources, human resources, and time). The Escuela Nueva model was scaled up from its pilot inception to all 63 provinces with a heavy emphasis on the 20 provinces with the most disadvantaged school-going populations. A well-thought-out approach to targeting ensured that the majority of the project was implemented in schools likely attended by ethnic minority and otherwise disadvantaged children. Projects in all provinces were intended to have a demonstration effect (which, as elaborated in the Efficacy section, resulted in a large number of schools voluntarily adopting the VNEN approach with support from MOET). The design also included retroactive financing that allowed the MOET project management unit (PMU) to begin working on project implementation activities before effectiveness. The effect of this design feature was to mitigate some of the implementation risks associated with the three- year implementation period. The design included a modified financial management arrangement so that grant funds would be sent from the center to a provincial bank account (rather than through the budget) and then directly to school bank accounts. This was a new and 55 appropriate design feature but was also identified as a risk because the schools, with limited experience and capacity, would then be responsible for reporting expenditures. Mitigation measures were identified and implemented as designed. Project implementation was conducted through the government education administrative structure and included a PMU located within MOET with personnel that were line staff of the ministry, provincial project management units to support training, school-level activities, and the distribution of school grants directly to school bank accounts. All these design features were appropriate and contributed to the successful implementation of the project. Other noteworthy aspects of project design included a study tour to Escuela Nueva Foundation in Colombia for senior MOET officials to learn firsthand about the program; a conscious and structured approach to scaling up that was underpinned by an ambition to have the VNEN approach, in whole or in part, mainstreamed into basic education; an impact evaluation that collected three rounds of data on five stakeholders (students, parents, teachers, principals, and schools); and significant hands-on technical support that included classroom observations, interviews, and other exchanges with principals and teachers. The relevance of design is rated substantial. Efficacy The PDO for VNEN was “to introduce and use new teaching and learning practices in the classroom targeting the most disadvantaged groups of primary students” (World Bank 2012, 3). As noted in comments about project design, the chosen indicators (both at the PDO and intermediate level) were both simple and few. There were three PDO indicators, all of which were essentially output oriented, and four intermediate-level indicators. Target values for all three PDO indicators were met or exceeded (table B.1). The target value for the second indicator (number of VNEN learning, teacher, and Teacher Training Institute guides) was exceeded by 472 percent. The Implementation Completion and Results Report attributes this to strong MOET capacity and technical assistance in VNEN material development (World Bank 2016b); however, while both MOET capacity and technical assistance were strong relative to the baseline value achieved during the pilot, the original target looks particularly low, which probably better explains the level of performance realized. The target value for the third PDO indicator (number of primary education teachers and education administrators completing VNEN training) was exceeded by 76 percent, an achievement that was attributed to the modified cascade training that allowed for more teachers to be trained at school level. Clear criteria and robust data were used to underpin the targeting and identification of schools in priority areas (priority 1 provinces) with the most disadvantaged children. The program was also introduced, on a lesser scale, to priority 2 and 3 provinces in the expectation that it would 56 be adopted by other schools nationwide. The scaling up reached 446,781 students in grades two to five, of which 352,956 (approximately 79 percent) were located in 20 priority 1 provinces. Table B.1. Baseline, Target, and Actual Values for PDO Indicators (number) Indicator Baseline Value Target Actual Students participating in VNEN 1,400 400,000 446,781 VNEN learning guides, teacher guides, 8 25 143 and TTI guides developed Primary education teachers and 100 30,000 52,792 education administrators completing VNEN training Source: Implementation Completion and Results Report. Note: PDO = project development objective; TTI = Teacher Training Institute; VNEN = Vietnam Escuela Nueva. The target values for the four intermediate indicators were met or exceeded (table B.2). The Implementation Completion and Results Report attributes additional workshops (target number was 20, actual number reached 63) to a highly participative and collegiate engagement of all relevant stakeholders, including teachers, administrators, and technical experts (World Bank 2016b). A high level of collegiality among stakeholders—national and local administrators, school principals, teachers, and parents—remained evident in the three field visits to VNEN schools. The Primary Education for Disadvantaged Children pilot was scaled up and introduced to 1,447 schools, of which 1,143 (approximately 80 percent) were in 20 priority 1 provinces. Given healthy progress (and the ambition of the key champions of the approach in the MOET), in December 2013 MOET indicated that it would expand the pilot to grade six. VNEN was ultimately introduced to 1,214 lower secondary schools. The project supported the development of grade six materials while all of the training and provision of materials was supported through the exchequer. Approximately 133,681 grade one students in VNEN schools also received materials for intensive Vietnamese.2 Furthermore, in addition to the 1,447 schools directly supported through the project, a significant number of schools voluntarily opted into the VNEN model. Between the 2014 and 2016 school years 452,255 students were exposed to the VNEN approach in voluntary schools that were not directly supported under the project (MOET data). This is indicative of the popularity of VNEN and a general appetite for change; however, the rapid expansion of the program beyond the parameters of the project caused problems that are referenced elsewhere in this report. 57 Table B.2. Baseline, Target, and Actual Values for Intermediate Indicators (number) Intermediate Indicator Baseline Value Target Actual Domestic workshops held for material 2 20 63 development Schools receiving full complement of 24 1,447 1,447 VNEN materials for each successive grade Schools receiving VNEN campus grants 0 1,447 1,447 Qualitative study of classroom 0 Study completed by Study completed 2014 observations comparing VNEN and non- project close VNEN classrooms (May 30, 2016) Source: Implementation Completion and Results Report. Note: VNEN = Vietnam Escuela Nueva. The impact evaluation reported findings of a positive impact using a rigorous procedure that followed a cohort of students from grade three through grade five. The evaluation measured the extent to which students acquired cognitive and noncognitive skills. By comparing results from a randomly chosen sets of VNEN schools and traditional schools (the counterfactual), the evaluation found the following: • VNEN students perform as well as or better than students educated under the traditional, frontal and didactic regime for Vietnamese language and mathematics; students from VNEN schools already showed a higher mean score at the baseline of the study, which (due to early implementation delays) was 18 months after the initiation of the project. The evaluation notes that this difference was retained throughout the years, though it narrows over the last two years of the program. Regression analysis shows a statistically significant effect of the VNEN program, with some specifications showing effect sizes of about one-fifth of a standard deviation, about 15 points in Vietnamese and 18 points in mathematics.3 • The program had a positive effect on the socioemotional skills of children enrolled in program-supported schools (this finding was arrived at using parent feedback regarding the behavioral development of their children over the three-year development of the VNEN program). The evaluation found that students in the VNEN program did better than their counterparts in traditional schools and that VNEN students did better at the lower end of the distribution, where there is a greater concentration of disadvantaged groups that are often a special focus of educational policy makers. The achievement of objectives is rated substantial. Efficiency A significant level of efficiency was demonstrated in implementing the VNEN project. A six- month delay in effectiveness was associated with signing the grant agreement and confirming 58 the designated account agreements with the commercial bank in charge of distributing the grants. Nonetheless, the project was almost fully disbursed and was fully implemented within the planned three-year cycle and planned budget, a considerable achievement given the scale and innovative nature of the project. Implementation efficiency was attributable to factors that included buy-in at senior levels in MOET and a tight, professional PMU. Senior-level buy-in was a striking feature of the overall effort. The symbiotic relationship between the management structures of MOET and the PMU also ensured efficiency. Relevant staff—for example, project management, financial management, and procurement—were familiar with World Bank procedures. Other aspects of the project were efficiently delivered including the teacher training modality (cascade approach) and the meal allowance procured under the project at relatively low cost ($129 compared with a government standard unit cost of $200). Project documentation provides robust analysis and assessment of efficiency. Annex 7 of the project appraisal document presented a detailed economic and financial analysis in support of the project (World Bank 2012, 60). Its analytic findings on the labor market returns to schooling and cognitive achievement using the Vietnam Household Living Standard Survey 2006 showed significant returns of wage earning to schooling as well as cognitive skills. An additional year of schooling implied an 8–9 percent rise in hourly wages, and a one standard deviation increase in math score is associated with a 7–8 percent higher wage. The analysis suggested that focusing on the quality of education or the actual learning of students is vital: • Annex 3 of the Implementation Completion and Results Report updated the above analysis, presenting a cost-benefit analysis of the project estimating private monetary returns to schooling based on benefits from similar education interventions, project costs, and projections on earning differentials for the beneficiaries (World Bank 2016b). The analysis found statistically significant estimated impacts of the interventions ranging from 0.117 to 0.383 standard deviations increase in student test scores, in line with results from other studies in the literature. Drawing on other research into the link between earnings and school quality, as measured by cognitive skills attainment, the authors estimate a 2.4–7.7 percent increase in earnings as a result of the VNEN project (Patrinos and Psacharopoulos 2010). Within the range of estimated percentage increases in earnings based on the returns to individual project components, the VNEN project yields an internal rate of return of between 17 and 25 percent and a net present value of between $973,135,542 and $3,358,080,864. Even under the worst-case scenario, which assumes no increased earnings for cohorts 3 and 4 due to the comparatively short exposure to VNEN and only 2.4 percent increased earnings for cohorts 1 and 2, the project still yields an internal rate of return of 14 percent and net present value of $521,507,528. 59 • The impact evaluation collected detailed data on teaching and learning processes rather than data on costs. However, the authors suggest that the lack of precise cost data is not problematic because the operating costs of the program are substantively the same over the treatment and control group—that is, no change in the student-teacher ratio and learning guides simply replacing textbooks. The impact evaluation notes that the program cost approximately $85 million and benefited about half a million children over four years, about $40 per student per year. The average per-student expenditure for primary education is about $1,000 per year (purchasing power parity). The main element of fixed costs was for developing the teaching method and materials; the marginal cost of reaching additional students was the same as any other public school in Vietnam. The impact evaluation states that, considering the effect on test scores and noncognitive development, and that the program was neither more nor less expensive than traditional teaching, the overall conclusion was one of positive impact. Based on the evidence, project efficiency is rated substantial. Outcome The overall outcome rating of the project is satisfactory. The relevance of both the project objective and design are rated substantial, reflecting, in the first instance, close alignment with the trajectory of identified sectoral needs regarding education reform as well as the introduction and application of an innovative pedagogical approach to demonstrate a new teaching and learning method relevant to sector needs. Based on analysis and available evidence, the project efficiency rating is substantial. Finally, overall efficacy is rated substantial. All PDO indicators were met or exceeded, since the bar for some indicators was low. Risk to Development Outcome Overall implementation risk for VNEN was rated moderate in the project appraisal document (World Bank 2012, 12). That assessment rated stakeholder, social and environmental, and program and donor risks as low. Risks associated with governance, design and delivery monitoring, and sustainability were rated moderate. Risk associated with capacity was rated substantial, reflecting existing limited capacity at the school level to implement the project. Separately, financial management risk was rated substantial, given weak capacity at the school level to manage grant funds, and the risk rating for procurement was also rated substantial, given past performance of projects in the sector. Perceived risks, particularly regarding aspects rated substantial, were mitigated by factors that included strong leadership, high levels of organization, and ownership of the project at all levels; a symbiotic relationship between MOET and project management; experienced and qualified procurement and financial management personnel; prudent use of regional 60 consultants through the provincial offices to monitor grant distribution; and progress at the school level. VNEN generated significant support and interest throughout the majority of the project life cycle. For example, by December 2014 the communications strategy had generated over 210 news articles national and local papers. The VNEN website, which was developed as an educational and communication tool, was used by teachers, administrators, and communities— between its launch in November 2013 and project close the website had been visited by 3,646,844 users. As noted earlier, the positive image of the VNEN attracted nonproject schools even in the early days (for example, by December 2014, approximately 1,000 volunteer schools nationwide had started to apply the VNEN model). In August 2014, the government issued Circular 30/2014/TT-BDG to introduce a new primary student assessment paradigm based on the VNEN model using a portfolio-based formative student assessment. Video-based teacher training materials were incorporated into broader teacher training—that is, into the pedagogical preservice training program. However, and despite the successes detailed above, a number of factors came into play that have weakened the chances of the further integration of the VNEN model as a whole into the broader basic education system in Vietnam. While it is true that the philosophy of a child- centered approach to pedagogy has been firmly rooted in educational reform, a number of factors discussed in the main body of this report combined to limit potential. These included the coincidental retirement of senior champions from MOET at project close; a change at the ministerial level at MOET; and bad publicity associated with the voluntary, and hence under- resourced, uptake of VNEN in a significant number of primary and lower secondary schools. The continuing risk to development outcome is rated modest. Successful implementation of the project meant that at least part of the VNEN approach has filtered into ongoing education reform in Vietnam and is evident in ongoing World Bank–supported projects (for example, Vietnam Enhancing Teacher Education Project [P150060] and Vietnam Renovation of General Education Project [P150058]). As discussed earlier, the Project Performance Assessment Report mission learned that there has been some negative reaction to the approach that threatens the extent to which it will be more generally applied as part of the education reform process. The risk to development outcome is rated modest. Bank Performance Quality at entry is rated satisfactory. The World Bank team provided varied and in-depth support at entry to ensure a well-planned project. The World Bank had a long-standing engagement with the government of Vietnam and other donors in support of basic education. Over time and with the provision of sustained 61 support, strong and trusted relationships had been built between World Bank personnel and senior government officials; this provided a solid foundation for the more experimental approach that was scaled up under the VNEN project. Early aide-mémoire detail close, collaborative working relationships, identification of challenges, and problem solving. Throughout preparation the World Bank worked closely with and took on-board comments from the United Nations Educational, Scientific, and Cultural Organization, which was designated as the GPE coordinating agency, and the local Education Sector Group of development partners. In March 2012 the proposed project went through a GPE quality assurance review that provided guidance on project design with a particular focus on sustainability. The team also took advice from the World Bank’s quality review team with particular reference to properly aligning the results framework with the PDO. The project team focused project design on the scaling up of the Escuela Nueva approach to pedagogical practice that had been piloted under the Primary Education for Disadvantaged Children Project. Project design took into consideration comments from the peer reviewers during the concept and appraisal review meetings. In the first instance, this involved simplifying the PDO to keep a focus on the introduction and use of the new teaching materials. In recognition of the relative simplicity of the PDO and associated indicators, design included an impact evaluation that was intended to deliver a formative assessment of the project. As such, project design focused on rolling out the approach in support of an expected demonstration effect on the benefits of a more participative, child-centered pedagogy that would complement and support important systemwide curriculum reform efforts that were to be rolled out in 2015. The quality of supervision is rated satisfactory. The team systematically documented support provided and project progress in aide-mémoire, back-to-office reports, and Implementation and Status Results Reports (ISRs) and provided ongoing support to the client to ensure successful project implementation. Aide-mémoire and back-to-office reports provide a close narrative of the evolution of the project and the manner in which various challenges were identified and tackled over the project life cycle. The World Bank fielded six review and implementation support missions, three of which were jointly undertaken with the United Nations Educational, Scientific, and Cultural Organization, including a midterm review, through which the project team actively supported the government in project implementation. Reporting was candid in relation to challenges that faced the project. For example, the first ISR (April 2013) stated that “the project has suffered on consequence of the long delay between start of the project activities on July 1, 2012 and the first disbursement of $13 million on February 5, 2013 (a period of 7 months)” (World Bank 2013c, 1). The World Bank team consistently reported on financial management and procurement matters and, as attested to by interviews conducted for this assessment, provided valuable advice and support to ensure smooth implementation of what was an ambitious scaling up effort. 62 No safeguards issues arose during the planning or implementation of the project. The project was considered to be of low social and environmental risk and was rated a category C operation; only small repairs to schools were envisaged that would have no negative impact and no need for environmental mitigation measures. The indigenous peoples safeguard, OP/BP 4.10, was triggered. However, it was anticipated that the project would have no adverse impact on ethnic minority groups. On the contrary, there was a focus on poor and ethnic minority regions and the project sought to increase accessibility to high-quality education services for students and teachers in the project locations. The ICR confirms that the first year was devoted to teacher training and the development of didactic material and that intensive Vietnamese language learning in grade one was provided in the pilot schools so that students entering grade two would be able to read and follow written instructions (World Bank 2016b, 15 [footnote 6]). The ICR does not elaborate on the extent to which this practice, or other practices specific to ethnic minority groups, was observed in the rollout of the scaled-up efforts; however, interviewees (teachers, parents, school principals) confirmed this to be the case and the Independent Evaluation Group (IEG) mission observed certain practices (such as clustered/group seating) in action. That said, and as per the findings of the impact evaluation, the program was more closely observed in some places than in others. The overall Bank performance rating is satisfactory. Borrower Performance Given that the PMU was almost exclusively staffed by MOET officials—including up to the director level and with reference to overall financial management and procurement—who continued to execute their regular jobs in parallel with project management responsibilities, this assessment does not distinguish between government performance and implementing agency performance and arrives at a single, overall rating for borrower performance. The MOET PMU demonstrated a high level of commitment and enthusiasm for the project throughout its life cycle. This is evident in the successful completion of all activities, more or less full disbursement, and the meeting of all targets associated with an ambitious scaling up within a three-year period. The decision to set up the PMU within MOET and to staff it with, among others, high-level MOET officials, enabled a sharp focus on implementation that was conscious of the short implementation period. This decision also meant that any issues arising could be efficiently dealt with because all actors were working within a familiar system (and all had prior experience with World Bank projects). The commitment and enthusiasm of the MOET PMU was also evident in the manner in which the expansion of the VNEN approach was pursued—through active engagement with the media, encouragement for others to engage, and the use of its own resources to expand the range of materials available (for example, to grade six). 63 Despite early delays, the retroactive funding aspect of project design allowed MOET/PMU to forge ahead with preparatory actions (for example, developing learning guides). Once the project was ready to move past early delays, it was possible to do so at pace. Ultimately MOET/PMU delivered on a financial management system that provided timely and accurate information on project funds, collected data to enable regular reporting on project outputs (and contributed to data collection for the impact evaluation), and concluded more than 500 procurement contracts. Interim financial reports were submitted on time and the project audit reports were unqualified (World Bank 2016b, 23). Interviews conducted during the IEG mission with financial management specialists at MOET and the Word Bank confirmed the smooth running of the financial management arrangements. The PMU and the provinces hired the required financial management consultants. The PMU provided guidance to the provincial PMUs. School-level grant activities were monitored, and the required project accounting software was developed and installed to facilitate financial monitoring. The procurement expert at the World Bank’s country office noted the smooth approach facilitated by the common management team—MOET officials straddled the project and their regular jobs. This helped in progressing the project, which was particularly important given the three-year implementation period. The head of accounting for MOET, who also occupied the role of financial controller for the project, informed IEG that World Bank procurement requirements were relatively easy to manage and implement compared with national requirements that were stringent and required significant levels of consultation and cross- ministerial approval. There were few negatives associated with the performance of MOET PMU with regard to the management of the project. As noted earlier, effectiveness was delayed largely for technical reasons, although the situation was retrieved by the availability of a retroactive funding. Early ISRs note the challenges associated with poor planning and division of labor that negatively affected procurement compliance. Relevant interviewees at MOET noted that the challenges were associated with managing complex national requirements rather than in meeting World Bank procurement requirements. In any event, these challenges were addressed and the project ultimately delivered more than 500 procurement contracts in line with World Bank requirements. Overall borrower performance is rated satisfactory. Monitoring and Evaluation Monitoring and evaluation (M&E) design. VNEN was designed with three complementary components: the District Fundamental School Quality Level Audit (DFA), a video study, and an impact evaluation: 64 • Project preparation was facilitated by the availability of high-quality census data of primary schools, a series that dated back to 2004/05. Those data were intended to serve as the primary source for M&E.4 A Fundamental School Quality Level Input Index comprising 47 indicators that measured five main dimensions of school quality (for example, school organization and management, teaching staff, and infrastructure) was developed. M&E for the project would require modification of the DFA database to include variables specific to VNEN such as seating arrangements, availability of learning guides, and teacher training, as well as data on the number of ethnic children needing support in the Vietnamese language and the number of ethnic children who have parity in learning levels with Kinh children. • A video study would allow for video recording of sampled VNEN classroom periods that could be used to analyze, measure, and code teacher practices and classroom dynamics. • The impact evaluation would span the project life cycle, gathering data at the beginning and end through standardized testing for students from grades three and five. The evaluation would adopt a regression discontinuity design through which a cut-off point (threshold) constructed from the four priority tags used to identify eligible schools and subsequently identifying two groups of schools, the treatment group (just above the cut- off point for inclusion in the program), and the control group (just below the cut-off point and not included in the program). The evaluation would allow for comparison of outcomes for each group over time. The intent was to implement a test assessment on students in grades three and five at the start of the project in 2012 that would act as the baseline and would be followed up with another assessment on students in grades three and five in 2015.5 Overall, M&E design was multifaceted and robust, and was built using complementary approaches to data collection and analysis. The three data collection and analysis methods included in M&E design were developed and agreed in consultation with stakeholders. The basic data collection core of the M&E arrangements supported the simple, output-focused indicators at both the PDO and intermediate levels (which in turn reflected the short period to implement the project). These basic data were usefully supplemented by data from the video and impact evaluation exercises. M&E implementation. Data relevant to all PDO and intermediate indicators were collected despite changes in the implementation of M&E relative to the original design (for example, including a change in the approach to core data collection and a delay in implementing the impact evaluation). The changes in implementation of M&E from the original plan were as follows: 65 • Before project effectiveness (January 2013) MOET replaced the proposed DFA approach, a paper-based system, with the online Education Quality Management System. Data for the more efficient Education Quality Management System were collected three times a year. This system was also used to produce education statistics for the system as a whole, including data on learning and teaching facilities. Statistics relevant to the VNEN model were also collected—for example, frequency of VNEN training and teacher meetings and number of schools voluntarily implementing the VNEN model. • The start of the impact evaluation (published in 2017 after project close) did not commence as planned due to funding issues related to the analysis of any data that would be collected. The impact evaluation finally began when it was agreed that project funds would be used to support the evaluation process and funds secured from Dubai Cares (the first such funding for Vietnam) would finance the analysis of the impact evaluation. By that time, the VNEN program had already been in operation for a year. Consequently, it was not possible to undertake the planned random assignment of the program; instead, a propensity score matching method was used to determine the randomized sample of treatment and control group schools to be followed for two years. Data collection for the quantitative survey was done by a team of field investigators who were trained to use handheld electronic devices to record responses. Data were uploaded to a cloud-based repository for checking and quality control. The impact evaluation conducted survey rounds over three school years (2013/14, 2014/15, and 2015/16) with three comprehensive sets of data collected on five stakeholder groups— students, parents, teachers, principals, and schools. Tests were administered for mathematics and the Vietnamese language. In 2014, MOET worked with consultants to conduct video studies in 16 VNEN classrooms. The video studies focused on key VNEN aspects such as the development and application of skills among students, including leadership, self-management, teamwork, creativity, and communication. The studies also observed teacher-student and student-student interaction. M&E use. Data from the monitoring framework were used to inform ongoing project management and reporting on progress through, for example, project ISRs, of which there were six. The results of the impact evaluation are being used to inform further policy decisions related to the reform of basic education with a focus on curricular reform and pedagogy. In addition, the Research to Improve Vietnam’s Education System team will continue the project impact evaluation, collecting five more years of data from VNEN primary schools and expanding data collection to lower secondary schools applying the VNEN model. Following the completion of the video studies, MOET developed video teacher training that continues to be relevant. Material from the video study was incorporated into the summer 66 teacher training programs in June/July 2015. MOET has introduced material from the video studies into the pedagogical preservice training program. The quality of M&E is rated substantial. References ———. 2011a. Learning for All: Investing in People’s Knowledge and Skills to Promote Development—World Bank Group Education Strategy 2020. Washington, DC: World Bank. ———. 2011b. Vietnam—Country Partnership Strategy for FY12–16. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/849061468133768943/Vietnam-Country-partnership- strategy-for-the-period-FY12-FY16. ———. 2012. “Vietnam—Global Partnership for Education—Vietnam Escuela Nueva Project.” Project Appraisal Document 72285-VN, World Bank, Washington, DC. ———. 2013c. “Vietnam—Vietnam: Global Partnership for Education–Vietnam Escuela Nueva Project.” Implementation Status and Results Report ISR9427, World Bank, Washington, DC. ———. 2016. “Vietnam—Global Partnership for Education—Vietnam Escuela Nueva Project.” Implementation Completion and Results Report ICR3864, World Bank, Washington, DC. ———. 2017. Vietnam—Country Partnership Framework for FY18–22. Washington, DC: World Bank. 1However, the World Bank’s attention to the education sector in Vietnam has shifted toward the provision of support at the tertiary level and to ensuring that young people are prepared for the world of work. 2Material for grade one intensive learning in Vietnamese had already been financed, developed, and evaluated under the World Bank–supported Primary Education for Disadvantaged Children project, as was the material for grade two under the Vietnam Escuela Nueva pilot supported under the Primary Education for Disadvantaged Children Project. 3 The impact evaluation notes that these findings involved a number of technical issues related to measurement and statistical inference and will be presented for discussion and review in the academic literature. 4School principals completed the census and their entries were verified by the district and provincial education officials based on a seven-step process. 5 The design did not allow for the same students to be assessed at the beginning and end of the project but would enable comparison of different cohorts of students. 67 Appendix C. Basic Project Information Vietnam School Readiness and Promotion Project Project Cost Actual project cost for World Bank–financed activities: $95,191,137 versus the appraisal revised cost of $99,944,415. In effect, all project resources were disbursed, as the scope of the project was set by the available funding. No borrower contribution was expected or provided. World Bank Project Financing Country: Vietnam Project name: Vietnam School Readiness and Promotion Project Project ID: P117393 Implementation Completion and Results Report date: December 26, 2017 Original commitment: $100 million Actual amount disbursed or final: $95,191 million Financial source: International Development Association Environmental assessment category: C Table C.1. Project Dates Project Stage Expected Actual Approval February 28, 2013 February 28, 2013 Effectiveness August 31, 2013 July 26, 2013 Restructuring — — Midterm review — May 31, 2016 Closing June 30, 2017 June 30, 2017 Table C.2. Key Staff Responsible Sector Director/Global Project Stage Task Manager/Leader Practice Senior Director Country Director Appraisal James A. Stevens and Xiaoqing Yu Victoria Kwakwa Christian Bodewig Completion An Thi My Tran Jaime Saavedra Ousmane Dione 68 Vietnam Global Partnership for Education—Vietnam Escuela Nueva Project Project Cost Actual project cost for World Bank–financed activities was $84,288,432.89 versus appraisal cost of $84,600,000. In effect, all project resources were disbursed, since the scope of the project was set by the available funding. No borrower contribution was expected or provided. World Bank Project Financing Country: Vietnam Project name: Vietnam Global Partnership for Education/Vietnam Escuela Nueva Project Project ID: P120867 Implementation Completion and Results Report date: November 7, 2016 Original commitment: $84.600 million Actual amount disbursed or final: $84.288 million Financial source: Global Partnership for Education Environmental assessment category: C Table C.3. Project Dates Project Stage Expected Actual Approval March 1, 2012 July 31, 2012 Effectiveness — January 9, 2013 Restructuring July 15, 2015 June 28, 2015 Midterm review November 24, 2014 December 3, 2014 Closing May 31, 2016 May 31, 2016 Restructuring. The project underwent one level 2 restructuring, which had two purposes. The first purpose was to ensure that eligible expenditures for equipment to schools could be financed out of category 1 instead of category 2 “subgrants” by modifying category 1 description to include expenditures covered under “Part C” of the project. The second purpose was to reallocate $10,868,100 from category 2 to category 1 (World Bank 2013, 5). This was justified by the fact that school-level equipment is procured at the central level (financed from category 1 type financing), while category 2 financing was limited to school-level expenditures. The budget for category 2 was reduced from $43,800,000 to $32,931,900 and $10,868,100 was transferred to category 1. World Bank 2013 PROPOSED PROJECT RESTRUCTURING OF THE GLOBAL PARTNERSHIP FOR EDUCATION GRANT (TF013048). Author, Washington, DC. 69 Table C.4. Key Staff Responsible Sector Director/Global Project Stage Task Manager/Leader Practice Senior Director Country Director Appraisal Suhas D. Parandekar Luis Benveniste Victoria Kwakwa Completion Suhas D. Parandekar Harry Anthony Patrinos Ousmane Dione 70 Appendix D. List of Persons Interviewed Name Title and Institution Suhas D. Parandekar World Bank Senior Education Specialist Christian Bodewig World Bank Program Leader Keiko Inoue World Bank Program Leader Eduardo Vellez Bustillo World Bank Sector Manager An Thi My Tran World Bank Senior Education Specialist Rosfita Roesli World Bank Senior Education Specialist Marilou Hyson World Bank Early Childhood Development Consultant Wendy Jarvie World Bank Consultant Bui Hong Quang Director Second Secondary Education Sector Development Program (Asian Development Bank–funded) Nguyen Thi Huy Second Secondary Education Sector Development Program (Asian Development Bank–funded) Nguyen Van Hoai World Bank Procurement Pham Van Cung World Bank Financial Management Tran Thi Kim Lien Vietnam Escuela Nueva (VNEN) Chief Accountant; Managing School Readiness Promotion financing Ly Thi Hang Early Childhood Development/FD/Ministry of Education and Training (MOET)School Readiness Promotion Nguyễn Đại Dương School Readiness and Promotion Project (SRPP)(Dean of Quality Assurance Department, MOET) Nguyen Thanh Giang SRPP Đoàn Văn Ninh Renovation of General Education Project (RGEP) Director Le Ninh Ha Accountant VNEN and RGEP Nguyen Minh Thuyet Chief Editor of RGEP (before 2018) Dinh Quang Bao Chief Editor of RGEP (since 2019) Le Tien Thanh Former Director Primary Education for Disadvantaged Children (PEDC) and VNEN Chief Consultant Nguyen Vinh Hien Former Vice Minister Dang Tu An Former Director PEDC and VNEN Chief Consultant Nguyen Dinh Khue VNEN Training Consultant Pham Ngoc Dinh Former Director PEDC and VNEN Director Tran Phuong Anh Officer from Lào Cai Department of Education and Training (DOET) via Primary Education and Early Childhood Education (ECE) Hoang Thi Tham Officer from Lào Cai DOET via Primary Education and ECE Hac Xuan Phuong Officers from Thanh Hoa DOET via Primary education and ECE Nguyen Van Chieu Officers from Daklak DOET via Primary Education and ECE Nguyễn Hữu Độ Vice Minister 71 Name Title and Institution Trinh Thi Hoai Thu Deputy Director of PEDC Hoang Mai Le Staff, PEDC Nguyen Thi Hanh Staff, PEDC Vu Huyen Trinh Staff, Kindergarten Department Tran Dai Hai ODA Department Prof Le Hai Anh Deputy Dean of Department of Education, VNU UED Nguyen Thi Ban Director of Teacher Training Center, VNU UED Nguyen Trung Kien SRPP Coordinator Nguyen Thanh Giang SRPP Consultant for Training Toshi Yuki Matsumoto United Nations Educational, Scientific, and Cultural Organization Le Thi Binh United Nations Educational, Scientific, and Cultural Organization Le anh Lan United Nations Children’s Fund Provinces Thanh Hoa: Hac Xuan Phuong Staff of Primary Education of Thanh Hoa Department of Education; Lam Son Tow Kindergarten and Lam Son Town Primary School Daklak: Nguyen Van Chieu Director of Primary Education of Daklak Department of Education Practical Kindergarten of College of Kindergarten Teacher Education Nguyen Viet Xuan Primary School Lào Cai: Tran Phuong Anh, Primary Education of Lào Cai Department of Education Hoang Thi Tham Quang Kim Kindergarten Quang Kim Primary School 72