Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004579 IMPLEMENTATION COMPLETION AND RESULTS REPORT ON A SMALL GRANT IN THE AMOUNT OF USD 0.65 MILLION TO THE Globe International Center FOR Transparency and Accountability in Mongolian Education (TAME) (P150842) June 10, 2019 Education Global Practice East Asia And Pacific Region Regional Vice President: Victoria Kwakwa Country Director: Martin Raiser Senior Global Practice Director: Jaime Saavedra Chanduvi Practice Manager: Tobias Linden Task Team Leader(s): Rabia Ali ICR Main Contributor: Akiko Sawamoto ABBREVIATIONS AND ACRONYMS AFE “All for Education!� National Civil Society Coalition BB Bank Budget BTT Budget Trust Tool CPS Country Partnership Strategy CSO Civil Society Organization ECD Education and Culture Department EQRP Education Quality Reform Project GA Grant Agreement GIC Globe International Center GP Global Practice GPSA Global Partnership for Social Accountability GSST Good School Support Tool ICR Implementation Completion and Results Report IP Indigenous Peoples IRI Intermediate Results Indicator ISR Implementation Status and Results Report K&L Knowledge and Learning M&E Monitoring and Evaluation MECSS Ministry of Education, Culture, Sciences and Sports MNT Mongolian Tughrik MOF Ministry of Finance MTR Mid-Term Review PDO Project Development Objective PIU Project Implementation Unit PTA Parent Teacher Association RF Results Framework RETF Recipient-Executed Trust Fund TAME Transparency and Accountability in Mongolian Education TTL Task Team Leader WB World Bank TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 11 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 16 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .... 18 V. LESSONS LEARNED AND RECOMMENDATIONS ................................................................. 22 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 25 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 32 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 33 ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Transparency and Accountability in Mongolian P150842 Education (TAME) Country Financing Instrument Mongolia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Organizations Borrower Implementing Agency Globe International Center Globe International Center Project Development Objective (PDO) Original PDO The project's objective is to strengthen civic engagement in the education sector to improve transparency of budgeting and procurement processes for the delivery of better quality education services in western and central Mongolia. Page 1 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-18163 650,000 650,000 650,000 Total 650,000 650,000 650,000 Total Project Cost 650,000 650,000 650,000 KEY DATES Approval Effectiveness Original Closing Actual Closing 09-Dec-2014 09-Dec-2014 09-Dec-2018 09-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 13-Nov-2015 0.16 Change in Results Framework Change in Components and Cost Change in Disbursements Arrangements Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 11-Jun-2015 Satisfactory Satisfactory 0.16 02 15-Dec-2015 Satisfactory Moderately Satisfactory 0.16 03 29-Jun-2016 Satisfactory Moderately Satisfactory 0.33 Page 2 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) 04 05-Dec-2016 Satisfactory Satisfactory 0.33 05 10-Jun-2017 Satisfactory Satisfactory 0.52 06 08-Dec-2017 Satisfactory Satisfactory 0.52 07 26-Jun-2018 Moderately Satisfactory Satisfactory 0.62 08 01-Dec-2018 Moderately Satisfactory Satisfactory 0.62 ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Victoria Kwakwa Country Director: Klaus Rohland Martin Raiser Director: Daniel Dulitzky Practice Manager: Roberto Adrian Senderowitsch Tobias Linden Task Team Leader(s): Kathrin Frauscher Rabia Ali ICR Contributing Author: Akiko Sawamoto Page 3 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. The Transparency and Accountability in Mongolian Education (TAME) Project was initiated to strategically address Mongolia’s sectoral, institutional, and country contexts. Since the start of its transition from a centrally planned to a market-driven economy in 1990, Mongolia has experienced rapid economic expansion, which has largely been driven by the use and trade of its vast mineral resources. It has also made significant progress in reducing poverty, whose incidence declined from 38.8 to 21.6 percent between 2010 and 2014 (World Bank, 2016). In the education sector, Mongolia has undertaken a series of reforms, as exemplified in its massive changes from 2004–2015, during which time the education system was restructured from a 10-year (4+4+2), to an 11-year (5+4+2), and then to a 12-year (5+4+3) system. Since 2009, there has been steady annual growth of both the kindergarten-aged and school-aged populations, by 10–15 percent and 8 percent, respectively. Despite its economic and educational achievements, the country continues to face challenges and constraints in improving the quality of educational services delivery. Moreover, parents and the community overall are generally uninformed about school performance and have limited ability to hold key education stakeholders (including teachers, schools, and government authorities) accountable for lack of quality and inefficient use of resources. 2. Financed by the Global Partnership for Social Accountability (GPSA), this small recipient-executed trust fund (RETF) project was designed to demonstrate how educational services in the project’s target locations could be enhanced through improved transparency and social accountability. In line with the GPSA’s mission to address critical governance and development challenges through constructive engagement between governments and civil society, the project’s theory of change envisioned that if stakeholders—government, school administrators, parents, teachers, and students—were more knowledgeable and constructively engaged in educational services and oversight, then providers of those services would improve their performance and be more accountable. 3. TAME was also intended to help contribute to higher-level objectives. First, the project’s selection of its target aimags (provinces), which demonstrated its emphasis on engagement with marginalized and vulnerable groups, was consistent with the direction of the Country Partnership Strategy (CPS) for Mongolia. The targeted communities in 20 (out of 141) soums (districts) in eight (out of 21) aimags in Western and Central Mongolia are largely inhabited by nomads who live from livestock and move around within their aimag. The western and central provinces also include nearly 90 percent of Mongolia’s ethnic minorities (the Kazakh people in Bayan - Olgii, the Tuvas and Tsaatan in Khuvsgul, and the Bayad and Durvud in Uvs and Khovd). Both nomadic and ethnic minority groups are often underserved in terms of accessing quality education due to their distance from rural and urban centers and linguistic differences (for ethnic minorities). The selected aimags were also among the worst performers in terms of transparency and corruption in Mongolia. Hence, the project was well aligned with the CPS, which makes “addressing vulnerabilities through improved access to services and better service delivery� an explicit goal. Second, with its focus on the improvement of educational services through enhanced social accountability processes, especially at school and local government levels, TAME was also envisioned to complement the World Bank (WB)-financed Education Quality Reform Project (EQRP) (P148110). This project, which became effective in August 2015 and is still under implementation, aims to improve the quality of Page 4 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) education for primary school children in Mongolia, with particular emphasis on enhancing native language and mathematical skills, as well as to strengthen school-level planning. Project Development Objectives (PDOs) 4. The project's development objective is “to strengthen civic engagement in the education sector to improve transparency of budgeting and procurement processes for the delivery of better quality education services in western and central Mongolia.� Key Expected Outcomes and Outcome Indicators 5. PDO Indicators: Below are the key development outcomes and outcome indicators.1 They are grouped based on a breakdown of the PDO into two different objectives. Objective 1: To strengthen civic engagement in the education sector The key outcome indicator is: • PDO Indicator 1: Percentage of stakeholders at project schools engaged in planning and oversight of school- level processes Objective 2: To improve transparency of budgeting and procurement processes The key outcome indicator is: • PDO Indicator 2: In target areas, percentage of stakeholders participating in Civil Society Organization (CSO)- initiated budget monitoring activities for education 6. Intermediate Results Indicators (IRIs): The following are the intermediate indicators by objective (as specified by breaking down the PDO into discrete objectives): Objective 1: To strengthen civic engagement in the education sector • IRI 1: Number of PTAs established • IRI 2: Number of project schools that completed the Good School Support Tool (GSST) processes • IRI 3: Number of recommendations and policy actions/corrective measures developed to address priority legislative and policy impediments that undermine effective stakeholder engagement at schools Objective 2: To improve transparency of budgeting and procurement processes • IRI 4: In project areas, number of meetings of the Citizen's Representative Hural observed by CSOs each year 1 Based on the “unpacking� of TAME’s compound PDO, the ultimate end-goal of the project was “the delivery of better quality education services in western and central Mongolia.� However, given the limited scale and scope, and the grant size of this small RETF project, combined with the shortcomings of the results framework that lacks indicators to track progress in the quality aspects of education services, it was clear that the ultimate goal was too ambitious for the project to achieve. Thus, this ICR limits its assessment of the project’s achievements to its second-tier objectives, which are the following: (i) to strengthen civic engagement in the education sector; and (ii) to improve transparency of budgeting and procurement processes. Page 5 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) • IRI 5: Number of recommendations and policy actions/corrective measures developed to address priority legislative and policy impediments that undermine transparency of education budget processes Components 7. The project consisted of four components: Component 1: Organization of Parent Teacher Associations (PTAs) and Monitoring of School Performance at the Soum (District) Level (Estimated: US$ 180,407; Actual: US$ 230,983) 8. This component focused on strengthening the capacities of students, parents, teachers, and school officials to form and manage effective PTAs and engage with district-level officials to improve the local educational environment. In this context, 28 PTAs2 were proposed to be established in 20 targeted soums. Each PTA was designed to consist of seven representatives (four parents and three teachers). Where present, PTAs were to include representatives of ethnic minorities to ensure their voices and needs were reflected in decision making. Key activities under this component included the following: • One inception training seminar for 50 CSOs and government representatives in Ulaanbaatar to present this project and obtain feedback on the project design and interventions. • One “train-the-trainers� session for 20 participants who subsequently conducted training in the targeted aimags for the PTA representatives on current education legislation, government reform initiatives, and social accountability tools employed under the project, such as GSSTs, focus group interviews, and public hearings. • The design and implementation of social accountability tools to ensure participation of parents, teachers, and students in decision making and monitor the availability of school supplies, budget, and general school environment. Each soum’s PTA conducted an annual GSST to assess the physical, governance, and psychological aspects of the learning environment. The findings of the GSSTs were presented to school and local government authorities through public hearings and focus group meetings. • Preparation and dissemination of news programs and articles on the results of the monitoring using community and mainstream radio and newspapers. • Support to selected PTAs to disseminate the results of the project activities and establish a national PTA network. This initiative included peer-learning regional workshops in the targeted aimags. The workshops brought together PTAs, CSOs, and local governments to share their experiences and challenges in improving transparency and social accountability in the education sector. In October 2018 (Year 4), the project organized a two-day national forum in Ulaanbaatar to commemorate the project’s accomplishments, as well as highlight the results and knowledge gained from the project. The forum was attended by 56 PTA personnel (two persons from each of the 28 participating schools); focal persons from the targeted aimags’ Education and Culture Department (ECD) and the Social Development Department, national CSOs, and representatives of the central 2 As presented in Section II, TAME overachieved the targeted number of PTAs, having 31 by the end of the project. Page 6 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) government, including the Ministry of Education, Culture, Sciences, and Sports (MECSS) and the Ministry of Finance (MOF); the WB’s Acting Country Manager for Mongolia and Ulaanbaatar-based officers; and researchers concerned with social accountability in the education sector. Component 2: Monitoring of Education Budget and Procurement at the Aimag (Provincial) Level (Estimated: US$ 88,925; Actual: US$ 45,183) 9. This component was designed to strengthen the capacity of national and provincial level CSOs, media organizations, local government officials, as well as MECSS and MOF staff to analyze and monitor government budget allocation and procurement in education within the eight aimags targeted by the project. Key activities under this component included the following: • Provision of training for selected CSOs in the project’s targeted aimags to analyze and monitor the allocation of the government education budget and procurement at the regional level. The selected CSOs were asked to produce budget monitoring reports (two per year) and procurement monitoring reports (one per year). • Training was offered to journalists from community and national radio stations, as well as aimag and national mainstream media outlets on relevant education and social accountability issues. • Production of quarterly project progress reports to be shared and discussed with the Project’s Steering Group Committee comprised of the Globe International Center (GIC) and its implementing partner (“All for Education!� National Civil Society Coalition [AFE]), MECSS, MOF, the Independent Authority against Corruption, regional- level CSOs, and selected representatives of PTAs. • Implementation of advocacy and public–private dialogue activities to influence national-level policymaking (including parliament) based on education budget and procurement information generated by the project. Component 3: Managing Knowledge and Learning (Estimated: US$ 81,580; Actual: US$ 75,596) 10. This component served as a quality assurance mechanism to produce the appropriate knowledge and learning (K&L) tools for the practice of social accountability and results for use by the project’s direct and indirect beneficiaries.3 Key activities under this component consisted of the following: • Establishment and implementation of the project’s monitoring and evaluation (M&E) system.4 • Preparation of the project’s Operations Manual. 3 The direct beneficiaries of the project were parents, students, and teachers in the targeted communities of 20 soums in eight aimags in Western and Central Mongolia. It was designed to reach approximately 3,600 teachers and 65,700 students. The indirect beneficiaries included Mongolian citizens and CSOs, who might benefit from the information and knowledge generated by the project, outside the targeted provinces. Through media outreach, the project estimated that it would reach up to 500,000 people, which is equivalent to about 15 percent of Mongolia’s total population. Another group of indirect beneficiaries was CSOs working on similar education projects in Mongolia and other countries. 4 The Partnership for Transparency Fund, a Washington-based consulting entity, provided technical support for the development of the project’s M&E system, Operations Manual, and knowledge products. Page 7 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) • Development of K&L products, including various manuals, guidebooks, and training modules used for Components 1 and 2. • Production and dissemination of case studies, lessons learned, and good practices through the GPSA’s Knowledge Portal and the project’s web-based dedicated portal as a repository of its K&L products and a platform through which the project experiences were shared to aid in replication in other provinces and countries. Component 4: Project Coordination and Management (Estimated: US$ 299,088; Actual: US$ 298,238) 11. This component supported the management and coordination of project activities and reporting, including audits. Among the costs financed under Component 4 were the operating costs of the PIU5 and auditing costs. GIC produced grant progress reports on a semiannual basis throughout the life of the project. GIC also prepared semi-annual unaudited Project Interim Financial Reports. Annual audits of the Project Financial Statements and eligibility of expenditures were performed by an independent audit firm with qualifications, under terms of reference acceptable to the Bank. Significant Changes during Implementation and Rationale for the Changes 12. During the inception phase of Year 1, GIC, in consultation with the WB GPSA and the education supervision team, ascertained that the original rollout plan for project activities was not feasible, given the remoteness and geographical spread of the project areas, the novel concept, and the limited implementation capacity of GIC’s partners. The simultaneous rollout in all eight target aimags—a big-bang approach originally foreseen by TAME— would not have allowed the project the opportunity for learning and mid-course correction or to engage the government effectively prior to and during the rollout. The thinly spread supervision and technical capacity not only ran the risk of compromising the quality of intermediate outputs but also increasing the likelihood of failing to achieve the PDO. In addition, the conditions described in the initial disbursement letter6 were determined to be far more complex than was justifiable, considering the project’s design, scope, and grant size. 13. Following the receipt of an official letter of request from GIC on August 27, 2015, restructuring took place on November 13, 2015 to make amendments to the following: 14. Change in the implementation schedule: Adjustments were made to the implementation schedule by adopting a phased-in approach, which enabled TAME to roll out project activities of Components 1 and 2 during Years 2– 4 (i.e., 2016, 2017, and 2018). No change was made to the closing date. 15. Change in the results framework (RF) indicators and targets: The PDO indicators were amended to bring them in line with the data collection instruments and processes designed since the project had become effective, with its annual targets reflecting the revised implementation schedule. The original RF was almost entirely output (not outcome) oriented and the intended baseline survey was not fashioned to strategically align with or reflect the interventions being planned. The change in the implementation schedule also necessitated amendments to 5 The operating costs were comprised of the following: overhead and staff costs (including staff salaries); office rental; secretarial service; transportation; basic utilities (electricity and water); and communication expenses (telephone, internet access, among others). 6 Initially, each year’s disbursement was conditional on achievement of every single intermediate output GIC planned to deliver in that year. For example, the second disbursement had more than ten component-specific conditions associated with it. Page 8 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) annual RF targets. As a result of the revisions to the RF, the total number of indicators (including both PDO indicators and IRIs) was reduced from 13 to 8, which was better for monitoring the progress of project activities, given its design and complexity. A table summarizing the modifications of RF indicators and targets is attached below. Table 1. Changes to PDO and Intermediate Results Indicators Indicators Modification at Restructuring Rationale (November 2015) Original PDO Indicator 1: Number of public PDO Indicator 1: Percentage of stakeholders PDO Indicator 1 was amended to reflect schools with improved level of citizen at project schools engaged in planning and progress towards achieving the PDO of satisfaction on education inputs oversight of school-level processes improved citizen engagement (which was more directly linked to the PDO), rather than improved citizen satisfaction. Improved satisfaction was also condition on improved engagement, and hence it was deemed critical to measure engagement first. Original PDO Indicator 2: Number of public PDO Indicator 2: In target areas, percentage PDO Indicator 2 was amended to capture schools in the 8 aimags monitored by aimag of stakeholders participating in CSO-initiated stakeholder engagement in CSO-initiated CSOs that contributed to the disclosure and budget monitoring activities for education activities. accountability in the budgeting and procurement process Both PDO Indicators 1 and 2 were modified to outcome-oriented indicators as the original ones were output-oriented and thus inappropriate as PDO indicators. Original PDO Indicator 3: Number of Original PDO Indicator 3: Dropped. PDO Indicator 3 was dropped as it did not government policies/instruments on capture progress attributable to the TAME disclosure and accountability on public project but many other influences in the education spending education sector and political landscape more broadly. Intermediate Results Indicator (IRI) 1: IRI 1: Annual targets were revised to reflect Annual targets for IRIs were adjusted to Number of PTAs established the new implementation schedule. reflect the change in the project’s implementation schedule. The amendment of the IRIs was also designed to improve their specificity. IRI 1: Annual targets were revised to reflect the new implementation schedule. Original IRI 2: Number of PTA- initiated IRI 2: Number of project schools that IRI 2: Amended to allow tracking of recommendations adopted by schools and completed the Good School Support Tool implementation of the GSST, which was the soums (GSST) processes activity central to the success of Component 1. Page 9 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) Indicators Modification at Restructuring Rationale (November 2015) Original IRI 3: Number of PTAs that have IRI 3: Number of recommendations and IRI 3: Amended to measure progress in adopted action plans on improving policy actions/corrective measures developed production of outputs of the GSST process; transparency, accountability and school to address priority legislative and policy the original indicator (i.e., adoption of action performance and services impediments that undermine effective plans by PTAs) was an intended outcome stakeholder engagement at schools that would be monitored in biannual reports but was beyond what was realistically achievable on the ground. IRI 4: Number of recommendations IRI 4: In project areas, number of meetings of IRI 4: Amended to allow tracking of CSO submitted to parliament by the TAME the Citizen’s Representative Hural observed monitoring of local government budget Consultative Group by CSOs each year meetings, which was an activity central to success of Component 2. IRI 5: Level of involvement of non-state IRI 5: Number of recommendations and IRI 5: Amended to measure progress in actors in monitoring education budget and policy actions/corrective measures developed production of policy recommendations procurement to address priority legislative and policy resulting from CSO-initiated activities; the impediments that undermine transparency of original indicator was designed to measure education budget processes engagement (i.e., a PDO-level outcome). Original IRI 6: Original IRIs 6, 7, 9 & 10: Dropped Original IRIs 6, 7, 9 & 10: These indicators Recommendations/policies/reforms were monitored in project reports and were developed by CSOs have been adopted by *As original IRIs 6, 7, 9, and 10 were dropped from the RF. government stakeholders dropped, original IRI 8 became IRI 6 (Number of case stories, tools and other knowledge Original IRI 7: Number of media reports products developed). disseminated on education legislation and reform, and social accountability tools Original IRI 9: Number of stories, tools and other knowledge products downloaded from the project website and shared in other forms Original IRI 10: Knowledge and skills shared among PTAs and CSOs 16. Change in disbursement arrangements: Disbursement conditions were streamlined to enhance the focus on the achievement of outcomes that were most critical for the project’s overall success, in line with the adoption of the phased-in approach. 17. Change in the project description in the grant agreement (GA): The following amendments were made to the original GA so that its project description would accurately reflect implementable project activities: (i) deletion of “community scorecards�—The GSST and community scorecards were designed to collect similar information. To avoid the duplication of activities, and as the GSST was the more comprehensive school accountability instrument, it was decided not to use community scorecards. There was no change to the cost of project Page 10 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) components; and (ii) the description of privately-owned schools would be deleted as there were no such schools in the project areas. 18. Furthermore, based on the findings and recommendations of the mid-term review (MTR), which was conducted in October 2017 by an M&E consultant from the GPSA, TAME made additional adjustments to activities to increase the likelihood of the project’s success in achieving the PDO. Key changes included the following: 19. Scaling up PTA activities: Based on the MTR’s finding that PTAs continued to play a critical role in achieving the PDO as demonstrated in their contribution to improving school governance and accountability, PTA activities were scaled up through two primary initiatives. First, TAME, in close consultation with and supported by the WB’s education supervision team and the GPSA, embarked on building synergy with the WB-financed EQRP to formalize linkages between PTAs (formed under Component 1 of TAME) and the school grants component of EQRP. It was envisioned that strengthening the link would be a win-win for both projects: having PTAs formally involved in the school grants would better position them to support the effort of enhancing learning outcomes for school children, while the arrangement would allow EQRP to harness social accountability mechanisms created under TAME for achieving one of its development objectives, which is to improve school-level planning. Second, to encourage PTAs to explore alternative avenues for engaging in education budget processes beyond the school level, the project expanded peer-learning activities under Component 2 to aid PTAs in building their capacity to engage in more informed discussions with local public authorities on monitoring and planning the allocation of local development funds for schools’ needs. 20. Scaling down budget monitoring exercises using Budget Trust Tools (BTTs): In response to the MTR’s finding that BTTs had their limits in serving as an effective tool for motivating public financial management processes without a politically informed strategy, the project scaled down its BTT activities under Component 2 by continuing them in only three highly committed aimags (out of eight targeted aimags). 21. Discontinuing community radios: Despite the initial vision of community radios as a key channel for disseminating the results of project activities under Component 1, the MTR found that they were largely non- operational, which accounted for the lack of project disbursements for this item. Hence, the community radios were discontinued. II. OUTCOME Assessment of Achievement of Each Objective/Outcome 22. The project has achieved two major outcomes: (i) improved stakeholder engagement and oversight in target schools; and (ii) increased transparency of budgeting and procurement processes in the education sector in target aimags. In what follows, the PDO results are assessed in light of the key two TAME objectives by providing quantitative evidence from the project’s outcome indicator results and qualitative evidence from the project documents and WB teams’ dialogue with stakeholders. Objective 1: To strengthen civic engagement in the education sector Page 11 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) 23. PDO Indicator 1: Percentage of stakeholders at project schools engaged in planning and oversight of school- level processes. The baseline and end targets were 14.7 percent and 54.7 percent, respectively; the final outcome (45.7 percent)7 fell short of its target by 9.0 percent. Although the project slightly exceeded the Year 2 target (baseline plus 10 percent) at 28.8 percent, it marginally underperformed relative to the Year 3 target (baseline plus 20 percent) at 31.8 percent. 24. It should be noted that because the Project Implementation Unit (PIU) applied an inappropriate methodology to construct the values for PDO Indicator 1, Year 1 and Year 2 values and the end-line target were revised upwards in Year 4, based on the supervision team's evaluation of the raw data and methodologies used and extensive exchanges with GIC. This issue is detailed in Section IV. 25. All three IRIs linked to Objective 1 (IRI 1, 2, and 3) were met (see Annex 1). Notably, IRI 1 (Number of PTAs established) was overachieved, with 31 PTAs established in the targeted aimags by project end. Having adopted a phased-in approach to restructuring, TAME successfully rolled out project activities under Component 1. This included the establishment of PTAs (IRI 1) in line with the implementation schedule for Years 2–4 (2016, 2017, and 2018). The plan aimed to establish 28 PTAs by the end of the project. Observing the usefulness of PTAs, three schools in non-project neighboring soums (two in Bulgan aimag and one in Bayan-Olgii aimag) requested non- financial technical support to initiate formation of PTAs. 26. The project has contributed significantly to improved stakeholder engagement and oversight in target schools through the following initiatives: a. Conducting GSSTs under PTA leadership and administration: As the primary means of establishing a school accountability mechanism (and to achieve Objective 1), GSSTs conducted under PTA leadership and administration proved particularly successful. The GSSTs created a space for the more active participation of various stakeholder groups (including parents, teachers, and students) in planning and oversight of school-level processes by (a) jointly conducting annual surveys and focus group discussions to elicit fellow stakeholders’ views on physical, psychological, and governance aspects of the school environment; (b) presenting and disseminating GSST results to school administrators and other stakeholders; and (c) developing school action/activity plans based on the GSST results and presenting these plans to principals.8 Experiences in the project schools confirmed that physical improvement of the learning environment based on the action plans—including the construction and refurbishment of school buildings and the renewal or repair of school and dormitory furniture and other equipment— significantly increased awareness of school conditions and children’s needs among parents who had previously been uninformed or indifferent. Seeing the physical transformation firsthand built PTAs’ confidence in their ability to facilitate much-needed stakeholder engagement; it also helped to convince 7 The estimation of the final result of PDO Indicator 1 was based on the 2018 GSST data collected at the 28 participating schools in the targeted soums. Although three additional schools in non-project neighboring soums spontaneously established PTAs in 2018 with non-financial technical support from AFE and GIC, GSSTs were not conducted at these schools. 8 In Mongolia, the principal is mainly responsible for overseeing the budget, disciplinary issues, representing the school in interactions with education authorities at different levels, and managing school property but is not directly responsible for instructional management (e.g., guiding curriculum, managing the teaching schedule, evaluating teacher performance). Most of these tasks are the responsibility of school manager(s) (World Bank, 2011). Page 12 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) some skeptical parents and teachers that PTAs are not merely an alternative form of parent councils as organized by class teachers.9 b. Innovative initiatives developed by PTAs to enhance parental engagement: As well as initiating efforts to improve the physical learning environment on the basis of GSST results, the PTAs also developed a wide range of innovative initiatives to promote parental engagement in education. These include raising donations to provide school supplies for students from low-income households at Bulgan aimag center school. Another PTA initiative facilitated “hand-to-hand� service for herder children of Bugat soum school in Bulgan aimag, where school teachers accompany their 45 dormitory students when they return home for school breaks. These children’s parents reside about 100 km from the school, and automobile transportation is provided by the soum governor’s office. Upon their visit to students’ homes, the teachers inform the parents about the curricular content of the next semester and provide support to help them check their children’s homework during the break. c. PTAs’ formal engagement in the school grants component of the WB-financed EQRP: PTAs’ formal involvement in the school grants component of the WB-financed EQRP is a critical first step toward mainstreaming and upscaling school accountability mechanisms in Mongolia’s education system. Under the PTA activities of the TAME project, GIC has been working closely with the EQRP team since October 2017 to formalize the role of PTAs in implementing school grants in the 31 targeted schools. With the endorsement of the MECSS (the agency implementing for EQRP), each PTA will be more extensively engaged in improving student learning outcomes at primary level, based on grant proposals they have developed. Following their review and approval of all PTA proposals in March 2019, the MECSS issued the Minister’s order on April 23, 2019, which authorized the financing of 31 grants (equivalent to about US$ 2,000 each) to be awarded to an equal number of PTAs. Implementation of the grants scheme will follow during the 2019–2020 School Year under MECSS monitoring and supervision. Objective 2: To improve transparency of budgeting and procurement processes 27. PDO Indicator 2: Percentage of stakeholders participating in CSO-initiated budget monitoring activities for education in target areas. The baseline and end targets were 0 percent and 40 percent, respectively; the final outcome (50.3 percent) exceeded its target by 10.3 percent. Although the BTTs were discontinued in Year 4, the project had already overachieved its target in three of the six aimags in which BTTs had been rolled out in Years 2–3 (2016 and 2017)10—by 10 percent in Year 2 and by 44.8 percent in Year 3. The significant overachievement 9 In Mongolian schools, each classroom has a parent council, which is comprised of 3–5 parents who are generally appointed by a class teacher. Interviews during the ICR mission with PTAs, parents and teachers of non-PTA members, and students revealed that they observed functional and structural differences between PTAs and parent councils. PTAs were often regarded as a mechanism that takes a bottom-up, school-wide approach to improving students’ learning environment through increased parental engagement and school accountability, while parent councils tended to be associated (especially by parents) with fundraising by class teachers for various purposes (e.g., graduation parties, minor classroom repairment). Apart from their differences in terms of scope and approach, the interviewees also indicated that the selection process of PTA boards was more transparent than that of parent council members. Although some project schools have made efforts to build linkages between the two entities to improve their coherence, many schools found it challenging to justify their coexistence without MECSS’s full endorsement of PTAs. 10 In accordance with the revised implementation schedule adopted at the project’s restructuring, CSO -initiated budget monitoring activities for education at aimag level (under Component 2) were not rolled out in the two remaining aimags Page 13 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) in Year 3 can be attributed to a combination of strong leadership from aimag-level CSOs (especially in Bulgan and Khuvsgul aimags) and their ties with key players in the local education sector, and the exemplary commitment that successfully delivered multiple budget monitoring activities and high target stakeholder engagement (including teachers, parents, CSOs, and aimag public officials). By continuing to work closely in Year 4 with highly committed aimags that expressed a desire to sustain BTTs11 as well as the remaining aimags, the project objectives were successfully achieved. 28. Because PIU’s estimates for PDO Indicator 2 were based on incomplete data for Year 2, that value was revised upward in Year 4, following the supervision team's evaluation of the raw data and methodologies used and extensive exchanges with GIC.12 29. The two IRIs linked to Objective 2 (IRI 4 and 5) were met (see Annex 1). Having adopted a phased-in approach to restructuring, TAME successfully rolled out project activities under Component 2 in all of the eight targeted aimags in line with the implementation schedule for Years 2–4 (2016, 2017, and 2018). However, as noted above, BTTs were scaled down in Year 4 on the MTR’s recommendation, thereby reducing the number of aimags in which BTTs were implemented during the lifetime of the project from 8 to 6. 30. The project has contributed to the improved transparency of budgeting and procurement processes in the education sector primarily by promoting local stakeholders’ participation in CSO-initiated budget monitoring activities at aimag level (under Component 2), and by strengthening PTAs’ capacity to engage in budgeting and procurement discussions with school administrators and public officials at school and soum levels (under Component 1). 31. At aimag level, with high levels of CSO commitment and incentive, the project facilitated spaces for dialogue between citizens and public sector decision-makers to share information on education budgets and procurement. In the early to middle phases of the project, a few CSOs were replaced because of underperformance (Bulgan aimag) and withdrawal due to other commitments (Arkhangai aimag). Nevertheless, in close coordination with the PIU, each CSO successfully organized a series of budget monitoring activities that included observing Citizens’ Representatives (Khural) meetings; conducting BTTs on local education budget and procurement processes; providing training on conducting BTTs; and organizing open hours on budget transparency for other local stakeholders (including other local CSOs, teachers, parents, and public officials). During the lifetime of the project, 13 Citizens’ Representatives meetings were observed, and aimag-level CSOs developed 13 recommendations to address key legislative and policy impediments that undermine the transparency of education budget processes. 13 These CSOs engaged in dialogue with local public authorities (Uvurkhangai and Sukhbaatar) until Year 4. Following the PIU’s decision to continue BTTs only in the three highly committed aimags (Bulgan, Khuvsgul, and Khovd) at the end of Year 3, BTTs were not introduced to the two aimags that joined in the final year. 11 The engagement rate of stakeholders in CSO-initiated budget monitoring activities in these three aimags in Year 4 alone was an astonishing 121.4 percent. 12 This issue is detailed in Section IV. 13 Most of these recommendations were designed to improve the implementation of and compliance with the Law of Mongolia on Transparency of Information and the Right to Receive Information and the Law of Mongolia on Glass Accounts ("Accounts Law"). Both of these laws “oblige all government and legal entities with state involvement to make information on budgets and financial matters, including the utilisation of financing and other government i ndebtedness, available to the public� (Hogan Lovells, 2015). The Accounts Law, which was adopted by the Parliament of Mongolia in July 2014 and became effective in January 2015, specifies the type of information that must be disclosed, including “details o f procured goods, work, and services with a value of more than MNT 5 million (approximately US$ 2,700),� among others (ibid.). Page 14 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) (including Citizens’ Representatives, aimag governors’ offices, provincial ECDs and social development departments, and targeted school principals), sharing and discussing the results of BTT exercises and recommendations for good practice around budget transparency. The high level of commitment exhibited by aimag-level CSOs prompted further initiatives to expand the citizen’s role in monitoring budgeting processes. For example, with support from the Khovd aimag-level CSO, a citizen’s monitoring group was created to monitor expenditures financed by the aimag’s Local Development Fund. 14 In Khuvsgul aimag, CSO efforts to engage actively with the local administration resulted in an invitation to the aimag Khural meeting to approve the 2017 budget, and the Khural agreed to adopt the CSO’s recommendation to organize a public consultation meeting prior to budget discussions.15 32. The project also contributed to increased disclosure of education budget and procurement information at school level. With PIU and AFE support, each project school’s PTA worked with school administrators to set up a mechanism for disclosing information on school budgets and expenditure through information boards at school sites. Additionally, parents and teachers were encouraged to engage in discussions with principals on school budgets and procurement and how resources could be better allocated to improve the learning environment. Interviews during the ICR mission with stakeholder groups (including principals, PTAs, parents and teachers of non-PTA members, and students) revealed that the PTAs’ efforts raised awareness of how budgets were allocated at each school. As stakeholders became more informed about school budgeting and procurement processes, it became more apparent that low levels of school autonomy in planning and managing school budgets limited the ability of PTAs (and even principals) to reallocate or increase education spending at school level.16 33. To address the issue of low school autonomy over budgets and to improve PTAs’ engagement with key decision- makers at soum-level, the project initiated peer learning activities specifically designed for PTAs. Beginning in Year 3, these activities were intended to build PTA capacity to improve education services by exploring and accessing different forms of local fundraising beyond school level. These peer learning activities also empowered PTA members to engage in informed discussions about the planning and allocation of local development funds, and how to monitor their use in meeting school needs. In Bugat soum of Bulgan aimag, joint school-wide efforts by the project school to raise funds for the demolition of old school buildings and the construction of new ones. Through media campaigning and extensive discussion and negotiation with public authorities, including aimag Citizens’ Representatives, the case was ultimately brought to parliament, and MNT 2.1 billion (approximately US$ 796,000) in funding was secured. Overall Outcome Rating 34. Based on convincing achievement of the two key objectives and quantitative and qualitative evidence of improvements, the overall outcome was rated Moderately Satisfactory. TAME created a collaborative problem- solving space where key education stakeholders—especially parents and teachers, school administrators and staff, government officials, and CSOs—could engage in discussions to improve educational service delivery through increased civic participation and enhanced transparency of budgeting and procurement processes. To that end, 31 first-time PTAs proved to be a critical mechanism for strengthening school governance and 14 ISR, No. 8, December 2018. 15 ISR, No. 5, June 2017. 16 For example, the allocation of school budgets is essentially determined by provincial ECDs based on the number of enrolled students at each school. Page 15 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) accountability in Mongolia. As shown by the consistently strong performance on PDO Indicator 2, aimag-level CSOs have also contributed to making citizens more informed about education budgets and procurement processes at aimag level and have empowered them to hold public authorities and their services to account. Given the underperformance of PDO Indicator 1 (which missed its Year 3 target by 2.9 percent and its end target by 9.0 percent, respectively), the overall outcome rating remains the same as the DO rating of the final Implementation Status and Results Report (ISR). Other Outcomes and Impacts 35. As well as the major achievements discussed in the previous section, the project also achieved some unintended outcomes and impacts, all of which were positive. • An emerging role for PTAs as a channel for school-wide grievance redress. With increased recognition at school level, some PTAs have received concerns and complaints regarding issues, such as school violence and the limited transparency of school funds management by principals. To strengthen this new role, the national PTA coalition of four Ulaanbaatar-based members and seven local PTA representatives will establish a mechanism and develop guidelines for receiving, reviewing, monitoring, and resolving grievances in a timely and professional manner. • Increase in student enrollment. The project school in Bugat soum witnessed increased enrollment as a result of the PTA’s efforts to promote awareness of their activities among the residents of neighboring bags (brigades) and to significantly improve the learning environment. Previously, recruiting new students was challenging, as a number of local parents chose to send their children to urban schools, either in the Bulgan aimag center or in the neighboring Erdenet City. However, following the announcement of a new school building and dormitory and the PTA’s collaboration with governors in the neighboring bags, 18 new students enrolled in Bugat soum school in 2018. • A new collaborative initiative by Bulgan aimag ECD to found a coalition of education stakeholders. The coalition includes the aimag governor (chairperson), parent representatives, ECD officers, CSOs, and other relevant stakeholders. One of its immediate priorities is to set up PTAs in kindergartens, as ECD observation of PTAs indicated that earlier engagement of parents in PTA activities would enhance PTAs’ effectiveness in changing parents’ behaviors and mindsets regarding their role in their children’s learning. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME 36. Timely provision of support for identified risks and course corrections: The education sector and the GPSA task teams closely monitored the project’s progress and provided timely support for the implementing agency and its partners when risks were identified. This allowed for course corrections and learning at different stages of project implementation. • Early on, the task teams engaged in collaborative efforts under the technical guidance of GPSA’s capacity building advisor to address risks identified in the project design and the limited capacity of the PIU and its implementing partner, which resulted in project restructuring in Year 1. As detailed in Section I: Context, the restructuring entailed, among other things, adjustments to the results framework and the adoption of a phased-in approach for project activities of Components 1 and 2. In addition, following technical guidance from the WB’s task teams, the project rolled out GSSTs in the first batch of schools with and without PTAs in Year 1. The assessment of the results in Year 2 showed that schools Page 16 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) with PTAs performed better, which confirmed the notion that PTAs should be viewed as a prerequisite for implementing the social accountability tool. The learning from this assessment informed the sequencing (i.e., establishing PTAs, followed by the implementation of GSSTs) in the second and third batch of project schools. • In response to the findings of the MTR—which suggested the scaling down of BTTs (Component 2) due to their limited effectiveness in certain aimags—the project took remedial actions. In consultation with the task teams, instead of discontinuing the BTTs altogether, the PIU decided to continue to work with three aimag-level CSOs in Bulgan, Khuvsgul, and Khovd, all of which indicated their keen interest in continuing the BTTs in Year 4. These aimag-level CSOs’ high level of commitment allowed the project to overachieve the end-line target for PDO Indicator 2 (which was set at 40 percent) by over 10 percent at 50.3 percent. Furthermore, as a follow-up to another of the MTR’s recommendations to enhance PTAs’ capacity to mobilize external local funding resources—considering the low level of school autonomy over budgets in Mongolia—the project provided capacity building support to help them explore and access different forms of local fundraising to improve education services. One such initiative was peer- learning activities designed for PTAs. Beginning in Year 3, these activities were organized in coordination with the WB-supported Third Sustainable Livelihoods Project (P125232), which assisted PTA members with learning how to engage in informed discussions about the planning and allocation of local development funds, as well as how to monitor their usage for school needs. 37. Strong commitment of GIC and AFE and ownership of local stakeholders: The high level of commitment and ownership of GIC, AFE, and local stakeholders (especially PTAs and aimag-level CSOs) were key drivers throughout the project’s implementation. GIC’s and AFE’s commitment to collaborative and systematic support facilitated strong ownership by local stakeholders at the aimag and soum levels. • Since the project began, both GIC and AFE ensured that the key stakeholders (PTAs and parent-teacher- student groups conducting GSSTs for Component 1 and aimag-level CSOs for Component 2) were provided with adequate training, manuals, and tools for undertaking activities effectively and maximizing available resources. In addition to producing the Operations Manual, GIC in collaboration with AFE prepared a wide array of easy-to-use templates (e.g., meeting minutes, financial reports, boilerplate letters addressed to local authorities, media/press conferences). In response to the PTAs’ and CSOs’ feedback, GIC also revised these boilerplates and the format of the survey tools (i.e., GSSTs and BTTs) as needed. Moreover, they conducted training programs in a way that catered to local needs and addressed challenges by taking into account different local contexts and pre-existing power dynamics among the stakeholders, including PTA members, principals, local authorities (e.g., aimag and soum governors, ECD officials), and CSOs. • Both PTAs and aimag-level CSOs took ownership of their activities by making extensive and proactive efforts to collaborate with the relevant local authorities to achieve their common goals: to foster more active stakeholder engagement in education planning and spending, as well as to improve the transparency of budgeting and procurement processes, both of which were envisioned as driving forces for enhancing the quality of education service delivery. • PTAs’ formal involvement in the EQRP grants scheme empowered them and increased their sense of ownership and accountability for their activities. Although MECSS has yet to fully endorse the Page 17 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) mainstreaming of PTAs after the completion of EQRP, the Bank’s education task team’s engagement in dialogue with key stakeholders strongly motivated local stakeholders (including PTAs, school management, and local authorities) to embark on unique and innovative efforts to sustain and expand PTA-associated initiatives, as highlighted in the preceding section on unintended outcomes and impacts. 38. Insufficient buy-in from the government: Limited constructive engagement with, and insufficient buy-in from, the government, especially MECSS, has been a major challenge since the very beginning and though this remains a risk to sustainability, it did not affect the overall project outcomes. Stakeholder interviews showed that MOF was supportive of the project, keen to see results, and remained engaged through representation on the project’s Steering Committee. This is not surprising given that MOF has been striving for years towards greater transparency in public spending through long-standing efforts to improve performance of financial management systems in the country. However, as discussed in the next section on the project’s potential risks to development outcomes, MECSS’s lack of ownership and sense of wariness toward the project activities, combined with the high turnover of MECSS officials on the project’s Steering Committee (as a result of MECSS’s frequent leadership transitions), has made it difficult to engage with the ministry’s education reform agenda. • One of the major challenges identified in Year 1 was the lack of MECSS’s experience in working closely with CSOs and their limited knowledge and understanding of social accountability interventions. To enhance MECSS’s capacity in school governance and encourage its ownership, the project took various actions, such as inviting the ministry to critique the social accountability instruments and co-own their implementation (albeit with the project preserving ultimate editorial control), soliciting their input and feedback for the gradual rollout of PTAs, having a local senior education expert (who later became Chief Technical Advisor for EQRP) on board to help GIC and its partners design BTTs and facilitate their conversations with ministry contacts, and formalizing the arrangement for the established PTAs to be part of the EQRP grants scheme. In order to sensitize MECSS officials on the benefits of social accountability interventions in the education sector, and to share global best practices, the World Bank invited and hosted presentations by participants from MECSS and the EQRP project team at the 2017 and 2018 GPSA Global Partners Forum. • Toward the end of the project, GIC and AFE realized that the ministry’s skepticism originated partly from its concern the project did not adequately engage school administrators (especially principals) in project activities as important stakeholders/collaborators, due to the project’s emphasis on PTAs’ independence and redressing power imbalances within schools. Based on this realization, the national PTA coalition, which has succeeded PTA-associated initiatives under Component 1 since the project’s completion, will maximize its efforts to cultivate a sense of collaboration with MECSS and also among school stakeholders by working more closely with school administrators to increase their incentives to join in the efforts to improve school governance. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME Bank Performance 39. Quality at Entry: Both the GPSA and education sector teams have faced challenges resulting from the project design, which nevertheless was well aligned with the WB's CAS for Mongolia and its sectoral context. Its theory of change was also in line with the GPSA's mission to address key governance and Page 18 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) development challenges through constructive engagement between governments and civil society. However, the project's original PDO and key indicators were highly ambitious and aspirational, largely due to the novelty of the project concept and its interventions (e.g., the establishment of the country's first PTAs) and because Mongolia has one of the lowest population densities in the world. The originally planned simultaneous rollout of key project activities (e.g., the formation of PTAs, the implementation of GSSTs and BTTs) also did not fully account for the limited implementation capacity of GIC and its implementing partner in the early phase. These shortcomings in the project design led to a major restructuring in Year 1, which involved changes in the results framework, implementation schedule, and disbursement arrangements, among other things. 40. Quality of Supervision: The issues with the project design, coupled with the transfer of responsibility for project supervision from Governance Global Practice (GP) to Education GP, which took approximately six months after the project became effective in December 2014, adversely affected the project's implementation pace and supervision in the early stage of the project. As a result, the GPSA and education sector teams dedicated the project's first year mainly to facilitating the restructuring and to designing/testing the social accountability tools used for Components 1 and 2. Moreover, no supervision budget was allocated during FY 15 and FY 16. In order to carry out the restructuring in FY 16, the team incurred a cost overrun in the amount US$12,000 in order to cover the fee of a local consultant to support GIC in developing a plan for the restructuring, fine-tuning design of activities, and providing intensive monitoring in the period immediately following the restructuring. The Education GP management team agreed to absorb this cost overrun in FY 16. During these critical initial years of the project, thus, the WB teams were able to provide only limited in-country supervision, relying primarily on updates provided remotely by GIC over phone and email.17 Nonetheless, given that GIC required day- to-day handholding from the WB teams (especially in the initial phase) because of their low technical capacity as an implementing agency, the project would have benefited from more intensive, on-site capacity-building supervision efforts by securing supervision budgets much sooner. 41. To address these challenges, which originated during project preparation, the WB teams made concerted efforts to minimize risks through close supervision and progress monitoring. In effect, key to the project's success were the consistent monitoring, supervision, and technical expertise provided by the Bank's teams, as well as their active engagement in supporting GIC and its implementing partners in their efforts to carry out the project. The teams' persistent work to strengthen links between TAME and the WB-supported EQRP culminated in MECSS's endorsement of the PTAs, established under TAME, to play an official and enhanced role in supporting student learning and improving its quality. As previously mentioned, the teams also took the initiative to restructure the project and make the needed adjustments. The MTR conducted in Year 3 also facilitated course corrections to help TAME address identified risks and achieve its outcomes. Furthermore, the supervision team supported the PIU when they encountered data reporting challenges that were due to inconsistencies in the PIU's application of the methodology for estimating the PDO indicators, which is detailed in the section on M&E quality. Throughout the project, the WB's procurement and financial management teams provided essential support and supervision for the PIU, which improved their project readiness and helped build their capacity in these areas. 17Subsequently, in FY 17, 18, and 19, bank budget (BB) in the amount of US$20,000, US$25,000, and US$35,000 was allocated by the Country Management Unit for supervision of this project. Page 19 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) Justification of MODERATELY SATISFACTORY Bank Performance Rating 42. The overall Bank performance rating is Moderately Satisfactory. The project was able to accomplish a great deal in the four-year implementation period. This was due in significant part to the Bank teams' strong commitment to supporting the project's success in achieving the PDO and their continuing efforts to provide quality and timely supervision. Their initiatives to facilitate the project restructuring early in the project implementation, the midcourse corrections through the MTR, and the linkage with EQRP to improve the project's sustainability and potential for scale-up illustrate how the Bank proactively addressed identified risks to achieving the planned outcomes during the project implementation. Nevertheless, the issues with the initial project design, which necessitated major adjustments and affected the project implementation in the early stage, demonstrate that quality at entry could have been better. Hence, the quality of the Bank's performance for the TAME project was determined to be Moderately Satisfactory. M&E Quality 43. The project overcame many challenges and setbacks in its M&E design, implementation, and utilization. The ultimate goal of the project (“the delivery of better quality education services in Western and Central Mongolia�) was too ambitious and unrealistic for the level of the implementing agency’s capacity and the project scale; no indicator was designed to define and capture the “quality� aspects of education services. However, the overhauling of the RF as part of the restructuring in November 2015, which involved modifications to RF indicators and targets, allowed the project to link the two PDO indicators to data captured by the two social accountability tools used (i.e., GSSTs and BTTs) to assess progress in achieving the project’s second-tier objectives (“to strengthen civic engagement in the education sector� and “to improve transparency of budgeting and procurement processes�). 44. Given the PDO indicators’ focus on the “share� of stakeholders engaged in planning and oversight of school-level processes (PDO Indicator 1) and of those participating in CSO-initiated budget monitoring activities for education (PDO Indicator 2), the project needed to rely on qualitative evidence to determine the outcomes of stakeholder participation (what has changed as a result of their participation in terms of their attitude, behavior, collective actions, etc.) and to examine “how� the project and its various initiatives succeeded in making those changes happen. However, the project faced difficulties collecting quality documentation related to procedural changes and outcomes from PTAs and aimag- level CSOs due to their limited reporting capacity. To address these challenges, GIC and AFE prepared user-friendly templates for reporting and provided relevant training. They also made the most of their training sessions and focus group interviews with stakeholders to gather qualitative information, such as their manner of engagement in different activities and the resulting changes. These efforts allowed GIC and AFE to provide more timely, responsive, and effective assistance to the PTAs and aimag-level CSOs. Nevertheless, as the MTR pointed out and based on the supervision team’s observation, the complexity of feedback loops and reporting between partners and the multiple processes of interpretation of different local contexts often made it difficult for the project to utilize M&E data for informed decision- making and course corrections among the partners. It also made it challenging for the WB teams to provide timely feedback and technical guidance when needed. 45. Another challenge in M&E implementation was the PIU’s shortcomings in the handling of M&E data for estimating the PDO indicator values. The WB’s education sector team identified M&E issues for both Page 20 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) PDO Indicators 1 and 2. For PDO Indicator 1, the estimation methodology applied to M&E data (which was used by the PIU to establish the baseline value and the Year 2 update to the indicator) was not appropriate: it narrowly took into account only one activity (school budget), instead of a full range of activities in which stakeholders engaged in planning and oversight of school-level processes (including annual school plans, school budgets, and GSSTs) as specified in the methodology. After the correction of this inconsistency, the value of the baseline changed from 5 to 14.7 percent, while the Year 2 value went from 19 to 28.8 percent, and the end target value was revised from 45 to 54.7 percent. For PDO Indicator 2, its Year 2 value was inaccurate as it accounted for only one event, instead of the total number of events that occurred during the reporting period; complete data on only one event was available at the time of the PIU’s progress report, and the final data was not updated. After the recalculation, the Year 2 value was modified from 11 to 19.7 percent. Of critical note is that, despite the modifications to the PDO indicator values, the RF annual targets were not amended as the agreed-upon changes to the methodology were fully consistent with all project documents, including the RF. Justification of MODEST M&E Quality Rating 46. The overall M&E quality rating is Modest. Despite the best of intentions, the project’s M&E design, implementation, and utilization had some shortcomings. The adjustments made to the RF at the time of restructuring significantly increased the appropriateness of the M&E design. Nonetheless, the identified issue with the handling of M&E data in estimating the PDO indicator values and the remaining challenge of streamlining the M&E system made it difficult for the project to help facilitate informed decision- making and remedial actions in a timely fashion. Compliance Issues 47. The project had adequate due diligence to safeguard against social risks.18 Given that the majority of the project beneficiaries in the targeted communities19 are indigenous/ethnic minority peoples, the project triggered Indigenous Peoples (IP) OP/BP 4.10, but it was not required to prepare an IP instrument (IP Plan or IP Planning Framework). Nonetheless, the project design ensured that elements of OP/BP 4.10 were incorporated, which included, among others, a social assessment, free, prior, and informed consultations with ethnic minority communities, and the provision of culturally appropriate benefits to ethnic minorities. Social safeguard measures integrated into project activities included the following: (i) the development of PTA training modules to address power imbalances between parents and teachers (especially those from ethnic minority backgrounds) and encourage women to speak up using interactive methods, instead of forced preaching, as these require a tremendous cultural and behavioral change; (ii) the translation of training materials, the PTA manual, and the GSST manual into the Kazakh language; (iii) instructions provided for training facilitators on how to handle different situations that might arise during training sessions, for example, where women or those from ethnic minority backgrounds hold back their opinions or are not actively engaged due to cultural norms; and (iv) the recruitment of Kazakh-speaking facilitators to allow for more active contributions from Kazakh participants in training sessions. 48. With the adequate support and guidance provided by the WB’s procurement and financial management 18Environmental Assessment OP/BP 4.01 was not triggered for TAME. 19These communities were located in 20 soums in eight aimags in Western and Central Mongolia. Four of the eight aimags (Bayan-Olgii, Khuvsgul, Khovd, and Uvs) are inhabited by ethnic minorities. Page 21 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) specialists to PIU, no significant procurement or financial management issues arose throughout the project’s implementation. Audit reports were submitted on time. As a result of the timeliness of the restructuring, which entailed the streamlining of disbursement arrangements, the project has met all disbursement-linked indicator conditions and the grant was fully disbursed by December 2018. Risk to Development Outcomes 49. The risk to development outcomes is deemed to be Medium to Low. Despite the arrangement with the ongoing EQRP, there are still some challenges in strengthening the role of PTAs as a catalyst for school governance reforms at scale while simultaneously improving service delivery in the education sector. One major anticipated risk is the lack of official endorsement by MECSS for (and their seeming disinterest in) mainstreaming PTAs beyond the scope and lifetime of EQRP. The high turnover of the ministers and officials of MECSS during the project’s implementation20 hindered the project from gaining much-needed support. MECSS’s inclination to keep the pre-existing parent councils as the primary medium of parental engagement adds a great deal of uncertainty in predicting ex-ante how likely it is that the PTAs established under TAME will be allowed to fully leverage their experience and lessons learned to enhance their role in school governance reforms. 50. To mitigate this risk to the sustainability of the outcomes, AFE has set up a national PTA coalition to provide continued support and capacity building for the 31 PTAs and any schools that have expressed interest in forming PTAs. Furthermore, to address MECSS’s concern that the project places too much emphasis on rendering PTAs independent from principals’ (and, in a broader sense, school management’s) control, the national PTA coalition will strengthen their engagement with school management going forward, by developing training programs designed for them and fostering a sense of partnership between the PTAs and school management.21 As discussed in the next section on lessons learned, one of the key features of high performing schools under TAME was a strong sense of partnership and collaboration among the stakeholders. Key Ratings 51. The table below summarizes the key ratings of the project. Table 2. Key Project Ratings Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest V. LESSONS LEARNED AND RECOMMENDATIONS 52. Taking into account the power dynamics among school stakeholders is key to facilitating effective and collaborative engagement in school governance. The project’s experience showed that a careful 20 During project implementation from December 2014–December 2018, MECSS had four different ministers, which also involved ministry reorganization. Due to this frequent leadership transition, four different MECSS officials, with varying levels of commitment, came on board to serve as the project’s Steering Committee members. 21 In addition to principals, strengthening the engagement of other key players in school management, such as school managers and school social workers, will be essential, given the former’s role in instructional management and the latter’s role in wo rking closely with students to ensure child protection and advocacy. Page 22 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) examination of the power dynamics among school stakeholders has critical implications for determining the level of success and potential for sustainability of stakeholders’ collaborative engagement in improving school governance and the learning environment. Under TAME, high-performing PTAs not only exhibited a strong ownership of their activities but also likely had the right composition of individuals in their respective boards. Selecting board members with diverse backgrounds and technical skills would balance out power dynamics, especially between PTAs and principals. Schools that appointed teachers or school staff as the key board members (i.e., PTA coordinator and chairperson) were more at risk of being under the thumb of principals because of their hierarchical relationship. By contrast, when the key active members included professionals with technical skills and credentials that served well in building a more equal partnership and collaboration with school management (e.g., tax accountant, bag governor, local public official), PTAs were more likely to not only better position themselves in influencing school-level politics as a catalyst for school governance reforms but also work with school management more collaboratively as partners. 53. Strategic support for strengthening the technical capacity of non-governmental implementing agencies to facilitate constructive engagement with the key line ministry throughout the project cycle is essential for obtaining high-level buy-in and maximizing the potential for scale-up. In the context of an unstable political climate and a country with limited experience in constructive government and civic engagement, such as Mongolia, providing strategic support is needed to help strengthen the capacity of non-governmental implementing agencies to maneuver political economy challenges. Despite WB teams’ efforts to encourage increased government ownership of the project (e.g., by recruiting a consultant to assist GIC and its implementing partners with facilitating conversations between them and ministry contracts at the early phase of the project, building synergies between the ministry-led EQRP and TAME), there was insufficient buy-in from government (especially MECSS) throughout, other than a general endorsement for GIC to carry out its work and a presence on the project’s Steering Committee (with the high turnover of appointed MECSS officials because of frequent leadership transitions). The MECSS’s issuance of the Minister’s order in April 2019, after the project completion, which authorized the financing of grants to be awarded to all 31 PTAs, officialized the ministry’s endorsement of the PTAs’ involvement in the school grants scheme under EQRP. This action has increased the potential for PTAs’ large-scale expansion and demonstrated the ministry’s incremental buy-in for PTAs. Nevertheless, as the MECSS has thus far sustained pre-existing parent councils, more strategic efforts would have been needed to communicate effectively with the MECSS on the advantage of leveraging PTAs as the primary form of parental engagement. PTAs have served as a school-wide mechanism for strengthening accountability and transparency while parent councils, typically organized by class teachers at classroom or grade level without transparent parent selection processes, have not. Thus, in maximizing its potential for scale-up, the project could have benefited from having a technical assistance component that provides capacity building in navigating governance challenges for GIC and AFE, as well as for the project beneficiaries. The project could have also benefited by strategically engaging to a greater extent potential champions for increased transparency and accountability in the education sector, for example, the MOF. 54. A combination of just-in-time supervision support and the implementing agency’s ownership of project interventions can help improve the project’s achievements and potential for sustainability. Rigorous and timely supervision support for the identified risks by task teams during the project implementation, in combination with the implementing agency’s strong sense of ownership of the Page 23 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) project, can lead to strong results, even when the capacity of the implementing agency is somewhat limited at the beginning. Under TAME, despite its extensive experience in the area of governance, GIC was considered under-capacitated as the implementing agency because of its lack of experience in the education sector. To ensure effective quality control, the project not only allowed GIC to forge an alliance with its implementing partners but also provided close supervision, monitoring, and hand- holding support, especially at the early stage. The overhauling of the project design—which was facilitated by the WB supervision teams to make it more manageable and appropriate for the project’s scale and for GIC and its partners’ capacity level—and their subsequent follow-through in rolling out the key project components—empowered GIC and its partners and increased their sense of ownership. With WB’s continuing support for course corrections and adaptive learning for the remainder of the project period, the project made great strides in achieving results by the project’s closing; it further led to creating the national PTA coalition, which will serve as an important institutional mechanism for sustaining PTA initiatives and providing oversight and support for local PTAs. 55. To improve quality at entry, during preparation of future GPSA-financed operations, a co-Task Team Leader (TTL)-ship arrangement could be considered between the Governance GP TTL leading preparation of the project and the sector TTL responsible for supervision of implementation upon project effectiveness. Such an arrangement would ensure that the project meets GPSA’s requirements while being well-aligned with sectoral priorities as well as fully integrated into the Bank’s sector-specific portfolio of activities. In particular, this arrangement allows for the supervision-stage TTL to start addressing upstream (at project initiation) potential issues around government ownership, mainstreaming/scale-up of activities, and sustainability, while ensuring technical soundness of the project’s design from both the governance and technical, sectoral perspectives, reducing the need for restructuring during implementation. The second challenge relates to adequacy of resources for supervision. This GPSA grant consisted of an RETF only, with no resources available from the TF to support supervision. If future GPSA-financed operations rely solely on Bank Budget (BB) for supervision of implementation, appointing a field-based TTL is a far more efficient use of the BB than an HQ-based TTL, especially given the high variable costs needed to cover the extremely large distances involved between remote project sites. . Page 24 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: One: To strengthen civic engagement in the education sector Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of stakeholders at Text 14.7 28 40% increase over 45.7 project schools engaged in baseline or 54.7% planning and oversight of school-level processes 13-Nov-2015 09-Dec-2014 13-Nov-2015 09-Dec-2018 Comments (achievements against targets): After the project restructuring in November 2015, the original PDO Indicator 1 ("Number of public schools with improved level of citizen satisfaction on education inputs") and its target (28 schools) were replaced with this new indicator and target. Objective/Outcome: Two: To improve transparency of budgeting and procurement processes Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Page 25 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) In target areas, percentage of Percentage 0.00 48.00 40.00 50.30 stakeholders participating in CSO-initiated budget 09-Dec-2014 09-Dec-2014 13-Nov-2015 09-Dec-2018 monitoring activities for education Comments (achievements against targets): After the project restructuring in November 2015, the original PDO Indicator 2 ("Number of public schools in the 8 aimags monitored by aimag CSOs that contributed to the disclosure and accountability in the budgeting and procurement process") and its target (48 schools) were replaced with this new indicator and target. A.2 Intermediate Results Indicators Component: One: Organization of Parent Teacher Associations (PTAs) and Monitoring of School Performance at the Soum (District) Level Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of PTAs established Number 0.00 28.00 31.00 09-Dec-2014 09-Dec-2014 09-Dec-2018 Comments (achievements against targets): After the project restructuring in November 2015, the annual targets (for Years 2 and 3) of Intermediate Results Indicator 1 were revised to reflect the new implementation schedule; its end-target (for Year 4) was not amended. Page 26 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of project schools that Number 0.00 84.00 56.00 58.00 completed the Good School Support Tool (GSST) processes 09-Dec-2014 09-Dec-2014 13-Nov-2015 09-Dec-2018 Comments (achievements against targets): After the project restructuring in November 2015, the original Intermediate Results Indicator 2 ("Number of PTA- initiated recommendations adopted by schools and soums") was replaced with this new indicator. Both the annual and end targets were revised to allow tracking of implementation of the GSST. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of recommendations Number 0.00 28.00 10.00 18.00 and policy actions/corrective measures developed to 09-Dec-2014 09-Dec-2014 13-Nov-2015 09-Dec-2018 address priority legislative and policy impediments that undermine effective stakeholder engagement at schools Comments (achievements against targets): Page 27 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) After the project restructuring in November 2015, the original Intermediate Results Indicator 3 ("Number of PTAs that have adopted action plans on improving transparency, accountability and school performance and services") was replaced with this new indicator. Both the annual and end targets were revised. Component: Two: Monitoring of Education Budget and Procurement at the Aimag (Provincial) Level Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion In project areas, number of Number 0.00 4.00 8.00 13.00 meetings of the Citizen's Representative Hural observed 09-Dec-2014 09-Dec-2014 13-Nov-2015 09-Dec-2018 by CSOs each year Comments (achievements against targets): After the project restructuring in November 2015, the original Intermediate Results Indicator 4 ("Number of recommendations submitted to parliament by the TAME Consultative Group") was replaced with this new indicator. Both the annual and end targets were revised. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of recommendations Number 0.00 100.00 10.00 13.00 and policy actions/corrective measures developed to 09-Dec-2014 09-Dec-2014 13-Nov-2015 09-Dec-2018 address priority legislative and policy impediments that Page 28 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) undermine transparency of education budget processes Comments (achievements against targets): After the project restructuring in November 2015, the original Intermediate Results Indicator 5 ("Level of involvement of non-state actors in monitoring education budget and procurement") was replaced with this new indicator. Both the annual and end (originally set at 100 percent) targets were revised. Component: Three: Managing Knowledge and Learning Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of case stories, tools Number 0.00 8.00 26.00 and other knowledge products developed 09-Dec-2014 09-Dec-2014 09-Dec-2018 Comments (achievements against targets): Page 29 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) A. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1: To strengthen civic engagement in the education sector 1. Percentage of stakeholders at project schools engaged in planning Outcome Indicators and oversight of school-level processes 1. Number of PTAs established 2. Number of project schools that completed the Good School Support Tool (GSST) processes Intermediate Results Indicators 3. Number of recommendations and policy actions/corrective measures developed to address priority legislative and policy impediments that undermine effective stakeholder engagement at schools 1. 31 PTAs were established. Key Outputs by Component 2. GSSTs were completed 58 times at the project schools. (linked to the achievement of the Objective/Outcome 1) 3. 18 recommendations and policy actions/corrective measures were developed. Objective/Outcome 2: To improve transparency of budgeting and procurement processes 1. In target areas, percentage of stakeholders participating in CSO- Outcome Indicators initiated budget monitoring activities for education 1. In project areas, number of meetings of the Citizen's Representative Hural observed by CSOs each year 2. Number of recommendations and policy actions/corrective Intermediate Results Indicators measures developed to address priority legislative and policy impediments that undermine transparency of education budget processes Key Outputs by Component 1. 13 meetings of Citizen’s Representative Hural were observed by (linked to the achievement of the Objective/Outcome 2) CSOs. Page 30 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) 2. 13 recommendations and policy actions/corrective measures developed. Page 31 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1: 0.18 0.23 128.0 Component 2: 0.09 0.05 50.8 Component 3: 0.08 0.08 92.7 Component 4: 0.30 0.30 99.7 Total 0.65 0.65 100.0 Page 32 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Transparency and Accountability in Mongolian Education Project Grant Recipient Portion of the Implementation Completion Report Project Background Transparency and Accountability in Mongolian Education (TAME) was a four-year project (2014-2018) funded by the World Bank/ Global Partnership for Social Accountability and implemented by Globe International Center NGO and its Civil Society Organization (CSO) partners. The Project development objective was to strengthen civic engagement in the education sector to improve transparency of budgeting and procurement processes for the delivery of better quality education services in western and central Mongolia. The project’s direct beneficiaries were targeted communities located in 20 (out of 141) soums (rural districts) in eight aimags (provinces) in western and central Mongolia. The targeted aimags in western Mongolia are Khuvsgul, Uvs, Bayan-Olgii and Khovd. In central Mongolia, the aimags are Arkhangai, Bulgan, Uvurkhangai and Sukhbaatar. The total grant issued by the World Bank/ Global Partnership for Social Accountability was $650,000. The project’s direct beneficiaries are 65,700 students, 131,400 parents and legal guardians, and 3,600 teachers. The project’s indirect beneficiaries are individuals and CSOs residing/ based outside the above- enumerated aimags and soums. The project expected to reach up to: (a) 500,000 individuals equivalent to about 15% of the nation’s population; and (b) 60 CSOs working on similar projects. Project Components TAME consisted of the following four components: 1. Organization of Parents Teachers Association (PTAs) at Soum level: Covered activities to strengthen the capacities of parents, teachers and school officials to form and manage effective Parent Teacher Associations, and engage with soum level officials to improve the local education environment. 2. Monitoring of education budget and procurement at Aimag level: The Project used the Budget Trust Tool (BTT) to strengthen the capacity of national and regional level CSOs, media organizations and relevant government staff to analyze and monitor government budget allocation, expenditures, and procurement in the education sector. 3. Managing knowledge and learning: Project Monitoring and Evaluation (M&E) is firmly tied to this component. 4. Project Coordination: Comprised of activities to provide supplemental project support, to include day-to-day project administration; vertical coordination across levels of implementation; (national – aimag – soum); horizontal coordination with agencies involved in the project (e.g., Ministry of Education, Culture and Science (MECS), Ministry of Finance (MOF), CSOs, etc.); monitoring of Page 33 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) implementation progress; internal and external evaluations (including audits) of project activities and results. Project Interventions and Social Accountability Approach The project implementing partners formulated the TAME theory of change as such: “Quality of education will be improved, if stakeholders – government, school administrators, parents, teachers and students – are more knowledgeable and constructively engaged in education governance and more accountable.� In order to engage the stakeholders in the education sector issues, the project applied three social accountability tools: 1) Organizing Parent Teacher Associations at the target 28 schools; 2) Conducting Good School Support Tool in order to address the environmental issues of the target schools and to support the schools to improve the environment as a community; and 3) Applying Budget Trust Tool in order to increase the transparency of the budgeting and procurement processes at the aimag level. Since the project interventions were introduced through brand new tools in Mongolia, the capacity building and behavioral change were at the core of the project activities throughout the project implementation. Not only the tools themselves were new, but also the method for engaging the stakeholders in a constructive manner, and encouraging the public officials on one hand, and the citizens and NGOs on the other hand, to collaborate together was a new approach. Changes and Outcomes in School Governance through Parent Teacher Associations (PTAs) The capacity building activities at the school level No. Description 2015 2016 2017 2018 1 Number of target schools 0 10 20 28 2 Number of PTAs established 0 5 20 31 3 Capacity building of Phase I participants 0 10 schools 10 schools 10 schools 4 Capacity building of Phase II participants 0 0 10 schools 10 schools 5 Capacity building of Phase III 0 0 0 11 schools participants (8 target + 3 non-target) PTAs are actively engaged in school staff, teachers and local residents, local government organizations and NGOs. The target schools’ cooperation with soum authorities, Department of Education, Culture and Art and Local Development Fund has increased. Table 1. Changes and outcomes in PTAs cooperation Stakeholders Cooperation Page 34 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) Parents � There is improvement seen in parents and teachers’ cooperation and attitude � Parents-Parents cooperation improved � Parents-Children cooperation improved Teachers � Teachers’ reach to students increased � Teachers’ attitude and cooperation has changed positively School � PTAs representatives were included in the school council authority � Two-way communication of parents-school improved � PTA participatory decision-making practice was established � Participation in the meetings increased � As a result of the school assessment, participatory decision making increased Local PTA activities played a critical role in improving local authorities’ increased engagement in government problems in school environment. The major local stakeholders are as follows: organizations � Aimag, Soum Governor’s office � Department of Education, Culture and Art � Family Development Division � Local Development Fund � Police Department, Road Police � World Vision International Mongolia � Private companies As a result of the PTA activities, there has been positive changes in transparency in school budget and improvement in parents’ awareness. For instance, parents started participating in discussions to plan school budget and to address, solve the urgent problems of school and monitor the budget allocation and discuss the report and participate in the activities. The target schools have amended their policies, activities, action plans and school rules and regulations to support PTAs activities. Establishment of the favorable legal and regulatory environment ensures sustainable implementation of the PTAs and GSST in the schools. Figure 1. Activities undertaken to improve school legal environment with involvement of PTAs Activities undertaken to improve school legal and regulatory environment � Clauses related to the PTAs were added to the school rules and regulations � Evaluation on the school staff participation � Assessment on the school environment (external and internal) � Statement in the school plan � School authorities improved their engagement with teachers and school staff � Quarterly report of school accountancy and finance is released � Representative of PTAs included in school lunch break procurement team members � Regular input and comments of parents, students and teachers are reflected into school annual plan � Financial and annual report is introduced to parents, the public, school authorities and PTA. � PTAs made the contract with the teachers and publicized the contract on the school website. � The rules of dormitory, on whose formulation teachers, parents and students themselves involved, are disclosed publicly on information boards Main activities of the project include improving school budget and financial transparency, monitoring implementation and addressing the common issues through broad involvement of parents, teachers and Page 35 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) students. Therefore, PTAs are organizing trainings and activities to improving members, parents and teachers’ awareness and knowledge of school budget, budget monitoring, evaluation methodologies and discussing the issues with their involvement and conducting monitoring-evaluation and introducing their reports. Changes in the learning environment through Good School Support Tool (GSST) The number of GSST respondents and percentage of the engagement No. Description 2015 2016 2017 2018 1 Number of GSST conducted 28 (in order to 10 30 58 determine the baseline) 2 Number of total respondents 1151 452 1397 2069 3 Number of respondents who participated in 169 130 444 946 either “Budget planning�, “Annual plan� or “Conducting GSST� 4 Percentage (of the stakeholders who 14.7% 28.8% 31.8% 45.7% participated) Under implementation of the GSST, the school joint team conducted assessment on the psychological, physical environments and school governance to improve the learning environment. By the result of the school environment assessment, joint team identified number of problems in the local schools. For instance, the teachers’ workload is high and the children are not sufficiently drawn into the extracurricular activities because of inadequacy of budget and material resources. Responding this demand, PTAs under GSST in collaboration with parents gained experience of organizing competitions, celebrations, campaigns, events and trainings. As a result of this assessment, a series of action taken by PTAs to improve school physical and psychological environments and support teachers saw considerable positive results. With physical environment assessment, the target schools were identified to have inadequate budget and high in degradation condition and insufficient in facilities and book resources. It was mainly because of the lower the number of students, the less centralized budget allocation. Changes and outcomes for Transparency in Education Sector No. Description 2015 2016 2017 2018 1 Number of BTT conducted 0 3 6 3 2 Number of target participants 0 1934 3786 5505 3 Number of stakeholders who participated in 0 380 2454 2771 Page 36 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) the NGO-initiated budget monitoring activities 4 Percentage (of the stakeholders who 0 19.7% 64.8% 50.3% participated) Budget transparency is considered to be one of the most critical aspect of the good governance across any sectors of social development. This project aimed to improve transparency and accountability of budget and procurement in education sector. Under this component, project aimed to implement BTT selected NGOs at the aimag level. However, it can also be seen that project might have generated some positive impact on particular aspects of the transparency in education. For example, Transparency of the budget information in Khuvsgul aimag has increased significantly over the project period. This also can be supported by the qualitative data. Figure 2. Scope of budget transparency among stakeholders Before the project intervention After project intervention Local Other authority Local Other government authority /Governor's government organizations /Governor's office, etc./ organizations office, etc./ CSOs CSOs School Teachers Citizens, Parents community According to the qualitative information collected through interviews, training and other advocacy activities of the project impacted in increasing numbers of the stakeholders except students in budget transparency process. With training and wider cooperation between the stakeholders, school management, teachers, parents became knowledgeable about budget process and allocated budgets and its expenditure. By the findings of the qualitative interviews, only there is low change in participation of the students in discussions of education sector’s budget. For the implementation of the BTT, following outcomes are concluded: Page 37 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) � Positive change in knowledge and attitudes of the stakeholders, community and government for more increasing transparency and open discussions; � Building basic capacity of the CSOs for further continuity of the BTT and capacity building of the stakeholders in terms of budget transparency actions; � Increased transparency of the budget information and active discussions of the stakeholders on education budget. Page 38 of 39 The World Bank Transparency and Accountability in Mongolian Education (TAME) (P150842) ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) 1. World Bank. 2012. Mongolia: Country Partnership Strategy for the Period FY2013–2017. 2. ———. 2014. Project Proposal Paper for GPSA Grant US$ 65,000 to Globe International Center Mongolia for a Transparency and Accountability in Mongolian Education (TAME) Project. 3. ———. 2015. Restructuring Paper on a Proposed Project Restructuring of Mongolia Global Partnership for Social Accountability—Transparency and Accountability in Mongolian Education Project. 4. ———. 2017, Mid-Term Review: Transparency and Accountability in Mongolian Education (TAME) Project. 5. Implementation Status and Results Reports. 6. Aide-Memoires. 7. GPSA Grantee Annual Reports (2015–2017) and GPSA Grantee Mid-Term Technical Progress Reports (2016–2017)* on the Transparency and Accountability in Mongolian Education Project. * GPSA Grantee Annual Report and Mid-Term Technical Progress Report for 2018 have not yet been submitted by GIC as of June 10, 2019. Page 39 of 39