Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004658 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF0A2955) ON A SMALL GRANT FROM LOCAL SOLUTIONS TO POVERTY MULTI DONOR TRUST FUND IN THE AMOUNT OF USD 4,987,000 TO THE Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI) FOR INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru - P159191) October 7, 2019 Social Development Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective October 7, 2019) Currency Unit = United States Dollars IDR 14,137 = USD1 IDR 1 = USD0.00007 FISCAL YEAR January 1 - December 31 Regional Vice President: Victoria Kwakwa Country Director: Rodrigo A. Chaves Regional Director: Benoit Bosquet Practice Manager: Nina Bhatt Task Team Leader(s): Dewi Susanti ICR Main Contributor: Paul Gerard M Boon ABBREVIATIONS AND ACRONYMS ACDP Analytical and Capacity Development Partnership APBD Anggaran Pembangunan dan Belanja Daerah (District Government Budget) APBDes Anggaran Pembangunan dan Belanja Desa (Village Government Budget) ASA Advisory Services & Analytics BPS Badan Pusat Statistik (Central Bureau of Statistics) CEM Community Empowerment Mechanism CPF Country Partnership Framework CY Calendar Year DFAT Government of Australia’s Department of Foreign Affairs and Trade GoI Government of Indonesia GP Global Practice IDR Indonesian Rupiah IE Impact Evaluation ISR Implementation Status and Results Report IRI Intermediate Results Indicator MDTF Multi Donor Trust Fund MoEC Ministry of Education and Culture MoV Ministry of Villages, Disadvantaged Regions and Transmigration OECD Organizations for Economic Co-operation and Development PDO Project Development Objective PPM Pay for Performance Mechanism RF Result Framework SMERU Social Monitoring and Early Response Unit, a research institute TAS Teacher Absence Survey TNP2K Tim National Percepatan Penanggulangan Kemiskinan (National Team for Acceleration of Poverty Reduction) TSA Teacher Special Allowance (Tunjangan Khusus Guru) USD United States Dollar WB World Bank TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 10 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 17 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 24 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 31 ANNEX 3. EFFICIENCY ANALYSIS ........................................................................................... 32 ANNEX 4. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 36 ANNEX 5. SUMMARY OF RECIPIENT COMPLETION REPORT .................................................. 37 ANNEX 6. SUMMARY OF IMPACT EVALUATION .................................................................... 39 ANNEX 7. SUMMARY OF QUALITATIVE RESEARCH ............................................................... 41 REFERENCES ........................................................................................................................ 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name INDONESIA: IMPROVING TEACHER PERFORMANCE AND P159191 ACCOUNTABILITY (KIAT GURU) Country Financing Instrument Indonesia Investment Project Financing Original EA Category Revised EA Category Organizations Borrower Implementing Agency Tim Nasional Percepatan Penanggulangan Kemiskinan Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI) (TNP2K) Project Development Objective (PDO) Original PDO The Project Development Objective is to improve the presence and service quality of primary school teachers in targeted areas. The objective will be reached by making teachers more accountable and by empowering communities to participate in education. PDO as stated in Legal Agreement (if different from Project Paper) The Project Development Objective is to improve the presence and service quality of primary school teachers in targeted areas. Page 1 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A2955 4,987,000 4,987,000 4,987,000 Total 4,987,000 4,987,000 4,987,000 Total Project Cost 4,987,000 4,987,000 4,987,000 KEY DATES Approval Effectiveness Original Closing Actual Closing 27-Jun-2016 30-Jun-2016 31-Dec-2017 29-Mar-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 22-Sep-2017 2.51 Additional Financing Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Procurement Change in Implementation Schedule 13-Jun-2018 4.40 Change in Results Framework Change in Loan Closing Date(s) Change in Implementation Schedule 10-Dec-2018 4.99 Change in Results Framework Change in Loan Closing Date(s) Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial Page 2 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 06-Oct-2016 Satisfactory Satisfactory 0.00 02 20-Apr-2017 Moderately Satisfactory Moderately Satisfactory 1.73 03 09-Aug-2017 Moderately Satisfactory Moderately Satisfactory 2.25 04 04-Feb-2018 Moderately Satisfactory Moderately Unsatisfactory 3.30 05 06-Jun-2018 Satisfactory Moderately Satisfactory 4.40 06 07-Dec-2018 Satisfactory Moderately Satisfactory 4.99 ADM STAFF Role At Approval At ICR Regional Vice President: Victoria Kwakwa Victoria Kwakwa Country Director: Rodrigo A. Chaves Rodrigo A. Chaves Director: Ede Jorge Ijjasz-Vasquez Benoit Bosquet Practice Manager: Kevin A Tomlinson Nina Bhatt Task Team Leader(s): Robert Wrobel Dewi Susanti ICR Contributing Author: Paul Gerard M Boon Page 3 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES CONTEXT AT APPRAISAL 1. National Context. Indonesia’s national poverty rate fell from 24 percent in 1998 to 11.3 percent in 2015, although the pace of poverty reduction had slowed since 2012. The sustained period of growth and macroeconomic stability experienced from 2005 to 2015 was accompanied by increased inequality. Rural areas were lagging, having consistently higher rates of poverty (14.1 percent) compared to urban areas (8.2 percent), with poorer connectivity, and lower quality of basic services (BPS, 2016). 2. Sectoral and Institutional Context. The Government of Indonesia (GoI) recognized the importance of education in reducing poverty and inequality, and since 20091 had allocated 20 percent of the national and 20 percent of district government budgets to education. While regional disparities persisted, education access had improved, particularly for those living in rural and remote areas (SMERU, 2004; BPS, 2016; WB, 2013). Access however, did not necessarily correlate with learning outcomes. Studies show that Indonesian student learning outcomes remained near the bottom ranks compared to other countries (WB, 2013; OECD 2014). In addition to learning outcomes, the quality of education and equity in terms of outreach remained problematic in rural compared to urban areas (ACDP, 2014; Stern & Nordstrum, 2014). 3. Challenges in frontline education service delivery. There were multiple challenges associated with improving the quality of education especially in poor rural and remote areas. Teacher absenteeism had declined over the past decade yet remained high in remote areas (19.3%) compared to the national average (9.4%) (SMERU, 2004; ACDP, 2014). When teachers were absent, students tended to not show up at schools, their dropout rates were higher, and learning outcomes poorer (UNICEF, 2012; WB, 2013; SMERU, 2004; Suryadarma et al, 2004; Suryahadi and Sambodho, 2013). For the recipients of education services, communities seldom had the resources (time and money) to monitor teacher or school performance. They did not have the tools to submit feedback or complaints, and to demand accountability. Similarly, communities did not have the correct information on which to act, including on minimum service standards, education budgets, and teacher incentives among others. There is a Teacher Special Allowance (Tunjangan Khusus Guru, hereafter TSA) in place, which can be up to one times the base salary. Yet, a study showed that TSA-recipients had higher absenteeism compared to non-recipients (SMERU, 2010). This raises questions about teacher accountability, since the monetary incentive is in place. 4. Government of Indonesia policies to improve education service delivery. At the time of appraisal, GoI’s strategy for improving the quality of basic education had a greater focus on performance by making teacher allowance performance-based. The introduction of the Civil Servant Law (UU 5/2014) provided the authorizing environment for performance-based evaluations and incentives for civil servants. The Village Law (UU 6/2014) also provided the legal and regulatory framework enabling communities to provide investments in basic services delivery using Village Funds granted by the central government. 5. Rationale for World Bank Engagement. This Project, Kinerja dan Akuntabilitas Guru (“Improving Teacher Performance and Accountability,� hereafter KIAT Guru), was initiated in 2014 and pre-piloted in 31 primary schools by the National Team for the Acceleration of Poverty Reduction under the Secretariat of the Vice President of 1 Based on Law 20, 2003. Page 4 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Indonesia Office (Tim Nasional Percepatan Penanggulangan Kemiskinan, hereafter TNP2K), in collaboration with the Directorate General for Teachers and Education Personnel, under the Ministry of Education and Culture (MoEC). In 2016, MoEC and TNP2K requested the Bank’s support to implement KIAT Guru at a larger scale using the TSA. The Project sought to empower communities to demand better services and make TSA performance- based on either teacher presence or service quality. The World Bank was uniquely positioned to support the GoI to operationalize, test, and rigorously evaluate the most effective option to improve education service delivery. The World Bank had supported decades of Community Driven Development in Indonesia accompanied by evaluations and reviews on education expenditure effectiveness of the Teacher Certification Allowance. These factors and experiences provided the framework for this Project, which was aligned with the fourth pillar on delivery of local services as articulated in the FY 2016-21 Country Partnership Framework (CPF) for Indonesia. It was in line with the CPF’s focus on supporting local governments to establish performance-based fiscal transfer systems and strengthening the capacity of central government agencies to support and oversee the performance of local governments. BaKTI was selected as the Grant Recipient for this Project and received funding under the Local Solutions to Poverty MDTF2 (TF-A2955). Theory of Change 6. Results Chain. The Project design accounted for the fact that despite significant increased spending on teacher salaries and allowances, student learning outcomes remain poor. The system did not link teacher allowances with any performance measure. This Project was based on the premise that when parents were engaged in improving learning, communities were committed to support and monitor education service delivery, and teacher allowance was tied with teacher performance, then student learning outcomes would improve. The Project tested two approaches: (a) Community Empowerment Mechanism (CEM), which brought community members and teachers to develop Service Agreements and Community Score Cards, and form User Committees to represent community members; and (b) Pay for Performance Mechanism (PPM), which tied the payment of TSA with either teacher presence or teacher service quality. Table 1 describes the theory of change on how the activities and outputs could lead to desired outcomes and impact. Project Development Objectives (PDOs) 7. PDO. The PDO was to improve the presence and service quality of primary school teachers in targeted areas. The PDO indicators were: (i) improved teacher presence, and (ii) improved teacher service quality. Key Expected Outcomes and Outcome Indicators 8. Key results. The key-results for the PDO were: a. Improved teacher presence rate. b. Improved community satisfaction on teacher service quality. 9. Project Beneficiaries. The beneficiary targets were: (a) 15,000 elementary school students benefiting from improved education services; (b) 1,800 User Committee members; (c) 1,400 school providers (principals and teachers); and (d) 200 Village Cadres. 2 Local Solutions to Poverty (LSP) is a Multi-Donor Trust Fund that’s assisting the Government of Indonesia to improve the quality of life of poor and vulnerable Indonesians. When the Project was initiated, there were only two remaining donors to the MDTF: The Government of Australia’s Department for Foreign Affairs and Trade and the Millennium Challenge Account-Indonesia. Page 5 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) 10. Participating Districts. At appraisal, participating districts were to be selected from a list of 306 districts that received the TSA, as set by the Ministry of Education and Culture (MoEC), and a list of 122 disadvantaged districts, as set by Ministry of Villages, Disadvantaged Regions and Transmigration (MoV)3. Based on this criteria, five districts were selected: Sintang, Ketapang, and Landak in West Kalimantan province, and Manggarai Barat and Manggarai Timur in East Nusa Tenggara province. Table 1. KIAT Guru Theory of Change. Problem: Despite significant increased spending on teacher salaries and allowances, student performance remains lackluster due to inadequate teacher presence and performance standards and accountability. Intermediate Results Indicators (IRIs) PDO-level Activities Long-term Goals Outputs Outcomes Outcomes Component One: Community Empowerment Mechanism (CEM) Number of Village Cadre and User Committee members trained on Develop training modules; conduct conducting monitoring and Improvements in community trainings evaluation of teacher service monitoring and evaluation of performance teacher presence and teacher Service Agreements and Community service quality based on the Conduct meetings between Score Cards developed Community Score Cards community members and teachers Number of Community Score Card agreed Number of village monthly meetings Assist community members in conducted, where User Committees User Committees appointed selecting User Committees share monitoring and evaluation results Service Agreements, Community Improvements on stakeholders Score Cards, and User Committees Improvements in Conduct evaluation of teacher engagement and on indicators that Improvements in reviewed and revised teacher presence service quality directly contribute to teacher student learning Number of end-of-semester and teacher service presence and service quality outcomes meetings conducted quality Component Two: Pay for Performance Mechanism (PPM) Number of MoEC and district government officials receiving capacity building on planning, Improvements in government Provide technical assistance for implementing, monitoring, capacity to oversee teacher MoEC and District Governments aggregating reports from schools, presence and teacher service quality and conducting performance-based teacher allowance payment Review relevant policies and regulations Relevant government regulations Percentage of Tunjangan Khusus Conduct workshops and meetings issued, including decrees, and allowance for teachers assigned to on policy discussions technical guidelines revised and PPM models are paid according to Develop relevant managerial and promulgated in the project areas KIAT Guru formula administrative instruments with governnent counterparts Indicators tracked in the Project's Results Framework 3 Eligible districts had a minimum quota of 120 recipients for TSA, a minimum of 40 remote primary schools, and below the national average Human Development Index and education standards. Participating villages had at least 300 households, with participating schools had a minimum of three TSA recipients and more than 70 students. Page 6 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) 11. With a total grant of USD 4,987,000, BaKTI implemented KIAT Guru in 203 primary schools. In all schools, the Project facilitated the formation of a Service Agreement between teachers, parents, and community leaders which delineated actions to improve education that each stakeholder commits to. The Service Agreement included teacher-specific Community Score Cards, consisting of between five to eight service performance indicators, one of them had to be teacher presence, with specified target actions that each teacher must perform. In addition, the Project also facilitated the formation of a User Committee that is responsible for monitoring and evaluating each teacher. The User Committee, as representatives of community members, conducted spot-check monitoring of Community Score Card indicators. At the end of every month, the User Committee met with teachers to present them with findings from the spot-check monitoring and teacher evaluation scores for the Community Score Card indicators. The achievement of the PDO indicators were measured mainly through the monthly Community Score Card scores for teacher presence and teacher service quality. 12. The Project implemented two mechanisms, Community Empowerment (CEM) and Pay for Performance (PPM), which were combined into three intervention groups and impact evaluated. All three intervention groups included CEM, described in previous paragraph, and different PPM versions: (1) CEM (without PPM); (2) CEM + PPM based on teacher presence; and (3) CEM + PPM based on teacher service quality. The PPM made conditional the amount of TSA received by eligible teachers. In other words, the Project did not advocate for the Government to provide additional allowance to teachers, but rather, to make available teacher allowance performance-based. Teachers in Group 2 schools were required to record their presence using a mobile-phone application, and at the end of every month the User Committee verified whether teacher absences were excused or not. Teachers’ TSA were cut depending on their tardiness and types of absences. On the other hand, TSA for teachers in Group 3 were paid based on the User Committee’s evaluation of their Community Score Cards. 13. An Integrated Advisory Services and Analytics (ASA – P160640) evaluated the three intervention groups against a control group, where no KIAT Guru intervention was conducted. The design for testing the intervention groups was robust and efficacy of the interventions was explicitly tested through an Impact Evaluation (IE), which randomly assigned a total of 270 schools into the three intervention and one control groups. The ASA provided the interventions design and the Project monitoring, learning, and evaluation design. In addition to the IE (see Annex 6), the ASA also included a qualitative research in nine villages (see Annex 7). These complementary efforts informed the PDO indicators and provided additional data points collected through Project monitoring. Components 14. KIAT Guru originally consisted of two components: a. Component One: Community Empowerment Mechanism (CEM), in the amount of USD 2,536,000, to provide community members with an explicit role to monitor and evaluate teacher service performance and to ensure teacher accountability. This component financed: (i) training and coaching of Village Cadres;4 (ii) capacity building for parents and community members; (iii) facilitated meetings between community members and teachers to develop Service Agreements and Community Score Cards; (iv) assisting 4 Village Cadres were residents at the village level who were identified based on the following criteria: (i) previously active in some village-level community organizations; (ii) had basic literacy and numeracy skills; and (iii) willing to improve their capacities to facilitate meetings. Page 7 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) community members in selecting User Committees;5 (v) facilitating evaluation of teacher service quality; and (vi) setting up mechanisms for community members to share their inputs on education service improvements to village, sub-district, and district governments. b. Component Two: Pay for Performance Mechanism (PPM), in the amount of USD 1,087,000, to link the payment of the TSA with teacher presence or teacher service quality and to support activities to engage stakeholders and inform policy. This component financed: (i) reviewing relevant existing regulations; (ii) organizing meetings and workshops with relevant government and non-government stakeholders on policy review and recommendations; and (iii) providing technical inputs to government and non- government stakeholders in developing relevant managerial and administrative instruments. Significant Changes 15. The Project experienced three restructurings. a. September 2017 Level 2 Restructuring. This Restructuring resulted in: (i) an additional financing of USD 1,364,000; (ii) revision of the Project components and budgets per component; (iii) an extension of the closing date from December 30, 2017 to June 29, 2018; (iv) changes to the Results Framework (RF) to better track progress towards achieving the PDO; and (v) revision of the Intermediate Results Indicators (IRIs), including to extend the implementation period. Changes in regulatory framework resulted in expansion of the Project’s geographical scope and delayed implementation (as further described in paragraph 40). MoEC and TNP2K requested to extend the implementation period to enable evaluation of the PPM impacts over a period greater than six months. The additional period also provided capacity development trainings for sustainability of the Project activities by beneficiaries. b. June 2018 Level 2 Restructuring. This Restructuring resulted in: (i) no cost extension of the Grant Agreement from June 29, 2018 to December 31, 2018; (ii) modification of the implementation schedule; and (iii) amendment of the RF to adjust to the implementation period. During implementation, MoEC and five district governments co-financed the Project, in the amount of USD 573,000 (70%) more than allocated at the start of the Project (USD 820,000). Hence, BaKTI as the Grant Recipient was able to save Project’s funding and requested an extension to provide additional capacity development trainings and technical assistance to strengthen sustainability of the Project activities. c. December 2018 Level 2 Restructuring. This Restructuring resulted in: (i) no cost extension of the Grant Agreement from December 31, 2018 to March 29, 2019; (ii) modification of the implementation schedule; and (iii) amendment of the RF to adjust to the implementation period. BaKTI requested another extension to implementation period, as a result of additional savings from spending efficiency and gain in currency exchange. Additional technical assistance was provided to the government to expand the Project. 16. Increase in Grant Financing and Project Components. The Project was funded by the Local Solutions to Poverty MDTF, which at the time of appraisal only had one remaining donor, i.e. the Government of Australia’s Department of Foreign Affairs and Trade (DFAT). The 2017 Restructuring increased DFAT’s grant amount from USD 3,623,000 to 5 The User Committee formation needed to meet the criteria of: a minimum of 9 community members with a gender-balanced composition, 1/3 parents, 1/3 concerned citizens, and 1/3 community and religious leaders to provide linkage to the village council. Page 8 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) USD 4,987,000 (38% additional)6. It also added a third component “Technical Assistance and Project Management� to clearly delineate Project costs directly paid for activities with beneficiaries of CEM and PPM activities, from the costs of Technical Assistance (USD 2,339,875) and Project Management and operations (USD 904,095) in both original components. This provided a clear cost structure for future replication planning. Table 2 presents the comparison between original and revised budget per component. The costs for Integrated ASA, in the total of USD 1,685,0007, are excluded from this Project’s efficiency analysis. Table 2. Costs by Component (USD): Original and Revised Project Components Original Component Costs Revised Component Costs 1. Community Empowerment Mechanism (CEM) 2,536,000 1,589,440 2. Pay for Performance Mechanism (PPM) 1,087,000 153,590 3. Technical Assistance and Project Management ---- 3,243,970 Total Project Costs 3,623,000 4,987,000 17. PDO Indicators. The initial PDO indicators were based on data collected through the baseline and endline surveys implemented as part of the Integrated ASA. Since the PDO indicators needed to be updated more frequently, that is, over the course of implementation, the team identified different sources of data as more appropriate for the Project to monitor progress on PDO indicators. During the September 2017 Restructuring, the PDO indicators were changed, to be based on community evaluations of teacher presence and service quality scores collected through the Project monitoring. a. PDO indicator 1 was changed from “improved teacher presence rate� based on Teacher Absence Survey, to “improved teacher presence score� based on a required Community Score Card indicator on teacher presence. The baseline value and endline target became 65 and 85 respectively. b. PDO indicator 2 was changed from “improved community satisfaction on teacher service quality� based on survey questionnaires, to “improved teacher service quality score� based on Community Score Card indicators on teacher service quality (including a required indicator on teacher presence). The baseline value and endline target became 56 and 85 respectively. Table 3. Original and Revised IRIs Original IRIs IRIs after 2017 Restructuring8 1 Issuance of relevant government regulations From “text� mentioning the regulations to “yes/ no� when the extensions of the regulations have been issued 2 Percentage of villages assigned to CEM models Number of Community Score Cards agreed completed the initial phase 3 Percentage of villages assigned to CEM models Cumulative number of village monthly meetings 6 Since this is a small grant, the original grant amount, at USD 3,623,000 did not get reflected in the Data Sheet, as the Additional Financing was added to the same TF number. 7 The Integrated ASA costs consisted of USD 785,000 from the Local Solutions to Poverty MDTF, USD 450,000 from the Local Solutions to Development SDTF from USAID, and a USD 450,000 co-financing from SMERU Research Institute to fund KIAT Guru endline survey, as part of the RISE (Research in Improving Systems in Education) Indonesia study. 8 The rationales for changes to each of the IRI, and detail explanation for each are explained in the Results Framework. Page 9 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) completed the implementation phase conducted 4 Percentage of villages assigned to CEM models Cumulative number of end-of-semester meetings completed the evaluation phase conducted 5 Percentage of Tunjangan Khusus allowance for Percentage of Tunjangan Khusus allowance for teachers assigned to PPM models are paid teachers assigned to PPM models are paid according to KIAT Guru formula according to KIAT Guru formula – End target (from 70 to 85%) and end date (From December 2017 to April 2018) changed 6 ---- Number of MoEC and district officials received capacity building 7 ---- Number of Village Cadres and User Committee members received capacity building 18. Intermediate Results Indicators (IRIs). The Project had five IRIs, of which three were linked to Component One, the CEM (indicators 2, 3 and 4), and two were linked to Component Two, the PPM (indicators 1 and 5). The September 2017 Restructuring revised the five IRIs without changing the substance of the indicators. Two IRIs were added to align with the additional Component Three, TA and Project Management. Table 3 presents the original and revised IRIs. II. OUTCOME Assessment of Achievement of Each Objective/Outcome Objective 1: KIAT Guru’s Contribution to ‘Improved Teacher Presence’ 19. Achievement of this objective is Substantial. The Community Score Card indicator for improved teacher presence rose from a baseline of 65 to 95.13, surpassing an end-target of 85 (112% achievement). Figure 1 shows the monthly trend on teacher presence score. Data was also collected through unannounced Teacher Absence Survey (TAS) as part of the two rounds of IE surveys by independent enumerators. In addition, the Project implementation team conducted another TAS after one semester of implementation. The three rounds of TAS in 203 schools (without the control schools) found that the percentage of classrooms with teachers increased from 75% to 87% (12% increase), also surpassing the original target of a 7.5% increase for the original PDO indicator. Similarly, teacher presence was found to be one of the most significant changes mentioned by stakeholders in all nine qualitative research schools conducted through the ASA, based on randomly sampled survey to 20 parents in each of the schools. 20. Group 2, which combined CEM with PPM based on teacher presence, had the most significant impact on improved teacher presence. Teacher presence in Group 2 was recorded using KIAT Kamera, a mobile-phone-based camera application, and verified by the User Committee, which determined the amount of TSA received by eligible teachers. The IE findings, as summarized in Table 4, identified Group 2 as having the most positive (statistically-significant) impacts on teacher presence, particularly for TSA recipients. In contrast, the IE found teachers in Group 3 (which combined CEM with PPM based on teacher service quality score, evaluated by the User Committee) responded differently, with those who did not receive the TSA becoming more absent from class (albeit non-significant statistically). Qualitative research found that stakeholders in Group 2 considered the tamper-proof KIAT Kamera to be objective, and the User Committee’s role to verify teacher presence to be appropriate. Interviews in nine qualitative study schools found 78% of the education stakeholders observed improvements in teacher attendance and 55% noted an improvement in teacher discipline. The key to success, it appears, was to implement a system of recording attendance that teachers view as objective and reliable, and to attach a monetary incentive to the desired Page 10 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) behavior, in this case presence. Figure 1. Monthly Teacher Presence Score Table 4. Summary of IE Results on Teacher Presence per Intervention Groups Objective 2. KIAT Guru’s contribution to ‘Improved teacher service quality’ 21. Achievement of this objective is Substantial. The Community Score Card indicators for improved teacher service quality rose from 56 points to 96.83 compared to the target of 85 (114% achievement). Figure 2 shows the monthly trend on teacher service quality score. The evidence of a significantly improved teacher service quality was verified through monthly monitoring and evaluation of the Community Score Card, consisted of 5 to 8 indicators on teacher performance (including presence) as agreed between teachers and User Committees. IE results also showed Page 11 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) improvement in parents’ satisfaction with education quality, from a baseline of 28% to an end-line of 64%. Parents also reported an increase on average annual meetings between parents and teachers on learning outcomes from 1.8 to 3 meetings per year, and on other topics from 1.1 to 2.5 meetings per year. The stakeholders’ interviews in the qualitative research reported that teachers had displayed increased seriousness about their professional responsibilities since KIAT Guru was initiated. The most common examples of improved teacher performance included the following behaviors: spending more time preparing lessons; assigning and correcting homework more regularly; including a greater variety of instructional approaches in lessons; receiving and responding to feedback from principals; investing more time and effort in supporting students, inside and outside of the classroom; and using more positive behavioral reinforcement and avoiding corporal punishment of students. Prevalence of these behaviors varied across schools conditional to the contents of the Service Agreements developed. In summary, teachers were aware of the Service Agreements, knew that their monthly performance score were dependent on their adherence to those terms, and adjusted their behavior accordingly. Figure 2. Monthly Teacher Service Quality Score Overall Outcome Rating 22. The overall outcome rating is Satisfactory (Table 5). The Project’s relevance at entry and completion remained high, as demonstrated by Government’s issuing regulations for the Project, co-financing some of the Project’s activities in the amount higher than initially committed, and expanding the Project’s scope starting in 2019. The Project’s efficacy was high, as demonstrated through its above-target achievements on the two PDO, all IRIs indicators, and the number of beneficiaries. The Project also achieved its longer-term impact on improving learning outcomes. The Project was extended for a total of 15 months with 38% of additional grant financing. GoI paid for 32 percent of the total Project implementation costs. Project’s efficiency analysis (Annex 3) shows that the one-time-investment cost of implementation by Project facilitators was at USD 5,058 per school, USD 40 per student, 8% of annual Village Funds. The cost of sustaining the Project’s activities is USD 2,182 per school, USD 17 per student, 3.5% of annual Village Funds. The cost is USD 506 per school or USD 4 per student higher for Group 2 schools, to cover for purchasing of mobile phone and maintenance of the application. After one year of intervention, Group 2 improved learning outcomes by 0.2 standard deviation, at USD 43 per student. This means it costs USD 22 per student per 0.1 standard Page 12 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) deviation increase. After two years of intervention, the learning outcomes decreased to 0.14 standard deviation, with an annual cost dropped to USD 21 per student, or USD 16 per 0.1 standard deviation increase. Compared to other rigorously evaluated interventions in education that improved learning outcomes, the cost of KIAT Guru was on par with interventions that adopted similar approaches (paragraph 65, Annex 3). There were no serious financial management or procurement issues during Project implementation. At Project completion, the Financial Management, Procurement, and Safeguards registered no findings. Table 5. Project’s Rating Relevance Efficacy Efficiency Overall Rating High Substantial Substantial Satisfactory 23. Capacity building. During the 2017 Restructuring, additional IRIs for the capacity building for MoEC and district government officials and Village Cadres and User Committee members were added. Achievement against the final targets were as follows: (i) Capacity building activities for MoEC and district officials were conducted for 1,103 staff, compared to a target of 750 (147% achievement); and (ii) Capacity building activities conducted for 2,628 Village Cadres and User Committee members, compared to a target of 2,000 (131% achievement). 24. KIAT Guru beneficiary targets were exceeded. The Project has achieved these outcomes in two provinces, five disadvantaged districts and 203 schools. The Project hired facilitators to facilitate the development of Service Agreements and Community Score Cards, to identify Village Cadres, to establish the User Committees, and to develop the capacity of the Project’s beneficiaries. Over the course of the Project, Village Cadres, parents and community members improved their skills and knowledge to monitor and evaluate teacher presence and service quality. Beneficiaries included: a. 1,843 User Committee members (45% female) trained and coached for monitoring and evaluation (compared to the target of 1,800); b. 203 Village Cadres (26% female) trained and mentored to facilitate the monthly meetings (compared to the target of 200); c. 25,942 elementary school students (47% female) benefitted from improved education services (compared to the target of 15,000); and d. 2,621 school providers (48% female) were involved (compared to the target of 1,400). 25. Ensuring gender balance in KIAT Guru activities required continuous efforts to counter power dynamics, and cultural and leadership habits in villages and schools. The Project required that the user committee composition and village meeting participation be gender-balanced. Female participation increased over the course of the Project’s timeline, from 38% during initial village meetings; to 46% during monthly meetings; and 49% during evaluation meetings. However, during spot check monitoring, it was apparent that women’s active participation in discussions and decision-making was not widespread and required a lot of encouragement. Women participation in capacity development trainings was much less, at 14% for training on sustaining KIAT Guru activities, and 27% on conducting evaluation meetings. Since this capacity development trainings were added after the Project’s approval, the Project’s facilitation manual did not include requirement for gender-balance participation. Page 13 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Other Outcomes and Impacts 26. All KIAT Guru intervention groups improved learning outcomes. Parental efforts in providing support to student learning significantly improved for all intervention groups, along with student learning outcomes (see Table 6). The largest impacts were found in Group 2, CEM + PPM based on teacher presence, with mathematics and Indonesian language at 0.19 and 0.17 standard deviations respectively (equal to 3 months more of learning annually compared to the control group). Compared to various rigorously-evaluated education interventions examined in Kraft (2019), these are substantial effect sizes (at around the 70th percentile). The IE proves that tying teacher pay to teacher presence, verified by community representatives, can significantly improve parent efforts and student learning outcomes. Impact on teacher efforts, although significant, were not as pronounced. Since all three intervention groups implemented CEM, the effect of CEM could not be separated from the effect of PPM, as the interaction between the two mechanisms might be additive or multiplicative. While Group 2 tied TSA with teacher presence only, teacher service quality was also scored monthly and reported to the village and district stakeholders. Hence, the effect of PPM on presence could not be separated from the evaluation on teacher service quality. In contrast, the qualitative research found that teachers in Group 3, CEM + PPM based on teacher service quality, resulted in issues with the community members (see Annex 7), with the IE found non-TSA-recipient teachers in Group 3 showing up less in classrooms (see Annex 6). 27. The IE and qualitative research attributed the positive results to five key elements: (i) actively engaging external stakeholders in monitoring and evaluating teacher performance; (ii) increasing parental involvement in learning; (iii) keeping teacher performance evaluation focused on a few, simple indicators; (iv) using objective rather than subjective measures to evaluate teacher performance; and (v) paying teacher allowance based on objective performance indicators. These key-elements will need to be considered if or when the approach is being replicated as part of GoI’s efforts to improve teacher performance. Table 6. Impact on Parental Efforts and Student Learning Outcomes per Intervention Group Group 2: Group 3: Group 1: Main outcomes CEM + PPM CEM + PPM CEM on Presence on Quality Parent efforts Overall ++ ++ ++ Mathematics 0 ++ + Learning outcomes Indonesian + ++ + ++ stronger positive effect + positive effect 0 no effect - Negative effect 28. Sustainability of impacts over time. The different mechanisms were tested and the results went through a rigorous evaluation, which provided evidence that the CEM plus PPM based on teacher presence showed the greatest potential to produce long-lasting change.9 GoI, with support from the Bank, has expanded the Project to a Phase 2, which included an ASA to conduct a third round of data collection on Phase 1 schools. The third round of surveys aimed to identify whether Phase 1 impacts on learning outcomes could be sustained after Project facilitators left 9 This is in line with recent research in Indonesia that underscores the importance of developing reform approaches that fit the capacity of local actors and can be institutionalized within existing social and governmental structures (Cannon, 2018; DFAT, 2014; van der Werf et al, 2000). Page 14 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) the pilot villages at the end of 2017. The surveys focused on evaluating the impacts of Groups 1 and 2 as the most promising interventions. Initial analysis indicated that some of the impacts were sustained after two years of implementation. Group 2 still had the strongest (and statistically significant) impacts with 0.18 and 0.09 standard deviation increases in math and language scores compared to control schools. After two years of implementation, teachers in Group 2 also met more often with parents and school principals and were more satisfied with the government at various levels compared to control schools. 29. National Priority Program. Supporting regulations were issued on an annual basis as a direct result of KIAT Guru’s Technical Assistance, which included funding technical specialists to conduct existing policy reviews and analysis, and leading discussions with MoEC and district government counterparts to identify regulatory gaps and develop mechanisms to make teacher allowances performance-based. The issuance of the national regulations, starting in 2017, included the Minister of MoEC decrees to enact KIAT Guru as a national priority program (despite its small geographical coverage), while the Director General decrees provided technical guidelines for implementation. The issuance of the national regulations was significant as they enabled, for the first time, the payment of civil servant allowances conditioned on their performance. The regulations also empowered community representatives to hold teachers accountable in the participating schools. 30. GoI co-financed Project implementation in the total amount of USD 1,595,761, bringing the total cost to USD 6,582,761 (Table 7). At the time the Project was approved, MoEC committed USD 820,000 for implementation. By the end of the Project, MoEC allocated USD 1,208,345 in total. In addition, MoEC allocated a total of USD 5,143,252 of TSA, paid based on the Project’s formula for 203 pilot schools (Table 7 MoEC column). 31. National budget savings. The KIAT Guru implementation of the performance-based incentive based on teacher presence or teacher performance resulted in national government budget savings, as the allowance of underperforming teachers was cut in accordance with regulations issued for KIAT Guru. A total of 6.74% of TSA allocation was not paid in Group 2 (CEM + PPM on presence) and a total of 5% of TSA allocation was not paid in Group 3 (CEM + PPM on quality). Over two years of implementation, the reduction amounted to an average of 5.9% and total savings of USD 100,078 out of an allocated USD 1.65 million for both groups10 in the five districts over two years of implementation. Table 7. Government of Indonesia’s Funding Contributions CY MoEC District Village Total TSA Implementation Government Government 2016 $741,316 $741,316 2017 $2,487,857 $392,898 $115,794 $65,951 $3,062,500 2018 $2,655,395 $74,132 $68,905 $136,765 $2,935,197 Total $5,143,252 $1,208,345 $184,699 $202,717 $6,739,013 32. District government contributions. The Head of Districts also issued various regulations11 and allocated resources, including budgets and government officials time for managing the scheme. In 2017 and 2018 the district governments provided a total of USD 184,699 from district government budget (APBD) for KIAT Guru implementation in the 203 schools (Table 7, District Government column). Moreover, all five district governments decided to expand the scope 10Based on data collected by BaKTI from MoEC, on TKG payments and process monitoring data. 11Annually each district government minimally issued regulations on: (i) KIAT Guru Pilot Program Implementation; (ii) Formation and tasks and responsibilities of KIAT Guru District Coordination Team; and (iii) Village Fund use for Basic Social Services. Page 15 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) of implementation in 2019, allocating USD 259,000 to support the expansion to an additional 207 schools. The district governments will expand Group 2 intervention to a total of 410 schools, with CEM + PPM based on teacher presence and verified by community members. 33. Village governments contributions. The Head of Villages issued regulations and allocated resources to empower community representatives. Where village budgets were previously allocated mostly for infrastructure, KIAT Guru village governments allocated a total of USD 202,717 from the Village Funds to finance KIAT Guru activities (Table 7, Village Government column). In 2017, 169 of 176 (96%) village governments provided financial support, and in 2018, all of them did. 34. Evidence-based policy influencing. Based on the KIAT Guru IE results, process monitoring and qualitative research, the Task Team recommended to the government to tie TSA to teacher presence, as recorded by camera application and verified by community representatives. The conclusive findings, complemented by districts’ reflections on the Project, have been formulated into policy recommendations to advocate for the adoption of a nation-wide policy on performance-based incentives and mechanisms to improve teacher performance. The MoV is also keen to champion the adoption of this pilot to other disadvantaged districts and hosted a knowledge sharing session with 13 such districts in November 2018. 35. Implementation of Phase 2 to expand the scope and institutionalize KIAT Guru mechanisms. Starting in 2019, MoEC and the five district governments have allocated resources to expand the scope of the pilot to tie TSA with teacher presence, covering a total of 410 primary schools in remote area (203 Phase 1 + 207 new schools). Phase 2 will institutionalize the community empowerment and pay-for-performance mechanisms within the government’s teacher management systems using disruptive technology. e-KIAT Guru digitizes diagnostic literacy and numeracy competency assessments, service agreements between education stakeholders to improve learning environment, teacher presence records, teacher performance evaluation results, and complaint handling and redress mechanism. Accessible using mobile-phone- and web-based applications, e-KIAT Guru can function in both online and offline modes, an important feature for remote areas. 36. Potential influencing of Teacher Certification Allowance. In addition, MoEC requested to adapt KIAT Guru mechanisms and test them in 50 secondary schools in urban areas of the five pilot districts using Teacher Certification Allowance. The Task Team provided policy recommendations to, and is involved in on-going discussions with, MoEC and TNP2K on adapting the KIAT Guru model or up-taking elements of KIAT Guru in adapting the system of performance assessment for the allocation of the Teacher Certification Allowance. On an annual basis, GoI spends USD 5.5 billion (18% of the total education budget) and finds it crucial that these allowances lead to improved teacher performance and student learning outcomes. However, decision makers in MoEC have different ideas on what indicators should be tied with performance-based Teacher Certification Allowance. KIAT Guru Phase 2 will advocate that MoEC tests different indicators to identify potential challenges for implementation. 37. Collaboration, dissemination and potential adoption by other programs and development partners. On-going collaborations with the World Bank’s Education GP (funded by DFAT’s ID-TEMAN) occurred prior, during, and post the Project’s implementation, to situate the Project within the broader education reform. The Project provides empirical evidence that feed into an upcoming Bank’s lending operations on Efficient Systems for Education Excellence and Equity (P163217). KIAT Guru’s Student Learning Assessment instrument was used as a model for INOVASI (DFAT-funded project in education) to provide technical assistance to the development of MoEC’s INAP (Indonesian National Assessment Programme). The second round of KIAT Guru Phase 1 surveys was partially funded Page 16 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) by USAID and Research in Improving Systems in Education (RISE12), implemented by SMERU Research Institute. RISE Indonesia has adapted the approach in Way Kanan district, using an android-based app to provide incentives for schools to perform, with indicators including teacher presence. KOMPAK (DFAT-funded project for community empowerment in Indonesia) is interested to adapt the approach for their work in Papua and West Papua provinces, to address teacher absenteeism and improving literacy and numeracy. Similarly, Tanoto Foundation, a national NGO working in five provinces, is interested in advocating this approach in the 24 district governments where they operate. Globally and nationally, initial findings from Phase 1 have been presented in various events, including Global Partnership for Social Accountability, South-South Knowledge Exchange with Government of Nigeria, Indonesia Development Forum, RISE Indonesia Update, Eastern Indonesia Forum, and Jakarta Development Network. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME Key Factors During Preparation 38. Using political leverage to manage risks. The team identified several issues related to the political context in Indonesia at appraisal: (i) the anticipated cabinet reshuffling and the removal of the Minister of MoEC who championed the Project; and (ii) the uncertainties around the new MoV facilitation structure as a potential anchor for the facilitation support. The mitigation strategy included looking beyond the sole interest of MoEC and identified other relevant policies (such as the then recent enactment of the Village Law and the Civil Servants Law) as potential ways to ensure wider support and potential influencing of the government systems to improve basic service delivery. The choice of the partners at national level, i.e. TNP2K, under the Vice President, and MoEC was critical given that the findings from the Project will inform the efforts of MoEC and TNP2K to advocate for the adoption of a nation- wide policy on payment of teacher allowances, both Special and Certification Allowances. Strong ownership from the Director General for Teachers and Education Personnel of MoEC and the Executive Secretary of TNP2K was also part of the political risk mitigation anticipating impending reshuffling at the national level. 39. Implementation arrangements. KIAT Guru used a hybrid implementation system, whereby the PPM was implemented within the government system, while the CEM was implemented by an NGO.13 The combination of working with government agencies, applying the government system for incentive allocation and payment as well as involving an NGO to effectively support the community empowerment was seen as a pre-requisite for future uptake by GoI. Significant government involvement, as owners of the Project, was seen as necessary for sustainability as recent studies (Bold et al, 2018; Banerjee et al, 2017; Banerjee, Glennester & Duflo, 2008; Banerjee & Duflo, 2006; Duflo et al, 2012) showed that projects successfully implemented only by NGOs did not achieve the same results when implemented within the government system. However, given the uncertainties that existed around the new MoV facilitation structure to support the implementation of Village Law,14 BaKTI, the Project’s Grant Recipient and 12 RISE is a multi-year research on education systems in several developing countries. Funded by DFID, DFAT, and Gates Foundation, and led by Lant Pritchett as Research Director, RISE has gained reputation as the leading research in education sector. 13 The role of the Project facilitators, among others, included recruiting Village Cadres and developing capacities of the Village Cadres and User Committee members. Their roles included institutionalizing the CEM processes into the government system, for example by piggy-backing on reporting system from the school principals to the education departments, by incorporating the role of supervisors instead of the Project facilitators, and by setting up forums at the village and sub-district levels. As such, handover of KIAT Guru activities from the NGO to the government system was important for sustainability, and could provide useful insights for other community development projects. 14 The Village Law foresaw the allocation of facilitators on district (technical advisors including in basic social services) and sub-district level (the so-called village facilitators covering 3-4 villages each) but the structure (and accompanying budget and personnel) was not yet in place at the Page 17 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Implementing Agency, was selected to implement the Project. BaKTI facilitators were hired in such a way that it would be possible to be taken over within the MoV’s facilitator structure in the Village Law implementation or facilitators being hired by the district governments. Key Factors During Implementation 40. Changes in Regulatory Framework. Until 2016, there was a national quota for the TSA allocation based on proposals from the districts. During initiation, the Project’s geographical spread covered 10 sub-districts, with the understanding that the district governments would be able to assign TSA allocation to Pilot schools. In 2017, when the Project just began, the government introduced a national index that identified very remote and disadvantaged villages to allocate TSAs. Private- and public-school teachers with either permanent or contract status who had registered with MoEC and were assigned to very remote villages automatically received the TSA. Its value ranged from USD 103 up to one times the teacher’s base salary per month. This national policy change required the district governments to appoint only teachers working in schools located in very remote and disadvantaged villages as recipients of TSA. The Project’s geographical scope expanded from 10 to 33 sub-districts and delayed issuance of key regulations for KIAT Guru implementation. The main impact was the reduction of the period for PPM implementation, from 11 months to 6 months, which was too short of an implementation period to observe Project’s achievements on PDO indicators. Mitigation of this risk was done by extending the implementation period by 6 months and allocating additional resources through restructuring (see paragraph 14). 41. Working within the Government System: Working within the government system, with their regulations and financing mechanisms resulted in some implementation challenges, including: • Slow internal processes hampered timely issuance of the national regulations. • Budget cuts for implementation, due to GoI’s fiscal constraints and changing priorities. • Delayed disbursement of performance-based teacher allowances. • Personnel changes within MoEC including a split of sub-directorates relevant to Project related matters. The team mitigated most of these issues by lobbying the new staff in MoEC and by leveraging TNP2K as an institution under the Vice President. 42. Strategy for transition and sustainability. The team put an early transition strategy in place, supported by implementation agreements, with Cooperation Agreements signed between five district governments and MoEC and TNP2K. The strategy aimed to sustain key activities, structures, and financing. This included having Project Facilitators mentoring Village Cadre in each participating village. The role of the Village Cadres was to facilitate and sustain monthly meetings between teachers and community members. During implementation, Project Facilitators were tasked to encourage the village governments and community members to sustain funding for Village Cadres, monthly meetings at the village level, and quarterly meetings at the sub-district level. The strategy ensured the phased withdrawal of the implementation team while maintaining some technical assistance to support local government implementing agencies as they took over the implementation to oversee monthly evaluation on teacher performance at the village level as well as payment of performance-based teacher allowance at the district level. Bank Performance in Ensuring Quality start of Project implementation. Page 18 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) 43. Quality assurance at entry. The implementation and intervention designs at entry benefitted from the fact that the Project was designed with an Integrated ASA. The Task Team conducted literature reviews and consultations with experts from Social, Education, and Governance Global Practices, and other global and national experts on CEM and PPM. The peer reviewers for the ASA Concept Note were leading global experts with experience in Indonesia or in implementing similar interventions in other developing countries. 44. Integrated ASA. The Project monitoring, learning, and evaluation design included: a. Process monitoring—conducted by the Project Grant Recipient—collected monthly scores on teacher presence and teacher service quality and implementation progress of the IRIs in 203 schools. b. Impact Evaluation (IE), analyzed data from two rounds of surveys, conducted by independent enumerators, in 270 schools. The first round was before the Project implementation started in the schools, and the second round after one year of implementation, when the Project facilitators left the villages. The surveys implemented different tools based on WB experience worldwide but adapted to the specifics of the KIAT research questions. The main tools were the Teacher Absence Survey, Student Learning Assessment, and survey questionnaires to key stakeholders. Annex 6 presents the summary of the IE. c. Qualitative research, conducted by independent researchers and implemented in three phases in nine villages, tracked changes in stakeholders’ views about education service delivery as well as the community facilitation processes. It was designed to contribute to a better understanding of the heterogeneous effects of the interventions. Annex 7 presents the summary of qualitative research. Table 8 provides an overview for three sources of data utilized to evaluate the achievements of PDO indicators. Table 8. Sources of Monitoring, Learning and Evaluation Data Source of Data Collected by Scope Process Monitoring BaKTI/ Project implementation team 203 implementation schools in 5 districts Impact Evaluation Enumerators hired through ASA 270 primary schools (203 implementation + 67 control schools) in 5 districts Qualitative Research Researchers hired through ASA 9 primary schools in 3 districts 45. Sample selection to anticipate replication. The choice of the participating districts15 and schools, as described above, was in line with national regulations and criteria for TSA recipients in the most remote areas. This allows for the possibility to use the results for replication in similar areas targeted by MoEC (areas receiving TSA) and MoV (very disadvantaged villages). At the start of the Project in 2016, there were 122 disadvantaged districts, with 13,425 very disadvantaged villages. There were 56,000 TSA recipients with an annual budget allocation of IDR 1.7 trillion (USD 121 million) in 2016. It was envisioned that the Project would influence TSA policies applicable for 42,800 schools in border, remote, and underdeveloped areas. It was also anticipated that lessons learned from the Project would be 15 The district selection took into account the prerequisites for implementation as identified during the Pre-Pilot including: (a) sufficient level of community willingness to contribute time and resources and to demand for improved education service in their village; (b) sufficient level of transparency in the management of teacher allowance and district finances; and (c) sufficient level of reform in the district bureaucracy and teacher management systems. Page 19 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) used to inform policies for teacher management and disbursement of teacher certification allowance, which in 2016 amounted to IDR 80 trillion (USD 5.95 billion). 46. Adequacy of implementation support. The World Bank team implemented semi-annual joint implementation support missions with partners TNP2K, MoEC, and DFAT. The team consisted of technical, FM, Procurement and Safeguard specialists. These missions were able to identify issues, the Aide Memoires and ISRs show candor in identifying issues, and helped to put solutions in place in a timely manner. As most of the Task Team members were located on the ground, close collaborative relationship with MoEC and TNP2K ensured an effective information flow on emerging issues especially related to the ones potentially impacting the implementation of the teacher allowance allocation and payments. 47. Restructuring in response to changes. During the 2017 Restructuring, a third component was added to delineate Project costs from management costs. This change was necessary so operational costs at the national, district, sub- district, and village levels can be tracked to inform the government at different levels on the costs of sustaining implementation and scaling-up. At the request of the partners, the team was able to carry out a restructuring, to anticipate for replication by providing a clear cost picture by clearly demarcation of Project costs associated with the CEM and PPM mechanisms separated from Project management and TA costs. The restructuring created the opportunity to retrieve data on the relative cost-effectiveness of alternative implementation models for community engagement in education as needed by the government when replicating or partially up-taking. The team quickly adjusted the IRIs to better track them and added new IRIs to track the capacity-building efforts for government officials and community members. IV. COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME Compliance Issues 48. Project Implementing Agency Performance. The appointment of BaKTI was subject to a full World Bank review and approval process, which determined requirements for BaKTI to fulfill prior to and during implementation. Nevertheless, assessment during the Joint Missions identified several issues affecting the quality of implementation and potential validity of the findings. In one mission, it was found that variations in implementation of the CEM and PPM occurred. Some principals and teachers did not sufficiently understand that the amount of their TSA was influenced by either their presence or service quality. Based on the feedback, BaKTI conducted additional information sessions for all schools, to inform stakeholders how teachers’ behaviors were rewarded or sanctioned. BaKTI also adjusted its management and implementation mechanisms to ensure proper implementation. 49. Financial Management. BaKTI possessed an adequate internal control system and improved their system to ensure that management had access to updated financial information in a timely manner. Audited Financial Statements and follow-up on audit findings, as stated in the management letters, were implemented swiftly and financial reports were timely. 50. Procurement. The Bank carried out an initial procurement capacity assessment identifying BaKTI as having adequate previous experience in carrying out procurement under previous Bank-financed projects. Procurements were carried out according to the procedures specified in the Bank’s Procurement and Consultant Guidelines and utilized Systematic Tracking of Exchanges in Procurement (STEP) to upload documents to allow the Bank’s ex-post review of Page 20 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) contracts procured and awarded. The Bank provided hands-on support, as needed, in particular for the application of STEP to ensure full use of the system. 51. Safeguards. KIAT Guru was assessed as low risk (Category C under OP 4.01) and the Project did not support physical construction or purchase of equipment with environmental risks. The Safeguard Guidelines socialization and awareness-raising to relevant stakeholders was carried out and the community facilitation manual details social safeguard aspects including procedures for consultations with indigenous stakeholders and participating communities during implementation to ensure broad community support and social inclusion. A Complaint Handling System, operated by BAKTI, was set up using an SMS gateway platform. Most issues raised were related to payments of teachers’ allowances (67% of the complaints) and clarification on eligibility of teachers to receive the special allowance (21% of the complaints), which were under the jurisdictions of MoEC and district governments. Risk to development outcomes 52. National regulatory framework to institutionalize PPM. Every year, the national and district governments issue new regulations to update the previous year’s regulations. Similarly, for teacher allowances, MoEC and district governments, on annual basis, re-issue regulations on eligibility criteria, technical guidelines, and name of eligible schools and teachers. TNP2K was influential in ensuring support, especially from MoEC and district stakeholders, to issue the annual regulations to ensure the implementation of the PPM. These KIAT Guru regulations were separate from and additional to the business-as-usual regulations. MoEC will need to adopt the KIAT Guru regulations as national policies for TSA. Templates for these regulations were produced by the Project. 53. Eligibility criteria for TSA schools. Since 2016, MoEC referred to the Developing Villages Index (Indeks Desa Membangun) issued by the MoV, whereby all eligible teachers appointed to schools in very disadvantaged villages receive TSA. However, the MoV’s priority is to graduate the status of all villages to have developing or developed status, even though they are still remote and difficult to access. For example, between 2017 and 2018, the number of very disadvantaged villages in five pilot districts was reduced from 616 to 475, a decrease of 23%. As a direct result, 43% of KIAT Guru schools were no longer eligible for TSA in 2019, as the status of the villages were upgraded from very disadvantaged to disadvantaged villages. This MoV policy makes it difficult for MoEC to plan and budget, and target TSA allocation for teachers assigned to special areas (which in addition to remote locations, also include disaster- and conflict-prone areas). To this end, MoEC has informally requested technical assistance so they can develop their own criteria. The Indonesia Social Development Unit is currently supporting the Ministry of Planning (BAPPENAS) to develop One Village Index (Indeks Satu Desa), which is based on available data collected by the government on a regular basis. Once this index is launched in October 2019, the Task Team will follow up with MoEC to identify indicators relevant for MoEC to identify special areas. 54. Sustained community involvement. Psychological challenges, such as different social status or educational attainments, may hamper objective community evaluation of teachers or the school principal. Besides this, communities’ willingness to contribute and participate in the CEM on an ongoing basis may fluctuate or dissipate over time. Qualitative research and Project monitoring findings indicate that community participation was affected by the village governments’ supports. Three indicators of support include: (i) village government staff’s participation in KIAT Guru meetings, (ii) timely enactment of village head decrees to appoint Village Cadres and User Committee members, and (iii) timely allocation of funds for KIAT Guru activities. LESSONS LEARNED AND RECOMMENDATIONS Page 21 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) 55. The KIAT Guru experience shows that when communities are empowered through a sustained CEM process and when this is strengthened by supply-side regulations that incentivize and sanction frontline service providers based on community inputs, long-term development impacts can be achieved. Parents’ engagement with and their involvement in frontline education service delivery improved their relationships with teachers, their understanding of rights and responsibilities regarding education, and their support towards social accountability and participation in education as citizens. The process built social capital through the service user groups who improved their understanding of organizational issues and their ability to engage constructively with the service providers in a structured manner. When community members felt supported by the government at various levels, their participations could be sustained. 56. A clear and coherent regulatory framework at the national, district, and village governments is needed to make teacher allowance performance-based and to sustain community participation activities through Village Funds. The regulatory framework for PPM needs to come at the national and district levels. The Village Law (UU 6/ 2014) and its regulatory framework, provides the villages (and its leadership) with an opportunity to allocate part of the national allocated Village Funds to support improvements in basic service delivery, can be utilized for sustaining the CEM. A Ministry of Village Decree 16/ 2018 regarding the priorities for the 2019 allocation of the Village Funds explicitly mentions community involvement in educational oversight. A stronger push from the national level is needed so village governments invest more in improving basic service delivery. The issuance of clearer and more comprehensive district government regulations can ensure the allocation of Village Funds for basic social services and community empowerment, including for incentives and capacity building for community members and Village Cadres. 57. Gender parity should be promoted. Gender-balanced participation in memberships, meetings, and in capacity development activities should be required and enforced. Women often act as the main caretakers of children’s education in most households, and as such, their participations in improving education service delivery will provide crucial inputs and insights. However, strategy to improve women participation in educational oversight is only effective when the criteria for gender balanced composition of the user groups is enforced. All facilitators need to be sufficiently equipped with the skills to promote and encourage women participation, and increase community awareness on the importance of women involvement in education and women access to key positions in community organizations. Capacity development trainings need to be conducted at a location accessible for women both from the perspective of transportation access and safety. 58. GoI stronger commitment in improving human capital provides an opportunity for scaling up KIAT Guru. The Government has continued to demonstrate commitment to improving teachers’ performance, with the 2019-24 medium-term development outcomes and 2019 budget prioritizing human development capacity. MoEC has exhibited stronger ownership of KIAT Guru over time, requested expansion to Phase 2, and continued to engage the Task Team to adopt various aspects of KIAT Guru. For example, MoEC and TNP2K continue to emphasize the importance of developing models for urban areas and have requested technical assistance to adapt the CEM and PPM for different contexts; these include, citizen engagement to improve education service delivery in urban/ semi- urban areas for elementary and secondary education, and performance-based payment using teacher professional allowance. 59. The CEM should be institutionalized into either village governance or school governance systems. The Project was implemented under a hybrid scheme, combining efforts of GoI and NGO. The question remains whether the Page 22 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) government can maintain and expand CEM16 which was previously implemented by the NGO. Phase 2 will address this sustainability and expansion challenge by testing two institutional arrangement models. The first model will institutionalize the CEM through existing school governance system by revitalizing school committees and strengthening the capacities of school supervisors. The second model will continue to set up the CEM through village governance system, involving actors such as Village Cadres and User Committees (including school committee members). The role of the Project facilitators will be limited to three visits17. Gradual phase-out of Project staff at the local level, while maintaining some on-going technical assistance to district and national government on policy recommendations, and the set-up of governance and administration mechanisms to sustain and scale up the pilot could be an effective strategy. . 16 This reflection stems from the fact that “Effects of community monitoring on service delivery improvements� have been mixed, some encouraging (Banerjee & Duflo, 2006; Joshi, 2010; Ringold et al, 2012), but particularly not effective when later adopted by the government (Duflo & Hanna, 2005; Banerjee & Duflo, 2006; Banerjee, Glennester & Duflo, 2008). 17 The three visits will be conducted during: (i) the first information session to introduce the Project and Village Cadres, (ii) the formation of Service Agreement and User Committee, and (iii) the first monthly meeting, to provide support to Village Cadres who will facilitate the meeting. Page 23 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: The Project Development Objective is to improve the presence and service quality of primary school teachers in targeted areas. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Improved teacher presence Number 65.00 80.68 85.00 95.13 score 31-Aug-2016 31-Dec-2017 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): At appraisal in June 2016, this indicator was "improved percentage of teacher presence rate" based on Teacher Absence Surveys conducted through the Integrated ASA before and after Project implementation. Since the Project's ISRs required more frequent reporting, in September 2017 Restructuring, this indicator was changed to “improved teacher presence score,� using data source from Community Score Card indicato r for teacher presence, with a baseline score of 65 and an end target of 85. At completion, the score was 95.13 (112% achievement). Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Improved teacher service Number 56.00 85.00 85.00 96.83 Page 24 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) quality score 31-Aug-2016 31-Dec-2017 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): At appraisal in June 2016, this indicator was "improved community satisfaction of teacher service quality" based on survey questionnaires conducted through the Integrated ASA before and after Project implementation. Since the Project's ISRs required more frequent reporting, in the September 2017 Restructuring, this indicator was revised to “improved teacher service quality score.� The score was based on the User Committees’ assessments on teacher service performance on Community Score Card indicators. The average score for baseline was 56, with an end target of 85. At completion, the score was 96.83 (114% achievement). A.2 Intermediate Results Indicators Component: Component 1: Community Empowerment Mechanism (CEM) implemented and utilized to measure teachers’ presence and quality of teaching Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of Community Score Number 0.00 203.00 203.00 203.00 Card agreed 31-Aug-2016 30-Apr-2018 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): The September 2017 Restructuring rephrased this indicator from “percentage of villages assigned to CEM models completed the initial phase� to “number of Community Score Cards agreed.� The Community Score Card was based on Service Agreements being conceded between the provider group (principal and teachers) and the user group (parents and community representatives). The Community Score Card consisted of between five to eight service performance indicators for the provider group, one of which had to be an indicator on teacher presence. The Community Score Card was evaluated on a monthly basis by the User Committees, elected to represent parents and community leaders. The Page 25 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) establishment of Service Agreements, Community Score Cards, and User Committees marked the completion of the CEM initial phase. A total of 203 Community Score Cards was agreed as a target, and it was fully achieved. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Cumulative number of village Number 0.00 2000.00 2000.00 4844.00 monthly meetings conducted 31-Aug-2016 31-Dec-2017 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): The September 2017 Restructuring rephrased this indicator from “percentage of villages assigned to CEM models completed the implementation phase� to “cumulative number of village monthly meetings conducted.� The implementation phase of the CEM consisted of monthly meetings where the User Committees presented the evaluation scores for the provider group on their Community Score Cards, along with the supporting evidence, to the village government representatives and wider communities. A target of 2,000 monthly meetings was agreed, with achievement at completion 4,844 (242% achievement). Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Cumulative number of end-of- Number 0.00 350.00 350.00 394.00 semester meetings conducted 31-Aug-2016 31-Dec-2017 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): Page 26 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) The September 2017 Restructuring rephrased this indicator from “percentage of villages assigned to CEM models completed the evalu ation phase� to “cumulative number of end-of-semester meetings conducted.� The evaluation phase of CEM happened at the end of each school semester to evaluate Service Agreement, Community Score Card, and User Committee. The meeting preceded by a focus group discussion with students from the pilot schools to get their inputs on whether or not learning environments at school and at home had improved in the past semester. A target of 350 meetings were agreed, with a 394 meetings completed (112% achievement). The first 203 meetings were facilitated by project facilitators, while the next 191 meetings were conducted by village cadres and user committee members independently. Component: Component 2: Pay for Performance Mechanism implemented and utilized to strengthen teachers’ accountability Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Relevant government Yes/No N Y Y Y regulations issued 31-Aug-2016 31-Dec-2017 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): This indicator initially tracked individual regulations, decrees, and guidelines issued by the national, district, and village governments to enact and support the Project implementation. Since the Project issued hundreds of regulations in total, the September 2017 Restructuring chnaged the type of indicator from “text� to “yes/ no.� The targets were achieved for 2017, 2018, and 2019. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Percentage of Tunjangan Percentage 0.00 75.00 85.00 94.00 Page 27 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Khusus allowance for teachers 31-Aug-2016 31-Dec-2017 31-Mar-2019 31-Mar-2019 assigned to PPM models are paid according to KIAT Guru formula Comments (achievements against targets): The September 2017 Restructuring approved change to the end target from 75% to 85%. At completion, 94% of Tunjangan Khusus allowance was paid according to KIAT Guru formula (110% achievement). Component: Component 3: Technical Assistance and project management supports provided to strengthen the capacity of government officials and community members to improve teacher presence and service quality Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Number of MoEC and district Number 0.00 375.00 750.00 1084.00 government officials receiving capacity building 31-Aug-2016 30-Apr-2018 31-Mar-2019 31-Mar-2019 Comments (achievements against targets): This indicator was added during the September 2017 Restructuring. At completion, 1,084 beneficiaries received capacity building, above the 750 target (by 144% achievement). The participants were 86% male and 14% female. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Page 28 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Number of Village Cadre and Number 0.00 2000.00 2000.00 2628.00 User Committee members received capacity building 31-Aug-2016 30-Apr-2018 31-Mar-2019 31-Dec-2018 Comments (achievements against targets): This indicator was added during the September 2017 Restructuring. At completion, 2,628 beneficiaries received capacity building, above the 2,000 target (131% achievement). Gender breakdown for the two training were: (i) on sustaining KIAT Guru activities, attended by 86% male and 14% female participants; and (ii) on conducting evaluation meeting, attended by 73% male and 26% female. Page 29 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) B. KEY OUTPUTS BY COMPONENT Component 1: Community Empowerment Mechanism (CEM) implemented and utilized to measure teachers’ presence and quality of teac hing Outcome Indicators 1. 2,628 Village Cadres and User Committee members received capacity building (target 2,000, 131% achievement) 1. 203 Community Score Card agreed (target 203, 100% achievement) Intermediate Results 2. 4,844 village monthly meetings conducted (target 2,000, 242% achievement) Indicators 3. 394 end-of-semester meetings conducted (target 350, 112% achievement) 1. Teacher presence, based on Community Score Card indicator, improved from 65.00 to 95.13 (target 85, 112% Key Outputs by achievement) Component 2. Teacher service quality, based on Community Score Card indicators, improved from 56.00 to 96.83 (target 85, 114% achievement) Component 2: Pay for Performance Mechanism implemented and utilized to strengthen teachers’ accountability Outcome Indicators 1. 1,084 MoEC and district government officials receiving capacity building (target 750, 144% achievement) 2. Relevant government regulations issued in 2017, 2018, and 2019 (100% achievement) Intermediate Results 3. 94 percent of Tunjangan Khusus allowance for teachers assigned to PPM models were paid according to KIAT Guru Indicators formula (target 85 percent, 110% of target achievement) 1. Teacher presence, based on Community Score Card indicator, improved from 65.00 to 95.13 (target 85, 112% Key Outputs by achievement) Component 2. Teacher service quality, based on Community Score Card indicators, improved from 56.00 to 96.83 (target 85, 114% achievement) Component 3: Technical Assistance and project management supports provided to strengthen the capacity of government officials and community members to improve teacher presence and service quality 1. 2,628 Village Cadres and User Committee members received capacity building (target 2,000, 131% achievement) Outcome Indicators 3. 1,103 MoEC and district government officials receiving capacity building (target 750, 147% achievement) Page 30 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (USDM) Closing (USDM) (USDM) Community Empowerment 2,536,000 1,589,440 63% Mechanism Pay for Performance 1,087,000 153,590 16% Mechanism Technical Assistance and 0 3,243,970 N/A Project Management Total 3,623,000 4,987,000 138% Page 31 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) ANNEX 3. EFFICIENCY ANALYSIS 60. GoI paid for 32 percent of the total Project implementation costs. The implementation budget for KIAT Guru activities was based on standard GoI costs (Standar Biaya Umum), with technical assistance salary grade based on the Indonesian National Consultant Association (Ikatan Nasional Konsultan Indonesia). The project was funded by Local Solutions to Poverty MDTF, with a total of USD 4,987,000 from July 2016 until March 2019. Over the same period, MoEC, district and village governments allocated a total of USD 1,595,761 (Table 9), bringing the total Project cost to USD 6,582,761 (Table 10). In addition, the Project made performance-based the Teacher Special Allowance (TSA) paid by MoEC in the total amount of USD 5,143,252. Table 9. Government of Indonesia’s Funding Contributions Calendar Year MoEC District Village Total 2016 $741,316 $0 $0 $741,316 2017 $392,898 $115,794 $65,951 $574,642 2018 $74,132 $68,905 $136,765 $279,802 Total $1,208,345 $184,699 $202,717 $1,595,761 Table 10. Expenditures from MDTF and GoI by Category of Expenses Category MDTF GoI Total 1. TA/ Consulting services 2,862,063 2,862,063 2. Training & workshop 1,159,228 1,595,761 2,756,791 3. Goods 111,078 111,078 4. Non-consulting services 714,289 714,289 5. Indirect cost 140,342 140,342 Total 4,987,000 1,595,761 6,582,761 61. The one-time investment cost for the Project facilitators to conduct KIAT Guru approach was a total of USD 1,026,759, or at USD 5,058 per school (8% of Village Fund) and USD 40 per student (Table 11). This cost covered trainings, salaries and transportation costs of 41 Project facilitators working in 203 schools, with an average of 132 students per school. Over three years of implementation, the cost of training and workshop was USD 2,756,791 (Table 10 row 2), of which USD 431,667 was spent for training of Project facilitators. This boils down to an annual cost of training at USD 143,889, or at USD 709 per school, USD 6 per student. The average monthly salary for the facilitator was USD 815, bringing the average monthly total salaries of USD 33,432. The facilitators were employed over 15 months, with a total spending on salary USD 501,483, averaging USD 2,470 per school, USD 19 per student. Each facilitator visited a school with an average of 11 visits, and transportation cost averaging USD 112 per visit. Over the course of 15 months, the total transportation costs reached USD 252,888, with an average of USD 1,246 per school, USD 10 per student. The one-time investment cost of Initial Phase meetings at the village level was USD 633 per school, USD 5 per student. This cost covered seven meetings which resulted in the Service Agreement, the Community Score Card, and the establishment of the User Committee. To scale up KIAT Guru to other 42,800 schools in border, remote, and underdeveloped areas, the Government will need to budget one-time investment cost of USD 216,482,400, without considering the economy of scale. Since the one-time investment is only 8% of annual Village Fund, it is affordable for 13,425 village government to absorb this cost. Page 32 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Table 11. One-time Investment cost to introduce KIAT Guru Phase 1 One-time Investment (203 Annual Cost per Annual Cost per Schools) School Student Training 143,889 709 6 Salary 501,483 2,470 19 Transport 252,888 1,246 10 Initial Meetings 128,499 6333 5 Total 1,026,759 5,058 40 63. The average annual cost to sustain CEM was USD 2,182 per school (3.5% of Village Fund) or USD 17 per student (Table 12), with additional USD 506 per school or USD 4 per student for Group 2 schools. This cost covers an annual refresher training, monthly meetings, and evaluation meeting. The annual cost of training per school at USD 709, or USD 6 per student. The annual cost to conduct monthly meetings was USD 834 per school, USD 7 per student. The average cost per village to conduct evaluation meetings at the end of every semester was USD 639 per school, USD 5 per student. In these evaluation meetings, User Committee and school providers reviewed the content of Service Agreement and Community Score Card indicators, and reappointed User Committee members. The village governments allocated a total of USD 202,717 from the village budgets (APBD) to finance KIAT Guru activities (Table 9). In 2017, 169 of 176 (96%) village governments provided the financial supports, and in 2018, all of them did, with an average of USD 674 (31% of total annual cost to sustain CEM). Given that these costs are only a very small fraction of Village Fund, the cost of maintaining KIAT Guru activities in the villages are completely sustainable. Group 2 schools had a total of USD 86,341 of additional cost to cover the purchasing of one smart phone per school and two personnel to develop and maintain the application. On annual basis, this cost came down to USD 33,874, or at USD 506 per school, USD 4 per student. Table 12. Average Annual Cost per School for CEM Total Cost Annual Cost Annual Cost (203 Schools) per School per Student Refresher Training 143,927 709 5 Monthly Meetings 169,302 834 7 Evaluation Meetings 129,717 639 5 Total 442,946 2,182 17 Group 2 Additional Cost 86,341 506 4 Total for Group 2 529,287 2,688 21 64. As a comparison, to improve teacher presence and service performance, the Government could improve top-down supervisions to schools, with an additional annual cost of USD 181,728, or at USD 895 per school and USD 7 per student (Table 13). This cost covers the additional salary and transport costs for school supervisors. MoEC sets an ideal ratio of 1 supervisor for every 10 schools, with visits conducted monthly. However, in practice, only 12 supervisors were assigned to the 203 schools (ratio of 1:17). In order to meet the ideal scenario, first of all, the Government will have to hire 8 new supervisors, with an average monthly salary of USD 651 per supervisor, bringing the total additional monthly spending to USD 5,208, and total additional annual spending to USD 62,500. In addition, Page 33 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) the Government must provide transportation funding for 20 supervisors to visit the schools every month, with a total of USD 10,087 monthly, and a total of USD 121,044 annually. Currently, the 12 supervisors only received USD 1,816 on average per year. Hence, the Government will need to add the annual transport costs for the supervisors in the amount of USD 119,228. The additional cost for supervisors will need to be maintained annually by the Government, and need to be budgeted at the district or national level. A study indicated that teacher absenteeism rate in schools visited within the last month by supervisors was at 18% (UNICEF 2012). Prior to KIAT Guru intervention, teacher absenteeism rate in the 203 schools was at 25%. After one year, teacher absenteeism rate in KIAT Guru schools dropped to 17%. While the effect on teacher absenteeism was mild, KIAT Guru interventions improved student learning outcomes at a rate almost three times faster than control group. In contrast, two studies on teacher absenteeism in Indonesia did not establish clear correlation between improved teacher presence with learning outcomes (SMERU 2004; ACDP 2014). Table 13. Additional cost for GoI to improve top-down supervisions to schools Additional Cost for Supervisors Annual Per school Per student Salary 62,500 308 2 Transport 119,228 587 5 Total 181,728 895 7 65. Compared to other rigorously evaluated interventions in education, the cost of implementing KIAT Guru was on par with interventions that adopted similar approaches. In the first year of intervention, Group 2 improved learning outcomes by 0.2 standard deviation, at USD 43 per student. This means it costs USD 22 per student per 0.1 standard deviation increase. After two years of intervention, the learning outcomes decreased to 0.14 standard deviation, with an annual cost dropped to USD 21 per student, or USD 16 per 0.1 standard deviation increase. Since KIAT Guru interventions uniquely combined CEM and PPM, comparison to other rigorously evaluated interventions in education (Glewwe & Muralidharan, 2015; JPAL, 2019) is based on similar interventions for each. For CEM comparison, a study on school committee formation through election and linkage with village council in Indonesia costs USD 7.50 per 0.1 standard deviation increase, excluding the cost of providing grant to the committee. Three studies on Conditional Cash Transfer (CCT) grants improved learning outcomes with costs averaging USD 77 per 0.1 standard deviation increase. For PPM comparison, camera monitoring and teacher-presence-based payment in India costs USD 44 per 0.1 standard deviation increase, excluding the cost of staff, transportation, and monthly meetings. Teacher incentive intervention in Kenya costs USD 16 per 0.1 standard deviation increase, while in India costs USD 1 per 0.1 standard deviation increase. 66. In KIAT Guru Phase 2, the one-time investment cost by Project facilitators will be reduced to USD 1,769 per school (3% of Village Funds), or USD 14 per student (Table 14), with the annual cost for sustaining it at USD 2,688 per school (4% of Village Funds), USD 21 per student. This cost cover trainings, salaries and transportation costs. For this calculation18, the cost of training is based on Phase 1, at USD 143,889 for 203 schools, or at USD 709 per school, or USD 6 per student. The CEM process will be reduced from a total of seven to four meetings. Phase 2 facilitators will be employed for a total duration of six months, covering five schools, with three visits to each school. Hence, to cover 203 schools, it will still take 41 Project facilitators, but the total transportation cost per school will drop to USD 340 (compared to USD 1,246 in Phase 1). As the facilitation process has been simplified, less experienced facilitators can 18 Note that in reality, Phase 2 trainings will be set up differently and more efficiently, with training at the national level for the trainers only, and training for school and village stakeholders at the district or sub-district levels. Page 34 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) be employed, with an average monthly salary of USD 297 per facilitator (compared to USD 815 in Phase 1). Hence, the one-time cost to implement KIAT Guru in 203 schools will be reduced to USD 359,120 in total, or at USD 1,769 per school, USD 14 per student. Compared to Phase 1 costs, this is a reduction of 65% percent in financing and 60% percent in implementation period. Sustaining of KIAT Guru activities at the village level will average USD 2,688 per school (4% of Village Fund) or USD 21 per student. Note that the cost for developing e-KIAT Guru, the MIS system to digitalize the CEM and PPM processes at the village, district, and national level, is estimated to be USD 200,000 for every two years. This cost also does not consider the economy of scale. Table 14. One-time investment cost to introduce KIAT Guru Phase 2 One-time Investment Cost per school Cost per student Training 143,869 709 6 Salary 72,943 359 3 Transport 68,880 339 3 Initial Meetings 73,428 362 3 Total 359,120 1,769 14 Page 35 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) ANNEX 4. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Comments from the Ministry of Education and Culture 67. A meeting with MoEC and BaKTI was held on September 2, 2019 to present the ICR draft. The meeting was chaired by Director General of Teacher and Education Personnel (Guru dan Tenaga Kependidikan) and attended by Director of Teacher Development for Basic Education (Pembinaan Guru Pendidikan Dasar), Director of Education Personnel (Tenaga Kependidikan), and representatives from Directorate for Empowerment of Secondary Education Teacher (Pembinaan Guru Pendidikan Menengah), Head of Sub-Directorate for Regulations, Governance, and Personnel (Kepala Bagian Hukum, Tata Laksana dan Kepegawaian), and Head of Sub-Directorate for Planning and Budgeting (Kepala Bagian Perencanaan dan Penganggaran). 68. The Director General for Teachers and Education Personnel requested for a grand scheme to scale up KIAT Guru to improve learning outcomes in other disadvantaged districts. The presentation on the draft ICR highlighted the Project’s impact in improving teacher presence and service quality, in making efficient TSA payment, and in increasing support from five district governments as evidenced in allocation of district government funding for the Project. The Director General identified the potential to adopt as a national policy, the implementation of diagnostics Student Learning Assessment to inform education stakeholders on student learning outcomes, as benchmarked against national curriculum target. The Director of Teacher Development for Basic Education summarized that while the cut to TSA was very modest, it significantly improved learning outcomes. As such, the KIAT Guru approach should be adapted for Teacher Certification Allowance. 69. The Head of Legal and Organization Bureau emphasized the importance to adapt KIAT Guru lessons for implementation of Civil Servant Law in education sector. A separate interview was conducted with the Head of Legal and Organization Bureau, under the Secretary General of MoEC, who in her previous position championed KIAT Guru. She advised that KIAT Guru should not stop as it stands but really tries to influence the adaptation of Civil Servant Law for education and provides technical assistance to develop regulations and tools for assessing teacher performance. Page 36 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) ANNEX 5. SUMMARY OF RECIPIENT COMPLETION REPORT 70. Yayasan BaKTI was selected through a competitive process as the Grant recipient for KIAT Guru (P159191) and to receive funding under LSP MDTF grant (TF-A2955). With total funding of USD 4,987,000, BaKTI implemented KIAT Guru in 203 primary schools in five targeted districts – Sintang, Ketapang and Landak in West Kalimantan Province, and Manggarai Barat and Manggarai Timur in East Nusa Tenggara Province. 71. KIAT Guru aimed to improve teacher attendance, teacher service quality, and student learning outcomes in remote areas, using two mechanisms: (a) Community Empowerment Mechanisms, and (b) Performance-based Payment Mechanisms. 72. Community Empowerment Mechanism (CEM) was implemented by: • Informing the communities their rights and obligations to participate in education. • Facilitating meetings between the communities and schools to produce a Service Agreement and Community Score Card, a list of agreed commitment of services between parents/ community members and teachers to improve education in the village. • Supporting the communities to form User Committee to provide input in improving basic education services. The UC consisted of a minimum of 9 people representing parents of each class, community/ religious/ cultural/ youth leaders, and a minority group in the village. • Identifying and developing the capacity of Village Cadres to facilitate the process of regular monthly meetings between the community and the school. • Strengthening the position and role of UC and Village Cadres as part of village government institutions by issuing Village Government Decrees. • Providing training for UC to verify attendance and evaluate the quality of teacher services. • Improving the capacity of UC to assess and report on the results of teacher attendance assessments and services to village communities and the district education office. • Providing the community with information about student learning outcomes after the Project was implemented for one semester, to provide inputs for evaluating Service Agreements and Community Score Cards. 73. Performance-based Payment Mechanisms (PPM), which linked TSA payments to the presence or quality of teacher services. Transparency and payment mechanisms were improved so that they were based on the right criteria, target, amount and on time. • Teacher attendance was recorded every day using the KIAT Kamera, which was verified by the UC at the end of every month, whether teachers who were absent had an official permit. Teacher tardiness or absences were subject to cut in the TSA payment for the current month. • The quality of teacher services was assessed by the UC at the end of each month based on the implementation of the agreed Community Score Card indicators. 74. Based on thirty-three months of implementation BaKTI highlighted four key lessons: • Involving external parties, and not only the school providers, to improve student learning outcomes. The external parties included district government, village government, district government as well as the community and parents of students. • Increasing the role of parents in children's education by ensuring children are studious at home. • Assessing teacher performance with simple and objective indicators, such as using teacher attendance Page 37 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) recorded using KIAT Kamera, which provided evidence to both teachers and parents that the teachers came to school. • Linking teacher allowance payments with simple and objective performance indicators. Page 38 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) ANNEX 6. SUMMARY OF IMPACT EVALUATION Research Design 75. A randomized controlled trial (RCT) strategy was used to evaluate a set of three interventions that combines a community empowerment mechanism (CEM) and different pay-for-performance mechanisms (PPM). The three treatment arms were as follows: a. Group 1 implemented CEM in treatment schools by facilitating the formation of a service agreement between teachers, village leadership, and parents. Based on the service agreement, teacher-specific Community Score Cards were developed and used as a guideline to monitor teacher performance. The monitoring was done by a user committee that included community leaders and parent representatives. Monitoring results did not affect teacher pay. There were 68 schools that were randomly assigned to Group 1. b. Group 2 implemented the CEM described above and added a PPM based on teacher presence. Teachers took pictures at the beginning and end of a school day and recorded the times on a manually-entered teacher attendance form. At the end of each month, members of the UC verified both entries and any letters provided by teachers to account for their absences. There were four types of possible entries: full presence (0), partial presence (up to 1.5), excused absence (2), and unexcused absence. The user committee used the pictures to verify self-reported presence reports, which determined the cut to teachers’ TSA. Teachers whose total cut exceeded 15 percent lost their entire TSA while those whose cuts were between 1-15 percent lost the percentage of their TSA equal to their absence. There were 68 schools that were randomly assigned to Group 2. c. Group 3 implemented the CEM described above and added PPM based on teacher service quality as evaluated using the Community Score Card (instead of the camera-verified absences). The score assigned by the user committee to an individual teacher determined the share of TSA that the individual teacher received. No camera was provided in Treatment 3. There were 67 schools assigned to Group 3. In addition, 67 schools were assigned to a control group that received no intervention. Empirical Strategy 76. The following regression model was used to estimate the causal impact of the interventions: 𝑟 𝑘 𝑘 𝑌𝑖𝑗𝑡 = 𝛼𝑘 + 𝛿𝑌𝑖𝑗𝑡−1 𝑘 + ∑ 𝛾𝑇𝑗𝑘𝑟 + 𝑋𝑖𝑗𝑡 𝑘 𝛽 + 𝜀𝑖𝑗𝑡 𝑅 where Yijt = the outcome variable for individual i in school j at time t ∈ {0,1}, αk = the strata fixed effects, and X = control variables. Trj is the dummy variable for treatment regime r, and γr is the ITT estimate of interest for treatment r. The control variables depend on the outcome of interest which we describe below. Our baseline results include estimates with and without the control variables. Standard errors are clustered at the school level. In addition, for robustness, we implement the randomization inference procedure to calculate the p-values of the sharp null of no effect for each individual treatment, holding other treatments’ assignments constant. Page 39 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) Results 77. Student learning outcomes improved in all groups, but the effects were much more pronounced in Group 2. The CEM-only treatment (Group 1) improved Indonesian and mathematics outcomes by 0.09 and 0.07 standard deviations (s.d.) respectively. Linking monetary incentives to the overall Community Score Card rating (Group 3) yielded similar learning impacts of around 0.11 and 0.09 s.d. for Indonesian and mathematics. However, having monetary incentives tied to the objective measures provided by the tamper-proof camera yielded impacts that were 1.5 and 2 times as large (0.17 and 0.20 for Indonesian and mathematics). The p-values from the randomization inference procedure are consistent with those from the regressions. 78. Overall, the interventions had no impact on the likelihood that a teacher was present at school, but Groups 1 and 2 had positive impacts on whether teachers were observed to be working when in school. Interestingly, the effect of Group 3 was negative, albeit non-significant. No changes were found in the likelihood that teachers performed academic activities when they were observed inside the classroom. In other words, even when teachers were in school or inside the classroom, they were chatting or conducting administrative activities unrelated to teaching. Furthermore, teachers' TSA status significantly affected their response to the interventions. The positive impacts of Group 2 on the likelihood of working when in school or teaching when inside the classroom were exclusively driven by the TSA teachers. Importantly, the overall negative impacts of Group 3 on both of those outcomes were driven by the non-TSA teachers. 79. Parents in treatment communities invested more money and time in their children’s education. Education expenditures increased by about Rp 29,000 (approximately USD 2) for Group 2 compared to the control-group average of Rp 324,154 (USD 23), constituting an increase of 8.6 percent. For the other interventions the estimates were smaller and not significantly different from those observed in the control or Group 2. Across all treatments, parents reported that their children were more likely to be accompanied when doing their homework by around 0.35 hours more from a base of about 2.5 hours per week. In Group 2 schools, the share of children with a paid tutor increased by 1.5 percentage points (p.p.) from a low base of 0.3 percent. 80. The IE also found that both teachers and parents were satisfied with the results of the interventions. All interventions increased teachers’ satisfaction rating of appreciation from non-school actors (the ministry, district office, and the village). Teachers in treated schools also reported greater satisfaction with the salary that they received. Interventions with a pay-for-performance component (Groups 2 and 3) increased satisfaction in their current job by around 6 percent (from a base of 85 percent). The increase in job satisfaction for Group 1 was much smaller and statistically insignificant. Parents’ assessment of the school quality was positively affected by these interventions. At baseline, there was already a high degree of satisfaction, even among control schools: 91 percent of parents rated their children’s school as either good or very good. The interventions increased this by about 5 percent. The interventions also affected their perception of whether teacher absenteeism was a main education problem in their community: The fraction of parents who reported this concern fell by between 6 percent (Treatment 1) and 8 percent (Treatment 3) from a base of 27 percent. Page 40 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) ANNEX 7. SUMMARY OF QUALITATIVE RESEARCH 81. The qualitative evaluation analyzed the impact of KIAT Guru in nine schools located in three of the five pilot districts: Ketapang (West Kalimantan), Landak (West Kalimantan), and Manggarai Barat (East Nusa Tenggara). In each district, three villages of similar characteristics were sampled. Researchers visited the nine schools three times between the end of 2016, before KIAT Guru started, to early 2018, shortly after KIAT Guru Project facilitators handed over implementation responsibilities to the stakeholders. The qualitative research was structured to gauge changes in stakeholders’ views about education service delivery as well as the community facilitation processes. The qualitative data collected through the case studies of the nine schools complements the quantitative measurements of the impact of KIAT Guru. 82. The data collected by the qualitative researchers indicated that implementation of the KIAT Guru had a strong and pervasive impact on the nine schools included in the study. Researchers concluded that in all nine schools, the quality of teaching and school-community relations improved over the course of the Project. Education stakeholders opined that introduction of KIAT Guru had exerted a particularly powerful impact in the following areas: (1) teacher attendance; (2) parent attitudes about student learning; (3) teacher performance; (4) student attitudes about learning; (5) teacher discipline; and (6) stakeholder relations. 83. Although researchers observed positive outcomes of all three approaches, Group 2 showed the greatest potential to produce long lasting change in the villages. The data generated by the quantitative researchers underscore the benefits of the changes introduced in Group 2 schools, where CEM was combined with PPM based on teacher presence. Creating a formal structure that empowers members of local school communities and centering their work on monitoring the presence of teachers appears to offer the greatest potential to produce positive changes in the educational services provided to rural school communities. 84. Researchers concluded that the positive outcomes of KIAT Guru outweighed the issues that surfaced in the communities they studied. As would be expected, all of these communities did face some challenges as they implemented the changes specified in Project guidelines. In some settings, those difficulties were more pronounced than in others. Although evidence collected in the nine schools indicates that Group 1 (CEM) did leverage some improvements in schools, this approach to reform was considered more unpredictable and difficult to enforce as it relied on social rewards and sanctions. Hypothetically, one might have expected schools in Group 3 (CEM + PPM based on teacher service performance) to yield the most widespread changes because it delegated greater authority to community representatives. Yet the depth of change associated with Group 3 proved to be too demanding, and ultimately limiting. Group 3 initially showed great promise, but a number of issues connected to the evaluation of teacher performance surfaced over the course of the Project and interfered with the school improvement process. By the final round of data collection, none of the researchers expressed support for Group 3. 85. The qualitative study offers insights that may be helpful for expansion and adaptation of KIAT Guru in other settings. The qualitative researchers documented conditions and behaviors that both supported and impeded smooth implementation of the initiative. Researchers attributed the success of Group 2 to five key elements: (1) actively engaging external stakeholders in monitoring and evaluating teacher performance; (2) increasing parental involvement in learning; (3) keeping teacher performance indicators simple; (4) using objective rather than subjective measures to evaluate teacher performance; and (5) paying teacher allowance based on objective performance indicator. Page 41 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) 86. Another key finding from the qualitative study is the importance of clearly communicating the goals of KIAT Guru and clarifying the duties of all parties involved. In the schools that achieved the most impressive gains, this was done with noteworthy transparency. Because the initiative requires the buy-in of actors operating at multiple levels of the education system, it is imperative that all stakeholders understand the overarching structure of KIAT Guru implementation, and how they fit into that system. 87. In summary, results from the qualitative study of KIAT Guru indicate that community empowerment linked to a pay-for-performance based on teacher presence (Group 2) shows the greatest potential for leveraging change in the teacher behavior. Responsibilities for key stakeholders need to be carefully delineated and communicated, and training should be provided to educators, community representatives, and government officials. If these conditions can be met, KIAT Guru can establish a more stable foundation for educational improvement in schools throughout Indonesia. Page 42 of 43 The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT GURU) (P159191) REFERENCES ACDP (2014). Study on Teacher Absenteeism in Indonesia 2014. Jakarta, Indonesia: Education Sector Analytical and Capacity Development Partnership. Banerjee, A., and E. Duflo. 2006. Addressing absence. Massachusetts Institute of Technology. http://economics.mit.edu/files/795. Banerjee, A., R. Banerji, J. Berry, E. Duflo, H. Kannan, S. Mukerji, M. Shotland and M. Walton, (2017). From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application. NBER Working Paper No. 22931. Banerjee, A., R. Glennerster, and E. Duflo, (2008). Putting a band-aid on a corpse: Incentives for nurses in the Indian public health care system. Journal of European Economic Association, 2008, 6(2-3): 487-500. BPS (Indonesian Center of Statistics Agency). (2016). Profil Kemiskinan di Indonesia, September 2015. (Poverty Profile in Indonesia, September 2015). Jakarta: BPS. Bold, T., M. Kimenyi, G. Mwabu, A. Ng’ang’a, and J. Sandefur, (2018). Experimental Evidence on Scaling Up Education Reforms in Kenya. Journal of Public Economics, Volume 168, December 2018, pages 1-20. Duflo, E., R. Hanna, and S.P. Ryan. (2012). Incentives Work: Getting Teachers to Come to School. American Economic Review 102 (4): 1241-78. Glewwe, P. and K. Muraliradhan. (2015). Improving School Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications. Retrieved from https://econweb.ucsd.edu/~kamurali/papers/Published_Book_Chapters/School_Education_Developing_Countr ies.pdf. JPAL. (2019). Conducting Cost Effectiveness Analysis. Retrieved from https://www.povertyactionlab.org/research- resources/cost-effectiveness. OECD. (2014). PISA 2012 Results in Focus: What 15-year-olds Know and What They Can Do With What They Know. Paris, France: OECD. SMERU. (2004). When teachers are absent. Jakarta: SMERU Research Institute. Retrieved from http://www.smeru.or.id/report/field/ketikaguruabsen/ketikaguruabsen-eng.pdf SMERU (2010). Teacher Absenteeism and Remote Area Allowance: Baseline Survey. Jakarta, Indonesia: SMERU Research Institute. Stern, J. & Nordstrum, L. (2014). Indonesia 2014: The National Early Grade Reading Assessment (EGRA) and Snapshot of School Management Effectiveness (SSME) Survey. North Carolina, USA: Research Triangle Park Press. Suryadarma, D., A. Suryahadi, S. Sumarto and F.H. Rogers (2004). The Determinants of Student Performance in Indonesian Public Primary Schools: The Role of Teachers and Schools. Jakarta, Indonesia: SMERU Research Institute. Suryahadi, A. and P. Sambodho (2013). Assessment of Policies to Improve Teacher Quality and Reduce Teacher Absenteeism. Jakarta, Indonesia: SMERU Research Institute. UNICEF (2012). We Like Being Taught – A Study on Teacher Absenteeism in Papua and West Papua. Jakarta, Indonesia: UNICEF. World Bank (2013). Early Childhood and Development in Poor Villages of Indonesia: Strong Foundations – Later Success. Jakarta, Indonesia: World Bank. Page 43 of 43