Document of The World Bank Report No: ICR 2655 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-7476; IDA-4349; TF-90794) ON AN IBRD IN THE AMOUNT OF US$24.5 MILLION AND AN IDA CREDIT IN THE AMOUNT OF SDR 40.33 MILLION (US$61.5 MILLION EQUIVALENT) AND GRANT IN THE AMOUNT OF US$52 MILLION TO THE REPUBLIC OF INDONESIA FOR A BETTER EDUCATION THROUGH REFORMED MANAGEMENT AND UNIVERSAL TEACHER UPGRADING (BERMUTU) PROJECT June 30, 2014 Human Development Sector Unit/Education South East Asia Country Unit East Asia and Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 25, 2013) Currency Unit = IDR IDR11,494.30 = US$1 IDR 1.00 = US$0.00009 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS ABBREVIATIONS AND ACRONYMS ACAP Anti-Corruption Action Plan ACDP Analytical and Capacity Development ADB Asian Development Bank AIG Accreditation incentive grant APBD Local Budget AusAID Government of Australia Agency for International Development Balitbang Office of Research and Development within the Ministry of National Education BAN-PT National Board of Accreditation for Higher Education BERMUTU Better Education through Reformed Management and Universal Teacher Upgrading Project BPSDMP-PMP Board of Education, Human Resource Development and Education Quality Assurance (HRD & QA Board) CAS Country Assistance Strategy CPD Continuing Professional Development DAPODIK Educational Core Data DFAT Department of Foreign and Trade of the Australian Government DGBE Directorate General of Basic Education DGQITEP Directorate General of the Quality Improvement of Teachers and Education Personnel DGSE Directorate General of Secondary Education DIA Dana Insentif Akreditasi (Accreditation Incentive Grant) DIKTI Directorate General of Higher Education DIPA Daftar Isian Pelaksanaan Anggaran (Budget Implementation Registration Form) FM Financial Management FY Financial Year HYLITE Hybrid Learning for Indonesian Teachers IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDA International Development Association ii IDR Indonesian Rupiah ISR Implementation Status and Results Report IVPF Isolated Vulnerable People’s Framework GDP Gross Domestic Product GoI Government of Indonesia GPO Guru Pintar Online KKG/MGMP Teacher Working Groups KKKS/MKKS School Principal Working Groups KKPS/MKPS Supervisor Working Groups LPMP Educational Quality Assurance Institution LPTK Teacher Training Institutions M&E Monitoring and evaluation MenPAN Minister of State Apparatus MoEC Ministry of Education and Culture MoNE Ministry of National Education NUPTK Teacher Database P2TK DIKDAS Education Personnel Management of Basic Education P4TK Center for Development and Empowerment of Teachers and Ed. Personnel (in-service) PAD Project Appraisal Document PCU Project Coordination Unit PDO Project Development Objectives PDSP Center for Education Data and Statistics PGSD Pendidikan Guru Sekolah Dasar (Basic Education Teacher Training) PIGP Program Induksi Guru Pemula (Teacher Induction Program for Beginners) PISA Program for International Student Assessment PKG Teachers Annual Performance Appraisal POM Project operational manual PMPTK Directorate General Quality Improvement of Teacher and Education Personnel PUSBANGPRODIK Center for Teaching Profession Development PUSBANGTENDIK Center for Teaching Personnel Development PUSLITJAK Center for Policy Research PUSPENDIK Center for Educational Assessment QER Quality enhancement review RENSTRA Medium-Term Education Strategic Plan RF Results framework RPL Recognizing the prior learning SATKER Implementing unit SDR Special Drawing Rights SSQ School System and Quality program TF Trust Fund TIMSS Trends in International Mathematics and Science Study TPM Teacher Performance Management iii TTL Task team leader USAID United States Agency for International Development USD United States Dollar UKG Uji Kompetensi Guru (Teacher Competency Test) WB World Bank Vice President: Axel van Trotsenburg Country Director: Rodrigo A. Chavez Sector Manager: Luis Benveniste Project Team Leader: Susiana Iskandar ICR Team Leader: Susiana Iskandar ICR Primary Author: Juan Prawda Sector Quality Reviewers: Franco Russo/Sandra Beemer iv Republic of Indonesia Better Education Through Reformed Management and Universal Teacher Upgrading CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 5 3. Assessment of Outcomes .......................................................................................... 12 4. Assessment of Risks to Development Outcome ....................................................... 19 5. Assessment of Bank and Borrower Performance ..................................................... 20 6. Lessons Learned ....................................................................................................... 23 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 24 Annex 1. Project Costs and Financing .......................................................................... 25 Annex 2. Outputs by Component ................................................................................. 26 Annex 3. Economic and Financial Analysis ................................................................. 38 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 47 Annex 5. Beneficiary Survey Results ........................................................................... 50 Annex 6. Stakeholder Workshop Report and Results................................................... 51 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 52 Annex 8. Comments of the Royal Netherlands Government ....................................... 58 Annex 9. List of Supporting Documents ...................................................................... 59 MAP .............................................................................................................................. 63 v A. Basic Information BERMUTU-Better Education through Country: Indonesia Project Name: Reformed Management and Universal Teacher Upgrading IBRD-74760, IDA- Project ID: P097104 L/C/TF Number(s): 43490, TF-90794 ICR Date: 06/30/2014 ICR Type: Core ICR REPUBLIC OF Lending Instrument: SIL Borrower: INDONESIA Original Total USD 138.00M Disbursed Amount: USD 128.41M Commitment: Revised Amount: USD 138.00M Environmental Category: C Implementing Agencies: Ministry of Education and Culture Cofinanciers and Other External Partners: Ministry of Foreign Affairs and Development Cooperation Note: Due to a system’s glitch, the Dutch TF commitment (of total US$ 52 million) and disbursement (of total US$ 50.87 million) were not captured in the Portal version of this table. B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 08/24/2006 Effectiveness: 02/05/2008 02/05/2008 11/22/2011 Appraisal: 04/11/2007 Restructuring(s): 10/05/2012 Approval: 07/05/2007 Mid-term Review: 07/05/2011 10/18/2011 Closing: 12/31/2013 12/31/2013 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Low or Negligible Bank Performance: Satisfactory Borrower Performance: Moderately Satisfactory vi C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Satisfactory Implementing Quality of Supervision: Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 19 15 Primary education 47 Secondary education 20 Sub-national government administration 3 3 Tertiary education 78 15 Theme Code (as % of total Bank financing) Education for all 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Axel van Trotsenburg James W. Adams Country Director: Rodrigo A. Chaves Joel Hellman Sector Manager: Luis Benveniste Christopher J. Thomas Project Team Leader: Susiana Iskandar Mae Chu Chang ICR Team Leader: Susiana Iskandar ICR Primary Author: Juan Prawda vii F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objective is to contribute to the improvement of the overall quality and performance of teachers through enhancing teachers' knowledge of subject matter and pedagogical skills in the classroom. Improvement of teacher quality and performance is to be measured by: (i) the increased number of teachers meeting academic qualifications mandated by the Teacher Law; (ii) the increased number of primary and junior secondary teachers in BERMUTU Districts using classroom instruction with specific subject and age appropriate pedagogy; and (iii) the reduction in absenteeism of teachers in BERMUTU Districts. Revised Project Development Objectives (as approved by original approving authority) N/A (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Increased number of teachers meeting academic qualifications mandated by the Indicator 1 : Teacher Law Value quantitative or 900,000 1,400,000 1,713,163 Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments (incl. % Target exceeded by 22.4 percent achievement) Increased number of primary and junior secondary teachers in BERMUTU Indicator 2 : Districts using classroom instruction with specific subject and age appropriate pedagogy Value quantitative or 17,000 190,000 214,380 Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Target exceeded by 12.8 percent. Findings of a study commissioned to assess the Comments impact of this indicator reveals that teacher’s participation in working groups (incl. % improved their abilities in teaching preparation, delivery and formative achievement) assessment of their students. viii Indicator 3 : Reduction in absenteeism of teachers in BERMUTU Districts In BERMUTU Value Districts: 6.6% quantitative or 19% 15% primary schools; Qualitative) 6.3% junior secondary schools Date achieved 05/30/2007 12/31/2013 12/31/2013 Target achieved. Study by MoEC in BERMUTU districts had the rate decreased Comments from 8.4% in 2010 to 6.6 % in 2012 in primary school and from 12.8% to 6.3% (incl. % in junior secondary school. National representative study by SMERU in primary achievement) schools found 19% in 2003 and 15% in 2008. (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Increased number of pre-service programs accredited (initially secondary Indicator 1 : programs only) Value (quantitative 0 75 81 or Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments Target exceeded by 8%. The 81 primary teacher education PGSD pre-service (incl. % programs have been accredited at the S1 level (bachelor degree), including 26 achievement) PGSD participating in the BERMUTU Accreditation Incentive Grants. Increased number of new graduates from accredited programs meeting the Indicator 2 : mandated competency standards Value (quantitative 0 15,000 13,179 or Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments Data not yet complete, as the first S1-level accreditation for PGSD was awarded (incl. % in 2010 and the 1st cohort of graduates could only be identified in 2013, i.e. achievement) 13,179 from 18 accredited S1-PGSD. Increased numbers of teachers gaining recognition for prior learning (RPL) from Indicator 3 : accredited universities. Value (quantitative 0 700,000 879,832 or Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments Target exceeded by 25.7%. About 87,501 have been granted RPL in 81 Teacher (incl. % Training universities (LPTKs) and 792,331 in the Open University. achievement) ix Indicator 4 : Increased numbers of active KKG/MGMPs in BERMUTU Districts Value (quantitative 1,200 4,500 6,107 or Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments Target exceeded by 35.7%. There are 6,107 active professional working groups (incl. % of primary and junior secondary school teachers, principals and supervisors. achievement) Increased numbers of beginning teachers being provided with school induction Indicator 5 : programs in BERMUTU Districts. Value (quantitative 0 3,000 1,429 or Qualitative) Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments Data currently being updated by Pusbangtendik. The Ministerial regulation and (incl. % its operational manuals are in place and the socialization and training of trainers achievement) in both BERMUTU and a non-BERMUTU has been completed. Revised policies, plans and procedures for continuing education and career Indicator 6 : development of certified teachers developed and piloted for implementation by the government in 2011 Value Available and Available and being (quantitative Limited policies in place implemented by implemented or Qualitative) GoI Date achieved 05/30/2007 12/31/2013 12/31/2013 Target achieved. MenPAN Decree 16/2009 on Functional Credits for Teachers, Comments the instruments for teacher performance assessment and teacher continuous (incl. % professional development and manuals have been developed, trialed and ready to achievement) be implemented. A teacher database established to track progress in teacher placement, academic Indicator 7 : qualifications, certification, and payment of professional allowances. Value Management information Available and Teacher database (quantitative system does not allow implemented by completed or Qualitative) complete analysis GoI Date achieved 05/30/2007 12/31/2013 12/31/2013 Comments Target achieved. A teacher relational database, overlaying databases of teachers, (incl. % schools data base and student performance has been established. achievement) x G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) Moderately Moderately 1 04/09/2008 8.00 Unsatisfactory Unsatisfactory 2 09/16/2008 Satisfactory Moderately Satisfactory 8.00 3 02/04/2009 Satisfactory Moderately Satisfactory 9.66 4 09/09/2009 Satisfactory Moderately Satisfactory 13.05 5 03/03/2010 Satisfactory Moderately Satisfactory 23.03 6 12/20/2010 Satisfactory Moderately Satisfactory 49.88 7 10/09/2011 Satisfactory Moderately Satisfactory 56.96 8 04/21/2012 Satisfactory Moderately Satisfactory 70.87 9 12/19/2012 Satisfactory Moderately Satisfactory 69.80 10 06/30/2013 Satisfactory Moderately Satisfactory 81.68 11 12/25/2013 Satisfactory Satisfactory 81.68 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Reallocation of funds across categories needed to accommodate the larger than 11/22/11 N S MS 91.28 originally anticipated number of teachers participating in the working group program (under Category 4) estimates 10/05/12 N S MS 121.29 Reallocation of funds Note: Due to a system’s glitch, the Portal version of this table does not capture the Dutch TF disbursements at the time of restructuring. xi I. Disbursement Profile Note: The project also benefitted from a US$52 million grant from the Dutch government. Amount disbursed is US$50.87. Due to a system glitch, this amount is not reflected in the disbursement graph. xii 1. Project Context, Development Objectives and Design 1. On July 5, 2007, the Bank approved Loan number 7476-IND in the amount of US$24.5 million and Credit number 4349-IND in the amount of Special Drawing Rights (SDR) 40.33 million (US$61.50 million equivalent to support the Indonesia “Better Education Through Reformed Management and Universal Teacher Upgrading Project (BERMUTU), which became effective on February 5, 2008. Further financial support in the amount of US$52 million was provided by the Dutch government through its Ministry of Foreign Affairs and Development Cooperation in the form of a Bank- administered trust fund to help finance project activities. The trust fund was approved on November 7, 2007 and became effective the same day. Furthermore, the Bank was engaged in Indonesia through a number of Bank-executed trust funds which often complemented and informed ongoing project implementation. 1.1 Context at Appraisal. 2. Background. At the time of project preparation in 2006, Indonesia ranked as a lower-middle income country, with a gross domestic product (GDP) per capita of US$4,248. Indonesia had restored its macroeconomic and financial stability and economic growth had increased from 3.8 percent in 2001 to 5.6 percent in 2005 while the percentage of the population living in poverty in the period 1999-2007 had fallen from 27 to below 16 percent in 2007. 3. Despite all of these achievements, Indonesia continued to under-perform compared with its neighboring countries in terms of access to quality health, education and other basic services, as reflected in the millennium development goals’ (MDG) indicators. Indonesia’s centralized development planning and service delivery systems had increased overall access to basic services but had fallen short in ensuring quality. 4. Main issues in the education sector. The quality of education in Indonesia, as measured by learning outcomes, was of serious concern. In the standardized tests of the Trends in International Mathematics Science Study (TIMSS 2003), assessing mathematics and science knowledge of 4th and 8th graders (loosely, ages 9/10 and 13/14, respectively), Indonesian students ranked 34th out of 45 countries. In the 2003 Program for International Student Assessment (PISA), assessing scholastic performance on mathematics, science, and reading of 15 year-old students (regardless of their school grade), Indonesia was ranked below its major economic competitors. 5. On a proficiency scale from zero to six for mathematics, over five percent of Indonesian students did not reach level one, and none reached level five or above. In reading, only 31 percent could complete more than the most basic reading tasks. Even after taking family socioeconomic status into account, student outcomes in Indonesia were lower than those of neighboring countries, which suggested that there were deficiencies in the classroom experiences that teachers were providing for their students. 1 6. A groundswell of public support in favor of improving educational quality led to revisions to Indonesia’s constitution in 2003 requiring that 20 percent of government spending be directed to education. In the period 2004-05, there was mounting pressure through Parliament to increase education spending substantially. 7. In 2006, an average Indonesian primary school teacher salary ranged between US$1,002 to US$3,022 per annum, a mere 50 percent of GDP per capita, half of the annual earnings of primary school teachers in the Philippines and Thailand. The Government of Indonesia (GoI) agreed that holding teachers accountable for a high standard of professional conduct required a professional level of remuneration and that quality improvement would be possible only if the teaching force earned a living wage. 8. In 2000, the GoI decentralized teacher management responsibilities to the local governments through a general allocation of funds to finance, among others, teacher salaries. This decentralization scheme created some issues, such as: (i) limited teacher mobility across districts; (ii) restricted career advancement for certain levels/rank as this decision was taken by the Ministry of Education and Culture (MoEC) 1; and, (iii) uncontrolled growth in the number of teachers contracted by either the provincial or district governments and hired by schools as “honorary” teachers. 9. Government’s strategy. In December 2005, the GoI passed a teacher law (UU 14/2005) requiring teachers to: (i) have a minimum academic qualification of at least four years of post-secondary education; (ii) obtain practical experience as a classroom teacher; and (iii) pass a certification examination assessing their proficiency in four competency domains (as defined by the National Education Standards Agency in the pedagogical, professional, personal, and social domains) before being certified as a teacher. Once certified, teachers would double their salary and receive an additional functional allowance of approximately 20 to 30 percent of their base salary depending on their civil service grade. Teachers assigned to disadvantaged areas would also get a special allowance equivalent to 100 percent of their base civil service salary to help incentivize being posted in hard-to-staff locations. The challenge for the GoI was to manage this transformation of the incumbent workforce of some 2.7 million teachers, 65 percent of whom did not have the newly prescribed minimum four-year qualification. 10. Rationale for Bank’s assistance. There were four reasons underpinning the Bank’s assistance. First, service delivery was one of the two main pillars of the FY04-07 Country Assistance Strategy (CAS) for Indonesia and accordingly, the GoI requested the Bank’s support in the area of civil service reform. The 2.7 million teachers constituted more than 70 percent of the core civil service workforce in Indonesia at the time of project identification. 11. Second, the extensive analytical work undertaken by the Bank in 2005 (mostly financed by Dutch and the Government of Australia Agency for International 1 The ministry was restructured in 2011, which included (amongst other things) changing its name from Ministry of National Education (MoNE) to Ministry of Education and Culture (MoEC). For simplicity’s sake, the ICR will use MoEC when referring to both the pre- and post-restructured ministry. 2 Development [AusAID] trust funds) concerning managing the transition to decentralization, employment and deployment and teacher certification contributed to the enactment of the Teacher Law approved by the Indonesian Parliament in 2005. These analytical activities cemented a strong relationship of the Bank with the MoEC. The GoI was interested in the Bank’s international expertise in teacher management and education quality assurance to mitigate possible implementation risks. 12. Third, against the backdrop of the decentralized education service delivery in Indonesia, the Bank was seen as an effective facilitator to ensure required close cross- sector collaborations and the sharing of lessons of experience from other sectors that had also managed policy reforms in a decentralized system. Fourth, the proposed project financing along with the Bank-executed Trust Funds, were considered as complementary, rather than substitute financial resources to the GoI’s funding of certification, scholarships and incentive allowances to teachers. 1.2 Original Project Development Objectives (PDO) and Key Indicators. 13. The project’s PDO was to contribute to the improvement of the overall quality and performance of the teachers in the Republic of Indonesia through enhancing their knowledge of subject matter and pedagogical skills in the classroom. Progress towards achieving the PDOs were to be measured against the following three outcome indicators: (i) Increased number of teachers meeting academic qualifications mandated by the Teachers Law; (ii) Increased number of primary and junior secondary teachers in BERMUTU Districts using classroom instruction with specific subject and age appropriate pedagogy; and (iii) Reduction in absenteeism of teachers in BERMUTU Districts. 14. Agreement was reached with the GoI that the intermediate outcome indicators for the end of the project were: Component 1: (i) pre-service programs accredited (initially secondary programs only); and (ii) new graduates from accredited programs who would meet the mandated competency standards; Component 2: (i) teachers gaining recognition for prior learning (RPL) from accredited universities; (ii) active teacher working groups (KKG) in BERMUTU Districts; and (iii) beginning teachers being provided with school induction programs in BERMUTU districts; Component 3: revised policies, plans, and procedures for the continuing education and career development of certified teachers developed and pilot tested in time for implementation by the GoI in 2011; and Component 4: a teacher database to track progress in teacher placement, academic qualifications, certification, and payment of professional allowances. 1.3 Revised PDO (as approved by the original approving authority) and Key Indicators, and reasons/justification 15. The PDO was not modified during the project implementation period. 3 1.4 Main Beneficiaries 16. The proposed regulatory framework aimed at benefiting in the longer-term a teaching workforce of about 2.7 million kindergarten, basic (primary and junior secondary) and secondary teachers (including vocational education teachers), 65 percent of them lacking the newly prescribed minimum four-year qualification. In addition, the working groups aimed at targeting teachers, principals and supervisors of public and private basic education schools (primary and junior secondary) in 75 Districts (selected from a total of 441 according to agreed criteria) of 16 Provinces in the country (out of 34). Finally, all public and private teacher-training institutes would benefit from the strengthened accreditation system and 40 of them, from accreditation grants. 1.5 Original Components 17. Component 1: Reforming University-based Teacher Education (US$20.5 million), through the following three sub-components: (i) Providing assistance to the National Board of Accreditation for Higher Education (BAN-PT); (ii) Grants and scholarships; and (iii) Overseas scholarship and non-degree training program to improve the quality of university teaching and coursework. 18. Component 2: Strengthening Structures for Teacher Improvement at the Local Level (US$134.2 million). This component was to be implemented in 75 of Indonesia’s 441 districts in 16 out of 34 provinces and was comprised of the following sub-components: (i) Providing technical assistance to establish agreed mechanisms for RPL of teachers; (ii) Providing technical assistance to the Directorate General for Quality Improvement of Teacher and Education Personnel (PMPTK) to develop subject-based modules and modules on management; (iii) Teacher Working Groups (KKG); and (iv) School Principal Working Groups (KKKS/MKKS) and Supervisor Working Groups (KKPS/MKPS). 19. Component 3: Reforming teacher accountability and incentives systems for performance appraisal and career advancement (US$5.3 million) through two sub-components providing funds for technical assistance, studies, workshops, and operational costs for the development and piloting of these policies, processes, and instruments. 20. Component 4: Improving program coordination, monitoring and evaluation (US$33.8 million) through the following three sub-components: (i) Monitoring of project activities and teacher certification; (ii) Evaluation and assessment; and (iii) Project coordination and monitoring. 1.6 Revised Components 21. The original components remained unchanged throughout the implementation period. 4 1.7 Other significant changes 22. The first Loan Amendment of November 23, 2011 reflected reallocations needed to accommodate the larger than originally anticipated number of teachers participating in the working group program (under Category 4) estimates, while the second Loan Amendment of October 5, 2012 was made in response to GoI’s official exchange rate change, which affected the already agreed upon disbursement plans and relevant fund availability. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 23. Weighing the following achievements and shortcomings, the ICR rates the BERMUTU preparation, design and quality at entry as substantial for the reasons explain below. 24. Project preparation. BERMUTU’s strategic relevance and approach was appropriate to support MoEC's core agenda concerning the implementation of the Teacher Law, as a first required step to improve teacher professionalism, performance, and accountability and improve education quality. The preparation of BERMUTU benefited from a rich analytical foundation including studies on teacher management, accountability and incentive systems, pre- and in-service teacher training institutions and classroom processes. Specifically, the Bank’s 2005 analytical work “Managing the Transition to Decentralization” and papers on teacher employment, deployment and certification identified key issues and proposed policy measures, which ultimately contributed to the development of the Teacher Law in 2005. Therefore, the Bank was well positioned to continue to draw from that analytical work (amongst other sources) and inform project preparation to be relevant to the country and the education sector needs. 25. Preparation activities also included a pilot of teacher employment-deployment in five districts and a survey of ongoing teacher support groups, both financed by Dutch and AusAID trust funds managed by the Bank. The project was also in line with the FY04-07 CAS’ Objective II “Making Service Delivery Responsive to the Needs of the Poor”. The Bank preparation team anchored in the Bank office in Jakarta worked very closely, and in a continuous harmonious fashion with the MoEC, providing technical support and analytical inputs for informed decision-making, including the conceptualization and operationalization of the Teacher Law. The interaction with other donors, in particular the Dutch Embassy and AusAID providing support to BERMUTU’s preparation activities through trust funds managed by the Bank, was satisfactory. In addition, the Bank preparation team kept close coordination with other donors, like USAID and New Zealand, financing related projects in the education sector. 26. The project design and quality at entry. The project benefitted from three quality reviews, of which one was a Quality Enhancement Review (QER). The QER 5 panel report acknowledged the QER’s strong support for the project because it represented the Bank’s largest and most innovative global initiative aimed at teacher skills upgrading 2. It also endorsed the project’s design and approach to teacher skills upgrading and noted the project’s significantly simplified design and evidence of government ownership to implement the different components. The QER panel report also suggested that it was important to: (i) pay particular attention to sequencing of activities and propose mitigation measures to safeguard PDO achievement against possible delays; (ii) clarify the district and training institution selection process to receive grants; (iii) evaluate the government’s existing grants program: (iv) assess GoI’s own implementation arrangements as to avoid creating parallel implementation and oversight functions; and (v) clarify responsibility for the different sub-components/activities, including the types of certification standards to be adopted. The preparation team incorporated the QER’s suggestions and expanded the discussion in the relevant sections of the Project Appraisal Document (PAD), such as the risk matrix, institutional arrangements and financial management sections and detailed project description. 27. The QER and decision meeting also acknowledged the importance of the proposed project’s support to GoI’s efforts in implementing the law and capitalizing on the opportunities it presented. As confirmed by the Decision Meeting, the project’s focus on the new Teacher Law provided “an imperative for reform that has not been previously present for education projects” 3. Quality at entry also benefitted from the “Readiness Filter Agreement” established between the Bank and GoI to facilitate project readiness and start-up and which entailed the existence of a draft Project Operations Manual (POM) and a co-financing agreement. The 2008 procurement plan and request for consultancy proposals was also in place, along with the following manuals: (i) the Accreditation Incentive Grants; (ii) the Distance Education Modules; and (iii) the ones for Teacher, Principal and School Supervisor Working Groups. 28. Although the initial project design was thought to be somewhat complex and required a shared vision by the many implementing actors, the QER panel also felt that the final design had been substantially simplified and that the complexity and implementation risks were mitigated in the final design. In addition, the QER suggested that the PAD specify who would be in charge of implementation of each activity, which the PAD tried to address in Annex 4 “Detailed Project Description” and Annex 6 “Institutional and Implementation Arrangements”. Lastly, there was an acknowledgement by the decision meeting that some risks should be highlighted even though they may be outside of the project’s control, providing the project with a more holistic approach to identifying and mitigating (whenever possible) potential risks. The PAD also identified decentralized funds management and potential for corruption as possible risks and appropriate mitigation measures were proposed at the time. Although these mitigation measures were not always sufficient to address the project’s decentralized institutional arrangements and capacity constraints, these challenges were largely overcome by mid- term and ultimately led to all key project activities being implemented by project closing. 2 QER panel report, March 5, 2007 3 Decision meeting minutes, April 19, 2007 6 2.2 Implementation 29. The project was able to implement all envisaged activities and meet (and in some cases exceeded) most output targets: (i) 81 pre-service programs were accredited, exceeding the end-target of 75; (ii) the target of 15,000 new graduates from accredited programs meeting mandated competency standards 13,179, missing the end target by 1,821. Due to the sequenced nature of this activity and the fact that each PGSD was allowed three to four years to become accredited, the number of actual graduates was affected by the pace of PGSD accreditation. It is expected, however, that the end target will be exceeded in 2014; (iii) 879,832 teachers gaining recognition for prior learning (RPL) from accredited universities, exceeding the 700,000 end target; (iv) 6,107 KKG/MGMPs (teacher working groups) are active in BERMUTU districts, exceeding the 4,500 final target; (v) 1,429 beginning teachers were provided with school induction programs in BERMUTU districts, falling short of the 3,000 end target; (vi) revised policies, plans and procedures for continuing education and career development of certified teachers are being implemented through MenPAN Decree 16/2009, meeting the end target; and (vii) a teacher database has been established, meeting the end target. In addition, the new regulatory system for teacher accountability and performance incentives is in place, functional coordination among key units within MoEC has markedly improved and the regulatory framework amongst teachers is in full execution. 30. By doing that, BERMUTU: (i) started changing the way basic education teachers- to-be are prepared (LPTK); (ii) improved the curriculum-subject knowledge and teaching skills of all in-service teachers (KKG; MGMP); (iii) set up standards (Teacher Performance Management (TPM) scheme) and targets to manage the teaching labor force; (iv) accompanied these changes with financial incentives to improve teachers' welfare based on teachers’ performance results (TPM scheme designed for the professional career allowance including all the assessment instruments). The project also helped focus on basic education teachers, principals and supervisors from disadvantaged districts first as a feasible way to get the process started in a more equitable fashion. 31. Initial implementation was not without its challenges: (i) there were significant delays in the government budget document (DIPA) for fiscal year 2008, limiting implementation activities to three key Satkers that year. Budget allocation levels did steadily improve in subsequent years to finance 60 percent of agreed project activities in 2009 and by 2010, 24 Satkers received their DIPA allocation. The 2008 and 2009 periods were also marked by partial allocation of counterpart funds, which combined with the DIPA delays, slowed implementation during that period. This was partially mitigated by intensive consultations between the Bank and MoEC to speed up the budget exercise and ensure adequate funding, with counterpart funding reaching a satisfactory level by 2010; (ii) it took time for Satkers to become familiar with donor-financed processes and implementation procedures. Although appropriate capacity building and technical assistance support was included as part of the project, delays in getting consultants on board and the quality of some selected consultants to support Satkers slowed initial project implementation; (iii) limited coordination, communication and shared vision among Satkers at the beginning also affected their ability to set short and medium term 7 goals and translating these into a proposed DIPA. This issue was resolved by 2010 to a large extent as Satkers became more and more familiar with this new approach; and (iv) the extensive restructuring of the ministry in 2011 which required mapping each project component and sub-component to the new structure and therefore delayed government budget allocation and required staff to establish new working relationships across the restructured ministry units. 32. The restructuring was seen by the Bank as a welcomed move for the project to further strengthen the integration between teacher management and basic education management. Furthermore, MoEC and the Bank were actively engaged in day-to-day activities which facilitated implementation through these earlier challenges. The project teams were proactive in updating the relevant implementation plans based on budget availability to maintain proper sequencing. As a matter of fact, the initial implementation and budget challenges were satisfactorily resolved by the 2011 mid-term review (MTR) and the project was largely on track to achieving its development objectives. The project was cited in the Bank publication “Teacher Reform in Indonesia – the role of politics and evidence in policy making” 4 as having had a significant impact on teacher working groups and being effective in increasing teacher knowledge levels. Finally, and mentioned previously, the project had carried out all envisaged activities and disbursed the majority of funds 5 from the three sources of financing by project closing without the need for a closing date extension. This is an impressive achievement given the project’s initial delays, perceived complexity and extensive ministry restructuring. 33. Therefore, overall project implementation is rated as satisfactory. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 34. The M&E design was appropriate. The M&E design had two aspects: (i) the MoEC was to continue to collect data through its existing teacher and school monitoring system to report on results framework and project output achievements; and (ii) through a series of impact evaluation studies. The project was also supporting the improvement and subsequent integration of the teacher database with the school and teachers working group databases. The PDOs were stated with sufficient operational precision allowing for gauging their attainment through the observation and measurement of a metric comprising three outcome and seven intermediate outcome indicators. Baseline data was included at the onset of project implementation and end-of-project outcome and intermediate targets were deemed achievable at the time of project design. The results framework was not modified during the entire implementation period and there were no issues of attribution as there was connectivity between the indicators, the PDOs and the key interventions included in the project design. The Bank’s regional operational service department praised the project’s M&E framework as a good practice example that was being used in Bank training programs. 4 The World Bank, ISBN: 978-0-8213-9960-6, 2014 5 At the time of ICR writing, 73.96 percent of loan, 96.6 percent of credit and 97.82 percent trust fund proceeds had been disbursed, reaching a combined disbursement rate of 93.1 percent. 8 35. M&E implementation and utilization was moderately satisfactory. MoEC regularly provided the Bank with data and information that allowed the teams to make judgments on progress toward achieving the PDO and the project’s overall implementation. This information was consistently reported on by the Bank in aide- memoires and Implementation Status and Results (ISR) reports. MoEC also used budget information to adjust implementation and procurement plans, especially during the budget challenges that occurred at the beginning of the project. This was critical to ensure the sequencing was maintained and all envisaged activities could be completed by project closing. Three impact evaluations on pre-service teacher training, teacher certification and teacher working groups were successfully carried out, as well as a teacher absenteeism study in BERMUTU districts. These reports provided the government and MoEC in particular with relevant insight into overall impact of BERMUTU interventions and (in the case of teacher working groups) led to a shift in focus of the KKG/MGMP reporting from administrative matters to classroom teaching impact. A video study was conducted to observe classroom teaching of eight-grade mathematics to students who participated in the 2007 and 2011 TIMSS, which also provided valuable insights into teaching practices 6. 36. Several monitoring instruments were also developed to assess progress of the five areas of expected district-support to BERMUTU: (i) financial allocation of at least 18 percent of the district’s annual budget to the working groups; (ii) provision of equipment to the schools for the delivery of pedagogical activities learned in the working groups; (iii) BERMUTU-related data entry at the district level; (iv) on-sight monitoring and supervision; and (v) ensuring consistency of BERMUTU with other related donor- financed education projects in the district. Finally, a complaint handling mechanism concerning teacher professional allowance payment delays which was established but not functioning is now operational. The teacher database was successfully integrated with the core education data (Dapodik) and the Center for Education Data and Statistics (PDSP) by project closing after encountering some delays 7. 37. Going forward, there exists an opportunity to further strengthen the institutional capacity at the central and district levels to produce, collect, analyze and update progress on teacher placement, academic qualifications, certification and payment of professional allowance. There were also a few activities not directly financed by the project but related to its activities that could not be completed by project closing, which include: (i) improvements in navigating MoEC’s main information portal (Gerai Informasi) as part of the ACAP Complaint Handling aspect; (ii) the analysis of the teacher working group data profiles as part of overall M&E functions; and (iii) improvements in the coordination of all of the M&E functions and deliveries carried out by all the Satkers as part of overall 6 The PAD envisaged project support for Trends in International Mathematics and Science Study (TIMSS), the Program of International Student Assessment (PISA) and Progress in International Reading Literacy Study (PIRLS). While the government funded the participation in those three internatioal studies, the WB supported the TIMSS video study which would provide useful insights into teaching practices than conducting more rounds of assessment. In addition MoEC also conduct the Early Grade Reading Assessment, supported by USAid. 7 One of the main reasons for the delay was the restructuring of MoNE, which initially affected staffing and budget allocation of the restructured departments in charge of this activity. 9 project coordination. 2.4 Safeguard and Fiduciary Compliance 38. Financial management. Financial management faced challenges at different times since the beginning of the project due to the time it took to put in place the necessary FM support and set up appropriate financial control procedures and guidelines related to training and workshop expenditure documentation 8. The main challenges included: (i) initial concerns regarding the verification process for cluster block grants and delays in following up on external audit report recommendations regarding sub- national PIU and block grant recipients; (ii) differences between the financial report from the accreditation incentive grant recipients and the actual cost incurred; (iii) delays in following up on Bank observations on travel expenditures, submitting the Inspector General’s 2013 intra-audit report and socializing the financial management manual with district education authorities; and (iv) delays in following up on the external audit findings. 39. Despite these challenges, the PCU submitted all the interim un-audited quarterly financial management reports (some delayed), and all the annual audit reports were unqualified (clean) and on time. Furthermore, the agreed action plans that were put in place to address the external audit and the Bank FM review findings have been followed up. In fact, MOEC has resolved almost 100 percent of audit findings for period 2009- 2012 by project closing. The remaining 0.04 percent is still being finalized, and will be covered in the 2013 audit report that is being prepared for submission to the Bank by June 30, 2014. 40. The project disbursed 73.96 percent of loan, 96.6 percent of credit and 97.82 percent trust fund proceeds. There were IDR:US$ exchange rate gains, which increased the funds available particularly in terms of the loan and the trust fund (since they were in US dollars). The Dutch government had expressed its preference to utilize trust fund money first to cover eligible expenditures. Lastly, certain activities were less costly than originally planned. The combination of these factors led to an overall disbursement rate of 93.1 percent, which is a considerable achievement given the financial management challenges that the project faced during implementation. 41. Procurement. The procurement under BERMUTU was limited to the selection of consultants, as it did not finance the purchase of goods, supplies and works. Procurement ratings ranged from moderately unsatisfactory at the onset of the implementation period to moderately satisfactory towards the second half of the implementation period. 42. The recommendations made by the Bank procurement assessment during preparation included: (i) provision of targeted technical assistance through employment 8 It took approximately one year for the FM support and guidelines to be established. Given that Satkers were not familiar with this type of project implementation, documentation/record keeping for eligible expenditures was weak at that time. Record-keeping significantly improved once the necessary guidelines and support were in place. 10 of procurement advisors to enhance capacity for managing procurement in respective MoEC units; (ii) use of a web-based tool to increase public disclosure of procurement process and results; and (iii) enhancement of control of procurement process by putting in place the detailed procurement plan, an operational manual and a reporting mechanism and implementation of the agreed Anti-corruption Action Plan. As with financial management, there were delays at project start-up in incorporating these recommendations specifically in regards to hiring the procurement advisor and establishing the procurement selection committees. 43. By MTR, however, these recommendations were incorporated so that procurement performance was rated satisfactory: MoEC prepared its annual procurement plans and submitted them for Bank review on a timely basis. Issues related to weaknesses in record keeping and hiring of four consultants that were not included in the approved procurement plan led to a downgrading of procurement performance to moderately satisfactory. However, these issues were identified by the Bank’s procurement review and flagged to MoEC. While the record-keeping recommendations were not addressed by the end of the project, overall project procurement was carried out in line with Bank procedures and rated moderately satisfactory at the time of project closing. 44. Anti-corruption Plan. During the preparation phase, agreement was reached with the GoI, that in order to mitigate potential fiduciary risks during the implementation period, the GoI, through the MoEC would develop a comprehensive Anti-Corruption Action Plan (ACAP), satisfactory to the Bank. The ACAP was included as an Annex of the BERMUTU PAD. The different Bank supervision missions rated the implementation of the ACAP as moderately satisfactory because, whilst no systematized complaints were ever handled, the BERMUTU website had limited access to the public for filing a complaint. 45. The project was given a C environmental rating because it did not finance any civil works. It did however trigger Operational Policy 4.10 (Indigenous People) and thus an Isolated Vulnerable Peoples’ Framework (IVPF) was developed during the preparation phase, satisfactory to the Bank. The IVPF aimed at ensuring that both teachers and communities who or which were Isolated Vulnerable People would benefit from BERMUTU, including developing screening and implementation criteria for Grants and a grievance mechanism in accordance with IVPF. 46. A social safeguard specialist was included in various supervision missions attesting to the fact that the IVPF for teachers working in geographically remote and difficult areas was being implemented well by: (i) developing a screening procedure and a set of criteria for selection of teacher working groups, thus ensuring that a significant number of functional working groups in remote areas participated in BERMUTU; (ii) providing modules to strengthen teacher professional development and also ensuring "reconciliation" between teachers with differences in understanding, view, religion and local conditions to teach; and (iii) providing a larger block grant. For example, BERMUTU has successfully trained teachers in remote areas with a significant percentage of indigenous students through 92 primary school teacher-working groups and 11 71 junior secondary teacher-working groups. The grievance mechanism required by the IVPF was not implemented. 47. Early in project implementation, safeguards was rated moderately satisfactory due to the: (i) absence of implementation of the grievance mechanism; and (ii) limited implementation of the IVPF and attention to the social safeguard requirements. Based on recommendations made during the MTR, the PCU hired a social safeguard consultant to help follow up the mid-term review recommendations and compliance with the IVPF moved forward with no grievances reported during project implementation. 2.5 Post-completion Operation/Next Phase 48. BERMUTU is only the first (completed) stage of a longer process needed before expected improvements in student learning outcomes could be observed and measured in a standardized fashion across the country and education levels. The legal framework, processes, instruments, manuals, training materials and modules required to professionalize Indonesia’s teachers have been developed, trialed and are just started to be implemented. The pieces are in place and there exist opportunities to expand and further improve the achievements made under BERMUTU. 49. The teacher reform supported by BERMUTU continues as mandated by the Teacher Law with strong political support. Accordingly, after the closing date, BERMUTU-related policy outcomes are continued to being implemented nation-wide. This stage includes, for example, the accreditation of LPTKs, the RPL, the induction for new teachers, and the teacher performance management (TPM) system designed for the four-level professional career structure progression. 50. BERMUTU’s important achievements have, in turn generated new challenges, some of which are being addressed by the government as it continues the reform process, such as: (i) the fine-tuning and simplification of processes; (ii) the alignment of the regulatory framework to the new 2013 Curriculum under implementation and the 2014 Civil Service Law requiring, inter-alia, all civil servants to enter a performance-based contract; and (iii) the dissemination and expansion of BERMUTU-related interventions to new districts and provinces and to a new “clientele” comprising early-childhood, secondary education (including vocational education) and special education teachers. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 51. Relevance of Objectives. The project development objectives were highly relevant to the country’s sectoral needs at the time of project development. They fit squarely into the government’s objectives to improve quality of education as spelled out by the 2003 Education Law and 2005 Teacher Law, while supporting the Bank’s FY04- 07 CAS Objective II “Making Service Delivery Responsive to the Needs of the Poor”. The objectives continue to fit within the FY09-12 and current Country Partnership 12 Strategy (FY13-15) 9: The FY09-12 continued the support to increase the number of teachers meeting mandated academic qualifications and system accreditation, while the FY13-15 includes as one of its engagement areas in the Pro-Jobs pillar the continued support for enhancing skills and technology. Bank engagement therefore continues to support development results that will improve the performance of teachers, strengthen institutions and service delivery and increase human resource capacity in research and development. 52. Relevance of Design. The project design is judged to be high. It was appropriately aligned with the relevant government and Bank strategies and designed to enhance the quality of the teaching force through upgrading pre- and in-service teacher training as defined by the Teacher Law and on standards defined by the National Education Standards Agency. The PDOs were stated with sufficient operational precision allowing for gauging their attainment through the observation and measurement of a metric comprising three outcome and seven intermediate outcome indicators. Baseline data was included at the onset of project implementation and end-of-project outcome and intermediate targets were deemed achievable at the time of project design. However, the design proved to be more complex than initially assessed, especially for Satkers, which had to become familiar with donor-financed implementation approaches. Although this led to initial delays, the majority of challenges were resolved by MTR when the project was back on track to meeting its PDOs. By project closing, all envisaged key activities were completed. 53. Relevance of Implementation. The relevance of implementation is judged to be substantial. As mentioned previously, all activities were completed by end-of-project and were very successful. In addition to completing all activities, meeting (or exceeding) almost all targets and reaching a combined disbursement rate of 93.1 percent, implementation of the different impact studies provided valuable insights, which were (and remain) highly relevant 10. Despite Satkers’ unfamiliarity with donor-financed implementation approaches, they were able to overcome this challenge to a large extend and help carry out project activities and manage funds. Furthermore, as part of government’s decision to restructure MoEC, it also made a firm commitment to ensure BERMUTU implementation would not be adversely affected. Finally, all audit reports were submitted and received a qualified (clean) opinion and procurement was carried out in accordance with Bank guidelines. There were, however, challenges relate to initial capacity of Satkers, DIPA allocations,procurement record keeping, and financial management but these challenges did not undermine overall project implementation and relevance. 9 Report No. 752906-ID, December 13, 2012 10 Findings on teacher working group effectiveness showed increased support for teachers in terms of training and professional development activities. Findings on improved cognitive and pedagogical ability of teachers as a result of the teacher working groups is mixed but has laid the foundation for further, more rigorous analysis. Lastly, teacher absenteeism in BERMUTU districts has dropped from 19 percent in 2003 to 6.6 percent in primary schools and 6.3 percent in junior secondary schools. 13 3.2 Achievement of Project Development Objectives 54. The ICR rates the achievement of the PDOs as satisfactory for the reasons explained below and documented in the ICR Data Sheet: 55. With respect to the first outcome indicator (Increased number of teachers meeting academic qualifications mandated by the Teacher Law), the end-of-project target of 1.4 million teachers was exceeded by 22.4 percent. BERMUTU developed some instruments supporting the effort by which over 1.71 million teachers acquired the mandated four- year college degree (S1 certificate) by the closing date of December 31, 2013. In addition, the government has decided to fine tune the existing distance learning program and the RPL to further enhance upgrading the remaining one million teachers, some of them working in hard-to-reach locations, to meet the 2.7 million 2015 target. 56. With respect to the second outcome indicator (Increased number of primary and junior secondary teachers in BERMUTU Districts using classroom instruction with specific subject and age appropriate pedagogy), the end-of-project target of 190,000 teachers was exceeded by 12.8 percent. BERMUTU was the instrument by which over 214,000 teachers benefited from the professional working group (KKG/MGMP) activities for the adoption of improved instructional practice. Findings of a study commissioned to assess the impact of this indicator reveals that teacher’s participation in these working groups improved their pedagogical skills as indicated by their abilities in teaching preparation, delivery and formative assessment of their students, as compared to equivalent teachers that did not participate in similar groups. MoEC has notified that it would conduct additional performance appraisals in 2015 on a random sample of BERMUTU teachers using the Teacher Annual Performance Appraisal (PKG) instrument developed and validated under Component 3 in 2013. At the time of the ICR mission, the PKG was being fine-tuned to align it with the new 2013 Curriculum (under implementation) and the 2014 Civil Service Law. The MoEC is expected to carry out a nation-wide administration of the revised PKG in January 2016. In the meantime, some districts, such as Karawang, are implementing the current non-revised PKG. 57. With respect to the third outcome indicator (Reduction in absenteeism of teachers in BERMUTU Districts), the target was achieved: two studies conducted by Puslitjak (MoEC) in BERMUTU districts found that absenteeism rates decreased from 8.4 percent in 2010 to 6.6 percent in 2012 in primary school and from 12.8 percent to 6.3 percent in junior secondary school. The SMERU study (funded by the Bank) on a national representative random sample of public primary schools in 2003 and 2008 found the teacher absenteeism rates to be 19 and 14 percent respectively. Another national representative study by Puslitjak in collaboration with the Analytical and Capacity Development Partnership (AC DP , ADB program jointly funded by the European Commission and the AusAid) is underway. 58. In addition to exceeding PDO-level indicator targets, the project contributed to the following long-term activities: (i) changing the way basic education teachers-to-be are prepared (LPTK); (ii) improvement in the curriculum-subject knowledge and teaching 14 skills of all in-service teachers (KKG; MGMP); (iii) setting up standards (Teacher Performance Management - TPM scheme) and targets to manage the teaching labor force; (iv) accompanied these changes with financial incentives to improve teachers' welfare based on teachers’ performance results (TPM scheme designed for the professional career allowance including all the assessment instruments). Furthermore, BERMUTU focused on basic education teachers, principals and supervisors from disadvantaged districts first as a feasible way to get the process started in a more equitable fashion. Therefore, the project is judged to have achieved its stated PDOs of improving the overall quality and performance of the teachers in the Republic of Indonesia through enhancing their knowledge of subject matter and pedagogical skills in the classroom. 59. Preliminary findings of three studies commissioned by MoEC that support the ICR’s PDO ratings can be found in Annex 2. 60. With respect to the intermediate outcome indicators: (i) Indicator 1 (Increased number of pre-service programs accredited), the end-of-project target of 75 was exceeded by 8 percent; (ii) Indicator 2: (Increased number of new graduates from accredited programs meeting the mandated competency standards) reached 13,179 graduates, missing the 15,000 graduates end-target due to the sequenced nature of the activity and three to four year period allowed for PGSD’s to become accredited; (iii) Indicator 3 (Increased numbers of teachers gaining recognition for prior learning (RPL) from accredited universities), the end-of-project target of 700,000 was exceeded by 25.7 percent, reaching almost 880,000 teachers; (iv) Indicator 4 (Increased numbers of active KKG/MGMPs in BERMUTU Districts), the end-of-project target of 4,500 was exceeded by 35.7 percent (there are 6,135 active professional working groups); (v) Indicator 5 (Increased numbers of beginning teachers being provided with school induction programs in BERMUTU Districts), reached 1,429 new teachers, missing the end-target of 3,000 but all mechanisms are in place to complete induction for all remaining teachers in both BERMUTU and non-BERMUTU districts; (vi) Indicator 6 (Revised policies, plans and procedures for continuing education and career development of certified teachers developed and piloted for implementation by the government in 2011) was met; and (vii) Indicator 7 (A teacher database established to track progress in teacher placement, academic qualifications, certification, and payment of professional allowances) was also met. 3.3 Efficiency 61. The BERMUTU program was efficient from both an operations and expenditure perspective. A good indication of its overall efficiency is the fact that the project achieved all of the PDO and intermediate outcome indicators with less financial resources than the original projection at the appraisal stage. At the same time the project had commendable disbursement of 93.1 percent of project loan, credit and grant funds. The bulk of these funds (US$91.46 million or 66.3 percent) was allocated for component 2, “Strengthening structures for teacher improvement at the local level”. This component reached a 95.6 percent expenditure rate with respect to the appraised estimates (as shown 15 in Annex 2). 62. The emphasis on component 2 is justifiable given the significant size of the target groups and its contribution in the form of improvements in teacher pedagogical skills. The latter is achieved through project support for the working group program of primary and junior secondary teachers (KKG and MGMP) in 75 participating districts. This program benefited 214,380 primary and junior secondary school teachers, principals and supervisors from 6,107 working groups. 63. While teacher working groups existed prior to BERMUTU, many were inactive or not operating effectively. The project provided a stimulus to revive and strengthen more than 6,000 KKG/MGMP by introducing a well-structured program. There were four important characteristics of BERMUTU support: (i) provision of grants for each KKG/MGMP for three consecutive years to ensure sufficient time for the working groups to become well-established and fully functional; (ii) development of a strong support system to ensure quality and sustainability (as described in Annex 2); (iii) formation of an M&E system; and (iv) a special model of support for teachers in difficult and remote areas. BERMUTU granted each of the working groups a total US$4,800 for KKG and US$6,300 for MGMP, for three consecutive years. While some non-BERMUTU working groups also received support, it tended to be much less comprehensive. For instance, there was a separate grant program to non-BERMUTU KKG/MGMP in 2007-2009 that provided US$1,000 to KKG and US$1,500 to MGMP per year with no guarantee for continuous funding for the following year and without a support system as provided under BERMUTU. 64. BERMUTU also introduced a different focus of grant spending. A survey conducted in 2007 of the non-BERMUTU grant program showed that most of the expenditure was on meals (31 percent), transportation (22 percent), and honorarium (20 percent); very little was spent on items that would support the session activities (e.g. supplies to create learning aids, bringing in expert resource personnel and classroom action research). The BERMUTU grant was specifically designed to ensure the funds would go mainly toward supporting high quality session activities and could not be used for meals and teacher transport (with the exception of teachers in remote KKG/MGMP), In this regard, BERMUTU improved efficiency of the system by focusing the use of grant financing to enhance the quality of working group activities. 65. While the cost of supporting teacher working groups through BERMUTU was more expensive, the project was arguably more cost-effective given that the BERMUTU- supported teacher working groups met more frequently, operated more effectively, and were more focused on improving teaching and learning. KKG/MGMP program became an integral part of the performance management system developed under component 3. As a result, the KKG/MGMP program will be sustainable as it is the most accessible option for teachers to do the required Continuing Professional Development (CPD). An impact evaluation study commissioned by MoEC examined a sample of KKG/MGMP in 20 BERMUTU districts and another 20 non-BERMUTU districts. The study found that working groups under BERMUTU were more effective, better organized, held more 16 accountable and that activities involved more participation from the individual teachers compared to the non-BERMUTU counterparts. In addition, BERMUTU also had an interesting spillover effect to non-BERMUTU groups through the adoption BERMUTU modules and guidelines. The widespread use of BERMUTU modules and guidelines among non-BERMUTU working groups provides additional evidence of BERMUTU impact on overall system efficiency. 66. The study also found a positive impact of BERMUTU on teachers’ competencies. Teachers in working groups receiving a direct grant under the BERMUTU program were tested at the baseline (2010) and midline (2011) of a three-stage study and showed a significant improvement in cognitive and pedagogical ability. 67. Component 2 also financed the development of a system to recognize teachers’ working experience and prior learning (RPL). In collaboration with selected teacher training colleges (LPTK), MoEC designed the application of the RPL system for programs tailored for teachers who have to upgrade to the 4-year college degree. This RPL has benefited 879,832 teachers by reducing the number of credits they had to take in order to earn an S1 degree. With the RPL system, in-service teachers can complete their study in a shorter period and they do not have to leave their teaching responsibilities. In addition to support for the accreditation of LPTK, BERMUTU, under component 1, also provided support for the development of distance learning materials for upgrading channel (HY-LITE) organized by the Consortium of LPTKs. This program, along with the distance learning by the Open University, have contributed to improved academic qualifications of 879,832 teachers, thus to the achievement of the first PDO indicator, “Increased number of teachers meeting academic qualifications mandated by the Teacher Law”. The distance learning channel was the most accessible route for incumbent teachers to upgrade their academic credentials. This upgrading channel allows teacher to fulfill their classroom tasks while pursuing their four-year college degree. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory 68. The relevance of project design was high as it was appropriately aligned with the relevant government and Bank strategies and designed to enhance the quality of the teaching force through upgrading pre- and in-service teacher training as defined by the 2005 Teacher Law and on standards defined by the National Education Standards Agency. Furthermore, the PDOs were stated with sufficient operational precision allowing for gauging their attainment through the observation and measurement of a metric comprising three outcome and seven intermediate outcome indicators. The design also recognized the potential complexities associated with project implementation and included appropriate mitigation measure. The relevance of implementation is judged to be high since all key activities were successfully completed by end-of-project, meeting (or exceeding) almost all targets and reaching a combined disbursement rate of 93.1 percent. Furthermore, implementation of the different impact studies provided valuable insights for decision-making purposes and which were (and remain) highly relevant. Implementation challenges, especially those that arose during the initial phases of the life 17 of the project, was addressed so that the pace picked up and progress towards PDO achievement was on track by MTR. Efficacy is rated satisfactory as the PDOs were achieved as supported by exceeding two of the outcome indicators and meeting the third indicator. Specifically, the project supported improved quality of teachers by supporting 1,713,163 teachers meeting the mandated qualifications, improved teacher performance as a result of their active participation in working groups and a significant drop in primary school teacher absenteeism from 8.4 percent in 2010 to 6.6 percent in 2012 and junior secondary school teacher absenteeism from 12.8 percent in 2010 to 6.3 percent in 2012. These are substantial achievements especially in the context of an ambition government reform. Project Achievement of PDO Efficiency Overall Rating Relevance (Efficacy) High Satisfactory Substantial Satisfactory 3.5 Overarching Themes, Other Outcomes and Impacts 69. Poverty Impacts, Gender Aspects, and Social Development. BERMUTU did not include gender-specific interventions targeting instead basic education teachers, principals and supervisors in 75 districts regardless of gender. BERMUTU was implemented in 75 poor and educational disadvantaged districts providing special attention to teachers working in rural and hard to reach schools by: (i) introducing a different design for the teacher working group grant (KKG/MGMP-remote); and (ii) providing the distance learning modality to those teachers that were in their upgrading process in order for them to meet the required academic qualifications. BERMUTU targeted districts with the highest numbers of teachers who were not meeting the required academic qualifications. 70. Institutional Change/Strengthening. The major institutional accomplishment of BERMUTU is the establishment of a regulatory framework and an assortment of instruments and processes to implement the Teacher Law. As per Decree of the Ministry of State Apparatus and Bureaucratic Reform (MenPAN &RB) 16/2009 on Functional Credits for Teachers, and five MOEC’s regulations on recognition of prior learning, new teacher induction, and teacher credit points are in place. The TPM scheme including manuals and instruments were developed, trialed and implemented towards the end of the BERMUTU period and it is currently under review to align it to the 2013 Curriculum (under implementation) and the Civil Service Law of January 2014. Accordingly, the nation-wide implementation of the revised TPM has been scheduled to start in January 2016. There is some initial capacity buildup at the districts to manage large amounts of funds and continue participating in the government’s continued roll out of the Teacher Law, using many of the mechanisms and approaches put in place by the project. Other Unintended Outcomes and Impacts (positive or negative). None. 18 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable. 4. Assessment of Risks to Development Outcome 71. There is a low to negligible risk for changes to occur that might be detrimental to the achieved PDOs because there is ample evidence of the financial, institutional and technical sustainability of the key activities financed under BERMUTU. As mentioned before, the BERMUTU programs are continued to be implemented by the GoI and there are some programs financed by other donors, such as the school principal and supervisor quality improvement funded by the Australian Department of Foreign and Trade (DFAT) grant 72. With respect to the financial sustainability, the district governments and teachers are automatically bound to implement the regulations under Component 3 of BERMUTU as these are to be implemented nationally. As BERMUTU is an integral part of the continuous teacher reform, its annual budget is guaranteed, as there are units within the MoEC officially mandated to manage the teacher quality and performance as stipulated by the Teacher Law, including implementing the regulations and policies developed under BERMUTU’s financial and technical support. As an early indication, the ICR team was informed that: (i) the Directorate of Education Personnel Management (P2TK Dikdas) of DGBE allocated funds in 2014 to disseminate and expand the Teacher Working Group Grant program to 50 districts; and (ii) 40 out of the 75 BERMUTU districts have signed Memorandums of Understanding with the MoEC to continue the BERMUTU-related programs, including providing teacher working group grants in their districts using their own funds. 73. With respect to the institutional sustainability, BERMUTU-related activities continue to be housed within the different units of the MoEC at the national and district level and supported by an existing regulatory framework and instruments. Some districts are enacting their own regulations to complement and synergize MoEC’s current BERMUTU undertaking. With respect to the technical sustainability, the internal capacity built during BERMUTU’s implementation period at both the national, provincial and district level mainstream institutions in terms of trained staff and procedures. This is expected to remain in place and expand nation-wide as this mandated by the Teacher Law and would be reflected in the MOEC Strategic Plan 2015-2019. 74. There is however a moderate risk to dilute the TPM associated with the teacher professional career and transforms it to an entitlement for every certified teacher regardless of the relevance and quality of the actions taken by the teachers to access credits in the performance progression scheme. The continuing administration of appropriate competency assessment instruments by third qualifying parties could significantly mitigate this perceived risk during the BERMUTU dissemination and expansion process, currently under implementation. 19 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 75. The preparation period started in 2005 by the Bank commissioning analytical work in the various areas (already mentioned in section 2.1) financed by the Dutch and AusAID TFs (managed by the Bank). It was informed, amongst others, by the Bank’s analytical work which contributed to the development of the 2005 Teacher Law, the FY04-07 CAS and responded fully to the government’s request for assistance in teacher management and education quality improvement. There were 4 field missions from identification in August 2006 to appraisal in April 2007 including a preparation mission in November 2006 and a pre-appraisal mission in February 2007. 76. Only one task team leader (TTL) and one Co-TTL were responsible for the entire project preparation period. The preparation period also benefited from three quality review meetings, of which one was a QER, carried out in September 2006. As mentioned in Section 2.1., the QER panel endorsed the project design while mainly recommending: (i) a simplified project design; (ii) further clarification on: (a) the management of implementation and sequencing of activities in order to get clearer understanding on which institutions were accountable for what and to whom; (b) risks and mitigating measures; (c) the accreditation and certification standards being proposed; and (d) the competitive and targeted grants for teacher training institutions and the criteria and procedures to be used; and (iii) the inclusion of baseline data. All of these recommendations were accommodated into the project design. The Bank’s regional operational service department praised the project’s M&E framework as a good practice example that was being used in Bank training programs. The Bank also worked with government to ensure project implementation capacity building of the Satker would be carried out through training and technical assistance. (b) Quality of Supervision Rating: Satisfactory 77. There was only one TTL during the entire implementation period (different from the one during the preparation period), whose tenure covered the period March 2008 (immediately after effectiveness) until December 2013 (closing date). 78. This operation benefited from a continuous on-sight and close communication from the TTL, World Bank staff in the Jakarta office and technical experts (consultants) with the key counterpart players on an ongoing basis. This helped resolve budget administration issues, provided timely technical inputs on the development of policies and operational instruments (including several impact evaluation studies), and address Satkers’ capacity constraints/quality of technical assistance. Specifically, the Bank team: (i) advised MoEC in 2008 to focus on three Satkers first when GoI was only able to allocated US$6.9 million compared to the expected US$21 million; (ii) proactively 20 facilitated the restructuring of the project in response to MoEC organizational changes and more realistic project needs; and (iii) encouraged MoEC to fully follow up on audit and FM recommendations, which were almost completely resolved. The external audit report due June 30, 2014 is expected to confirm complete resolution of all previously- made recommendations. The project files attest to the carrying out of 9 field supervision missions during the entire implementation period. A total of 11 ISRs were filed in the Bank system, implying that in the 71 month span implementation period these reporting documents were updated, on average, twice a year. 79. Overall, the supervision missions included the appropriate skill-mix to carry out their envisaged terms of reference as shown in Annex 4 of the ICR, including procurement, financial management and the social safeguards. The findings of each supervision mission were conveyed to Bank management and to the GoI clearly and straightforward, including accomplishments and implementation shortcomings, potential risks and implementation flags. The Bank team intensified its support during the project’s initial phase to address the challenges that began to surface, and actively contributed to their resolution by MTR. This allowed the team to then shift its focus to using the preliminary findings of the various impact evaluation studies in policy dialogues with MoEC. Although financial management was rated moderately unsatisfactory and procurement moderately satisfactory (see Section 2.4. for details), it was meant to send a strong message to the government while recognizing the considerable efforts made the project management unit, which was rated satisfactory. 80. Weighing the many supervision strengths with one detected shortcoming, the ICR rates the Bank supervision performance as satisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory 81. The ICR team rates the overall Bank performance as satisfactory because the rating at the preparation stage was rated satisfactory and during implementation as satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 82. GoI and MoEC displayed full ownership of the project design. This ownership will be again reflected by the full commitment, support and engagement provided by the MoEC in: (i) addressing the challenges during the first two years of the implementation period; and (ii) then, supporting all the BERMUTU-related processes and studies in order to attain the stated PDOs mid-implementation period onwards. As mentioned before, the “Readiness Filter Agreement” reached between the Bank and the GoI/MoEC in the early 2000s requiring completion of a POM and funding arrangements as a condition for negotiations and applicable to all projects in Indonesia further highlighted the government’s commitment to the project’s objectives. This commitment is confirmed by 21 the 2014 Civil Service Law which includes, inter-alia, the requirement that all public teachers (as civil servants) enter a performance-based contract with the MoEC (with observable and measurable teaching performance an professional progression credit targets) including incentives (monetary) if these targets are met or penalties if they are not. 83. Challenges facing MoEC during 2008-2010 included: (i) unpredictability of education funding from the national budget (DIPA); and (ii) delays in MoEC timely releasing BERMUTU approved budget funds. To partially address these issues, the Director of Teacher Development and Training under the then director general in his capacity as the PCU Manager, was able to secure some funds in the DIPA of his Directorate to recruit technical assistance needed by certain Satkers, like those working on teacher training accreditation, development of new teacher induction and principal and supervisor learning materials. In addition, the PCU, jointly with the Bank team, diligently alerted the respective Satkers to support them in preparing for the next DIPA, resulting in a much improved situation starting 2010. 84. Rating: Given the above-mentioned strengths and challenges, and the fact that the latter were satisfactorily resolved mid-implementation period onwards, the ICR rates the GoI performance satisfactory. (b) Implementing Agency or Agencies Performance Rating: Moderately satisfactory 85. MoEC was entrusted with the overall responsibility of implementing and managing BERMUTU. The Directorate General of the Quality Improvement of Teachers and Education Personnel (DGQITEP) became the BERMUTU’s PCU until the end of 2010. The proposed implementation arrangements were ambitious involving 8 central and 20 district project implementation units with different levels of institutional capacity. In addition, in early 2011, the ministry was restructured as previously mentioned, which required remapping of departments and budget allocations, and leading to some delays in the short-term. 86. Implementation strengths: (i) almost all challenges during project start-up were satisfactorily addressed by MTR; (ii) despite these initial challenges, the project was on track to achieving its PDOs by MTR; (iii) PCU’s attention to addressing those initial challenges allowed it to focus on results and actively engage with the Bank on the findings of the various impact evaluations; (iv) social safeguards and adherence to the RVPF is judged satisfactory; and (v) the PCU’s continued efforts to resolve the external audit and Bank FM review findings even beyond the project implementation period (the external audit due June 30, 2014 is expected to confirm that all findings have been addressed). In terms of ownership and commitment to project implementation, it is important to highlight efforts made by the PCU in providing the necessary technical assistance during the early stages, resulting in more reliance on mainstreamed institution staff and less dependency on technical assistance. Furthermore, the PCU (along with the Pusbangprodik Satker) was able to display some initial, capacity to undertake: (i) with 22 district support, on-sight monitoring of BERMUTU’s activities especially the ones concerning the working groups of teachers, principals and supervisors; (ii) the fine-tuning of some of BERMUTU’s processes and instruments; and (iii) the following up on the several commissioned evaluation studies including the analysis of their preliminary findings. 87. Implementation weaknesses: (i) delays in fully addressing the previously- mentioned start-up challenges, which affected physical and financial project execution; (ii) delays in fully addressing all audit and FM findings prior to project closing despite continued follow-up by the Bank team; (iii) weaknesses in procurement filing; and (iv) the time it took to complete the ministry restructuring, leading to slowed implementation during 2011, picking up again in 2012 once the restructuring was completed. 88. Weighing the strengths with the weaknesses described above, the ICR rates the overall implementing agency performance as moderately satisfactory. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately satisfactory 89. Despite the fact that the achievement of the PDOs was rated satisfactory, all outcome and most of the intermediate outcome indicators were achieved or exceeded, all BERMUTU sub-components implemented and documented disbursements reached 88.9 percent of the total financing (93.1 percent of the total international financing), the ICR rates the overall Borrower performance as moderately satisfactory because the ratings of the borrower performance at the preparation was rated satisfactory and the government and implementing agency performances at the implementation stage were both rated moderately satisfactory. 6. Lessons Learned 90. Improving teaching quality and performance requires an articulated, focused and holistic approach, just like the one included in BERMUTU. The holistic approach of the BERMUTU design stems from including interventions supporting: (i) changes in the way basic education teachers-to-be are prepared (LPTK); (ii) improvements in the curriculum-subject knowledge and teaching skills of all practicing teachers (KKG; MGMP); (iii) the appropriate managing of teachers by setting targets and standards (TPM scheme); (iv) financial incentives and improving teachers welfare based on teachers’ performance results (TPM scheme designed for the professional career allowance including all the assessment instruments); and (v) changes in the recruitment procedures and regulations to only employ the best candidates and set standards for their performance. BERMUTU focused on basic education teachers, principals and supervisors from disadvantaged districts first as a feasible way to get the process started. With the experienced gained during the period 2008-2013, the GoI/MoEC is now in a position to disseminate and extend BERMUTU to other districts and provinces including more hard- to-reach communities and to a “new clientele of teachers”. All the 12 sub-components of the BERMUTU design nicely articulated between themselves generating a welcoming 23 synergy that contributed to the attainment of the PDOs. 91. MoEC being in the driver seat owning and implementing the program without the need of parallel structures greatly contributed to the satisfactory achievement of the stated PDOs and ensured technical, institutional and financial sustainability of BERMUTU’s key interventions after the closing date. Ensuring and maintaining “client ownership” from the early stages is a key to success. This empowerment has been down streamed to the provinces, districts and cluster of working groups showing cohesion and initiative. 92. BERMUTU-follow up programs currently implemented by MoEC and some districts, largely navigating in new and unchartered territory would greatly benefit from a continued technical support from international and multilateral donors providing sound technical advice and linkages with worldwide best practices. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/Implementing Agencies. No issues were raised. Please see Annex 7 for an excerpt of the Borrower’s ICR. (b) Co-financiers, the Royal Netherlands Government No issues were raised. Please see Annex 8 for the Dutch government’s contribution to the ICR. (c) Other partners and stakeholders. Not applicable. 24 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Actual / Latest Percentage of Components Estimate Estimate (US$ Appraisal (US$ million) million) Component 1 20.45 15.08 73.7 Component 2 134.20 127.84 95.3 Component 3 5.28 8.37 158.5 Component 4 33.83 21.98 65.0 Unallocated 1.24 0.0 NA Fee 0.06 0.06 100.0 Total financing 195.06 173.34 88.9 Note: Amounts shown by component are inclusive of counterpart funding. (b) Financing Appraisal Actual / Latest Type of Co- Percentage of Source of Funds Estimate Estimate financing Appraisal (US$ million) (US$ million) IBRD Loan 24.50 18.12 73.9 IDA Credit 61.50 59.42 96.6 TF Grant 52.00 50.87 97.8 Inter. financing 138.00 128.41 93.1 Government National Counterpart 39.00 26.62 68.3 Government Counterpart 18.06 18.31 101.4 District Total GoI Counterpart 57.06 44.93 78.7 Total 195.06 173.34 88.8 25 Annex 2. Outputs by Component 93. BERMUTU aimed at supporting GoI/MoEC's core agenda concerning the implementation of the Teacher Law to improve teacher professionalism, performance, accountability and welfare. Specifically, the Teacher Law aimed at: (i) improving teachers’ welfare by doubling their salary contingent on compliance with the criteria set for in the Law 11; and (ii) ensuring that teachers participating in BERMUTU certified their four year pre-service teacher training, improved their content knowledge and pedagogy competencies as well as their personal and social teaching-related skills and participated in further continuous professional development. 94. The BERMUTU goals, as reflected in the stated PDOs, were not only achieved but also exceeded as shown by the corresponding outcome and intermediate outcome indicators in the ICR Data Sheet and fully detailed in section 3.2. Accordingly, MoEC has successfully put in place the regulatory framework and assortment of instruments for teacher certification, teacher performance appraisal and incentives as well as career progression. The expectation is that this new system will play a key role in ensuring that all Indonesian students are taught by highly qualified teachers and thus, learning outcomes will also likely improve. 95. Despite these commendable achievements, BERMUTU - a first, completed, stage of a longer process to improve the quality of basic education in Indonesia – has, in turn, generated new challenges that need to be addressed in the second post-BERMUTU (currently under implementation) and further future stages. Some of these challenges are summarized in the section of lessons learned. 96. Teacher reforms take a long time to mature as reflected mainly by the most important outcome of a reform – improved students’ learning outcomes as compared to a baseline. BERMUTU’s short life-span of about 78 months since effectiveness has not yet allowed for the: (i) coverage of the entire universe of basic and secondary teachers in Indonesia; and (ii) gathering of hard evidence through standardized testing of changes in students’ learning outcomes from a random sample of teachers benefited by BERMUTU as contrasted with a comparable set of students from random teachers not benefited by BERMUTU and a baseline. This is a big challenge for the MoEC given that there are 2.7 million teachers working in a very diverse society, one million of which had not yet benefited from some of BERMUTU’s interventions. Whilst a very sound start has begun, the waves of change will take some time to impact on every teacher in the system and will take yet more government funding and extension of the impact from 75 districts to the current existing 515 districts. 11 The Teacher Law requires teachers, including primary school teachers, to have a bachelor (S1) degree/Diploma 4 certificate. Accordingly, the teacher training universities have to upgrade their Diploma 2 program for primary school teachers to become S1 PGSD and get them accredited.” 26 97. This section describes the outputs of the four components. The assessment is primarily based on the Aide-Memoires, ISRs and the April 2014 ICR mission which included visits to project sites in the BERMUTU District of Solok and held discussions with the BERMUTU Director, the PCU Director, the Coordinators of the 4 components and the Bank fiduciary and social safeguards team. 98. Two cost tables are presented to substantiate this Annex: (a) the documented expenditure by component and by source of financing at appraisal of BERMUTU and at the time of writing this ICR (table A2.1); and (b) the documented expenditure by category and by source of financing at appraisal and at the time of writing of this ICR (table A2.2). 99. Component 1 – Reforming University-based Teacher Education. This component aimed at: (i) supporting teacher quality through institutional accreditation to ensure that teacher training reflected international best practices; and (ii) expanding existing teachers’ access to upgrading opportunities. This component included the following 3 sub- components: a. Sub-component 1.1: Providing assistance to the National Board of Accreditation for Higher Education (BAN-PT) to: (i) facilitate the development of accreditation instruments and a strategic plan for the accreditation process; and (ii) provide training, workshops and support for incremental operating costs associated with the accreditation of teachers training providers; b. Sub-component 1.2: Grants and scholarships providing: (i) technical assistance to the Directorate General of Higher Education (DIKTI); and (ii) accreditation incentive grants (AIG) to selected public and private universities and teacher training institutes (about 40 out of 303 in the country) to improve curriculum and teacher preparation programs in line with international best practices and teacher accreditation standards. Three types of AIG were included under this sub- component: i. Tier A grants targeted to large teacher education institutions offering many good-quality study programs covering primary, junior, and senior secondary teacher programs; ii. Tier B grants targeted to small or medium-sized teacher education institutions offering a limited number of good-quality study programs with at least primary and junior secondary programs; and iii. Distance Learning Development Grants opened to the 10 Teacher Training Institutions (LPTK) mandated by the DIKTI to offer in- service distance education to teachers. c. Sub-component 1.3: Overseas scholarship and non-degree training program to improve the quality of university teaching and coursework providing about 30 PhD scholarships and about 90 three-month overseas, non-degree training opportunities to teachers who were assigned to teach and carry out research in the new D4/S1 Primary programs. Eligibility and selection criteria were to be 27 detailed in the project operation manual. 100. This component estimated at appraisal to cost US$20.45 million (10.5 percent of the total cost), 14.2 percent to be financed by IBRD, 50.3 percent by IDA, 16 percent by the TF090794 and 19.5 percent by the GoI, ended up documenting expenditures amounting to US$15.08 million, 73.7 percent of the total appraised amount. 101. The significant under-spending is mainly explained by two factors. First, there were some gains due to the exchange rates fluctuation. The appraised cost of this component was estimated at a US dollar equivalent to 9,000 IDR/US$. During the implementation period the IDR exchange rate fluctuated up and down reaching a high of about 12,400 in early 2009 and about 12,100 during the second semester of 2013 to a low of about 8,500 in 2011. Second, sub-component 1.3 was appraised for 30 candidates while the real number was only 19. 102. This Component supported: (i) the establishment of the S1-PGSD (four-year education of primary school teachers which used to be a two-year program until 2006) in participating pre-service teacher education universities/faculties (LPTKs) through the issuing of Accreditation Incentive Grants; (ii) the continuing timely accreditation by the National Board of Accreditation for Higher Education (BAN-PT); and (iii) the strengthening of the distance learning programs for teachers. 103. The Accreditation Incentive Grant was competitively provided on a performance- based fashion (tranches were released as per attainment of agreed intermediate targets), to 26 teacher training universities to support their efforts to get accredited status for their newly established S1-PGSD study programs and to improve the accreditation status of up to two other priority study programs for secondary school teachers. 104. This component supported the development and validation of accreditation instruments and procedures to measure institutional baselines scores and changes towards the required accreditation scores. From zero S1-PGSD institutions accredited in 2006, BERMUTU supported the accreditation of 81. It also supported BAN-PT’s staff (accreditation assessors) capacity improvements through immersion programs in similar institutions in United States, India, Japan and New Zealand. 105. In order to facilitate teachers getting their bachelor degree as mandated by the new Teacher’s Law, this Component strengthened distance learning programs for in-service teachers through different options; (i) the HYLITE (organized by members of public LPTK Consortium); (ii) the teacher learning portal; and (iii) the Guru Pintar On-line (organized by the Open University) providing video-streaming to improve their subject knowledge and teaching skills. 106. This Component provided three-month overseas, non-degree training scholarships to 90 LPTK’s teachers for them to enhance their teaching and research in the new D4/S1 Primary programs. In addition, 20 lecturers in the primary education teacher program (S1- PGSD) of teacher education universities (LPTKs) were provided scholarships to get their doctoral degree in basic education in various universities in the United States, Australia, 28 Netherlands and the People Republic of China in the period 2008-2012. These participants were selected competitively based on agreed criteria. To date, one candidate dropped out of the program, while only 3 participants have successfully completed their study (1 from USA and 2 from Australia) and have returned to Indonesia. Problems associated with getting the appropriate visas on time for 10 students explain why the original target of 30 was not met. 107. The shortcoming with this component is that most of the degree scholarship (PhD) recipients had not completed their studies and/or obtained their envisaged PhD degree by December 31, 2013. This is mainly because the scholarship recipients did not start their studies at the same time, but in three batches: 2009, 2010 and 2012. MoEC is currently financing the cost associated with allowing the remaining 16 candidates complete their degrees. 108. Rating of Component 1. Weighing the above achievements with the one shortcoming, the ICR rates the implementation of this component as satisfactory. 109. Component 2 – Strengthening Structure of Teacher Improvement at Local Level. This component aimed at supporting initiatives taken by the MoEC to improve the quality of teachers through the implementation of teacher classroom assessment and professional development elements of the certification process. This component included the following four sub-components implemented in 75 of Indonesia’s 441 districts 12 in 16 of Indonesia’s 34 provinces: a. Sub-component 2.1: Providing technical assistance to establish agreed mechanisms for RPL of teachers, including the development of an eligibility examination; b. Sub-component 2.2: Providing technical assistance to the Directorate General for Quality Improvement of Teacher and Education Personnel (PMPTK) to develop subject-based modules and modules on management, which would be available in teacher training facilities (P4TK) to support the learning activities of teachers, core-teachers, and principals at the school cluster level. This sub-component was to be carried out in parallel with the AusAID-funded Basic Education Project for grades 1 to 9, providing capacity building support to P4TKs; c. Sub-component 2.3: Teacher Working Groups (KKG), a window aimed at financing a range of teacher training activities, including teacher classroom inter- visitation, use of the centrally developed training modules and other innovative approaches; and 12 The BERMUTU provinces and districts were selected according to the following criteria: (i) percentage of unqualified basic education teachers; (ii) percentage of basic education teachers working in rural, remote and hard to access schools; (iii) areas belonging to the lower-income decile (a proxy for poverty); (iv) percentage of the total provincial (district) budget spent on education in the province (district); and (v) districts with the lowest score of the Indonesian Education Development Index (a function of net enrollemnt rates and outcomes of the national (non-standardized) tests). Currently there are about 515 districts in the country. 29 d. Sub-component 2.4: School Principal Working Groups (KKKS/MKKS) and Supervisor Working Groups (KKPS/MKPS), a window aimed at financing school principals and supervisors to enable them to: (i) establish school-based teacher induction programs for beginning teachers; (ii) be trained in assessing the performance of classroom teachers; (iii) implement the teacher induction program; and (iv) implement a performance assessment reporting process by the principal at the end of the probationary year. 110. This component estimated at appraisal to cost US$134.2 million (68.8 percent of the total cost), 12.4 percent to be financed by the IBRD, 30 percent by IDA, 25 percent by the TF and 32.6 percent by the GoI, ended up documenting expenditures amounting to US$127.84 million, 95.3 percent of the total appraised amount, a commendable disbursement achievement. 111. This component supported the strengthening of the regulatory framework-governing teachers. As a consequence, a regulation for Recognition of Prior Learning (RPL) is now in place and a substantial number of teachers have benefited through teacher training universities (LPTK) and the Open University (UT) as further detailed below. The component financed the development of guidelines and instruments for the carrying out of RPL. 112. In addition, this component supported the development of regulations on induction and continuing professional development along developing corresponding learning packages (12 induction modules), which are currently also in place benefitting teachers participating in about 6,107 working groups. It also developed manuals for masters and key teachers for their facilitation of these working groups. 113. Several field observations in the second half of the implementation period conducted by MoEC’s M&E team as well as the social safeguard consultant working noted that: (i) classroom processes had started to reflect teachers’ knowledge, both in the curriculum subject content as well as in the pedagogical process to deliver this content, gained in the working groups’ activities; and (ii) teachers have started to migrate from spending most of their working group activities completing the reports/deliverables to fully comprehend the learning materials and synergizing this acquired learning with the schools’ principals and supervisors’ working groups. While quantitative data showed that teachers benefited from participating in the working groups, especially the opportunity to participate in learning activities, which is not common, to improve their content knowledge and pedagogical skills. 114. This component has been instrumental in granting RPL to 879,832 teachers pursuing their S1 degree (four-year college), 87,501 of them in LPTKs institutions, and the rest, 792,331, through the Open University thus exceeding the end-of-project target of 700,000. Even though RPL does not reduce the period to the S1 degree, it does reduce the load of studies and, more important, prevents teachers of abandoning their classroom task while pursuing their degree. This RPL option is being currently utilized by the MoEC to reach about 2.7 million teachers to acquire their mandated S1 college degree. The ICR team was informed that the RPL option would cease to operate in 2016, assuming that all of the 30 existing teachers will have been upgraded to the required minimum 4 years training. 115. In August 2013 a joint Bank and MoEC team visited two universities in Bandung to review the progress of the implementation of RPL and the quality of the procedures used in determining RPL. The team found that the sampled universities had adopted the processes and procedures for determining RPL as recommended in the Regulation. 116. Rating of Component 2. Weighing the achievements mentioned above, the ICR rates the implementation of this component as satisfactory. 117. Component 3 – Reforming Teacher Accountability and Incentive Systems for Performance Appraisal and Career Advancement. This component aimed at developing an integrated framework designed to sustain and continually enhance the quality and accountability of teachers after they have been certified. The goals of this component were to be achieved by: (i) reviewing and, to the extent possible, reforming existing policies, procedures, and instruments used by the Ministry of Administrative Reform for performance appraisal for progression and promotion; (ii) linking incentives created under the Teacher Law to career advancement; and (iii) improving lines of accountability between managers (principals/supervisors) and teachers. The above activities to be financed by BERMUTU were packaged into the following two sub-components: a. Sub-component 3.1: Providing funds for technical assistance, studies, workshops, and operational costs to: (i) establish and support a taskforce, under the PMPTK Project Coordination Unit (PCU), to coordinate and oversee activities within Component 3; (ii) develop a career-oriented teacher profile for progression and promotion; (iii) develop an approach to a performance-based appraisal; and (iv) develop an approach to addressing the under-performance of certified teachers; and b. Sub-component 3.2: Providing funds for technical assistance, workshops, and operational costs to: (i) pilot test and refine policies, processes, and instruments developed under Sub-component 3.1; (ii) print and distribute manuals and related documentation for training and the trial; (iii) develop a plan and strategies for providing continuing education to certified teachers; and (iv) prepare an integrated framework for sustaining and enhancing teacher quality, based on the findings of the trials in selected districts. 118. This component estimated at appraisal to cost US$5.28 million (5.6 percent of the total cost), 15.7 percent to be financed by the IBRD, 39.6 percent by IDA, 41.3 percent by the TF and 3.4 percent by the GoI, ended up documenting expenditures amounting to US$8.37 million, 158.5 percent of the total appraised amount. The over-spending is mainly explained by the fact that the appraised cost assumed a two-year implementation period for this component, taking in reality six years. 119. This Component supported the legislation and systems to assist the professional strengthening and accountability of teachers. Accordingly, the Regulation 16/2009 issued by the Ministry of State Apparatus and Bureaucratic Reform concerning teacher credit points 31 became operational in January 2013. Based on this, teachers are financially compensated based on their observable teacher’s competencies and knowledge. This performance-based teacher credit system is being managed by the TPM scheme. Credits are granted based on the results of administering the UKG, PKG and CPD assessment instruments. Possibilities to get credits include the RPL and a series of CPD options (including the KKG). 120. The TPM system was designed for the four-level professional career structure progression including the administration of: (i) the UKG (to test minimum standards in professional [what to teach] and pedagogic [how to teach it] competencies); (ii) the PKG (to assess minimum standards in key tasks in all legislated competency domains [pedagogic, professional, personal and social]); and (iii) the CPD (to assess proficiency gains in the professional career progression through the continuing professional development options as compared to the minimum professional standards). MoEC is currently financing the dissemination and expansion of KKG-MGPM in 50 districts – 16 BERMUTU districts for dissemination purposes and 34 new, non-BERMUTU districts, for expansion purposes – in 25 provinces (8 BERMUTU provinces and 17 non-BERMUTU provinces, each one with 2 districts per province). 121. PKG has been integrated with a system of on-line reporting (e-kinerja), either directly from the school or via the District Education Office, enabling the MoEC to: (i) monitor standards of operation from the national down to the school level; and (ii) facilitate follow-up in terms of supporting areas in most need and enable teachers with internet access to link directly to on-line versions for professional support. PKG has also provided the MoEC with valuable insights into the complexity of combining the management of professional development with financial incentives. In addition, by MoEC fine tuning the PKG with the 2013 Curriculum and the 2014 Civil Service Law, it will ensure that the linking of professional performance to income increases is carried out in consistency with the general administrative and reporting requirements for all civil service personnel management, which at times might enter into conflict with managing needs-based professional development. 122. However, having just recently started the implementation of the TPM scheme in 2013, there exist opportunities for improvement in terms of the effectiveness and timeliness of the ongoing performance-based teacher career professional progression, which MoEC will take into consideration as it moves forward. 123. Rating of Component 3. The rating for this component is based on the success in the ratification of key policies and regulation on teacher accountability and incentive systems for performance appraisal and career advancement detailed above (the Functional Credits for Teachers (in place), Academic Upgrading for In-service Teachers (in place), instruments for continuous professional development and teacher performance appraisal (developed but now under revision to concur with the 2013 Curriculum and the 2014 Civil Service Law) and issuance of the MoEC decree on new teacher induction program. Accordingly, the ICR rates the implementation of this component as highly satisfactory. 124. Component 4 – Improving Program Coordination, Monitoring and Evaluation. This component aimed at: (i) monitoring project activities and teacher certification to be 32 carried out by the Office of Research and Development within the Ministry of National Education (Balitbang) in an integrated fashion along existing monitoring systems functioning in the MoEC (school database and teachers’ working- groups database); (ii) developing of an improved teacher database (mostly financed by the GoI); (iii) assessing and evaluating the impact of BERMUTU; and (iv) supporting the management and coordination of BERMUTU’s activities. This component included the following three sub-components: a. Sub-component 4.1: Monitoring of project activities and teacher certification to be carried out by the Office of Research and Development within the Ministry of National Education (Balitbang) in an integrated fashion using existing monitoring systems. This sub-component aimed at financing the: (i) incremental operating costs associated with establishing a monitoring system for the certification of teachers and the grants programs under Components 1 and 2; (ii) the strengthening of the GoI’s system for the routine monitoring of teachers in order to increase its response rate, timeliness, and ability to monitor the teacher certification process as well as time spent on each on task and the teaching methods applied in classrooms; b. Sub-component 4.2: Evaluation and assessment aimed at financing: (i) a series of evaluation studies to measure the effect of BERMUTU-financed activities on teacher behavior and learning outcomes; and (ii) a series of Rapid Assessments to provide in-time feedback regarding the process of teacher certification and BERMUTU implementation; and c. Sub-component 4.3 aimed at financing project coordination and monitoring. 125. This component estimated at appraisal to cost US$33.83 million (17.3 percent of the total cost), 10.1 percent to be financed by the IBRD, 25 percent by the IDA, 38.3 percent by the TF and 26.6 percent by the GoI, ended up documenting expenditures amounting to US$21.98 million, 65 percent of the total appraised amount. 126. The under-spending is largely explained by the following 3 factors: (i) gains from fluctuations of the IDR exchange rate; (ii) the different impact evaluation studies were appraised under an assumed sample size of 600 schools while the real sample was in reality only 360 schools as per refined study design; and (iii) as mentioned further down, BERMUTU’s implementation was an efficient one (achieved more [see PDOs] with less). 127. The implementation of this component was also delayed by the restructuring of the MoEC, budget allocation challenges and capacity constraints as explained under Section 2, which slowed initial project implementation especially activities such as the envisaged integration of the teachers’ database with the school database and teachers’ working groups database. As mentioned before, however, most activities were completed by project closing. 128. As a result of the organizational restructuring of the MoEC, the unit responsible for integrating the teachers’ database with the school database and teachers’ working- groups database - the Center for Education Data and Statistics (PDSP) - was moved from the 33 Research and Development Board to the Secretary General with changes of roles and responsibilities of the PDSP, changes of leadership, and unavailability of fully dedicated staff. Despite these changes, the envisaged integration was accomplished few weeks before project closing. Although it too early to ascertain the effectiveness and degree of utilization of this integrated teacher database, it represents an important tool for MoEC’s decision- making process. 129. This component financed the establishment of a baseline and mid-term data for the impact evaluation of the teacher certification and trend analysis of student and teacher performance in selected schools participating in BERMUTU. This effort was successfully completed. 130. This component also supported the carrying out of the following studies which were satisfactorily completed and whose preliminary findings are explained below: a. LPTK (Pre-service Teacher Training) impact evaluation study aimed at evaluating longer term effects of the Teacher Law by analyzing whether certification and the prospect of a higher salary attracted better high school graduates to the teacher training universities; b. KKG/MGMP impact evaluation study (Puspendik Baseline, 2010; Puslitjak, 2013 final report) aimed at evaluating the impact of teacher working groups introduced under BERMUTU on student learning. At the design stage, it was planned that the study would rely on a randomized evaluation design to induce exogenous variation in the in-service training of teachers. This could not take place however, since the control groups received the same treatments. The analysis approach is shifted to trend evaluation of the teachers and the student performances; c. Teacher certification impact evaluation study (Puslitjak, 2013). This joint study by the WB and MoEC evaluates the impact of the Law on Teachers and Lecturers (14/2005) and the associated increase in teacher salaries on student learning. The study specifically analyses the effects of the law through its impact on in-service teachers. The study relies on a randomized evaluation design to induce exogenous variation in the certification process; and d. Absenteeism Study in BERMUTU districts. Studies by SMERU (funded by the WB) of teacher absenteeism on randomly selected public primary schools in 2003 and 2008 using nationally representative samples of primary schools. BERMUTU supported a similar study in project districts in 2010 and again in 2012. 131. Findings of the above-mentioned studies attesting to some welcoming noticeable changes due to BERMUTU and supporting the ICR’s PDOs rating: a. Accredited LPTKs tend to attract a better quality intake of students (basic teachers-to- be) than non-accredited LPTK (compared also against a baseline) and the intake of the former set of LPTKs is larger than the one of the latter set; 34 b. Quality of accredited LPTKs in terms of number of certified lecturers, number of lectures doing research and number of graduates becoming certified basic education teachers is better than non-accredited LPTKs (also compared against a baseline); c. The Puspendik Baseline, 2010 report found that the test results of basic education teachers in BERMUTU districts that received a grant were slightly better than the results of teachers in BERMUTU districts that did not receive grants in terms of teacher professional competencies but did not have an impact on the pedagogy, personal and social aspects of the teacher. The Puslitjak, 2013 final report found that the grants had positive impact in the: (a) teacher working groups’ profile and activities; (b) teacher subject knowledge competency; and (c) student learning outcomes. However, during 2010-2012, teacher-working groups in non-BERMUTU districts improved their profile and activities as well, leading to conclude that the grant and working groups were not the only way to improve teacher performance; and d. The Puslitjak 2013 final report of the third study found that: (a) teacher competencies acquired under his/her certification process and/or participation in working groups was positively correlated with his/her education background; (b) teachers certification program decreased number of teachers with second jobs; (c) the professional competencies between teachers with and without second jobs were almost the same; (d) the end-line study showed that, regrettably, the teacher professional scores were very low, indicating limited teacher professional competencies’ gains, opposite to the expectation of the BERMUTU design; and (e) students under certified teachers showed better learning outcomes than similar students under non-certified teachers. 132. Rating of Component 4. Cognizant on the completion of all of the above-mentioned studies and the development of a teachers’ database counterweighted by the delay in finalizing the integration of the above database with the school and teachers’ working-groups databases, the ICR rates the implementation of this component as moderately satisfactory. 133. The information in Table A2.1 corresponding to Component 4, the one associated with BEMUTU’s monitoring, impact evaluation studies and management, reflects a sign of efficient implementation, as spending was 64.3 percentage of the estimated at appraisal, the PDOs were achieved and even surpassed, all the subcomponents were implemented and most of their corresponding intermediate outcome also achieved or exceeded. 134. Table A2.2 reflects a significant under-spending of category 2 – distance learning development grants – which is partially explained by the combination of the fluctuations of the exchange rate (as explained above) and that the average amount of the grant estimated at appraisal was about 42 percent greater than the one actually provided. 35 Table A2.2: Documented expenditures by component and by source at appraisal and closing of BERMUTU (in US$ millions) Component At the time of appraisal At the time of BERMUTu’s closing Percentages with respect to the appraised estimates December 31, 2013 IBRD IDA TF Govt Total IBRD IDA TF Govt Total IBRD IDA TF Govt Total (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (6)/(1) (7)/(2) (8)/(3) (9)/(4) (10)/(5) Component 1 2.90 10.28 3.27 4.00 20.45 3.42 10.66 0.34 0.66 15.08 117.7 103.7 10.4 16.5 73.7 Component 2 16.63 40.09 33.60 43.88 134.20 11.71 41.26 38.49 36.39 127.84 70.4 102.9 114.6 82.9 95.3 Component 3 0.83 2.09 2.18 0.18 5.28 0.70 3.81 1.08 2.78 8.37 84.7 182.0 49.6 1543.3 158.5 Component 4 3.42 8.47 12.95 9.00 33.83 2.23 3.70 10.96 5.10 21.98 65.4 43.7 84.6 56.7 65.0 Unallocated 0.66 0.58 1.24 Fee 0.06 0.06 0.06 0.06 100.0 100.0 Total 24.50 61.50 52.00 57.06 195.06 18.12 59.42 50.87 44.93 173.34 74.0 96.6 97.8 78.7 88.8 36 Table A2.3: Documented expenditures by category and by source at appraisal and at the closing of BERMUTU (in US$ millions) Category of expenditure At the time of appraisal At the time of BERMUTu’s closing Percentages with respect to the appraised estimates December 31, 2013 IBRD IDA TF Govt Total IBRD IDA TF Govt Total IBRD IDA TF Govt Total (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (6)/(1) (7)/(2) (8)/(3) (9)/(4) (10)/(5) Category 1: 0.88 5.93 0.89 7.70 0.99 4.89 5.89 113.02 82.5 76.4 Accreditation incentive grants Category 2: 1.24 2.37 0.59 4.20 0.00 0.6 0.60 0.0 25.1 14.2 Distance learning development grants Category 3: 4.88 12.75 9.87 22.50 50.00 7.61 15.13 8.86 17.06 48.67 155.9 118.7 89.8 75.8 97.3 Training workshops and incremental operating costs under sub-comp 2.2 in the provinces Category 4: 11.10 25.56 20.24 18.00 74.90 3.74 24.03 29.47 18.31 75.55 33.7 94.0 145.63 101.7 100.9 Working groups grants Category 5: 0.20 0.50 10.00 10.70 0.93 0.35 5.51 6.79 464.0 70.7 55.1 63.4 Consultants’ services Category 6: 5.48 13.81 10.41 16.56 46.26 4.79 14.42 7.02 9.55 35.79 87.4 104.4 67.5 57.7 77.4 Training workshops, fellowships and incremental operating costs under sub-comp 2.2 at the national level Unallocated 0.66 0.58 1.24 Fee 0.06 0.06 0.06 0.06 100.0 100.0 Total 24.50 61.50 52.00 57.06 195.06 18.12 59.42 50.87 44.93 173.34 74.0 96.6 97.8 78.7 88.8 37 Annex 3. Economic and Financial Analysis Background 135. The economic rationale for the BERMUTU program is based on the expected gains from supporting teacher skills development and qualification upgrading, with improved competency of teachers eventually contributing to better learning outcomes for students in the long term. In the economic analysis of the PAD, it was shown that teacher qualifications had a significant effect on teachers’ competency scores, i.e. teachers with a four year college degree (S1) scored higher than teachers without a degree. In addition, it showed a positive relationship between teacher qualifications and student achievement. Using grade 8 students’ math scores in TIMSS 2003, it showed that after controlling for students’ family backgrounds and other teacher characteristics such as the number of years that they have been teaching, students taught by teachers with an S1 degree scored higher than students taught by teachers with only a teaching diploma (one or two year diploma program)13. Considering these potential gains, BERMUTU program had a sound economic rationale to support the government’s teacher reform and activities related to the certification program that attempts to upgrade teacher academic qualification to a four-year degree through teacher training courses. 136. The economic analysis in the PAD did not include any calculation of economic rate of return for project components. However, it has some analysis of cost effectiveness of upgrading teachers’ qualifications compared with the alternative way of reducing class size to improve student learning achievement in Indonesia. Although at the time of preparation, it was assumed that improvements in student-teacher ratios (STR) would lead to potential efficiency gains, a Bank study (2013) showed that due to chronic inefficiency of education spending, the already low STRs continued to decline in both primary and secondary education 14. Given that BERMUTU was not designed to tackle that issue, the ICR will not measure efficiency of the BERMUTU program based on the STR indicator. As stated before, BERMUTU’s key contribution was to support the teacher reform activities related to the teacher certification program, which was the cornerstone feature of the reform. It contributed to the system as a whole by strengthening the accreditation system for teacher training colleges, developing distance learning materials for upgrading channel, improving the policies and regulations on teacher accountability and incentive systems for performance appraisal and career advancement (recognition of prior learning for in service teachers), supporting the development of new regulations on a teacher induction program, and strengthening the role of teacher working groups (KKG/MGMP). Thus, most if not all of 13 See Annex Table 10.3, BERMUTU PAD, p. 97 14 World Bank (2013). Spending more or spending better: Improving education financing in Indonesia. p. 45 38 BERMUTU’s contributions are difficult to quantify due to its very nature of reforming the system through better policies and regulations. 137. The economic analysis of the BERMUTU program refers to several preliminary research findings and the Bank sector study (Teacher Reform in Indonesia: The Role of Politics and Evidence in Policy Making, 2014), which provides detailed analysis of the process of teacher reform in Indonesia. The next sections are organized as follows: (i) contribution to the PDO; (ii) program efficiency; and (iii) program sustainability Contribution to PDO 138. Increased number of teachers meeting academic qualifications mandated by teacher law. The proportion of teachers with a four-year college degree (S1) degree has increased steadily during the period of 2006-2012 (see Figure 1). BERMUTU played an instrumental role to support teachers to acquire the mandated S1 degree. When BERMUTU was started in 2008, there were only around 900,000 teachers with an S1 degree. By the time of BERMUTU’s closing, it had nearly doubled to over 1.7 million teachers. BERMUTU’s contribution for this outcome is focused on support for accreditation of pre-service teacher training colleges and other support to pre-service and in-service teachers (component 1). In addition, BERMUTU also contributed to the improvement of policies and regulations supporting the teacher certification program through, for instance, recognition of prior learning (component 2), which affected 879,832 teachers. 139. BERMUTU support for teacher academic upgrading has been instrumental in terms of the accreditation of 81 pre-service teacher training colleges (LPTK). Component 1 of the BERMUTU program supported the development and validation of accreditation instruments and procedures to measure institutional baseline scores and changes towards the required accreditation scores. The program provided competitive accreditation incentive grants to 26 teacher training universities to support their efforts in obtaining accreditation and/or improving their accreditation status. In turn, improved accreditation of the LPTK is correlated positively with the quality of high school graduates entering teacher training institutions. A three-round survey on LPTK commissioned by MoEC found that accredited LPTKs tend to attract an intake of more and better quality students than non-accredited LPTKs 15. In this regard, accreditation status seems to act as a signal of quality to prospective students. And BERMUTU contribution can be linked to component 1 that supports the accreditation process to LPTKs. In addition, the LPTK study also showed that the quality of accredited LPTKs in terms of number of certified lecturers, number of lectures doing research and number of graduates becoming certified as basic education teachers is better 15 Puslitjak-MoEC (2009, 2011, 2012). LPTK (Pre-service Teacher Training) survey. Unpublished reports. 39 than non-accredited LPTKs. While these indicators do not capture the full extent of quality, they do provide some evidence that accredited LPTK has made some progress in a positive direction. 140. In addition to support for the accreditation of LPTK, BERMUTU also provided support for the development of distance learning materials for the Open University upgrading channel. The latter has greatly contributed to improved academic qualifications of 879,832 teachers. The Open University channel was the most accessible route for incumbent teachers to upgrade their academic courses. This upgrading channel allows teachers to fulfill their classroom tasks while pursuing their S1 degree. In this regard, the Open University upgrading channel has also contributed to efficiency of the certification program by minimizing the likelihood of teachers abandoning their teaching duties while pursuing their S1 degree. Figure 1: The proportion of primary and secondary school teachers holding a four year university degree, 2006-2012 Source: NUPTK (2006,2008,2010,2012) 141. Increased number of primary and junior secondary teachers using appropriate classroom instruction. BERMUTU was a key instrument by which over 214,000 teachers in the treatment areas benefited from the professional working group (KKG/MGMP) activities for the adoption of improved instructional practice. Arguably, the most important economic impact of the BERMUTU program is to activate previously inactive and low-functioning teacher working groups and to leverage working group activities more effectively for skills development and teachers’ upgrading. Teacher working groups (KKGs and MGMPs) have played an integral role in supporting teachers in training and professional development activities. They also provided a forum for teachers to discuss teaching problems and work 40 cooperatively to undertake common tasks such as curriculum development, the creation of teaching aids, and the design of test items as well as more advanced activities such as lesson study and classroom action research. 142. All regular primary and junior secondary school working groups in the BERMUTU program received a direct grant through the provincial quality assurance agency (LPMP). Primary working groups received US$4,800 over three years, while for junior secondary school working groups the annual amount was US$6,300. This funding was conditional on adhering to the program guidelines, which meant expenditures were to be directed toward the purchase of learning materials and activities associated with modules of work provided by the program. These modules predominantly focused on classroom action research, lesson study, active learning, group work, and other effective classroom methodologies. A further requirement was that each working group would meet 16 times each year (fortnightly) to conduct activities—a more intensive pattern than in the past. 143. An impact evaluation study commissioned by MoEC examined a sample of KKG/MGMP in 20 BERMUTU districts and another 20 non-BERMUTU districts. The study found that working groups under BERMUTU were more effective, better organized, held more accountable and that activities involved more participation from the individual teachers compared to the non-BERMUTU counterparts. In addition, BERMUTU also had an interesting spillover effect to non-BERMUTU groups through the adoption BERMUTU modules and guidelines. The widespread use of BERMUTU modules and guidelines among non-BERMUTU working groups provides additional evidence of BERMUTU impact on overall system efficiency. 144. The study also found a positive impact of BERMUTU on teachers’ competencies. Teachers in working groups receiving a direct grant under the BERMUTU program were tested at the baseline (2010) and midline (2011) and showed a significant improvement in cognitive and pedagogical ability, as shown in Figure 2 below. (Puslitjak-MoEC 2011) 41 Figure 2: Standardized Gains in Abilities of Primary School Teachers Attending Working Groups in BERMUTU Districts, 2010–11 Baseline (2010) standard deviations Source: Rahmawati (2011) “BERMUTU KKG/MGMP Study report”. Background paper, Education Assessment Center, Jakarta, Indonesia. Note: Teachers tested were drawn from teacher working groups in districts participating in BERMUTU program. 145. As mentioned before, there were some weaknesses in the KKG/MGMP study. To re- confirm the extent of the positive impact of direct grants on teachers’ subject knowledge competencies, another analysis is expected to be conducted. However, the KKG/MGMP study already highlights areas of initial efficiency gains and therefore provides the government with the opportunity to further analyze and fine-tune the direct grants to teachers approach. 146. Reduction in teacher absenteeism. Teacher absenteeism was a serious problem before the BERMUTU program was implemented. Two studies commissioned by the Bank on randomly selected public primary schools in 2003 and 2008 using nationally representative samples of primary schools found teacher absenteeism rates to be 19 percent and 14 percent respectively16. These studies were conducted during BERMUTU preparation for program implementation to inform policy on the need to seriously tackle the high absenteeism problem. A government teacher absenteeism study conducted by Puslitjak- MOEC (2012) in 20 BERMUTU districts found the percentages have reduced to 6.6 percent for primary school teachers and 6.3 percent for junior secondary school teachers. There are at least three hypothetical explanations to this reduction in absenteeism. First, the teacher certification program, which effectively doubled the salaries of teachers, makes teachers less likely to have extra jobs. It was anticipated that absenteeism (as a natural consequence of 16 SMERU (2010). “Teacher Absenteeism and Remote Area Allowance Baseline Survey”. Jakarta. Available online: http://www.smeru.or.id/report/research/absenteeism/absenteeism_eng.pdf 42 teachers being less occupied with making extra money outside school hours) would decline and teachers would spend more time to teach in school. Second, the teacher certification program has been implemented with a supporting regulation requiring teachers to have 24 hours classroom time per week in order to receive the certification allowance. Third, the improved monitoring and evaluation system (through LPMP and District Education Office) for teacher performance may have a significant impact on the reduction of absenteeism. BERMUTU’s key contribution in this area is through the capacity improvement of LPMP and District Education Office to establish an effective M&E system. 147. A World Bank commissioned Randomized Control Trial (RCT) impact evaluation study on teacher certification reveals that because of the certification program, teachers rely less on second jobs and have fewer difficulties financially supporting their households (De Ree et al. 2012). Figure 3 presents the effects of certification on teacher characteristics. Overall, the livelihoods of teachers have improved. Certification is related to a 27 percentage point decrease in the likelihood of teachers holding a second job because the professional allowance decreases the need to rely on such jobs to supplement income. This income effect is also reflected in a decrease in the number of teachers who report problems financially supporting their households. Figure 3: Effects of certification on selected teacher characteristics in Indonesia, 2009- 11 Source: De Ree et al. 2012. 148. To achieve the most important outcomes of the teacher reform, i.e. teachers’ improved knowledge and ultimately improved student learning outcomes, a long-term perspective needs to be applied, to which the project lifespan of six years contributed. To date, there is inconclusive evidence to show that Indonesia’s teacher reform has reached its ultimate outcomes of teachers improved knowledge and ultimately student’s learning outcomes. Figure 3 indicates that the certification process and the payment of the professional allowance alone does not make teachers more knowledgeable (measured by the insignificant t-stat in the first two bars). This finding indicates that the 90-hour training, 43 which was part of the certification procedure for many teachers, is perhaps not yet as sufficient given the short time span to bring the competence of teachers to the next level. Given that there has been a system weakness in terms of school instruction time deficit, the 90-hour training program is only the first step to improve teacher competences and improve instructional time. As a result of this study, the GOI decided to redesign the way teacher were certified, including making a teacher competency test compulsory; improving the accreditation system for LPTKs; and improving training programs for teachers failing the competency test. This new design of the teacher certification program, together with better quality inputs to teacher training colleges, will arguably bring a positive impact on the quality of the overall system. Efficiency analysis 149. First, to put the costs of BERMUTU into perspective, the investment is relatively small compared to the total education budget, or even compared to the total GOI budget (less than one percent of total GOI budget) and most importantly it achieved all of its PDOs. 150. The BERMUTU program was efficient from both an operations and expenditure perspective. A good indication of its overall efficiency is the fact that the project achieved all of the PDOs and the intermediate outcome indicators with less financial resources than the original projection at the appraisal stage. At the same time the project had commendable disbursement of 93.1 percent of program loan, credit and grant funds. The bulk of these funds (US$91.46 million or 66.3 percent of the total) was allocated to component 2 for “strengthening structures for teacher improvement at the local level”. This component reached a 95.6 percent expenditure rate with respect to the appraised estimates (as shown in Annex 2). 151. The emphasis on component 2 is justified given the significant size of the target groups and its contribution in the form of improvements in teacher pedagogical skills. The latter is achieved through project support for the working group program of primary and junior secondary teachers (KKG and MGMP) in 75 participating districts. The support aimed at providing teachers the opportunity to conduct a wide range of professional development activities, continuously learn from their peers and identify methods to improve the quality of teaching and learning. This program benefited 214,380 primary and junior secondary school teachers, principals and supervisors from 6,107 working groups. 152. While teacher working groups existed prior to BERMUTU, many were inactive or not operating effectively. The project provided a stimulus to revive and strengthen these KKG/MGMP by introducing a well-structured program. There were four important characteristics of BERMUTU support: (i) provision of grants for each KKG/MGMP for three consecutive years to ensure sufficient time for the working groups to become well- 44 established and fully functional; (ii) development of a strong support system to ensure quality and sustainability (as described in Annex 2); (iii) formation of an M&E system; and (iv) a special model of support for teachers in difficult and remote areas. BERMUTU granted each of the working groups a total US$4,800 for KKG and US$6,300 for MGMP, for three consecutive years. While some non-BERMUTU working groups also received support, it tended to be much less comprehensive. For instance, there was a separate grant program to non-BERMUTU KKG/MGMP in 2007-2009 that provided US$1,000 to KKG and US$1,500 to MGMP per year with no guarantee for continuous funding for the following year and without a support system as provided under BERMUTU. This undermined non-BERMUTU KKG/MGMPs effectiveness and commitment to carry out their mandate. 153. BERMUTU also introduced a different focus of grant spending. A survey conducted in 2007 of the non-BERMUTU grant program showed that most of the expenditure was on meals (31 percent), transportion (22 percent), and honorarium (20 percent); very little was spent on items that would support the session activities (e.g. supplies to create learning aids, bringing in expert resource personnel and classroom action research). The BERMUTU grant was specifically designed to ensure the funds would go mainly toward supporting high quality session activities and could not be used for meals and teacher transport (with the exception of teachers in remote KKG/MGMP), In this regard, BERMUTU improved efficiency of the system by focusing the use of grant financing to enhance the quality of working group activities. 154. While the cost of supporting teacher working groups through BERMUTU was more expensive, the project was arguably more cost-effective given that the BERMUTU- supported teacher working groups met more frequently, operated more effectively, and were more focused on improving teaching and learning. An impact evaluation study commissioned by MoEC examined a sample of KKG/MGMP in 20 BERMUTU districts and another 20 non-BERMUTU districts. The study found that working groups under BERMUTU were more effective, better organized, held more accountable and that activities involved more participation from the individual teachers compared to the non-BERMUTU counterparts. Learning from BERMUTU experience, in the 2012-2013 school year MOEC decided to adopt the BERMUTU approach to provide direct, proposal-based grants to KKG/MGMP amounting to US$3,000. However, this is not a multi-year grant program since the Indonesian financing system does not allow for this type of financing mechanism and is arguably undermining the program’s effectiveness given the uncertainty of funding. 155. Component 2 also financed the development of a system to recognize teachers’ working experience and prior learning (RPL). In collaboration with selected teacher training colleges (LPTK), MoEC designed the application of the RPL system for programs tailored for teachers who have to upgrade to the 4-year college degree. This RPL has benefited 879,832 teachers by reducing the number of credits they had to take in order to earn an S1 45 degree. With the RPL system, in-service teachers can complete their study in a shorter period and they do not have to leave their teaching responsibilities. In addition to support for the accreditation of LPTK, BERMUTU, under component 1, also provided support for the development of distance learning materials for upgrading channel (HY-LITE) organized by the Consortium of LPTKs. This program, along with the distance learning by the Open University, have contributed to the academic upgrading of 879,832 teachers, thus to the achievement of the first PDO indicator, i.e. “Increased number of teachers meeting academic qualifications mandated by the Teacher Law”. The distance learning channel was the most accessible route for incumbent teachers to upgrade their academic credentials. This upgrading channel allows teachers to fulfill their classroom tasks while pursuing their four- year college degree. Sustainability analysis 156. Given what BERMUTU has achieved, there is a clear need to sustain the reforms it has supported, especially in terms of teacher working group activities. There are some evidences that program activities under BERMUTU have been continued with financial support from the central and local government. MoEC has allocated funds in 2014 to disseminate and expand the teacher working group grant program to 50 new districts. Another positive sign is that more than half of BERMUTU districts have signed a MoU with the MoEC to continue the BERMUTU-related programs in their districts using their own funds. More importantly, the teacher working group program will be sustainable as it is the most accessible option for teachers to do the required CPD and has been recognized by GoI as such. The spending to strengthen teacher working groups is fairly small compared to the education budget and the budget for the certification program. BERMUTU has therefore shown that investing in teacher working groups was cost effective and it brought significant impact on teachers’ motivation and pedagogical knowledge 46 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Responsibility/Specialty At the preparation stage Mae Chu Chang Lead Education specialist EASHD Task Team Leadership Susiana Iskandar Senior Education Specialist EASHD Co-Task Team Leadership Steen Andersen Consultant. EASHD Teacher Reforms Niel L. Baumgart Teacher Quality Specialist EASHD Peer Reviewer Steven Burgess Sr. Social Development Specialist EASHD Governance Dandan Chen Senior Economist EASHD Economic analysis Helen J Craig Senior Education Specialist EASHD Peer Reviewer Muhammad Firdaus Consultant EASHD Civil Service Policy Titie Hadiyati Consultant EASHD Operations Manual Robin Horn Education Sector Manager ECSHD Peer reviewer Dingyong Hou Senior Education Specialist EASHD Project Preparation Ratna Kesuma Education Specialist EASHD Teacher Working Group Grant Jups Kluyskens Consultant EASHD Civil Service Policy Richard Kraft Consultant EASHD Teacher Reforms Juan Martinez Sr. Social Development Specialist EASIS Social Safeguards Sulistiowati Nainggolan Social Development Specialist EASHD Social Safeguard Vicente B Paqueo Country Sector Coordinator EASHD Quality Control Natasha Beschorner Senior ICT Policy Specialist TWICT Information Technology Menno Pradhan Senior Economist DECRG Economic analysis Yogana Prasta Senior Disbursement Officer EACIF Disbursement Andrew Ragatz Senior Education Specialist EASHD Teacher Reforms Ralph Rawlison Consultant EASHD Employment Compensation Rizal H Rivai Senior Procurement Specialist EAPCO Procurement Rosfita Roesli Operation Officer EASHD Project Preparation F. Halsey Rogers Senior Economist DECRG Economic Analysis Imad Saleh Lead Procurement Specialist EAPCO Procurement Christopher Smith Consultant EASHD Institutional Accreditation Rajiv Sondhi Sr. Financial Management Specialist EAPCO Financial Management Ritchie Stevenson Consultant EASHD Teacher Reforms Unggul Suprayitno Financial Management Specialist EAPCO Financial Management Erly Carina Tatontos Team Assistant EASHD Mission Support Christopher J. Thomas Sector Manager EASHD Quality Control At the implementation stage Susiana Iskandar Senior education specialist EASHE Task Team leadership Khadrian Adrima Program Assistant EASHE Project Implementation Support Hafid Inunuh Alatas Consultant EASHE Operations Analysis Dayu Nirman Amurwanti Operation Officer EACIF Governance Novira Asra Sr. Financial Management Specialist EASFM Financial management Amanda Beatty Consultant EASHE Impact Evaluation Studies Irene E. Bouwmeester Program Assistant EACIF Implementation Support Mae Chu Chang Sector Leader EASHE Quality Assurance Siwage Dharma Negara Operation Officer EASHE Teacher Data System Enda Ginting Public Sector Specialist EASHE Financial Management Titie Hadiyati Operation Officer EASHE Project Supervision 47 David Harding Consultant EASHE Teacher Training William Hardy Consultant EASHE Procurement Djoko Hartono Consultant EASHE Monitoring and Evaluation Joppe Jaitze De Ree Consultant EASHE Impact Evaluation Studies Megha Kapoor Research Analyst EASHE Project Implementation Sulistiowati Nainggolan Consultant EASHE Social Safeguards Siwage Dharma Negara Operation Officer EASIS Operation Dyah Kelasworo Nugraheni Program Assistant EASHE Supervision Support Ade Sonya Oktaviane Team assistant EACIF Supervision Support Budi Permana Procurement Analyst EACIF Procurement Tatong Permana Anggrimulja Consultant EASFM Financial Management Concepcion V Pijano Consultant EASHE Higher Edu. Accreditation System Rima Prama Artha Consultant EASHE Research Assistant Shahnaz Natasya Arina Consultant EASHE Research Assitant Andreas Priyono Consultant EASHE Learning Module Review Rizal H Rivai Senior Procurement Specialist EAPCO Procurement Rivandra Royono Research Assistant EASHE Operation Assistant/Analyst Isono Sadoko Consultant EASID Social Safeguard Imad Saleh Lead Procurement Specialist EAPCO Procurement Quality Assurance Ahsan Ali Lead Procurement Specialist EAPCO Procurement Quality Assurance GB Surya Ningnagara Team Assistant EACIF Supervision Support Christopher Smith Consultant EASHE Higher Education /Grants Ritchie Stevenson Consultant EASHE Teacher Quality Management Susie Sugiarti Research Analyst EASHE Studies Unggul Suprayitno Financial Management Specialist EASFM Financial Management Erly Carina Tatontos Team Assistant EACIF Supervision Support Anna Coronado Program Assistant EASHD Supervision Support Paulus Bagus Tjahjanto Procurement Officer EAPCO Procurement Franco Russo Operation Officer EASHE Sector Quality Review Eduardo Velez Bustillo Sector Manager EASHE Quality Assurance - overall Luis Benveniste Sector Manager EASHE Quality Assurance- overall At the completion stage Juan Prawda Consultant EASHE ICR Primary Author Samer Al-Samarrai Senior Education economist EASHE ICR- Economic Analysis Siwage Dharma Negara Operation Officer EASHE ICR-Economic Analysis Angela Demas Senior Education Specialist EASHE ICR Peer Reviewer Helen J. Craig Lead HD Specialist EASHE ICR Peer Reviewer Franco Russo Operation Officer EASHE Sector Quality Reviewer Sandra Beemer Sr. Operations Officer - Consultant EASHE Sector Quality Reviewer Andrew Ragatz Senior Education Specialist EASHE ICR Mission Member Susie Sugiarti Research Assistant EASHE ICR Mission Member Titie Hadiyati Operation Officer EASHE ICR Mission Member/Disbursement Dyah K. Nugraheni Program Assistant EASHE ICR Mission Support Anna Coronado Program Assistant EASHD ICR Mission Support Susiana Iskandar Senior Education Specialist EASHE ICR Mission Leader 48 Annex 4. Bank Lending and Implementation Support/Supervision Processes (cont.) Staff Time and Cost (Bank Budget Only) USD Thousands (including Stage of Project Cycle No. of staff weeks travel and consultant costs) Lending FY 2006 13.98 55.176 FY 2007 84.62 450.763 FY 2008 7.05 36.6 Total: 105.65 545.54 Supervision/ICR FY 2009 27.62 73.85 FY 2010 46.71 195.16 FY 2011 24.17 94.67 FY 2012 22.11 95.60 FY 2013 13.60 81.77 FY 2014 20.17 94.69 Total: 149.49 635,75 49 Annex 5. Beneficiary Survey Results Not applicable 50 Annex 6. Stakeholder Workshop Report and Results Not applicable 51 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR 157. The World Bank requested each of the BERMUTU Satkers at the central and provincial levels to answer questions with regard to the implementation of the program. The response from the PCU and Satker Pusbangprodik are presented below. The responses from other Satkers are available in the project file due the size of the documents. A. Project Coordinating Unit (Satker P2TK Dikdas) What worked well in the program and why? 1) The number of teachers meeting academic qualification mandated by the Teacher Law No. 14/2005 was achieved surpassing the target expected, mainly as a result of the implementation of RPL (Recognition of Prior Learning) carried out mostly through distant education at the Open University (UT) as well as implemented through the teacher training programmes at LPTKs in certain universities as they were officially appointed for this purpose by the Directorate General of Higher Education. 2) The number of primary and junior secondary teachers using the pedagogic skills with specific subject and age appropriate pedagogy was achieved surpassing the target expected. The improvement in the pedagogic skills of teachers participating in the 6,107 teacher, principal and supervisor professional working groups with the primary aim to improve their pedagogic competency. The appraisal of a random sample of classroom performance at 900 schools in 75 districts and municipalities (Gintings, 2013), has validated that improvement was occurring. According to the monitoring and evaluation findings, the achievement was on account on, among others, the joint efforts and functions among LPTK (teacher training programs at the university) in developing pre-service programs accredited at the S1 level BERMUTU Accreditation Incentive Grants, P4TK (Teacher Training Center), LPMP (quality assurance agency), and the Office of Education (at district and municipalities) in effectively implementing CPD (Continues Professional Development). 3) The outcomes of the joint efforts among LPTKs, P4TKs and LPMPs in implementing CPD are by far very effective, because each institution, especially P4TK and LPMP as a companion activity in the working group. The role of lecturers and LPMP P4TK becomes very important, because they are compiled into a module and the development team (NCT). Both of these institutions, can play an optimal role in describing each function in ensuring the quality of teachers / principals / school supervisor who will then have an impact on the improvement of student achievement (component 2). With the increasing ability of educators that will enhance the quality of learning, thus increasing the quality of learning will have an impact on the quality of student learning and achievement. 4) The absenteeism rate of teachers at primary and junior high were dropped far lower than expected (6.6% of 15% for primary and 63% of 15% for junior high) was 52 achieved, among others, due to good cooperation between supervisors and principals, which leads supervisory learning processes better in schools at BERMUTU region, which in turn, improving the rate of teacher absenteeism far lower than it was anticipated. In addition, the liability rules that every teacher has to teach 24 hours per week also encouraged them to be more diligent in teaching. 5) The entity of active professional working groups consist of primary and junior secondary school teachers, principals and supervisors, in 3,295 KKG, 1,383 MGMP, 931 KKKS, 220 MKKS, 178 KKPS and 100 MKPS in 75 BERMUTU districts. In addition, there are total of 193 forums for KKG, MGMP, MKKS and MKPS, made up of 6,107 active professional working groups of primary and junior secondary school teachers, principals and supervisors in total. It means that the number of active teacher working groups and forums for primary and junior secondary school was achieved surpassing the target expected (6,107 of 4,500) in the Project Development Objectives. The exceeding output of the objective is due to, partly, the fact that there were a number of grants for remote areas of Java left unused because there were no longer KKGs/MGMPs which met that classification in Java. The unused budget was then reallocated for KKG/MGMP as regular local grants which have a far lower unit cost compared to the remote ones; each grant for a remote area can be reallocated to about approximately 5 regular local grants. The effort of empowering the 6,107 professional working groups of school teachers, principals and supervisors at the local level was so fundamental, since the teacher professional management system had already been established and operationalized as the main agency to deliver continuous professional development activities. 6) The tremendous efforts have been dedicated by MoEC to bring about policies and regulations to improve teacher welfare, professional abilities, and career development brought into being. There were remarkable outputs achieved in designating and developing a great deal of regulations in support of teacher quality improvement and career development at the post issuance of the Teacher Law no.14/2005. There were more than a dozen teacher regulations delivered; among others: career structure, teacher certification, teacher induction, teacher performance appraisal and career development, and teacher, principal, supervisor performance assessment, which, among others: MenPAN Decree 16/2009 on Functional Credits for Teachers, the instruments for teacher performance assessment (PKG) and Teacher Continuous Professional Development (CPD), and legalized by the issuance of, among others, the joint decree between the MoEC and the National Education Board No: 03/V/PB/2010; No:14 in 2010 about the implementation guidelines; the Education Ministerial Decree No: 35 and No 38 in 2010 about the technical guidelines for PKG); and its equivalently for private teachers on the Functional Credits. 7) Adjacent to the provision of policies and regulations by MoEC, of the “inter- government-coordination” between central and district level defined, the joint cooperation between the two parties in delivering a local regulation of teacher quality improvement in education development was overwhelming. This was evidenced by the enactment of local regulations in 41 districts /cities of BERMUTU partners on 53 teacher quality improvement programs and its implications of funding from the local budget (APBD). This is the first time this happened in the history of post-government autonomy at districts and municipalities during the period of 14 years (since 2000), that the presence of central government policies are fully supported simultaneously by the 41 local governments including funding in a form of local regulation, due to the common interests of both parties in implementing teacher quality improvement program in three principal programs i.e, induction, teacher performance assessment and continues professional development. 8) The impacts of the BERMUTU project, which are deemed of great magnitude, are the best practices at district and municipalities in forms of, among others, program replication implementing the BERMUTU model. For example, the local District and municipality Governments of Gorontalo have officially declared the continuation of the teacher quality improvement program in the region beyond the life of the BERMUTU project. In addition, the City Government of Bogor has extended the CPD Program, not only for KKG and MGMPs at basic education, but also for the secondary level. More encouragingly, the significant number of KKGs and MGMPs in Sumedang, Pekalongan, and districts and municipalities have been able to continue the CPD activities independently with their own budget, which indicates the assurance of the sustainability of BERMUTU program in the future. 9) The coordination between central government and district/city governments had been conspicuously successful for the reason that: (i) since the beginning, both parties realized their own rights and obligations agreed upon in implementing BERMUTU Program, as they were suggested in the MoU; (ii) there were national coordination meeting regularly carried out by central government during the period of BERMUTU implementation in 2008-2013, at least twice a year, as the forum of communication among central and local governments to assess the progress of current activities, problems faced, solutions made, and follow-up agreed upon, to bring about the quality improvement of program milestones as well as program outputs. What did not work so well in the program and why? 1) Shortfall of achievement in induction program (1,400 of 3,000), among others caused by: (i) at the time of induction program carried out there was no local regulations supported the implementation of the program; in addition, the program was more focused on the beginner teacher candidates for public servants and very few take into account as a private teacher program participants; (ii) although according to the Decree of the Minister of State Apparatus, Induction program implemented starting in 2013, there were no special programs at the time conducted by the Center for Teaching Personnel Development (Pusbang Tendik) to support achieving the number of participants having completed the induction program. Instead, the Teacher Induction Program for Beginners (PIGP) was carried out mostly by the Office of Education at districts and municipalities on their own, without any companion activities and coordination with Pusbang Tendik. 54 2) It took time for all participating teacher training colleges (LPTK) to support the Recognition of Prior Learning (RPL) Program as suggested by the Director General of Higher Education. This was mainly due to their unfamiliarity with the management process, particularly in terms of planning and implementation process which would provide academic funding. What would be done differently going forward? 1) Given the importance of empowering MGMPs/KKG as part of the development of quality teachers in carrying out their duties and careers, the program coordination functions should exist at the level of the Directorate General of Primary Education. If the coverage of the program will be extended to secondary education, of course, the Directorate General of Secondary Education will also be given the same task. 2) To increase motivation to participation in the managing program activities, it is recommended to apply reward systems; for those whose district/city of education offices perform well, management should reward them by increasing the number of block-grants on the basis of the KKG/MGMP involved. 3) Scholarship funds should not be given to lecturers, but to the colleges whose authorized in implementing the program; it is to increase the responsibility of the implementing agencies and to improve support for higher education in the implementation of the CPD. Notice that some LPTKs were not fully supporting the CPD mainly because they were not given the opportunity to perform program management. 4) Special attention given in coordinating the regulation-making about teacher quality improvement at district/city local government, to alleviate the fiscal burden of the central government in improving the quality of education through improving the teacher quality in the future. How can the WB as well as the Royal Netherlands support be improved? 1) The World Bank expedite the process of issuance of NOL (No Objection Letter) on various proposed use of funds without violating applicable regulations, that established the program more effective and efficient. Some activities cannot be carried out because the delay of NOL issuances. 2) The success of the central government organizes the government of districts and municipalities in developing regulations on teacher professional quality improvement indicates the growing importance to support efforts in developing governance network with local governments, allows for the bank to be financially involved. B. Center for Teaching Profession Development (Pusbangprodik) – Satker in charge of component 2 and 3 implementation 55 What worked well? 158. In implementing Component 2 and 3 of BERMUTU, there are six areas that worked well: (a) Comprehensive project requirements, e.g. not merely professional development but underlying legislation plus field trialing; (b) Mutually supporting components within a comprehensive project design, e.g. KKG/MGMP strengthening was concurrent with development of the in-service CPD) system (Component 2 and 3); (c) Rapid build-up of KKG/MGMP activity linked to study modules, coordinated with the CPD system and requirements; (d) Wide acceptability during trialing of the performance appraisal based on direct observation of teacher performance, resulting in increased willingness of principals to undertake regular teacher observation as well as increased willingness of teachers to question and reflect on their own experience; (e) Management of Components 2 and 3 by a single manager, permitting coordinated development of components; and (f) Regular and effective liaising between MOEC and Bank team leader and consultants, which allowed timely intervention to keep the components on target. 159. Supporting block grants from BERMUTU for the teachers working group have good impact on the teachers and for continuity for the program, it is in line with the regulation for teachers career development that every single teachers have the obligation doing Continuous Professional Development for teachers credit earning at least in teachers working group. Lesson learnt from BERMUTU activities are that teachers will have standard programs that can be developed based on teachers need, that teachers will not have any difficulty, and that the activities of teachers working group will be sustained because teachers need the activities in teacher working group for supporting their career development. What did not work well, and why? 160. Preparation of all stakeholders to implement the project took longer than anticipated, resulting in uncertainties at the school level (with subsequent weak implementation of objective appraisal and needs-based in-service support). In addition to that, there were conflicting messages between different departmental and ministerial initiatives and no clear and timely public statements. The reason of this situation included: (i) low operational priority in MOEC during and after the re-structuring, which involved the establishment of Board of Education, Human Resource Development and Education Quality Assurance (HRD & QA) Board, changes in key personnel as well as unexpected introduction of changes that required retrospective legislation (i.e. teacher competency testing system (UKG) and the 2013 Curriculum); and (ii) initial weak perception of BERMUTU within MoEC as preparation for enduring change in teacher performance management, resulting in poor inter- departmental and inter-ministerial coordination and public profile. Only State Ministry of National Development Planning appeared to understand the link with economic development planning and with wider change in civil service practice. 161. There were initial delay in finalizing designs for the CPD system as well as performance appraisal, in-service support, quality assurance of performance management, integration of management of professional performance with management of the career structure for all educational professionals. Some delays were also experienced in finalizing 56 revision of regulations related to annual teachers’ performance appraisal and CPD, as well as the teacher competencies test. 162. Implementation of RPL could be strengthened. The program study availabilities within the universities, which were selected to conduct the RPL program did not always fully meet teachers’ needs, i.e., where the local university may not offer the program study the teacher seeks. Supporting such local universities expand the program studies they offer is being considered now as government continues the implementation of the Teacher Law. 57 Annex 8. Comments of the Royal Netherlands Government 163. BERMUTU project is jointly financed by the Royal Netherlands Government (Trust Fund US$52 million) and GoI (US$57.06 million equivalent); the World Bank. The project is implemented by 28 working units of Ministry of Education and Culture with the Directorate of Teacher and Education Personnel Development as the secretariat of Project Coordinating Unit. Project Period is from 2007 to 2013. BERMUTU was designed to support the MoEC in operationalizing the Law. To what extent have the objectives been met / results achieved Main impact/success of the project 164. The achieving of the Project Development Objective is solid as evidenced by significant achievement of the project’s Key Performance Indicators.  Increased number of teachers meeting academic qualifications mandated by the Teacher Law. At the time of this report over 1.7 million (of total 2.9 million teachers) teachers have acquired the mandated 4-year college degree. This exceeds the final target of 1.4 million.  Improvement in the pedagogic skills of teachers: 214,380 teachers are participating in the 6,107 teacher, principal and supervisor professional working groups with the primary aim to improve their pedagogic competency. The appraisal of a random sample of classroom performance has validated that improvement is occurring.  Reduction in Teacher Absenteeism in BERMUTU Districts. Studies (by SMERU) of teacher absenteeism on randomly selected public primary schools in 2003 and 2008 using nationally representative samples of primary schools found the rates to be 19% and 14% respectively. Another national representative study is being conducted in collaboration with the ACDP, a program jointly funded by the European Union and AusAid. In the BERMUTU districts, MoEC conducted a study on selected schools in 20 districts in 2013 and found absenteeism rates of 6.6 percent in primary schools and 6.3 percent in junior secondary schools. What follow-up will be given and by whom (GoI, WB, other partners) to sustain the results of the project / TF after its ending. 165. As the Project was designed to help implementing the Teacher Law, all of the projects outputs are institutionalized/continued to be used by the Government. To mention some: the teacher relational data base; the policies of Recognition of Prior Learning to be used for teacher portfolio; induction of new teachers; teacher performance assessment and continuous professional development as a part of teacher professional career support system. 58 Annex 9. List of Supporting Documents Bank preparation documents Bjork, Christopher (2006). “Indonesian Teaching Culture, Classroom Realities, and the Potential for Reform.” Background Paper, August. Chen D., Willeneberg T., Prasad M., Bjork C., Gallos F., Kesuma R., and Grintings A. (2006), Momitoring and evaluation and student learning, August. Gallos F., Kraft R., Stevenson R., Bjork C., Iskandar S., Rochsantiningsih D., and Rosyidi U. (2006), Certification, professional development support, in-service training and upgrading, pilto studies design, distance learning, August. Gallos, Florenda (with many government counterparts) (2006). “A Report on the Pilot Video Study on Mathematics Teaching in Junior Secondary Education in Indonesia.” August. Kluyskens, Jups and Muhammad Firdaus (2006). “Teacher Management in the Context of the Civil Service.” Background Paper, December. Kraft R., Smith C., Rawlinson R. and Wibowo U.H, Policy and standard in teaching quality, August 2006. Kraft, Richard J. (2006). “Preparing Pre- and In-Service Teachers for Indonesian Schools.” Background Paper, August. Kraft, Richard J. (2006). “University-Based Teacher Education Reform: Accreditation, Program Improvement, Open and Distance Education, and Scholarships.” Background paper, December. Joni, T. Raka and Udik Budi Wibowo (2006). “Teacher Management: A Review of Regulations, Policies and Practices” Background Paper, May. Leung, Frederick (2006). “Design of the Classroom Video Study” October. McMahon, Walter W. (2005). “Improving the Quality of Education: Financing Improved Teacher Quality and Efficient Deployment.” Background Paper, December. Rawlinson R., Rogers H., Chen D., Paqueo V., Prasad M., Ragatz A., and Suharwoto G., Incentives structure/economic analysis, August 2006. Peck, Frank (2005). “Report on Study of Teacher Employment and Deployment: Organization and Management Component.” Background Paper, December. Ragatz, Andy (2006) Teacher Working Groups in Indonesia: A Study of the Current Situation and Opportunities for Increased Effectiveness, (Note: Initial study results for design 59 inputs were available December 2006; final paper published later) Rawlinson, Ralph (2006). “Reforming Teacher Accountability and Incentive Systems for Performance and Career Advancement.” Background Paper, August Samani, Muchlas (2006). “The Implementation of Teachers Certification in Indonesia: Issues, Risks and Alternative Solutions.” Background Paper, February. Stevenson, Ritchie (2006a). “A Model for Teacher Certification under the New Teacher Law.” Background Paper, February. Stevenson, Ritchie (2006b). “Pilot Study for Teacher Certification and Upgrading: Development of Model and Plan for Implementation, Background Paper.” March. Stevenson, Ritchie (2006c). “Pilot Study for Teacher Employment and Deployment, and Teacher Certification and Quality Improvement: Progress and Plans for Implementation of Teacher Certification Using the Teacher Law Regulations of May 2006.” Background Paper, June. Stevenson, Ritchie (2006d). “Teacher Certification and Assessment in Indonesia, Background Paper.” August 2006 Stevenson, Ritchie (2005). “Report on Study of Teacher Employment and Deployment: Teacher Competency Component.” Background Paper, December. UNESCO-UIS/OECD 2205 Education Trends in Perspective and World Bank staff estimates. Willenberg, Tony (2006). “Systems Analysis of the Teacher Management Information Systems of the Directorate General of Quality Improvement of Teacher and Education Personnel.” Background Paper, August. World Bank. Country Assistant Strategy (Indonesia), 2003 (Report No. 27108- IND, 29 October 2003) Chaudhury, Nazmul; Hammer, Jeffrey; Kremer, Michael; Muralidharan, Karthik; Rogers, F. Halsey. 2006. “Missing in Action: Teacher and Health Worker Absence in Developing Countries”. The Journal of Economic Perspectives, Volume 20 , Number 1, Winter 2006, pp. 91-116(26) World Bank. Statements of mission objectives, Aide-Memoires and Back-to-Office Reports from the identification mission carried out in August 2006 up to the appraisal mission carried out April 2007. World Bank. Project Concept Note and Minutes of the Review Meeting dated August 24, 2006. 60 World Bank. Project Appraisal Document (PAD) for the Indonesia Better Education Through Reformed Management and Universal Teacher Upgrading Project (BERMUTU) Report No. 39299, May 30, 2007. World Bank. Agreed Minutes of Negotiations dated May 23, 2007. Bank project implementation documents World Bank (IDA). Indonesia BERMUTU Financial Agreement Project No Cr.43490-IND, November 7, 2007 World Bank (IBRD). Indonesia BERMUTU Loan Agreement Project Ln-No7476 dated November 7, 2007. Grant Agreement TF090794 dated November 7, 2007. Trust Fund Administration Agreement between the Government of Netherlands and the International Bank for Reconstruction and Development and the International Development Association concerning the Netherlands Trust Fund for Indonesia (Trust Fund No TF070812) August 27, 2007. Amendments to the Loan and/or Credit Agreements dated November 23, 2011 and October 5, 2012. World Bank. Statements of Mission Objectives, Aide-Memoires, Back-to-Office Reports, Financial Management Reports, Procurement Review Reports, Implementation Status and Results Reports (ISRs) and Audit Reports of all the supervision missions recorded in the Data Sheet of this ICR and in the Project’s electronic files from March 2008 to December 2013. Bank and Borrower other project implementation and post-implementation-related documents Chang, Mae Chu, Sheldon Shaeffer, Samer Al-Samarrai, Andrew B. Ragatz, Joppe de Ree, and Ritchie Stevenson. (2014). Teacher Reform in Indonesia: The Role of Politics and Evidence in Policy Making. Directions in Development. Washington, DC: World Bank. Hanushek, Eric A., (2011). "The economic value of higher teacher quality," Economics of Education Review, Elsevier, vol. 30(3), pages 466-479, June. Joppe de Ree, Samer Al-Samarrai and Susiana Iskandar (2012). Teacher Certification in Indonesia: a Doubling of Pay or a Way to Improve Learning? Policy Brief, World Bank, Jakarta. 61 OECD (2014) “How much are teachers paid and how much does it matter?”, Education Indicators in Focus. April. Puslitjak (2013). Analysis of Teacher Absenteeism Study. Study Report, Research and Development Board, MoEC, Jakarta Puslitjak (2013) LPTK (Pre-service Teacher Training) impact evaluation study. Study Report, Research and Development Board, MoEC, Jakarta Rahmawati, Puspendik (2011). “BERMUTU KKG/MGMP Study report”. Background paper, Education Assessment Center, Jakarta, Indonesia. World Bank, Country Partnership Strategy (Report No. 752906-ID dated December 13, 2012). World Bank. 2013. “Spending More or Spending Better: Improving Education Financing in Indonesia”, World Bank, Jakarta, Indonesia 62 MAP 63