73264 Teacher Certification in Indonesia: a Doubling of Pay, or a Way to Improve Learning? Policy Brief October 2012 KEMENTERIAN PENDIDIKAN DAN KEBUDAYAAN certi cation may crowd out other quality investments in the system and limit resources available to expand pre- and post-basic education. The Government of Indonesia and the World Bank are in the process of evaluating the impact of the teacher certi cation program. This brief provides initial results of this evaluation exercise by analyzing the direct e ects that certi cation may have by changing the motivation and behavior of teachers who become Photo: Fitrawardi Faisal certi ed and receive the professional allowance. The ndings show that certi cation has led to some positive changes in teacher behavior but these changes have not translated into improved student learning. Certi cation may have other impacts on student learning that go beyond the direct e ect presented in this brief. It provides nancial incentives for unquali ed teachers to upgrade their academic quali cations, and for high-ability senior secondary school Origins of Teacher Certification in Indonesia graduates to opt for a career in teaching. Both mechanisms should improve the quality of the teaching force in the medium to longer term. This brief does not provide conclusive evidence on these The 2005 Teacher Law aimed to improve the quality of the mechanisms. Instead we provide some suggestive evidence that the Indonesian education system by addressing the weaknesses in process of academic upgrading of in-service teachers might not raise teacher competencies, their low motivation and poor levels of pay. the quality of teachers as much as anticipated. More research into A central component of the law was the teacher certi cation these indirect mechanisms however is clearly warranted. program. This program was designed to certify teachers who demonstrated minimum levels of teaching competency. Strong The ndings presented in this brief are based on two rounds of data incentives were also introduced which entitled certi ed teachers to a collection undertaken in November 2009 and April 2011. A nal professional allowance equivalent to their basic pay. round of data was collected in April 2012 and results from an analysis of all three rounds are expected in 2013. Since 2005, approximately one million teachers have been certi ed. About one-third was certi ed after a successful assessment of a portfolio of past experience and training. The How Could Certification Improve Quality? remaining two-thirds were certi ed after passing a 90 hour training course (PLPG). The program aims to certify all teachers by 2015 and Teacher certi cation is expected to raise the standard of teaching requires a large number of teachers to enroll and complete a by introducing incentives and sanctions to ensure that teachers university bachelor’s degree. The Open University (UT) has been the have a minimum level of competency. There are three main ways in traditional supplier of distance learning courses for upgrading which certi cation can lead to improvements in the quality of teachers’ knowledge and skills and, as of 2011, close to 500,000 teaching: teachers were actively enrolled to upgrade academic quali cations. • The behavioral channel. Once teachers are certi ed they are eligible to receive an additional allowance equivalent to their The nancial implications of the certi cation program are basic pay. This large increase in salary can improve the motivation enormous. If all three million existing teachers are certi ed the and self esteem of existing teachers, which in turn could improve regulation would add approximately USD 5-7 billion each year to the student learning outcomes. Teachers may for example refuse teacher salary bill. Simple budget projections suggest that second jobs, arrive on time or prepare better for class. Many 1 existing teachers qualify for certi cation because they have a Figure 1: The proportion of teachers with a university bachelor’s bachelor’s degree, high rank (rank IV) in the civil service, or degree has increased as a result of certi cation. because they are very senior. No additional academic upgrading Percentage of primary and secondary school teachers holding a four is required for these teachers so that, for them, the impact of year university bachelor’s degree, 2006-2010 certi cation is likely to be restricted to a change in behavior. • The upgrading channel. In-service teachers who do not qualify for certi cation can choose to obtain a bachelor’s degree to meet 100% the certi cation requirements. The professional allowance 90% 80% 74% 76% provides a nancial incentive for these teachers to undertake the 70% 62% necessary academic upgrading. Raising the academic Percentage 60% quali cations of teachers in this way is expected to improve the 50% quality of teaching and student learning. The upgrading channel 40% could become increasingly e ective if upgrading programs are 30% 24% 27% carefully designed and monitored, and if competency testing, a 20% 17% 10% pre-requisite for participating in the certi cation program, is 0% adequately enforced. Both are needed to ensure that teachers Primary Junior Secondary who obtain a bachelor’s degree also have the required 2006 2008 2010 competencies to teach. • The attraction channel. The prospect of a higher salary has the potential to attract brighter high school graduates into teacher training colleges across Indonesia. Certi cation may only impact Source: NUPTK 2006, 2008, 2010 education quality through this channel in the long term when higher ability students graduate and enter the teaching force. The data collected for the study however shows that a bachelor’s Attracting higher ability students into the teaching force degree is a poor proxy for some professional competencies [see however will also be determined by the likelihood that new box 1 for a description of the data we use throughout the graduates are able to get jobs as teachers. Given the current remainder of this brief]. Primary school teachers perform poorly on oversupply of teachers in Indonesia this is unlikely to be average on the subject matter tests and those with a university automatic and may well dampen the incentive e ects associated bachelor’s degree fare only marginally better than teachers with with higher pay. The Government, therefore, should control the lower level quali cations (Figure 2)1. The evidence suggests that, for increased demand and make sure that the system does not primary school teachers at least, a bachelor’s degree has not been an produce too many new teachers. e ective criterion for selecting high quality teachers into the certi cation program. The Intentions and the Reality of Certification Figure 2: Teachers in general have di culties with the subject matter tests administered for the study and teachers with a Since 2005, teachers have been selected into the certi cation bachelor’s degree score only marginally better program based on their academic quali cations, seniority, or rank in the civil service. These three conditions are used as proxies for the Test scores (percentage correct answers) of teachers with, and teachers four teacher competencies spelled out in the law: pedagogical, without a four year post secondary degree. personal, social and professional competencies. Before 2012, the certi cation program did not select teachers by explicitly evaluating these competencies through rigorous testing. The use of proxy indicators of teacher competency turns out to be important in 100% explaining the impact of certi cation on teacher quality. 80% % Question Correct The number of existing teachers with a bachelor’s degree has 60% increased in recent years as a result of a degree being included in 35% 31% 40% the certi cation criteria (Figure 1). Between 2006 and 2010, the proportion of teachers with a university bachelor’s degree has 20% increased from 17 percent to 27 percent in primary schools and from 0% 62 percent to 76 percent in junior secondary schools. Moreover, Degree No Degree approximately 500,000 existing teachers are currently undertaking courses to obtain a bachelor’s degree. 1 Note that this di erence, however small, is statistically signi cant at the 1% level. 2 Teachers who upgrade their academic quali cations are expected to improve their professional capacities in the process. Most The Impact of Doubling Pay on Performance primary school teachers with a bachelor’s degree have acquired their The certi cation process includes signi cant improvements in the degrees through sequential upgrading. The rst wave of upgrading pay of teachers. To what extent has the increase in pay for certi ed happened in the 1990s, when new regulations prescribed that teachers changed their behavior and has this had any e ect on primary school teachers were required to have a two-year post student learning? secondary diploma. Primary school teachers started upgrading once more when a bachelor’s degree became a criterion for certi cation. The doubling of pay means increased recognition for the work An explorative analysis of the recent teacher assessments UKA2 and teachers do and less pressure to take on additional paid work to UKG3 found that teachers who acquired a bachelor’s degree through support the needs of the household. Increased pay may also sequential upgrading scored lower on the tests than teachers who improve motivation and lead to teachers arriving at school on time, obtained a bachelor’s degree through a standard four-year academic preparing better for their teaching and being more e ective training. It is vital therefore to closely monitor the learning gains that generally. teachers make in the academic upgrading process. If applicable, policies should be designed to ensure the quality of the upgrading The Government of Indonesia in partnership with the World Bank is programs. undertaking a rigorous impact evaluation study to assess the causal e ects of certi cation on teacher behavior and student learning It is likely that the weak relationship between academic outcomes. The study relies on the principles of a randomized quali cations and subject matter pro ciency also means that the controlled trial [see box 2 for details]. current process of academic upgrading has limited e ects on student learning. However, subject matter knowledge is only one of Before investigating the e ects of certi cation on student learning the required competencies to be an e ective teacher and upgrading outcomes we rst investigate what happens to a teacher after being has the potential to expand their competencies in other areas. For certi ed. We want to investigate the extent to which certi cation has example, their pedagogical skills may be greatly improved. Further made a di erence in the lives of teachers, and, subsequently, if there investigation into the e ects of academic upgrading on overall are indications that certi cation could have led to better teacher competencies is clearly warranted. performance in the classroom. Box 1: Data The analysis presented in this brief relies on a sample of 90,000 students and 3,000 core subject teachers. schools, and math, Students and teachers have biology, physics, been tested (and interviewed) Indonesian and three times over the past English teachers in years: in November 2009, in junior secondary April 2011 and in April 2012. schools. The 3,000 core subject The data set is unique and teachers that were can be used for a variety of tested were also purposes. In this brief we use interviewed. it to evaluate the e ects of certi cation at the school The subject level. However, because we The 90,000 students and 3,000 core knowledge tests used in this study, both track students and their subject teachers were sampled for 240 for teachers and students, were teachers for three public primary and 120 public junior developed by the Government’s center consecutive years we can also secondary schools across 22 districts in for educational assessment (PUSPENDIK) use it to evaluate the broader Indonesia. They were administered a and generally include a math, science, impact of teachers while subject matter test. Core subject Indonesian and English (only for junior students progress through teachers are class teachers in primary secondary school) component. school. 2 The Ujian Kompetensi Awal or UKA was an explicit competency test held in 2012 for that year’s batch of teachers wishing to enter the certi cation process. 3 3 The Ujian Kompetensi Guru or UKG is an explicit competency test held in 2012 for all teachers who were certi ed before 2012. Figure 3: Certifying teachers and paying them double, decreases the likelihood of them having second jobs or nancial problems. The causal e ects of certi cation on teacher characteristics. The bars represent t-statistics associated with each indicator. Primary Jun. Sec. Being a Having Teaching Teaching Absence Problems school school member of second job hours sample hours other at least once �nancially teacher test teacher test teacher school school last week supporting score score working (self reported) household 5 group 4 3 threshold for statistical signi�cance 2 1 0 -1 -2 threshold for statistical signi�cance -3 -4 -5 27 percentage points less 38 percentage points less likely to have a second job likely to have �nancial problems The certi cation process has not led to improvements in the unlikely to have occurred purely by chance. The rst two bars subject knowledge of teachers. Figure 3 shows the causal e ects represent the e ects of certi cation on teacher test scores. The bars of certi cation on teacher characteristics. Bars pointing downward are well within the boundaries for statistical signi cance and show indicate a decrease and bars pointing upward indicate an increase in that the certi cation process itself, which includes the 90 hours of the indicator shown. When bars pass through the dashed line they PLPG training for a majority of teachers, has not led to measurable are statistically signi cant which means that the measured e ect is improvements in teacher’s subject knowledge. Figure 4: Certifying teachers and paying them double does not make them teach better, here measured by the e ects of certi cation on student learning gains. The causal e ects of certi cation on student learning. The bars represent t-statistics associated with each indicator. Primary school Junior Secondary Junior Secondary Junior Secondary Junior Secondary Mathematics Science Bahasa Indonesia English 3 threshold for statistical signi�cance 2 1 0 -1 -2 threshold for statistical signi�cance -3 4 Certi cation does seem to have reduced the proportion of teachers holding second jobs and reporting nancial di culties. The magnitudes of these e ects are substantial. Certi cation can cause a 27 percentage point decrease in the likelihood of having a second job and a 38 percentage point decrease in the likelihood of teachers facing nancial di culties. The doubling of pay — on average from about US$ 250 to US$ 500 per month — provides teachers with su cient income to reduce their need to take on second jobs. However, other areas of teacher behavior do not seem to have been a ected by certi cation. For Photo: Mudi Astuti example, no statistically signi cant e ect of teacher certi cation was found on membership of teacher working groups, the number of teaching hours (in the sample school or in other schools) and self-reported absence levels. Reductions in second job holdings caused by certi cation do not lead to improved student learning outcomes. The fact that of two comparable groups of teachers, of which one group is certi cation made a di erence in the livelihoods of teachers certi ed and the other group is not [see box 2 for details on the suggests that certi cation might also matter for harder to measure methodology]. The di erences between these groups are small and concepts such as classroom preparation and motivation. An not statistically signi cant. all-embracing way of evaluating whether this has been the case is to investigate student learning outcomes. Figure 4 shows that there is This shows that teachers are unable to translate increased no measurable effect of certification on student learning outcomes. motivation into improved performance in the classroom when their The bars represent the differences in learning outcomes of students core competencies are low. Box 2: Methodology Randomized controlled trials (RCT’s) have the explicit objective to evaluate the e ects of a program or intervention under minimal assumptions. RCT’s deal with a problem called controlled trial to control for this A causal comparison. As a result of the selection. so-called selection bias. The 360 intervention a group of teachers in sample schools were randomly treatment schools is certi ed only Selection problem. In our case we divided into two groups, treatment because they were granted preferential know that better educated and more and control. The random division access to the certi cation process. This experienced teachers are selected into into treatment and control groups group can be compared to a matching the certi cation program rst. ensures that both sets of schools are group in control schools who were not Comparing learning outcomes of similar, prior to the project’s granted the preferential access, but students of certi ed teachers with intervention. were otherwise similar. those of uncerti ed teachers, therefore, does not measure a true The project’s intervention. The It is these two groups that are causal e ect of certi cation. Instead it research project granted preferential e ectively compared when we present would measure the combined e ect of access to the certi cation process for the causal e ects of certi cation in certi cation and all the other reasons all quali ed teachers in treatment gure 3 and 4. In essence we estimate why certi ed teachers are di erent schools. In control schools this did the average causal e ect of certi cation from uncerti ed teachers. not happen and everything stayed on the teachers (and their students) business as usual. The pace at which who were granted preferential access Random division into treatment and teachers are certi ed in control to the certi cation process. In the control. Our analysis relies on the schools is consequently much lower, literature this concept is called a local principles of the randomized but not zero. average treatment e ect. 5 Conclusions and Recommendations e ects on student learning however needs to be further understood and reviewed, with special attention to the di erent ways of upgrading. Indonesia’s certi cation program has raised the income levels of teachers and made the teaching profession signi cantly more The policy brief shows that the academic criteria used to certify attractive. However, di erences between the design and teachers have not guaranteed minimum teacher competency implementation of the program have limited the impact of levels. Many teachers with university bachelor’s degrees have certi cation on the much needed improvements in teacher quality. di culties with the competency tests that were administered for this study. This is particularly worrying given that the tests were designed The main evidence we provide is on the direct e ects of certi cation to measure competencies in terms of subject knowledge considered via the behavioral channel: does certifying teachers and paying them necessary for e ective teaching. more make them teach better? The government has recognized the need to improve the criteria Although certi cation has led to some changes in teacher behavior, used for certi cation. The Ministry of Education and Culture has potential increases in motivation have not translated into improved introduced changes to the certi cation program to reduce the student learning outcomes. reliance on academic quali cations as the main criteria for certi cation. As of 2012, explicit competency testing of the 285,000 In the medium term, we expect improvements in student learning teachers wishing to enter the certi cation program was introduced outcomes to occur as currently unquali ed teachers upgrade their (Ujian Kompetensi Awal or UKA). Approximately 15% of these teachers academic quali cations. The analysis provides some evidence that did not meet the pass score of 30 out of 100 and will need to indicates that the e ects of academic upgrading are likely to be undertake retraining before being assessed again for certi cation. weaker than anticipated. The process of academic upgrading and its Competency tests like the UKA are crucial to the functioning of the system and should be used to restrict entry into the certi cation program to candidates who demonstrate the necessary attributes of an e ective teacher. Preliminary results from the UKA showed that many teachers performed relatively poorly and cut-o points for passing may have been set too low. More detailed analysis of the results and its ability to identify competent teachers is clearly warranted. The teacher certi cation program has the potential to substantially improve student learning achievement across Indonesia. Turning that potential into reality will require changes to the existing program to ensure that only competent teachers are certi ed. The recent changes in the program have been in this direction and further analytical work needs to be done to assess whether these have gone Photo: Khairullah far enough in raising the quality of education in Indonesian schools. Printed on recycle paper Researched and prepared by Joppe De Ree, Samer Al-Samarrai and Susiana Iskandar, World Bank. Preparation of this document received partial funding from the Government of the Netherlands under the supervision of the World Bank. The ndings, interpretations, and conclusions expressed in this paper do not necessarily re ect the views of the Government of Indonesia or the Government of the Netherlands. For more information, please contact Mae Chu Chang, mchang@worldbank.org or Megha Kapoor, mkapoor1@worldbank.org. Human Development Sector, World Bank O ce Indonesia Stock Exchange Building, Tower 2, 12th Floor Jl. Jenderal Sudirman Kav. 52 – 53 Phone: (021) 5299 3000 Fax: (021) 5299 3111 Website: www.worldbank.org/id/education