The Use of Video in Teacher Professional Development Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia The Word Bank Office Jakarta Indonesia Stock Exchange Building, Tower II, 12th floor Jl. Jend. Sudirman Kav. 52-53 Jakarta 12190 Tel : (+6221) 5299-3000 Fax : (+6221) 5299-3111 http://www.worldbank.org/id Published in December 2015 The Use of Video in Teacher Professional Development: Design, Implementation and Impact Evaluation of an Innovative In-Service Course for Mathematics Teachers in Indonesia is a product of the staff of the World Bank. The findings, interpretation, and conclusions expressed herein do not necessarily reflect the view of the World Bank, its board of Executive Directors, officers or any of its member countries. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map of this work do no imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The work is funded by the Government of the Kingdom of the Netherlands under the Dutch Education Support Program (DESP) Trust Fund. DESP provides support to the Government of Indonesia through the World Bank for the purpose of developing policies, studies, and programs that help the government achieve its education strategic plan. The Use of Video in Teacher Professional Development Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Content Acknowledgements vi Preface vii 1 Background 1 2 Purpose, Research Questions and Outcomes of the Study 3 2.1 RESEARCH QUESTIONS 4 2.1.1 Teacher Working Group Component 4 2.1.2 Online component 4 2.2 OUTCOMES 5 3 Scope and Methodology 6 3.1 APPROACH 6 3.1.1 Part 1:Video data analysis 6 3.1.2 Part 2: Development of professional development vignettes 6 3.1.3 Part 3: Development of working group program 6 3.1.4 Part 4: Development of Online program 7 3.1.5 Part 5: Evaluation 7 3.1.6 Part 6: Future Development 7 3.2 CONCEPTUAL FRAMEWORK OF BELIEFS-KNOWLEDGE-PRACTICES 8 3.3 SAMPLE SELECTION 9 3.3.1 Targeted Selection Stage for Provinces and Districts 9 3.3.2 Stratification and Randomized Selection Stage for Teacher Working Groups 10 4 Course Design 11 4.1 VIDEO STUDY RESULTS USED AS A BASIS FOR AREAS FOR IMPROVEMENT 12 4.2 FOUNDATIONAL CONCEPTS OF PROFESSIONAL DEVELOPMENT 12 4.2.1 Characteristics of Good Professional Development Programs 12 4.2.2 Principles of Adult Learning 13 4.3 TEACHER WORKING GROUPS FOR DELIVERY OF PROFESSIONAL DEVELOPMENT 13 4.3.1 What is a Teacher Working Group? 14 4.4 COURSE DESIGN PRINCIPLES 16 4.4.1 In-On-In Approach 16 4.4.2 ELPSA Framework 16 4.4.3 Incorporation of Videos into the Course 17 4.4.4 Facilitators as an Integral Part of Session Activities 17 5 Evaluation Instruments 18 5.1 TEACHER SURVEY WITH MATHEMATICAL BELIEFS 18 5.2 TEACHER PRE AND POST TESTS ON SUBJECT AND PEDAGOGICAL KNOWLEDGE 18 5.3 STUDENT PRE AND POST PERCEPTIONS SURVEY ON CLASSROOM PRACTICES 19 5.4 WHY NO STUDENT TESTING? 19 6 Implementation 20 6.1 FUNDING OF ACTIVITIES 20 6.2 FACEBOOK GROUP FOR PARTICIPANTS 20 6.3 SEQUENTIAL STEPS OF THE PILOT ACTIVITIES 20 6.3.1 Pre-meetings with District Officials 20 6.4 INITIAL FACILITATOR WORKSHOP 21 6.5 CONDUCTING SESSIONS 21 6.5.1 Pre testing and data gathering in the first session 21 6.5.2 Final Session Post testing and Demonstration of Self-shot Videos 21 6.5.3 Final Session Post testing and Demonstration of Self-filmed Videos 21 6.5.4 Follow-up Workshops and Focus Group Sessions 21 6.5.5 Evaluation Results 22 6.6 ATTENDANCE AND DELIVERABLES OF TREATMENT GROUP PARTICIPANTS 22 6.6.1 Attendance of Teachers in the Treatment Group 22 6.6.2 Attendance of Teachers in the Control Group 23 6.6.3 Attendance per Session 23 6.6.4 Deliverables 24 7 Evaluation of the Program 25 7.1 QUANTITATIVE ANALYSIS 25 7.1.1 BELIEFS: Pre-post results of the Mathematical Beliefs Statements 26 7.1.2 KNOWLEDGE: Pre test and post test results of the Teacher Assessment 30 7.1.3 PRACTICES: Pre-survey and post-survey results of Student Perceptions on Practices 32 7.1.4 General Conclusions on Quantitative Analysis 34 7.2 QUALITATIVE ANALYSIS 35 7.2.1 Observation of Session Activities 35 7.2.2 Interviews and Surveys of Facilitators 36 7.2.3 Interviews and Surveys of Teachers (participants) 38 7.3 FEEDBACK THROUGH WORKSHOPS WITH FOCUS GROUP DISCUSSIONS 39 8 Key Lessons Learned and Considerations 40 8.1 CONTENT 40 8.2 IMPLEMENTATION 41 8.2.1 Using Working Groups for the Delivery of CPD 41 8.2.2 Importance of Facilitators 41 8.2.3 Responsibility for the Provision of CPD 42 8.3 EFFECTIVENESS 42 8.4 SCALING UP AND SUSTAINABILITY 43 9 Wider Policy Implications 44 References 46 Appendices 47 APPENDIX 1: OUTLINE OF ACTIVITIES 47 APPENDIX 2: GENERAL PROFESSIONAL DEVELOPMENT CONCEPTS 48 APPENDIX 3: INSTRUMENTS 49 Teacher Survey – Beliefs and Practices Questions 49 Student Perception Survey 52 Observation Instrument 53 Teacher Post-Session Survey 56 Facilitator Post-Session Interview 57 APPENDIX 4: SUMMARY OF ONLINE COURSE CONDUCTED BY P4TK MATHEMATICS 58 Implementation 59 Discussion on the Findings 61 Recommendations 63 APPENDIX 5: REGRESSION RESULTS 65 Regressions for Teacher Mathematics Competency Testing 65 Regressions for Teacher Mathematics Competency Testing 76 Regressions for Student Perceptions Survey 77 FIGURES Figure 1 : Components and Processes 5 Figure 2 : Conceptual Model of Beliefs-Knowledge-Practices 8 Figure 3 : Theory of Results Chain 11 Figure 4 : Example of ELPSA for the topic of right triangles 17 Figure 5 : Attendance Rate of Teachers Used in Analysis 23 Figure 6 : Distribution of pre test scores in the teacher assessment 30 Figure 7 : Example item with open-ended questioning 31 Figure 8 : Change in response from pre to post survey 34 Figure 9 : Participant Mapping 62 TABLES Table 1 : Survey results for teacher beliefs on the teaching of mathematics 27 Table 2 : Survey results for teaching practices used in learning activities 28 Table 3 : Survey responses for statements on The Process of Learning Mathematics 29 Table 4 : Post test average scores by items in terms of their relationship to the CPD course content 31 Table 5 : Statements given to students and whether there was a statistically significant change 33 Table 6 : Summary of Online Geometry Course Survey 60 Acknowledgements This report is the result of a joint program between the World Bank and the Ministry of Education and Culture (MOEC). The report was produced by Andrew B. Ragatz and Susie Sugiarti under the overall guidance of Susiana Iskandar, but it is just the final piece of an extensive effort to develop, implement and evaluate the teacher professional development course. The architects of the course are Tom Lowrie and Sitti Maesuri Pattahudin (formerly of Charles Sturt University and currently in University of Canberra). They put their hearts into the conceptual design, selection of video, development of the course materials and evaluation instruments and in conducting the facilitator workshop. A great deal of the material within this report comes directly from their concept note and design documents. The implementation was led by Susie Sugiarti and Tukiman Tarunasayoga, with a core team that consisted of senior mathematics instructors from the Center for Development and Empowerment of Mathematics Teacher and Education Personnel (P4TK), the Educational Quality Assurance Institution (LPMP), and some junior secondary schools. Team members included Adi Wijaya, Budiharjo, Erwin Roosilawati, Al Krismanto, Rachmadi Widdiharto, Suhendro, Sri Yuniati, Suwarkono, Sri Kurnianingsih, Ade Sunawan, and Isfarudi. The Ministry of Education and Culture was heavily involved in all stages. Abi Sujak, Unifa Rosyidi and Dian Wahyuni (of Pusat Pengembangan Profesi Pendidik, Pusbangprodik) not only provided leadership and guidance, but also continually participated in and supported the activities. Professor Widodo, head of P4TK, went above and beyond to support the activities and even established a core team within P4TK to develop an online version of the course. The online course received extensive technical support from Petra Wiyakti Bodrogini and Winastwan Gora Swajati. The report benefited from the excellent comments and advice of several World Bank experts, including Barbara Bruns (peer reviewer), Helen Craig (peer reviewer), Muna Salih Meky (peer reviewer), Susiana Iskandar, Ratna Kesuma, Samer Al-Samarrai and Cristobal Ridao-Cano, as well as production support from Santi Sugiarti Santobri. But the biggest contributors to this effort were the over 400 facilitators and teachers who participated in the program. Their energy and enthusiasm drove the program. Their constant feedback and insights helped to understand what worked, what needed improvement and how to best move forward in supporting Indonesia’s teachers through professional development and support. Their courage to film their own lessons and share their experiences were essential to the success of the course. Preface In-service training of teachers is notoriously difficult and there are often questions about the extent to which training leads to actual changes in teaching practices. The following report documents the development and piloting of an innovative professional development course utilizing video. Video was first employed as a tool for discussion through clips of actual classroom situations. By the end of the course, teachers had the opportunity to film their own lessons and use the video to share as well as reflect on their own teaching practices. The course took place over one semester and was conducted in Indonesia’s teacher working groups. These groups are typically made up of 20-30 teachers from clusters of nearby schools. At the secondary level the groups are by subject; in the case of this pilot the participants were all mathematics teachers. The fact that the course involved six sessions over the semester allowed for the use of an approach where teachers learned practical material that could be directly applied in their own classrooms. They then applied what they learned in their own classrooms. In the subsequent working group session they shared their experiences and reflected on what worked, what didn’t and how they could improve the use of the practice. This cycle was specifically intended to ensure what teachers learned would actually make it to the classroom and encourage teachers to refine and continue the new practices. So did the course lead to changes? The pilot was evaluated through both quantitative and qualitative methods. The quantitative component measured changes in three areas: (1) teachers’ beliefs on teaching and learning, (2) their subject matter knowledge and (3) students’ perceptions of practices used by their teachers. The results indicate changes in teacher beliefs and an improvement in subject knowledge on items related to concepts included in the course. Students also perceived changes in the practices used by the teachers. The quantitative component captured perceptions, opinions and understandings of the teachers through interviews and surveys after each session as well as through workshops held after the course. The sessions were also observed by experts to determine how the course was actually implemented. Through this process there was a much better understanding of what worked, what didn’t and how the course could be improved. And after the pilot? The course was embraced by the Ministry of Education and Culture for scaling up. The Ministry’s mathematics teacher training center (P4TK) converted into an online course. It is serving as a model for development of further in-service courses in mathematics as well as in other subject areas. The University of Canberra is now expanding the work by creating new courses through a project supported by Australia’s Government Partnerships for Development (DPFD) program and being conducted in the region of Nusa Tenggara Barat (NTB). viii The Use of Video in Teacher Professional Development 1 Background The Ministry of Education and Culture (MOEC) and the World Bank jointly conducted a large-scale video study to gain a better understanding of what takes place in the classroom, particularly in terms of the practices involved in the teaching of mathematics and the relationship of these practices with student learning outcomes. The study involved the production of more than 600 hours of video footage of 205 eighth grade mathematics teachers across Indonesia while these teachers were engaged in their day-to-day classroom activities. The study was conducted in two phases, in 2007 and 2011 respectively, with the produced video footage forming the basis for detailed coding and analysis. This study provided many important insights into the teaching practices of mathematics teachers, identifying both relative strengths and areas for improvement. The study focused on five key dimensions that frame classroom practices in the teaching of mathematics, these being: (i) the structure of lessons; (ii) the content of lessons; (iii) the actions of participants; (iv) instructional practices; and (v) classroom climate and resources. Key areas of teaching practice were explored in depth, including the nature and quality of teacher-student interaction; the quality and effectiveness of the language used in instruction; teachers’ management of students’ questions; teachers’ management of student misconceptions; the time teachers spent on different tasks; and the manner in which they combined different techniques and approaches in the instruction of their students. Mathematical problems were explored in terms of the problem type, problem approach and level of complexity. The study also assessed teachers’ beliefs and the level of teachers’ knowledge to determine the relationship of these factors with the teaching practices they used and with student learning outcomes. The findings of this study highlighted the general need to strengthen both teacher subject and pedagogical knowledge, identifying specific areas where teaching practices should be improved. The study has become particularly relevant in the context of the promotion of particular practices (such as discovery learning, problem- based teaching and higher order thinking) through the 2013 curriculum reform process.1 The lessons learnt through the study formed the basis for the development of sessions to build the capacities of teachers through Continuous Professional Development (CPD) activities. The video footage produced through the study captured day-to-day teaching activities in their natural setting, providing useful real-life examples of what teachers do and how they behave in their classrooms. This unique and invaluable data provided a solid basis for the development of materials to build the capacities of teachers. The CPD course was developed on the basis of an understanding that educational activities are conducted in the context of a specific culture. Thus, the design 1 As of this writing, the 2013 curriculum is being evaluated to determine whether it should be continued. It was rolled out to 7th grade classrooms in 6,221 schools in 2013 and another 211,779 in 2014. Now it is being rolled back in the second wave of schools, but continued in the first wave for piloting and evaluation. Regardless of what ultimately happens, the video study results and support that teachers require for improving their practices is relevant. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 1 of the course and pilot was informed by the findings of substantial research into the Indonesian education system. The design process involved close cooperation with classroom teachers and other educationalists to ensure that the teacher development activities and change process are sustainable and culturally respectful. Elsewhere, it has been argued that the system of pedagogy applied in schools reflect the important a priori choices of the particular culture or society in which those schools are located (Matsumoto and Juang, 2013). In this context, Matsumoto and Juang (2013, pp. 82-83) stated that “regardless of the way education occurs, the choices a society and culture make concerning its structure, organization, planning, and implementation all encourage and reinforce a certain view of culture”. This study aimed to establish opportunities for professional development through shared, culturally rich learning experiences. The challenges: On average, Indonesian teachers have low levels of knowledge of mathematics. This has an impact both on what they teach and the manner in which they teach. In general, Indonesian teachers spend a considerable amount of time on introducing new concepts. However, they spend less time on practicing and reinforcing these concepts and very little time on assessment and reflection. Their approach to teaching is very teacher-centered. They rarely situate mathematics within real-life situations, nor do they use routine problems to demonstrate concepts. Their use of the type of higher order thinking usually associated with solving non-routine problems thus tends to be minimal. Teachers often use language that does not accurately convey mathematical concepts. Indonesian teachers generally use questioning mearly to test if a student knows the answer to a specific problem, rather than as a means of initiating discussion or to seeking information related to student thinking. The opportunities: With its ability to capture the complex interactions involved in classroom practice, video is a powerful medium for promoting discussion and discourse. The use of video footage encourages teachers to engage in comparative reflection, descriptive reflection, and critical reflection. It also creates an opportunity to analyze teaching from different perspectives, including from the perspective of both the teacher and the student. The use of video footage enables teachers to gain new insights into their own teaching practice by promoting reflection on the challenges faced by their colleagues. In turn, this enables them to formulate new ideas, activities and approaches that they can apply in their own practice (Zhang et al., 2011). In order to overcome the challenges faced by Indonesian teachers, higher levels of professional awareness need to be developed to encourage critical thinking and personal theorizing. 2 The Use of Video in Teacher Professional Development 2 Purpose, Research Questions and Outcomes of the Study The central goal of the study was to determine the level of success of a professional development initiative aimed at enhancing teaching and learning opportunities for teacher educators, classroom teachers and their students in Indonesian schools. The professional development initiative was designed around the use of classroom video data, which was used as a catalyst to promote the use of new pedagogical practices. The video data, and related learning framework and tools, were designed to encourage classroom teachers to reflect more deeply on their practices. This involved the utilization of teacher working groups to develop new levels of professional engagement. The main objective was to conduct a comprehensive analysis of a pilot that will determine the effectiveness of a new approach to promote new and contextually innovative teaching practices. Although this is a pilot, it should not be viewed as a stand-alone product. Rather, it must be seen within a larger context. Two major long-term goals include: 1. The development of a professional development program to promote new and contextually innovative teaching practices; and 2. The development of a dynamic framework for pedagogy that can be used by policy makers and the teaching profession to enable the identification and description of effective practices. The professional development program was posited within video learning artifacts. Episodes from the TIMMS video footage were purposefully selected to serve as vignettes for inclusion in professional development materials. These vignettes form the foundation for the creation of detailed lesson units and learning opportunities intended to enhance teachers’ mathematics discipline knowledge and pedagogical content knowledge. The program includes a series of scaffolded professional resources that model and depict effective practices. In addition, the program encourages teachers to reflect more deeply on their own teaching practices by encouraging participants to develop their own personal theories of teaching and to establish professional support networks that promote reflection and professional development. The dynamic pedagogies framework was utilized to provide a common structure for teachers to develop teaching strategies and to facilitate opportunities for learning. Importantly, the framework was intended to support teachers in the development of learning activities that encourage a deep understanding of important concepts, that value diversity and that promote more interactive, engaged learning. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 3 2.1 RESEARCH QUESTIONS The study was divided into two components, each addressing specific research questions. The pilot evaluated the implementation of the program to determine how well the teachers utilized it and to assess the effectiveness of both face-to-face and online delivery methods. 2.1.1 Teacher Working Group Component In the environment of the teacher working groups (MGMP), the study attempted to address five interrelated research questions to determine the professional and cultural needs of classroom teachers. In addition, these research questions sought to determine the impact that the professional development program had in terms of a number of measures: 1. Do the video lessons enable classroom teachers to engage in critical reflection? If so, does this reflection affect their individual professional practice? 2. To what extent do teachers’ beliefs on matters related to teaching and learning change and evolve as a result of the program? 3. To what extent do teachers’ knowledge of their subject and general pedagogical knowledge develop as a result of the program? 4. Do teachers utilize different questioning techniques to assess student performance and to promote student engagement in mathematics lessons as a result of the program? 5. Do students’ attitudes to and beliefs regarding their mathematics lessons reflect changes in classroom practice? 6. In what capacity do teachers assess and examine their own personal theories and practices as a result of video artefacts in which they are personally involved? 2.1.2 Online component This component of the project utilized the same five research questions posed in the Teacher Working Group component. In addition, the following research questions were posed in order to evaluate the program’s applicability in a digital communication environment. 1. Does the manner in which teachers engage with and reflect on their practices vary between face-to-face and digital communication environments? 2. Do teachers’ shared experiences and critical reflections vary between the two communication environments? Originally, it was envisioned that the study team would develop the online component. However, following an expression of interest by the Ministry’s mathematics teacher training center (P4TK Matematika) in incorporating the course into their program, a better solution was devised. P4TK Matematika had already engaged in some pioneering work to develop e-learning systems and therefore had a good foundational capacity to convert the course for use in an online environment. The study team provided technical assistance to adapt the course for use in an online environment and to ensure the important principles and concepts of the course (including the use of practical activities, the in-on-in of practicing methods, interaction amongst teachers, and reflection on personal practice) were retained. However, while the study team provided extensive technical assistance, P4TK Matematika led the process and performed the evaluation. In addition to the research questions above, the following questions were added for the online course. 4 The Use of Video in Teacher Professional Development Figure 1: Components and Processes Teacher Working Groups Consultation with stakeholders to develop 10 Working Groups in Treatment framework including (150-200 teachers) • Ministry of Content Video Pedagogy Education and Culture Evaluation • Teacher Capture through analysis education from lessons learned of research universities and modify online questions • Professional environment associations • Unions • Mathematics education experts Online Environment 150 teachers Content Video Pedagogy construction of construction of dynamic professional development pedagogies framework 2.2 OUTCOMES The research design of the project involved the analysis of the research questions described above using a mixed methods approach that involve both qualitative and quantitative methodologies. The major outcome of the project was developed in concert with the research design and consequently was constructed in an ongoing manner. Multiple instruments were designed to gather key information on teachers’ mathematical beliefs, Mathematical Knowledge in Teaching (MKiT), and teaching practices, with the goal of determining the extent to which these changed as a result of the pilot. Members of the team also closely observed the MGMP activities to understand how the course was used in practice. In addition, the team conducted interviews and surveys after each session to gain an understanding of the perspectives and opinions of the facilitators and teachers on the content, the process and effectiveness of the course. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 5 3 Scope and Methodology 3.1 APPROACH The study involved multiple key steps, which can be broken down into five parts that are briefly discussed below: 3.1.1 Part 1:Video data analysis The Video Study provides an important initial analysis of current teaching practices and identifies areas where teachers could improve their practices. The findings provide a picture of what takes place in Indonesia’s mathematics classrooms, why these events take place, and how they relate to student learning outcomes. The TIMMS Video Data was examined to select appropriate lessons on the basis of the video stimulus for the professional development vignettes. The vignettes did not necessarily constitute notions of ‘best practice.’ Rather, these vignettes were intended to serve as visual stimulus to encourage discourse and discussion on matters related to discipline knowledge and pedagogical practices. 3.1.2 Part 2: Development of professional development vignettes The selected videos were framed around detailed professional development vignettes. Each vignette included: (i) a comprehensive analysis of the video lesson, including text stimulus within the video; (ii) pre- and post-video lessons that provided an exemplary sequence of learning activities aligned to the video lesson content; and (iii) complementary resources that promote effective pedagogical practices2 and discipline content knowledge. 3.1.3 Part 3: Development of working group program The vignettes formed the basis for teacher working group activities. These activities were designed to engage teachers in critiquing lessons and to encourage them to reflect upon their own practice. The video lessons were intended to provide stimulus for discussion and to encourage teachers to question their own pedagogical and content knowledge. The pre-video lesson is designed to promote conversations to ensure that students are equipped with the prior knowledge necessary to engage in the video lesson. Thus, the pre-lesson specifically considers the content knowledge the students should possess in order to engage effectively in the learning process. In turn, this helps to promote teachers’ discipline content knowledge. The post-video lesson relates to the pedagogical issues that the classroom teachers were encouraged to adapt and then implement in their respective classrooms. Initially, these lessons were explicitly designed to provide a framework for teacher practices. However, over the duration of the working groups, teachers were encouraged to develop their own lessons to trial and then to report on them. Consequently, the process ensured that teachers took a greater level of ownership over the process. 2 The vignettes describe effective ways of developing pedagogical practices which promote student engagement, deep levels of questions and techniques for promoting constructive student engagement. 6 The Use of Video in Teacher Professional Development 3.1.4 Part 4: Development of Online program Following the 11-week intensive program, the research group analyzed the evaluative stages. P4TK Mathematics took the lead in converting the course for use in an online environment and in piloting it with teachers, with the necessary modifications being implemented to develop online resources. The online component is housed on the P4TK servers to facilitate the provision of online training. The online component includes important support mechanisms for teachers, with these mechanisms including online facilitators, virtual synchronous meetings, and a forum where teachers can discuss their experiences. A critical design aspect of the CPD is that teachers discuss and reflect on the content within the sessions. While an online forum cannot fully substitute for the face-to-face interaction that would be experienced in the teacher working groups, the CPD course was designed to ensure that teachers would have ample opportunity to interact with other teachers. A helpline where teachers could obtain both technical and content support was also available. 3.1.5 Part 5: Evaluation The purpose of the pilot is to gain insights into CPD from both a content and delivery standpoint. Therefore, significant emphasis was placed on the evaluation of the pilot, with instruments being developed to facilitate both quantitative and qualitative analysis. While the teacher was the main focus of the evaluation, student perceptions also played an important role in the evaluation. The process itself was also closely evaluated, with a team of observers attending almost all sessions. Feedback surveys for both facilitators and teachers were also conducted after each session. At the completion of the course, multiple workshops were held to gather feedback from teachers and facilitators. Teachers were assessed with pre- and post-assessment instruments to determine the extent of the pilot’s impact on their knowledge, practices and perspectives. Assessments of student attitudes were also conducted both at the start and after the completion of the course. In the evaluation of the pilot, the process of delivery was as important as its outcomes. The CPD activities require a great deal of observation of teachers in action within the working group. It also required feedback from teachers regarding the processes both in the working group and through their between-session assignments and their in- class activities, to the extent that these activities related to the course. Due to budget constraints, the pilot could not be evaluated using a large-scale randomized control-trial (RCT) approach. However, extensive efforts were made to ensure that the pilot could reliably determine the effectiveness and impact of the CPD activities. To this end, the sampling approach followed many of the techniques used in larger RCTs, including stratification, random selection of teacher working groups, and use of control and treatment groups. A more complete explanation of the sampling approach can be found in Section 3.2 Conceptual Framework of Beliefs-Knowledge-Practices. 3.1.6 Part 6: Future Development As previously mentioned, it is important to keep in mind that this single course was not intended to provide a comprehensive CPD experience, nor was it designed to reach a large number of teachers during the pilot stage. Rather, it was intended to serve as a proof of concept by determining the extent to which the unique content, approach and activities used in the sessions were an effective and relevant means of facilitating teacher professional development. It was also designed and implemented within the context of an envisioned comprehensive framework that would require the development of additional courses. Therefore, this pilot should be seen as an embryonic work that will hopefully inspire further development. It should be pointed out that while the focus of this pilot is on the teaching of mathematics, the CPD approach and the framework concept utilized by the pilot could easily be extended for the teaching of subjects other than mathematics. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 7 3.2 CONCEPTUAL FRAMEWORK OF BELIEFS-KNOWLEDGE-PRACTICES A conceptual framework of beliefs-knowledge-practices was developed on the basis of the findings of the Indonesia Video Study for TIMSS and as part of a doctoral dissertation by Ragatz (2013). A full description and explanation of the conceptual framework can be found in the Technical Report for the Indonesia Video Study for TIMSS. This conceptual framework is particularly useful in considering what is required for teachers to incorporate new practices into their lessons. The main premise of this framework is that teachers are most effective when they adopt teaching practices that are aligned with their mathematical knowledge and their mathematical beliefs. Figure 2 below depicts a “Congruent Zone” that represents this alignment. By contrast, teachers are much less effective when they adopt practices that are not aligned with their knowledge or beliefs, when they are operating in the “Dissonant Zone”. With programs intended to encourage teachers to adopt new practices or to participate in reforms such as Indonesia’s 2013 curriculum reform, which encourage new approaches, it is therefore critical to implement measures to ensure that teacher’s knowledge and beliefs are developed so that they are aligned with the new practices. If teachers attempt to implement new practices that are not aligned with their knowledge and/or beliefs, then they are operating in the “Dissonant Zone” and will be much less effective. In the longer term, without constant supervision and/or coercion, teachers are likely to abandon the new practices and revert back to practices that are within their “Congruent Zone”. Figure 2: Conceptual Model of Beliefs-Knowledge-Practices Practices Used (actions) Dissonant Zone Kn ow e zon edl ce ge n a bse bs e fa nc lie Congruent Zone ez Be on e (practices-knowledge- beliefs all aligned in congruence) M at h em fs ati on lie ca ( pti Be l K too ) ce al no ls er tic wl ) (p ma ed Practice absence zone he ge at in M Te a ch in g With this conceptual framework in mind, the pilot course was designed to simultaneously develop both subject and pedagogical knowledge while also building the teachers’ beliefs in the effectiveness and applicability of the new practices. The impact evaluation component also attempted to measure knowledge, beliefs and practices to understand whether the course was effective in changing teacher beliefs, in increasing teacher knowledge and in the actually changing teaching practices in the classroom. The instruments and approach are discussed later in Chapter 5: Evaluation Instruments. 8 The Use of Video in Teacher Professional Development 3.3 SAMPLE SELECTION The pilot was implemented in five districts. Although a full-blown evaluation approach was not feasible due to budget constraints, many rigorous evaluation methods were still applied, albeit with a small sample. Budget constraints dictated that the evaluation should involve a sample within a relatively small geographical area. It was also important to link the sample for the CPD course to the sample for the video study in order to develop synergies. The selection of the sample involved two stages, with the first stage involving purposeful targeting, while the second stage involved stratification and randomized selection methods. 3.3.1 Targeted Selection Stage for Provinces and Districts The targeted selection of provinces and districts was implemented through the following means: First, a sample of five provinces was selected on the basis that these provinces had a large number of mathematics teachers at junior secondary level (Statistik Pendidikan, 2010) and would also provide some regional variation. No Province Number Percentage 1 Jawa Timur 9,626 13.9% 2 Jawa Barat 9,542 13.8% 3 Jawa Tengah 8,430 12.2% 4 Sumatera Utara 4,849 7.0% 5 Sulawesi Selatan 3,118 4.5% 6 DKI Jakarta 2,751 4.0% 7 Sumatera Selatan 2,608 3.8% 8 Banten 2,203 3.2% 9 Lampung 2,146 3.1% 10 Aceh 2,062 3.0% 11 Sumatera Barat 1,921 2.8% 12 Others (22 provinces) 20,003 28.9%   Total 69,259 100.0% Source: Statistik Pendidikan, MoEC (2010) Next, four districts were selected based on their participation in the teacher working group program of the Better Education through Reformed Management and Universal Teacher Upgrading (BERMUTU) project in order to see the sustainability of the program, One district in the province of DKI Jakarta was also included because it provided synergies with a continuous professional pilot that was being conducted by MOEC. Jakarta Selatan was selected because it had a large number of teachers that had participated in the video study. The list of selected districts is as follows: No Province District 1 Jawa Tengah Kab. Blora 2 Jawa Barat Kab. Sumedang 3 Sumatera Barat Kab. Solok 4 Sulawesi Selatan Kab. Gowa 5 DKI Jakarta Jakarta Selatan Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 9 3.3.2 Stratification and Randomized Selection Stage for Teacher Working Groups The randomized selection stage used the following methodology: First, a list of all mathematical teacher working groups at the junior secondary level was obtained from the district educational agency, together with data on the membership of the working groups, with data including the number of members; their status as civil servants, contract teachers, or school hired teachers; experience levels, and urban/rural location, amongst others. The working groups were then paired up with the most similar counterpart working group for urban and rural categories and the two pairs that would be most beneficial for the study based on their makeup were then put into a pool. Once the two pairs in the district were identified, one working group from each pair was then randomly selected to be the treatment group, with the other to be the control group. This process produced two treatment working groups and two control working groups for each district. In total, there were 40 teacher working groups in the study, with 10 treatment MGMP and 10 control groups. 10 The Use of Video in Teacher Professional Development 4 Course Design The course was designed based on a chain of processes, beginning with strong foundational evidence from the Indonesia video study on what should change, how to most effectively support teachers in making the change, dimensions in which the changes could be measured. This section describes the design process. Figure 3: Theory of Results Chain Foundation evidence for course Comprehensive Design of course study to identify to change areas in which practices based on teachers need Indonesian context support and evidence Cycle of learning, applying, reflecting, refining and sharing Teachers learn Reflection of Filming of own practical concept Application of experiences and Continue to refine lessons to reflect that can be easily what is learned in discussion with practices in the on and share with applied in the own classrooms other teachers on classroom peers classroom how to improve Changes that take place (measured in pilot) Explicit building of Gradual building Practices take root subject knowledge and reinforcement and become a in tandem with of beliefs in the part of teachers’s pedagogical new practices repertoire knowledge Expected ultimate outcome (but not measured in pilot) Students learn more by teacher having increased knowledge and ability to apply new practices Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 11 4.1 VIDEO STUDY RESULTS USED AS A BASIS FOR AREAS FOR IMPROVEMENT The findings of the video study and the desire to put these findings into practice served as the impetus for creation of the course, which focused on the key areas described in the following box. Key Areas to Focus on for the Course: • Actively engaging students in the learning process; • Paying attention to the language used; • Paying attention to the accuracy of the mathematical content (e.g. What is volume? What is 3D space?) • Enriching teachers’ roles, with teachers moving from direct teaching to posing “strategic questions”; listing to students carefully; and eliciting students’ understanding; • transforming the manner in which teachers fulfill their duties, with a shift from a very procedural pattern to a more investigative and meaningful pattern; • Incorporating non-routine tasks into lessons to provide more challenging tasks that produce higher-order thinking. 4.2 FOUNDATIONAL CONCEPTS OF PROFESSIONAL DEVELOPMENT The core function of teacher professional development is to improve the processes of teaching and learning. This may involve aspects such as curriculum development, instructional development, student assessment and improvements in school-parent collaboration. Gordon (2004) defines successful professional development as: “... a combination of experiences that empower individual educators, educational teams, and the educational organization to improve curriculum, instruction, and student assessment in order to facilitate student growth and development.” Fullan (1991) states: “There is no single strategy that can contribute more to meaning and improvement than ongoing professional development.” However, it is extremely challenging to create an effective professional development program, with many experts concluding that often programs are ineffective. If professional development is critical to facilitating improvements, but it is acknowledged that many programs fail, it is necessary to determine the key elements that ensure their success. 4.2.1 Characteristics of Good Professional Development Programs • Strong leadership and support: Leaders establish an atmosphere of support and trust, offer incentives and rewards for participation, and provide sustained moral and material support. Leaders serve as role models by participating fully in professional development activities. • Collegiality and collaboration: Teachers are involved in program planning and delivery. Schools form collaborative partnerships with other schools, universities, businesses and critical friends. Also, there is cooperation and coordination between the school and the district educational agency. 12 The Use of Video in Teacher Professional Development • Data-based development: A variety of needs assessments are conducted and the data from these assessments is analyzed carefully. As programs are planned and implemented, participants continue to gather data as a basis for ongoing program improvement. Much of the data is gathered at the classroom level, before being shared and analyzed at the team and school levels. • Program integration: School-wide professional development goals and school improvement goals are integrated. There is also an integration between individual, team, school and district goals. • A development perspective: Effective development programs are characterized by long-range planning and development. Participants take an incremental approach, with the aim being to achieve continuous, ongoing improvements. • Relevant learning activities: Many or most learning activities take place at the school level. Activities that take place at other sites are focused on the school’s improvement goals. Learning activities are participatory and experiential. • Professional development as a way of life: Norms exist among teachers that there is an obligation to attend professional development activities, with the teachers believing that these activities have a strong potential to improve teaching and learning. Teachers think of professional development activities as a “way of life” in their schools. 4.2.2 Principles of Adult Learning It is also important to keep in mind that in teacher professional development activities, the participants are adults. Adults learn in a different manner than do children, and the approach to professional development must take this into account. Key principles of adult learning include: • Adults are motivated to learn when they feel a need or interest they are experiencing in their personal and/ or work lives; • Adults bring considerable life experience and prior knowledge to the learning situation; • Adults learn best when they are actively involved in the learning process; • Adults have widely varying learning styles; • As adults develop personally and professionally, they have an increasing need to be self-directed; • Adults have affiliation needs; • Adults enjoy learning in groups in which they can share experiences and ideas. 4.3 TEACHER WORKING GROUPS FOR DELIVERY OF PROFESSIONAL DEVELOPMENT Teacher working groups were selected as the venue for the delivery of the course. These groups were selected for this purpose primarily on the basis of the benefits of their use from the point of view of both effectiveness and sustainability. This section briefly examines the theory and core concepts of teacher professional development in order to provide a context for the implementation of the course. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 13 4.3.1 What is a Teacher Working Group? Teacher working groups, known in Indonesian as Kelompok Kerja Guru (KKG) when they involve primary school teachers, and Musyawarah Guru Mata Pelajaran (MGMP) when they involve secondary school teachers, are clusters of teachers within a defined geographical area. For secondary school working groups, the number of schools within a cluster tend to range from three to six, and involve an average of approximately 20 teachers. In the secondary school groups, the groups are subject specific, so, for example, a single group may consist of 20 mathematics teachers. The frequency of meetings varies, but they tend to be held every second week. The primary purpose of teacher working groups is to provide an environment for professional development. Therefore, in order to examine the current state and activities of these groups, it is first necessary to understand the theory behind the professional development of teachers and to determine what elements are necessary if this development is to be effective. Traditional in-service training vs. teacher working groups Keeping in mind the above characteristics, it is useful to compare traditional in-service training to activities in teacher working groups. While both approaches should attempt to incorporate as many aspects of good practice as possible, there are relative advantages for each method of achieving professional development. Status of in-service training It has been widely recognized that periodic in-service training is essential in order to keep teachers abreast of recent pedagogical developments and as a means for them to update their knowledge and skills on a continuous, ongoing basis. The traditional method of providing in-service training is to have teachers attend courses, typically of one or two weeks in duration, once every one or two years. These are usually conducted by expert trainers at a center where teachers would typically be provided with food and accommodation over the period of training. There are advantages of this traditional approach, including: • Bringing teachers to a training center allows for a period of immersion during which teachers can focus on learning; • The instructors may be experts in their field and they may have the specific skills required to facilitate the training of teachers; • Conducting the courses over multiple cycles provides economies of scale, with instructors able to modify the course over time to increase its effectiveness. However, traditional in-service training has also been criticized by many education experts as being ineffective in achieving teacher professional development to a high level. General weaknesses include: • Many components of the training program lack of relevance to the real needs of teachers and to the demands of the classroom situations they might face; • The process of training at the teachers’ institution is itself often very mechanical in nature; • There is a lack of emphasis on, and inadequate time devoted to, teaching the practical aspects of classroom work, including teaching strategies, classroom management techniques, reading and language teaching skills;3 • Studies show that learning acquired over a brief intensive period is rarely applied in the classroom or is quickly lost if not reinforced with additional follow-up activities; • Activities at in-service training institutions are costly and may require that teachers be away from the classroom for extended periods of time; 3 http://www.un.org.in/JANSHALA/aprjun01/teachtrn.htm 14 The Use of Video in Teacher Professional Development • teachers are often assigned to courses, rather than being able to select courses that they themselves consider of the most interest or relevance: Advantages of Teacher Working Groups Working groups can provide a very different type of learning and professional development experience. They have an advantage over traditional in-service training in that: • Training activities can be performed on a regular basis (every two weeks rather than once a year); • Regular meetings allow for the reinforcement of concepts and multiple iterations of training and professional development activities; • The structure of training involves a “for teachers, by teachers” approach, which has been shown to be the most successful method for professional development; • Training activities can be more easily modified to meet the needs of a local context, particularly since all teachers come from the same general location; • Teachers form a support system upon which they can rely on and to which they have easy access; • The teacher working group methodology conforms to the principles of adult learning described above, in that: a Teachers are able to structure learning around their interests and needs; a Teachers are able to apply life experience and prior knowledge to the learning situation; a Teachers are able to be actively involved in the learning process; a The learning style can be structured to adapt to the needs of working group members; a The teachers are able to be self-directed; a The working group provides an affiliation for its members; • Learning activities tend to be participatory and experiential. Teacher Working Group Limitations There are also limitations to what working groups can accomplish and challenges to ensuring that they are an effective means of achieving their intended goal. Working groups are intended to focus on teacher professional development. They may involve a range of different activities, from informal discussions between teachers about the issues they face in the classroom to more structured activities that generate significant outputs, such as research papers, articles for journals, and so on. In comparison with in-service training, teacher working groups may have certain limitations, including the following: • Working groups tend to rely on the expertise of their members, which is often limited and must be supported through the provision of materials or outside expertise; • The working group may lack the capacity required for more training-oriented activities; • The activities may become less structured; • For most current working group activities within Indonesia, there is no specific output required or final exam to test the extent to which teachers have gained knowledge. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 15 4.4 COURSE DESIGN PRINCIPLES It is notoriously difficult to achieve behavioral change. However, in the context of teacher professional development, such change is most likely to occur when teachers are provided with support not only to develop a higher level of understanding, but also with support to enable teachers to implement what has been learned. It also requires time for the new behavior to become entrenched through repeated practice. Therefore, a number of key design principles were defined prior to the development of the course. The seven key principles applied in the design of the course are listed below. Key Design Principles for Behavioral Change in Teaching Practices Behavioral change is most likely to occur when: 1. Activities to facilitate this change are conducted over an extended period of time; 2. There is an iterative cycle of learning, practice, reflection and modification; 3. There are opportunities to engage in discussions with peers; 4. The content and subject of the activities relate directly and are relevant to the teacher’s classroom context; 5. The content is practical and has practical applications, rather than being purely theoretical; 6. The approach strengthens both the teacher’s knowledge and beliefs in relation to the practices; 7. Activities relate to both subject matter and pedagogical content, so that the teacher strengthens both simultaneously. 4.4.1 In-On-In Approach With the stated goal of achieving behavioral change, the course was designed with an in-on-in approach, in which teachers first engage in learning activities in a working group session. Following this they are assigned with “homework” tasks to complete in their own classrooms. In the subsequent working group session, teachers share their experiences and reflect on what worked, what didn’t work, and how they could do better in the future. The in-on-in approach dictates that what is learned has to be practical and to have practical applications, rather than being purely theoretical. Through the learning process, both subject matter and pedagogical aspects are addressed simultaneously through the activities, in order to build teacher knowledge and to enable them to apply this knowledge in the pedagogical context of the classroom. Teachers must be provided with reasons and a rationale for the activities they engage in, with measures to develop a foundational understanding that helps convince teachers that the practices are useful and beneficial. 4.4.2 ELPSA Framework The ELPSA (Experiences, Language, Pictures, Symbols and Application) learning framework is embedded in the design of the course. The ELPSA framework is built on the basis of constructivist and social learning theories. The ELPSA framework views learning as an active process in which students construct their own ways of knowing and developing understanding through both individual reflection and through social interactions with others. It should be emphasized that the form of learning defined by the ELPSA is not a linear process. Learning is a complex and unpredictable process that does not take place in a linear sequence. Thus, the various elements of the model should be considered as interrelated and overlapping. While the components of the ELPSA framework can be discussed individually, these components cannot be implemented in isolation. Rather, the various components are incorporated and intertwined with each other throughout the lesson. 16 The Use of Video in Teacher Professional Development Figure 4: Example of ELPSA for the topic of right triangles EXPERENCES Student construct right-angled triangles LANGUAGE to scale and label Terminology including: right angle, hypotenuse, 90o PICTURES reinforced 5 3 c Student encounter 4 a triangles that do not b have the right angle as an experience to move forward trigonometry Formula : c2=b2+a2 reinforced APPLICATION Student are given task to determine the are of the floor in their classroom SYMBOLS Introduction of formula for right -angle triangle A=1/2 bh 4.4.3 Incorporation of Videos into the Course One of the advantages of developing the course on the basis of the video study is that those designing the course could draw upon more than 600 hours of footage containing real-life examples of classroom practices. Example clips were incorporated into the session activities for tools to stimulate discussion. These examples were not necessarily intended to portray best practices, but rather to serve as stimulants to provoke discussion on specific issues and subjects. 4.4.4 Facilitators as an Integral Part of Session Activities The least costly method of delivering CPD through teacher working groups would be to design the course so that it could be utilized without any facilitation. If this approach were adopted, the materials would need to be completely self-explanatory. In addition, teachers would need to have a sufficient level of motivation to conduct the activities in a fairly self-directed manner. While this option was considered, it would have severely limited the complexity of the activities included in the course. Also, it would have few if any regulators to control the quality and consistency of delivery. It was decided that the course could be developed much more effectively if it involved the selection of two members within the working group to serve facilitators and leaders, together with measures to develop the capacities of the selected two facilitators. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 17 5 Evaluation Instruments The instruments were designed to capture changes in teacher beliefs, knowledge and practices. 5.1 TEACHER SURVEY WITH MATHEMATICAL BELIEFS Teachers were asked to participate in a survey at the beginning of the first session to gather background information. In addition, the survey included a set of statements related to beliefs related to teaching mathematics and the process of learning mathematics. There were also statements on learning activities in the classroom. The items were ranked on a Likert scale from 1 (“not at all important”) to 5 (“very important”).4 Teachers rank these items in the first session and again in the last session. Results for each teacher could then be compared to determine if changes in beliefs and/or perception had occurred and, if so, to what extent. Section Number of Items Teaching Mathematics 10 Learning Activities 14 The Process of Learning Mathematics 8 Total 32 5.2 TEACHER PRE AND POST TESTS ON SUBJECT AND PEDAGOGICAL KNOWLEDGE Teachers were also given competency tests on subject knowledge and pedagogy both at the beginning and at the end of the delivery of the course. Teachers were tested on matters related directly related to the course, related only indirectly, and not related at all. The purpose of each of these test items was to determine the extent of both direct and indirect effects. In the case of test items not at all related to the course, these items were included to check whether the expected outcome of no change was recorded, therefore essentially functioning as control items. Teachers in both the treatment and control groups participated in these tests to determine the relative changes between the two groups. The tests were trialed with a group of teachers in Yogyakarta and modified on the basis of the results of this trial. The tests were not standardized, so that receiving a particular score on the pre test would be equivalent to receiving the same score on the post test in terms of ability. It is recognized that teachers in the treatment group could possibly attempt to give the “correct answer” by stating that their beliefs had changed in areas related to the course’s focus. This was taken into consideration in the analysis and triangulated with the student perceptions survey on classroom practices, which is explained next. 4 In the case of teachers’ learning activities, the scale was on frequency of use and ranged from “never” to “very often”. 18 The Use of Video in Teacher Professional Development 5.3 STUDENT PRE AND POST PERCEPTIONS SURVEY ON CLASSROOM PRACTICES The student perceptions survey was adapted from the “Asking Students about Teaching” instrument from the Measures of Effective Teaching (MET) project of the Bill and Melinda Gates foundation. The purpose of the instrument for this pilot was to measure students’ perceptions of their teacher’s practices before and after the pilot in order to determine whether any change in practices had taken place, particularly in comparing the treatment and control groups. Some statements were more related to the pilot course than others, so changes would not be expected for many of the statements in either the treatment or control groups. More relevant statements, on the other hand, would have an expected change in the treatment group, but not the control group. The reason for using student perceptions is that it provides a relatively objective measure as to whether teachers in fact changed their practices. 5.4 WHY NO STUDENT TESTING? While improvements to student learning may be regarded as the ultimate goal of measures to improve classroom practices, this evaluation did not implement any measures to determine the extent to which such improvements had occurred. The first reason for this was simply a matter of constraints. The proper implementation of pre and post tests to the students of more than 400 teachers in the treatment and control groups was not feasible from both a cost and logistical standpoint. Second, while of course the quality of student learning is important, the primary objective of the program was to determine whether the methods applied would lead to changes in classroom practices. Third, the practices included in the module had been chosen based on the Indonesia Video Study based on the fact that the practices had been identified as having a positive relationship with student learning outcomes. Therefore, it was decided that a survey to capture students’ perceptions of changes in the classroom would be both more useful in terms of the frame of reference of the program and more achievable from a logistical standpoint. It was determined that a survey to capture student perceptions could be safely administered and collected by the teachers themselves, while it might have been tempting for teachers to manipulate the results of a test to measure the quality of student learning, which might have raised concerns regarding the validity of the test’s results. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 19 6 Implementation The following section describes the process of implementation, including key decisions, components, approaches and steps taken in this process. 6.1 FUNDING OF ACTIVITIES Teacher working groups are supposed to function independently of this pilot. Therefore, this pilot was intended to work with existing working groups, without providing funds to cover the cost of meeting sessions. However, funds were provided to cover the cost of the initial facilitator workshop and the workshops held after the pilot was completed. The materials were also provided without charge. Teachers in both the treatment and control groups also received a small sum of money to recompense them for their participation in the pre-post test activities. With a view to the sustainability and scaling up of the activities, it should be acknowledged that the pilot did involve certain costs. However, for the most part, the pilot was integrated into the existing structure. This approach could be scaled up in a manner that would not require much additional funding. It requires the provision of course materials (at a cost of approximately two dollars per participant) and of a three-day training session for the facilitators. Actual sessions would not involve additional costs for existing working groups that are already fully functional. 6.2 FACEBOOK GROUP FOR PARTICIPANTS In order to provide a forum to enable facilitators to share their experiences, a Facebook group was established. Nearly all facilitators already had a Facebook account, making it an easy way for participating facilitators to share ideas and to ask questions. The forum was used most extensively at the beginning of the process, when facilitators began to prepare to implement the course in their respective working groups. It was also frequently used towards the end of the process, when teachers began to film their own lessons. This forum also enabled the gathering of additional information for the evaluation. The nature of the difficulties and the concerns that teachers faced and of the successes they achieved and the innovative approaches they adopted could be identified through this means. It is important to note that provision of this forum should be considered part of the pilot, as it possibly played a role in some of the pilot’s achievements, possibly eliminating or ameliorating some constraints. 6.3 SEQUENTIAL STEPS OF THE PILOT ACTIVITIES 6.3.1 Pre-meetings with District Officials With the MOEC’s new requirement that teachers obtain a certain number of credit points for each stage of their progression and promotion, there is an additional motivation for teachers to actively engage in CPD. However, the activities in which teachers participate must be recognized so that they may be provided with credit points. The district education agency is responsible for the provision of professional development to teachers within that district. Therefore, this agency is the one to 20 The Use of Video in Teacher Professional Development recognize and provide credits for the activities in which teachers engage. Therefore, it was critically important to meet with officials of the education agency in each of the selected districts to ask whether they would grant credit for participation in the activity. In order for a participant to receive one credit point, it was necessary for the course to have a duration of at least 30 hours. All officials of district education agencies in the participating districts were enthusiastic to have the course piloted in their district and were willing to provide certificates and credit to the participants. Therefore, it is clear that there is demand for the type of activities conducted through the project. While district education agencies are responsible for facilitating professional development, most districts are unsure how or do not have the capacity to deliver quality professional development activities. As a result, most teachers do not participate in any professional development activities at all. Even when these activities are offered to them, they are often not relevant to their needs or of low quality. 6.4 INITIAL FACILITATOR WORKSHOP A three-day workshop was held at the offices of P4TK Mathematics in Yogyakarta. Two facilitators from each teacher working group participated in this workshop. Three school-level working groups from South Jakarta also participated, so the total number of facilitators participating was 26. Prof. Tom Lowrie and Sitti Maesuri Pattahudin, who designed the course, led the workshop. Participants were introduced to the underlying philosophy of the course, including the ELPSA approach, and each of the activities within the course. Measures were also taken to develop the necessary skills for them to fulfill their role as facilitators. 6.5 CONDUCTING SESSIONS All sessions took place over the second school semester (in the period from January to May 2014). Each working group had its own schedule, with some variation in the frequency of meetings. For each session, a team member attended to document the activities and discussions that took place in that session. Facilitators and teachers also provided feedback after each session. 6.5.1 Pre testing and data gathering in the first session In order to obtain baseline data, all participants were subjected to an hour-long mathematics competency assessment in the first session and required to complete a survey questionnaire intended to gather background information and to assess mathematical beliefs on teaching and learning. Teachers were also given the student perceptions survey instrument and asked to conduct the survey amongst their students. Each working group provided details regarding the planned schedule for each working group session. This was used to track working group activities and to schedule visits by the study team so that the working group sessions could be observed. 6.5.2 Final Session Post testing and Demonstration of Self-shot Videos In the last session of the course, the post-survey on teacher beliefs and the post test of subject and pedagogical knowledge were conducted. Some teachers also presented the videos that they taped of their own lessons. 6.5.3 Final Session Post testing and Demonstration of Self-filmed Videos The student perception surveys were administered and collected by the control and treatment teachers. They were then collected by the district office and sent in to the project team for analysis. 6.5.4 Follow-up Workshops and Focus Group Sessions After the pilot activities were completed, there were two additional workshops held to gather feedback from participants. In addition, the participants shared the videos they had produced and their perspectives specifically on the utility and relevance of the ELPSA framework in the context of the development of the 2013 curriculum. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 21 6.5.5 Evaluation Results The evaluation covered multiple perspectives in order to gain a full understanding of how the course was implemented and of the extent of its effectiveness. The data collected was also intended to identify areas for improvement to the content and delivery of the course. The following section includes three subsections: • Attendance and Deliverables: This subsection provides an overview of treatment group attendance to create a sense of the level of participation throughout the semester; • Quantitative analysis: This subsection provides an overview of the measurable results of the implementation of the course in terms of its impact on: (i) changes in beliefs about teaching and learning, as measured through a teacher beliefs survey; (ii) changes in subject matter knowledge, as measured through a competency assessment; and (iii) changes in classroom practices, as measured through a student perception survey; • Qualitative analysis: This subsection includes summaries of: (i) observations of session activities; (ii) interviews and surveys of facilitators; (iii) interviews and surveys of teachers (participants); and (iv) feedback received through workshops and focus group discussions; 6.6 ATTENDANCE AND DELIVERABLES OF TREATMENT GROUP PARTICIPANTS One challenge faced was that the participants of the teacher working groups were not fully consistent in their attendance. While the majority of participants who participated in the initial session and the pre test also participated in the final session and post test, there were also teachers who participated in only the pre test. While it may be expected that some participants would drop out and not complete the full course, there were also a large number of teachers who did not participate in the initial session, but participated in the final session and post test. This is important, in that it captures the effect of the rotation of teachers. For some schools, not all teachers would participate, while some were rotated during the period in which the course was implemented. The nature of in-on-in training requires a certain degree of consistency in the attendance of participants, both in terms of effectively facilitating the participants’ professional development and in determining whether they ultimately receive credit for their participation in the course. Fortunately, the majority of participants in both the treatment and control groups participated in both pre and post testing. 6.6.1 Attendance of Teachers in the Treatment Group A total of 157 teachers in the treatment group attended both the first and last sessions. These are the teachers who are included in the analysis, since pre and post data is required for the purposes of comparison. In addition, there were 46 teachers who attended the first session, but not the last session. This means that 77 percent of all teachers who commenced participation in the pilot also completed it. There were also another 19 teachers who attended the last session, but not the first. If these teachers were also included, then the proportion of teachers who attended both pre and post drops to 71 percent.5 When considering the attendance of the core group of 157 teachers that attended both the pre and post sessions, half attended all sessions, while another 34 percent missed only one session. The remaining 16 percent missed two or more sessions. 5 There were also another 22 teachers who did not attend either the first or last session, but joined in at least one of the middle sessions, so in total 244 teachers attended at least one session over the course of the semester. 22 The Use of Video in Teacher Professional Development Figure 5 - Attendance Rate of Teachers Used in Analysis Missed 4 - 2% Missed 3 - 2% Missed 2 - 12% Attended all- 50% Missed 1- 34% 6.6.2 Attendance of Teachers in the Control Group In the case of the control group, the frequency of working group meetings and the rate of attendance at these meetings was not tracked. However, the number of teachers participating in the pre and post sessions was recorded. While a total of 168 teachers in the control group participated in the pre test session, 42 of those teachers did not participate in the post test. This means that 75 percent of teachers in the control group who participated in the first session also participated in the last session. This is very close to the rate of 77 percent recorded for teachers in the treatment group. In addition, 20 teachers in the control group did not participate in the pre test, but did participate in the post test. For a semester-long course, some attrition should be expected. The retention rates of 77 percent amongst teachers in the treatment group and 75 percent amongst teachers in the control group may be considered reasonable for the purposes of conducting the impact evaluation. For the purposes of analysis, only those teachers who participated in both the pre- and post test are included, which means that the analysis involved 156 treatment group teachers and 126 control group teachers. While the number of teachers in the treatment group was higher, these numbers were considered reasonable for the purposes of conducting a comparative analysis between teachers in the two groups. 6.6.3 Attendance per Session The average participation rate for all meetings was 73 percent, with attendance being highest at the first meeting. It should be noted that failure to attend does not necessarily imply lack of interest on the part of the participants. Some teachers were very interested in participating, but could not attend the activities due to work or other conflicts. Good coordination between facilitators, the educational agency, and schools in the planning of the MGMP meetings schedule is required. Facilitators were not responsible for pushing teachers to participate in MGMP. In terms of attendance rates for each sections of the course, there were clear drops in Sessions 3 and 5, where the rate of attendance fell below 60 percent. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 23 Topic % participation Introduction on ELPSA and CPD 88 Questioning 78 Rich problems 57 Volume and capacity 70 Inquiry 58 Lesson plan and video presentation 85 However, in terms of the effectiveness of implementing a course through the MGMP, the attendance rates are cause for concern. For the purpose of conducting an impact evaluation, the rate is also important, in that the varied levels of participation should be considered, as teachers recording a high rate of attendance (5-6 sessions) should be distinguished from teachers who had a low rate of attendance (Three or fewer sessions). 6.6.4 Deliverables Teachers in the treatment group were also expected to submit deliverables at various points in the course. For the video recording of their own lessons, teachers were able to work in teams, and nearly all teachers fulfilled their assigned tasks, with a total of 98 video lessons being recorded. In the area of lesson planning, 98 participants completed the lesson plan assignments. While only 52 teachers actually submitted open-ended questions, most did in fact participate in the open-ended question activities. While many teachers did conduct hands-on activities in their classrooms, but only four teachers actually submitted the assignment deliverable, required the writing of a summary report. It is clear that getting teachers to complete more comprehensive tasks, particularly those that require somewhat extensive writing, presents challenges for teachers. Generally, the rate of submission of deliverables was quite varied. Teachers were generally active and involved in the work, but it is also clear that many of them did not complete all homework activities. 24 The Use of Video in Teacher Professional Development 7 Evaluation of the Program The program was evaluated using a mixed methods approach, employing both quantitative and qualitative methods to get a more complete picture of the way the program was implemented, the views of the participants and the impact the program had. While it was designed to be an impact evaluation with treatment and control groups and care was taken to follow impact evaluation procedures such as randomized assignment, it is also recognized that this evaluation should not be considered a full-fledged randomized controlled trial (RCT). With assignment being at the teacher working group level, the total of 20 teacher working groups (10 treatment and 10 control) over five districts is relatively small. In one case a district only had four mathematics teacher working groups in total. While randomized selection still took place, it was somewhat limited since a total of four working groups were to be included per district (with two treatment and two control). Still, the analysis of results for data gathered before implementation indicate the treatment and control groups were quite similar and allowed for good comparison. Incorporating the quantitative results with the qualitative provided a rich set of information for evaluation. 7.1 QUANTITATIVE ANALYSIS The quantitative analysis examined three areas: (i) teacher beliefs; (ii) teacher knowledge; and (iii) teaching practices. In each of these areas, data was gathered both before the course began and after it was completed. • For teacher beliefs, a survey was conducted to ask teachers to rate statements on a scale of 1 to 5 in terms of the extent to which they agreed with that statement. The statements included those related to: (i) teaching mathematics; (ii) the process of learning mathematics; and (iii) learning activities. • For teacher knowledge, pre and post tests were conducted. While tests were not identical, they contained the same general types of questions, with the inclusion of anchor items (which appeared on both the pre and post tests) and parallel items (which were similar, but were changed so that the answer would not be the same). This test also contained groupings of questions, with questions directly, indirectly and unrelated to what was learned in the course. For example, one of the main sessions in the course covered open-ended questioning. The tests contained items to evaluate teachers’ ability both to create open-ended questions and to answer open-ended questions. This can be considered directly related. Indirect items have a relationship with what was learned through the course, while unrelated items have no relationship to what was learned through the course. The mix of these questions allows for analysis to determine the direct and indirect impacts of the course, while the unrelated questions served as controls, with the assumption being that answers to these questions should not change as a result of the intervention. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 25 • For teaching practices, students were asked to participate in a perceptions survey, with students rating on a scale of 1 to 5 how frequently certain activities took place or the extent to which teachers used certain techniques. For all of these instruments, the results of the pre test and surveys were compared to the results of the post test and surveys to determine whether any changes were discernible amongst the treatment group in relation to the control group and, if so, to what extent. This involved first looking at averages, followed by the implementation of a simple correlational analysis. Following this, a more rigorous regression analysis was conducted, with control variables to determine whether the changes were statistically significant.6 Another end goal was to gain an understanding of the beliefs-knowledge-practices relationship. This section provides an overview of the results of the quantitative analysis. While this provides an important perspective, a great deal of qualitative data was also gathered to gain a more in-depth perspective of the process of using the course and of teachers’ and facilitators’ perspectives on what worked, what didn’t, and how the course impacted their practices. With this mixed methods approach, the complementary views create a more complete and comprehensive picture of the course and its impact. 7.1.1 BELIEFS: Pre-post results of the Mathematical Beliefs Statements In order to enable a comparison between the results of the pre survey and post survey, the average results for both the treatment and control groups were first examined to gain an indication of whether the treatment and control groups were similar in the pre-session and to determine the extent to which results changed in the post- session. In this initial process, the relatively large shifts were highlighted. The second round of analysis used regressions to determine the extent to which changes amongst the treatment group were statistically significant relative to the control group. This round involved simple correlations as well as controlling for teacher and school characteristics. The following three tables show the average results for the treatment and control groups at the pre-session and post-session points, and the extent of the changes. It is important to note that the average results for the treatment and control groups in the pre survey are almost always within 0.1 of each other, indicating similar beliefs before the pilot was conducted. For the purpose of conducting an impact evaluation, this is a desirable result, indicating that the groups appear to be similar in composition, which enables a meaningful comparison. The statements where there was a statistically significant difference between the treatment and control groups are highlighted in either green (in cases where the results for the treatment group increased relative to those for the control group) or red (where the results for the treatment group decreased relative to those for the control group). Beliefs on the Teaching of Mathematics Of the three sections, most of the statistically significant changes were recorded in the teaching mathematics section. Treatment teachers showed a statistically significant increase for statements regarding hands-on activities, inquiry-oriented activities, performance-based assessment and informal questioning to assess student understanding. All areas except the performance-based assessment were part of the pilot course. Items related to the use of calculators and computers and the establishment of connections between mathematics and other disciplines were not covered in the course. These items were intended to act as control statements, so the lack of difference between the two groups is in fact a positive result, as it establishes the credence of the changes in those items that were in fact covered in the course. 6 It is important to note that the impact evaluation design attempted to follow principles of a Randomized Controlled Trial (RCT), but with the understanding that because of budget and logistical limitations a true RCT was not possible. In a full-blown RCT there would not be a need to include control variables and the direct correlational comparison between treatment and control groups would have been sufficient. Because of the pseudo-nature of this evaluation, the correlational analysis was considered useful, but that regressions with control variables were also necessary. 26 The Use of Video in Teacher Professional Development Table 1: Survey results for teacher beliefs on the teaching of mathematics TEACHING MATHEMATICS PRE   POST   CHANGE In your opinion, how important are each of the following for effective mathematics instruction in the grades you teach? Please indicate from 1 (not important) to 5 (very   important): Cntrl Trtmnt   Cntrl Trtmnt   Cntrl Trtmnt 1 Make connections between mathematics and other disciplines. 4.4 4.3   4.4 4.4   0.1 0.2 2 Have students work in cooperative learning groups. 4.4 4.2   4.3 4.4   (0.1) 0.2 3 Have students participate in appropriate hands-on activities.** 3.9 3.9   3.9 4.3   0.1 0.4 4 Engage students in inquiry-oriented activities.** 4.0 4.0   4.1 4.3   0.1 0.4 5 Use calculators. 2.0 1.9   2.0 1.9   (0.1) (0.0) 6 Use computers. 3.3 3.4   3.6 3.6   0.3 0.2 7 Engage students in applications of mathematics in a variety of contexts. 4.3 4.3   4.4 4.4   0.0 0.1 8 Use performance-based assessment.** 3.8 3.9   3.9 4.2   0.1 0.3 9 Use informal questioning to assess student understanding.** 4.2 4.2   4.1 4.4   (0.1) 0.2 Note: items with ** indicate statistical significance at the level of 5 percent or higher between treatment and control groups. Green highlighting indicates that results for the treatment group increased relative to the control group, while red indicates that the results for the treatment group decreased relative to the control group. Beliefs on Learning Activities In the case of learning activities, only a few of the changes were found to be statistically significant, but these changes were related to activities emphasized in the course. A statistically significant increase was recorded amongst teachers in the treatment group for statements related to students participating in student-led discussions and working in cooperative learning groups, as well as for statements related to students working on extended mathematics investigations or projects. Again, these items relate to what was emphasized in the course. Interestingly, statements on working in cooperative learning groups and engaging in hands-on mathematical activities appeared in both the Learning Mathematics and Learning Activities sections, but while the rate of change for statements related to hands-on activities in Learning Mathematics reached the level of statistical significance, they did not do so for Learning Activities. Similarly, the statements related to cooperative learning groups reached the level of statistical significance in Learning Activities, but they did not do so for Learning Mathematics. The relationships for all statements were always positive, so there were no contradictory results. However, it is unclear why these results failed to reach the level of statistical significance across the various sections. Another important general aspect of the results is that statements regarding the design and implementation of investigations (19); working on models or simulations (20); and the use of manipulatives to explore a concept (21) increased in the cases of both the treatment and control groups. This is likely due to the fact that all teachers in Indonesia have been learning about the new curriculum, which emphasizes these teaching methods. Therefore, it is not the case that the beliefs of the control group remained completely static during the period the pilot was conducted. The course emphasized aspects of investigation and their results did in fact increase, but it is unknown Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 27 if the increase would have been statistically significant had the control group not been simultaneously exposed to similar ideas through other channels. Table 2: Survey results for teaching practices used in learning activities LEARNING ACTIVITIES PRE   POST   CHANGE In your mathematics lessons, how often do you usually ask students to do each of the following? Please indicate from 1   (never) to 5 (very often): Cntrl Trtmnt   Cntrl Trtmnt   Cntrl Trtmnt 10 Work on problems for which there is no immediately obvious method of solution. 3.0 2.8   2.9 3.0   (0.1) 0.1 11 Use manipulatives to solve exercises or problems. 3.6 3.6   3.6 3.6   0.0 0.0 12 Participate in student-led discussions.** 3.8 3.8   3.9 4.0   0.1 0.2 13 Work in cooperative learning groups.** 4.0 3.7   3.9 4.0   (0.0) 0.3 14 Read other (non-textbook) mathematics-related materials in class. 3.9 3.7   4.0 4.0   0.2 0.3 15 Share ideas or solve problems with each other in small groups. 3.9 3.7   3.9 3.9   0.0 0.2 16 Investigate mathematical concepts through other disciplines.* 3.2 3.1   3.2 3.3   0.1 0.3 17 Engage in hands-on mathematical activities. 3.4 3.4   3.6 3.7   0.1 0.2 18 Play mathematics games. 3.0 2.9   3.2 3.1   0.2 0.2 19 Design or implement their own investigation. 2.8 2.8   3.1 3.0   0.3 0.2 20 Work on models or simulations. 3.2 3.1   3.5 3.5   0.3 0.3 21 Use manipulatives to explore a concept. 3.7 3.6   4.0 3.9   0.3 0.3 22 Work on extended mathematics investigations or projects (a week or more in duration).** 2.5 2.4   2.7 2.8   0.2 0.5 23 Write reflections in a notebook or journal. 3.2 3.2   3.4 3.4   0.2 0.2 Note: items with ** indicate statistical significance at the level of 5 percent or higher between treatment and control groups. Green highlighting indicates that results for the treatment group increased relative to the control group, while red indicates that the results for the treatment group decreased relative to the control group. 28 The Use of Video in Teacher Professional Development Beliefs on the Process of Learning Mathematics Statements related to the process of learning mathematics were less directly related to the subject covered in the course. While teachers were exposed to certain approaches and practices, the relationship between the course and the statements below was less strong. The course did introduce the ELPSA approach to how students learn, but the statements below were less strongly related to this concept. In retrospect, it would have been good to include some statements more directly related to ELPSA. In the case of only one statement was there a statistically significant difference between the treatment and control groups. In the pre-survey, teachers in both the treatment and control groups expressed a strong belief in the importance of finding the correct answer to be successful in maths (26), with averages of 4.0 for the control group and 4.1 for the treatment group. The result for the control group increased slightly in the post survey to 4.1, while the result for the treatment group declined slightly to 3.8. While it is unclear whether this was due to teachers’ participation in the course, one possible explanation is that in learning about the process and in varying the ways in which problems can be approached, the emphasis on the final answer became less important. The process itself may have become viewed as having more value in learning than the actual result. Table 3: Survey responses for statements on The Process of Learning Mathematics THE PROCESS OF LEARNING PRE   POST   CHANGE MATHEMATICS Rate each of the following belief statements according to how strongly you believe each statement. Rate each item 1 (strongly disbelieve) to 5   (strongly believe). Cntrl Trtmnt   Cntrl Trtmnt   Cntrl Trtmnt 24 What is learnt in the classroom is enough to be successful in maths. 3.0 2.7   2.8 2.7   (0.2) 0.0 25 You need to be good at memorizing to be successful in maths. 4.1 3.9   3.8 3.8   (0.3) (0.2) 26 It is important to find the correct answer to be successful in maths. *** 4.0 4.1   4.1 3.8   0.1 (0.2) 27 Maths can only be learnt from teachers. 1.7 1.6   1.7 1.5   (0.1) (0.1) 28 It is necessary to solve problems correctly and quickly to be successful in maths. 3.7 3.8   3.6 3.7   (0.1) (0.1) 29 Maths problems must be solved in the way shown by the teacher. 2.2 2.1   2.2 1.9   (0.1) (0.2) 30 The exercises in the maths book can only be solved with the methods shown in the book. 2.1 1.9   2.0 1.9   (0.2) 0.0 31 In a mathematics lesson, it is sufficient to know the topics that will be asked in the exam. 2.3 1.9   2.3 2.0   (0.0) 0.1 Note: items with ** indicate statistical significance at the level of 5 percent or higher between treatment and control groups. Green highlighting indicates that results for the treatment group increased relative to the control group, while red indicates that the results for the treatment group decreased relative to the control group. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 29 7.1.2 KNOWLEDGE: Pre test and post test results of the Teacher Assessment A comparison of the pre test results for the treatment and control groups shows that the results were very similar, with an average rate of 53 percent being recorded for the treatment group compared to 55 percent for the control group. The distribution of scores across percentiles indicates that there is a fairly even distribution, although the treatment group had a slightly larger proportion of teachers with scores of less than 40 percent, so in a sense the treatment group started slightly behind the control group. However, in general, this provided a reasonable distribution, with comparable results between treatment and control as well as a good range of achievement levels, from low to high. Figure 6: Distribution of pre test scores in the teacher assessment TREATMENT 4% 8% 18% 14% 19% 14% 7% 9% 6% CONTROL 1% 7% 16% 18% 14% 22% 12% 9% 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 A comparison of the post test scores for the treatment and control groups shows that the treatment group recorded an average score of 54 percent compared to 50 percent for the control group, showing that the treatment group recorded a clear increase. This gap held up under the regression analysis, showing the difference was clearly statistically significant. The difference was stronger when the focus was exclusively on subject items. Whereas the pre test results show a rate of 57 percent for the treatment group and 60 percent for the control group, the post test results show a rate of 77 percent for the treatment group compared to 73 percent for the control group. The percentages themselves are not comparable, because the pre and post tests were not standardized to enable such a comparison. However, a clear shift can be noted, with the treatment group recording a result 3 percent lower than the control group in the pre test, but 4 percent higher in the post test. This indicates a significant improvement in the results of the treatment group relative to those of the control group. The results of the regression analysis again showed the difference to be statistically significant. When breaking down the test items into those that are directly related to the course, those that are indirectly related, and those that are unrelated, another interesting pattern can be observed. As in the case of the results from the pre test, the scores for the control group were again higher than those for the treatment group on items that were unrelated to the course. This would be expected, as if the control group recorded higher average scores on the pre test, they would also be expected to record higher average scores on the post test for items where changes could not be expected to occur as a result of the course. For the directly related items, though, the scores for the treatment group were much higher than those for the control group. Again, this could be expected if the course were effective in improving knowledge on the subjects covered. In the case of the indirect items, the scores for the treatment group were also higher than for the control group. In examining these changes through the regression analysis, the results for the directly related items showed a strong statistically significant change amongst the treatment group relative to the control group. For indirectly related items, the change was still statistically significant, but to a lesser degree. For the unrelated items, there was no relationship. All of these results provide further confirmation that the course facilitated the expected changes in the area of teacher knowledge. 30 The Use of Video in Teacher Professional Development Table 4: Post test average scores by items in terms of their relationship to the CPD course content Direct (%) Indirect (%) Unrelated (%) Treatment 50 53 71 Control 42 51 74 Total 46 52 72 In the area of the items directly related to the course, the vast improvement shown by the treatment teachers in answering open-ended questions is of particular interest. In the pre test, the open-ended questions were answered incorrectly by almost all teachers, with these items simply being left blank in most cases. The following sample item, contained in Figure 7, appeared on both the pre and post test. In the pre test, it was answered correctly by only 4 percent of all teachers, with 79 percent of the teachers leaving it entirely blank. This is obviously a very poor result and it would have been a bad question to include had it not been related to the content of the course, which emphasized teachers’ use of questioning in their lessons, particularly the use of open-ended questions. Figure 7: Example item with open-ended questioning A teacher plans to provide the following problem for her class. “Find the volume of the rectangular prism with the dimensions shown below.” 7cm 5cm 10cm This is a closed question. Rephrase the problem so that it becomes an open ended question. In the post test, an important result emerged. In the case of the control group, 7 percent of teachers answered the item correctly. This slight increase could be due to the fact that the item had previously been included in the pre test and therefore some teachers were better able to answer it because they were familiar with the item. However, in the treatment group, 33 percent of teachers were able to convert the item to an open-ended question. While the figure of 33 percent is still low, this indicates that there was a nearly 30 percent improvement in an area that was previously particularly difficult for teachers. This particular item asked teachers to convert a problem to an open-ended problem. However, similar results were obtained for items that were themselves open-ended items. This improvement in terms of teachers being more comfortable with, and able to correctly answer, open-ended items is an important achievement of the course. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 31 7.1.3 PRACTICES: Pre-survey and post-survey results of Student Perceptions on Practices A great deal of data was collected through the qualitative analysis, in which the teachers explained how they thought their practices had changed as a result of their participation in the course. They also filmed themselves implementing some of the practices they had learned in the course. Such information is useful, although it is also anecdotal in nature and relies on the teacher being honest and accurate in their assessment of how their practices changed. In attempting to determine whether changes in teaching practices actually occurred in the classroom, a unique approach was used. Students of the teachers were surveyed at the beginning and end of the course and asked 27 questions related to what had taken place in their mathematics classroom. This approach relied on students being able to notice the differences and to reliably rate these differences on a scale of 1 (strongly agree) to 5 (strongly disagree). In many cases, students only participated in either the pre or post survey, but not both. There were also some classrooms in which data was collected for only one of the two surveys. This data was not included in the analysis. Only the data from students who participated in both the pre and post-surveys were retained for the purpose of analysis, as it was important to obtain a measure of how each individual student felt teaching practices had changed, as opposed to averages in the class.7 The scale of the survey was from 1 (strongly disagree) and 5 (strongly agree).8 The changes in the average scores between the pre and post surveys were very small, with no change being greater than 0.18. However, when a regression analysis was conducted, statistically significant results did emerge to show differences between the treatment and the control groups. The statements are highlighted below, with green highlighting indicating a statistically significant positive change in the responses of the treatment group relative to the control group, and red highlighting indicating a statistically significant negative relationship. The results appear to indicate that teachers in the treatment group changed their practices in ways that students interpreted to be related to: (i) asking students to explain their answers and explain what they think; (ii) giving students time to explain their ideas and respecting students’ ideas and suggestions; (iii) expecting the students to work through the hard problems; and (iv) providing more feedback and using multiple ways to explain the topics. These suggest that the course was effective in increasing student involvement and in encouraging teachers to use questioning and rich problems. There was also a statement in which a negative relationship was recorded. Students in the treatment group classes showed a statistically significant decrease in the belief that they learn a lot almost every day in their mathematics class. This may be due to more time being spent on problems and therefore less material being covered. This shows that there may be a trade-off involved in increasing student involvement and working on richer problems. 7 A student who rated things in one extreme or another and only participated in either the pre-survey or post-survey would skew the results, whereas of that same student participated in both surveys then his/her results would be valid for comparison. 8 In the conversion of the survey to Indonesian the scale was flipped so that 1 was strongly agree and 5 was strongly disagree. In doing correlations this direction of the scale is confusing because a decrease from pre to post would actually mean more agreement. For ease of interpretation, the results were flipped back so that an increase would mean more agreement. 32 The Use of Video in Teacher Professional Development Table 5: Statements given to students and whether there was a statistically significant change 1 My teacher in this class makes me feel s/he really cares about me. 2 My teacher seems to know if something is bothering me. 3 My teacher really tries to understand how students feel about things.** 4 If you don’t understand something, my teacher explains it another way.** 5 My teacher knows when the class understands, and when we do not. 6 When s/he is teaching us, my teacher thinks we understand when we do not. 7 My teacher has several good ways to explain each topic that we cover in class.** 8 My teacher explains difficult things clearly 9 My teacher asks questions to be sure we are following along when s/he is teaching. 10 My teacher asks students to explain more about the answers they give.** 11 My teacher doesn’t let people give up when the work gets hard.** 12 My teacher wants me to explain my answers -- why I think what I think.** 13 In this class we learn a lot almost every day.** 14 In this class we learn to correct our mistakes. 15 This class does not keep my attention -- I get bored. 16 My teacher makes learning enjoyable. 17 My teacher makes lessons interesting. 18 I like the way we learn in this class. 19 My teacher wants us to share our thoughts. 20 Students get to decide how activities are done in this class. 21 My teacher gives us time to explain our ideas.** 22 Students speak up and share their ideas about class work. 23 My teacher respects my ideas and suggestions.** 24 My teacher takes the time to summarise what we learn each day. 25 My teacher checks to make sure we understand what s/he is teaching us. 26 We get helpful comments to let us know what we did wrong on assignments 27 The comments that I get on my work in this class help me understand how to improve.** Note: items with ** indicate statistical significance at the 5% level or higher between treatment and control groups. Highlight in green indicates treatment groups increased relative to the control group, while red indicates that the treatment group decreased relative to the control group. It is also important to consider that 3-4 months passed between the pre and post surveys. This would mean that the teacher would have taught approximately 39-52 lessons in the intervening period. The fact that statistically significant results emerged indicates that many of the teachers in the treatment group likely changed their practices not just in the case of lessons immediately following the working group session, but that they continued to incorporate those changes in subsequent lessons. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 33 7.1.4 General Conclusions on Quantitative Analysis The results indicate that the course had a positive impact in the three areas of beliefs, knowledge and practices. More interestingly, it appears that specific aspects of change can be noted across the three components. In the case of questioning, teachers in the treatment group expressed an increased belief in the value of questioning to assess student understanding, while they also showed an increase in the area of competency assessment in items related to open-ended questioning. The students also indicated that there was an increase in the use of questioning by their teachers. The students felt that they had to explain their answers and that teachers valued their ideas. Such a consistency across the three components is an encouraging indication of the course’s positive impact, particularly since Session 2 focused specifically on this area. Teachers in the treatment group also expressed an increased belief in the importance of hands-on and inquiry- oriented activities, with these teachers also tending to show an improvement in application-oriented items in the competency test. Unfortunately, there was not an item in the student perceptions survey that would indicate whether such activities had increased in the classroom. This was a design oversight and it would be beneficial to include such items in any subsequent evaluations. Caution in Interpretation While the analysis of the teacher and student surveys indicate that the course facilitated many of the changes targeted in the course, caution should be applied in the interpretation of the extent of the change. The shifts in teacher beliefs are based on a scale of 1 to 5. For statements in which statistically significant changes were recorded, the level of change was to an order of magnitude of 1 or 2 on the ratings scale. Taking into account all the responses by teachers in both the treatment and control groups, in 84 percent of the responses, there were no changes recorded or only a change of one on the ratings scale. A change in response to an order of magnitude of 3 or more on the ratings scale would be an indicator of a strong reversal, but this occurred in only 3 percent of the cases. A complete reversal of belief was almost never noted. This should by no means discount the fact that some statistically significant changes were recorded for the treatment group relative to the control group, but these changes should also be seen as incremental. Figure 8: Change in response from pre to post survey 50 42% 43% 40 30 24% 20% 22% 20 18% 10 8% 5% 7% 6% 0% 1% 2% 0% 0% 1% 1% 0% 0 down 4 down 3 down 2 down 1 no change up 1 up 2 up 3 up 4 The same is true of the student perceptions survey. Extremely dramatic changes should not be expected, particularly because the course was intended to introduce new methods into the teachers’ existing lessons, rather than to completely change the format of these lessons. The fact that statistically significant changes were recorded in the response of teachers in the treatment group is a positive indicator that teachers did in fact make changes in their practices. However, it should be emphasized that these changes involved the introduction of new approaches rather than radical overhauls to teaching methods. 34 The Use of Video in Teacher Professional Development 7.2 QUALITATIVE ANALYSIS The qualitative analysis involved data derived from the process of observation by the core team of session activities in the teacher working groups, surveys and interviews with facilitators and teachers, and the gathering of feedback through workshops and focus groups. 7.2.1 Observation of Session Activities The study team attended nearly all sessions in the capacity of observers. They did not provide any inputs and simply recorded what occurred and then later conducted post-session surveys and interviews. The following subsections contain a summary of their observations. Tools and materials The course sessions tended to be implemented more smoothly when tools and materials such as a laptop, projector, sound system and writing utensils were available. Supply of electricity was an issue in some instances, with some locations having difficulty because the supplies were not consistently available. Observers recommended that steps should be made to ensure materials, instruments, and office supplies are available in sufficient quantities to support the activities. Facilitators In some cases, activities were not conduced effectively by the facilitators. It appeared that in a few cases, facilitators were unsure how to conduct the sessions. At times, the facilitators did not provide clear instructions. Some presentations were unclear and in these cases the materials were not deeply discussed by the participants. The observers considered that in part, this may be due to insufficient training. Interestingly, the activities that were covered in more detail during the facilitator workshop were the ones conducted more effectively in the working group sessions, while some activities that were not explicitly covered in the workshop were conducted less effectively and in some cases were not conducted at all. This highlights the importance of facilitator training and indicates that ideally all session topics should be covered. Facilitators tended to be most effective in presenting the material when they: (i) had the confidence to facilitate and to respond to questions and discuss the results; (ii) had a strategy to handle the training; and (iii) allocated sufficient time for the participants to engage in activities. Teachers’ Participation In all sessions, participating teachers were actively engaged and showed an eagerness to learn. For example, the participants were enthusiastic about engaging in the discussion and simulation of the relation between the volume of a prism and a pyramid. The video clips inspired the teachers to improve the manner in which they conducted inquiry-based learning in their classes. Teachers’ Understanding The following points highlight aspects related to teachers’ understanding in general and on specific topics: 1. The presentations became more meaningful when every teacher used teaching aids; 2. The participants were able to make cubes, pyramids, and to understand the material. They also showed enthusiasm for group work (session 3); 3. The participants regarded the concepts of volume and capacity as new. This led to a question: How to explain these concepts to students in class? (session 4). Observers noted that the teachers might not really understand the difference between volume and capacity. More time was needed to explain this difference to eliminate teachers’ misconceptions. More practice was needed; Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 35 4. The most difficult aspect of the analysis of ELPSA components was on differentiating between “Language” and “Symbol”; 5. Most teacher working groups had difficulties due to lack of time and were therefore not able to discuss the example lesson plan. This could lead to low level of understanding of the lesson plan; 6. In understanding the concepts, practicing in groups or individually tended to be very helpful; 7. Not all participants submitted their homework; 8. The game of misteri bangun ruang (house building mystery) was particularly inspiring for teachers, who expanded upon this in their classroom practice; 9. The topic on effective questioning was extremely effective in building the awareness of teachers on the need to avoid giving standard questions and requiring students to answer in choir. 7.2.2 Interviews and Surveys of Facilitators The facilitators provided feedback through surveys and interviews conducted at the end of each session, and through workshops held after the completion of the course. The following are summaries of comments received. Course Sessions Session 1: Facilitators found the introduction to the ELPSA approach interesting, with participants reacting very well to the concepts. The teachers found that the material was very beneficial in terms of personal development, enabling self-reflection, and facilitating improvements in the learning process. Session 2: In the session on questioning, teachers were interested in all activities, with these activities related to how to ask questions effectively and what to avoid when formulating these questions; how to arrange effective questions in learning activities; how to build open-ended questions; how to develop teacher’s ability to improve the learning process; and how to apply the “Bag of Tricks” game in the classroom context. Session 3: In the session on rich problems, teachers were even more interested in the material and activities, especially in the implementation of 3-dimensional geometry, nets geometry, and tower building activities. Teachers understood the importance of involving the students in learning activities. Session 4: In the session on volume and capacity, challenging questions related to geometry inspired teachers to develop questions that would attract the students’ interest in learning mathematics. Some teachers found that their existing teaching methods to be ineffective and stated that they planned to change these methods. Session 5: In the session on inquiry, the materials used the scientific approach, which participants found to be very relevant to the new curriculum for 2013. Teachers stated that they seldom conducted inquiry activities in their class. Teachers found the contents on: (i) prim volume; (ii) the investigation method; and (iii) designing RPP based on ELPSA to be particularly useful. The teachers learned how to involve the students in mathematical learning by investigation. This session motivated and inspired many of the teachers. However, the materials for this session were found to be too dense and required more time. It was suggested that the material covered in this session should be covered in two separate meetings. Session 6: In the session on lesson plans (RPP) and video presentations, teachers expressed the opinion that the ELPSA model could be applied in the classroom context because it is in line with the scientific approach mandated by the new curriculum for 2013. The teachers appeared to be interested in producing videos and showing these videos to other participants. Finally, the teachers were able to engage in self-reflection after producing and viewing the videos. 36 The Use of Video in Teacher Professional Development Difficulties The facilitators reported difficulties in the following areas: • Some facilitators did not feel that they fully understood the ELPSA framework. Of the different aspects of the ELPSA model, the area considered to be the most difficult was “Application”; • During the session on the second topic, some facilitators struggled to provide good examples of effective questions and explain how to analyze lesson plans. In the facilitator training sessions, there was no exercise on RPP nor on how to fill in the form; • During the session on the third topic, some facilitators indicated that they were not sufficiently confident to explain the difference between volume and capacity to the participants; • Facilitators complained about the limited availability of teaching aids, with not all teachers having cube or weight scales; • During the session on the fifth topic, facilitators had difficulty explaining the volume of a hexagonal prism. In general, facilitators felt that they needed more practice to manage the question and answer sessions. It was suggested that training for facilitators should include simulations of sessions so that they could understand their roles clearly. It was also recommended that the facilitators should be provided with a better understanding of the concepts and materials, including how to fill in the RPP evaluation sheet, before being required to facilitate meetings. It was suggested that to enable better preparation, there should be a longer buffer time between the provision of the training and the implementation of the pilot. In the pilot, the actual buffer time was around 2-3 weeks. One suggestion was to conduct a test to determine the extent to which facilitators were ready to fulfill their functions. Implementation in the working group sessions A theme raised through by facilitators’ comments is that time allocation and management are critically important. The sessions required more than 3-4 hours of intensive activities, with 15 minutes breaks in each session. On average, MGMP meetings run for four hours, but with the pace of the activities much slower than was the case with this course. Sometimes facilitators were unable to complete the materials assigned for the sessions within the time allocated to the working group sessions. Implementation in the classroom Facilitators expressed the belief that most of the participating teachers would be able to implement the concepts and knowledge they had acquired through the course in their respective classrooms. The constraints most commonly expressed were the time limitations on lessons and the limited availability of teaching aids. Suggestions for Improvement Some important suggestions generated through the feedback process include: • The facilitators need manuals to facilitate group activities; • It would be better to conduct the meetings during the first semester or early in the second semester, as teachers are generally much more busy during the second semester; • Most schools do not have mathematics laboratories, despite the fact that there is a clear need for teaching aids and place to store them. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 37 7.2.3 Interviews and Surveys of Teachers (participants) At the end of each session, facilitators and teachers were asked to complete brief surveys to generate feedback on the content, use of the practices in their own classrooms, and what aspects of the content that they found most and least effective. Activities and topics Most of teachers expressed the belief that the activities provided significant benefits. However, some teachers felt that some activities were less beneficial than others. For example, they stated that videotaping could be manipulated, some game activities were not effective, and that they did not have sufficient time to prepare teaching aids. Teachers stated that in all six meetings, many of the topics interesting. In particular, they found the following topics to be of value: the ELPSA approach, effective open-ended questions, proofing prism and pyramid volume, and presenting lesson plans. Some teachers expressed agreement with the statement that the sessions provided additional knowledge and enriched lesson activities. In addition, teachers expressed the belief that the activities related to the following were beneficial: the use of the ELPSA approach to attract students’ attention; the effective use of questioning; the concept of the volume of a prism and pyramid; the use of videos for self-reflection; methods to diagnose students’ learning difficulties; method for learning 3-dimention nets; and methods to address misconceptions related to volume and capacity. Teachers’ understanding and difficulties Most of teachers stated that it was easy to understand the activities, topics and concepts presented through the course. They agreed that the teaching aids and videos used were particularly helpful. However, they also stated that they had difficulties in understanding some concepts. In particular, they stated that they had difficulties in explaining volume and capacity, the ELPSA approach, how to make nets of pyramid, developing lesson plan, and developing effective problems. ELPSA was a new concept for most participating teachers. While some teachers stated that the approaches in the course were not entirely new to them, other teachers stated that some of the materials covered through the course were entirely new, including the concept of volume and capacity, the use of questioning techniques, determining the capacity of a three-dimensional object, and the use of video as a learning tool and to encourage self-reflection. Implementation in the classroom Not all activities included in the course could be applied in the classroom. Teachers stated that the main constraint was the time limitation, with the activities in the course consume a significant amount of time, with much extra preparation time required. They also stated that some activities required extra expenditure, especially when teaching aids were required. Above of all, teachers’ readiness was identified as the key factor for classroom implementation. Homework assignments and tasks Many of the participating teachers completed their assigned homework tasks. They found some benefits in completing the homework, with these tasks providing them with the opportunity to review materials, to achieve professional development, to sharpen their creativity, and to increase their knowledge. Implementing the practices in the classroom helped the teachers to better understand the concepts. Some teachers applied the concepts of questioning, the use of open-ended questions, the demonstration method and the development of three-dimensional nets in the classroom. On the other hand, some teachers did not apply all these concepts and approaches in their classroom practice. Teachers stated that the main reasons that they did 38 The Use of Video in Teacher Professional Development not apply these concepts were that they did not fully understand them as they were still learning them and that they did not have the necessary teaching aids. Comments regarding ELPSA Teachers expressed a number of opinions on the ELPSA framework, which can be summarized as follows: (i) the ELPSA approach would be useful if it were utilized in the teaching of other subjects; (ii) ELPSA should be piloted in classrooms that lacked lack of facilities; (iii) additional problem solving materials are needed; and (iv) a comparative study of the teachers who have implemented ELPSA would be useful. Filming own lessons Teachers were specifically asked: “What is your opinion regarding the video you produced of your own teaching practices and regarding the videos produced by your colleagues?” Teachers’ responses to this question included the following: • “We very much appreciate the use of the video in the teaching learning process.” • “The enquiry type of learning is suitable for implementation at grades 7, 8 and 9.” • “The use of the video enabled us to engage in self-reflection and enabled us to become more professional teachers.” • “Students are very interested in the process of learning.” • “The use of the video methodology made it possible to watch and reflect on our teaching practices over and over again.” • “Developing good videos is a time-consuming process and requires patients.” Topics that could enhance the course Teachers were also asked “What things do you want to know related to the course topics?” • Teachers’ responses included the following: “We want to dig deeper into the language of mathematics so that we can minimize any misconceptions.” • “We want to learn how to use video clips for the teaching of other subjects.” • “We want to learn how to use the language of mathematics properly.” • “We want to learn more deeply about effective questioning.” • “We are eager to learn how to be good teachers who use effective teaching methods to encourage productive student learning.” 7.3 FEEDBACK THROUGH WORKSHOPS WITH FOCUS GROUP DISCUSSIONS Two workshops were held with the facilitators after the completion of the course. In the first workshop, additional information was gathered and facilitators were able to share their general thoughts on the pilot, with particular focus on the process and content. The second workshop involved presentations by the facilitators, who shared their own videos and provided their own perspectives regarding the ELPSA framework. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 39 8 Key Lessons Learned and Considerations Many lessons were learned through the implementation of the course in terms of its content, implementation and effectiveness. These lessons learned also raise questions regarding the provision of continuous professional development in general. The following is a summary of the key lessons and considerations, as well as a discussion of sustainability in terms of scaling up and expanding the practices covered in the course in the area of mathematics and for other subjects. 8.1 CONTENT The content of the sessions was generally very well received. There were almost no comments expressing disagreement with the course’s approach or topics and there was near universal agreement that the content was beneficial. In general, participants expressed the belief that the ELPSA framework and the topics of the course were highly relevant to their own practice. There were mixed responses regarding the level of difficulty of the content. Some participants indicated that the concepts and approaches were not entirely new, but that they still found it valuable to develop their ability to implement them in the classroom. Other teachers indicated that the content was challenging and felt that they did not acquire a sufficient level of knowledge to be confident about implementing the new practices in their own classroom. These teachers tended to regard the approaches as valuable, but indicated that they required more support if they were to implement them effectively. Some teachers also indicated that they would like to go into more depth in some of the topics. In particular, some teachers felt that it would be beneficial to spend more time on the topic of questioning and ways in which a culture of questioning could be created in the classroom. On the basis of this feedback, it might be desirable to consider expanding the course to include a second session on questioning. Another approach would be to develop a separate course consisting of three sessions that could go into much greater depth on this topic. The issue of the time required both to conduct the working group sessions and to implement the sample lesson plan also arose. It appears that in some of the sessions, the content was too dense and could not be completed effectively within the 3-4 hour duration of the working group sessions. This led to some activities not being conducted or being rushed. Revisions to the course should reduce the content to ensure that all activities can be implemented appropriately. 40 The Use of Video in Teacher Professional Development 8.2 IMPLEMENTATION 8.2.1 Using Working Groups for the Delivery of CPD Teacher working groups have both advantages and disadvantages as a system of delivery for CPD, as previously discussed in Section 4.3. Some specific issues that the pilot revealed included the following: • The level of attendance at working groups can vary significantly: While in some working groups, teachers from all schools in the cluster attend, others rotate teachers within the school. In many districts, there is a policy of having a day dedicated to working groups for particular subjects. For example, working groups for mathematics teachers may be conducted on Thursdays, so no mathematics classes are scheduled for that day. However, in Jakarta, this is not the case, so teachers need to find time in their schedules to attend. This negatively effected the level of attendance. Attendance was also affected by conflicting obligations, as discussed in the next point. • A course implemented through a working group may be affected by teachers’ conflicting obligations: Teachers have many conflicting obligations, being required to engage in training on the new curriculum, to prepare for national examinations, and to conduct many other activities. This creates significant challenges in terms of maintaining the attendance of participants in working groups throughout the semester. • Teachers have different interests and needs, which may not all be addressed through a single course: While some topics may be of interest to most or all teachers, different teachers will have different professional development interests and needs. It may be worth considering adopting a certain degree of flexibility to enable teachers to engage only in the specific activities that meet their needs or to enable multiple sub- groups to conduct separate activities/sessions. • The quality of the facilitator played a critical role in ensuring the effectiveness of working group activities: This point is discussed in greater detail below. • The support provided to working groups is not consistent across different schools and districts: School and educational agency authorities have varying perceptions of the degree to which working groups are the responsibility either of the district, the school, or the teachers themselves. One district commented that working groups are out of their jurisdiction and that the district could not oblige teachers to attend working groups. 8.2.2 Importance of Facilitators At the design stage, consideration was given to the idea of developing stand-alone courses that would not require the training and use of facilitators, but could be used in a fully self-directed manner by teachers. While this approach would be less expensive and could be more easily and widely distributed to working groups, the content itself would have had to be simplified and include even more detailed step-by-step instructions. There would have also been a greater reliance on the motivation of the members of the working groups to implement the course on their own and to engage in all the associated activities. Through the piloting of the course, it became clear that the decision to train and deploy facilitators was correct, with facilitators playing a vital role in ensuring the effectiveness of the course. By training and deploying facilitators, the necessary leadership to conduct the activities and to facilitate and understanding of the content was provided. It also allowed for the establishment of a support network through which facilitators were able to share their experiences and pose questions with facilitators in other working groups. In addition, the level of commitment and understanding of the facilitators were important in determining how effectively the sessions were conducted. In some cases, facilitators did not have a sufficient level of understanding of the materials. In these cases, it was found that the sessions were much less effective. This indicates the importance of facilitator training, underscoring the need to allow sufficient time to go through all key activities. The most effective sessions tended to be those in which simulations were conducted during the training process. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 41 This indicates that additional practice and a greater use of simulations in the training process would be beneficial. Training for facilitators should be more practical. Facilitators should engage in simulation sessions so they clearly understand their role in facilitating teacher working group meetings, especially in providing input for the development and implementation of lesson plans. Facilitators have to prepare themselves beforehand so they understand the topic and are able to manage the meetings well. It would be good if facilitators could provide some other examples to enrich the courses. 8.2.3 Responsibility for the Provision of CPD One challenge created by the decentralization of the responsibility for the provision of educational services to district governments is that continuous professional development is now the responsibility of these governments, through district-level educational agencies. The capacity of districts to provide high quality CPD varies, but is generally very low. How local governments perceive their role in the provision of CPD also varies. In the case of the pilot, the study team was able to meet with district-level officials and propose that teachers receive credit for successful completion of the course. But how could this process be scaled up? Under current arrangements, if a working group or set of teachers were to want to conduct the course, they would need to reach an agreement with the district to receive credit. Alternatively, should CPD be a supply-driven process in which districts offer the CPD activity to the working groups? Should it be voluntary or obligatory for all teachers to participate? The answer to these questions is beyond the scope of this study. However, challenges in the implementation of the course did underscore the need to clarify the role of the MOEC, provincial and district level agencies, and of schools and teachers in the provision of CPD, particularly in terms of how teachers will receive credit for their participation. For the delivery of CPD through online courses, such as courses offered by P4TK Mathematics, it appears that there is already a mechanism for teachers to receive credit. However, it is important for the rules and processes to be clearly defined. Educational agencies, school authorities and other stakeholders should support and opportunities for teachers to attend the teacher working group activities and to create opportunities for them to do so. At present, teachers need a letter of permission from their principal to attend teacher working group meetings. Teachers’ CPD activities should be mandatory and scheduled. 8.3 EFFECTIVENESS The results of the impact evaluation indicate that the course led to statistically significant changes in teacher beliefs, improvements in teacher knowledge, and changes in the classroom. The qualitative results also indicate that teachers valued the professional development activities and felt that they benefited from these activities. Input from the facilitators, teachers and observers have factors to identify factors that lead to effective implementation, but also indicate that there were aspects that hindered the effectiveness of the course. Working group meetings tend to be most effective when these conditions are present: (i) the topics meet with the needs of the participants; (ii) the facilitators prepare and perform the activities well; (iii) learning tools, stationeries, laptops, projector, and other facilities are available; (iv) all participants are actively involved in discussions, questioning, engaging in practice, and exploring through trial and error; (v) the activities can be effectively implemented within the allocated time (180-240 minutes). 42 The Use of Video in Teacher Professional Development A key area for revision relates to be content load, given the time constraints that apply to working group sessions. In particular, the content of the course has to be adjusted to achieve a better fit with the time available so that the teacher learning process is more effective. Most lesson plan analysis, which was intended to be conducted during the sessions, had to be assigned as additional homework because of the lack of time. Each course required 3-4 hours of intensive activities, with 15 minutes break each session. Regular MGMP meetings usually run for four hours, but with a slower pace of activities. Specifically, Session 2, on questioning, could be conducted over two meetings, since the activities required more time than most others. Session 5, on enquiry learning, could also be conducted over two meetings. 8.4 SCALING UP AND SUSTAINABILITY There is significant interest in CPD courses of this type. Teachers who participated in the pilot were generally very enthusiastic and requested additional support. Teachers in the control group were also very interested in utilizing the course in their working group in the future. Many of the educational agencies in the participating districts are developing plans to support the rollout of the course to the rest of the teacher working groups in their district. In dissemination events with district officials and teachers, there were many requests for support. However, despite this enthusiasm, scaling up the implementation of the course requires careful consideration. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 43 9 Wider Policy Implications Policies related to teaching practices over the past decade have tended to emphasize more modern, student-centered teaching approaches, with these approaches involving a higher level of student interaction and a more intense focus on reasoning and higher order thinking. For example, Regulation 19/2005 states: “The process of learning in the educational unit should be organized to be interactive, inspiring, fun, challenging, motivating learners to actively participate and provide enough space for innovation, creativity, and independence in accordance with their talents, interests, and physical and psychological development of learners.” A similar statement was also included in Regulation 32/2013. The video study results indicate that eighth grade mathematics teachers tend to use traditional methods and that the approaches tend to be procedural, rather than encouraging reasoning and higher order thinking. Importantly, the results of the video study indicate that when teachers used more interactive methods that encourage reasoning, students tend to have better learning outcomes. This shows that there are benefits to the teaching methods that have begun to be encouraged over the past decade. However, it is also clear that in mathematics classrooms (and likely with other subjects), there is a gap between the methods that are encouraged and methods that are actually put into practice. How can teachers incorporate more progressive methods into their arsenal of practices? Teachers clearly require support, and there are no quick fix solutions. Changing practices is notoriously difficult and requires long-term, continuous, and intensive support. This pilot was specifically designed to strengthen teacher knowledge and to improve teaching practices in the areas that encourage greater student interaction and that utilize inquiry-based methods that promote student reasoning. The results indicate that the approach used in this model changed teacher beliefs, improved teacher knowledge and led to changes in practices used in the classroom. Although the results were positive, a course of this type must be seen merely as a seed for the development of a much larger support system. Teachers will need a much higher level of support provided on an ongoing basis in order to fully build and strengthen their subject and pedagogical knowledge. It is clear that the system for the provision of CPD at present is not working effectively in Indonesia. Since decentralization, teachers have received less CPD and the quality and relevance of the offerings has declined. While district governments are responsible for the provision of CPD, the majority do not have sufficient capacity or the willingness to provide opportunities for effective CPD to teachers. In some cases, local governments do not even necessarily consider it their obligation to provide CPD. Even for those that do see it as their obligation, in many cases, the offerings tend to be haphazard, of low quality and of limited relevance. Very often, they consist of short workshops in which a speaker is brought in to give a presentation. While teachers receive credit for participation in these workshops, they often fail to meet the needs of teachers and have little impact on classroom practices. At least in the short-term, it is not realistic to expect local governments to be able to develop high quality CPD activities. In addition, a larger framework is required to give structure to CPD as a whole, rather than having patchy, poorly integrated offerings. 44 The Use of Video in Teacher Professional Development The P4TK Mathematics centers do tend to offer good courses and programs, but these benefit only a small number of teachers each year. With a country as large and diverse as Indonesia, in-house P4TK training at their centers for the majority of teachers is simply not a feasible large-scale solution. Online methods may provide a way to reach a larger number of teachers, with the pilot geometry course serving as an example of how online methods may be effectively utilized. This methodology certainly has good prospects, but it cannot be relied upon as the only method for the provision of CPD. Teacher working groups offer many advantages, as specified in detail in this report. Among other advantages, they can be a very effective means of building skills through the in-on-in process, which ensures that what is learned can be applied in the classroom and then reflected upon with fellow teachers. However, there are many challenges to the use of teacher working groups as a large-scale method of providing CPD. Working groups function very differently from district to district, or even from school to school. This makes it difficult to develop and implement a program consistently. It may also necessitate the development of high quality courses that can be run by teachers themselves. The development of the geometry course used in this pilot is required extensive time and resources. However, the payoff is that it can have a significant positive impact in terms of improving the level of teachers’ subject knowledge and pedagogical knowledge. Most importantly, it may result in real positive changes to what occurs in the classroom. There can be no single method for the provision of CPD. A holistic approach, with many different offerings that utilize a range of different methodologies, is necessary. Within Indonesia’s teacher professional management system, individual teachers are required to develop an annual CPD plan. To achieve this effectively, they should have a catalog of offerings that can be used to address their specific needs. This requires a system that provides many options under a holistic dynamic pedagogies framework. This CPD course has planted a seed for the professional development of mathematics teachers. The approach adopted through the course will be continued through a project in the province of NTB under a DFAT Global Partnership for Development (GPFD) grant. However, such approaches need to be applied for teachers of other subjects and for every level of schooling, from early childhood through senior secondary. The central government is best positioned to develop the necessary framework and to produce high quality activities that can be implemented across Indonesia. However, ultimately, it is districts that are responsible for the provision of CPD. In this capacity, they must provide support for their teachers by establishing mechanisms that will give access to and credit for CPD activities. None of this is easy, requiring inputs and support from many stakeholders. However, it is critically important if Indonesian students are to benefit from improved learning outcomes. For more than a decade, Indonesian teachers have been provided with very little support and only limited opportunities to develop the skills necessary to improve Indonesia’s human capital. The level and quality of teachers’ subject knowledge and pedagogical knowledge is still generally very low, which ultimately impacts the ability of Indonesian students to grow up and become productive citizens. It is vitally important for the nation’s future to address this. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 45 References Kemmis, S., & Grootenboer, P. (2008). Situating praxis in practice: Practice architecture and the cultural, social and material conditions for practice. In S. Kemmis & T. J. Smith (Eds.), Enabling praxis: Challenges for Education (pp. 37-62). Rotterdam, The Netherlands: Sense. Kemmis, S., & McTaggart, R. (1992). The action research planner (3 rd ed.). Victoria, Australia: Deakin University Press. Matsumoto, D., & Juang, L. (2013). Culture and psychology. (5 th ed.). Belmont, CA: Cengage. Mattsson, M., Vidar Eilertsen, T., & Rorrison, D. (Eds.). (2011). A practicum turn in teacher education. Rotterdam, The Netherlands: Springer. Ponte, P., & Rönnerman, K. (2009). Pedagogy as human science, building and action research: Swedish and Dutch reflections. Educational Action Research, 17(1), 155-167. Ragatz, A. (2013), Teaching Practices of Indonesia’s Eighth Grade Mathematics Teachers: Formation, Execution and Effect, Dissertation Thesis, University of Oxford, Department of Education Schatzki, T. R. (2002). The site of the social: A philosophical account of the constitution of social life and change. University Park, PA: University of Pennsylvania Press. World Bank. (2014). Teacher Reform in Indonesia: The Role of Politics and Evidence in Policy. Indonesia: The World Bank. Zhang, M., Lundeberg, M., Koehler, M.J., & Eberhardt, J. (2011). Understanding affordances and challenges of three types of video for teacher professional development. Teacher and Teacher Education, 27, 454-462. 46 The Use of Video in Teacher Professional Development Appendices APPENDIX 1: OUTLINE OF ACTIVITIES Week Professional Activities For Teachers Between In-class Activities Development Session Sessions 1 Session 1 Main assignment introduced (to be Student attitude test (pre). completed at the conclusion of the Introduction to the program): To develop a lesson plan to program, ELPSA & be used within the RIPPLE geometry PCB unit using ELPSA framework. 3 Session 2 Teachers develop 3 appropriate DESIGNING QUESTIONS: Teacher opened-ended questions. Write and delivers the 3 open-ended Teacher and student submit a reflection of the strengths questions they have developed for questioning and/or weaknesses of classroom their between session activity with implementation. the students in their classroom. 5 Session 3 Write and submit a reflection of the VIDEO STIMULUS: Teachers strengths and/or weaknesses that replicate in their classroom the Volume and capacity occurred of classroom implementation hands on activities demonstrating of video stimulus. the definition of volume and capacity from the PD session. 7 Session 4 Teachers are required to find a closed REDESIGNING PROBLEMS: Teachers question about 3D shapes from a will use the open-ended problem Problem based textbook and rewrite it as an open- they redesigned from the textbook learning ended problem. in their classroom. 9 Session 5 Teachers will write and submit a Student attitude test (post). reflection on five things that they have Inquiry learning found significant from the professional development sessions. Teachers will complete main assignment. 11 Session 6 DEVELOPING ONLINE ENVIRONMENT: Teachers MAIN ASSIGNMENT implement and video lesson DUE: Group plans they have developed in presentation of their classrooms. Videos will be the lesson plans uploaded as a teaching resource developed by the and development of online teachers using the environment. ELPSA framework. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 47 APPENDIX 2: GENERAL PROFESSIONAL DEVELOPMENT CONCEPTS This study draws on research into the nature of teachers’ professional knowledge and how such knowledge informs professional practice and decision-making. To date, studies that focus on effective teaching practices have yet to address the complex interactions that exist within and across the teaching profession. In order to consider the elements which influence and inform the day-to-day practices of the classroom teacher, the focus must shift from the individual classroom practices of practitioners to a more encompassing meta-level of analysis that locates these practices within cultural, contextual and curriculum arrangements that make their particular practices possible (Schatzki, 2002). In this view of practices, practitioners’ dispositions and actions are not formed entirely by individuals in their own right, but rather in interaction with these arrangements (Kemmis & Grootenboer, 2008, p. 50). Thus, teachers’ professional development needs to be framed within learning opportunities that consider cultural practices, the contextual aspects of teachers’ content-discipline knowledge and prevailing curriculum and pedagogical practices. Teachers’ practice, in essence, takes place within any educational setting. By placing the individual teacher or school leader at the center of decision making, it acknowledges that professionals need to take more responsibility for their own learning. At the same time, it challenges education systems to provide individuals with the professional autonomy they need to design learning not only in relation to their own needs, but also in relation to the context in which they operate. In this respect, it must be emphasized that individuals need not learn in isolation. Rather, they can do so with other like-minded community members that share common interests and goals. Indeed, it is envisaged that groups of individuals (such as groups of teachers within a school or across schools) will work towards achieving identified professional learning goals, although these pathways will still be personally driven. These new levels of professional responsibility should lead to higher levels of expertise within the profession, since professional learning maps to professional practice(s). Such teacher working groups, which have already been established in Indonesia, provide an appropriate medium for this type of communication and engagement. From this basis, the course draws on the principles of Action Research (Kemmis & McTaggart, 1992). Action Research is an approach to improving education by changing it and learning from the consequences of these changes. This approach advocates participation, collaboration, the establishment of self-critical communities, and self-reflection through a systematic learning process in which individuals theorize and reach their own conclusions regarding the relationship between circumstance, action and consequence in their own specific situation. This type of thinking involves people making critical analyses of the situations in which they work by identifying issues, gathering and recording evidence, keeping personal journals, sharing in supportive group discussions and working toward the improvement of their own practices.9 Both conceptually and practically, video can be used to good effect to promote the principles of Action Research (Lowrie, 2013). These concepts are elaborated further in Section 4.2 (Foundational Concepts of Professional Development). 9 Evidence of the success of this type of approach to professional learning is found in Norway, The Netherlands and Sweden. These countries are involved in a Charles Sturt University initiative within the Research Institute for Professional Practice, Learning and Education (RIPPLE) known as the Pedagogy, Education and Practice (PEP) group. Researchers within this group have utilized action research principles within their projects with great success across the spectrum of education fields (see for example, Mattsson, Vidar Eilertsen, & Rorrison, 2011; Ponte & Rönnerman, 2009). 48 The Use of Video in Teacher Professional Development APPENDIX 3: INSTRUMENTS Teacher Survey – Beliefs and Practices Questions Teaching Mathematics In your opinion, how important are each of the following for effective mathematics instruction in the grades you teach? Please indicate from 1 (not important) to 5 (very important): Please choose the appropriate response for each item: Not Very important important   1 2 3 4 5 8 Make connections between mathematics and other disciplines. o o o o o 9 Have students work in cooperative learning groups. o o o o o 10 Have students participate in appropriate hands-on activities. o o o o o 11 Engage students in inquiry-oriented activities. o o o o o 12 Use calculators. o o o o o 13 Use computers. o o o o o 14 Engage students in applications of mathematics in a variety of contexts. o o o o o 15 Use performance-based assessment. o o o o o 16 Use informal questioning to assess student understanding. o o o o o Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 49 Learning Activities In your mathematics lessons, how often do you usually ask students to do each of the following? Please indicate from 1 (never) to 5 (very often): Please choose the appropriate response for each item: Never Very often   1 2 3 4 5 17 Work on problems for which there is no immediately obvious method of solution. o o o o o 18 Use manipulatives to solve exercises or problems. o o o o o 19 Participate in student-led discussions. o o o o o 20 Work in cooperative learning groups. o o o o o 21 Read other (non-textbook) mathematics-related materials in class. o o o o o 22 Share ideas or solve problems with each other in small groups. o o o o o 23 Investigate mathematical concepts through other disciplines. o o o o o 24 Engage in hands-on mathematical activities. o o o o o 25 Play mathematics games. o o o o o 26 Design or implement their own investigation. o o o o o 27 Work on models or simulations. o o o o o 28 Use manipulatives to explore a concept. o o o o o 29 Work on extended mathematics investigations or projects (a week or more in duration). o o o o o 30 Write reflections in a notebook or journal. o o o o o 50 The Use of Video in Teacher Professional Development The Process of Learning Mathematics Rate each of the following belief statements according to how strongly you believe each statement. Rate each item 1 (strongly disbelieve) to 5 (strongly believe). Please choose the appropriate response for each item: Strongly Strongly Disbelieve Believe   1 2 3 4 5 31 What is learnt in the classroom is enough to be successful in maths. o o o o o 32 You need to be good at memorizing to be successful in maths. o o o o o 33 It is important to find the correct answer to be successful in maths. o o o o o 34 Maths can only be learnt from teachers. o o o o o 35 It is necessary to solve problems correctly and quickly to be successful in maths. o o o o o 36 Maths problems must be solved in the way shown by the teacher. o o o o o The exercises in the maths book can only 37 be solved with the methods shown in the book. o o o o o In a mathematics lesson, it is sufficient to 38 know the topics that will be asked in the exam. o o o o o Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 51 Student Perception Survey Please rate this 1=Strongly Disagree 5= Strongly Agree No Question 1 2 3 4 5 1 My teacher in this class makes me feel s/he really cares about me. 2 My teacher seems to know if something is bothering me. 3 My teacher really tries to understand how students feel about things. 4 If you don’t understand something, my teacher explains it another way. 5 My teacher knows when the class understands, and when we do not. 6 When s/he is teaching us, my teacher thinks we understand when we do not*. 7 My teacher has several good ways to explain each topic that we cover in class. 8 My teacher explains difficult things clearly 9 My teacher asks questions to be sure we are following along when s/he is teaching. 10 My teacher asks students to explain more about the answers they give. 11 My teacher doesn’t let people give up when the work gets hard. 12 My teacher wants me to explain my answers ─ why I think what I think. 13 In this class we learn a lot almost every day. 14 In this class we learn to correct our mistakes. 15 This class does not keep my attention ─ I get bored.* 16 My teacher makes learning enjoyable. 17 My teacher makes lessons interesting. 18 I like the way we learn in this class. 19 My teacher wants us to share our thoughts. 20 Students get to decide how activities are done in this class. 21 My teacher gives us time to explain our ideas. 22 Students speak up and share their ideas about class work. 23 My teacher respects my ideas and suggestions. 24 My teacher takes the time to summarize what we learn each day. 25 My teacher checks to make sure we understand what s/he is teaching us. 26 We get helpful comments to let us know what we did wrong on assignments 27 The comments that I get on my work in this class help me understand how to improve. *Reverse coded item. Agreement represents an unfavorable response. Adapted from MET project Policy AND PRACTICE BRIEF Asking Students about Teaching. Student Perception Surveys and Their Implementation. Bill and Melinda Gates Foundation. September 2012. 52 The Use of Video in Teacher Professional Development Observation Instrument OBSERVATION INSTRUMENT MGMP SESSION USING THE GEOMETRY COURSE GENERAL DATA Date Session Number (1-6) Begin Time End Time Number of Teachers attending Number of teachers attending who were previously registered Number of teachers attending who were not registered MATERIALS 1. What materials did the facilitators use? (e.g. projector, computer, teacher aids, photocopies) 2. Were the facilitators missing any necessary tools to conduct the session? FACILITATORS 3. From the way the session was conducted, do you think the facilitators had a good general understanding of the objective of the session? (Describe using specific examples from the session) 4. Were the facilitators able to effectively conduct the session? (Explain) a. What activities/aspects were most difficult? b. What activities/aspects went best? 5. Facilitator teamwork/interaction a. How did the two facilitators work together? b. Did one facilitator always lead or did they take turns with sessions? Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 53 TEACHER PARTICIPATION 6. Describe the level of involvement of the teachers in the session. a. Did they appear to be engaged in terms of paying attention when the facilitators or other teachers talked? b. Did they actively participate in the discussions? c. Did they actively participate in the group activities? d. Did it seem that they were enjoying the session or were they bored/uninterested? 7. If there was an assignment for the teachers from the previous session: a. Was the previous assignment discussed in this session? If so, describe the discussion. (Amount of time spent discussing, points raised, any confusion or conflict, etc.) b. Approximately what proportion of the teachers appeared to have done the assignment? (% of total teachers) TEACHER UNDERSTANDING 8. In your estimation, what percent of the teachers were able to understand and follow the session topic? (Judgment through observation and possibly informally asking some teachers.) 9. If some teachers seemed confused, what were the areas of confusion? 10. From what you observed, do you think the teachers already knew the material and concepts or was the session a new learning experience for them? 54 The Use of Video in Teacher Professional Development OBSERVER OPINION ON CHANGES/IMPROVEMENTS IN YOUR OPINION AS THE OBSERVER: 11. What aspect(s) of the session do you think were particularly effective? 12. What aspect(s) do you think could be changed to make the session more effective? 13. GENERAL CONCLUDING COMMENTS Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 55 Teacher Post-Session Survey TEACHER SESSION SURVEY INSTRUMENT MGMP SESSION USING THE GEOMETRY COURSE Name____________________________________________________________ Date ____________________________________________________________ 14. Overall, what was your impression about the session? a. What was your favorite activity and why? b. What was your least favorite activity and why? 15. Were you able to understand the session topic and concepts? Were there any concepts that you found to be particularly difficult? 16. Which of the concepts were new to you? 17. Do you think you will be able to incorporate the ideas of the session into your own classroom activities? Why or why not? 18. If there was an out-of-MGMP-session assignment (homework or in-class activity to conduct) from the previous session: a. Did you do the assignment? b. Did you find it useful? If so, in what ways was it useful? c. Have you attempted to implement any of the concepts in your classroom? If so, what concepts? Do you think it was 56 The Use of Video in Teacher Professional Development Facilitator Post-Session Interview POST-SESSION INTERVIEW WITH FACILITATORS (Interview both facilitators) • How do you think the session went overall? • What aspects do you think were particularly helpful/useful for the teachers? • Did you feel prepared to run the session? (Why or why not?) • What activities were particularly difficult to conduct? • Do you think the teachers understood the content of the session? • Do you think the teachers will be able to implement the concepts in their own classrooms? Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 57 APPENDIX 4: SUMMARY OF ONLINE COURSE CONDUCTED BY P4TK MATHEMATICS During the training for the implementation of the geometry course pilot program in January 2014, P4TK Mathematics expressed an interest in converting the course into an online course and hosting it on their server. This online course can be accessed through the URL at http://etraining.p4tkmatematika.org/. Through their participation in the design and implementation of this online course, P4TK hoped to achieve the following goals: • To utilize the new video case studies approach to facilitate the professional development of mathematics teachers and to adapt the newly developed course for use in online sessions. • To develop a speedy, economical mode for the delivery of professional development activities through the use of online learning methodologies. P4TK considered that the online mode of delivery may be an effective option to improve the capacities of more than one million primary school classroom mathematics teachers and nearly 110,000 mathematics teachers at the junior secondary school level and above. With the large number of mathematics teachers in Indonesia, the MoEC hope to use online learning to accelerate the teacher training process while controlling the associated costs. • To strengthen the implementation of the new 2013 curriculum. P4TK Mathematics was mandated with the task of training teachers in the areas of knowledge and skills required for the implementation of the 2013 curriculum. P4TK considered the ELPSA geometry course for eighth grade teachers to be a potentially useful tool for this purpose. Through the delivering ELPSA geometry course online, there were also opportunities to benefit in terms of: 1. Increased access: As stated above, by adapting the course for online use, training providers such as P4TK can provide increased access, enabling more teachers to benefit from the training process. 2. Potential to enhance the quality of learning: The P4TK mathematics has expressed the belief that the use of the course in an online mode has the potential to enhance the quality of learning. In addition, online courses can be readily evaluated over time, refined, and adjusted for future use. 3. Flexibility of delivery: By making the course available online, it is sufficiently flexible for use in either online distance education programs or in face-to-face training programs conducted in various settings, such as MGMP meetings, programs conducted on P4TK premises, or in collaborative partnerships involving teachers from different schools, or in other sites. 4. Extended use in the classroom: Teachers will be able to make use some of the online materials in their classrooms with their students. 5. Use of technology: Information, mobile and other technologies can be readily used to gather information and knowledge, to develop information and knowledge, and to present information and knowledge. 6. Dissemination: By making the materials available online for use by Indonesian mathematics teachers through periodic online courses, the geometry course will be disseminated to a broader audience, potentially benefiting 73,000 junior secondary school mathematics teachers, as well as teachers of other levels and other stakeholders. The purpose of this section is to discuss issues related to the implementation of the course on the basis of the findings of the World Bank team, who observed the process and engaged with P4TK Mathematics throughout this process. This brief report is intended for internal circulation within the World Bank. 58 The Use of Video in Teacher Professional Development Implementation The process involves three stages: (i) the preparatory phase; (ii) the online course delivery phase; and (iii) the evaluation phase. These three phases took place in the period from March to October 2014. i. Preparatory Phase: During this phase, planning and consultation processes took place to secure agreement on the online learning syllabus, the structure of the program, and the development of online learning resources (additional readings, multimedia, etc.). Milestones Time Frame Preparatory Phase Consultations 3rd week of March – 1st week of April Online course design (conversion from face to face course) 2nd – 3rd week of April Content conversion and online media development 4th week of April – 3rd week of May Review and revisions 4th week of May Internal coordination for budgetary preparation June 2014 Technical preparation July 2014 Recruitment of participants 1st – 2nd week of August Online Learning Delivery Phase Delivery (1st phase)* August 19 – September 8, 2014 Conclusion Phase Evaluation 2nd – 4th week of September 2014 Dissemination/Knowledge Sharing October – December 2014 *During the preparatory phase, P4TK teams decided to deliver the course in two parts due to budgetary and time constraints. Thus, the course was implemented according to the following structure and schedule: Phase 1 Session/Book 1 to 3 August – September, 2014 83 hours Phase 2 Session/Book 1, 4 to 5 First quarter of 2015 83 notional hours ii. Online Course Delivery Phase: The first phase was conducted in the period from August 19 to September 8, 2014, with 112 enrolled participants. A fully online Learning Management System was used for the delivery of the program (http://diklatonline.p4tkmatematika.org), with additional support provided through the use of other means of communication, particularly a telephone support line and a dedicated Facebook group. The learning activities involved the use of readings, forum discussions, weblogs to record reflections, involvement in a project, the production of video documentation, and participation in online quizzes and surveys. The usage of web 2.0 technologies was also introduced in these online trainings, through the use of facilities such as cloud file storage (using Google Drive), online blogging platforms (using Kompasiana and Wordpress), online video publication (using YouTube) and online community development (using Facebook). The structure of the course was as follows: Activities Session Notional hours 1 Pre-test 1 2 Orientation 4 3 Policy 1 4 Session 1: Curriculum 2013 and ELPSA 15 5 Session 2: Effective Questioning 25 6 Session 3: Rich Problems 26 7 ICT use for support: Blogging, Video Screencast 8 8 Action Planning 1 9 Reflection and evaluation 1 10 Post-test 1 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 59 iii. Evaluation Phase: Two types of evaluation were conducted, both during and at the end of the course, with these being: (i) a participant evaluation; and (ii) a course evaluation. 1. Participant Evaluation: The learning progress of each participant was assessed in terms of participants’ level of attendance (35%); assessment of assignments (50%); and post-test results (15%). About 73% of participants, or 86 individuals, passed the course. The remaining 16 did not pass for a number of reasons, as follows: (i) other competing activities affected attendance; (ii) they failed to submit assignments or the quality of these assignments was below standard; (iii) poor connectivity constrained participation. 2. Course Evaluation: A survey was conducted at the end of the course, with this survey showing that in general, participants felt comfortable participating in the online course and that the content and materials were useful and relevant to their work. A summary of the results of this survey can be seen in Table 1 below. Table 6: Summary of Online Geometry Course Survey Items Score I feel this is simple and I don’t face many problems in accessing and using the online system in general. 78.8 I feel this is simple and don’t face many problems in operating the menu/service in the online course. 78.4 I feel this is easy and don’t face many problems in using the available content. 80.2 This online course provide options of rich/varied learning resources and media. 88.8 Topic/materials selection is apt. 88 Time duration and schedule set for each materials are apt. 73.3 Discussion forum gives deeper understanding and skills. 82.4 Coaching and guidance provided by facilitator helps me with the learning process. 77.1 Help from peers help me in the learning process. 79.5 Coaching and guidance from the class administrators help me with the learning process. 78.8 In general, I am satisfied with my enrollment in the online course. 87.1 In general, this course support my work. 93.7 I am gaining knowledge and perspective through this online course. 94.4 The activities enhance my thinking, reasoning, and exercise. 95.1 I feel that this activity make me learn independently and not to depend on my facilitator. 91.7 My knowledge and skills obtained from this course can be implemented for my teaching in my classroom. 93.7 Of all the sessions, participants were most positive about Session 2 (Effective Questioning), as the following figure shows: Series 1 4.7 4.65 4.6 4.55 4.5 4.45 4.4 4.35 4.3 4.25 4.2 Sesi 1 Sesi 2 Sesi 3 Sesi Penunjang 60 The Use of Video in Teacher Professional Development Despite the largely positive feedback from participants, a number of challenges were identified by the internal team, including the following: • Time constraints: Facilitators were not given time to focus exclusively on their facilitation tasks. Rather, they were required to multitask, with other ongoing work obligations, which made it challenging for them to focus on their duties; • Technical issues: A number of technical problems affected the submission of assignments. In particular, participants with poor internet connectivity had difficulties uploading their assignments. However, this was overcome when they changed the setting for upload timeout. Discussion on the Findings Did the online course achieve its objectives? The following sections attempt to determine the extent to which the online course allowed P4TK Mathematics to: 1. Benefit from the different approach of the course. Although a more formal analysis of the benefits is need, a number of participants expressed their opinions on what they had obtained through their participation. The opinions of these participants are quoted below: Very useful and better than face to face interaction. Participants were really into independent learning. Teacher participants were able to develop their skills because the materials supported the teaching and learning process in school. Teacher knowledge has been enhanced through an improved ability to engage in effective questioning. In addition, teachers enhanced their levels of IT literacy and knowledge of the implementation of the Curriculum 2013 (Ahmad Zaini, SMPN 11 Banjarbaru, Kalsel). Absolutely amazing. In 30 years of teaching, maybe this training has been the most impressive, although it only took place in the virtual world (Fadiloes Bahar, M.Pd, SMPN 8 Tangerang, Banten). Great! An inspirational and meaningful distance training for Indonesian Math teachers! Teachers are tested through reading sessions, observation, and individual and group assignments, all of which were packed into this DOL Math 2014. (Sri Sudarini, S.Pd, SMPN 4 Yogyakarta, DIY). 2. Conduct an economical, rapid mode for the delivery of online learning. P4TK Mathematics stated that the cost of delivering the course was relatively low compared to the cost involved in face-to-face training. The cost of the production of materials, the payments to facilitators, administrators, and committee members and the printing and distribution of certificates amounted to a relatively modest Rp 1,115.000 per participant. 3. Support the implementation of Curriculum 2013. Despite the current controversy regarding the Curriculum 2013, P4TK Mathematics was successful in fulfilling their mandate to support and train teachers for the implementation of the curriculum. One of the statements from a participant quoted above expressed this explicitly (see the comment by Ahmad Zaini above). Aligned with the face-to-face version of the Geometry course, it was a successful means of implementing the ‘5 Ms’ principle, which translates as Observe, Question, Present, Analyze/Reason, and Implement/Try. 4. Increased access. The course did facilitate increased access, allowing participation from different locations across Indonesia. Based on a geographical mapping of the participants’ location, it can be seen that most participants were located in Java, with the smallest number being located in Maluku and Papua (See Figure 1). Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 61 Figure 9: Participant Mapping MALUKU, PAPUA - 1% BALI, NT- 2% SULAWESI - 9% KALIMANTAN - 6% SUMATRA - 13% JAWA - 69% 5. Enhance potential to enhance learning quality. Although there were significant increases from the pre-test to post-test score, with the average rising from 53.68 to 76, this increase did not necessarily reflect enhancements to the quality of learning per se. The activities of the course were varied to enable various learning styles, with activities including tests, reading, engagement with audiovisuals, discussion, and individual and group assignments. Participants found the interaction with peers to be the most beneficial activity (score: 88.8, see Table 1). 6. Flexibility of delivery. Can the course be described as a form of ‘anytime, anywhere’ learning? The nature of the delivery of the course may justify its description as a form of ‘anywhere’ learning. Certainly, the course facilitated interaction between facilitators and participants from a number of widely diverse locations. However, the scheduling of the course was very tight, with this scheduling being described as ‘flexibly scheduled’ or fleksibel terjadwal. This means that the learning process was almost synchronous, enabling participants with varying work schedules to take part on a day-to-day basis. However, this does not fully meet the criteria for ‘anytime’ learning. Teachers were not entirely free to learn at their own pace, with activities taking place within a scheduled timeframe. It also did not embrace the learning process utilized by teachers at KKG and MGMP, due to time and logistical constraints. The P4TK Mathematics team aims to enable individual participants take part in similar courses if they have sufficient levels of ICT literacy and showed a high level of commitment. These participants will be invited to participate on the basis of their inclusion on a list of online participants in previous online training programs. 7. Extended use in the classroom and the use of technology to gather information, to develop knowledge, to present information. Teachers were able to use the online materials in their classrooms with and for their students. They produced videos for the ‘Tugu Project’ in the classroom with their students during Session 3 (for example, see https://www. youtube.com/watch?v=Q92_NJldMhg). A few teachers did extend the use of technology by creating videos and by publishing their reflections on their blogs (for example, see http://grupdolaljabar.blogspot.com/). However, the quality of the output is still only fair. 8. Dissemination of the face-to-face course. This achievement of this goal is yet to become evident. The involvement of 112 participants was a dissemination activity of its own. However, this may be expanded during the second phase of the training, which will be conducted in January. 62 The Use of Video in Teacher Professional Development Recommendations Some recommendations can be made to improve implementation and to develop opportunities in future iterations of this project: 1. Process Automation: For the development of a training program development with a larger number of participants, certain aspects of the system should be automated. Some opportunities for automation include the following: a. The summary reporting system could be customized to facilitate the examination of task completion and the specific activity of the participants; b. An automatic reminder system could be developed to send reminders through SMS (LMS needs to be integrated with the SMS-Gateway), web and email to remind participants of their obligation to complete course assignments and other activities; c. The use of the restricted and conditional formatting of training activities need to be explored for the next iteration of the training program. Through this feature, the participants would be guided by structured instructions to complete each task and activity. 2. Online Activities to Enrich Participants’ Learning Experiences. a. Although the use of web 2.0 has already been instigated, for the upcoming training sessions, the facilitator may consider using other web 2.0 facilities to enrich the experience of participants. For example, the use of Diigo for social bookmarking or the use of the Voicethread discussion tool for multimedia discussions using audio and/or video materials could be considered. b. Self-evaluation tools to enable participants to measure their own levels of knowledge on any topic or session are necessary. Through this feature, participants will be able to determine the level of knowledge gained on any particular topic. With the availability of these results, participants can engage in activities to improve their knowledge in areas in which they are weak. In addition to self-evaluation, especially in the area of self-time management, the online classes need to display information in the form of a progress bar. For example, they could use a graphic plug-in, such as the one available at https://moodle. org/plugins/view.php?plugin=block_progress. By using this plug-in, participants would find it easier to manage their time, enabling them to complete the required assignments for the online training process. c. To store all the learning products that have been produced by the participants, the organizers need to provide an e-portfolio system in the LMS so that all artifacts produced by the participants will be well documented. This will facilitate the sharing of knowledge and the assessment process. Organizers can use the Exabis portfolio system, available at https://moodle.org/plugins/view.php?plugin=block_exaport d. The use of gamification to increase levels of participation in online classes needs to be explored in the next iteration of the online training process. For example, this could involve giving badges or stamps to participants who perform well or are particularly active in any given session or time period. 3. Quality of Facilitation. a. The facilitators need to provide feedback on assignments submitted by the participants so that participants are able make improvements. This ensured that participants go beyond merely completing the requirement to submit assignments, but actually improve the quality of their work. b. Improved standards for both the quality and quantity of discussion: The quantity of discussion can be improved by determining the number of times participants have to post in a forum discussion. For example, standards could be applied to require participants to post at least once and to provide feedback at least twice in a discussion forum. To improve the quality of discussion, the facilitator could be encouraged to ask questions that explore certain topics more deeply, with additional discussion techniques to encourage participants to put forward ideas and to respond meaningfully to discussion questions. Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia 63 4. Program Feedback/Evaluation: There is a need for a feature to enable participants to provide feedback to the program organizers on the quality of course activities, learning contents, the reliability of LMS, and the level of service of facilitators at the end of the training activities. The results of this feedback could be used to improve the training program in the future to ensure ongoing, continuous improvements. 5. Technical Support: The use of a help desk and/or support ticket system which is integrated in the LMS system will help training providers to deliver excellent service to assist participants in the completion of their tasks. Organizers can use the recommended plug-in https://moodle.org/plugins/view.php?plugin=block_helpdesk, which can be integrated with the Moodle LMS. The organizers could select issues that are addressed through the system for display on a FAQ (Frequently Asked Questions) page, which could be used as a reference to address similar questions submitted by other participants through the helpdesk and/or support ticket system. 6. Program Management: A Quality Assurance function needs to be integrated with the online training management organization structure to ensure that all activities that take place in the online training process comply with agreed upon quality standards. These activities range from the process of designing the online course, the development of multimedia contents, the recruitment of participants, the process of facilitation and the delivery of trainings, the level of reliability of service (in terms of technology) and the process of assessing participants. The Quality Assurance team would be able to conduct monitoring and evaluation activities on the basis of some of these facilities, which would also facilitate ongoing improvements to the program. 64 The Use of Video in Teacher Professional Development   Statement 1 Statement 2 Statement 3 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.020 0.033 0.029 0.044 0.041 0.063 0.199 0.208 0.22 1 on page 27.)   (0.087) (0.089) (0.090) (0.085) (0.086) (0.085) (0.097)** (0.098)** (0.095)** Matching Pre Statement 0.300 0.296 0.302 0.259 0.246 0.204 0.28 0.266 0.191 (0.051)*** (0.052)*** (0.053)*** (0.055)*** (0.055)*** (0.056)*** (0.052)*** (0.052)*** (0.054)*** Teacher years teaching 0.033 0.032 0.072 0.059 0.141 0.126 (0.047) (0.048) (0.045) (0.045) (0.052)*** (0.051)** Teacher Degree -0.096 -0.123 0.128 0.116 -0.017 -0.007 (0.065) (0.068)* (0.063)** (0.064)* (0.072) (0.073) Teacher Gender -0.071 -0.086 0.056 0.086 0.069 0.069 (0.094) (0.096) (0.091) (0.09) (0.104) (0.102) District of Gowa -0.198 -0.001 -0.232 APPENDIX 5: REGRESSION RESULTS (0.171) (0.162) (0.182) District of Blora -0.235 -0.027 -0.214 (0.148) (0.140) (0.159) District of Sumedang -0.151 0.094 0.309 Regressions for Teacher Mathematics Competency Testing (0.140) (0.134) (0.158)* District of Solok -0.222 -0.387 -0.173 (0.153) (0.143)*** (0.162) Constant 3.132 3.28 3.486 3.183 2.741 2.973 2.865 2.358 2.698 (0.234)*** (0.342)*** (0.364)*** (0.250)*** (0.336)*** (0.349)*** (0.214)*** (0.329)*** (0.346)*** Observations 268 267 267 266 265 265 264 263 263 R-squared 0.114 0.126 0.137 0.077 0.102 0.155 0.114 0.139 0.204 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.108 0.11 0.106 0.07 0.085 0.125 0.107 0.122 0.176 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (note: Beliefs statements are not stated in full; the numbered statement corresponds to the items listed in Table 65 66   Statement 4 Statement 5 Statement 6 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   1 on page 27.) Treatment Group 0.206 0.198 0.208 -0.017 0.000 -0.002 -0.063 -0.071 -0.061   (0.089)** (0.089)** (0.089)** (0.107) (0.109) (0.108) (0.113) (0.113) (0.113) Matching Pre Statement 0.285 0.264 0.252 0.477 0.474 0.44 0.26 0.259 0.248 (0.049)*** (0.049)*** (0.050)*** (0.060)*** (0.062)*** (0.065)*** (0.049)*** (0.049)*** (0.050)*** Teacher years teaching 0.123 0.11 0.024 0.014 0.18 0.178 (0.047)*** (0.048)** (0.059) (0.059) (0.059)*** (0.060)*** Teacher Degree 0.009 0.002 -0.073 -0.099 -0.003 -0.039 (0.065) (0.068) (0.080) (0.082) -0.082 (0.086) Teacher Gender 0.141 0.157 -0.022 -0.045 0.267 0.266 (0.094) (0.096) (0.115) (0.116) (0.119)** (0.121)** District of Gowa -0.237 -0.520 -0.134 (0.170) (0.206)** (0.215) District of Blora -0.078 -0.401 -0.199 (0.148) (0.179)** (0.188) District of Sumedang -0.055 -0.127 -0.184 (0.141) (0.172) (0.177) District of Solok -0.269 -0.425 -0.375 (0.151)* (0.186)** (0.192)* Constant 3.005 2.444 2.624 0.994 1.039 1.489 2.755 1.725 2.001 (0.205)*** (0.300)*** (0.328)*** (0.143)*** (0.326)*** (0.363)*** (0.181)*** (0.362)*** (0.412)*** Observations 265 264 264 267 266 266 266 265 265 R-squared 0.13 0.157 0.173 0.195 0.199 0.233 0.095 0.138 0.151 Adj. R-squared 0.123 0.140 0.143 0.189 0.184 0.206 0.089 0.121 0.121 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (note: Beliefs statements are not stated in full; the numbered statement corresponds to the items listed in Table The Use of Video in Teacher Professional Development   Statement 7 Statement 8 Statement 9 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   1 on page 27.) Treatment Group 0.074 0.053 0.063 0.22 0.216 0.213 0.289 0.283 0.293   (0.081) (0.082) (0.082) (0.089)** (0.091)** (0.091)** (0.095)*** (0.095)*** (0.096)*** Matching Pre Statement 0.213 0.212 0.196 0.272 0.263 0.258 0.253 0.225 0.221 (0.053)*** (0.053)*** (0.054)*** (0.050)*** (0.051)*** (0.053)*** (0.060)*** (0.060)*** (0.061)*** Teacher years teaching 0.057 0.051 0.062 0.047 0.15 0.131 (0.043) (0.044) (0.047) (0.048) (0.050)*** (0.051)** Teacher Degree 0.031 0.028 0.066 0.099 0.019 0.038 (0.060) (0.062) (0.066) (0.068) (0.070) (0.073) Teacher Gender 0.189 0.196 0.063 0.062 0.225 0.248 (0.086)** (0.088)** (0.094) (0.095) (0.102)** (0.105)** District of Gowa 0.003 -0.244 -0.257 (0.157) (0.170) (0.183) District of Blora -0.037 0.037 0.069 (0.136) (0.148) (0.159) District of Sumedang 0.136 0.145 0.026 (0.130) (0.142) (0.151) District of Solok -0.101 0.136 -0.102 (0.140) (0.153) (0.163) Constant 3.429 2.898 2.967 2.906 2.549 2.529 2.998 2.223 2.265 (0.235)*** (0.324)*** (0.343)*** (0.198)*** (0.306)*** (0.339)*** (0.267)*** (0.355)*** (0.379)*** Observations 267 266 266 267 266 266 267 266 266 R-squared 0.061 0.083 0.099 0.133 0.142 0.166 0.095 0.136 0.151 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.054 0.066 0.068 0.126 0.126 0.137 0.088 0.119 0.121 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (note: Beliefs statements are not stated in full; the numbered statement corresponds to the items listed in Table 67 68   Statement 10 Statement 11 Statement 12 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.037 0.079 0.071 0.000 0.005 0.011 0.125 0.109 0.113   (0.137) (0.138) (0.139) (0.102) (0.104) (0.104) (0.106) (0.108) (0.109) Table 2 on page 28.) Matching Pre Statement 0.255 0.235 0.229 0.309 0.305 0.304 0.256 0.262 0.251 (0.056)*** (0.056)*** (0.057)*** (0.059)*** (0.059)*** (0.060)*** (0.060)*** (0.060)*** (0.062)*** Teacher years teaching 0.000 0.021 0.085 0.092 0.096 0.083 (0.073) (0.074) (0.055) (0.056)* (0.057)* (0.058) Teacher Degree -0.141 -0.143 0.001 0.003 0.105 0.124 (0.101) (0.106) (0.076) (0.079) (0.079) (0.083) Teacher Gender -0.358 -0.378 0.066 0.081 0.213 0.221 (0.147)** (0.149)** (0.109) (0.112) (0.114)* (0.116)* District of Gowa 0.430 0.251 -0.113 (0.265) (0.199) (0.208) District of Blora 0.042 0.163 0.032 (0.230) (0.173) (0.180) District of Sumedang 0.184 0.007 0.174 (0.218) (0.165) (0.173) District of Solok 0.281 0.068 0.042 (0.236) (0.177) (0.185) Constant 2.159 2.972 2.794 2.524 2.142 2.006 2.892 2.067 2.061 (0.197)*** (0.444)*** (0.489)*** (0.226)*** (0.365)*** (0.395)*** (0.240)*** (0.389)*** (0.418)*** Observations 267 266 266 267 266 266 267 266 266 R-squared 0.072 0.098 0.112 0.094 0.103 0.113 0.069 0.095 0.104 Adj. R-squared 0.065 0.081 0.08 0.088 0.086 0.082 0.062 0.078 0.073 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 13 Statement 14 Statement 15 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.153 0.136 0.145 -0.024 -0.057 -0.048 0.004 -0.031 -0.016   (0.088)* (0.089)* (0.09)* (0.104) (0.104) (0.105) (0.095) (0.095) (0.094) Table 2 on page 28.) Matching Pre Statement 0.199 0.197 0.182 0.212 0.212 0.215 0.266 0.277 0.246 (0.053)*** (0.053)*** (0.055)*** (0.056)*** (0.055)*** (0.057)*** (0.052)*** (0.053)*** (0.052)*** Teacher years teaching 0.025 0.025 0.121 0.115 0.032 0.026 (0.047) (0.048) (0.055)** (0.056)** (0.051) (0.051) Teacher Degree 0.115 0.131 0.022 0.030 0.084 0.100 (0.065)* (0.069)* (0.076) -(0.081) (0.070) (0.072) Teacher Gender 0.115 0.136 0.301 0.314 0.265 0.278 (0.094) (0.096) (0.110)*** (0.113)*** (0.101)*** (0.101)*** District of Gowa 0.21 0.025 0.221 (0.170) (0.200) (0.179) District of Blora 0.183 0.068 0.111 (0.148) (0.175) (0.155) District of Sumedang 0.190 0.09 0.459 (0.141) (0.166) (0.148)*** District of Solok 0.079 -0.029 0.041 (0.152) (0.179) (0.159) Constant 3.139 2.728 2.59 3.226 2.319 2.249 2.886 2.207 2.100 (0.218)*** (0.320)*** (0.349)*** (0.230)*** (0.371)*** (0.419)*** (0.212)*** (0.338)*** (0.360)*** Observations 266 265 265 268 267 267 268 267 267 R-squared 0.057 0.073 0.083 0.052 0.091 0.094 0.091 0.118 0.166 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.049 0.055 0.050 0.045 0.074 0.062 0.084 0.101 0.137 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 69 70   Statement 16 Statement 17 Statement 18 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.123 0.115 0.097 0.067 0.060 0.071 -0.102 -0.108 -0.101   (0.105) (0.108) (0.107) (0.107) (0.110) (0.106) (0.101) (0.103) (0.103) Table 2 on page 28.) Matching Pre Statement 0.318 0.323 0.315 0.329 0.325 0.241 0.477 0.479 0.455 (0.052)*** (0.052)*** (0.052)*** (0.055)*** (0.056)*** (0.058)*** (0.052)*** (0.052)*** (0.053)*** Teacher years teaching -0.041 -0.026 0.026 0.022 0.018 0.025 (0.057) (0.057) (0.059) (0.057) (0.055) (0.055) Teacher Degree -0.023 -0.045 0.087 0.077 0.105 0.058 (0.079) (0.082) (0.081) (0.082) (0.076) (0.078) Teacher Gender -0.017 -0.066 0.050 0.045 0.061 0.050 (0.114) (0.114) (0.117) (0.114) (0.110) (0.111) District of Gowa 0.118 -0.117 0.203 (0.203) (0.204) (0.195) District of Blora -0.297 -0.278 -0.199 (0.177)* (0.176) (0.172) District of Sumedang 0.159 0.416 -0.009 (0.168) (0.171)** (0.161) District of Solok 0.131 -0.123 -0.279 (0.181) (0.179) (0.174) Constant 2.246 2.432 2.513 2.451 2.182 2.480 1.827 1.524 1.730 (0.180)*** (0.348)*** (0.382)*** (0.206)*** (0.356)*** (0.379)*** (0.172)*** (0.346)*** (0.386)*** Observations 268 267 267 265 264 264 267 266 266 R-squared 0.128 0.131 0.169 0.120 0.124 0.201 0.247 0.253 0.277 Adj. R-squared 0.121 0.114 0.140 0.114 0.107 0.173 0.241 0.239 0.251 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 19 Statement 20 Statement 21 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group -0.051 -0.049 -0.032 0.010 0.006 0.012 -0.073 -0.065 -0.059   (0.122) (0.123) (0.121) (0.108) (0.109) (0.110) (0.093) (0.095) (0.096) Table 2 on page 28.) Matching Pre Statement 0.253 0.250 0.218 0.234 0.233 0.217 0.259 0.245 0.233 (0.057)*** (0.058)*** (0.057)*** (0.054)*** (0.055)*** (0.056)*** (0.047)*** (0.049)*** (0.050)*** Teacher years teaching 0.124 0.130 0.110 0.100 0.082 0.085 (0.065)* (0.065)** (0.058)* (0.059)* (0.052) (0.053) Teacher Degree 0.152 0.137 0.033 0.052 0.011 0.029 (0.090)* (0.093) (0.081) (0.085) (0.070) (0.073) Teacher Gender 0.126 0.126 0.181 0.184 0.061 0.073 (0.131) (0.130) (0.116) (0.118) (0.101) (0.102) District of Gowa 0.510 -0.029 0.236 (0.231)** (0.212) (0.185) District of Blora 0.002 0.040 0.174 (0.202) (0.183) (0.159) District of Sumedang 0.463 0.240 0.242 (0.191)** (0.172) (0.151) District of Solok 0.002 0.030 0.176 (0.206) (0.186) (0.162) Constant 2.359 1.544 1.430 2.715 2.009 1.975 3.026 2.684 2.495 (0.185)*** (0.393)*** (0.431)*** (0.191)*** (0.361)*** (0.408)*** (0.188)*** (0.315)*** (0.347)*** Observations 268 267 267 266 265 265 267 266 266 R-squared 0.069 0.093 0.144 0.067 0.087 0.099 0.105 0.116 0.126 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.062 0.076 0.114 0.060 0.069 0.067 0.099 0.099 0.095 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 71 72   Statement 22 Statement 23 Statement 24 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.159 0.198 0.196 0.011 0.040 0.046 -0.023 -0.034 -0.016   (0.126)** (0.127)** (0.127)** (0.124) (0.124) (0.125) (0.129) (0.132) (0.132) Matching Pre Statement 0.326 0.307 0.301 0.363 0.351 0.352 0.281 0.279 0.243 (0.055)*** (0.055)*** (0.055)*** (0.052)*** (0.051)*** (0.053)*** (0.062)*** (0.063)*** (0.065)*** Teacher years teaching 0.097 0.119 0.218 0.224 0.031 0.017 (0.068) (0.069)* (0.066)*** (0.067)*** (0.070) (0.071) Teacher Degree 0.142 0.107 -0.005 0.006 0.098 0.148 Table 2 and Table 3 on pages 28 and 30.) (0.093) (0.097) (0.091) (0.095) (0.097) (0.102) Teacher Gender -0.161 -0.178 0.129 0.142 0.119 0.164 (0.134) (0.136) (0.131) (0.134) (0.138) (0.140) District of Gowa 0.508 0.336 0.331 (0.241)** (0.237) (0.249) District of Blora -0.069 0.203 0.415 (0.210) (0.209) (0.220)* District of Sumedang 0.109 0.162 0.411 (0.199) (0.198) (0.212)* District of Solok 0.052 0.176 0.140 (0.215) (0.212) (0.223) Constant 1.899 1.677 1.608 2.254 1.338 1.107 1.973 1.555 1.270 (0.164)*** (0.384)*** (0.428)*** (0.187)*** (0.387)*** (0.444)** (0.206)*** (0.420)*** (0.454)*** Observations 268 267 267 267 266 266 268 267 267 R-squared 0.120 0.143 0.166 0.158 0.195 0.202 0.074 0.081 0.101 Adj. R-squared 0.113 0.127 0.137 0.152 0.180 0.174 0.067 0.064 0.069 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 25 Statement 26 Statement 27 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group -0.009 -0.025 -0.014 -0.278 -0.318 -0.303 -0.083 -0.071 -0.073   (0.113) (0.115) (0.116) (0.112)** (0.114)*** (0.114)*** (0.096) (0.098) (0.098) Table 3 on page 30.) Matching Pre Statement 0.438 0.437 0.443 0.411 0.414 0.396 0.358 0.362 0.362 (0.057)*** (0.058)*** (0.058)*** (0.054)*** (0.056)*** (0.057)*** (0.053)*** (0.054)*** (0.054)*** Teacher years teaching -0.063 -0.068 -0.015 -0.030 0.059 0.047 (0.062) (0.063) (0.062) (0.063) (0.053) (0.054) Teacher Degree 0.038 0.068 0.106 0.145 -0.143 -0.114 (0.085) (0.090) (0.085) (0.088) (0.073)* (0.076) Teacher Gender 0.078 0.107 0.218 0.255 -0.028 -0.035 (0.122) (0.124) (0.121)* (0.122)** (0.104) (0.105) District of Gowa 0.178 -0.168 -0.190 (0.219) (0.218) (0.185) District of Blora 0.288 0.284 0.021 (0.193) (0.190) (0.163) District of Sumedang 0.148 0.000 0.100 (0.183) (0.180) (0.155) District of Solok 0.152 0.031 0.127 (0.197) (0.194) (0.166) Constant 2.045 2.086 1.823 2.449 2.013 1.959 1.046 1.069 1.049 (0.251)*** (0.411)*** (0.455)*** (0.234)*** (0.374)*** (0.416)*** (0.115)*** (0.286)*** (0.324)*** Observations 268 267 267 268 267 267 268 267 267 R-squared 0.181 0.184 0.192 0.191 0.207 0.224 0.153 0.168 0.180 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.175 0.169 0.163 0.185 0.192 0.197 0.146 0.152 0.151 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 73 74   Statement 28 Statement 29 Statement 30 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.005 -0.022 -0.020 -0.033 0.003 -0.005 0.024 0.059 0.046   (0.128) (0.129) (0.127) (0.100) (0.102) (0.102) (0.104) (0.106) (0.107) Table 3 on page 30.) Matching Pre Statement 0.273 0.276 0.257 0.389 0.394 0.400 0.340 0.351 0.350 (0.057)*** (0.058)*** (0.057)*** (0.049)*** (0.050)*** (0.050)*** (0.052)*** (0.053)*** (0.054)*** Teacher years teaching -0.023 -0.051 0.101 0.103 0.052 0.054 (0.070) (0.069) (0.054)* (0.055)* (0.056) (0.057) Teacher Degree -0.144 -0.134 -0.017 -0.023 -0.095 -0.103 (0.097) (0.099) (0.076) (0.079) (0.078) (0.082) Teacher Gender 0.276 0.237 -0.123 -0.155 -0.153 -0.187 (0.137)** (0.136)* (0.108) (0.109) (0.112) (0.114) District of Gowa -0.686 -0.054 -0.127 (0.242)*** (0.192) (0.199) District of Blora -0.346 -0.189 -0.225 (0.210) (0.169) (0.175) District of Sumedang 0.084 0.094 0.020 (0.200) (0.160) (0.167) District of Solok -0.134 0.066 0.045 (0.215) (0.173) (0.179) Constant 2.623 2.429 2.809 1.207 1.065 1.122 1.239 1.401 1.521 (0.231)*** (0.405)*** (0.444)*** (0.132)*** (0.303)*** (0.340)*** (0.135)*** (0.312)*** (0.353)*** Observations 267 266 266 268 267 267 268 267 267 R-squared 0.080 0.111 0.162 0.192 0.209 0.223 0.138 0.152 0.165 Adj. R-squared 0.073 0.093 0.133 0.186 0.194 0.196 0.132 0.135 0.136 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 31 Correlation Regression Regression 1 2   Treatment Group -0.135 -0.117 -0.109   (0.125) (0.127) (0.128) Table 3 on page 30.) Matching Pre Statement 0.334 0.334 0.342 (0.057)*** (0.058)*** (0.059)*** Teacher years teaching 0.027 0.011 (0.067) (0.069) Teacher Degree -0.014 0.030 (0.093) (0.098) Teacher Gender -0.093 -0.089 (0.133) (0.135) District of Gowa -0.040 (0.238) District of Blora 0.158 (0.210) District of Sumedang 0.280 (0.200) District of Solok 0.238 (0.214) Constant 1.487 1.562 1.367 (0.157)*** (0.380)*** (0.430)*** Observations 267 266 266 R-squared 0.131 0.131 0.142 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.124 0.114 0.112 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 75 Regressions for Teacher Mathematics Competency Testing   Correlation Regression Regression 1 2 Treatment Group 0.286 0.275 0.272   (0.105)*** (0.109)** (0.105)*** Pre-test score (Z score) 0.577 0.582 0.554 (0.053)*** (0.054)*** (0.053)*** Teacher years teaching -0.028 -0.067 (0.057) (0.054) Teacher Degree -0.028 0.044 (0.080) (0.078) Teacher Gender -0.002 0.028 (0.115) (0.111) District of Gowa -0.144 (0.194) District of Blora 0.207 (0.174) District of Sumedang 0.674 (0.157)*** District of Solok 0.237 (0.177) Constant -0.153 -0.011 -0.310 (0.075)** (0.310) (0.334) Observations 243 242 242 R-squared 0.338 0.338 0.419 Adj. R-squared 0.332 0.324 0.397 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% 76 The Use of Video in Teacher Professional Development   Statement 1 Statement 2 Statement 3 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.061 0.031 0.014 0.019 -0.080 -0.064 0.081 0.062 0.050   (0.028)** (0.031) (0.032) (0.043) (0.046)* (0.049) (0.039)** (0.042)** (0.044)** Table 7 on page 33) Matching Pre Statement 0.323 0.316 0.296 0.273 0.246 0.230 0.247 0.224 0.211 (0.016)*** (0.017)*** (0.017)*** (0.017)*** (0.018)*** (0.018)*** (0.017)*** (0.018)*** (0.018)*** Teacher years teaching 0.003 0.019 -0.092 -0.071 -0.028 -0.009 (0.015) (0.015) (0.022)*** (0.022)*** (0.020) (0.020) Teacher Degree 0.005 0.008 0.050 0.066 0.007 0.019 (0.019) (0.020) (0.028)* (0.030)** (0.025) (0.027) Teacher Gender 0.029 0.050 0.225 0.207 0.076 0.067 (0.032) (0.034) (0.049)*** (0.052)*** (0.044)* (0.046) District of Gowa 0.306 0.203 0.278 Regressions for Student Perceptions Survey (0.052)*** (0.078)*** (0.070)*** District of Blora 0.132 0.135 0.143 (0.053)** (0.080)* (0.072)** District of Sumedang 0.130 0.021 0.153 (0.050)*** (0.075) (0.067)** District of Solok 0.163 0.373 0.349 (0.053)*** (0.080)*** (0.071)*** Constant -1.090 -1.150 -1.421 -1.632 -1.762 -2.020 -1.502 -1.573 -1.853 (0.034)*** (0.087)*** (0.110)*** (0.054)*** (0.134)*** (0.168)*** (0.049)*** (0.121)*** (0.150)*** Observations 4,028 3,740 3,740 4,028 3,737 3,737 4,028 3,739 3,739 R-squared 0.093 0.087 0.096 0.059 0.064 0.073 0.052 0.047 0.055 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.092 0.086 0.094 0.059 0.063 0.071 0.051 0.046 0.053 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 77 78   Statement 4 Statement 5 Statement 6 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.036 0.042 0.010 0.144 0.055 -0.012 -0.044 -0.035 -0.018   (0.029) (0.032) (0.034) (0.041)*** (0.044)** (0.046)* (0.049) (0.053) (0.056) Table 7 on page 33) Matching Pre Statement 0.218 0.220 0.214 0.223 0.207 0.196 0.320 0.308 0.290 (0.017)*** (0.018)*** (0.018)*** (0.017)*** (0.018)*** (0.018)*** (0.016)*** (0.017)*** (0.017)*** Teacher years teaching 0.004 0.011 0.010 0.029 0.020 -0.015 (0.015) (0.015) (0.021) (0.021) (0.025) (0.026) Teacher Degree -0.023 -0.021 0.016 -0.008 -0.008 0.004 (0.019) (0.020) (0.026) (0.028) (0.032) (0.034) Teacher Gender 0.026 0.026 0.286 0.238 -0.066 0.006 (0.034) (0.035) (0.046)*** (0.048)*** (0.056) (0.058) District of Gowa 0.172 0.192 -0.272 (0.053)*** (0.073)*** (0.089)*** District of Blora 0.059 -0.121 0.049 (0.055) (0.075) (0.091) District of Sumedang 0.180 0.092 0.102 (0.052)*** (0.071) (0.086) District of Solok 0.137 0.193 -0.425 (0.055)** (0.075)*** (0.092)*** Constant -1.155 -1.187 -1.319 -1.729 -2.239 -2.232 -1.739 -1.721 -1.713 (0.035)*** (0.091)*** (0.114)*** (0.049)*** (0.128)*** (0.157)*** (0.060)*** (0.152)*** (0.186)*** Observations 4,043 3,753 3,753 4,032 3,742 3,742 4,042 3,751 3,751 R-squared 0.040 0.040 0.045 0.045 0.054 0.063 0.093 0.087 0.104 Adj. R-squared 0.039 0.039 0.043 0.045 0.053 0.061 0.092 0.086 0.102 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 7 Statement 8 Statement 9 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.042 0.025 -0.013 0.086 0.063 0.031 0.094 0.071 0.027   -0.028 -0.03 -0.032 (0.032)*** (0.034)** (0.036)* (0.028)*** (0.030)** (0.031)* Table 7 on page 33) Matching Pre Statement 0.174 0.167 0.157 0.257 0.26 0.248 0.237 0.234 0.221 (0.015)*** (0.016)*** (0.016)*** (0.015)*** (0.016)*** (0.016)*** (0.016)*** (0.017)*** (0.017)*** Teacher years teaching 0.002 0.015 0.001 0.02 0.007 0.021 -0.014 -0.015 -0.016 -0.017 -0.014 -0.014 Teacher Degree -0.014 -0.02 -0.006 -0.007 -0.019 -0.032 -0.018 -0.019 -0.021 -0.022 -0.018 (0.019)* Teacher Gender 0.072 0.047 0.041 0.017 0.099 0.071 (0.032)** -0.033 -0.036 -0.038 (0.032)*** (0.033)** District of Gowa 0.174 0.271 0.167 (0.051)*** (0.057)*** (0.050)*** District of Blora -0.005 0.061 -0.051 -0.052 -0.059 -0.052 District of Sumedang 0.122 0.147 0.088 (0.049)** (0.056)*** (0.049)* District of Solok 0.189 0.303 0.156 (0.052)*** (0.059)*** (0.051)*** Constant -1.235 -1.347 -1.426 -1.219 -1.267 -1.45 -1.175 -1.33 -1.373 (0.034)*** (0.086)*** (0.107)*** (0.037)*** (0.096)*** (0.120)*** (0.034)*** (0.087)*** (0.108)*** Observations 4,027 3,739 3,739 4,031 3,742 3,742 4,040 3,752 3,752 R-squared 0.032 0.03 0.039 0.068 0.067 0.078 0.054 0.059 0.068 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.031 0.029 0.036 0.067 0.065 0.076 0.054 0.058 0.066 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 79 80   Statement 10 Statement 11 Statement 12 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.064 0.063 0.024 0.071 0.058 0.009 0.111 0.094 0.074   (0.034)* (0.037)* (0.039)* (0.033)** (0.036)** (0.038)* (0.039)*** (0.042)** (0.044)* Table 7 on page 33) Matching Pre Statement 0.222 0.204 0.193 0.198 0.197 0.190 0.236 0.217 0.213 (0.016)*** (0.017)*** (0.017)*** (0.017)*** (0.017)*** (0.017)*** (0.017)*** (0.017)*** (0.017)*** Teacher years teaching -0.007 0.015 -0.001 0.009 0.050 0.063 (0.017) (0.018) (0.017) (0.017) (0.020)** (0.020)*** Teacher Degree -0.022 -0.044 0.007 -0.006 0.019 0.022 (0.022) (0.023)* (0.022) (0.023) (0.025) (0.027) Teacher Gender 0.059 0.044 0.018 0.025 0.134 0.160 (0.039) (0.040) (0.038) (0.040) (0.044)*** (0.047)*** District of Gowa 0.282 0.239 0.285 (0.061)*** (0.060)*** (0.070)*** District of Blora -0.048 0.012 0.134 (0.063) (0.062) (0.073)* District of Sumedang 0.014 0.171 0.170 (0.059) (0.059)*** (0.068)** District of Solok 0.103 0.046 0.113 (0.063)* (0.062) (0.072) Constant -1.406 -1.479 -1.560 -1.259 -1.287 -1.395 -1.637 -2.074 -2.313 (0.043)*** (0.106)*** (0.131)*** (0.039)*** (0.102)*** (0.127)*** (0.049)*** (0.121)*** (0.150)*** Observations 4,037 3,748 3,748 4,051 3,762 3,762 4,018 3,733 3,733 R-squared 0.046 0.042 0.055 0.035 0.035 0.043 0.050 0.048 0.053 Adj. R-squared 0.045 0.041 0.052 0.034 0.033 0.041 0.049 0.047 0.051 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 13 Statement 14 Statement 15 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.079 0.030 -0.008 0.094 0.073 0.023 0.020 0.034 0.117   (0.034)** (0.036) (0.038) (0.029)*** (0.032)** (0.033) (0.048) (0.052) (0.055)** Table 7 on page 33) Matching Pre Statement 0.249 0.259 0.246 0.190 0.189 0.169 0.259 0.253 0.244 (0.015)*** (0.016)*** (0.016)*** (0.016)*** (0.016)*** (0.016)*** (0.016)*** (0.017)*** (0.017)*** Teacher years teaching 0.000 0.019 -0.023 -0.005 -0.025 -0.054 (0.017) (0.017) (0.015) (0.015) (0.025) (0.025)** Teacher Degree 0.004 -0.002 -0.055 -0.064 0.054 0.108 (0.022) (0.023) (0.019)*** (0.020)*** (0.031)* (0.033)*** Teacher Gender 0.105 0.116 0.044 0.028 -0.042 0.027 (0.038)*** (0.040)*** (0.033) (0.035) (0.055) (0.057) District of Gowa 0.346 0.291 -0.232 (0.061)*** (0.053)*** (0.087)*** District of Blora 0.087 0.017 0.281 (0.063) (0.055) (0.090)*** District of Sumedang 0.173 0.164 0.076 (0.059)*** (0.051)*** (0.085) District of Solok 0.159 0.207 -0.071 (0.062)** (0.054)*** (0.090) Constant -1.372 -1.495 -1.729 -1.270 -1.185 -1.343 -2.767 -2.718 -2.938 (0.040)*** (0.102)*** (0.128)*** (0.035)*** (0.089)*** (0.112)*** (0.075)*** (0.156)*** (0.186)*** Observations 4,019 3,736 3,736 4,026 3,739 3,739 4,042 3,752 3,752 R-squared 0.064 0.073 0.083 0.039 0.042 0.055 0.058 0.056 0.069 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.064 0.072 0.081 0.038 0.041 0.053 0.058 0.055 0.067 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 81 82   Statement 16 Statement 17 Statement 18 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.021 -0.009 -0.049 0.050 0.026 -0.020 -0.014 -0.020 -0.093   (0.030) (0.032) (0.034) (0.030)* (0.032) (0.034) (0.031) (0.033)** (0.035)*** Table 7 on page 33) Matching Pre Statement 0.307 0.310 0.287 0.309 0.315 0.295 0.327 0.332 0.302 (0.015)*** (0.016)*** (0.016)*** (0.015)*** (0.015)*** (0.016)*** (0.015)*** (0.016)*** (0.016)*** Teacher years teaching -0.011 0.008 -0.023 -0.006 0.000 0.022 (0.015) (0.016) (0.015) (0.015) (0.016) (0.016) Teacher Degree -0.052 -0.058 -0.013 -0.026 -0.024 -0.041 (0.020)*** (0.021)*** (0.019) (0.020) (0.020) (0.021)* Teacher Gender 0.081 0.070 0.063 0.043 0.055 0.031 (0.034)** (0.036)** (0.034)* (0.035) (0.035) (0.036) District of Gowa 0.305 0.250 0.369 (0.055)*** (0.054)*** (0.056)*** District of Blora 0.055 -0.015 0.003 (0.056) (0.055) (0.057) District of Sumedang 0.151 0.118 0.223 (0.053)*** (0.052)** (0.054)*** District of Solok 0.232 0.173 0.259 (0.056)*** (0.055)*** (0.057)*** Constant -1.106 -1.113 -1.317 -1.139 -1.123 -1.237 -1.109 -1.154 -1.348 (0.036)*** (0.091)*** (0.115)*** (0.037)*** (0.090)*** (0.114)*** (0.038)*** (0.093)*** (0.117)*** Observations 4,017 3,729 3,729 4,033 3,745 3,745 4,041 3,754 3,754 R-squared 0.096 0.099 0.110 0.101 0.106 0.116 0.106 0.108 0.128 Adj. R-squared 0.095 0.097 0.108 0.101 0.105 0.114 0.105 0.107 0.125 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 19 Statement 20 Statement 21 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.066 0.034 0.032 0.030 -0.043 -0.059 -0.001 -0.032 -0.075   (0.041) (0.045) (0.047) (0.045) (0.049) (0.052) (0.031) (0.033)* (0.035)** Table 7 on page 33) Matching Pre Statement 0.324 0.310 0.309 0.236 0.224 0.225 0.249 0.259 0.236 (0.016)*** (0.017)*** (0.017)*** (0.016)*** (0.017)*** (0.017)*** (0.016)*** (0.017)*** (0.017)*** Teacher years teaching 0.080 0.084 0.020 0.019 -0.003 0.010 (0.021)*** (0.022)*** (0.023) (0.024) (0.016) (0.016) Teacher Degree -0.063 -0.054 0.033 0.044 -0.011 0.003 (0.027)** (0.029)* (0.030) (0.031) (0.020) (0.021) Teacher Gender 0.195 0.219 0.233 0.198 0.127 0.109 (0.047)*** (0.050)*** (0.052)*** (0.054)*** (0.035)*** (0.037)*** District of Gowa 0.151 -0.037 0.262 (0.075)** (0.082) (0.056)*** District of Blora 0.122 0.007 0.130 (0.077) (0.085) (0.057)** District of Sumedang 0.115 0.117 0.303 (0.073) (0.080) (0.054)*** District of Solok 0.073 0.208 0.384 (0.077) (0.084)** (0.057)*** Constant -1.450 -1.967 -2.138 -1.920 -2.421 -2.424 -1.228 -1.392 -1.662 (0.048)*** (0.127)*** (0.159)*** (0.057)*** (0.140)*** (0.174)*** (0.038)*** (0.094)*** (0.119)*** Observations 4,014 3,724 3,724 4,024 3,735 3,735 4,022 3,752 3,752 R-squared 0.094 0.098 0.099 0.051 0.052 0.056 0.057 0.064 0.079 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.094 0.097 0.097 0.051 0.051 0.054 0.057 0.063 0.077 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 83 84   Statement 22 Statement 23 Statement 24 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.054 0.036 -0.038 0.049 0.016 -0.023 0.067 0.044 0.001   (0.035) (0.038) (0.040) (0.029)* (0.032)* (0.033) (0.033)** (0.036) (0.038) Table 7 on page 33) Matching Pre Statement 0.237 0.230 0.212 0.286 0.275 0.255 0.241 0.239 0.224 (0.016)*** (0.017)*** (0.017)*** (0.016)*** (0.017)*** (0.017)*** (0.016)*** (0.016)*** (0.016)*** Teacher years teaching 0.056 0.071 0.031 0.046 -0.018 0.001 (0.018)*** (0.018)*** (0.015)** (0.015)*** (0.017) (0.017) Teacher Degree 0.007 -0.005 -0.003 -0.007 0.000 -0.001 (0.023) (0.024) (0.019) (0.020) (0.022) (0.023) Teacher Gender 0.149 0.118 0.152 0.134 0.107 0.097 (0.040)*** (0.042)*** (0.033)*** (0.035)*** (0.038)*** (0.040)** District of Gowa 0.249 0.237 0.323 (0.065)*** (0.054)*** (0.060)*** District of Blora -0.025 0.031 0.081 (0.066) (0.055) (0.062) District of Sumedang 0.250 0.162 0.208 (0.062)*** (0.052)*** (0.058)*** District of Solok 0.230 0.223 0.273 (0.066)*** (0.054)*** (0.062)*** Constant -1.420 -1.875 -1.986 -1.155 -1.510 -1.665 -1.354 -1.466 -1.694 (0.043)*** (0.109)*** (0.136)*** (0.037)*** (0.090)*** (0.113)*** (0.041)*** (0.102)*** (0.128)*** Observations 4,014 3,743 3,743 4,006 3,735 3,735 4,010 3,739 3,739 R-squared 0.050 0.052 0.064 0.074 0.076 0.086 0.058 0.061 0.072 Adj. R-squared 0.049 0.051 0.062 0.074 0.075 0.083 0.058 0.060 0.070 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in The Use of Video in Teacher Professional Development   Statement 25 Statement 26 Statement 27 Correlation Regression Regression Correlation Regression Regression Correlation Regression Regression 1 2 1 2 1 2   Treatment Group 0.048 0.018 -0.033 0.013 -0.014 -0.068 0.006 -0.023 -0.070   (0.026)* (0.028) (0.029) (0.027) (0.029) (0.031)** (0.024) (0.026) (0.028)** Table 7 on page 33) Matching Pre Statement 0.226 0.218 0.198 0.231 0.231 0.217 0.200 0.196 0.181 (0.014)*** (0.014)*** (0.015)*** (0.016)*** (0.016)*** (0.016)*** (0.015)*** (0.016)*** (0.016)*** Teacher years teaching 0.013 0.029 0.029 0.042 0.033 0.047 (0.013) (0.013)** (0.014)** (0.014)*** (0.013)*** (0.013)*** Teacher Degree -0.024 -0.037 -0.017 -0.029 -0.018 -0.025 (0.017) (0.018)** (0.018) (0.019) (0.016) (0.017) Teacher Gender 0.125 0.095 0.136 0.119 0.149 0.126 (0.029)*** (0.030)*** (0.031)*** (0.032)*** (0.028)*** (0.029)*** District of Gowa 0.215 0.226 0.201 (0.046)*** (0.049)*** (0.044)*** District of Blora -0.040 -0.014 -0.005 (0.047) (0.050) (0.045) District of Sumedang 0.120 0.169 0.159 (0.045)*** (0.047)*** (0.043)*** District of Solok 0.197 0.154 0.200 (0.047)*** (0.050)*** (0.045)*** Constant -1.119 -1.332 -1.412 -1.111 -1.395 -1.490 -1.067 -1.399 -1.499 (0.031)*** (0.079)*** (0.098)*** (0.034)*** (0.083)*** (0.104)*** (0.030)*** (0.076)*** (0.094)*** Observations 4,027 3,756 3,756 4,025 3,754 3,754 4,024 3,754 3,754 R-squared 0.065 0.067 0.082 0.050 0.058 0.070 0.043 0.052 0.065 Design, Implementation and Impact Evaluation of an Innovative In-service Course for Mathematics Teachers in Indonesia Adj. R-squared 0.064 0.065 0.080 0.049 0.057 0.068 0.042 0.051 0.063 Standard errors in parentheses * significant at 10%; ** significant at 5%; *** significant at 1% (Note: Beliefs statements are not stated in full here; the numbered statement corresponds to the items listed in 85